Skip to content
Python

Release tree: demo, inference, training (renamed, torchrun) and dataset pipeline only.

mohamdevb5b2f58c7774
diff --git a/.gitignore b/.gitignore
index f305feb..d2cf080 100644
--- a/.gitignore
+++ b/.gitignore
@@ -49,7 +49,7 @@ runs/
# Notebook checkpoints
.ipynb_checkpoints/
-competitors/
+/competitors/
*.png
# Saved teacher runs for `demo.py --show-reference` (renders + packed .npz).
diff --git a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md b/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md
deleted file mode 100644
index dd691d3..0000000
--- a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md
+++ /dev/null
@@ -1,15 +0,0 @@
-# 01: Archive the AIC and MPII original frames to $STORE
-
-**What to build:** every built split has a tar in `$STORE`, so the `$SCRATCH` 30-day purge (AIC and MPII originals last read 1–2 Sept 2026, due around 1–2 Oct) costs a restore instead of a rebuild. The archive template's placeholder account has been fixed to `vsi@v100`; job 189727 was submitted with `REPO=$PWD` on the free `archive` partition.
-
-**Blocked by:** None (can start immediately; already queued as job 189727).
-
-**Status:** done (2026-09-26)
-
-- [x] `hmr_archive_<jobid>.out` ends with an `ls -lh` listing five tars: aic, coco, mpii, sa1b, harmony4d
-- [x] The aic and mpii tars hold real frames (hard links/copies), not links: `tar tvf` shows regular files under `original_images/`
-- [x] If the job timed out on sa1b, the partial sa1b tar was deleted before resubmitting (the script skips any existing tar, including a half-written one)
-
-## Answer
-
-Job 189727 wrote five split tars, but `40_archive.slurm`'s unanchored `--exclude=images` also dropped every nested `images/` folder -- i.e. all AIC and MPII original photos (0 in either tar; mpii.tar 228 MB). Fixed in the template (exclude only a top-level `images` symlink, anchored; tested on a fake tree) and rsynced. The photos were archived separately by job 226766: `sam3d_gt_aic_originals.tar` 13 GB, `sam3d_gt_mpii_originals.tar` 2.6 GB. `sam3d_gt_harmony4d.tar` verified to hold all 496,036 crops. STORE total 155 GB.
diff --git a/.scratch/v3-framing-robustness/issues/02-lr-schedule-sweep.md b/.scratch/v3-framing-robustness/issues/02-lr-schedule-sweep.md
deleted file mode 100644
index a6545f7..0000000
--- a/.scratch/v3-framing-robustness/issues/02-lr-schedule-sweep.md
+++ /dev/null
@@ -1,12 +0,0 @@
-# 02: Learning-rate x schedule-length sweep on the reference model
-
-**What to build:** find out whether the underfitting reference model (train loss 2.340 above val 2.302 at epoch 299; 3DPW val still falling as the LR reached zero) gains from a better recipe. 2x2 grid on g8h, current corpus, baseline preset, 1 seed per cell: peak LR {6e-4, 1.2e-3} x length {300, 600 epochs}. The (6e-4, 300) cell is the existing g8h_s0, so three new runs. Launched through the usual Slurm launcher with the generation-8 flags plus the LR/epoch overrides.
-
-**Blocked by:** None (can start immediately).
-
-**Status:** ready-for-human
-
-- [ ] Launch commands printed in the launcher's form (`RUN_NAME=... SEED=... EPOCHS=... MIX=... EXTRA_TRAIN_ARGS=... sbatch`), with enough `afterany` links for 600 epochs (~8 x 20 h)
-- [ ] Verified that the LR the trainer actually uses is the intended peak after its sqrt(world size) scaling, and that the one-cycle schedule spans the full 600 epochs (read from `history.jsonl` `lr` at epoch ~60 and at the end)
-- [ ] Each finished run scored on the published protocol (EMDB-1 with `--bbox annotated`, SMPL-24 adapter) and on 3DPW test J14 with the adapter
-- [ ] A 2x2 table recorded in the status doc, naming the winning recipe or stating that no cell beats g8h_s0 by more than ~0.5 mm
diff --git a/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md b/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md
deleted file mode 100644
index 90d24be..0000000
--- a/.scratch/v3-framing-robustness/issues/03-profile-cpu-tier-backbones.md
+++ /dev/null
@@ -1,28 +0,0 @@
-# 03: Profile candidate CPU-tier backbones on the reference phones
-
-**What to build:** pick the CPU-tier model-family backbone from measured latency, before any training. Random-weight exports of the current architecture (continuous head, exact landmarks, bounded scales, CLIFF focal) with HGNetV2-B0, B1 and B4 (anchor) at 224 px, profiled on AI Hub.
-
-**Blocked by:** None (can start immediately).
-
-**Status:** done (2026-09-25), uncommitted
-
-- [x] The B4 random-weight export's latency matches the trained g8h's within noise on the S23 CPU (63 ms) — the check that random weights are a valid latency proxy
-- [x] CPU-only latency and peak memory for B0, B1 and B4 on Galaxy S23, A73, A53, A14 and Pixel 8; S23 NPU latency and placement for each
-- [~] Every export places 100% of layers on the NPU on the S23 (yes, all four); fp16 parity NOT measured: it is meaningless on random weights, so it moves to ticket 11 on the trained members
-- [x] A table recorded in the status doc and a recommended CPU-tier size against the 33 ms budget on mid-range phones (A73 / Pixel 8)
-
-## Answer
-
-Exports: g8h's exact architecture and config (continuous head, bounded scales, CLIFF focal), backbone swapped, network only (no FK/mesh), 224 px, opset 17, onnxsim. AI Hub, ORT runtime compiled for the S23. Raw rows: `benchmark/results/family_latency_aihub.json`; scripts: `tools/profiling/`.
-
-| model | params (backbone) | S23 NPU | S23 CPU | A73 CPU | Pixel 8 CPU | A53 CPU | A14 CPU |
-|---|---|---|---|---|---|---|---|
-| g8h trained (B4) | 32.2 M (13.6 M) | 2.44 ms | 61.7 | 84.9 | 83.2 | 187.9 | 195.2 |
-| random B4 | 32.2 M (13.6 M) | 2.47 | 60.0 | 81.7 | 78.7 | 194.2 | 199.4 |
-| random B1 | 20.3 M (2.2 M) | 1.88 | 26.9 | 36.9 | 33.2 | 77.6 | 76.8 |
-| random B0 | 20.0 M (1.9 M) | 1.60 | 23.9 | 32.7 | 29.9 | 79.7 | 68.6 |
-
-- Random weights are a valid latency proxy: random B4 is within 2-6% of the trained g8h on every phone.
-- All four place 100% of layers on the S23 NPU.
-- **Recommendation: HGNetV2-B0 is the CPU-tier member** (under 33 ms on mid-range: A73 32.7, Pixel 8 29.9). B1 is 3.3 ms slower and just over budget on the A73 (36.9); worth training as the middle point of the accuracy-vs-latency curve.
-- **Budget phones (A53, A14) stay at 70-80 ms with either backbone.** B0 and B1 differ by <2 ms there: below B4 the backbone is no longer the cost. ~18 M of the 20 M parameters are the decoder and heads, unchanged by the backbone swap. Reaching 30 FPS on budget phones needs a slimmer decoder (width / depth), which is a new work item, not a backbone choice.
diff --git a/.scratch/v3-framing-robustness/issues/04-context-crop-builder.md b/.scratch/v3-framing-robustness/issues/04-context-crop-builder.md
deleted file mode 100644
index 316cc13..0000000
--- a/.scratch/v3-framing-robustness/issues/04-context-crop-builder.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# 04: Context-crop builder
-
-**What to build:** the crop builder can store a context crop, a square covering a chosen multiple of the person's tight box (default 1.2, today's) at a chosen size. It records the stored square in original-frame coordinates next to the existing 1.2x square, so any window can be recomputed exactly. The rebuild setting is context 2.0 at 448 px.
-
-**Blocked by:** None (can start immediately).
-
-**Status:** done (2026-09-25), uncommitted
-
-- [x] With default flags the builder's output (crops and npz) is byte-identical to the current builder's on a real local split
-- [x] With context 2.0 / 448, the 1.2x window cut from the context crop and resized to 224 matches today's 224 crop of the same person (PSNR reported, mean and min)
-- [x] The recorded geometry maps the annotation targets' 2D keypoints onto the context crop to 0.000 px
-- [x] Where the context square leaves the frame, the existing frame-edge padding applies and is flagged as such in the npz
-- [x] Works through the array build job and the streaming script unchanged apart from the new flags
-
-## Answer
-
-`--context C` on the builder (default 1.2 = unchanged); context builds record `bbox_context` and `context_padded`, need an explicit `--output-dir`, and refuse to mix with a folder of the other kind. `tools/verify_context_crops.py` on 200 real AIC rows: default byte-identical to HEAD; geometry within 0.5 px of the exact square (int() rounding); 1.2x window vs today's crop 42.2 dB mean / 33.4 min after a sigma-1 blur, 0.094 px max shift; keypoints unchanged, none lost; context_padded 0 wrong.
-
-Notes for later tickets:
-- At 2.0x, 195/200 AIC context squares reach past the frame edge, so most context crops contain some black frame-edge padding. Ticket 07 must treat it as real frame edge (deployment crops pad the same way), and may want to report how often v3 windows touch it.
-- 42/200 rows have visible keypoints outside today's 1.2x crop: the annotation box does not always enclose all keypoints. Pre-existing.
-- Today's 224 crops of people larger than ~400 px are aliased (INTER_LINEAR downscale by 3-4x). Context crops alias less. Default left unchanged.
-- Tickets 08/09 must pass `--output-dir` for every context split (the builder now refuses otherwise).
diff --git a/.scratch/v3-framing-robustness/issues/05-input-size-setting.md b/.scratch/v3-framing-robustness/issues/05-input-size-setting.md
deleted file mode 100644
index 845d46c..0000000
--- a/.scratch/v3-framing-robustness/issues/05-input-size-setting.md
+++ /dev/null
@@ -1,25 +0,0 @@
-# 05: Input size as a checkpoint setting (prefactor)
-
-**What to build:** one input-size setting (default 224) drives every place that builds the network input: the training dataset, 3DPW validation, the benchmark harness and the inference package. It is saved in the run config, recovered by config-from-checkpoint (a checkpoint without it means 224), and stamped into the ONNX metadata beside the CLIFF form, so the phone app can read its crop size from the model file.
-
-**Blocked by:** None (can start immediately).
-
-**Status:** done (2026-09-25), uncommitted
-
-- [x] baseline and v2 augmented samples are bit-identical against `git show HEAD`, compared on alternating calls
-- [x] g8h_s0 evaluated through the harness gives the same 3DPW test J14 and EMDB-1 numbers as recorded, to the decimal
-- [x] A random-weight model configured for 288 runs end to end through the 3DPW and EMDB harnesses, and its ONNX metadata reads 288
-- [x] `ddp_smoke` passes at 224 and at 288
-
-## Answer
-
-`--image-size` on the DDP trainer (multiple of 32), recorded in run_config.json. `config_from_checkpoint` reads the size from the weights (`mem_pos_embed` grid, 7x7 = 224, 9x9 = 288) and refuses a run_config.json that contradicts it. The dataset (both trainers and the self/overfit tests), `val3dpw`, the 3DPW / EMDB / SMPL-fit harnesses, `eval_all.py` grouping, `pth_to_onnx.py` metadata (`image_size`), `instanthmr.inference` and `ddp_smoke.py` all follow it.
-
-Evidence: `tools/verify_input_size.py` PASS (baseline and v2 0/24 samples differ from the pre-change trainer; labels and CLIFF identical at 224 and 288; 288 checkpoint round-trips through config, export metadata, inference on a real frame and both harnesses). g8h_s0 full regression identical to 4 decimals (3DPW test J14+adapter PA 40.7983, EMDB SMPL24+adapter 51.9492). `ddp_smoke` passes at 224 and 288; `--self-test` passes.
-
-Open, outside this repo (story 21 is NOT satisfied end to end yet):
-- neurocomp-app's own exporter (`tools/modelexport/pose.py`) hard-codes `INPUT_SHAPES` 224 and its metadata stamp clears all props and never writes `image_size`.
-- the Android engine hard-codes `POSE_INPUT_SIZE = 224` in `Preprocess.kt`.
-Both must read `image_size` before a non-224 model can ship. File this in the phone-app tracker.
-
-Note for ticket 10: `RandomPixelate` uses absolute resolutions (32-112 px), so at 288 it degrades relatively harder than at 224, which slightly confounds a 224-vs-288 comparison.
diff --git a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
deleted file mode 100644
index a4e2c57..0000000
--- a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md
+++ /dev/null
@@ -1,37 +0,0 @@
-# 06: Profile a 288 px input on the reference phones
-
-**What to build:** decide whether the 288 px arm is deployable before GPU hours are spent on it. A random-weight B4 export at 288, profiled like ticket 03.
-
-**Blocked by:** 05
-
-**Status:** done (2026-09-25), uncommitted
-
-- [x] NPU latency on Galaxy S22, S23, S24 and S25; CPU latency on S23, A73, A53, A14 and Pixel 8
-- [~] 100% NPU placement on all four Snapdragon phones (yes); fp16 parity deferred to trained weights (ticket 11), as in ticket 03
-- [x] Result recorded next to ticket 03's table, with the 224-vs-288 latency ratio per phone
-
-## Answer
-
-Random-weight exports at 288 px (`tools/profiling/export_family.py --image-size 288`), same method as ticket 03. Raw rows: `benchmark/results/family_latency_aihub_288.json`.
-
-| model @ 288 | S22 NPU | S23 NPU | S24 NPU | S25 NPU | S23 CPU | A73 CPU | Pixel 8 CPU | A53 CPU | A14 CPU |
-|---|---|---|---|---|---|---|---|---|---|
-| B4 | 5.5 | 3.27 | 2.31 | 1.88 | 93.3 | 119.0 | 115.0 | 288.7 | 294.2 |
-| B0 | 3.7 | 2.05 | 1.31 | 1.00 | 29.3 | 38.2 | 32.1 | 93.5 | 79.0 |
-
-Cost of 288 over 224 (same backbone): B4 x1.32 on the S23 NPU, x1.46-1.55 on CPUs; B0 x1.28 NPU, x1.07-1.23 CPU. All exports 100% on the NPU.
-
-- **The 288 arm is deployable on the NPU tier:** B4 at 288 is 1.9-5.5 ms across Snapdragon 8 Gen 1-Elite, far inside 33 ms. Ticket 10 may train it.
-- **Not for the CPU tier with B4** (93-294 ms). B0 at 288 stays near budget on mid-range CPUs (Pixel 8 32.1, A73 38.2) because below B4 the decoder, not the image, dominates cost -- a 288 CPU-tier member is plausible if 288 proves worth it.
-
-## Addendum (2026-10-04): 384 and 448 px
-
-Same method; 16 of 24 profiles returned (B4@448 and two B0@448 phones missing: run timed out). Raw: `benchmark/results/family_latency_aihub_384_448.json`.
-
-| model | input | S23 NPU | S23 CPU | Pixel 8 CPU | A73 CPU | A53 CPU | A14 CPU |
-|---|---|---|---|---|---|---|---|
-| B4 | 384 | 4.18 | 146 | 158 | 186 | 445 | 454 |
-| B0 | 384 | 2.31 | 37 | 46 | 50 | 107 | 105 |
-| B0 | 448 | 2.81 | 42 | - | 59 | 130 | - |
-
-NPU tier affords 384+ (B4 4.2 ms, 100% NPU). CPU tier is capped at 224-288 (B0 over 33 ms on mid-range from 384). Above ~270 px the current context corpus (2.0x box at 448 px) has no extra detail: 384 needs a ~640 px rebuild. Fast SAM 3D Body (Table 4): PA-MPJPE 33.5 @256 -> 31.0 @384 -> 31.0 @448, i.e. saturation at 384.
diff --git a/.scratch/v3-framing-robustness/issues/07-v3-preset.md b/.scratch/v3-framing-robustness/issues/07-v3-preset.md
deleted file mode 100644
index 72da005..0000000
--- a/.scratch/v3-framing-robustness/issues/07-v3-preset.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# 07: v3 preset, proven on local data
-
-**What to build:** `--preset v3` trains from context crops. Per sample it draws the effective person box log-uniformly between 0.6x and 2.0x the tight box, shifts its centre by up to +/-15% of the box, clamps the window inside the stored context, and maps the context crop to the network input in one resampling step (with v2's rotation and flip). The CLIFF vector (perspective-correct form) describes the sampled box. The bounds are config values with these defaults.
-
-**Blocked by:** 04, 05
-
-**Status:** done (2026-09-25), uncommitted
-
-- [x] A small local split rebuilt with ticket 04's builder (context 2.0 / 448) trains under v3 at 224 and at 288
-- [x] Verification on real samples: 2D labels land on the right pixels; CLIFF crop-centre error at the rounding floor (mean / p95 / max px, as in the crop-centre-fix check); no black pixel inside the stored context; box-scale draws cover 0.6–2.0x; output size equals the configured input size
-- [x] baseline and v2 samples bit-identical against HEAD (alternating calls)
-- [x] The v3 dataset refuses a split without context geometry, with a message naming the split
-- [x] `ddp_smoke` passes for v3 at 224 and at 288
-- [x] Files that must be rsynced to the cluster listed in the status doc
-
-## Answer
-
-Implemented as specified, with two decisions taken with the user (2026-09-25):
-- **Clamp (Q22, option b):** keep the rotated window inside the stored square only on sides where the square ends inside the photo. Tail restored: P(>1.6x) 17.0% (unclamped 18.5%, all-sides clamp 10.5%), P(>1.8x) 6.6% (8.8% / 1.9%), loose windows keep their shift.
-- **geom_p (Q23):** 80/20 as v2; the 20% identity windows keep root-rotation supervision. Revisit only if v3 helps.
-
-Evidence: `tools/verify_v3_preset.py` PASS (see README section `--preset v3`). `ddp_smoke --context-crops` passes at 224 and 288. A 1-epoch `train_distill_jz.py --preset v3 --image-size 288` run on the local context split completed and recorded preset / image_size / v3 fields in run_config.json. Rsync list and cluster checks: STATUS.md, "Code ready, not launched".
-
-Findings:
-- 95% of 2.0x context squares (AIC, COCO) run past the frame; windows >1.5x are ~38% black on average (frame edge, representative of deployment). Harmony4D / SA-1B may differ; measure after the rebuild.
-- Naming: `--preset v3` is unrelated to the old `v3_s{0,1,2}` runs (v2 + rebalance). Noted in STATUS.
-- `apply_v3_preset` forces `cliff_focal`.
diff --git a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md b/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md
deleted file mode 100644
index 5c3a128..0000000
--- a/.scratch/v3-framing-robustness/issues/08-rebuild-four-splits.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# 08: Rebuild coco, sa1b, aic and mpii as context crops on Jean Zay
-
-**What to build:** the four splits whose original frames are on the cluster, rebuilt with the array build job at context 2.0 / 448 into new directories that the baseline and v2 presets never pick up, then verified and archived.
-
-**Blocked by:** 01, 04
-
-**Status:** done (2026-10-04); archive to STORE submitted
-
-- [x] Launch commands printed in the build job's form, with the new flags and output directories
-- [x] Crop counts match the current splits minus images that are missing at source (report the difference)
-- [x] Verification passes per split (reprojection RMS ~0.000 px, zero missing or orphan)
-- [x] Each split archived as one tar to `$STORE`
-- [x] `$SCRATCH` usage recorded (expected ~250 GB total with ticket 09)
-
-## Answer
-
-Context crops (2.0x box, 448 px) in `$SCRATCH/instanthmr/data_ctx/sam3d_gt_{coco,mpii,aic,sa1b}`: 39,665 / 7,135 / 287,391 / 3,401,720 -- every split equal to the 224 corpus.
-
-What it took:
-- `prepost` had no usable node for a week (pp1/pp4 draining), so builds moved to `21_build_gpu.slurm` (gpu_p13, qos_gpu-dev, 2 h, 10 processes, PART/PARTS to split one split across parallel jobs).
-- MPII photos exist only under `$DATA_ROOT/sam3d_gt_mpii/original_images`, not `$IMG_ROOT/mpii`.
-- SA-1B jobs died OUT_OF_MEMORY at ~40 min: `build_split.py` cached every decoded frame of a parquet shard. Bounded to 8 entries (output byte-identical, `verify_context_crops.py` PASS); the next round completed in 36 min.
-- Every kill was followed by a truncated-npz check on files written in the 3 min before it; none found.
diff --git a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
deleted file mode 100644
index 1b2019b..0000000
--- a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# 09: Rebuild Harmony4D as context crops
-
-**What to build:** Harmony4D rejoins the corpus as context crops. Its original frames exist nowhere, so its zips are streamed scene by scene: download, undistort, build context crops, delete. The route depends on the `prepost` download test: HTTP 206 means the streaming job runs on Jean Zay (keeping no 4K originals); otherwise the same script runs on the laptop and the crops are rsynced as one tar.
-
-**Blocked by:** None (04 done; route decided)
-
-**Status:** done (2026-10-05)
-
-- [x] Route chosen from the `prepost` result and recorded
-- [x] Undistortion blockers in the undistortion notes handled before the first full scene
-- [x] 496,036 train crops (± source-missing), verified and archived to `$STORE`
-
-## Route decided (2026-09-27): laptop
-
-`prepost` job 226661 returned `HTTP 302, 0 bytes`: Hugging Face's redirect to its file host `us.aws.cdn.hf.co` cannot be followed from Jean Zay compute nodes (the login node could, 2026-09-25; same failure as STATUS 7.1 on 2026-09-03). So stream on the laptop, scene by scene (65 GB free: download one zip, undistort, build context crops with `--context 2.0 --body-size 448 --output-dir`, delete), then rsync one tar of crops to Jean Zay's context data root.
-
-## Route changed (user, 2026-10-04): Jean Zay
-
-Laptop built scenes 01-02 (65,068 crops, verified vs June: labels identical, 38.5 dB, 0.08 px shift) then was stopped to free the machine. Remaining 13 scenes: login-node download + `datasets_pipeline/jeanzay/22_h4d_scene.slurm` per scene, into `data_ctx_h4d/` (kept out of `data_ctx` while generation 9 trains). Fixes found on the way: zips unpack without the `train/` prefix (stream_archives.py now moves them); pycolmap >= 3 renamed `Image.projection_matrix()` (vendored `lib/datasets/exo_camera.py` patched).
-
-## Answer
-
-496,036 context crops in `$SCRATCH/instanthmr/data_ctx_h4d/sam3d_gt_harmony4d`, every scene equal to the June count, 0 undistortion exceptions. Full corpus served through `data_all/` (symlinks); see STATUS "Full context corpus". Archive: `sam3d_gt_harmony4d_ctx.tar`.
diff --git a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md b/.scratch/v3-framing-robustness/issues/10-v3-ablation.md
deleted file mode 100644
index 24b34db..0000000
--- a/.scratch/v3-framing-robustness/issues/10-v3-ablation.md
+++ /dev/null
@@ -1,27 +0,0 @@
-# 10: The v3 ablation
-
-**What to build:** measure what the v3 preset and a larger input buy. Three arms, 2 seeds each, on the rebuilt corpus's data mix: a g8h control (baseline preset on the same mix), v3 at 224 and v3 at 288. Current generation-8 recipe unless ticket 02 has already produced a winner.
-
-**Blocked by:** 06, 07, 08 (09 descoped: Harmony4D added afterwards as its own measurement)
-
-**Status:** launching (2026-10-04) as generation 9
-
-- [ ] Launch commands printed in the launcher's form, with the rsync list from ticket 07 and a cluster-side `grep -c` check that the new trainer is in place
-- [ ] Every checkpoint scored on EMDB-1 published protocol, on EMDB under 1.2x / 1.4x / 1.6x enlarged boxes, and on 3DPW test J14 with the adapter
-- [ ] A table of accuracy against latency (from tickets 03 and 06), stating whether v3 reduces the EMDB published-protocol penalty by more than ~0.5 mm without losing on 3DPW
-- [ ] Results recorded in the benchmark results and status docs
-
-## Scope change (user, 2026-10-04)
-
-- No control runs; reference is g8h_s0/s1. g9 vs g8 changes crops AND mix (no Harmony4D), so it is not a single-variable comparison.
-- Arms: v3 @ 224 and v3 @ 288, each on HGNetV2-B4 and B0 (folds ticket 11's CPU-tier member in). Seed 0 only. **300 epochs** (user choice, account deadline), 4 chained 20 h links per run.
-- Without Harmony4D now; Harmony4D (ticket 09, building on the laptop) is added later and its effect measured on its own.
-- Launch recipe and prerequisites: STATUS.md, "Generation 9".
-
-## Restart (2026-10-05)
-
-First launch (data_ctx only, no Harmony4D) crashed at epoch 12 on an empty npz. Restarted from scratch on `data_all` (all five context splits incl. Harmony4D) with g8h's mix MIX7, so g9 vs g8h is now a cleaner comparison (crops + preset + backbone/input size). Old runs kept as `_ctxonly_ep12_*`. No separate Harmony4D-delta generation.
-
-## Final scope (user, 2026-10-12)
-
-14 runs: HGNetV2 B0..B6 x input {224, 288}, 600 epochs, v3, `data_all` (incl. Harmony4D), MIX7, seed 0. Folds ticket 11's model family into this generation. Dataset loader now skips unreadable samples (logged) instead of killing the run. Launch block and costs: STATUS.md "Generation 9, final form".
diff --git a/.scratch/v3-framing-robustness/issues/11-train-model-family.md b/.scratch/v3-framing-robustness/issues/11-train-model-family.md
deleted file mode 100644
index 4d1b394..0000000
--- a/.scratch/v3-framing-robustness/issues/11-train-model-family.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# 11: Train the model family
-
-**What to build:** the paper's accuracy-vs-latency curve. The CPU-tier backbone(s) chosen in ticket 03, trained with the winning v3 setting from ticket 10 and the winning recipe from ticket 02, beside the NPU-tier (B4) member.
-
-**Blocked by:** 02, 03, 10
-
-**Status:** ready-for-human
-
-- [ ] Each member scored on EMDB-1 published protocol and 3DPW test J14, 2 seeds
-- [ ] Each member profiled on the reference phones: latency, NPU placement, fp16 parity
-- [ ] The accuracy-vs-latency table recorded, with each member's deployment tier
diff --git a/.scratch/v3-framing-robustness/issues/12-build-egohumans.md b/.scratch/v3-framing-robustness/issues/12-build-egohumans.md
deleted file mode 100644
index 00a32ee..0000000
--- a/.scratch/v3-framing-robustness/issues/12-build-egohumans.md
+++ /dev/null
@@ -1,23 +0,0 @@
-# 12: Build EgoHumans as context crops on Jean Zay
-
-**What to build:** EgoHumans (`egohumans_train`) joins the corpus as context crops (2.0x box, 448 px), the way Harmony4D does in ticket 09: download on a Jean Zay LOGIN node, undistort and build on compute nodes. It is ~272k annotated persons over ~268k 4K frames from 89 takes in 7 multi-person sports sequences (tagging, lego, fencing, basketball, volleyball, badminton, tennis): close interactions and wide framings the current corpus is thin on.
-
-Known facts (STATUS.md 6-8, `datasets_pipeline/registry.py`, `undistort.md`):
-- **Annotations are missing everywhere.** The `egohumans_train` parquets failed to download on Jean Zay (Xet proxy error, 2026-09-03) and are not on the laptop either. Hugging Face file downloads work from a login node (Harmony4D, 2026-09-25), so fetch them there first.
-- **Images:** Google Drive folder `1JD963urzuzV_R_6FOVOtlx8UupwUuknR`, one `.tar.gz` per take under `<sequence>/<take>.tar.gz`. gdown file downloads ran at ~20 MB/s from Jean Zay, but folder *listing* got rate-limited (403) after repeated requests: list each of the 7 sequence folders separately, ~15 min apart if refused.
-- **Only the referenced takes are needed** (~107 GB of the advertised 550 GB). Shopping list: `datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/egohumans_train --list-groups 2`, intersected with the Drive listing.
-- **Unpack with `tar xzf <take>.tar.gz --strip-components=8`. Not optional:** the tarballs carry an 8-level absolute prefix. Only `exo/` cameras are referenced.
-- **Undistort** with `sam-3d-body/data/scripts/egohumans/undistort_images.py`. It shares the vendored `lib/` with Harmony4D, so the pycolmap >= 3 patch (ticket 09) applies; check it in one take first.
-- `stream_archives.py` does not support `gdrive`; write a per-take job like `22_h4d_scene.slurm` instead.
-- Keep the output out of `data_ctx` while a generation trains from it (pair-index fingerprint, see ticket 09 / STATUS).
-
-**Blocked by:** 09 (proves the login-download / compute-undistort route and the patched `lib/` on Jean Zay).
-
-**Status:** ready-for-agent (job script + one verified take), then ready-for-human (full download and launch)
-
-- [ ] `egohumans_train` parquets fetched on a login node; row count recorded
-- [ ] Shopping list of referenced takes built; download size recorded
-- [ ] One take downloaded, unpacked, undistorted and built; its crops checked (labels reproject to 0.000 px; a few crops visually on the person; no undistortion exceptions)
-- [ ] Per-take Jean Zay job written and the remaining takes built
-- [ ] Final crop count within a few hundred of the annotation row count (minus missing frames), recorded in STATUS
-- [ ] Archived to `$STORE` (`sam3d_gt_egohumans_ctx.tar`)
diff --git a/.scratch/v3-framing-robustness/spec.md b/.scratch/v3-framing-robustness/spec.md
deleted file mode 100644
index 7a9b0db..0000000
--- a/.scratch/v3-framing-robustness/spec.md
+++ /dev/null
@@ -1,108 +0,0 @@
-# Spec: framing-robust, resolution-flexible training (v3 preset) and a deployable model family
-
-Status: ready-for-agent
-
-## Problem Statement
-
-The reference model (g8h) scores 51.9 mm SMPL-24 PA-MPJPE on EMDB-1 when it is given tight boxes built from the ground-truth joints, which is the framing every training crop has. Under the published protocol, which uses EMDB's own looser and off-centre boxes, it scores 64.3 / 66.9 mm (seeds 0 / 1): a penalty of 12–15 mm that puts it level with SMPLer-X-H and about 21 mm behind CameraHMR. The penalty has grown since generation 5 (+7.0 mm) and generation 6 (+8.3 mm), and it differs by 2.6 mm between two seeds that agree to 0.3 mm on tight boxes, which is the signature of behaviour that training never constrains.
-
-Enlarging our own boxes shows where it breaks: 1.4x the tight box costs 2.2 mm, 1.6x costs 16.2 mm. 1.6x is exactly the loosest framing training can produce, and at that point training has only ever shown black borders, because every stored crop is cut at exactly 1.2x the tight box and saved at the network's 224 px input size. No pixel of real scene outside that square exists in the corpus. The v2 preset cannot fix this: it adds zoom-in (truncation), which is the opposite direction, and its zoom-in upsamples the 224 crop, training the model on blur a real crop would not have.
-
-The same fragility hits the product. On the phone, crops come from the NanoDet detector, whose boxes are often too large, and people often stand very close to the camera and are heavily truncated.
-
-Separately:
-
-- The model is underfitting. Training loss sits above validation loss, and it was still falling when the learning rate reached zero. Learning rate and schedule length have never been tuned.
-- The input resolution has never been varied, because the stored crops cap it at 224.
-- g8h runs in 1.4–4.1 ms on Snapdragon 8 Gen 1 and newer NPUs, but in 63–196 ms on phone CPUs, against a 33 ms (30 FPS) budget. The CPU deployment tier has no model that fits.
-
-## Solution
-
-The corpus is rebuilt once, from original frames, with twice the context around each person and at 448 px. A new v3 preset samples each training crop from that larger stored area with a wide, bounded range of framings: loose and off-centre like EMDB's and the detector's boxes, and tight and truncated like a person close to the camera. The surrounding area is always real scene, never black. Because the stored crops have resolution to spare, the network input can be 224 or larger without upsampling.
-
-Accuracy is judged on the published protocol, and v3 is compared against g8h on two seeds each, at 224 and at 288. Independently, and on the existing corpus, a small learning-rate x schedule-length sweep tests whether the underfitting model gains from a better recipe. The winning recipe is then trained at smaller backbone sizes. The sizes are picked beforehand by profiling random-weight exports on the same hosted phones, so the model family gives one accuracy-vs-latency curve with a member for each deployment tier.
-
-## User Stories
-
-1. As the researcher, I want the crop builder to accept a context factor and an output size, so that I can store crops with real scene around each person at a resolution above the network input.
-2. As the researcher, I want the builder with default flags to produce byte-identical output to today's, so that the existing corpus, the baseline and v2 presets, and every recorded result stay valid.
-3. As the researcher, I want each stored crop to record where the standard 1.2x window sits inside it, so that the trainer and any verification can recover today's framing exactly.
-4. As the researcher, I want a verification that cuts the 1.2x window out of a rebuilt crop and compares it with today's 224 crop, so that I can prove the rebuild shows the same person in the same place.
-5. As the researcher, I want the rebuilt crop's stored geometry to map the annotation targets' 2D keypoints onto the new crop to 0.000 px, so that no label is silently shifted by the rebuild.
-6. As the researcher, I want the rebuilt corpus written beside the existing one rather than over it, so that runs in flight and old checkpoints keep reading the 224 corpus.
-7. As the researcher, I want the four splits whose original frames are on Jean Zay (coco, sa1b, aic, mpii) rebuilt with the existing array build job, so that the rebuild reuses tooling that already works on the cluster.
-8. As the researcher, I want Harmony4D rebuilt by streaming its original zips scene by scene, so that the one split without original frames rejoins the corpus at the new framing.
-9. As the researcher, I want the Harmony4D streaming route chosen from the result of a download test on the `prepost` partition, so that the rebuild runs on Jean Zay when the network allows and on my laptop when it does not.
-10. As the researcher, I want each rebuilt split verified (crop count, reprojection, missing images) and archived to `$STORE`, so that the 30-day `$SCRATCH` purge costs a restore, not a rebuild.
-11. As the researcher, I want the AIC and MPII original frames archived before any rebuild starts, so that the purge due around 1–2 October 2026 cannot delete the only on-cluster copy.
-12. As the researcher, I want a v3 preset selectable with `--preset v3`, so that it sits beside baseline and v2 with the same interface.
-13. As the researcher, I want v3 to sample the effective person box over a wide range, from tight truncation to loose framing, so that the model sees the framings that EMDB boxes, detector boxes and close-up people produce.
-14. As the researcher, I want v3 to shift the box centre randomly, so that off-centre framing like EMDB's 5.7% offset is part of training.
-15. As the researcher, I want the v3 sampler to keep the sampled window inside the stored context, so that black borders never appear except where the real frame itself ends.
-16. As the researcher, I want the v3 crop sampled directly from the stored high-resolution crop in one resampling step, so that zoom-in never upsamples a 224 intermediate.
-17. As the researcher, I want the CLIFF conditioning to describe the box actually sampled in v3, so that the camera and depth placement stays correct under the new framings.
-18. As the researcher, I want the network input size to be a training option (224 by default, 288 for the resolution ablation), so that I can test whether resolution buys accuracy.
-19. As the researcher, I want the input size recorded in the run config and read back when a checkpoint is loaded, so that evaluation and deployment cannot silently feed a 288 model a 224 image.
-20. As the researcher, I want the validation, the benchmark harness and the inference package to build their input at the checkpoint's configured size, so that the three input builders stay in agreement.
-21. As the researcher, I want the ONNX export to carry the input size in its metadata, so that the phone app configures its crop size from the model file.
-22. As the researcher, I want baseline and v2 samples to stay bit-identical to the current code, so that controls in flight remain valid.
-23. As the researcher, I want the v3 dataset to reject a split that lacks the rebuilt geometry, with a clear message, so that a v3 run can never train silently on 224 crops with black-border zoom-out.
-24. As the researcher, I want a verification of v3 samples on real data (label alignment, CLIFF correctness, no black border inside the stored context, coverage of the configured box-scale range), so that the preset is proven before any GPU hours are spent.
-25. As the researcher, I want the DDP smoke test to pass for v3 at 224 and at 288, so that the cluster job cannot abort on its second step.
-26. As the researcher, I want the EMDB-1 published-protocol score (EMDB's own boxes, SMPL-24 via the adapter) reported for every new checkpoint beside the 3DPW test J14 score, so that decisions use the numbers that go in the paper.
-27. As the researcher, I want to score a checkpoint on EMDB under enlarged boxes (1.2x, 1.4x, 1.6x), so that I can see whether the framing cliff has moved.
-28. As the researcher, I want matched g8h controls on the rebuilt corpus's data mix, so that a v3 result is attributable to the preset and not to a changed mixture.
-29. As the researcher, I want v3 at 224 and v3 at 288 trained with two seeds each, so that a change above ~0.5 mm can be told from seed noise.
-30. As the researcher, I want a learning-rate x schedule-length sweep (peak learning rate {6e-4, 1.2e-3} x {300, 600} epochs) on g8h with the baseline preset on the current corpus, so that I know whether the underfitting model gains from a better recipe, independently of the rebuild.
-31. As the researcher, I want the existing g8h_s0 run reused as the (6e-4, 300) cell of that sweep, so that the sweep costs three new runs, not four.
-32. As the researcher, I want the sweep's winning recipe carried into the v3 runs once both are known, so that the final model family uses the best recipe on the best data.
-33. As the researcher, I want random-weight exports of the current architecture with HGNetV2-B0 and B1 backbones profiled on the same five hosted phones as g8h, so that the CPU-tier backbone size is chosen from measured latency before any training.
-34. As the researcher, I want each model-family member profiled for latency, NPU placement and fp16 error on QNN, so that every member satisfies the portability rule before it is trained at scale.
-35. As the researcher, I want the chosen CPU-tier backbone(s) trained with the v3 recipe, so that the paper's accuracy-vs-latency curve has a member for each deployment tier.
-36. As the researcher, I want every architecture or input-resolution change to pass the fp16-parity and full-NPU-placement check, so that a gain on the desktop never becomes a model that falls off the NPU on the phone.
-37. As the researcher, I want the published-protocol numbers, the sweep results and the family latencies recorded in the repo's results and status docs, so that the paper's tables have a single source.
-38. As the phone-app developer, I want the NPU runtime libraries for Snapdragon 8 Gen 1, Gen 3 and Elite shipped alongside Gen 2's, so that the same model runs on every Snapdragon phone that can run it.
-39. As the phone-app developer, I want phones without a usable fp16 NPU (Snapdragon 888 / 778G class, Exynos, Tensor) served by the CPU-tier model instead of the fake-model fallback, so that the app works on any Android phone.
-
-## Implementation Decisions
-
-- **Two storage parameters, one rebuild.** The builder gains a context factor (the stored square's side as a multiple of the tight box; today 1.2) and keeps its existing output-size option. The rebuild uses context 2.0 at 448 px. The standard 1.2x window then covers ~269 px of real detail. That supports 224 and 256 input without upsampling; 288 upsamples by ~7%. Defaults reproduce today's builder byte for byte.
-- **Recorded geometry.** Each rebuilt annotation stores the stored square in original-frame coordinates, next to the existing 1.2x square, so any window can be recomputed exactly. Where the stored square leaves the original frame, the builder's existing padding behaviour applies, and that padding is real frame-edge truncation, not an augmentation artefact.
-- **Separate corpus.** Rebuilt splits live in their own directories, named so that the baseline and v2 presets never pick them up. The pair-index fingerprint therefore does not change for runs on the old corpus.
-- **v3 box sampling.** Per sample, the effective person box is drawn as a multiple of the tight box, **log-uniformly** between 0.6x (the same truncation depth as v2's 2.0 zoom-in on a 1.2x crop) and 2.0x (the full stored context). Log-uniform keeps tight and loose framings equally likely instead of over-sampling the loose end. The centre is shifted by up to +/-15% of the box on each axis. The window is clamped to stay inside the stored context, so a large box and a large shift never combine into a black border. The bounds are config values with these defaults; they are not per-run tuning knobs. v3 keeps rotation and flip as v2 has them.
-- **Wide on purpose.** The zoom-out range covers EMDB's framing (average ~1.4x equivalent, with a tail past the 1.6x cliff) and the phone detector's oversized boxes. The zoom-in range covers people close to the camera. The upper bound is the stored context, so "wide" never means synthetic black margins.
-- **One resampling step.** The v3 warp maps the stored crop straight to the network input, composing the box sampling with rotation and flip. No intermediate resize to 224 happens first. This is what removes v2's upsampling blur.
-- **CLIFF follows the sampled box.** v3 builds the conditioning vector from the sampled box in original-frame coordinates, in the perspective-correct form (`cliff_focal`), with the crop-centre correction for zoom already verified for v2.
-- **Input size is config.** A single input-size setting drives the dataset, the validation, the benchmark harness and the inference package. It is stored in the run config, recovered by the existing config-from-checkpoint path, and stamped into ONNX metadata beside the CLIFF form. A checkpoint without it means 224.
-- **Presets stay orthogonal.** v3 is a value of the existing preset option. The loss-budget option is unchanged. baseline and v2 code paths are not modified beyond what is needed to route to v3.
-- **Harmony4D route.** If the `prepost` download test returns HTTP 206, Harmony4D is rebuilt on Jean Zay with the existing streaming job, passing the new builder flags and keeping no 4K originals. If it fails, the same streaming script runs on the laptop, scene by scene, deleting each zip after its crops are built, and the crops are rsynced as one tar.
-- **Controls and seeds.** Each v3 arm is compared against g8h retrained on the rebuilt corpus's exact data mix, with 2 seeds per arm. The acceptance target is a meaningful reduction of the EMDB published-protocol penalty with no loss on 3DPW test J14, where "meaningful" means beyond the ~0.5 mm that 2 seeds can resolve.
-- **Recipe sweep runs now.** The learning-rate x schedule-length sweep uses the current corpus and the baseline preset. It shares nothing with the rebuild and starts immediately.
-- **Model family sizing.** CPU-tier backbones are chosen from random-weight exports profiled on the Galaxy S23, A73, A53, A14 and Pixel 8 (CPU only), against a 33 ms budget on mid-range phones. The NPU-tier member stays at the HGNetV2-B4 size.
-- **Not reopened.** RepViT stays dropped because it overflows fp16 on the NPU. Weight-decay and dropout tuning stay dropped while the model underfits; they return if training ever overfits. No large seed sweep: seed spread is ~0.2 mm at generation 8.
-
-## Testing Decisions
-
-A good test here observes a component from its outside, on real data, and reports a number: pixels, millimetres, milliseconds or a byte diff. It never asserts on internal variables. Four seams, three of which already exist:
-
-- **Crop builder output (new check).** Run the builder on a small real split. Default flags: output byte-identical to the current builder's. New flags: the 1.2x window cut from the rebuilt crop and resized to 224 matches today's crop to within JPEG and resampling noise (report PSNR). The recorded geometry reprojects the annotation targets' 2D keypoints to 0.000 px. Prior art: the builder's own reprojection validation and `verify_split.py`.
-- **One training sample.** For baseline and v2: a few dozen augmented samples bit-identical against `git show HEAD`, compared on alternating calls because of the ~5e-10 cold-kernel difference in the rig. For v3: 2D labels land on the right pixels; the CLIFF vector matches the exact preimage of the crop centre (as in the `--crop-centre-fix` verification: mean, p95 and max error in px); no black pixel appears inside the stored context; the drawn box scales cover the configured range; the output size equals the configured input size. Prior art: `tools/verify_cont_head.py`, `tools/verify_exact_landmarks.py`.
-- **Exported model on phones.** `tools/ddp_smoke.py` for every new configuration, then ONNX export, then AI Hub profiling for latency on the five reference phones, NPU placement (must be 100% on Snapdragon 8 Gen 1+) and fp16 error. Prior art: the AI Hub profiling in the phone-app repo and this session's portability run.
-- **Benchmark harness, unchanged.** EMDB-1 with `--bbox annotated` and the SMPL-24 adapter, 3DPW test J14 with the adapter, plus EMDB under enlarged boxes (1.2x / 1.4x / 1.6x) as the framing-cliff diagnostic. Evaluating a 288 checkpoint through the harness is itself the test that the input size round-trips through the checkpoint config.
-
-## Out of Scope
-
-- Core ML / Apple deployment. There is no Mac or iPhone yet; it is deferred to when there is.
-- int8 quantization for older Snapdragon NPUs. Those phones are served by the CPU tier.
-- Decoder feedback (roadmap item 4). It follows once the v3 recipe is known and is gated by the same portability check.
-- Changing the loss budget, the continuous head or the exact-landmark readout.
-- Weight-decay and dropout tuning, while the model underfits.
-- Rebuilding EgoHumans or EgoExo4D, which are not in the current corpus.
-- The phone app's own changes (stories 38–39) beyond specifying what the model family needs from it. They are tracked in the phone-app repo.
-
-## Further Notes
-
-- Measurements this spec rests on (2026-09-25): EMDB-1 annotated-box scores for g8h_s0/s1 are in `benchmark/results/gen8_emdb1_annotated.json`. The box-scale diagnostic: 51.95 / 54.19 / 68.16 mm at 1.2 / 1.4 / 1.6x, stride 5. Hosted-phone latencies are from AI Hub jobs on the compiled g8h model (compile job `jpvldwoz5`): NPU S22 4.1 ms, S23 2.4 ms, S24 1.7 ms, S25 1.4 ms, S21 and A73 all-CPU. CPU: S23 63 ms, Pixel 8 82 ms, A73 84 ms, A53 188 ms, A14 196 ms.
-- Storage: the 448 px rebuild is roughly 4x today's crop pixels, ~250 GB on `$SCRATCH` (400 TB, 1% used), archived as one tar per split to `$STORE` (50 TiB).
-- Compute: 4,385 of 50,000 V100 hours used. A 300-epoch run is at most 4 links x 20 h x 4 GPUs = 320 GPU-hours.
-- Vocabulary follows `CONTEXT.md`: reference model, annotation target, baseline preset, v2 preset, published protocol, deployment tier, model family.
diff --git a/CLAUDE.md b/CLAUDE.md
deleted file mode 100644
index 21e711b..0000000
--- a/CLAUDE.md
+++ /dev/null
@@ -1,418 +0,0 @@
-# CLAUDE.md
-
-Behavioural guidelines plus what you need to know about this repo before touching it.
-
-**Tradeoff:** these guidelines bias toward caution over speed. For trivial tasks, use judgment.
-
----
-
-## 0. Never commit
-
-**The user commits. Always.** Never run `git commit`, `git push`, `git merge`, or
-`git rebase`. When work is ready, print the commands and let the user run them.
-Keep commit messages to one or two short sentences.
-
-`git add` and read-only git (`status`, `diff`, `log`, `show`) are fine.
-
----
-
-## 0.5 Write for a human, not a compiler
-
-**The user is a person reading a terminal, not a log parser.** Answers that are
-technically correct and unreadable are failures. This has come up more than
-once — take it seriously.
-
-- **Define every name before you use it.** `scale_lo`, `m_noflip`, `fk_scale`,
- `n_skip_run` mean nothing to someone who has not just read that function. If
- a variable, flag or file has to appear, say what it *is* in plain words
- first, once.
-- **Give commands in the form the user actually runs them.** This repo launches
- through `datasets_pipeline/jeanzay/52_train_ddp.slurm` with `RUN_NAME=... MIX=...
- EXTRA_TRAIN_ARGS=... sbatch`. Read the launcher and match it. A bare `srun
- python ...` line the user has never typed is not an answer.
-- **Say what to run, in what order, and what happens next.** If a command is
- optional, say so. If you offer a check, say what a good result looks like.
-- **Lead with the answer.** The finding first, the evidence after. Do not make
- the user read a derivation to learn whether it worked.
-- **Explain the mechanism, don't just name it.** "The bone scales enter the
- kinematics multiplicatively, so +10 makes a 336 m skeleton" teaches
- something; "unbounded scale parameters caused divergence" does not.
-- **Cut the ceremony.** No restating the request, no listing what you are about
- to do, no summary of the summary.
-
-If the user says they cannot follow an answer, that is a defect to fix in the
-next answer, not a preference to note.
-
----
-
-## 1. Think before coding
-
-Don't assume. Don't hide confusion. Surface tradeoffs.
-
-- State your assumptions explicitly. If uncertain, ask.
-- If multiple interpretations exist, present them — don't pick silently.
-- If a simpler approach exists, say so. Push back when warranted.
-- If something is unclear, stop. Name what's confusing. Ask.
-
-## 2. Simplicity first
-
-Minimum code that solves the problem. Nothing speculative.
-
-- No features beyond what was asked.
-- No abstractions for single-use code.
-- No "flexibility" or "configurability" that wasn't requested.
-- No error handling for impossible scenarios.
-- If you write 200 lines and it could be 50, rewrite it.
-
-Ask yourself: *"Would a senior engineer say this is overcomplicated?"* If yes, simplify.
-
-## 3. Surgical changes
-
-Touch only what you must. Clean up only your own mess.
-
-- Don't "improve" adjacent code, comments, or formatting.
-- Don't refactor things that aren't broken.
-- Match existing style, even if you'd do it differently.
-- If you notice unrelated dead code, mention it — don't delete it.
-- Remove imports and variables that *your* changes orphaned; leave pre-existing
- dead code alone unless asked.
-
-The test: every changed line should trace directly to the user's request.
-
-## 4. Goal-driven execution
-
-Define success criteria. Loop until verified.
-
-- "Add validation" → "write tests for invalid inputs, then make them pass"
-- "Fix the bug" → "write a test that reproduces it, then make it pass"
-- "Change a loss" → "show the new term is numerically what you claim, on real data"
-
-For multi-step tasks, state a brief plan with a verification per step.
-
-In this repo "verify" usually means **run it on real data and show numbers**, not
-read the code and assert it looks right. `data/` holds a local corpus and
-`checkpoints/mhr_model.pt` is the MHR rig, so most claims about geometry, losses
-or augmentation can be measured in under a minute. Do that.
-
----
-
-## What the training labels are
-
-**This is not a distillation setup.** Training is supervised by the released
-ground-truth annotations of `facebook/sam-3d-body-dataset` — the human MHR fits
-Meta used to build SAM 3D Body. The whole cluster corpus (`sam3d_gt_sa1b` /
-`_aic` / `_harmony4d` / `_coco` / `_mpii`) and every number in this repo's result
-tables are of that kind.
-
-Distilling from the `sam-3d-body-dinov3` model is an **option**:
-`tools/annotate_dataset.py` runs it over your own images and is the only thing
-that writes `data/sam3d_distill_mix/`. Use it for imagery the released dataset
-does not cover, and remember it caps the student at the teacher's accuracy.
-
-Two consequences for reading this file and the code:
-
-- **`train_distill_*.py` and `data/sam3d_distill_mix/` are historical names.**
- They say nothing about which labels a run used.
-- **Where the docs say "teacher" they almost always mean "the training
- target".** Parameter ranges, oracle ablations and
- `adapter_j14_h36m_teacher.npz` were all measured against the *dataset's*
- annotations, not against model inference.
-
-## Where to read first
-
-Docs are the source of truth; the code is bigger than any context window.
-
-| order | file | for |
-|---|---|---|
-| 1 | `instanthmr_distill_train/README.md` | which training script to run, what `--preset v2` and `--losses rebalanced` change, the pair-index cache |
-| 2 | `datasets_pipeline/jeanzay/STATUS.md` | Jean Zay cluster + data state, what is built, job templates *(git-excluded, local only)* |
-| 3 | `datasets_pipeline/JEAN_ZAY.md` | site runbook: env, quotas, proxy quirks *(git-excluded, local only)* |
-| 4 | `benchmark/README.md` | how published numbers are produced; the 3DPW GT (SMPL forward + H36M regressor), the MHR->J14 adapter, and why **J14+adapter** is the row to quote |
-| 5 | `datasets_pipeline/README.md` | how the corpus is built, including the SA-1B visibility mask |
-| 6 | `docs/architecture.md` | the student model |
-| 7 | `docs/todo.md` | open work: reproducing the teacher's regression pathway, the crop-resolution rebuild, model selection. Cleared and rewritten 2026-09-12 |
-
-`datasets_pipeline/jeanzay/` and `JEAN_ZAY.md` are in `.git/info/exclude` on
-purpose — they carry the login name, jump host and project code. Keep
-site-specific material there and portable material in the repo docs.
-
-## Repo-specific traps
-
-**One trainer, two layers.** `train_distill_mhr_only.py` holds the dataset,
-augmentation, model and losses. `train_distill_jz.py` imports it and adds DDP,
-mixture sampling, resume and 3DPW validation. Substance changes go in the base
-file so both entry points get them. **Never fork the base file** — that is how
-`train_distill.py` drifted until its ONNX export silently zeroed the SimCC 2D
-head, with keypoints collapsing to the bbox centre and no error raised.
-
-**`--preset baseline` and `--losses legacy` must stay bit-identical.** Runs in
-flight use them as controls. The two flags are orthogonal axes — `--preset` is
-the input pipeline and the absolute-pose terms, `--losses` is the loss budget —
-and both default to the old behaviour. If you change the dataset or a loss, gate
-it behind a config flag whose default reproduces the old behaviour, and prove it
-— diff a few dozen augmented samples against `git show HEAD:<file>`. When you
-do, compare **alternating** calls: the TorchScript MHR forward differs by ~5e-10
-on its first invocation in a process (cold-kernel autotuning), which reads as a
-regression if you only compare once.
-
-**Know the MHR parameter layout before touching a pose loss.** From
-`character_torch.parameter_transform.parameter_names`: `0:3` root translation
-(always 0 in the corpus, 1 unit = 10 cm), `3:6` root rotation (extrinsic-XYZ
-Euler, driving joint-parameter slots 10/11/12 *and nothing else*), `6:136` the
-130 local joint angles, `136:204` 68 bone scales. Consequences that cost a
-session to re-derive: `6:136` is exactly invariant to the in-plane rotation
-augmentation, so it never needed the `m_ident` mask; and the 45 `shape_params`
-move the 127-joint skeleton by `0.00e+00 cm`, so they are mesh-only and every
-loss and metric in the trainer is blind to them.
-
-**The teacher's root Euler triple is REVERSED relative to the rig, and its own
-source hides it.** `sam_3d_body/models/heads/mhr_head.py` decodes the global
-rotation with `roma.rotmat_to_euler("ZYX", R)`. roma's *uppercase* letters mean
-extrinsic and it returns the triple in the order of the string, so `"ZYX"` hands
-back `(theta_z, theta_y, theta_x)` -- the reverse of the rig's extrinsic-XYZ
-`model_params[3:6]`. Feeding `(0.3, 0.5, 0.7)` through the teacher's own forward
-returns `(0.7, 0.5, 0.3)`. Measured against `checkpoints/mhr_model.pt` (md5
-8f21d804..., byte-identical to the teacher's `assets/mhr_model.pt`): drive
-`3:6`, rigid-fit the 125 joints it moves, and the recovered rotation matches
-`batch6DFromXYZ` to **0.00000-0.014 deg** while roma's `"ZYX"` of the same
-triple is **36-92 deg away**. Do the fit about the moving sub-skeleton, not the
-centroid and not joint 0 -- the root drives joint-parameter slots 10/11/12, so
-joints 0 and 1 do not move and a centroid Procrustes leaves an 11 mm residual
-that reads as a convention mismatch. Either convention round-trips if encode and
-decode agree, so a wrong choice trains fine and only shows up as a rotation loss
-measuring distance in a permuted space. `instanthmr_distill_train/mhr_cont.py`
-uses the rig's and says so; do not "fix" it to match the teacher's line.
-
-**The vertex subset now has two independent consumers, and mixing them breaks
-a loss silently.** `--w-verts` skins 595 farthest-point samples; the teacher's
-exact 70-landmark mapping (`--exact-landmarks`) needs the 468 vertices its first
-70 rows reference. **Those two sets overlap in exactly 10 vertices** -- the 468
-are not a subset of the 595, and raising `n_verts` does not reach them, because
-farthest-point sampling optimises coverage, not landmark support. So
-`_build_vertex_subset` skins their UNION (1053) and keeps `verts_loss_pos` /
-`lm_vert_pos` to index the two consumers separately. If `loss_verts` ever
-averages over the union instead of `verts_loss_pos`, it silently changes value
-and stops being comparable with generations 6-8 -- no error, just a different
-number. The union is free (181.2 ms for 595 alone vs 180.9 ms for 1053 at batch
-64), because the cost is the forward-kinematics pass, not the vertex count.
-`instanthmr_distill_train/assets/mhr_landmarks70.npz` is a new file and must be
-rsynced. Scoring stays on the fitted readout by default and is now a switch
-(`--landmarks exact` on `eval_3dpw_ckpt.py` / `eval_emdb_ckpt.py`,
-`--val-landmarks exact` on the trainer). Measured 2026-09-22 it is worth
-**0.13 mm** of 3DPW-test J14+adapter PA-MPJPE and 0.1 mm of EMDB-1 — and the
-same 0.13 mm on `g7h_s0`, which trained WITHOUT `--exact-landmarks`, so it is a
-property of the operator, not of the training flag. The mismatch it removes is
-real (both adapters are fitted from the annotation `joints_3d`, i.e. the exact
-readout), but it is an offset, not a re-ranking. `fitted` stays default so the
-metric keeps measuring the deployed path and every recorded row stays
-comparable; **never put fitted-scored and exact-scored rows in one table.**
-
-**`--cont-head` adds two files that must be rsynced.**
-`instanthmr_distill_train/mhr_cont.py` and
-`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB, built by
-`tools/build_cont_head_assets.py` from the teacher checkpoint plus the corpus).
-Neither existed before 2026-09-12 and the job dies at startup without them.
-Gate any change to that head with `tools/verify_cont_head.py` (24 checks,
-including GT reachability to 0.0003 mm of forward kinematics) *and*
-`tools/ddp_smoke.py --cont-head`.
-
-**The student's MHR outputs are unbounded and 77 of them are body SIZE.** The
-head is a plain `nn.Linear`. `0:3` root translation (exactly `0.000e+00` in all
-24,000 sampled annotations — the person is placed by `cam_trans`), `130:136`
-the `*_length/_width_flexible` parameters, and `136:204` the 68 bone scales all
-scale the skeleton, the last **multiplicatively along the kinematic chain**:
-uniform +10 gives a 336 m skeleton, +25 gives 28,300 km. That, not the learning
-rate, killed four 300-epoch runs at epochs 24-32 on 2026-09-06. Watch for the
-two traps: `130:136` *look* like pose parameters (they sit inside `6:136`) but
-drive joint **translation** channels, so bounding only `136:204` still reaches
-375 m; and `6:130` are genuine rotations that stay at ~1.5 m at any magnitude,
-so leave them alone. `--bound-scales` fixes it from
-`assets/mhr_size_bounds.npz`; that file is **new and must be rsynced** or the
-job dies at startup.
-
-**The anomaly guard deletes the gradient that would fix the anomaly.** A batch
-over `anomaly_loss_threshold` was skipped whole — but `loss_scale`/`loss_pose`
-in that same batch are the only force pulling the sizes back, so the sample was
-quarantined for good. Worse, `bad` is all-reduced with `MAX`: one bad sample in
-4x64 = 256 vetoes the step for every rank, so 2% runaway samples stop 99.4% of
-steps. Use `--anomaly-safe-fallback`, which steps on the non-FK terms instead
-(`safe_loss_subset`, cosine +1.0000 with `teacher - prediction`). Never "fix" a
-divergence here by lowering the LR alone: `OneCycleLR(pct_start=0.1)` peaks at
-10% of the run, so a lower peak only moves *when* it dies.
-
-**A single-process smoke test cannot clear a training change for the cluster.**
-`52_train_ddp.slurm` wraps the student in `DistributedDataParallel` with the
-default `find_unused_parameters=False`, and that Reducer exists **nowhere** in a
-laptop run -- not in `train_distill_jz.py` on one GPU, not in `--self-test`, not
-in `--overfit-test`. A parameter that receives no gradient therefore passes
-every local check and then aborts the job on the SECOND optimiser step with
-`Expected to have finished reduction in the prior iteration ... Parameter
-indices which did not receive grad for rank N: <i>`, where `<i>` indexes
-`list(model.parameters())`. On 2026-09-11 that burned all eight queued
-generation-7 `hgnetv2_b4` links while the `repvit_m2_3` arms ran nine hours
-beside them: `forward_features()` stops before the timm classifier head, and
-`num_classes=0` empties that head for repvit but leaves hgnetv2_b4 a 2048x2048
-`last_conv` -- 4,194,304 orphaned parameters, index 241. `InstantHMRStudent`
-now replaces any head that still has parameters with `nn.Identity()` (a no-op
-for a parameterless one, so every existing state_dict is byte-identical --
-verified, 1542 keys, same sha256). **Run `python tools/ddp_smoke.py --backbone
-<name> --w-verts <w>` before submitting any architecture change.** It builds a
-real Reducer at world size 1 and takes under a minute; world 1 is enough,
-because the unused-parameter check is world-size independent.
-
-**The pair index is a host-RAM bomb, and it kills jobs with no traceback.**
-Four generation-6 jobs were OOM-killed at 74-76 GiB per node between epochs 14
-and 33 on 2026-09-08. Not CUDA -- `sacct` says `OUT_OF_ME+`, exit `0:125`,
-`Detected N oom_kill events`, and the only Python output is a dataloader worker
-dying of `ConnectionResetError`, which is the symptom. Measured: a list of
-`(PosixPath, PosixPath)` pairs costs **694 bytes per pair**, so 4.2 M crops is
-2.94 GB per process; `build_jz_loaders` runs `persistent_workers=True` with 9
-workers per rank, i.e. 40 processes per node, and CPython writes a refcount into
-an object header just to *read* it, so every worker steadily converts its
-copy-on-write share into a private copy. It grows over epochs, which is why the
-kill lands hours in. `PairIndex` in `train_distill_jz.py` stores the index as
-three contiguous numpy byte arrays instead (87 bytes/pair, 8.0x smaller,
-verified identical on all 961,139 local pairs and in iteration order). Do
-**not** call `list()` on it -- `SAM3DStudentDataset` copies only real lists for
-exactly this reason. Two corollaries: adding *any* per-node memory (the
-`--w-verts` subset costs ~2 GiB across 4 ranks) decides only which arm dies
-first, not whether one does; and `sacct` MaxRSS undersamples at ~30 s, so a
-number well under the `mem=160000M` cgroup limit does not mean you are safe.
-
-**rsync to the cluster lands files where you point it, and a misplaced trainer
-fails silently.** `rsync ... a/b.py c/d.slurm host:REPO/` puts BOTH in
-`REPO/`, not in `REPO/a/` and `REPO/c/`. A stray `train_distill_jz.py` in the
-repo root is simply never imported, so the job runs the old code and raises
-nothing -- on 2026-09-08 that silently discarded both a
-`--constraint=v100-32g` fix and a logging change for a full generation. Check
-the destination directory per file, and verify on the cluster with `grep -c
-'<a string only the new version has>'` before submitting.
-
-**Body shape is invisible to every geometric loss, and the fix is a vertex
-term.** `MHRForwardPass.get_joints` zeroes the 45 `shape_params`, and that is
-not a shortcut: they move the 127-joint skeleton by *exactly* `0.000e+00 cm`.
-Measured `d(loss)/d(shape_params)` is `8.9e-03` through vertices and
-`0.000e+00` through joints, so `--w-verts` is the ONLY route by which shape gets
-a geometric gradient; ignoring identity costs 5.95 mm mean / 22.68 mm max PVE.
-Do **not** reach for `models/mhr_assets/assets/lod6.fbx` to make it cheap — it
-needs `pymomentum` (which segfaults here) and it is a different topology, so the
-teacher correspondence would have to be fitted. `n_verts` instead takes a
-farthest-point subset of the rig's OWN 18,439-vertex mesh and rewrites the three
-flattened influence tables of the rig's own `linear_blend_skinning`; the op is
-untouched, so the answer is bit-exact against the full mesh (4.6e-05 cm) and the
-vertex ids match the teacher's. 595 vertices cost +0.6 ms per batch-64 step
-against 16.7 ms for all 18,439. `loss_verts` belongs in `FK_LOSS_KEYS` — a
-blown-up bone scale explodes vertices exactly as it explodes joints.
-
-**`parameter_transform[:, :204]` has rank 192, not 204.** Twelve directions
-move the parameters and leave the skeleton *exactly* unchanged — each
-`*_flexible` parameter against its `scale_*` partner, plus the spine rotations.
-Along them every geometric loss has zero gradient, so the model is regressing
-12 dimensions of solver-arbitrary teacher noise that no image can determine.
-Do **not** "fix" this by zeroing the `*_flexible` parameters even though the rig
-pins them to `[0,0]`: the teacher genuinely uses them (`leg_length_flexible`
-mean 0.32, max 1.64), so zeroing shortens every leg.
-
-**`DistillationLoss` is never `.to(device)`'d.** `run_overfit_test`,
-`run_self_tests`, `train_instant_hmr` and `train_distill_jz.train` all construct
-it bare, so any buffer you register lands on the CPU and raises on step 1. Build
-tensors on `mhr_module.device` instead.
-
-**3DPW GT is the published protocol, and it is a cache on disk.** The pickles'
-`jointPositions` field is the 24 SMPL *kinematic* joints, which is not what any
-paper reports; `benchmark/make_3dpw_gt.py` precomputes the real reference (SMPL
-forward + `J_regressor_h36m`) into `<3DPW root>/gt_h36m/`, and both the harness
-and `val3dpw.py` default to it. Two consequences: a job whose 3DPW directory
-lacks that cache raises at startup, and **numbers from before 2026-09-05 are ~13
-mm apart from later ones** — `--val-3dpw-gt jointpositions` keeps a resumed run
-on its original metric.
-
-**Anything fitted over the 70 MHR keypoints needs float64 and a truncated
-`lstsq`.** 40 of the 70 are finger joints millimetres apart, so the design
-matrix is near-singular. Centring in float32 leaves a 0.04 mm residual in the
-exact direction centring annihilates, `lstsq` keeps it as signal, and the J14
-adapter went from 37 mm to 430 mm in-sample looking like a broken GT. Even in
-float64 the unregularised fit reaches landmarks through ±2000 weights on
-fingers; `RCOND` in `eval_3dpw.fit_adapter` truncates it to a rank-24 map with
-weights under 1 at no measurable cost.
-
-**The CLIFF conditioning is computed in three places and they must agree.**
-`SAM3DStudentDataset.__getitem__`, `val3dpw._preprocess` and
-`instanthmr.inference.InstantHMR._preprocess` each build the 3-vector
-separately. Feeding a model trained on one variant with the other raises
-nothing — it silently mis-places the person in depth. Since 2026-09-06 the
-perspective-correct variant (`--cliff-focal`, config `cliff_focal`) is
-mandatory for new runs, because the rebuilt corpus carries real per-image
-focals spanning `f/diag` 0.6-1.8 where the pixel form is ambiguous; the old
-corpus was constant at 1.000. See `instanthmr_distill_train/README.md`.
-
-**Never rebuild a config from `DistillConfig()` defaults to load a checkpoint.**
-Two 2026-09-06 flags make the defaults wrong, in opposite ways.
-`--bound-scales` puts a `tanh` remap inside `forward()`, so a default-config
-export reads the head's raw pre-`tanh` numbers as bone scales — measured 21.0
-mm MPJPE / 14.4 mm PA-MPJPE of silent corruption. `--cliff-focal` changes what
-the conditioning vector *means*, not the graph, so a mismatch raises nothing.
-Use `T.config_from_checkpoint()`: it reads the bounds out of the weights
-(`scale_lo`/`scale_hi` are the flag) and the conditioning form out of the
-`run_config.json` beside the checkpoint, and `tools/pth_to_onnx.py` stamps the
-latter into the ONNX metadata so `instanthmr.inference` configures itself.
-Deployment has no annotation to read a focal from, so it falls back to
-`1.05 x diag` for focal-aware graphs and `1.00 x diag` for pixel-conditioned
-ones (that IS the old corpus's synthetic focal); `demo.py --focal` overrides.
-
-**One reported number: J14 PA-MPJPE, adapter-applied, on the H36M GT.** The
-training log, `summarize_runs.py` and `benchmark/eval_3dpw_ckpt.py` all print
-it and they agree to the decimal. J12 was removed on 2026-09-06 — it was the
-rig-clean row against the old GT and is not against this one. Runs started
-before that date recorded J14 *without* the adapter, ~26 mm higher; do not put
-old and new rows in one table.
-
-**PA-MPJPE is blind to half of what a demo shows.** Procrustes removes
-translation, rotation and scale, so `cam_trans` drift, body-size error and
-temporal jitter are all invisible to it — and to root-relative MPJPE for the
-first two. A change can be worth shipping on stability and move the benchmark
-by 0.1 mm. Measure jitter directly (frame-to-frame acceleration on 3DPW's
-consecutive frames, and the variance of rigid bone lengths within a person
-track, which cannot change and so is pure noise).
-
-**Seed spread is narrow at generation 8.** g8h_s0 vs g8h_s1 differ by 0.10 mm
-on 3DPW test J14 (41.76 / 41.86) and 0.24 mm on EMDB-1 (46.93 / 46.69,
-fit-free J14/H36M). The old "34–49 mm" warning was the 70-keypoint metric on
-early generations and no longer applies. A seed sweep buys ~0.2 mm, so it is
-not a way to gain accuracy; keep 2 seeds per arm only to tell a >0.5 mm change
-from noise.
-
-**Two corpora, two data roots, never mixed.** `$SCRATCH/instanthmr/data` holds
-the 224 px crops (baseline / v2 presets); `$SCRATCH/instanthmr/data_ctx` holds the
-context crops (2.0x box, 448 px, `build_split.py --context`) that only
-`--preset v3` can read. Each root has its own pair index. The builder refuses a
-context build without `--output-dir`, and the v3 dataset refuses any root that
-contains a standard split. Generation 9 is the first to train on `data_ctx`; see
-STATUS.md. A third kind, `data_win224` / `data_win288` (`build_split.py
---windows 4`, one tile of four windows per person), is the only corpus
-`--preset v4` reads, and the image size must match the folder. When `prepost` has no free node, `datasets_pipeline/jeanzay/21_build_gpu.slurm`
-builds on the 2 h GPU dev queue instead.
-
-**You cannot reach Jean Zay.** No SSH, no Slurm. Print commands for the user and
-read the logs they paste back. Files reach the cluster by `rsync` of *specific
-files* — never the whole tree, since some files are edited on both sides.
-
-**ONNX export constrains the model.** `torch.linalg.lstsq` and friends do not
-export; use explicit small-matrix algebra. Pin `onnxruntime-gpu==1.26.0` — 1.27+
-is CUDA 13 and silently falls back to CPU.
-
----
-
-## Agent skills
-
-### Issue tracker
-
-Issues and specs live as local markdown files under `.scratch/<feature>/`, not GitHub Issues (the repo is public). See `docs/agents/issue-tracker.md`.
-
-### Triage labels
-
-The five default roles, label string = role name (`needs-triage`, `needs-info`, `ready-for-agent`, `ready-for-human`, `wontfix`). See `docs/agents/triage-labels.md`.
-
-### Domain docs
-
-Single-context: one `CONTEXT.md` plus `docs/adr/` at the repo root, created lazily. See `docs/agents/domain.md`.
diff --git a/CONTEXT.md b/CONTEXT.md
deleted file mode 100644
index dac2244..0000000
--- a/CONTEXT.md
+++ /dev/null
@@ -1,62 +0,0 @@
-# InstantHMR
-
-A real-time, single-image human mesh recovery model that predicts MHR rig parameters and is built to run on phones. This glossary fixes the words used about its models, training runs, evaluation and deployment.
-
-## Language
-
-### Models and runs
-
-**Reference model**:
-The model currently shipped to devices and used as the comparison point for every new experiment (g8h as of 2026-09-24).
-_Avoid_: baseline, new baseline, the current model
-
-**Generation**:
-A numbered batch of training runs that differ from the previous batch along one or two stated axes (e.g. gen8 = continuous regression head + exact landmarks).
-_Avoid_: version, round
-
-**Arm**:
-One configuration inside a generation, trained under one or more seeds (e.g. g8h = gen8 with the HGNetV2-B4 backbone).
-_Avoid_: variant, experiment
-
-**Model family**:
-A set of models trained with one recipe at several backbone sizes, so that each deployment tier gets its own point on one accuracy-vs-latency curve.
-_Avoid_: model zoo, variants
-
-### Training
-
-**Annotation target**:
-The human MHR fit released for each person in `facebook/sam-3d-body-dataset`, which every current run is trained to match.
-_Avoid_: reference target, teacher label, pseudo-label, GT (unqualified)
-
-**Baseline preset**:
-The original input pipeline selected by `--preset baseline`; only ever the name of a preset, never of a model.
-_Avoid_: baseline (on its own), default
-
-**v2 preset**:
-The alternative input pipeline selected by `--preset v2`, whose CLIFF vector follows the augmented crop.
-
-**v3 preset**:
-The input pipeline that samples each training crop from a context crop over a wide range of framings, loose, off-centre and truncated, always surrounded by real scene.
-
-**Context crop**:
-A stored training crop covering twice the person's tight box at high resolution, from which the network's input window is sampled.
-_Avoid_: big crop, raw crop, original (originals are the full source frames)
-
-**v4 preset**:
-v3's framings without v3's texture: each training crop is one of four precomputed windows, on the baseline loss weights.
-
-**Precomputed window**:
-A training crop cut from the full-resolution photo with a single resize, exactly as the runtime cuts its crop; window 0 frames the person like a perfect detector box, the others loose, off-centre or truncated.
-_Avoid_: tile (the tile is the file holding a person's four windows side by side)
-
-### Evaluation
-
-**Published protocol**:
-The evaluation setting whose numbers may sit beside other papers' rows: EMDB-1 with EMDB's own person boxes scored on SMPL-24 joints, and 3DPW test scored on J14 with the adapter.
-_Avoid_: benchmark (unqualified), our protocol
-
-### Deployment
-
-**Deployment tier**:
-A class of device runtime the product must support, each with its own latency target: Snapdragon NPU, Apple Core ML, and CPU (the tier that covers every other device).
-_Avoid_: platform, backend, target
diff --git a/README.md b/README.md
index 3faecb4..fe77402 100644
--- a/README.md
+++ b/README.md
@@ -318,8 +318,7 @@ via `--model`.
InstantHMR can be trained from two different label sources. **Both produce the
exact same `images/*.png` + `annotations/*.npz` layout** that
-[`notebooks/distill_transformer_decoder.ipynb`](notebooks/distill_transformer_decoder.ipynb)
-consumes, so the training notebook is identical either way:
+[`training/`](training/README.md) consumes, so training is identical either way:
1. **Released ground-truth annotations — this is what every shipped model uses.**
Train directly on
@@ -327,8 +326,8 @@ consumes, so the training notebook is identical either way:
— the human MHR fits used to build SAM 3D Body — converted to the `.npz`
schema by [`tools/parquet_to_npz.py`](tools/parquet_to_npz.py) and laid out
by [`datasets_pipeline/build_split.py`](datasets_pipeline/build_split.py)
- into `data/sam3d_gt_<dataset>/`. The cluster corpus is entirely of this kind
- (sa1b / aic / harmony4d / coco / mpii).
+ into `data/sam3d_gt_<dataset>/`. The released models were trained only on
+ this kind (sa1b / aic / harmony4d / coco / mpii).
2. **Teacher distillation — an option, not the default.** Run
`facebook/sam-3d-body-dinov3` over your own images with
[`tools/annotate_dataset.py`](tools/annotate_dataset.py) to generate
@@ -337,9 +336,9 @@ consumes, so the training notebook is identical either way:
imagery the released dataset does not cover; note that it caps the student at
the teacher's own accuracy, which training on the ground truth does not.
-The trainer entry points and the `data/sam3d_distill_mix` directory keep
-"distill" in their names for historical reasons. The name does **not** mean the
-labels are teacher inference — only `tools/annotate_dataset.py` produces those.
+The `data/sam3d_distill_mix` directory keeps "distill" in its name for
+historical reasons. The name does **not** mean the labels are teacher inference —
+only `tools/annotate_dataset.py` produces those.
### Train on the original annotations (COCO example)
@@ -368,9 +367,8 @@ python tools/parquet_to_npz.py \
--output_dir data/sam3d_gt_coco \
--validate
-# 4. Point the notebook's config at the output and train:
-# cfg.images_dir = "data/sam3d_gt_coco/images"
-# cfg.annotations_dir = "data/sam3d_gt_coco/annotations"
+# 4. Train on it (see training/README.md for the released models' settings):
+python training/train_ddp.py --data_root data --output_dir runs/coco
```
The converter recovers `cam_trans` from the MHR params, derives the 1.2× square
@@ -430,17 +428,14 @@ python tools/parquet_to_npz.py --annotation_dir $ANN_DIR/egohumans_train \
> live in the [dataset setup guides](https://github.com/facebookresearch/sam-3d-body/tree/main/data).
To train on several datasets at once, convert each into its own
-`data/sam3d_gt_<name>/` and either point the notebook at one of them or merge the
-`images/` + `annotations/` folders into a single directory — the filenames are
-prefixed per dataset, so they never collide.
+`data/sam3d_gt_<name>/` under the same `--data_root`; `train_ddp.py --mix` sets
+how often each one is sampled.
## Documentation
-- **[`docs/architecture.md`](docs/architecture.md)** — network design
- (RepViT backbone, 9-query decoder, CLIFF condition).
+- **[`training/README.md`](training/README.md)** — training and ONNX export.
- **[`docs/annotation.md`](docs/annotation.md)** — training data generation
with the SAM3D teacher.
-- **[`docs/training.md`](docs/training.md)** — training and ONNX export.
## License
diff --git a/benchmark/METRICS.md b/benchmark/METRICS.md
deleted file mode 100644
index 2a5eaad..0000000
--- a/benchmark/METRICS.md
+++ /dev/null
@@ -1,293 +0,0 @@
-# What we measure, and in what space
-
-InstantHMR predicts **MHR**. Every published 3DPW and EMDB number is defined on
-**SMPL**. This document explains how we get from one to the other, exactly
-which space each metric lives in, and why the reported numbers are comparable
-to a published table.
-
-Read this before quoting a number, and before adding a new one.
-
----
-
-## 1. The problem in one paragraph
-
-The model outputs 204 MHR model parameters plus 45 identity coefficients. Those
-drive a 127-joint skeleton and an 18,439-vertex mesh. The benchmarks want the
-**24 SMPL joints** (MPJPE, PA-MPJPE) and the **6,890 SMPL vertices** (PVE).
-These are different rigs: different joint definitions, different topology, no
-shared vertex. Nothing can be compared until one side is moved into the other's
-space. **Which side you move is the whole ballgame** — see §5.
-
----
-
-## 2. How MHR becomes SMPL
-
-Three stages. Stage 2 is the one that makes it work.
-
-```
- student ──▶ MHR params (204 + 45)
- │
- │ ① rig forward: identity blendshapes,
- │ pose correctives, linear blend skinning
- ▼
- MHR mesh (18,439 verts)
- │
- │ ② official barycentric surface map
- │ SMPL vertex i = w₁·v_a + w₂·v_b + w₃·v_c
- │ on MHR triangle t(i)
- ▼
- target vertices (6,890, SMPL topology)
- │
- │ ③ labelled fit for θ (72), β (10), t (3)
- │ global orientation solved in closed form
- ▼
- SMPL params ──▶ 24 joints + 6,890 vertices
-```
-
-### ① The rig forward
-
-`MHRRig.vertices` calls the TorchScript rig's own top-level forward, so the
-mesh includes identity blendshapes and pose correctives — the same mesh the
-dataset annotations describe. Not the trainer's skeleton-only path, and not its
-595-vertex training subset.
-
-Note this stage consumes the **45 `shape_params`**. Those move the 127-joint
-skeleton by exactly `0.00e+00 cm`, so every joint metric in this repo is blind
-to them. They *do* move the mesh, which is why PVE is the first number here
-that responds to body shape at all.
-
-### ② The surface map — where the correspondence comes from
-
-`mhr2smpl_mapping.npz` from
-[facebookresearch/MHR](https://github.com/facebookresearch/MHR/tree/main/tools/mhr_smpl_conversion)
-(`tools/mhr_smpl_conversion/assets`, Apache-2.0). For each of the 6,890 SMPL
-vertices it stores **one MHR triangle id and three barycentric weights**. So
-each SMPL vertex is a fixed point on the MHR surface, and pushing a posed MHR
-mesh through it resamples that same surface in SMPL topology.
-
-It is a topology constant — indices and weights, no coordinate frame, no units
-— so it applies in whatever frame you hand it.
-
-**Why not derive it ourselves?** We tried, three ways, and all three failed:
-
-| attempt | residual | why it fails |
-|---|---|---|
-| nearest MHR vertex + mean offset in camera space | 60 mm | the offset rotates with the person; it is not a constant |
-| affine combination of the K nearest MHR vertices | 65 mm, weights to ±405 | a local neighbourhood on a smooth mesh spans a tangent plane, leaving the normal direction unresolved |
-| offset in a local surface frame (normal + tangents) | ~10 mm std | works in principle, but sensitive to under-converged fits upstream |
-
-**Verify the map against the rig before trusting it** (`mhr_smpl.py` does):
-triangle ids top out at 36,871 against the rig's 36,874 faces, barycentric rows
-sum to 1, and the reconstructed SMPL-topology mesh has a mean nearest-vertex
-spacing of **13.88 mm** against real SMPL's **14.37 mm**. A map built for a
-different mesh would scatter neighbouring vertices onto unrelated triangles and
-that number would blow up.
-
-### ③ The fit
-
-Now every SMPL vertex has a known target, so this is a plain labelled
-regression: minimise `‖SMPL(θ,β,t) − target‖²` plus an edge term (edge
-*vectors*, which constrain local shape independently of position, down-weighted
-after 30 % of the schedule so absolute positions drive the endgame).
-
-Two details that are not optional:
-
-- **Correspondence must be known.** Fitting the surface by ICP instead — letting
- nearest-neighbour search discover the pairing — has a local minimum that its
- own residual cannot see: the mesh settles overlapping the target while limbs
- are matched to the **wrong limbs**. Measured: 12 mm surface residual, 76 mm
- PA-MPJPE. The surface looks fine and the pose is wrong.
-- **Global orientation is solved in closed form**, by Procrustes between SMPL's
- rest mesh and the target. Known correspondence makes that a three-line SVD.
- Without it the fit starts at identity and has to walk to an arbitrary
- camera-space orientation through axis-angle: it stalls at **40–51 mm** of
- vertex residual instead of **13 mm**.
-
-Typical fit residual on real predictions: **14–16 mm** surface distance.
-
----
-
-## 3. Which space each metric is computed in
-
-**Everything is computed in SMPL space.** Predictions are converted; the ground
-truth is never touched.
-
-| metric | space | prediction side | GT side | alignment |
-|---|---|---|---|---|
-| **MPJPE** (SMPL24) | SMPL | 24 kinematic joints of the fitted SMPL | 24 kinematic joints of GT SMPL | mid-hip (joints 1 & 2) |
-| **PA-MPJPE** (SMPL24) | SMPL | same | same | Procrustes (rotation + translation + **scale**) |
-| **PVE** | SMPL | 6,890 fitted vertices | 6,890 GT vertices | mid-hip |
-| **PA-PVE** | SMPL | same | same | Procrustes |
-| **J14 (H36M)** | SMPL | `J_regressor_h36m` @ fitted vertices → 14 | `J_regressor_h36m` @ GT vertices → 14 | mid-hip / Procrustes |
-
-Nothing is computed in MHR space. See §5.
-
-### How the 24 joints are obtained — there is no regression
-
-This is the part people expect to be complicated, and it is not. Once SMPL is
-fitted, **both sides come out of the same SMPL model by the same code path**:
-`SMPLModel.forward` returns the 24 kinematic-tree joints carried by the global
-transforms, and it is called on the fitted parameters for the prediction and on
-the GT parameters for the ground truth. No adapter, no learned map, nothing
-fitted per checkpoint.
-
-We use the **kinematic** joints, not `J_regressor @ posed vertices`. That is not
-a shortcut — it is EMDB's own definition. EMDB ships a `kp2d` field, and it
-reprojects the kinematic joints to **0.00 px** and the regressed ones to
-**1.04 px**; in 3D the two sit 3.8 mm apart on average. 3DPW's `jointPositions`
-field is the same kinematic set.
-
-### How the 14 joints are obtained — and why an adapter is not a compromise
-
-3DPW's published protocol is 14 joints in the Human3.6M convention. **There is
-no route from MHR to that convention without a conversion, and no such route
-exists for SMPL-native methods either.** This is the single most important
-thing to be clear about, because it is easy to state backwards.
-
-`J_regressor_h36m.npy` — the file NLF, CameraHMR, SPIN, HMR2.0 and everyone else
-uses — **is itself a fitted linear regressor**, 17 x 6890, mapping SMPL vertices
-to H36M joints. Those methods do not skip a conversion; they use one that ships
-with the benchmark ecosystem, because their meshes are already SMPL. We predict
-MHR, so we need one more step. That is a property of the rig, not a weakness in
-the protocol, and a linear map from our native output to the benchmark's joint
-convention is the same class of object theirs is.
-
-So we report two routes, and **neither is "adapter-free"** — that phrase should
-never appear next to either number:
-
-| row | path | what it needs |
-|---|---|---|
-| **J14 + joint adapter** (primary) | MHR70 keypoints → fixed 14x70 linear map → J14 | the adapter, fitted once on 3DPW *train* from the dataset's own MHR annotations, never from a student, rank-truncated (`RCOND`) |
-| **J14 via mesh fit** | MHR mesh → barycentric map → fitted SMPL → `J_regressor_h36m` → J14 | Meta's surface map + the standard H36M regressor |
-
-Both convert MHR into the H36M convention. They differ only in *where* the
-conversion happens — joint space or mesh space.
-
-Evidence the joint adapter is a rig conversion and not an error sponge, which
-is the only real objection to it:
-
-- it is fitted from **dataset annotations, never from a student**, so it is one
- fixed map for every checkpoint and cannot absorb a particular model's error;
-- it reads anatomically — `right_hip = +1.16 right_hip - 0.32 left_hip`
- (widening SMPL's 120 mm hips toward H36M's 292 mm), `right_knee = +0.84
- right_knee`, nearly identity because the knee is the same point in both;
-- the **SMPL24 version transfers across datasets**: fitted on 3DPW train,
- applied unchanged to EMDB, where it was never fitted. A map that had
- overfitted 3DPW would not survive that.
-
-The mesh route exists because it is the only one that also yields **PVE** and
-the SMPL24 joint set, not because the adapter is suspect. Expect the two J14
-numbers to differ by a few mm; report the adapter row as the headline and the
-mesh row alongside it, each labelled by its path.
-
-## 4. What PVE means (checked against the papers)
-
-**PVE is pelvis-aligned, not Procrustes-aligned.** CameraHMR §4.3:
-
-> "…MPJPE, PA-MPJPE, and PVE (Per Vertex Error), which measures the Euclidean
-> distance (in mm) between predicted and actual 3D vertices and joints **after
-> aligning the pelvis**."
-
-NLF's appendix confirms the naming: *"MVE is analogous to MPJPE but for mesh
-vertices instead of joints, similarly **P-MVE** is the Procrustes-aligned
-version of it."*
-
-So the column to put next to a published PVE is our **`PVE_mm`**, not
-`PA_PVE_mm`. We report PA-PVE too, because it isolates shape from pose, but it
-belongs in a different column and must be labelled `P-MVE` / `PA-PVE` if quoted.
-
-The reference point is the **midpoint of the two hips**, not SMPL joint 0. NLF:
-*"on 3DPW and EMDB the reference point is the midpoint between the two hip
-joints."*
-
----
-
-## 5. The direction rule — read this before inventing a new metric
-
-The conversion runs **prediction → SMPL**. Never the other way for a reported
-number.
-
-Solving an MHR ground truth for EMDB and scoring predictions on MHR joints
-would remove the conversion cost entirely, and it is *not* a comparable number:
-
-1. **MPJPE is not rig-independent.** The published column is defined on SMPL's
- joints and SMPL's vertices. A number on MHR's 70 keypoints is a different
- quantity, not a tighter measurement of the same one.
-2. **40 of MHR's 70 keypoints are finger joints millimetres apart.** Averaging
- error over them drags the mean down regardless of body accuracy.
-3. **The reference would be something we generated**, with an optimiser told to
- look like an MHR model. Whatever MHR cannot represent silently leaves the
- error term.
-
-The MHR ground truth is still worth having — `make_mhr_gt.py` builds and caches
-it — but for the **oracle row**: fit MHR to the GT surface, convert it back
-through the forward path, and score it. That is the best any MHR-rigged model
-could achieve here, and it decomposes the error into rig cost and model error.
-It is an upper bound, labelled as one.
-
----
-
-## 6. Conversion floors — quote these next to results
-
-Measured as a round trip through the rig (GT SMPL → fit MHR to that surface →
-convert back → score against the untouched GT), cold-started at every stage:
-
-| route | MPJPE floor | PA-MPJPE floor | PVE floor |
-|---|---|---|---|
-| linear MHR70 → SMPL24 joint adapter | 25.11 mm | 18.77 mm | not possible |
-| **mesh conversion** | **11.44 mm** | **10.66 mm** | **13.61 mm** |
-
-A floor does not subtract from a result linearly. Treating model error and
-conversion error as independent, dropping the floor from 18.8 to 10.7 mm at a
-~50 mm model error predicts √(50²+18.8²)=53.4 → √(50²+10.7²)=51.1, about 2.3 mm
-— which is what is observed. The mesh route's real win is that **PVE exists**.
-
----
-
-## 7. Running it
-
-One command per dataset. Both write JSON to `benchmark/results/`.
-
-```bash
-# EMDB-1 in SMPL space (24,103 frames, ~25 min on an RTX 4070)
-python benchmark/eval_smpl_fit.py --dataset emdb \
- --ckpt <run>/best_student_model_v3.pth \
- --emdb-root /path/to/EMDB \
- --out benchmark/results/emdb1_smplfit.json
-
-# 3DPW test in SMPL space (35,463 person-frames, ~30 min)
-python benchmark/eval_smpl_fit.py --dataset 3dpw --split test \
- --ckpt <run>/best_student_model_v3.pth \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles \
- --out benchmark/results/3dpw_test_smplfit.json
-```
-
-`--ckpt` takes several checkpoints and evaluates them in one pass; the config
-(`--bound-scales`, `--cliff-focal`) is restored **per checkpoint** from the
-weights and `run_config.json`, so checkpoints with different flags can share an
-invocation safely. Results are written after *each* checkpoint.
-
-The legacy joint-adapter rows come from `eval_emdb_ckpt.py` and
-`eval_3dpw_ckpt.py`; `eval_all.py` runs everything and prints the combined
-table.
-
-### One-time setup
-
-```bash
-# Meta's surface maps -> benchmark/data/mhr_smpl_conversion/
-# {mhr2smpl,smpl2mhr}_mapping.npz from facebookresearch/MHR (Apache-2.0)
-
-# EMDB 24-joint GT cache (7 MB); --check must print ~0.00 px against kp2d
-python benchmark/make_emdb_gt.py --emdb-root /path/to/EMDB \
- --smpl-dir benchmark/data/smpl --check
-
-# 3DPW H36M GT cache, for the J14 adapter rows
-python benchmark/make_3dpw_gt.py --sequence-dir /path/to/3DPW/sequenceFiles \
- --smpl-dir benchmark/data/smpl --check
-
-# MHR ground truth, cached so it is never recomputed (~25 MB each)
-python benchmark/make_mhr_gt.py --dataset emdb --emdb-root /path/to/EMDB
-python benchmark/make_mhr_gt.py --dataset 3dpw --split test \
- --sequence-dir /path/to/3DPW/sequenceFiles
-```
diff --git a/benchmark/README.md b/benchmark/README.md
deleted file mode 100644
index 0c21f96..0000000
--- a/benchmark/README.md
+++ /dev/null
@@ -1,840 +0,0 @@
-# InstantHMR benchmarks
-
-Evaluation harness for publishing numbers on `models/instanthmr_mhr_only_ckpt90.onnx`.
-
-> **A note on the word "teacher" below.** The models under test are trained on
-> the **released ground-truth annotations** of `facebook/sam-3d-body-dataset`,
-> not on teacher inference. Where this file says "teacher labels" or names a
-> file `adapter_*_teacher.npz`, it means *the dataset's own MHR annotations* —
-> the filenames are historical. Only `tools/annotate_dataset.py` produces
-> genuine teacher output, and it writes `data/sam3d_distill_mix/`.
-
-Three benchmarks are implemented — two 3D, one 2D:
-
-| Benchmark | What it measures | Data cost | Status |
-|---|---|---|---|
-| **3DPW** test | MPJPE / PA-MPJPE, the standard in-the-wild 3D HMR metric | 4.6 GB images + 30 MB GT + 500 MB SMPL | ready — published-protocol GT; **check contamination below** |
-| **EMDB-1** | MPJPE / PA-MPJPE on the 24 SMPL joints, the "EMDB (24)" column | 16 GB images (of a 50 GB download) + 7 MB GT | ready — **and genuinely clean**, see below |
-| **COCO** val2017 | OKS AP / PCK, 2D keypoint localisation in the wild | 820 MB, direct download, no registration | ready — clean |
-
-## Why these three
-
-**3DPW** is the benchmark essentially every monocular HMR paper reports, so it
-is the one that makes your numbers comparable. The GT pickles store
-`jointPositions` (24 SMPL joints in world coordinates) directly, which is
-tempting and wrong — published numbers use SMPL run *forward* on the GT
-`poses`/`betas` with the Human3.6M regressor applied to the vertices. The two
-references are **46 mm apart in J12 PA-MPJPE**, so the harness builds the real
-one once with `make_3dpw_gt.py` and scores against that.
-
-**EMDB-1** is the newer of the two 3D benchmarks and the one this repo can
-report honestly. Every recent paper carries an "EMDB (24)" column (SAM 3D Body
-Table 2, NLF Table 3, Fast SAM 3D Body Table 1), it is out-of-domain for almost
-all of them, and — unlike 3DPW — **no EMDB frame appears in any corpus this
-repo trains on**, so it is the only 3D number here with no contamination
-caveat attached. It is structurally a simpler 3DPW: one subject per sequence,
-provided person boxes, real calibration.
-
-**COCO val2017** needs no registration at all and the joint mapping is free:
-MHR70 indices 0–14 already follow COCO's ordering, with the wrists at 41/62.
-
-### The ones deliberately skipped
-
-- **Human3.6M** — the licence forbids redistribution, the download is manual and
- slow, and protocol variants (P1 vs P2, 17 vs 14 joints) differ per paper. High
- effort, high chance of an unfair comparison.
-- **RICH** — the reason to skip it is not the size alone. Its GT is **SMPL-X**,
- so it needs the SMPL-X models *and* a second forward pass (different joint
- count, hand and face parameters) that none of our SMPL code covers; the test
- split alone is 125K images at 4K, against EMDB-1's few thousand; and NLF
- *trains* on RICH (the ∗ in SAM 3D Body's Table 2), so the column is not
- out-of-domain for every method in it.
-- **MPII** — 2D only, so it adds little once COCO is in.
-
-### EMDB-1 — the joint convention, settled against the data
-
-Use **EMDB-1** (17 sequences, 24,323 frames), the camera-frame pose split every
-paper reports. EMDB-2 (25 sequences) is global-trajectory and not what this
-model does. Three sequences are in both; `benchlib/emdb.py` selects on the
-pickles' own `emdb1` flag and asserts it finds exactly 17.
-
-It is structurally a simpler 3DPW, one subject per sequence:
-
-| EMDB pkl | 3DPW equivalent |
-|---|---|
-| `smpl{poses_root (N,3), poses_body (N,69), betas (10,), trans}` | `poses` (N,72), `betas`, `trans` |
-| `camera{intrinsics, extrinsics (N,4,4), width, height}` | `cam_intrinsics`, `cam_poses` |
-| `good_frames_mask` | `campose_valid` |
-| `bboxes` (N,4) xyxy — **provided** | derived from projected joints |
-
-`betas` is 10-dim, exactly the count the published protocol uses, so
-`make_3dpw_gt.py`'s forward pass drops in unchanged, and EMDB's `gender` field
-picks the same gendered pair 3DPW needs.
-
-**The protocol is the 24 SMPL joints, mid-hip aligned — not the H36M
-regressor.** NLF's Table 3 reads "Results on EMDB1 (24j)" and its metrics
-section states that "on 3DPW and EMDB the reference point is the midpoint
-between the two hip joints"; SAM 3D Body's Table 2 column header is "EMDB (24)".
-
-**Which 24, though — and which way the extrinsics point — came from the data,
-not the docs.** Both would have produced quietly wrong numbers rather than a
-crash, and EMDB's own `kp2d` field settles both at once, because it is the 24
-joints already projected:
-
-| hypothesis | reprojection error vs `kp2d` |
-|---|---|
-| `extrinsics` as world→camera, **kinematic** joints | **0.00 px** ✅ |
-| `extrinsics` as world→camera, `J_regressor @ vertices` | 1.04 px |
-| `inv(extrinsics)` as world→camera | 3477 px, every joint behind the camera |
-
-So `extrinsics` is world→camera (3DPW's sense), and "EMDB (24)" means the SMPL
-**kinematic-tree** joints — `smpl_forward`'s second output — not the regressed
-ones, which sit 3.8 mm away on average and 19.3 mm at worst. This is the same
-distinction that bites on 3DPW, pointing the other way: there the published GT
-*is* the regressed one.
-
-`make_emdb_gt.py --check` re-runs that comparison over all 24,323 frames and
-reports **0.0024 px mean / 0.095 px max**, which certifies the SMPL forward, the
-gendered model choice, the beta padding and the world frame in one number.
-
-**The MHR70 → SMPL-24 adapter is required, not optional.** Ten of SMPL's 24
-joints — the three spines, the collars, the feet, the hands — have no MHR
-landmark at all, so there is no raw row to fall back on.
-`benchmark/results/adapter_smpl24_teacher.npz` is the student-free conversion,
-fitted by `fit_adapter_mhr.py --target smpl24` from 22,209 annotation/GT pairs
-on 3DPW **train** (where `jointPositions` **is** the SMPL-24 kinematic set,
-verified to 0.001 mm). EMDB is a different dataset, so it is a real holdout.
-It reads anatomically: `pelvis = 0.38 right_hip + 0.34 left_hip`, `left_knee =
-0.93 left_knee`, `left_ankle` from heel + ankle + toe.
-
-**The conversion floor: 25.11 mm MPJPE / 18.77 mm PA-MPJPE** on a held-out half.
-That is what an MHR-rigged model carries on a 24-joint benchmark before it makes
-a single mistake, using this linear conversion. Quote it next to the result: the table it joins runs 61.7 (SAM 3D Body) to 118.5 (HMR2.0b)
-MPJPE, so ~25 mm of rig overhead is material.
-
-### Two routes into SMPL space, and why the mesh one wins
-
-The linear adapter above converts *joints*. `benchmark/eval_smpl_fit.py`
-converts the **mesh** instead — the route SAM 3D Body and Fast SAM 3D Body use
-for this same rig — which lowers the conversion floor ~2x and is the only way
-to reach PVE at all:
-
-| conversion | MPJPE floor | PA-MPJPE floor | PVE floor |
-|---|---|---|---|
-| linear MHR70 → SMPL24 adapter | 25.11 | 18.77 | not possible |
-| **mesh fit** (`eval_smpl_fit.py`) | **11.44** | **10.66** | **13.61** |
-
-Measured as a round trip through the rig: GT SMPL → fit MHR to that surface →
-convert back → score against the untouched GT. Every stage is cold-started, so
-this is the floor the evaluator actually operates at, not a best case.
-
-**The direction is the part that makes it publishable.** The prediction is
-fitted into SMPL and the ground truth is left exactly as the benchmark defines
-it, so MPJPE/PA-MPJPE/PVE mean what they mean in every published table. The
-tempting inverse — solving an MHR ground truth for EMDB and scoring in MHR
-space — removes the conversion entirely and is **not** a comparable number: it
-changes the joint set (MPJPE is not rig-independent), it averages error over 40
-finger joints millimetres apart, and it scores against a reference we generated
-with an optimiser told to look like an MHR model, so whatever MHR cannot
-represent silently leaves the error term. Keep that as a diagnostic if you want
-it; do not put it in the table.
-
-**Three components, and the one that was hard.**
-
-1. The student's MHR parameters → the rig's own 18,439-vertex mesh.
-2. **Meta's official barycentric surface map**
- (`facebookresearch/MHR`, `tools/mhr_smpl_conversion/assets`, Apache-2.0) →
- the same surface resampled in SMPL topology. Copy
- `{mhr2smpl,smpl2mhr}_mapping.npz` into
- `benchmark/data/mhr_smpl_conversion/`.
-3. A labelled fit for SMPL `theta`/`beta`/`trans`.
-
-Step 2 is what makes step 3 tractable, and it is worth being explicit about
-why. Fitting the surface by ICP has to *discover* the correspondence, and its
-search has a local minimum that its own residual cannot detect: the mesh
-settles overlapping the target while limbs are matched to the **wrong limbs**,
-which measured a healthy-looking 12 mm surface residual alongside 76 mm
-PA-MPJPE. Three attempts to build a correspondence here all failed — a mean
-offset in camera space is not pose-invariant (60 mm), an affine combination of
-the K nearest MHR vertices is ill-conditioned because a local neighbourhood on
-a smooth mesh spans a tangent plane and leaves the normal unresolved (65 mm,
-weights reaching 405), and a surface-frame offset is sensitive to
-under-converged fits. With the official map, correspondence is **known** —
-vertex i pairs with vertex i — and the failure mode does not exist.
-
-Verify the map against this rig before trusting it: triangle ids top out at
-36,871 against the rig's 36,874 faces, barycentric rows sum to 1, and the
-reconstructed SMPL-topology mesh has a mean nearest-vertex spacing of 13.88 mm
-against real SMPL's 14.37 mm. A map built for a different mesh would scatter
-neighbouring vertices onto unrelated triangles.
-
-**PVE is the first metric in this repo that can see body shape at all.** The
-45 MHR `shape_params` move the 127-joint skeleton by exactly `0.00e+00 cm`, so
-every joint metric here — MPJPE, PA-MPJPE, the adapter rows, the training
-losses — is blind to identity by construction (see the note in `CLAUDE.md`).
-The mesh conversion feeds those parameters into `MHRRig.vertices`, so they
-reach the SMPL surface and PVE responds to them. If `--w-verts` is doing
-anything, this is the benchmark row where it shows up.
-
-One more detail in step 3: **solve the global orientation in closed form.**
-Known correspondence makes that a plain Procrustes between SMPL's rest mesh and
-the target. Without it, Adam starts from identity and has to walk the whole way
-through axis-angle, and the same fit stalls at 40-51 mm of vertex residual
-instead of 13 mm.
-
-## Setup
-
-```bash
-# COCO — fully automatic (~820 MB)
-python benchmark/download.py --coco
-
-# 3DPW — accept the licence at
-# https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html
-# then unzip sequenceFiles.zip (and imageFiles.zip if you don't have it) and check:
-python benchmark/download.py --check-3dpw \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles
-```
-
-Then build the ground truth. This needs two files that cannot be redistributed
-here — put both in `benchmark/data/smpl/` (gitignored):
-
-| file | where |
-|---|---|
-| `SMPL_{MALE,FEMALE}.pkl` | `smpl.is.tue.mpg.de`, free research licence. 3DPW's `genders` field is per-subject, so the gendered pair is required. |
-| `J_regressor_h36m.npy` | `visiondata.cis.upenn.edu/spin/data.tar.gz` (SPIN); also ships inside HMR2.0 / 4D-Humans |
-
-```bash
-python benchmark/make_3dpw_gt.py \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --smpl-dir benchmark/data/smpl --check
-```
-
-That writes `<3DPW root>/gt_h36m/{test,validation,train}.npz` — 17 H36M joints
-per person-frame in the same world frame as `jointPositions`, 14 MB for all
-three splits. `--check` additionally regresses the SMPL kinematic joints and
-compares them to `jointPositions`: **0.001 mm max over 74,620 person-frames**,
-which is what certifies the forward pass, the gendered model choice and the
-world frame all the way through.
-
-Then fit the MHR70 → J14 adapter once. It is student-free and dataset-wide, so
-this is a one-time step, not something to repeat per checkpoint:
-
-```bash
-python benchmark/fit_adapter_mhr.py \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --target j14 --out benchmark/results/adapter_j14_h36m_teacher.npz
-```
-
-### EMDB
-
-Registration is a form at `emdb.ait.ethz.ch`, approved by email; the download is
-ten per-subject zips, 50 GB. **Only EMDB-1 is needed** — 17 of the 81 sequences,
-16 GB of images — so unpack selectively rather than extracting all ten zips:
-
-```python
-# python3, from the download directory
-import zipfile, pickle
-out = '/path/to/EMDB'
-for p in range(10):
- z = zipfile.ZipFile(f'P{p}.zip')
- for n in z.namelist(): # pass 1: the annotations
- if n.endswith('.pkl'):
- z.extract(n, out)
-want = {'/'.join(f.split('/')[-3:-1])
- for f in __import__('glob').glob(out + '/P*/*/*.pkl')
- if pickle.load(open(f, 'rb'))['emdb1']}
-for p in range(10): # pass 2: only EMDB-1 frames
- z = zipfile.ZipFile(f'P{p}.zip')
- for i in z.infolist():
- if i.filename.endswith('.jpg') and '/'.join(i.filename.split('/')[:2]) in want:
- z.extract(i, out)
-```
-
-Then build the GT — the same gendered SMPL models as 3DPW, nothing new to fetch:
-
-```bash
-python benchmark/make_emdb_gt.py --emdb-root /path/to/EMDB \
- --smpl-dir benchmark/data/smpl --check
-```
-
-That writes `<EMDB root>/gt_smpl24/emdb1.npz` (7 MB, 24 world joints per frame).
-`--check` must print a reprojection error of ~0.00 px against EMDB's `kp2d`; a
-larger number means the extrinsics or the joint convention moved.
-
-The MHR70 → SMPL-24 adapter is fitted once, from 3DPW train, student-free:
-
-```bash
-python benchmark/fit_adapter_mhr.py \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --target smpl24 --out benchmark/results/adapter_smpl24_teacher.npz
-```
-
-## Running
-
-```bash
-# COCO val2017 (~10 min on an RTX 4070)
-python benchmark/eval_coco.py \
- --onnx models/instanthmr_mhr_only_ckpt90.onnx \
- --coco-root benchmark/data/coco
-
-# 3DPW test — add --stride 10 for a 10x faster smoke run
-python benchmark/eval_3dpw.py \
- --onnx models/instanthmr_mhr_only_ckpt90.onnx \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles \
- --exclude-seen data/sam3d_distill_mix/annotations data/sam3d_gt_3dpw/annotations
-
-# same thing for a training checkpoint instead of an exported graph. --adapter
-# adds the J14+adapter row, which is the one to quote.
-python benchmark/eval_3dpw_ckpt.py \
- --ckpt instanthmr_distill_train/runs/*/*/best_student_model_v3.pth \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles --split test \
- --adapter benchmark/results/adapter_j14_h36m_teacher.npz
-```
-
-```bash
-# EMDB-1, all 17 sequences (~4 min on an RTX 4070). --adapter defaults to the
-# committed SMPL-24 map, so the short form is the full protocol.
-python benchmark/eval_emdb_ckpt.py \
- --ckpt instanthmr_distill_train/runs/*/*/best_student_model_v3.pth \
- --emdb-root /path/to/EMDB \
- --out benchmark/results/emdb1.json
-
-# ...and again with EMDB's own person boxes, which is what published
-# "Oracle: annotated bounding boxes" rows use. State which one you quote.
-python benchmark/eval_emdb_ckpt.py --ckpt <ckpt> --emdb-root /path/to/EMDB \
- --bbox annotated
-```
-
-```bash
-# EMDB-1 in SMPL space via the mesh conversion: MPJPE, PA-MPJPE and PVE.
-# ~25 min for all 24,103 frames on an RTX 4070 (fit is 16 min of that).
-python benchmark/eval_smpl_fit.py \
- --ckpt instanthmr_distill_train/runs/*/*/best_student_model_v3.pth \
- --emdb-root /path/to/EMDB --fit-batch 128 \
- --out benchmark/results/emdb1_smplfit.json
-```
-
-All four write a JSON report to `benchmark/results/`.
-
-### The evaluator configures itself from the checkpoint (fixed 2026-09-06)
-
-`eval_3dpw_ckpt.py` used to build a **default** `DistillConfig`, which silently
-disagreed with any run that used one of the 2026-09-06 flags:
-
-* `--bound-scales` puts a `tanh` remap of the 77 body-size parameters inside
- `forward()`. Exporting or scoring without it reads the head's raw pre-`tanh`
- numbers as bone scales — **21.0 mm MPJPE / 14.4 mm PA-MPJPE** of corruption,
- measured on 64 COCO teacher targets.
-* `--cliff-focal` changes what the conditioning vector *means*, not the graph,
- so feeding the wrong form raises nothing.
-
-Both are now restored per checkpoint by `T.config_from_checkpoint()`: the
-bounds from the weights (`scale_lo` / `scale_hi` are the flag), the
-conditioning form from the `run_config.json` beside the checkpoint. The script
-prints what it inferred:
-
-```
-[cfg] bno_s0/best_student_model_v3.pth: backbone=repvit_m2_3 bound_scales=True cliff_focal=True [run_config.json]
-[3DPW] test / GT=h36m: 35,463 person-frames (stride 1), CLIFF conditioning angular (real per-sequence focal)
-```
-
-Check that line. `cliff_focal=False [DEFAULT ...]` on a run you launched with
-`--cliff-focal` means `run_config.json` did not travel with the checkpoint —
-pass `--cliff-focal` by hand. The checkpoint load is **strict**, so a bounded
-checkpoint scored through the wrong config raises instead of quietly producing
-a number. Checkpoints that disagree on the conditioning cannot share one
-invocation, because the crops are built once; the script says so and exits.
-
-The focal itself comes from 3DPW's own calibration (`cam_intrinsics`,
-`K[1,1]` = 1969.23 px on 10 of the 12 validation sequences, 1961.85 on the two
-landscape ones) — not a heuristic. The naive `sqrt(H^2+W^2)` stand-in would be
-2202.9 px, 12% high. `eval_3dpw.py` (the ONNX path) now passes the same
-per-frame focal through `benchlib/runner.py`.
-
-## Current numbers
-
-### 3DPW (2026-09-05)
-
-3DPW **test**, all 35,463 person-frames, published-protocol GT, the two Jean Zay
-checkpoints pulled at ~epoch 130. Full reports in
-`benchmark/results/3dpw_test_{b3_s1,v3_s1}_teacher_adapter.json`.
-
-| run | preset | J12 PA | J14 PA | **J14+adapter PA** | J14+adapter MPJPE |
-|---|---|---|---|---|---|
-| b3_s1 | baseline | 62.49 | 67.89 | **41.83** | 69.03 |
-| v3_s1 | v2 | 63.20 | 69.05 | **43.23** | 70.59 |
-
-**Quote the adapter row.** Raw J14 is dominated by convention, not quality:
-per-joint it is 131 mm at `head` and 98/87 mm at the hips against ~47 mm at the
-knees.
-
-Two caveats before this goes in a table. The 1.4 mm gap between the two runs is
-**not** a result — measured seed spread in this repo is far wider, so
-baseline-vs-v2 needs 2-3 seeds per arm through the same path. And these are the
-only numbers here that are genuinely held out: both checkpoints were trained on
-Jean Zay, whose corpus is coco/mpii/aic/sa1b/harmony4d with `3dpw_train`
-annotations-only, so all 24 test sequences are unseen. The local `data/` corpus
-is **not** clean — see the contamination section.
-
-### EMDB-1 (2026-09-07)
-
-All 17 EMDB-1 sequences, all 24,103 evaluatable frames (24,323 total, minus 206
-where `good_frames_mask` is 0 and 14 where the subject is under 60 % inside the
-frame), `b3_s1` at epoch 137 — the same checkpoint as the 3DPW table above.
-Reports in `benchmark/results/emdb1_b3_s1_{gt-joints,annotated}.json`.
-
-| person box | **SMPL24+adapter PA** | SMPL24+adapter MPJPE | PCK@50 | PCK@100 |
-|---|---|---|---|---|
-| projected GT joints ×1.2 (this repo's 3DPW convention) | **53.82** | 79.90 | 0.632 | 0.900 |
-| EMDB's own `bboxes` (published "Oracle" protocol) | **60.69** | 87.51 | 0.582 | 0.863 |
-| *rig conversion floor (teacher, held-out half)* | *18.77* | *25.11* | | |
-
-Where that sits in the published EMDB (24) table — Fast SAM 3D Body Table 1,
-which is the most recent one carrying every baseline:
-
-| method | PA-MPJPE | MPJPE |
-|---|---|---|
-| Fast SAM 3D Body (oracle) | 37.3 | 64.3 |
-| SAM 3D Body (oracle) | 38.2 | 61.7 |
-| NLF-L +fit | 40.9 | 68.4 |
-| PromptHMR | 41.0 | 71.7 |
-| GENMO | 42.5 | 73.0 |
-| CameraHMR | 43.3 | 70.3 |
-| TRAM | 45.7 | 74.4 |
-| WHAM | 50.4 | 79.7 |
-| **InstantHMR b3_s1** | **53.8 / 60.7** | **79.9 / 87.5** |
-| SMPLer-X-H | 64.5 | 92.7 |
-| HMR2.0b | 79.2 | 118.5 |
-
-Read that with the 18.8 mm conversion floor in mind: those baselines all predict
-SMPL directly and pay no rig-conversion cost, and the floor already contains the
-error in the dataset's own MHR fits. It is not separable from the model's error without perfect
-MHR ground truth, so the honest statement is "53.8 mm, of which up to 18.8 mm is
-the MHR→SMPL conversion", not a corrected number.
-
-**The box convention is worth 6.9 mm, so state it.** EMDB's own boxes are 17.5 %
-larger than the projected-joint boxes (mean crop side 1248 vs 1069 px) and their
-centre sits 5.7 % of a crop away, so the person lands ~15 % smaller and
-off-centre in a 224² crop the model was trained to see filled. That is framing
-sensitivity, not a modelling difference — but it means the row that belongs next
-to the published "Oracle" numbers is the 60.69 one, and the 53.82 one is what
-compares to this repo's own 3DPW numbers.
-
-**The whole EMDB path was cross-checked against 3DPW.** Scoring the same
-checkpoint on 3DPW test under the *same* GT convention (`--gt jointpositions`,
-i.e. SMPL kinematic joints) gives J14 PA 55.81 / J12 PA 49.26, against EMDB-1's
-60.17 / 55.55 on the identical rows. A 4-6 mm gap is what a harder dataset
-looks like — EMDB has handstands, cartwheels and dancing where 3DPW mostly
-walks. A wrong extrinsics sense or a mismatched joint convention would show up
-as hundreds of millimetres, not four.
-
-**EMDB is the one clean 3D number here.** The corpus is coco / mpii / aic / sa1b
-/ harmony4d / 3dpw; no EMDB frame appears in any of it, at any split. Contrast
-the 3DPW section below, where the local corpus contains crops from all 24 test
-sequences.
-
-Hardest sequences are the ones you would guess: `P5_42_indoor_dancing` (75.4 mm)
-and `P8_68_outdoor_handstand` (75.0 mm), against 44.5 mm on `P1_14_outdoor_climb`.
-Per-joint, the error is concentrated in the extremities the adapter has to
-extrapolate — `left/right_hand` at 97-98 mm and the wrists at 79-82 mm, against
-30-40 mm at the collars, neck, pelvis and hips.
-
-### `--fit-iters 400` is too low by ~4x — every number below it is inflated (2026-09-11)
-
-**The default was not converged**, and it is the reason this repo carried three
-mutually inconsistent conversions. On 3DPW test the fit leaves **16.13 mm** of
-surface residual at 400 iterations and **7.96 mm** at 4000; the metrics move
-with it, by far more than any model change on the roadmap:
-
-| fit-iters | residual | PVE | J14 PA | SMPL24 PA |
-|---|---|---|---|---|
-| 400 | 16.13 | 82.34 | 45.95 | 49.16 |
-| 1500 | 8.94 | 80.96 | 42.71 | 46.56 |
-| 4000 | 7.96 | 81.01 | 42.33 | 45.93 |
-
-Worse, **the sign is not predictable**: on EMDB, PVE gets *worse* as the fit
-converges (92.45 -> 93.84 -> 94.07) because at 400 iterations the fit is
-accidentally regularised toward the SMPL mean shape. A number you cannot
-defend without saying how long Adam ran is not a protocol. Use >= 1500, or
-skip the fit entirely (next section). The EMDB table two sections down was
-produced at 400 and is inflated.
-
-### The conversion, settled: one surface, one operator on both sides
-
-`barycentric_transfer` already returns the prediction **in SMPL topology**,
-vertex i <-> SMPL vertex i. So the SMPL parameter fit is not needed to reach
-either PVE or a regressed joint set — it exists only to recover the 24
-*kinematic* joints, which come off the kinematic chain. Measured on identical
-predictions, a converged fit converges *toward* the fit-free answer, so all it
-adds in those cases is a hyperparameter with a 3-4 mm blast radius.
-
-The protocol, in order of what the ground truth actually *is*:
-
-| metric | GT is | conversion |
-|---|---|---|
-| **PVE / PA-PVE** | vertices | none — compare directly |
-| **3DPW J14** | `J_regressor_h36m @ GT verts` | same regressor on **both** sides, so its bias cancels exactly |
-| **EMDB SMPL-24** | kinematic joints | a vertex regressor approximates these; costs ~1.1 mm (see below) |
-
-Nothing here is fitted by us: only the rig, Meta's published barycentric map,
-and the benchmark's own regressor. There is no optimiser and no free parameter,
-so the failure above cannot recur.
-
-**Implemented 2026-09-22 as `--fit-iters 0`.** It reports `PVE_direct_mm`,
-`PA_PVE_direct_mm` and `J14_h36m_direct_{MPJPE,PA_MPJPE}_mm`, rooted at the
-H36M pelvis regressed from each surface by the same operator. The old `PVE_mm`,
-`PA_PVE_mm`, `MPJPE_mm`, `PA_MPJPE_mm` and `J14_h36m_viamesh_*` keys keep their
-old meaning — the *fitted* mesh, and `null` when no fit ran — so every row
-recorded before that date stays comparable. The fit-free rows are computed on
-every run, fitted or not, and are identical either way; on g8h_s0 / EMDB-1
-stride 10 they came out the same to the decimal under `--fit-iters 0` and
-`--fit-iters 1500`, which is the check that the two paths share one surface.
-
-What the fit costs, same checkpoint and frames: PVE 86.29 fit-free against
-89.29 at 1500 iterations (9.30 mm residual), and J14/H36M PA-MPJPE 46.70
-against 45.66. The fit still buys the 24 SMPL **kinematic** joints, which the
-surface cannot give, so `--fit-iters >= 1500` remains the only route to the
-EMDB SMPL-24 row.
-
-**The one approximation, quantified.** EMDB's 24 kinematic joints are not a
-linear function of posed vertices. Regressing them from the surface carries, on
-EMDB GT bodies, **10.15 mm mid-hip / 7.02 mm Procrustes** of bias. Errors add in
-quadrature, so at a ~50 mm error level that costs **~1.1 mm** of PA-MPJPE:
-sqrt(50.73^2 + 7.02^2) = 51.2 against 51.82 measured. Quote it; do not
-reach for the fit to remove it unless you run >= 1500 iterations.
-
-**Conversion floor, measured on EMDB (2026-09-11).** `make_mhr_gt.py`'s cached
-MHR fit of EMDB's own GT, pushed through this exact forward path — the oracle
-row, the best any MHR-rigged model could score:
-
-| | MPJPE | PA-MPJPE | PVE |
-|---|---|---|---|
-| oracle (conversion floor) | 18.64 | **12.36** | 19.92 |
-
-The oracle's own surface fit leaves 9.40 mm, so this is an **upper bound** on
-conversion cost, not a tight estimate. At a 51.8 mm result, a 12.4 mm floor
-contributes ~1.5 mm in quadrature: **conversion is not what separates this
-model from SOTA.** Do not subtract it — errors do not add linearly.
-
-### The EMDB person-box convention: quote `--bbox annotated` for published tables
-
-Not because it scores better — it scores **8-12 mm worse**, and it changes the
-ranking. Measured over all 24,103 frames, PA-MPJPE penalty for switching from
-projected-GT-joint boxes to EMDB's own:
-
-| run | gt-joints | annotated | penalty |
-|---|---|---|---|
-| b3_s1 | 53.16 | 60.15 | +6.99 |
-| g6bv_s | 52.01 | 60.33 | +8.32 |
-| g6b_s | 54.78 | 63.30 | +8.52 |
-| g6v_s (v2) | 52.50 | 64.36 | **+11.86** |
-| g6vv_s (v2) | 51.81 | 63.81 | **+12.00** |
-
-Published "Oracle: annotated bounding boxes" rows use EMDB's own boxes, so that
-is the only row that may sit beside them. Picking the cheaper convention while
-comparing against papers that used theirs is the same error as picking a
-flattering joint conversion.
-
-**`--preset v2` is markedly more box-sensitive** (~12 mm against ~8.4 mm), which
-reverses the ranking: g6vv_s is best under projected boxes and *fourth* under
-EMDB's. v2 sets `cliff_follows_aug`, so the model leans harder on the CLIFF
-vector describing its crop; EMDB's boxes are 17.5 % larger and 5.7 % off-centre
-against the padded boxes it trained on. Truncation augmentation does not cover
-this mismatch. Check both conventions before claiming a v2 win.
-
-### EMDB-1, SMPL space via mesh conversion (2026-09-08)
-
-All 17 sequences, all 24,103 frames, projected-GT-joint boxes, `--fit-iters 400`
-— **inflated, see the fit-iters section above; kept for provenance only.**
-Report in `benchmark/results/emdb1_smplfit_gt-joints.json`.
-
-| run | epoch | MPJPE | **PA-MPJPE** | PVE | PA-PVE | mean fit residual |
-|---|---|---|---|---|---|---|
-| b3_s1 (baseline) | 137 | 77.38 | **52.00** | 92.25 | 62.57 | 14.61 |
-| bno_s0 (v2, `--cliff-focal --bound-scales`) | 85 | 79.55 | **54.23** | 94.44 | 64.82 | 14.72 |
-| *conversion floor (round trip)* | | *11.44* | *10.66* | *13.61* | | |
-
-**The two runs are not comparable to each other.** bno_s0 is at epoch 85 against
-b3_s1's 137, so this is a maturity difference as much as a configuration one —
-and with one seed per arm it would prove nothing even at matched epochs (seed
-spread in this repo is 34-49 mm on the 70-keypoint metric).
-
-Against the published EMDB (24) table (Fast SAM 3D Body Table 1), b3_s1 lands
-next to **WHAM** — ahead of it on MPJPE (77.4 vs 79.7) and PVE (92.3 vs 94.4),
-just behind on PA-MPJPE (52.0 vs 50.4) — and well ahead of SMPLer-X-H (64.5 PA)
-and HMR2.0b (79.2 PA). WHAM is a *video* method; this is single-frame.
-
-**The mesh route beats the linear adapter on 16 of 17 sequences**, mean
-−1.73 mm, with per-sequence Pearson r = 0.989 and Spearman r = 0.961 between
-the two. Two conversions sharing no machinery agreeing that closely on the
-ranking is the evidence that neither distorts the result. The gain is largest
-where the pose is unusual — `outdoor_sitting` −4.9 mm, `outdoor_climb`
-−3.8 mm — which is where a fixed linear joint map is furthest from its fitting
-distribution.
-
-Note the headline gain is only ~1.8 mm even though the conversion floor drops
-18.8 -> 10.7 mm. That is arithmetic, not disappointment: floors do not subtract
-linearly. Treating model and conversion error as independent, sqrt(50^2 +
-18.8^2) = 53.4 against sqrt(50^2 + 10.7^2) = 51.1, so ~2.3 mm predicted against
-1.8 mm observed. The real win is that **PVE exists at all**.
-
-### Generation 6 — the `preset x --w-verts` 2x2 (2026-09-11)
-
-Scored through the fit-free conversion above, all frames, `best_student_model_v3.pth`
-(selected on 3DPW validation). All four are seed 0, `--losses rebalanced
---cliff-focal --bound-scales --anomaly-safe-fallback --crop-centre-fix`.
-
-**3DPW test** — 35,463 person-frames, J14/H36M, published-protocol GT:
-
-| run | preset | w_verts | ep | MPJPE | PA-MPJPE | PVE | PA-PVE | J14+adapter PA |
-|---|---|---|---|---|---|---|---|---|
-| g6vv_s | v2 | 0.35 | 239 | 67.71 | **42.69** | 81.01 | **57.68** | **41.77** |
-| g6v_s | v2 | 0 | 225 | **67.27** | 43.16 | **80.57** | 58.07 | 42.21 |
-| g6bv_s | baseline | 0.35 | 247 | 70.31 | 43.30 | 83.51 | 58.51 | 42.55 |
-| g6b_s | baseline | 0 | 102 | 73.98 | 45.27 | 87.92 | 61.07 | 44.44 |
-| b3_s1 (gen5) | baseline | 0 | 137 | 69.10 | 43.03 | 82.81 | 58.31 | 41.81 |
-| bno_s0 (gen5) | v2 | 0 | 85 | 69.84 | 44.80 | 83.39 | 60.04 | 43.78 |
-
-**EMDB-1** — 24,103 frames, SMPL-24 kinematic, projected-GT-joint boxes (for the
-`--bbox annotated` row that belongs beside published tables, see above):
-
-| run | preset | w_verts | ep | MPJPE | PA-MPJPE | PVE | PA-PVE |
-|---|---|---|---|---|---|---|---|
-| g6vv_s | v2 | 0.35 | 239 | **78.71** | **51.81** | **94.71** | **61.67** |
-| g6bv_s | baseline | 0.35 | 247 | 79.20 | 52.01 | 95.01 | 61.69 |
-| g6v_s | v2 | 0 | 225 | 79.44 | 52.50 | 95.29 | 62.68 |
-| g6b_s | baseline | 0 | 102 | 83.31 | 54.78 | 99.82 | 64.30 |
-| b3_s1 (gen5) | baseline | 0 | 137 | 78.01 | 53.16 | 94.17 | 63.66 |
-| bno_s0 (gen5) | v2 | 0 | 85 | 80.55 | 55.36 | 96.82 | 65.88 |
-
-**`--preset v2` is the win, and it lands where the mechanism says it should.**
-At nearly matched epochs (g6bv_s 247 vs g6vv_s 239), 3DPW MPJPE drops **2.60 mm**
-and PVE **2.50 mm** while PA-MPJPE moves only 0.61 mm. That is the signature of
-v2's absolute-pose supervision — `w_reproj` 0.01 -> 0.5 and the unsquared
-Euclidean `loss_cam` fix *where the body sits*, and Procrustes discards exactly
-that. Read it next to the box-sensitivity finding above before shipping it.
-
-**`--w-verts 0.35` does not deliver what it was added for.** At matched epochs it
-moves PVE by -0.58 mm on EMDB and **+0.44 mm on 3DPW** — the wrong sign, net
-~zero. PVE is the only metric that can see the 45 `shape_params`, so this was
-the row where the surface term should have shown up. It did not, at one seed.
-
-**g6b_s is not a fair member of the 2x2.** Its best checkpoint is epoch 102
-against the others' 225-247 and its run died at 171 of a 300-epoch
-`OneCycleLR`, so it never annealed. Its ~2 mm deficit is maturity.
-
-**One seed per arm.** Every gap above except the v2 MPJPE/PVE effect is under
-1 mm, well inside seed spread. Only the 2.5 mm v2 result is safe to act on.
-
-### What these metrics cannot see
-
-Both headline metrics are alignment-invariant, and that hides exactly the
-failure a demo shows first. PA-MPJPE removes translation, rotation **and**
-scale; root-relative MPJPE removes translation. So camera-translation drift,
-systematic body-size error and frame-to-frame jitter are all invisible here —
-a change can be worth shipping on stability and move PA-MPJPE by 0.1 mm.
-
-3DPW is video, so measure stability directly on consecutive frames:
-
-- **rigid bone lengths** within a person track. A femur cannot change length,
- so its variance is pure prediction noise. GT floor: 1.06%.
-- **`cam_trans` acceleration**, `|t[i+1] - 2t[i] + t[i-1]|`. GT floor: 7.0
- mm/frame².
-
-Measured on 3DPW test (35,463 person-frames, J14 + teacher adapter):
-
-| checkpoint | PA | MPJPE | bone-len CV | cam accel | (z only) |
-|---|---|---|---|---|---|
-| `distill_mhr_only_ckpt90` (old corpus, *contaminated*) | 41.14 | 68.85 | 1.65% | 148.1 | 144.3 |
-| b2_s0 | 44.84 | 72.69 | 1.96% | 234.4 | 229.9 |
-| b3_s1 | 41.83 | 69.03 | 1.85% | 219.6 | 215.9 |
-| v3_s1 | 43.23 | 70.59 | 1.79% | 224.6 | 222.1 |
-| ground truth | — | — | 1.06% | 7.0 | |
-
-Every model is 21-33x jitterier than ground truth, and the rebuilt corpus made
-it ~50% worse — traced to the focal spread and addressed by `--cliff-focal`
-(see `instanthmr_distill_train/README.md`). The legacy row is optimistic on
-both axes: it trained on all 60 3DPW sequences.
-
-## Protocols, stated precisely
-
-Write these into the paper — they are the details a reviewer will ask about.
-
-**3DPW test.** All 24 test sequences, every frame where `campose_valid` is set
-and the person is at least 60 % inside the image. Person boxes come from the
-projected GT joints padded by `--bbox-scale` (default 1.2), so no detector is in
-the loop. MPJPE is mid-hip-aligned; PA-MPJPE is Procrustes (scale + rotation +
-translation). Predictions use `joints_3d_local`, the rig-local pose, so the
-camera translation never enters the metric.
-
-**EMDB-1.** All 17 sequences of the `emdb1` split, every frame where
-`good_frames_mask` is set and the subject is at least 60 % inside the image.
-Metrics are over the 24 SMPL kinematic joints, aligned at the midpoint of the
-two hips (SMPL indices 1 and 2, not the `pelvis` joint), reached through the
-rig mesh and Meta's barycentric map, then a vertex regressor (see "The
-conversion, settled" — the ~1.1 mm kinematic approximation is quoted with the
-result). PA-MPJPE is Procrustes with uniform scale. PVE is vertex-to-vertex
-against the GT SMPL mesh, mid-hip aligned. Person boxes: say which of the two
-conventions you used, and use `--bbox annotated` for any published comparison.
-
-**COCO val2017.** Top-down with **ground-truth boxes**, one crop per annotated
-person with `num_keypoints ≥ 1` and `iscrowd = 0`. These AP numbers are *not*
-comparable to COCO-leaderboard AP, which includes detection. InstantHMR emits no
-per-keypoint confidence, so every keypoint is scored 1.0 — read AP next to the
-confidence-free `mean_OKS` and `PCK` numbers.
-
-### The ground truth — what it is, and what it used to be
-
-The harness scores against the **published-protocol GT**: gendered SMPL run
-forward on each frame's `poses`/`betas`/`trans`, then `J_regressor_h36m.npy`
-applied to the 6890 vertices. This is the reference every 3DPW paper uses —
-NLF's included ("obtained through the same Human3.6M-style joint regressor that
-all prior works use"). `benchmark/make_3dpw_gt.py` precomputes it; `--gt
-jointpositions` restores the pickles' raw `jointPositions` field, which is what
-this harness used before and is **not** comparable to any published table.
-
-The difference is not cosmetic. Over 1,437 test person-frames, the same
-landmarks under the two references:
-
-| | raw | root-relative | after Procrustes |
-|---|---|---|---|
-| J14 mean | 64.2 mm | 86.3 mm | 47.2 mm |
-| J12 mean | 63.6 mm | 88.1 mm | 46.0 mm |
-| right hip | 154.5 mm | 86.8 mm | 114.5 mm |
-| head | 116.3 mm | 42.0 mm | 62.2 mm |
-
-The hips dominate, because the two conventions disagree about where a hip is:
-H36M's are marker-derived and **292 mm apart**, SMPL's kinematic ones sit near
-the femoral heads, **120 mm apart**. Shoulder width goes the other way (303 vs
-387 mm), and H36M's `head` is the top of the skull where SMPL's is inside it
-(neck→head 192 vs 100 mm).
-
-Two details worth keeping straight:
-
-- **`jointPositions` is the kinematic joint, not the regressed one.** Comparing
- `J_regressor @ posed vertices` to it gives a 4.5 mm residual and looks like a
- bug in the forward pass. The joints that match are the ones carried by the
- global transforms. `make_3dpw_gt.py --check` compares those, and gets 0.001 mm.
-- **`betas` is 300-dim for 48 of the 87 subject tracks and 10-dim for the other
- 39, and entries 10: are all exactly zero.** So `--num-betas 10`, the published
- protocol, is also lossless here.
-
-`v_template_clothed` in the pickles is not a substitute for any of this — it is
-a static T-pose surface with no blend weights and no pose blendshapes, so it
-cannot be posed. Using it makes the residual worse (17.9 mm), not better.
-
-Selection inside `instanthmr_distill_train/val3dpw.py` follows the same default,
-so **3DPW numbers from before and after this change are not comparable** — the
-new GT is several mm harsher. The cache has to be rsynced next to the cluster's
-`sequenceFiles/` or the trainer raises on startup.
-
-### The joint-convention caveat — do not skip this
-
-InstantHMR predicts MHR70 joints; the GT is now in the H36M convention. The two
-place several landmarks differently, which puts a floor under the error that has
-nothing to do with model quality. The harness therefore reports three rows:
-
-- **J14** — the 14-joint LSP set most 3DPW papers use. `neck` and `head` carry
- the largest systematic penalty (MHR has no head-top; `nose` is substituted).
-- **J12** — limbs only (shoulders, elbows, wrists, hips, knees, ankles). Against
- `jointpositions` this was the clean row, because MHR and SMPL agree on those
- landmarks. Against the H36M GT it is **not** clean: the hips alone move 292 vs
- 120 mm apart between the conventions. Read it as a consistent tracking metric,
- not as a rig-free measure of the model.
-- **J14+adapter** — a linear MHR70 → J14 regressor (14x70, no bias, applied to
- centred keypoints, so it is equivariant to rotation and translation and can
- only move landmarks, never fix a pose). It absorbs the rig offset and is the
- row to compare against a published table, not an optional extra.
-
- ```bash
- python benchmark/eval_3dpw.py --onnx ... --split train --stride 10 \
- --fit-adapter benchmark/results/adapter_j14.npz
- python benchmark/eval_3dpw.py --onnx ... --split test \
- --adapter benchmark/results/adapter_j14.npz
- ```
-
- Fit on train, report on test — never fit on the split you report. The two
- splits share no sequences, so this is a real holdout, and an adapter is
- specific to one model *and* one GT convention: applying b3_s1's to another
- checkpoint, or an `h36m` one to `jointpositions`, is meaningless.
-
- **Fit it from the dataset annotations, not from the student.** `--fit-adapter` uses one
- checkpoint's own predictions as the input side, so the map can absorb that
- checkpoint's systematic error along with the rig offset — a different adapter
- per checkpoint, and a fair reader will call it tuning.
- `benchmark/fit_adapter_mhr.py` instead pairs the **released dataset's** MHR
- labels in `data/sam3d_gt_3dpw` with the H36M GT on the same 3DPW train
- frames, giving one adapter for every checkpoint that never saw a student.
- (Those are `facebook/sam-3d-body-dataset` annotations built by
- `datasets_pipeline/build_split.py`; only `data/sam3d_distill_mix` is teacher
- inference. Earlier revisions of this file called them teacher labels — wrong,
- and it changed what the residual below means.)
-
- ```bash
- python benchmark/fit_adapter_mhr.py --sequence-dir /path/to/3DPW/sequenceFiles \
- --out benchmark/results/adapter_j14_h36m_teacher.npz
- ```
-
- That it is a rig conversion rather than an error sponge is visible two ways.
- The annotations' own J14 error against the GT drops from **44.90 mm to 14.83 mm**
- on a held-out half — near the teacher's accuracy floor, which absorbing error
- could not reach. And every row reads anatomically:
- `right_hip = +1.16 right_hip - 0.32 left_hip` (widening SMPL's 120 mm hips
- toward H36M's 292 mm), `head = +0.57 left_ear +0.55 right_ear -0.47 neck`
- (extrapolating to the top of the skull), `right_elbow` from the cubital fossa,
- `right_shoulder` from the acromion, `right_knee = +0.84 right_knee` — nearly
- identity, because the knee is the same point in both conventions.
-
- The fit must also be truncated: see `RCOND` in `fit_adapter`, without which
- the map reaches the hips through +-2000 weights on near-collinear finger
- joints and stops being a joint regressor in any readable sense.
-
-### Train/test contamination — READ THIS BEFORE PUBLISHING
-
-**COCO val2017 is clean.** All 25,594 COCO images in `data/` are
-`COCO_train2014_*`. COCO's 2017 split re-partitions the 2014 data such that
-val2017 is a 5k subset of val2014, which is disjoint from train2014. Verified by
-image-id intersection: **0 of 5000** val2017 images appear in training.
-
-**3DPW test is clean for every checkpoint trained on Jean Zay** — that corpus
-is coco/mpii/aic/sa1b/harmony4d plus `3dpw_train`, so the 24 test sequences are
-unseen for `b3_s1`, `v3_s1` and everything after them. The paragraph below
-applies to the **local** corpus and to `distill_mhr_only_ckpt90`, which was
-trained from it.
-
-**The local `data/` corpus is contaminated.** `data/sam3d_gt_3dpw` holds only train sequences,
-but `data/sam3d_distill_mix` contains teacher-distilled crops from **all 60**
-3DPW sequences — including all 24 test sequences (9,060 person-crops, ~28.5k
-distinct frames across the dataset). `--exclude-seen` quantifies this: it splits
-the report into frames the model saw during training and frames it did not.
-
-Note that even the "not in training data" subset is only *frame*-level held out.
-Neighbouring frames of the same person in the same scene were seen, so it still
-overstates true generalisation. **To publish a 3DPW number, retrain with the 24
-test sequences removed from `sam3d_distill_mix`.** The contamination-split
-report tells you how large the effect is in the meantime.
-
-## Layout
-
-```
-benchmark/
- download.py COCO fetcher + 3DPW presence check
- make_3dpw_gt.py SMPL forward + H36M regressor -> <3DPW root>/gt_h36m/
- make_emdb_gt.py SMPL forward -> <EMDB root>/gt_smpl24/ (+ a kp2d check)
- fit_adapter_mhr.py MHR70 -> J14 / SMPL24 adapter from TEACHER labels
- eval_3dpw.py 3D from an ONNX graph: MPJPE, PA-MPJPE, PCK3D, AUC
- eval_3dpw_ckpt.py the same, for a .pth straight out of runs/
- eval_emdb_ckpt.py EMDB-1 for a .pth; reuses the 3DPW crop and metrics
- eval_smpl_fit.py EMDB-1 in SMPL space via mesh conversion: +PVE, lower floor
- mhr_smpl.py the MHR rig, differentiable SMPL, and both fit directions
- eval_coco.py 2D: OKS AP/AR, mean OKS, PCK, NME, per-keypoint breakdown
- benchlib/
- joints.py MHR70 <-> COCO17 / SMPL24 / H36M17 maps, J14 & J12 sets
- metrics.py Procrustes alignment and the metric definitions
- threedpw.py 3DPW pickle loading, the GT switch, projection, boxes
- emdb.py EMDB-1 loading; extrinsics sense and the 24-joint choice
- runner.py batched ONNX inference with overlapped image decoding
-```
diff --git a/benchmark/benchlib/detboxes.py b/benchmark/benchlib/detboxes.py
deleted file mode 100644
index 5bb7563..0000000
--- a/benchmark/benchlib/detboxes.py
+++ /dev/null
@@ -1,81 +0,0 @@
-"""Replace a benchmark's person boxes with the deployed detector's.
-
-The published protocols crop around a box nobody has at runtime (projected GT
-joints on 3DPW, EMDB's annotated boxes). ``instanthmr.pipeline`` feeds the
-pose model RF-DETR's raw box, so this scores the same path: run RF-DETR on
-each frame, keep the person detection that overlaps the GT person best, and
-crop around it exactly as the phone would. A frame where no detection reaches
-``MIN_IOU`` with the GT person is a missed detection: it is dropped and
-counted, since there is nothing the pose model could be given.
-
-Detections are cached per image (all persons, so the same cache serves every
-person in a frame and every later run).
-"""
-
-from __future__ import annotations
-
-import json
-from pathlib import Path
-
-import cv2
-import numpy as np
-from tqdm import tqdm
-
-from .threedpw import bbox_from_points
-
-MIN_IOU = 0.3
-CACHE_DIR = Path.home() / ".cache" / "instanthmr" / "detboxes"
-
-
-def _iou(a: np.ndarray, b: np.ndarray) -> float:
- ix = max(0.0, min(a[2], b[2]) - max(a[0], b[0]))
- iy = max(0.0, min(a[3], b[3]) - max(a[1], b[1]))
- inter = ix * iy
- union = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter
- return inter / union if union > 0 else 0.0
-
-
-def detect_all(image_paths, cache_name: str, variant: str = "medium",
- confidence: float = 0.5) -> dict[str, list]:
- """{image path: [[x1, y1, x2, y2, conf], ...]}, from the cache when present."""
- CACHE_DIR.mkdir(parents=True, exist_ok=True)
- path = CACHE_DIR / f"{cache_name}_rfdetr-{variant}_c{confidence}.json"
- cache = json.loads(path.read_text()) if path.is_file() else {}
- todo = sorted(set(image_paths) - cache.keys())
- if todo:
- import sys
- sys.path.insert(0, str(Path(__file__).resolve().parents[2])) # the repo, for instanthmr
- from instanthmr.detector import RFDETRDetector
- det = RFDETRDetector(variant=variant, confidence=confidence, max_persons=20)
- for i, p in enumerate(tqdm(todo, desc=f"RF-DETR {variant}", unit="img")):
- bgr = cv2.imread(p, cv2.IMREAD_COLOR)
- dets = det.detect(cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)) if bgr is not None else []
- cache[p] = [[*map(float, d["bbox"]), d["confidence"]] for d in dets]
- if i % 2000 == 1999:
- path.write_text(json.dumps(cache))
- path.write_text(json.dumps(cache))
- return cache
-
-
-def apply_detector_boxes(ds, cache_name: str, variant: str = "medium") -> dict:
- """Swap every ``ds.samples[i]["bbox"]`` for the matched RF-DETR box and drop
- missed detections from ``ds.samples`` and ``ds.gt`` alike. Each sample needs
- ``image_path`` and ``uv`` (the projected GT joints)."""
- cache = detect_all([s["image_path"] for s in ds.samples], cache_name, variant)
- keep, ious = [], []
- for i, s in enumerate(ds.samples):
- ref = bbox_from_points(np.asarray(s["uv"], dtype=np.float64), 1.0)
- cands = cache.get(s["image_path"], [])
- best = max(cands, key=lambda d: _iou(np.asarray(d[:4]), ref), default=None)
- iou = _iou(np.asarray(best[:4]), ref) if best is not None else 0.0
- if iou < MIN_IOU:
- continue
- s["bbox"] = np.asarray(best[:4], dtype=np.float32)
- keep.append(i)
- ious.append(iou)
- n = len(ds.samples)
- ds.samples = [ds.samples[i] for i in keep]
- ds.gt = np.asarray(ds.gt)[keep]
- return dict(n_frames=n, n_missed=n - len(keep),
- miss_rate=(n - len(keep)) / max(n, 1),
- mean_iou=float(np.mean(ious)) if ious else 0.0)
diff --git a/benchmark/benchlib/emdb.py b/benchmark/benchlib/emdb.py
deleted file mode 100644
index f07a12a..0000000
--- a/benchmark/benchlib/emdb.py
+++ /dev/null
@@ -1,135 +0,0 @@
-"""EMDB-1 ground truth loading -- the 3DPW loader's structure, one subject.
-
-EMDB ships one pickle per sequence under ``<root>/P<n>/<seq>/<name>_data.pkl``
-with the frames beside it in ``images/<frame:05d>.jpg``. The fields we use:
-
- smpl{poses_root (F,3), poses_body (F,69), betas (10,), trans (F,3)}
- camera{intrinsics (3,3), extrinsics (F,4,4), width, height}
- good_frames_mask (F,) reliable-GT mask, 3DPW's ``campose_valid``
- bboxes{bboxes (F,4) xyxy, invalid_idxs} person boxes, **provided**
- kp2d (F,24,2) the 24 joints projected -- see below
- emdb1 / emdb2 split membership
-
-Two conventions were settled against the data rather than the docs, because
-guessing either gives quietly wrong numbers instead of a crash:
-
-* ``extrinsics`` is **world -> camera** (the same sense as 3DPW's
- ``cam_poses``). Applied as-is it reprojects the GT joints onto ``kp2d`` to
- 0.002 px; inverted, every joint lands behind the camera.
-* ``kp2d`` is the projection of the SMPL **kinematic-tree** joints, not
- ``J_regressor @ posed vertices`` -- the latter reprojects ~1 px off and sits
- 3.8 mm away in 3D. So "EMDB (24)" means the kinematic joints, which is what
- ``make_emdb_gt.py`` caches into ``<root>/gt_smpl24/emdb1.npz``.
-
-Person boxes come in two flavours, because published tables use both:
-``gt-joints`` reproduces this repo's 3DPW protocol (projected GT joints padded
-by ``bbox_scale``, no detector in the loop) and ``annotated`` uses EMDB's own
-``bboxes`` field, which is what the "Oracle: evaluated with annotated bounding
-boxes" rows report.
-"""
-
-from __future__ import annotations
-
-import pickle
-from pathlib import Path
-
-import numpy as np
-
-from .threedpw import bbox_from_points, project
-
-EMDB1_SEQUENCES = 17 # sanity check: the EMDB-1 split has 17 sequences
-
-
-def gt_path(emdb_root: str | Path, gt_dir: str | Path | None = None) -> Path:
- base = Path(gt_dir) if gt_dir else Path(emdb_root) / "gt_smpl24"
- return base / "emdb1.npz"
-
-
-def load_gt(emdb_root: str | Path, gt_dir: str | Path | None = None) -> dict:
- path = gt_path(emdb_root, gt_dir)
- if not path.is_file():
- raise FileNotFoundError(
- f"EMDB ground truth not found: {path}\n"
- "Build it once with benchmark/make_emdb_gt.py.")
- with np.load(path) as z:
- return {k: z[k] for k in z.files if k != "layout"}
-
-
-def build_samples(emdb_root: str | Path, stride: int = 1,
- bbox: str = "gt-joints", bbox_scale: float = 1.2,
- min_visible_frac: float = 0.6,
- gt_dir: str | Path | None = None):
- """Enumerate every evaluatable frame of EMDB-1.
-
- Args:
- emdb_root: directory holding ``P0/ ... P9/``.
- stride: keep every Nth frame (1 = the full protocol).
- bbox: ``gt-joints`` (projected GT joints padded by ``bbox_scale``, this
- repo's 3DPW convention) or ``annotated`` (EMDB's own boxes).
- min_visible_frac: drop a frame when fewer than this fraction of the
- projected joints fall inside the image.
-
- Returns:
- ``(samples, gt)`` -- a list of dicts and an (N, 24, 3) array of
- camera-space SMPL joints in metres, index-aligned.
- """
- if bbox not in ("gt-joints", "annotated"):
- raise ValueError(f"unknown bbox source: {bbox}")
- root = Path(emdb_root)
- table = load_gt(root, gt_dir)
-
- samples: list[dict] = []
- gts: list[np.ndarray] = []
- n_seq = 0
-
- for pkl in sorted(root.glob("P*/*/*_data.pkl")):
- with open(pkl, "rb") as f:
- seq = pickle.load(f)
- if not seq["emdb1"]:
- continue
- n_seq += 1
- name = str(seq["name"])
- K = np.asarray(seq["camera"]["intrinsics"], dtype=np.float64)
- T_all = np.asarray(seq["camera"]["extrinsics"], dtype=np.float64)
- W = int(seq["camera"]["width"])
- H = int(seq["camera"]["height"])
- good = np.asarray(seq["good_frames_mask"]).astype(bool)
- joints_w = np.asarray(table[name], dtype=np.float64)
- boxes = np.asarray(seq["bboxes"]["bboxes"], dtype=np.float64)
- # EMDB flags the frames whose provided box is unusable; they only
- # matter for bbox="annotated", where there is nothing to crop around.
- bad_box = set(np.asarray(seq["bboxes"]["invalid_idxs"]).ravel().tolist())
- img_dir = pkl.parent / "images"
-
- for f in range(0, joints_w.shape[0], stride):
- if not good[f]:
- continue
- if bbox == "annotated" and f in bad_box:
- continue
- T = T_all[f]
- joints_cam = joints_w[f] @ T[:3, :3].T + T[:3, 3]
- if (joints_cam[:, 2] <= 0).any(): # behind the camera
- continue
- uv = project(joints_cam, K)
- inside = ((uv[:, 0] >= 0) & (uv[:, 0] < W)
- & (uv[:, 1] >= 0) & (uv[:, 1] < H))
- if inside.mean() < min_visible_frac:
- continue
- box = (bbox_from_points(uv, bbox_scale) if bbox == "gt-joints"
- else boxes[f].astype(np.float32))
- samples.append(dict(
- image_path=str(img_dir / f"{f:05d}.jpg"),
- bbox=box, uv=uv.astype(np.float32),
- sequence=name, person=0, frame=f,
- gender=str(seq["gender"]),
- # EMDB is calibrated, so this is the real focal.
- focal=float(K[1, 1]),
- ))
- gts.append(joints_cam)
-
- if n_seq != EMDB1_SEQUENCES:
- raise RuntimeError(f"found {n_seq} EMDB-1 sequences under {root}, "
- f"expected {EMDB1_SEQUENCES}")
- if not samples:
- raise RuntimeError(f"no valid EMDB-1 samples found in {root}")
- return samples, np.stack(gts)
diff --git a/benchmark/benchlib/feedback.py b/benchmark/benchlib/feedback.py
deleted file mode 100644
index 08d8a19..0000000
--- a/benchmark/benchlib/feedback.py
+++ /dev/null
@@ -1,104 +0,0 @@
-"""Previous-pose feedback and jitter for the video benchmarks (3DPW, EMDB).
-
-A ``--prev-token`` model can be fed its own earlier prediction. Here every
-person-frame at frame ``f`` gets the prediction made at frame ``f - lag`` of
-the same track (sequence, person), or the previous pose withheld when that
-frame has none (the first ``lag`` frames, a missed detection, a non-finite
-prediction). At 30 fps the default lag of 5 frames is 167 ms, inside the
-training pairs (Harmony4D and semelle, 5-20 frames apart).
-
-Frames are processed in rounds of ``lag`` frames per track, all tracks
-together: round ``r`` only needs round ``r - 1``, so batches stay large.
-
-``accel`` is the standard video-HMR jitter number: the second difference over
-three consecutive frames of pelvis-centred joints, in mm per frame^2, for the
-prediction (``accel_mm``) and against the GT's (``accel_err_mm``).
-"""
-
-from __future__ import annotations
-
-import math
-from collections import defaultdict
-
-import numpy as np
-import torch
-
-import train_distill_mhr_only as T
-
-
-def _track_rounds(samples, lag):
- """Index batches in dependency order, and frame -> index per track."""
- tracks = defaultdict(dict)
- for i, s in enumerate(samples):
- tracks[(s["sequence"], s["person"])][s["frame"]] = i
- rounds = defaultdict(list)
- for frames in tracks.values():
- f0 = min(frames)
- for f, i in frames.items():
- rounds[(f - f0) // lag].append(i)
- return [sorted(rounds[r]) for r in sorted(rounds)], tracks
-
-
-def predict_feedback(model, ds, mhr_module, device, lag=5, use_amp=True,
- exact_landmarks=False, batch_size=64, num_workers=6):
- """Same return as ``eval_3dpw_ckpt.predict``: (N, 70, 3) mm, indices, dropped."""
- model.eval()
- rounds, tracks = _track_rounds(ds.samples, lag)
- batches = [r[k:k + batch_size] for r in rounds for k in range(0, len(r), batch_size)]
- loader = torch.utils.data.DataLoader(ds, batch_sampler=batches,
- num_workers=num_workers, pin_memory=True)
- prev = {} # index -> PREV_DIM vector
- kp_all = {}
- n_dropped = 0
- zeros = np.zeros(T.PREV_DIM, np.float32)
- for ids, b in zip(batches, loader):
- pp, hp = [], []
- for i in ids:
- s = ds.samples[i]
- j = tracks[(s["sequence"], s["person"])].get(s["frame"] - lag)
- v = prev.get(j) if j is not None else None
- pp.append(zeros if v is None else v)
- hp.append(0.0 if v is None else 1.0)
- img = b["image"].to(device, non_blocking=True)
- cc = b["cliff_cond"].to(device, non_blocking=True)
- pp = torch.from_numpy(np.stack(pp)).to(device)
- hp = torch.tensor(hp, device=device)
- with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
- out = model(img, cc, pp, hp)
- kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),
- out["shape_params"].float(),
- exact=exact_landmarks).float().cpu().numpy()
- t = out["cam_trans"].float().cpu().numpy()
- ok = b["ok"].bool().numpy()
- for k, i in enumerate(ids):
- if not ok[k]:
- continue
- if not (np.isfinite(kp[k]).all() and np.isfinite(t[k]).all()):
- n_dropped += 1
- continue
- kp_all[i] = kp[k]
- z = max(float(t[k, 2]), 0.1)
- prev[i] = np.concatenate([kp[k].reshape(-1),
- [t[k, 0] / z, t[k, 1] / z, math.log(z)]]).astype(np.float32)
- idx = np.array(sorted(kp_all))
- return np.stack([kp_all[i] for i in idx]) * 1000.0, idx, n_dropped
-
-
-def accel(pred, gt, root_p, root_g, samples):
- """(pred, gt) are (N, J, 3) mm in the same joint set, ``samples`` the
- matching sample dicts; triplets are three consecutive frames of one track."""
- p = pred - root_p[:, None]
- g = gt - root_g[:, None]
- pos = {(s["sequence"], s["person"], s["frame"]): n for n, s in enumerate(samples)}
- tri = [(pos[(q, r, f - 1)], n, pos[(q, r, f + 1)])
- for (q, r, f), n in pos.items()
- if (q, r, f - 1) in pos and (q, r, f + 1) in pos]
- if not tri:
- return dict(accel_mm=None, accel_err_mm=None, n_triplets=0)
- a, b, c = (np.array(x) for x in zip(*tri))
- ap = p[a] - 2 * p[b] + p[c]
- ag = g[a] - 2 * g[b] + g[c]
- return dict(accel_mm=float(np.linalg.norm(ap, axis=-1).mean()),
- accel_err_mm=float(np.linalg.norm(ap - ag, axis=-1).mean()),
- n_triplets=len(tri))
diff --git a/benchmark/benchlib/runner.py b/benchmark/benchlib/runner.py
deleted file mode 100644
index 50a9bc5..0000000
--- a/benchmark/benchlib/runner.py
+++ /dev/null
@@ -1,124 +0,0 @@
-"""Batched InstantHMR inference over a flat list of (image, bbox) samples.
-
-The public API is :func:`run_samples`. It groups samples by source image so
-each frame is decoded once and all its people go through the ONNX graph in a
-single call, and it overlaps JPEG decoding with GPU work via a small prefetch
-pool. Predictions come back in the input order.
-"""
-
-from __future__ import annotations
-
-import sys
-from concurrent.futures import ThreadPoolExecutor
-from dataclasses import dataclass
-from pathlib import Path
-
-import cv2
-import numpy as np
-from tqdm import tqdm
-
-REPO_ROOT = Path(__file__).resolve().parents[2]
-if str(REPO_ROOT) not in sys.path:
- sys.path.insert(0, str(REPO_ROOT))
-
-from instanthmr import InstantHMR # noqa: E402
-
-
-@dataclass
-class Sample:
- """One person to evaluate."""
- image_path: str
- bbox: np.ndarray # (4,) xyxy in original image pixels
- meta: dict # carried through untouched
-
-
-def build_model(onnx_path: str, device: str = "cuda",
- mhr_script: str | None = None) -> InstantHMR:
- """Create the ONNX session, attaching the MHR rig when the graph needs it.
-
- The ``mhr_only`` graphs have no 3D head: their joints are derived by running
- the predicted MHR parameters through the rig, so a backend is mandatory.
- """
- from instanthmr.inference import onnx_output_names
-
- names = onnx_output_names(onnx_path) or []
- mhr = None
- if "joints_3d" not in names:
- from instanthmr.mhr_renderer import build_mhr
- script = mhr_script or str(REPO_ROOT / "checkpoints" / "mhr_model.pt")
- if not Path(script).exists():
- raise FileNotFoundError(
- f"this graph has no joints_3d output and needs the MHR rig, "
- f"but {script} is missing"
- )
- mhr = build_mhr(script_path=script, device=device)
- return InstantHMR(onnx_path, device=device, mhr=mhr)
-
-
-def _group_by_image(samples: list[Sample]) -> list[tuple[str, list[int]]]:
- """Group sample indices by image path, preserving first-seen order."""
- order: list[str] = []
- groups: dict[str, list[int]] = {}
- for i, s in enumerate(samples):
- if s.image_path not in groups:
- groups[s.image_path] = []
- order.append(s.image_path)
- groups[s.image_path].append(i)
- return [(p, groups[p]) for p in order]
-
-
-def _load_rgb(path: str) -> np.ndarray | None:
- img = cv2.imread(path, cv2.IMREAD_COLOR)
- if img is None:
- return None
- return cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
-
-
-def run_samples(model: InstantHMR, samples: list[Sample],
- num_workers: int = 6, desc: str = "inference"):
- """Run the model over every sample.
-
- Returns:
- ``(predictions, ok)`` where ``predictions`` is a list the same length as
- ``samples`` (``None`` where the image could not be read) and ``ok`` is a
- boolean mask of the usable entries.
- """
- groups = _group_by_image(samples)
- preds: list[object | None] = [None] * len(samples)
-
- with ThreadPoolExecutor(max_workers=num_workers) as pool:
- # Keep a bounded window of decoded frames in flight ahead of the GPU.
- window = max(2 * num_workers, 8)
- pending: dict[int, object] = {}
- next_submit = 0
-
- def submit_until(limit: int) -> None:
- nonlocal next_submit
- while next_submit < min(limit, len(groups)):
- pending[next_submit] = pool.submit(_load_rgb, groups[next_submit][0])
- next_submit += 1
-
- submit_until(window)
- for gi in tqdm(range(len(groups)), desc=desc, unit="img"):
- path, idxs = groups[gi]
- image = pending.pop(gi).result()
- submit_until(gi + 1 + window)
- if image is None:
- continue
- dets = [{"bbox": samples[i].bbox, "confidence": 1.0} for i in idxs]
- # The dataset's own focal when it has one (3DPW is calibrated), so
- # a graph exported from a --cliff-focal checkpoint is conditioned
- # on the real camera rather than a diagonal-derived stand-in. Every
- # sample in a group is the same frame, hence the same focal.
- out = model.predict_batch(image, dets,
- focal=samples[idxs[0]].meta.get("focal"))
- for i, p in zip(idxs, out):
- preds[i] = p
-
- ok = np.array([p is not None for p in preds], dtype=bool)
- return preds, ok
-
-
-def stack_joints(preds, ok, attr: str) -> np.ndarray:
- """Stack one joint array (e.g. ``joints_3d_local``) over the valid preds."""
- return np.stack([getattr(p, attr) for p, k in zip(preds, ok) if k]).astype(np.float64)
diff --git a/benchmark/download.py b/benchmark/download.py
deleted file mode 100644
index 32eb140..0000000
--- a/benchmark/download.py
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/usr/bin/env python3
-"""Fetch the benchmark data.
-
-COCO val2017 is a direct public download and is handled fully automatically.
-3DPW requires accepting a licence on the MPI page, so this script cannot fetch
-it for you — it verifies what you already have and tells you what is missing.
-
- python benchmark/download.py --coco
- python benchmark/download.py --check-3dpw --sequence-dir ... --image-root ...
-"""
-
-from __future__ import annotations
-
-import argparse
-import sys
-import urllib.request
-import zipfile
-from pathlib import Path
-
-COCO_FILES = [
- ("http://images.cocodataset.org/annotations/annotations_trainval2017.zip",
- "annotations_trainval2017.zip", "annotations/person_keypoints_val2017.json"),
- ("http://images.cocodataset.org/zips/val2017.zip",
- "val2017.zip", "val2017"),
-]
-
-THREEDPW_HELP = """
-3DPW must be downloaded from the official page (free, licence form):
-
- https://virtualhumans.mpi-inf.mpg.de/3DPW/licence.html
-
-Fill the form; the confirmation mail carries the download links. You need:
-
- sequenceFiles.zip ~ 30 MB <-- the ground truth (required)
- imageFiles.zip ~ 4.6 GB <-- the RGB frames
-
-Unzip so the layout is:
-
- <3dpw_root>/sequenceFiles/{train,validation,test}/*.pkl
- <3dpw_root>/imageFiles/<sequence>/image_XXXXX.jpg
-"""
-
-
-def _progress(block_num, block_size, total):
- if total <= 0:
- return
- done = min(block_num * block_size, total)
- pct = 100.0 * done / total
- sys.stdout.write(f"\r {pct:5.1f}% ({done/1e6:.0f} / {total/1e6:.0f} MB)")
- sys.stdout.flush()
-
-
-def download_coco(root: Path) -> None:
- root.mkdir(parents=True, exist_ok=True)
- for url, zip_name, marker in COCO_FILES:
- if (root / marker).exists():
- print(f"[coco] {marker} already present — skipping")
- continue
- zip_path = root / zip_name
- if not zip_path.exists():
- print(f"[coco] downloading {url}")
- urllib.request.urlretrieve(url, zip_path, _progress)
- print()
- print(f"[coco] extracting {zip_name}")
- with zipfile.ZipFile(zip_path) as z:
- # The annotations zip carries train2017 too; we only need val.
- members = [m for m in z.namelist()
- if "train2017" not in m or "val" in m]
- z.extractall(root, members=members)
- zip_path.unlink()
- print(f"[coco] ok -> {root / marker}")
- print(f"\n[coco] ready. Use --coco-root {root}")
-
-
-def check_3dpw(sequence_dir: Path | None, image_root: Path | None) -> bool:
- ok = True
- if sequence_dir is None or not sequence_dir.is_dir():
- print(f"[3dpw] MISSING sequenceFiles: {sequence_dir}")
- ok = False
- else:
- for split, expected in (("test", 24), ("validation", 12), ("train", 24)):
- pkls = list((sequence_dir / split).glob("*.pkl"))
- flag = "ok " if pkls else "MISSING"
- print(f"[3dpw] {flag} sequenceFiles/{split:<11s} "
- f"{len(pkls)} pkl (official: {expected})")
- if split == "test" and not pkls:
- ok = False
- if image_root is None or not image_root.is_dir():
- print(f"[3dpw] MISSING imageFiles: {image_root}")
- ok = False
- else:
- seqs = [d for d in image_root.iterdir() if d.is_dir()]
- print(f"[3dpw] ok imageFiles {len(seqs)} sequences "
- f"(official: 60)")
- if not ok:
- print(THREEDPW_HELP)
- else:
- print("\n[3dpw] ready.")
- return ok
-
-
-def main():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--coco", action="store_true", help="download COCO val2017")
- p.add_argument("--coco-root", default="benchmark/data/coco")
- p.add_argument("--check-3dpw", action="store_true")
- p.add_argument("--sequence-dir", default=None)
- p.add_argument("--image-root", default=None)
- a = p.parse_args()
-
- if a.coco:
- download_coco(Path(a.coco_root))
- if a.check_3dpw:
- check_3dpw(
- Path(a.sequence_dir) if a.sequence_dir else None,
- Path(a.image_root) if a.image_root else None,
- )
- if not (a.coco or a.check_3dpw):
- p.print_help()
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_3dpw.py b/benchmark/eval_3dpw.py
deleted file mode 100644
index 31450c8..0000000
--- a/benchmark/eval_3dpw.py
+++ /dev/null
@@ -1,279 +0,0 @@
-#!/usr/bin/env python3
-"""3DPW test-set evaluation for InstantHMR (MPJPE / PA-MPJPE / PCK / AUC).
-
-By default the GT is the published protocol: SMPL run forward on the sequence's
-poses/betas and the Human3.6M regressor applied to the vertices, precomputed by
-``benchmark/make_3dpw_gt.py``. ``--gt jointpositions`` falls back to the raw
-SMPL kinematic joints in the pickles, which is what this harness used before
-and is not comparable to any published table. Predicted MHR70 joints are mapped
-onto the joint set by name; see ``benchlib/joints.py`` for the caveat about
-``neck`` and ``head``, which are the only two joints where the rigs disagree on
-the underlying anatomy.
-
-Usage:
- python benchmark/eval_3dpw.py \
- --onnx models/instanthmr_mhr_only_ckpt90.onnx \
- --sequence-dir /path/to/3DPW/sequenceFiles \
- --image-root /path/to/3DPW/imageFiles
-
-Add ``--stride 10`` for a fast smoke run over 1/10th of the frames.
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-import numpy as np
-
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-
-from benchlib import joints as J
-from benchlib import metrics as M
-from benchlib import threedpw as P
-from benchlib.runner import Sample, build_model, run_samples, stack_joints
-
-MM = 1000.0 # metres -> millimetres
-RCOND = 3e-3 # singular-value cutoff for the joint adapter; see fit_adapter
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--onnx", required=True, help="path to the exported graph")
- p.add_argument("--sequence-dir", required=True, help="3DPW sequenceFiles/")
- p.add_argument("--image-root", required=True, help="3DPW imageFiles/")
- p.add_argument("--split", default="test",
- choices=["test", "train", "validation"])
- p.add_argument("--stride", type=int, default=1,
- help="evaluate every Nth frame (1 = full protocol)")
- p.add_argument("--bbox-scale", type=float, default=1.2,
- help="padding on the projected-GT-joint person box")
- p.add_argument("--gt", default="h36m", choices=["h36m", "jointpositions"],
- help="h36m: SMPL forward + H36M regressor, the published "
- "protocol. jointpositions: the raw pickle field.")
- p.add_argument("--gt-dir", default=None,
- help="where make_3dpw_gt.py wrote its npz "
- "(default: <sequence-dir>/../gt_h36m)")
- p.add_argument("--device", default="cuda")
- p.add_argument("--mhr-script", default=None)
- p.add_argument("--num-workers", type=int, default=6)
- p.add_argument("--limit", type=int, default=0, help="cap sample count (debug)")
- p.add_argument("--out", default="benchmark/results/3dpw.json")
- p.add_argument("--adapter", default=None,
- help="npz with a fitted MHR70->J14 joint regressor to also report")
- p.add_argument("--exclude-seen", nargs="*", default=None,
- metavar="DIR",
- help="training annotation dirs; frames found there are "
- "reported separately as a held-out subset")
- p.add_argument("--fit-adapter", default=None,
- help="fit that regressor on THIS split and write it here "
- "(run on train/validation, never on test)")
- return p.parse_args()
-
-
-def evaluate(pred_mhr: np.ndarray, gt_raw: np.ndarray, set_name: str,
- gt_src: str) -> dict:
- """Metrics for one joint set. Inputs are (N, 70, 3) and (N, J, 3) in mm."""
- spec = J.JOINT_SETS[set_name]
- pred = pred_mhr[:, spec["mhr"], :]
- gt = gt_raw[:, J.gt_index(spec, gt_src), :]
-
- root_p, root_g = J.pelvis(pred), J.pelvis(gt)
- e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
- e_pa = M.pa_mpjpe(pred, gt)
-
- return dict(
- joint_set=set_name,
- num_joints=len(spec["names"]),
- num_samples=int(pred.shape[0]),
- MPJPE_mm=float(e_mpjpe.mean()),
- PA_MPJPE_mm=float(e_pa.mean()),
- PCK3D_50mm=M.pck3d(e_pa, 50.0),
- PCK3D_100mm=M.pck3d(e_pa, 100.0),
- AUC_0_150mm=M.auc3d(e_pa),
- per_joint_PA_MPJPE_mm={
- n: float(v) for n, v in zip(spec["names"], e_pa.mean(axis=0))
- },
- )
-
-
-def fit_linear_adapter(X_mhr: np.ndarray, Y_target: np.ndarray) -> np.ndarray:
- """Least-squares map (J, 70) from MHR70 keypoints onto any target joint set.
-
- Both sides are centred per sample, so the map is bias-free and equivariant
- to rotation and translation; see ``fit_adapter`` for why the centring is in
- float64 and why the singular values are truncated.
- """
- X = X_mhr.astype(np.float64)
- Y = Y_target.astype(np.float64)
- X = X - X.mean(axis=1, keepdims=True)
- Y = Y - Y.mean(axis=1, keepdims=True)
- A = X.transpose(0, 2, 1).reshape(-1, X.shape[1])
- B = Y.transpose(0, 2, 1).reshape(-1, Y.shape[1])
- W, *_ = np.linalg.lstsq(A, B, rcond=RCOND)
- return W.T
-
-
-def fit_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
- gt_src: str) -> np.ndarray:
- """Least-squares regressor W: (14, 70) mapping MHR70 joints to SMPL J14.
-
- Fitted on centred poses and used without a bias, so the mapping stays
- equivariant to rotation and translation — it only removes the systematic
- difference in where the two rigs place each landmark.
-
- The 70 MHR keypoints are 40 finger joints millimetres apart, so the design
- matrix is badly conditioned and the unregularised solution reaches those
- landmarks through +-2000 weights on fingers that nearly cancel. It scores
- well and is not a joint regressor. Truncating the small singular values
- (``RCOND``) gives a rank-24 map with weights under 0.9, where the H36M hip
- reads as ``0.94 * MHR hip + 0.99 * neck``-style combinations of nearby
- landmarks, and costs nothing measurable: on a sequence-level holdout inside
- 3DPW train, 45.74 mm against the unregularised fit's 45.76 mm.
-
- Centring must happen in float64. Doing it in float32 leaves a ~0.04 mm
- residual in the all-ones direction, which is exactly the direction the
- centring annihilates; ``lstsq`` then sees a singular value of 3e-7 relative
- instead of zero, keeps it, and loads that null direction with weights of
- +-2500. The fitted map still scores ~37 mm in sample in float64 and blows up
- to 430 mm in float32.
- """
- gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]
- return fit_linear_adapter(pred_mhr, gt) # (14, 70)
-
-
-def evaluate_adapter(pred_mhr: np.ndarray, gt_raw: np.ndarray,
- W: np.ndarray, gt_src: str) -> dict:
- # W was fitted on centred keypoints, so apply it to centred keypoints. Both
- # metrics below are translation-invariant, so this costs nothing and makes
- # the result independent of whatever W does to the all-ones direction.
- centred = pred_mhr - pred_mhr.mean(axis=1, keepdims=True)
- pred = np.einsum("jk,nkc->njc", W, centred)
- gt = gt_raw[:, J.gt_index(J.JOINT_SETS["J14"], gt_src), :]
- root_p, root_g = J.pelvis(pred), J.pelvis(gt)
- e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
- e_pa = M.pa_mpjpe(pred, gt)
- return dict(
- joint_set="J14+adapter", num_joints=14,
- num_samples=int(pred.shape[0]),
- MPJPE_mm=float(e_mpjpe.mean()),
- PA_MPJPE_mm=float(e_pa.mean()),
- PCK3D_50mm=M.pck3d(e_pa, 50.0),
- PCK3D_100mm=M.pck3d(e_pa, 100.0),
- AUC_0_150mm=M.auc3d(e_pa),
- )
-
-
-def main():
- args = parse_args()
-
- print(f"[3DPW] loading GT from {args.sequence_dir} ({args.split} split)")
- samples, gt_joints = P.build_samples(
- args.sequence_dir, args.image_root, split=args.split,
- stride=args.stride, bbox_scale=args.bbox_scale,
- gt=args.gt, gt_dir=args.gt_dir,
- )
- if args.limit:
- samples, gt_joints = samples[:args.limit], gt_joints[:args.limit]
- seqs = sorted({s["sequence"] for s in samples})
- print(f"[3DPW] {len(samples)} person-frames across {len(seqs)} sequences")
-
- model = build_model(args.onnx, args.device, args.mhr_script)
- preds, ok = run_samples(
- model,
- [Sample(s["image_path"], s["bbox"], s) for s in samples],
- num_workers=args.num_workers, desc="3DPW",
- )
- if not ok.all():
- print(f"[3DPW] WARNING: {int((~ok).sum())} samples had unreadable images")
-
- # joints_3d_local is the body-centred rig pose; root alignment and
- # Procrustes both make the camera translation irrelevant.
- pred_mhr = stack_joints(preds, ok, "joints_3d_local") * MM
- gt_smpl = gt_joints[ok] * MM
-
- kept_meta = [s for s, k in zip(samples, ok) if k]
- W_adapt = np.load(args.adapter)["W"] if args.adapter else None
-
- def _rows(pm, gm):
- rows = [evaluate(pm, gm, s, args.gt) for s in ("J14", "J12")]
- if W_adapt is not None:
- rows.append(evaluate_adapter(pm, gm, W_adapt, args.gt))
- return rows
- report = dict(
- model=str(args.onnx), dataset="3DPW", split=args.split, gt=args.gt,
- stride=args.stride, bbox_scale=args.bbox_scale,
- num_sequences=len(seqs), sequences=seqs,
- num_samples=int(pred_mhr.shape[0]),
- results=_rows(pred_mhr, gt_smpl),
- )
-
- # Frames present in the training data are leaked; score them separately so
- # the honest number is visible next to the optimistic one.
- if args.exclude_seen:
- seen = P.seen_frames(args.exclude_seen)
- is_seen = np.array([(m["sequence"], m["frame"]) in seen for m in kept_meta])
- report["contamination"] = dict(
- training_dirs=list(args.exclude_seen),
- seen_frames=int(is_seen.sum()),
- unseen_frames=int((~is_seen).sum()),
- seen_fraction=float(is_seen.mean()),
- seen_sequences=sorted({m["sequence"] for m, f in zip(kept_meta, is_seen) if f}),
- )
- if (~is_seen).any():
- report["results_unseen_frames"] = _rows(pred_mhr[~is_seen], gt_smpl[~is_seen])
- if is_seen.any():
- report["results_seen_frames"] = _rows(pred_mhr[is_seen], gt_smpl[is_seen])
-
- if args.fit_adapter:
- W = fit_adapter(pred_mhr, gt_smpl, args.gt)
- Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)
- np.savez(args.fit_adapter, W=W, split=args.split, gt=args.gt)
- print(f"[3DPW] fitted joint adapter on '{args.split}' -> {args.fit_adapter}")
-
- # Per-sequence PA-MPJPE on the headline joint set.
- spec = J.JOINT_SETS["J14"]
- e = M.pa_mpjpe(pred_mhr[:, spec["mhr"], :],
- gt_smpl[:, J.gt_index(spec, args.gt), :]).mean(axis=1)
- kept = [s for s, k in zip(samples, ok) if k]
- per_seq = {}
- for name in seqs:
- m = np.array([s["sequence"] == name for s in kept])
- if m.any():
- per_seq[name] = dict(n=int(m.sum()), PA_MPJPE_mm=float(e[m].mean()))
- report["per_sequence_J14"] = per_seq
-
- out = Path(args.out)
- out.parent.mkdir(parents=True, exist_ok=True)
- out.write_text(json.dumps(report, indent=2))
-
- def _table(title, rows):
- print(f"\n{title}")
- print(f" {'joint set':<14s} {'N':>7s} {'MPJPE':>10s} {'PA-MPJPE':>10s}")
- for r in rows:
- print(f" {r['joint_set']:<14s} {r['num_samples']:7d} "
- f"{r['MPJPE_mm']:9.1f} {r['PA_MPJPE_mm']:9.1f}")
-
- print("\n" + "=" * 60)
- print(f"3DPW {args.split} | {report['num_samples']} person-frames "
- f"| {len(seqs)} sequences | GT: {args.gt} (mm)")
- print("=" * 60)
- _table("ALL frames:", report["results"])
- if "results_unseen_frames" in report:
- c = report["contamination"]
- _table(f"Frames NOT in training data "
- f"({c['unseen_frames']}/{report['num_samples']}, "
- f"{100*(1-c['seen_fraction']):.1f}%):",
- report["results_unseen_frames"])
- if "results_seen_frames" in report:
- _table("Frames SEEN in training (leaked — for reference only):",
- report["results_seen_frames"])
- print("=" * 60)
- print(f"written to {out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_3dpw_ckpt.py b/benchmark/eval_3dpw_ckpt.py
deleted file mode 100644
index fe4b59b..0000000
--- a/benchmark/eval_3dpw_ckpt.py
+++ /dev/null
@@ -1,288 +0,0 @@
-#!/usr/bin/env python3
-"""3DPW evaluation for a training checkpoint (.pth), not an ONNX export.
-
-``eval_3dpw.py`` scores exported graphs; this scores checkpoints straight out of
-``runs/``. The crop and the MHR forward pass come from ``val3dpw.py`` (which was
-cross-validated against ``eval_3dpw.py``) and the metrics are ``eval_3dpw.py``'s
-own functions, so the two scripts report the same numbers for the same weights.
-
- # headline: published-protocol GT on the test split
- python benchmark/eval_3dpw_ckpt.py --ckpt runs/*/best_student_model_v3.pth \
- --sequence-dir .../sequenceFiles --image-root .../imageFiles \
- --split test
-
- # the MHR->J14 adapter, fitted on train and applied to test
- python benchmark/eval_3dpw_ckpt.py --ckpt <ckpt> ... --split train \
- --stride 10 --fit-adapter benchmark/results/adapter_j14_h36m.npz
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
-sys.path.insert(0, str(ROOT / "benchmark"))
-
-import eval_3dpw as E # noqa: E402
-import train_distill_mhr_only as T # noqa: E402
-import val3dpw # noqa: E402
-from benchlib import joints as J # noqa: E402
-from benchlib.detboxes import apply_detector_boxes # noqa: E402
-from benchlib import feedback as FB # noqa: E402
-
-MM = 1000.0
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ckpt", nargs="+", required=True)
- p.add_argument("--sequence-dir", required=True)
- p.add_argument("--image-root", required=True)
- p.add_argument("--split", default="test")
- p.add_argument("--stride", type=int, default=1)
- p.add_argument("--bbox", default="gt-joints", choices=["gt-joints", "detector"],
- help="gt-joints: projected GT joints padded 1.2x, the published "
- "protocol. detector: RF-DETR's box matched to each GT "
- "person, as the deployed pipeline crops "
- "(benchlib/detboxes.py); missed detections are dropped.")
- p.add_argument("--detector-variant", default="medium",
- help="RF-DETR variant for --bbox detector (the demo's default)")
- p.add_argument("--gt", nargs="+", default=["h36m"],
- choices=["h36m", "jointpositions"])
- p.add_argument("--gt-dir", default=None)
- p.add_argument("--adapter", default=None,
- help="npz with a fitted MHR70->J14 regressor to also report")
- p.add_argument("--fit-adapter", default=None,
- help="fit that regressor on THIS split and write it here "
- "(run on train, never on test)")
- p.add_argument("--landmarks", default="fitted", choices=["fitted", "exact"],
- help="how the 70 MHR keypoints are read off the prediction. "
- "fitted (default): the (70, 127) skeleton matrix, which "
- "is what the exported graph's consumer uses and what "
- "every recorded number in benchmark/results was scored "
- "with. exact: the teacher's mesh+joint mapping, the "
- "operator the stored annotations -- and so the adapter "
- "-- came out of. Worth 0.13 mm of J14+adapter PA-MPJPE "
- "on 3DPW test, with or without --exact-landmarks "
- "training.")
- p.add_argument("--backbone", default=None,
- help="override the backbone; normally read from the "
- "run_config.json beside the checkpoint")
- p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true",
- default=None,
- help="force the focal-aware CLIFF conditioning; normally "
- "read from run_config.json")
- p.add_argument("--no-cliff-focal", dest="cliff_focal", action="store_false",
- help="force the pixel-normalised CLIFF conditioning")
- p.add_argument("--prev", default="masked", choices=["masked", "feedback"],
- help="--prev-token checkpoints only. masked (default): the "
- "previous pose withheld, the single-frame model. "
- "feedback: the model's own prediction --prev-lag frames "
- "earlier in the same track (benchlib/feedback.py).")
- p.add_argument("--prev-lag", type=int, default=5,
- help="frames between a prediction and the one it feeds "
- "(5 = 167 ms at 30 fps, inside the training pairs)")
- p.add_argument("--batch-size", type=int, default=64)
- p.add_argument("--num-workers", type=int, default=6)
- p.add_argument("--device", default="cuda")
- p.add_argument("--out", default=None)
- return p.parse_args()
-
-
-def predict(model, loader, mhr_module, device, use_amp: bool = True,
- exact_landmarks: bool = False):
- """(N, 70, 3) predicted keypoints in mm, plus the indices they came from.
-
- The indices let the same forward pass be scored against more than one GT
- source: the sample list is identical whichever GT ``build_samples`` returns,
- only the joint values differ.
- """
- model.eval()
- preds, keeps, n_dropped, base = [], [], 0, 0
- for b in loader:
- img = b["image"].to(device, non_blocking=True)
- cc = b["cliff_cond"].to(device, non_blocking=True)
- with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
- out = model(img, cc)
- kp = mhr_module.get_native_keypoints(out["mhr_params"].float(),
- out["shape_params"].float(),
- exact=exact_landmarks)
- # Same guard as val3dpw.evaluate: a non-finite prediction makes the
- # Procrustes SVD raise rather than return a large error.
- finite = torch.isfinite(kp).flatten(1).all(dim=1).cpu()
- keep = b["ok"].bool() & finite
- n_dropped += int((b["ok"].bool() & ~finite).sum())
- if bool(keep.any()):
- preds.append(kp[keep.to(kp.device)].float().cpu().numpy())
- keeps.append(base + np.nonzero(keep.numpy())[0])
- base += int(keep.shape[0])
- return (np.concatenate(preds) * MM, np.concatenate(keeps), n_dropped)
-
-
-def main():
- args = parse_args()
- device = torch.device(args.device)
-
- # Rebuild each checkpoint's real training configuration instead of the
- # defaults: `--bound-scales` changes the forward pass and `--cliff-focal`
- # changes what the conditioning vector means. Getting either wrong scores
- # a different model than the one that was trained. See
- # T.config_from_checkpoint.
- cfgs = {}
- for ck_path in args.ckpt:
- st = torch.load(ck_path, map_location="cpu", weights_only=False)
- st = st.get("model_state_dict", st)
- cfgs[ck_path], prov = T.config_from_checkpoint(
- st, ck_path, backbone=args.backbone, cliff_focal=args.cliff_focal)
- print(f"[cfg] {Path(ck_path).parent.name}/{Path(ck_path).name}: "
- f"backbone={cfgs[ck_path].backbone} "
- f"bound_scales={cfgs[ck_path].bound_scales} "
- f"cliff_focal={cfgs[ck_path].cliff_focal} "
- f"[{prov.get('cliff_focal', 'default')}]")
- del st
-
- # One shared 3DPW loader, so every checkpoint must agree on the conditioning.
- focals = {c.cliff_focal for c in cfgs.values()}
- if len(focals) > 1:
- raise SystemExit(
- "the checkpoints disagree on --cliff-focal, and the crop/"
- "conditioning is built once for all of them.\n"
- "Evaluate the two groups in separate invocations.")
- cliff_focal = focals.pop()
- sizes = {c.image_size for c in cfgs.values()}
- if len(sizes) > 1:
- raise SystemExit(
- f"the checkpoints disagree on the input size ({sorted(sizes)}), and the "
- "crops are built once for all of them.\n"
- "Evaluate the groups in separate invocations.")
- input_size = sizes.pop()
-
- cfg = next(iter(cfgs.values()))
- exact_lm = args.landmarks == "exact"
- mhr = T.MHRForwardPass(
- cfg.mhr_model_path, device, kp_regressor=np.load(cfg.kp_regressor_path),
- landmark_assets=cfg.landmark_assets_path if exact_lm else None)
- print(f"[readout] 70 keypoints via the "
- f"{'teacher exact mesh+joint mapping' if exact_lm else 'fitted (70, 127) skeleton matrix'}")
- W_adapt = W_adapt_gt = None
- if args.adapter:
- ad = np.load(args.adapter)
- W_adapt, W_adapt_gt = ad["W"], str(ad["gt"]) if "gt" in ad else "unknown"
-
- # One dataset per GT source: same samples and same crops, different joint
- # values, so a single forward pass scores against all of them.
- sets, det_stats = {}, None
- for gt in args.gt:
- ds = val3dpw.ThreeDPWValSet(args.sequence_dir, args.image_root,
- split=args.split, stride=args.stride,
- gt=gt, gt_dir=args.gt_dir,
- cliff_focal=cliff_focal,
- input_size=input_size)
- if args.bbox == "detector":
- det_stats = apply_detector_boxes(ds, f"3dpw_{args.split}",
- args.detector_variant)
- print(f"[detector] RF-DETR {args.detector_variant}: {det_stats['n_missed']} of "
- f"{det_stats['n_frames']} person-frames missed "
- f"({100 * det_stats['miss_rate']:.2f}%), mean IoU with the GT "
- f"person {det_stats['mean_iou']:.3f}")
- sets[gt] = ds
- print(f"[3DPW] {args.split} / GT={gt}: {len(ds):,} person-frames "
- f"(stride {args.stride}), CLIFF conditioning "
- f"{'angular (real per-sequence focal)' if cliff_focal else 'pixel-normalised'}",
- flush=True)
- first = sets[args.gt[0]]
- loader = torch.utils.data.DataLoader(
- first, batch_size=args.batch_size, shuffle=False,
- num_workers=args.num_workers, pin_memory=True)
-
- report = {}
- hdr = (f"\n {'run':<24s} {'ckpt':<6s} {'ep':>4s} {'GT':<15s} {'J12 PA':>8s} "
- f"{'J12 MPJPE':>10s} {'J14 PA':>8s} {'J14 MPJPE':>10s}"
- + (f" {'J14+ad PA':>10s} {'J14+ad MPJPE':>13s}"
- if W_adapt is not None else ""))
- print(hdr)
-
- for ck_path in args.ckpt:
- model = T.InstantHMRStudent(cfgs[ck_path], pretrained=False).to(device)
- ck = torch.load(ck_path, map_location="cpu", weights_only=False)
- # Strict: an unexpected key means the config restored above is wrong
- # and the model being scored is not the model that was trained.
- model.load_state_dict(ck["model_state_dict"])
- fb = args.prev == "feedback" and cfgs[ck_path].prev_token
- if fb:
- pred, idx, n_dropped = FB.predict_feedback(
- model, first, mhr, device, lag=args.prev_lag, use_amp=cfg.use_amp,
- exact_landmarks=exact_lm, batch_size=args.batch_size,
- num_workers=args.num_workers)
- else:
- pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp,
- exact_landmarks=exact_lm)
- del model
- torch.cuda.empty_cache()
-
- for gt, ds in sets.items():
- gt_arr = np.asarray(ds.gt, dtype=np.float64)[idx] * MM
- rows = {s: E.evaluate(pred, gt_arr, s, gt) for s in ("J14", "J12")}
- # An adapter absorbs the offset between the MHR rig and one specific
- # GT convention, so scoring it against the other one is meaningless.
- jit = None
- if W_adapt is not None and W_adapt_gt == gt:
- rows["J14+adapter"] = E.evaluate_adapter(pred, gt_arr, W_adapt, gt)
- pj = np.einsum("jk,nkc->njc", W_adapt, pred - pred.mean(axis=1, keepdims=True))
- gj = gt_arr[:, J.gt_index(J.JOINT_SETS["J14"], gt), :]
- jit = FB.accel(pj, gj, J.pelvis(pj), J.pelvis(gj), [ds.samples[i] for i in idx])
- report[f"{ck_path}|{gt}"] = dict(
- ckpt=ck_path, gt=gt, split=args.split, stride=args.stride,
- bbox=args.bbox, detector=det_stats,
- epoch=ck.get("epoch"), source=ck.get("source"),
- landmarks=args.landmarks,
- prev="feedback" if fb else "masked",
- prev_lag=args.prev_lag if fb else None, accel_J14_adapter=jit,
- n=int(pred.shape[0]), n_dropped=n_dropped,
- results=list(rows.values()))
-
- run = Path(ck_path).parent.name
- name = Path(ck_path).stem.replace("best_student_model_", "")
- line = (f" {run:<24s} {name:<6s} {str(ck.get('epoch')):>4s} {gt:<15s} "
- f"{rows['J12']['PA_MPJPE_mm']:8.2f} {rows['J12']['MPJPE_mm']:10.2f} "
- f"{rows['J14']['PA_MPJPE_mm']:8.2f} {rows['J14']['MPJPE_mm']:10.2f}")
- if W_adapt is not None:
- # Both metrics adapter-applied. The raw J14 columns read ~35 mm
- # higher on MPJPE because they compare the MHR rig's own joint
- # positions against the H36M convention; the adapter is what
- # removes that fixed convention offset, so it belongs on MPJPE
- # every bit as much as on PA-MPJPE.
- line += (f" {rows['J14+adapter']['PA_MPJPE_mm']:10.2f}"
- f" {rows['J14+adapter']['MPJPE_mm']:13.2f}"
- if "J14+adapter" in rows
- else f" {'n/a':>10s} {'n/a':>13s}")
- if jit and jit["n_triplets"]:
- line += f" accel {jit['accel_mm']:.2f} err {jit['accel_err_mm']:.2f} mm/f^2"
- line += " [feedback]" if fb else ""
- print(line + (f" ({n_dropped} dropped)" if n_dropped else ""), flush=True)
-
- if args.fit_adapter:
- if args.split == "test":
- raise SystemExit("refusing to fit the adapter on the test split")
- Wf = E.fit_adapter(pred, gt_arr, gt)
- Path(args.fit_adapter).parent.mkdir(parents=True, exist_ok=True)
- np.savez(args.fit_adapter, W=Wf, split=args.split, gt=gt, ckpt=ck_path)
- print(f" fitted adapter on '{args.split}' -> {args.fit_adapter}")
-
- if args.out:
- Path(args.out).parent.mkdir(parents=True, exist_ok=True)
- Path(args.out).write_text(json.dumps(report, indent=2))
- print(f"\nwritten to {args.out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_all.py b/benchmark/eval_all.py
deleted file mode 100644
index c33ffd4..0000000
--- a/benchmark/eval_all.py
+++ /dev/null
@@ -1,213 +0,0 @@
-#!/usr/bin/env python3
-"""One command per checkpoint: run every benchmark route, print one table.
-
- python benchmark/eval_all.py --ckpt <run>/best_student_model_v3.pth
-
-Runs four evaluations and consolidates them:
-
- 3DPW test J14 (H36M) + adapter eval_3dpw_ckpt.py
- 3DPW test SMPL24 + PVE, J14 adapter-free eval_smpl_fit.py
- EMDB-1 SMPL24 + adapter eval_emdb_ckpt.py
- EMDB-1 SMPL24 + PVE eval_smpl_fit.py
-
-``--table-only`` skips the compute and re-prints the table from whatever is
-already in ``benchmark/results/``, which is what you want after a run or when
-adding a checkpoint to an existing comparison.
-
-Paths default to this machine's layout; override per site. See METRICS.md for
-what each row means and which space it is computed in.
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import subprocess
-import sys
-from pathlib import Path
-
-ROOT = Path(__file__).resolve().parent.parent
-RESULTS = ROOT / "benchmark" / "results"
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ckpt", nargs="*", default=[])
- p.add_argument("--emdb-root", default="/home/madjel/Downloads/EMDB_root")
- p.add_argument("--sequence-dir",
- default="/home/madjel/Downloads/sequenceFiles/sequenceFiles")
- p.add_argument("--image-root", default="/home/madjel/Downloads/imageFiles")
- p.add_argument("--stride", type=int, default=1)
- p.add_argument("--fit-iters", type=int, default=400)
- p.add_argument("--fit-batch", type=int, default=128)
- p.add_argument("--table-only", action="store_true")
- p.add_argument("--skip", nargs="*", default=[],
- choices=["3dpw-adapter", "3dpw-mesh", "emdb-adapter", "emdb-mesh"])
- return p.parse_args()
-
-
-def run(cmd: list[str]) -> None:
- print(f"\n$ {' '.join(cmd)}\n", flush=True)
- subprocess.run(cmd, cwd=ROOT, check=True)
-
-
-def group_by_conditioning(ckpts: list[str]) -> list[list[str]]:
- """Split checkpoints into groups that can share one crop pipeline.
-
- ``eval_3dpw_ckpt.py`` and ``eval_emdb_ckpt.py`` build the dataset once for
- all their checkpoints, so they refuse a mix of ``--cliff-focal`` settings --
- the conditioning vector means something different in each, and feeding the
- wrong one raises nothing, it just mis-places the person in depth. Runs from
- before and after 2026-09-06 therefore have to go in separate invocations,
- which this works out automatically instead of making the caller notice.
- """
- import torch
- sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
- import train_distill_mhr_only as T
- groups: dict[tuple[bool, int], list[str]] = {}
- for c in ckpts:
- st = torch.load(c, map_location="cpu", weights_only=False)
- cfg, _ = T.config_from_checkpoint(st.get("model_state_dict", st), c)
- # the input size too: the harnesses build one crop set per invocation
- groups.setdefault((bool(cfg.cliff_focal), int(cfg.image_size)), []).append(c)
- del st
- return list(groups.values())
-
-
-def merge_json(parts: list[Path], dest: Path) -> None:
- """Concatenate per-group reports into the one file the table reads."""
- out: dict = {}
- for f in parts:
- if f.is_file():
- out.update(json.loads(f.read_text()))
- f.unlink()
- dest.parent.mkdir(parents=True, exist_ok=True)
- dest.write_text(json.dumps(out, indent=2))
-
-
-def _load(name: str) -> dict:
- f = RESULTS / name
- return json.loads(f.read_text()) if f.is_file() else {}
-
-
-def _run_name(key: str) -> str:
- parts = key.split("|")[0].split("/")
- return parts[-3] if len(parts) >= 3 else key
-
-
-def table() -> None:
- """The full comparison, one row per checkpoint per dataset."""
- dpw_ad = _load("3dpw_test_alladapter.json")
- dpw_me = _load("3dpw_test_smplfit.json")
- emdb_ad = _load("emdb1_alladapter.json")
- emdb_me = _load("emdb1_smplfit_gt-joints.json")
-
- runs: list[str] = []
- for src in (dpw_me, emdb_me, dpw_ad, emdb_ad):
- for k in src:
- n = _run_name(k)
- if n not in runs:
- runs.append(n)
-
- def pick(src, name, *fields):
- for k, v in src.items():
- if _run_name(k) == name:
- out = []
- for f in fields:
- if f.startswith("@"): # a joint_set row
- r = next((r for r in v.get("results", [])
- if r["joint_set"] == f[1:]), None)
- out.append(r)
- else:
- out.append(v.get(f))
- return out
- return [None] * len(fields)
-
- def fmt(v, nd=2):
- return "-" if v is None else f"{v:.{nd}f}"
-
- print("\n" + "=" * 108)
- print("3DPW test — all 24 sequences, published-protocol GT")
- print("=" * 108)
- print(f"{'run':<12}{'ep':>5} | {'J14 PA':>8}{'J14 MPJPE':>11} (adapter)"
- f" | {'J14 PA':>8}{'J14 MPJPE':>11} (via mesh fit)"
- f" | {'SMPL24 PA':>10}{'MPJPE':>8}{'PVE':>8}")
- for n in runs:
- (ep,) = pick(dpw_me, n, "epoch")
- a, = pick(dpw_ad, n, "@J14+adapter")
- m = pick(dpw_me, n, "J14_h36m_viamesh_PA_MPJPE_mm", "J14_h36m_viamesh_MPJPE_mm",
- "PA_MPJPE_mm", "MPJPE_mm", "PVE_mm")
- print(f"{n:<12}{fmt(ep,0):>5} | "
- f"{fmt(a['PA_MPJPE_mm']) if a else '-':>8}"
- f"{fmt(a['MPJPE_mm']) if a else '-':>11} "
- f" | {fmt(m[0]):>8}{fmt(m[1]):>11} "
- f" | {fmt(m[2]):>10}{fmt(m[3]):>8}{fmt(m[4]):>8}")
-
- print("\n" + "=" * 108)
- print("EMDB-1 — all 17 sequences, the 'EMDB (24)' protocol")
- print("=" * 108)
- print(f"{'run':<12}{'ep':>5} | {'SMPL24 PA':>10}{'MPJPE':>9} (joint adapter)"
- f" | {'SMPL24 PA':>10}{'MPJPE':>9}{'PVE':>8}{'PA-PVE':>9} (mesh fit)")
- for n in runs:
- (ep,) = pick(emdb_me, n, "epoch")
- a, = pick(emdb_ad, n, "@SMPL24+adapter")
- m = pick(emdb_me, n, "PA_MPJPE_mm", "MPJPE_mm", "PVE_mm", "PA_PVE_mm")
- print(f"{n:<12}{fmt(ep,0):>5} | "
- f"{fmt(a['PA_MPJPE_mm']) if a else '-':>10}"
- f"{fmt(a['MPJPE_mm']) if a else '-':>9} "
- f" | {fmt(m[0]):>10}{fmt(m[1]):>9}{fmt(m[2]):>8}{fmt(m[3]):>9}")
- print("=" * 108)
- print("Both J14 columns convert MHR->H36M; neither is adapter-free. We predict")
- print("MHR, so a conversion is mandatory -- the joint adapter is the primary row.")
- print("PVE is pelvis-aligned (the published definition); PA-PVE is the")
- print("Procrustes variant and is NOT what papers call PVE. See METRICS.md.")
- print("Conversion floors — mesh: 11.44 MPJPE / 10.66 PA / 13.61 PVE;")
- print(" adapter: 25.11 MPJPE / 18.77 PA / no PVE.\n")
-
-
-def main():
- args = parse_args()
- if not args.table_only:
- if not args.ckpt:
- raise SystemExit("--ckpt is required unless --table-only")
- py = [sys.executable]
- common = ["--stride", str(args.stride)]
- groups = group_by_conditioning(args.ckpt)
- if "3dpw-adapter" not in args.skip:
- parts = []
- for gi, grp in enumerate(groups):
- dst = RESULTS / f"_dpw_ad_{gi}.json"
- run(py + ["benchmark/eval_3dpw_ckpt.py", "--ckpt", *grp,
- "--sequence-dir", args.sequence_dir,
- "--image-root", args.image_root, "--split", "test",
- "--adapter", "benchmark/results/adapter_j14_h36m_teacher.npz",
- "--out", str(dst)] + common)
- parts.append(dst)
- merge_json(parts, RESULTS / "3dpw_test_alladapter.json")
- if "emdb-adapter" not in args.skip:
- parts = []
- for gi, grp in enumerate(groups):
- dst = RESULTS / f"_emdb_ad_{gi}.json"
- run(py + ["benchmark/eval_emdb_ckpt.py", "--ckpt", *grp,
- "--emdb-root", args.emdb_root, "--out", str(dst)] + common)
- parts.append(dst)
- merge_json(parts, RESULTS / "emdb1_alladapter.json")
- mesh = ["--fit-iters", str(args.fit_iters), "--fit-batch", str(args.fit_batch)]
- if "3dpw-mesh" not in args.skip:
- run(py + ["benchmark/eval_smpl_fit.py", "--dataset", "3dpw",
- "--split", "test", "--ckpt", *args.ckpt,
- "--sequence-dir", args.sequence_dir,
- "--image-root", args.image_root,
- "--out", "benchmark/results/3dpw_test_smplfit.json"]
- + common + mesh)
- if "emdb-mesh" not in args.skip:
- run(py + ["benchmark/eval_smpl_fit.py", "--dataset", "emdb",
- "--ckpt", *args.ckpt, "--emdb-root", args.emdb_root,
- "--out", "benchmark/results/emdb1_smplfit_gt-joints.json"]
- + common + mesh)
- table()
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_coco.py b/benchmark/eval_coco.py
deleted file mode 100644
index bd59c3b..0000000
--- a/benchmark/eval_coco.py
+++ /dev/null
@@ -1,229 +0,0 @@
-#!/usr/bin/env python3
-"""COCO val2017 keypoint evaluation for InstantHMR (OKS AP/AR, PCK, NME).
-
-This is a *top-down with ground-truth boxes* protocol: one crop per annotated
-person, so the numbers measure pose accuracy alone and are not comparable to
-detector-in-the-loop AP from the COCO leaderboard. It is the standard ablation
-protocol for single-person pose regressors, and it is the honest one for a
-model that never sees a detector at eval time.
-
-InstantHMR regresses joint *positions* with no per-keypoint confidence, so
-every predicted keypoint is emitted with score 1.0. That is fine for OKS AP
-here (exactly one prediction per GT person) but it means AP cannot be improved
-by confidence ranking the way heatmap methods do — read AP alongside the mean
-OKS and PCK numbers, which are confidence-free.
-
-Usage:
- python benchmark/eval_coco.py \
- --onnx models/instanthmr_mhr_only_ckpt90.onnx \
- --coco-root benchmark/data/coco
-"""
-
-from __future__ import annotations
-
-import argparse
-import contextlib
-import io
-import json
-import sys
-from pathlib import Path
-
-import numpy as np
-
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-
-from benchlib import joints as J
-from benchlib import metrics as M
-from benchlib.runner import Sample, build_model, run_samples
-
-# Official COCO per-keypoint OKS tolerances.
-OKS_SIGMAS = np.array(
- [.26, .25, .25, .35, .35, .79, .79, .72, .72, .62, .62,
- 1.07, 1.07, .87, .87, .89, .89]
-) / 10.0
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--onnx", required=True)
- p.add_argument("--coco-root", default="benchmark/data/coco",
- help="dir holding val2017/ and annotations/")
- p.add_argument("--ann", default=None,
- help="override path to person_keypoints_val2017.json")
- p.add_argument("--images", default=None, help="override path to val2017/")
- p.add_argument("--min-keypoints", type=int, default=1,
- help="skip GT persons with fewer labelled keypoints")
- p.add_argument("--device", default="cuda")
- p.add_argument("--mhr-script", default=None)
- p.add_argument("--num-workers", type=int, default=6)
- p.add_argument("--limit", type=int, default=0)
- p.add_argument("--hmr2-subset",
- default="benchmark/data/hmr2_evaluation_data/coco_val.npz",
- help="4D-Humans' coco_val.npz; enables the HMR2.0 / SAM 3D Body PCK")
- p.add_argument("--out", default="benchmark/results/coco.json")
- return p.parse_args()
-
-
-def oks(pred: np.ndarray, gt: np.ndarray, vis: np.ndarray,
- area: np.ndarray) -> np.ndarray:
- """Per-sample OKS. pred/gt are (N, 17, 2), vis (N, 17), area (N,)."""
- d2 = ((pred - gt) ** 2).sum(axis=-1)
- var = (2 * OKS_SIGMAS) ** 2
- e = d2 / var[None, :] / (area[:, None] + np.spacing(1)) / 2.0
- m = vis > 0
- out = np.full(len(pred), np.nan)
- has = m.any(axis=1)
- out[has] = np.array([np.exp(-e[i][m[i]]).mean() for i in np.where(has)[0]])
- return out
-
-
-def main():
- args = parse_args()
- from pycocotools.coco import COCO
- from pycocotools.cocoeval import COCOeval
-
- root = Path(args.coco_root)
- ann_file = Path(args.ann) if args.ann else root / "annotations" / "person_keypoints_val2017.json"
- img_dir = Path(args.images) if args.images else root / "val2017"
- for pth in (ann_file, img_dir):
- if not pth.exists():
- raise FileNotFoundError(
- f"{pth} not found — run: python benchmark/download.py --coco"
- )
-
- print(f"[COCO] loading {ann_file}")
- with contextlib.redirect_stdout(io.StringIO()):
- coco = COCO(str(ann_file))
-
- samples: list[Sample] = []
- for ann_id in sorted(coco.getAnnIds(catIds=[1], iscrowd=False)):
- a = coco.anns[ann_id]
- if a["num_keypoints"] < args.min_keypoints or a["area"] <= 0:
- continue
- x, y, w, h = a["bbox"]
- if w <= 1 or h <= 1:
- continue
- info = coco.imgs[a["image_id"]]
- samples.append(Sample(
- image_path=str(img_dir / info["file_name"]),
- bbox=np.array([x, y, x + w, y + h], dtype=np.float32),
- meta=dict(ann_id=ann_id, image_id=a["image_id"],
- keypoints=np.array(a["keypoints"], dtype=np.float64).reshape(17, 3),
- area=float(a["area"]), bbox_wh=(float(w), float(h))),
- ))
- # Group people from the same image together so each JPEG is decoded once.
- samples.sort(key=lambda s: s.image_path)
- if args.limit:
- samples = samples[:args.limit]
- print(f"[COCO] {len(samples)} annotated persons")
-
- model = build_model(args.onnx, args.device, args.mhr_script)
- preds, ok = run_samples(model, samples, num_workers=args.num_workers, desc="COCO")
- if not ok.all():
- print(f"[COCO] WARNING: {int((~ok).sum())} samples had unreadable images")
-
- kept = [s for s, k in zip(samples, ok) if k]
- pred_kp = np.stack([
- p.joints_2d[J.COCO17_FROM_MHR70] for p, k in zip(preds, ok) if k
- ]).astype(np.float64)
- gt_kp = np.stack([s.meta["keypoints"] for s in kept])
- gt_xy, gt_vis = gt_kp[:, :, :2], gt_kp[:, :, 2]
- areas = np.array([s.meta["area"] for s in kept])
- diag = np.array([np.hypot(*s.meta["bbox_wh"]) for s in kept])
-
- # --- confidence-free metrics -----------------------------------------
- valid = gt_vis > 0
- sample_oks = oks(pred_kp, gt_xy, gt_vis, areas)
- summary = {
- "num_samples": len(kept),
- "mean_OKS": float(np.nanmean(sample_oks)),
- # 0.05 is the threshold most 2D-pose papers quote; 0.1/0.2 are kept so
- # older reports in benchmark/results/ stay comparable. All three are
- # normalised by the BBOX DIAGONAL -- a paper normalising by head
- # segment (PCKh) or torso is not comparable at the same threshold.
- "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.05),
- "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.1),
- "[email protected]_bboxdiag": M.pck2d(pred_kp, gt_xy, valid, diag, 0.2),
- "NME_bboxdiag": M.nme2d(pred_kp, gt_xy, valid, diag),
- }
- # --- the published protocol: HMR2.0 Table 2, SAM 3D Body Table 2 -----
- # 4D-Humans' EvaluatorPCK, reproduced: only the 4,648 persons listed in
- # their coco_val.npz (no simple rule selects them -- >=6 keypoints alone
- # gives 5,430), distance / max(w, h) of the tight GT box, PCK per keypoint
- # then averaged over the 17 (mmpose's avg_acc, not a pooled mean).
- hmr2 = Path(args.hmr2_subset)
- if hmr2.is_file():
- z = np.load(hmr2)
- listed = {}
- for n, c in zip(z["imgname"], z["center"]):
- listed.setdefault(Path(str(n)).name, []).append(c)
- sub = np.array([
- any(np.abs(np.array([s.bbox[0] + s.bbox[2], s.bbox[1] + s.bbox[3]]) / 2 - c).max() < 0.02
- for c in listed.get(Path(s.image_path).name, []))
- for s in kept])
- side = np.array([max(s.meta["bbox_wh"]) for s in kept])[sub]
- d = np.linalg.norm(pred_kp[sub] - gt_xy[sub], axis=-1) / side[:, None]
- v = valid[sub]
- summary["hmr2_protocol"] = {"num_samples": int(sub.sum()),
- "num_listed": int(len(z["imgname"]))}
- for t in (0.05, 0.1):
- summary["hmr2_protocol"][f"PCK@{t}"] = float(np.mean(
- [(d[v[:, j], j] < t).mean() for j in range(17) if v[:, j].any()]))
-
- per_kp = {}
- for i, name in enumerate(J.COCO17_NAMES):
- v = valid[:, i]
- d_i = (np.linalg.norm(pred_kp[v, i] - gt_xy[v, i], axis=-1) / diag[v]
- if v.any() else None)
- per_kp[name] = {
- "n": int(v.sum()),
- "[email protected]": float("nan") if d_i is None else float((d_i < 0.05).mean()),
- "[email protected]": float("nan") if d_i is None else float((d_i < 0.1).mean()),
- }
-
- # --- OKS AP through the official evaluator ---------------------------
- results = [dict(
- image_id=int(s.meta["image_id"]), category_id=1, score=1.0,
- keypoints=np.concatenate(
- [pred_kp[i], np.ones((17, 1))], axis=1).reshape(-1).tolist(),
- ) for i, s in enumerate(kept)]
-
- with contextlib.redirect_stdout(io.StringIO()):
- dt = coco.loadRes(results)
- ev = COCOeval(coco, dt, "keypoints")
- ev.params.imgIds = sorted({r["image_id"] for r in results})
- ev.evaluate(); ev.accumulate()
- ev.summarize()
- ap_keys = ["AP", "AP50", "AP75", "AP_medium", "AP_large",
- "AR", "AR50", "AR75", "AR_medium", "AR_large"]
- summary["oks_ap"] = {k: float(v) for k, v in zip(ap_keys, ev.stats)}
-
- report = dict(model=str(args.onnx), dataset="COCO val2017",
- protocol="top-down, ground-truth boxes, score=1.0",
- summary=summary, per_keypoint=per_kp)
- out = Path(args.out)
- out.parent.mkdir(parents=True, exist_ok=True)
- out.write_text(json.dumps(report, indent=2))
-
- print("\n" + "=" * 68)
- print(f"COCO val2017 (GT boxes) | {summary['num_samples']} persons")
- print("=" * 68)
- print(f" OKS AP {summary['oks_ap']['AP']*100:6.2f} "
- f"AP50 {summary['oks_ap']['AP50']*100:6.2f} "
- f"AP75 {summary['oks_ap']['AP75']*100:6.2f}")
- print(f" mean OKS {summary['mean_OKS']:6.4f}")
- print(f" [email protected] {summary['[email protected]_bboxdiag']*100:6.2f} % "
- f"[email protected] {summary['[email protected]_bboxdiag']*100:6.2f} % "
- f"[email protected] {summary['[email protected]_bboxdiag']*100:6.2f} %")
- print(f" NME {summary['NME_bboxdiag']:6.4f} (bbox-diagonal units)")
- if "hmr2_protocol" in summary:
- h = summary["hmr2_protocol"]
- print(f" HMR2.0 / SAM 3D Body protocol ({h['num_samples']}/{h['num_listed']} persons):"
- print("=" * 68)
- print(f"written to {out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_emdb_ckpt.py b/benchmark/eval_emdb_ckpt.py
deleted file mode 100644
index bf09a70..0000000
--- a/benchmark/eval_emdb_ckpt.py
+++ /dev/null
@@ -1,292 +0,0 @@
-#!/usr/bin/env python3
-"""EMDB-1 evaluation for a training checkpoint (.pth).
-
-EMDB-1 is the 17-sequence, camera-frame split every paper reports as
-"EMDB (24)": 24 SMPL joints, the reference point the midpoint between the two
-hips, MPJPE and PA-MPJPE. (EMDB-2 is the global-trajectory split and is not
-what this model does.) One subject per sequence, so this is structurally a
-simpler 3DPW and reuses its crop, its forward pass and its metrics.
-
- python benchmark/eval_emdb_ckpt.py \
- --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \
- --emdb-root /path/to/EMDB \
- --adapter benchmark/results/adapter_smpl24_teacher.npz
-
-**Quote the SMPL24+adapter row.** InstantHMR regresses the 70 MHR keypoints,
-and 10 of SMPL's 24 joints (the three spines, the collars, the feet, the hands)
-have no MHR landmark at all, so a raw row does not exist for this joint set --
-the conversion has to be a fitted linear map. ``--adapter`` supplies the
-student-free one from ``fit_adapter_mhr.py --target smpl24``, fitted on 3DPW
-train teacher labels; EMDB is a different dataset, so it is a real holdout.
-Its conversion floor is 25.11 mm MPJPE / 18.77 mm PA-MPJPE -- quote that next
-to the result.
-
-The J14 and J12 rows below are the same 12/14 limb landmarks read straight off
-the MHR rig against SMPL's. They are internally comparable to this repo's 3DPW
-``--gt jointpositions`` numbers and to nothing published.
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-import cv2
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
-sys.path.insert(0, str(ROOT / "benchmark"))
-
-import eval_3dpw as E # noqa: E402
-import train_distill_mhr_only as T # noqa: E402
-import val3dpw # noqa: E402
-from benchlib import emdb as D # noqa: E402
-from benchlib.detboxes import apply_detector_boxes # noqa: E402
-from benchlib import joints as J # noqa: E402
-from benchlib import metrics as M # noqa: E402
-from eval_3dpw_ckpt import predict # noqa: E402
-from benchlib import feedback as FB # noqa: E402
-
-MM = 1000.0
-SMPL24_HIPS = (J.SMPL["left_hip"], J.SMPL["right_hip"])
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ckpt", nargs="+", required=True)
- p.add_argument("--emdb-root", required=True)
- p.add_argument("--stride", type=int, default=1)
- p.add_argument("--bbox", default="gt-joints",
- choices=["gt-joints", "annotated", "detector"],
- help="gt-joints: projected GT joints padded by --bbox-scale, "
- "this repo's 3DPW convention. annotated: EMDB's own "
- "boxes, the published 'Oracle' protocol. detector: "
- "RF-DETR's box, as the deployed pipeline crops "
- "(benchlib/detboxes.py); missed detections are dropped.")
- p.add_argument("--detector-variant", default="medium",
- help="RF-DETR variant for --bbox detector (the demo's default)")
- p.add_argument("--bbox-scale", type=float, default=1.2)
- p.add_argument("--gt-dir", default=None)
- p.add_argument("--adapter", default="benchmark/results/adapter_smpl24_teacher.npz",
- help="npz with the MHR70 -> SMPL24 regressor")
- p.add_argument("--landmarks", default="fitted", choices=["fitted", "exact"],
- help="how the 70 MHR keypoints are read off the prediction; "
- "see eval_3dpw_ckpt.py. fitted is the deployed readout "
- "and what every recorded number was scored with.")
- p.add_argument("--backbone", default=None)
- p.add_argument("--cliff-focal", dest="cliff_focal", action="store_true",
- default=None)
- p.add_argument("--no-cliff-focal", dest="cliff_focal", action="store_false")
- p.add_argument("--prev", default="masked", choices=["masked", "feedback"],
- help="--prev-token checkpoints only; see eval_3dpw_ckpt.py")
- p.add_argument("--prev-lag", type=int, default=5)
- p.add_argument("--batch-size", type=int, default=64)
- p.add_argument("--num-workers", type=int, default=8)
- p.add_argument("--device", default="cuda")
- p.add_argument("--out", default=None)
- return p.parse_args()
-
-
-class EMDBValSet(torch.utils.data.Dataset):
- """(crop, cliff_cond, ok) for every evaluatable EMDB-1 frame.
-
- The crop and the conditioning vector come from ``val3dpw._preprocess``, not
- a copy of it: that function is the transcription of the training dataset's
- own crop, and the three places that build the CLIFF vector must agree or
- the person is silently mis-placed in depth.
- """
-
- def __init__(self, emdb_root, stride=1, bbox="gt-joints", bbox_scale=1.2,
- gt_dir=None, cliff_focal=False, input_size=val3dpw.INPUT_SIZE):
- self.samples, gt = D.build_samples(
- emdb_root, stride=stride, bbox=bbox, bbox_scale=bbox_scale,
- gt_dir=gt_dir)
- self.gt = np.asarray(gt, dtype=np.float32)
- self.cliff_focal = cliff_focal
- self.input_size = input_size
-
- def __len__(self):
- return len(self.samples)
-
- def __getitem__(self, i):
- s = self.samples[i]
- bgr = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)
- if bgr is None:
- return {"image": torch.zeros(3, self.input_size, self.input_size),
- "cliff_cond": torch.zeros(3), "ok": torch.tensor(0.0)}
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
- h, w = rgb.shape[:2]
- crop, cliff = val3dpw._preprocess(
- rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None,
- size=self.input_size)
- return {"image": torch.from_numpy(crop),
- "cliff_cond": torch.from_numpy(cliff),
- "ok": torch.tensor(1.0)}
-
-
-def evaluate_smpl24(pred_mhr: np.ndarray, gt: np.ndarray,
- W: np.ndarray) -> tuple[dict, np.ndarray]:
- """The published EMDB row, plus the per-sample PA error for the breakdown.
-
- ``W`` was fitted on centred keypoints, so it is applied to centred
- keypoints; both metrics are translation-invariant, so this costs nothing.
- """
- pred = np.einsum("jk,nkc->njc", W,
- pred_mhr - pred_mhr.mean(axis=1, keepdims=True))
- root_p = 0.5 * (pred[:, SMPL24_HIPS[0]] + pred[:, SMPL24_HIPS[1]])
- root_g = 0.5 * (gt[:, SMPL24_HIPS[0]] + gt[:, SMPL24_HIPS[1]])
- e_mpjpe = M.mpjpe(pred, gt, root=(root_p, root_g))
- e_pa = M.pa_mpjpe(pred, gt)
- return dict(
- joint_set="SMPL24+adapter", num_joints=24,
- num_samples=int(pred.shape[0]),
- MPJPE_mm=float(e_mpjpe.mean()),
- PA_MPJPE_mm=float(e_pa.mean()),
- PCK3D_50mm=M.pck3d(e_pa, 50.0),
- PCK3D_100mm=M.pck3d(e_pa, 100.0),
- AUC_0_150mm=M.auc3d(e_pa),
- per_joint_PA_MPJPE_mm={n: float(v) for n, v
- in zip(J.SMPL24_NAMES, e_pa.mean(axis=0))},
- ), e_pa.mean(axis=1)
-
-
-def main():
- args = parse_args()
- device = torch.device(args.device)
-
- # Same rule as eval_3dpw_ckpt.py: rebuild each checkpoint's real training
- # configuration. --bound-scales changes the forward pass, --cliff-focal
- # changes what the conditioning vector means, and the defaults are wrong
- # for both.
- cfgs = {}
- for ck_path in args.ckpt:
- st = torch.load(ck_path, map_location="cpu", weights_only=False)
- st = st.get("model_state_dict", st)
- cfgs[ck_path], prov = T.config_from_checkpoint(
- st, ck_path, backbone=args.backbone, cliff_focal=args.cliff_focal)
- print(f"[cfg] {Path(ck_path).parent.name}/{Path(ck_path).name}: "
- f"backbone={cfgs[ck_path].backbone} "
- f"bound_scales={cfgs[ck_path].bound_scales} "
- f"cliff_focal={cfgs[ck_path].cliff_focal} "
- f"[{prov.get('cliff_focal', 'default')}]")
- del st
-
- focals = {c.cliff_focal for c in cfgs.values()}
- if len(focals) > 1:
- raise SystemExit(
- "the checkpoints disagree on --cliff-focal, and the crop/"
- "conditioning is built once for all of them.\n"
- "Evaluate the two groups in separate invocations.")
- cliff_focal = focals.pop()
- sizes = {c.image_size for c in cfgs.values()}
- if len(sizes) > 1:
- raise SystemExit(
- f"the checkpoints disagree on the input size ({sorted(sizes)}), and the "
- "crops are built once for all of them.\n"
- "Evaluate the groups in separate invocations.")
- input_size = sizes.pop()
-
- cfg = next(iter(cfgs.values()))
- exact_lm = args.landmarks == "exact"
- mhr = T.MHRForwardPass(
- cfg.mhr_model_path, device, kp_regressor=np.load(cfg.kp_regressor_path),
- landmark_assets=cfg.landmark_assets_path if exact_lm else None)
- print(f"[readout] 70 keypoints via the "
- f"{'teacher exact mesh+joint mapping' if exact_lm else 'fitted (70, 127) skeleton matrix'}")
- ad = np.load(args.adapter)
- if str(ad["gt"]) != "smpl24":
- raise SystemExit(f"{args.adapter} targets '{ad['gt']}', not smpl24; "
- "fit one with fit_adapter_mhr.py --target smpl24")
- W_adapt = ad["W"]
-
- ds = EMDBValSet(args.emdb_root, stride=args.stride,
- bbox="gt-joints" if args.bbox == "detector" else args.bbox,
- bbox_scale=args.bbox_scale, gt_dir=args.gt_dir,
- cliff_focal=cliff_focal, input_size=input_size)
- det_stats = None
- if args.bbox == "detector":
- det_stats = apply_detector_boxes(ds, "emdb1", args.detector_variant)
- print(f"[detector] RF-DETR {args.detector_variant}: {det_stats['n_missed']} of "
- f"{det_stats['n_frames']} frames missed ({100 * det_stats['miss_rate']:.2f}%), "
- f"mean IoU with the GT person {det_stats['mean_iou']:.3f}")
- seqs = sorted({s["sequence"] for s in ds.samples})
- print(f"[EMDB1] {len(ds):,} frames across {len(seqs)} sequences "
- f"(stride {args.stride}, boxes: {args.bbox}), CLIFF conditioning "
- f"{'angular (real per-sequence focal)' if cliff_focal else 'pixel-normalised'}",
- flush=True)
- loader = torch.utils.data.DataLoader(
- ds, batch_size=args.batch_size, shuffle=False,
- num_workers=args.num_workers, pin_memory=True)
-
- report = {}
- print(f"\n {'run':<24s} {'ckpt':<6s} {'ep':>4s} "
- f"{'SMPL24+ad PA':>13s} {'SMPL24+ad MPJPE':>16s} "
- f"{'J14 PA':>8s} {'J12 PA':>8s}")
-
- for ck_path in args.ckpt:
- model = T.InstantHMRStudent(cfgs[ck_path], pretrained=False).to(device)
- ck = torch.load(ck_path, map_location="cpu", weights_only=False)
- model.load_state_dict(ck["model_state_dict"])
- fb = args.prev == "feedback" and cfgs[ck_path].prev_token
- if fb:
- pred, idx, n_dropped = FB.predict_feedback(
- model, ds, mhr, device, lag=args.prev_lag, use_amp=cfg.use_amp,
- exact_landmarks=exact_lm, batch_size=args.batch_size,
- num_workers=args.num_workers)
- else:
- pred, idx, n_dropped = predict(model, loader, mhr, device, cfg.use_amp,
- exact_landmarks=exact_lm)
- del model
- torch.cuda.empty_cache()
-
- gt_arr = np.asarray(ds.gt, dtype=np.float64)[idx] * MM
- head, e_sample = evaluate_smpl24(pred, gt_arr, W_adapt)
- # The GT is the 24 SMPL joints, i.e. threedpw's "jointpositions" layout.
- rows = [head] + [E.evaluate(pred, gt_arr, s, "jointpositions")
- for s in ("J14", "J12")]
-
- kept = [ds.samples[i] for i in idx]
- pj = np.einsum("jk,nkc->njc", W_adapt, pred - pred.mean(axis=1, keepdims=True))
- hips = lambda x: 0.5 * (x[:, SMPL24_HIPS[0]] + x[:, SMPL24_HIPS[1]])
- jit = FB.accel(pj, gt_arr, hips(pj), hips(gt_arr), kept)
- per_seq = {}
- for name in seqs:
- m = np.array([s["sequence"] == name for s in kept])
- if m.any():
- per_seq[name] = dict(n=int(m.sum()),
- PA_MPJPE_mm=float(e_sample[m].mean()))
-
- report[ck_path] = dict(
- ckpt=ck_path, dataset="EMDB1", split="emdb1", stride=args.stride,
- bbox=args.bbox, detector=det_stats,
- adapter=str(args.adapter), landmarks=args.landmarks,
- prev="feedback" if fb else "masked",
- prev_lag=args.prev_lag if fb else None, accel_SMPL24_adapter=jit,
- epoch=ck.get("epoch"), source=ck.get("source"),
- n=int(pred.shape[0]), n_dropped=n_dropped,
- num_sequences=len(seqs), results=rows,
- per_sequence_SMPL24_adapter=per_seq)
-
- run = Path(ck_path).parent.name
- name = Path(ck_path).stem.replace("best_student_model_", "")
- print(f" {run:<24s} {name:<6s} {str(ck.get('epoch')):>4s} "
- f"{head['PA_MPJPE_mm']:13.2f} {head['MPJPE_mm']:16.2f} "
- f"{rows[1]['PA_MPJPE_mm']:8.2f} {rows[2]['PA_MPJPE_mm']:8.2f}"
- + (f" accel {jit['accel_mm']:.2f} err {jit['accel_err_mm']:.2f} mm/f^2"
- if jit["n_triplets"] else "")
- + (" [feedback]" if fb else "")
- + (f" ({n_dropped} dropped)" if n_dropped else ""), flush=True)
-
- if args.out:
- Path(args.out).parent.mkdir(parents=True, exist_ok=True)
- Path(args.out).write_text(json.dumps(report, indent=2))
- print(f"\nwritten to {args.out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/eval_footlock.py b/benchmark/eval_footlock.py
deleted file mode 100644
index 4219ac9..0000000
--- a/benchmark/eval_footlock.py
+++ /dev/null
@@ -1,202 +0,0 @@
-#!/usr/bin/env python3
-"""Foot lock on the held-out semelle participants.
-
- python benchmark/eval_footlock.py --ckpt <best_student_model_v3.pth> --out benchmark/results/<name>_footlock.json
-
-Inputs are datasets_pipeline/semelle/label.py `eval` files: per labelled frame
-(every 5th video frame, 8 fps) the participant's detector box, insole contact
-per foot and the 8 mocap foot keypoints in the camera frame. Frames are
-undistorted to the centred pinhole the training split uses, the box with them,
-and the model sees the runtime crop (val3dpw._preprocess) at the calibrated focal.
-
-Two modes for a --prev-token checkpoint, one for any other:
- masked previous pose always withheld (single-frame model)
- feedback previous pose = the model's own prediction on the previous
- labelled frame; withheld on the first frame and after a gap of
- more than 20 video frames (the longest pair it was trained on)
-
-Metrics, over consecutive labelled frames 5 video frames apart (125 ms):
- slide_mm mean displacement of a foot (mean of its 4 keypoints, camera
- frame) while the insole says it is in contact in both frames
- mocap_slide_mm the same for the mocap feet: what a planted foot really moves
- foot_err_mm foot keypoint error against the mocap after removing each
- trial's median offset (the depth bias label.py also removes)
- contact_f1 per-foot contact head against the insole (prev-token only)
-"""
-from __future__ import annotations
-
-import argparse
-import json
-import sys
-from pathlib import Path
-
-import cv2
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
-sys.path.insert(0, str(ROOT / "datasets_pipeline/semelle"))
-import train_distill_mhr_only as T # noqa: E402
-import val3dpw # noqa: E402
-
-FEET = T.FOOT_KP.tolist()
-MAX_GAP = 20
-
-
-def run_trial(path, model, mhr, cfg, mode, device):
- import label as L
- z = np.load(path)
- cam = L.vc.Camera(**json.loads(str(z["camera"])))
- stem = path.stem # <session>_<trial>_<cam>
- session, rest = stem.split("_", 1)
- trial, cam_name = rest.rsplit("_", 1)
- cap = cv2.VideoCapture(str(L.MANIP / session / trial / f"{cam_name}_manip.mp4"))
- w, h = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)), int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
- K, dist, Kc = L.centred_K(cam, w, h)
- maps = cv2.initUndistortRectifyMap(K, dist, None, Kc, (w, h), cv2.CV_32FC1)
- want = {int(f): i for i, f in enumerate(z["frame"])}
- feet = np.full((len(want), 8, 3), np.nan)
- logits = np.full((len(want), 2), np.nan)
- prev, last = None, -10 ** 9
- f = 0
- while want and f <= max(want):
- if f not in want:
- cap.grab(); f += 1
- continue
- ok, bgr = cap.read()
- if not ok:
- break
- i = want[f]
- rgb = cv2.cvtColor(cv2.remap(bgr, *maps, cv2.INTER_LINEAR), cv2.COLOR_BGR2RGB)
- box = L.undistort_box(z["bbox"][i], K, dist, Kc)
- img, cliff = val3dpw._preprocess(rgb, box, h, w, focal=cam.focal_px if cfg.cliff_focal else None,
- size=cfg.image_size)
- img = torch.from_numpy(img)[None].to(device)
- cliff = torch.from_numpy(cliff)[None].to(device)
- args = ()
- if cfg.prev_token:
- use = mode == "feedback" and prev is not None and f - last <= MAX_GAP
- pp = torch.from_numpy(prev if use else np.zeros(T.PREV_DIM, np.float32))[None].to(device)
- args = (pp, torch.tensor([1.0 if use else 0.0], device=device))
- with torch.no_grad():
- o = model(img, cliff, *args)
- kp = mhr.regress_keypoints(mhr.to_vision(
- mhr.get_joints(o["mhr_params"].float(), o["shape_params"].float())[..., :3]))[0].cpu().numpy()
- t = o["cam_trans"][0].float().cpu().numpy()
- feet[i] = (kp + t)[FEET]
- if "contact_logits" in o:
- logits[i] = o["contact_logits"][0].float().cpu().numpy()
- zt = max(float(t[2]), 0.1)
- prev = np.concatenate([kp.reshape(-1), [t[0] / zt, t[1] / zt, np.log(zt)]]).astype(np.float32)
- last = f
- f += 1
- return z, feet, logits
-
-
-def lock(z, feet, planted):
- """Causal foot lock: while a foot is called planted on consecutive labelled
- frames, shift its 4 keypoints so their mean sits at the running average of
- the predicted foot since contact onset (one frame's error does not hold for
- the whole stance); released, the raw prediction comes back (``jump`` is
- that step). Camera frame, so it assumes the static semelle cameras."""
- fr = z["frame"]
- out = feet.copy()
- jump = []
- for s in range(2):
- sl = slice(4 * s, 4 * s + 4)
- total, n = None, 0
- for k in range(len(fr)):
- ok = np.isfinite(feet[k, sl]).all()
- if n and planted[k, s] and ok and fr[k] - fr[k - 1] == 5:
- total += feet[k, sl].mean(0); n += 1
- out[k, sl] = feet[k, sl] - feet[k, sl].mean(0) + total / n
- continue
- if n and k and fr[k] - fr[k - 1] == 5 and not planted[k, s]:
- jump.append(np.linalg.norm(out[k, sl].mean(0) - out[k - 1, sl].mean(0)))
- total, n = (feet[k, sl].mean(0).copy(), 1) if planted[k, s] and ok else (None, 0)
- return out, jump
-
-
-def score(z, feet, logits):
- fr = z["frame"]
- mocap = z["feet_cam"].astype(np.float64)
- c, v = z["contact"], z["contact_valid"]
- pair = np.where(np.diff(fr) == 5)[0]
- slide, mslide = [], []
- for k in pair:
- for s in range(2):
- sl = slice(4 * s, 4 * s + 4)
- if v[k, s] and v[k + 1, s] and c[k, s] and c[k + 1, s]:
- d = np.linalg.norm(feet[k + 1, sl].mean(0) - feet[k, sl].mean(0))
- dm = np.linalg.norm(mocap[k + 1, sl].mean(0) - mocap[k, sl].mean(0))
- if np.isfinite(d) and np.isfinite(dm):
- slide.append(d); mslide.append(dm)
- ok = np.isfinite(feet).all((1, 2)) & np.isfinite(mocap).all((1, 2))
- off = np.median((mocap[ok] - feet[ok]).mean(1), axis=0) if ok.any() else np.zeros(3)
- err = np.linalg.norm(mocap[ok] - off - feet[ok], axis=-1).ravel()
- tp = fp = fn = 0
- if np.isfinite(logits).any():
- m = v > 0
- pr = logits > 0
- tp, fp, fn = int((pr & (c > 0) & m).sum()), int((pr & (c == 0) & m).sum()), int((~pr & (c > 0) & m).sum())
- return dict(slide=slide, mslide=mslide, err=err.tolist(), tp=tp, fp=fp, fn=fn)
-
-
-def main() -> int:
- p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ckpt", required=True)
- p.add_argument("--eval-dir", default=str(ROOT / "data/semelle_work/eval"))
- p.add_argument("--out", required=True)
- p.add_argument("--max-trials", type=int, default=None)
- p.add_argument("--lock", action="store_true",
- help="also score the feet after lock(): '+lock' planted = the "
- "contact head (prev-token only), '+oracle' = the insole")
- args = p.parse_args()
- device = "cuda"
- state = torch.load(args.ckpt, map_location="cpu", weights_only=False)["model_state_dict"]
- cfg, prov = T.config_from_checkpoint(state, args.ckpt)
- model = T.InstantHMRStudent(cfg, pretrained=False).to(device).eval()
- model.load_state_dict(state)
- mhr = T.MHRForwardPass(cfg.mhr_model_path, device, kp_regressor=np.load(cfg.kp_regressor_path))
- files = sorted(Path(args.eval_dir).glob("*.npz"))[: args.max_trials]
- modes = ("masked", "feedback") if cfg.prev_token else ("single",)
- res = {}
- variants = [""]
- if args.lock:
- variants += (["+lock"] if cfg.prev_token else []) + ["+oracle"]
- for mode in modes:
- acc = {v: dict(slide=[], mslide=[], err=[], jump=[], tp=0, fp=0, fn=0) for v in variants}
- for fp in files:
- z, feet, logits = run_trial(fp, model, mhr, cfg, mode, device)
- for var in variants:
- jump = []
- if var == "+lock":
- feet_v, jump = lock(z, feet, logits > 0)
- elif var == "+oracle":
- feet_v, jump = lock(z, feet, (z["contact"] > 0) & (z["contact_valid"] > 0))
- else:
- feet_v = feet
- s = score(z, feet_v, logits)
- s["jump"] = jump
- for k in ("slide", "mslide", "err", "jump"):
- acc[var][k] += s[k]
- for k in ("tp", "fp", "fn"):
- acc[var][k] += s[k]
- for var in variants:
- a = acc[var]
- f1 = 2 * a["tp"] / max(2 * a["tp"] + a["fp"] + a["fn"], 1) if a["tp"] + a["fn"] else None
- res[mode + var] = dict(slide_mm=1000 * float(np.mean(a["slide"])),
- slide_median_mm=1000 * float(np.median(a["slide"])),
- mocap_slide_mm=1000 * float(np.mean(a["mslide"])),
- foot_err_mm=1000 * float(np.mean(a["err"])),
- contact_f1=f1, n_contact_pairs=len(a["slide"]))
- if var:
- res[mode + var]["release_jump_mm"] = 1000 * float(np.mean(a["jump"])) if a["jump"] else None
- print(mode + var, json.dumps(res[mode + var]))
- Path(args.out).write_text(json.dumps(dict(ckpt=args.ckpt, trials=len(files), **res), indent=1))
- return 0
-
-
-if __name__ == "__main__":
- raise SystemExit(main())
diff --git a/benchmark/eval_smpl_fit.py b/benchmark/eval_smpl_fit.py
deleted file mode 100644
index b2e3510..0000000
--- a/benchmark/eval_smpl_fit.py
+++ /dev/null
@@ -1,492 +0,0 @@
-#!/usr/bin/env python3
-"""EMDB-1 / 3DPW in SMPL space: MPJPE, PA-MPJPE and PVE via mesh conversion.
-
-The joint-adapter evaluators (``eval_emdb_ckpt.py``, ``eval_3dpw_ckpt.py``)
-score a linear MHR70 -> SMPL24 map, which carries an 18.8 mm PA-MPJPE floor and
-cannot produce vertices, so it cannot reach PVE at all. This converts the
-predicted **mesh** instead, which is what SAM 3D Body and Fast SAM 3D Body do
-for this same rig.
-
-Direction matters and only one is publishable: the prediction is fitted into
-SMPL, and the ground truth is left exactly as the benchmark defines it. Fitting
-the GT to the MHR rig instead would change the joint set, the mesh and the
-reference all at once, and no such number belongs in a published table however
-small it comes out.
-
-Three stages, none of which sees the ground truth:
-
-1. the student's MHR parameters -> the rig's own 18,439-vertex mesh;
-2. Meta's official barycentric surface map -> the same surface sampled in SMPL
- topology, so every SMPL vertex has a KNOWN target (``mhr_smpl.py`` explains
- why that matters: ICP has to discover the correspondence and reliably finds
- a wrong-limb minimum its own residual cannot see);
-3. a labelled fit for SMPL theta/beta/trans, global orientation solved in
- closed form by Procrustes.
-
-Measured end to end on a round trip through the rig, the conversion floor is
-**11.4 mm MPJPE / 10.7 mm PA-MPJPE / 13.6 mm PVE**, against 25.1 / 18.8 / n/a
-for the linear adapter. Quote it next to the result.
-
- python benchmark/eval_smpl_fit.py \\
- --ckpt instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth \\
- --emdb-root /path/to/EMDB --out benchmark/results/emdb1_smplfit.json
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import pickle
-import sys
-from pathlib import Path
-
-import cv2
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-sys.path.insert(0, str(ROOT / "instanthmr_distill_train"))
-sys.path.insert(0, str(ROOT / "benchmark"))
-
-import train_distill_mhr_only as T # noqa: E402
-import val3dpw # noqa: E402
-from benchlib import emdb as D # noqa: E402
-from benchlib import joints as J # noqa: E402
-from benchlib import metrics as M # noqa: E402
-from eval_emdb_ckpt import EMDBValSet # noqa: E402
-from mhr_smpl import (MHRRig, SMPLModel, barycentric_transfer, # noqa: E402
- fit_smpl_to_targets, load_surface_map)
-
-MM = 1000.0
-HIPS = (1, 2) # SMPL left/right hip; the mid-hip is the reference point
-
-
-def _chunked_mean(fn, A, B, chunk: int = 256) -> float:
- """Mean of ``fn(A, B)`` computed in slices along the sample axis.
-
- PVE runs over 6890 vertices per frame, and ``metrics.similarity_transform``
- casts to float64 for the SVD -- at 24,103 frames that is ~4 GB per array
- and ~16 GB of intermediates, which exhausted system memory and killed the
- process (twice) right after the last fit batch, with all the expensive work
- already done. Chunking bounds it to ~40 MB per slice and changes no number:
- the metric is a mean over independent samples.
- """
- tot, n = 0.0, 0
- for i in range(0, A.shape[0], chunk):
- e = fn(A[i:i + chunk], B[i:i + chunk], i)
- tot += float(e.sum())
- n += e.size
- return tot / n
-
-
-def parse_args():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ckpt", nargs="+", required=True)
- p.add_argument("--dataset", default="emdb", choices=["emdb", "3dpw"])
- p.add_argument("--emdb-root", default=None, help="EMDB: dir holding P0/../P9/")
- p.add_argument("--sequence-dir", default=None, help="3DPW: sequenceFiles/")
- p.add_argument("--image-root", default=None, help="3DPW: imageFiles/")
- p.add_argument("--split", default="test", help="3DPW split")
- p.add_argument("--stride", type=int, default=1)
- p.add_argument("--bbox", default="gt-joints", choices=["gt-joints", "annotated"],
- help="EMDB only; 3DPW always uses projected-GT-joint boxes")
- p.add_argument("--fit-iters", type=int, default=400,
- help="0 skips the SMPL parameter fit entirely and reports "
- "the fit-free rows only (PVE, PA-PVE, J14/H36M) -- the "
- "settled protocol, no optimiser and no free parameter, "
- "and the only way to reach PVE without a "
- "hyperparameter that has a 3-4 mm blast radius. "
- "Otherwise USE >= 1500: 400 converges on the round trip "
- "but NOT on real predictions -- 16.1 mm residual on "
- "3DPW vs 8.0 at 4000, worth 3.6 mm of J14 PA-MPJPE, "
- "and the sign differs per dataset. The fit buys the 24 "
- "SMPL KINEMATIC joints and nothing else. See "
- "benchmark/README.md.")
- p.add_argument("--fit-batch", type=int, default=64)
- p.add_argument("--batch-size", type=int, default=64)
- p.add_argument("--num-workers", type=int, default=10)
- p.add_argument("--smpl-dir", default="benchmark/data/smpl")
- p.add_argument("--h36m-regressor",
- default="benchmark/data/smpl/J_regressor_h36m.npy")
- p.add_argument("--device", default="cuda")
- p.add_argument("--out", default=None)
- return p.parse_args()
-
-
-def predict_params(model, loader, device, use_amp=True):
- """(N, 204) MHR model params, (N, 45) shape params, and the kept indices."""
- model.eval()
- mp, sh, keeps, base = [], [], [], 0
- for b in loader:
- img = b["image"].to(device, non_blocking=True)
- cc = b["cliff_cond"].to(device, non_blocking=True)
- with torch.amp.autocast(device_type="cuda", dtype=torch.float16, enabled=use_amp):
- out = model(img, cc)
- p = out["mhr_params"].float()
- s = out["shape_params"].float()
- finite = (torch.isfinite(p).all(1) & torch.isfinite(s).all(1)).cpu()
- keep = b["ok"].bool() & finite
- if bool(keep.any()):
- k = keep.to(p.device)
- mp.append(p[k].cpu())
- sh.append(s[k].cpu())
- keeps.append(base + np.nonzero(keep.numpy())[0])
- base += int(keep.shape[0])
- return torch.cat(mp), torch.cat(sh), np.concatenate(keeps)
-
-
-def emdb_gt_smpl(emdb_root: Path, samples, smpl_dir, device):
- """GT SMPL joints and vertices in camera space, metres, per sample.
-
- ``make_emdb_gt.py`` caches only the 24 joints (7 MB); the 6890 vertices PVE
- needs would be 2 GB, so they are rebuilt here from the sequence parameters
- instead of stored.
- """
- seq_info = {}
- for pkl in sorted(Path(emdb_root).glob("P*/*/*_data.pkl")):
- with open(pkl, "rb") as f:
- d = pickle.load(f)
- if d["emdb1"]:
- seq_info[str(d["name"])] = d
-
- idx_by_seq: dict[str, list[int]] = {n: [] for n in seq_info}
- for i, s in enumerate(samples):
- idx_by_seq[s["sequence"]].append(i)
-
- J = torch.zeros(len(samples), 24, 3)
- V = torch.zeros(len(samples), 6890, 3)
- genders = [""] * len(samples)
- models = {}
- for name, d in seq_info.items():
- ii = idx_by_seq[name]
- if not ii:
- continue
- g = "male" if str(d["gender"]).lower().startswith("m") else "female"
- if g not in models:
- models[g] = SMPLModel(smpl_dir, g, device)
- sm = models[g]
- frames = np.array([samples[i]["frame"] for i in ii])
- s = d["smpl"]
- poses = np.concatenate([s["poses_root"], s["poses_body"]], axis=1)[frames]
- trans = np.asarray(s["trans"], np.float32)[frames]
- beta = np.asarray(s["betas"], np.float32).ravel()[:10]
- Ecam = np.asarray(d["camera"]["extrinsics"], np.float64)[frames]
- for k in range(0, len(ii), 64):
- sl = slice(k, k + 64)
- th = torch.as_tensor(poses[sl], dtype=torch.float32, device=device)
- tr = torch.as_tensor(trans[sl], dtype=torch.float32, device=device)
- be = torch.as_tensor(beta, dtype=torch.float32,
- device=device)[None].expand(th.shape[0], 10)
- v, j = sm.forward(th, be, tr)
- Tc = torch.as_tensor(Ecam[sl], dtype=torch.float32, device=device)
- R, t = Tc[:, :3, :3], Tc[:, :3, 3]
- v = torch.einsum("nij,nvj->nvi", R, v) + t[:, None, :]
- j = torch.einsum("nij,nvj->nvi", R, j) + t[:, None, :]
- tgt = torch.tensor(ii[sl])
- J[tgt] = j.cpu()
- V[tgt] = v.cpu()
- for i in ii:
- genders[i] = g
- return J, V, genders
-
-
-def threedpw_gt_smpl(sequence_dir, split, samples, smpl_dir, device):
- """GT SMPL joints and vertices in camera space for 3DPW, metres.
-
- The joints are the 24 SMPL **kinematic** ones -- the same convention EMDB
- uses, and the same thing 3DPW stores in ``jointPositions`` -- so both
- datasets are scored identically here. This deliberately does not use the
- H36M regressor: that is the 14-joint 3DPW convention reported by
- ``eval_3dpw_ckpt.py``, and it has no vertex analogue, so PVE could not be
- defined against it.
- """
- import pickle as _pk
- by_seq: dict[str, list[int]] = {}
- for i, s in enumerate(samples):
- by_seq.setdefault(s["sequence"], []).append(i)
-
- J = torch.zeros(len(samples), 24, 3)
- V = torch.zeros(len(samples), 6890, 3)
- genders = [""] * len(samples)
- models = {}
- for name, ii in by_seq.items():
- with open(Path(sequence_dir) / split / f"{name}.pkl", "rb") as f:
- seq = _pk.load(f, encoding="latin1")
- cam = np.asarray(seq["cam_poses"], np.float64)
- for pid in sorted({samples[i]["person"] for i in ii}):
- jj = [i for i in ii if samples[i]["person"] == pid]
- g = str(seq["genders"][pid])
- g = "male" if g.lower().startswith("m") else "female"
- if g not in models:
- models[g] = SMPLModel(smpl_dir, g, device)
- sm = models[g]
- frames = np.array([samples[i]["frame"] for i in jj])
- poses = np.asarray(seq["poses"][pid], np.float32)[frames]
- trans = np.asarray(seq["trans"][pid], np.float32)[frames]
- beta = np.asarray(seq["betas"][pid], np.float32).ravel()[:10]
- for k in range(0, len(jj), 64):
- sl = slice(k, k + 64)
- th = torch.as_tensor(poses[sl], dtype=torch.float32, device=device)
- tr = torch.as_tensor(trans[sl], dtype=torch.float32, device=device)
- be = torch.as_tensor(beta, dtype=torch.float32,
- device=device)[None].expand(th.shape[0], 10)
- v, j = sm.forward(th, be, tr)
- Tc = torch.as_tensor(cam[frames[sl]], dtype=torch.float32, device=device)
- R, t = Tc[:, :3, :3], Tc[:, :3, 3]
- v = torch.einsum("nij,nvj->nvi", R, v) + t[:, None, :]
- j = torch.einsum("nij,nvj->nvi", R, j) + t[:, None, :]
- tgt = torch.tensor(jj[sl])
- J[tgt] = j.cpu()
- V[tgt] = v.cpu()
- for i in jj:
- genders[i] = g
- return J, V, genders
-
-
-class ThreeDPWSet(torch.utils.data.Dataset):
- """3DPW crops, built by the same ``val3dpw._preprocess`` EMDB uses."""
-
- def __init__(self, sequence_dir, image_root, split, stride, cliff_focal,
- input_size=val3dpw.INPUT_SIZE):
- from benchlib import threedpw as P3
- self.samples, _ = P3.build_samples(
- sequence_dir, image_root, split=split, stride=stride,
- gt="jointpositions")
- self.cliff_focal = cliff_focal
- self.input_size = input_size
-
- def __len__(self):
- return len(self.samples)
-
- def __getitem__(self, i):
- s = self.samples[i]
- bgr = cv2.imread(s["image_path"], cv2.IMREAD_COLOR)
- if bgr is None:
- return {"image": torch.zeros(3, self.input_size, self.input_size),
- "cliff_cond": torch.zeros(3), "ok": torch.tensor(0.0)}
- rgb = cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
- h, w = rgb.shape[:2]
- crop, cliff = val3dpw._preprocess(
- rgb, s["bbox"], h, w, s["focal"] if self.cliff_focal else None,
- size=self.input_size)
- return {"image": torch.from_numpy(crop),
- "cliff_cond": torch.from_numpy(cliff),
- "ok": torch.tensor(1.0)}
-
-
-def main():
- args = parse_args()
- device = torch.device(args.device)
- rig = MHRRig("checkpoints/mhr_model.pt", device)
- tri, bary = load_surface_map("mhr2smpl", device)
-
- report = {}
- for ck_path in args.ckpt:
- st = torch.load(ck_path, map_location="cpu", weights_only=False)
- cfg, prov = T.config_from_checkpoint(st.get("model_state_dict", st), ck_path)
- print(f"[cfg] {Path(ck_path).parts[-3]}: bound_scales={cfg.bound_scales} "
- f"cliff_focal={cfg.cliff_focal}", flush=True)
-
- if args.dataset == "emdb":
- ds = EMDBValSet(args.emdb_root, stride=args.stride, bbox=args.bbox,
- cliff_focal=cfg.cliff_focal, input_size=cfg.image_size)
- else:
- ds = ThreeDPWSet(args.sequence_dir, args.image_root, args.split,
- args.stride, cfg.cliff_focal, cfg.image_size)
- loader = torch.utils.data.DataLoader(
- ds, batch_size=args.batch_size, shuffle=False,
- num_workers=args.num_workers, pin_memory=True)
- tag = "EMDB1" if args.dataset == "emdb" else f"3DPW/{args.split}"
- print(f"[{tag}] {len(ds):,} frames, boxes={args.bbox}", flush=True)
-
- model = T.InstantHMRStudent(cfg, pretrained=False).to(device)
- model.load_state_dict(st["model_state_dict"])
- mp, sh, idx = predict_params(model, loader, device, cfg.use_amp)
- del model
- torch.cuda.empty_cache()
- print(f"[{tag}] {len(idx):,} predictions", flush=True)
-
- kept = [ds.samples[i] for i in idx]
- if args.dataset == "emdb":
- Jgt, Vgt, genders = emdb_gt_smpl(Path(args.emdb_root), kept,
- args.smpl_dir, device)
- else:
- Jgt, Vgt, genders = threedpw_gt_smpl(args.sequence_dir, args.split,
- kept, args.smpl_dir, device)
-
- # `barycentric_transfer` already returns the prediction in SMPL
- # topology, vertex i <-> SMPL vertex i, so this surface IS the answer
- # for every vertex-space metric. Keeping it costs one more (N, 6890, 3)
- # and removes the optimiser from PVE entirely.
- Vd = torch.zeros_like(Vgt)
- fitting = args.fit_iters > 0
- # Fit per gender group, so one SMPL model serves each batch. The fit
- # exists only to recover the 24 SMPL KINEMATIC joints, which come off
- # the chain and not off the surface.
- Jp = torch.zeros_like(Jgt)
- # 2.9 GB at 35,463 frames, so it is allocated only when a fit fills it.
- Vp = torch.zeros_like(Vgt) if args.fit_iters > 0 else None
- resid = torch.zeros(len(kept))
- if not fitting:
- for k in range(0, len(kept), args.fit_batch):
- sel = np.arange(k, min(k + args.fit_batch, len(kept)))
- verts = rig.vertices(mp[sel].to(device), sh[sel].to(device))
- Vd[sel] = barycentric_transfer(verts, rig.faces, tri, bary).cpu()
- for g in ("male", "female") if fitting else ():
- gi = np.array([i for i, x in enumerate(genders) if x == g])
- if not len(gi):
- continue
- sm = SMPLModel(args.smpl_dir, g, device)
- for k in range(0, len(gi), args.fit_batch):
- sel = gi[k:k + args.fit_batch]
- verts = rig.vertices(mp[sel].to(device), sh[sel].to(device))
- tgt = barycentric_transfer(verts, rig.faces, tri, bary)
- Vd[sel] = tgt.cpu()
- th, be, tr, r = fit_smpl_to_targets(sm, tgt, iters=args.fit_iters)
- v, j = sm.forward(th, be, tr)
- Jp[sel] = j.cpu()
- Vp[sel] = v.cpu()
- resid[sel] = r.mean(1).cpu()
- if (k // args.fit_batch) % 20 == 0:
- print(f" {g} {k:6d}/{len(gi)} residual "
- f"{r.mean() * MM:5.2f} mm", flush=True)
-
- jp, jg = Jp.numpy() * MM, Jgt.numpy() * MM # (N, 24, 3), small
- rp = 0.5 * (jp[:, HIPS[0]] + jp[:, HIPS[1]])
- rg = 0.5 * (jg[:, HIPS[0]] + jg[:, HIPS[1]])
-
- # Everything vertex-shaped is done in ONE chunked pass, scaling inside
- # the slice. The obvious version -- materialise `Vp.numpy() * MM` and
- # `Vgt.numpy() * MM`, then einsum the H36M regressor over them -- needs
- # ~17.6 GB at 35,463 frames and killed the process twice, right after
- # the last fit batch with all the expensive work already done. Two
- # separate causes, both fixed here: the scaled copies double 5.9 GB of
- # float32, and `J_regressor_h36m.npy` is float64, so the einsum upcasts
- # the entire (N, 6890, 3) array to a 5.9 GB float64 intermediate.
- Jreg = np.load(args.h36m_regressor).astype(np.float32) # (17, 6890)
- sel = J.J14_FROM_H36M17
- # The root the published PVE uses (WHAM's evaluate_3dpw / _emdb, which
- # SAM 3D Body's table follows). 3DPW: mid-hip of the H36M-regressed J14
- # -- the root below, unchanged. EMDB: mid-hip of SMPL joints 1/2. The
- # prediction's surface has no kinematic chain, so SMPL's own
- # J_regressor rows 1/2 stand in on BOTH sides (4.9 mm from the
- # kinematic hips on GT; scoring GT at its kinematic hips instead moves
- # PVE by 0.05 mm). The H36M root reads ~4.4 mm low on EMDB.
- hipreg = ({g: SMPLModel(args.smpl_dir, g, device).m["J_regressor"][1:3]
- .float().cpu().numpy() for g in set(genders)}
- if args.dataset == "emdb" else None)
- pve_pub_sum = 0.0
- CH = 256
- pve_sum = pve_n = pa_pve_sum = 0.0
- pve_d_sum = pa_pve_d_sum = 0.0
- hp_parts, hg_parts, hpd_parts = [], [], []
- for i in range(0, Vd.shape[0], CH):
- b = Vgt[i:i + CH].numpy() * MM
- n = b.shape[0]
- # Fit-free: the prediction's own surface against the GT surface,
- # rooted at the H36M pelvis regressed from EACH side by the same
- # operator, so the regressor's bias cancels instead of being
- # compared against a kinematic joint it does not equal.
- a_d = Vd[i:i + CH].numpy() * MM
- h_pd = np.einsum("jv,nvc->njc", Jreg, a_d)[:, sel]
- h_gd = np.einsum("jv,nvc->njc", Jreg, b)[:, sel]
- e_d = M.mpjpe(a_d, b, root=(J.pelvis(h_pd), J.pelvis(h_gd)))
- pve_d_sum += float(e_d.sum())
- if hipreg is not None:
- R = np.stack([hipreg[g] for g in genders[i:i + n]])
- mid = lambda v: np.einsum("njv,nvc->njc", R, v).mean(1)
- e_d = M.mpjpe(a_d, b, root=(mid(a_d), mid(b)))
- pve_pub_sum += float(e_d.sum())
- pa_pve_d_sum += float(M.pa_mpjpe(a_d, b).sum())
- pve_n += a_d.shape[0] * a_d.shape[1]
- hpd_parts.append(h_pd)
- hg_parts.append(h_gd)
- if fitting:
- a = Vp[i:i + CH].numpy() * MM
- pve_sum += float(M.mpjpe(a, b, root=(rp[i:i + n], rg[i:i + n])).sum())
- pa_pve_sum += float(M.pa_mpjpe(a, b).sum())
- hp_parts.append(np.einsum("jv,nvc->njc", Jreg, a)[:, sel])
- h_pd = np.concatenate(hpd_parts) # already mm
- h_g = np.concatenate(hg_parts)
- h_p = np.concatenate(hp_parts) if fitting else None
- del hp_parts, hg_parts, hpd_parts
- hpd_r, hg_r = J.pelvis(h_pd), J.pelvis(h_g)
- hp_r = J.pelvis(h_p) if fitting else None
-
- res = dict(
- ckpt=ck_path, dataset=tag, bbox=args.bbox, stride=args.stride,
- n=int(len(kept)), fit_iters=args.fit_iters,
- epoch=st.get("epoch"), source=st.get("source"),
- fit_residual_mm=float(resid.mean() * MM) if fitting else None,
- # The fit-free rows: the prediction's own surface, in SMPL
- # topology, against the benchmark's own GT surface. No optimiser,
- # no free parameter, so these are the PVE numbers to quote. The
- # *_mm keys below keep their old meaning -- the FITTED mesh -- so
- # rows recorded before 2026-09-22 stay comparable.
- PVE_direct_mm=pve_d_sum / pve_n,
- # The PVE to put beside a published table: fit-free surface, the
- # benchmark's own root (see `hipreg`). Equal to PVE_direct on 3DPW.
- PVE_published_mm=pve_pub_sum / pve_n,
- PA_PVE_direct_mm=pa_pve_d_sum / pve_n,
- J14_h36m_direct_MPJPE_mm=float(
- M.mpjpe(h_pd, h_g, root=(hpd_r, hg_r)).mean()),
- J14_h36m_direct_PA_MPJPE_mm=float(M.pa_mpjpe(h_pd, h_g).mean()),
- MPJPE_mm=float(M.mpjpe(jp, jg, root=(rp, rg)).mean()) if fitting else None,
- PA_MPJPE_mm=float(M.pa_mpjpe(jp, jg).mean()) if fitting else None,
- PVE_mm=pve_sum / pve_n if fitting else None,
- PA_PVE_mm=pa_pve_sum / pve_n if fitting else None,
- # The published 3DPW protocol is 14 joints in the H36M convention,
- # reached here THROUGH the mesh conversion: MHR mesh -> barycentric
- # map -> fitted SMPL -> J_regressor_h36m, against the same
- # regressor on the GT mesh.
- #
- # This is NOT an "adapter-free" number and must never be labelled
- # one. We predict MHR, so a conversion to SMPL is mandatory; the
- # only question is which conversion, never whether. This route puts
- # it in mesh space, `eval_3dpw_ckpt.py --adapter` puts it in joint
- # space. Both are legitimate, and note that J_regressor_h36m is
- # itself a fitted linear map (SMPL vertices -> H36M joints) -- SMPL-
- # native methods use it for exactly the same reason we need ours.
- J14_h36m_viamesh_MPJPE_mm=float(
- M.mpjpe(h_p, h_g, root=(hp_r, hg_r)).mean()) if fitting else None,
- J14_h36m_viamesh_PA_MPJPE_mm=float(
- M.pa_mpjpe(h_p, h_g).mean()) if fitting else None,
- )
-
- e = (M.pa_mpjpe(jp, jg) if fitting else M.pa_mpjpe(h_pd, h_g)).mean(1)
- res["per_sequence_PA_MPJPE_mm"] = {
- n: float(e[[i for i, s in enumerate(kept) if s["sequence"] == n]].mean())
- for n in sorted({s["sequence"] for s in kept})}
- report[ck_path] = res
- print(f"\n {Path(ck_path).parts[-3]:<10s} fit-free: PVE "
- f"{res['PVE_direct_mm']:6.2f} PVE(published root) "
- f"{res['PVE_published_mm']:6.2f} PA-PVE {res['PA_PVE_direct_mm']:6.2f}"
- f" | J14/H36M: MPJPE {res['J14_h36m_direct_MPJPE_mm']:6.2f} "
- f"PA {res['J14_h36m_direct_PA_MPJPE_mm']:6.2f}", flush=True)
- if fitting:
- print(f" {Path(ck_path).parts[-3]:<10s} fitted: SMPL24 MPJPE "
- f"{res['MPJPE_mm']:6.2f} PA {res['PA_MPJPE_mm']:6.2f} PVE "
- f"{res['PVE_mm']:6.2f} PA-PVE {res['PA_PVE_mm']:6.2f} | "
- f"J14/H36M via mesh: MPJPE {res['J14_h36m_viamesh_MPJPE_mm']:6.2f}"
- f" PA {res['J14_h36m_viamesh_PA_MPJPE_mm']:6.2f}"
- f" (fit residual {res['fit_residual_mm']:.2f} mm)", flush=True)
- print(flush=True)
- # Written after every checkpoint, not at the end: a full sweep is ~20
- # minutes per checkpoint and losing all of it to a crash in the last
- # one has already happened once.
- if args.out:
- # Merge, never clobber: adding one checkpoint later must not throw
- # away the others, and a crash mid-sweep must not cost the runs
- # that already finished.
- dst = Path(args.out)
- dst.parent.mkdir(parents=True, exist_ok=True)
- merged = json.loads(dst.read_text()) if dst.is_file() else {}
- merged.update(report)
- dst.write_text(json.dumps(merged, indent=2))
- print(f"written to {args.out}", flush=True)
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/fit_adapter_mhr.py b/benchmark/fit_adapter_mhr.py
deleted file mode 100644
index 34b343e..0000000
--- a/benchmark/fit_adapter_mhr.py
+++ /dev/null
@@ -1,160 +0,0 @@
-#!/usr/bin/env python3
-"""Fit an MHR70 -> SMPL/H36M joint adapter from dataset labels, student-free.
-
-``eval_3dpw_ckpt.py --fit-adapter`` fits the map against one checkpoint's own
-predictions, which lets it absorb that checkpoint's systematic error along with
-the skeleton difference. This fits the same map against the **dataset's own**
-MHR annotations in ``data/sam3d_gt_3dpw`` instead, so the result is one adapter
-for every checkpoint and describes only the rig conversion.
-
-Those annotations are the released ``facebook/sam-3d-body-dataset`` labels for
-``3dpw_train``, built by ``datasets_pipeline/build_split.py``. They are NOT
-teacher inference -- only ``data/sam3d_distill_mix`` is that, via
-``tools/annotate_dataset.py``. The distinction matters for what the numbers
-below mean: the residual after fitting is a rig-conversion floor, not a
-teacher's error.
-
-Both sides come from the same 3DPW frame: the dataset's 70 MHR keypoints, and
-the H36M joints regressed from GT SMPL by ``make_3dpw_gt.py``. The annotation
-filename carries the person index, and it matches 3DPW's own (verified: the
-correct pairing sits at 0.5-2.4 deg of residual rotation, the wrong one at
-7-108 deg).
-
-Two targets, both fitted on 3DPW train and both reported on a different split
-or a different dataset:
-
- j14 the 14 LSP joints in the H36M convention, from the
- ``make_3dpw_gt.py`` cache -- the 3DPW row.
- smpl24 the 24 SMPL kinematic joints, straight out of the pickles'
- ``jointPositions``. That field *is* the kinematic joint set
- (verified to 0.001 mm by ``make_3dpw_gt.py --check``), which is
- also what EMDB's ``kp2d`` projects -- so this adapter transfers to
- EMDB unchanged, and 3DPW train is a clean holdout for it.
-
- python benchmark/fit_adapter_mhr.py \\
- --annotations data/sam3d_gt_3dpw/annotations \\
- --sequence-dir /path/to/3DPW/sequenceFiles \\
- --target j14 --out benchmark/results/adapter_j14_h36m_teacher.npz
-"""
-
-from __future__ import annotations
-
-import argparse
-import pickle
-import re
-import sys
-from pathlib import Path
-
-import numpy as np
-
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-
-import eval_3dpw as E # noqa: E402
-from benchlib import joints as J # noqa: E402
-from benchlib import metrics as M # noqa: E402
-from benchlib import threedpw as P # noqa: E402
-
-NAME_RX = re.compile(r"3dpw_(?P<seq>.+?)_image_(?P<frame>\d+)_p(?P<pid>\d+)\.npz$")
-
-
-def build_pairs(ann_dir: Path, sequence_dir: Path, split: str, stride: int,
- target: str):
- """(N, 70, 3) annotated MHR keypoints and the paired GT, camera space, mm.
-
- The GT is (N, 17, 3) H36M joints for ``target="j14"`` and (N, 24, 3) SMPL
- kinematic joints for ``target="smpl24"``.
- """
- table = P.load_gt_h36m(sequence_dir, split) if target == "j14" else None
- seqs: dict[str, dict] = {}
- mhr, gt = [], []
- skipped = 0
-
- files = sorted(f for f in ann_dir.iterdir() if NAME_RX.match(f.name))
- for f in files[::stride]:
- m = NAME_RX.match(f.name)
- seq, frame, pid = m["seq"], int(m["frame"]), int(m["pid"])
- key = f"{seq}|{pid}"
- if table is not None and key not in table:
- skipped += 1
- continue
- if seq not in seqs:
- with open(sequence_dir / split / f"{seq}.pkl", "rb") as fh:
- s = pickle.load(fh, encoding="latin1")
- seqs[seq] = dict(cam=np.asarray(s["cam_poses"]),
- valid=[np.asarray(v).astype(bool)
- for v in s["campose_valid"]],
- jp=[np.asarray(j, np.float64).reshape(-1, 24, 3)
- for j in s["jointPositions"]])
- s = seqs[seq]
- if frame >= len(s["cam"]) or not s["valid"][pid][frame]:
- skipped += 1
- continue
-
- z = np.load(f)
- T = s["cam"][frame]
- gt_w = table[key][frame] if table is not None else s["jp"][pid][frame]
- g = gt_w @ T[:3, :3].T + T[:3, 3] # world -> camera
- mhr.append(z["joints_3d"])
- gt.append(g)
-
- return (np.stack(mhr) * 1000.0, np.stack(gt) * 1000.0, len(files), skipped)
-
-
-def main():
- ap = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("--annotations", default="data/sam3d_gt_3dpw/annotations")
- ap.add_argument("--sequence-dir", required=True)
- ap.add_argument("--split", default="train")
- ap.add_argument("--stride", type=int, default=1)
- ap.add_argument("--target", default="j14", choices=["j14", "smpl24"])
- ap.add_argument("--out", default=None,
- help="default: adapter_{j14_h36m,smpl24}_teacher.npz")
- args = ap.parse_args()
-
- out = Path(args.out) if args.out else Path(
- "benchmark/results/adapter_"
- + ("j14_h36m" if args.target == "j14" else "smpl24") + "_teacher.npz")
-
- mhr, gt, n_files, skipped = build_pairs(
- Path(args.annotations), Path(args.sequence_dir), args.split,
- args.stride, args.target)
- print(f"[adapter] {len(mhr)} annotation/GT pairs from {n_files} annotations "
- f"({skipped} skipped: no GT track or campose_valid=0)")
-
- # Target-side joints the adapter must reproduce, and the metric to read it
- # by. j14 also has a raw row -- the same 14 landmarks taken straight off the
- # MHR rig -- which is the annotation's deviation plus the rig offset; the
- # adapter's job is the second part. smpl24 has no raw row: 10 of the 24
- # (spines, collars, feet, hands) have no MHR landmark at all, which is
- # exactly why the conversion has to be a fitted map.
- if args.target == "j14":
- Y = gt[:, J.J14_FROM_H36M17, :]
- gt_name, hips = "h36m", J.J14_HIP_IDX
- before = M.pa_mpjpe(mhr[:, J.J14_FROM_MHR70, :], Y).mean()
- print(f"[adapter] annotation vs GT J14 PA-MPJPE, no adapter: {before:.2f} mm")
- else:
- Y = gt
- gt_name = "smpl24"
- hips = (J.SMPL["left_hip"], J.SMPL["right_hip"])
-
- # Fit on one half, score on the other: the conversion floor an MHR-rigged
- # model carries into this joint convention before it makes any mistake.
- half = len(mhr) // 2
- W_half = E.fit_linear_adapter(mhr[:half], Y[:half])
- P_held = np.einsum("jk,nkc->njc", W_half,
- mhr[half:] - mhr[half:].mean(axis=1, keepdims=True))
- root_p = 0.5 * (P_held[:, hips[0]] + P_held[:, hips[1]])
- root_g = 0.5 * (Y[half:, hips[0]] + Y[half:, hips[1]])
- print(f"[adapter] held-out half, adapter applied: "
- f"{M.mpjpe(P_held, Y[half:], root=(root_p, root_g)).mean():.2f} mm MPJPE, "
- f"{M.pa_mpjpe(P_held, Y[half:]).mean():.2f} mm PA-MPJPE")
-
- W = E.fit_linear_adapter(mhr, Y)
- out.parent.mkdir(parents=True, exist_ok=True)
- np.savez(out, W=W, split=args.split, gt=gt_name, source="dataset_mhr")
- print(f"[adapter] W {W.shape}, |W|max {np.abs(W).max():.2f} -> {out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/make_emdb_gt.py b/benchmark/make_emdb_gt.py
deleted file mode 100644
index 93f335b..0000000
--- a/benchmark/make_emdb_gt.py
+++ /dev/null
@@ -1,119 +0,0 @@
-#!/usr/bin/env python3
-"""Build EMDB ground truth: gendered SMPL forward -> the 24 kinematic joints.
-
-EMDB-1 (17 sequences, the camera-frame split every paper reports) ships SMPL
-parameters, not joints, so the reference has to be produced once. Unlike 3DPW
-there is no convention to choose: the dataset's own ``kp2d`` field is the
-projection of the **kinematic-tree** joints -- ``J_regressor @ posed vertices``
-reprojects 1.0 px off, the kinematic joints 0.00 px -- so those are what EMDB
-means by "24 joints", and they are ``smpl_forward``'s second output already.
-
-Writes ``<emdb-root>/gt_smpl24/emdb1.npz``, one (F, 24, 3) array of **world**
-joints per sequence, in the same frame the ``camera.extrinsics`` map from.
-
- python benchmark/make_emdb_gt.py --emdb-root /path/to/EMDB \
- --smpl-dir benchmark/data/smpl --check
-
-``--check`` reprojects with the sequence's own intrinsics/extrinsics and
-compares against ``kp2d``; it also prints how far the regressed joints sit from
-the kinematic ones, which is the only convention question EMDB leaves open.
-"""
-
-from __future__ import annotations
-
-import argparse
-import pickle
-from pathlib import Path
-
-import numpy as np
-import torch
-
-from make_3dpw_gt import load_smpl, smpl_forward
-
-
-def emdb1_sequences(root: Path) -> list[tuple[str, Path]]:
- """(name, sequence directory) for every EMDB-1 sequence under ``root``."""
- out = []
- for pkl in sorted(root.glob("P*/*/*_data.pkl")):
- with open(pkl, "rb") as f:
- d = pickle.load(f)
- if d["emdb1"]:
- out.append((str(d["name"]), pkl.parent))
- return out
-
-
-def main():
- ap = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("--emdb-root", required=True)
- ap.add_argument("--smpl-dir", default="benchmark/data/smpl")
- ap.add_argument("--out", default=None,
- help="default <emdb-root>/gt_smpl24")
- ap.add_argument("--device", default="cuda")
- ap.add_argument("--chunk", type=int, default=256)
- ap.add_argument("--check", action="store_true")
- args = ap.parse_args()
-
- device = torch.device(args.device if torch.cuda.is_available() else "cpu")
- root = Path(args.emdb_root)
- out_dir = Path(args.out) if args.out else root / "gt_smpl24"
- out_dir.mkdir(parents=True, exist_ok=True)
-
- models = {g: load_smpl(Path(args.smpl_dir) / f"SMPL_{g.upper()}.pkl", device)
- for g in ("male", "female")}
- n_dirs = models["male"]["shapedirs"].shape[-1]
-
- data: dict[str, np.ndarray] = {}
- px_err, kin_vs_reg = [], []
- for name, seq_dir in emdb1_sequences(root):
- with open(next(seq_dir.glob("*_data.pkl")), "rb") as f:
- d = pickle.load(f)
- m = models["male" if d["gender"].lower().startswith("m") else "female"]
- s = d["smpl"]
- poses = np.concatenate([s["poses_root"], s["poses_body"]], axis=1)
- # EMDB stores exactly 10 betas -- the count the published protocol uses --
- # so the zero-pad to the model's shapedirs is lossless.
- b = np.zeros(n_dirs, np.float32)
- raw = np.asarray(s["betas"], np.float32).ravel()
- b[:raw.shape[0]] = raw
- b = torch.as_tensor(b, device=device)
- p = torch.as_tensor(np.asarray(poses, np.float32), device=device)
- t = torch.as_tensor(np.asarray(s["trans"], np.float32), device=device)
-
- kin, reg = [], []
- for i in range(0, p.shape[0], args.chunk):
- V, Jk = smpl_forward(m, p[i:i + args.chunk], b, t[i:i + args.chunk])
- kin.append(Jk.cpu().numpy())
- if args.check:
- reg.append((m["J_regressor"] @ V).cpu().numpy())
- J = np.concatenate(kin).astype(np.float32)
- data[name] = J
- print(f" {name:<34s} {J.shape[0]:5d} frames {d['gender']}", flush=True)
-
- if args.check:
- R = np.concatenate(reg)
- kin_vs_reg.append(np.linalg.norm(R - J, axis=-1).ravel())
- K = np.asarray(d["camera"]["intrinsics"])
- T = np.asarray(d["camera"]["extrinsics"])
- Jc = np.einsum("nij,nkj->nki", T[:, :3, :3], J) + T[:, None, :3, 3]
- uv = (Jc[..., :2] / Jc[..., 2:3]) * np.array([K[0, 0], K[1, 1]]) \
- + np.array([K[0, 2], K[1, 2]])
- px_err.append(np.linalg.norm(uv - d["kp2d"], axis=-1).ravel())
-
- dst = out_dir / "emdb1.npz"
- np.savez_compressed(dst, layout="smpl24_kinematic", **data)
- n = sum(v.shape[0] for v in data.values())
- print(f"[EMDB] {len(data)} sequences, {n} frames -> {dst}")
-
- if args.check:
- e = np.concatenate(px_err)
- print(f"[check] reprojection vs kp2d: mean {e.mean():.4f} px, "
- f"max {e.max():.4f} px")
- d = np.concatenate(kin_vs_reg) * 1000.0
- print(f"[check] J_regressor @ vertices vs kinematic joints: "
- f"mean {d.mean():.3f} mm, max {d.max():.3f} mm "
- f"(kp2d picks the kinematic ones)")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/make_mhr_gt.py b/benchmark/make_mhr_gt.py
deleted file mode 100644
index 255dc8f..0000000
--- a/benchmark/make_mhr_gt.py
+++ /dev/null
@@ -1,154 +0,0 @@
-#!/usr/bin/env python3
-"""Solve and cache an MHR ground truth for EMDB-1 and 3DPW, once.
-
-Each benchmark's ground truth is a SMPL body. This fits the MHR rig to that
-exact surface and stores the resulting parameters, so the expensive part -- an
-optimisation per frame -- is paid once instead of on every checkpoint.
-
- <root>/gt_mhr/<tag>.npz model_params (N, 204), shape_params (N, 45),
- trans (N, 3), residual_mm (N,), keys (N,)
-
-Only 249 floats per frame, so both datasets together are ~60 MB.
-
-**What this is for, and what it is not for.** It gives two things:
-
-* the **oracle row** -- push this MHR body back through the forward conversion
- and score it, and you have the best result any MHR-rigged model could
- possibly achieve on this benchmark. That separates rig cost from model error
- honestly, and it is an upper bound, labelled as such.
-* the MHR-space diagnostic, if you want to look at error on the rig's own 70
- keypoints.
-
-It is **not** a ground truth to report against. Scoring predictions on MHR
-joints against this reference would change the joint set (MPJPE is not
-rig-independent), average error over 40 finger joints millimetres apart, and
-compare against a surface we produced with an optimiser told to look like an
-MHR model -- so whatever MHR cannot represent silently leaves the error. Report
-in SMPL space; see ``eval_smpl_fit.py``.
-
- python benchmark/make_mhr_gt.py --dataset emdb --emdb-root /path/to/EMDB
- python benchmark/make_mhr_gt.py --dataset 3dpw --split test \\
- --sequence-dir /path/to/3DPW/sequenceFiles
-"""
-
-from __future__ import annotations
-
-import argparse
-import pickle
-import sys
-from pathlib import Path
-
-import numpy as np
-import torch
-
-sys.path.insert(0, str(Path(__file__).resolve().parent))
-
-from mhr_smpl import (MHRRig, SMPLModel, barycentric_transfer, # noqa: E402
- fit_mhr_to_targets, load_surface_map)
-
-
-def emdb_frames(root: Path):
- """(key, gender, poses, betas, trans) per EMDB-1 sequence."""
- for pkl in sorted(root.glob("P*/*/*_data.pkl")):
- with open(pkl, "rb") as f:
- d = pickle.load(f)
- if not d["emdb1"]:
- continue
- s = d["smpl"]
- good = np.asarray(d["good_frames_mask"]).astype(bool)
- idx = np.nonzero(good)[0]
- yield (str(d["name"]), idx,
- "male" if str(d["gender"]).lower().startswith("m") else "female",
- np.concatenate([s["poses_root"], s["poses_body"]], axis=1)[idx],
- np.asarray(s["betas"], np.float32).ravel()[:10],
- np.asarray(s["trans"], np.float32)[idx])
-
-
-def threedpw_frames(seq_dir: Path, split: str):
- for pkl in sorted((seq_dir / split).glob("*.pkl")):
- with open(pkl, "rb") as f:
- seq = pickle.load(f, encoding="latin1")
- name = str(seq["sequence"])
- for pid, poses in enumerate(seq["poses"]):
- valid = np.asarray(seq["campose_valid"][pid]).astype(bool)
- idx = np.nonzero(valid)[0]
- g = str(seq["genders"][pid])
- yield (f"{name}|{pid}", idx,
- "male" if g.lower().startswith("m") else "female",
- np.asarray(poses, np.float32)[idx],
- np.asarray(seq["betas"][pid], np.float32).ravel()[:10],
- np.asarray(seq["trans"][pid], np.float32)[idx])
-
-
-def main():
- ap = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("--dataset", required=True, choices=["emdb", "3dpw"])
- ap.add_argument("--emdb-root", default=None)
- ap.add_argument("--sequence-dir", default=None)
- ap.add_argument("--split", default="test")
- ap.add_argument("--smpl-dir", default="benchmark/data/smpl")
- ap.add_argument("--mhr", default="checkpoints/mhr_model.pt")
- ap.add_argument("--iters", type=int, default=400)
- ap.add_argument("--batch", type=int, default=96)
- ap.add_argument("--stride", type=int, default=1)
- ap.add_argument("--device", default="cuda")
- ap.add_argument("--out", default=None)
- args = ap.parse_args()
-
- device = torch.device(args.device)
- rig = MHRRig(args.mhr, device)
- tri, bary = load_surface_map("smpl2mhr", device)
- models: dict[str, SMPLModel] = {}
-
- if args.dataset == "emdb":
- root = Path(args.emdb_root)
- src = emdb_frames(root)
- out = Path(args.out) if args.out else root / "gt_mhr" / "emdb1.npz"
- else:
- root = Path(args.sequence_dir)
- src = threedpw_frames(root, args.split)
- out = Path(args.out) if args.out else root.parent / "gt_mhr" / f"{args.split}.npz"
-
- store: dict[str, np.ndarray] = {}
- all_res = []
- for key, idx, gender, poses, beta, trans in src:
- idx, poses, trans = idx[::args.stride], poses[::args.stride], trans[::args.stride]
- if gender not in models:
- models[gender] = SMPLModel(args.smpl_dir, gender, device)
- sm = models[gender]
- MP, SH, TR, RS = [], [], [], []
- for k in range(0, len(idx), args.batch):
- sl = slice(k, k + args.batch)
- th = torch.as_tensor(poses[sl], dtype=torch.float32, device=device)
- tr = torch.as_tensor(trans[sl], dtype=torch.float32, device=device)
- be = torch.as_tensor(beta, dtype=torch.float32,
- device=device)[None].expand(th.shape[0], 10)
- with torch.no_grad():
- V, _ = sm.forward(th, be, tr)
- # GT SMPL surface, resampled in MHR topology.
- tgt = barycentric_transfer(V, sm.faces, tri, bary)
- mp, sh, t, res = fit_mhr_to_targets(rig, tgt, iters=args.iters)
- MP.append(mp.cpu().numpy()); SH.append(sh.cpu().numpy())
- TR.append(t.cpu().numpy()); RS.append(res.mean(1).cpu().numpy())
- store[f"{key}|model_params"] = np.concatenate(MP).astype(np.float32)
- store[f"{key}|shape_params"] = np.concatenate(SH).astype(np.float32)
- store[f"{key}|trans"] = np.concatenate(TR).astype(np.float32)
- store[f"{key}|frames"] = idx.astype(np.int32)
- r = np.concatenate(RS)
- store[f"{key}|residual_mm"] = (r * 1000).astype(np.float32)
- all_res.append(r)
- print(f" {key:<36s} {len(idx):5d} frames residual "
- f"{r.mean() * 1000:6.2f} mm", flush=True)
-
- a = np.concatenate(all_res) * 1000
- out.parent.mkdir(parents=True, exist_ok=True)
- np.savez_compressed(out, dataset=args.dataset, split=args.split,
- iters=args.iters, **store)
- print(f"[mhr-gt] {len(all_res)} tracks, {a.shape[0]} frames | "
- f"surface residual mean {a.mean():.2f} mm, p95 {np.percentile(a, 95):.2f} mm")
- print(f"[mhr-gt] -> {out}")
-
-
-if __name__ == "__main__":
- main()
diff --git a/benchmark/mhr_smpl.py b/benchmark/mhr_smpl.py
deleted file mode 100644
index ee9a45f..0000000
--- a/benchmark/mhr_smpl.py
+++ /dev/null
@@ -1,412 +0,0 @@
-"""MHR <-> SMPL mesh conversion: the two rigs, and a solver that fits either to
-the other's surface.
-
-Why this exists. InstantHMR regresses MHR, and every published 3DPW/EMDB number
-is defined on SMPL -- 24 joints for MPJPE, 6890 vertices for PVE. The linear
-MHR70 -> SMPL24 joint adapter carries an 18.8 mm floor and cannot produce
-vertices at all, so it cannot reach PVE. Converting the *mesh* instead is what
-SAM 3D Body and Fast SAM 3D Body do for this same rig (their "iterative
-fitting"), and NLF does the equivalent for its nonparametric points.
-
-Direction matters, and both are used here for different jobs:
-
-``fit_mhr_to_smpl`` MHR params that reproduce a given SMPL surface. Used to
- build matched-surface pairs for calibration, and to
- compute the ORACLE row -- the best any MHR-rigged model
- could score on a SMPL benchmark.
-``fit_smpl_to_targets`` SMPL params reproducing a predicted MHR surface, via
- Meta's official barycentric map. This is the reported
- path: it puts the prediction in the benchmark's own
- space, where the GT is untouched.
-
-Never fit anything per-frame against the ground truth of a split being
-reported. Calibration runs on 3DPW *train*; the oracle row is labelled as an
-upper bound, not as a result.
-
-Units. The rig is centimetres, Y-up/Z-back; annotations and SMPL are metres,
-Y-down/Z-forward. ``MHRRig.vertices`` returns the vision frame, so everything
-in this module is metres unless a name says otherwise.
-"""
-
-from __future__ import annotations
-
-from pathlib import Path
-
-import numpy as np
-import torch
-
-MHR_NUM_VERTS = 18439
-MHR_NUM_MODEL_PARAMS = 204
-MHR_NUM_SHAPE_PARAMS = 45
-SMPL_NUM_VERTS = 6890
-
-
-class MHRRig:
- """The TorchScript MHR rig, full 18,439-vertex mesh.
-
- ``MHRForwardPass`` in the trainer evaluates the skeleton branch only, and
- its optional vertex subset exists to keep a training step cheap. Conversion
- needs the whole surface, so this calls the rig's own top-level forward --
- identity blendshapes, pose correctives and skinning included, which is the
- mesh the dataset's annotations describe.
- """
-
- def __init__(self, path: str | Path, device):
- self.device = device
- self.m = torch.jit.load(str(path), map_location=device).eval()
- for p in self.m.parameters():
- p.requires_grad_(False)
- self.faces = self.m.character_torch.mesh.faces.long().to(device)
-
- def vertices(self, model_params: torch.Tensor, shape_params: torch.Tensor,
- correctives: bool = True) -> torch.Tensor:
- """(B, 18439, 3) skin in the vision frame, metres, body-centred.
-
- ``model_params[:, :3]`` is the rig's global translation. The corpus
- converter zeroes it and carries the translation in ``cam_trans``
- instead, so a caller that wants camera space adds that separately.
- """
- B = model_params.shape[0]
- face = torch.zeros(B, 72, device=model_params.device,
- dtype=model_params.dtype)
- v, _ = self.m(shape_params, model_params, face, correctives)
- v = v / 100.0
- return torch.stack([v[..., 0], -v[..., 1], -v[..., 2]], dim=-1)
-
-
-def _chunked_nearest(A: torch.Tensor, B: torch.Tensor, chunk: int = 512):
- """Per-point nearest neighbour of A in B: (distances, indices).
-
- Chunked over A because the full (6890, 18439) matrix is 500 MB per sample
- and the backward pass through it will not fit on an 11 GB card.
- """
- ds, ix = [], []
- for i in range(0, A.shape[1], chunk):
- d = torch.cdist(A[:, i:i + chunk], B)
- m = d.min(-1)
- ds.append(m.values)
- ix.append(m.indices)
- return torch.cat(ds, 1), torch.cat(ix, 1)
-
-
-def fit_mhr_to_smpl(rig: MHRRig, smpl_verts: torch.Tensor,
- init_model: torch.Tensor, init_shape: torch.Tensor,
- init_trans: torch.Tensor, iters: int = 600,
- reassoc: int = 25, verbose: bool = False):
- """Solve MHR parameters that reproduce ``smpl_verts``.
-
- Point-to-point ICP: the correspondence is re-associated every ``reassoc``
- steps under ``no_grad`` and held fixed in between, so the gradient never
- flows through an argmin and the (6890, 18439) distance matrix is built once
- per re-association rather than once per step.
-
- Args:
- smpl_verts: (B, 6890, 3) target surface, metres, any frame.
- init_*: starting parameters; the dataset's own MHR annotation is the
- natural choice and starts ~50 mm out.
-
- Returns:
- ``(model_params, shape_params, trans, residual)`` -- residual is the
- (B, 6890) per-vertex distance from each SMPL vertex to the fitted MHR
- surface, in metres.
- """
- P = init_model.clone().requires_grad_(True)
- S = init_shape.clone().requires_grad_(True)
- T = init_trans.clone().requires_grad_(True)
- opt = torch.optim.Adam([{"params": [P], "lr": 3e-3},
- {"params": [S], "lr": 1e-2},
- {"params": [T], "lr": 3e-3}])
- idx = None
- for it in range(iters):
- if it % reassoc == 0:
- with torch.no_grad():
- v = rig.vertices(P, S) + T[:, None, :]
- dist, idx = _chunked_nearest(smpl_verts, v)
- if verbose and it % (reassoc * 6) == 0:
- print(f" it {it:4d} mean {dist.mean() * 1000:6.2f} mm", flush=True)
- v = rig.vertices(P, S) + T[:, None, :]
- tgt = torch.gather(v, 1, idx[..., None].expand(-1, -1, 3))
- loss = ((smpl_verts - tgt) ** 2).sum(-1).mean()
- opt.zero_grad()
- loss.backward()
- opt.step()
-
- with torch.no_grad():
- v = rig.vertices(P, S) + T[:, None, :]
- dist, _ = _chunked_nearest(smpl_verts, v)
- return P.detach(), S.detach(), T.detach(), dist
-
-
-class SMPLModel:
- """Gendered SMPL as a differentiable layer over (theta, beta, trans).
-
- ``make_3dpw_gt.smpl_forward`` cannot be reused here for two reasons: it
- builds the kinematic chain with in-place writes into one tensor (fine for
- the no-grad GT cache it was written for, but autograd refuses it), and it
- takes a single shape vector for the whole batch. This version stacks the
- chain instead and carries per-sample betas, so a whole batch fits in one
- call. It is verified against that function to float precision below.
- """
-
- def __init__(self, smpl_dir: str | Path, gender: str, device):
- import pickle
- from make_3dpw_gt import _install_chumpy_stub, load_smpl
- path = Path(smpl_dir) / f"SMPL_{gender.upper()}.pkl"
- self.m = load_smpl(path, device)
- self.n_dirs = self.m["shapedirs"].shape[-1]
- self.device = device
- _install_chumpy_stub()
- with open(path, "rb") as f:
- self.faces = torch.as_tensor(
- np.asarray(pickle.load(f, encoding="latin1")["f"], np.int64),
- device=device)
- # Unique undirected edges, for the edge term in fit_smpl_to_targets.
- e = torch.cat([self.faces[:, [0, 1]], self.faces[:, [1, 2]],
- self.faces[:, [2, 0]]], dim=0)
- e = torch.unique(torch.stack([e.min(1).values, e.max(1).values], 1), dim=0)
- self.edges = (e[:, 0], e[:, 1])
-
- def forward(self, theta: torch.Tensor, beta10: torch.Tensor,
- trans: torch.Tensor):
- """(B, 6890, 3) vertices and (B, 24, 3) kinematic joints, metres.
-
- ``beta10`` is the 10 coefficients the published protocol uses; the rest
- of the shape space stays zero, which is lossless on 3DPW and EMDB
- (both store exactly those 10 as non-zero).
- """
- from make_3dpw_gt import rodrigues
- m = self.m
- N = theta.shape[0]
- b = torch.zeros(N, self.n_dirs, device=theta.device, dtype=theta.dtype)
- b = torch.cat([beta10, b[:, beta10.shape[1]:]], dim=1)
-
- v_shaped = m["v_template"] + torch.einsum("vck,nk->nvc", m["shapedirs"], b)
- J = torch.einsum("jv,nvc->njc", m["J_regressor"], v_shaped) # (N,24,3)
- R = rodrigues(theta.reshape(N, 24, 3))
- eye = torch.eye(3, device=theta.device).expand(N, 23, 3, 3)
- pose_feature = (R[:, 1:] - eye).reshape(N, 207)
- v_posed = v_shaped + torch.einsum("vck,nk->nvc", m["posedirs"], pose_feature)
-
- def rt(rot, t):
- bottom = torch.zeros(N, 1, 4, device=theta.device, dtype=theta.dtype)
- bottom[..., 3] = 1.0
- return torch.cat([torch.cat([rot, t[..., None]], dim=-1), bottom], dim=1)
-
- parents = m["parents"]
- Gs = [rt(R[:, 0], J[:, 0])]
- for i in range(1, 24):
- Gs.append(Gs[parents[i]] @ rt(R[:, i], J[:, i] - J[:, parents[i]]))
- G = torch.stack(Gs, dim=1) # (N,24,4,4)
- J_posed = G[:, :, :3, 3] + trans[:, None, :]
-
- Jh = torch.cat([J, torch.zeros(N, 24, 1, device=theta.device,
- dtype=theta.dtype)], dim=-1)
- G = G - torch.nn.functional.pad(G @ Jh[..., None], (3, 0, 0, 0))
- T = torch.einsum("vj,njab->nvab", m["weights"], G)
- v_h = torch.cat([v_posed, torch.ones(N, v_posed.shape[1], 1,
- device=theta.device,
- dtype=theta.dtype)], dim=-1)
- verts = torch.einsum("nvab,nvb->nva", T, v_h)[..., :3]
- return verts + trans[:, None, :], J_posed
-
-
-# --------------------------------------------------------------------------
-# facebookresearch/MHR, tools/mhr_smpl_conversion/assets (Apache-2.0). Each
-# target vertex is a barycentric point on a source triangle, so the map is a
-# topology constant: indices and weights, no coordinate frame baked in.
-#
-# This is what makes the conversion tractable. Fitting by ICP has to *discover*
-# the correspondence, and its search has a local minimum whose own residual
-# cannot see it -- the mesh overlaps the target while limbs match the wrong
-# limbs (measured: 12 mm surface residual, 76 mm PA-MPJPE). With the mapping,
-# correspondence is KNOWN, vertex i pairs with vertex i, and the fit is
-# well-posed from a cold start.
-#
-# Verified against this repo's rig before use: triangle ids max at 36,871
-# against the rig's 36,874 faces, barycentric rows sum to 1, and the
-# reconstructed SMPL-topology mesh has a mean nearest-vertex spacing of
-# 13.88 mm against real SMPL's 14.37 mm -- a mapping built for a different mesh
-# would scatter neighbouring vertices onto unrelated triangles.
-
-MAP_DIR = Path(__file__).resolve().parent / "data" / "mhr_smpl_conversion"
-
-
-def load_surface_map(direction: str, device, map_dir: Path | None = None):
- """(triangle_ids, barycentric weights) for ``mhr2smpl`` or ``smpl2mhr``."""
- d = Path(map_dir) if map_dir else MAP_DIR
- path = d / f"{direction}_mapping.npz"
- if not path.is_file():
- raise FileNotFoundError(
- f"{path} not found. Copy it from facebookresearch/MHR, "
- "tools/mhr_smpl_conversion/assets/ (Apache-2.0).")
- z = np.load(path)
- return (torch.as_tensor(z["triangle_ids"], dtype=torch.long, device=device),
- torch.as_tensor(z["baryc_coords"], dtype=torch.float32, device=device))
-
-
-def barycentric_transfer(src_verts: torch.Tensor, src_faces: torch.Tensor,
- tri: torch.Tensor, bary: torch.Tensor) -> torch.Tensor:
- """(B, T, 3) target-topology vertices sampled off the source surface.
-
- Frame-agnostic: it is a weighted sum of source vertices, so whatever frame
- and units go in come out. No centimetre conversion here -- ``MHRRig`` has
- already put the rig in metres.
- """
- corners = src_verts[:, src_faces[tri], :] # (B, T, 3, 3)
- return (corners * bary[None, :, :, None]).sum(dim=2)
-
-
-def _matrix_to_axis_angle(R: torch.Tensor) -> torch.Tensor:
- """(B, 3, 3) rotation matrices -> (B, 3) axis-angle."""
- c = ((R[:, 0, 0] + R[:, 1, 1] + R[:, 2, 2] - 1) / 2).clamp(-1, 1)
- ang = torch.acos(c)
- ax = torch.stack([R[:, 2, 1] - R[:, 1, 2],
- R[:, 0, 2] - R[:, 2, 0],
- R[:, 1, 0] - R[:, 0, 1]], dim=-1)
- return ax / (2 * torch.sin(ang).clamp_min(1e-8))[:, None] * ang[:, None]
-
-
-def fit_smpl_to_targets(smpl: SMPLModel, targets: torch.Tensor,
- iters: int = 300, w_edge: float = 1.0,
- edge_until: float = 0.3, verbose: bool = False):
- """SMPL parameters reproducing ``targets``, which are IN SMPL TOPOLOGY.
-
- ``targets[:, i]`` is the position vertex i should take, so this is a plain
- labelled regression -- no correspondence search, no local minimum of the
- kind that defeats ICP, and safe to start from the mean pose.
-
- The edge term compares edge *vectors* rather than positions, which
- constrains local shape independently of where the body is, and is weighted
- down after ``edge_until`` of the schedule so the endgame is driven by
- absolute vertex positions. This mirrors the staged weighting in Meta's own
- PyTorch backend.
- """
- N = targets.shape[0]
- dev = targets.device
-
- # Closed-form global orientation. Known correspondence makes this a plain
- # Procrustes between SMPL's rest mesh and the target, and it matters a lot:
- # the body can be facing anywhere in camera space, and Adam started from
- # identity has to walk the whole way there through axis-angle. Without it
- # the same fit stalls at 40-51 mm of vertex residual.
- with torch.no_grad():
- z3 = torch.zeros(N, 3, device=dev)
- v0, J0 = smpl.forward(torch.zeros(N, 72, device=dev),
- torch.zeros(N, 10, device=dev), z3)
- A = v0 - v0.mean(1, keepdim=True)
- B = targets - targets.mean(1, keepdim=True)
- U, _, Vt = torch.linalg.svd(A.transpose(1, 2) @ B)
- d = torch.sign(torch.linalg.det(Vt.transpose(1, 2) @ U.transpose(1, 2)))
- D = torch.eye(3, device=dev).expand(N, 3, 3).clone()
- D[:, 2, 2] = d
- R = Vt.transpose(1, 2) @ D @ U.transpose(1, 2)
- root = _matrix_to_axis_angle(R)
- th0 = torch.zeros(N, 72, device=dev)
- th0[:, :3] = root
- v1, _ = smpl.forward(th0, torch.zeros(N, 10, device=dev), z3)
- tr0 = targets.mean(1) - v1.mean(1)
-
- th = th0.clone().requires_grad_(True)
- be = torch.zeros(N, 10, device=dev, requires_grad=True)
- tr = tr0.clone().requires_grad_(True)
- opt = torch.optim.Adam([{"params": [th], "lr": 5e-2},
- {"params": [be], "lr": 2e-2},
- {"params": [tr], "lr": 2e-2}])
- sched = torch.optim.lr_scheduler.MultiStepLR(
- opt, milestones=[int(iters * 0.5), int(iters * 0.8)], gamma=0.1)
- e0, e1 = smpl.edges
- tgt_edge = targets[:, e1] - targets[:, e0]
- for it in range(iters):
- v, _ = smpl.forward(th, be, tr)
- loss = ((v - targets) ** 2).sum(-1).mean()
- w = w_edge if it < edge_until * iters else 0.1 * w_edge
- loss = loss + w * (((v[:, e1] - v[:, e0]) - tgt_edge) ** 2).sum(-1).mean()
- opt.zero_grad()
- loss.backward()
- opt.step()
- sched.step()
- if verbose and it % 50 == 0:
- with torch.no_grad():
- print(f" it {it:4d} vertex {(v - targets).norm(dim=-1).mean() * 1000:6.2f} mm",
- flush=True)
- with torch.no_grad():
- v, J = smpl.forward(th, be, tr)
- resid = (v - targets).norm(dim=-1)
- return th.detach(), be.detach(), tr.detach(), resid
-
-
-def _matrix_to_mhr_root(R: torch.Tensor) -> torch.Tensor:
- """(B, 3, 3) vision-frame rotations -> the rig's ``model_params[3:6]``.
-
- The rig drives its root through an extrinsic-XYZ Euler triple, and in the
- vision frame the last two axes are negated -- determined empirically, not
- assumed: setting ``params[3:6] = [0.2, -0.3, 0.4]`` produces a mesh whose
- recovered extrinsic-XYZ Euler is ``[0.2, 0.3, -0.4]``. So the map is
- ``[e_x, -e_y, -e_z]``.
- """
- sy = (-R[:, 2, 0]).clamp(-1, 1)
- y = torch.asin(sy)
- x = torch.atan2(R[:, 2, 1], R[:, 2, 2])
- z = torch.atan2(R[:, 1, 0], R[:, 0, 0])
- return torch.stack([x, -y, -z], dim=-1)
-
-
-def fit_mhr_to_targets(rig: MHRRig, targets: torch.Tensor, iters: int = 400,
- verbose: bool = False):
- """MHR parameters reproducing ``targets``, which are IN MHR TOPOLOGY.
-
- The mirror of ``fit_smpl_to_targets``: ``targets[:, i]`` is where MHR vertex
- i should be, so correspondence is known and the fit is cold-startable. Feed
- it a GT SMPL mesh pushed through the official ``smpl2mhr`` barycentric map.
-
- Used to build the ORACLE reference -- the MHR body that best reproduces the
- benchmark's ground-truth surface. Scoring that back through the forward
- conversion gives the best score any MHR-rigged model could achieve, which
- is the honest way to separate rig cost from model error. It is an upper
- bound and must be labelled as one; it is not a result.
-
- ``model_params[:3]`` (the rig's own global translation) is held at zero and
- a separate translation is fitted instead, matching how the corpus stores
- pose and placement separately.
- """
- N = targets.shape[0]
- dev = targets.device
- with torch.no_grad():
- V0 = rig.vertices(torch.zeros(N, MHR_NUM_MODEL_PARAMS, device=dev),
- torch.zeros(N, MHR_NUM_SHAPE_PARAMS, device=dev))
- A = V0 - V0.mean(1, keepdim=True)
- B = targets - targets.mean(1, keepdim=True)
- U, _, Vt = torch.linalg.svd(A.transpose(1, 2) @ B)
- d = torch.sign(torch.linalg.det(Vt.transpose(1, 2) @ U.transpose(1, 2)))
- D = torch.eye(3, device=dev).expand(N, 3, 3).clone()
- D[:, 2, 2] = d
- p0 = torch.zeros(N, MHR_NUM_MODEL_PARAMS, device=dev)
- p0[:, 3:6] = _matrix_to_mhr_root(Vt.transpose(1, 2) @ D @ U.transpose(1, 2))
- v1 = rig.vertices(p0, torch.zeros(N, MHR_NUM_SHAPE_PARAMS, device=dev))
- t0 = targets.mean(1) - v1.mean(1)
-
- free = p0[:, 3:].clone().requires_grad_(True) # [:3] stays zero
- sh = torch.zeros(N, MHR_NUM_SHAPE_PARAMS, device=dev, requires_grad=True)
- tr = t0.clone().requires_grad_(True)
- opt = torch.optim.Adam([{"params": [free], "lr": 3e-2},
- {"params": [sh], "lr": 2e-2},
- {"params": [tr], "lr": 2e-2}])
- sched = torch.optim.lr_scheduler.MultiStepLR(
- opt, milestones=[int(iters * 0.5), int(iters * 0.8)], gamma=0.1)
- zeros3 = torch.zeros(N, 3, device=dev)
- for it in range(iters):
- mp = torch.cat([zeros3, free], dim=1)
- v = rig.vertices(mp, sh) + tr[:, None, :]
- loss = ((v - targets) ** 2).sum(-1).mean()
- opt.zero_grad()
- loss.backward()
- opt.step()
- sched.step()
- if verbose and it % 100 == 0:
- print(f" it {it:4d} {(v - targets).norm(dim=-1).mean() * 1000:6.2f} mm",
- flush=True)
- with torch.no_grad():
- mp = torch.cat([zeros3, free], dim=1)
- v = rig.vertices(mp, sh) + tr[:, None, :]
- resid = (v - targets).norm(dim=-1)
- return mp.detach(), sh.detach(), tr.detach(), resid
diff --git a/benchmark/purge_3dpw_test.sh b/benchmark/purge_3dpw_test.sh
deleted file mode 100755
index 6dc5b56..0000000
--- a/benchmark/purge_3dpw_test.sh
+++ /dev/null
@@ -1,61 +0,0 @@
-#!/usr/bin/env bash
-# Remove every 3DPW *test* sequence crop from a distillation folder, so the
-# 3DPW test split can be used as a real benchmark.
-#
-# Dry run (default — prints what it would remove, deletes nothing):
-# ./benchmark/purge_3dpw_test.sh /path/to/data/sam3d_distill_mix
-# Actually delete:
-# ./benchmark/purge_3dpw_test.sh /path/to/data/sam3d_distill_mix --delete
-#
-# Crops are named imageFiles_<sequence>_image_<frame>_p<n>.{npz,png}
-# and the pattern anchors on the literal "_image_" that follows the sequence
-# name, so no sequence can match another one's files by prefix.
-
-set -euo pipefail
-
-ROOT="${1:?usage: $0 <distill_dir> [--delete]}"
-MODE="${2:---dry-run}"
-
-# The 24 official 3DPW test sequences.
-TEST_SEQS="
-downtown_arguing_00 downtown_bar_00 downtown_bus_00 downtown_cafe_00
-downtown_car_00 downtown_crossStreets_00 downtown_downstairs_00
-downtown_enterShop_00 downtown_rampAndStairs_00 downtown_runForBus_00
-downtown_runForBus_01 downtown_sitOnStairs_00 downtown_stairs_00
-downtown_upstairs_00 downtown_walkBridge_01 downtown_walking_00
-downtown_walkUphill_00 downtown_warmWelcome_00 downtown_weeklyMarket_00
-downtown_windowShopping_00 flat_guitar_01 flat_packBags_00
-office_phoneCall_00 outdoors_fencing_01
-"
-
-ANN="$ROOT/annotations"
-IMG="$ROOT/images"
-[[ -d "$ANN" && -d "$IMG" ]] || { echo "ERROR: $ROOT needs annotations/ and images/"; exit 1; }
-
-before_a=$(find "$ANN" -maxdepth 1 -type f | wc -l)
-before_i=$(find "$IMG" -maxdepth 1 -type f | wc -l)
-total=0
-
-for s in $TEST_SEQS; do
- n=$(find "$ANN" -maxdepth 1 -name "imageFiles_${s}_image_*" | wc -l)
- total=$((total + n))
- printf ' %-30s %6d\n' "$s" "$n"
- if [[ "$MODE" == "--delete" ]]; then
- find "$ANN" -maxdepth 1 -name "imageFiles_${s}_image_*" -delete
- find "$IMG" -maxdepth 1 -name "imageFiles_${s}_image_*" -delete
- fi
-done
-
-echo
-echo " test-sequence crops found: $total"
-if [[ "$MODE" == "--delete" ]]; then
- after_a=$(find "$ANN" -maxdepth 1 -type f | wc -l)
- after_i=$(find "$IMG" -maxdepth 1 -type f | wc -l)
- echo " annotations: $before_a -> $after_a"
- echo " images : $before_i -> $after_i"
- [[ "$after_a" -eq "$after_i" ]] \
- && echo " OK: annotations and images still paired" \
- || echo " WARNING: counts diverged — inspect before training"
-else
- echo " DRY RUN — nothing deleted. Re-run with --delete to apply."
-fi
diff --git a/benchmark/results/3dpw_test.json b/benchmark/results/3dpw_test.json
deleted file mode 100644
index 8b2912f..0000000
--- a/benchmark/results/3dpw_test.json
+++ /dev/null
@@ -1,354 +0,0 @@
-{
- "model": "models/instanthmr_mhr_only_ckpt90.onnx",
- "dataset": "3DPW",
- "split": "test",
- "stride": 1,
- "bbox_scale": 1.2,
- "num_sequences": 24,
- "sequences": [
- "downtown_arguing_00",
- "downtown_bar_00",
- "downtown_bus_00",
- "downtown_cafe_00",
- "downtown_car_00",
- "downtown_crossStreets_00",
- "downtown_downstairs_00",
- "downtown_enterShop_00",
- "downtown_rampAndStairs_00",
- "downtown_runForBus_00",
- "downtown_runForBus_01",
- "downtown_sitOnStairs_00",
- "downtown_stairs_00",
- "downtown_upstairs_00",
- "downtown_walkBridge_01",
- "downtown_walkUphill_00",
- "downtown_walking_00",
- "downtown_warmWelcome_00",
- "downtown_weeklyMarket_00",
- "downtown_windowShopping_00",
- "flat_guitar_01",
- "flat_packBags_00",
- "office_phoneCall_00",
- "outdoors_fencing_01"
- ],
- "num_samples": 35463,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 77.25832087421568,
- "PA_MPJPE_mm": 54.8939122400647,
- "PCK3D_50mm": 0.5790421405005619,
- "PCK3D_100mm": 0.8924653864591264,
- "AUC_0_150mm": 0.6371804272928843,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.99963246267586,
- "right_knee": 42.38728858483827,
- "right_hip": 50.39103164429582,
- "left_hip": 45.39265522544049,
- "left_knee": 46.095110047060146,
- "left_ankle": 62.30361585517269,
- "right_wrist": 63.81340497982142,
- "right_elbow": 41.90725242230058,
- "right_shoulder": 43.96807667630508,
- "left_shoulder": 50.08607425327331,
- "left_elbow": 48.06366572018603,
- "left_wrist": 66.96362486775135,
- "neck": 32.920260588579204,
- "head": 115.22307803320544
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 71.98718459643236,
- "PA_MPJPE_mm": 48.292409136297024,
- "PCK3D_50mm": 0.6549784282209626,
- "PCK3D_100mm": 0.9406353100414516,
- "AUC_0_150mm": 0.6786479562676105,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.99815377789105,
- "right_knee": 43.630305291279456,
- "right_hip": 46.529599094848884,
- "left_hip": 42.83447178924256,
- "left_knee": 47.52243530675108,
- "left_ankle": 57.855173112259216,
- "right_wrist": 62.29407156666568,
- "right_elbow": 36.678818675428985,
- "right_shoulder": 38.21018561144652,
- "left_shoulder": 40.09340440903251,
- "left_elbow": 40.734850438026946,
- "left_wrist": 66.12744056269153
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.85722222445656,
- "PA_MPJPE_mm": 44.083164610596384,
- "PCK3D_50mm": 0.7056892294181863,
- "PCK3D_100mm": 0.9425034543044863,
- "AUC_0_150mm": 0.705719636913712
- }
- ],
- "contamination": {
- "training_dirs": [
- "data/sam3d_distill_mix/annotations",
- "data/sam3d_gt_3dpw/annotations"
- ],
- "seen_frames": 10353,
- "unseen_frames": 25110,
- "seen_fraction": 0.29193807630488117,
- "seen_sequences": [
- "downtown_arguing_00",
- "downtown_bar_00",
- "downtown_bus_00",
- "downtown_cafe_00",
- "downtown_car_00",
- "downtown_crossStreets_00",
- "downtown_downstairs_00",
- "downtown_enterShop_00",
- "downtown_rampAndStairs_00",
- "downtown_runForBus_00",
- "downtown_runForBus_01",
- "downtown_sitOnStairs_00",
- "downtown_stairs_00",
- "downtown_upstairs_00",
- "downtown_walkBridge_01",
- "downtown_walkUphill_00",
- "downtown_walking_00",
- "downtown_warmWelcome_00",
- "downtown_weeklyMarket_00",
- "downtown_windowShopping_00",
- "flat_guitar_01",
- "flat_packBags_00",
- "office_phoneCall_00",
- "outdoors_fencing_01"
- ]
- },
- "results_unseen_frames": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 25110,
- "MPJPE_mm": 80.50931989672257,
- "PA_MPJPE_mm": 57.05366140421865,
- "PCK3D_50mm": 0.5548216419184161,
- "PCK3D_100mm": 0.881657848324515,
- "AUC_0_150mm": 0.6249301231264464,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.47931140891157,
- "right_knee": 43.66784597368266,
- "right_hip": 51.590643380821774,
- "left_hip": 45.99633112852736,
- "left_knee": 47.61141586722965,
- "left_ankle": 65.61625292500608,
- "right_wrist": 69.17219482531468,
- "right_elbow": 43.22191976400349,
- "right_shoulder": 44.582150790065214,
- "left_shoulder": 50.70150039417648,
- "left_elbow": 51.159805717594764,
- "left_wrist": 72.72767706391078,
- "neck": 34.115691861368255,
- "head": 117.10851855844838
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 25110,
- "MPJPE_mm": 75.3534859518377,
- "PA_MPJPE_mm": 50.59458176134388,
- "PCK3D_50mm": 0.6247710075667064,
- "PCK3D_100mm": 0.9298320722155847,
- "AUC_0_150mm": 0.6652752619656309,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.2944636488828,
- "right_knee": 44.98211013553142,
- "right_hip": 47.548593595632475,
- "left_hip": 43.394665628452856,
- "left_knee": 49.07030713900323,
- "left_ankle": 60.92885186640498,
- "right_wrist": 67.01462392288674,
- "right_elbow": 38.323823357280524,
- "right_shoulder": 39.95733150137268,
- "left_shoulder": 41.6934507272976,
- "left_elbow": 43.797015072444125,
- "left_wrist": 71.12974454093712
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 25110,
- "MPJPE_mm": 70.43668183981028,
- "PA_MPJPE_mm": 46.514347723018886,
- "PCK3D_50mm": 0.6766484610570632,
- "PCK3D_100mm": 0.9324372759856631,
- "AUC_0_150mm": 0.6915008065892562
- }
- ],
- "results_seen_frames": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 10353,
- "MPJPE_mm": 69.3734000343482,
- "PA_MPJPE_mm": 49.655691288465555,
- "PCK3D_50mm": 0.6377861489423355,
- "PCK3D_100mm": 0.9186778159539678,
- "AUC_0_150mm": 0.6668921183601361,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.98545895355014,
- "right_knee": 39.2814452511299,
- "right_hip": 47.48151259627433,
- "left_hip": 43.92850938109478,
- "left_knee": 42.41748625256036,
- "left_ankle": 54.26919908481471,
- "right_wrist": 50.81628211491889,
- "right_elbow": 38.71867935650709,
- "right_shoulder": 42.478711178718214,
- "left_shoulder": 48.593429570758246,
- "left_elbow": 40.554337473790454,
- "left_wrist": 52.98358520334844,
- "neck": 30.020880770194864,
- "head": 110.65016085085732
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 10353,
- "MPJPE_mm": 63.82261132933802,
- "PA_MPJPE_mm": 42.70875660901737,
- "PCK3D_50mm": 0.7282430213464697,
- "PCK3D_100mm": 0.9668373096364983,
- "AUC_0_150mm": 0.7110818743515204,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 51.42872068017997,
- "right_knee": 40.351659522983596,
- "right_hip": 44.05814619089098,
- "left_hip": 41.47578664459172,
- "left_knee": 43.768251813285275,
- "left_ankle": 50.400322004696164,
- "right_wrist": 50.84491966241465,
- "right_elbow": 32.689050727849335,
- "right_shoulder": 33.97268601750797,
- "left_shoulder": 36.21267775476452,
- "left_elbow": 33.30792549161378,
- "left_wrist": 53.99493279743053
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 10353,
- "MPJPE_mm": 58.175658335580685,
- "PA_MPJPE_mm": 38.18661211828219,
- "PCK3D_50mm": 0.7761242428005685,
- "PCK3D_100mm": 0.9669178016034,
- "AUC_0_150mm": 0.7402057597232443
- }
- ],
- "per_sequence_J14": {
- "downtown_arguing_00": {
- "n": 1793,
- "PA_MPJPE_mm": 53.86209217665553
- },
- "downtown_bar_00": {
- "n": 2458,
- "PA_MPJPE_mm": 67.11104059483931
- },
- "downtown_bus_00": {
- "n": 2534,
- "PA_MPJPE_mm": 57.3890391264324
- },
- "downtown_cafe_00": {
- "n": 2076,
- "PA_MPJPE_mm": 52.69031252948052
- },
- "downtown_car_00": {
- "n": 1546,
- "PA_MPJPE_mm": 60.43672446675064
- },
- "downtown_crossStreets_00": {
- "n": 1042,
- "PA_MPJPE_mm": 55.938241587076654
- },
- "downtown_downstairs_00": {
- "n": 638,
- "PA_MPJPE_mm": 47.09329939736882
- },
- "downtown_enterShop_00": {
- "n": 1385,
- "PA_MPJPE_mm": 46.63087615806622
- },
- "downtown_rampAndStairs_00": {
- "n": 1954,
- "PA_MPJPE_mm": 56.34401250060689
- },
- "downtown_runForBus_00": {
- "n": 1302,
- "PA_MPJPE_mm": 52.232966837233924
- },
- "downtown_runForBus_01": {
- "n": 1437,
- "PA_MPJPE_mm": 66.2344342328675
- },
- "downtown_sitOnStairs_00": {
- "n": 2631,
- "PA_MPJPE_mm": 52.626225175849925
- },
- "downtown_stairs_00": {
- "n": 1197,
- "PA_MPJPE_mm": 63.31683556710035
- },
- "downtown_upstairs_00": {
- "n": 825,
- "PA_MPJPE_mm": 47.17526695041395
- },
- "downtown_walkBridge_01": {
- "n": 1182,
- "PA_MPJPE_mm": 44.298063167659926
- },
- "downtown_walkUphill_00": {
- "n": 387,
- "PA_MPJPE_mm": 44.89834428674509
- },
- "downtown_walking_00": {
- "n": 2594,
- "PA_MPJPE_mm": 52.14282318214196
- },
- "downtown_warmWelcome_00": {
- "n": 1122,
- "PA_MPJPE_mm": 64.06683409393634
- },
- "downtown_weeklyMarket_00": {
- "n": 1055,
- "PA_MPJPE_mm": 47.945786223105564
- },
- "downtown_windowShopping_00": {
- "n": 1821,
- "PA_MPJPE_mm": 44.496857077869684
- },
- "flat_guitar_01": {
- "n": 748,
- "PA_MPJPE_mm": 60.36081947705571
- },
- "flat_packBags_00": {
- "n": 1273,
- "PA_MPJPE_mm": 64.67982670780314
- },
- "office_phoneCall_00": {
- "n": 1524,
- "PA_MPJPE_mm": 45.326043123588434
- },
- "outdoors_fencing_01": {
- "n": 939,
- "PA_MPJPE_mm": 55.91377958820054
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_adapter.json b/benchmark/results/3dpw_test_adapter.json
deleted file mode 100644
index 8f64214..0000000
--- a/benchmark/results/3dpw_test_adapter.json
+++ /dev/null
@@ -1,334 +0,0 @@
-{
- "model": "models/instanthmr_mhr_only_ckpt90.onnx",
- "dataset": "3DPW",
- "split": "test",
- "stride": 1,
- "bbox_scale": 1.2,
- "num_sequences": 24,
- "sequences": [
- "downtown_arguing_00",
- "downtown_bar_00",
- "downtown_bus_00",
- "downtown_cafe_00",
- "downtown_car_00",
- "downtown_crossStreets_00",
- "downtown_downstairs_00",
- "downtown_enterShop_00",
- "downtown_rampAndStairs_00",
- "downtown_runForBus_00",
- "downtown_runForBus_01",
- "downtown_sitOnStairs_00",
- "downtown_stairs_00",
- "downtown_upstairs_00",
- "downtown_walkBridge_01",
- "downtown_walkUphill_00",
- "downtown_walking_00",
- "downtown_warmWelcome_00",
- "downtown_weeklyMarket_00",
- "downtown_windowShopping_00",
- "flat_guitar_01",
- "flat_packBags_00",
- "office_phoneCall_00",
- "outdoors_fencing_01"
- ],
- "num_samples": 35463,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 77.25832087421568,
- "PA_MPJPE_mm": 54.8939122400647,
- "PCK3D_50mm": 0.5790421405005619,
- "PCK3D_100mm": 0.8924653864591264,
- "AUC_0_150mm": 0.6371804272928843,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.99963246267586,
- "right_knee": 42.38728858483827,
- "right_hip": 50.39103164429582,
- "left_hip": 45.39265522544049,
- "left_knee": 46.095110047060146,
- "left_ankle": 62.30361585517269,
- "right_wrist": 63.81340497982142,
- "right_elbow": 41.90725242230058,
- "right_shoulder": 43.96807667630508,
- "left_shoulder": 50.08607425327331,
- "left_elbow": 48.06366572018603,
- "left_wrist": 66.96362486775135,
- "neck": 32.920260588579204,
- "head": 115.22307803320544
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 71.98718459643236,
- "PA_MPJPE_mm": 48.292409136297024,
- "PCK3D_50mm": 0.6549784282209626,
- "PCK3D_100mm": 0.9406353100414516,
- "AUC_0_150mm": 0.6786479562676105,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.99815377789105,
- "right_knee": 43.630305291279456,
- "right_hip": 46.529599094848884,
- "left_hip": 42.83447178924256,
- "left_knee": 47.52243530675108,
- "left_ankle": 57.855173112259216,
- "right_wrist": 62.29407156666568,
- "right_elbow": 36.678818675428985,
- "right_shoulder": 38.21018561144652,
- "left_shoulder": 40.09340440903251,
- "left_elbow": 40.734850438026946,
- "left_wrist": 66.12744056269153
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.85722222445656,
- "PA_MPJPE_mm": 44.083164610596384,
- "PCK3D_50mm": 0.7056892294181863,
- "PCK3D_100mm": 0.9425034543044863,
- "AUC_0_150mm": 0.705719636913712
- }
- ],
- "contamination": {
- "training_dirs": [
- "data/sam3d_distill_mix/annotations",
- "data/sam3d_gt_3dpw/annotations"
- ],
- "seen_frames": 10353,
- "unseen_frames": 25110,
- "seen_fraction": 0.29193807630488117,
- "seen_sequences": [
- "downtown_arguing_00",
- "downtown_bar_00",
- "downtown_bus_00",
- "downtown_cafe_00",
- "downtown_car_00",
- "downtown_crossStreets_00",
- "downtown_downstairs_00",
- "downtown_enterShop_00",
- "downtown_rampAndStairs_00",
- "downtown_runForBus_00",
- "downtown_runForBus_01",
- "downtown_sitOnStairs_00",
- "downtown_stairs_00",
- "downtown_upstairs_00",
- "downtown_walkBridge_01",
- "downtown_walkUphill_00",
- "downtown_walking_00",
- "downtown_warmWelcome_00",
- "downtown_weeklyMarket_00",
- "downtown_windowShopping_00",
- "flat_guitar_01",
- "flat_packBags_00",
- "office_phoneCall_00",
- "outdoors_fencing_01"
- ]
- },
- "results_unseen_frames": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 25110,
- "MPJPE_mm": 80.50931989672257,
- "PA_MPJPE_mm": 57.05366140421865,
- "PCK3D_50mm": 0.5548216419184161,
- "PCK3D_100mm": 0.881657848324515,
- "AUC_0_150mm": 0.6249301231264464,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.47931140891157,
- "right_knee": 43.66784597368266,
- "right_hip": 51.590643380821774,
- "left_hip": 45.99633112852736,
- "left_knee": 47.61141586722965,
- "left_ankle": 65.61625292500608,
- "right_wrist": 69.17219482531468,
- "right_elbow": 43.22191976400349,
- "right_shoulder": 44.582150790065214,
- "left_shoulder": 50.70150039417648,
- "left_elbow": 51.159805717594764,
- "left_wrist": 72.72767706391078,
- "neck": 34.115691861368255,
- "head": 117.10851855844838
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 25110,
- "MPJPE_mm": 75.3534859518377,
- "PA_MPJPE_mm": 50.59458176134388,
- "PCK3D_50mm": 0.6247710075667064,
- "PCK3D_100mm": 0.9298320722155847,
- "AUC_0_150mm": 0.6652752619656309,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.2944636488828,
- "right_knee": 44.98211013553142,
- "right_hip": 47.548593595632475,
- "left_hip": 43.394665628452856,
- "left_knee": 49.07030713900323,
- "left_ankle": 60.92885186640498,
- "right_wrist": 67.01462392288674,
- "right_elbow": 38.323823357280524,
- "right_shoulder": 39.95733150137268,
- "left_shoulder": 41.6934507272976,
- "left_elbow": 43.797015072444125,
- "left_wrist": 71.12974454093712
- }
- }
- ],
- "results_seen_frames": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 10353,
- "MPJPE_mm": 69.3734000343482,
- "PA_MPJPE_mm": 49.655691288465555,
- "PCK3D_50mm": 0.6377861489423355,
- "PCK3D_100mm": 0.9186778159539678,
- "AUC_0_150mm": 0.6668921183601361,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.98545895355014,
- "right_knee": 39.2814452511299,
- "right_hip": 47.48151259627433,
- "left_hip": 43.92850938109478,
- "left_knee": 42.41748625256036,
- "left_ankle": 54.26919908481471,
- "right_wrist": 50.81628211491889,
- "right_elbow": 38.71867935650709,
- "right_shoulder": 42.478711178718214,
- "left_shoulder": 48.593429570758246,
- "left_elbow": 40.554337473790454,
- "left_wrist": 52.98358520334844,
- "neck": 30.020880770194864,
- "head": 110.65016085085732
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 10353,
- "MPJPE_mm": 63.82261132933802,
- "PA_MPJPE_mm": 42.70875660901737,
- "PCK3D_50mm": 0.7282430213464697,
- "PCK3D_100mm": 0.9668373096364983,
- "AUC_0_150mm": 0.7110818743515204,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 51.42872068017997,
- "right_knee": 40.351659522983596,
- "right_hip": 44.05814619089098,
- "left_hip": 41.47578664459172,
- "left_knee": 43.768251813285275,
- "left_ankle": 50.400322004696164,
- "right_wrist": 50.84491966241465,
- "right_elbow": 32.689050727849335,
- "right_shoulder": 33.97268601750797,
- "left_shoulder": 36.21267775476452,
- "left_elbow": 33.30792549161378,
- "left_wrist": 53.99493279743053
- }
- }
- ],
- "per_sequence_J14": {
- "downtown_arguing_00": {
- "n": 1793,
- "PA_MPJPE_mm": 53.86209217665553
- },
- "downtown_bar_00": {
- "n": 2458,
- "PA_MPJPE_mm": 67.11104059483931
- },
- "downtown_bus_00": {
- "n": 2534,
- "PA_MPJPE_mm": 57.3890391264324
- },
- "downtown_cafe_00": {
- "n": 2076,
- "PA_MPJPE_mm": 52.69031252948052
- },
- "downtown_car_00": {
- "n": 1546,
- "PA_MPJPE_mm": 60.43672446675064
- },
- "downtown_crossStreets_00": {
- "n": 1042,
- "PA_MPJPE_mm": 55.938241587076654
- },
- "downtown_downstairs_00": {
- "n": 638,
- "PA_MPJPE_mm": 47.09329939736882
- },
- "downtown_enterShop_00": {
- "n": 1385,
- "PA_MPJPE_mm": 46.63087615806622
- },
- "downtown_rampAndStairs_00": {
- "n": 1954,
- "PA_MPJPE_mm": 56.34401250060689
- },
- "downtown_runForBus_00": {
- "n": 1302,
- "PA_MPJPE_mm": 52.232966837233924
- },
- "downtown_runForBus_01": {
- "n": 1437,
- "PA_MPJPE_mm": 66.2344342328675
- },
- "downtown_sitOnStairs_00": {
- "n": 2631,
- "PA_MPJPE_mm": 52.626225175849925
- },
- "downtown_stairs_00": {
- "n": 1197,
- "PA_MPJPE_mm": 63.31683556710035
- },
- "downtown_upstairs_00": {
- "n": 825,
- "PA_MPJPE_mm": 47.17526695041395
- },
- "downtown_walkBridge_01": {
- "n": 1182,
- "PA_MPJPE_mm": 44.298063167659926
- },
- "downtown_walkUphill_00": {
- "n": 387,
- "PA_MPJPE_mm": 44.89834428674509
- },
- "downtown_walking_00": {
- "n": 2594,
- "PA_MPJPE_mm": 52.14282318214196
- },
- "downtown_warmWelcome_00": {
- "n": 1122,
- "PA_MPJPE_mm": 64.06683409393634
- },
- "downtown_weeklyMarket_00": {
- "n": 1055,
- "PA_MPJPE_mm": 47.945786223105564
- },
- "downtown_windowShopping_00": {
- "n": 1821,
- "PA_MPJPE_mm": 44.496857077869684
- },
- "flat_guitar_01": {
- "n": 748,
- "PA_MPJPE_mm": 60.36081947705571
- },
- "flat_packBags_00": {
- "n": 1273,
- "PA_MPJPE_mm": 64.67982670780314
- },
- "office_phoneCall_00": {
- "n": 1524,
- "PA_MPJPE_mm": 45.326043123588434
- },
- "outdoors_fencing_01": {
- "n": 939,
- "PA_MPJPE_mm": 55.91377958820054
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_alladapter.json b/benchmark/results/3dpw_test_alladapter.json
deleted file mode 100644
index 5143458..0000000
--- a/benchmark/results/3dpw_test_alladapter.json
+++ /dev/null
@@ -1,146 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 137,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.63785012091508,
- "PA_MPJPE_mm": 67.89201715239723,
- "PCK3D_50mm": 0.386628316837267,
- "PCK3D_100mm": 0.8279313248013019,
- "AUC_0_150mm": 0.555249314824265,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.1026158925688,
- "right_knee": 48.77531527426408,
- "right_hip": 98.43968262009881,
- "left_hip": 86.54123107665853,
- "left_knee": 46.110874847033735,
- "left_ankle": 67.11189987887853,
- "right_wrist": 70.71902986402077,
- "right_elbow": 57.68874380634787,
- "right_shoulder": 51.452820373726844,
- "left_shoulder": 53.46686079217727,
- "left_elbow": 66.61272199119497,
- "left_wrist": 74.65868335746043,
- "neck": 31.322680447479044,
- "head": 131.48507991165152
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.43934163084393,
- "PA_MPJPE_mm": 62.492379121774945,
- "PCK3D_50mm": 0.43160947090394686,
- "PCK3D_100mm": 0.8874202220154339,
- "AUC_0_150mm": 0.5877625294150285,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.453416312484485,
- "right_knee": 50.619841347293885,
- "right_hip": 98.35461927230007,
- "left_hip": 85.97593910675288,
- "left_knee": 47.09601993678759,
- "left_ankle": 62.49923408065495,
- "right_wrist": 66.505798652139,
- "right_elbow": 48.842995632922765,
- "right_shoulder": 51.30888543605562,
- "left_shoulder": 51.65398595718133,
- "left_elbow": 54.28141719363774,
- "left_wrist": 70.316396533089
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 69.02998274054096,
- "PA_MPJPE_mm": 41.82524856095757,
- "PCK3D_50mm": 0.7230534037487764,
- "PCK3D_100mm": 0.9491320934092273,
- "AUC_0_150mm": 0.7205023578153955
- }
- ]
- },
- "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 85,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.59075208981304,
- "PA_MPJPE_mm": 68.65276639038719,
- "PCK3D_50mm": 0.38712380307846006,
- "PCK3D_100mm": 0.8200015307705012,
- "AUC_0_150mm": 0.5514413607692109,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.6002112208962,
- "right_knee": 47.796202194113405,
- "right_hip": 98.89813178363754,
- "left_hip": 86.54804326749168,
- "left_knee": 47.93220834357654,
- "left_ankle": 70.43573883389739,
- "right_wrist": 70.9671299706096,
- "right_elbow": 59.931862055983714,
- "right_shoulder": 48.88554359499399,
- "left_shoulder": 50.86661346152063,
- "left_elbow": 70.13890219496395,
- "left_wrist": 74.72033982944755,
- "neck": 34.32004777862704,
- "head": 131.09775493566124
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 103.55279371039903,
- "PA_MPJPE_mm": 63.212442975844404,
- "PCK3D_50mm": 0.42238389307165214,
- "PCK3D_100mm": 0.8809745368412147,
- "AUC_0_150mm": 0.5840778621607435,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.31700202915948,
- "right_knee": 49.58846684843106,
- "right_hip": 97.74150472679717,
- "left_hip": 85.53733176084349,
- "left_knee": 48.44964497448344,
- "left_ankle": 64.69117986025115,
- "right_wrist": 67.87638478520945,
- "right_elbow": 50.475248931588496,
- "right_shoulder": 51.10462090980213,
- "left_shoulder": 51.255945550101416,
- "left_elbow": 56.66098902303212,
- "left_wrist": 70.8509963104334
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 70.63804778208632,
- "PA_MPJPE_mm": 43.80571997315016,
- "PCK3D_50mm": 0.6998984857457068,
- "PCK3D_100mm": 0.9421328467094476,
- "AUC_0_150mm": 0.7086226431104736
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_b3_s1.json b/benchmark/results/3dpw_test_b3_s1.json
deleted file mode 100644
index 06c5dea..0000000
--- a/benchmark/results/3dpw_test_b3_s1.json
+++ /dev/null
@@ -1,74 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 137,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.63785012091508,
- "PA_MPJPE_mm": 67.89201715239723,
- "PCK3D_50mm": 0.386628316837267,
- "PCK3D_100mm": 0.8279313248013019,
- "AUC_0_150mm": 0.555249314824265,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.1026158925688,
- "right_knee": 48.77531527426408,
- "right_hip": 98.43968262009881,
- "left_hip": 86.54123107665853,
- "left_knee": 46.110874847033735,
- "left_ankle": 67.11189987887853,
- "right_wrist": 70.71902986402077,
- "right_elbow": 57.68874380634787,
- "right_shoulder": 51.452820373726844,
- "left_shoulder": 53.46686079217727,
- "left_elbow": 66.61272199119497,
- "left_wrist": 74.65868335746043,
- "neck": 31.322680447479044,
- "head": 131.48507991165152
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.43934163084393,
- "PA_MPJPE_mm": 62.492379121774945,
- "PCK3D_50mm": 0.43160947090394686,
- "PCK3D_100mm": 0.8874202220154339,
- "AUC_0_150mm": 0.5877625294150285,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.453416312484485,
- "right_knee": 50.619841347293885,
- "right_hip": 98.35461927230007,
- "left_hip": 85.97593910675288,
- "left_knee": 47.09601993678759,
- "left_ankle": 62.49923408065495,
- "right_wrist": 66.505798652139,
- "right_elbow": 48.842995632922765,
- "right_shoulder": 51.30888543605562,
- "left_shoulder": 51.65398595718133,
- "left_elbow": 54.28141719363774,
- "left_wrist": 70.316396533089
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 73.44449613823136,
- "PA_MPJPE_mm": 42.380049729446,
- "PCK3D_50mm": 0.7164247646440355,
- "PCK3D_100mm": 0.9470897232930902,
- "AUC_0_150mm": 0.7166858272872446
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_b3_s1_teacher_adapter.json b/benchmark/results/3dpw_test_b3_s1_teacher_adapter.json
deleted file mode 100644
index a372e83..0000000
--- a/benchmark/results/3dpw_test_b3_s1_teacher_adapter.json
+++ /dev/null
@@ -1,74 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 137,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.63785012091508,
- "PA_MPJPE_mm": 67.89201715239723,
- "PCK3D_50mm": 0.386628316837267,
- "PCK3D_100mm": 0.8279313248013019,
- "AUC_0_150mm": 0.555249314824265,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.1026158925688,
- "right_knee": 48.77531527426408,
- "right_hip": 98.43968262009881,
- "left_hip": 86.54123107665853,
- "left_knee": 46.110874847033735,
- "left_ankle": 67.11189987887853,
- "right_wrist": 70.71902986402077,
- "right_elbow": 57.68874380634787,
- "right_shoulder": 51.452820373726844,
- "left_shoulder": 53.46686079217727,
- "left_elbow": 66.61272199119497,
- "left_wrist": 74.65868335746043,
- "neck": 31.322680447479044,
- "head": 131.48507991165152
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.43934163084393,
- "PA_MPJPE_mm": 62.492379121774945,
- "PCK3D_50mm": 0.43160947090394686,
- "PCK3D_100mm": 0.8874202220154339,
- "AUC_0_150mm": 0.5877625294150285,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.453416312484485,
- "right_knee": 50.619841347293885,
- "right_hip": 98.35461927230007,
- "left_hip": 85.97593910675288,
- "left_knee": 47.09601993678759,
- "left_ankle": 62.49923408065495,
- "right_wrist": 66.505798652139,
- "right_elbow": 48.842995632922765,
- "right_shoulder": 51.30888543605562,
- "left_shoulder": 51.65398595718133,
- "left_elbow": 54.28141719363774,
- "left_wrist": 70.316396533089
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 69.02998274054096,
- "PA_MPJPE_mm": 41.82524856095757,
- "PCK3D_50mm": 0.7230534037487764,
- "PCK3D_100mm": 0.9491320934092273,
- "AUC_0_150mm": 0.7205023578153955
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_gen5.json b/benchmark/results/3dpw_test_gen5.json
deleted file mode 100644
index 089baf0..0000000
--- a/benchmark/results/3dpw_test_gen5.json
+++ /dev/null
@@ -1,290 +0,0 @@
-{
- "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 85,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.59075208981304,
- "PA_MPJPE_mm": 68.65276639038719,
- "PCK3D_50mm": 0.38712380307846006,
- "PCK3D_100mm": 0.8200015307705012,
- "AUC_0_150mm": 0.5514413607692109,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.6002112208962,
- "right_knee": 47.796202194113405,
- "right_hip": 98.89813178363754,
- "left_hip": 86.54804326749168,
- "left_knee": 47.93220834357654,
- "left_ankle": 70.43573883389739,
- "right_wrist": 70.9671299706096,
- "right_elbow": 59.931862055983714,
- "right_shoulder": 48.88554359499399,
- "left_shoulder": 50.86661346152063,
- "left_elbow": 70.13890219496395,
- "left_wrist": 74.72033982944755,
- "neck": 34.32004777862704,
- "head": 131.09775493566124
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 103.55279371039903,
- "PA_MPJPE_mm": 63.212442975844404,
- "PCK3D_50mm": 0.42238389307165214,
- "PCK3D_100mm": 0.8809745368412147,
- "AUC_0_150mm": 0.5840778621607435,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.31700202915948,
- "right_knee": 49.58846684843106,
- "right_hip": 97.74150472679717,
- "left_hip": 85.53733176084349,
- "left_knee": 48.44964497448344,
- "left_ankle": 64.69117986025115,
- "right_wrist": 67.87638478520945,
- "right_elbow": 50.475248931588496,
- "right_shoulder": 51.10462090980213,
- "left_shoulder": 51.255945550101416,
- "left_elbow": 56.66098902303212,
- "left_wrist": 70.8509963104334
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 70.63804778208632,
- "PA_MPJPE_mm": 43.80571997315016,
- "PCK3D_50mm": 0.6998984857457068,
- "PCK3D_100mm": 0.9421328467094476,
- "AUC_0_150mm": 0.7086226431104736
- }
- ]
- },
- "instanthmr_distill_train/runs/bno_s1/bno_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/bno_s1/bno_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 83,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 106.30543762749703,
- "PA_MPJPE_mm": 69.44755093221157,
- "PCK3D_50mm": 0.38019908073203057,
- "PCK3D_100mm": 0.8145914655516212,
- "AUC_0_150mm": 0.5472726100845551,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.74238502662348,
- "right_knee": 50.229744214062734,
- "right_hip": 99.11057157887134,
- "left_hip": 87.11689008905907,
- "left_knee": 48.94307517094425,
- "left_ankle": 71.62276277304298,
- "right_wrist": 72.18656746368522,
- "right_elbow": 60.28634507967081,
- "right_shoulder": 49.20902541330741,
- "left_shoulder": 51.6274906772661,
- "left_elbow": 69.61796047904427,
- "left_wrist": 78.3774062085848,
- "neck": 35.00746248300054,
- "head": 131.1880263937989
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 104.2722236813728,
- "PA_MPJPE_mm": 64.21451119174024,
- "PCK3D_50mm": 0.4131536155053624,
- "PCK3D_100mm": 0.8724656684431661,
- "AUC_0_150mm": 0.5785831150989112,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.47348616999159,
- "right_knee": 51.722590577312005,
- "right_hip": 98.26981164392382,
- "left_hip": 86.4184403211381,
- "left_knee": 49.3418828076764,
- "left_ankle": 66.84643443124799,
- "right_wrist": 68.57455496187201,
- "right_elbow": 51.34128655762564,
- "right_shoulder": 50.95717427196927,
- "left_shoulder": 51.53147956682218,
- "left_elbow": 56.7424678225824,
- "left_wrist": 74.35452516872154
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 72.11047276121764,
- "PA_MPJPE_mm": 45.08471664866667,
- "PCK3D_50mm": 0.6821717605069267,
- "PCK3D_100mm": 0.9376895839124078,
- "AUC_0_150mm": 0.7012210168812281
- }
- ]
- },
- "instanthmr_distill_train/runs/dt_s0/dt_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/dt_s0/dt_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 85,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 113.34418144842128,
- "PA_MPJPE_mm": 69.84164513504265,
- "PCK3D_50mm": 0.38238244286801937,
- "PCK3D_100mm": 0.8120697225679884,
- "AUC_0_150mm": 0.545767439056037,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.41474637936595,
- "right_knee": 47.67699784860789,
- "right_hip": 97.78243974951047,
- "left_hip": 86.92655040131828,
- "left_knee": 47.27021771033743,
- "left_ankle": 71.70788367819415,
- "right_wrist": 76.16593910378955,
- "right_elbow": 62.16631485447152,
- "right_shoulder": 48.92740512341149,
- "left_shoulder": 51.04701763616356,
- "left_elbow": 71.05377174626769,
- "left_wrist": 78.67982288070175,
- "neck": 34.30443174226576,
- "head": 134.65949303619172
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 111.1924824935285,
- "PA_MPJPE_mm": 64.17582989732823,
- "PCK3D_50mm": 0.4218763218001861,
- "PCK3D_100mm": 0.8720520918516012,
- "AUC_0_150mm": 0.5794967585479823,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.0988301138224,
- "right_knee": 49.43896704040137,
- "right_hip": 97.50509750422344,
- "left_hip": 86.45899166106351,
- "left_knee": 47.99247268585938,
- "left_ankle": 65.93102969455582,
- "right_wrist": 73.0855595216394,
- "right_elbow": 52.04128443883315,
- "right_shoulder": 50.16486648188655,
- "left_shoulder": 50.43834591421628,
- "left_elbow": 57.25801910095047,
- "left_wrist": 74.69649461048695
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 82.33977956088862,
- "PA_MPJPE_mm": 45.745626042711464,
- "PCK3D_50mm": 0.6781514737694417,
- "PCK3D_100mm": 0.9323359155014683,
- "AUC_0_150mm": 0.6976277345467224
- }
- ]
- },
- "instanthmr_distill_train/runs/dt_s1/dt_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/dt_s1/dt_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 87,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 108.93126868900502,
- "PA_MPJPE_mm": 69.53100413938488,
- "PCK3D_50mm": 0.37985465736925006,
- "PCK3D_100mm": 0.8164586027287999,
- "AUC_0_150mm": 0.5471804779720436,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.55625612825625,
- "right_knee": 47.71172195018994,
- "right_hip": 96.54945478544958,
- "left_hip": 86.4034397102533,
- "left_knee": 47.583384395413084,
- "left_ankle": 72.05032976730145,
- "right_wrist": 75.2052788934457,
- "right_elbow": 62.21923341843739,
- "right_shoulder": 49.43185737953577,
- "left_shoulder": 51.118420086942656,
- "left_elbow": 71.58790258368994,
- "left_wrist": 76.93281864570858,
- "neck": 34.66586332776017,
- "head": 132.41809687900442
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 106.99366688568519,
- "PA_MPJPE_mm": 64.08310421047089,
- "PCK3D_50mm": 0.4158794612224948,
- "PCK3D_100mm": 0.8770032616153925,
- "AUC_0_150mm": 0.5797055935021175,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.29752536225554,
- "right_knee": 49.64317526219177,
- "right_hip": 95.86754622285763,
- "left_hip": 86.00415756921923,
- "left_knee": 48.316752199732825,
- "left_ankle": 65.89142714634777,
- "right_wrist": 72.50887822047521,
- "right_elbow": 52.09341400163957,
- "right_shoulder": 51.17750531428965,
- "left_shoulder": 51.22386739723045,
- "left_elbow": 57.616979263684016,
- "left_wrist": 73.35602256572712
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 76.87984598188521,
- "PA_MPJPE_mm": 45.3190899609874,
- "PCK3D_50mm": 0.6844477745416753,
- "PCK3D_100mm": 0.9339452386994896,
- "AUC_0_150mm": 0.7000717694862341
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_smplfit.json b/benchmark/results/3dpw_test_smplfit.json
deleted file mode 100644
index ae04c95..0000000
--- a/benchmark/results/3dpw_test_smplfit.json
+++ /dev/null
@@ -1,88 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 137,
- "source": "ema",
- "fit_residual_mm": 17.004608154296875,
- "MPJPE_mm": 70.12309265136719,
- "PA_MPJPE_mm": 49.299208058306824,
- "PVE_mm": 83.93521674934446,
- "PA_PVE_mm": 59.70094080149259,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 55.128664281708716,
- "downtown_bar_00": 60.09846807136917,
- "downtown_bus_00": 57.47289959628734,
- "downtown_cafe_00": 46.52143377778766,
- "downtown_car_00": 53.570638641576814,
- "downtown_crossStreets_00": 49.85964262237165,
- "downtown_downstairs_00": 41.02150680058183,
- "downtown_enterShop_00": 36.96853688737289,
- "downtown_rampAndStairs_00": 45.00065479850545,
- "downtown_runForBus_00": 45.257167005245286,
- "downtown_runForBus_01": 58.42430960101843,
- "downtown_sitOnStairs_00": 47.7291710517197,
- "downtown_stairs_00": 57.84252563548758,
- "downtown_upstairs_00": 39.42688494174954,
- "downtown_walkBridge_01": 40.05517954627335,
- "downtown_walkUphill_00": 37.15261530936371,
- "downtown_walking_00": 47.660608196674175,
- "downtown_warmWelcome_00": 53.35867689309556,
- "downtown_weeklyMarket_00": 41.23658721494934,
- "downtown_windowShopping_00": 38.65736838805555,
- "flat_guitar_01": 63.87007688526666,
- "flat_packBags_00": 55.583542431958634,
- "office_phoneCall_00": 42.071492928142334,
- "outdoors_fencing_01": 50.52562153596208
- },
- "J14_h36m_viamesh_MPJPE_mm": 69.96729793143032,
- "J14_h36m_viamesh_PA_MPJPE_mm": 46.13575630968572
- },
- "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 85,
- "source": "ema",
- "fit_residual_mm": 16.382274627685547,
- "MPJPE_mm": 71.3781509399414,
- "PA_MPJPE_mm": 50.89205090035432,
- "PVE_mm": 84.60048436590854,
- "PA_PVE_mm": 61.25163162048173,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 55.51393117547959,
- "downtown_bar_00": 60.513748658835624,
- "downtown_bus_00": 59.656931380784485,
- "downtown_cafe_00": 46.960072212207756,
- "downtown_car_00": 58.120584069229224,
- "downtown_crossStreets_00": 51.57475394673719,
- "downtown_downstairs_00": 43.31706603789759,
- "downtown_enterShop_00": 41.44317245415561,
- "downtown_rampAndStairs_00": 47.67656300606207,
- "downtown_runForBus_00": 45.31364935332185,
- "downtown_runForBus_01": 57.0176377230583,
- "downtown_sitOnStairs_00": 49.564813096273184,
- "downtown_stairs_00": 57.30311493027224,
- "downtown_upstairs_00": 40.619809459390126,
- "downtown_walkBridge_01": 39.999748915846226,
- "downtown_walkUphill_00": 38.30589736074419,
- "downtown_walking_00": 47.66982111316505,
- "downtown_warmWelcome_00": 57.76132724620347,
- "downtown_weeklyMarket_00": 46.6148834939354,
- "downtown_windowShopping_00": 42.05428317264101,
- "flat_guitar_01": 59.238372173536504,
- "flat_packBags_00": 61.617872574583046,
- "office_phoneCall_00": 43.61936431501065,
- "outdoors_fencing_01": 50.118543453226955
- },
- "J14_h36m_viamesh_MPJPE_mm": 70.8501205444336,
- "J14_h36m_viamesh_PA_MPJPE_mm": 47.68866579180827
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_v3_s1.json b/benchmark/results/3dpw_test_v3_s1.json
deleted file mode 100644
index b4b2be9..0000000
--- a/benchmark/results/3dpw_test_v3_s1.json
+++ /dev/null
@@ -1,74 +0,0 @@
-{
- "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 126,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.70162783403123,
- "PA_MPJPE_mm": 69.05148473277063,
- "PCK3D_50mm": 0.37404175780793664,
- "PCK3D_100mm": 0.8228153286523983,
- "AUC_0_150mm": 0.5488736167026034,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.26131519537937,
- "right_knee": 48.22895082504128,
- "right_hip": 98.35751407606348,
- "left_hip": 86.86613781457758,
- "left_knee": 46.25701446183899,
- "left_ankle": 69.39201526188755,
- "right_wrist": 71.39134970517831,
- "right_elbow": 57.63600754408093,
- "right_shoulder": 52.59923939395279,
- "left_shoulder": 55.207572449651074,
- "left_elbow": 67.88209503589113,
- "left_wrist": 76.36662877587462,
- "neck": 32.38007584338846,
- "head": 135.8948698759832
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.98084160960175,
- "PA_MPJPE_mm": 63.198364128279884,
- "PCK3D_50mm": 0.42087997819323425,
- "PCK3D_100mm": 0.8823257103647933,
- "AUC_0_150mm": 0.5837227290354721,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.60270339441506,
- "right_knee": 49.996919602186196,
- "right_hip": 98.25229959444044,
- "left_hip": 86.3963348166251,
- "left_knee": 47.03163228047883,
- "left_ankle": 64.51319472082247,
- "right_wrist": 67.54483867159513,
- "right_elbow": 48.887356763210875,
- "right_shoulder": 52.06354180217583,
- "left_shoulder": 52.42872324128663,
- "left_elbow": 54.99294113405385,
- "left_wrist": 71.66988351806822
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 74.66172176671088,
- "PA_MPJPE_mm": 43.63754202578393,
- "PCK3D_50mm": 0.7020536494777253,
- "PCK3D_100mm": 0.9413050221357471,
- "AUC_0_150mm": 0.70911897400432
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/3dpw_test_v3_s1_teacher_adapter.json b/benchmark/results/3dpw_test_v3_s1_teacher_adapter.json
deleted file mode 100644
index aef8a8e..0000000
--- a/benchmark/results/3dpw_test_v3_s1_teacher_adapter.json
+++ /dev/null
@@ -1,74 +0,0 @@
-{
- "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/v3_s1/v3_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 126,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.70162783403123,
- "PA_MPJPE_mm": 69.05148473277063,
- "PCK3D_50mm": 0.37404175780793664,
- "PCK3D_100mm": 0.8228153286523983,
- "AUC_0_150mm": 0.5488736167026034,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.26131519537937,
- "right_knee": 48.22895082504128,
- "right_hip": 98.35751407606348,
- "left_hip": 86.86613781457758,
- "left_knee": 46.25701446183899,
- "left_ankle": 69.39201526188755,
- "right_wrist": 71.39134970517831,
- "right_elbow": 57.63600754408093,
- "right_shoulder": 52.59923939395279,
- "left_shoulder": 55.207572449651074,
- "left_elbow": 67.88209503589113,
- "left_wrist": 76.36662877587462,
- "neck": 32.38007584338846,
- "head": 135.8948698759832
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.98084160960175,
- "PA_MPJPE_mm": 63.198364128279884,
- "PCK3D_50mm": 0.42087997819323425,
- "PCK3D_100mm": 0.8823257103647933,
- "AUC_0_150mm": 0.5837227290354721,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.60270339441506,
- "right_knee": 49.996919602186196,
- "right_hip": 98.25229959444044,
- "left_hip": 86.3963348166251,
- "left_knee": 47.03163228047883,
- "left_ankle": 64.51319472082247,
- "right_wrist": 67.54483867159513,
- "right_elbow": 48.887356763210875,
- "right_shoulder": 52.06354180217583,
- "left_shoulder": 52.42872324128663,
- "left_elbow": 54.99294113405385,
- "left_wrist": 71.66988351806822
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 70.59166700598287,
- "PA_MPJPE_mm": 43.23008138157838,
- "PCK3D_50mm": 0.7050406661268687,
- "PCK3D_100mm": 0.9443907331987866,
- "AUC_0_150mm": 0.7119192574437612
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/adapter_j14.npz b/benchmark/results/adapter_j14.npz
deleted file mode 100644
index 82c864c..0000000
Binary files a/benchmark/results/adapter_j14.npz and /dev/null differ
diff --git a/benchmark/results/adapter_smpl24_teacher.npz b/benchmark/results/adapter_smpl24_teacher.npz
deleted file mode 100644
index 84b2041..0000000
Binary files a/benchmark/results/adapter_smpl24_teacher.npz and /dev/null differ
diff --git a/benchmark/results/coco_gen5_bno_s0.json b/benchmark/results/coco_gen5_bno_s0.json
deleted file mode 100644
index 1a659f9..0000000
--- a/benchmark/results/coco_gen5_bno_s0.json
+++ /dev/null
@@ -1,112 +0,0 @@
-{
- "model": "models/gen5/bno_s0.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.6670050119307734,
- "[email protected]_bboxdiag": 0.7338855090522612,
- "[email protected]_bboxdiag": 0.8519240636223705,
- "[email protected]_bboxdiag": 0.9132448874880892,
- "NME_bboxdiag": 0.06468322584629825,
- "oks_ap": {
- "AP": 0.39577907358699205,
- "AP50": 0.6323345348337878,
- "AP75": 0.43885406127788734,
- "AP_medium": 0.370084407129405,
- "AP_large": 0.4431423548172553,
- "AR": 0.5558721662468514,
- "AR50": 0.7618073047858942,
- "AR75": 0.621536523929471,
- "AR_medium": 0.5005462988254575,
- "AR_large": 0.6334448160535117
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.813730084348641,
- "[email protected]": 0.8638706654170571
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.8036726978125844,
- "[email protected]": 0.8552524979746152
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.8108548168249661,
- "[email protected]": 0.8583446404341927
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.8047095761381475,
- "[email protected]": 0.8631083202511773
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.8101965601965602,
- "[email protected]": 0.8645577395577395
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.742592255953463,
- "[email protected]": 0.8427558625704418
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.7481346678798908,
- "[email protected]": 0.8504094631483167
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.7656826568265682,
- "[email protected]": 0.8646217712177122
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.7578803106441298,
- "[email protected]": 0.8524440383736867
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.7683143219264892,
- "[email protected]": 0.8585551330798479
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.7774177467597209,
- "[email protected]": 0.861914257228315
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.5186119873817034,
- "[email protected]": 0.8088328075709779
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.5104515050167224,
- "[email protected]": 0.8068561872909699
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.7492171932820951,
- "[email protected]": 0.8682038144036436
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.7365714285714285,
- "[email protected]": 0.8682857142857143
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.7314846127832263,
- "[email protected]": 0.8674332093337842
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.7338545207341944,
- "[email protected]": 0.8626784500339905
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_gen5_bno_s1.json b/benchmark/results/coco_gen5_bno_s1.json
deleted file mode 100644
index e58547e..0000000
--- a/benchmark/results/coco_gen5_bno_s1.json
+++ /dev/null
@@ -1,112 +0,0 @@
-{
- "model": "models/gen5/bno_s1.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.6682343657306236,
- "[email protected]_bboxdiag": 0.7370666275745804,
- "[email protected]_bboxdiag": 0.8534046763908232,
- "[email protected]_bboxdiag": 0.9128783991790662,
- "NME_bboxdiag": 0.06547462989899253,
- "oks_ap": {
- "AP": 0.39293350706049857,
- "AP50": 0.6344218780922184,
- "AP75": 0.425842275446914,
- "AP_medium": 0.3713137366196817,
- "AP_large": 0.4360823819559765,
- "AR": 0.552912468513854,
- "AR50": 0.7662153652392947,
- "AR75": 0.6199622166246851,
- "AR_medium": 0.49964490576345266,
- "AR_large": 0.6277220364176885
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.8310684161199625,
- "[email protected]": 0.873008434864105
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.820955981636511,
- "[email protected]": 0.8606535241695922
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.8260515603799186,
- "[email protected]": 0.8635006784260516
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.8150706436420723,
- "[email protected]": 0.8690737833594977
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.8215601965601965,
- "[email protected]": 0.870085995085995
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.7453190329031085,
- "[email protected]": 0.8480276313397564
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.7534121929026387,
- "[email protected]": 0.8549590536851683
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.7615313653136532,
- "[email protected]": 0.8609317343173432
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.7615349474645957,
- "[email protected]": 0.8547281863864779
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.7723700887198987,
- "[email protected]": 0.8577946768060837
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.7851445663010967,
- "[email protected]": 0.8576769690927218
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.5091482649842272,
- "[email protected]": 0.8069400630914827
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.509824414715719,
- "[email protected]": 0.8043478260869565
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.7338457159123256,
- "[email protected]": 0.8642186165670367
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.7397142857142858,
- "[email protected]": 0.8631428571428571
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.7267500845451471,
- "[email protected]": 0.8660804869800474
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.7233174711080897,
- "[email protected]": 0.8677770224337186
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_gen5_dt_s0.json b/benchmark/results/coco_gen5_dt_s0.json
deleted file mode 100644
index 558e3ff..0000000
--- a/benchmark/results/coco_gen5_dt_s0.json
+++ /dev/null
@@ -1,112 +0,0 @@
-{
- "model": "models/gen5/dt_s0.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.08904053315167523,
- "[email protected]_bboxdiag": 0.031092868137506415,
- "[email protected]_bboxdiag": 0.11507732903320383,
- "[email protected]_bboxdiag": 0.35413032324268856,
- "NME_bboxdiag": 0.26528959799858415,
- "oks_ap": {
- "AP": 5.461410363272982e-06,
- "AP50": 9.687857239735715e-06,
- "AP75": 3.890369390573635e-06,
- "AP_medium": 0.000760422094841063,
- "AP_large": 0.0029702970297029695,
- "AR": 0.0002676322418136021,
- "AR50": 0.0006297229219143577,
- "AR75": 0.00015743073047858942,
- "AR_medium": 0.00038240917782026763,
- "AR_large": 0.00011148272017837236
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.012652296157450796,
- "[email protected]": 0.039831302717900655
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.00594112881447475,
- "[email protected]": 0.027275182284634082
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.007598371777476255,
- "[email protected]": 0.03717774762550882
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.009733124018838305,
- "[email protected]": 0.024803767660910518
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.004914004914004914,
- "[email protected]": 0.032862407862407864
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.010179967278676604,
- "[email protected]": 0.047991274313761136
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.012556869881710645,
- "[email protected]": 0.057870791628753415
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.03505535055350553,
- "[email protected]": 0.14391143911439114
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.028780264961169484,
- "[email protected]": 0.13453631795340337
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.07604562737642585,
- "[email protected]": 0.2948035487959442
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.0992023928215354,
- "[email protected]": 0.31331006979062814
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.09127234490010515,
- "[email protected]": 0.29989484752891693
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.07901337792642141,
- "[email protected]": 0.27738294314381273
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.006831767719897523,
- "[email protected]": 0.0389980074010817
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.008,
- "[email protected]": 0.037142857142857144
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.0010145417653026716,
- "[email protected]": 0.004058167061210686
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.0006798096532970768,
- "[email protected]": 0.0030591434398368456
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_gen5_dt_s1.json b/benchmark/results/coco_gen5_dt_s1.json
deleted file mode 100644
index f48e3c7..0000000
--- a/benchmark/results/coco_gen5_dt_s1.json
+++ /dev/null
@@ -1,112 +0,0 @@
-{
- "model": "models/gen5/dt_s1.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.09038467977205145,
- "[email protected]_bboxdiag": 0.031400718317085685,
- "[email protected]_bboxdiag": 0.117364216081507,
- "[email protected]_bboxdiag": 0.35738473942681226,
- "NME_bboxdiag": 0.2647629929200928,
- "oks_ap": {
- "AP": 9.943598533839752e-06,
- "AP50": 1.776493438817566e-05,
- "AP75": 1.3218945392536584e-05,
- "AP_medium": 0.006930693069306931,
- "AP_large": 0.00792079207920792,
- "AR": 0.0004722921914357682,
- "AR50": 0.0012594458438287153,
- "AR75": 0.00047229219143576825,
- "AR_medium": 0.00040972411909314386,
- "AR_large": 0.0005574136008918618
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.01007497656982193,
- "[email protected]": 0.041002811621368325
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.003780718336483932,
- "[email protected]": 0.018633540372670808
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.0067842605156037995,
- "[email protected]": 0.032021709633649934
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.005965463108320251,
- "[email protected]": 0.029827315541601257
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.009520884520884522,
- "[email protected]": 0.03163390663390663
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.011997818578440284,
- "[email protected]": 0.05199054717324123
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.01019108280254777,
- "[email protected]": 0.050773430391264786
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.027675276752767528,
- "[email protected]": 0.1113929889298893
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.03060758337140247,
- "[email protected]": 0.13613522156235724
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.0788339670468948,
- "[email protected]": 0.30164765525982257
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.09621136590229312,
- "[email protected]": 0.3200398803589232
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.09695057833859096,
- "[email protected]": 0.32723449001051524
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.08486622073578595,
- "[email protected]": 0.29891304347826086
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.010816965556504412,
- "[email protected]": 0.051522914887560486
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.0074285714285714285,
- "[email protected]": 0.039142857142857146
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.0010145417653026716,
- "[email protected]": 0.0037199864727764627
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.0010197144799456153,
- "[email protected]": 0.004418762746430999
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_gen8_lr1x_e600.json b/benchmark/results/coco_gen8_lr1x_e600.json
deleted file mode 100644
index f2b8c6e..0000000
--- a/benchmark/results/coco_gen8_lr1x_e600.json
+++ /dev/null
@@ -1,118 +0,0 @@
-{
- "model": "models/gen8/g8h_lr1x_e600.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.6848534474258408,
- "[email protected]_bboxdiag": 0.7603899435608004,
- "[email protected]_bboxdiag": 0.867507146522026,
- "[email protected]_bboxdiag": 0.9307043905299421,
- "NME_bboxdiag": 0.054894041614812886,
- "hmr2_protocol": {
- "num_samples": 4648,
- "num_listed": 4648,
- "[email protected]": 0.7818773042797575,
- "[email protected]": 0.8911537817560902
- },
- "oks_ap": {
- "AP": 0.4102344555600176,
- "AP50": 0.6329140564493448,
- "AP75": 0.43868074453405653,
- "AP_medium": 0.38315629813148766,
- "AP_large": 0.47727727040946333,
- "AR": 0.5763539042821159,
- "AR50": 0.7699937027707808,
- "AR75": 0.6275188916876574,
- "AR_medium": 0.5205408358372029,
- "AR_large": 0.6554069119286512
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.8380974695407685,
- "[email protected]": 0.8959700093720713
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.8431001890359168,
- "[email protected]": 0.8979206049149339
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.8458616010854817,
- "[email protected]": 0.8966078697421981
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.8511773940345368,
- "[email protected]": 0.9029827315541601
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.8464373464373465,
- "[email protected]": 0.8974201474201474
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.7671332485002726,
- "[email protected]": 0.8725686238865661
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.7737943585077343,
- "[email protected]": 0.8682438580527753
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.7774446494464945,
- "[email protected]": 0.8666974169741697
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.7699862951119233,
- "[email protected]": 0.8565555047967108
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.7743979721166033,
- "[email protected]": 0.8555133079847909
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.7766699900299102,
- "[email protected]": 0.8579262213359921
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.5655099894847528,
- "[email protected]": 0.8199789695057834
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.5553929765886287,
- "[email protected]": 0.8158444816053512
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.7637346996868773,
- "[email protected]": 0.8682038144036436
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.7654285714285715,
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.7605681433885695,
- "[email protected]": 0.8677713899222185
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.7630863358259687,
- "[email protected]": 0.8725356900067981
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_gen8_lr2x_e600.json b/benchmark/results/coco_gen8_lr2x_e600.json
deleted file mode 100644
index dee0f16..0000000
--- a/benchmark/results/coco_gen8_lr2x_e600.json
+++ /dev/null
@@ -1,118 +0,0 @@
-{
- "model": "models/gen8/g8h_lr2x_e600.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.6881792023605318,
- "[email protected]_bboxdiag": 0.7619145349263359,
- "[email protected]_bboxdiag": 0.8704243934618485,
- "[email protected]_bboxdiag": 0.9316865791981236,
- "NME_bboxdiag": 0.05416568979720726,
- "hmr2_protocol": {
- "num_samples": 4648,
- "num_listed": 4648,
- "[email protected]": 0.7818118002235858,
- "[email protected]": 0.8915644948488242
- },
- "oks_ap": {
- "AP": 0.41533212120286433,
- "AP50": 0.6408538772904333,
- "AP75": 0.44903830115901155,
- "AP_medium": 0.3894469467410182,
- "AP_large": 0.4746945211689104,
- "AR": 0.5785107052896725,
- "AR50": 0.7701511335012594,
- "AR75": 0.6306675062972292,
- "AR_medium": 0.5233269598470364,
- "AR_large": 0.656558900037161
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.8357544517338332,
- "[email protected]": 0.8947985004686035
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.8414798811774237,
- "[email protected]": 0.8946799891979476
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.8469470827679783,
- "[email protected]": 0.8974219810040706
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.847723704866562,
- "[email protected]": 0.9007849293563579
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.8442874692874693,
- "[email protected]": 0.9007985257985258
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.7771314306489729,
- "[email protected]": 0.8760225413561171
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.7712465878070973,
- "[email protected]": 0.8687898089171975
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.7629151291512916,
- "[email protected]": 0.8662361623616236
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.7672453174965738,
- "[email protected]": 0.8558702603928735
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.7703422053231939,
- "[email protected]": 0.8565272496831432
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.7848953140578265,
- "[email protected]": 0.861914257228315
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.5711882229232387,
- "[email protected]": 0.8269190325972661
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.5650083612040134,
- "[email protected]": 0.8229515050167224
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.7651579846285226,
- "[email protected]": 0.8704810703102761
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.7708571428571429,
- "[email protected]": 0.8811428571428571
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.7724044639837674,
- "[email protected]": 0.880622252282719
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.769544527532291,
- "[email protected]": 0.8793337865397689
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/coco_val2017.json b/benchmark/results/coco_val2017.json
deleted file mode 100644
index 6472b0d..0000000
--- a/benchmark/results/coco_val2017.json
+++ /dev/null
@@ -1,94 +0,0 @@
-{
- "model": "models/instanthmr_mhr_only_ckpt90.onnx",
- "dataset": "COCO val2017",
- "protocol": "top-down, ground-truth boxes, score=1.0",
- "summary": {
- "num_samples": 6352,
- "mean_OKS": 0.715087947740202,
- "[email protected]_bboxdiag": 0.8967235945173349,
- "[email protected]_bboxdiag": 0.9517847980649418,
- "NME_bboxdiag": 0.04808343393139624,
- "oks_ap": {
- "AP": 0.44039902588970287,
- "AP50": 0.697060689022555,
- "AP75": 0.47301161863312285,
- "AP_medium": 0.43565038188009775,
- "AP_large": 0.46188555986388313,
- "AR": 0.5974968513853904,
- "AR50": 0.8148614609571788,
- "AR75": 0.6572732997481109,
- "AR_medium": 0.5542201584266594,
- "AR_large": 0.6594574507617986
- }
- },
- "per_keypoint": {
- "nose": {
- "n": 4268,
- "[email protected]": 0.9395501405810684
- },
- "left_eye": {
- "n": 3703,
- "[email protected]": 0.9343775317310289
- },
- "right_eye": {
- "n": 3685,
- "[email protected]": 0.9383989145183175
- },
- "left_ear": {
- "n": 3185,
- "[email protected]": 0.930298273155416
- },
- "right_ear": {
- "n": 3256,
- "[email protected]": 0.9302825552825553
- },
- "left_shoulder": {
- "n": 5501,
- "[email protected]": 0.9014724595528085
- },
- "right_shoulder": {
- "n": 5495,
- "[email protected]": 0.902820746132848
- },
- "left_elbow": {
- "n": 4336,
- "[email protected]": 0.8996771217712177
- },
- "right_elbow": {
- "n": 4378,
- "[email protected]": 0.901553220648698
- },
- "left_wrist": {
- "n": 3945,
- "[email protected]": 0.8869455006337136
- },
- "right_wrist": {
- "n": 4012,
- "[email protected]": 0.8898305084745762
- },
- "left_hip": {
- "n": 4755,
- "[email protected]": 0.8441640378548896
- },
- "right_hip": {
- "n": 4784,
- "[email protected]": 0.844690635451505
- },
- "left_knee": {
- "n": 3513,
- "[email protected]": 0.8778821520068317
- },
- "right_knee": {
- "n": 3500,
- "[email protected]": 0.8805714285714286
- },
- "left_ankle": {
- "n": 2957,
- "[email protected]": 0.8782549881636794
- },
- "right_ankle": {
- "n": 2942,
- "[email protected]": 0.8820530251529571
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/emdb1_alladapter.json b/benchmark/results/emdb1_alladapter.json
deleted file mode 100644
index d275942..0000000
--- a/benchmark/results/emdb1_alladapter.json
+++ /dev/null
@@ -1,344 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 137,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 79.89899497418331,
- "PA_MPJPE_mm": 53.81798394620607,
- "PCK3D_50mm": 0.6322069175344701,
- "PCK3D_100mm": 0.900053935194789,
- "AUC_0_150mm": 0.6589641163661865,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.38875643902142,
- "left_hip": 38.58502567465045,
- "right_hip": 38.64374049286347,
- "spine1": 43.75229274731907,
- "left_knee": 54.25041422518342,
- "right_knee": 54.1425459887338,
- "spine2": 43.637535615918374,
- "left_ankle": 61.87196282700251,
- "right_ankle": 62.62819743536405,
- "spine3": 41.83282129953466,
- "left_foot": 69.52197418470328,
- "right_foot": 70.2120609591868,
- "neck": 34.26217727492899,
- "left_collar": 29.577013017049016,
- "right_collar": 30.66917362317821,
- "head": 39.570892402613865,
- "left_shoulder": 39.64857745770012,
- "right_shoulder": 41.82903043998137,
- "left_elbow": 53.31573380982405,
- "right_elbow": 50.12875326034317,
- "left_wrist": 81.75573306282612,
- "right_wrist": 79.08061992194783,
- "left_hand": 96.88062749117694,
- "right_hand": 98.44595505789538
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 90.72479809063141,
- "PA_MPJPE_mm": 60.16621895590192,
- "PCK3D_50mm": 0.5267808986433224,
- "PCK3D_100mm": 0.8703866145885811,
- "AUC_0_150mm": 0.6129356328093468,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.84055062203499,
- "right_knee": 52.78757477579679,
- "right_hip": 49.741148343188165,
- "left_hip": 47.454675146924686,
- "left_knee": 53.824850575924025,
- "left_ankle": 65.37445015384029,
- "right_wrist": 80.73570274834391,
- "right_elbow": 45.04158922948974,
- "right_shoulder": 55.16986767345743,
- "left_shoulder": 43.801427220783474,
- "left_elbow": 50.0949029673105,
- "left_wrist": 86.1400763275341,
- "neck": 44.892623032925655,
- "head": 103.42762656507293
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.68860096199708,
- "PA_MPJPE_mm": 55.54807307469167,
- "PCK3D_50mm": 0.5913268057918102,
- "PCK3D_100mm": 0.899746227993749,
- "AUC_0_150mm": 0.644360403983085,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.701581175567185,
- "right_knee": 53.355717753595975,
- "right_hip": 47.103644910914305,
- "left_hip": 44.961480475975875,
- "left_knee": 54.17709666329682,
- "left_ankle": 62.46243257507469,
- "right_wrist": 77.97302519602071,
- "right_elbow": 42.233270361387255,
- "right_shoulder": 48.56697075655756,
- "left_shoulder": 42.793863660337045,
- "left_elbow": 48.05990297063805,
- "left_wrist": 83.18789039693452
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.44573927833153
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.07407797193647
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.7116134427868
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 54.05099939228138
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.68807307972911
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 75.35818170729262
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.274601114210945
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.39621370386727
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.46998460391101
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.54244714508236
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 51.04704954722545
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.79030469061686
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.45811541891769
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 54.80319982067974
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 75.03112152950358
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 64.85139366363609
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.71147187108503
- }
- }
- },
- "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 85,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 82.16065099955965,
- "PA_MPJPE_mm": 55.825338706859185,
- "PCK3D_50mm": 0.6148646779792281,
- "PCK3D_100mm": 0.8940138848552739,
- "AUC_0_150mm": 0.6506397387734272,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.70396733646249,
- "left_hip": 41.072286545463925,
- "right_hip": 40.58765229043174,
- "spine1": 46.46333685183644,
- "left_knee": 58.32141425285344,
- "right_knee": 56.989772761846005,
- "spine2": 43.355409456514096,
- "left_ankle": 66.60249721625911,
- "right_ankle": 66.47063521157392,
- "spine3": 40.91482345371773,
- "left_foot": 72.94529885905705,
- "right_foot": 72.4786906097902,
- "neck": 37.233127996944575,
- "left_collar": 31.747304809215237,
- "right_collar": 31.876652034764007,
- "head": 43.86986225417686,
- "left_shoulder": 42.42372187894316,
- "right_shoulder": 42.3622200550824,
- "left_elbow": 55.729122225866355,
- "right_elbow": 51.488487100984884,
- "left_wrist": 83.37699882632741,
- "right_wrist": 78.6395288964919,
- "left_hand": 97.93653692171387,
- "right_hand": 97.21878111830387
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 92.61108651904796,
- "PA_MPJPE_mm": 61.927314981267074,
- "PCK3D_50mm": 0.5202968213796741,
- "PCK3D_100mm": 0.8618014355059536,
- "AUC_0_150mm": 0.6073539806410698,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.57235357611404,
- "right_knee": 54.33595960450975,
- "right_hip": 50.526315138235056,
- "left_hip": 47.84138836982132,
- "left_knee": 56.92537511368124,
- "left_ankle": 70.61200686646174,
- "right_wrist": 80.12905500727975,
- "right_elbow": 46.838162649020745,
- "right_shoulder": 54.52294203786921,
- "left_shoulder": 44.46141139571641,
- "left_elbow": 52.90357655334994,
- "left_wrist": 87.68342051742559,
- "neck": 45.56855188312777,
- "head": 106.06189102512643
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 87.29156555160438,
- "PA_MPJPE_mm": 57.31832852318586,
- "PCK3D_50mm": 0.5813384226029955,
- "PCK3D_100mm": 0.8932982063090348,
- "AUC_0_150mm": 0.6388870300801355,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.6110898068444,
- "right_knee": 54.38292599948846,
- "right_hip": 47.6902899756637,
- "left_hip": 45.7720338767824,
- "left_knee": 57.13106223561326,
- "left_ankle": 67.23894697013618,
- "right_wrist": 77.86331140249156,
- "right_elbow": 44.199689767243605,
- "right_shoulder": 49.40597095256337,
- "left_shoulder": 45.15945622082252,
- "left_elbow": 49.80201024945468,
- "left_wrist": 83.5631548211261
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.98543497401005
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 52.091088721749536
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.583596666258245
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 55.92338370551133
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 51.08274846782155
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 78.11362474445052
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.839872627535655
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.59408499334506
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 51.93820655996959
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 45.12661562019302
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 51.956708404866745
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.03109592282591
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.752346278928854
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 52.18738325398466
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 87.26453095273783
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 62.152433163627016
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 52.703730588090174
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/emdb1_b3_s1_annotated.json b/benchmark/results/emdb1_b3_s1_annotated.json
deleted file mode 100644
index 108e590..0000000
--- a/benchmark/results/emdb1_b3_s1_annotated.json
+++ /dev/null
@@ -1,173 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 137,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 87.5145198705048,
- "PA_MPJPE_mm": 60.68695037973602,
- "PCK3D_50mm": 0.5824395994966048,
- "PCK3D_100mm": 0.8630167060808475,
- "AUC_0_150mm": 0.6278615988997042,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.86098621792327,
- "left_hip": 41.79013147929267,
- "right_hip": 41.16490895868687,
- "spine1": 48.40987721094612,
- "left_knee": 58.16313551683889,
- "right_knee": 57.22590059890525,
- "spine2": 46.81683316737049,
- "left_ankle": 69.84579402852766,
- "right_ankle": 69.70267345558224,
- "spine3": 45.18799735489693,
- "left_foot": 76.68425821217114,
- "right_foot": 77.31345782617193,
- "neck": 42.938995983362986,
- "left_collar": 37.18888642853474,
- "right_collar": 37.55237455507592,
- "head": 48.43977502391476,
- "left_shoulder": 43.78942526296338,
- "right_shoulder": 45.138807870567796,
- "left_elbow": 58.9530251328787,
- "right_elbow": 55.218168805087,
- "left_wrist": 94.77740166022281,
- "right_wrist": 92.7091943098265,
- "left_hand": 112.59585738228681,
- "right_hand": 114.01894267162925
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 98.12348511185374,
- "PA_MPJPE_mm": 65.5655235547273,
- "PCK3D_50mm": 0.48783790992229775,
- "PCK3D_100mm": 0.8378150911860408,
- "AUC_0_150mm": 0.5875892459225012,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.26933843183772,
- "right_knee": 54.829084865597956,
- "right_hip": 51.412027756054854,
- "left_hip": 48.73087643103119,
- "left_knee": 56.7033084565791,
- "left_ankle": 72.41752237552517,
- "right_wrist": 94.73684429306688,
- "right_elbow": 51.668938476699914,
- "right_shoulder": 56.772514583590144,
- "left_shoulder": 46.49234777994068,
- "left_elbow": 56.461653215982516,
- "left_wrist": 99.10996551016746,
- "neck": 53.15409908635089,
- "head": 105.15880850375798
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 93.66851373450805,
- "PA_MPJPE_mm": 61.76109450753499,
- "PCK3D_50mm": 0.5403269302576442,
- "PCK3D_100mm": 0.8634125765810619,
- "AUC_0_150mm": 0.6135420611988245,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.49097778731077,
- "right_knee": 55.341688859316235,
- "right_hip": 50.346738035900245,
- "left_hip": 47.99181424877826,
- "left_knee": 57.16501452499131,
- "left_ankle": 70.0484272084777,
- "right_wrist": 90.96316636646812,
- "right_elbow": 48.39876355325566,
- "right_shoulder": 54.60786812013797,
- "left_shoulder": 49.63662210397929,
- "left_elbow": 53.613263181921,
- "left_wrist": 94.52879009988325
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 63.95078685745811
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 62.80989428662845
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 54.788964020413744
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 55.025884444805776
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 52.066525759678704
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 84.33874324494832
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 57.08465013107306
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 67.55319596038812
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 49.19546174141772
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 54.248099071883765
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 55.74420089059678
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 58.74884027499412
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 51.620113245460516
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 61.38240839177174
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 84.0380008365223
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 66.56964635175764
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.1854288712578
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/emdb1_b3_s1_gt-joints.json b/benchmark/results/emdb1_b3_s1_gt-joints.json
deleted file mode 100644
index 7b447ac..0000000
--- a/benchmark/results/emdb1_b3_s1_gt-joints.json
+++ /dev/null
@@ -1,173 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 137,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 79.89899497418331,
- "PA_MPJPE_mm": 53.81798394620607,
- "PCK3D_50mm": 0.6322069175344701,
- "PCK3D_100mm": 0.900053935194789,
- "AUC_0_150mm": 0.6589641163661865,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.38875643902142,
- "left_hip": 38.58502567465045,
- "right_hip": 38.64374049286347,
- "spine1": 43.75229274731907,
- "left_knee": 54.25041422518342,
- "right_knee": 54.1425459887338,
- "spine2": 43.637535615918374,
- "left_ankle": 61.87196282700251,
- "right_ankle": 62.62819743536405,
- "spine3": 41.83282129953466,
- "left_foot": 69.52197418470328,
- "right_foot": 70.2120609591868,
- "neck": 34.26217727492899,
- "left_collar": 29.577013017049016,
- "right_collar": 30.66917362317821,
- "head": 39.570892402613865,
- "left_shoulder": 39.64857745770012,
- "right_shoulder": 41.82903043998137,
- "left_elbow": 53.31573380982405,
- "right_elbow": 50.12875326034317,
- "left_wrist": 81.75573306282612,
- "right_wrist": 79.08061992194783,
- "left_hand": 96.88062749117694,
- "right_hand": 98.44595505789538
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 90.72479809063141,
- "PA_MPJPE_mm": 60.16621895590192,
- "PCK3D_50mm": 0.5267808986433224,
- "PCK3D_100mm": 0.8703866145885811,
- "AUC_0_150mm": 0.6129356328093468,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.84055062203499,
- "right_knee": 52.78757477579679,
- "right_hip": 49.741148343188165,
- "left_hip": 47.454675146924686,
- "left_knee": 53.824850575924025,
- "left_ankle": 65.37445015384029,
- "right_wrist": 80.73570274834391,
- "right_elbow": 45.04158922948974,
- "right_shoulder": 55.16986767345743,
- "left_shoulder": 43.801427220783474,
- "left_elbow": 50.0949029673105,
- "left_wrist": 86.1400763275341,
- "neck": 44.892623032925655,
- "head": 103.42762656507293
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.68860096199708,
- "PA_MPJPE_mm": 55.54807307469167,
- "PCK3D_50mm": 0.5913268057918102,
- "PCK3D_100mm": 0.899746227993749,
- "AUC_0_150mm": 0.644360403983085,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.701581175567185,
- "right_knee": 53.355717753595975,
- "right_hip": 47.103644910914305,
- "left_hip": 44.961480475975875,
- "left_knee": 54.17709666329682,
- "left_ankle": 62.46243257507469,
- "right_wrist": 77.97302519602071,
- "right_elbow": 42.233270361387255,
- "right_shoulder": 48.56697075655756,
- "left_shoulder": 42.793863660337045,
- "left_elbow": 48.05990297063805,
- "left_wrist": 83.18789039693452
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.44573927833153
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.07407797193647
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.7116134427868
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 54.05099939228138
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.68807307972911
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 75.35818170729262
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.274601114210945
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.39621370386727
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.46998460391101
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.54244714508236
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 51.04704954722545
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.79030469061686
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.45811541891769
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 54.80319982067974
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 75.03112152950358
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 64.85139366363609
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.71147187108503
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/emdb1_smplfit_gt-joints.json b/benchmark/results/emdb1_smplfit_gt-joints.json
deleted file mode 100644
index f22bee7..0000000
--- a/benchmark/results/emdb1_smplfit_gt-joints.json
+++ /dev/null
@@ -1,70 +0,0 @@
-{
- "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/b3_s1/b3_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 137,
- "source": "ema",
- "fit_residual_mm": 14.610766410827637,
- "MPJPE_mm": 77.38072967529297,
- "PA_MPJPE_mm": 51.999232991718365,
- "PVE_mm": 92.24835687335322,
- "PA_PVE_mm": 62.57454616195972,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 40.679812730232776,
- "P2_23_outdoor_hug_tree": 48.46140407281954,
- "P3_31_outdoor_workout": 53.641220567187126,
- "P3_32_outdoor_soccer_warmup_a": 53.625993557350874,
- "P3_33_outdoor_soccer_warmup_b": 48.168861427570775,
- "P5_42_indoor_dancing": 74.88580298687171,
- "P5_44_indoor_rom": 48.84046610706719,
- "P6_49_outdoor_big_stairs_down": 56.64582810483881,
- "P6_50_outdoor_workout": 44.77893859356306,
- "P6_51_outdoor_dancing": 43.987160466263575,
- "P7_57_outdoor_rock_chair": 48.498681297758864,
- "P7_59_outdoor_rom": 49.07154407413522,
- "P7_60_outdoor_workout": 43.90422666512502,
- "P8_64_outdoor_skateboard": 52.09684144342112,
- "P8_68_outdoor_handstand": 72.55139613251777,
- "P8_69_outdoor_cartwheel": 62.702449073220386,
- "P9_76_outdoor_sitting": 48.80991828197765
- }
- },
- "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/bno_s0/bno_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 85,
- "source": "ema",
- "fit_residual_mm": 14.715494155883789,
- "MPJPE_mm": 79.54940795898438,
- "PA_MPJPE_mm": 54.234037765862624,
- "PVE_mm": 94.44203375607358,
- "PA_PVE_mm": 64.8163185715288,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 40.38579015681633,
- "P2_23_outdoor_hug_tree": 50.952089101184306,
- "P3_31_outdoor_workout": 52.97866412872431,
- "P3_32_outdoor_soccer_warmup_a": 55.258014050152454,
- "P3_33_outdoor_soccer_warmup_b": 51.0758103503119,
- "P5_42_indoor_dancing": 77.7890516978954,
- "P5_44_indoor_rom": 53.694246453659666,
- "P6_49_outdoor_big_stairs_down": 59.71285853164975,
- "P6_50_outdoor_workout": 50.39122909218151,
- "P6_51_outdoor_dancing": 44.74699902324005,
- "P7_57_outdoor_rock_chair": 49.20744661036477,
- "P7_59_outdoor_rom": 51.584609243641225,
- "P7_60_outdoor_workout": 43.26185962015645,
- "P8_64_outdoor_skateboard": 50.56812695561018,
- "P8_68_outdoor_handstand": 87.12619774976028,
- "P8_69_outdoor_cartwheel": 59.58728879933646,
- "P9_76_outdoor_sitting": 47.7313144333539
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/family_latency_aihub.json b/benchmark/results/family_latency_aihub.json
deleted file mode 100644
index 4100378..0000000
--- a/benchmark/results/family_latency_aihub.json
+++ /dev/null
@@ -1,266 +0,0 @@
-[
- {
- "model": "g8h_trained_b4",
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jp0moor2g",
- "ms": 2.44,
- "peak_mb": 160.2,
- "layers": {
- "NPU": 918
- }
- },
- {
- "model": "g8h_trained_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jp8ejj7zp",
- "ms": 61.74,
- "peak_mb": 199.1,
- "layers": {
- "CPU": 791
- }
- },
- {
- "model": "g8h_trained_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jgk266yyg",
- "ms": 84.89,
- "peak_mb": 169.4,
- "layers": {
- "CPU": 791
- }
- },
- {
- "model": "g8h_trained_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "j5ql4427p",
- "ms": 187.85,
- "peak_mb": 177.8,
- "layers": {
- "CPU": 791
- }
- },
- {
- "model": "g8h_trained_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "jglywwke5",
- "ms": 195.18,
- "peak_mb": 176.4,
- "layers": {
- "CPU": 791
- }
- },
- {
- "model": "g8h_trained_b4",
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "j568oo1vg",
- "ms": 83.2,
- "peak_mb": 233.4,
- "layers": {
- "CPU": 791
- }
- },
- {
- "model": "random_b4",
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jp3zoomx5",
- "ms": 2.47,
- "peak_mb": 80.9,
- "layers": {
- "NPU": 890
- }
- },
- {
- "model": "random_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jgolddv4g",
- "ms": 60.04,
- "peak_mb": 149.3,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jpvl22w75",
- "ms": 81.72,
- "peak_mb": 131.9,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jgjr33l7p",
- "ms": 194.15,
- "peak_mb": 134.0,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4",
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "jpe766v75",
- "ms": 199.35,
- "peak_mb": 134.5,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4",
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jgzlzz7z5",
- "ms": 78.69,
- "peak_mb": 187.4,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b1",
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "j5wlyy9zp",
- "ms": 1.88,
- "peak_mb": 97.8,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b1",
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jg9zoo4qp",
- "ms": 26.94,
- "peak_mb": 136.7,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b1",
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jp1noo8kg",
- "ms": 36.88,
- "peak_mb": 123.1,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b1",
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jgdd66vkg",
- "ms": 77.58,
- "peak_mb": 129.8,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b1",
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "j57eoodqp",
- "ms": 76.84,
- "peak_mb": 122.1,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b1",
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jp4yeewqp",
- "ms": 33.22,
- "peak_mb": 171.4,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0",
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jpxl001jp",
- "ms": 1.6,
- "peak_mb": 81.8,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b0",
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "j5m099zyg",
- "ms": 23.86,
- "peak_mb": 134.3,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0",
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jgnz119vg",
- "ms": 32.68,
- "peak_mb": 120.4,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0",
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jprlxx4vp",
- "ms": 79.72,
- "peak_mb": 126.1,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0",
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "jp2roo7xg",
- "ms": 68.58,
- "peak_mb": 120.6,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0",
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jpyo884r5",
- "ms": 29.89,
- "peak_mb": 166.4,
- "layers": {
- "CPU": 722
- }
- }
-]
\ No newline at end of file
diff --git a/benchmark/results/family_latency_aihub_288.json b/benchmark/results/family_latency_aihub_288.json
deleted file mode 100644
index f9fe6ec..0000000
--- a/benchmark/results/family_latency_aihub_288.json
+++ /dev/null
@@ -1,218 +0,0 @@
-[
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S22 5G",
- "job": "jg9zoorwp",
- "ms": 3.7,
- "peak_mb": 150.2,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jp1noo98g",
- "ms": 2.05,
- "peak_mb": 90.1,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S24",
- "job": "jgdd66krg",
- "ms": 1.31,
- "peak_mb": 169.7,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S25",
- "job": "j57eoomvp",
- "ms": 1.0,
- "peak_mb": 167.3,
- "layers": {
- "NPU": 827
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jp4yee78p",
- "ms": 29.25,
- "peak_mb": 139.7,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jpxl00q3p",
- "ms": 38.2,
- "peak_mb": 125.0,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "j5m0997dg",
- "ms": 93.51,
- "peak_mb": 125.8,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "jgnz114kg",
- "ms": 78.97,
- "peak_mb": 182.6,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b0_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jprlxxr0p",
- "ms": 32.06,
- "peak_mb": 142.9,
- "layers": {
- "CPU": 722
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S22 5G",
- "job": "jp2roo1rg",
- "ms": 5.5,
- "peak_mb": 174.3,
- "layers": {
- "NPU": 890
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jpyo88l85",
- "ms": 3.27,
- "peak_mb": 159.2,
- "layers": {
- "NPU": 890
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S24",
- "job": "jp0moow9g",
- "ms": 2.31,
- "peak_mb": 232.4,
- "layers": {
- "NPU": 890
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "npu",
- "device": "Samsung Galaxy S25",
- "job": "jp8ejjnkp",
- "ms": 1.88,
- "peak_mb": 141.0,
- "layers": {
- "NPU": 890
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jgk2661wg",
- "ms": 93.32,
- "peak_mb": 173.5,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "j5ql44nnp",
- "ms": 119.02,
- "peak_mb": 160.3,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jglywwdj5",
- "ms": 288.74,
- "peak_mb": 146.3,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "j568oox6g",
- "ms": 294.21,
- "peak_mb": 147.4,
- "layers": {
- "CPU": 762
- }
- },
- {
- "model": "random_b4_288",
- "image_size": 288,
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jp3zood35",
- "ms": 115.04,
- "peak_mb": 198.5,
- "layers": {
- "CPU": 762
- }
- }
-]
\ No newline at end of file
diff --git a/benchmark/results/family_latency_aihub_384_448.json b/benchmark/results/family_latency_aihub_384_448.json
deleted file mode 100644
index 731e065..0000000
--- a/benchmark/results/family_latency_aihub_384_448.json
+++ /dev/null
@@ -1,194 +0,0 @@
-[
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jpv2692zg",
- "ms": 2.31,
- "peak_mb": 118.5,
- "layers": {
- "NPU": 809
- }
- },
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jgj3vw31p",
- "ms": 36.9,
- "peak_mb": 150.8,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jpe6dl68g",
- "ms": 50.2,
- "peak_mb": 135.9,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jgzz34z4g",
- "ms": 106.88,
- "peak_mb": 139.0,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "j5wye1y4g",
- "ms": 105.39,
- "peak_mb": 141.7,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "jg9olxomg",
- "ms": 45.76,
- "peak_mb": 194.2,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jp1o4von5",
- "ms": 4.18,
- "peak_mb": 119.8,
- "layers": {
- "NPU": 872
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jgd6xz66p",
- "ms": 146.33,
- "peak_mb": 196.7,
- "layers": {
- "CPU": 757
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "j57oy7ong",
- "ms": 186.47,
- "peak_mb": 160.4,
- "layers": {
- "CPU": 757
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "jp4el9e2g",
- "ms": 444.66,
- "peak_mb": 177.4,
- "layers": {
- "CPU": 757
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Samsung Galaxy A14 5G",
- "job": "jpx0kd08p",
- "ms": 453.94,
- "peak_mb": 175.4,
- "layers": {
- "CPU": 757
- }
- },
- {
- "model": "random_b4_384",
- "image_size": 384,
- "unit": "cpu",
- "device": "Google Pixel 8",
- "job": "j5m9nd97g",
- "ms": 157.76,
- "peak_mb": 186.7,
- "layers": {
- "CPU": 757
- }
- },
- {
- "model": "random_b0_448",
- "image_size": 448,
- "unit": "npu",
- "device": "Samsung Galaxy S23",
- "job": "jg9olxqqg",
- "ms": 2.81,
- "peak_mb": 119.6,
- "layers": {
- "NPU": 809
- }
- },
- {
- "model": "random_b0_448",
- "image_size": 448,
- "unit": "cpu",
- "device": "Samsung Galaxy S23",
- "job": "jp1o4vmk5",
- "ms": 41.9,
- "peak_mb": 163.1,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_448",
- "image_size": 448,
- "unit": "cpu",
- "device": "Samsung Galaxy A73 5G",
- "job": "jgd6xzmkp",
- "ms": 58.9,
- "peak_mb": 147.7,
- "layers": {
- "CPU": 717
- }
- },
- {
- "model": "random_b0_448",
- "image_size": 448,
- "unit": "cpu",
- "device": "Samsung Galaxy A53 5G",
- "job": "j57oy78qg",
- "ms": 130.25,
- "peak_mb": 147.4,
- "layers": {
- "CPU": 717
- }
- }
-]
\ No newline at end of file
diff --git a/benchmark/results/footlock_g11_2026-11-01.md b/benchmark/results/footlock_g11_2026-11-01.md
deleted file mode 100644
index e0ac863..0000000
--- a/benchmark/results/footlock_g11_2026-11-01.md
+++ /dev/null
@@ -1,106 +0,0 @@
-# Foot lock: generation-11 fine-tunes (2026-11-01)
-
-Native foot contact for InstantHMR, measured before the final evaluation. All
-models are B4 at 288 px. g11 arms are 120-epoch fine-tunes of g10m75_b4
-(epoch 474), LR 1e-4.
-
-| arm | previous-pose token | semelle data | notes |
-|---|---|---|---|
-| g10m75_b4 | no | no | baseline |
-| g11Ap_b4 | yes, `--prev-pairs-only --prev-mask-p 0.2` | yes, 4 participants, `--w-foot-mocap 0.35` | the paper model |
-| g11Cp_b4 | yes, same flags | no (contact from Harmony4D motion only) | token-only ablation |
-| g11A_b4 / g11C_b4 | yes, old recipe (stills fed their own GT as the previous pose) | A yes / C no | superseded; lag on video |
-
-Checkpoint: **`last.pth` (epoch 119)**. `best_student_model_v3.pth` is chosen on
-3DPW validation and lands at epoch 3-6 (the initialisation), blind to foot lock.
-
-## Protocol
-
-- **Foot lock** (`benchmark/eval_footlock.py`): 6 held-out sessions of 4
- participants never seen in training, same room and cameras as training,
- 156 trial-cameras, 166,663 foot pairs in insole contact. Frames every 5 video
- frames (125 ms at 40 fps), undistorted, calibrated focal.
- - *slide*: mean displacement of a foot (mean of its 4 keypoints, camera
- frame) between consecutive labelled frames while the insole says it is in
- contact in both. Mocap feet on the same pairs: 1.83 mm.
- - *foot error*: foot keypoints vs mocap markers after removing each trial's
- median offset.
- - *contact F1*: per-foot contact head (logit > 0) vs insole contact.
- - *masked*: previous pose withheld. *feedback*: the model's own prediction on
- the previous labelled frame (125 ms earlier).
-- **3DPW test**: every frame, GT-joint boxes, J14 PA-MPJPE with the adapter,
- H36M GT. **EMDB-1**: SMPL24 PA-MPJPE with the adapter. *feedback* = the
- model's own prediction 5 frames earlier in the same track.
-- *jitter*: mean second difference of pelvis-centred joints over 3 consecutive
- frames (mm/frame^2).
-
-## Foot lock (held-out sessions)
-
-| model | prev pose | slide mm (median) | foot err mm | contact F1 |
-|---|---|---|---|---|
-| mocap | - | 1.83 | - | - |
-| g10m75_b4 | - | 24.37 (16.90) | 50.46 | - |
-| g11A ep4 (old) | masked / feedback | 20.20 / 20.20 | 44.63 / 44.72 | 0.971 |
-| g11C ep51 (old) | masked / feedback | 24.28 / 23.49 | 50.03 / 49.57 | 0.851 / 0.844 |
-| g11Ap ep6 | masked / feedback | 15.89 / 15.86 | 34.36 / 34.76 | 0.977 |
-| **g11Ap ep119** | masked | 13.65 (9.66) | 30.68 | 0.981 |
-| **g11Ap ep119** | **feedback** | **11.07 (7.66)** | **30.41** | **0.981** |
-| g11Cp ep3 | masked / feedback | 23.08 / 23.08 | 50.04 / 50.04 | 0.734 |
-| g11Cp ep119 | masked / feedback | 24.49 / 22.08 | 48.98 / **62.66** | 0.802 / 0.665 |
-
-## 3DPW test and EMDB-1
-
-| model | prev pose | 3DPW J14 PA | 3DPW jitter | EMDB PA | EMDB jitter |
-|---|---|---|---|---|---|
-| g10m75_b4 | - | 39.64 | 21.03 | 48.75 | 25.10 |
-| g11Ap ep119 | masked | 39.67 | 20.86 | 49.20 | 24.40 |
-| g11Ap ep119 | feedback | 40.01 | **20.14** | 49.33 | **23.72** |
-| g11Cp ep119 | masked | 39.81 | 21.28 | 49.42 | 25.14 |
-| g11Cp ep119 | feedback | 41.54 | 20.68 | 51.77 | 24.79 |
-| g11C ep51 (old) | masked / feedback lag 1 / lag 5 | 39.85 / 41.22 / 43.58 | 20.8 / 16.9 / 23.1 | - | - |
-
-## Post-hoc foot lock (`eval_footlock.py --lock`)
-
-While a foot is called planted, its keypoints are held at the running average of
-its predicted position since contact onset (causal); released, the raw
-prediction returns. *lock* = Ap's own contact head, *oracle* = insole contact.
-Slide is near zero by construction; foot error is the test. *Release jump* is
-the foot's displacement on the release pair and includes genuine lift-off
-motion (not yet compared with the raw prediction on the same pairs).
-
-| model | variant | slide mm | foot err mm | release jump mm |
-|---|---|---|---|---|
-| g11Ap ep119 feedback | none | 11.07 | 30.41 | - |
-| g11Ap ep119 feedback | lock | 1.73 | 27.26 | 126.9 |
-| g11Ap ep119 feedback | oracle | 1.63 | 25.58 | 98.0 |
-| g11Ap ep119 masked | lock / oracle | 1.81 / 1.70 | 26.52 / 25.05 | 134.1 / 105.1 |
-| g10m75_b4 | oracle | 2.75 | 35.62 | 118.3 |
-
-## Contact detection on jumps (held-out, g11Ap ep119)
-
-22 jump trial-cameras; frames where the insoles give a valid label for both feet.
-
-| insole says | frames | model: 0 planted | 1 planted | 2 planted |
-|---|---|---|---|---|
-| both feet off (flight) | 500 | 83.2% | 15.0% | 1.8% |
-| one foot off | 274 | 3.3% | 68.2% | 28.5% |
-| both feet on | 10,191 | 0.3% | 0.7% | 99.0% |
-
-Feedback; masked is within 2.5 points. On flight frames where exactly one foot
-stays "planted", it is the left foot 70 times and the right 5 times.
-
-## Demo path
-
-`demo.py` with `models/instanthmr_g11Ap_b4_ep119.onnx` (own RF-DETR boxes, no
-undistortion, previous pose fed back from 5 frames earlier), one held-out
-walking trial, 532 insole-labelled frames: 93.5% agreement, F1 0.956 (withheld:
-93.0%, 0.953). 43 fps end to end on an RTX 4070 (InstantHMR 7.0 ms).
-
-## Result files
-
-`benchmark/results/`: `g10m75_b4_footlock_held6.json`,
-`g11{A_b4_ep4,C_b4_ep51,Ap_b4_ep6,Ap_b4_ep119,Cp_b4_ep3,Cp_b4_ep119}_footlock_held6.json`,
-`g11Ap_b4_ep119_footlock_held6_lock.json`, `g10m75_b4_footlock_held6_lock.json`,
-`g11{Ap,Cp}_b4_ep119_{3dpw_test,emdb1}_{masked,feedback}_lag5.json`,
-`g10m75_b4_{3dpw_test,emdb1}_accel.json`, `g11C_b4_ep51_3dpw_test_*.json`.
-The 2-session files (`*_footlock.json` without `held6`) are superseded.
diff --git a/benchmark/results/g10m75_b4_3dpw_test_accel.json b/benchmark/results/g10m75_b4_3dpw_test_accel.json
deleted file mode 100644
index 809ba11..0000000
--- a/benchmark/results/g10m75_b4_3dpw_test_accel.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 474,
- "source": "raw",
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_J14_adapter": {
- "accel_mm": 21.027141982274202,
- "accel_err_mm": 21.63795300187405,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 101.24964422138368,
- "PA_MPJPE_mm": 65.83705658970332,
- "PCK3D_50mm": 0.41220628341007326,
- "PCK3D_100mm": 0.8368561196579131,
- "AUC_0_150mm": 0.5667884395899874,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.570666559021504,
- "right_knee": 46.98402739644765,
- "right_hip": 97.4498275649219,
- "left_hip": 86.71354442491283,
- "left_knee": 44.18913310451295,
- "left_ankle": 65.14210614963991,
- "right_wrist": 64.74940597357465,
- "right_elbow": 57.071413138748206,
- "right_shoulder": 49.37393670117943,
- "left_shoulder": 51.82139078103376,
- "left_elbow": 65.42587463375546,
- "left_wrist": 69.81207302713642,
- "neck": 32.03193640559233,
- "head": 127.38345639536948
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.69035901696749,
- "PA_MPJPE_mm": 60.559663082008555,
- "PCK3D_50mm": 0.4578551353993364,
- "PCK3D_100mm": 0.898107417120191,
- "AUC_0_150mm": 0.5991543814103995,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.61767775620856,
- "right_knee": 48.866588645180904,
- "right_hip": 96.73631708625226,
- "left_hip": 85.71009197102313,
- "left_knee": 45.173298959462514,
- "left_ankle": 60.27974930555247,
- "right_wrist": 62.15530850073175,
- "right_elbow": 49.06121862265806,
- "right_shoulder": 49.60822337244735,
- "left_shoulder": 49.779920886046426,
- "left_elbow": 53.21300810444542,
- "left_wrist": 66.51455377409394
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.14096052071149,
- "PA_MPJPE_mm": 39.636821484619226,
- "PCK3D_50mm": 0.7504562098928058,
- "PCK3D_100mm": 0.9565502878251376,
- "AUC_0_150mm": 0.733658732519426
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g10m75_b4_emdb1_accel.json b/benchmark/results/g10m75_b4_emdb1_accel.json
deleted file mode 100644
index 4e647c6..0000000
--- a/benchmark/results/g10m75_b4_emdb1_accel.json
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_SMPL24_adapter": {
- "accel_mm": 25.098613798819507,
- "accel_err_mm": 24.76626165412154,
- "n_triplets": 24061
- },
- "epoch": 474,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 74.13625349919556,
- "PA_MPJPE_mm": 48.75233528628786,
- "PCK3D_50mm": 0.6715588654247743,
- "PCK3D_100mm": 0.9266170186283865,
- "AUC_0_150mm": 0.6829983462550283,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.35479073274574,
- "left_hip": 38.00743222891997,
- "right_hip": 37.473826200937374,
- "spine1": 43.6497883966273,
- "left_knee": 49.28376291616478,
- "right_knee": 48.63197224873031,
- "spine2": 43.22088834724593,
- "left_ankle": 51.57758735762877,
- "right_ankle": 52.553530046466285,
- "spine3": 40.71130540772466,
- "left_foot": 58.37637674440748,
- "right_foot": 59.14379561003081,
- "neck": 32.44479197844708,
- "left_collar": 27.53405846523394,
- "right_collar": 27.989654198336545,
- "head": 37.73802718408648,
- "left_shoulder": 37.513489500124514,
- "right_shoulder": 38.92188648520798,
- "left_elbow": 49.26931596241498,
- "right_elbow": 45.13216981316247,
- "left_wrist": 72.63339476372882,
- "right_wrist": 69.00254815333818,
- "left_hand": 87.19331150825518,
- "right_hand": 85.6983426209447
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.42463667022884,
- "PA_MPJPE_mm": 55.500987276120306,
- "PCK3D_50mm": 0.570563237534154,
- "PCK3D_100mm": 0.8931579352896202,
- "AUC_0_150mm": 0.6361901906774518,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 55.30724020789289,
- "right_knee": 46.71144784085963,
- "right_hip": 49.91673748842551,
- "left_hip": 46.64445610911143,
- "left_knee": 49.356216249842866,
- "left_ankle": 55.05524372099114,
- "right_wrist": 70.71677049913313,
- "right_elbow": 40.62957460877809,
- "right_shoulder": 51.74420217045175,
- "left_shoulder": 41.662203716660066,
- "left_elbow": 45.45032270821481,
- "left_wrist": 77.12615813929534,
- "neck": 42.729726474116134,
- "head": 103.96352193191153
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.03615955942358,
- "PA_MPJPE_mm": 50.139690933442054,
- "PCK3D_50mm": 0.6388865839660347,
- "PCK3D_100mm": 0.9256558657981717,
- "AUC_0_150mm": 0.6713943608500972,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.16598150966339,
- "right_knee": 47.201932297594325,
- "right_hip": 46.59139924308438,
- "left_hip": 43.97131026586643,
- "left_knee": 49.941796229079884,
- "left_ankle": 52.014138972569434,
- "right_wrist": 69.2292533523314,
- "right_elbow": 38.73087235561699,
- "right_shoulder": 44.59638543513222,
- "left_shoulder": 39.5845355716731,
- "left_elbow": 43.31685725751665,
- "left_wrist": 74.33182871117651
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.276678816500755
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 47.77583256290774
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.348424127168315
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.499614675949566
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.172355784380024
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 61.5761361247485
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 43.58554539943247
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.983871461399346
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 44.20038435097333
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.334846079193106
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 43.5218270741477
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 45.609613431892974
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.59133544061146
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 48.21329933423208
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 61.43424534423505
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 51.709271793878465
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.80480235699461
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g10m75_b4_footlock.json b/benchmark/results/g10m75_b4_footlock.json
deleted file mode 100644
index 3595478..0000000
--- a/benchmark/results/g10m75_b4_footlock.json
+++ /dev/null
@@ -1,12 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "trials": 48,
- "single": {
- "slide_mm": 27.666296599509735,
- "slide_median_mm": 19.639506924676837,
- "mocap_slide_mm": 1.799128054429777,
- "foot_err_mm": 56.66949423833437,
- "contact_f1": null,
- "n_contact_pairs": 50999
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g10m75_b4_footlock_held6.json b/benchmark/results/g10m75_b4_footlock_held6.json
deleted file mode 100644
index e151c0a..0000000
--- a/benchmark/results/g10m75_b4_footlock_held6.json
+++ /dev/null
@@ -1,12 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "trials": 156,
- "single": {
- "slide_mm": 24.372936237044282,
- "slide_median_mm": 16.901182594280865,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 50.46389324405153,
- "contact_f1": null,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g10m75_b4_footlock_held6_lock.json b/benchmark/results/g10m75_b4_footlock_held6_lock.json
deleted file mode 100644
index fe55a89..0000000
--- a/benchmark/results/g10m75_b4_footlock_held6_lock.json
+++ /dev/null
@@ -1,21 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "trials": 156,
- "single": {
- "slide_mm": 24.372936237044282,
- "slide_median_mm": 16.901182594280865,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 50.46389324405153,
- "contact_f1": null,
- "n_contact_pairs": 166663
- },
- "single+oracle": {
- "slide_mm": 2.745684603899639,
- "slide_median_mm": 0.26541061973360186,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 35.61514037016256,
- "contact_f1": null,
- "n_contact_pairs": 166663,
- "release_jump_mm": 118.28241398430083
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11A_b4_ep4_footlock.json b/benchmark/results/g11A_b4_ep4_footlock.json
deleted file mode 100644
index f308de4..0000000
--- a/benchmark/results/g11A_b4_ep4_footlock.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11A_b4/best_student_model_v3.pth",
- "trials": 48,
- "masked": {
- "slide_mm": 22.107962795353803,
- "slide_median_mm": 16.301859751490113,
- "mocap_slide_mm": 1.799128054429777,
- "foot_err_mm": 48.51918476871163,
- "contact_f1": 0.9803737528275499,
- "n_contact_pairs": 50999
- },
- "feedback": {
- "slide_mm": 22.088440617714742,
- "slide_median_mm": 16.282389958058232,
- "mocap_slide_mm": 1.799128054429777,
- "foot_err_mm": 48.580522672435535,
- "contact_f1": 0.980364551069062,
- "n_contact_pairs": 50999
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11A_b4_ep4_footlock_held6.json b/benchmark/results/g11A_b4_ep4_footlock_held6.json
deleted file mode 100644
index 9bd79df..0000000
--- a/benchmark/results/g11A_b4_ep4_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11A_b4/best_student_model_v3.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 20.199585302302513,
- "slide_median_mm": 14.590456783687506,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 44.62899372619635,
- "contact_f1": 0.9709687571770029,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 20.19544812635625,
- "slide_median_mm": 14.577429420539673,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 44.718540747378526,
- "contact_f1": 0.9709660041396126,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_3dpw_test_feedback_lag5.json b/benchmark/results/g11Ap_b4_ep119_3dpw_test_feedback_lag5.json
deleted file mode 100644
index ecb406e..0000000
--- a/benchmark/results/g11Ap_b4_ep119_3dpw_test_feedback_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Ap_b4/last.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 119,
- "source": null,
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 5,
- "accel_J14_adapter": {
- "accel_mm": 20.135477306024317,
- "accel_err_mm": 20.783979304278585,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.33478917052959,
- "PA_MPJPE_mm": 65.91252858912782,
- "PCK3D_50mm": 0.41283269081255713,
- "PCK3D_100mm": 0.8378108370494801,
- "AUC_0_150mm": 0.5662442233880162,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.87811843421172,
- "right_knee": 47.480660920065084,
- "right_hip": 96.78890472195758,
- "left_hip": 85.12311686351617,
- "left_knee": 45.571670315003956,
- "left_ankle": 65.98857209769301,
- "right_wrist": 64.913877018297,
- "right_elbow": 58.25949344204909,
- "right_shoulder": 48.070523928431555,
- "left_shoulder": 50.59562854313111,
- "left_elbow": 66.46674352540639,
- "left_wrist": 69.53451288616223,
- "neck": 32.240662602022034,
- "head": 127.86291494984252
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 97.82294905449851,
- "PA_MPJPE_mm": 60.539843559584305,
- "PCK3D_50mm": 0.45487315417947344,
- "PCK3D_100mm": 0.899785222156426,
- "AUC_0_150mm": 0.5991699966556085,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.77009812891358,
- "right_knee": 49.315992230556816,
- "right_hip": 95.61485701748335,
- "left_hip": 83.90144103328858,
- "left_knee": 46.50904989156167,
- "left_ankle": 60.64088253021299,
- "right_wrist": 62.26537865136643,
- "right_elbow": 49.99483586149725,
- "right_shoulder": 49.08535753609565,
- "left_shoulder": 48.93124888987266,
- "left_elbow": 54.24841164894461,
- "left_wrist": 66.20056929521805
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.13649200160337,
- "PA_MPJPE_mm": 40.00765862128708,
- "PCK3D_50mm": 0.7460492021865848,
- "PCK3D_100mm": 0.9551645376871669,
- "AUC_0_150mm": 0.7312829195250037
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_3dpw_test_masked_lag5.json b/benchmark/results/g11Ap_b4_ep119_3dpw_test_masked_lag5.json
deleted file mode 100644
index 54723b6..0000000
--- a/benchmark/results/g11Ap_b4_ep119_3dpw_test_masked_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Ap_b4/last.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 119,
- "source": null,
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_J14_adapter": {
- "accel_mm": 20.862696749200293,
- "accel_err_mm": 21.480745043366262,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.68573761916001,
- "PA_MPJPE_mm": 65.6908204018717,
- "PCK3D_50mm": 0.4143030361624389,
- "PCK3D_100mm": 0.8385923356738009,
- "AUC_0_150mm": 0.567792471701862,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.01021561197346,
- "right_knee": 47.14496617469539,
- "right_hip": 96.94880121001742,
- "left_hip": 85.66142069262379,
- "left_knee": 44.70870152068196,
- "left_ankle": 65.96805796133752,
- "right_wrist": 64.51923116213435,
- "right_elbow": 57.295964476434456,
- "right_shoulder": 48.773750844554165,
- "left_shoulder": 51.29877943876152,
- "left_elbow": 66.01757939746504,
- "left_wrist": 69.10873506372853,
- "neck": 31.664554235083646,
- "head": 127.55072783671254
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.05434635114169,
- "PA_MPJPE_mm": 60.36758850247231,
- "PCK3D_50mm": 0.45990891915517584,
- "PCK3D_100mm": 0.9006194249405484,
- "AUC_0_150mm": 0.6004565109356746,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.26246063889291,
- "right_knee": 48.98904955659549,
- "right_hip": 96.0511909049572,
- "left_hip": 84.54052047841017,
- "left_knee": 45.60953337151846,
- "left_ankle": 60.5960248569769,
- "right_wrist": 62.07799530455755,
- "right_elbow": 48.91965682839804,
- "right_shoulder": 49.344858314768594,
- "left_shoulder": 49.53594867377272,
- "left_elbow": 53.55467422940461,
- "left_wrist": 65.92914887141504
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.25360055779696,
- "PA_MPJPE_mm": 39.67334761076662,
- "PCK3D_50mm": 0.748321187877909,
- "PCK3D_100mm": 0.9566691239561556,
- "AUC_0_150mm": 0.733558283826942
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_emdb1_feedback_lag5.json b/benchmark/results/g11Ap_b4_ep119_emdb1_feedback_lag5.json
deleted file mode 100644
index 4c2a6fb..0000000
--- a/benchmark/results/g11Ap_b4_ep119_emdb1_feedback_lag5.json
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Ap_b4/last.pth": {
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 5,
- "accel_SMPL24_adapter": {
- "accel_mm": 23.71694207064039,
- "accel_err_mm": 23.413286234279354,
- "n_triplets": 24061
- },
- "epoch": 119,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 75.28451342675072,
- "PA_MPJPE_mm": 49.32961464663227,
- "PCK3D_50mm": 0.6649898352902128,
- "PCK3D_100mm": 0.9237577618277116,
- "AUC_0_150mm": 0.6794633938844004,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 35.984782052733735,
- "left_hip": 37.91461543626717,
- "right_hip": 37.10285380062053,
- "spine1": 43.3968465220128,
- "left_knee": 50.068259141558244,
- "right_knee": 49.152762959346084,
- "spine2": 43.0021799735047,
- "left_ankle": 52.945051696958174,
- "right_ankle": 53.48035286306277,
- "spine3": 40.57270966883438,
- "left_foot": 59.92935324448284,
- "right_foot": 61.21016730621103,
- "neck": 33.05460166568911,
- "left_collar": 28.273605467739078,
- "right_collar": 28.028362952847363,
- "head": 38.28001132987784,
- "left_shoulder": 36.83429191726316,
- "right_shoulder": 38.163907856892465,
- "left_elbow": 49.9615098428841,
- "right_elbow": 46.809072771123745,
- "left_wrist": 73.65416358980372,
- "right_wrist": 70.23516924165543,
- "left_hand": 88.4359149483448,
- "right_hand": 87.42020526946044
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 87.33718185033395,
- "PA_MPJPE_mm": 55.96037845764732,
- "PCK3D_50mm": 0.56288488095732,
- "PCK3D_100mm": 0.8926096929250064,
- "AUC_0_150mm": 0.633269449794096,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 55.80040242087239,
- "right_knee": 48.070309778470175,
- "right_hip": 49.203084231211406,
- "left_hip": 46.899005878765216,
- "left_knee": 50.337457886581944,
- "left_ankle": 56.177682138007114,
- "right_wrist": 71.21785717378056,
- "right_elbow": 41.5367673553261,
- "right_shoulder": 51.345826089244056,
- "left_shoulder": 41.18318571360114,
- "left_elbow": 46.58576174574198,
- "left_wrist": 77.49742720751759,
- "neck": 43.49747743771099,
- "head": 104.0930533502317
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 81.12590351695273,
- "PA_MPJPE_mm": 50.50412879185908,
- "PCK3D_50mm": 0.6311697022500657,
- "PCK3D_100mm": 0.9258460219336458,
- "AUC_0_150mm": 0.6690548269768766,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.98966846207546,
- "right_knee": 48.66611716307579,
- "right_hip": 45.96025854757653,
- "left_hip": 44.223283999783035,
- "left_knee": 50.83362155867761,
- "left_ankle": 53.13952190672274,
- "right_wrist": 69.25736532474082,
- "right_elbow": 38.89421854154825,
- "right_shoulder": 44.11637489604615,
- "left_shoulder": 39.3929947892343,
- "left_elbow": 44.28655178621686,
- "left_wrist": 74.28956852661139
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.92137593645755
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 47.87833054304481
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.58969493098249
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.372711663756775
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 44.90651146598574
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 62.56660539383333
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 44.7686746785312
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.38740045901465
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 44.62617472300928
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.74512530787968
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 45.21053643981933
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 46.58308594832151
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.5671559145812
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.62822467955338
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 63.49831562828297
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 51.99055678175822
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.02086488172738
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_emdb1_masked_lag5.json b/benchmark/results/g11Ap_b4_ep119_emdb1_masked_lag5.json
deleted file mode 100644
index 32a64e6..0000000
--- a/benchmark/results/g11Ap_b4_ep119_emdb1_masked_lag5.json
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Ap_b4/last.pth": {
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_SMPL24_adapter": {
- "accel_mm": 24.400431340190718,
- "accel_err_mm": 24.076295016988766,
- "n_triplets": 24061
- },
- "epoch": 119,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 74.73421185418586,
- "PA_MPJPE_mm": 49.19718649041132,
- "PCK3D_50mm": 0.6676641220318356,
- "PCK3D_100mm": 0.9240913994108617,
- "AUC_0_150mm": 0.6804539344810064,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.17180135649348,
- "left_hip": 38.167765861038525,
- "right_hip": 37.3489377581274,
- "spine1": 43.34648059455688,
- "left_knee": 49.79220733012618,
- "right_knee": 49.205437215797915,
- "spine2": 42.904102871963396,
- "left_ankle": 52.8064164354047,
- "right_ankle": 53.182587618501024,
- "spine3": 40.5324330265759,
- "left_foot": 59.788440417513286,
- "right_foot": 61.01934639444416,
- "neck": 32.732649303759764,
- "left_collar": 28.068073409916316,
- "right_collar": 28.15360125170892,
- "head": 37.75024996384262,
- "left_shoulder": 37.211459039385616,
- "right_shoulder": 38.73597788150285,
- "left_elbow": 49.57795356165581,
- "right_elbow": 46.11656990278919,
- "left_wrist": 73.14233194686011,
- "right_wrist": 70.07280674910159,
- "left_hand": 87.46363548503955,
- "right_hand": 87.44121039376645
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.84952407265665,
- "PA_MPJPE_mm": 55.836256511021446,
- "PCK3D_50mm": 0.5654482844459195,
- "PCK3D_100mm": 0.892316309173132,
- "AUC_0_150mm": 0.6342407995180438,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 55.38994656274253,
- "right_knee": 47.84877136789184,
- "right_hip": 48.981101675007615,
- "left_hip": 46.253719701017,
- "left_knee": 50.03947588212598,
- "left_ankle": 56.20862527797976,
- "right_wrist": 71.28073471741013,
- "right_elbow": 41.35290172855042,
- "right_shoulder": 51.91486894438225,
- "left_shoulder": 41.5075301287873,
- "left_elbow": 46.43905391817685,
- "left_wrist": 77.42551069201505,
- "neck": 43.25250571026476,
- "head": 103.81284484794887
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.49522234326268,
- "PA_MPJPE_mm": 50.44852951496572,
- "PCK3D_50mm": 0.6334792349500062,
- "PCK3D_100mm": 0.9252340649158473,
- "AUC_0_150mm": 0.6695531364274916,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.68045602595872,
- "right_knee": 48.38045413789266,
- "right_hip": 45.885785069889494,
- "left_hip": 43.80164130717893,
- "left_knee": 50.519234692305886,
- "left_ankle": 53.20763393290701,
- "right_wrist": 69.4510044708268,
- "right_elbow": 38.70498354846364,
- "right_shoulder": 44.654604988560145,
- "left_shoulder": 39.750288310367516,
- "left_elbow": 44.031571731301604,
- "left_wrist": 74.31469596393619
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.68429330592589
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 47.91816886079579
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.40346874499568
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.49502519253246
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.2908999587836
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 62.02821292057404
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 44.40229018666644
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.69560570250014
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 44.48936282788257
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.785014836629266
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 43.836374199344974
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 46.3409362704575
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.59781563363172
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.67697877421014
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 63.30392290063722
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.08425607398132
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.125035941712255
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_footlock_held6.json b/benchmark/results/g11Ap_b4_ep119_footlock_held6.json
deleted file mode 100644
index fb7f873..0000000
--- a/benchmark/results/g11Ap_b4_ep119_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 13.652990772598686,
- "slide_median_mm": 9.659798582033709,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 30.67944011345725,
- "contact_f1": 0.9807184748636739,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 11.070269377794858,
- "slide_median_mm": 7.659354394283289,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 30.410201997983858,
- "contact_f1": 0.9806047477931363,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep119_footlock_held6_lock.json b/benchmark/results/g11Ap_b4_ep119_footlock_held6_lock.json
deleted file mode 100644
index f15e90e..0000000
--- a/benchmark/results/g11Ap_b4_ep119_footlock_held6_lock.json
+++ /dev/null
@@ -1,56 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/last.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 13.652990772598686,
- "slide_median_mm": 9.659798582033709,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 30.67944011345725,
- "contact_f1": 0.9807184748636739,
- "n_contact_pairs": 166663
- },
- "masked+lock": {
- "slide_mm": 1.8064639797433504,
- "slide_median_mm": 0.11818192252201057,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 26.51853092638421,
- "contact_f1": 0.9807184748636739,
- "n_contact_pairs": 166663,
- "release_jump_mm": 134.13089255566646
- },
- "masked+oracle": {
- "slide_mm": 1.696517696546154,
- "slide_median_mm": 0.1310135382905873,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 25.049827973792087,
- "contact_f1": 0.9807184748636739,
- "n_contact_pairs": 166663,
- "release_jump_mm": 105.13933541883355
- },
- "feedback": {
- "slide_mm": 11.070269377794858,
- "slide_median_mm": 7.659354394283289,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 30.410201997983858,
- "contact_f1": 0.9806047477931363,
- "n_contact_pairs": 166663
- },
- "feedback+lock": {
- "slide_mm": 1.7318894857429854,
- "slide_median_mm": 0.11086644036484843,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 27.262499993864918,
- "contact_f1": 0.9806047477931363,
- "n_contact_pairs": 166663,
- "release_jump_mm": 126.86397171804693
- },
- "feedback+oracle": {
- "slide_mm": 1.633914015249923,
- "slide_median_mm": 0.1223839931868227,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 25.579087266189212,
- "contact_f1": 0.9806047477931363,
- "n_contact_pairs": 166663,
- "release_jump_mm": 98.04958085295783
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Ap_b4_ep6_footlock_held6.json b/benchmark/results/g11Ap_b4_ep6_footlock_held6.json
deleted file mode 100644
index fc39620..0000000
--- a/benchmark/results/g11Ap_b4_ep6_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11Ap_b4/best_student_model_v3.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 15.88753207793247,
- "slide_median_mm": 11.196064289331627,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 34.364854760861625,
- "contact_f1": 0.976902671057446,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 15.85523034805657,
- "slide_median_mm": 11.147126174622802,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 34.755370259991054,
- "contact_f1": 0.976804396219891,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag1.json b/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag1.json
deleted file mode 100644
index 83b2f4c..0000000
--- a/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag1.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 51,
- "source": "ema",
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 1,
- "accel_J14_adapter": {
- "accel_mm": 16.922179341566927,
- "accel_err_mm": 17.62723954645827,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.01927160106395,
- "PA_MPJPE_mm": 66.8945105269123,
- "PCK3D_50mm": 0.4070902872611696,
- "PCK3D_100mm": 0.8316937975596296,
- "AUC_0_150mm": 0.5627832916269843,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.97474604034674,
- "right_knee": 47.253359164118265,
- "right_hip": 97.62291057674348,
- "left_hip": 85.90755823303252,
- "left_knee": 44.28209767172753,
- "left_ankle": 67.581050694151,
- "right_wrist": 70.73039601994356,
- "right_elbow": 57.3703035562264,
- "right_shoulder": 50.06519915692683,
- "left_shoulder": 51.525869834458135,
- "left_elbow": 66.27689503880684,
- "left_wrist": 72.72247817734338,
- "neck": 32.84089869839365,
- "head": 127.36938451455383
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 99.60044253987596,
- "PA_MPJPE_mm": 61.80835111050184,
- "PCK3D_50mm": 0.4485543618231208,
- "PCK3D_100mm": 0.8909003750387728,
- "AUC_0_150mm": 0.5941813806241791,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.33470042731949,
- "right_knee": 49.07727142819435,
- "right_hip": 97.08454460586312,
- "left_hip": 85.040785663487,
- "left_knee": 45.070804679003864,
- "left_ankle": 62.625887909928714,
- "right_wrist": 67.86660792511417,
- "right_elbow": 48.89064960832629,
- "right_shoulder": 50.84754200376089,
- "left_shoulder": 50.373473952939676,
- "left_elbow": 54.07535565986538,
- "left_wrist": 69.41258946221915
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.60274110806344,
- "PA_MPJPE_mm": 41.21846437067753,
- "PCK3D_50mm": 0.735460701495724,
- "PCK3D_100mm": 0.9503426106082395,
- "AUC_0_150mm": 0.7260508161228857
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag5.json b/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag5.json
deleted file mode 100644
index 048e902..0000000
--- a/benchmark/results/g11C_b4_ep51_3dpw_test_feedback_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 51,
- "source": "ema",
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 5,
- "accel_J14_adapter": {
- "accel_mm": 23.14270352642641,
- "accel_err_mm": 23.831018188977833,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.06980918384288,
- "PA_MPJPE_mm": 68.77158717099566,
- "PCK3D_50mm": 0.3884491280650658,
- "PCK3D_100mm": 0.8223601258454486,
- "AUC_0_150mm": 0.5533922485056472,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.54445929017486,
- "right_knee": 47.59279867961613,
- "right_hip": 97.12129205359685,
- "left_hip": 85.87664089732759,
- "left_knee": 45.202584292803614,
- "left_ankle": 69.22957762915091,
- "right_wrist": 78.0865207835035,
- "right_elbow": 59.05913208903247,
- "right_shoulder": 51.147066330909475,
- "left_shoulder": 52.39046549796714,
- "left_elbow": 68.09399466148005,
- "left_wrist": 81.09902093734304,
- "neck": 34.21936362826638,
- "head": 127.13930362276696
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 101.9939948029579,
- "PA_MPJPE_mm": 63.983003644043244,
- "PCK3D_50mm": 0.42513558732575735,
- "PCK3D_100mm": 0.8795622667757005,
- "AUC_0_150mm": 0.5831298803326442,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.75257280361086,
- "right_knee": 49.33421396702008,
- "right_hip": 96.5304581469697,
- "left_hip": 85.04540797490667,
- "left_knee": 45.84111610266849,
- "left_ankle": 64.13097886121258,
- "right_wrist": 75.44850204394847,
- "right_elbow": 50.451952259797565,
- "right_shoulder": 52.859507161853635,
- "left_shoulder": 51.985039428507896,
- "left_elbow": 55.661555936246295,
- "left_wrist": 77.75473904177666
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 68.35026037497795,
- "PA_MPJPE_mm": 43.57750753220123,
- "PCK3D_50mm": 0.7127710571581648,
- "PCK3D_100mm": 0.9416877147610588,
- "AUC_0_150mm": 0.7137326617175218
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11C_b4_ep51_3dpw_test_masked_lag5.json b/benchmark/results/g11C_b4_ep51_3dpw_test_masked_lag5.json
deleted file mode 100644
index d37e825..0000000
--- a/benchmark/results/g11C_b4_ep51_3dpw_test_masked_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 51,
- "source": "ema",
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_J14_adapter": {
- "accel_mm": 20.80364185873147,
- "accel_err_mm": 21.4230828438584,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.86569017706783,
- "PA_MPJPE_mm": 65.7584775737024,
- "PCK3D_50mm": 0.413060292216032,
- "PCK3D_100mm": 0.838207628876777,
- "AUC_0_150mm": 0.5671918586919502,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.95652449180901,
- "right_knee": 46.89910873517343,
- "right_hip": 96.84541540751013,
- "left_hip": 85.81045978935211,
- "left_knee": 44.388897054449636,
- "left_ankle": 65.94168797390452,
- "right_wrist": 64.7715016035552,
- "right_elbow": 57.190063309069,
- "right_shoulder": 49.706456786615355,
- "left_shoulder": 51.59085785941296,
- "left_elbow": 65.66693284040427,
- "left_wrist": 68.81714935886565,
- "neck": 32.172615914318136,
- "head": 126.8610149073941
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.42539594998296,
- "PA_MPJPE_mm": 60.48863570191304,
- "PCK3D_50mm": 0.4572911673199297,
- "PCK3D_100mm": 0.8986337873276372,
- "AUC_0_150mm": 0.599516033521535,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.15875697835752,
- "right_knee": 48.74115430713258,
- "right_hip": 96.03158575189435,
- "left_hip": 84.82077436641218,
- "left_knee": 45.23479477411858,
- "left_ankle": 60.844471016028564,
- "right_wrist": 62.429558125890225,
- "right_elbow": 48.59442311173604,
- "right_shoulder": 50.00461215538298,
- "left_shoulder": 49.73915177647036,
- "left_elbow": 53.410639583682794,
- "left_wrist": 65.85370647585034
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.39726563637626,
- "PA_MPJPE_mm": 39.845168887995364,
- "PCK3D_50mm": 0.7466816521042052,
- "PCK3D_100mm": 0.9549993756067692,
- "AUC_0_150mm": 0.7323620607497578
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11C_b4_ep51_footlock.json b/benchmark/results/g11C_b4_ep51_footlock.json
deleted file mode 100644
index 0fed71f..0000000
--- a/benchmark/results/g11C_b4_ep51_footlock.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth",
- "trials": 48,
- "masked": {
- "slide_mm": 28.2444123363245,
- "slide_median_mm": 20.509945109687855,
- "mocap_slide_mm": 1.799128054429777,
- "foot_err_mm": 59.53556022026623,
- "contact_f1": 0.8347293640783502,
- "n_contact_pairs": 50999
- },
- "feedback": {
- "slide_mm": 27.49499905360166,
- "slide_median_mm": 19.789050188776354,
- "mocap_slide_mm": 1.799128054429777,
- "foot_err_mm": 58.84774316195442,
- "contact_f1": 0.8196253228566135,
- "n_contact_pairs": 50999
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11C_b4_ep51_footlock_held6.json b/benchmark/results/g11C_b4_ep51_footlock_held6.json
deleted file mode 100644
index 5f27987..0000000
--- a/benchmark/results/g11C_b4_ep51_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11C_b4/best_student_model_v3.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 24.279807675995404,
- "slide_median_mm": 17.174716393436228,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 50.02693770257766,
- "contact_f1": 0.8505048951765851,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 23.494512860298446,
- "slide_median_mm": 16.56258194738066,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 49.572739203804296,
- "contact_f1": 0.8439830375171342,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep119_3dpw_test_feedback_lag5.json b/benchmark/results/g11Cp_b4_ep119_3dpw_test_feedback_lag5.json
deleted file mode 100644
index b40ba1e..0000000
--- a/benchmark/results/g11Cp_b4_ep119_3dpw_test_feedback_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Cp_b4/last.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/last.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 119,
- "source": null,
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 5,
- "accel_J14_adapter": {
- "accel_mm": 20.676193061293965,
- "accel_err_mm": 21.33714009079852,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.67346200956271,
- "PA_MPJPE_mm": 67.17560441223577,
- "PCK3D_50mm": 0.3982662009901668,
- "PCK3D_100mm": 0.8306524707844393,
- "AUC_0_150mm": 0.5594437949282116,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.13783994658543,
- "right_knee": 48.61511671561069,
- "right_hip": 97.09024228054102,
- "left_hip": 86.03390267113143,
- "left_knee": 47.42858024929464,
- "left_ankle": 70.42927621079468,
- "right_wrist": 65.94131599979929,
- "right_elbow": 58.10182986038263,
- "right_shoulder": 49.60824853792554,
- "left_shoulder": 51.66657871904823,
- "left_elbow": 66.20874514619135,
- "left_wrist": 71.61796482046834,
- "neck": 32.061783939687466,
- "head": 127.51703667384002
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.67075856987624,
- "PA_MPJPE_mm": 62.13922527706384,
- "PCK3D_50mm": 0.43687787271240447,
- "PCK3D_100mm": 0.8885505080412449,
- "AUC_0_150mm": 0.5901399125970761,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.84150167399308,
- "right_knee": 50.38268484564124,
- "right_hip": 96.14199770940169,
- "left_hip": 85.06894438901432,
- "left_knee": 48.204780826737675,
- "left_ankle": 65.34692273520221,
- "right_wrist": 62.97421188668362,
- "right_elbow": 49.69285609006077,
- "right_shoulder": 50.66522785784326,
- "left_shoulder": 50.87982018590939,
- "left_elbow": 53.700426542033675,
- "left_wrist": 67.77132858224505
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 67.23091177891753,
- "PA_MPJPE_mm": 41.536743253492524,
- "PCK3D_50mm": 0.7291241172892472,
- "PCK3D_100mm": 0.9471561909595917,
- "AUC_0_150mm": 0.7224049704040207
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep119_3dpw_test_masked_lag5.json b/benchmark/results/g11Cp_b4_ep119_3dpw_test_masked_lag5.json
deleted file mode 100644
index 2486857..0000000
--- a/benchmark/results/g11Cp_b4_ep119_3dpw_test_masked_lag5.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Cp_b4/last.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/last.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 119,
- "source": null,
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_J14_adapter": {
- "accel_mm": 21.284027376132443,
- "accel_err_mm": 21.88788290259966,
- "n_triplets": 34705
- },
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 101.34836492253064,
- "PA_MPJPE_mm": 65.74531665917735,
- "PCK3D_50mm": 0.415704899674107,
- "PCK3D_100mm": 0.8380062117055603,
- "AUC_0_150mm": 0.5676772740745825,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.91584960076675,
- "right_knee": 46.65686680799448,
- "right_hip": 97.31021229438578,
- "left_hip": 86.47472972591719,
- "left_knee": 43.94843951184188,
- "left_ankle": 65.51156797224577,
- "right_wrist": 65.33155818011923,
- "right_elbow": 57.05304374037338,
- "right_shoulder": 49.28856702967222,
- "left_shoulder": 51.48899217437485,
- "left_elbow": 66.02553639119265,
- "left_wrist": 69.84608214231449,
- "neck": 32.20927937057594,
- "head": 126.37370828670828
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.98177803063545,
- "PA_MPJPE_mm": 60.58508154733318,
- "PCK3D_50mm": 0.46033189521473084,
- "PCK3D_100mm": 0.897773736006542,
- "AUC_0_150mm": 0.5992947670129614,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.60334974109297,
- "right_knee": 48.462630494889034,
- "right_hip": 96.61429523164087,
- "left_hip": 85.50298429457202,
- "left_knee": 44.83421197570693,
- "left_ankle": 60.33799780983,
- "right_wrist": 62.9604994442266,
- "right_elbow": 48.8488272481752,
- "right_shoulder": 49.646131975025334,
- "left_shoulder": 49.674627469705754,
- "left_elbow": 53.75206144846139,
- "left_wrist": 66.78336143467202
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.47611225891512,
- "PA_MPJPE_mm": 39.80500775051873,
- "PCK3D_50mm": 0.7489073118461496,
- "PCK3D_100mm": 0.9555009043630988,
- "AUC_0_150mm": 0.7329648178779441
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep119_emdb1_feedback_lag5.json b/benchmark/results/g11Cp_b4_ep119_emdb1_feedback_lag5.json
deleted file mode 100644
index 38b4cea..0000000
--- a/benchmark/results/g11Cp_b4_ep119_emdb1_feedback_lag5.json
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Cp_b4/last.pth": {
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/last.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "prev": "feedback",
- "prev_lag": 5,
- "accel_SMPL24_adapter": {
- "accel_mm": 24.786672613695742,
- "accel_err_mm": 24.51570694260223,
- "n_triplets": 24061
- },
- "epoch": 119,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.1979265994633,
- "PA_MPJPE_mm": 51.770060411478006,
- "PCK3D_50mm": 0.6302932553347439,
- "PCK3D_100mm": 0.9139699069272151,
- "AUC_0_150mm": 0.6650260262966418,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.05648444919698,
- "left_hip": 39.46572746171653,
- "right_hip": 38.38729724900891,
- "spine1": 43.45119434283571,
- "left_knee": 54.922869271956685,
- "right_knee": 52.27222323243107,
- "spine2": 43.98413859552691,
- "left_ankle": 61.51524421143229,
- "right_ankle": 60.330918167505594,
- "spine3": 41.74194042939997,
- "left_foot": 69.04894671123058,
- "right_foot": 69.17965776525533,
- "neck": 33.53235624781543,
- "left_collar": 28.358145797173513,
- "right_collar": 28.590874930665443,
- "head": 38.97530361409454,
- "left_shoulder": 37.76907540645984,
- "right_shoulder": 39.44080512414226,
- "left_elbow": 50.56118179100191,
- "right_elbow": 46.25917208976721,
- "left_wrist": 76.96145191696397,
- "right_wrist": 71.68099276054217,
- "left_hand": 90.39128601170523,
- "right_hand": 88.6041622976458
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 89.6217185392296,
- "PA_MPJPE_mm": 58.43683274240557,
- "PCK3D_50mm": 0.5248813129367417,
- "PCK3D_100mm": 0.8837340935627456,
- "AUC_0_150mm": 0.6183331673151571,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.67096795050039,
- "right_knee": 50.55603287410185,
- "right_hip": 52.80720155443429,
- "left_hip": 48.624095503852814,
- "left_knee": 54.48948819892771,
- "left_ankle": 65.15636600093762,
- "right_wrist": 72.15330489697097,
- "right_elbow": 40.926938140862454,
- "right_shoulder": 53.2276177031594,
- "left_shoulder": 42.388930684177915,
- "left_elbow": 47.44714513208908,
- "left_wrist": 81.58029662700937,
- "neck": 43.82130984912394,
- "head": 103.26596327753002
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 83.87400016234747,
- "PA_MPJPE_mm": 53.65806106340466,
- "PCK3D_50mm": 0.5852210651509494,
- "PCK3D_100mm": 0.9150762698972466,
- "AUC_0_150mm": 0.6498887614489607,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.56950050144152,
- "right_knee": 50.89892425671809,
- "right_hip": 49.37659533411967,
- "left_hip": 45.9179594396927,
- "left_knee": 54.46561779538317,
- "left_ankle": 62.08476066261144,
- "right_wrist": 70.96882571986636,
- "right_elbow": 39.147573720908895,
- "right_shoulder": 46.44322161223819,
- "left_shoulder": 40.61530665609468,
- "left_elbow": 45.649491044583655,
- "left_wrist": 78.7589560171975
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.959962118156646
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.89781967203327
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.351129210279936
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.199231497941305
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.90591676177114
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 64.8352109491138
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.170081661388366
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.952728916537914
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 47.437148151117036
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.61950054589303
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 48.33261935492051
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.152193915731026
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.20075413436996
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.538723321769595
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 68.17401362307172
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 53.69257824399373
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.22879890346247
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep119_emdb1_masked_lag5.json b/benchmark/results/g11Cp_b4_ep119_emdb1_masked_lag5.json
deleted file mode 100644
index 24e8922..0000000
--- a/benchmark/results/g11Cp_b4_ep119_emdb1_masked_lag5.json
+++ /dev/null
@@ -1,182 +0,0 @@
-{
- "instanthmr_distill_train/runs/g11Cp_b4/last.pth": {
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/last.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "prev": "masked",
- "prev_lag": null,
- "accel_SMPL24_adapter": {
- "accel_mm": 25.140746341582933,
- "accel_err_mm": 24.820544251230693,
- "n_triplets": 24061
- },
- "epoch": 119,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 74.68504123046212,
- "PA_MPJPE_mm": 49.420807192044805,
- "PCK3D_50mm": 0.6640407833049827,
- "PCK3D_100mm": 0.9226254684755701,
- "AUC_0_150mm": 0.6791235107038387,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.65659231846469,
- "left_hip": 38.739875689767274,
- "right_hip": 37.963848297291435,
- "spine1": 43.72215003755321,
- "left_knee": 50.44457857373066,
- "right_knee": 49.16147421499149,
- "spine2": 43.42945879397658,
- "left_ankle": 53.641208173538594,
- "right_ankle": 54.35780223570673,
- "spine3": 40.94879170419838,
- "left_foot": 60.448475746083275,
- "right_foot": 61.93892206232942,
- "neck": 32.73678623105209,
- "left_collar": 27.89020926403403,
- "right_collar": 28.356942327449513,
- "head": 37.7185929190615,
- "left_shoulder": 37.22695745547703,
- "right_shoulder": 38.943176511548685,
- "left_elbow": 49.300882415001304,
- "right_elbow": 45.77829886335639,
- "left_wrist": 73.06282754448938,
- "right_wrist": 69.59711888563338,
- "left_hand": 87.26965681827244,
- "right_hand": 86.76474552606898
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.79178746605311,
- "PA_MPJPE_mm": 55.95574704030166,
- "PCK3D_50mm": 0.563664274156744,
- "PCK3D_100mm": 0.8912346418051103,
- "AUC_0_150mm": 0.6334037620037355,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.37667463533803,
- "right_knee": 47.46537815053654,
- "right_hip": 49.90965980762136,
- "left_hip": 47.13574028628853,
- "left_knee": 50.223274237281935,
- "left_ankle": 57.03131392785176,
- "right_wrist": 70.85789511828996,
- "right_elbow": 40.91167491799967,
- "right_shoulder": 52.20677231881975,
- "left_shoulder": 41.783096303707886,
- "left_elbow": 45.85258316661215,
- "left_wrist": 77.32376633729686,
- "neck": 42.99214045728123,
- "head": 103.31048889929762
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.67330945986194,
- "PA_MPJPE_mm": 50.741564114020456,
- "PCK3D_50mm": 0.6323417555214427,
- "PCK3D_100mm": 0.9228726714516865,
- "AUC_0_150mm": 0.6677238455570825,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 53.78685414466862,
- "right_knee": 47.9477414325536,
- "right_hip": 46.724975219861825,
- "left_hip": 44.56104179668429,
- "left_knee": 50.650517621845005,
- "left_ankle": 54.04721083501075,
- "right_wrist": 69.32374433929482,
- "right_elbow": 38.58844504149409,
- "right_shoulder": 45.07685404591891,
- "left_shoulder": 39.72153434472141,
- "left_elbow": 43.81015925533049,
- "left_wrist": 74.65969129086169
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.3116474487617
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.42436399055881
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.42636150320221
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.27975657434216
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.93679399848008
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 63.046206643045096
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 44.30459385891545
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.616232276175865
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 43.672294259740916
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.49892132450445
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 44.36057603872932
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 46.642389948524176
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.481158673633246
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.086616236174095
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 63.61953715377291
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.86101928505943
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.43771183679875
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep119_footlock_held6.json b/benchmark/results/g11Cp_b4_ep119_footlock_held6.json
deleted file mode 100644
index 0c06fe2..0000000
--- a/benchmark/results/g11Cp_b4_ep119_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/last.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 24.491146700529235,
- "slide_median_mm": 17.461713171184627,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 48.98297783231104,
- "contact_f1": 0.8022863574569359,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 22.084397028751717,
- "slide_median_mm": 14.97646978128771,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 62.66020488962684,
- "contact_f1": 0.6651817839152931,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/g11Cp_b4_ep3_footlock_held6.json b/benchmark/results/g11Cp_b4_ep3_footlock_held6.json
deleted file mode 100644
index 65e218b..0000000
--- a/benchmark/results/g11Cp_b4_ep3_footlock_held6.json
+++ /dev/null
@@ -1,20 +0,0 @@
-{
- "ckpt": "instanthmr_distill_train/runs/g11Cp_b4/best_student_model_v3.pth",
- "trials": 156,
- "masked": {
- "slide_mm": 23.07777380974508,
- "slide_median_mm": 16.299346334307465,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 50.040571615580056,
- "contact_f1": 0.7341644483987046,
- "n_contact_pairs": 166663
- },
- "feedback": {
- "slide_mm": 23.082563123409642,
- "slide_median_mm": 16.30244750229059,
- "mocap_slide_mm": 1.833724014677916,
- "foot_err_mm": 50.0351350748647,
- "contact_f1": 0.7338291248702871,
- "n_contact_pairs": 166663
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_3dpw_test_detector_288.json b/benchmark/results/gen10_3dpw_test_detector_288.json
deleted file mode 100644
index 64682df..0000000
--- a/benchmark/results/gen10_3dpw_test_detector_288.json
+++ /dev/null
@@ -1,642 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 481,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 106.73322811415207,
- "PA_MPJPE_mm": 69.75096916072816,
- "PCK3D_50mm": 0.3759362311338305,
- "PCK3D_100mm": 0.8174409951954964,
- "AUC_0_150mm": 0.5471819919043043,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.20770733647707,
- "right_knee": 49.877940348859944,
- "right_hip": 98.49547836583446,
- "left_hip": 87.63858583347924,
- "left_knee": 49.12667990526123,
- "left_ankle": 71.28453465806761,
- "right_wrist": 75.42698236059015,
- "right_elbow": 60.98774370791998,
- "right_shoulder": 50.99653468058157,
- "left_shoulder": 52.654595504365275,
- "left_elbow": 69.55761996625608,
- "left_wrist": 79.12869041934793,
- "neck": 34.73213250794872,
- "head": 126.39834265520503
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 104.85696909573582,
- "PA_MPJPE_mm": 65.13906800712654,
- "PCK3D_50mm": 0.40669941116521346,
- "PCK3D_100mm": 0.8709210923640207,
- "AUC_0_150mm": 0.5753168365837354,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.33966155137414,
- "right_knee": 51.43445566267487,
- "right_hip": 97.34979567023127,
- "left_hip": 86.62041826868825,
- "left_knee": 49.757821545880766,
- "left_ankle": 66.20758630903444,
- "right_wrist": 72.56468035011356,
- "right_elbow": 51.941024840724566,
- "right_shoulder": 53.10241683059991,
- "left_shoulder": 53.436236921669575,
- "left_elbow": 57.042808797684145,
- "left_wrist": 75.87190933684289
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 71.56401316374992,
- "PA_MPJPE_mm": 44.95815734013557,
- "PCK3D_50mm": 0.693789734215507,
- "PCK3D_100mm": 0.9371269230147047,
- "AUC_0_150mm": 0.7044047675579378
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 524,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 104.23288991237436,
- "PA_MPJPE_mm": 68.49232125616992,
- "PCK3D_50mm": 0.38419447724735917,
- "PCK3D_100mm": 0.8244273419933028,
- "AUC_0_150mm": 0.5528796103170284,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.44903569185718,
- "right_knee": 48.60700628018377,
- "right_hip": 97.79671634635844,
- "left_hip": 87.87463901130415,
- "left_knee": 48.28411363762043,
- "left_ankle": 69.74289293181482,
- "right_wrist": 72.61980597583334,
- "right_elbow": 59.61764136188258,
- "right_shoulder": 50.091878333574655,
- "left_shoulder": 52.39075524678232,
- "left_elbow": 68.44193910050973,
- "left_wrist": 75.38269969348565,
- "neck": 34.415670114322005,
- "head": 125.17770386084966
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 102.06837694295032,
- "PA_MPJPE_mm": 63.795426553313305,
- "PCK3D_50mm": 0.41941267504623864,
- "PCK3D_100mm": 0.8787722228513192,
- "AUC_0_150mm": 0.5818620834718361,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.491359940184,
- "right_knee": 50.20670909371312,
- "right_hip": 96.83320135699799,
- "left_hip": 86.82157593865625,
- "left_knee": 48.986215224587006,
- "left_ankle": 64.77556335403308,
- "right_wrist": 69.61980563496893,
- "right_elbow": 51.17740419709609,
- "right_shoulder": 51.81680555718693,
- "left_shoulder": 52.35038123418101,
- "left_elbow": 56.295299031847385,
- "left_wrist": 72.17079807630786
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 68.71909935783944,
- "PA_MPJPE_mm": 43.52858476985652,
- "PCK3D_50mm": 0.7066967824384878,
- "PCK3D_100mm": 0.9431204199492049,
- "AUC_0_150mm": 0.711794881562489
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 492,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 102.90207453462736,
- "PA_MPJPE_mm": 66.72376678512934,
- "PCK3D_50mm": 0.406809615477943,
- "PCK3D_100mm": 0.8322184835886568,
- "AUC_0_150mm": 0.5622873210052356,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.57002958052782,
- "right_knee": 48.01301841437031,
- "right_hip": 98.57314647026017,
- "left_hip": 88.03850075014603,
- "left_knee": 45.48957256791564,
- "left_ankle": 68.07360007952249,
- "right_wrist": 67.11043081894735,
- "right_elbow": 56.882976875306646,
- "right_shoulder": 48.80848581669065,
- "left_shoulder": 51.54480550739813,
- "left_elbow": 66.13697203084998,
- "left_wrist": 70.61597295751562,
- "neck": 32.75972924685509,
- "head": 126.51549387550494
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.62239593696297,
- "PA_MPJPE_mm": 61.61489656259991,
- "PCK3D_50mm": 0.4484722379496471,
- "PCK3D_100mm": 0.8911552108104027,
- "AUC_0_150mm": 0.5938695918459288,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.74300164033769,
- "right_knee": 49.91164248191233,
- "right_hip": 97.79054873793744,
- "left_hip": 87.00550804171196,
- "left_knee": 46.527513333675394,
- "left_ankle": 63.315645580380455,
- "right_wrist": 64.34283570756496,
- "right_elbow": 48.48224055876918,
- "right_shoulder": 49.65083577588827,
- "left_shoulder": 49.85038839749633,
- "left_elbow": 53.389811824874556,
- "left_wrist": 67.36878667065024
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 65.66032365863555,
- "PA_MPJPE_mm": 40.87043735868663,
- "PCK3D_50mm": 0.7376409401944449,
- "PCK3D_100mm": 0.9523856706084087,
- "AUC_0_150mm": 0.7270201667043257
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 484,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 101.10989075996896,
- "PA_MPJPE_mm": 65.47815865864816,
- "PCK3D_50mm": 0.41644895417118266,
- "PCK3D_100mm": 0.8392412281411262,
- "AUC_0_150mm": 0.5690944186456695,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.749845423753975,
- "right_knee": 47.433754408997,
- "right_hip": 97.60636814250499,
- "left_hip": 87.05833318025881,
- "left_knee": 44.39830220990828,
- "left_ankle": 65.1862516326931,
- "right_wrist": 65.65532734494887,
- "right_elbow": 55.867836736652805,
- "right_shoulder": 49.574616389868496,
- "left_shoulder": 51.30825578095304,
- "left_elbow": 65.42070764635841,
- "left_wrist": 67.49940486697626,
- "neck": 32.190441156402585,
- "head": 123.74477630079748
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 98.64725925312783,
- "PA_MPJPE_mm": 60.57970853456063,
- "PCK3D_50mm": 0.45831681953723624,
- "PCK3D_100mm": 0.8979730494847695,
- "AUC_0_150mm": 0.5995533817210388,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.36706108523276,
- "right_knee": 49.375800000428555,
- "right_hip": 96.90124369456714,
- "left_hip": 86.03511065470914,
- "left_knee": 45.46823321260738,
- "left_ankle": 60.39772696406034,
- "right_wrist": 62.97483827763772,
- "right_elbow": 48.20586863450878,
- "right_shoulder": 49.507792102736765,
- "left_shoulder": 49.834270440868345,
- "left_elbow": 53.6133627211208,
- "left_wrist": 64.2751946262499
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 63.84637834953265,
- "PA_MPJPE_mm": 39.512290567853505,
- "PCK3D_50mm": 0.7550714204830387,
- "PCK3D_100mm": 0.9564864843004351,
- "AUC_0_150mm": 0.7348726887425789
- }
- ]
- },
- "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 159,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 109.35367275911065,
- "PA_MPJPE_mm": 71.6248189300115,
- "PCK3D_50mm": 0.37513750262872675,
- "PCK3D_100mm": 0.7995373440962842,
- "AUC_0_150mm": 0.5384064149803244,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.7660482859598,
- "right_knee": 52.39470884287265,
- "right_hip": 99.12181214172188,
- "left_hip": 88.81758389904535,
- "left_knee": 50.06865590799054,
- "left_ankle": 75.29418822359116,
- "right_wrist": 81.67815002363479,
- "right_elbow": 63.006854054241344,
- "right_shoulder": 49.57917857034092,
- "left_shoulder": 51.627064587477605,
- "left_elbow": 72.90758869705074,
- "left_wrist": 83.44348044922626,
- "neck": 35.90024661968483,
- "head": 128.14190471732317
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 107.93498087230692,
- "PA_MPJPE_mm": 67.10317116439745,
- "PCK3D_50mm": 0.4049607443475635,
- "PCK3D_100mm": 0.8507483108745706,
- "AUC_0_150mm": 0.5662018299416889,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.43056454746954,
- "right_knee": 54.08405313760191,
- "right_hip": 97.94746900296342,
- "left_hip": 87.60387711297243,
- "left_knee": 50.80544838634012,
- "left_ankle": 69.94451480421077,
- "right_wrist": 79.2417256872088,
- "right_elbow": 54.10007031487996,
- "right_shoulder": 52.63313029335664,
- "left_shoulder": 52.71013027626079,
- "left_elbow": 59.76299972940898,
- "left_wrist": 79.97407068009615
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 76.08385660209942,
- "PA_MPJPE_mm": 47.76771113585957,
- "PCK3D_50mm": 0.6662952747626057,
- "PCK3D_100mm": 0.9223635893039132,
- "AUC_0_150mm": 0.6893846274670242
- }
- ]
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 376,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 100.78359761930798,
- "PA_MPJPE_mm": 65.77732066287084,
- "PCK3D_50mm": 0.41384044842033746,
- "PCK3D_100mm": 0.8417243638481324,
- "AUC_0_150mm": 0.56776903159012,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.73766640048825,
- "right_knee": 46.988455796726086,
- "right_hip": 96.2293049721673,
- "left_hip": 85.39535701496472,
- "left_knee": 45.17979228857756,
- "left_ankle": 65.99471609331263,
- "right_wrist": 66.74068171004052,
- "right_elbow": 57.18624554182059,
- "right_shoulder": 48.67004167157678,
- "left_shoulder": 50.78756897237406,
- "left_elbow": 65.86662999847148,
- "left_wrist": 69.16672434688954,
- "neck": 32.7858295603646,
- "head": 125.15347491241755
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 98.35122280523517,
- "PA_MPJPE_mm": 60.67594334341044,
- "PCK3D_50mm": 0.4532635979315291,
- "PCK3D_100mm": 0.9003982183973125,
- "AUC_0_150mm": 0.5994069644690077,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.915898455287696,
- "right_knee": 48.76632933064926,
- "right_hip": 95.11672130936478,
- "left_hip": 84.17129313347468,
- "left_knee": 46.10643187068251,
- "left_ankle": 60.8820160783589,
- "right_wrist": 64.15376948607975,
- "right_elbow": 48.844686848058124,
- "right_shoulder": 49.619807690320016,
- "left_shoulder": 50.52334010363763,
- "left_elbow": 53.50848216326889,
- "left_wrist": 66.50254365174315
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 65.35611358134756,
- "PA_MPJPE_mm": 40.30602849404404,
- "PCK3D_50mm": 0.7451793196046395,
- "PCK3D_100mm": 0.9537081223611628,
- "AUC_0_150mm": 0.7302069271058648
- }
- ]
- },
- "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 383,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 101.75578819096737,
- "PA_MPJPE_mm": 66.09335917498443,
- "PCK3D_50mm": 0.4111834123299416,
- "PCK3D_100mm": 0.8376599479107689,
- "AUC_0_150mm": 0.565470950417126,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.2928328793787,
- "right_knee": 47.32521699052752,
- "right_hip": 97.72855434135153,
- "left_hip": 87.177138879566,
- "left_knee": 44.78543492250695,
- "left_ankle": 67.0873383940765,
- "right_wrist": 67.07772827934612,
- "right_elbow": 57.0547456928177,
- "right_shoulder": 48.8310795988766,
- "left_shoulder": 50.51369905473233,
- "left_elbow": 66.36331482137733,
- "left_wrist": 68.87057156918843,
- "neck": 33.236229989324166,
- "head": 124.96314303671222
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 99.4101569895191,
- "PA_MPJPE_mm": 61.05199687223043,
- "PCK3D_50mm": 0.451522572000151,
- "PCK3D_100mm": 0.8968406748952553,
- "AUC_0_150mm": 0.5966940597820727,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.408248812936336,
- "right_knee": 49.216900116819446,
- "right_hip": 96.6592202159503,
- "left_hip": 85.92787595088147,
- "left_knee": 45.691681104827275,
- "left_ankle": 61.825135037645325,
- "right_wrist": 64.67477085785104,
- "right_elbow": 48.79900711044531,
- "right_shoulder": 49.87641601882194,
- "left_shoulder": 49.693226333222796,
- "left_elbow": 53.97855558459741,
- "left_wrist": 65.87292532276648
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 64.84651656927035,
- "PA_MPJPE_mm": 40.301455119044384,
- "PCK3D_50mm": 0.7437860944400407,
- "PCK3D_100mm": 0.9544199006745717,
- "AUC_0_150mm": 0.7299392274909239
- }
- ]
- },
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 379,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 100.37965587923244,
- "PA_MPJPE_mm": 65.60646072021552,
- "PCK3D_50mm": 0.4122692786773865,
- "PCK3D_100mm": 0.8424300758690975,
- "AUC_0_150mm": 0.5681046997720117,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.90919478394263,
- "right_knee": 47.52992010162072,
- "right_hip": 96.57137078651314,
- "left_hip": 85.97588840555316,
- "left_knee": 45.07981688404867,
- "left_ankle": 66.27180507142705,
- "right_wrist": 65.27949262658849,
- "right_elbow": 57.90133447592936,
- "right_shoulder": 49.27160194257226,
- "left_shoulder": 50.662091821326825,
- "left_elbow": 67.06265448783446,
- "left_wrist": 67.70008732246127,
- "neck": 32.89485478936691,
- "head": 123.38033658383237
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 98.16097676926454,
- "PA_MPJPE_mm": 60.681424810220825,
- "PCK3D_50mm": 0.4524544219227721,
- "PCK3D_100mm": 0.8987138111954102,
- "AUC_0_150mm": 0.5985198616067786,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.942907905319224,
- "right_knee": 49.4045994817474,
- "right_hip": 95.72952964424253,
- "left_hip": 84.75069510533703,
- "left_knee": 45.89567294509718,
- "left_ankle": 61.49273364691198,
- "right_wrist": 62.93516828625031,
- "right_elbow": 49.84070442472038,
- "right_shoulder": 49.96934000673494,
- "left_shoulder": 49.57455895319653,
- "left_elbow": 54.85286885568073,
- "left_wrist": 64.78831846741167
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 64.4424923170005,
- "PA_MPJPE_mm": 39.77446172460066,
- "PCK3D_50mm": 0.7492316029571153,
- "PCK3D_100mm": 0.95557451833638,
- "AUC_0_150mm": 0.7329804621027622
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_3dpw_test_gt-joints_288.json b/benchmark/results/gen10_3dpw_test_gt-joints_288.json
deleted file mode 100644
index b6d61ed..0000000
--- a/benchmark/results/gen10_3dpw_test_gt-joints_288.json
+++ /dev/null
@@ -1,602 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 481,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 106.39861634424358,
- "PA_MPJPE_mm": 69.65849196852879,
- "PCK3D_50mm": 0.37517573648188657,
- "PCK3D_100mm": 0.8181706486841416,
- "AUC_0_150mm": 0.5468059719801426,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.1549463637793,
- "right_knee": 48.69558291578664,
- "right_hip": 97.7845919078818,
- "left_hip": 86.95649362211593,
- "left_knee": 47.53594073652043,
- "left_ankle": 69.88287137690065,
- "right_wrist": 74.7786961452429,
- "right_elbow": 60.22397025919391,
- "right_shoulder": 51.576738727335204,
- "left_shoulder": 53.822368990846634,
- "left_elbow": 69.08419042085075,
- "left_wrist": 79.73266717230209,
- "neck": 32.816283795679226,
- "head": 135.1735451249677
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 104.40257931183609,
- "PA_MPJPE_mm": 64.07082244569104,
- "PCK3D_50mm": 0.41482437094060476,
- "PCK3D_100mm": 0.8779667070843791,
- "AUC_0_150mm": 0.5799815133689239,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.643969157223346,
- "right_knee": 50.4189596691331,
- "right_hip": 97.41583205257864,
- "left_hip": 86.46149386504096,
- "left_knee": 48.389067684241255,
- "left_ankle": 64.80678516759107,
- "right_wrist": 70.84640393642232,
- "right_elbow": 51.11802193213622,
- "right_shoulder": 51.83734561697545,
- "left_shoulder": 52.20204407157766,
- "left_elbow": 56.1594327182752,
- "left_wrist": 75.5505134770971
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 71.33510588915172,
- "PA_MPJPE_mm": 44.34485073367059,
- "PCK3D_50mm": 0.6940594019521352,
- "PCK3D_100mm": 0.9406061045516252,
- "AUC_0_150mm": 0.7065152997132987
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 524,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 103.7700233901552,
- "PA_MPJPE_mm": 68.14377634253853,
- "PCK3D_50mm": 0.3893232785881462,
- "PCK3D_100mm": 0.8250248750206453,
- "AUC_0_150mm": 0.5545116081913635,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.06016162315319,
- "right_knee": 47.76728247852774,
- "right_hip": 97.64330000759553,
- "left_hip": 87.15973799084092,
- "left_knee": 46.884270104298864,
- "left_ankle": 67.94902831201394,
- "right_wrist": 71.21355188210572,
- "right_elbow": 58.26763803588583,
- "right_shoulder": 50.38976661361343,
- "left_shoulder": 53.14164484273692,
- "left_elbow": 67.69376823106079,
- "left_wrist": 75.90282232130136,
- "neck": 32.77354202718358,
- "head": 131.16635432522173
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 101.5432252795076,
- "PA_MPJPE_mm": 62.75993408081658,
- "PCK3D_50mm": 0.43261756384588634,
- "PCK3D_100mm": 0.8842643506377539,
- "AUC_0_150mm": 0.5870661349599869,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.46125358089467,
- "right_knee": 49.36471842214643,
- "right_hip": 97.26181824367421,
- "left_hip": 86.46026697234966,
- "left_knee": 47.600310992667445,
- "left_ankle": 63.267659620435936,
- "right_wrist": 67.5645606319212,
- "right_elbow": 49.756607993829114,
- "right_shoulder": 50.67445544473804,
- "left_shoulder": 51.070909412967694,
- "left_elbow": 55.502246610806395,
- "left_wrist": 72.13440104336819
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 68.43632581260393,
- "PA_MPJPE_mm": 42.859903324702785,
- "PCK3D_50mm": 0.7112141024246599,
- "PCK3D_100mm": 0.946568052819639,
- "AUC_0_150mm": 0.7148565045596301
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 492,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.78881297830642,
- "PA_MPJPE_mm": 66.51973335120698,
- "PCK3D_50mm": 0.40788185674405114,
- "PCK3D_100mm": 0.8327633227387902,
- "AUC_0_150mm": 0.5631718318475893,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.26948863579298,
- "right_knee": 47.46051260838262,
- "right_hip": 98.33903396805457,
- "left_hip": 87.68804115072673,
- "left_knee": 44.494236736864075,
- "left_ankle": 65.61735883969321,
- "right_wrist": 67.21180489923528,
- "right_elbow": 56.86538749186849,
- "right_shoulder": 49.46526990171911,
- "left_shoulder": 52.33417092399012,
- "left_elbow": 65.898221928299,
- "left_wrist": 72.02131565377383,
- "neck": 31.369228298040415,
- "head": 129.24219588045713
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.47358770086609,
- "PA_MPJPE_mm": 61.15218564338712,
- "PCK3D_50mm": 0.4531295528673077,
- "PCK3D_100mm": 0.8937225653028038,
- "AUC_0_150mm": 0.5960367901241307,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.13404204366826,
- "right_knee": 49.39284395328845,
- "right_hip": 97.95505661034973,
- "left_hip": 87.04078412958091,
- "left_knee": 45.507909367589676,
- "left_ankle": 61.00330884674717,
- "right_wrist": 63.941899308270806,
- "right_elbow": 48.58478823193253,
- "right_shoulder": 49.196578526445656,
- "left_shoulder": 49.26330995669283,
- "left_elbow": 53.370148586045374,
- "left_wrist": 68.4355581600341
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.80751587693167,
- "PA_MPJPE_mm": 40.41211366551497,
- "PCK3D_50mm": 0.7417489455811087,
- "PCK3D_100mm": 0.9548019867789769,
- "AUC_0_150mm": 0.7292645245495695
- }
- ]
- },
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 484,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.7053068443338,
- "PA_MPJPE_mm": 65.30017546829447,
- "PCK3D_50mm": 0.4154068022607063,
- "PCK3D_100mm": 0.8397726402971306,
- "AUC_0_150mm": 0.5695769628655608,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.20454510754775,
- "right_knee": 46.920423740103985,
- "right_hip": 97.2312935330236,
- "left_hip": 86.58218352312187,
- "left_knee": 43.4942770586795,
- "left_ankle": 63.42686504196035,
- "right_wrist": 65.61397646388207,
- "right_elbow": 55.7328975593997,
- "right_shoulder": 49.85450570101728,
- "left_shoulder": 51.87661034875145,
- "left_elbow": 65.47163341041592,
- "left_wrist": 68.07803959637484,
- "neck": 31.340965862140123,
- "head": 126.37423960970426
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.34347552967077,
- "PA_MPJPE_mm": 60.091402111499235,
- "PCK3D_50mm": 0.46140813429959865,
- "PCK3D_100mm": 0.9005230803936497,
- "AUC_0_150mm": 0.6017427220070957,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.073963515620775,
- "right_knee": 48.87087292920695,
- "right_hip": 96.89599241058708,
- "left_hip": 85.85079701040257,
- "left_knee": 44.48300316194517,
- "left_ankle": 58.87495116417157,
- "right_wrist": 62.42525590758948,
- "right_elbow": 48.02277902352207,
- "right_shoulder": 49.212677727146556,
- "left_shoulder": 49.37972566568719,
- "left_elbow": 53.58231464204072,
- "left_wrist": 64.42449218007067
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 63.50029678474325,
- "PA_MPJPE_mm": 38.96943402397896,
- "PCK3D_50mm": 0.7596791021628175,
- "PCK3D_100mm": 0.9601717685636136,
- "AUC_0_150mm": 0.7376122916972855
- }
- ]
- },
- "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 159,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 111.0665094632581,
- "PA_MPJPE_mm": 71.89407143814988,
- "PCK3D_50mm": 0.3728533964977582,
- "PCK3D_100mm": 0.7974770485133399,
- "AUC_0_150mm": 0.5373025900558913,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.86203442836424,
- "right_knee": 51.79742571611586,
- "right_hip": 98.29292142613957,
- "left_hip": 88.71985611732175,
- "left_knee": 50.13760942325833,
- "left_ankle": 73.92275021170205,
- "right_wrist": 82.03886782594732,
- "right_elbow": 63.97880431599796,
- "right_shoulder": 50.38959369779265,
- "left_shoulder": 51.9738662694138,
- "left_elbow": 73.83879261728723,
- "left_wrist": 84.50887843994433,
- "neck": 35.97701031653122,
- "head": 131.07858932828185
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 109.9440806564324,
- "PA_MPJPE_mm": 67.0584517702804,
- "PCK3D_50mm": 0.4055141978963991,
- "PCK3D_100mm": 0.8510419310267039,
- "AUC_0_150mm": 0.566637376711575,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.4772030497291,
- "right_knee": 53.433557993283706,
- "right_hip": 97.53534066961694,
- "left_hip": 87.87656719472717,
- "left_knee": 50.86243581672414,
- "left_ankle": 68.58256187573126,
- "right_wrist": 79.37883743328965,
- "right_elbow": 54.85202641220365,
- "right_shoulder": 52.54925917212221,
- "left_shoulder": 52.66983322891003,
- "left_elbow": 60.56907990801099,
- "left_wrist": 80.91471848901597
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 78.2582122513115,
- "PA_MPJPE_mm": 47.88274007740074,
- "PCK3D_50mm": 0.6643946809753425,
- "PCK3D_100mm": 0.9220394697088716,
- "AUC_0_150mm": 0.6888873988349772
- }
- ]
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 376,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.61269498522314,
- "PA_MPJPE_mm": 65.69319560572443,
- "PCK3D_50mm": 0.41360814692174136,
- "PCK3D_100mm": 0.8411704754653744,
- "AUC_0_150mm": 0.5679401559696606,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.89553778481446,
- "right_knee": 46.64295666240954,
- "right_hip": 96.32062370953902,
- "left_hip": 85.41059018197956,
- "left_knee": 44.80576890319366,
- "left_ankle": 65.48391017848762,
- "right_wrist": 66.5722903815099,
- "right_elbow": 57.09353851205642,
- "right_shoulder": 48.53982977836506,
- "left_shoulder": 50.84881824599475,
- "left_elbow": 66.00912240293543,
- "left_wrist": 69.57812835991473,
- "neck": 32.36462228153937,
- "head": 125.13900109740254
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.0422629175358,
- "PA_MPJPE_mm": 60.60910818210954,
- "PCK3D_50mm": 0.45418464314919776,
- "PCK3D_100mm": 0.8990896615251577,
- "AUC_0_150mm": 0.5993820911026758,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.044510337500874,
- "right_knee": 48.40323406335127,
- "right_hip": 95.2962010162751,
- "left_hip": 84.24703123433983,
- "left_knee": 45.679155794269384,
- "left_ankle": 60.4845529675519,
- "right_wrist": 63.892983973930775,
- "right_elbow": 48.895295959415655,
- "right_shoulder": 49.31735866265223,
- "left_shoulder": 50.452003432889654,
- "left_elbow": 53.75995537682707,
- "left_wrist": 66.83701536631068
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.09964348223286,
- "PA_MPJPE_mm": 40.19495769190858,
- "PCK3D_50mm": 0.7442948586252875,
- "PCK3D_100mm": 0.9540708424474603,
- "AUC_0_150mm": 0.7305278000527842
- }
- ]
- },
- "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 383,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 101.31907860715981,
- "PA_MPJPE_mm": 65.92076094787913,
- "PCK3D_50mm": 0.41219419837980026,
- "PCK3D_100mm": 0.8384694711993587,
- "AUC_0_150mm": 0.5660942000821002,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.6903209251555,
- "right_knee": 47.01265298534931,
- "right_hip": 97.72294203125473,
- "left_hip": 86.39189582344204,
- "left_knee": 44.41712361509261,
- "left_ankle": 66.019666747955,
- "right_wrist": 66.89448189028586,
- "right_elbow": 57.4295955371266,
- "right_shoulder": 48.69330961629026,
- "left_shoulder": 50.552388393906774,
- "left_elbow": 66.26333167825068,
- "left_wrist": 69.16763783575591,
- "neck": 32.757280783530646,
- "head": 125.87802540691196
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 98.86496385373847,
- "PA_MPJPE_mm": 60.78174406941384,
- "PCK3D_50mm": 0.4527700232166859,
- "PCK3D_100mm": 0.8985115942437658,
- "AUC_0_150mm": 0.5978022982608862,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.717435356785245,
- "right_knee": 48.921661399056866,
- "right_hip": 96.66106194582612,
- "left_hip": 85.28852718139414,
- "left_knee": 45.297720144508865,
- "left_ankle": 60.9673676727434,
- "right_wrist": 64.35083609401615,
- "right_elbow": 49.21560764617818,
- "right_shoulder": 49.372491864903104,
- "left_shoulder": 49.45322829091888,
- "left_elbow": 54.046665970783984,
- "left_wrist": 66.0883252658511
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.45170435888161,
- "PA_MPJPE_mm": 40.04693600698129,
- "PCK3D_50mm": 0.7440189171007207,
- "PCK3D_100mm": 0.9559722205437458,
- "AUC_0_150mm": 0.731086895136113
- }
- ]
- },
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 379,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 100.16115634013397,
- "PA_MPJPE_mm": 65.6939451488142,
- "PCK3D_50mm": 0.4118699167341414,
- "PCK3D_100mm": 0.8404957279417985,
- "AUC_0_150mm": 0.5676412788768875,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.55296979474068,
- "right_knee": 47.01340593081006,
- "right_hip": 96.19403979643175,
- "left_hip": 85.5832032005072,
- "left_knee": 44.494672153836746,
- "left_ankle": 65.18587521818144,
- "right_wrist": 65.34941612223491,
- "right_elbow": 57.99429055188764,
- "right_shoulder": 49.85304975319167,
- "left_shoulder": 51.17715388501612,
- "left_elbow": 67.23126144505112,
- "left_wrist": 68.64692330953368,
- "neck": 32.3681220847678,
- "head": 126.07084883720796
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 97.80456703844256,
- "PA_MPJPE_mm": 60.503021804691926,
- "PCK3D_50mm": 0.45542302305689497,
- "PCK3D_100mm": 0.900252845688934,
- "AUC_0_150mm": 0.5994155198557698,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.66083811529596,
- "right_knee": 48.916405929555026,
- "right_hip": 95.66666152680556,
- "left_hip": 84.65358638634301,
- "left_knee": 45.34322552211043,
- "left_ankle": 60.562082063133545,
- "right_wrist": 62.737138419126424,
- "right_elbow": 49.73746158930055,
- "right_shoulder": 49.797502920146826,
- "left_shoulder": 49.438782008248744,
- "left_elbow": 55.02572233426398,
- "left_wrist": 65.49685484197315
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 64.3578519319439,
- "PA_MPJPE_mm": 39.69392420489313,
- "PCK3D_50mm": 0.7481258132218288,
- "PCK3D_100mm": 0.9562703179571465,
- "AUC_0_150mm": 0.7333155436489851
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_b4mid_3dpw_test_detector_288.json b/benchmark/results/gen10_b4mid_3dpw_test_detector_288.json
deleted file mode 100644
index 8665093..0000000
--- a/benchmark/results/gen10_b4mid_3dpw_test_detector_288.json
+++ /dev/null
@@ -1,162 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 184,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 102.20045349669901,
- "PA_MPJPE_mm": 66.33197059362605,
- "PCK3D_50mm": 0.4079905204070078,
- "PCK3D_100mm": 0.835237475128201,
- "AUC_0_150mm": 0.5641032297723718,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.72331970283871,
- "right_knee": 47.65261623169293,
- "right_hip": 98.088218788668,
- "left_hip": 87.25298905346035,
- "left_knee": 44.73711055040664,
- "left_ankle": 66.7657173346979,
- "right_wrist": 67.47226724449584,
- "right_elbow": 57.277416975857065,
- "right_shoulder": 49.33282438048045,
- "left_shoulder": 50.74925922509534,
- "left_elbow": 66.45363465917728,
- "left_wrist": 69.64848728504178,
- "neck": 32.80838783223187,
- "head": 125.68533904662051
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 99.74689180524018,
- "PA_MPJPE_mm": 61.24716324396635,
- "PCK3D_50mm": 0.44835900049069566,
- "PCK3D_100mm": 0.8946254671045182,
- "AUC_0_150mm": 0.5955378657539484,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.839297958781366,
- "right_knee": 49.53952269279832,
- "right_hip": 97.15877584185019,
- "left_hip": 86.11108258649297,
- "left_knee": 45.70199350862173,
- "left_ankle": 61.68577591219942,
- "right_wrist": 64.58972498453979,
- "right_elbow": 48.91162832910641,
- "right_shoulder": 50.210205603260185,
- "left_shoulder": 49.89296659298869,
- "left_elbow": 54.07254683797817,
- "left_wrist": 66.25243807897895
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 65.24878583077343,
- "PA_MPJPE_mm": 40.54341420362009,
- "PCK3D_50mm": 0.7425263277092062,
- "PCK3D_100mm": 0.9534209845188217,
- "AUC_0_150mm": 0.7285377834784982
- }
- ]
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 192,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 102.34678425412685,
- "PA_MPJPE_mm": 66.99667535911469,
- "PCK3D_50mm": 0.39986371062976206,
- "PCK3D_100mm": 0.8326956985942379,
- "AUC_0_150mm": 0.5602119966999369,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.58266528331262,
- "right_knee": 47.44960120595661,
- "right_hip": 97.64388667164263,
- "left_hip": 86.72762390246551,
- "left_knee": 45.87825752724323,
- "left_ankle": 68.2567558004759,
- "right_wrist": 67.39941495347357,
- "right_elbow": 58.900895698740115,
- "right_shoulder": 49.20448925435452,
- "left_shoulder": 50.85058399804745,
- "left_elbow": 67.81161821035319,
- "left_wrist": 71.40079364875015,
- "neck": 33.88269235717518,
- "head": 126.96417651561498
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.04087990583882,
- "PA_MPJPE_mm": 61.78034830040261,
- "PCK3D_50mm": 0.4382030158909901,
- "PCK3D_100mm": 0.8925447287962858,
- "AUC_0_150mm": 0.5922182883366635,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.314249972115526,
- "right_knee": 49.362176817453324,
- "right_hip": 96.37263064814142,
- "left_hip": 85.3566659686558,
- "left_knee": 46.80763491334563,
- "left_ankle": 62.42424719027513,
- "right_wrist": 64.99955086590556,
- "right_elbow": 50.47290092588583,
- "right_shoulder": 50.53143050080936,
- "left_shoulder": 50.48220559828428,
- "left_elbow": 55.132237810738246,
- "left_wrist": 68.10824839322117
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 66.51728161754653,
- "PA_MPJPE_mm": 41.49439030046061,
- "PCK3D_50mm": 0.7280258666709805,
- "PCK3D_100mm": 0.9507154180888753,
- "AUC_0_150mm": 0.7224632069578941
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_b4mid_3dpw_test_gt-joints_288.json b/benchmark/results/gen10_b4mid_3dpw_test_gt-joints_288.json
deleted file mode 100644
index 11a33d8..0000000
--- a/benchmark/results/gen10_b4mid_3dpw_test_gt-joints_288.json
+++ /dev/null
@@ -1,152 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 184,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.42286501840562,
- "PA_MPJPE_mm": 66.4197633430155,
- "PCK3D_50mm": 0.4051607107609138,
- "PCK3D_100mm": 0.8337945786554195,
- "AUC_0_150mm": 0.5630635213880996,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.09995892203392,
- "right_knee": 47.123549768979125,
- "right_hip": 97.56622312762947,
- "left_hip": 87.31113227718082,
- "left_knee": 44.03031947441535,
- "left_ankle": 65.76737792323684,
- "right_wrist": 67.87953543582202,
- "right_elbow": 57.25069715967993,
- "right_shoulder": 49.89909674488968,
- "left_shoulder": 51.67848934122772,
- "left_elbow": 66.02595899969232,
- "left_wrist": 69.74789458262622,
- "neck": 31.52570326903292,
- "head": 129.9707497757706
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.00945353928743,
- "PA_MPJPE_mm": 60.89046252999226,
- "PCK3D_50mm": 0.4515715910479467,
- "PCK3D_100mm": 0.8963661656750228,
- "AUC_0_150mm": 0.5968361997162573,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.235585057702316,
- "right_knee": 49.02830219057866,
- "right_hip": 97.08427685558765,
- "left_hip": 86.51715752651755,
- "left_knee": 45.00207826163817,
- "left_ankle": 60.802082635753855,
- "right_wrist": 64.38373094478276,
- "right_elbow": 48.7176261215449,
- "right_shoulder": 49.68789198927227,
- "left_shoulder": 49.617944905764766,
- "left_elbow": 53.67020813913239,
- "left_wrist": 65.93866573163184
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.76083643344705,
- "PA_MPJPE_mm": 40.358764075096914,
- "PCK3D_50mm": 0.7401376082113752,
- "PCK3D_100mm": 0.9552773313030483,
- "AUC_0_150mm": 0.7288509046424836
- }
- ]
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 192,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.31762891912466,
- "PA_MPJPE_mm": 66.89167147001749,
- "PCK3D_50mm": 0.400393166318215,
- "PCK3D_100mm": 0.8324571686385408,
- "AUC_0_150mm": 0.5606019436627077,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.23792468154363,
- "right_knee": 47.346648926703686,
- "right_hip": 97.5270705108192,
- "left_hip": 86.75277201470011,
- "left_knee": 45.678030365385716,
- "left_ankle": 67.51228660772404,
- "right_wrist": 67.32768338336831,
- "right_elbow": 58.81746468272669,
- "right_shoulder": 49.052257549589086,
- "left_shoulder": 50.65046287257298,
- "left_elbow": 67.91687045191209,
- "left_wrist": 72.14533903288212,
- "neck": 33.42022341353992,
- "head": 127.09836608677733
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 99.95120758698395,
- "PA_MPJPE_mm": 61.68652071778024,
- "PCK3D_50mm": 0.4394204288037297,
- "PCK3D_100mm": 0.8918121234338137,
- "AUC_0_150mm": 0.5925182054050578,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.174471767495206,
- "right_knee": 49.18212790144903,
- "right_hip": 96.35587012557018,
- "left_hip": 85.40299315268881,
- "left_knee": 46.522595722051896,
- "left_ankle": 61.84350424175099,
- "right_wrist": 64.80348434781398,
- "right_elbow": 50.5741968978992,
- "right_shoulder": 50.11265189842603,
- "left_shoulder": 49.974322710782694,
- "left_elbow": 55.32728910535417,
- "left_wrist": 68.96474074208078
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.45504514351028,
- "PA_MPJPE_mm": 41.40204064288754,
- "PCK3D_50mm": 0.7281230739483002,
- "PCK3D_100mm": 0.9505540986380171,
- "AUC_0_150mm": 0.7228658908597018
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_b4mid_emdb1_annotated_288.json b/benchmark/results/gen10_b4mid_emdb1_annotated_288.json
deleted file mode 100644
index aac287b..0000000
--- a/benchmark/results/gen10_b4mid_emdb1_annotated_288.json
+++ /dev/null
@@ -1,348 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 184,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 88.5999015986422,
- "PA_MPJPE_mm": 65.19736744388115,
- "PCK3D_50mm": 0.5607946452032251,
- "PCK3D_100mm": 0.837397488556058,
- "AUC_0_150mm": 0.6100733567721682,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.44859738756545,
- "left_hip": 44.11322258602585,
- "right_hip": 43.04859843407978,
- "spine1": 51.92732604001735,
- "left_knee": 57.84360334905335,
- "right_knee": 56.15064153135521,
- "spine2": 52.20080260309749,
- "left_ankle": 68.94163108941588,
- "right_ankle": 69.74320078244183,
- "spine3": 51.97978719520583,
- "left_foot": 76.73376597163862,
- "right_foot": 78.13677720707092,
- "neck": 51.79071329439178,
- "left_collar": 46.188197867665444,
- "right_collar": 44.87430227864462,
- "head": 57.28682884045944,
- "left_shoulder": 49.19538969262912,
- "right_shoulder": 48.265903222356904,
- "left_elbow": 61.62163764215709,
- "right_elbow": 58.522546712572755,
- "left_wrist": 100.59269097174109,
- "right_wrist": 103.27620113344842,
- "left_hand": 121.12138180301955,
- "right_hand": 127.73307101709244
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 98.2745890071266,
- "PA_MPJPE_mm": 67.88090924151832,
- "PCK3D_50mm": 0.4879149602005678,
- "PCK3D_100mm": 0.8241860823489666,
- "AUC_0_150mm": 0.5810193235597382,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.94642876344594,
- "right_knee": 52.98069777920062,
- "right_hip": 49.92183097163062,
- "left_hip": 47.46243764456257,
- "left_knee": 55.92839748933326,
- "left_ankle": 71.09922858302397,
- "right_wrist": 104.28530128830309,
- "right_elbow": 53.662242503280346,
- "right_shoulder": 59.23703538692064,
- "left_shoulder": 51.998276276678055,
- "left_elbow": 57.92775815488624,
- "left_wrist": 104.34310123947719,
- "neck": 62.2758115563521,
- "head": 109.26418174416207
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 94.27749292940297,
- "PA_MPJPE_mm": 63.97434062678503,
- "PCK3D_50mm": 0.5381107469333002,
- "PCK3D_100mm": 0.8483141794244147,
- "AUC_0_150mm": 0.6065721975446772,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.81236095455142,
- "right_knee": 54.019286414933454,
- "right_hip": 50.00191969430879,
- "left_hip": 48.36297128548567,
- "left_knee": 56.82071542080945,
- "left_ankle": 69.41206005306327,
- "right_wrist": 99.00601234625906,
- "right_elbow": 50.26860402703479,
- "right_shoulder": 59.49647636464276,
- "left_shoulder": 56.57430128515137,
- "left_elbow": 55.7019294017041,
- "left_wrist": 99.21545027347628
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 83.66465111026757
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 71.37326182817792
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.688683394151425
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 58.00567019484552
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 59.334158185760536
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.28605944428783
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 54.51891609808384
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 82.33369358672651
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 56.6732976623086
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 61.884599556717625
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 57.44820197490315
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 59.475651727713405
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 57.2139181361469
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 62.02752043444365
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 89.31068610607244
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.73357758275769
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.293554917594854
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 192,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 81.213786000732,
- "PA_MPJPE_mm": 54.66406797793713,
- "PCK3D_50mm": 0.6158984358793511,
- "PCK3D_100mm": 0.8997825305287032,
- "AUC_0_150mm": 0.6536374582381437,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.42016838997137,
- "left_hip": 40.68612166861483,
- "right_hip": 40.049503664718436,
- "spine1": 46.00617803838946,
- "left_knee": 55.720877225261766,
- "right_knee": 56.39166250470147,
- "spine2": 44.1018375915778,
- "left_ankle": 63.977606025007866,
- "right_ankle": 65.39916801138418,
- "spine3": 41.47311284550179,
- "left_foot": 71.76264930196912,
- "right_foot": 72.39268173497406,
- "neck": 34.68861913820637,
- "left_collar": 29.88688581237687,
- "right_collar": 30.422205953545145,
- "head": 40.78393959831271,
- "left_shoulder": 40.53548876359596,
- "right_shoulder": 40.47230808873917,
- "left_elbow": 52.69419304380265,
- "right_elbow": 49.31996968295051,
- "left_wrist": 82.70060058251639,
- "right_wrist": 79.08588086964544,
- "left_hand": 97.10981021941835,
- "right_hand": 96.85616271530888
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 92.20362226881441,
- "PA_MPJPE_mm": 60.688215372751245,
- "PCK3D_50mm": 0.5166102619116767,
- "PCK3D_100mm": 0.8678646997113578,
- "AUC_0_150mm": 0.6095134915419635,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.64216443589855,
- "right_knee": 53.51626363672083,
- "right_hip": 51.05003371216731,
- "left_hip": 48.334843010202285,
- "left_knee": 55.012783626379424,
- "left_ankle": 67.39842216654294,
- "right_wrist": 80.96253393309217,
- "right_elbow": 44.997586619356866,
- "right_shoulder": 53.41865827306896,
- "left_shoulder": 42.96311589159333,
- "left_elbow": 50.20080593841506,
- "left_wrist": 87.45901562474427,
- "neck": 43.64372191371885,
- "head": 104.0350664366165
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 87.06530323971708,
- "PA_MPJPE_mm": 56.30674639689889,
- "PCK3D_50mm": 0.5714053575626824,
- "PCK3D_100mm": 0.8979553029360108,
- "AUC_0_150mm": 0.6389601927926698,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.0496263454732,
- "right_knee": 53.82770495276158,
- "right_hip": 48.456754576109084,
- "left_hip": 46.53011186175702,
- "left_knee": 55.474762514319465,
- "left_ankle": 64.50193288932718,
- "right_wrist": 78.54652429503486,
- "right_elbow": 41.9418875667631,
- "right_shoulder": 47.812210117983064,
- "left_shoulder": 43.51188952546321,
- "left_elbow": 47.14146152256496,
- "left_wrist": 83.88609059522997
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 46.118911177932524
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 52.581662613497286
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.978697360698405
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.97475169245756
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.958131549708526
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 70.62522256586392
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 50.7199963718349
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 63.88502584845409
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 49.83385136617413
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.27300598577904
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 48.9396493159645
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.555973971143246
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 49.514208947547054
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.98786571913443
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 74.42394715552393
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 61.25258106314955
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.547785564228526
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_b4mid_emdb1_detector_288.json b/benchmark/results/gen10_b4mid_emdb1_detector_288.json
deleted file mode 100644
index a573b9b..0000000
--- a/benchmark/results/gen10_b4mid_emdb1_detector_288.json
+++ /dev/null
@@ -1,358 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 184,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 78.16762011732557,
- "PA_MPJPE_mm": 51.74323745707022,
- "PCK3D_50mm": 0.6485766854127613,
- "PCK3D_100mm": 0.9122960494341457,
- "AUC_0_150mm": 0.6693270667177313,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.2498439891717,
- "left_hip": 39.98407633787233,
- "right_hip": 39.421264207060524,
- "spine1": 44.618931225143214,
- "left_knee": 52.55010662295095,
- "right_knee": 52.126981273688294,
- "spine2": 43.36147491398545,
- "left_ankle": 57.361750663870296,
- "right_ankle": 57.857956693433515,
- "spine3": 41.06629271802394,
- "left_foot": 64.91331782817348,
- "right_foot": 64.95966860689485,
- "neck": 34.06492989863523,
- "left_collar": 29.676755196932874,
- "right_collar": 29.89465323852864,
- "head": 39.26350404969212,
- "left_shoulder": 39.095638724535604,
- "right_shoulder": 39.325626698156725,
- "left_elbow": 51.50366020414947,
- "right_elbow": 47.490139563469185,
- "left_wrist": 76.82620909548672,
- "right_wrist": 74.16739020609778,
- "left_hand": 91.73303628191104,
- "right_hand": 92.32449073182134
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 89.24273413530881,
- "PA_MPJPE_mm": 57.7382071831398,
- "PCK3D_50mm": 0.5524920403487369,
- "PCK3D_100mm": 0.8836641175945488,
- "AUC_0_150mm": 0.6261748464461608,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.92767309672148,
- "right_knee": 49.56761952783651,
- "right_hip": 49.0621122488843,
- "left_hip": 46.198117802348634,
- "left_knee": 51.98259800863033,
- "left_ankle": 61.27480481800226,
- "right_wrist": 75.4683454144404,
- "right_elbow": 42.70832786998944,
- "right_shoulder": 52.31707539270207,
- "left_shoulder": 43.13809782119893,
- "left_elbow": 47.846437424758115,
- "left_wrist": 81.00222071459237,
- "neck": 44.47260494174515,
- "head": 103.36886548210708
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 83.28739971284743,
- "PA_MPJPE_mm": 52.826303790149524,
- "PCK3D_50mm": 0.6160522113448587,
- "PCK3D_100mm": 0.9133617996250781,
- "AUC_0_150mm": 0.6588814382308346,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.32034795547459,
- "right_knee": 50.18647744854045,
- "right_hip": 46.36817768952671,
- "left_hip": 44.342571125642465,
- "left_knee": 52.641397263057385,
- "left_ankle": 58.143894857405265,
- "right_wrist": 73.41148506459481,
- "right_elbow": 40.23645708642926,
- "right_shoulder": 46.02793489360532,
- "left_shoulder": 42.16918167898673,
- "left_elbow": 45.35931012583006,
- "left_wrist": 77.70841029270119
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 44.78969851363475
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.83464400435064
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 52.88557514054094
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.197121444079706
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.33578189184168
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 71.799514527021
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.36011645696588
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 59.17150999320814
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 46.565869916995645
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 41.760248535115345
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 46.82861820703194
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 48.218679009779834
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.166594379557
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.034942888603794
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 69.87073846264678
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.63844822427287
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.320761330594024
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 192,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 79.17012128851799,
- "PA_MPJPE_mm": 53.162638546812225,
- "PCK3D_50mm": 0.6309709782684163,
- "PCK3D_100mm": 0.9068440602652226,
- "AUC_0_150mm": 0.6614987804959989,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.90222051748616,
- "left_hip": 40.39632259986049,
- "right_hip": 39.73234191968821,
- "spine1": 45.61360372816786,
- "left_knee": 53.7935057908784,
- "right_knee": 54.658749591261994,
- "spine2": 43.76258101426058,
- "left_ankle": 61.17863474347928,
- "right_ankle": 62.83852065426702,
- "spine3": 40.91589619575921,
- "left_foot": 69.30543523979158,
- "right_foot": 70.01071787420872,
- "neck": 34.124129569235514,
- "left_collar": 29.410541507815296,
- "right_collar": 29.832069315492024,
- "head": 39.77531726727477,
- "left_shoulder": 39.91374956062693,
- "right_shoulder": 39.75027104578926,
- "left_elbow": 51.3080125096544,
- "right_elbow": 48.20790177847495,
- "left_wrist": 79.28398480714834,
- "right_wrist": 76.09922364725647,
- "left_hand": 93.73985992425982,
- "right_hand": 93.34973432135618
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 90.53288806282629,
- "PA_MPJPE_mm": 59.20472687320279,
- "PCK3D_50mm": 0.533350194899872,
- "PCK3D_100mm": 0.875627101496712,
- "AUC_0_150mm": 0.6176921666665068,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.36308293491288,
- "right_knee": 51.9178527476561,
- "right_hip": 50.63133445478744,
- "left_hip": 48.11447571950113,
- "left_knee": 53.29159266236018,
- "left_ankle": 64.75670954780378,
- "right_wrist": 77.98031203760941,
- "right_elbow": 43.87060372692679,
- "right_shoulder": 52.65906066570793,
- "left_shoulder": 42.20340475375346,
- "left_elbow": 48.37948395466859,
- "left_wrist": 83.66651834544545,
- "neck": 43.373512423154004,
- "head": 103.658232250552
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 84.95833820891082,
- "PA_MPJPE_mm": 54.58380162830657,
- "PCK3D_50mm": 0.5927549816010553,
- "PCK3D_100mm": 0.9053704089425814,
- "AUC_0_150mm": 0.6486400682653479,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.6489256782715,
- "right_knee": 52.19607051022766,
- "right_hip": 47.92419724557667,
- "left_hip": 46.15805015824043,
- "left_knee": 53.69840524159013,
- "left_ankle": 61.851275153841975,
- "right_wrist": 75.63081292659182,
- "right_elbow": 41.011506300908884,
- "right_shoulder": 46.6369416807779,
- "left_shoulder": 42.29196894991116,
- "left_elbow": 45.64245513968905,
- "left_wrist": 80.31501055405158
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 45.068321956327026
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.43997920833623
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 52.1518744120853
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.1996122210565
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.4002382232012
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.28738044193197
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.242615415725496
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 62.89322077297734
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 47.51535594292225
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 42.61063162460898
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 48.19039415576344
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 50.97261768892995
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.04133227810523
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.55347128664571
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 73.38811677666534
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.46890748658437
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.767677778164625
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_b4mid_emdb1_gt-joints_288.json b/benchmark/results/gen10_b4mid_emdb1_gt-joints_288.json
deleted file mode 100644
index 453b625..0000000
--- a/benchmark/results/gen10_b4mid_emdb1_gt-joints_288.json
+++ /dev/null
@@ -1,348 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 184,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.6415235752167,
- "PA_MPJPE_mm": 51.069633226419825,
- "PCK3D_50mm": 0.6561683192963531,
- "PCK3D_100mm": 0.9161722607144339,
- "AUC_0_150mm": 0.6726036646488925,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.13041808491286,
- "left_hip": 39.169572186526814,
- "right_hip": 38.40084825682526,
- "spine1": 43.21307865687097,
- "left_knee": 51.870072129387005,
- "right_knee": 51.121785234371316,
- "spine2": 43.33926592051827,
- "left_ankle": 56.39700056761229,
- "right_ankle": 56.31371015734757,
- "spine3": 41.48338570104681,
- "left_foot": 63.53795531754066,
- "right_foot": 63.12638743532176,
- "neck": 33.56052355469663,
- "left_collar": 29.109671089916443,
- "right_collar": 29.632675786633097,
- "head": 38.924586169768105,
- "left_shoulder": 38.50323347492888,
- "right_shoulder": 39.495125995117164,
- "left_elbow": 50.85894782686017,
- "right_elbow": 47.313512584085764,
- "left_wrist": 76.364954208997,
- "right_wrist": 73.45178230419674,
- "left_hand": 91.8099035717785,
- "right_hand": 91.5428012188152
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 87.23983439822314,
- "PA_MPJPE_mm": 57.36432865238798,
- "PCK3D_50mm": 0.5538403636773135,
- "PCK3D_100mm": 0.8850469117655775,
- "AUC_0_150mm": 0.6277049092881147,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.15546395276529,
- "right_knee": 48.99553103720511,
- "right_hip": 48.46414243911828,
- "left_hip": 45.714235572312674,
- "left_knee": 51.708344812156554,
- "left_ankle": 60.259524573389506,
- "right_wrist": 74.31575718755879,
- "right_elbow": 42.27639313452701,
- "right_shoulder": 53.20397956699581,
- "left_shoulder": 43.269841913642765,
- "left_elbow": 47.29892870716698,
- "left_wrist": 80.51548583073658,
- "neck": 44.58093604305017,
- "head": 104.34203636280625
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 81.70909538448277,
- "PA_MPJPE_mm": 52.1244645479706,
- "PCK3D_50mm": 0.6249809843864526,
- "PCK3D_100mm": 0.9175102684313156,
- "AUC_0_150mm": 0.6625731236775504,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 55.65958357480698,
- "right_knee": 49.54802258346513,
- "right_hip": 45.41998289720122,
- "left_hip": 43.504562535544004,
- "left_knee": 52.23957248248953,
- "left_ankle": 57.10411658569907,
- "right_wrist": 72.50783563038149,
- "right_elbow": 39.88004625917208,
- "right_shoulder": 46.10502311036497,
- "left_shoulder": 41.30151213458978,
- "left_elbow": 44.876791823688684,
- "left_wrist": 77.34652495824413
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.90061238288795
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.74561728746214
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 50.91276448609074
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.799879869906384
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.1311262051114
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.77702579095806
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.9381452477994
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 57.661835976984996
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.971165234860635
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.859822237789274
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.90246529397492
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 48.626790711511156
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.4445419451792
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.10648345594927
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 68.78472910786144
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.06224649539653
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.22385104556175
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 192,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 79.58365603538039,
- "PA_MPJPE_mm": 53.07121295209272,
- "PCK3D_50mm": 0.6317073255058153,
- "PCK3D_100mm": 0.9073559307969963,
- "AUC_0_150mm": 0.6617499316330142,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.78246166240224,
- "left_hip": 40.21315124071172,
- "right_hip": 39.43816709036744,
- "spine1": 45.52151445121305,
- "left_knee": 53.60406292847368,
- "right_knee": 54.49514310783855,
- "spine2": 43.98976060399271,
- "left_ankle": 60.81831956167732,
- "right_ankle": 62.231982591956395,
- "spine3": 41.20245730701446,
- "left_foot": 68.19577400264957,
- "right_foot": 69.24503398597467,
- "neck": 34.10780811096899,
- "left_collar": 29.395319476833908,
- "right_collar": 29.99783474874383,
- "head": 39.80687870110729,
- "left_shoulder": 39.42033542207971,
- "right_shoulder": 39.84231521268835,
- "left_elbow": 51.136799672939375,
- "right_elbow": 48.23378543684173,
- "left_wrist": 79.76985759733225,
- "right_wrist": 76.21999620203182,
- "left_hand": 94.51083889081279,
- "right_hand": 93.52951284357128
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 90.62061575431098,
- "PA_MPJPE_mm": 59.14858519188725,
- "PCK3D_50mm": 0.5325507790968522,
- "PCK3D_100mm": 0.8766573218508662,
- "AUC_0_150mm": 0.6179281275769065,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.56332153084,
- "right_knee": 51.59273660392156,
- "right_hip": 50.75068173360921,
- "left_hip": 48.09563350198529,
- "left_knee": 53.059997323937964,
- "left_ankle": 64.29681325399567,
- "right_wrist": 77.8691848810783,
- "right_elbow": 43.781914485553386,
- "right_shoulder": 53.08756918759728,
- "left_shoulder": 42.155640240045095,
- "left_elbow": 48.383500156462254,
- "left_wrist": 84.17387817324303,
- "neck": 43.45069463642368,
- "head": 103.81862697772873
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.3370749781246,
- "PA_MPJPE_mm": 54.45913919549979,
- "PCK3D_50mm": 0.5930762422381722,
- "PCK3D_100mm": 0.9070897122073324,
- "AUC_0_150mm": 0.6492248321384166,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.966870139455935,
- "right_knee": 51.938806300851816,
- "right_hip": 47.939030546319465,
- "left_hip": 46.00113450142451,
- "left_knee": 53.522377629095764,
- "left_ankle": 61.26110751150672,
- "right_wrist": 75.68361961005694,
- "right_elbow": 40.95390200378828,
- "right_shoulder": 46.87843964213239,
- "left_shoulder": 41.9634223859373,
- "left_elbow": 45.58170034038195,
- "left_wrist": 80.8192597350464
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.910986918042674
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.45334510126923
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.540335807123256
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.35743755069831
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.789105009096176
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.78584796387995
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.43123470790272
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.68383972204025
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.25772676026624
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.94774139270611
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 48.0785581843615
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 51.07283733753988
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.816280879066866
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.08484754930317
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 72.46227195806664
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.82407490514476
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.963540147346066
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_emdb1_annotated_288.json b/benchmark/results/gen10_emdb1_annotated_288.json
deleted file mode 100644
index d6c8642..0000000
--- a/benchmark/results/gen10_emdb1_annotated_288.json
+++ /dev/null
@@ -1,1386 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 94.45962391862395,
- "PA_MPJPE_mm": 69.15292253682375,
- "PCK3D_50mm": 0.5166248323168624,
- "PCK3D_100mm": 0.8260296090389855,
- "AUC_0_150mm": 0.5911587880685892,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 45.155750894726246,
- "left_hip": 45.92466882934077,
- "right_hip": 44.5963185388058,
- "spine1": 53.171757645132075,
- "left_knee": 65.3149763662308,
- "right_knee": 64.04391643763643,
- "spine2": 51.61026554328148,
- "left_ankle": 79.1514948525433,
- "right_ankle": 82.37265372976736,
- "spine3": 50.65689678059247,
- "left_foot": 87.38835131312726,
- "right_foot": 92.25452341917591,
- "neck": 48.33327900958358,
- "left_collar": 43.39260223263352,
- "right_collar": 42.84741181813571,
- "head": 54.868397531539784,
- "left_shoulder": 48.03431129562065,
- "right_shoulder": 47.786359208878736,
- "left_elbow": 65.11147019576173,
- "right_elbow": 61.46114934696543,
- "left_wrist": 107.92868811082947,
- "right_wrist": 111.51598173959265,
- "left_hand": 128.8253427104907,
- "right_hand": 137.92357333338023
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 104.52467783440706,
- "PA_MPJPE_mm": 72.21784617379674,
- "PCK3D_50mm": 0.4397674266985141,
- "PCK3D_100mm": 0.8073831947416149,
- "AUC_0_150mm": 0.5592186834114956,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 82.32898254783564,
- "right_knee": 60.96510225332543,
- "right_hip": 53.76757171910138,
- "left_hip": 51.3751696534623,
- "left_knee": 63.61122492003102,
- "left_ankle": 81.10471839437476,
- "right_wrist": 112.68288294641313,
- "right_elbow": 56.838273132568425,
- "right_shoulder": 59.91995339629147,
- "left_shoulder": 50.41881278534153,
- "left_elbow": 61.83311990212367,
- "left_wrist": 111.66606808221877,
- "neck": 58.28885396113405,
- "head": 106.24911273893282
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 101.24713166622197,
- "PA_MPJPE_mm": 69.51310294274248,
- "PCK3D_50mm": 0.482211757872464,
- "PCK3D_100mm": 0.8254297528661716,
- "AUC_0_150mm": 0.578853009418807,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 81.3959894355432,
- "right_knee": 61.6541066344525,
- "right_hip": 53.293696246182186,
- "left_hip": 51.574257433962906,
- "left_knee": 64.1928639729915,
- "left_ankle": 79.62756454770508,
- "right_wrist": 107.84503869837022,
- "right_elbow": 53.52729868238552,
- "right_shoulder": 58.79101899472403,
- "left_shoulder": 55.03952557462917,
- "left_elbow": 59.91430184113585,
- "left_wrist": 107.30157325082752
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 74.80671540200073
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 73.50287530670971
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 61.6643237128329
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 63.998835565703665
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 62.79843672749021
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 98.89548498561267
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 60.69330308946684
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 78.84682686029183
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 63.993226251354315
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 66.63359625567523
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 59.479638396332106
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 67.71615079076409
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 58.80598408365355
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 58.785595803193594
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 90.34482299108159
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 72.46619026140407
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 68.1662695405488
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 524,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 90.97928089553483,
- "PA_MPJPE_mm": 66.54871519969497,
- "PCK3D_50mm": 0.5368488016706081,
- "PCK3D_100mm": 0.836204690978993,
- "AUC_0_150mm": 0.6024887478870921,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.99952568574835,
- "left_hip": 44.74100433064114,
- "right_hip": 43.72406709606072,
- "spine1": 52.692258588750846,
- "left_knee": 61.40391340618424,
- "right_knee": 58.379794819336055,
- "spine2": 51.57522805988803,
- "left_ankle": 73.68118279535281,
- "right_ankle": 76.15973808977502,
- "spine3": 50.558309778347734,
- "left_foot": 82.20764861089557,
- "right_foot": 84.18609405039145,
- "neck": 49.90290685046982,
- "left_collar": 44.16767627212506,
- "right_collar": 43.186057642569814,
- "head": 56.37376223303643,
- "left_shoulder": 48.275391725514595,
- "right_shoulder": 47.122178687197426,
- "left_elbow": 62.40583386689923,
- "right_elbow": 59.40306383348312,
- "left_wrist": 104.6724497768617,
- "right_wrist": 104.04278668172861,
- "left_hand": 125.9012607145247,
- "right_hand": 128.40703119689533
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 102.17118532664738,
- "PA_MPJPE_mm": 69.73218619747206,
- "PCK3D_50mm": 0.4607725179438244,
- "PCK3D_100mm": 0.8165462509112678,
- "AUC_0_150mm": 0.5702395498887168,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 76.84525675084822,
- "right_knee": 55.51119146350326,
- "right_hip": 52.45342372403111,
- "left_hip": 50.55239471091257,
- "left_knee": 59.849320921299096,
- "left_ankle": 75.85826618441708,
- "right_wrist": 105.46279776914496,
- "right_elbow": 54.64760302453113,
- "right_shoulder": 59.09750017497527,
- "left_shoulder": 50.175746131867506,
- "left_elbow": 58.922645238358626,
- "left_wrist": 108.61952841071755,
- "neck": 59.83819699471202,
- "head": 108.4167352652905
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 97.9976307168935,
- "PA_MPJPE_mm": 66.25034197667475,
- "PCK3D_50mm": 0.5066070613616562,
- "PCK3D_100mm": 0.8405074057171307,
- "AUC_0_150mm": 0.5938289482547793,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 75.52130773991777,
- "right_knee": 56.15140793210517,
- "right_hip": 51.95660474257117,
- "left_hip": 50.87028585816059,
- "left_knee": 60.67208746706091,
- "left_ankle": 74.4707952166475,
- "right_wrist": 100.80061227972597,
- "right_elbow": 51.79346766985979,
- "right_shoulder": 57.93198249841685,
- "left_shoulder": 54.92384234703191,
- "left_elbow": 56.3897930129963,
- "left_wrist": 103.521916955603
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 72.84519701035593
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 78.32775090133048
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 61.5092587142006
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 61.099453470951815
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 60.519080069000054
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 89.1240004652546
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 57.701284160444935
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 80.72348386978729
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 55.85185397563299
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 61.716819404303834
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 57.33438452824986
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 62.81000581435473
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 59.66693026725339
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 62.10707816089662
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 86.30387416012319
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 65.32626363657965
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 62.968794771251375
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 492,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 88.4237206093721,
- "PA_MPJPE_mm": 64.49928273760551,
- "PCK3D_50mm": 0.5584384378154863,
- "PCK3D_100mm": 0.8421790510171625,
- "AUC_0_150mm": 0.6124775771900076,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.52371731507839,
- "left_hip": 44.12068164093575,
- "right_hip": 43.08134669154997,
- "spine1": 52.55523257596412,
- "left_knee": 56.50660547841881,
- "right_knee": 54.95418650269281,
- "spine2": 51.48934666200081,
- "left_ankle": 69.15003849285623,
- "right_ankle": 69.9722927938743,
- "spine3": 50.54923648140238,
- "left_foot": 76.30418106187233,
- "right_foot": 78.56273942441857,
- "neck": 51.547178275322985,
- "left_collar": 44.81807410755135,
- "right_collar": 43.80799547036015,
- "head": 57.1599692177262,
- "left_shoulder": 48.57618907374005,
- "right_shoulder": 47.17249127187744,
- "left_elbow": 61.19190848983235,
- "right_elbow": 58.137567434548735,
- "left_wrist": 99.20526505783319,
- "right_wrist": 100.90881685104604,
- "left_hand": 120.20910938742672,
- "right_hand": 124.4786159442024
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 99.46382445873705,
- "PA_MPJPE_mm": 67.48196858665513,
- "PCK3D_50mm": 0.48615169421708027,
- "PCK3D_100mm": 0.8256263298581682,
- "AUC_0_150mm": 0.582083974861343,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.79790335679944,
- "right_knee": 52.34325431012094,
- "right_hip": 51.210053033548895,
- "left_hip": 49.222096547901415,
- "left_knee": 55.04606949292538,
- "left_ankle": 71.85519626232688,
- "right_wrist": 102.07798173196775,
- "right_elbow": 53.57639105925462,
- "right_shoulder": 57.72598971402876,
- "left_shoulder": 50.14326274246303,
- "left_elbow": 57.31872168031749,
- "left_wrist": 102.68016755544643,
- "neck": 61.27730542191245,
- "head": 109.47316730415821
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 94.83944200552573,
- "PA_MPJPE_mm": 63.52272209136361,
- "PCK3D_50mm": 0.5377753806580093,
- "PCK3D_100mm": 0.8503367492290033,
- "AUC_0_150mm": 0.6078333621076929,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.41856770719906,
- "right_knee": 53.305937201098295,
- "right_hip": 50.71927171119947,
- "left_hip": 49.56533800393746,
- "left_knee": 55.84064391239195,
- "left_ankle": 69.71467155136895,
- "right_wrist": 97.63651415220158,
- "right_elbow": 50.83877932603055,
- "right_shoulder": 57.49958627618545,
- "left_shoulder": 54.66759607668704,
- "left_elbow": 55.167171329519846,
- "left_wrist": 97.89858784854351
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 75.44427646628932
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 73.40568200645237
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 60.968901908126114
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 57.350992690864715
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 60.839947887905026
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.17117798793748
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 52.7820096850376
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 82.89485308425287
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 59.028369088521615
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 64.03737578353947
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 58.942995221524434
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 58.64376204308768
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.72075070677768
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 61.30858314022962
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 81.1044272886978
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.120010045628234
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.350524176891334
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 484,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 83.65941554154135,
- "PA_MPJPE_mm": 60.740529146271086,
- "PCK3D_50mm": 0.5961619577092755,
- "PCK3D_100mm": 0.8610580978854637,
- "AUC_0_150mm": 0.6315353485199496,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.92869294726276,
- "left_hip": 42.103597911798474,
- "right_hip": 41.14006162296645,
- "spine1": 49.59269767538443,
- "left_knee": 53.31049394213209,
- "right_knee": 51.90519344779513,
- "spine2": 50.5397900304624,
- "left_ankle": 61.163789005619755,
- "right_ankle": 61.789886096418485,
- "spine3": 49.85512805526686,
- "left_foot": 68.45523513942334,
- "right_foot": 70.41017383934839,
- "neck": 48.73922618158494,
- "left_collar": 42.636832518083295,
- "right_collar": 42.16057613958167,
- "head": 53.84078990384543,
- "left_shoulder": 47.00700314351727,
- "right_shoulder": 47.201793370744426,
- "left_elbow": 58.18007073436363,
- "right_elbow": 55.26427022581382,
- "left_wrist": 94.19329545513054,
- "right_wrist": 96.09118061562911,
- "left_hand": 112.47577652406522,
- "right_hand": 118.78714498426798
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 95.16908444246708,
- "PA_MPJPE_mm": 64.58062809989676,
- "PCK3D_50mm": 0.5117560943806639,
- "PCK3D_100mm": 0.8428618844127287,
- "AUC_0_150mm": 0.596817307289702,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.614014024758845,
- "right_knee": 49.24136479500793,
- "right_hip": 50.87690918275486,
- "left_hip": 47.87197874594562,
- "left_knee": 52.217715588931654,
- "left_ankle": 62.80668766524385,
- "right_wrist": 97.7013795682189,
- "right_elbow": 51.25066582461683,
- "right_shoulder": 58.15796188030236,
- "left_shoulder": 49.36363791181911,
- "left_elbow": 55.03120979362426,
- "left_wrist": 98.81022347691763,
- "neck": 59.70839006367502,
- "head": 108.47665487673774
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 90.29272386594673,
- "PA_MPJPE_mm": 60.29603810348175,
- "PCK3D_50mm": 0.5677509023772974,
- "PCK3D_100mm": 0.8692763003222282,
- "AUC_0_150mm": 0.6245059843976054,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.87381358656977,
- "right_knee": 50.20566350592132,
- "right_hip": 50.24896571412658,
- "left_hip": 48.01528206034561,
- "left_knee": 53.053144502297634,
- "left_ankle": 61.26860200297716,
- "right_wrist": 93.09660863616183,
- "right_elbow": 48.59003203056994,
- "right_shoulder": 57.33548368137759,
- "left_shoulder": 53.10518126996541,
- "left_elbow": 53.42392703132124,
- "left_wrist": 94.33575322014683
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 75.84960361956114
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 64.61311383441941
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.74437763569734
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.928249528549735
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 55.72001198534987
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 74.38983429866022
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.24837583776664
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 76.9033689658453
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 52.71433574042551
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 58.7768587207741
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.71111379075149
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 58.57933762155924
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.53151477290587
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 60.47946290103302
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 81.68396455591157
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.41600574868248
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.13567563946509
- }
- }
- },
- "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 159,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 116.04602798005325,
- "PA_MPJPE_mm": 73.8828711461943,
- "PCK3D_50mm": 0.49118367008256236,
- "PCK3D_100mm": 0.8116054018172012,
- "AUC_0_150mm": 0.5760374160357499,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.18836966373432,
- "left_hip": 48.67646860405237,
- "right_hip": 47.69495804379468,
- "spine1": 55.41150198876239,
- "left_knee": 76.95766122860559,
- "right_knee": 71.00791711805256,
- "spine2": 51.01542272057889,
- "left_ankle": 89.56041543495908,
- "right_ankle": 90.64716740160712,
- "spine3": 48.88531654387884,
- "left_foot": 99.48405534594477,
- "right_foot": 99.54794909727276,
- "neck": 44.74798443061487,
- "left_collar": 39.278214587817914,
- "right_collar": 40.05967743058107,
- "head": 53.22908609322195,
- "left_shoulder": 47.636392338467125,
- "right_shoulder": 47.92403285296371,
- "left_elbow": 70.27019845118876,
- "right_elbow": 66.47295839054654,
- "left_wrist": 123.41620451458564,
- "right_wrist": 120.34663119208864,
- "left_hand": 145.6967982631811,
- "right_hand": 147.0335257721649
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 126.25804854008744,
- "PA_MPJPE_mm": 77.77478506580454,
- "PCK3D_50mm": 0.4015267808986433,
- "PCK3D_100mm": 0.7856401989082569,
- "AUC_0_150mm": 0.5377242368628797,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 90.67438445124493,
- "right_knee": 67.86265134680455,
- "right_hip": 58.80723690901898,
- "left_hip": 56.9506060870034,
- "left_knee": 74.74615562903753,
- "left_ankle": 91.83980840007226,
- "right_wrist": 122.19678827938482,
- "right_elbow": 61.554044955975606,
- "right_shoulder": 60.46940768967485,
- "left_shoulder": 50.85491477302783,
- "left_elbow": 67.82859205464682,
- "left_wrist": 127.25641328011103,
- "neck": 52.43681519969391,
- "head": 105.36917186556713
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 122.74619725516156,
- "PA_MPJPE_mm": 75.99473020804712,
- "PCK3D_50mm": 0.43552669792142057,
- "PCK3D_100mm": 0.8023205963296408,
- "AUC_0_150mm": 0.5544584866292912,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 89.1279808279554,
- "right_knee": 68.34733505509116,
- "right_hip": 57.2405116055678,
- "left_hip": 56.02170773911921,
- "left_knee": 75.21348277330083,
- "left_ankle": 89.5533287982869,
- "right_wrist": 118.7448353089064,
- "right_elbow": 56.65501291708321,
- "right_shoulder": 59.96904448835669,
- "left_shoulder": 56.079635479651905,
- "left_elbow": 62.401745551510615,
- "left_wrist": 122.58214195173522
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 56.07070124093597
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 69.73515833054432
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 61.91152952974689
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 59.38219151519477
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 56.8467293582546
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 104.57609311151433
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 64.84845675748757
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 81.09336936020148
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 78.29072849059622
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 55.191562086320445
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 69.72410491613105
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 77.02969851997423
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 68.44463870603963
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 62.495544324051615
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 108.73047134984658
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 80.70464280529114
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 92.09982155156052
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 376,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 80.11871139566982,
- "PA_MPJPE_mm": 53.37056522421867,
- "PCK3D_50mm": 0.630606148612206,
- "PCK3D_100mm": 0.9062149939841514,
- "AUC_0_150mm": 0.6606093294057447,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.81581902027198,
- "left_hip": 40.853428659865976,
- "right_hip": 40.07286061743478,
- "spine1": 45.188250823148685,
- "left_knee": 54.66357547035274,
- "right_knee": 54.50708819375279,
- "spine2": 43.44733414554974,
- "left_ankle": 60.94507930775727,
- "right_ankle": 61.97617559793488,
- "spine3": 40.7090225131446,
- "left_foot": 68.65059585315652,
- "right_foot": 68.59596014763231,
- "neck": 34.92458981525891,
- "left_collar": 29.84260113630748,
- "right_collar": 30.0681893958741,
- "head": 40.34670382339373,
- "left_shoulder": 40.40651392950457,
- "right_shoulder": 40.363213966136755,
- "left_elbow": 51.54290362050678,
- "right_elbow": 48.35164736130119,
- "left_wrist": 80.18941535281044,
- "right_wrist": 76.63753756535834,
- "left_hand": 95.18192296958733,
- "right_hand": 94.61313609520585
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 92.02587243408509,
- "PA_MPJPE_mm": 59.57220473409454,
- "PCK3D_50mm": 0.5320914408994731,
- "PCK3D_100mm": 0.8726240361306535,
- "AUC_0_150mm": 0.6155420544433824,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.781734256994184,
- "right_knee": 51.27933101263981,
- "right_hip": 52.26511527117973,
- "left_hip": 48.72806600901015,
- "left_knee": 54.01246657181446,
- "left_ankle": 64.59712450754955,
- "right_wrist": 78.71571548243503,
- "right_elbow": 44.31324752804424,
- "right_shoulder": 52.194670953843485,
- "left_shoulder": 42.06270359791605,
- "left_elbow": 49.00560056370773,
- "left_wrist": 85.23101870307887,
- "neck": 44.02166773223272,
- "head": 103.80240408687747
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.4526659350975,
- "PA_MPJPE_mm": 55.0642278472783,
- "PCK3D_50mm": 0.5878417624362112,
- "PCK3D_100mm": 0.9037118477644553,
- "AUC_0_150mm": 0.645813843723554,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.80311521458449,
- "right_knee": 51.47618990295472,
- "right_hip": 49.68798772596861,
- "left_hip": 46.98312705036054,
- "left_knee": 54.25580553730022,
- "left_ankle": 61.60037990838713,
- "right_wrist": 76.63553409999122,
- "right_elbow": 42.04757093255615,
- "right_shoulder": 46.85168988421369,
- "left_shoulder": 42.80957905718771,
- "left_elbow": 46.29465868498036,
- "left_wrist": 81.32509616885481
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.99653966453705
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 51.22802653293374
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.10332012780033
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.27149531652043
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.59622076052561
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 70.80625460473064
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.52822805360342
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.33441056946079
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.37130320805703
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.20260200551605
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.762865208170986
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 52.4873895307563
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.48090033735151
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.71328233032176
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 72.99671160163554
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 57.327485975911316
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.069861558433495
- }
- }
- },
- "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 383,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 79.1364529277748,
- "PA_MPJPE_mm": 52.541201734736084,
- "PCK3D_50mm": 0.6384578683151475,
- "PCK3D_100mm": 0.9093612136801781,
- "AUC_0_150mm": 0.6652087100209272,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.88078370684721,
- "left_hip": 40.7707714293256,
- "right_hip": 39.89864957366807,
- "spine1": 45.81847794327041,
- "left_knee": 53.127242685684536,
- "right_knee": 53.08574698128348,
- "spine2": 44.7910390784409,
- "left_ankle": 56.75905942499106,
- "right_ankle": 58.16959865513955,
- "spine3": 42.05493612888672,
- "left_foot": 64.30613556719297,
- "right_foot": 65.05118230757668,
- "neck": 34.21864349665207,
- "left_collar": 29.614457371391357,
- "right_collar": 30.168431953048632,
- "head": 39.68366013986951,
- "left_shoulder": 39.31176176482065,
- "right_shoulder": 40.35379242035913,
- "left_elbow": 52.13079804640948,
- "right_elbow": 48.253344917193644,
- "left_wrist": 79.23459197966791,
- "right_wrist": 77.1130344325212,
- "left_hand": 93.20148748668534,
- "right_hand": 94.99121414274032
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.29255367357425,
- "PA_MPJPE_mm": 58.77989410199223,
- "PCK3D_50mm": 0.5388659384427547,
- "PCK3D_100mm": 0.876595088933802,
- "AUC_0_150mm": 0.6198663340185008,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.756051871751175,
- "right_knee": 50.81489947212034,
- "right_hip": 51.94700981488938,
- "left_hip": 49.19551035998004,
- "left_knee": 52.99740719905924,
- "left_ankle": 60.702120579352375,
- "right_wrist": 78.60382199636399,
- "right_elbow": 43.731525253853555,
- "right_shoulder": 53.16943724322554,
- "left_shoulder": 42.824918454553725,
- "left_elbow": 49.11512498535816,
- "left_wrist": 83.44684974877296,
- "neck": 43.75810225063536,
- "head": 102.85573819797534
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.3599365554387,
- "PA_MPJPE_mm": 54.21152318667249,
- "PCK3D_50mm": 0.5981689692845981,
- "PCK3D_100mm": 0.9070482235959563,
- "AUC_0_150mm": 0.6506083434935819,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.919026759087465,
- "right_knee": 51.354651366710975,
- "right_hip": 49.17400620803124,
- "left_hip": 47.02273622907863,
- "left_knee": 53.41940934446503,
- "left_ankle": 57.60876317646949,
- "right_wrist": 76.79156277793369,
- "right_elbow": 41.42651033238488,
- "right_shoulder": 47.2412790328728,
- "left_shoulder": 42.39979066354495,
- "left_elbow": 46.729545512776184,
- "left_wrist": 80.45099683671447
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.40218971379543
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.22090188850195
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.83152776016149
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.55589378352272
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.71891183946268
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.31659308750521
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.74904299508661
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.1324759126367
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.16089209721233
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.50048969808818
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.10683218928586
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 52.71980431574263
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.03444415040397
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.58449148838511
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 74.21396761055068
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.82741451783965
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.21365622880793
- }
- }
- },
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 379,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.88665615262045,
- "PA_MPJPE_mm": 52.07917163041946,
- "PCK3D_50mm": 0.6454210402577826,
- "PCK3D_100mm": 0.908858164267242,
- "AUC_0_150mm": 0.6675373698629404,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.15900264424531,
- "left_hip": 39.089109902236345,
- "right_hip": 37.96916180513328,
- "spine1": 44.205245153188784,
- "left_knee": 52.9924020533549,
- "right_knee": 51.96389102006279,
- "spine2": 43.79822082826003,
- "left_ankle": 57.57739087602462,
- "right_ankle": 58.771774901865655,
- "spine3": 41.55301964626333,
- "left_foot": 65.55052813689574,
- "right_foot": 66.32479338963965,
- "neck": 33.68765359874907,
- "left_collar": 29.260208526652065,
- "right_collar": 30.16990812955694,
- "head": 39.03939397883591,
- "left_shoulder": 38.3902486317579,
- "right_shoulder": 40.883712730911164,
- "left_elbow": 51.37068092924423,
- "right_elbow": 47.777308505394295,
- "left_wrist": 78.71651499926232,
- "right_wrist": 76.60386803964425,
- "left_hand": 92.91143882206377,
- "right_hand": 94.13464188082445
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.61625455693527,
- "PA_MPJPE_mm": 58.34617962377626,
- "PCK3D_50mm": 0.5434682108332691,
- "PCK3D_100mm": 0.8808802697945128,
- "AUC_0_150mm": 0.6223312737519912,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.8655919164838,
- "right_knee": 49.43007016970009,
- "right_hip": 50.06943590607147,
- "left_hip": 47.05840815518325,
- "left_knee": 52.91211704750162,
- "left_ankle": 60.61590127360832,
- "right_wrist": 78.36436767048248,
- "right_elbow": 43.5775818560265,
- "right_shoulder": 53.79348110491367,
- "left_shoulder": 42.55244591579653,
- "left_elbow": 48.84626122531505,
- "left_wrist": 83.44141000239335,
- "neck": 43.83801230207689,
- "head": 102.48143018731467
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.9178227755528,
- "PA_MPJPE_mm": 53.851077539371246,
- "PCK3D_50mm": 0.602618622854693,
- "PCK3D_100mm": 0.9097518981039705,
- "AUC_0_150mm": 0.652414882544849,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.720652835760596,
- "right_knee": 50.032352638025685,
- "right_hip": 47.652486397801724,
- "left_hip": 45.3439756574678,
- "left_knee": 53.40706568226776,
- "left_ankle": 57.931933843727975,
- "right_wrist": 76.43412519972611,
- "right_elbow": 41.114539548721396,
- "right_shoulder": 48.198355617377544,
- "left_shoulder": 42.1556699271808,
- "left_elbow": 46.07688494769644,
- "left_wrist": 80.14488817670104
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.16091805009316
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.75422226495046
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.44043117656213
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.837421497423925
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 47.43192462504323
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.87740166532663
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.06121725009011
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.60026589223815
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 47.344348569669535
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.38062916206199
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 45.085874233409655
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.53897839054536
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.01926209949478
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.05420202359375
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 68.04237396497881
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 55.560788460379136
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.075729788160054
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_emdb1_detector_288.json b/benchmark/results/gen10_emdb1_detector_288.json
deleted file mode 100644
index 9f1d318..0000000
--- a/benchmark/results/gen10_emdb1_detector_288.json
+++ /dev/null
@@ -1,1426 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 86.36019540610455,
- "PA_MPJPE_mm": 59.35715081702926,
- "PCK3D_50mm": 0.5816583350690828,
- "PCK3D_100mm": 0.8783586752759842,
- "AUC_0_150mm": 0.6333649127757883,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.17085392841998,
- "left_hip": 43.525508269408654,
- "right_hip": 42.64667972914601,
- "spine1": 48.45505298556794,
- "left_knee": 61.133645309493645,
- "right_knee": 59.39718047245011,
- "spine2": 45.88930729506386,
- "left_ankle": 71.94458638866328,
- "right_ankle": 74.79102229264433,
- "spine3": 43.35930489033893,
- "left_foot": 80.9623697017724,
- "right_foot": 84.37800542056193,
- "neck": 36.1692957543859,
- "left_collar": 32.03686624610615,
- "right_collar": 32.38728758728458,
- "head": 43.209096017646445,
- "left_shoulder": 41.926937763055825,
- "right_shoulder": 42.2834399393085,
- "left_elbow": 56.038527042109216,
- "right_elbow": 52.97784385650853,
- "left_wrist": 88.36831947766974,
- "right_wrist": 87.48524180988208,
- "left_hand": 105.1680827541698,
- "right_hand": 107.86716467704487
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 97.44849094502838,
- "PA_MPJPE_mm": 64.70002507534664,
- "PCK3D_50mm": 0.4886214181569316,
- "PCK3D_100mm": 0.8494956407891213,
- "AUC_0_150mm": 0.5922540139007137,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 76.29052906582484,
- "right_knee": 57.17220701423028,
- "right_hip": 53.22176821255679,
- "left_hip": 50.7905757758844,
- "left_knee": 60.25579755057648,
- "left_ankle": 75.45468196109333,
- "right_wrist": 88.7210666977337,
- "right_elbow": 48.16306584956884,
- "right_shoulder": 55.103738955697985,
- "left_shoulder": 45.172526713871484,
- "left_elbow": 52.92770926058818,
- "left_wrist": 92.54978696500646,
- "neck": 45.67312752722943,
- "head": 104.30376950499087
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 92.38667807680024,
- "PA_MPJPE_mm": 61.00327611317812,
- "PCK3D_50mm": 0.5396827049920155,
- "PCK3D_100mm": 0.8748455182947997,
- "AUC_0_150mm": 0.6182177548136252,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 74.3428161875868,
- "right_knee": 57.4651644513926,
- "right_hip": 51.054345165129824,
- "left_hip": 49.25991659614704,
- "left_knee": 60.54422809522697,
- "left_ankle": 72.68018644860318,
- "right_wrist": 86.17969143226448,
- "right_elbow": 44.942188356129925,
- "right_shoulder": 49.968599871076,
- "left_shoulder": 46.25190182814624,
- "left_elbow": 50.31495875896146,
- "left_wrist": 89.03531616747283
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 48.05068830150099
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 54.59845699833797
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 61.98486530730729
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 56.785408431910504
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.30526788582369
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 89.30683373550065
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 54.900253544812244
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 65.60249753126486
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 58.99803152747547
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 48.77302611262348
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 52.75389292681569
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 56.508247628403836
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 51.93269993229314
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.59048325883051
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 79.19614587510411
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 68.84578334663757
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 63.22926971371727
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 524,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 83.52163917263809,
- "PA_MPJPE_mm": 56.11431552695574,
- "PCK3D_50mm": 0.6063563146566687,
- "PCK3D_100mm": 0.8925553704089426,
- "AUC_0_150mm": 0.6473147955287094,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 41.22354591499183,
- "left_hip": 42.35481866691604,
- "right_hip": 41.954075351845525,
- "spine1": 48.018529698143695,
- "left_knee": 57.777879544748465,
- "right_knee": 55.03495085342147,
- "spine2": 45.014054881021664,
- "left_ankle": 65.98614458139804,
- "right_ankle": 67.68572019034663,
- "spine3": 42.15605073934213,
- "left_foot": 74.54715697002241,
- "right_foot": 75.08451871220423,
- "neck": 36.36487138269545,
- "left_collar": 31.8116958431103,
- "right_collar": 31.837303343519647,
- "head": 42.7117329196016,
- "left_shoulder": 40.52615403358803,
- "right_shoulder": 40.11917443550819,
- "left_elbow": 54.090253730392654,
- "right_elbow": 50.33266223980275,
- "left_wrist": 82.3734821029785,
- "right_wrist": 80.56996057552166,
- "left_hand": 99.09652632981538,
- "right_hand": 100.07230960600242
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 95.04766962720016,
- "PA_MPJPE_mm": 61.6499059426591,
- "PCK3D_50mm": 0.5146695628886839,
- "PCK3D_100mm": 0.8627845389353409,
- "AUC_0_150mm": 0.6062360280164365,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.73812837670701,
- "right_knee": 52.83103385216617,
- "right_hip": 52.144406657392935,
- "left_hip": 49.64478330661134,
- "left_knee": 56.99928814653413,
- "left_ankle": 69.67677996382557,
- "right_wrist": 81.92638352011569,
- "right_elbow": 45.68762816782299,
- "right_shoulder": 53.31891994457145,
- "left_shoulder": 43.57126989274324,
- "left_elbow": 50.386786513807515,
- "left_wrist": 85.95762896047687,
- "neck": 46.486885518839046,
- "head": 104.7287603756135
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 89.43618486736666,
- "PA_MPJPE_mm": 57.198091543726136,
- "PCK3D_50mm": 0.5717836561827397,
- "PCK3D_100mm": 0.8928105255849476,
- "AUC_0_150mm": 0.6362306911866479,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.45185773131985,
- "right_knee": 53.11217917995485,
- "right_hip": 49.78507245680848,
- "left_hip": 48.12992443113187,
- "left_knee": 57.47598155852748,
- "left_ankle": 67.0388739327788,
- "right_wrist": 79.19969721551347,
- "right_elbow": 42.428841386299844,
- "right_shoulder": 47.561617052011016,
- "left_shoulder": 44.220194662452734,
- "left_elbow": 47.66952923033551,
- "left_wrist": 82.30332968757972
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 45.61181664285712
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 52.922013571026206
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 59.40477429566595
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 53.420461185753744
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.01949788583968
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 78.22891534005207
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.798062367159666
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 64.40179443744721
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 52.84311397731389
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 45.24183489526006
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 50.0470066723417
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 52.64914280990403
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 49.578923395275865
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.42619137870427
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 76.4315837309105
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 63.70752846159056
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 60.06573034732305
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 492,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 78.74461207244048,
- "PA_MPJPE_mm": 51.528570140653244,
- "PCK3D_50mm": 0.6454019995834201,
- "PCK3D_100mm": 0.912993820731792,
- "AUC_0_150mm": 0.6690983957195298,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.9136767999096,
- "left_hip": 40.47148433079545,
- "right_hip": 40.055781659879116,
- "spine1": 45.653318711842765,
- "left_knee": 51.31958950684752,
- "right_knee": 51.4750443871456,
- "spine2": 43.11869146746732,
- "left_ankle": 57.49578861417643,
- "right_ankle": 59.19028188962307,
- "spine3": 40.36905392301823,
- "left_foot": 65.37227352497462,
- "right_foot": 66.90815738685133,
- "neck": 34.577706915936815,
- "left_collar": 29.396742868583814,
- "right_collar": 29.75534529828994,
- "head": 39.90155032752195,
- "left_shoulder": 38.55782515664978,
- "right_shoulder": 38.89356198800002,
- "left_elbow": 51.2516598851357,
- "right_elbow": 47.36640119811567,
- "left_wrist": 74.30253658889178,
- "right_wrist": 72.33088406783426,
- "left_hand": 90.17757705357506,
- "right_hand": 89.83074982461156
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 90.31671159904738,
- "PA_MPJPE_mm": 57.3960646310737,
- "PCK3D_50mm": 0.5519415597940905,
- "PCK3D_100mm": 0.8843336209718213,
- "AUC_0_150mm": 0.6270866188591663,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.010885099190006,
- "right_knee": 49.40588287592014,
- "right_hip": 49.69679032024971,
- "left_hip": 47.2359017726153,
- "left_knee": 51.35332678204827,
- "left_ankle": 62.17149116169564,
- "right_wrist": 73.41896609248039,
- "right_elbow": 42.68680807216356,
- "right_shoulder": 50.759110982952784,
- "left_shoulder": 42.19079935943876,
- "left_elbow": 47.31648136404793,
- "left_wrist": 77.29151607961984,
- "neck": 44.400539482168895,
- "head": 103.6064053904406
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 83.95724548526482,
- "PA_MPJPE_mm": 52.43400582432592,
- "PCK3D_50mm": 0.6172325210025689,
- "PCK3D_100mm": 0.9153926265361383,
- "AUC_0_150mm": 0.6598392360014603,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.080266053201306,
- "right_knee": 50.03123741472005,
- "right_hip": 46.9991834428908,
- "left_hip": 45.32429074780136,
- "left_knee": 51.97449231596455,
- "left_ankle": 58.77784259321732,
- "right_wrist": 71.58091575001832,
- "right_elbow": 40.36698474592163,
- "right_shoulder": 45.09209429879848,
- "left_shoulder": 41.35450793529113,
- "left_elbow": 44.660629205864005,
- "left_wrist": 73.96562538822212
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 44.55181691811845
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.55521451689604
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 55.3608614332472
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.51112207314189
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.91037508785191
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 71.79274745681104
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.04477868242617
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 60.11182400385648
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 47.90637123492156
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 42.15007907867123
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 46.586321590275794
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 49.16260553786834
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.94278563465783
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 47.51324841837869
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 65.48214792268561
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 54.31335306279603
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.37047323026588
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 484,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 75.57517180400826,
- "PA_MPJPE_mm": 49.188970523442464,
- "PCK3D_50mm": 0.6732277997639381,
- "PCK3D_100mm": 0.9235714781642713,
- "AUC_0_150mm": 0.681643385226644,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.76494914364384,
- "left_hip": 38.661692494981786,
- "right_hip": 38.33899780751174,
- "spine1": 43.590608446181534,
- "left_knee": 48.693919336878395,
- "right_knee": 48.33038665929497,
- "spine2": 43.10625754694728,
- "left_ankle": 51.12328525703001,
- "right_ankle": 51.64621602572816,
- "spine3": 40.754431212417565,
- "left_foot": 58.89029658299511,
- "right_foot": 59.30724451670488,
- "neck": 33.321065148898164,
- "left_collar": 28.526638897978092,
- "right_collar": 29.397640216627998,
- "head": 38.28544104893039,
- "left_shoulder": 37.3425972954533,
- "right_shoulder": 38.80988900864712,
- "left_elbow": 50.05459478922153,
- "right_elbow": 45.955044612681725,
- "left_wrist": 73.33228830936048,
- "right_wrist": 70.81434998288026,
- "left_hand": 87.15345303618544,
- "right_hand": 88.33400518543841
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 87.52030278961686,
- "PA_MPJPE_mm": 55.5301788896483,
- "PCK3D_50mm": 0.5714821316987533,
- "PCK3D_100mm": 0.8941262237033951,
- "AUC_0_150mm": 0.637160077057679,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.162950415758246,
- "right_knee": 46.11857456438417,
- "right_hip": 49.57608957802908,
- "left_hip": 46.24688520676111,
- "left_knee": 48.65116440134171,
- "left_ankle": 54.522815181061254,
- "right_wrist": 72.22424807163992,
- "right_elbow": 41.45965937779312,
- "right_shoulder": 51.97302243777457,
- "left_shoulder": 41.61907912450625,
- "left_elbow": 46.2897693764571,
- "left_wrist": 77.3833564044678,
- "neck": 44.32855319258784,
- "head": 102.86633712251388
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 81.27800512468139,
- "PA_MPJPE_mm": 50.2842200477356,
- "PCK3D_50mm": 0.6423870027077692,
- "PCK3D_100mm": 0.9240401305283621,
- "AUC_0_150mm": 0.671601346493674,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 51.04154316101264,
- "right_knee": 46.78196293467675,
- "right_hip": 46.61359048404333,
- "left_hip": 44.02357370170303,
- "left_knee": 49.330124986438435,
- "left_ankle": 51.584933128264964,
- "right_wrist": 70.3118528048344,
- "right_elbow": 39.51377846872424,
- "right_shoulder": 45.228300751207385,
- "left_shoulder": 40.12754055595759,
- "left_elbow": 44.46739246564281,
- "left_wrist": 74.38604713032164
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 43.14525470503906
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.63124268755206
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 53.46645092572258
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.50637271231829
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 44.58458310879921
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.51196571955484
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 44.438172164809664
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 54.283740403861856
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 45.02070476116322
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 39.920652479440285
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 43.34205808334225
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 45.37915121344483
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.08349948704955
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.36959107759936
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 63.49534962925613
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 48.76035674927394
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.02486702205984
- }
- }
- },
- "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 159,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 111.37231793572217,
- "PA_MPJPE_mm": 71.35701703978829,
- "PCK3D_50mm": 0.5069239047420676,
- "PCK3D_100mm": 0.8257203360411025,
- "AUC_0_150mm": 0.5864722963181953,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.34918742724728,
- "left_hip": 48.82035681676445,
- "right_hip": 48.43658419936253,
- "spine1": 55.75530378806384,
- "left_knee": 73.56256056149807,
- "right_knee": 69.50168708040728,
- "spine2": 51.80355993263622,
- "left_ankle": 86.294038181837,
- "right_ankle": 86.89030089807757,
- "spine3": 48.98559671202493,
- "left_foot": 95.82849826894862,
- "right_foot": 94.38728552041927,
- "neck": 42.6858535357048,
- "left_collar": 37.72376647073241,
- "right_collar": 38.580886161564194,
- "head": 50.733469225541235,
- "left_shoulder": 47.07882147359614,
- "right_shoulder": 46.63421698648318,
- "left_elbow": 66.47575673727309,
- "right_elbow": 62.91759478784899,
- "left_wrist": 117.25228074666268,
- "right_wrist": 114.24297887828718,
- "left_hand": 139.1048125083684,
- "right_hand": 140.52301205556844
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 121.98633193605717,
- "PA_MPJPE_mm": 75.42408735608659,
- "PCK3D_50mm": 0.4205611926086827,
- "PCK3D_100mm": 0.7961734162525664,
- "AUC_0_150mm": 0.5481690162475751,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 87.36080712074616,
- "right_knee": 65.75207582919393,
- "right_hip": 60.6198600683398,
- "left_hip": 58.26548967855848,
- "left_knee": 71.20238104860131,
- "left_ankle": 89.19814766215679,
- "right_wrist": 115.9911126864057,
- "right_elbow": 57.516110959500125,
- "right_shoulder": 59.748238845498385,
- "left_shoulder": 49.634329403621386,
- "left_elbow": 63.17993056603776,
- "left_wrist": 120.78112574958729,
- "neck": 50.60594372076321,
- "head": 106.08166964620187
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 118.26657533033976,
- "PA_MPJPE_mm": 73.15498981426235,
- "PCK3D_50mm": 0.4605741859334861,
- "PCK3D_100mm": 0.8155141290009026,
- "AUC_0_150mm": 0.5679417146517414,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 85.22307005189593,
- "right_knee": 65.95553865516908,
- "right_hip": 58.552152920612116,
- "left_hip": 56.86714225686114,
- "left_knee": 71.54146829481897,
- "left_ankle": 86.78073886867207,
- "right_wrist": 113.08852296630829,
- "right_elbow": 53.082326859755526,
- "right_shoulder": 57.3366527546273,
- "left_shoulder": 53.6039894322701,
- "left_elbow": 58.672561641824096,
- "left_wrist": 117.1557130683335
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 52.64896936268227
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 66.98254470132622
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 59.8123429091429
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 59.01179780321712
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 55.62412498827506
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 94.12111384446503
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 62.123977501437544
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 77.34657192841055
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 76.98443210679922
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 53.74855800040186
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 69.28527859535708
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 72.50772090844319
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 65.05969239936657
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 61.32890533410167
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 108.31377246209591
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 81.76857857000522
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 88.691431445385
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 376,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 78.03734890145603,
- "PA_MPJPE_mm": 51.57059446982616,
- "PCK3D_50mm": 0.6467072832048879,
- "PCK3D_100mm": 0.9143581198361452,
- "AUC_0_150mm": 0.6693086453762993,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.975988330483126,
- "left_hip": 40.040535296600595,
- "right_hip": 39.2548188318151,
- "spine1": 44.70117623689213,
- "left_knee": 52.682735619580995,
- "right_knee": 52.7894351467389,
- "spine2": 43.20184803028694,
- "left_ankle": 58.00527699510531,
- "right_ankle": 59.3440303144582,
- "spine3": 40.29060962076097,
- "left_foot": 65.63679618952523,
- "right_foot": 65.63334857030883,
- "neck": 34.060617414763726,
- "left_collar": 29.025255059507398,
- "right_collar": 29.262001985220937,
- "head": 39.21456048128227,
- "left_shoulder": 39.65320074122343,
- "right_shoulder": 39.43681904017622,
- "left_elbow": 49.93771837120045,
- "right_elbow": 47.01588890930422,
- "left_wrist": 76.07219172653706,
- "right_wrist": 73.18225428880594,
- "left_hand": 90.68001922483984,
- "right_hand": 90.59714085040929
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 90.3871220276991,
- "PA_MPJPE_mm": 57.847875272495536,
- "PCK3D_50mm": 0.547362156693546,
- "PCK3D_100mm": 0.8807867408575595,
- "AUC_0_150mm": 0.6242060745793168,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.51414914876979,
- "right_knee": 49.67463103492852,
- "right_hip": 51.692295737748,
- "left_hip": 48.42779004471409,
- "left_knee": 52.01258410950236,
- "left_ankle": 61.71846829944016,
- "right_wrist": 75.06731938337762,
- "right_elbow": 42.842183430057226,
- "right_shoulder": 51.451763492459975,
- "left_shoulder": 41.33186557187064,
- "left_elbow": 46.77781689004721,
- "left_wrist": 80.62637070936145,
- "neck": 43.35036704155637,
- "head": 103.382648921104
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 84.40421186119713,
- "PA_MPJPE_mm": 53.0695731618992,
- "PCK3D_50mm": 0.6071478164271332,
- "PCK3D_100mm": 0.9121190029854891,
- "AUC_0_150mm": 0.6560251784462465,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.336565970286244,
- "right_knee": 49.81966598182401,
- "right_hip": 48.86400854454167,
- "left_hip": 46.38212780444356,
- "left_knee": 52.292339641903084,
- "left_ankle": 58.67825588896805,
- "right_wrist": 73.27242301798303,
- "right_elbow": 40.74560389367415,
- "right_shoulder": 45.4470956833974,
- "left_shoulder": 41.47728622623572,
- "left_elbow": 44.330225186845986,
- "left_wrist": 77.18928010268738
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 43.76084696585426
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.288686443763865
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 51.69255486590644
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.30131722750054
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.18912494087321
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.44462833760275
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 45.79696371625728
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 59.3517991828907
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 45.711011368705975
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 42.46613632744154
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 46.87341754605619
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 49.8724407258073
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.67054865859928
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.51140398437274
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 70.25637166955849
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 55.41227452090537
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.57004562516038
- }
- }
- },
- "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 383,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 77.43465515911765,
- "PA_MPJPE_mm": 50.96418789361965,
- "PCK3D_50mm": 0.6537058251753107,
- "PCK3D_100mm": 0.9164323404846213,
- "AUC_0_150mm": 0.6729504717879115,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.51332846676612,
- "left_hip": 40.42074665596479,
- "right_hip": 39.77983648353846,
- "spine1": 45.569536816809325,
- "left_knee": 51.44592418184902,
- "right_knee": 51.637188384198964,
- "spine2": 44.227502225797274,
- "left_ankle": 55.0685346510069,
- "right_ankle": 56.007132009574065,
- "spine3": 41.27668908180914,
- "left_foot": 62.37859685336282,
- "right_foot": 63.0315913726984,
- "neck": 33.46053861577096,
- "left_collar": 28.796233093445764,
- "right_collar": 29.18365701278341,
- "head": 39.079606521528405,
- "left_shoulder": 38.98780932737523,
- "right_shoulder": 39.271149039838726,
- "left_elbow": 50.832498760018304,
- "right_elbow": 46.5072868013335,
- "left_wrist": 75.13487301135929,
- "right_wrist": 72.99051922168049,
- "left_hand": 88.92703685034057,
- "right_hand": 90.61269400802165
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 89.73081876622899,
- "PA_MPJPE_mm": 57.31222427968328,
- "PCK3D_50mm": 0.5551343470110394,
- "PCK3D_100mm": 0.883402267384771,
- "AUC_0_150mm": 0.6275363139591694,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.31711190439754,
- "right_knee": 49.3356496399676,
- "right_hip": 51.50581614609806,
- "left_hip": 48.896774956295104,
- "left_knee": 51.292127079942716,
- "left_ankle": 59.16468153455262,
- "right_wrist": 74.4307818790412,
- "right_elbow": 42.03665542123582,
- "right_shoulder": 52.38873863884536,
- "left_shoulder": 42.24874591749714,
- "left_elbow": 47.45375177375781,
- "left_wrist": 79.13560497158996,
- "neck": 42.97615239875516,
- "head": 103.18854765358981
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 83.42182692347514,
- "PA_MPJPE_mm": 52.415005724305914,
- "PCK3D_50mm": 0.6197944872595987,
- "PCK3D_100mm": 0.9146497257515795,
- "AUC_0_150mm": 0.6601445039451906,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 55.22114415447575,
- "right_knee": 49.80791572239083,
- "right_hip": 48.5132136251677,
- "left_hip": 46.51432615443718,
- "left_knee": 51.6370242385951,
- "left_ankle": 56.02747496790258,
- "right_wrist": 72.79883365443187,
- "right_elbow": 39.76914941582212,
- "right_shoulder": 45.66384055342877,
- "left_shoulder": 41.36753606687784,
- "left_elbow": 45.31093339718402,
- "left_wrist": 76.34867674095725
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 42.65514589096134
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.693954134617826
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 52.93083147094491
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.53185429833531
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.55883442453656
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 63.64943258044303
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 45.498685642775776
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 58.17308777673389
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 45.35437657873772
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 41.56200659551345
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 45.212316485964585
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 48.915722980451434
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.158752336241406
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.35333641159865
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 73.07661242623804
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.03087672125004
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.04038223983059
- }
- }
- },
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 379,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 75.21653836129266,
- "PA_MPJPE_mm": 49.755940911716884,
- "PCK3D_50mm": 0.6657276261889884,
- "PCK3D_100mm": 0.920610289523016,
- "AUC_0_150mm": 0.6788998371754988,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.496152207130166,
- "left_hip": 38.5113292965768,
- "right_hip": 37.68284848070092,
- "spine1": 43.41387094478198,
- "left_knee": 51.01734322611215,
- "right_knee": 50.26976366734501,
- "spine2": 42.83405251540932,
- "left_ankle": 54.922729070477914,
- "right_ankle": 55.41992893059886,
- "spine3": 40.34618608808517,
- "left_foot": 62.19536152625109,
- "right_foot": 62.62191029044732,
- "neck": 32.10125774026579,
- "left_collar": 27.509543594168527,
- "right_collar": 28.454150124339538,
- "head": 37.64961105437637,
- "left_shoulder": 37.7195689257353,
- "right_shoulder": 39.61715774880208,
- "left_elbow": 49.55430099966515,
- "right_elbow": 46.01624896017141,
- "left_wrist": 73.57058846160164,
- "right_wrist": 71.2851810455134,
- "left_hand": 87.09239822105147,
- "right_hand": 87.84109876159864
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 86.86432634744224,
- "PA_MPJPE_mm": 56.37353850240044,
- "PCK3D_50mm": 0.5633171660665933,
- "PCK3D_100mm": 0.8907876335287291,
- "AUC_0_150mm": 0.632605918313869,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.09214931204945,
- "right_knee": 47.76591352475783,
- "right_hip": 49.460477554531884,
- "left_hip": 46.32129290368397,
- "left_knee": 51.15051625604136,
- "left_ankle": 58.407876762214826,
- "right_wrist": 73.01800512975785,
- "right_elbow": 41.799219843973496,
- "right_shoulder": 52.942892374135475,
- "left_shoulder": 41.708553932604374,
- "left_elbow": 46.86019273503794,
- "left_wrist": 78.20699513705927,
- "neck": 41.77347681196014,
- "head": 102.72197675579844
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 80.84436093558031,
- "PA_MPJPE_mm": 51.507641952776716,
- "PCK3D_50mm": 0.6281503853363882,
- "PCK3D_100mm": 0.9210476983961675,
- "AUC_0_150mm": 0.6649063927094042,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.67901268821243,
- "right_knee": 48.306659565955535,
- "right_hip": 46.68350355680783,
- "left_hip": 44.340529674590975,
- "left_knee": 51.59582594803433,
- "left_ankle": 55.3748874225392,
- "right_wrist": 71.44852640875504,
- "right_elbow": 39.32230792299174,
- "right_shoulder": 46.35276452681138,
- "left_shoulder": 40.59369116433555,
- "left_elbow": 44.174847336797136,
- "left_wrist": 75.21914721748944
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 41.84802923090198
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.22438031951986
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 51.431885593401354
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.695923492450056
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.2355839773884
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 65.01459182412023
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 43.73989899406976
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 58.81678663284938
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 45.346401286022065
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 40.50217453820777
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 44.06792370842814
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 47.856520882388246
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.93957422688802
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 48.86746456367921
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 64.2988977823099
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 53.14371204057995
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.600779880767284
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_emdb1_gt-joints_288.json b/benchmark/results/gen10_emdb1_gt-joints_288.json
deleted file mode 100644
index 45e6342..0000000
--- a/benchmark/results/gen10_emdb1_gt-joints_288.json
+++ /dev/null
@@ -1,1386 +0,0 @@
-{
- "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 83.92834903184675,
- "PA_MPJPE_mm": 58.751911786878345,
- "PCK3D_50mm": 0.5887510545022058,
- "PCK3D_100mm": 0.8831162096004647,
- "AUC_0_150mm": 0.6373788298337913,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.73316963853984,
- "left_hip": 42.105094279179845,
- "right_hip": 41.281129393185765,
- "spine1": 47.52378838791497,
- "left_knee": 60.34233066458636,
- "right_knee": 58.01963490613796,
- "spine2": 46.345831413154315,
- "left_ankle": 69.46304947870532,
- "right_ankle": 71.65870446461322,
- "spine3": 44.124636440657866,
- "left_foot": 77.94854678829442,
- "right_foot": 80.49452618497489,
- "neck": 35.87975893096458,
- "left_collar": 31.802125584773197,
- "right_collar": 32.30946576995018,
- "head": 43.0679603442414,
- "left_shoulder": 41.546614701840646,
- "right_shoulder": 42.00426152659356,
- "left_elbow": 56.77920604123176,
- "right_elbow": 53.02748414950827,
- "left_wrist": 89.85182009232813,
- "right_wrist": 87.59546436792508,
- "left_hand": 107.38088070000722,
- "right_hand": 108.76039863576929
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 94.8815319752873,
- "PA_MPJPE_mm": 64.37833197108672,
- "PCK3D_50mm": 0.49418566746285286,
- "PCK3D_100mm": 0.8511062641876234,
- "AUC_0_150mm": 0.5945055121539645,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 72.84242833556407,
- "right_knee": 55.85977721389233,
- "right_hip": 52.905753032054754,
- "left_hip": 50.23681474853344,
- "left_knee": 59.752772688290094,
- "left_ankle": 72.89876157293581,
- "right_wrist": 88.46108514820337,
- "right_elbow": 47.7697198283535,
- "right_shoulder": 56.061901731944,
- "left_shoulder": 45.555457998299616,
- "left_elbow": 53.299717934651326,
- "left_wrist": 94.05249340506685,
- "neck": 46.18883241002267,
- "head": 105.41113154740228
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 90.06856735842071,
- "PA_MPJPE_mm": 60.32961295780091,
- "PCK3D_50mm": 0.5513214122723312,
- "PCK3D_100mm": 0.8783104454493909,
- "AUC_0_150mm": 0.622964437122225,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.99684074028161,
- "right_knee": 56.265374419376826,
- "right_hip": 50.204237807006166,
- "left_hip": 48.234024124729025,
- "left_knee": 60.08133782915697,
- "left_ankle": 70.16863307472946,
- "right_wrist": 86.40635266662676,
- "right_elbow": 45.104162890766325,
- "right_shoulder": 49.44125065643362,
- "left_shoulder": 45.06681777373729,
- "left_elbow": 51.198305231126774,
- "left_wrist": 90.7880182796401
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 46.038846304255806
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 54.940274763348654
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 56.00108324631387
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 56.523829821180634
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.788880173913896
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 85.35684571680031
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 56.06876157210267
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 65.33608203203687
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 57.2324324832803
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 49.01324581528504
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 53.92540290081699
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 59.2254161821622
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 51.90649850478667
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.934905602715446
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 77.50694285849761
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 67.96418670806709
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 62.10231836498085
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 524,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 80.60510979187373,
- "PA_MPJPE_mm": 54.993650125407605,
- "PCK3D_50mm": 0.6183445352584049,
- "PCK3D_100mm": 0.897874054405399,
- "AUC_0_150mm": 0.6530262261557589,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.61402999352406,
- "left_hip": 41.12877703919749,
- "right_hip": 40.4043050077848,
- "spine1": 46.14043216885901,
- "left_knee": 56.507227582127825,
- "right_knee": 54.25101638949458,
- "spine2": 44.669903276529006,
- "left_ankle": 64.64218316601806,
- "right_ankle": 65.27543327821816,
- "spine3": 42.39946824354014,
- "left_foot": 73.05376439453156,
- "right_foot": 72.61144586757796,
- "neck": 34.75257765898708,
- "left_collar": 30.698217016767707,
- "right_collar": 31.30128138517326,
- "head": 40.66816083682148,
- "left_shoulder": 39.4794543472988,
- "right_shoulder": 40.0664034601648,
- "left_elbow": 53.956812973591546,
- "right_elbow": 49.72496942396486,
- "left_wrist": 81.82904397487937,
- "right_wrist": 79.47645702974874,
- "left_hand": 98.49175567068687,
- "right_hand": 98.70448282429453
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.1295611747323,
- "PA_MPJPE_mm": 60.78367134761022,
- "PCK3D_50mm": 0.5202849674907095,
- "PCK3D_100mm": 0.8692308604145305,
- "AUC_0_150mm": 0.6102803616812714,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.08148217733152,
- "right_knee": 52.21173879289632,
- "right_hip": 50.59103312142057,
- "left_hip": 48.2395568703151,
- "left_knee": 55.96232402367248,
- "left_ankle": 67.99351548640969,
- "right_wrist": 80.66535401511157,
- "right_elbow": 45.02161828158875,
- "right_shoulder": 54.296846451434575,
- "left_shoulder": 44.16618442825935,
- "left_elbow": 50.15620491029579,
- "left_wrist": 85.6049932650607,
- "neck": 45.41397242061508,
- "head": 103.56657462213155
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.17522223100981,
- "PA_MPJPE_mm": 56.31933729990539,
- "PCK3D_50mm": 0.5802078579429947,
- "PCK3D_100mm": 0.8974678117523407,
- "AUC_0_150mm": 0.6405463514781323,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.92605494283208,
- "right_knee": 52.56359699805319,
- "right_hip": 48.1825312492314,
- "left_hip": 46.61837400545518,
- "left_knee": 56.53189586927218,
- "left_ankle": 65.77903316398344,
- "right_wrist": 78.30081864178275,
- "right_elbow": 42.09132491994855,
- "right_shoulder": 47.72631014281385,
- "left_shoulder": 43.29779308555731,
- "left_elbow": 47.659215924933704,
- "left_wrist": 82.15509865500101
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.82819253595642
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.76047942123261
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.91226809840116
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 53.831911922329695
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.37961705254229
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 74.23418205317233
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.96999079926048
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 62.69591045477341
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 49.518239572243694
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 45.85106549809772
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.171047643558246
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 52.610025262951254
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.68638498836195
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.28008184677216
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 76.65100459581367
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 62.3578765260251
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.24661736287698
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 492,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.18474175662534,
- "PA_MPJPE_mm": 50.49397533885077,
- "PCK3D_50mm": 0.65654517418302,
- "PCK3D_100mm": 0.9194775200874027,
- "AUC_0_150mm": 0.674738320621312,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.39096163334939,
- "left_hip": 39.08157065860265,
- "right_hip": 38.56004764047348,
- "spine1": 44.098201910432145,
- "left_knee": 50.54203673728069,
- "right_knee": 50.194568324703326,
- "spine2": 43.04924893828296,
- "left_ankle": 55.51606647230906,
- "right_ankle": 56.44042383184989,
- "spine3": 40.88749844740145,
- "left_foot": 63.0364421462635,
- "right_foot": 64.0770077864364,
- "neck": 33.72133161097206,
- "left_collar": 28.873028114304017,
- "right_collar": 29.199327608732986,
- "head": 39.25877493377026,
- "left_shoulder": 37.886874639627806,
- "right_shoulder": 38.598569255005344,
- "left_elbow": 50.66685121571532,
- "right_elbow": 46.959231462387336,
- "left_wrist": 73.15213662055746,
- "right_wrist": 72.09355633623686,
- "left_hand": 88.85298983080452,
- "right_hand": 89.71866197691895
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 87.30044959097395,
- "PA_MPJPE_mm": 56.718528354613895,
- "PCK3D_50mm": 0.5589375359321009,
- "PCK3D_100mm": 0.8878533199779518,
- "AUC_0_150mm": 0.6305752711433708,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.74726087553786,
- "right_knee": 48.44470250086413,
- "right_hip": 48.590458112367095,
- "left_hip": 46.18321696174198,
- "left_knee": 50.82476201069403,
- "left_ankle": 60.02791576808218,
- "right_wrist": 73.30992056096062,
- "right_elbow": 42.129921614685166,
- "right_shoulder": 51.66639055632412,
- "left_shoulder": 42.57961021510322,
- "left_elbow": 46.871180772310176,
- "left_wrist": 76.57435586486831,
- "neck": 44.19632813482103,
- "head": 103.91337301623462
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 81.33767974631881,
- "PA_MPJPE_mm": 51.465509138913774,
- "PCK3D_50mm": 0.6291436750611957,
- "PCK3D_100mm": 0.920950365791257,
- "AUC_0_150mm": 0.6653846462694378,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.15476424153055,
- "right_knee": 49.05340204937179,
- "right_hip": 45.59813420000352,
- "left_hip": 43.96140703763108,
- "left_knee": 51.34819997082364,
- "left_ankle": 56.726487308944456,
- "right_wrist": 71.34128351471523,
- "right_elbow": 39.90528304213521,
- "right_shoulder": 44.82656827581919,
- "left_shoulder": 40.6593369783226,
- "left_elbow": 44.6705626253075,
- "left_wrist": 73.34068042236055
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.183095691974586
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 47.52396349951072
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 50.03359234685762
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.91735125523349
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.0474972450889
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 66.73782921984493
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 45.99392775682312
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.6673468128953
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 47.67982018947211
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.14920961901022
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.852098284348195
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 48.86218338666897
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.20948170012932
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 48.27992366548726
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 65.40585420441921
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 53.57709815872016
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.59406897039507
- }
- }
- },
- "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10b_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 484,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 73.68882669984916,
- "PA_MPJPE_mm": 48.641582199601004,
- "PCK3D_50mm": 0.6766792515454507,
- "PCK3D_100mm": 0.9272479912597326,
- "AUC_0_150mm": 0.6843368000860108,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 35.67217458323834,
- "left_hip": 37.79480282688669,
- "right_hip": 37.31763487417782,
- "spine1": 42.225240323277156,
- "left_knee": 48.08352214249748,
- "right_knee": 47.78670563344909,
- "spine2": 42.68151892330568,
- "left_ankle": 50.357892826012694,
- "right_ankle": 50.72393481623491,
- "spine3": 40.89240401708636,
- "left_foot": 57.85233043294667,
- "right_foot": 57.604205021026054,
- "neck": 32.65178507776647,
- "left_collar": 28.101268824118268,
- "right_collar": 29.012023918089014,
- "head": 37.63786049520765,
- "left_shoulder": 37.21605110915164,
- "right_shoulder": 38.875421168129286,
- "left_elbow": 49.65448119085736,
- "right_elbow": 45.937354681492856,
- "left_wrist": 73.08416600556129,
- "right_wrist": 70.60630200864549,
- "left_hand": 87.35450863708074,
- "right_hand": 88.27438325418646
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 84.91243792021574,
- "PA_MPJPE_mm": 55.151179097118266,
- "PCK3D_50mm": 0.5741282946402642,
- "PCK3D_100mm": 0.8979617237925333,
- "AUC_0_150mm": 0.6389367558697303,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 52.62376013200289,
- "right_knee": 45.92186182084901,
- "right_hip": 48.17208657835952,
- "left_hip": 45.17853290232076,
- "left_knee": 48.48064463615928,
- "left_ankle": 53.805693510194445,
- "right_wrist": 71.9946802156334,
- "right_elbow": 41.39933182206767,
- "right_shoulder": 52.26414339178021,
- "left_shoulder": 41.9258662872651,
- "left_elbow": 46.08268525439413,
- "left_wrist": 77.38937240989199,
- "neck": 43.983875396281974,
- "head": 102.89397300245527
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 79.1842944047682,
- "PA_MPJPE_mm": 49.794918218981934,
- "PCK3D_50mm": 0.6466207526034103,
- "PCK3D_100mm": 0.92836991245903,
- "AUC_0_150mm": 0.6742030952288542,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 49.82520615056693,
- "right_knee": 46.54580614360534,
- "right_hip": 45.225143237239976,
- "left_hip": 42.99990742173302,
- "left_knee": 49.07438103621896,
- "left_ankle": 51.08595889084857,
- "right_wrist": 69.97062703150272,
- "right_elbow": 39.319464260665896,
- "right_shoulder": 45.09315816446268,
- "left_shoulder": 39.77049012275868,
- "left_elbow": 44.2521981243505,
- "left_wrist": 74.37667804382991
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.139779511575064
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 47.69979436167506
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 49.481724325197014
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.1977822842888
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 44.47397026515244
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 64.78583215114088
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.38181754980584
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 53.843564359800666
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 43.47088565721161
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.967869728293095
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 43.77540738294967
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 45.126883966926094
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 43.80621329138337
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.99929324405432
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 62.85775022044266
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 48.56479441987629
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 51.05629678959965
- }
- }
- },
- "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10com_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 159,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 112.74756843428398,
- "PA_MPJPE_mm": 71.39650990716837,
- "PCK3D_50mm": 0.5094127287059702,
- "PCK3D_100mm": 0.8254816136303917,
- "AUC_0_150mm": 0.5868705162744655,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.57623333063146,
- "left_hip": 49.467984134261506,
- "right_hip": 48.818864801505484,
- "spine1": 55.667443505632406,
- "left_knee": 73.83923493493157,
- "right_knee": 69.30801054691179,
- "spine2": 51.88228145528801,
- "left_ankle": 85.79853499009812,
- "right_ankle": 86.62166519414116,
- "spine3": 49.3584055842293,
- "left_foot": 95.27411962188211,
- "right_foot": 94.10667834824385,
- "neck": 42.42794270393526,
- "left_collar": 37.54767350656964,
- "right_collar": 38.904866014160575,
- "head": 50.56221294135745,
- "left_shoulder": 46.3785154988802,
- "right_shoulder": 46.612035573876256,
- "left_elbow": 66.48935949523566,
- "right_elbow": 62.95370857624108,
- "left_wrist": 117.60035083035079,
- "right_wrist": 114.44573002911983,
- "left_hand": 139.90532848313507,
- "right_hand": 140.96905767142212
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 122.12063664868687,
- "PA_MPJPE_mm": 75.43404060163941,
- "PCK3D_50mm": 0.4212101635243983,
- "PCK3D_100mm": 0.7979030470421583,
- "AUC_0_150mm": 0.5488601051822334,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 86.94005583008195,
- "right_knee": 65.8715273437041,
- "right_hip": 60.358750135341516,
- "left_hip": 58.19699871389234,
- "left_knee": 71.6079565477234,
- "left_ankle": 88.4337301909952,
- "right_wrist": 116.04995297690229,
- "right_elbow": 57.59783419503374,
- "right_shoulder": 60.65595184220364,
- "left_shoulder": 49.78881402013055,
- "left_elbow": 63.23184405474108,
- "left_wrist": 120.82897441477131,
- "neck": 50.53059090899042,
- "head": 105.98358724844026
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 118.85826875560848,
- "PA_MPJPE_mm": 73.17346785089532,
- "PCK3D_50mm": 0.46116320236761676,
- "PCK3D_100mm": 0.8164025225075717,
- "AUC_0_150mm": 0.5684713766501203,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 85.13565084882644,
- "right_knee": 66.06003447484802,
- "right_hip": 58.37079876523564,
- "left_hip": 56.79053907241929,
- "left_knee": 71.92958020936179,
- "left_ankle": 86.12238508733917,
- "right_wrist": 113.39367551593405,
- "right_elbow": 53.387286605100286,
- "right_shoulder": 57.578489520321476,
- "left_shoulder": 53.015326161367305,
- "left_elbow": 58.90589047326527,
- "left_wrist": 117.39195747672503
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 52.343077722952614
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 66.57960068969788
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 59.87764360327815
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 58.354004041023
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 55.979587528073296
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 97.38315188208585
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 61.35694306896457
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 75.02523251076889
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 81.1619039827765
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 54.036927775018334
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 69.85414374169333
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 71.76232357196119
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 65.49607869229717
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 61.93462986345885
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 106.50004368123611
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 80.68918519257116
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 88.08611220389551
- }
- }
- },
- "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10f_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 376,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 78.53938903070869,
- "PA_MPJPE_mm": 51.72950054670942,
- "PCK3D_50mm": 0.6453951098756725,
- "PCK3D_100mm": 0.9138022237895698,
- "AUC_0_150mm": 0.6685488220580225,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.13567673007808,
- "left_hip": 40.22876000448766,
- "right_hip": 39.51237833764695,
- "spine1": 44.75474028783639,
- "left_knee": 52.58940502134984,
- "right_knee": 52.86137737112788,
- "spine2": 43.38613675218977,
- "left_ankle": 57.966188154007384,
- "right_ankle": 58.791526365613805,
- "spine3": 40.44022638989568,
- "left_foot": 65.4580304151958,
- "right_foot": 65.09054145075687,
- "neck": 34.37538098123898,
- "left_collar": 29.15031846131856,
- "right_collar": 29.410262750526467,
- "head": 39.56255892987009,
- "left_shoulder": 39.865932130651544,
- "right_shoulder": 39.539565206496036,
- "left_elbow": 49.959237897424785,
- "right_elbow": 46.9920353768995,
- "left_wrist": 77.29333242580911,
- "right_wrist": 73.37726850744666,
- "left_hand": 92.17401170871214,
- "right_hand": 90.5931214644469
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 90.88919164395098,
- "PA_MPJPE_mm": 58.093946942446514,
- "PCK3D_50mm": 0.5444698644507797,
- "PCK3D_100mm": 0.8796711731201213,
- "AUC_0_150mm": 0.6229892601854063,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.02188794740276,
- "right_knee": 49.70649707528342,
- "right_hip": 52.2738011986845,
- "left_hip": 48.79984502893699,
- "left_knee": 52.00966525804215,
- "left_ankle": 61.687893339043676,
- "right_wrist": 75.31832836336466,
- "right_elbow": 42.917142507969835,
- "right_shoulder": 51.64509247803077,
- "left_shoulder": 41.24324201118555,
- "left_elbow": 47.028578090036355,
- "left_wrist": 82.1228846184345,
- "neck": 43.6100833780604,
- "head": 103.93031589977558
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 84.99154703372642,
- "PA_MPJPE_mm": 53.25038903582839,
- "PCK3D_50mm": 0.6051874593757347,
- "PCK3D_100mm": 0.9112005421178553,
- "AUC_0_150mm": 0.6551602687212899,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.71247964544024,
- "right_knee": 49.83747677085364,
- "right_hip": 49.30477207093875,
- "left_hip": 46.601945974527084,
- "left_knee": 52.262373168479684,
- "left_ankle": 58.68980137648884,
- "right_wrist": 73.48798765561382,
- "right_elbow": 40.71187892391729,
- "right_shoulder": 45.65855325416682,
- "left_shoulder": 41.56824991454706,
- "left_elbow": 44.546171529623614,
- "left_wrist": 78.62297814534394
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.66710665300051
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.355090246336644
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.39787165649168
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.73398427485219
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.53005282201343
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.41377360182258
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.12116036347262
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.62909945473794
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.45592416138394
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.455949762546915
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.983520806784085
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.416505948260884
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.169215926454804
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.047762713258415
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 72.00815438677436
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.593187573399916
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.47031572628662
- }
- }
- },
- "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m50_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 383,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.2709756968681,
- "PA_MPJPE_mm": 50.937672517077495,
- "PCK3D_50mm": 0.6529564092989808,
- "PCK3D_100mm": 0.9170642658590217,
- "AUC_0_150mm": 0.6731018625709823,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.16837706809641,
- "left_hip": 40.117170093101684,
- "right_hip": 39.17884098846019,
- "spine1": 45.12545444721083,
- "left_knee": 51.337906515885024,
- "right_knee": 51.00233435300082,
- "spine2": 44.372327856906374,
- "left_ankle": 54.4353691469771,
- "right_ankle": 55.21894123825709,
- "spine3": 41.552424169378256,
- "left_foot": 61.6330360848477,
- "right_foot": 62.31189915297904,
- "neck": 33.65466151757938,
- "left_collar": 29.036759169154305,
- "right_collar": 29.316622384719174,
- "head": 39.22484634818254,
- "left_shoulder": 38.77849844687163,
- "right_shoulder": 39.34566439759891,
- "left_elbow": 50.68762095043062,
- "right_elbow": 46.802655145750805,
- "left_wrist": 75.89793015694804,
- "right_wrist": 73.60419505994405,
- "left_hand": 90.12665572884335,
- "right_hand": 91.57394998873754
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 89.41039483649519,
- "PA_MPJPE_mm": 57.28200758651668,
- "PCK3D_50mm": 0.5539677929836831,
- "PCK3D_100mm": 0.8835266505058648,
- "AUC_0_150mm": 0.6275597947441769,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.216836048287625,
- "right_knee": 48.842278062253015,
- "right_hip": 51.21922756729105,
- "left_hip": 48.58273825386055,
- "left_knee": 51.201849704607426,
- "left_ankle": 58.244239767159485,
- "right_wrist": 74.95498111640289,
- "right_elbow": 42.24325856915804,
- "right_shoulder": 52.67900586672068,
- "left_shoulder": 42.44335468572963,
- "left_elbow": 47.51799489000796,
- "left_wrist": 80.04343030480895,
- "neck": 43.3212354228813,
- "head": 103.437675952065
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 83.31801430362718,
- "PA_MPJPE_mm": 52.284579374944045,
- "PCK3D_50mm": 0.6207422312575198,
- "PCK3D_100mm": 0.9159060421247701,
- "AUC_0_150mm": 0.6608398588673431,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.189990518580444,
- "right_knee": 49.39094091050461,
- "right_hip": 48.232308920725,
- "left_hip": 46.21483093147516,
- "left_knee": 51.6379216565422,
- "left_ankle": 55.15392847913113,
- "right_wrist": 73.22921974084778,
- "right_elbow": 39.89184884543998,
- "right_shoulder": 45.82396616989967,
- "left_shoulder": 41.23639757388701,
- "left_elbow": 45.26357486723375,
- "left_wrist": 77.15002388506178
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.75128520534701
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.619706182525
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.717892706485785
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 48.960745360005525
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.82295903842518
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 64.4191570799788
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 45.62177975788789
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 57.68495620961139
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 44.61986511246689
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 41.220062121362076
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 45.70790998316626
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.20449012653489
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.57302970712051
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.92256916820521
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 73.36243133878057
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 51.736346133447334
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.789184014523
- }
- }
- },
- "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g10m75_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 379,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 74.64705614317401,
- "PA_MPJPE_mm": 49.78386681105657,
- "PCK3D_50mm": 0.6674895241256276,
- "PCK3D_100mm": 0.9205268362167919,
- "AUC_0_150mm": 0.6789931896221368,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 35.88385207435999,
- "left_hip": 37.94785686502172,
- "right_hip": 37.06285126246968,
- "spine1": 42.76267969432163,
- "left_knee": 50.71380958273024,
- "right_knee": 49.4927005345643,
- "spine2": 43.11070060575879,
- "left_ankle": 54.312535575178835,
- "right_ankle": 55.23578793505453,
- "spine3": 40.82588549448561,
- "left_foot": 61.943605063752905,
- "right_foot": 62.31787095304231,
- "neck": 32.27320070629859,
- "left_collar": 27.68697287883084,
- "right_collar": 28.638018284063218,
- "head": 37.77404072173092,
- "left_shoulder": 37.76272400637788,
- "right_shoulder": 39.738857088277825,
- "left_elbow": 49.728635281214196,
- "right_elbow": 46.048989195618624,
- "left_wrist": 74.59084409256067,
- "right_wrist": 71.95198701071394,
- "left_hand": 88.1033961289566,
- "right_hand": 88.9050024299762
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.49980519509622,
- "PA_MPJPE_mm": 56.52353229359914,
- "PCK3D_50mm": 0.561169030529691,
- "PCK3D_100mm": 0.8903011480491462,
- "AUC_0_150mm": 0.6318002367336342,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.715732640490835,
- "right_knee": 47.259121584150066,
- "right_hip": 49.51084499813961,
- "left_hip": 46.36396132027236,
- "left_knee": 51.031452951875615,
- "left_ankle": 57.67261217469871,
- "right_wrist": 73.58258248828862,
- "right_elbow": 41.814459256852466,
- "right_shoulder": 53.50704470505506,
- "left_shoulder": 42.038295533746556,
- "left_elbow": 47.07056687542915,
- "left_wrist": 79.24491691654684,
- "neck": 42.21480201661142,
- "head": 103.3030586482306
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.59142235589435,
- "PA_MPJPE_mm": 51.5080079825892,
- "PCK3D_50mm": 0.6277710935015005,
- "PCK3D_100mm": 0.9214205700535203,
- "AUC_0_150mm": 0.6650865305215654,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.34935453311017,
- "right_knee": 47.79381292361822,
- "right_hip": 46.55293368883342,
- "left_hip": 44.156463168849804,
- "left_knee": 51.48580083970807,
- "left_ankle": 54.77494604267369,
- "right_wrist": 71.9173531339937,
- "right_elbow": 39.15081653137483,
- "right_shoulder": 46.486812845112695,
- "left_shoulder": 40.74338856266846,
- "left_elbow": 44.373185151847395,
- "left_wrist": 76.31122836928
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.97784985305192
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.946491083303535
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.38950632537735
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.899611825578724
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.25073153181695
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 64.61845465317312
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 43.55307080194591
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 57.131354436397054
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 45.43696388205539
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 40.91653856627333
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 44.297608747778284
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 48.19417436756266
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.869697548561774
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 48.75064823787249
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.38597505041768
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 54.41338294621959
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 52.690622532346396
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen10_latency_local.json b/benchmark/results/gen10_latency_local.json
deleted file mode 100644
index da9e0c5..0000000
--- a/benchmark/results/gen10_latency_local.json
+++ /dev/null
@@ -1,88 +0,0 @@
-{
- "machine": {
- "cpu": "x86_64",
- "onnxruntime": "1.26.0"
- },
- "rows": [
- {
- "model": "g10b_b0_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 12.436088999493222,
- "p90_ms": 12.513574599779531
- },
- "cpu_1t": {
- "median_ms": 33.831027500127675,
- "p90_ms": 37.82274789955409
- },
- "cuda": {
- "median_ms": 2.8496525001173723,
- "p90_ms": 3.3210485996278294
- }
- },
- {
- "model": "g10b_b1_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 13.285763500334724,
- "p90_ms": 14.33074509996004
- },
- "cpu_1t": {
- "median_ms": 46.143122000103176,
- "p90_ms": 50.29675979994863
- },
- "cuda": {
- "median_ms": 3.2948635002867377,
- "p90_ms": 3.9304144998823176
- }
- },
- {
- "model": "g10b_b2_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 17.04834700012725,
- "p90_ms": 19.90896669931317
- },
- "cpu_1t": {
- "median_ms": 63.29742699972485,
- "p90_ms": 66.5183552999224
- },
- "cuda": {
- "median_ms": 3.601899999921443,
- "p90_ms": 4.04615989928061
- }
- },
- {
- "model": "g10b_b4_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 24.630298499687342,
- "p90_ms": 25.78717449978285
- },
- "cpu_1t": {
- "median_ms": 92.95684900007473,
- "p90_ms": 102.16233960018144
- },
- "cuda": {
- "median_ms": 3.463431000000128,
- "p90_ms": 3.4955191003973596
- }
- },
- {
- "model": "g10m75_b4_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 24.50892599972576,
- "p90_ms": 24.606630499602034
- },
- "cpu_1t": {
- "median_ms": 92.36904200042773,
- "p90_ms": 92.79409869996016
- },
- "cuda": {
- "median_ms": 3.7649220007551776,
- "p90_ms": 4.449848999774986
- }
- }
- ]
-}
\ No newline at end of file
diff --git a/benchmark/results/gen6_3dpw_j14adapter.json b/benchmark/results/gen6_3dpw_j14adapter.json
deleted file mode 100644
index 2049f00..0000000
--- a/benchmark/results/gen6_3dpw_j14adapter.json
+++ /dev/null
@@ -1,290 +0,0 @@
-{
- "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 102,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 109.31060473532123,
- "PA_MPJPE_mm": 69.03721010395624,
- "PCK3D_50mm": 0.39503546956385127,
- "PCK3D_100mm": 0.8118159369322554,
- "AUC_0_150mm": 0.5504897620951336,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.5568811480596,
- "right_knee": 47.95772431021306,
- "right_hip": 99.03357432412425,
- "left_hip": 88.40061029784397,
- "left_knee": 47.56725775636462,
- "left_ankle": 70.29666472194043,
- "right_wrist": 73.90936250413914,
- "right_elbow": 60.21047147727324,
- "right_shoulder": 48.7594881479906,
- "left_shoulder": 50.23575878992591,
- "left_elbow": 70.73785149884611,
- "left_wrist": 77.19405228378567,
- "neck": 34.421872822977555,
- "head": 128.23937137190305
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 107.94474039350453,
- "PA_MPJPE_mm": 64.11260767952042,
- "PCK3D_50mm": 0.428681536625027,
- "PCK3D_100mm": 0.8689455676808693,
- "AUC_0_150mm": 0.5801218989714861,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.42470164585065,
- "right_knee": 49.69509364988518,
- "right_hip": 98.15725842851955,
- "left_hip": 87.51582399401904,
- "left_knee": 48.39252151040565,
- "left_ankle": 65.18918021243668,
- "right_wrist": 70.85586623493876,
- "right_elbow": 51.488877615927244,
- "right_shoulder": 50.59903888387248,
- "left_shoulder": 51.03028577756485,
- "left_elbow": 57.469193567248524,
- "left_wrist": 73.53345063357644
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 75.0772344122537,
- "PA_MPJPE_mm": 44.465876009326585,
- "PCK3D_50mm": 0.696097743724848,
- "PCK3D_100mm": 0.9355042076047068,
- "AUC_0_150mm": 0.7058889572840962
- }
- ]
- },
- "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 247,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 107.50022500460784,
- "PA_MPJPE_mm": 67.88658382068638,
- "PCK3D_50mm": 0.4046611961762964,
- "PCK3D_100mm": 0.8177819135436935,
- "AUC_0_150mm": 0.5560052789491378,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.87522395933384,
- "right_knee": 47.35189113996917,
- "right_hip": 98.6727272726729,
- "left_hip": 90.38633017011507,
- "left_knee": 47.60500718940778,
- "left_ankle": 67.85544440533677,
- "right_wrist": 70.28749044470575,
- "right_elbow": 59.395211074024495,
- "right_shoulder": 47.97138190216553,
- "left_shoulder": 49.411469179216006,
- "left_elbow": 69.10694208638806,
- "left_wrist": 74.10098938192529,
- "neck": 33.505073863521076,
- "head": 126.88699142082751
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 106.07782437778351,
- "PA_MPJPE_mm": 63.02085164461351,
- "PCK3D_50mm": 0.44028282999182244,
- "PCK3D_100mm": 0.8755886416828808,
- "AUC_0_150mm": 0.5857787110539867,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.46720310312233,
- "right_knee": 49.029458093319604,
- "right_hip": 97.94139039788224,
- "left_hip": 89.66296993519647,
- "left_knee": 48.356036572418745,
- "left_ankle": 63.2247706954782,
- "right_wrist": 67.10022184348587,
- "right_elbow": 50.488811815830545,
- "right_shoulder": 49.32164792363375,
- "left_shoulder": 49.684912271450415,
- "left_elbow": 56.347339522093506,
- "left_wrist": 70.62545756145039
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 71.7470439611344,
- "PA_MPJPE_mm": 42.578590821670765,
- "PCK3D_50mm": 0.7166362526738129,
- "PCK3D_100mm": 0.9450453390052409,
- "AUC_0_150mm": 0.716836370379409
- }
- ]
- },
- "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 225,
- "source": "raw",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 103.79603864551636,
- "PA_MPJPE_mm": 67.23530884349267,
- "PCK3D_50mm": 0.4025342308482483,
- "PCK3D_100mm": 0.8281629545482011,
- "AUC_0_150mm": 0.5595563286509688,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.38307779824238,
- "right_knee": 47.103437333772014,
- "right_hip": 98.13827811820883,
- "left_hip": 88.11436838234566,
- "left_knee": 46.94140350787356,
- "left_ankle": 66.80158151802696,
- "right_wrist": 68.3841582458417,
- "right_elbow": 57.80165606918464,
- "right_shoulder": 49.52095693867013,
- "left_shoulder": 49.73127411968748,
- "left_elbow": 68.30390384132103,
- "left_wrist": 73.84761435117764,
- "neck": 33.726795140357595,
- "head": 126.49581844418778
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 101.60904983902788,
- "PA_MPJPE_mm": 62.23479546950118,
- "PCK3D_50mm": 0.44078335166229593,
- "PCK3D_100mm": 0.8846896765643064,
- "AUC_0_150mm": 0.5899720108099945,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.42316944731556,
- "right_knee": 48.75024941003139,
- "right_hip": 97.4505108204137,
- "left_hip": 87.21530864401828,
- "left_knee": 47.86222433250393,
- "left_ankle": 62.25955806480468,
- "right_wrist": 66.02621371548021,
- "right_elbow": 50.039525918380654,
- "right_shoulder": 49.07085543967181,
- "left_shoulder": 49.70148336401379,
- "left_elbow": 55.46603484511562,
- "left_wrist": 70.55241163226455
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 67.98746005488917,
- "PA_MPJPE_mm": 42.234429549831546,
- "PCK3D_50mm": 0.720356427826185,
- "PCK3D_100mm": 0.9478188534528945,
- "AUC_0_150mm": 0.7183568101289705
- }
- ]
- },
- "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 239,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.38210751397787,
- "PA_MPJPE_mm": 67.15630163795679,
- "PCK3D_50mm": 0.4068445583122852,
- "PCK3D_100mm": 0.8279494523467115,
- "AUC_0_150mm": 0.5599351878526994,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.7665394538419,
- "right_knee": 46.58257021084827,
- "right_hip": 98.5812079766113,
- "left_hip": 87.86653676233917,
- "left_knee": 45.38419871281326,
- "left_ankle": 67.16328284270159,
- "right_wrist": 69.49749995124274,
- "right_elbow": 58.21540961885245,
- "right_shoulder": 48.25533546902176,
- "left_shoulder": 49.95558925474159,
- "left_elbow": 68.64531375386449,
- "left_wrist": 72.17055782128531,
- "neck": 33.67057865370319,
- "head": 129.43360244952788
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.11241461195807,
- "PA_MPJPE_mm": 61.72775478336907,
- "PCK3D_50mm": 0.4480937878916053,
- "PCK3D_100mm": 0.8883319704104747,
- "AUC_0_150mm": 0.5929310239750109,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.87709732727048,
- "right_knee": 48.32575395623139,
- "right_hip": 97.87117835471858,
- "left_hip": 87.07298605453178,
- "left_knee": 46.09251651935203,
- "left_ankle": 61.71549070086557,
- "right_wrist": 66.47290693562417,
- "right_elbow": 49.05797557258036,
- "right_shoulder": 49.54315937498645,
- "left_shoulder": 50.0186382492938,
- "left_elbow": 55.23074208132174,
- "left_wrist": 68.45461227365254
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 68.4703090038315,
- "PA_MPJPE_mm": 41.78863155958053,
- "PCK3D_50mm": 0.7237603780197469,
- "PCK3D_100mm": 0.9499720030132008,
- "AUC_0_150mm": 0.721090106115662
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen6_3dpw_smplfit.json b/benchmark/results/gen6_3dpw_smplfit.json
deleted file mode 100644
index 716954c..0000000
--- a/benchmark/results/gen6_3dpw_smplfit.json
+++ /dev/null
@@ -1,174 +0,0 @@
-{
- "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 102,
- "source": "ema",
- "fit_residual_mm": 16.407779693603516,
- "MPJPE_mm": 75.09175109863281,
- "PA_MPJPE_mm": 51.997389429496,
- "PVE_mm": 88.99562652986062,
- "PA_PVE_mm": 62.232839273349775,
- "J14_h36m_viamesh_MPJPE_mm": 75.06359100341797,
- "J14_h36m_viamesh_PA_MPJPE_mm": 48.0625726362123,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 52.35380204853086,
- "downtown_bar_00": 63.43621329400987,
- "downtown_bus_00": 70.42712580551195,
- "downtown_cafe_00": 47.011574173008455,
- "downtown_car_00": 60.77430019284298,
- "downtown_crossStreets_00": 51.18487757623839,
- "downtown_downstairs_00": 43.35631228850949,
- "downtown_enterShop_00": 44.915920302835474,
- "downtown_rampAndStairs_00": 48.29072450053912,
- "downtown_runForBus_00": 45.24491591290331,
- "downtown_runForBus_01": 56.95027485318371,
- "downtown_sitOnStairs_00": 47.6351770869155,
- "downtown_stairs_00": 60.041978509870916,
- "downtown_upstairs_00": 41.888490125483685,
- "downtown_walkBridge_01": 43.297105145822556,
- "downtown_walkUphill_00": 39.704988938823426,
- "downtown_walking_00": 49.266615736123995,
- "downtown_warmWelcome_00": 56.68671025624492,
- "downtown_weeklyMarket_00": 46.43459014394701,
- "downtown_windowShopping_00": 41.19524987284674,
- "flat_guitar_01": 65.11440430066091,
- "flat_packBags_00": 57.99867125376623,
- "office_phoneCall_00": 41.297467198364956,
- "outdoors_fencing_01": 49.71312080688268
- }
- },
- "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 247,
- "source": "ema",
- "fit_residual_mm": 16.44468116760254,
- "MPJPE_mm": 71.14830780029297,
- "PA_MPJPE_mm": 49.70935360363883,
- "PVE_mm": 84.52926939081257,
- "PA_PVE_mm": 59.69143506594802,
- "J14_h36m_viamesh_MPJPE_mm": 71.13408660888672,
- "J14_h36m_viamesh_PA_MPJPE_mm": 46.101878495835884,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 52.031236927354335,
- "downtown_bar_00": 61.81048529655713,
- "downtown_bus_00": 62.00489752242751,
- "downtown_cafe_00": 45.17636336684825,
- "downtown_car_00": 60.668824113978154,
- "downtown_crossStreets_00": 49.38730167442858,
- "downtown_downstairs_00": 41.66461687753154,
- "downtown_enterShop_00": 40.737916968050165,
- "downtown_rampAndStairs_00": 45.575119645039635,
- "downtown_runForBus_00": 43.856374789751726,
- "downtown_runForBus_01": 56.691850799168535,
- "downtown_sitOnStairs_00": 46.95034291987287,
- "downtown_stairs_00": 61.9412683432234,
- "downtown_upstairs_00": 38.167424639991026,
- "downtown_walkBridge_01": 38.987812005278904,
- "downtown_walkUphill_00": 36.832141521119155,
- "downtown_walking_00": 46.391550816843775,
- "downtown_warmWelcome_00": 54.93313575074026,
- "downtown_weeklyMarket_00": 43.59573050730472,
- "downtown_windowShopping_00": 38.98571115726211,
- "flat_guitar_01": 62.50786710991815,
- "flat_packBags_00": 55.29524804866438,
- "office_phoneCall_00": 39.142479316853695,
- "outdoors_fencing_01": 48.53746768617269
- }
- },
- "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 225,
- "source": "raw",
- "fit_residual_mm": 16.19407844543457,
- "MPJPE_mm": 68.57659149169922,
- "PA_MPJPE_mm": 49.277110368518564,
- "PVE_mm": 81.60751473960043,
- "PA_PVE_mm": 59.19252199670127,
- "J14_h36m_viamesh_MPJPE_mm": 68.0377197265625,
- "J14_h36m_viamesh_PA_MPJPE_mm": 45.818179457288885,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 53.654612794436275,
- "downtown_bar_00": 61.32844290121133,
- "downtown_bus_00": 58.07648931695738,
- "downtown_cafe_00": 45.83578195474714,
- "downtown_car_00": 56.24965576365669,
- "downtown_crossStreets_00": 48.07396249981064,
- "downtown_downstairs_00": 41.53538553211249,
- "downtown_enterShop_00": 38.1628359228303,
- "downtown_rampAndStairs_00": 47.140114824312974,
- "downtown_runForBus_00": 44.17973487466733,
- "downtown_runForBus_01": 55.4910134237619,
- "downtown_sitOnStairs_00": 47.52558021833354,
- "downtown_stairs_00": 59.88327898559083,
- "downtown_upstairs_00": 38.66000085104978,
- "downtown_walkBridge_01": 37.6729155352188,
- "downtown_walkUphill_00": 38.12688843232087,
- "downtown_walking_00": 46.38571317138245,
- "downtown_warmWelcome_00": 57.24636250782035,
- "downtown_weeklyMarket_00": 43.718720459737646,
- "downtown_windowShopping_00": 37.57489380475218,
- "flat_guitar_01": 62.42108643644976,
- "flat_packBags_00": 57.026405168890136,
- "office_phoneCall_00": 40.822180300111086,
- "outdoors_fencing_01": 46.8167621028375
- }
- },
- "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 400,
- "epoch": 239,
- "source": "ema",
- "fit_residual_mm": 16.134763717651367,
- "MPJPE_mm": 68.75614166259766,
- "PA_MPJPE_mm": 48.85976575938138,
- "PVE_mm": 82.17483175804935,
- "PA_PVE_mm": 58.922767700609775,
- "J14_h36m_viamesh_MPJPE_mm": 68.67670440673828,
- "J14_h36m_viamesh_PA_MPJPE_mm": 45.700203014070375,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 52.81382585719828,
- "downtown_bar_00": 59.8867111583646,
- "downtown_bus_00": 59.6588530827453,
- "downtown_cafe_00": 45.07045578806903,
- "downtown_car_00": 55.3646862528783,
- "downtown_crossStreets_00": 48.26229494702743,
- "downtown_downstairs_00": 41.648202117137984,
- "downtown_enterShop_00": 37.82838810971019,
- "downtown_rampAndStairs_00": 46.62054654021483,
- "downtown_runForBus_00": 44.48715914045716,
- "downtown_runForBus_01": 56.257055296433,
- "downtown_sitOnStairs_00": 46.83417974089869,
- "downtown_stairs_00": 58.466226055982474,
- "downtown_upstairs_00": 39.68883644049444,
- "downtown_walkBridge_01": 38.646604867544205,
- "downtown_walkUphill_00": 38.29069005784523,
- "downtown_walking_00": 45.79220479206771,
- "downtown_warmWelcome_00": 56.37613800716904,
- "downtown_weeklyMarket_00": 42.11701588308054,
- "downtown_windowShopping_00": 38.40437632341867,
- "flat_guitar_01": 57.64387081332988,
- "flat_packBags_00": 55.160215101019304,
- "office_phoneCall_00": 39.43539451340168,
- "outdoors_fencing_01": 48.00674327896801
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen6_emdb1_smpl24adapter.json b/benchmark/results/gen6_emdb1_smpl24adapter.json
deleted file mode 100644
index 66aa961..0000000
--- a/benchmark/results/gen6_emdb1_smpl24adapter.json
+++ /dev/null
@@ -1,686 +0,0 @@
-{
- "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 102,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 85.24908665569862,
- "PA_MPJPE_mm": 55.44934350591747,
- "PCK3D_50mm": 0.6303433874068235,
- "PCK3D_100mm": 0.8986848110193751,
- "AUC_0_150mm": 0.6570538000222164,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.44276850297125,
- "left_hip": 41.537323716220456,
- "right_hip": 40.83889567412325,
- "spine1": 47.591581757428976,
- "left_knee": 59.06540487001995,
- "right_knee": 57.68392632974216,
- "spine2": 43.522468816932076,
- "left_ankle": 64.72579890189495,
- "right_ankle": 67.53222921055465,
- "spine3": 40.546135936069575,
- "left_foot": 72.56786270235342,
- "right_foot": 74.28700017742666,
- "neck": 36.16823799839183,
- "left_collar": 30.84593635100392,
- "right_collar": 30.87846376826569,
- "head": 42.869070904273045,
- "left_shoulder": 39.87788877285762,
- "right_shoulder": 40.93670745505024,
- "left_elbow": 54.33933161260382,
- "right_elbow": 51.9918869815113,
- "left_wrist": 79.99968499743431,
- "right_wrist": 79.17244427923963,
- "left_hand": 95.39213276028248,
- "right_hand": 97.97106166536739
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 94.87598513934071,
- "PA_MPJPE_mm": 61.39192918123466,
- "PCK3D_50mm": 0.5332916471571411,
- "PCK3D_100mm": 0.8685640791602705,
- "AUC_0_150mm": 0.6141937701695354,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.30459760585951,
- "right_knee": 54.58202973959393,
- "right_hip": 51.20044830862761,
- "left_hip": 48.2350682050327,
- "left_knee": 57.78033752675744,
- "left_ankle": 69.22462357276686,
- "right_wrist": 80.90026764736525,
- "right_elbow": 46.725279563116594,
- "right_shoulder": 52.81483639467326,
- "left_shoulder": 42.86777081428923,
- "left_elbow": 50.904126458628454,
- "left_wrist": 84.23369414174194,
- "neck": 45.221201369742765,
- "head": 105.49272718908968
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 89.60147429303947,
- "PA_MPJPE_mm": 56.762792878143536,
- "PCK3D_50mm": 0.5970695210831294,
- "PCK3D_100mm": 0.9003650997801104,
- "AUC_0_150mm": 0.6466039117961045,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.86427842066897,
- "right_knee": 54.83300212466379,
- "right_hip": 48.6203023849111,
- "left_hip": 46.57846367605056,
- "left_knee": 57.991104920598715,
- "left_ankle": 65.92114345863513,
- "right_wrist": 78.82243478171007,
- "right_elbow": 44.040242793090364,
- "right_shoulder": 47.426811932820634,
- "left_shoulder": 42.93378383216155,
- "left_elbow": 47.067751536020246,
- "left_wrist": 80.05419467639135
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.66515964403605
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 53.386934652987286
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.30633511102723
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 53.5295522291753
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.29547818782338
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 73.83213823491313
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 49.11944288349437
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 64.321551743386
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 50.815677401932994
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 45.324366828245026
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 49.33170640233391
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.68090116841388
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.94319803020966
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.86315457060261
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 94.36464376588654
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 62.46249814840102
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.968853360928975
- }
- }
- },
- "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 247,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 81.39060362181309,
- "PA_MPJPE_mm": 52.78712376976438,
- "PCK3D_50mm": 0.6439827683967417,
- "PCK3D_100mm": 0.9095340828942455,
- "AUC_0_150mm": 0.6663711160748741,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.01883427645608,
- "left_hip": 41.26715291990504,
- "right_hip": 40.59560413306814,
- "spine1": 47.211135495716064,
- "left_knee": 54.07129993331573,
- "right_knee": 51.22826390004105,
- "spine2": 43.47053100257457,
- "left_ankle": 60.42976835229707,
- "right_ankle": 60.109835959627475,
- "spine3": 40.56446121949907,
- "left_foot": 67.66038088157181,
- "right_foot": 67.20682151149438,
- "neck": 35.49749139737654,
- "left_collar": 30.45434777688604,
- "right_collar": 29.945759278738805,
- "head": 41.14971274272008,
- "left_shoulder": 39.42971880286286,
- "right_shoulder": 38.60963627394683,
- "left_elbow": 52.432809884889636,
- "right_elbow": 48.739740787964216,
- "left_wrist": 76.5847725330413,
- "right_wrist": 74.88977793086886,
- "left_hand": 91.947187695367,
- "right_hand": 93.37592578411665
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.69099578645202,
- "PA_MPJPE_mm": 58.58589537201083,
- "PCK3D_50mm": 0.5510576632428684,
- "PCK3D_100mm": 0.878559871029688,
- "AUC_0_150mm": 0.6242627884820733,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.9398698320647,
- "right_knee": 48.70823929264372,
- "right_hip": 51.26898188028841,
- "left_hip": 47.340733332433125,
- "left_knee": 54.47894716026778,
- "left_ankle": 64.29507565211546,
- "right_wrist": 76.3910190457949,
- "right_elbow": 44.26002490301836,
- "right_shoulder": 51.01331403806369,
- "left_shoulder": 41.39904010703302,
- "left_elbow": 48.234218818912936,
- "left_wrist": 79.76576475431467,
- "neck": 45.215729070168166,
- "head": 104.89157732103257
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.34770762458652,
- "PA_MPJPE_mm": 53.48934341210172,
- "PCK3D_50mm": 0.6205866489648592,
- "PCK3D_100mm": 0.9117986695985285,
- "AUC_0_150mm": 0.6588061362102341,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.094435246763915,
- "right_knee": 48.960518042555194,
- "right_hip": 48.42810003524638,
- "left_hip": 45.36082824221826,
- "left_knee": 54.91487326770962,
- "left_ankle": 61.44566114113791,
- "right_wrist": 74.11045153762892,
- "right_elbow": 41.399519390619325,
- "right_shoulder": 44.423552162499995,
- "left_shoulder": 41.47769336689967,
- "left_elbow": 45.399908223627406,
- "left_wrist": 75.85658028831402
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.27623425052127
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 51.30494088711467
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.190737385397
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.959443277910445
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.399577689238605
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 70.33963063922049
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.56238898164989
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.62643348383337
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.89072501280143
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.27561230353556
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 52.00803125854006
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 51.11736699906316
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.57351587489808
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.41187331370503
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 77.39830882410666
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.65838547226543
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.32869985598444
- }
- }
- },
- "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 225,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 81.67234323853624,
- "PA_MPJPE_mm": 53.14881633351334,
- "PCK3D_50mm": 0.6394622384488791,
- "PCK3D_100mm": 0.9079972755811863,
- "AUC_0_150mm": 0.6634924533108135,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.19292169475212,
- "left_hip": 41.45500628056827,
- "right_hip": 40.541136540400686,
- "spine1": 46.20372459798169,
- "left_knee": 54.69418407839382,
- "right_knee": 53.766838428818396,
- "spine2": 42.712495783690606,
- "left_ankle": 58.50568266377871,
- "right_ankle": 59.61915907264363,
- "spine3": 40.310623689717474,
- "left_foot": 64.72012099983473,
- "right_foot": 65.31113701222193,
- "neck": 35.90498704809899,
- "left_collar": 30.613988391446586,
- "right_collar": 31.018642126117832,
- "head": 42.037296140906605,
- "left_shoulder": 40.34064854072661,
- "right_shoulder": 41.83985460570097,
- "left_elbow": 53.809594392978966,
- "right_elbow": 50.90747734441213,
- "left_wrist": 78.22019178792188,
- "right_wrist": 76.18306708250148,
- "left_hand": 92.8576913233281,
- "right_hand": 94.80512237737673
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.36362163829955,
- "PA_MPJPE_mm": 59.34886674839958,
- "PCK3D_50mm": 0.54367565389015,
- "PCK3D_100mm": 0.8759312711517832,
- "AUC_0_150mm": 0.6205009950575019,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.510409617905154,
- "right_knee": 51.30217662740902,
- "right_hip": 50.842734490587226,
- "left_hip": 46.83592032884032,
- "left_knee": 54.163994918998945,
- "left_ankle": 62.64579203303455,
- "right_wrist": 77.96599707863224,
- "right_elbow": 45.41432359258422,
- "right_shoulder": 55.00612888634326,
- "left_shoulder": 42.39492242978502,
- "left_elbow": 50.2108397276734,
- "left_wrist": 83.31423575103972,
- "neck": 45.04362486192491,
- "head": 104.23303413283618
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.37051494700557,
- "PA_MPJPE_mm": 54.52220104838302,
- "PCK3D_50mm": 0.6091012183822208,
- "PCK3D_100mm": 0.9083447427014618,
- "AUC_0_150mm": 0.6533335430069609,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 58.69641833400611,
- "right_knee": 51.3839495031858,
- "right_hip": 48.12415629259117,
- "left_hip": 45.04815504463009,
- "left_knee": 54.59121129051678,
- "left_ankle": 59.68592516087166,
- "right_wrist": 75.773909756642,
- "right_elbow": 43.09055455498249,
- "right_shoulder": 48.35399819031066,
- "left_shoulder": 42.37838978201944,
- "left_elbow": 47.7645840092948,
- "left_wrist": 79.37516066154522
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.04874761935634
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.26032023685678
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.06371790301442
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.849367348358705
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.16156904715849
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.3372732833773
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.52030048095535
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 59.71073877006052
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.350753318583045
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.158471445028894
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.116466258401836
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 51.02608445976944
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 43.69734517983646
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 48.708660854338866
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 84.31221432789145
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.78676368209051
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.144779093609436
- }
- }
- },
- "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "epoch": 239,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 80.8388164639782,
- "PA_MPJPE_mm": 52.492057030998936,
- "PCK3D_50mm": 0.6409264406920301,
- "PCK3D_100mm": 0.9122636877843698,
- "AUC_0_150mm": 0.6665234082760411,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.5173001947221,
- "left_hip": 40.93687239313311,
- "right_hip": 40.470826420546786,
- "spine1": 46.292761894118335,
- "left_knee": 54.5434503733806,
- "right_knee": 53.1692009948552,
- "spine2": 42.78351268865345,
- "left_ankle": 58.67837996388447,
- "right_ankle": 60.14529867036297,
- "spine3": 39.88261944112938,
- "left_foot": 66.37824342178689,
- "right_foot": 66.60436949815637,
- "neck": 36.20816598036446,
- "left_collar": 30.61573052378412,
- "right_collar": 30.349233402107203,
- "head": 42.12061096013422,
- "left_shoulder": 39.739920516958506,
- "right_shoulder": 39.54386087105362,
- "left_elbow": 52.105730853592775,
- "right_elbow": 49.65800015055759,
- "left_wrist": 75.16283400303837,
- "right_wrist": 73.57376719343458,
- "left_hand": 89.72087722264682,
- "right_hand": 91.6078011115718
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.59636844538235,
- "PA_MPJPE_mm": 58.58564589932577,
- "PCK3D_50mm": 0.550325685599303,
- "PCK3D_100mm": 0.8768588379632648,
- "AUC_0_150mm": 0.6237037437831608,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.70154315537687,
- "right_knee": 50.64757250784787,
- "right_hip": 50.828102396842795,
- "left_hip": 47.34674906877681,
- "left_knee": 53.83874414824775,
- "left_ankle": 63.660653499345635,
- "right_wrist": 74.4958157667914,
- "right_elbow": 44.5602167982945,
- "right_shoulder": 51.99381055264517,
- "left_shoulder": 41.45297960362028,
- "left_elbow": 48.030263424886854,
- "left_wrist": 78.53945031747034,
- "neck": 45.57499282103069,
- "head": 106.52814852938386
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.66504711722767,
- "PA_MPJPE_mm": 53.203264963279565,
- "PCK3D_50mm": 0.6213680178124438,
- "PCK3D_100mm": 0.9131712511582237,
- "AUC_0_150mm": 0.6596636790405334,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.62621825720753,
- "right_knee": 50.72520946117408,
- "right_hip": 48.02264267888712,
- "left_hip": 45.093007144744185,
- "left_knee": 54.112639805033346,
- "left_ankle": 59.93621495766321,
- "right_wrist": 72.17271131220497,
- "right_elbow": 41.82290591142253,
- "right_shoulder": 45.720285663919114,
- "left_shoulder": 41.50812120327427,
- "left_elbow": 45.170328955465074,
- "left_wrist": 74.52889420835932
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.22893034085069
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.10120372897257
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.713349838206234
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.84240167090533
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.54020753726515
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.32164948268385
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.44759077309669
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.96885962140703
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.42842266255204
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.1381303991415
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 52.07759183937822
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.28301493840896
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.115564045640156
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 47.02788058784131
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 82.59746344528897
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.80553886673358
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 53.78672892188069
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen6_emdb1_smplfit.json b/benchmark/results/gen6_emdb1_smplfit.json
deleted file mode 100644
index d2f5dea..0000000
--- a/benchmark/results/gen6_emdb1_smplfit.json
+++ /dev/null
@@ -1,146 +0,0 @@
-{
- "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6b_s/g6b_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 102,
- "source": "ema",
- "fit_residual_mm": 14.30030345916748,
- "MPJPE_mm": 82.50424194335938,
- "PA_MPJPE_mm": 53.630448317554475,
- "PVE_mm": 97.47888544609019,
- "PA_PVE_mm": 63.253926476830515,
- "J14_h36m_viamesh_MPJPE_mm": 80.97623443603516,
- "J14_h36m_viamesh_PA_MPJPE_mm": 49.348192690589,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 40.19883884313969,
- "P2_23_outdoor_hug_tree": 52.456288460241744,
- "P3_31_outdoor_workout": 51.53924413457358,
- "P3_32_outdoor_soccer_warmup_a": 52.70549120145435,
- "P3_33_outdoor_soccer_warmup_b": 48.09691879414547,
- "P5_42_indoor_dancing": 72.86494091057435,
- "P5_44_indoor_rom": 48.81177007852985,
- "P6_49_outdoor_big_stairs_down": 62.43381219712345,
- "P6_50_outdoor_workout": 49.38750120275361,
- "P6_51_outdoor_dancing": 44.805854253346425,
- "P7_57_outdoor_rock_chair": 46.358417588952314,
- "P7_59_outdoor_rom": 52.27356134260238,
- "P7_60_outdoor_workout": 43.69644338518449,
- "P8_64_outdoor_skateboard": 47.828347816320175,
- "P8_68_outdoor_handstand": 92.90519698038229,
- "P8_69_outdoor_cartwheel": 59.74083776717741,
- "P9_76_outdoor_sitting": 48.22672952341287
- }
- },
- "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6bv_s/g6bv_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 247,
- "source": "ema",
- "fit_residual_mm": 14.43234634399414,
- "MPJPE_mm": 78.12178039550781,
- "PA_MPJPE_mm": 50.73391808179524,
- "PVE_mm": 92.54210270906181,
- "PA_PVE_mm": 60.60177938772959,
- "J14_h36m_viamesh_MPJPE_mm": 76.23255157470703,
- "J14_h36m_viamesh_PA_MPJPE_mm": 46.21968662656892,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 37.38836410475635,
- "P2_23_outdoor_hug_tree": 50.25822391648926,
- "P3_31_outdoor_workout": 50.2125611592255,
- "P3_32_outdoor_soccer_warmup_a": 48.83342154213949,
- "P3_33_outdoor_soccer_warmup_b": 46.20071574026429,
- "P5_42_indoor_dancing": 69.61155737449016,
- "P5_44_indoor_rom": 46.18514740454893,
- "P6_49_outdoor_big_stairs_down": 59.273861728340805,
- "P6_50_outdoor_workout": 46.88291877013638,
- "P6_51_outdoor_dancing": 42.430497456679575,
- "P7_57_outdoor_rock_chair": 48.84798872367035,
- "P7_59_outdoor_rom": 49.12946709631032,
- "P7_60_outdoor_workout": 42.909592200731446,
- "P8_64_outdoor_skateboard": 46.96549114523738,
- "P8_68_outdoor_handstand": 75.75563453564564,
- "P8_69_outdoor_cartwheel": 55.808674554177585,
- "P9_76_outdoor_sitting": 47.93093974219468
- }
- },
- "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6v_s/g6v_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 225,
- "source": "raw",
- "fit_residual_mm": 14.628873825073242,
- "MPJPE_mm": 78.38782501220703,
- "PA_MPJPE_mm": 51.342255464859115,
- "PVE_mm": 92.74704889821241,
- "PA_PVE_mm": 61.62021675354329,
- "J14_h36m_viamesh_MPJPE_mm": 76.76594543457031,
- "J14_h36m_viamesh_PA_MPJPE_mm": 47.25285211533647,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 38.18086105192671,
- "P2_23_outdoor_hug_tree": 48.9477478015161,
- "P3_31_outdoor_workout": 50.25096136567123,
- "P3_32_outdoor_soccer_warmup_a": 52.17091963663792,
- "P3_33_outdoor_soccer_warmup_b": 48.89497629299315,
- "P5_42_indoor_dancing": 67.59165316745444,
- "P5_44_indoor_rom": 48.250051509854664,
- "P6_49_outdoor_big_stairs_down": 58.111135330682956,
- "P6_50_outdoor_workout": 46.77052387379077,
- "P6_51_outdoor_dancing": 42.87896974838975,
- "P7_57_outdoor_rock_chair": 46.755298978881,
- "P7_59_outdoor_rom": 49.44025848980838,
- "P7_60_outdoor_workout": 41.04483850742212,
- "P8_64_outdoor_skateboard": 46.89091605881439,
- "P8_68_outdoor_handstand": 83.20208888949294,
- "P8_69_outdoor_cartwheel": 57.45923317591712,
- "P9_76_outdoor_sitting": 48.82811972679905
- }
- },
- "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g6vv_s/g6vv_s/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 400,
- "epoch": 239,
- "source": "ema",
- "fit_residual_mm": 14.366242408752441,
- "MPJPE_mm": 77.56647491455078,
- "PA_MPJPE_mm": 50.68995444446009,
- "PVE_mm": 92.34008350832515,
- "PA_PVE_mm": 60.63477876512061,
- "J14_h36m_viamesh_MPJPE_mm": 76.6056900024414,
- "J14_h36m_viamesh_PA_MPJPE_mm": 46.800030606742794,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 37.348505120306996,
- "P2_23_outdoor_hug_tree": 48.915155503100486,
- "P3_31_outdoor_workout": 51.1337024534537,
- "P3_32_outdoor_soccer_warmup_a": 50.384480748001124,
- "P3_33_outdoor_soccer_warmup_b": 46.83623959506515,
- "P5_42_indoor_dancing": 68.04583433850948,
- "P5_44_indoor_rom": 47.43597349329044,
- "P6_49_outdoor_big_stairs_down": 57.029334946014046,
- "P6_50_outdoor_workout": 44.73937441685114,
- "P6_51_outdoor_dancing": 42.55079609281225,
- "P7_57_outdoor_rock_chair": 48.73104939849944,
- "P7_59_outdoor_rom": 48.61240233840695,
- "P7_60_outdoor_workout": 41.175174065305455,
- "P8_64_outdoor_skateboard": 45.14990130797402,
- "P8_68_outdoor_handstand": 81.86023152094495,
- "P8_69_outdoor_cartwheel": 57.25147003355788,
- "P9_76_outdoor_sitting": 47.962357497378235
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen7_3dpw_test.json b/benchmark/results/gen7_3dpw_test.json
deleted file mode 100644
index 31ed650..0000000
--- a/benchmark/results/gen7_3dpw_test.json
+++ /dev/null
@@ -1,146 +0,0 @@
-{
- "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 169,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 106.55963232427057,
- "PA_MPJPE_mm": 67.59828859909312,
- "PCK3D_50mm": 0.3942298008789845,
- "PCK3D_100mm": 0.8283099890831893,
- "AUC_0_150mm": 0.5574437873913111,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.46382356533427,
- "right_knee": 47.25597372496897,
- "right_hip": 98.07144410741707,
- "left_hip": 87.49522052231794,
- "left_knee": 44.323743383557606,
- "left_ankle": 66.46594863836347,
- "right_wrist": 69.61139208366498,
- "right_elbow": 58.25602058897865,
- "right_shoulder": 51.48491089262547,
- "left_shoulder": 53.19574911798813,
- "left_elbow": 67.50110591724483,
- "left_wrist": 73.81250672134334,
- "neck": 31.457780983702975,
- "head": 132.9804201397962
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 104.86066023996102,
- "PA_MPJPE_mm": 61.927791610931955,
- "PCK3D_50mm": 0.4436619387342676,
- "PCK3D_100mm": 0.8896525956630855,
- "AUC_0_150mm": 0.5917538164114106,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.39738917874813,
- "right_knee": 49.284650222721915,
- "right_hip": 98.12497192706117,
- "left_hip": 86.9643206519134,
- "left_knee": 45.27319245312406,
- "left_ankle": 61.848608069967995,
- "right_wrist": 65.83881379194949,
- "right_elbow": 49.28868611126632,
- "right_shoulder": 50.777514327694306,
- "left_shoulder": 50.06590186895298,
- "left_elbow": 54.691430452746104,
- "left_wrist": 69.57802027503759
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 70.8794869952138,
- "PA_MPJPE_mm": 41.55189456354803,
- "PCK3D_50mm": 0.728433256391974,
- "PCK3D_100mm": 0.9495832678727527,
- "AUC_0_150mm": 0.7225466771299639
- }
- ]
- },
- "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 107,
- "source": "ema",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.31958983589064,
- "PA_MPJPE_mm": 68.61191926982534,
- "PCK3D_50mm": 0.38391522754097834,
- "PCK3D_100mm": 0.8224024234514041,
- "AUC_0_150mm": 0.5514124476591491,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.17333511415941,
- "right_knee": 47.94372228787508,
- "right_hip": 98.87687339306508,
- "left_hip": 87.5800866771523,
- "left_knee": 46.8765331171045,
- "left_ankle": 69.20124114022532,
- "right_wrist": 72.72554456354305,
- "right_elbow": 59.11847627456244,
- "right_shoulder": 50.7956029353215,
- "left_shoulder": 52.96843907321996,
- "left_elbow": 67.68644517146863,
- "left_wrist": 76.67468336414615,
- "neck": 33.0633611593688,
- "head": 129.8825255063427
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 103.572053244377,
- "PA_MPJPE_mm": 63.41503431210888,
- "PCK3D_50mm": 0.4229102632790984,
- "PCK3D_100mm": 0.8805163127766968,
- "AUC_0_150mm": 0.5829835063593466,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.46866831168299,
- "right_knee": 49.84423219735444,
- "right_hip": 98.35840224615846,
- "left_hip": 87.07441880924338,
- "left_knee": 47.862718711425885,
- "left_ankle": 64.21569158868604,
- "right_wrist": 69.26732158409469,
- "right_elbow": 50.22142636256743,
- "right_shoulder": 51.26354789935885,
- "left_shoulder": 51.500077945407234,
- "left_elbow": 54.932207994752424,
- "left_wrist": 72.97169809457472
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 69.35152614973077,
- "PA_MPJPE_mm": 43.032953201743574,
- "PCK3D_50mm": 0.7107931405368171,
- "PCK3D_100mm": 0.9452286286310481,
- "AUC_0_150mm": 0.713739613858593
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen7_3dpw_test_pve.json b/benchmark/results/gen7_3dpw_test_pve.json
deleted file mode 100644
index 611d044..0000000
--- a/benchmark/results/gen7_3dpw_test_pve.json
+++ /dev/null
@@ -1,96 +0,0 @@
-{
- "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 107,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 82.27373083751675,
- "PA_PVE_direct_mm": 59.657708764309405,
- "J14_h36m_direct_MPJPE_mm": 68.7114028930664,
- "J14_h36m_direct_PA_MPJPE_mm": 44.02579726076463,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 49.62962688554044,
- "downtown_bar_00": 54.11500865395675,
- "downtown_bus_00": 50.39796642534758,
- "downtown_cafe_00": 41.10439154927856,
- "downtown_car_00": 48.80962068134795,
- "downtown_crossStreets_00": 44.477837978866226,
- "downtown_downstairs_00": 36.81204628219283,
- "downtown_enterShop_00": 33.22070985457979,
- "downtown_rampAndStairs_00": 43.9082529136217,
- "downtown_runForBus_00": 40.00761101306935,
- "downtown_runForBus_01": 51.15868443843556,
- "downtown_sitOnStairs_00": 41.84408874824941,
- "downtown_stairs_00": 53.271430467594065,
- "downtown_upstairs_00": 33.72264917320064,
- "downtown_walkBridge_01": 34.46676736487242,
- "downtown_walkUphill_00": 32.260865939077334,
- "downtown_walking_00": 42.06071908711886,
- "downtown_warmWelcome_00": 51.4704668615721,
- "downtown_weeklyMarket_00": 39.35185763634684,
- "downtown_windowShopping_00": 35.0696818991034,
- "flat_guitar_01": 48.523492054513405,
- "flat_packBags_00": 46.271696714928744,
- "office_phoneCall_00": 36.77163754344973,
- "outdoors_fencing_01": 48.09862701234846
- }
- },
- "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 169,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 83.25301935126727,
- "PA_PVE_direct_mm": 57.682906672823414,
- "J14_h36m_direct_MPJPE_mm": 70.24108123779297,
- "J14_h36m_direct_PA_MPJPE_mm": 42.573140311864826,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 47.25237549523189,
- "downtown_bar_00": 53.73080081795746,
- "downtown_bus_00": 50.87531008381287,
- "downtown_cafe_00": 40.45822422213792,
- "downtown_car_00": 47.36230945356504,
- "downtown_crossStreets_00": 43.407753243808436,
- "downtown_downstairs_00": 34.74479702121051,
- "downtown_enterShop_00": 32.34710946868719,
- "downtown_rampAndStairs_00": 41.95225779839528,
- "downtown_runForBus_00": 38.77355803170025,
- "downtown_runForBus_01": 48.46377407524906,
- "downtown_sitOnStairs_00": 38.92934264777041,
- "downtown_stairs_00": 51.262523536579835,
- "downtown_upstairs_00": 33.62923976951196,
- "downtown_walkBridge_01": 33.74466386916976,
- "downtown_walkUphill_00": 31.325529746085973,
- "downtown_walking_00": 41.548266517244514,
- "downtown_warmWelcome_00": 48.33051806836209,
- "downtown_weeklyMarket_00": 37.68580493959419,
- "downtown_windowShopping_00": 34.44469679633343,
- "flat_guitar_01": 45.86799152143159,
- "flat_packBags_00": 43.24300254496675,
- "office_phoneCall_00": 35.467951031373794,
- "outdoors_fencing_01": 43.78394915687843
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen7_3dpw_val_stride5.json b/benchmark/results/gen7_3dpw_val_stride5.json
deleted file mode 100644
index 6001c01..0000000
--- a/benchmark/results/gen7_3dpw_val_stride5.json
+++ /dev/null
@@ -1,146 +0,0 @@
-{
- "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "validation",
- "stride": 5,
- "epoch": 169,
- "source": "ema",
- "n": 1993,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 1993,
- "MPJPE_mm": 103.90234720126412,
- "PA_MPJPE_mm": 63.38082033985036,
- "PCK3D_50mm": 0.44731560461615655,
- "PCK3D_100mm": 0.8547415955845459,
- "AUC_0_150mm": 0.581793188602505,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.85948093194728,
- "right_knee": 48.547847304625115,
- "right_hip": 93.48160213603958,
- "left_hip": 79.84831634941624,
- "left_knee": 42.368813230012584,
- "left_ankle": 57.1097939616838,
- "right_wrist": 74.59096379795069,
- "right_elbow": 56.99275884049035,
- "right_shoulder": 46.58418912944933,
- "left_shoulder": 47.12430108487619,
- "left_elbow": 62.35690634119856,
- "left_wrist": 70.64812134640509,
- "neck": 28.415358839382485,
- "head": 122.40303146442776
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 1993,
- "MPJPE_mm": 101.33608351059434,
- "PA_MPJPE_mm": 58.47637562965319,
- "PCK3D_50mm": 0.49790934938953,
- "PCK3D_100mm": 0.9057534704800134,
- "AUC_0_150mm": 0.6126064883004495,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 53.655452915405846,
- "right_knee": 50.538598104998165,
- "right_hip": 94.88849772018665,
- "left_hip": 80.98289262973533,
- "left_knee": 42.88079426469913,
- "left_ankle": 54.47031725378426,
- "right_wrist": 68.33861065226719,
- "right_elbow": 49.76684152737039,
- "right_shoulder": 44.36023974855144,
- "left_shoulder": 42.89197693615923,
- "left_elbow": 53.18353818410083,
- "left_wrist": 65.75874761857979
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 1993,
- "MPJPE_mm": 71.46007771292817,
- "PA_MPJPE_mm": 39.93041587262252,
- "PCK3D_50mm": 0.7482976130743316,
- "PCK3D_100mm": 0.9567414522256469,
- "AUC_0_150mm": 0.7317373479991028
- }
- ]
- },
- "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "validation",
- "stride": 5,
- "epoch": 107,
- "source": "ema",
- "n": 1993,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 1993,
- "MPJPE_mm": 104.55982490458405,
- "PA_MPJPE_mm": 65.93545723714554,
- "PCK3D_50mm": 0.42409146297756434,
- "PCK3D_100mm": 0.8353164647695506,
- "AUC_0_150mm": 0.568342886739533,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.19624479582436,
- "right_knee": 50.74390423789055,
- "right_hip": 95.92368380209209,
- "left_hip": 81.13944986939818,
- "left_knee": 45.85929759842022,
- "left_ankle": 62.16279393248126,
- "right_wrist": 80.60291493429548,
- "right_elbow": 59.837420844358356,
- "right_shoulder": 48.020763238307765,
- "left_shoulder": 46.653383851617825,
- "left_elbow": 62.65661110302429,
- "left_wrist": 77.38705180161271,
- "neck": 29.98179263390863,
- "head": 117.93108867680598
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 1993,
- "MPJPE_mm": 101.93417681696192,
- "PA_MPJPE_mm": 61.96334827986432,
- "PCK3D_50mm": 0.46483525673189496,
- "PCK3D_100mm": 0.8771951831409934,
- "AUC_0_150mm": 0.5936517057011368,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.675849979517814,
- "right_knee": 52.61648682866706,
- "right_hip": 96.62411961742293,
- "left_hip": 82.1574260874414,
- "left_knee": 46.535775669858026,
- "left_ankle": 59.28138744913558,
- "right_wrist": 74.83031419970418,
- "right_elbow": 52.28211428433266,
- "right_shoulder": 46.93795907879271,
- "left_shoulder": 43.69254107647294,
- "left_elbow": 54.286340260518216,
- "left_wrist": 73.63986482650833
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 1993,
- "MPJPE_mm": 71.92407096255458,
- "PA_MPJPE_mm": 43.644587072587775,
- "PCK3D_50mm": 0.705684180345495,
- "PCK3D_100mm": 0.9385707117769335,
- "AUC_0_150mm": 0.7105734124736115
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen7_emdb1_fitted.json b/benchmark/results/gen7_emdb1_fitted.json
deleted file mode 100644
index d2cb2b8..0000000
--- a/benchmark/results/gen7_emdb1_fitted.json
+++ /dev/null
@@ -1,346 +0,0 @@
-{
- "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 107,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 83.10537542543,
- "PA_MPJPE_mm": 57.27129473548358,
- "PCK3D_50mm": 0.6022106515094939,
- "PCK3D_100mm": 0.8879150589829758,
- "AUC_0_150mm": 0.6438961107326576,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.554537776576815,
- "left_hip": 40.88342485356299,
- "right_hip": 40.556785981476324,
- "spine1": 46.094981880985294,
- "left_knee": 59.55280670872391,
- "right_knee": 58.12076431735593,
- "spine2": 44.88325954547466,
- "left_ankle": 67.34936901502714,
- "right_ankle": 68.76173911584449,
- "spine3": 43.29432355470121,
- "left_foot": 75.51432595448246,
- "right_foot": 76.52723029793168,
- "neck": 34.37678107176358,
- "left_collar": 30.34213756464158,
- "right_collar": 32.49547180852728,
- "head": 40.70670852197265,
- "left_shoulder": 40.90763132387344,
- "right_shoulder": 43.28605041537219,
- "left_elbow": 55.460948978650926,
- "right_elbow": 51.57767989347048,
- "left_wrist": 88.22072490895478,
- "right_wrist": 84.39317433802674,
- "left_hand": 105.79151249016327,
- "right_hand": 105.85870333404564
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 92.15800897593506,
- "PA_MPJPE_mm": 62.899444205848745,
- "PCK3D_50mm": 0.5054764967016554,
- "PCK3D_100mm": 0.8613598781420214,
- "AUC_0_150mm": 0.6020410484879504,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.22016985698207,
- "right_knee": 55.77798922333778,
- "right_hip": 50.027186001583736,
- "left_hip": 47.33849313518418,
- "left_knee": 58.78539985950967,
- "left_ankle": 70.57859302033872,
- "right_wrist": 86.06771608637555,
- "right_elbow": 46.70124162278418,
- "right_shoulder": 57.522100384180064,
- "left_shoulder": 45.67913084537015,
- "left_elbow": 52.14206245235403,
- "left_wrist": 92.10036070263786,
- "neck": 44.62703270690047,
- "head": 103.02474298434394
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 87.80247927294556,
- "PA_MPJPE_mm": 58.85905630411169,
- "PCK3D_50mm": 0.5654482844459195,
- "PCK3D_100mm": 0.8857541938071333,
- "AUC_0_150mm": 0.6304206766747906,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.07369445982394,
- "right_knee": 56.2752004524805,
- "right_hip": 47.63480892268604,
- "left_hip": 45.54955802912199,
- "left_knee": 59.2016881854235,
- "left_ankle": 68.2667954786239,
- "right_wrist": 84.114370923369,
- "right_elbow": 43.61140196684028,
- "right_shoulder": 51.081315578658966,
- "left_shoulder": 44.379727692648544,
- "left_elbow": 49.02587107130371,
- "left_wrist": 89.09424288835987
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 45.53992428925261
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 53.8866814882433
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 54.91493680596014
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 56.2548156489873
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 50.41464294583603
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 76.8118144949826
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.59038292874356
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 63.68729818036881
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 54.516750726545055
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 48.11876941245566
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 52.04310734475353
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 58.28440425091945
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 49.19472747222995
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 55.949613884246624
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 79.0143266412795
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 65.66463077080004
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.99795952378006
- }
- }
- },
- "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 169,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 80.32957778215987,
- "PA_MPJPE_mm": 52.35941511424271,
- "PCK3D_50mm": 0.6470805847128297,
- "PCK3D_100mm": 0.9092194609246429,
- "AUC_0_150mm": 0.6673331053690279,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.898463181811856,
- "left_hip": 38.0451426565691,
- "right_hip": 38.442613281696275,
- "spine1": 43.78267260306844,
- "left_knee": 53.67533264307888,
- "right_knee": 53.86078872362915,
- "spine2": 43.2957811667419,
- "left_ankle": 58.34624044814604,
- "right_ankle": 58.23222760451015,
- "spine3": 41.57724684219601,
- "left_foot": 67.21083656256306,
- "right_foot": 66.01507840584965,
- "neck": 34.25600680203047,
- "left_collar": 29.29816639457542,
- "right_collar": 30.528908365678806,
- "head": 39.98071279601301,
- "left_shoulder": 39.16366342358563,
- "right_shoulder": 40.806733633931735,
- "left_elbow": 53.047402854164275,
- "right_elbow": 48.45017650968431,
- "left_wrist": 78.5398470555618,
- "right_wrist": 75.23460123739079,
- "left_hand": 93.93856160906871,
- "right_hand": 93.99875794028063
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 90.7562584976495,
- "PA_MPJPE_mm": 58.70385827851572,
- "PCK3D_50mm": 0.5448432619531652,
- "PCK3D_100mm": 0.8811025302125995,
- "AUC_0_150mm": 0.6221603483207914,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.8380665355502,
- "right_knee": 51.677977231532985,
- "right_hip": 49.630205134726936,
- "left_hip": 46.77926031064371,
- "left_knee": 53.5835750893044,
- "left_ankle": 62.27408274576074,
- "right_wrist": 76.56575862330767,
- "right_elbow": 43.85209632775263,
- "right_shoulder": 54.53079484649968,
- "left_shoulder": 43.528723084500825,
- "left_elbow": 49.180537500980286,
- "left_wrist": 81.935543001211,
- "neck": 45.08486911448025,
- "head": 103.39252635296869
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 85.5182985290965,
- "PA_MPJPE_mm": 53.68111666147786,
- "PCK3D_50mm": 0.6142665504985548,
- "PCK3D_100mm": 0.9101322103749188,
- "AUC_0_150mm": 0.6560543929078565,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.53482232465091,
- "right_knee": 52.32039441267591,
- "right_hip": 46.77911068971432,
- "left_hip": 44.321286291079495,
- "left_knee": 54.25763613054847,
- "left_ankle": 59.2967859662969,
- "right_wrist": 74.3441763265356,
- "right_elbow": 41.05605774138131,
- "right_shoulder": 47.45061788138216,
- "left_shoulder": 41.741120933063954,
- "left_elbow": 46.60787542647106,
- "left_wrist": 78.46351581393422
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.85427418529601
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.35065248349734
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.188897533857755
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.64568177383507
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.57420879668373
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.3325101530597
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.16062244646457
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.940423483526395
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.84210750243531
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.480699639965
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 49.4756048216955
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.237131151912216
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.00279231421817
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.54326126849155
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 75.60157728969241
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.766612893000044
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 52.05201146181139
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen7_emdb1_pve.json b/benchmark/results/gen7_emdb1_pve.json
deleted file mode 100644
index b29f43a..0000000
--- a/benchmark/results/gen7_emdb1_pve.json
+++ /dev/null
@@ -1,82 +0,0 @@
-{
- "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7h_s0/g7h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 107,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 93.69069191261715,
- "PA_PVE_direct_mm": 67.22151318499363,
- "J14_h36m_direct_MPJPE_mm": 80.55783081054688,
- "J14_h36m_direct_PA_MPJPE_mm": 52.10808993766342,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 42.05997507281238,
- "P2_23_outdoor_hug_tree": 48.28855303931294,
- "P3_31_outdoor_workout": 52.9753788503885,
- "P3_32_outdoor_soccer_warmup_a": 53.807105745959774,
- "P3_33_outdoor_soccer_warmup_b": 49.437105613976584,
- "P5_42_indoor_dancing": 72.42647192354559,
- "P5_44_indoor_rom": 49.84704040734609,
- "P6_49_outdoor_big_stairs_down": 57.78107904333181,
- "P6_50_outdoor_workout": 50.566296866672474,
- "P6_51_outdoor_dancing": 44.24490444862562,
- "P7_57_outdoor_rock_chair": 47.703432916531405,
- "P7_59_outdoor_rom": 53.009989732917475,
- "P7_60_outdoor_workout": 44.866869645162296,
- "P8_64_outdoor_skateboard": 50.4121214968073,
- "P8_68_outdoor_handstand": 66.3252719493076,
- "P8_69_outdoor_cartwheel": 59.99572573228202,
- "P9_76_outdoor_sitting": 47.47939234327563
- }
- },
- "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g7r_s0/g7r_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 169,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 90.01686087531817,
- "PA_PVE_direct_mm": 62.07023338138236,
- "J14_h36m_direct_MPJPE_mm": 77.76502990722656,
- "J14_h36m_direct_PA_MPJPE_mm": 47.745280860170475,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 38.83938993695836,
- "P2_23_outdoor_hug_tree": 43.350175828550334,
- "P3_31_outdoor_workout": 50.91046222764864,
- "P3_32_outdoor_soccer_warmup_a": 50.649142410813404,
- "P3_33_outdoor_soccer_warmup_b": 45.009184806973856,
- "P5_42_indoor_dancing": 65.40033076518941,
- "P5_44_indoor_rom": 43.60482023671588,
- "P6_49_outdoor_big_stairs_down": 55.785333331192966,
- "P6_50_outdoor_workout": 44.21877121807696,
- "P6_51_outdoor_dancing": 40.06394514330322,
- "P7_57_outdoor_rock_chair": 44.74101257142711,
- "P7_59_outdoor_rom": 46.05954205306638,
- "P7_60_outdoor_workout": 41.67639591160367,
- "P8_64_outdoor_skateboard": 45.922038428816315,
- "P8_68_outdoor_handstand": 63.786410338411294,
- "P8_69_outdoor_cartwheel": 53.06378784576521,
- "P9_76_outdoor_sitting": 43.035803996322095
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_3dpw_test_exact.json b/benchmark/results/gen8_3dpw_test_exact.json
deleted file mode 100644
index 1f6bfbe..0000000
--- a/benchmark/results/gen8_3dpw_test_exact.json
+++ /dev/null
@@ -1,221 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 295,
- "source": "raw",
- "landmarks": "exact",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.80703127543617,
- "PA_MPJPE_mm": 66.9263006963097,
- "PCK3D_50mm": 0.3970959672253979,
- "PCK3D_100mm": 0.8319556398822112,
- "AUC_0_150mm": 0.5608453335734745,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.30196265594208,
- "right_knee": 46.69770418101452,
- "right_hip": 97.13177963405501,
- "left_hip": 86.76620846296395,
- "left_knee": 45.04349010931432,
- "left_ankle": 65.45255021766913,
- "right_wrist": 68.9802424230139,
- "right_elbow": 58.24450196072215,
- "right_shoulder": 50.29685526252414,
- "left_shoulder": 52.69742741295832,
- "left_elbow": 67.96360660529217,
- "left_wrist": 72.35950274158938,
- "neck": 31.618358713920944,
- "head": 129.41401936735573
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.58179597051439,
- "PA_MPJPE_mm": 61.61298233134312,
- "PCK3D_50mm": 0.4409760407560932,
- "PCK3D_100mm": 0.8934664298000733,
- "AUC_0_150mm": 0.5932573522790222,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.23154664057738,
- "right_knee": 48.561911448764754,
- "right_hip": 96.95194535331011,
- "left_hip": 86.20845142392274,
- "left_knee": 45.879241483409515,
- "left_ankle": 61.21563744245761,
- "right_wrist": 65.54448965061788,
- "right_elbow": 49.53963067224118,
- "right_shoulder": 49.90395357981121,
- "left_shoulder": 50.198974775345796,
- "left_elbow": 55.470541964028875,
- "left_wrist": 68.64946354163048
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.83506356723007,
- "PA_MPJPE_mm": 40.66449503511796,
- "PCK3D_50mm": 0.7379864728227811,
- "PCK3D_100mm": 0.9544595775879086,
- "AUC_0_150mm": 0.7277700741124227
- }
- ]
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 247,
- "source": "raw",
- "landmarks": "exact",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.56541964923919,
- "PA_MPJPE_mm": 67.18680458107954,
- "PCK3D_50mm": 0.39921084752317304,
- "PCK3D_100mm": 0.831047248440024,
- "AUC_0_150mm": 0.559534172762135,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.78247363580984,
- "right_knee": 47.48472813528978,
- "right_hip": 98.22455605292551,
- "left_hip": 86.50565978120595,
- "left_knee": 45.26362925963544,
- "left_ankle": 66.16853375398803,
- "right_wrist": 69.05703853015149,
- "right_elbow": 58.97323628590917,
- "right_shoulder": 50.165973668761254,
- "left_shoulder": 51.64742703953226,
- "left_elbow": 68.3788601152943,
- "left_wrist": 72.42705489572386,
- "neck": 31.25839777183021,
- "head": 129.27769520905642
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.4332471236804,
- "PA_MPJPE_mm": 61.98145997538403,
- "PCK3D_50mm": 0.43818204889603246,
- "PCK3D_100mm": 0.8904139525702846,
- "AUC_0_150mm": 0.5911397431034435,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.72623212603937,
- "right_knee": 49.29932646889992,
- "right_hip": 97.84753671612978,
- "left_hip": 85.83399537309099,
- "left_knee": 46.271981994386415,
- "left_ankle": 61.38539486992044,
- "right_wrist": 65.4452878518779,
- "right_elbow": 50.09260495446915,
- "right_shoulder": 50.32852037063604,
- "left_shoulder": 50.175296147724644,
- "left_elbow": 56.02207338564183,
- "left_wrist": 68.34926944579175
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.36154761309335,
- "PA_MPJPE_mm": 40.76549050577253,
- "PCK3D_50mm": 0.7363811779681841,
- "PCK3D_100mm": 0.9523064280276022,
- "AUC_0_150mm": 0.727364900731872
- }
- ]
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 92,
- "source": "ema",
- "landmarks": "exact",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 103.2678192077454,
- "PA_MPJPE_mm": 67.62020858112662,
- "PCK3D_50mm": 0.3937826547588835,
- "PCK3D_100mm": 0.82800383498294,
- "AUC_0_150mm": 0.5570481001097919,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.69092633837089,
- "right_knee": 46.951023926891274,
- "right_hip": 97.29378722694062,
- "left_hip": 86.60485625661998,
- "left_knee": 44.54696925900634,
- "left_ankle": 66.1053792432559,
- "right_wrist": 68.42825197369791,
- "right_elbow": 58.69479715116709,
- "right_shoulder": 51.07718749891772,
- "left_shoulder": 53.493189989051714,
- "left_elbow": 68.77373140388806,
- "left_wrist": 72.05191666147934,
- "neck": 32.73633895719916,
- "head": 134.2345642492867
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.59520049903726,
- "PA_MPJPE_mm": 61.74848208820537,
- "PCK3D_50mm": 0.4414225154856235,
- "PCK3D_100mm": 0.8922327496263711,
- "AUC_0_150mm": 0.5925871095695985,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.93229638085995,
- "right_knee": 48.682171588664076,
- "right_hip": 96.96746918608095,
- "left_hip": 85.8534455489899,
- "left_knee": 45.36179927744644,
- "left_ankle": 61.64562733574743,
- "right_wrist": 65.02419732134345,
- "right_elbow": 49.508224768170365,
- "right_shoulder": 51.23428612403596,
- "left_shoulder": 50.98312291221603,
- "left_elbow": 55.500933512942964,
- "left_wrist": 68.28821110196692
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.85320912362639,
- "PA_MPJPE_mm": 41.1313440195285,
- "PCK3D_50mm": 0.7328140798659367,
- "PCK3D_100mm": 0.9519378346042757,
- "AUC_0_150mm": 0.7249510133947618
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_3dpw_test_fitted.json b/benchmark/results/gen8_3dpw_test_fitted.json
deleted file mode 100644
index a4a822d..0000000
--- a/benchmark/results/gen8_3dpw_test_fitted.json
+++ /dev/null
@@ -1,221 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 295,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.90314007165716,
- "PA_MPJPE_mm": 66.75417606434624,
- "PCK3D_50mm": 0.39980502817826225,
- "PCK3D_100mm": 0.8327069259308495,
- "AUC_0_150mm": 0.5619064771993814,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.05109104892337,
- "right_knee": 46.94215548473336,
- "right_hip": 97.44293023077252,
- "left_hip": 86.91167415702769,
- "left_knee": 45.06409705256008,
- "left_ankle": 65.60436699289106,
- "right_wrist": 68.87685239691548,
- "right_elbow": 57.44952439390172,
- "right_shoulder": 50.244764130125965,
- "left_shoulder": 52.69003609042731,
- "left_elbow": 66.5267784650984,
- "left_wrist": 72.22817354585416,
- "neck": 31.50716013166217,
- "head": 129.0188607799542
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.72900937988003,
- "PA_MPJPE_mm": 61.44884911854648,
- "PCK3D_50mm": 0.44510240720375227,
- "PCK3D_100mm": 0.8933747849871697,
- "AUC_0_150mm": 0.5943006932259247,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.93447127681083,
- "right_knee": 48.84689469907562,
- "right_hip": 97.24594667562684,
- "left_hip": 86.35254867329824,
- "left_knee": 45.948983236860066,
- "left_ankle": 61.31441118064832,
- "right_wrist": 65.41141248152778,
- "right_elbow": 49.02660379503469,
- "right_shoulder": 49.59549580248439,
- "left_shoulder": 49.95472719809619,
- "left_elbow": 54.25456041864994,
- "left_wrist": 68.50013398444479
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.08732433547077,
- "PA_MPJPE_mm": 40.798329210872225,
- "PCK3D_50mm": 0.7363086677865461,
- "PCK3D_100mm": 0.9543346989417542,
- "AUC_0_150mm": 0.7269478372408915
- }
- ]
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 247,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.6525817461484,
- "PA_MPJPE_mm": 67.00182266943203,
- "PCK3D_50mm": 0.40165605198174353,
- "PCK3D_100mm": 0.8316233015497038,
- "AUC_0_150mm": 0.5606608094553277,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.50069966094325,
- "right_knee": 47.706682882858985,
- "right_hip": 98.52407097923475,
- "left_hip": 86.64301969615587,
- "left_knee": 45.282613479975716,
- "left_ankle": 66.31612624741163,
- "right_wrist": 68.9610354847499,
- "right_elbow": 58.11394782093181,
- "right_shoulder": 50.104734720366274,
- "left_shoulder": 51.622575673450754,
- "left_elbow": 66.91436972388529,
- "left_wrist": 72.31291445619766,
- "neck": 31.142680872636475,
- "head": 128.88004567325007
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.57060223253823,
- "PA_MPJPE_mm": 61.803102250578924,
- "PCK3D_50mm": 0.44230841534369153,
- "PCK3D_100mm": 0.8905196965851733,
- "AUC_0_150mm": 0.5922825364858543,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.41007351839793,
- "right_knee": 49.5612697203333,
- "right_hip": 98.12997769057627,
- "left_hip": 85.97188308538445,
- "left_knee": 46.334107186717304,
- "left_ankle": 61.47633401152167,
- "right_wrist": 65.31269169032657,
- "right_elbow": 49.512903222021734,
- "right_shoulder": 50.01400380973745,
- "left_shoulder": 49.91017946596769,
- "left_elbow": 54.78779058871191,
- "left_wrist": 68.21601301725084
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.61241265402009,
- "PA_MPJPE_mm": 40.89476284389129,
- "PCK3D_50mm": 0.7349350026788484,
- "PCK3D_100mm": 0.9523305980881482,
- "AUC_0_150mm": 0.7265379208108249
- }
- ]
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 92,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 103.3552608287201,
- "PA_MPJPE_mm": 67.4280683350019,
- "PCK3D_50mm": 0.3964615031360654,
- "PCK3D_100mm": 0.828668511647955,
- "AUC_0_150mm": 0.5582104721075551,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.39459665976419,
- "right_knee": 47.153198883168606,
- "right_hip": 97.5952870875838,
- "left_hip": 86.74699360285612,
- "left_knee": 44.54385047592911,
- "left_ankle": 66.22549073403927,
- "right_wrist": 68.34888418002461,
- "right_elbow": 57.886769696720066,
- "right_shoulder": 50.985546488740205,
- "left_shoulder": 53.45123802978203,
- "left_elbow": 67.35204434425665,
- "left_wrist": 71.93700750249884,
- "neck": 32.592662556909225,
- "head": 133.7793864477537
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.73619751003687,
- "PA_MPJPE_mm": 61.56971983790048,
- "PCK3D_50mm": 0.44555358166727765,
- "PCK3D_100mm": 0.8923455432422525,
- "AUC_0_150mm": 0.5936921534757262,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.58499098404639,
- "right_knee": 48.931886760641056,
- "right_hip": 97.24917343468958,
- "left_hip": 85.99413869176327,
- "left_knee": 45.410370669081324,
- "left_ankle": 61.69728339223045,
- "right_wrist": 64.91939784414778,
- "right_elbow": 48.96749551554467,
- "right_shoulder": 50.91719098766221,
- "left_shoulder": 50.72034745526738,
- "left_elbow": 54.29140198864562,
- "left_wrist": 68.15296033108598
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 67.14198639664794,
- "PA_MPJPE_mm": 41.271759981790396,
- "PCK3D_50mm": 0.7309429143453338,
- "PCK3D_100mm": 0.9518351118469551,
- "AUC_0_150mm": 0.7240582155400236
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_3dpw_test_pve.json b/benchmark/results/gen8_3dpw_test_pve.json
deleted file mode 100644
index dc84763..0000000
--- a/benchmark/results/gen8_3dpw_test_pve.json
+++ /dev/null
@@ -1,143 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 295,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 78.82837980688144,
- "PA_PVE_direct_mm": 56.717468171625484,
- "J14_h36m_direct_MPJPE_mm": 65.75281524658203,
- "J14_h36m_direct_PA_MPJPE_mm": 41.75702950321811,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 45.05975302986316,
- "downtown_bar_00": 53.2947004599301,
- "downtown_bus_00": 48.88079901057791,
- "downtown_cafe_00": 38.842131235486605,
- "downtown_car_00": 46.5961762766902,
- "downtown_crossStreets_00": 42.839533257582794,
- "downtown_downstairs_00": 34.91155288540495,
- "downtown_enterShop_00": 31.31448445328985,
- "downtown_rampAndStairs_00": 42.30817554678824,
- "downtown_runForBus_00": 39.22241943324179,
- "downtown_runForBus_01": 49.59826270996972,
- "downtown_sitOnStairs_00": 38.04187901754806,
- "downtown_stairs_00": 50.49035244421762,
- "downtown_upstairs_00": 32.37017969985136,
- "downtown_walkBridge_01": 32.04064056525192,
- "downtown_walkUphill_00": 30.627759564108693,
- "downtown_walking_00": 40.303516789158145,
- "downtown_warmWelcome_00": 47.5582151350902,
- "downtown_weeklyMarket_00": 36.519280146342474,
- "downtown_windowShopping_00": 33.068929583668236,
- "flat_guitar_01": 46.396848101246164,
- "flat_packBags_00": 42.43439241396279,
- "office_phoneCall_00": 34.98855966129716,
- "outdoors_fencing_01": 44.2511508755811
- }
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 247,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 78.22564847427604,
- "PA_PVE_direct_mm": 56.83515365731806,
- "J14_h36m_direct_MPJPE_mm": 65.17237854003906,
- "J14_h36m_direct_PA_MPJPE_mm": 41.85701040398819,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 46.22316575232354,
- "downtown_bar_00": 55.41782090657222,
- "downtown_bus_00": 48.45785130363753,
- "downtown_cafe_00": 38.60759548966954,
- "downtown_car_00": 45.887235672727385,
- "downtown_crossStreets_00": 42.19440236042504,
- "downtown_downstairs_00": 34.820235315625474,
- "downtown_enterShop_00": 31.273622254261312,
- "downtown_rampAndStairs_00": 41.95044997141832,
- "downtown_runForBus_00": 38.51170708200028,
- "downtown_runForBus_01": 49.1178795490624,
- "downtown_sitOnStairs_00": 38.163287714468765,
- "downtown_stairs_00": 51.0485895272681,
- "downtown_upstairs_00": 32.15508929765303,
- "downtown_walkBridge_01": 32.08295622323714,
- "downtown_walkUphill_00": 30.021231316922957,
- "downtown_walking_00": 40.17104678878722,
- "downtown_warmWelcome_00": 48.4890093139576,
- "downtown_weeklyMarket_00": 36.37250920429031,
- "downtown_windowShopping_00": 32.537393551950096,
- "flat_guitar_01": 46.279603379583435,
- "flat_packBags_00": 43.88742613280304,
- "office_phoneCall_00": 34.04630744938934,
- "outdoors_fencing_01": 45.81445800906054
- }
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 92,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 79.87915560472746,
- "PA_PVE_direct_mm": 57.195531897108154,
- "J14_h36m_direct_MPJPE_mm": 66.91474914550781,
- "J14_h36m_direct_PA_MPJPE_mm": 42.36544791889175,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 47.74858130416555,
- "downtown_bar_00": 55.799553512897525,
- "downtown_bus_00": 48.750644980870995,
- "downtown_cafe_00": 39.79573307973785,
- "downtown_car_00": 47.46523788472092,
- "downtown_crossStreets_00": 43.530625663157245,
- "downtown_downstairs_00": 35.30016224183736,
- "downtown_enterShop_00": 30.875809218541345,
- "downtown_rampAndStairs_00": 42.05712663310433,
- "downtown_runForBus_00": 37.889725313057646,
- "downtown_runForBus_01": 48.498553029290534,
- "downtown_sitOnStairs_00": 39.13283848163328,
- "downtown_stairs_00": 50.154009989812316,
- "downtown_upstairs_00": 32.71023720184678,
- "downtown_walkBridge_01": 32.07696352498332,
- "downtown_walkUphill_00": 30.845565873218845,
- "downtown_walking_00": 40.443879604787064,
- "downtown_warmWelcome_00": 49.109667423565824,
- "downtown_weeklyMarket_00": 36.998349737585684,
- "downtown_windowShopping_00": 33.217374624511734,
- "flat_guitar_01": 54.27986106886148,
- "flat_packBags_00": 43.270649682994325,
- "office_phoneCall_00": 34.2528269527834,
- "outdoors_fencing_01": 43.6702963839884
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_emdb1_annotated.json b/benchmark/results/gen8_emdb1_annotated.json
deleted file mode 100644
index 42b0f14..0000000
--- a/benchmark/results/gen8_emdb1_annotated.json
+++ /dev/null
@@ -1,346 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 295,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 88.76262010632146,
- "PA_MPJPE_mm": 64.32690564747114,
- "PCK3D_50mm": 0.5623694837433791,
- "PCK3D_100mm": 0.8426821004300986,
- "AUC_0_150mm": 0.6134404029481004,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.72739616523092,
- "left_hip": 43.53664928490318,
- "right_hip": 42.47946649645917,
- "spine1": 51.80011363644271,
- "left_knee": 58.04927382682903,
- "right_knee": 56.29593421755156,
- "spine2": 51.50166121350724,
- "left_ankle": 70.10281185148675,
- "right_ankle": 70.07250431760954,
- "spine3": 50.67987416584745,
- "left_foot": 77.79060701035993,
- "right_foot": 78.21471337079649,
- "neck": 49.27518692448127,
- "left_collar": 43.24291462481559,
- "right_collar": 42.80192669461389,
- "head": 55.14169371785151,
- "left_shoulder": 47.904915709508444,
- "right_shoulder": 47.244650493662036,
- "left_elbow": 61.798039706379,
- "right_elbow": 57.81949942904837,
- "left_wrist": 99.87869570494013,
- "right_wrist": 101.27008016526156,
- "left_hand": 119.02842811021682,
- "right_hand": 125.18869870150422
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 100.20960661675163,
- "PA_MPJPE_mm": 67.70112248806943,
- "PCK3D_50mm": 0.48429359712187575,
- "PCK3D_100mm": 0.8266783625037785,
- "AUC_0_150mm": 0.5808690468383478,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.23891257552113,
- "right_knee": 53.370854089752505,
- "right_hip": 51.9298041156822,
- "left_hip": 49.25334293941451,
- "left_knee": 56.7825543131599,
- "left_ankle": 72.06322892772938,
- "right_wrist": 102.62766164388755,
- "right_elbow": 53.70033232898691,
- "right_shoulder": 58.22250209407417,
- "left_shoulder": 49.761324857441736,
- "left_elbow": 58.86439891553489,
- "left_wrist": 103.59978537156461,
- "neck": 59.81198585359963,
- "head": 107.58902680662273
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 95.63595481944421,
- "PA_MPJPE_mm": 64.06419658736174,
- "PCK3D_50mm": 0.53339141738926,
- "PCK3D_100mm": 0.8486426309311428,
- "AUC_0_150mm": 0.6047484831005286,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.03874380342945,
- "right_knee": 54.265726018154375,
- "right_hip": 51.31842966016909,
- "left_hip": 49.38816719105823,
- "left_knee": 57.51416548085523,
- "left_ankle": 70.59377879986292,
- "right_wrist": 98.00290890921097,
- "right_elbow": 50.425163158690815,
- "right_shoulder": 57.98955549907401,
- "left_shoulder": 54.372273287172675,
- "left_elbow": 56.59655723988543,
- "left_wrist": 99.26489000077778
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 75.1569086897259
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 70.37429717601877
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.15779649961338
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 55.763399060412645
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 64.57149990299354
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 83.09306734173849
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 55.185263572409305
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 77.3131934192975
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 55.07497626438046
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 63.936823334609144
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 56.42255440924773
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 59.816166687025365
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 55.52638191914596
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 62.05609838052144
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 83.5236655338729
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.87959907752564
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 59.30274501084284
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 247,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 89.24478111621829,
- "PA_MPJPE_mm": 66.89051058318199,
- "PCK3D_50mm": 0.5427246262567592,
- "PCK3D_100mm": 0.8293072093377035,
- "AUC_0_150mm": 0.6015779501860072,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.810182621115835,
- "left_hip": 44.13638723000304,
- "right_hip": 42.90311175608087,
- "spine1": 53.19471820477527,
- "left_knee": 58.01026433731803,
- "right_knee": 56.6788951694521,
- "spine2": 53.05014683883006,
- "left_ankle": 70.70555056990528,
- "right_ankle": 71.9338142391788,
- "spine3": 52.454466492137556,
- "left_foot": 78.779893834194,
- "right_foot": 80.9591141900794,
- "neck": 52.30428934621626,
- "left_collar": 46.13822355677994,
- "right_collar": 45.72129105536187,
- "head": 59.05837199161462,
- "left_shoulder": 49.52196280870598,
- "right_shoulder": 49.190675699586116,
- "left_elbow": 63.988331123564954,
- "right_elbow": 60.81275012273367,
- "left_wrist": 106.9344451042545,
- "right_wrist": 106.81214625580598,
- "left_hand": 127.6084289102973,
- "right_hand": 130.6647925383788
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 100.28093707341523,
- "PA_MPJPE_mm": 69.73662325205572,
- "PCK3D_50mm": 0.46777816632191604,
- "PCK3D_100mm": 0.814531089787282,
- "AUC_0_150mm": 0.5708741153318393,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 72.38978289942779,
- "right_knee": 54.03149026043775,
- "right_hip": 51.356108506491154,
- "left_hip": 49.811121151005466,
- "left_knee": 56.808665864142974,
- "left_ankle": 72.92868528434977,
- "right_wrist": 108.19024936320407,
- "right_elbow": 56.45063342066514,
- "right_shoulder": 60.36568877055438,
- "left_shoulder": 50.575689764376556,
- "left_elbow": 60.60541881006109,
- "left_wrist": 110.6946994511135,
- "neck": 62.07527690734759,
- "head": 110.02921507560288
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 96.16395031415158,
- "PA_MPJPE_mm": 66.1919422644989,
- "PCK3D_50mm": 0.5132832704089394,
- "PCK3D_100mm": 0.837381930326792,
- "AUC_0_150mm": 0.5940371720113368,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 71.03626620506343,
- "right_knee": 54.86410905938645,
- "right_hip": 51.47818801727088,
- "left_hip": 50.6134082786491,
- "left_knee": 57.6991870489433,
- "left_ankle": 71.41209076510692,
- "right_wrist": 103.01994285775166,
- "right_elbow": 53.49245806856022,
- "right_shoulder": 60.90576162520488,
- "left_shoulder": 56.61177561619242,
- "left_elbow": 57.93463650794004,
- "left_wrist": 105.23548312391765
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 83.788047040943
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 74.10063803076984
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 61.92140648476573
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 60.40650615697361
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 70.6191221994521
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.43654095908283
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.06881540316626
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 81.41758018488571
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 57.0616796141908
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 65.66184766547114
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 59.24821940321826
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 62.59178945483681
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 64.53617334097041
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 64.85601161182977
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 78.86868543325276
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.5489513986187
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 59.203835018123364
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_emdb1_exact.json b/benchmark/results/gen8_emdb1_exact.json
deleted file mode 100644
index c7a8aee..0000000
--- a/benchmark/results/gen8_emdb1_exact.json
+++ /dev/null
@@ -1,518 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "exact",
- "epoch": 295,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.2122862106147,
- "PA_MPJPE_mm": 51.86187871821707,
- "PCK3D_50mm": 0.6486052911809042,
- "PCK3D_100mm": 0.9106058720214635,
- "AUC_0_150mm": 0.6687220258576655,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.37336492790027,
- "left_hip": 39.33342266290948,
- "right_hip": 38.41569861303099,
- "spine1": 44.072474214510756,
- "left_knee": 52.63767452067048,
- "right_knee": 51.89135889130645,
- "spine2": 43.438049056194444,
- "left_ankle": 57.916637701367364,
- "right_ankle": 58.17133971264106,
- "spine3": 41.10865172494157,
- "left_foot": 66.18403574350798,
- "right_foot": 65.13271394142697,
- "neck": 33.04596304191114,
- "left_collar": 28.790103338618998,
- "right_collar": 29.943410446848144,
- "head": 38.4201919212527,
- "left_shoulder": 39.160382962517765,
- "right_shoulder": 40.310962378193,
- "left_elbow": 51.06033143493064,
- "right_elbow": 47.12828205229361,
- "left_wrist": 77.93689340716665,
- "right_wrist": 74.97951933698579,
- "left_hand": 94.01397337657578,
- "right_hand": 94.21965382950846
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.26700471672483,
- "PA_MPJPE_mm": 58.04846687203332,
- "PCK3D_50mm": 0.5476141084986457,
- "PCK3D_100mm": 0.8809958452119179,
- "AUC_0_150mm": 0.6241700604796886,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.70058008180087,
- "right_knee": 49.7469813678576,
- "right_hip": 49.36845349573269,
- "left_hip": 46.6926216364305,
- "left_knee": 52.615103281622794,
- "left_ankle": 61.07949683957811,
- "right_wrist": 76.36450819868143,
- "right_elbow": 42.900602056434295,
- "right_shoulder": 53.620386800825784,
- "left_shoulder": 42.9658127911143,
- "left_elbow": 48.25849102637601,
- "left_wrist": 81.80086697939578,
- "neck": 44.35848979365691,
- "head": 103.20614185895931
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.7516811034589,
- "PA_MPJPE_mm": 53.13262471812324,
- "PCK3D_50mm": 0.614069479594518,
- "PCK3D_100mm": 0.909706952108313,
- "AUC_0_150mm": 0.6571525027670228,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.35272312021682,
- "right_knee": 50.22190958538767,
- "right_hip": 46.387767917232644,
- "left_hip": 44.4627090584536,
- "left_knee": 53.12340730386851,
- "left_ankle": 58.76608975776491,
- "right_wrist": 74.51906623245199,
- "right_elbow": 40.17096320609634,
- "right_shoulder": 46.76632849306561,
- "left_shoulder": 41.30958553547687,
- "left_elbow": 45.78289069169998,
- "left_wrist": 78.728055715764
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.87038161440268
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.011828465478594
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.54638652785373
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.8162855423868
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.54666704023567
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.9769615067491
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.46331044306099
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.78684386490259
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.34141434655844
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.07056245578691
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.62699524068066
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.81132028519969
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.7896250017526
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.847847851872906
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.63704871341942
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.24055207916096
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.39374100475584
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "exact",
- "epoch": 247,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.28414813977952,
- "PA_MPJPE_mm": 51.53263168740522,
- "PCK3D_50mm": 0.6501023385747279,
- "PCK3D_100mm": 0.9130346844791105,
- "AUC_0_150mm": 0.6703174971749823,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.20792047726475,
- "left_hip": 38.716219043743905,
- "right_hip": 37.71826485100924,
- "spine1": 44.13895094556884,
- "left_knee": 51.6824198158276,
- "right_knee": 50.81026130913571,
- "spine2": 43.51636532083446,
- "left_ankle": 56.58269305674233,
- "right_ankle": 57.94940773883263,
- "spine3": 40.941411987381144,
- "left_foot": 64.32969154549836,
- "right_foot": 64.83089831913564,
- "neck": 33.398371889139,
- "left_collar": 29.008309136636086,
- "right_collar": 29.773248710304106,
- "head": 38.83341543093524,
- "left_shoulder": 38.98440859778381,
- "right_shoulder": 39.89717400094668,
- "left_elbow": 50.40155415393976,
- "right_elbow": 47.33887398224507,
- "left_wrist": 77.64113624879326,
- "right_wrist": 75.59440283954187,
- "left_hand": 93.30570123496202,
- "right_hand": 94.18205986152431
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.08364586159168,
- "PA_MPJPE_mm": 57.94123351836941,
- "PCK3D_50mm": 0.5476674509989865,
- "PCK3D_100mm": 0.8827028052228235,
- "AUC_0_150mm": 0.6244276913729117,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.3384197102628,
- "right_knee": 49.05186997313463,
- "right_hip": 49.63849197802996,
- "left_hip": 47.73629172173575,
- "left_knee": 51.84981922816395,
- "left_ankle": 60.18837036374923,
- "right_wrist": 76.97181953184595,
- "right_elbow": 42.97412154549122,
- "right_shoulder": 53.330386199146446,
- "left_shoulder": 42.21684035478016,
- "left_elbow": 47.52509073124235,
- "left_wrist": 81.43586792261665,
- "neck": 44.63596844637286,
- "head": 104.28391155059977
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.585970102899,
- "PA_MPJPE_mm": 52.79957039658859,
- "PCK3D_50mm": 0.6165484241242446,
- "PCK3D_100mm": 0.9134512992850129,
- "AUC_0_150mm": 0.6588260998162455,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.95560319097742,
- "right_knee": 49.430052128331475,
- "right_hip": 46.60096925619823,
- "left_hip": 45.30674189212308,
- "left_knee": 52.290106935983175,
- "left_ankle": 57.35491780981008,
- "right_wrist": 74.9224683120239,
- "right_elbow": 40.54505911394943,
- "right_shoulder": 46.19713183858256,
- "left_shoulder": 40.643663436065,
- "left_elbow": 45.06925511687083,
- "left_wrist": 78.27887572814795
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.685622490231005
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.296834879501525
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.61132928374254
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.42109776532132
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.11748106097561
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.45238547335242
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.72255243631251
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.36371084542673
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 47.15499734226795
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.115983520259974
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.085263487843235
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.04292839251302
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.87057707469499
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.54241154000006
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.60604157926443
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 55.29812528170841
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.462693637860966
- }
- }
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "exact",
- "epoch": 92,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.98692565087396,
- "PA_MPJPE_mm": 51.84962867766953,
- "PCK3D_50mm": 0.6458220968344189,
- "PCK3D_100mm": 0.9126699304374283,
- "AUC_0_150mm": 0.6684713654972677,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.444756988342476,
- "left_hip": 39.45005178890784,
- "right_hip": 38.90195923943951,
- "spine1": 45.97677920213361,
- "left_knee": 51.89135337482178,
- "right_knee": 51.32533748503452,
- "spine2": 43.997376113951844,
- "left_ankle": 58.30806681008707,
- "right_ankle": 58.397857310165726,
- "spine3": 40.90675563338684,
- "left_foot": 66.3984394227497,
- "right_foot": 65.51264426386312,
- "neck": 33.796579521655396,
- "left_collar": 29.422123640551597,
- "right_collar": 30.0821734004735,
- "head": 39.910988647648956,
- "left_shoulder": 39.28233662875761,
- "right_shoulder": 39.24305140428493,
- "left_elbow": 50.61908713667689,
- "right_elbow": 47.36473426457613,
- "left_wrist": 75.92680441872527,
- "right_wrist": 74.06320220787723,
- "left_hand": 92.05131366120509,
- "right_hand": 93.11731569875312
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.60252567969596,
- "PA_MPJPE_mm": 58.025698370180635,
- "PCK3D_50mm": 0.5465828201587236,
- "PCK3D_100mm": 0.8803616621523106,
- "AUC_0_150mm": 0.6240246591481146,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.17056183584565,
- "right_knee": 49.32022479841245,
- "right_hip": 50.67297535530687,
- "left_hip": 47.98747636188927,
- "left_knee": 52.20123359856125,
- "left_ankle": 61.772561962929956,
- "right_wrist": 74.9487108850648,
- "right_elbow": 42.71019077461973,
- "right_shoulder": 52.66645038391631,
- "left_shoulder": 42.291714394457735,
- "left_elbow": 47.70949404253593,
- "left_wrist": 79.53621005278646,
- "neck": 44.8395997625749,
- "head": 105.53237297362767
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.7430433397665,
- "PA_MPJPE_mm": 52.705929420027026,
- "PCK3D_50mm": 0.6188890732827173,
- "PCK3D_100mm": 0.913938790468683,
- "AUC_0_150mm": 0.6594960516671506,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.50254589050034,
- "right_knee": 49.704024762920426,
- "right_hip": 47.46718688776811,
- "left_hip": 45.57044311261965,
- "left_knee": 52.5645620695132,
- "left_ankle": 59.17619528154309,
- "right_wrist": 73.29465152173177,
- "right_elbow": 40.05940322305987,
- "right_shoulder": 45.36457520741507,
- "left_shoulder": 40.632452033313456,
- "left_elbow": 44.78984276078414,
- "left_wrist": 76.34527028915525
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.89753277034631
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.1707680623559
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.9330794845804
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.25503752391202
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.681844007962
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.1885500872837
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.807422427979276
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.095920439303974
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.28817115449439
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.38848466498806
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 49.95600371117474
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 48.92178559601738
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.67208061186149
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.21815650713306
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 68.41873257384535
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.84277949304279
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.764547877180625
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_emdb1_fitted.json b/benchmark/results/gen8_emdb1_fitted.json
deleted file mode 100644
index 37d6faf..0000000
--- a/benchmark/results/gen8_emdb1_fitted.json
+++ /dev/null
@@ -1,518 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 295,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.28133498395243,
- "PA_MPJPE_mm": 51.94917326166577,
- "PCK3D_50mm": 0.6465827213763156,
- "PCK3D_100mm": 0.9107372526241547,
- "AUC_0_150mm": 0.6681061095772222,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.45904143454635,
- "left_hip": 39.4637925325054,
- "right_hip": 38.63344824832323,
- "spine1": 44.073005603096604,
- "left_knee": 52.729267865308756,
- "right_knee": 51.9637426421973,
- "spine2": 43.80597621290074,
- "left_ankle": 57.8990228759747,
- "right_ankle": 58.189714662699735,
- "spine3": 41.8082717277129,
- "left_foot": 66.19531003452462,
- "right_foot": 65.19604679768183,
- "neck": 33.18097087754447,
- "left_collar": 28.820628382423205,
- "right_collar": 30.045158100991195,
- "head": 38.43886388107154,
- "left_shoulder": 38.267183569440014,
- "right_shoulder": 40.142624948163586,
- "left_elbow": 51.79547269264506,
- "right_elbow": 47.86634361806799,
- "left_wrist": 77.91960721386734,
- "right_wrist": 75.09057323289153,
- "left_hand": 93.90372592604993,
- "right_hand": 93.89236519935419
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.24875987846532,
- "PA_MPJPE_mm": 58.06083009514393,
- "PCK3D_50mm": 0.5467013590483698,
- "PCK3D_100mm": 0.8819204485511585,
- "AUC_0_150mm": 0.6240450210702877,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.87164370027369,
- "right_knee": 49.900008685117164,
- "right_hip": 49.483329892570104,
- "left_hip": 46.825825585118544,
- "left_knee": 52.67234286849751,
- "left_ankle": 61.39731908675756,
- "right_wrist": 76.22799660286076,
- "right_elbow": 42.975780840293666,
- "right_shoulder": 53.59319190738354,
- "left_shoulder": 42.941824969128305,
- "left_elbow": 48.27005007868398,
- "left_wrist": 81.69373630469454,
- "neck": 44.48096773004882,
- "head": 102.51760308058665
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.80475609925979,
- "PA_MPJPE_mm": 53.221210121191284,
- "PCK3D_50mm": 0.6126934406505414,
- "PCK3D_100mm": 0.9096378044226859,
- "AUC_0_150mm": 0.6565272738547248,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.47732042540727,
- "right_knee": 50.42859296699345,
- "right_hip": 46.48760097811383,
- "left_hip": 44.56592049226495,
- "left_knee": 53.21556321457418,
- "left_ankle": 59.058105696247864,
- "right_wrist": 74.4182027612659,
- "right_elbow": 40.43695031747281,
- "right_shoulder": 46.68281764603542,
- "left_shoulder": 41.1975616908077,
- "left_elbow": 45.98214894049772,
- "left_wrist": 78.70373632461431
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.31888905393791
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.180015950378625
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.61758101633136
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.93586472078992
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.68536095147245
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.88227217037303
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.53485487091855
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.902593364750075
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.4289648281872
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.1923851505244
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.64224046282951
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.91192536080979
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.93165730133809
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.69479899747672
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.71627636180757
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.348237256175786
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.44213896954093
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 247,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 76.37327117980249,
- "PA_MPJPE_mm": 51.62199253812385,
- "PCK3D_50mm": 0.6482872118270202,
- "PCK3D_100mm": 0.9130502427083765,
- "AUC_0_150mm": 0.6696832902164054,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.313078718267725,
- "left_hip": 38.853223708548384,
- "right_hip": 37.9242056098596,
- "spine1": 44.17469308622304,
- "left_knee": 51.73802820291177,
- "right_knee": 50.91070415426388,
- "spine2": 43.87232251672769,
- "left_ankle": 56.626517117605175,
- "right_ankle": 57.94677503513802,
- "spine3": 41.61633851721419,
- "left_foot": 64.34195267479396,
- "right_foot": 64.9077459186065,
- "neck": 33.5222875642854,
- "left_collar": 29.04015758798671,
- "right_collar": 29.86363291516162,
- "head": 38.81802403831711,
- "left_shoulder": 38.048903670876236,
- "right_shoulder": 39.69094064043953,
- "left_elbow": 51.206432127118205,
- "right_elbow": 48.126823634558015,
- "left_wrist": 77.63397317086117,
- "right_wrist": 75.69509328173379,
- "left_hand": 93.20018923299263,
- "right_hand": 93.85577779048205
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.08711096923552,
- "PA_MPJPE_mm": 57.95926961569839,
- "PCK3D_50mm": 0.5464909525192477,
- "PCK3D_100mm": 0.8833340248101896,
- "AUC_0_150mm": 0.6242755027339466,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.50315630107591,
- "right_knee": 49.212925041995604,
- "right_hip": 49.752582859858904,
- "left_hip": 47.867220500041874,
- "left_knee": 51.9074770945956,
- "left_ankle": 60.488162758302906,
- "right_wrist": 76.83532588789609,
- "right_elbow": 43.07787045412104,
- "right_shoulder": 53.304932191659255,
- "left_shoulder": 42.1878557471594,
- "left_elbow": 47.53408080679513,
- "left_wrist": 81.32111647664708,
- "neck": 44.77204801368858,
- "head": 103.6650204859401
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.65530611265551,
- "PA_MPJPE_mm": 52.88531277727116,
- "PCK3D_50mm": 0.6151723851802681,
- "PCK3D_100mm": 0.9133856089836673,
- "AUC_0_150mm": 0.658233214176257,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.07963138978226,
- "right_knee": 49.64283648913686,
- "right_hip": 46.702688604288035,
- "left_hip": 45.409720621530035,
- "left_knee": 52.37970355054816,
- "left_ankle": 57.630180511248504,
- "right_wrist": 74.81881425668325,
- "right_elbow": 40.80768864128385,
- "right_shoulder": 46.12003898635273,
- "left_shoulder": 40.5170851938668,
- "left_elbow": 45.280920656496555,
- "left_wrist": 78.23444442603694
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.14160092210835
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.42307971396525
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.686485351496366
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.529491382182535
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.27332393431397
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.3663223453656
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.8557105799311
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.44852469947513
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 47.23841814430818
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.197762760102584
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.159607966384954
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.13855301879991
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.01093679534138
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.39391892118029
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.69748941977592
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 55.37908490363827
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.507770411225145
- }
- }
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 92,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.13034904317948,
- "PA_MPJPE_mm": 51.962309233158365,
- "PCK3D_50mm": 0.6437217358835,
- "PCK3D_100mm": 0.9127995823479789,
- "AUC_0_150mm": 0.6677079527422409,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.59484150726063,
- "left_hip": 39.620697482498414,
- "right_hip": 39.19451339334599,
- "spine1": 46.044052233981354,
- "left_knee": 51.95537991110134,
- "right_knee": 51.373140188549016,
- "spine2": 44.420090311721935,
- "left_ankle": 58.403168889934626,
- "right_ankle": 58.4542483134746,
- "spine3": 41.69881653559353,
- "left_foot": 66.4274479755913,
- "right_foot": 65.59225504513064,
- "neck": 33.94207443975719,
- "left_collar": 29.42279097143931,
- "right_collar": 30.168120773079973,
- "head": 39.94083804214448,
- "left_shoulder": 38.408684230764486,
- "right_shoulder": 39.07630332945401,
- "left_elbow": 51.3869130392211,
- "right_elbow": 48.115464110616905,
- "left_wrist": 75.91428948808245,
- "right_wrist": 74.15300522567144,
- "left_hand": 91.97205214591565,
- "right_hand": 92.8162340114697
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.59781799220282,
- "PA_MPJPE_mm": 58.033505790946236,
- "PCK3D_50mm": 0.5454596641793257,
- "PCK3D_100mm": 0.8813929504922328,
- "AUC_0_150mm": 0.62393269591276,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.3891214415158,
- "right_knee": 49.46143443856073,
- "right_hip": 50.81062898126727,
- "left_hip": 48.142056787782664,
- "left_knee": 52.2342533650758,
- "left_ankle": 62.13280535145852,
- "right_wrist": 74.82783984648376,
- "right_elbow": 42.734662595995054,
- "right_shoulder": 52.597866164144676,
- "left_shoulder": 42.22067259606931,
- "left_elbow": 47.71554624308956,
- "left_wrist": 79.45218225266969,
- "neck": 44.981762073071,
- "head": 104.76824893606343
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.81647574866606,
- "PA_MPJPE_mm": 52.799629803153906,
- "PCK3D_50mm": 0.6171880402162939,
- "PCK3D_100mm": 0.9137313474118022,
- "AUC_0_150mm": 0.6588585546170801,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.67638708977429,
- "right_knee": 49.90776972054975,
- "right_hip": 47.59500825702852,
- "left_hip": 45.69908926563715,
- "left_knee": 52.63237381010554,
- "left_ankle": 59.50804348992904,
- "right_wrist": 73.20733256837394,
- "right_elbow": 40.311591979043484,
- "right_shoulder": 45.2575783909611,
- "left_shoulder": 40.48179298122341,
- "left_elbow": 44.97557430888548,
- "left_wrist": 76.34301577633514
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.32975847787291
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 48.33729889892481
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.998846717711615
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.42189837942712
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.84017463484044
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.14307160743546
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.931757444599214
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 61.218639657162434
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.454304804642774
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.53182282055049
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.0317522735558
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.03654554137998
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.79147487796578
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.11983602452117
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 68.54853339521294
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 58.90072056019343
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.8352058130093
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8_emdb1_pve.json b/benchmark/results/gen8_emdb1_pve.json
deleted file mode 100644
index 5b4e56c..0000000
--- a/benchmark/results/gen8_emdb1_pve.json
+++ /dev/null
@@ -1,122 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 295,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 86.61826157660215,
- "PA_PVE_direct_mm": 61.45216096468052,
- "J14_h36m_direct_MPJPE_mm": 73.95945739746094,
- "J14_h36m_direct_PA_MPJPE_mm": 46.92595779880267,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 39.1488172578375,
- "P2_23_outdoor_hug_tree": 44.57636396142284,
- "P3_31_outdoor_workout": 49.75435500625264,
- "P3_32_outdoor_soccer_warmup_a": 47.5304612919426,
- "P3_33_outdoor_soccer_warmup_b": 45.46984275261015,
- "P5_42_indoor_dancing": 65.41958926492853,
- "P5_44_indoor_rom": 44.5641101350517,
- "P6_49_outdoor_big_stairs_down": 55.346004668200486,
- "P6_50_outdoor_workout": 43.23006708197633,
- "P6_51_outdoor_dancing": 40.62459213423858,
- "P7_57_outdoor_rock_chair": 42.645138006118195,
- "P7_59_outdoor_rom": 45.74840867851276,
- "P7_60_outdoor_workout": 41.648605486065925,
- "P8_64_outdoor_skateboard": 45.9690529338183,
- "P8_68_outdoor_handstand": 53.59263243840905,
- "P8_69_outdoor_cartwheel": 51.33240557887028,
- "P9_76_outdoor_sitting": 45.30673007677923
- }
- },
- "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s1/g8h_s1/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 247,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 86.2876905819106,
- "PA_PVE_direct_mm": 61.24528232010409,
- "J14_h36m_direct_MPJPE_mm": 73.50804138183594,
- "J14_h36m_direct_PA_MPJPE_mm": 46.685678262793004,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 39.27861507171429,
- "P2_23_outdoor_hug_tree": 44.70765158288387,
- "P3_31_outdoor_workout": 49.69947749044918,
- "P3_32_outdoor_soccer_warmup_a": 48.12621408377986,
- "P3_33_outdoor_soccer_warmup_b": 45.42930773102074,
- "P5_42_indoor_dancing": 64.14754776420781,
- "P5_44_indoor_rom": 43.79878867274744,
- "P6_49_outdoor_big_stairs_down": 52.324522252451196,
- "P6_50_outdoor_workout": 44.06462517132796,
- "P6_51_outdoor_dancing": 40.03121214461728,
- "P7_57_outdoor_rock_chair": 42.93569640041334,
- "P7_59_outdoor_rom": 46.03404013488489,
- "P7_60_outdoor_workout": 42.55385361146884,
- "P8_64_outdoor_skateboard": 45.657419348418586,
- "P8_68_outdoor_handstand": 53.82902059086308,
- "P8_69_outdoor_cartwheel": 50.75891419689283,
- "P9_76_outdoor_sitting": 44.387613355925474
- }
- },
- "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8r_s0/g8r_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 92,
- "source": "ema",
- "fit_residual_mm": null,
- "PVE_direct_mm": 87.5872682109864,
- "PA_PVE_direct_mm": 61.256274114720206,
- "J14_h36m_direct_MPJPE_mm": 74.8912353515625,
- "J14_h36m_direct_PA_MPJPE_mm": 46.94383966600102,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 37.93555713956423,
- "P2_23_outdoor_hug_tree": 43.16786199992927,
- "P3_31_outdoor_workout": 50.26512931625483,
- "P3_32_outdoor_soccer_warmup_a": 48.212848135852894,
- "P3_33_outdoor_soccer_warmup_b": 45.77416445317726,
- "P5_42_indoor_dancing": 64.61202383289185,
- "P5_44_indoor_rom": 44.79325439308256,
- "P6_49_outdoor_big_stairs_down": 54.62202243113308,
- "P6_50_outdoor_workout": 43.49767218622079,
- "P6_51_outdoor_dancing": 39.35014650479087,
- "P7_57_outdoor_rock_chair": 45.07179953987791,
- "P7_59_outdoor_rom": 45.2898338035344,
- "P7_60_outdoor_workout": 42.123894666538696,
- "P8_64_outdoor_skateboard": 43.98416958537533,
- "P8_68_outdoor_handstand": 55.83053052397044,
- "P8_69_outdoor_cartwheel": 53.64123001278987,
- "P9_76_outdoor_sitting": 45.02367136464931
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8e600_3dpw_test_fitted.json b/benchmark/results/gen8e600_3dpw_test_fitted.json
deleted file mode 100644
index 5e7f1ff..0000000
--- a/benchmark/results/gen8e600_3dpw_test_fitted.json
+++ /dev/null
@@ -1,221 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 481,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.67208981530685,
- "PA_MPJPE_mm": 66.88827906149398,
- "PCK3D_50mm": 0.40043747809588265,
- "PCK3D_100mm": 0.8333152057879238,
- "AUC_0_150mm": 0.5614256099464217,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.32566466583239,
- "right_knee": 47.15757278410884,
- "right_hip": 97.91236566919363,
- "left_hip": 86.9490282552268,
- "left_knee": 44.53516581618817,
- "left_ankle": 65.83516249290801,
- "right_wrist": 69.57656761565505,
- "right_elbow": 57.30814205770195,
- "right_shoulder": 50.57434084812882,
- "left_shoulder": 52.783406809770966,
- "left_elbow": 66.62647224986277,
- "left_wrist": 71.49801868409502,
- "neck": 31.509690874000956,
- "head": 129.84430803824233
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.33834875125983,
- "PA_MPJPE_mm": 61.536504952360026,
- "PCK3D_50mm": 0.4448274727650415,
- "PCK3D_100mm": 0.8943217813871734,
- "AUC_0_150mm": 0.5942502847887198,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.237419998496975,
- "right_knee": 49.111783974488894,
- "right_hip": 97.51347173184112,
- "left_hip": 86.22150295326539,
- "left_knee": 45.48550400353603,
- "left_ankle": 60.84145088776921,
- "right_wrist": 65.8537664444401,
- "right_elbow": 48.74105875119605,
- "right_shoulder": 50.52034004161104,
- "left_shoulder": 50.659167441813175,
- "left_elbow": 54.43509093493264,
- "left_wrist": 67.81750226492976
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.85016697108857,
- "PA_MPJPE_mm": 40.61170253850853,
- "PCK3D_50mm": 0.7406330944525683,
- "PCK3D_100mm": 0.9546932215065199,
- "AUC_0_150mm": 0.7285012834172203
- }
- ]
- },
- "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 518,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.81827863509393,
- "PA_MPJPE_mm": 66.79035028325032,
- "PCK3D_50mm": 0.4021394531926636,
- "PCK3D_100mm": 0.8312788781869231,
- "AUC_0_150mm": 0.5619045280009501,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.49785874555987,
- "right_knee": 47.14725298578676,
- "right_hip": 97.98861684334267,
- "left_hip": 87.53378889657029,
- "left_knee": 44.11917522358694,
- "left_ankle": 66.3427163612456,
- "right_wrist": 68.0548811400338,
- "right_elbow": 57.05052993762211,
- "right_shoulder": 50.038698547415784,
- "left_shoulder": 52.571548872467744,
- "left_elbow": 66.82050584337476,
- "left_wrist": 72.64890085535404,
- "neck": 31.740279229099713,
- "head": 128.51015048404423
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.53770117978908,
- "PA_MPJPE_mm": 61.55104488747315,
- "PCK3D_50mm": 0.4452363496226114,
- "PCK3D_100mm": 0.8919343165176851,
- "AUC_0_150mm": 0.5939325979310862,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.981980839671074,
- "right_knee": 49.08710054345032,
- "right_hip": 97.56254084161058,
- "left_hip": 86.68303216464714,
- "left_knee": 45.08787177560861,
- "left_ankle": 61.3008290027359,
- "right_wrist": 64.5650465655679,
- "right_elbow": 48.95435324199886,
- "right_shoulder": 50.52685218966079,
- "left_shoulder": 50.57954325715595,
- "left_elbow": 54.56724101613218,
- "left_wrist": 68.71614721143841
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.77250332436297,
- "PA_MPJPE_mm": 40.69199406281121,
- "PCK3D_50mm": 0.7385625259324607,
- "PCK3D_100mm": 0.9540486865586265,
- "AUC_0_150mm": 0.7278909244151527
- }
- ]
- },
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "epoch": 295,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.90314007131975,
- "PA_MPJPE_mm": 66.75417606476643,
- "PCK3D_50mm": 0.39980502817826225,
- "PCK3D_100mm": 0.8327069259308495,
- "AUC_0_150mm": 0.5619064771993814,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.05109104673127,
- "right_knee": 46.94215549198992,
- "right_hip": 97.44293023222869,
- "left_hip": 86.91167415608224,
- "left_knee": 45.064097051510274,
- "left_ankle": 65.60436699939369,
- "right_wrist": 68.87685239747502,
- "right_elbow": 57.44952437991582,
- "right_shoulder": 50.244764143404396,
- "left_shoulder": 52.69003609812139,
- "left_elbow": 66.52677846287735,
- "left_wrist": 72.22817355045031,
- "neck": 31.507160128258747,
- "head": 129.0188607682908
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.72900938161254,
- "PA_MPJPE_mm": 61.44884912037969,
- "PCK3D_50mm": 0.44510240720375227,
- "PCK3D_100mm": 0.8933747849871697,
- "AUC_0_150mm": 0.5943006932259247,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.9344712858018,
- "right_knee": 48.84689470618248,
- "right_hip": 97.24594667733341,
- "left_hip": 86.35254867154886,
- "left_knee": 45.94898323550374,
- "left_ankle": 61.31441118333219,
- "right_wrist": 65.41141248323005,
- "right_elbow": 49.0266037842799,
- "right_shoulder": 49.59549581483996,
- "left_shoulder": 49.954727201736944,
- "left_elbow": 54.2545604153924,
- "left_wrist": 68.50013398537457
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.08732433559445,
- "PA_MPJPE_mm": 40.79832921073415,
- "PCK3D_50mm": 0.7363086677865461,
- "PCK3D_100mm": 0.9543346989417542,
- "AUC_0_150mm": 0.7269478372408915
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8e600_3dpw_test_pve.json b/benchmark/results/gen8e600_3dpw_test_pve.json
deleted file mode 100644
index 3091f65..0000000
--- a/benchmark/results/gen8e600_3dpw_test_pve.json
+++ /dev/null
@@ -1,98 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 481,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 78.83505836762673,
- "PVE_published_mm": 78.83505836762673,
- "PA_PVE_direct_mm": 56.91410177852873,
- "J14_h36m_direct_MPJPE_mm": 65.5341567993164,
- "J14_h36m_direct_PA_MPJPE_mm": 41.6480581870448,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 46.724881619073784,
- "downtown_bar_00": 54.42938286092481,
- "downtown_bus_00": 50.64704855123716,
- "downtown_cafe_00": 38.94228866305341,
- "downtown_car_00": 45.52335143280792,
- "downtown_crossStreets_00": 41.59451640232422,
- "downtown_downstairs_00": 35.11467225342558,
- "downtown_enterShop_00": 29.990330992426575,
- "downtown_rampAndStairs_00": 40.89617187771701,
- "downtown_runForBus_00": 38.13501433228221,
- "downtown_runForBus_01": 49.92449802153219,
- "downtown_sitOnStairs_00": 38.329424588117945,
- "downtown_stairs_00": 51.95312557603397,
- "downtown_upstairs_00": 31.700704154831314,
- "downtown_walkBridge_01": 31.364117379866755,
- "downtown_walkUphill_00": 30.1434090742278,
- "downtown_walking_00": 39.205795579853636,
- "downtown_warmWelcome_00": 46.53793134752158,
- "downtown_weeklyMarket_00": 36.885936538418214,
- "downtown_windowShopping_00": 32.75120179747051,
- "flat_guitar_01": 46.340424264768004,
- "flat_packBags_00": 40.71895068904101,
- "office_phoneCall_00": 34.74016390333005,
- "outdoors_fencing_01": 44.10642225989501
- }
- },
- "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 518,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 78.43976667437313,
- "PVE_published_mm": 78.43976667437313,
- "PA_PVE_direct_mm": 56.785389489845365,
- "J14_h36m_direct_MPJPE_mm": 65.23712158203125,
- "J14_h36m_direct_PA_MPJPE_mm": 41.61939363877355,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 46.71709755844352,
- "downtown_bar_00": 53.83207822629599,
- "downtown_bus_00": 48.65236033605704,
- "downtown_cafe_00": 38.66125558703131,
- "downtown_car_00": 46.18466352478656,
- "downtown_crossStreets_00": 42.57564925326769,
- "downtown_downstairs_00": 33.91592730608885,
- "downtown_enterShop_00": 30.01681560754425,
- "downtown_rampAndStairs_00": 41.571448661341506,
- "downtown_runForBus_00": 38.444672503526775,
- "downtown_runForBus_01": 48.819265356758876,
- "downtown_sitOnStairs_00": 37.697398733723126,
- "downtown_stairs_00": 52.51551969286717,
- "downtown_upstairs_00": 32.05573658912798,
- "downtown_walkBridge_01": 31.813943698241207,
- "downtown_walkUphill_00": 30.111507249035302,
- "downtown_walking_00": 40.58534160883497,
- "downtown_warmWelcome_00": 46.69374967255896,
- "downtown_weeklyMarket_00": 36.38936390754917,
- "downtown_windowShopping_00": 33.241522601468155,
- "flat_guitar_01": 48.74412804708892,
- "flat_packBags_00": 41.76850909461474,
- "office_phoneCall_00": 33.65640120474975,
- "outdoors_fencing_01": 43.28125434152628
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8e600_emdb1_annotated.json b/benchmark/results/gen8e600_emdb1_annotated.json
deleted file mode 100644
index 88cd194..0000000
--- a/benchmark/results/gen8e600_emdb1_annotated.json
+++ /dev/null
@@ -1,346 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 86.56307577082555,
- "PA_MPJPE_mm": 62.870025736990414,
- "PCK3D_50mm": 0.5740969312257119,
- "PCK3D_100mm": 0.850348850073988,
- "AUC_0_150mm": 0.6204626292448314,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 41.92819348367251,
- "left_hip": 42.983309591841284,
- "right_hip": 41.99028748432828,
- "spine1": 50.546013737542765,
- "left_knee": 55.87615404726698,
- "right_knee": 55.503917699781574,
- "spine2": 50.49607895919742,
- "left_ankle": 66.1181759111917,
- "right_ankle": 66.98668454711286,
- "spine3": 49.78704163506046,
- "left_foot": 73.13540954015934,
- "right_foot": 75.7911614026987,
- "neck": 48.6956588109759,
- "left_collar": 42.73892796587593,
- "right_collar": 42.10077260781202,
- "head": 55.32774526303255,
- "left_shoulder": 47.76003947941104,
- "right_shoulder": 47.36225766052754,
- "left_elbow": 60.362470277701114,
- "right_elbow": 56.31569359814934,
- "left_wrist": 100.02892705884527,
- "right_wrist": 97.56416452991154,
- "left_hand": 118.92762398132496,
- "right_hand": 120.55390841434752
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 97.52043161985303,
- "PA_MPJPE_mm": 66.60282504447187,
- "PCK3D_50mm": 0.4932373563456831,
- "PCK3D_100mm": 0.8317903521197717,
- "AUC_0_150mm": 0.5863527132309093,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.54325731831655,
- "right_knee": 52.5715564247917,
- "right_hip": 50.65727821785841,
- "left_hip": 48.61720059795035,
- "left_knee": 54.35848136667145,
- "left_ankle": 68.57272375227623,
- "right_wrist": 99.14361640124834,
- "right_elbow": 51.80272340308523,
- "right_shoulder": 57.842049573846666,
- "left_shoulder": 49.321726616475765,
- "left_elbow": 57.59989885904841,
- "left_wrist": 104.93261432735564,
- "neck": 59.08336480050076,
- "head": 110.39305896318065
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 92.62130444081637,
- "PA_MPJPE_mm": 62.52354452474751,
- "PCK3D_50mm": 0.5455890691338561,
- "PCK3D_100mm": 0.8560656349831971,
- "AUC_0_150mm": 0.6123240581750631,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.09113869530734,
- "right_knee": 53.47881404589802,
- "right_hip": 50.16151816441917,
- "left_hip": 48.88523996220425,
- "left_knee": 55.289079967341344,
- "left_ankle": 66.70908987983816,
- "right_wrist": 94.9783202880758,
- "right_elbow": 49.31230615843084,
- "right_shoulder": 57.06256345327593,
- "left_shoulder": 53.488600402271814,
- "left_elbow": 54.9418653064444,
- "left_wrist": 99.88399797346307
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 78.73205914997531
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 69.90917751476255
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 57.19897705885768
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.67358303674525
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 57.442956505064636
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 80.65953393773327
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 51.99371657415196
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 78.22017006614858
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 52.73912827718323
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 60.6432327921372
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 59.01714813491954
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 60.25282361405862
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.7914121366103
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 65.03925056034467
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 71.88318204701882
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 54.47972068910211
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 59.022351604762875
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 518,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 86.8526631463249,
- "PA_MPJPE_mm": 64.05441730455037,
- "PCK3D_50mm": 0.5658821861732288,
- "PCK3D_100mm": 0.8431644055373466,
- "AUC_0_150mm": 0.6150335879306507,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.380681813664424,
- "left_hip": 43.14556979151991,
- "right_hip": 41.90776232971016,
- "spine1": 51.22295988018028,
- "left_knee": 55.764830320027066,
- "right_knee": 54.83281414553069,
- "spine2": 51.471690765291555,
- "left_ankle": 66.17087747941379,
- "right_ankle": 67.57230277686087,
- "spine3": 51.26440173016567,
- "left_foot": 73.71044418544813,
- "right_foot": 76.95746977694525,
- "neck": 50.78486534194797,
- "left_collar": 44.639826106707844,
- "right_collar": 43.99084423923209,
- "head": 56.82741409596142,
- "left_shoulder": 49.24272031887481,
- "right_shoulder": 48.14018206736064,
- "left_elbow": 61.415998933054354,
- "right_elbow": 57.17333195963461,
- "left_wrist": 101.3900004424182,
- "right_wrist": 100.84837252598894,
- "left_hand": 122.01449168342972,
- "right_hand": 124.4361625998391
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 97.48756148761787,
- "PA_MPJPE_mm": 67.07187484458336,
- "PCK3D_50mm": 0.49156299452943025,
- "PCK3D_100mm": 0.8295884922445932,
- "AUC_0_150mm": 0.5842171968955812,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.67528864650981,
- "right_knee": 52.191939340524335,
- "right_hip": 50.17118548228657,
- "left_hip": 47.97431966069622,
- "left_knee": 54.63651790459916,
- "left_ankle": 68.44809808221093,
- "right_wrist": 102.85687493782847,
- "right_elbow": 52.825735929818634,
- "right_shoulder": 57.91663750460134,
- "left_shoulder": 51.03963804392885,
- "left_elbow": 57.58853860242432,
- "left_wrist": 105.7341804255367,
- "neck": 61.42412632106316,
- "head": 108.5231669421385
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 93.25364651573624,
- "PA_MPJPE_mm": 63.20718597629368,
- "PCK3D_50mm": 0.5412362223236389,
- "PCK3D_100mm": 0.853462224619342,
- "AUC_0_150mm": 0.6090766821066756,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.70111560985947,
- "right_knee": 53.21055489696277,
- "right_hip": 49.88626010638046,
- "left_hip": 48.55953948036445,
- "left_knee": 55.60604452614569,
- "left_ankle": 66.79906795260878,
- "right_wrist": 98.00589178875549,
- "right_elbow": 50.51912111454967,
- "right_shoulder": 58.1470800514442,
- "left_shoulder": 55.91817111431265,
- "left_elbow": 55.15661183062778,
- "left_wrist": 99.97677324351281
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 79.28268429761474
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 73.42796633035849
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.043883427803706
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 56.909969339467104
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 59.87977625757823
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 80.19744683923537
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 54.23044843360599
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 85.56935324693771
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 55.77721722025817
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 64.04983187986235
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 54.686871127603425
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 60.572792685344226
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 57.99382653290719
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 61.5382452105661
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 73.29607622049595
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.39597209180131
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.897785317188756
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8e600_emdb1_fitted.json b/benchmark/results/gen8e600_emdb1_fitted.json
deleted file mode 100644
index 33d142e..0000000
--- a/benchmark/results/gen8e600_emdb1_fitted.json
+++ /dev/null
@@ -1,518 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 75.45319378157116,
- "PA_MPJPE_mm": 50.32863498976247,
- "PCK3D_50mm": 0.6644383824973378,
- "PCK3D_100mm": 0.9189433542159344,
- "AUC_0_150mm": 0.6767868765722734,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.29148536825942,
- "left_hip": 38.24552586212516,
- "right_hip": 37.600955289971296,
- "spine1": 42.841433811205626,
- "left_knee": 50.3974071974786,
- "right_knee": 50.636864207098895,
- "spine2": 42.87487848201726,
- "left_ankle": 54.447180676894604,
- "right_ankle": 55.30010689324191,
- "spine3": 41.06766402362964,
- "left_foot": 61.48244702825232,
- "right_foot": 62.10221206337029,
- "neck": 32.24941748448096,
- "left_collar": 28.165231607074126,
- "right_collar": 29.045948558461998,
- "head": 37.68983884655172,
- "left_shoulder": 37.81425445121847,
- "right_shoulder": 39.70770089314365,
- "left_elbow": 50.604477974573584,
- "right_elbow": 47.003193546634456,
- "left_wrist": 76.40272950121867,
- "right_wrist": 73.48478178514239,
- "left_hand": 90.80408930132057,
- "right_hand": 91.62741490093312
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.41165848688678,
- "PA_MPJPE_mm": 56.76824098666464,
- "PCK3D_50mm": 0.5623544194261533,
- "PCK3D_100mm": 0.8884311970649771,
- "AUC_0_150mm": 0.6310808777460634,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.107587898242244,
- "right_knee": 48.17377530664997,
- "right_hip": 48.27013921643617,
- "left_hip": 45.91431273776492,
- "left_knee": 50.0210048258065,
- "left_ankle": 58.42373661025704,
- "right_wrist": 74.65785131579995,
- "right_elbow": 41.81074013327866,
- "right_shoulder": 52.85493659343189,
- "left_shoulder": 43.177167245776566,
- "left_elbow": 47.45422252448673,
- "left_wrist": 80.89639938141364,
- "neck": 43.42408396871431,
- "head": 102.56941605524631
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.79194371698863,
- "PA_MPJPE_mm": 51.799443261140055,
- "PCK3D_50mm": 0.6299319586773431,
- "PCK3D_100mm": 0.9173201122958414,
- "AUC_0_150mm": 0.6642359024598286,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.64409338810367,
- "right_knee": 48.88869145759666,
- "right_hip": 45.3866750768073,
- "left_hip": 43.713823356970885,
- "left_knee": 50.70734109073142,
- "left_ankle": 55.59481655264034,
- "right_wrist": 73.1607309509648,
- "right_elbow": 39.977807550054926,
- "right_shoulder": 45.675124820251376,
- "left_shoulder": 40.460065309981935,
- "left_elbow": 45.291183998498376,
- "left_wrist": 78.092965581079
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.65367167514524
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.27719618777549
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 50.95631810035045
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.76531423523893
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.57629567928524
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.6821731285332
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.13458534299859
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.84716692393309
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.161930904869564
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.06490859601754
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.47200755913893
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.147604732459804
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.91034623837281
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.41865594510458
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 59.00598028526663
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 51.79904127762359
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.21967075691419
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 518,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 75.03129863376317,
- "PA_MPJPE_mm": 50.231048272159896,
- "PCK3D_50mm": 0.662592139291098,
- "PCK3D_100mm": 0.9195760555394211,
- "AUC_0_150mm": 0.6764055605445983,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.41865419530961,
- "left_hip": 37.83689958269942,
- "right_hip": 37.263008304124185,
- "spine1": 43.04592857790974,
- "left_knee": 49.34990316220861,
- "right_knee": 50.030527622090254,
- "spine2": 42.87024932905197,
- "left_ankle": 54.345121455629936,
- "right_ankle": 54.85922455941696,
- "spine3": 41.01227290744936,
- "left_foot": 61.50587319072112,
- "right_foot": 62.195691635973134,
- "neck": 33.31026799055825,
- "left_collar": 28.903970424939203,
- "right_collar": 29.52583959286704,
- "head": 38.32914476397436,
- "left_shoulder": 38.142237765326996,
- "right_shoulder": 39.11885299716117,
- "left_elbow": 50.74154487747758,
- "right_elbow": 46.30681101170729,
- "left_wrist": 74.9902338760167,
- "right_wrist": 73.0563780271927,
- "left_hand": 90.74249012240489,
- "right_hand": 91.6440325596265
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 85.8657145275826,
- "PA_MPJPE_mm": 56.33370553060074,
- "PCK3D_50mm": 0.565246768333521,
- "PCK3D_100mm": 0.8919636559764345,
- "AUC_0_150mm": 0.6329391660330254,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.48351146940709,
- "right_knee": 48.16205288093324,
- "right_hip": 47.60072663781627,
- "left_hip": 45.31644924800634,
- "left_knee": 49.38107178275438,
- "left_ankle": 58.09469210609631,
- "right_wrist": 74.34390980008577,
- "right_elbow": 41.345799078829984,
- "right_shoulder": 52.11257155832642,
- "left_shoulder": 42.82043785277687,
- "left_elbow": 46.69823757231741,
- "left_wrist": 79.00194940757115,
- "neck": 44.89936900123691,
- "head": 102.41109903225222
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.33336647272002,
- "PA_MPJPE_mm": 51.152573115354095,
- "PCK3D_50mm": 0.6337177944654192,
- "PCK3D_100mm": 0.9216591295689334,
- "AUC_0_150mm": 0.6672094759988383,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.043562963958,
- "right_knee": 48.887752965350906,
- "right_hip": 44.5391206807371,
- "left_hip": 43.03052454915732,
- "left_knee": 50.05114121690766,
- "left_ankle": 55.28490959413068,
- "right_wrist": 72.34281455553291,
- "right_elbow": 39.37444357360621,
- "right_shoulder": 45.02155924404227,
- "left_shoulder": 40.78841670415529,
- "left_elbow": 44.77082451455856,
- "left_wrist": 75.6958068221123
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 42.90745748634419
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.383834345775064
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.25140927621456
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.260652503327705
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.359897452014046
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.83079485059494
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.74991980467376
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 59.1226037425883
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 44.88345561138241
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.98589302328467
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 45.53582601931474
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.30386915165015
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.73238533385643
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.11142723550196
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 59.38407851610339
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 52.73235829313001
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 52.02449069677688
- }
- }
- },
- "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 295,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 77.28133498531433,
- "PA_MPJPE_mm": 51.9491732712096,
- "PCK3D_50mm": 0.6465827213763156,
- "PCK3D_100mm": 0.9107372526241547,
- "AUC_0_150mm": 0.6681061095772222,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.459041428548,
- "left_hip": 39.46379253972692,
- "right_hip": 38.633448237690324,
- "spine1": 44.07300560792153,
- "left_knee": 52.72926789360471,
- "right_knee": 51.96374265609758,
- "spine2": 43.805976220856614,
- "left_ankle": 57.899022895229834,
- "right_ankle": 58.18971465552947,
- "spine3": 41.80827173378941,
- "left_foot": 66.19531000919939,
- "right_foot": 65.19604680752869,
- "neck": 33.18097089510936,
- "left_collar": 28.820628396996476,
- "right_collar": 30.045158123315833,
- "head": 38.438863881765414,
- "left_shoulder": 38.26718356914064,
- "right_shoulder": 40.14262493829269,
- "left_elbow": 51.79547271490188,
- "right_elbow": 47.866343644847056,
- "left_wrist": 77.91960724642401,
- "right_wrist": 75.09057325487099,
- "left_hand": 93.90372595034053,
- "right_hand": 93.89236520730692
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 88.24875987466123,
- "PA_MPJPE_mm": 58.0608300944181,
- "PCK3D_50mm": 0.5467013590483698,
- "PCK3D_100mm": 0.8819204485511585,
- "AUC_0_150mm": 0.6240450210702877,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.87164369689338,
- "right_knee": 49.90000868892296,
- "right_hip": 49.48332988909885,
- "left_hip": 46.825825586548234,
- "left_knee": 52.672342881063734,
- "left_ankle": 61.397319075815396,
- "right_wrist": 76.22799662283796,
- "right_elbow": 42.975780811329976,
- "right_shoulder": 53.59319189262762,
- "left_shoulder": 42.9418249779795,
- "left_elbow": 48.27005010836092,
- "left_wrist": 81.69373627636021,
- "neck": 44.48096774412928,
- "head": 102.51760306988548
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 82.80475609909476,
- "PA_MPJPE_mm": 53.221210122957814,
- "PCK3D_50mm": 0.6126934406505414,
- "PCK3D_100mm": 0.9096378044226859,
- "AUC_0_150mm": 0.6565272738547248,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.47732042542089,
- "right_knee": 50.42859296831835,
- "right_hip": 46.48760097516529,
- "left_hip": 44.565920495992735,
- "left_knee": 53.215563230815704,
- "left_ankle": 59.058105680748255,
- "right_wrist": 74.41820278016962,
- "right_elbow": 40.436950297876166,
- "right_shoulder": 46.68281764042519,
- "left_shoulder": 41.19756169704192,
- "left_elbow": 45.98214897741906,
- "left_wrist": 78.7037363061006
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 43.31888915722836
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.180015950378625
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 51.61758101633136
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 49.93586472078992
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 46.68536095147245
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.88227217037303
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.53485487091855
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.902593364750075
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.4289648281872
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.1923851505244
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.64224046282951
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.91192536080979
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.93165730133809
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.69479899747672
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 66.71627636180757
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 56.348237256175786
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 55.442139025455056
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen8e600_emdb1_pve.json b/benchmark/results/gen8e600_emdb1_pve.json
deleted file mode 100644
index 1be2175..0000000
--- a/benchmark/results/gen8e600_emdb1_pve.json
+++ /dev/null
@@ -1,84 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 481,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 84.87610670870846,
- "PVE_published_mm": 89.23325418783574,
- "PA_PVE_direct_mm": 60.11191741514725,
- "J14_h36m_direct_MPJPE_mm": 72.01241302490234,
- "J14_h36m_direct_PA_MPJPE_mm": 45.65661630917442,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 37.99256784945708,
- "P2_23_outdoor_hug_tree": 43.75399998585647,
- "P3_31_outdoor_workout": 49.17296693829096,
- "P3_32_outdoor_soccer_warmup_a": 45.220382244989246,
- "P3_33_outdoor_soccer_warmup_b": 44.67102976862948,
- "P5_42_indoor_dancing": 63.153714350515436,
- "P5_44_indoor_rom": 42.939674223326584,
- "P6_49_outdoor_big_stairs_down": 53.40884785044657,
- "P6_50_outdoor_workout": 43.24825680371942,
- "P6_51_outdoor_dancing": 40.08151540563019,
- "P7_57_outdoor_rock_chair": 43.262392501333046,
- "P7_59_outdoor_rom": 44.99145225174039,
- "P7_60_outdoor_workout": 40.48482399201354,
- "P8_64_outdoor_skateboard": 44.94984976649381,
- "P8_68_outdoor_handstand": 48.69721892865197,
- "P8_69_outdoor_cartwheel": 47.65970172585432,
- "P9_76_outdoor_sitting": 45.005142388296726
- }
- },
- "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr2x_e600/g8h_lr2x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 518,
- "source": "raw",
- "fit_residual_mm": null,
- "PVE_direct_mm": 84.54918767526907,
- "PVE_published_mm": 88.66376636986152,
- "PA_PVE_direct_mm": 59.8689347697134,
- "J14_h36m_direct_MPJPE_mm": 71.76900482177734,
- "J14_h36m_direct_PA_MPJPE_mm": 45.28566342247253,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 38.73744131520006,
- "P2_23_outdoor_hug_tree": 44.540587008765726,
- "P3_31_outdoor_workout": 49.22495972283584,
- "P3_32_outdoor_soccer_warmup_a": 46.730456590053365,
- "P3_33_outdoor_soccer_warmup_b": 44.31893254533928,
- "P5_42_indoor_dancing": 63.19556704651091,
- "P5_44_indoor_rom": 43.069485931052355,
- "P6_49_outdoor_big_stairs_down": 53.014429580382995,
- "P6_50_outdoor_workout": 41.591717317406115,
- "P6_51_outdoor_dancing": 39.49775587240576,
- "P7_57_outdoor_rock_chair": 41.30756870916201,
- "P7_59_outdoor_rom": 45.114976641168006,
- "P7_60_outdoor_workout": 40.65706441389069,
- "P8_64_outdoor_skateboard": 44.77179547532716,
- "P8_68_outdoor_handstand": 47.84618225792263,
- "P8_69_outdoor_cartwheel": 47.986706388143425,
- "P9_76_outdoor_sitting": 42.62847741551841
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_3dpw_test_detector_224.json b/benchmark/results/gen9w_3dpw_test_detector_224.json
deleted file mode 100644
index 1085f7f..0000000
--- a/benchmark/results/gen9w_3dpw_test_detector_224.json
+++ /dev/null
@@ -1,402 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 481,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 102.802403320663,
- "PA_MPJPE_mm": 66.93925661490847,
- "PCK3D_50mm": 0.4003935001698562,
- "PCK3D_100mm": 0.8327280521539382,
- "AUC_0_150mm": 0.5613543513189554,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.92724793708123,
- "right_knee": 47.681260129687466,
- "right_hip": 97.80681224685675,
- "left_hip": 87.411378795221,
- "left_knee": 44.90926668159655,
- "left_ankle": 67.75528187409769,
- "right_wrist": 69.67874616608141,
- "right_elbow": 56.75989849839467,
- "right_shoulder": 50.374651542017716,
- "left_shoulder": 52.54578426706502,
- "left_elbow": 66.48660509530406,
- "left_wrist": 70.4250684433762,
- "neck": 32.89842934209658,
- "head": 126.48916158984225
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.27572797075382,
- "PA_MPJPE_mm": 61.88421964231757,
- "PCK3D_50mm": 0.4417369682557657,
- "PCK3D_100mm": 0.8914524591401503,
- "AUC_0_150mm": 0.5926583010971858,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.35325771958618,
- "right_knee": 49.5945508722776,
- "right_hip": 97.0333952043599,
- "left_hip": 86.33777184050125,
- "left_knee": 45.893168885833184,
- "left_ankle": 62.66351902522998,
- "right_wrist": 66.50331066802183,
- "right_elbow": 48.277622889192564,
- "right_shoulder": 51.2244104232857,
- "left_shoulder": 51.42343983263935,
- "left_elbow": 53.99730596883275,
- "left_wrist": 67.3088823780505
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 65.73074791967127,
- "PA_MPJPE_mm": 41.019988038110064,
- "PCK3D_50mm": 0.7383547406053351,
- "PCK3D_100mm": 0.9514838151317598,
- "AUC_0_150mm": 0.7263636373124212
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 495,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 110.20868308510333,
- "PA_MPJPE_mm": 72.69648884062465,
- "PCK3D_50mm": 0.353763932251646,
- "PCK3D_100mm": 0.7946741996538169,
- "AUC_0_150mm": 0.5304355676249408,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 73.76391619607557,
- "right_knee": 51.04657803114811,
- "right_hip": 101.07798552576386,
- "left_hip": 88.63211880117603,
- "left_knee": 51.62250108432617,
- "left_ankle": 76.98041103095609,
- "right_wrist": 81.59096265441863,
- "right_elbow": 64.43759884504529,
- "right_shoulder": 50.40388856598386,
- "left_shoulder": 52.51938250796732,
- "left_elbow": 71.83843532349752,
- "left_wrist": 85.4822881551009,
- "neck": 35.31914658455966,
- "head": 133.0356304627262
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 108.4110591777182,
- "PA_MPJPE_mm": 67.79154414465411,
- "PCK3D_50mm": 0.37848441097648433,
- "PCK3D_100mm": 0.8507058468274639,
- "AUC_0_150mm": 0.5591763131588016,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.05294944564129,
- "right_knee": 53.001185761925804,
- "right_hip": 99.86467082183319,
- "left_hip": 87.64359862326027,
- "left_knee": 52.3819919166503,
- "left_ankle": 71.23856804368269,
- "right_wrist": 77.82372776481559,
- "right_elbow": 54.7525531392323,
- "right_shoulder": 53.16781933664017,
- "left_shoulder": 53.655448317581616,
- "left_elbow": 58.55171200243005,
- "left_wrist": 81.36430456215597
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 75.41681917092193,
- "PA_MPJPE_mm": 48.38154241886281,
- "PCK3D_50mm": 0.649972903893751,
- "PCK3D_100mm": 0.9220885840464597,
- "AUC_0_150mm": 0.6831865725421599
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 476,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 109.40783584305251,
- "PA_MPJPE_mm": 71.70127941363054,
- "PCK3D_50mm": 0.3548376660142032,
- "PCK3D_100mm": 0.8019274633191517,
- "AUC_0_150mm": 0.5344579761178545,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 73.06189838228754,
- "right_knee": 50.93503063903913,
- "right_hip": 100.38910579744802,
- "left_hip": 88.89844010358713,
- "left_knee": 50.3141381628107,
- "left_ankle": 75.15120200974899,
- "right_wrist": 77.78687739823988,
- "right_elbow": 63.487060349921265,
- "right_shoulder": 50.688449321720626,
- "left_shoulder": 52.54135978546837,
- "left_elbow": 71.4493609701724,
- "left_wrist": 82.65568129362575,
- "neck": 35.94860110919917,
- "head": 130.51070646755878
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 107.69969469037723,
- "PA_MPJPE_mm": 66.81574278354587,
- "PCK3D_50mm": 0.38211980523157063,
- "PCK3D_100mm": 0.8554453063073265,
- "AUC_0_150mm": 0.5633054470870577,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.10671924334784,
- "right_knee": 52.6856064414195,
- "right_hip": 99.21367540465826,
- "left_hip": 87.93050975214916,
- "left_knee": 51.12801317306597,
- "left_ankle": 69.85845169323932,
- "right_wrist": 74.49470946424798,
- "right_elbow": 53.895246908481184,
- "right_shoulder": 53.097647808165966,
- "left_shoulder": 53.73756671187732,
- "left_elbow": 58.14509692580495,
- "left_wrist": 78.49566987609317
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 74.1847470712719,
- "PA_MPJPE_mm": 46.8788021555531,
- "PCK3D_50mm": 0.665357021531294,
- "PCK3D_100mm": 0.9274208551045828,
- "AUC_0_150mm": 0.6910548799865578
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 343,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 105.34652220708722,
- "PA_MPJPE_mm": 69.19137273367568,
- "PCK3D_50mm": 0.3799379620492745,
- "PCK3D_100mm": 0.8191537117621366,
- "AUC_0_150mm": 0.5486452077333356,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.33284959525155,
- "right_knee": 48.25097870632742,
- "right_hip": 99.84653148910938,
- "left_hip": 86.83617301422545,
- "left_knee": 48.24432160765498,
- "left_ankle": 69.9301046665157,
- "right_wrist": 73.85295091794026,
- "right_elbow": 62.31605031240204,
- "right_shoulder": 49.74112604363924,
- "left_shoulder": 51.217442660728146,
- "left_elbow": 70.23284472718139,
- "left_wrist": 77.70789254234514,
- "neck": 34.87503432501514,
- "head": 127.29491766312377
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 103.07528775856375,
- "PA_MPJPE_mm": 64.25772718309034,
- "PCK3D_50mm": 0.4128968029290756,
- "PCK3D_100mm": 0.873733155927981,
- "AUC_0_150mm": 0.5786008750942124,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.32277547992246,
- "right_knee": 50.19616087720301,
- "right_hip": 98.61729210783815,
- "left_hip": 86.13279157522193,
- "left_knee": 49.20897942425856,
- "left_ankle": 64.63698225414257,
- "right_wrist": 71.1235943113079,
- "right_elbow": 53.10789560397975,
- "right_shoulder": 50.59170380632294,
- "left_shoulder": 51.63403722297492,
- "left_elbow": 57.48970873003428,
- "left_wrist": 74.03080480387759
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 70.36033753463434,
- "PA_MPJPE_mm": 44.29333724889724,
- "PCK3D_50mm": 0.6953791028357895,
- "PCK3D_100mm": 0.9398911302716081,
- "AUC_0_150mm": 0.7065474081406775
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 416,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 104.34548140646658,
- "PA_MPJPE_mm": 67.41136752523109,
- "PCK3D_50mm": 0.3982864745943672,
- "PCK3D_100mm": 0.8296140220327741,
- "AUC_0_150mm": 0.5581152772987075,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.90078526085212,
- "right_knee": 47.79385060260674,
- "right_hip": 99.14072001474942,
- "left_hip": 88.07138968284741,
- "left_knee": 46.40155161064613,
- "left_ankle": 67.60772064824984,
- "right_wrist": 70.43146348843383,
- "right_elbow": 59.74154296351525,
- "right_shoulder": 49.07575686048597,
- "left_shoulder": 49.771345397534404,
- "left_elbow": 67.82203708720154,
- "left_wrist": 73.88962280922453,
- "neck": 33.2415039215723,
- "head": 125.86985500531573
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 102.42531607406545,
- "PA_MPJPE_mm": 62.482247488876524,
- "PCK3D_50mm": 0.4354994715585249,
- "PCK3D_100mm": 0.8853848186313366,
- "AUC_0_150mm": 0.5882814602274733,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.283398468259755,
- "right_knee": 49.85477801856356,
- "right_hip": 98.48421240619103,
- "left_hip": 87.33270084409006,
- "left_knee": 47.35997731045054,
- "left_ankle": 62.02892084989324,
- "right_wrist": 67.20455352622737,
- "right_elbow": 51.080221123787965,
- "right_shoulder": 50.0821785669609,
- "left_shoulder": 49.35842433137748,
- "left_elbow": 55.250975022271525,
- "left_wrist": 70.46662939844485
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 67.44931889046163,
- "PA_MPJPE_mm": 41.92902110491711,
- "PCK3D_50mm": 0.7224509439151042,
- "PCK3D_100mm": 0.9489177734280214,
- "AUC_0_150mm": 0.7202546199056841
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_3dpw_test_detector_288.json b/benchmark/results/gen9w_3dpw_test_detector_288.json
deleted file mode 100644
index ac56aa8..0000000
--- a/benchmark/results/gen9w_3dpw_test_detector_288.json
+++ /dev/null
@@ -1,322 +0,0 @@
-{
- "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 361,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 107.64900789558786,
- "PA_MPJPE_mm": 70.81032462044344,
- "PCK3D_50mm": 0.3647661646472653,
- "PCK3D_100mm": 0.8093364284905447,
- "AUC_0_150mm": 0.540269940881697,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 71.35775486625703,
- "right_knee": 49.795707521640566,
- "right_hip": 98.7890030491265,
- "left_hip": 87.0427080239817,
- "left_knee": 49.239000873166056,
- "left_ankle": 74.37423848001143,
- "right_wrist": 77.21461740726504,
- "right_elbow": 61.4200611188708,
- "right_shoulder": 50.26930087014049,
- "left_shoulder": 52.59859742615345,
- "left_elbow": 71.94560661663346,
- "left_wrist": 82.27841614398885,
- "neck": 35.68829825681443,
- "head": 129.33123403215822
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 105.89337858084649,
- "PA_MPJPE_mm": 65.98363711314647,
- "PCK3D_50mm": 0.3912236251085192,
- "PCK3D_100mm": 0.8645750764352848,
- "AUC_0_150mm": 0.5689662546284289,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.5890291321839,
- "right_knee": 51.74747921130889,
- "right_hip": 97.46833755084593,
- "left_hip": 85.83107503347951,
- "left_knee": 49.895725065574624,
- "left_ankle": 68.54512354579865,
- "right_wrist": 74.5849517925945,
- "right_elbow": 52.86372615599355,
- "right_shoulder": 52.80257969309457,
- "left_shoulder": 53.29149741149982,
- "left_elbow": 58.70511450830544,
- "left_wrist": 78.4790062570784
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 72.90699760019562,
- "PA_MPJPE_mm": 46.314441607505984,
- "PCK3D_50mm": 0.6742764935211997,
- "PCK3D_100mm": 0.9288646327062136,
- "AUC_0_150mm": 0.6948263527049402
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 317,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 105.55093735249915,
- "PA_MPJPE_mm": 69.6434831058787,
- "PCK3D_50mm": 0.3740718572560946,
- "PCK3D_100mm": 0.8177200446479124,
- "AUC_0_150mm": 0.5462573062948025,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.32688633224707,
- "right_knee": 49.8616120176448,
- "right_hip": 98.52850340783282,
- "left_hip": 87.51935335953557,
- "left_knee": 48.775941548709355,
- "left_ankle": 73.29108090434451,
- "right_wrist": 73.56617689328674,
- "right_elbow": 60.91350179498496,
- "right_shoulder": 49.63860158477713,
- "left_shoulder": 51.612081048992216,
- "left_elbow": 69.97581259039976,
- "left_wrist": 78.63573273550595,
- "neck": 35.069318301269,
- "head": 128.294160962772
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 103.61010811015035,
- "PA_MPJPE_mm": 64.7412565467062,
- "PCK3D_50mm": 0.40374815989129204,
- "PCK3D_100mm": 0.8742615974030876,
- "AUC_0_150mm": 0.5757349324167187,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.83100580934249,
- "right_knee": 51.86591702837063,
- "right_hip": 97.28754771608665,
- "left_hip": 86.43155061844294,
- "left_knee": 49.43682781165856,
- "left_ankle": 67.36104222561596,
- "right_wrist": 70.6834969905974,
- "right_elbow": 52.018215099384165,
- "right_shoulder": 51.91121183404787,
- "left_shoulder": 52.207626920398184,
- "left_elbow": 57.06159348685347,
- "left_wrist": 74.7990430196761
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 70.78917567890214,
- "PA_MPJPE_mm": 44.99818054197777,
- "PCK3D_50mm": 0.686570846207354,
- "PCK3D_100mm": 0.936975265703609,
- "AUC_0_150mm": 0.7022936325585352
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 323,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 103.24798635374374,
- "PA_MPJPE_mm": 67.45433962471266,
- "PCK3D_50mm": 0.3966061115874274,
- "PCK3D_100mm": 0.8312964071371952,
- "AUC_0_150mm": 0.5579857326020037,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.53432187190742,
- "right_knee": 46.96661050863571,
- "right_hip": 97.73625379014197,
- "left_hip": 87.26342498171378,
- "left_knee": 46.187861789839076,
- "left_ankle": 67.27451145983726,
- "right_wrist": 70.28111573668816,
- "right_elbow": 59.76681259365406,
- "right_shoulder": 49.581762805279745,
- "left_shoulder": 51.01287522810417,
- "left_elbow": 67.39545235210144,
- "left_wrist": 74.63879167385564,
- "neck": 34.53467583784215,
- "head": 126.18628411637673
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.95798732059849,
- "PA_MPJPE_mm": 62.367617660235446,
- "PCK3D_50mm": 0.43293039670856454,
- "PCK3D_100mm": 0.8880647718265202,
- "AUC_0_150mm": 0.5892243447143068,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.97035739548527,
- "right_knee": 49.003375094909806,
- "right_hip": 96.58791443838062,
- "left_hip": 86.19408688125208,
- "left_knee": 47.08982517279168,
- "left_ankle": 61.70960002531091,
- "right_wrist": 67.53252841894654,
- "right_elbow": 51.230140568357804,
- "right_shoulder": 50.84814436933092,
- "left_shoulder": 50.850504638295725,
- "left_elbow": 54.90892196078321,
- "left_wrist": 71.48601295898085
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 67.91719694117052,
- "PA_MPJPE_mm": 42.390200144467656,
- "PCK3D_50mm": 0.7160226960221299,
- "PCK3D_100mm": 0.946256693142663,
- "AUC_0_150mm": 0.7173840242296852
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 275,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 102.66003015392268,
- "PA_MPJPE_mm": 66.83733095095361,
- "PCK3D_50mm": 0.4027148680783603,
- "PCK3D_100mm": 0.8341475645857935,
- "AUC_0_150mm": 0.5612049770211451,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.37194243880539,
- "right_knee": 48.001798752551586,
- "right_hip": 98.50077991648777,
- "left_hip": 87.1177614821466,
- "left_knee": 46.1150202868564,
- "left_ankle": 67.53746205733097,
- "right_wrist": 67.54469133502666,
- "right_elbow": 58.43316575250908,
- "right_shoulder": 49.11138694735078,
- "left_shoulder": 50.42383073508272,
- "left_elbow": 66.65296945683575,
- "left_wrist": 71.69952737891084,
- "neck": 33.786747057625114,
- "head": 125.4255497158309
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.30681713543167,
- "PA_MPJPE_mm": 61.777415909597735,
- "PCK3D_50mm": 0.4404630468425622,
- "PCK3D_100mm": 0.8920422375721889,
- "AUC_0_150mm": 0.5924995555734138,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.120950146917885,
- "right_knee": 49.982319644178666,
- "right_hip": 97.12308431368596,
- "left_hip": 85.73714004232302,
- "left_knee": 47.04085585738183,
- "left_ankle": 61.828105789743454,
- "right_wrist": 65.17304581667655,
- "right_elbow": 50.09139200134846,
- "right_shoulder": 50.331461407610604,
- "left_shoulder": 50.25289725324927,
- "left_elbow": 54.087093813146794,
- "left_wrist": 68.56064482891021
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 66.41007264372921,
- "PA_MPJPE_mm": 41.272811844231256,
- "PCK3D_50mm": 0.7300378536648495,
- "PCK3D_100mm": 0.9510713072455796,
- "AUC_0_150mm": 0.7241036498468163
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_3dpw_test_gt-joints_224.json b/benchmark/results/gen9w_3dpw_test_gt-joints_224.json
deleted file mode 100644
index 739b2f3..0000000
--- a/benchmark/results/gen9w_3dpw_test_gt-joints_224.json
+++ /dev/null
@@ -1,377 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 481,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.67208981530685,
- "PA_MPJPE_mm": 66.88827906149398,
- "PCK3D_50mm": 0.40043747809588265,
- "PCK3D_100mm": 0.8333152057879238,
- "AUC_0_150mm": 0.5614256099464217,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 64.32566466583239,
- "right_knee": 47.15757278410884,
- "right_hip": 97.91236566919363,
- "left_hip": 86.9490282552268,
- "left_knee": 44.53516581618817,
- "left_ankle": 65.83516249290801,
- "right_wrist": 69.57656761565505,
- "right_elbow": 57.30814205770195,
- "right_shoulder": 50.57434084812882,
- "left_shoulder": 52.783406809770966,
- "left_elbow": 66.62647224986277,
- "left_wrist": 71.49801868409502,
- "neck": 31.509690874000956,
- "head": 129.84430803824233
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.33834875125983,
- "PA_MPJPE_mm": 61.536504952360026,
- "PCK3D_50mm": 0.4448274727650415,
- "PCK3D_100mm": 0.8943217813871734,
- "AUC_0_150mm": 0.5942502847887198,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.237419998496975,
- "right_knee": 49.111783974488894,
- "right_hip": 97.51347173184112,
- "left_hip": 86.22150295326539,
- "left_knee": 45.48550400353603,
- "left_ankle": 60.84145088776921,
- "right_wrist": 65.8537664444401,
- "right_elbow": 48.74105875119605,
- "right_shoulder": 50.52034004161104,
- "left_shoulder": 50.659167441813175,
- "left_elbow": 54.43509093493264,
- "left_wrist": 67.81750226492976
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 65.85016697108857,
- "PA_MPJPE_mm": 40.61170253850853,
- "PCK3D_50mm": 0.7406330944525683,
- "PCK3D_100mm": 0.9546932215065199,
- "AUC_0_150mm": 0.7285012834172203
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 495,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 110.08245156143067,
- "PA_MPJPE_mm": 73.34607561868295,
- "PCK3D_50mm": 0.3462945283011267,
- "PCK3D_100mm": 0.7920307282036408,
- "AUC_0_150mm": 0.5268022581073984,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 74.53270327456663,
- "right_knee": 51.53417283830616,
- "right_hip": 101.08422200146858,
- "left_hip": 88.70957488441523,
- "left_knee": 51.99029641820986,
- "left_ankle": 76.71053799801602,
- "right_wrist": 82.33706511115322,
- "right_elbow": 65.35077504874738,
- "right_shoulder": 50.67533347562955,
- "left_shoulder": 52.59781300162183,
- "left_elbow": 72.7112005121165,
- "left_wrist": 87.35389318766138,
- "neck": 35.09257115905695,
- "head": 136.16489975059199
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 108.17860678554457,
- "PA_MPJPE_mm": 68.20553519197144,
- "PCK3D_50mm": 0.37171841073795225,
- "PCK3D_100mm": 0.8473925875795429,
- "AUC_0_150mm": 0.5563874842748417,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.34191795562127,
- "right_knee": 53.40448625172198,
- "right_hip": 100.06664848897843,
- "left_hip": 87.81030938057812,
- "left_knee": 52.904248682702836,
- "left_ankle": 71.50730136167641,
- "right_wrist": 77.91771909574926,
- "right_elbow": 55.56947225171372,
- "right_shoulder": 53.38271574607082,
- "left_shoulder": 53.35183840781771,
- "left_elbow": 59.424000566188894,
- "left_wrist": 82.78576411483779
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 75.52762871702767,
- "PA_MPJPE_mm": 49.00321540548335,
- "PCK3D_50mm": 0.6408570703469613,
- "PCK3D_100mm": 0.9189718861912416,
- "AUC_0_150mm": 0.6791551810149113
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 476,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 109.17729708092469,
- "PA_MPJPE_mm": 71.88108129836831,
- "PCK3D_50mm": 0.3528244729919715,
- "PCK3D_100mm": 0.8013039747664568,
- "AUC_0_150mm": 0.5334530531009734,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 72.53014891405228,
- "right_knee": 50.72599111698719,
- "right_hip": 99.72732286716383,
- "left_hip": 88.77335959727537,
- "left_knee": 50.392123503433616,
- "left_ankle": 74.81961966173637,
- "right_wrist": 78.03473435723386,
- "right_elbow": 63.81378720141611,
- "right_shoulder": 50.79285225405011,
- "left_shoulder": 52.91078692586644,
- "left_elbow": 72.38474398462559,
- "left_wrist": 84.05892534172334,
- "neck": 36.02081568017122,
- "head": 131.349926771421
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 107.61912784552689,
- "PA_MPJPE_mm": 66.9300851175815,
- "PCK3D_50mm": 0.38219176794593424,
- "PCK3D_100mm": 0.854963859045578,
- "AUC_0_150mm": 0.5624557504883934,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.89315538217726,
- "right_knee": 52.6427550154613,
- "right_hip": 98.6455275837267,
- "left_hip": 87.94037124126588,
- "left_knee": 51.10963191066116,
- "left_ankle": 69.28481687879618,
- "right_wrist": 74.36796630739957,
- "right_elbow": 54.18830930398644,
- "right_shoulder": 53.42439908555606,
- "left_shoulder": 53.725874627689166,
- "left_elbow": 59.13414977966033,
- "left_wrist": 79.80406429459784
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 74.23364200605525,
- "PA_MPJPE_mm": 47.13912725530142,
- "PCK3D_50mm": 0.66274708851479,
- "PCK3D_100mm": 0.9257415173158342,
- "AUC_0_150mm": 0.6893667717024731
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 343,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.94333129884087,
- "PA_MPJPE_mm": 69.80009687306462,
- "PCK3D_50mm": 0.374142466393545,
- "PCK3D_100mm": 0.8156025797511289,
- "AUC_0_150mm": 0.5452827383794963,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.33962092017975,
- "right_knee": 48.102463048393815,
- "right_hip": 99.78735921884589,
- "left_hip": 86.67104693861037,
- "left_knee": 48.6737870774193,
- "left_ankle": 70.4991540380317,
- "right_wrist": 73.92964256542365,
- "right_elbow": 62.583923146319655,
- "right_shoulder": 50.21521713805111,
- "left_shoulder": 51.705707152473245,
- "left_elbow": 70.74267979844184,
- "left_wrist": 80.03063982765745,
- "neck": 34.74768266819348,
- "head": 130.17243268486337
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 103.61809157830562,
- "PA_MPJPE_mm": 64.6879117269615,
- "PCK3D_50mm": 0.4100071435956725,
- "PCK3D_100mm": 0.8718805515607817,
- "AUC_0_150mm": 0.5759799172786175,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.23047493676964,
- "right_knee": 50.110604071029506,
- "right_hip": 98.6536585089377,
- "left_hip": 86.12081956221932,
- "left_knee": 49.65131535728207,
- "left_ankle": 65.30921147677716,
- "right_wrist": 71.08967507607717,
- "right_elbow": 53.35416229960335,
- "right_shoulder": 50.706850196702746,
- "left_shoulder": 51.68758363926956,
- "left_elbow": 57.98994635007556,
- "left_wrist": 76.35063924879425
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 70.83988184569084,
- "PA_MPJPE_mm": 44.83733926397207,
- "PCK3D_50mm": 0.6868244971620321,
- "PCK3D_100mm": 0.9375123368017371,
- "AUC_0_150mm": 0.7031440310800989
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 416,
- "source": "raw",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 104.2083972335382,
- "PA_MPJPE_mm": 67.72726086734916,
- "PCK3D_50mm": 0.3926386052263728,
- "PCK3D_100mm": 0.8284590377898897,
- "AUC_0_150mm": 0.5561637487815886,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.17089839188434,
- "right_knee": 47.67697809499401,
- "right_hip": 99.05250281145562,
- "left_hip": 87.19124965834231,
- "left_knee": 46.535243148214725,
- "left_ankle": 68.03772275956916,
- "right_wrist": 70.64004788571204,
- "right_elbow": 60.20402818913845,
- "right_shoulder": 49.05190210072777,
- "left_shoulder": 50.26149801347135,
- "left_elbow": 68.86562626674183,
- "left_wrist": 75.30362965565924,
- "neck": 33.259720855027595,
- "head": 126.93060431194986
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 102.31697966493746,
- "PA_MPJPE_mm": 62.73635026553071,
- "PCK3D_50mm": 0.43106900149451544,
- "PCK3D_100mm": 0.8831011664739776,
- "AUC_0_150mm": 0.5864880676785953,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.561900568455364,
- "right_knee": 49.6804044490605,
- "right_hip": 98.53786514126264,
- "left_hip": 86.61622210421294,
- "left_knee": 47.54954155509747,
- "left_ankle": 62.60730799929371,
- "right_wrist": 67.46647758328741,
- "right_elbow": 51.52987761172092,
- "right_shoulder": 49.996811442166205,
- "left_shoulder": 49.447796806553335,
- "left_elbow": 56.191848293468006,
- "left_wrist": 71.65014963178997
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 67.61875479106949,
- "PA_MPJPE_mm": 42.30304879586423,
- "PCK3D_50mm": 0.7170552003899436,
- "PCK3D_100mm": 0.9470917374648024,
- "AUC_0_150mm": 0.7177955059540865
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_3dpw_test_gt-joints_288.json b/benchmark/results/gen9w_3dpw_test_gt-joints_288.json
deleted file mode 100644
index 30cd722..0000000
--- a/benchmark/results/gen9w_3dpw_test_gt-joints_288.json
+++ /dev/null
@@ -1,302 +0,0 @@
-{
- "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 361,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 107.86686939334284,
- "PA_MPJPE_mm": 71.02444442447438,
- "PCK3D_50mm": 0.36394874335826877,
- "PCK3D_100mm": 0.8072578663476219,
- "AUC_0_150mm": 0.5388794266133938,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.98449443373526,
- "right_knee": 49.79480702002293,
- "right_hip": 98.50476434684485,
- "left_hip": 87.08901899100073,
- "left_knee": 48.90025229273755,
- "left_ankle": 74.56946509553796,
- "right_wrist": 77.81195966315201,
- "right_elbow": 61.72384828257411,
- "right_shoulder": 50.432632190756806,
- "left_shoulder": 52.79037516084865,
- "left_elbow": 71.63535238427215,
- "left_wrist": 82.97753230809283,
- "neck": 35.32927891488839,
- "head": 131.7984408581773
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 105.99863313932453,
- "PA_MPJPE_mm": 65.91006545190102,
- "PCK3D_50mm": 0.3948622507966049,
- "PCK3D_100mm": 0.863376382896728,
- "AUC_0_150mm": 0.5691014775660472,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.18787081694754,
- "right_knee": 51.745602400439864,
- "right_hip": 97.29193581845561,
- "left_hip": 85.86503640390676,
- "left_knee": 49.51098363957994,
- "left_ankle": 68.59133303551225,
- "right_wrist": 74.86639513298728,
- "right_elbow": 52.71541756948042,
- "right_shoulder": 52.674342922005565,
- "left_shoulder": 52.92640833930272,
- "left_elbow": 58.34794487136244,
- "left_wrist": 79.19751447283177
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 73.40109543207136,
- "PA_MPJPE_mm": 46.430969238333354,
- "PCK3D_50mm": 0.6716255574220213,
- "PCK3D_100mm": 0.9289118235907847,
- "AUC_0_150mm": 0.6937904125686394
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 317,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 105.85146295192882,
- "PA_MPJPE_mm": 69.96007263543709,
- "PCK3D_50mm": 0.3713185976530871,
- "PCK3D_100mm": 0.8148251094702326,
- "AUC_0_150mm": 0.544144731362122,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.71689366458462,
- "right_knee": 49.79033122306574,
- "right_hip": 98.92418558076439,
- "left_hip": 87.4937112427745,
- "left_knee": 48.72513730901825,
- "left_ankle": 73.61793105662896,
- "right_wrist": 74.25844777646894,
- "right_elbow": 61.21305473513757,
- "right_shoulder": 49.61796407122349,
- "left_shoulder": 51.878782956379496,
- "left_elbow": 70.29629884140675,
- "left_wrist": 79.53774296442933,
- "neck": 35.01854604882444,
- "head": 130.35198942541274
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 103.8359613082194,
- "PA_MPJPE_mm": 64.8596783504169,
- "PCK3D_50mm": 0.4015100245326115,
- "PCK3D_100mm": 0.8719580971717,
- "AUC_0_150mm": 0.5745321718016567,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.08229445092073,
- "right_knee": 51.69305799355376,
- "right_hip": 97.6787452061199,
- "left_hip": 86.34265078889331,
- "left_knee": 49.29910484689149,
- "left_ankle": 67.6425397970894,
- "right_wrist": 70.98073079231627,
- "right_elbow": 52.329595872737684,
- "right_shoulder": 52.06529235886285,
- "left_shoulder": 52.451143270312315,
- "left_elbow": 57.33224371248775,
- "left_wrist": 75.41874111481738
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 71.27543939879975,
- "PA_MPJPE_mm": 45.336267020120985,
- "PCK3D_50mm": 0.6804053319153565,
- "PCK3D_100mm": 0.9358043191898195,
- "AUC_0_150mm": 0.6997670447981676
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 323,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 103.8226593151782,
- "PA_MPJPE_mm": 67.69325189028706,
- "PCK3D_50mm": 0.39113603312909634,
- "PCK3D_100mm": 0.8293130465958484,
- "AUC_0_150mm": 0.5562350894441679,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.87124608729032,
- "right_knee": 46.996741954419804,
- "right_hip": 98.07229669260956,
- "left_hip": 87.19126803366103,
- "left_knee": 46.356163127767346,
- "left_ankle": 67.80965198902578,
- "right_wrist": 70.64320903724494,
- "right_elbow": 59.53713756884794,
- "right_shoulder": 49.52687598353362,
- "left_shoulder": 50.97116390391279,
- "left_elbow": 67.93251277155284,
- "left_wrist": 74.75930274459418,
- "neck": 34.17311603713364,
- "head": 127.86484053242489
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 101.49229530263892,
- "PA_MPJPE_mm": 62.452216235674165,
- "PCK3D_50mm": 0.4315883221009691,
- "PCK3D_100mm": 0.886419178674487,
- "AUC_0_150mm": 0.5880983329891915,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 61.55136347422553,
- "right_knee": 48.99817403603347,
- "right_hip": 96.99768623546959,
- "left_hip": 86.20740555152773,
- "left_knee": 47.14285369263485,
- "left_ankle": 62.253163561212425,
- "right_wrist": 67.5965704390305,
- "right_elbow": 50.9371393509537,
- "right_shoulder": 50.48827209421642,
- "left_shoulder": 50.524687718080145,
- "left_elbow": 55.214974491062705,
- "left_wrist": 71.51430418364286
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 68.43797895882646,
- "PA_MPJPE_mm": 42.57006241192356,
- "PCK3D_50mm": 0.7119130200087818,
- "PCK3D_100mm": 0.9466022937387458,
- "AUC_0_150mm": 0.7158322732942533
- }
- ]
- },
- "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth|h36m": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 275,
- "source": "ema",
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 102.88165768492182,
- "PA_MPJPE_mm": 67.01856548558627,
- "PCK3D_50mm": 0.4001434090259063,
- "PCK3D_100mm": 0.8320180792052884,
- "AUC_0_150mm": 0.5598809351630328,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.22828314984679,
- "right_knee": 47.971143735533616,
- "right_hip": 98.56710780400871,
- "left_hip": 87.12965945716073,
- "left_knee": 46.10154535822198,
- "left_ankle": 67.14635903802294,
- "right_wrist": 67.58634175311445,
- "right_elbow": 59.08615523930151,
- "right_shoulder": 49.362315600017645,
- "left_shoulder": 50.636464613958644,
- "left_elbow": 66.7936267648197,
- "left_wrist": 72.12528571000344,
- "neck": 33.63042837813828,
- "head": 126.89520019605943
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 100.39415381180089,
- "PA_MPJPE_mm": 61.840858211065004,
- "PCK3D_50mm": 0.4389058079312711,
- "PCK3D_100mm": 0.8905196965851733,
- "AUC_0_150mm": 0.5916162354888133,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.866974644744644,
- "right_knee": 49.93429863051092,
- "right_hip": 97.28169057222394,
- "left_hip": 85.80033874506645,
- "left_knee": 47.012985802245986,
- "left_ankle": 61.61046664373474,
- "right_wrist": 65.10456238452748,
- "right_elbow": 50.655414753288866,
- "right_shoulder": 50.37174446406672,
- "left_shoulder": 50.18025430833187,
- "left_elbow": 54.20613708316662,
- "left_wrist": 69.06543050087178
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 66.78473949546623,
- "PA_MPJPE_mm": 41.36725055989475,
- "PCK3D_50mm": 0.7275893184445761,
- "PCK3D_100mm": 0.9505017301735007,
- "AUC_0_150mm": 0.7232251281305588
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_annotated_224.json b/benchmark/results/gen9w_emdb1_annotated_224.json
deleted file mode 100644
index 3b0a102..0000000
--- a/benchmark/results/gen9w_emdb1_annotated_224.json
+++ /dev/null
@@ -1,867 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 86.56307577082555,
- "PA_MPJPE_mm": 62.870025736990414,
- "PCK3D_50mm": 0.5740969312257119,
- "PCK3D_100mm": 0.850348850073988,
- "AUC_0_150mm": 0.6204626292448314,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 41.92819348367251,
- "left_hip": 42.983309591841284,
- "right_hip": 41.99028748432828,
- "spine1": 50.546013737542765,
- "left_knee": 55.87615404726698,
- "right_knee": 55.503917699781574,
- "spine2": 50.49607895919742,
- "left_ankle": 66.1181759111917,
- "right_ankle": 66.98668454711286,
- "spine3": 49.78704163506046,
- "left_foot": 73.13540954015934,
- "right_foot": 75.7911614026987,
- "neck": 48.6956588109759,
- "left_collar": 42.73892796587593,
- "right_collar": 42.10077260781202,
- "head": 55.32774526303255,
- "left_shoulder": 47.76003947941104,
- "right_shoulder": 47.36225766052754,
- "left_elbow": 60.362470277701114,
- "right_elbow": 56.31569359814934,
- "left_wrist": 100.02892705884527,
- "right_wrist": 97.56416452991154,
- "left_hand": 118.92762398132496,
- "right_hand": 120.55390841434752
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 97.52043161985303,
- "PA_MPJPE_mm": 66.60282504447187,
- "PCK3D_50mm": 0.4932373563456831,
- "PCK3D_100mm": 0.8317903521197717,
- "AUC_0_150mm": 0.5863527132309093,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.54325731831655,
- "right_knee": 52.5715564247917,
- "right_hip": 50.65727821785841,
- "left_hip": 48.61720059795035,
- "left_knee": 54.35848136667145,
- "left_ankle": 68.57272375227623,
- "right_wrist": 99.14361640124834,
- "right_elbow": 51.80272340308523,
- "right_shoulder": 57.842049573846666,
- "left_shoulder": 49.321726616475765,
- "left_elbow": 57.59989885904841,
- "left_wrist": 104.93261432735564,
- "neck": 59.08336480050076,
- "head": 110.39305896318065
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 92.62130444081637,
- "PA_MPJPE_mm": 62.52354452474751,
- "PCK3D_50mm": 0.5455890691338561,
- "PCK3D_100mm": 0.8560656349831971,
- "AUC_0_150mm": 0.6123240581750631,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.09113869530734,
- "right_knee": 53.47881404589802,
- "right_hip": 50.16151816441917,
- "left_hip": 48.88523996220425,
- "left_knee": 55.289079967341344,
- "left_ankle": 66.70908987983816,
- "right_wrist": 94.9783202880758,
- "right_elbow": 49.31230615843084,
- "right_shoulder": 57.06256345327593,
- "left_shoulder": 53.488600402271814,
- "left_elbow": 54.9418653064444,
- "left_wrist": 99.88399797346307
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 78.73205914997531
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 69.90917751476255
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 57.19897705885768
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.67358303674525
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 57.442956505064636
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 80.65953393773327
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 51.99371657415196
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 78.22017006614858
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 52.73912827718323
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 60.6432327921372
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 59.01714813491954
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 60.25282361405862
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.7914121366103
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 65.03925056034467
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 71.88318204701882
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 54.47972068910211
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 59.022351604762875
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 495,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 98.80407799930023,
- "PA_MPJPE_mm": 68.77680118628923,
- "PCK3D_50mm": 0.5144674245253011,
- "PCK3D_100mm": 0.833091316433639,
- "AUC_0_150mm": 0.5920096949516389,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 44.597327148287775,
- "left_hip": 46.03672965770137,
- "right_hip": 45.69516043856871,
- "spine1": 50.73676694844938,
- "left_knee": 70.7208462602123,
- "right_knee": 67.98737214959377,
- "spine2": 47.720310264972184,
- "left_ankle": 86.4461359229997,
- "right_ankle": 88.12745403714572,
- "spine3": 45.00518401049988,
- "left_foot": 96.18908913413586,
- "right_foot": 97.46984442213075,
- "neck": 41.5508337283668,
- "left_collar": 36.13130994331438,
- "right_collar": 36.60261241281916,
- "head": 50.03312689286353,
- "left_shoulder": 44.678787339011194,
- "right_shoulder": 44.934865772396705,
- "left_elbow": 64.62161747957974,
- "right_elbow": 61.620767125291415,
- "left_wrist": 111.09782138194512,
- "right_wrist": 107.32201694196984,
- "left_hand": 133.01162549906326,
- "right_hand": 132.30562355962203
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 109.44193486631913,
- "PA_MPJPE_mm": 73.59853738842263,
- "PCK3D_50mm": 0.42148280297058455,
- "PCK3D_100mm": 0.8016459124827378,
- "AUC_0_150mm": 0.5508344468659943,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 88.82964268081025,
- "right_knee": 64.97776156086667,
- "right_hip": 56.502544912534695,
- "left_hip": 53.814022236423554,
- "left_knee": 69.07332819381251,
- "left_ankle": 89.14289248830534,
- "right_wrist": 109.28613733826555,
- "right_elbow": 57.16852423265606,
- "right_shoulder": 58.38112119924028,
- "left_shoulder": 48.03170584069258,
- "left_elbow": 62.42753020377072,
- "left_wrist": 115.27399287998985,
- "neck": 49.85193275257287,
- "head": 107.61838691797574
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 105.60340986764425,
- "PA_MPJPE_mm": 70.7209696889217,
- "PCK3D_50mm": 0.4630958801808903,
- "PCK3D_100mm": 0.8245515772587092,
- "AUC_0_150mm": 0.5732860630832105,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 86.82998706646737,
- "right_knee": 65.4321166651344,
- "right_hip": 54.335372688660414,
- "left_hip": 52.365875269524665,
- "left_knee": 69.45872750438012,
- "left_ankle": 86.73598366357163,
- "right_wrist": 105.69350170385752,
- "right_elbow": 53.20562588036006,
- "right_shoulder": 54.454752243716364,
- "left_shoulder": 50.699530365941555,
- "left_elbow": 58.556652935529264,
- "left_wrist": 110.88351027991702
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 55.468295106026275
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 63.969955766784246
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 63.053629941791556
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 63.52127046028012
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 58.52762166469394
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 105.47790216889464
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 66.11726668585023
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 72.64219864166756
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 68.50574329491153
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 56.93464100309395
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 64.02906148647689
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 70.65531592232193
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 57.67215098638569
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 60.04337343789902
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 92.26085947637783
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 77.90838942047814
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 74.97796130270913
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 476,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 94.40980071470597,
- "PA_MPJPE_mm": 66.10490400484157,
- "PCK3D_50mm": 0.5347812858703619,
- "PCK3D_100mm": 0.8447306006167974,
- "AUC_0_150mm": 0.6039773191130002,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.16946609362433,
- "left_hip": 44.480258787041926,
- "right_hip": 43.69708624965789,
- "spine1": 49.35491966519267,
- "left_knee": 67.58892171147183,
- "right_knee": 66.2605819157014,
- "spine2": 46.86372141563387,
- "left_ankle": 82.30947069345058,
- "right_ankle": 84.95717316132449,
- "spine3": 44.405647819803676,
- "left_foot": 91.39220935896788,
- "right_foot": 93.44060623776072,
- "neck": 39.24452171606876,
- "left_collar": 34.83114179885335,
- "right_collar": 35.212826583174696,
- "head": 47.257260493127,
- "left_shoulder": 43.51991815396369,
- "right_shoulder": 44.46770756266806,
- "left_elbow": 62.00263496174787,
- "right_elbow": 59.44929353005582,
- "left_wrist": 107.16075353569664,
- "right_wrist": 102.44438518599966,
- "left_hand": 127.88020120471253,
- "right_hand": 125.1269882805005
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 104.20810219483367,
- "PA_MPJPE_mm": 70.77551394922503,
- "PCK3D_50mm": 0.4460440609052815,
- "PCK3D_100mm": 0.8172426668879392,
- "AUC_0_150mm": 0.5647078943650244,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 85.39788258771607,
- "right_knee": 62.542699440193445,
- "right_hip": 54.44723066599899,
- "left_hip": 51.510254707847814,
- "left_knee": 65.4455951146437,
- "left_ankle": 84.95407548738243,
- "right_wrist": 104.45983596981596,
- "right_elbow": 53.95660163589094,
- "right_shoulder": 57.55511622965743,
- "left_shoulder": 47.399404832855346,
- "left_elbow": 58.9964205493001,
- "left_wrist": 111.77475314056956,
- "neck": 47.91337760993337,
- "head": 104.5039473173453
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 100.48927702498293,
- "PA_MPJPE_mm": 67.96723620612089,
- "PCK3D_50mm": 0.4885214841859243,
- "PCK3D_100mm": 0.8377138392178014,
- "AUC_0_150mm": 0.5863392501446525,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 83.1013522601928,
- "right_knee": 62.9112623350598,
- "right_hip": 52.47553864339497,
- "left_hip": 50.231014288906394,
- "left_knee": 65.78250562777441,
- "left_ankle": 82.4043818678247,
- "right_wrist": 101.31522206420419,
- "right_elbow": 50.79505723367452,
- "right_shoulder": 53.864048206038156,
- "left_shoulder": 49.31177190850422,
- "left_elbow": 55.75776718797795,
- "left_wrist": 107.65691284989853
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 54.31993871703611
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 63.539171948765386
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 59.79112903238372
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 61.0811733610311
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 57.20895429080521
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 95.34013215691209
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 66.69570243012592
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 71.75341789855669
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 65.91662491040887
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 53.37964479027007
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 60.019662863194434
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 66.73957470761924
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 55.83094148140724
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 56.62191134785518
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 85.78140924967424
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 75.52708590157857
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 76.73403859954931
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 343,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 88.16388309849815,
- "PA_MPJPE_mm": 61.24348709443844,
- "PCK3D_50mm": 0.5705634844901741,
- "PCK3D_100mm": 0.8687784369857141,
- "AUC_0_150mm": 0.6257650299186421,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.7523243336816,
- "left_hip": 42.50242972927826,
- "right_hip": 41.77748775669497,
- "spine1": 47.221966615750254,
- "left_knee": 62.75083309496669,
- "right_knee": 61.80243642547612,
- "spine2": 45.70696716241673,
- "left_ankle": 73.70732857170623,
- "right_ankle": 75.91716463551685,
- "spine3": 43.57874965270277,
- "left_foot": 81.09754505938453,
- "right_foot": 83.21289620053423,
- "neck": 37.994381514329504,
- "left_collar": 32.920875120793454,
- "right_collar": 33.42149358301194,
- "head": 45.47892453146991,
- "left_shoulder": 43.50638973030334,
- "right_shoulder": 42.694191920863226,
- "left_elbow": 58.84845684033201,
- "right_elbow": 55.21107349848616,
- "left_wrist": 96.6723320579724,
- "right_wrist": 93.45673399686355,
- "left_hand": 114.882434482671,
- "right_hand": 114.72827375131565
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 99.74171679986522,
- "PA_MPJPE_mm": 66.9617547305426,
- "PCK3D_50mm": 0.47292868107704433,
- "PCK3D_100mm": 0.8368489992354241,
- "AUC_0_150mm": 0.5815515058167224,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 77.01680589701148,
- "right_knee": 58.464628865751095,
- "right_hip": 53.68254967532693,
- "left_hip": 50.31250156339712,
- "left_knee": 61.294358552742686,
- "left_ankle": 76.17756224903725,
- "right_wrist": 95.95600671656496,
- "right_elbow": 51.448020595573276,
- "right_shoulder": 56.983880024212205,
- "left_shoulder": 45.7793448220186,
- "left_elbow": 56.38994157838487,
- "left_wrist": 101.77197510518059,
- "neck": 46.28134754014817,
- "head": 105.9056430422471
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 94.80239996275485,
- "PA_MPJPE_mm": 63.29910761033962,
- "PCK3D_50mm": 0.5233822898947572,
- "PCK3D_100mm": 0.8622474380782476,
- "AUC_0_150mm": 0.6078678244219811,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 74.56215561901719,
- "right_knee": 58.79559009707537,
- "right_hip": 51.127624380830504,
- "left_hip": 48.711879607863956,
- "left_knee": 61.84980671101634,
- "left_ankle": 73.62982555661438,
- "right_wrist": 93.44472015558978,
- "right_elbow": 47.8304701070473,
- "right_shoulder": 51.26776112511871,
- "left_shoulder": 47.798941144201144,
- "left_elbow": 52.97169925560957,
- "left_wrist": 97.59881756409123
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 47.62589456467713
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 58.47777095803588
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 57.13846164629266
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 57.893838276522445
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 53.120265290646394
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 86.298826978497
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 60.16626076911829
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 71.59389896435307
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 55.3113747413293
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 49.574397333475126
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 55.571200431372276
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 63.31235305504479
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 51.49037720945502
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 52.878082047760735
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 88.86740864405617
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 68.90026643656105
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 64.47793071216343
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 416,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 83.833161323396,
- "PA_MPJPE_mm": 56.64331055732487,
- "PCK3D_50mm": 0.6055971594130745,
- "PCK3D_100mm": 0.8900223347024575,
- "AUC_0_150mm": 0.6458173011078353,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.094926757828304,
- "left_hip": 41.71466415881362,
- "right_hip": 40.80633006258003,
- "spine1": 46.30182445472659,
- "left_knee": 57.847863377915246,
- "right_knee": 56.40811936912709,
- "spine2": 43.88153798230367,
- "left_ankle": 63.70265762207686,
- "right_ankle": 64.32107302697987,
- "spine3": 41.89972880723249,
- "left_foot": 71.27176462056276,
- "right_foot": 72.3868823047739,
- "neck": 36.7217290100631,
- "left_collar": 32.0469643717298,
- "right_collar": 32.036128533715704,
- "head": 42.31847295418952,
- "left_shoulder": 40.48875929811232,
- "right_shoulder": 42.54869320019132,
- "left_elbow": 55.30952820151958,
- "right_elbow": 53.38066498751529,
- "left_wrist": 87.18710162930986,
- "right_wrist": 86.05383863417542,
- "left_hand": 103.76059883838933,
- "right_hand": 106.94960117196105
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 94.14949680349432,
- "PA_MPJPE_mm": 62.11866942697986,
- "PCK3D_50mm": 0.5099691206192472,
- "PCK3D_100mm": 0.8629779339856923,
- "AUC_0_150mm": 0.6042511296087012,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.07533718355917,
- "right_knee": 53.96464127174541,
- "right_hip": 50.19714058202571,
- "left_hip": 47.724228732132985,
- "left_knee": 56.96592783089215,
- "left_ankle": 67.29375370834457,
- "right_wrist": 87.62529856468747,
- "right_elbow": 48.150503506627494,
- "right_shoulder": 54.82576617555189,
- "left_shoulder": 43.83033198597072,
- "left_elbow": 52.65227318347304,
- "left_wrist": 91.67665982127255,
- "neck": 46.945615272979495,
- "head": 102.73389415845539
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 88.97795500804025,
- "PA_MPJPE_mm": 57.91186421129567,
- "PCK3D_50mm": 0.5669245875340553,
- "PCK3D_100mm": 0.8903352279799195,
- "AUC_0_150mm": 0.6329450133142754,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.00634022524534,
- "right_knee": 54.63082012137297,
- "right_hip": 48.037460312767706,
- "left_hip": 46.108128956531345,
- "left_knee": 57.656606623461684,
- "left_ankle": 64.4650407975658,
- "right_wrist": 84.52487163503653,
- "right_elbow": 44.90462616084134,
- "right_shoulder": 49.70034743621156,
- "left_shoulder": 44.47515403589896,
- "left_elbow": 49.78128125299899,
- "left_wrist": 87.65169297761584
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 48.641680358501695
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 52.54774106841503
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 55.19341685625968
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 53.02278342611785
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 50.110317188254264
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 76.66442414306871
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 51.43014940007747
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 65.20459619835408
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 51.1202477091565
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 47.035468119161905
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 52.087928039224025
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 57.38482064782934
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.52660094731107
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.2789658622323
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 81.24443024238474
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.84385945678372
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 60.513944526791995
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_annotated_288.json b/benchmark/results/gen9w_emdb1_annotated_288.json
deleted file mode 100644
index e2b7a24..0000000
--- a/benchmark/results/gen9w_emdb1_annotated_288.json
+++ /dev/null
@@ -1,694 +0,0 @@
-{
- "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 361,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 91.89670287617336,
- "PA_MPJPE_mm": 63.757212600643335,
- "PCK3D_50mm": 0.5491052289479871,
- "PCK3D_100mm": 0.857063090348366,
- "AUC_0_150mm": 0.6137960116507158,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.38792292916049,
- "left_hip": 43.66349015807839,
- "right_hip": 42.73738264175591,
- "spine1": 48.531454138331505,
- "left_knee": 65.74860860721955,
- "right_knee": 64.68150237307081,
- "spine2": 46.78077636974182,
- "left_ankle": 80.30480669806117,
- "right_ankle": 82.31382311602702,
- "spine3": 44.3247840429019,
- "left_foot": 88.91519791435911,
- "right_foot": 90.87581570590748,
- "neck": 38.35601938837669,
- "left_collar": 33.9500945786115,
- "right_collar": 34.17133698807522,
- "head": 45.75652538312053,
- "left_shoulder": 42.55598676069015,
- "right_shoulder": 43.75684567136546,
- "left_elbow": 59.63341984308999,
- "right_elbow": 56.2439555488187,
- "left_wrist": 100.27703965352704,
- "right_wrist": 96.06055535191769,
- "left_hand": 119.81429960209256,
- "right_hand": 118.33145895113807
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 101.83601243931935,
- "PA_MPJPE_mm": 68.84998179523183,
- "PCK3D_50mm": 0.4536098055369515,
- "PCK3D_100mm": 0.8280148884845395,
- "AUC_0_150mm": 0.5723822359149511,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 82.78691686866186,
- "right_knee": 61.527320677439235,
- "right_hip": 53.71209774889444,
- "left_hip": 51.45789367185153,
- "left_knee": 64.68808537293596,
- "left_ankle": 82.98007467497027,
- "right_wrist": 97.84534054214987,
- "right_elbow": 51.378010052186674,
- "right_shoulder": 57.55341284399523,
- "left_shoulder": 46.72264111790647,
- "left_elbow": 56.773650587106175,
- "left_wrist": 104.87429109450976,
- "neck": 47.525925309898916,
- "head": 104.07408457073934
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 98.12515949644451,
- "PA_MPJPE_mm": 65.70465310435183,
- "PCK3D_50mm": 0.5000311164585322,
- "PCK3D_100mm": 0.8501604226306545,
- "AUC_0_150mm": 0.595784043301619,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 80.77610506305012,
- "right_knee": 62.038632888544136,
- "right_hip": 51.41931959319529,
- "left_hip": 49.85185091016465,
- "left_knee": 65.15842954454202,
- "left_ankle": 80.47443718780067,
- "right_wrist": 95.07049968169703,
- "right_elbow": 48.017375580707956,
- "right_shoulder": 52.81929579637555,
- "left_shoulder": 47.99921347168562,
- "left_elbow": 53.8107219153025,
- "left_wrist": 101.01995561915643
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 49.86487771847274
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 59.82460905968223
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.91684163690936
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 61.60744176954229
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 56.10538721383652
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 90.73466417557056
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 63.86663662303396
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 71.2300768474913
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 63.1605232390724
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 51.529787420907724
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 57.978345246288015
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 64.66384933619422
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 52.88975820051211
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 57.73803589176279
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 83.99605217779109
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 74.4723571848433
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 69.01036966908056
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 317,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 90.56817439230251,
- "PA_MPJPE_mm": 62.68335661389248,
- "PCK3D_50mm": 0.5541668395358808,
- "PCK3D_100mm": 0.8607192742258917,
- "AUC_0_150mm": 0.6169875676922383,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.58311297904817,
- "left_hip": 43.65459003175729,
- "right_hip": 43.10356310264368,
- "spine1": 48.907353883796496,
- "left_knee": 65.44272582433068,
- "right_knee": 62.545603537100966,
- "spine2": 46.41320985148489,
- "left_ankle": 78.17353797138134,
- "right_ankle": 78.09233811421754,
- "spine3": 43.79024655087795,
- "left_foot": 86.49873779614148,
- "right_foot": 86.69366301009754,
- "neck": 38.75550606724046,
- "left_collar": 33.927210450607,
- "right_collar": 33.913046762243134,
- "head": 45.31615844992675,
- "left_shoulder": 42.44111784062704,
- "right_shoulder": 41.702042542288645,
- "left_elbow": 59.143672391761335,
- "right_elbow": 56.30132649961313,
- "left_wrist": 97.54735955124492,
- "right_wrist": 95.06439422498168,
- "left_hand": 117.00163378226522,
- "right_hand": 117.38840751774035
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 100.63589112067115,
- "PA_MPJPE_mm": 67.69225742341102,
- "PCK3D_50mm": 0.4626454324002347,
- "PCK3D_100mm": 0.83160661684082,
- "AUC_0_150mm": 0.5763736506402727,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 79.43352460999523,
- "right_knee": 59.16377942172885,
- "right_hip": 54.03832608703559,
- "left_hip": 51.185603358523345,
- "left_knee": 64.04684640754797,
- "left_ankle": 80.87049837508683,
- "right_wrist": 96.36630345890325,
- "right_elbow": 51.547775230537184,
- "right_shoulder": 55.82942069577965,
- "left_shoulder": 45.66550568600926,
- "left_elbow": 55.64845488484714,
- "left_wrist": 101.59293376169694,
- "neck": 47.79883269051294,
- "head": 104.50379925955035
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 96.92154434706322,
- "PA_MPJPE_mm": 64.17069524276937,
- "PCK3D_50mm": 0.5120144103776847,
- "PCK3D_100mm": 0.855567771646683,
- "AUC_0_150mm": 0.6016561316821759,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 77.39525283451466,
- "right_knee": 59.72622946206458,
- "right_hip": 51.81347333231742,
- "left_hip": 49.57019822114406,
- "left_knee": 64.44786394040139,
- "left_ankle": 78.5340091612592,
- "right_wrist": 93.43798272745066,
- "right_elbow": 47.54291220230494,
- "right_shoulder": 50.45029967585119,
- "left_shoulder": 46.76845005479937,
- "left_elbow": 52.48737271548853,
- "left_wrist": 97.87429858563647
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 50.42311309532114
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 57.78255777247055
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 57.50981692132104
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 58.99244476010798
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 54.62357002990997
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 85.96403719244915
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 60.59423052572172
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 67.81865718628754
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 61.915968032102484
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 50.445311908043266
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 57.58841853078742
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 62.50763054492119
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 55.76945534683384
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 55.11821617186037
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 88.34917948004514
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 73.02417497568214
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 68.93446706423859
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 323,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 84.15185423856586,
- "PA_MPJPE_mm": 57.49852583149515,
- "PCK3D_50mm": 0.6033066423266813,
- "PCK3D_100mm": 0.8858492718748704,
- "AUC_0_150mm": 0.643123719931352,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.97356671310843,
- "left_hip": 41.41444021677189,
- "right_hip": 40.82028363252888,
- "spine1": 46.018020542289605,
- "left_knee": 60.25945659541424,
- "right_knee": 58.722384077181175,
- "spine2": 43.56464832026149,
- "left_ankle": 69.7371009466064,
- "right_ankle": 70.4901442735581,
- "spine3": 41.181463615553234,
- "left_foot": 77.48434427887581,
- "right_foot": 78.4322564095153,
- "neck": 36.395107597279775,
- "left_collar": 31.626174236835315,
- "right_collar": 31.33464804904905,
- "head": 42.802056000577245,
- "left_shoulder": 41.040333404700014,
- "right_shoulder": 41.55465279404201,
- "left_elbow": 54.48255262301644,
- "right_elbow": 52.913770063150565,
- "left_wrist": 86.37621035839,
- "right_wrist": 84.7468066112279,
- "left_hand": 103.59387436093682,
- "right_hand": 105.00032423501271
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 94.87807283147559,
- "PA_MPJPE_mm": 63.236938986902594,
- "PCK3D_50mm": 0.5059536157324814,
- "PCK3D_100mm": 0.8546120518489103,
- "AUC_0_150mm": 0.5995583279210133,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 72.89573306303029,
- "right_knee": 56.23555230793994,
- "right_hip": 50.76751598087448,
- "left_hip": 47.642777542497114,
- "left_knee": 59.42777801478976,
- "left_ankle": 72.32051186191548,
- "right_wrist": 86.81275759262775,
- "right_elbow": 48.29244471600249,
- "right_shoulder": 54.511999224330935,
- "left_shoulder": 43.36640565830605,
- "left_elbow": 51.885325375511286,
- "left_wrist": 90.90350675137523,
- "neck": 45.628384586023635,
- "head": 104.62645314141196
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 89.76602092277679,
- "PA_MPJPE_mm": 59.00913702183445,
- "PCK3D_50mm": 0.5619839853960088,
- "PCK3D_100mm": 0.8834792349500062,
- "AUC_0_150mm": 0.6289464926286337,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.76743563027034,
- "right_knee": 56.524066579871175,
- "right_hip": 48.542531848040234,
- "left_hip": 46.01072158916307,
- "left_knee": 59.86351517410251,
- "left_ankle": 70.20175120276035,
- "right_wrist": 83.5308534647184,
- "right_elbow": 44.41150561241701,
- "right_shoulder": 48.97633399804551,
- "left_shoulder": 44.532603550435226,
- "left_elbow": 48.7994064122216,
- "left_wrist": 86.94891919996788
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 46.35350066965975
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 55.03873360166745
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 55.47877302820346
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 53.37219047568269
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 51.37453491041696
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.45883494320042
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 53.407439067655766
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 64.49541620998106
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 54.02261643256779
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 46.61354253002335
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.647154894652715
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 57.80507374496757
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 49.78489711321818
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.22652926291575
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 80.1205200700963
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 65.63093523107665
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 60.881202774814454
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 275,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 80.26983737203881,
- "PA_MPJPE_mm": 54.22448018730959,
- "PCK3D_50mm": 0.6245470826591434,
- "PCK3D_100mm": 0.9016633475777566,
- "AUC_0_150mm": 0.6569575509049649,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.40275099812783,
- "left_hip": 40.42429168260508,
- "right_hip": 39.51798235524411,
- "spine1": 45.17206631860313,
- "left_knee": 56.022011195178415,
- "right_knee": 55.839567939448244,
- "spine2": 43.74862169018239,
- "left_ankle": 62.78387016996401,
- "right_ankle": 63.99530300599515,
- "spine3": 41.1397963609844,
- "left_foot": 70.28881060665799,
- "right_foot": 71.03529697016094,
- "neck": 34.492076495216224,
- "left_collar": 29.776608811491318,
- "right_collar": 30.177677742591488,
- "head": 40.39285483802678,
- "left_shoulder": 39.72566104325211,
- "right_shoulder": 40.50917841711117,
- "left_elbow": 52.88215955482283,
- "right_elbow": 50.05378742980195,
- "left_wrist": 82.37153620649705,
- "right_wrist": 78.89021505454873,
- "left_hand": 97.21858930607797,
- "right_hand": 96.52681030284036
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.68495824712078,
- "PA_MPJPE_mm": 60.48139097450015,
- "PCK3D_50mm": 0.5229313482020614,
- "PCK3D_100mm": 0.8702384409765234,
- "AUC_0_150mm": 0.6112602194384278,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.23300723925892,
- "right_knee": 53.251920197080516,
- "right_hip": 51.15084093815264,
- "left_hip": 48.61027704689794,
- "left_knee": 55.163334292073834,
- "left_ankle": 66.1812658985285,
- "right_wrist": 80.77002867032783,
- "right_elbow": 45.72101056010095,
- "right_shoulder": 52.905562923498216,
- "left_shoulder": 42.812442575376664,
- "left_elbow": 50.279374459888906,
- "left_wrist": 87.3512098311026,
- "neck": 44.1884112249524,
- "head": 103.12078778576229
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.4759995099639,
- "PA_MPJPE_mm": 56.08571221604588,
- "PCK3D_50mm": 0.581542408275595,
- "PCK3D_100mm": 0.8992863958843298,
- "AUC_0_150mm": 0.6410496797123812,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.75913972035031,
- "right_knee": 53.52631763297199,
- "right_hip": 48.52652209401131,
- "left_hip": 46.76423178512845,
- "left_knee": 55.545395001714255,
- "left_ankle": 63.25665964991212,
- "right_wrist": 78.54834436866845,
- "right_elbow": 42.89368742441846,
- "right_shoulder": 47.27891246981358,
- "left_shoulder": 42.66838612571805,
- "left_elbow": 47.41462026005756,
- "left_wrist": 83.84633005978603
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 45.51383447593414
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 52.048128103211546
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 53.86529106760148
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.89333890798706
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 50.12785567305116
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.50550450137052
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.50633972946262
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 60.670426644688334
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.96378945173
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.63872831323301
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.34899677092646
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 52.72301762843724
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.94367249301546
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.539552520591144
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 78.42641430045558
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 61.24671988857725
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 58.36759059360562
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_detector_224.json b/benchmark/results/gen9w_emdb1_detector_224.json
deleted file mode 100644
index 315e8f7..0000000
--- a/benchmark/results/gen9w_emdb1_detector_224.json
+++ /dev/null
@@ -1,892 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 77.93272024703253,
- "PA_MPJPE_mm": 51.26658662518217,
- "PCK3D_50mm": 0.6540755398180934,
- "PCK3D_100mm": 0.9144223425675206,
- "AUC_0_150mm": 0.6717235208614692,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.15333533169134,
- "left_hip": 40.0614633008373,
- "right_hip": 39.5434656360952,
- "spine1": 44.794557442609374,
- "left_knee": 51.60447829493982,
- "right_knee": 51.80129545942806,
- "spine2": 43.24984927595156,
- "left_ankle": 55.490444413085946,
- "right_ankle": 57.3901399084674,
- "spine3": 40.64083855350627,
- "left_foot": 62.76068910503937,
- "right_foot": 64.8241715556129,
- "neck": 33.30451042107754,
- "left_collar": 28.71968988913112,
- "right_collar": 29.21379972213772,
- "head": 39.45397549888446,
- "left_shoulder": 38.605579817091645,
- "right_shoulder": 39.60827200860123,
- "left_elbow": 51.03987229609044,
- "right_elbow": 46.620280570669074,
- "left_wrist": 77.38627797096356,
- "right_wrist": 73.05033049184212,
- "left_hand": 91.73683207642085,
- "right_hand": 91.34392996419862
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 89.4670883789237,
- "PA_MPJPE_mm": 57.50753527970483,
- "PCK3D_50mm": 0.5554824887672211,
- "PCK3D_100mm": 0.8836373374594578,
- "AUC_0_150mm": 0.6274060607573115,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.78380139238089,
- "right_knee": 48.94005054769252,
- "right_hip": 50.007973843999636,
- "left_hip": 47.55559195004585,
- "left_knee": 50.884738883808126,
- "left_ankle": 59.87704281103771,
- "right_wrist": 74.40164921292127,
- "right_elbow": 41.845538620468275,
- "right_shoulder": 51.859657341517256,
- "left_shoulder": 42.783832964315515,
- "left_elbow": 47.978603878780554,
- "left_wrist": 81.72199147243376,
- "neck": 43.54909322329852,
- "head": 103.91592777316778
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 83.16979605940543,
- "PA_MPJPE_mm": 52.57603290868653,
- "PCK3D_50mm": 0.6219225161424703,
- "PCK3D_100mm": 0.9136291050475596,
- "AUC_0_150mm": 0.6601526787654005,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 56.88071479089014,
- "right_knee": 49.66047892645083,
- "right_hip": 47.22499292721182,
- "left_hip": 45.475930145307245,
- "left_knee": 51.61865925440633,
- "left_ankle": 56.605613217315344,
- "right_wrist": 72.72006041394528,
- "right_elbow": 39.941703814269964,
- "right_shoulder": 45.595942283694676,
- "left_shoulder": 41.23624633560497,
- "left_elbow": 45.36211706621922,
- "left_wrist": 78.5899357289225
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 42.80027409071535
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.31407347255747
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 54.63478336685305
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.71950281450704
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.60282068122893
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.88642846915626
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 45.92726397268056
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 59.45218646958912
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 48.66897167414233
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 41.93640571859846
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 47.354733686339614
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 49.603002587189465
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 45.247492639447216
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.3978696044395
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 60.88214152304064
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 53.1596753387607
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.999141190480394
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 495,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 95.23262235109229,
- "PA_MPJPE_mm": 66.05497057164119,
- "PCK3D_50mm": 0.5345952232173853,
- "PCK3D_100mm": 0.8464382420329098,
- "AUC_0_150mm": 0.6040456961251351,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 44.22741776238005,
- "left_hip": 45.318835135548596,
- "right_hip": 45.12412088521768,
- "spine1": 50.45760905412255,
- "left_knee": 67.9202802814058,
- "right_knee": 65.91444482424754,
- "spine2": 47.2138240167276,
- "left_ankle": 82.48851482780994,
- "right_ankle": 84.68621191946194,
- "spine3": 44.22940974351632,
- "left_foot": 92.47483617504012,
- "right_foot": 93.5543722712093,
- "neck": 40.21192707029494,
- "left_collar": 35.00963610750935,
- "right_collar": 35.555271791710446,
- "head": 48.11894474264623,
- "left_shoulder": 43.68438148051055,
- "right_shoulder": 43.686584909039674,
- "left_elbow": 61.58287085969108,
- "right_elbow": 58.93516037445934,
- "left_wrist": 103.56068920521386,
- "right_wrist": 101.58285070301746,
- "left_hand": 124.31349281222259,
- "right_hand": 125.46760676638618
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 105.89022186245927,
- "PA_MPJPE_mm": 70.89833199139689,
- "PCK3D_50mm": 0.44347011039366796,
- "PCK3D_100mm": 0.815481893653108,
- "AUC_0_150mm": 0.5637874022021141,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 85.60938518163074,
- "right_knee": 62.76360580709031,
- "right_hip": 55.73110490224694,
- "left_hip": 53.00947312384336,
- "left_knee": 66.24100946785089,
- "left_ankle": 85.73573488901707,
- "right_wrist": 103.21026112167122,
- "right_elbow": 54.48177326107945,
- "right_shoulder": 56.835219502579555,
- "left_shoulder": 47.0314321613496,
- "left_elbow": 58.94273001579015,
- "left_wrist": 107.21723346694431,
- "neck": 48.81179799413859,
- "head": 106.95588698432411
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 101.79125145230734,
- "PA_MPJPE_mm": 67.65694571136295,
- "PCK3D_50mm": 0.489047420676248,
- "PCK3D_100mm": 0.8404985072554328,
- "AUC_0_150mm": 0.5881997030188602,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 83.5446844620612,
- "right_knee": 63.17132774829755,
- "right_hip": 53.46659257075443,
- "left_hip": 51.43887375923552,
- "left_knee": 66.57819917852638,
- "left_ankle": 83.12837376684638,
- "right_wrist": 99.76339944222408,
- "right_elbow": 50.46397924339159,
- "right_shoulder": 52.42843296179403,
- "left_shoulder": 49.17066663357897,
- "left_elbow": 55.44898760980346,
- "left_wrist": 103.27983115984182
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 52.67127860816841
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 61.226534340243425
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 62.49005435330854
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 62.683063806536076
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 56.71348261604477
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 99.4847635168394
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 60.38898398167203
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 70.75433743792576
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 65.56420390672218
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 54.73895318320908
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 62.213801278357735
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 66.25250094803904
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.794596174781624
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 58.731162411293724
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 90.38425633280964
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 75.18000573984044
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 69.55083074630772
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 476,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 91.43899171758987,
- "PA_MPJPE_mm": 63.36693650710691,
- "PCK3D_50mm": 0.5537561619107131,
- "PCK3D_100mm": 0.8578941887106852,
- "AUC_0_150mm": 0.6157363609283909,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.84038311292027,
- "left_hip": 44.037213878304975,
- "right_hip": 43.378752005275125,
- "spine1": 48.99269332448613,
- "left_knee": 65.03994976639292,
- "right_knee": 63.83958032660458,
- "spine2": 46.24800543125666,
- "left_ankle": 78.16752486944955,
- "right_ankle": 80.78880120601812,
- "spine3": 43.44292508366987,
- "left_foot": 87.27330239502103,
- "right_foot": 89.41141326179941,
- "neck": 37.781917870612006,
- "left_collar": 33.50367304311045,
- "right_collar": 33.86938725573472,
- "head": 45.61237728322669,
- "left_shoulder": 42.4829496429172,
- "right_shoulder": 43.26778050713899,
- "left_elbow": 59.04690159214296,
- "right_elbow": 56.85707138731543,
- "left_wrist": 100.42127516762199,
- "right_wrist": 96.48629099428611,
- "left_hand": 119.96316332352241,
- "right_hand": 118.05314344173925
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 101.39460396886135,
- "PA_MPJPE_mm": 68.22615849631471,
- "PCK3D_50mm": 0.46694141101556225,
- "PCK3D_100mm": 0.8292230785253072,
- "AUC_0_150mm": 0.5764236905329824,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 81.43045180167078,
- "right_knee": 60.385505440291084,
- "right_hip": 53.88447034934697,
- "left_hip": 51.02990125896034,
- "left_knee": 63.12828611701542,
- "left_ankle": 81.11234080919839,
- "right_wrist": 98.30140471801248,
- "right_elbow": 51.43213367907262,
- "right_shoulder": 56.12642499149054,
- "left_shoulder": 46.07179550208336,
- "left_elbow": 56.05546277183182,
- "left_wrist": 104.79943752518528,
- "neck": 46.854144115893845,
- "head": 104.55445986835291
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 97.29514408992455,
- "PA_MPJPE_mm": 64.99934735583398,
- "PCK3D_50mm": 0.5119245990418663,
- "PCK3D_100mm": 0.8522460598486427,
- "AUC_0_150mm": 0.6000788366222987,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 79.21592140235464,
- "right_knee": 60.84201878482035,
- "right_hip": 51.728229047865426,
- "left_hip": 49.51777088206432,
- "left_knee": 63.47845133682494,
- "left_ankle": 78.46115182100472,
- "right_wrist": 95.53926998493515,
- "right_elbow": 48.2324450245917,
- "right_shoulder": 51.79221351116445,
- "left_shoulder": 47.4157492692701,
- "left_elbow": 52.823614884501275,
- "left_wrist": 100.9453323206106
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 51.104323584328924
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 59.74830891271705
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 59.66516634642243
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 59.32908890136233
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 54.448473794945606
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 90.46064303401337
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 64.06049765778022
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 69.67192521222434
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 61.85058413341746
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 50.47870239353256
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 58.48783768018488
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 63.450440089600924
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 54.42431831724349
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 54.2643832642692
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 83.1587149383846
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 74.22386715031978
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 71.79529916141044
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 343,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 86.36757142826029,
- "PA_MPJPE_mm": 59.1368474641432,
- "PCK3D_50mm": 0.5924130389502187,
- "PCK3D_100mm": 0.8787769909046727,
- "AUC_0_150mm": 0.6365587478415116,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 40.34187510692447,
- "left_hip": 42.15304982848357,
- "right_hip": 41.455681232637694,
- "spine1": 46.665458477835486,
- "left_knee": 60.96849083437088,
- "right_knee": 59.59101808488021,
- "spine2": 44.90093834630875,
- "left_ankle": 70.33524955810324,
- "right_ankle": 72.03684000177269,
- "spine3": 42.701909689960985,
- "left_foot": 77.59712484631818,
- "right_foot": 79.29210862928073,
- "neck": 36.991127669970375,
- "left_collar": 32.04198282803094,
- "right_collar": 32.772696586016565,
- "head": 44.249848834919185,
- "left_shoulder": 42.64599349529411,
- "right_shoulder": 41.96246772621233,
- "left_elbow": 57.23938823478103,
- "right_elbow": 53.74970368268758,
- "left_wrist": 91.6159052009957,
- "right_wrist": 89.63795789362807,
- "left_hand": 108.4577145813716,
- "right_hand": 109.87980776865083
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 97.66782794931949,
- "PA_MPJPE_mm": 64.96953126373381,
- "PCK3D_50mm": 0.4946082661350314,
- "PCK3D_100mm": 0.8472966941411015,
- "AUC_0_150mm": 0.592491675601377,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 73.60143158238142,
- "right_knee": 56.4389967932258,
- "right_hip": 52.43042317904653,
- "left_hip": 49.14251732559699,
- "left_knee": 59.63327030344196,
- "left_ankle": 73.12152554280689,
- "right_wrist": 92.10014495786292,
- "right_elbow": 50.04477973468864,
- "right_shoulder": 56.208888257294916,
- "left_shoulder": 44.82681573479825,
- "left_elbow": 54.63144646326971,
- "left_wrist": 96.33839513603827,
- "neck": 45.68687565635401,
- "head": 105.36792702546698
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 92.52557504388959,
- "PA_MPJPE_mm": 61.041919473112806,
- "PCK3D_50mm": 0.5480038880788725,
- "PCK3D_100mm": 0.8742310629729917,
- "AUC_0_150mm": 0.6202396230176063,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.99087626673987,
- "right_knee": 56.736736916745684,
- "right_hip": 49.95747091869593,
- "left_hip": 47.684154432436415,
- "left_knee": 60.22483567726383,
- "left_ankle": 70.51082078151717,
- "right_wrist": 89.57145346919917,
- "right_elbow": 46.37413204743002,
- "right_shoulder": 50.240710043082224,
- "left_shoulder": 46.494411263013646,
- "left_elbow": 51.378123382470086,
- "left_wrist": 92.33930847875959
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 45.61672490174277
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 55.679291687080855
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 55.99859071854547
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 55.81913623576881
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 51.83811667984022
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 80.08262151739717
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 57.30757261330848
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 68.17732788686986
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 54.30305931041489
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 47.41155601226178
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 54.537469104713914
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 60.044366968066
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 49.53942373207649
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.668877311842195
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 90.17081887314328
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 67.42130325562283
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 60.927954314528584
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 416,
- "source": "raw",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 81.17946078051318,
- "PA_MPJPE_mm": 54.49023830392354,
- "PCK3D_50mm": 0.6263573561063668,
- "PCK3D_100mm": 0.900883496493786,
- "AUC_0_150mm": 0.6566018392225633,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.50627909092445,
- "left_hip": 41.37784207521789,
- "right_hip": 40.601263174275005,
- "spine1": 45.411020147178334,
- "left_knee": 56.24301216836695,
- "right_knee": 55.04977629968738,
- "spine2": 42.94023029532674,
- "left_ankle": 61.55660540710874,
- "right_ankle": 61.58856823542339,
- "spine3": 40.89425963231863,
- "left_foot": 69.3172010202465,
- "right_foot": 69.75489178737477,
- "neck": 35.374264591667234,
- "left_collar": 30.992583891057688,
- "right_collar": 30.920243507093335,
- "head": 40.36730295257536,
- "left_shoulder": 39.57545408902788,
- "right_shoulder": 41.44402584629446,
- "left_elbow": 53.32453685421091,
- "right_elbow": 51.51586608918955,
- "left_wrist": 81.23882937874446,
- "right_wrist": 80.88904879296595,
- "left_hand": 97.28402584389124,
- "right_hand": 100.59858812399776
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 91.5290597678693,
- "PA_MPJPE_mm": 60.1337125007405,
- "PCK3D_50mm": 0.5290028863034487,
- "PCK3D_100mm": 0.8734579105543488,
- "AUC_0_150mm": 0.6146370235847562,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.7077786477625,
- "right_knee": 52.73703340142972,
- "right_hip": 49.43108769341662,
- "left_hip": 46.94390037709033,
- "left_knee": 55.338267528178434,
- "left_ankle": 65.59180580723309,
- "right_wrist": 82.19689660324315,
- "right_elbow": 46.11298919860812,
- "right_shoulder": 53.761121184878924,
- "left_shoulder": 43.11273615905586,
- "left_elbow": 50.26810795200838,
- "left_wrist": 85.45022904519355,
- "neck": 45.91938809188806,
- "head": 102.30063332038024
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 86.06808320595972,
- "PA_MPJPE_mm": 55.646359489636424,
- "PCK3D_50mm": 0.590734569186975,
- "PCK3D_100mm": 0.9013504131083802,
- "AUC_0_150mm": 0.6451020508720181,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.609566155084366,
- "right_knee": 53.366804217193504,
- "right_hip": 47.29184892403487,
- "left_hip": 45.29607596258607,
- "left_knee": 56.042141208286466,
- "left_ankle": 62.52884789841417,
- "right_wrist": 79.2922640170551,
- "right_elbow": 42.991788962336265,
- "right_shoulder": 48.09224252768964,
- "left_shoulder": 43.04466186923539,
- "left_elbow": 47.60146801776535,
- "left_wrist": 81.59860411595585
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 46.31959657073665
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.133350292731905
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 54.368302598830454
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 50.9813893164271
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.86156295372381
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 71.99969811968687
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.13447721233935
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 62.577595392524344
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 47.78500447092444
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 44.45550742479953
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 49.60420275216622
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 53.956962148348104
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.54430281127898
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 50.003952810148455
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 82.89068286910323
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.142572536992766
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 58.00801376494526
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_detector_288.json b/benchmark/results/gen9w_emdb1_detector_288.json
deleted file mode 100644
index ef41ac6..0000000
--- a/benchmark/results/gen9w_emdb1_detector_288.json
+++ /dev/null
@@ -1,714 +0,0 @@
-{
- "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 361,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 89.88001462362865,
- "PA_MPJPE_mm": 62.14829779020451,
- "PCK3D_50mm": 0.559473720752621,
- "PCK3D_100mm": 0.8656564604596265,
- "AUC_0_150mm": 0.6211403650225199,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.3998926536106,
- "left_hip": 43.23514052527652,
- "right_hip": 42.737193823034055,
- "spine1": 48.81111286770574,
- "left_knee": 64.66572811594146,
- "right_knee": 62.881830878407776,
- "spine2": 46.65410707767702,
- "left_ankle": 78.27946454078177,
- "right_ankle": 79.93866684278517,
- "spine3": 43.885610075995054,
- "left_foot": 86.62946626547422,
- "right_foot": 88.5823919305529,
- "neck": 37.52294377293753,
- "left_collar": 33.287637841065525,
- "right_collar": 33.55174663542328,
- "head": 44.81931039722878,
- "left_shoulder": 42.157586875116095,
- "right_shoulder": 42.83101945768526,
- "left_elbow": 58.163221387297845,
- "right_elbow": 54.593885177986586,
- "left_wrist": 95.5494237098845,
- "right_wrist": 92.47845791273399,
- "left_hand": 114.15463760732204,
- "right_hand": 113.74867059298487
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 99.87602829547231,
- "PA_MPJPE_mm": 67.34457038338077,
- "PCK3D_50mm": 0.4677775463445116,
- "PCK3D_100mm": 0.8367721010503764,
- "AUC_0_150mm": 0.5799334240082471,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 80.78047750813414,
- "right_knee": 59.93015732684019,
- "right_hip": 53.60761250152085,
- "left_hip": 51.271079847522174,
- "left_knee": 63.71833115447427,
- "left_ankle": 81.07929827327376,
- "right_wrist": 93.89229910270527,
- "right_elbow": 49.93170585158508,
- "right_shoulder": 56.64462222889465,
- "left_shoulder": 45.869644577123054,
- "left_elbow": 55.14241513381188,
- "left_wrist": 99.94414656471223,
- "neck": 46.76928662281015,
- "head": 104.24290867392317
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 95.95171474047633,
- "PA_MPJPE_mm": 63.93110681655201,
- "PCK3D_50mm": 0.5155037145039227,
- "PCK3D_100mm": 0.8610810247865028,
- "AUC_0_150mm": 0.6048097058632498,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 78.86436167404493,
- "right_knee": 60.32354574190792,
- "right_hip": 51.25792133949062,
- "left_hip": 49.622990796997975,
- "left_knee": 64.04343778958985,
- "left_ankle": 78.35190145870001,
- "right_wrist": 91.3217455838409,
- "right_elbow": 46.49540018149841,
- "right_shoulder": 51.537472126409845,
- "left_shoulder": 46.9886915358542,
- "left_elbow": 52.20780244939982,
- "left_wrist": 96.15801112088965
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 47.82647999052852
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 57.43144619804872
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 59.837096476076894
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 60.850199250429654
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 54.30761326999251
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 86.48715626780354
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 61.20026539296575
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 68.42785674243936
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 63.10620382781966
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 49.75701399154462
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 57.63075629535146
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 60.86506785817905
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 53.08237755329758
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 56.54837387465017
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 83.44831832132593
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 73.21719099524415
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 66.43926934316917
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 317,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 87.9075350376874,
- "PA_MPJPE_mm": 60.45347662032824,
- "PCK3D_50mm": 0.5697042282857738,
- "PCK3D_100mm": 0.8709869471637853,
- "AUC_0_150mm": 0.6264520944337313,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.11719627930378,
- "left_hip": 43.21645094160724,
- "right_hip": 42.64613211619856,
- "spine1": 48.63178231054832,
- "left_knee": 63.25816868312911,
- "right_knee": 60.276422936410334,
- "spine2": 45.857657646733166,
- "left_ankle": 74.72452030737033,
- "right_ankle": 75.32958620460016,
- "spine3": 43.0093593710802,
- "left_foot": 83.16578868087672,
- "right_foot": 83.68150904320454,
- "neck": 37.80670167568633,
- "left_collar": 32.73762058776788,
- "right_collar": 32.74992441460084,
- "head": 44.41713684337203,
- "left_shoulder": 41.79605713118931,
- "right_shoulder": 40.8484732057064,
- "left_elbow": 56.99921608632764,
- "right_elbow": 54.09836486313372,
- "left_wrist": 91.5399513530506,
- "right_wrist": 90.29105803345213,
- "left_hand": 110.21920956039801,
- "right_hand": 111.46515061212946
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 98.21326053740394,
- "PA_MPJPE_mm": 65.68104694607909,
- "PCK3D_50mm": 0.47828131044127714,
- "PCK3D_100mm": 0.8418841312821733,
- "AUC_0_150mm": 0.5852919466662572,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 77.11197146713526,
- "right_knee": 57.217803944290914,
- "right_hip": 53.563592342153655,
- "left_hip": 50.879081569825196,
- "left_knee": 62.0414668513186,
- "left_ankle": 77.8268564345301,
- "right_wrist": 91.46379788481748,
- "right_elbow": 49.546304014344045,
- "right_shoulder": 54.865793673684024,
- "left_shoulder": 44.712431453158324,
- "left_elbow": 53.45801633407019,
- "left_wrist": 95.4466219119112,
- "neck": 46.857606314487114,
- "head": 104.54331304938117
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 94.11300202800673,
- "PA_MPJPE_mm": 61.769127946112974,
- "PCK3D_50mm": 0.5305214191487885,
- "PCK3D_100mm": 0.8682704992015552,
- "AUC_0_150mm": 0.6127208041335475,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 74.8928982249361,
- "right_knee": 57.77475641815015,
- "right_hip": 51.08245577830001,
- "left_hip": 49.01476598044393,
- "left_knee": 62.40287855529652,
- "left_ankle": 75.48429314836426,
- "right_wrist": 88.70027084238862,
- "right_elbow": 45.46923814399258,
- "right_shoulder": 48.99209913672429,
- "left_shoulder": 45.3722580361786,
- "left_elbow": 50.35916692018572,
- "left_wrist": 91.68445416839494
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 49.182159962268344
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 55.95248435153752
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 56.39319704294962
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 57.59563594680395
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 53.04390972365793
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.65128991734748
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 55.8418913307253
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 65.50128345059231
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 57.53065260591323
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 48.235659021517954
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 56.78525892132384
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 60.21690219212064
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 54.923596400111414
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.63809435601619
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 87.12556008035637
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 71.18086393821
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 64.82726076497933
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 323,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 81.70291446004636,
- "PA_MPJPE_mm": 55.25000177304557,
- "PCK3D_50mm": 0.6208272582100951,
- "PCK3D_100mm": 0.896361869055058,
- "AUC_0_150mm": 0.6530614141767059,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.36809596880524,
- "left_hip": 40.84968376295284,
- "right_hip": 40.34568619339702,
- "spine1": 45.33971375500754,
- "left_knee": 57.4499167916032,
- "right_knee": 56.69320057050378,
- "spine2": 42.79635873455927,
- "left_ankle": 66.23867167989718,
- "right_ankle": 67.20490166050838,
- "spine3": 40.18898450313805,
- "left_foot": 74.06867628166592,
- "right_foot": 74.97694216244753,
- "neck": 35.210746438080506,
- "left_collar": 30.483753055433546,
- "right_collar": 30.166025771787584,
- "head": 41.42539351845108,
- "left_shoulder": 39.85333532064559,
- "right_shoulder": 40.30003418143373,
- "left_elbow": 52.350985968289194,
- "right_elbow": 51.168905219468414,
- "left_wrist": 81.79225389953217,
- "right_wrist": 80.1494535169007,
- "left_hand": 98.13599122766432,
- "right_hand": 99.44233237092202
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 92.51150950378644,
- "PA_MPJPE_mm": 61.13281827674059,
- "PCK3D_50mm": 0.5239027583539143,
- "PCK3D_100mm": 0.8659564971583301,
- "AUC_0_150mm": 0.609484679171102,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 69.83279371171066,
- "right_knee": 54.386321863332,
- "right_hip": 49.949757888314146,
- "left_hip": 46.92096921179578,
- "left_knee": 56.939028778782,
- "left_ankle": 68.92848963746131,
- "right_wrist": 81.96375459958284,
- "right_elbow": 46.487695985272474,
- "right_shoulder": 53.30152037354356,
- "left_shoulder": 42.07438806223504,
- "left_elbow": 49.689106519976015,
- "left_wrist": 86.16777773630653,
- "neck": 44.79358102850941,
- "head": 104.42427047754647
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 87.02721828901434,
- "PA_MPJPE_mm": 56.55963730819824,
- "PCK3D_50mm": 0.5837151982225925,
- "PCK3D_100mm": 0.8963132680691522,
- "AUC_0_150mm": 0.6407481192314326,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 66.56879575317143,
- "right_knee": 54.64333821995198,
- "right_hip": 47.54505039934619,
- "left_hip": 45.13750460047565,
- "left_knee": 57.367261731090444,
- "left_ankle": 66.75627507043153,
- "right_wrist": 78.85960824685846,
- "right_elbow": 42.74106867204931,
- "right_shoulder": 47.213641986421194,
- "left_shoulder": 42.741282079576685,
- "left_elbow": 46.82117737924292,
- "left_wrist": 82.32064355976314
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 44.42583264589824
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 51.282244845755855
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 55.161071580952814
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.31193487164133
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.647686459616416
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 74.49491926359632
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 52.29583824854792
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 63.386672577153654
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 48.94452183730741
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 44.9808821295364
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 50.29255742705601
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 53.24283853085669
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.94593109597065
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.49571013617698
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 78.88111991386596
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 64.19278282236355
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 58.91895341067835
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 275,
- "source": "ema",
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 78.31500077149303,
- "PA_MPJPE_mm": 52.66772764952136,
- "PCK3D_50mm": 0.6412986877733805,
- "PCK3D_100mm": 0.9088627369298062,
- "AUC_0_150mm": 0.6652944167097805,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 38.16107682370417,
- "left_hip": 40.32936476673414,
- "right_hip": 39.553525407854735,
- "spine1": 45.06865733314703,
- "left_knee": 54.73045648161787,
- "right_knee": 53.64387498095441,
- "spine2": 43.37882052704738,
- "left_ankle": 60.5835420090056,
- "right_ankle": 61.932630389354756,
- "spine3": 40.56942896498256,
- "left_foot": 67.63568905070605,
- "right_foot": 68.70566906964369,
- "neck": 33.62035222713069,
- "left_collar": 28.977020813382847,
- "right_collar": 29.486228657581364,
- "head": 38.97048078947517,
- "left_shoulder": 38.9090368298353,
- "right_shoulder": 39.78626220151134,
- "left_elbow": 51.38850209466253,
- "right_elbow": 48.4267230965889,
- "left_wrist": 78.80163991057078,
- "right_wrist": 75.24358981173182,
- "left_hand": 93.53949185735175,
- "right_hand": 92.58339949393792
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 89.96401815773763,
- "PA_MPJPE_mm": 58.914789532569,
- "PCK3D_50mm": 0.5390632903859315,
- "PCK3D_100mm": 0.8783437974231559,
- "AUC_0_150mm": 0.6200680157839621,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.31124899882076,
- "right_knee": 51.44346448500823,
- "right_hip": 50.84465517402539,
- "left_hip": 48.15254692824374,
- "left_knee": 53.963076411911736,
- "left_ankle": 64.18753284980791,
- "right_wrist": 76.83065739128713,
- "right_elbow": 44.10553878683052,
- "right_shoulder": 52.13329867825932,
- "left_shoulder": 41.65496531716521,
- "left_elbow": 48.550259974703614,
- "left_wrist": 83.37263194058477,
- "neck": 43.686478527206724,
- "head": 102.57069799211094
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 84.3438997583312,
- "PA_MPJPE_mm": 54.30554774307022,
- "PCK3D_50mm": 0.6013295841144206,
- "PCK3D_100mm": 0.9069499409845171,
- "AUC_0_150mm": 0.6511898282839819,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.92675911572212,
- "right_knee": 51.65591727801712,
- "right_hip": 48.08967114856443,
- "left_hip": 46.20407972614968,
- "left_knee": 54.322533769730455,
- "left_ankle": 61.11467717487386,
- "right_wrist": 74.85939995395367,
- "right_elbow": 41.54015028052913,
- "right_shoulder": 46.01650771474177,
- "left_shoulder": 41.12498133443737,
- "left_elbow": 45.826401574536035,
- "left_wrist": 79.985493845587
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 43.54676925403121
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.81617804135277
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 53.02026134036774
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.2998285082673
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.59228462173178
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 68.1191567617839
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.25192530171703
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 58.75156916277865
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 46.36786004875756
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 42.466393419364344
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 45.9137857697515
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 49.75410098337341
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 47.42560778317899
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.900767877957755
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 78.07832082241798
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.976134433477256
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.97351651190629
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_gt-joints_224.json b/benchmark/results/gen9w_emdb1_gt-joints_224.json
deleted file mode 100644
index eb16299..0000000
--- a/benchmark/results/gen9w_emdb1_gt-joints_224.json
+++ /dev/null
@@ -1,867 +0,0 @@
-{
- "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 481,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 75.45319378157116,
- "PA_MPJPE_mm": 50.32863498976247,
- "PCK3D_50mm": 0.6644383824973378,
- "PCK3D_100mm": 0.9189433542159344,
- "AUC_0_150mm": 0.6767868765722734,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 36.29148536825942,
- "left_hip": 38.24552586212516,
- "right_hip": 37.600955289971296,
- "spine1": 42.841433811205626,
- "left_knee": 50.3974071974786,
- "right_knee": 50.636864207098895,
- "spine2": 42.87487848201726,
- "left_ankle": 54.447180676894604,
- "right_ankle": 55.30010689324191,
- "spine3": 41.06766402362964,
- "left_foot": 61.48244702825232,
- "right_foot": 62.10221206337029,
- "neck": 32.24941748448096,
- "left_collar": 28.165231607074126,
- "right_collar": 29.045948558461998,
- "head": 37.68983884655172,
- "left_shoulder": 37.81425445121847,
- "right_shoulder": 39.70770089314365,
- "left_elbow": 50.604477974573584,
- "right_elbow": 47.003193546634456,
- "left_wrist": 76.40272950121867,
- "right_wrist": 73.48478178514239,
- "left_hand": 90.80408930132057,
- "right_hand": 91.62741490093312
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 86.41165848688678,
- "PA_MPJPE_mm": 56.76824098666464,
- "PCK3D_50mm": 0.5623544194261533,
- "PCK3D_100mm": 0.8884311970649771,
- "AUC_0_150mm": 0.6310808777460634,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 57.107587898242244,
- "right_knee": 48.17377530664997,
- "right_hip": 48.27013921643617,
- "left_hip": 45.91431273776492,
- "left_knee": 50.0210048258065,
- "left_ankle": 58.42373661025704,
- "right_wrist": 74.65785131579995,
- "right_elbow": 41.81074013327866,
- "right_shoulder": 52.85493659343189,
- "left_shoulder": 43.177167245776566,
- "left_elbow": 47.45422252448673,
- "left_wrist": 80.89639938141364,
- "neck": 43.42408396871431,
- "head": 102.56941605524631
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 80.79194371698863,
- "PA_MPJPE_mm": 51.799443261140055,
- "PCK3D_50mm": 0.6299319586773431,
- "PCK3D_100mm": 0.9173201122958414,
- "AUC_0_150mm": 0.6642359024598286,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 54.64409338810367,
- "right_knee": 48.88869145759666,
- "right_hip": 45.3866750768073,
- "left_hip": 43.713823356970885,
- "left_knee": 50.70734109073142,
- "left_ankle": 55.59481655264034,
- "right_wrist": 73.1607309509648,
- "right_elbow": 39.977807550054926,
- "right_shoulder": 45.675124820251376,
- "left_shoulder": 40.460065309981935,
- "left_elbow": 45.291183998498376,
- "left_wrist": 78.092965581079
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 41.65367167514524
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.27719618777549
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 50.95631810035045
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 47.76531423523893
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 45.57629567928524
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 67.6821731285332
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 46.13458534299859
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 58.84716692393309
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 46.161930904869564
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 43.06490859601754
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 47.47200755913893
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 49.147604732459804
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 44.91034623837281
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.41865594510458
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 59.00598028526663
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 51.79904127762359
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 54.21967075691419
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 495,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 95.41329950143607,
- "PA_MPJPE_mm": 65.93923730293947,
- "PCK3D_50mm": 0.5355851277157754,
- "PCK3D_100mm": 0.8473184527513864,
- "AUC_0_150mm": 0.6047466428798628,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 43.7423216800487,
- "left_hip": 44.84802612212388,
- "right_hip": 44.52511359707877,
- "spine1": 50.00695571295725,
- "left_knee": 66.85105609408488,
- "right_knee": 65.0204830245789,
- "spine2": 47.28361454635847,
- "left_ankle": 81.86573937240706,
- "right_ankle": 84.54204057235656,
- "spine3": 44.44319301745281,
- "left_foot": 91.87007853734274,
- "right_foot": 93.34236986455747,
- "neck": 40.546790493917854,
- "left_collar": 35.078845710048185,
- "right_collar": 35.75837693913816,
- "head": 48.99457124345012,
- "left_shoulder": 43.917674170602034,
- "right_shoulder": 44.06121908506595,
- "left_elbow": 61.59719153119287,
- "right_elbow": 58.9632324739968,
- "left_wrist": 104.1003255295113,
- "right_wrist": 101.2504220509526,
- "left_hand": 124.82206826693579,
- "right_hand": 125.10998563438598
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 106.0441768668011,
- "PA_MPJPE_mm": 70.95781098098507,
- "PCK3D_50mm": 0.4429798306079267,
- "PCK3D_100mm": 0.8149578297900083,
- "AUC_0_150mm": 0.5633976572509188,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 85.4394828233353,
- "right_knee": 61.963500348503686,
- "right_hip": 55.66559682805148,
- "left_hip": 53.03429120063521,
- "left_knee": 65.45732154445383,
- "left_ankle": 85.04942760620254,
- "right_wrist": 103.07489703709746,
- "right_elbow": 54.621141091789895,
- "right_shoulder": 57.62783351704533,
- "left_shoulder": 47.22371756980122,
- "left_elbow": 59.05243640566007,
- "left_wrist": 107.87418833091935,
- "neck": 48.94259379337621,
- "head": 108.38292563691958
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 102.04808263074143,
- "PA_MPJPE_mm": 67.52104596203705,
- "PCK3D_50mm": 0.4911525536240302,
- "PCK3D_100mm": 0.84176243621126,
- "AUC_0_150mm": 0.5890483895504018,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 83.42464931796681,
- "right_knee": 62.38526453576696,
- "right_hip": 53.269016540324834,
- "left_hip": 51.4099586642093,
- "left_knee": 65.87518005258465,
- "left_ankle": 82.37306231963079,
- "right_wrist": 99.55515720436617,
- "right_elbow": 50.53923408803409,
- "right_shoulder": 52.860322032987945,
- "left_shoulder": 49.26510975990263,
- "left_elbow": 55.53674836269248,
- "left_wrist": 103.75884866597777
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 52.17051819771386
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 61.15581783344582
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 60.995831869548056
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 63.61326458495019
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 56.88832080064986
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 103.88780833020698
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 62.270404631199
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 72.49280068867759
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 65.11922715357234
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 53.76403029896563
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 62.44102326283144
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 67.72893973967639
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 56.274159171565266
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 58.25281401271364
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 86.75836542555099
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 74.6967314697801
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 66.56765939294387
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 476,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 91.18124664746512,
- "PA_MPJPE_mm": 63.20443319361482,
- "PCK3D_50mm": 0.5583848483591254,
- "PCK3D_100mm": 0.8597442918585515,
- "AUC_0_150mm": 0.6176074989995891,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 42.3073817560868,
- "left_hip": 43.594246949406,
- "right_hip": 42.81940553003142,
- "spine1": 48.475566871842325,
- "left_knee": 64.30842557922038,
- "right_knee": 63.15926146624469,
- "spine2": 46.40549526654528,
- "left_ankle": 77.49907645045653,
- "right_ankle": 80.1417995188542,
- "spine3": 43.63859915025811,
- "left_foot": 86.6731518464849,
- "right_foot": 88.93654923720621,
- "neck": 37.91538399267599,
- "left_collar": 33.314448405511875,
- "right_collar": 33.82769795706792,
- "head": 45.64211741468662,
- "left_shoulder": 42.684651353278184,
- "right_shoulder": 43.58056932288431,
- "left_elbow": 59.40464663727825,
- "right_elbow": 56.808375628152355,
- "left_wrist": 100.3675821140375,
- "right_wrist": 97.09375727998422,
- "left_hand": 119.42234310845843,
- "right_hand": 118.8858638101043
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 101.50962366171527,
- "PA_MPJPE_mm": 68.29485628308944,
- "PCK3D_50mm": 0.46543109630692087,
- "PCK3D_100mm": 0.8291617522418667,
- "AUC_0_150mm": 0.5765195299512403,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 80.87672279281638,
- "right_knee": 59.548100332466646,
- "right_hip": 54.1343826313165,
- "left_hip": 51.47174722021055,
- "left_knee": 62.50245489524185,
- "left_ankle": 80.74661587005293,
- "right_wrist": 98.8480330024579,
- "right_elbow": 51.44850798201668,
- "right_shoulder": 56.45808335943594,
- "left_shoulder": 46.321948251405985,
- "left_elbow": 56.42709751344536,
- "left_wrist": 104.90062629533405,
- "neck": 46.96156869125297,
- "head": 105.4820991257985
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 97.35650569849777,
- "PA_MPJPE_mm": 64.92486918442634,
- "PCK3D_50mm": 0.5145728747458822,
- "PCK3D_100mm": 0.8533515883223388,
- "AUC_0_150mm": 0.6013912514348145,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 78.62649935888747,
- "right_knee": 59.94134140695337,
- "right_hip": 51.80283519954715,
- "left_hip": 49.79037573259599,
- "left_knee": 62.80030016144864,
- "left_ankle": 77.58158226097218,
- "right_wrist": 96.19592607998705,
- "right_elbow": 48.38814029038836,
- "right_shoulder": 52.13616698833467,
- "left_shoulder": 47.4959257643338,
- "left_elbow": 53.28411171340138,
- "left_wrist": 101.05522525626624
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 50.22747026922755
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 60.66047849617188
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.08169507815057
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 59.60667195308013
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 54.87308979659357
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 93.95179213765246
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 63.60166165194169
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 69.56536347989173
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 61.13061006024233
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 50.560195323164535
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 58.25059109963292
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 63.73049616794995
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 54.306223567700194
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 53.79515474109683
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 83.5839639750734
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 74.67850104823984
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 68.84021227090606
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 343,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 85.50047839829512,
- "PA_MPJPE_mm": 58.53128371420026,
- "PCK3D_50mm": 0.5928774426419948,
- "PCK3D_100mm": 0.881475680758965,
- "AUC_0_150mm": 0.6382147361302012,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.619211619773075,
- "left_hip": 41.37431921621365,
- "right_hip": 40.57991745946236,
- "spine1": 46.126295339737595,
- "left_knee": 59.79920000706359,
- "right_knee": 58.80555055813576,
- "spine2": 44.96719449520902,
- "left_ankle": 69.48648589419324,
- "right_ankle": 71.25704223953717,
- "spine3": 42.776656105514654,
- "left_foot": 76.69149280129876,
- "right_foot": 78.72888083980604,
- "neck": 36.72276831766147,
- "left_collar": 31.738128080287847,
- "right_collar": 32.389604392659386,
- "head": 43.87728479746904,
- "left_shoulder": 42.358061612369454,
- "right_shoulder": 41.61782280206008,
- "left_elbow": 56.948794330114985,
- "right_elbow": 53.4929164429843,
- "left_wrist": 90.6967748017553,
- "right_wrist": 88.71249474736656,
- "left_hand": 107.43925805337328,
- "right_hand": 108.54465418676101
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 96.85165034663717,
- "PA_MPJPE_mm": 64.53498529211366,
- "PCK3D_50mm": 0.49368484065409757,
- "PCK3D_100mm": 0.849879386679785,
- "AUC_0_150mm": 0.5933032983828428,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 72.62921403755631,
- "right_knee": 55.64561677970594,
- "right_hip": 52.40531742319207,
- "left_hip": 49.16529589883128,
- "left_knee": 58.687972568915946,
- "left_ankle": 72.13263047904198,
- "right_wrist": 90.96541671767332,
- "right_elbow": 49.64643115100276,
- "right_shoulder": 56.35346801730268,
- "left_shoulder": 45.11138834920038,
- "left_elbow": 54.1496330774696,
- "left_wrist": 95.41478961657818,
- "neck": 45.406717752207,
- "head": 105.77590222091378
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 91.79797087661343,
- "PA_MPJPE_mm": 60.42892858546434,
- "PCK3D_50mm": 0.5492020357078649,
- "PCK3D_100mm": 0.876962065579665,
- "AUC_0_150mm": 0.621987335712906,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 70.02150073025082,
- "right_knee": 56.09613644063622,
- "right_hip": 49.64130445681709,
- "left_hip": 47.40633358121243,
- "left_knee": 59.30186485105646,
- "left_ankle": 69.55318589903295,
- "right_wrist": 88.69330784958252,
- "right_elbow": 46.0612518405269,
- "right_shoulder": 49.74317651601932,
- "left_shoulder": 46.158436757226085,
- "left_elbow": 50.94971950944186,
- "left_wrist": 91.52092459376931
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 46.164615412961304
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 55.318601117627765
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 56.05507282317658
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 56.22275946082061
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 51.446825381711676
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 78.59623542843318
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 56.98048448581035
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 67.67023851351388
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 54.57184591962766
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 47.44879853163376
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 54.57462400279853
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 59.799926465920166
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.84116021471688
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.6144946616695
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 84.9540405800068
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 67.4456263048733
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 59.92901693659125
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_224_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 416,
- "source": "raw",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 81.03630026710945,
- "PA_MPJPE_mm": 54.44165274950423,
- "PCK3D_50mm": 0.6258989199131505,
- "PCK3D_100mm": 0.901220802389744,
- "AUC_0_150mm": 0.6567936039729138,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.07167548838527,
- "left_hip": 40.96902085962881,
- "right_hip": 40.22325267754232,
- "spine1": 45.00290613369919,
- "left_knee": 56.055549060627236,
- "right_knee": 54.95441671573737,
- "spine2": 43.01440933677799,
- "left_ankle": 61.386573377152985,
- "right_ankle": 61.65059006307621,
- "spine3": 40.98353484571552,
- "left_foot": 68.9463771362151,
- "right_foot": 69.84600880384232,
- "neck": 35.543694886602566,
- "left_collar": 30.894959582774128,
- "right_collar": 30.795244063319014,
- "head": 40.831098626214626,
- "left_shoulder": 39.67867734371399,
- "right_shoulder": 41.55592678348151,
- "left_elbow": 53.29158790381144,
- "right_elbow": 51.55849659878411,
- "left_wrist": 81.66053536102481,
- "right_wrist": 80.8321695988927,
- "left_hand": 97.47404635740429,
- "right_hand": 100.37891438367683
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 91.50852254651835,
- "PA_MPJPE_mm": 60.261419368593174,
- "PCK3D_50mm": 0.5278922007337558,
- "PCK3D_100mm": 0.8735427125254118,
- "AUC_0_150mm": 0.6138937903020276,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.78916590286097,
- "right_knee": 52.601309315332486,
- "right_hip": 49.71540133148126,
- "left_hip": 47.157917576508176,
- "left_knee": 55.256623409634805,
- "left_ankle": 65.59716731386467,
- "right_wrist": 82.13785700082016,
- "right_elbow": 46.02616910399252,
- "right_shoulder": 54.113394711328574,
- "left_shoulder": 43.295994595888516,
- "left_elbow": 50.206976561657484,
- "left_wrist": 85.96901116628584,
- "neck": 45.77932260361724,
- "head": 103.01356056703177
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.10199031274678,
- "PA_MPJPE_mm": 55.672044726484025,
- "PCK3D_50mm": 0.590441715415785,
- "PCK3D_100mm": 0.9020903345365031,
- "AUC_0_150mm": 0.6449458172118849,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 60.577207943084034,
- "right_knee": 53.25052772300932,
- "right_hip": 47.33065356144575,
- "left_hip": 45.33595106520317,
- "left_knee": 55.90542251427289,
- "left_ankle": 62.37443156875004,
- "right_wrist": 79.40945240139865,
- "right_elbow": 42.934680452789244,
- "right_shoulder": 48.30204573837914,
- "left_shoulder": 43.144962209892306,
- "left_elbow": 47.38617393755321,
- "left_wrist": 82.11302760203056
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 46.222777888505064
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.64475852826377
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 54.70199685465083
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.41733931730783
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.083919411419416
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 70.80514877637937
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 48.86430484575099
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 63.3610374782254
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.788310759377005
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.381678214455434
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 49.926233168259415
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.48135695216082
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.860705303009624
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.87090005467664
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 81.85380570646217
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 59.27342631372593
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.82037869596712
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_emdb1_gt-joints_288.json b/benchmark/results/gen9w_emdb1_gt-joints_288.json
deleted file mode 100644
index c9580c0..0000000
--- a/benchmark/results/gen9w_emdb1_gt-joints_288.json
+++ /dev/null
@@ -1,694 +0,0 @@
-{
- "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b0_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 361,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 89.4735776299812,
- "PA_MPJPE_mm": 61.959743297922074,
- "PCK3D_50mm": 0.5610418481793414,
- "PCK3D_100mm": 0.8657463109709718,
- "AUC_0_150mm": 0.621616782187913,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 41.96642556193023,
- "left_hip": 42.88608097585596,
- "right_hip": 42.1652763016479,
- "spine1": 48.30525197585183,
- "left_knee": 63.85657322215367,
- "right_knee": 62.71279907286846,
- "spine2": 46.58898639165771,
- "left_ankle": 78.46877324850936,
- "right_ankle": 79.35548798331912,
- "spine3": 43.883016110852004,
- "left_foot": 86.51959070132482,
- "right_foot": 88.38744174592493,
- "neck": 37.79052332914537,
- "left_collar": 33.28639952278085,
- "right_collar": 33.35179986214454,
- "head": 45.23224165389223,
- "left_shoulder": 42.26798455185012,
- "right_shoulder": 42.99157902759224,
- "left_elbow": 58.35407232076084,
- "right_elbow": 54.859535233432396,
- "left_wrist": 95.28798358230385,
- "right_wrist": 91.5891567234886,
- "left_hand": 113.91842152881505,
- "right_hand": 113.00843852202976
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 99.77156873693087,
- "PA_MPJPE_mm": 67.31447790021238,
- "PCK3D_50mm": 0.46613343922807476,
- "PCK3D_100mm": 0.8352042721415829,
- "AUC_0_150mm": 0.5794042311883085,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 80.09718961893094,
- "right_knee": 59.62683358529868,
- "right_hip": 53.63844210705563,
- "left_hip": 51.34767897737285,
- "left_knee": 62.96785580972124,
- "left_ankle": 81.46899973343122,
- "right_wrist": 93.10290744551273,
- "right_elbow": 49.76732477121191,
- "right_shoulder": 56.9114657934368,
- "left_shoulder": 45.86515785355661,
- "left_elbow": 55.274487976330796,
- "left_wrist": 99.86445039635852,
- "neck": 47.06885077868291,
- "head": 105.40104575607253
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 95.74634513067876,
- "PA_MPJPE_mm": 63.675979126750136,
- "PCK3D_50mm": 0.5168270892973212,
- "PCK3D_100mm": 0.8610269814269317,
- "AUC_0_150mm": 0.6056405997736418,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 77.98829798219931,
- "right_knee": 60.04863863769262,
- "right_hip": 51.063384878862486,
- "left_hip": 49.46045812442686,
- "left_knee": 63.31063738681314,
- "left_ankle": 78.66353142725545,
- "right_wrist": 90.59960691652287,
- "right_elbow": 46.310051560305666,
- "right_shoulder": 51.43439846398842,
- "left_shoulder": 46.774060401683954,
- "left_elbow": 52.35840937500452,
- "left_wrist": 96.1002743662463
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 47.53889109056079
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 57.51896901450996
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 58.113251646742306
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 60.41894110637095
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 54.639424209772734
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 86.68423372340176
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 61.6078069340802
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 69.43716592085373
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 62.18931723391786
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 49.789025904396624
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 58.200828804343125
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 61.77352621065217
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 52.42354213211462
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 55.80773243566966
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 82.6957633083744
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 72.59084332479958
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 65.60830819929198
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b1_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 317,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 88.03764661658637,
- "PA_MPJPE_mm": 60.41831973103175,
- "PCK3D_50mm": 0.5707225241671161,
- "PCK3D_100mm": 0.8710291941528717,
- "AUC_0_150mm": 0.6267325956390561,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 41.86047552118006,
- "left_hip": 42.87746228080597,
- "right_hip": 42.22412244129071,
- "spine1": 48.188429201129345,
- "left_knee": 63.009072619469066,
- "right_knee": 60.06388887125302,
- "spine2": 45.6612635124761,
- "left_ankle": 75.10158802289334,
- "right_ankle": 74.62566806654398,
- "spine3": 42.869560633075956,
- "left_foot": 83.32233408407829,
- "right_foot": 83.32434433985168,
- "neck": 37.969194898093825,
- "left_collar": 32.73430862696114,
- "right_collar": 32.70758478192214,
- "head": 44.69407719798415,
- "left_shoulder": 41.751708335400195,
- "right_shoulder": 40.98450048226682,
- "left_elbow": 56.93470945904146,
- "right_elbow": 54.485364031974775,
- "left_wrist": 91.92436666233198,
- "right_wrist": 90.3847561612622,
- "left_hand": 110.73003393521631,
- "right_hand": 111.61085937826014
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 98.29229831566185,
- "PA_MPJPE_mm": 65.72724775410568,
- "PCK3D_50mm": 0.4779043509699445,
- "PCK3D_100mm": 0.8419283906567647,
- "AUC_0_150mm": 0.5851683758891133,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 76.22315755409761,
- "right_knee": 57.10429066897349,
- "right_hip": 53.3302996272736,
- "left_hip": 50.49528759095279,
- "left_knee": 62.07025852422959,
- "left_ankle": 78.17132500057085,
- "right_wrist": 91.4025994500175,
- "right_elbow": 49.71897772884376,
- "right_shoulder": 54.844508860748824,
- "left_shoulder": 44.855648877089784,
- "left_elbow": 53.68729509391442,
- "left_wrist": 95.98851904785862,
- "neck": 47.00213842819814,
- "head": 105.28716210471057
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 94.23543525844572,
- "PA_MPJPE_mm": 61.71717122116292,
- "PCK3D_50mm": 0.5318805404583109,
- "PCK3D_100mm": 0.8692313543265707,
- "AUC_0_150mm": 0.613118029745996,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 73.99004636366409,
- "right_knee": 57.71620112647505,
- "right_hip": 50.81752252132023,
- "left_hip": 48.60204168180593,
- "left_knee": 62.484220902069396,
- "left_ankle": 75.69838806849187,
- "right_wrist": 88.73422403190237,
- "right_elbow": 45.69428239220022,
- "right_shoulder": 49.05244418299142,
- "left_shoulder": 45.36195634807564,
- "left_elbow": 50.347459933142595,
- "left_wrist": 92.10726710181629
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 48.737987046273616
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 54.53876487802224
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 57.004008879684775
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 58.7582569241043
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 53.31075470216184
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 81.73588282156733
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 57.0220368253622
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 65.3130003589239
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 58.95568203401182
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 48.178186455760326
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 56.615756538387764
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 60.82757533845631
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 53.74251429324582
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 54.00649452911257
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 87.01017409958928
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 71.5662222785782
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 62.670415646853975
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b2_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 323,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 81.47137269428303,
- "PA_MPJPE_mm": 55.10671038739444,
- "PCK3D_50mm": 0.6202979573773666,
- "PCK3D_100mm": 0.8975231299008423,
- "AUC_0_150mm": 0.6533997351866698,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 39.36010849169128,
- "left_hip": 40.85737979174461,
- "right_hip": 40.32155052288853,
- "spine1": 45.415902175118646,
- "left_knee": 57.1101678669842,
- "right_knee": 56.3085190094945,
- "spine2": 43.26081180020867,
- "left_ankle": 65.70489646075023,
- "right_ankle": 66.22069847353198,
- "spine3": 40.733456611265325,
- "left_foot": 73.37914609885806,
- "right_foot": 74.07316036625947,
- "neck": 35.23700631916798,
- "left_collar": 30.4791403729528,
- "right_collar": 30.294840572101982,
- "head": 41.51964958386219,
- "left_shoulder": 39.929657530072845,
- "right_shoulder": 40.67315912731431,
- "left_elbow": 52.258672960651815,
- "right_elbow": 50.83879597849367,
- "left_wrist": 81.81812222481747,
- "right_wrist": 79.77718664607215,
- "left_hand": 97.94287096347148,
- "right_hand": 99.04614934969221
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 92.19662005749441,
- "PA_MPJPE_mm": 61.07256711076473,
- "PCK3D_50mm": 0.5238026090409611,
- "PCK3D_100mm": 0.8645782089959163,
- "AUC_0_150mm": 0.6093027982252052,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 68.68430927684066,
- "right_knee": 54.28245487806341,
- "right_hip": 50.402177579858424,
- "left_hip": 47.214210627720874,
- "left_knee": 56.81284867943897,
- "left_ankle": 68.619174796597,
- "right_wrist": 81.46793100238136,
- "right_elbow": 46.03522077396803,
- "right_shoulder": 53.73217454398053,
- "left_shoulder": 42.24359227527992,
- "left_elbow": 49.59959465647656,
- "left_wrist": 86.28531949598393,
- "neck": 44.67793621782224,
- "head": 104.95899474629435
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 86.85590318363907,
- "PA_MPJPE_mm": 56.398798858932985,
- "PCK3D_50mm": 0.5846436819759643,
- "PCK3D_100mm": 0.8949957819911768,
- "AUC_0_150mm": 0.641068528033141,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 65.4341459850176,
- "right_knee": 54.452178177321166,
- "right_hip": 47.88876683854956,
- "left_hip": 45.339783373723954,
- "left_knee": 57.216817353819906,
- "left_ankle": 66.25766703026585,
- "right_wrist": 78.51618383557694,
- "right_elbow": 42.31466993655817,
- "right_shoulder": 47.458437853153725,
- "left_shoulder": 42.73458930880154,
- "left_elbow": 46.625083114505664,
- "left_wrist": 82.54726349990179
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.086831774757044
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 50.79304189716567
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 55.01404327464575
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 52.51953839080094
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 49.60438663821062
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 74.84134787823152
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 52.30344870859231
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 62.54974818979407
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 50.71454844421931
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 44.73794216602203
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 50.588119724721615
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 53.68047240239769
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 48.20121130421831
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 51.94136258521363
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 76.66533904224153
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 64.55963424980996
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 57.21101809850449
- }
- }
- },
- "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth": {
- "ckpt": "instanthmr_distill_train/runs/g9w_b4_288_s0/best_student_model_v3.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 275,
- "source": "ema",
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 78.371544963454,
- "PA_MPJPE_mm": 52.46441093656383,
- "PCK3D_50mm": 0.640195203916525,
- "PCK3D_100mm": 0.9095721141213403,
- "AUC_0_150mm": 0.665419387405039,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 37.703650633443225,
- "left_hip": 39.890466115786865,
- "right_hip": 38.80747110888024,
- "spine1": 44.70477190392395,
- "left_knee": 53.84400561125651,
- "right_knee": 53.47593074394138,
- "spine2": 43.46800025222844,
- "left_ankle": 60.097656737913184,
- "right_ankle": 60.743831191269145,
- "spine3": 40.73298156816384,
- "left_foot": 66.98009797134009,
- "right_foot": 67.43355667006932,
- "neck": 33.94815375586987,
- "left_collar": 29.172629374956063,
- "right_collar": 29.393683433487922,
- "head": 39.51376658383096,
- "left_shoulder": 39.00178264044968,
- "right_shoulder": 40.00744673989196,
- "left_elbow": 51.18440663793142,
- "right_elbow": 48.52372439487156,
- "left_wrist": 78.61968002410984,
- "right_wrist": 75.46663204066262,
- "left_hand": 93.47926887985425,
- "right_hand": 92.95226746340106
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 89.93831600001035,
- "PA_MPJPE_mm": 58.798339488549715,
- "PCK3D_50mm": 0.539615696919767,
- "PCK3D_100mm": 0.8780708981098974,
- "AUC_0_150mm": 0.6199196765188416,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 62.06488692464218,
- "right_knee": 51.32132617545651,
- "right_hip": 50.62142467020798,
- "left_hip": 47.9740082180099,
- "left_knee": 53.245626970864286,
- "left_ankle": 63.702636940871784,
- "right_wrist": 76.90816872799242,
- "right_elbow": 43.92232131570493,
- "right_shoulder": 52.239382444622215,
- "left_shoulder": 41.93487851199521,
- "left_elbow": 48.474456914116054,
- "left_wrist": 83.28254929365283,
- "neck": 43.77129630369932,
- "head": 103.71378942786035
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 84.45795256541614,
- "PA_MPJPE_mm": 53.97663924947734,
- "PCK3D_50mm": 0.6034103638551218,
- "PCK3D_100mm": 0.9086040465225629,
- "AUC_0_150mm": 0.6522139081424299,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 59.46453388508666,
- "right_knee": 51.48079145613311,
- "right_hip": 47.67650288377584,
- "left_hip": 45.84220884184089,
- "left_knee": 53.54838526715057,
- "left_ankle": 60.5440484434169,
- "right_wrist": 75.01427366646412,
- "right_elbow": 41.285969727750256,
- "right_shoulder": 46.0138135882835,
- "left_shoulder": 41.13717377891797,
- "left_elbow": 45.7257632369426,
- "left_wrist": 79.9862062179657
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 44.05573677671157
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 49.64271345594448
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 52.50048316704524
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 51.32739084987526
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 48.718216556000215
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 69.4547122752743
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 47.1710736464845
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 56.521316980645025
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 48.10371416575587
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 42.47769059105345
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 46.09252488295609
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 50.092129509254946
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 46.45775058529342
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 49.11901693755719
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 76.54197014285722
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 60.60633220403424
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 56.103027281090306
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/gen9w_latency_local.json b/benchmark/results/gen9w_latency_local.json
deleted file mode 100644
index c34ad22..0000000
--- a/benchmark/results/gen9w_latency_local.json
+++ /dev/null
@@ -1,152 +0,0 @@
-{
- "machine": {
- "cpu": "x86_64",
- "onnxruntime": "1.26.0"
- },
- "rows": [
- {
- "model": "g8h_lr1x_e600",
- "image_size": 224,
- "cpu_all": {
- "median_ms": 18.616634501086082,
- "p90_ms": 18.658106801012764
- },
- "cpu_1t": {
- "median_ms": 63.59058800080675,
- "p90_ms": 64.05181389709469
- },
- "cuda": {
- "median_ms": 3.158963500027312,
- "p90_ms": 3.220398101984756
- }
- },
- {
- "model": "g9w_b0_224_s0",
- "image_size": 224,
- "cpu_all": {
- "median_ms": 10.86602900068101,
- "p90_ms": 10.913577198516577
- },
- "cpu_1t": {
- "median_ms": 27.841736000482342,
- "p90_ms": 28.02969779986597
- },
- "cuda": {
- "median_ms": 2.5402979990758467,
- "p90_ms": 2.711577599620796
- }
- },
- {
- "model": "g9w_b0_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 12.715936500171665,
- "p90_ms": 12.816705599834677
- },
- "cpu_1t": {
- "median_ms": 33.3640889984963,
- "p90_ms": 33.885154298332054
- },
- "cuda": {
- "median_ms": 2.619978500661091,
- "p90_ms": 2.821950801080675
- }
- },
- {
- "model": "g9w_b1_224_s0",
- "image_size": 224,
- "cpu_all": {
- "median_ms": 11.776877001466346,
- "p90_ms": 11.822248599855811
- },
- "cpu_1t": {
- "median_ms": 30.819064999377588,
- "p90_ms": 30.96474520280026
- },
- "cuda": {
- "median_ms": 2.5805280019994825,
- "p90_ms": 2.6298580996808596
- }
- },
- {
- "model": "g9w_b1_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 13.95525450061541,
- "p90_ms": 14.018068499717629
- },
- "cpu_1t": {
- "median_ms": 38.344823000443284,
- "p90_ms": 38.469571399036795
- },
- "cuda": {
- "median_ms": 2.666424999915762,
- "p90_ms": 2.8537020974908955
- }
- },
- {
- "model": "g9w_b2_224_s0",
- "image_size": 224,
- "cpu_all": {
- "median_ms": 14.896990001943777,
- "p90_ms": 14.940757200747612
- },
- "cpu_1t": {
- "median_ms": 41.247660001317854,
- "p90_ms": 41.6223343971069
- },
- "cuda": {
- "median_ms": 2.8968929982511327,
- "p90_ms": 2.979932002199348
- }
- },
- {
- "model": "g9w_b2_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 18.35872800074867,
- "p90_ms": 18.44849649896787
- },
- "cpu_1t": {
- "median_ms": 55.231083499165834,
- "p90_ms": 55.82707279900205
- },
- "cuda": {
- "median_ms": 3.0081754994171206,
- "p90_ms": 3.0360214008396724
- }
- },
- {
- "model": "g9w_b4_224_s0",
- "image_size": 224,
- "cpu_all": {
- "median_ms": 18.748270000287448,
- "p90_ms": 18.80032479930378
- },
- "cpu_1t": {
- "median_ms": 63.839815999017446,
- "p90_ms": 64.31525379884988
- },
- "cuda": {
- "median_ms": 3.210486000170931,
- "p90_ms": 3.2762115999503294
- }
- },
- {
- "model": "g9w_b4_288_s0",
- "image_size": 288,
- "cpu_all": {
- "median_ms": 24.76131249932223,
- "p90_ms": 24.81809230048384
- },
- "cpu_1t": {
- "median_ms": 91.02867250112467,
- "p90_ms": 91.98357429922908
- },
- "cuda": {
- "median_ms": 3.4833885001717135,
- "p90_ms": 3.739856000538566
- }
- }
- ]
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_3dpw_test_detector.json b/benchmark/results/oldhf_3dpw_test_detector.json
deleted file mode 100644
index c6477b3..0000000
--- a/benchmark/results/oldhf_3dpw_test_detector.json
+++ /dev/null
@@ -1,82 +0,0 @@
-{
- "models/instanthmr.pth|h36m": {
- "ckpt": "models/instanthmr.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 35463,
- "n_missed": 139,
- "miss_rate": 0.003919578151876604,
- "mean_iou": 0.6723040298246045
- },
- "epoch": 297,
- "source": null,
- "landmarks": "fitted",
- "n": 35324,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 103.32592886676912,
- "PA_MPJPE_mm": 67.39686620625218,
- "PCK3D_50mm": 0.41548239157513306,
- "PCK3D_100mm": 0.8220311564779915,
- "AUC_0_150mm": 0.5610869778487702,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 67.61611410709556,
- "right_knee": 47.792614967808404,
- "right_hip": 94.98743357829603,
- "left_hip": 83.00371183120897,
- "left_knee": 46.52167480767435,
- "left_ankle": 72.1225141920695,
- "right_wrist": 69.41745380360979,
- "right_elbow": 59.03659936739649,
- "right_shoulder": 48.68815732228623,
- "left_shoulder": 50.66584534814893,
- "left_elbow": 68.51360840531422,
- "left_wrist": 73.53655399280728,
- "neck": 33.40453900141097,
- "head": 128.2493061624039
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35324,
- "MPJPE_mm": 100.91215875891452,
- "PA_MPJPE_mm": 62.111111466516085,
- "PCK3D_50mm": 0.4610533914618956,
- "PCK3D_100mm": 0.8798243875740762,
- "AUC_0_150mm": 0.5930841591753392,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 63.54257652879037,
- "right_knee": 49.05904315865144,
- "right_hip": 94.30946615340643,
- "left_hip": 82.30521205092276,
- "left_knee": 46.42816491967229,
- "left_ankle": 66.91255425099824,
- "right_wrist": 66.74391417286965,
- "right_elbow": 49.86991598885744,
- "right_shoulder": 50.20272864795174,
- "left_shoulder": 50.40743732770012,
- "left_elbow": 55.41381184275506,
- "left_wrist": 70.13851255561745
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35324,
- "MPJPE_mm": 69.30357814360359,
- "PA_MPJPE_mm": 43.065274477978825,
- "PCK3D_50mm": 0.7245984114402187,
- "PCK3D_100mm": 0.9313315916333694,
- "AUC_0_150mm": 0.7151531941051813
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_3dpw_test_gt-joints.json b/benchmark/results/oldhf_3dpw_test_gt-joints.json
deleted file mode 100644
index 546f8e4..0000000
--- a/benchmark/results/oldhf_3dpw_test_gt-joints.json
+++ /dev/null
@@ -1,77 +0,0 @@
-{
- "models/instanthmr.pth|h36m": {
- "ckpt": "models/instanthmr.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 297,
- "source": null,
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 110.012365166235,
- "PA_MPJPE_mm": 70.90676985482673,
- "PCK3D_50mm": 0.38817922905563546,
- "PCK3D_100mm": 0.7987620900657023,
- "AUC_0_150mm": 0.5419692309931387,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 74.89903885678324,
- "right_knee": 51.88532413888325,
- "right_hip": 96.00694968006601,
- "left_hip": 85.3536622172935,
- "left_knee": 49.639467464938136,
- "left_ankle": 80.47285733367677,
- "right_wrist": 75.00680437251044,
- "right_elbow": 61.57764357846618,
- "right_shoulder": 47.75851339017392,
- "left_shoulder": 49.955880954807256,
- "left_elbow": 70.7527787938197,
- "left_wrist": 79.18896798781593,
- "neck": 31.410180122258005,
- "head": 138.78670907608202
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 107.75829489158814,
- "PA_MPJPE_mm": 65.23201733286855,
- "PCK3D_50mm": 0.42744550658432734,
- "PCK3D_100mm": 0.8573959713880195,
- "AUC_0_150mm": 0.5748570598646051,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 71.86719782335308,
- "right_knee": 53.21057347820458,
- "right_hip": 96.39676285373322,
- "left_hip": 85.29202291120973,
- "left_knee": 49.36104632454278,
- "left_ankle": 74.75273394119331,
- "right_wrist": 71.12477896619525,
- "right_elbow": 51.707886994913466,
- "right_shoulder": 49.302906248619955,
- "left_shoulder": 48.56602273783773,
- "left_elbow": 56.85504272982152,
- "left_wrist": 74.347232984798
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 76.93266155571972,
- "PA_MPJPE_mm": 47.119662589986866,
- "PCK3D_50mm": 0.6759056723103758,
- "PCK3D_100mm": 0.9154571565535105,
- "AUC_0_150mm": 0.6907323151500407
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_3dpw_test_pve.json b/benchmark/results/oldhf_3dpw_test_pve.json
deleted file mode 100644
index b7e4497..0000000
--- a/benchmark/results/oldhf_3dpw_test_pve.json
+++ /dev/null
@@ -1,50 +0,0 @@
-{
- "/tmp/claude-1000/-home-madjel-Projects-gitpackages-sam3d-student-release/8b8b1032-085a-4960-bbc2-9b3487321cd3/scratchpad/ck/oldhf/oldhf/instanthmr.pth": {
- "ckpt": "/tmp/claude-1000/-home-madjel-Projects-gitpackages-sam3d-student-release/8b8b1032-085a-4960-bbc2-9b3487321cd3/scratchpad/ck/oldhf/oldhf/instanthmr.pth",
- "dataset": "3DPW/test",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 35463,
- "fit_iters": 0,
- "epoch": 297,
- "source": null,
- "fit_residual_mm": null,
- "PVE_direct_mm": 89.7278876608327,
- "PVE_published_mm": 89.7278876608327,
- "PA_PVE_direct_mm": 63.18450334010902,
- "J14_h36m_direct_MPJPE_mm": 76.23746490478516,
- "J14_h36m_direct_PA_MPJPE_mm": 47.940958757166,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "downtown_arguing_00": 44.45625727912758,
- "downtown_bar_00": 69.03086528635365,
- "downtown_bus_00": 52.346133759957645,
- "downtown_cafe_00": 60.25013649405067,
- "downtown_car_00": 58.477784617223975,
- "downtown_crossStreets_00": 48.979087579388654,
- "downtown_downstairs_00": 38.70197659597242,
- "downtown_enterShop_00": 34.82307313239689,
- "downtown_rampAndStairs_00": 45.06696299622308,
- "downtown_runForBus_00": 45.34645120118713,
- "downtown_runForBus_01": 54.91683275566302,
- "downtown_sitOnStairs_00": 40.223382568105094,
- "downtown_stairs_00": 64.58473474015543,
- "downtown_upstairs_00": 38.6965501233381,
- "downtown_walkBridge_01": 35.31036874642173,
- "downtown_walkUphill_00": 35.15567650760603,
- "downtown_walking_00": 46.48699201106517,
- "downtown_warmWelcome_00": 52.04370519656569,
- "downtown_weeklyMarket_00": 40.66608731187835,
- "downtown_windowShopping_00": 33.269417985470476,
- "flat_guitar_01": 54.084185989634996,
- "flat_packBags_00": 51.2095533258883,
- "office_phoneCall_00": 34.51863126759615,
- "outdoors_fencing_01": 44.212615113735296
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_emdb1_annotated.json b/benchmark/results/oldhf_emdb1_annotated.json
deleted file mode 100644
index 32104c7..0000000
--- a/benchmark/results/oldhf_emdb1_annotated.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "models/instanthmr.pth": {
- "ckpt": "models/instanthmr.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 297,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 134.77861626636104,
- "PA_MPJPE_mm": 88.8286799789983,
- "PCK3D_50mm": 0.40693758729895313,
- "PCK3D_100mm": 0.7345005462667165,
- "AUC_0_150mm": 0.5138784423836947,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 51.83661012145418,
- "left_hip": 52.93678934335618,
- "right_hip": 52.08922774267161,
- "spine1": 58.902822729577174,
- "left_knee": 95.95193283574329,
- "right_knee": 90.1856973827214,
- "spine2": 54.16195269027105,
- "left_ankle": 124.77722541240753,
- "right_ankle": 124.92705915728261,
- "spine3": 52.84619150528256,
- "left_foot": 137.3734386701164,
- "right_foot": 137.26173871588819,
- "neck": 49.65528886793337,
- "left_collar": 44.98042467141282,
- "right_collar": 47.63784127426329,
- "head": 58.67113657459795,
- "left_shoulder": 50.00546945724675,
- "right_shoulder": 53.63757409008912,
- "left_elbow": 80.54757524895214,
- "right_elbow": 76.80041191588333,
- "left_wrist": 144.8416941678954,
- "right_wrist": 143.35828674911,
- "left_hand": 172.70870553356545,
- "right_hand": 175.7932246382368
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 142.38689300595675,
- "PA_MPJPE_mm": 91.04509501726206,
- "PCK3D_50mm": 0.32044025343614607,
- "PCK3D_100mm": 0.7117223107971148,
- "AUC_0_150mm": 0.48090797348017367,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 125.3432507172952,
- "right_knee": 85.52201200928039,
- "right_hip": 60.781776686724776,
- "left_hip": 58.09785591847596,
- "left_knee": 91.55907871674756,
- "left_ankle": 125.8074762419021,
- "right_wrist": 145.28252028349226,
- "right_elbow": 72.49727761542921,
- "right_shoulder": 66.94377356280948,
- "left_shoulder": 54.44635598740269,
- "left_elbow": 79.11913840422415,
- "left_wrist": 148.10876098677454,
- "neck": 57.64458751441248,
- "head": 103.47746559669804
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 141.062043644966,
- "PA_MPJPE_mm": 91.72135580070635,
- "PCK3D_50mm": 0.34096032305798724,
- "PCK3D_100mm": 0.7122661079533668,
- "AUC_0_150mm": 0.48570661573828083,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 123.52463466317387,
- "right_knee": 85.19354937922041,
- "right_hip": 60.66533235262749,
- "left_hip": 58.52743505188823,
- "left_knee": 91.39620439841504,
- "left_ankle": 124.64484804767274,
- "right_wrist": 140.62880696852514,
- "right_elbow": 67.5369716655591,
- "right_shoulder": 69.10975963776751,
- "left_shoulder": 61.45879387179384,
- "left_elbow": 74.28845829591364,
- "left_wrist": 143.68147527591924
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 69.30280266343286
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 82.44581217378295
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 75.20698806342318
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 76.26400153079399
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 67.69024954217736
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 132.55233432015575
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 86.72818302190208
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 92.64173218168398
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 88.35376280187049
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 68.0087347729235
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 89.79694751092585
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 93.85308031825565
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 74.61069207914906
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 75.5846972764283
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 129.55043134472947
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 97.02440249066993
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 104.81657232860918
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_emdb1_detector.json b/benchmark/results/oldhf_emdb1_detector.json
deleted file mode 100644
index c028f92..0000000
--- a/benchmark/results/oldhf_emdb1_detector.json
+++ /dev/null
@@ -1,180 +0,0 @@
-{
- "models/instanthmr.pth": {
- "ckpt": "models/instanthmr.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "detector",
- "detector": {
- "n_frames": 24103,
- "n_missed": 98,
- "miss_rate": 0.004065883914865369,
- "mean_iou": 0.6621514634695929
- },
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 297,
- "source": null,
- "n": 24005,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24005,
- "MPJPE_mm": 123.70432641133017,
- "PA_MPJPE_mm": 80.84527156082444,
- "PCK3D_50mm": 0.45102756370200653,
- "PCK3D_100mm": 0.7757706727765049,
- "AUC_0_150mm": 0.5461749680286141,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.899070882070184,
- "left_hip": 50.12679987819629,
- "right_hip": 49.438309576736906,
- "spine1": 55.88856019726661,
- "left_knee": 86.5904807684985,
- "right_knee": 80.33398331121661,
- "spine2": 53.441574168713665,
- "left_ankle": 108.09811166332233,
- "right_ankle": 109.27746645614098,
- "spine3": 51.60868434787386,
- "left_foot": 119.42913688366353,
- "right_foot": 119.335319352363,
- "neck": 45.58282854691961,
- "left_collar": 41.55650821907194,
- "right_collar": 43.588494463606416,
- "head": 54.39861526848465,
- "left_shoulder": 48.83370813905048,
- "right_shoulder": 50.866692942443436,
- "left_elbow": 75.1236692008074,
- "right_elbow": 70.17376742078018,
- "left_wrist": 132.11284041789804,
- "right_wrist": 129.00224265975592,
- "left_hand": 157.50644788704415,
- "right_hand": 159.07320480786086
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24005,
- "MPJPE_mm": 131.5663759814014,
- "PA_MPJPE_mm": 84.21333519377056,
- "PCK3D_50mm": 0.3602701818073616,
- "PCK3D_100mm": 0.7475644954920105,
- "AUC_0_150mm": 0.508986223108281,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 109.61516977755971,
- "right_knee": 76.56669971222199,
- "right_hip": 59.66733428672744,
- "left_hip": 57.138667080368926,
- "left_knee": 83.08501331845957,
- "left_ankle": 110.33762606694938,
- "right_wrist": 130.76926976572705,
- "right_elbow": 65.58589468592399,
- "right_shoulder": 64.38525518372155,
- "left_shoulder": 53.509646431976506,
- "left_elbow": 72.81454837860827,
- "left_wrist": 135.47014054323978,
- "neck": 53.2165349153175,
- "head": 106.82489256598626
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24005,
- "MPJPE_mm": 129.23845566908562,
- "PA_MPJPE_mm": 83.41425563838337,
- "PCK3D_50mm": 0.3894327570645005,
- "PCK3D_100mm": 0.7585260015274595,
- "AUC_0_150mm": 0.5213398642307941,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 107.9773384185184,
- "right_knee": 76.6728644999444,
- "right_hip": 58.73904301714058,
- "left_hip": 56.66311638602539,
- "left_knee": 83.08625736644235,
- "left_ankle": 108.91124166127886,
- "right_wrist": 127.04714959401281,
- "right_elbow": 60.83058178633379,
- "right_shoulder": 63.70746358106421,
- "left_shoulder": 58.606501778985844,
- "left_elbow": 67.59926719149338,
- "left_wrist": 131.13024237936037
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1262,
- "PA_MPJPE_mm": 60.3833656590437
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 70.1433000974086
- },
- "P3_31_outdoor_workout": {
- "n": 1140,
- "PA_MPJPE_mm": 69.74535904886748
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 69.13001585549337
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 56.81933891072583
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 121.23950311603548
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 76.48414260111205
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1555,
- "PA_MPJPE_mm": 86.33727401096382
- },
- "P6_50_outdoor_workout": {
- "n": 1531,
- "PA_MPJPE_mm": 80.41984894871094
- },
- "P6_51_outdoor_dancing": {
- "n": 1425,
- "PA_MPJPE_mm": 64.17377020833929
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1554,
- "PA_MPJPE_mm": 87.5786218011748
- },
- "P7_59_outdoor_rom": {
- "n": 1836,
- "PA_MPJPE_mm": 84.35149543000469
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 70.03468385885563
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 69.00515052063099
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 114.90815368255527
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 90.39503373777713
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 96.44300020573198
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_emdb1_gt-joints.json b/benchmark/results/oldhf_emdb1_gt-joints.json
deleted file mode 100644
index e42a73b..0000000
--- a/benchmark/results/oldhf_emdb1_gt-joints.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "models/instanthmr.pth": {
- "ckpt": "models/instanthmr.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 297,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 130.3854596588656,
- "PA_MPJPE_mm": 82.0571670163171,
- "PCK3D_50mm": 0.4425244437068691,
- "PCK3D_100mm": 0.7705420487076298,
- "AUC_0_150mm": 0.5411218498210414,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.59187920665972,
- "left_hip": 50.4126501457034,
- "right_hip": 49.762050911165666,
- "spine1": 54.612692164149415,
- "left_knee": 90.09166365427974,
- "right_knee": 82.24455682927075,
- "spine2": 53.449334715576,
- "left_ankle": 111.63578442839923,
- "right_ankle": 111.96051253671182,
- "spine3": 52.00669208342192,
- "left_foot": 123.03845499470782,
- "right_foot": 123.76316525209222,
- "neck": 47.53526549385952,
- "left_collar": 42.51789215922212,
- "right_collar": 43.91447960243653,
- "head": 57.02544550539255,
- "left_shoulder": 48.29854874415506,
- "right_shoulder": 50.254633039524,
- "left_elbow": 75.6921061879758,
- "right_elbow": 71.40956568788079,
- "left_wrist": 132.12530841342553,
- "right_wrist": 130.20406948638814,
- "left_hand": 158.51340449003933,
- "right_hand": 160.311852659173
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 137.4546838606729,
- "PA_MPJPE_mm": 85.2887885622615,
- "PCK3D_50mm": 0.3544016453197883,
- "PCK3D_100mm": 0.7423349790482513,
- "AUC_0_150mm": 0.5048719483644597,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 112.90230744160056,
- "right_knee": 77.76579851390298,
- "right_hip": 59.7360313981596,
- "left_hip": 57.117470395531804,
- "left_knee": 85.69734064895307,
- "left_ankle": 113.69040067031324,
- "right_wrist": 131.2842203111102,
- "right_elbow": 65.73414330792406,
- "right_shoulder": 64.85578426165553,
- "left_shoulder": 53.93985198484884,
- "left_elbow": 72.82189014455065,
- "left_wrist": 135.33234242828175,
- "neck": 55.04960700928208,
- "head": 108.11585135554654
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 135.1134120391211,
- "PA_MPJPE_mm": 84.22686772788774,
- "PCK3D_50mm": 0.3867672074015683,
- "PCK3D_100mm": 0.7555733034615332,
- "AUC_0_150mm": 0.5193542141499362,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 111.3023944097616,
- "right_knee": 78.06397585463169,
- "right_hip": 58.00096104384721,
- "left_hip": 55.89400725476661,
- "left_knee": 85.73769890796152,
- "left_ankle": 111.54025285994577,
- "right_wrist": 127.81832219553837,
- "right_elbow": 61.883637076937525,
- "right_shoulder": 62.992167213801736,
- "left_shoulder": 57.951701927944086,
- "left_elbow": 68.24978921396766,
- "left_wrist": 131.28750477554897
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 62.25777058529678
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 72.58083553110158
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 70.85729372012992
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 71.1220512553712
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 59.81773999862998
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 122.87846608625289
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 77.60790822411231
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 84.46367417675029
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 80.2575222618084
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 64.7979797339095
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 89.52530190150982
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 87.2869229592149
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 71.97161263079856
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 70.63465836882817
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 119.51908090607776
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 93.44620858723786
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 91.95960957155165
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_emdb1_pve.json b/benchmark/results/oldhf_emdb1_pve.json
deleted file mode 100644
index 0527dc0..0000000
--- a/benchmark/results/oldhf_emdb1_pve.json
+++ /dev/null
@@ -1,43 +0,0 @@
-{
- "/tmp/claude-1000/-home-madjel-Projects-gitpackages-sam3d-student-release/8b8b1032-085a-4960-bbc2-9b3487321cd3/scratchpad/ck/oldhf/oldhf/instanthmr.pth": {
- "ckpt": "/tmp/claude-1000/-home-madjel-Projects-gitpackages-sam3d-student-release/8b8b1032-085a-4960-bbc2-9b3487321cd3/scratchpad/ck/oldhf/oldhf/instanthmr.pth",
- "dataset": "EMDB1",
- "bbox": "gt-joints",
- "stride": 1,
- "n": 24103,
- "fit_iters": 0,
- "epoch": 297,
- "source": null,
- "fit_residual_mm": null,
- "PVE_direct_mm": 144.21723528444417,
- "PVE_published_mm": 150.48613197099746,
- "PA_PVE_direct_mm": 95.13054542358611,
- "J14_h36m_direct_MPJPE_mm": 125.79401397705078,
- "J14_h36m_direct_PA_MPJPE_mm": 74.73550725538553,
- "MPJPE_mm": null,
- "PA_MPJPE_mm": null,
- "PVE_mm": null,
- "PA_PVE_mm": null,
- "J14_h36m_viamesh_MPJPE_mm": null,
- "J14_h36m_viamesh_PA_MPJPE_mm": null,
- "per_sequence_PA_MPJPE_mm": {
- "P1_14_outdoor_climb": 57.46953455053445,
- "P2_23_outdoor_hug_tree": 65.39207672154404,
- "P3_31_outdoor_workout": 68.43899828016741,
- "P3_32_outdoor_soccer_warmup_a": 66.03507987022952,
- "P3_33_outdoor_soccer_warmup_b": 56.33767162537286,
- "P5_42_indoor_dancing": 115.1491631769016,
- "P5_44_indoor_rom": 72.83147589287064,
- "P6_49_outdoor_big_stairs_down": 76.44488593620376,
- "P6_50_outdoor_workout": 75.72554039848404,
- "P6_51_outdoor_dancing": 58.5532042313216,
- "P7_57_outdoor_rock_chair": 82.00943258632788,
- "P7_59_outdoor_rom": 79.89864730225953,
- "P7_60_outdoor_workout": 64.36161008820217,
- "P8_64_outdoor_skateboard": 64.46712642460376,
- "P8_68_outdoor_handstand": 102.39527335660438,
- "P8_69_outdoor_cartwheel": 85.55792106233463,
- "P9_76_outdoor_sitting": 79.03644060252135
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_headreadout_3dpw_test_gt-joints.json b/benchmark/results/oldhf_headreadout_3dpw_test_gt-joints.json
deleted file mode 100644
index 0629359..0000000
--- a/benchmark/results/oldhf_headreadout_3dpw_test_gt-joints.json
+++ /dev/null
@@ -1,77 +0,0 @@
-{
- "models/instanthmr.pth|h36m": {
- "ckpt": "models/instanthmr.pth",
- "gt": "h36m",
- "split": "test",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "epoch": 297,
- "source": null,
- "landmarks": "fitted",
- "n": 35463,
- "n_dropped": 0,
- "results": [
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 108.43727806914217,
- "PA_MPJPE_mm": 70.83225990468031,
- "PCK3D_50mm": 0.38019505238860624,
- "PCK3D_100mm": 0.8021620119158398,
- "AUC_0_150mm": 0.5424949947832952,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 75.80475604312385,
- "right_knee": 53.36046621385818,
- "right_hip": 95.75183428859872,
- "left_hip": 85.86460382802085,
- "left_knee": 49.69196292616157,
- "left_ankle": 80.280954805861,
- "right_wrist": 73.84783750155245,
- "right_elbow": 60.65879670753468,
- "right_shoulder": 50.071056042068136,
- "left_shoulder": 52.57758635898964,
- "left_elbow": 69.32932382465516,
- "left_wrist": 78.30692805192474,
- "neck": 31.18762148228823,
- "head": 134.91791059088712
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 35463,
- "MPJPE_mm": 106.06813862380264,
- "PA_MPJPE_mm": 65.65687948728726,
- "PCK3D_50mm": 0.4224802376185508,
- "PCK3D_100mm": 0.85576046395774,
- "AUC_0_150mm": 0.5726258990515329,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 73.4848517283021,
- "right_knee": 54.495293405449914,
- "right_hip": 96.37094429621642,
- "left_hip": 85.83102612703178,
- "left_knee": 49.69621612888227,
- "left_ankle": 75.53604873299703,
- "right_wrist": 70.46667053874846,
- "right_elbow": 50.698769863166554,
- "right_shoulder": 50.62429468093989,
- "left_shoulder": 50.680928224574465,
- "left_elbow": 55.97742657968803,
- "left_wrist": 74.02008354145013
- }
- },
- {
- "joint_set": "J14+adapter",
- "num_joints": 14,
- "num_samples": 35463,
- "MPJPE_mm": 75.11661512370893,
- "PA_MPJPE_mm": 47.289477012950144,
- "PCK3D_50mm": 0.6752268964433756,
- "PCK3D_100mm": 0.91397875451678,
- "AUC_0_150mm": 0.6897150284888346
- }
- ]
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_headreadout_emdb1_annotated.json b/benchmark/results/oldhf_headreadout_emdb1_annotated.json
deleted file mode 100644
index baab7bc..0000000
--- a/benchmark/results/oldhf_headreadout_emdb1_annotated.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "models/instanthmr.pth": {
- "ckpt": "models/instanthmr.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "annotated",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 297,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 127.52831384645937,
- "PA_MPJPE_mm": 88.41258514422454,
- "PCK3D_50mm": 0.41353946258418733,
- "PCK3D_100mm": 0.7401689277959866,
- "AUC_0_150mm": 0.5183877079505117,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 52.29486666208945,
- "left_hip": 53.9684319649705,
- "right_hip": 53.45872671314912,
- "spine1": 59.729367133966676,
- "left_knee": 98.52406190325775,
- "right_knee": 91.18311864526042,
- "spine2": 53.59389638044434,
- "left_ankle": 119.31176255519384,
- "right_ankle": 121.41424267065021,
- "spine3": 51.94264363081134,
- "left_foot": 131.52310943568807,
- "right_foot": 133.73939454994098,
- "neck": 49.893285559436045,
- "left_collar": 44.48921758531999,
- "right_collar": 48.523798568700585,
- "head": 59.1544690833575,
- "left_shoulder": 52.31660686899669,
- "right_shoulder": 55.95176977479212,
- "left_elbow": 81.92620522350283,
- "right_elbow": 77.45632607531645,
- "left_wrist": 142.60539880031612,
- "right_wrist": 142.56069720292544,
- "left_hand": 171.23363421761624,
- "right_hand": 175.10701225568326
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 135.38170439706582,
- "PA_MPJPE_mm": 90.82384239221406,
- "PCK3D_50mm": 0.32767705264904784,
- "PCK3D_100mm": 0.7174892277784034,
- "AUC_0_150mm": 0.48585333947951104,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 122.46743661543225,
- "right_knee": 86.3986348837229,
- "right_hip": 60.208882724940246,
- "left_hip": 58.223108511844984,
- "left_knee": 94.43762324697511,
- "left_ankle": 120.27137200654727,
- "right_wrist": 145.68877529192346,
- "right_elbow": 74.06158782946251,
- "right_shoulder": 67.99780725300823,
- "left_shoulder": 55.70058849788488,
- "left_elbow": 78.95979021661455,
- "left_wrist": 146.23419683531938,
- "neck": 58.78169216988906,
- "head": 102.10229740743222
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 134.00259207635108,
- "PA_MPJPE_mm": 91.53071504798666,
- "PCK3D_50mm": 0.34936868163022583,
- "PCK3D_100mm": 0.717393408842606,
- "AUC_0_150mm": 0.4906061753790519,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 120.99923995486785,
- "right_knee": 85.84755037972249,
- "right_hip": 60.12211888872253,
- "left_hip": 58.49716822453872,
- "left_knee": 94.10945251046353,
- "left_ankle": 119.24936887172609,
- "right_wrist": 141.16002695188703,
- "right_elbow": 69.71593055574931,
- "right_shoulder": 69.93413432211214,
- "left_shoulder": 62.10990609064722,
- "left_elbow": 74.7489656588862,
- "left_wrist": 141.87471816651683
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 67.57972784336891
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 82.38557614332494
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 77.21442549453639
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 75.26850241955498
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 65.47960274663882
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 129.8345494187858
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 87.32620702037039
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 90.51085742395307
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 87.66749832698237
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 67.34053566971794
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 94.10576121506185
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 92.14862452965812
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 73.88350531039548
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 73.64002984697416
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 128.77990320646467
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 98.62074422973846
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 106.62465642813024
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/results/oldhf_headreadout_emdb1_gt-joints.json b/benchmark/results/oldhf_headreadout_emdb1_gt-joints.json
deleted file mode 100644
index ed3eca8..0000000
--- a/benchmark/results/oldhf_headreadout_emdb1_gt-joints.json
+++ /dev/null
@@ -1,175 +0,0 @@
-{
- "models/instanthmr.pth": {
- "ckpt": "models/instanthmr.pth",
- "dataset": "EMDB1",
- "split": "emdb1",
- "stride": 1,
- "bbox": "gt-joints",
- "detector": null,
- "adapter": "benchmark/results/adapter_smpl24_teacher.npz",
- "landmarks": "fitted",
- "epoch": 297,
- "source": null,
- "n": 24103,
- "n_dropped": 0,
- "num_sequences": 17,
- "results": [
- {
- "joint_set": "SMPL24+adapter",
- "num_joints": 24,
- "num_samples": 24103,
- "MPJPE_mm": 122.98126139327248,
- "PA_MPJPE_mm": 80.83828545422334,
- "PCK3D_50mm": 0.45887614266550497,
- "PCK3D_100mm": 0.7812184513684327,
- "AUC_0_150mm": 0.5502327823378075,
- "per_joint_PA_MPJPE_mm": {
- "pelvis": 48.14923705315608,
- "left_hip": 50.40042943901286,
- "right_hip": 50.157814344638496,
- "spine1": 54.29266901039462,
- "left_knee": 90.74858534948434,
- "right_knee": 81.63538514380706,
- "spine2": 51.684974194598226,
- "left_ankle": 106.3111797208181,
- "right_ankle": 108.10014388518466,
- "spine3": 50.2483923259102,
- "left_foot": 117.24135424909423,
- "right_foot": 119.38007719594698,
- "neck": 46.25778678930155,
- "left_collar": 41.39661879727389,
- "right_collar": 43.885379314110985,
- "head": 54.7843646761137,
- "left_shoulder": 50.40893747715841,
- "right_shoulder": 52.22515750995223,
- "left_elbow": 75.87244064188248,
- "right_elbow": 71.8592466990922,
- "left_wrist": 129.45478495000754,
- "right_wrist": 129.3383773832378,
- "left_hand": 156.6837471735221,
- "right_hand": 159.60176757765916
- }
- },
- {
- "joint_set": "J14",
- "num_joints": 14,
- "num_samples": 24103,
- "MPJPE_mm": 130.5664625715232,
- "PA_MPJPE_mm": 84.33579639460675,
- "PCK3D_50mm": 0.3684573941595889,
- "PCK3D_100mm": 0.7533353880074205,
- "AUC_0_150mm": 0.5134960349697372,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 109.4612208695725,
- "right_knee": 77.4242639097585,
- "right_hip": 58.19187417248823,
- "left_hip": 56.1060460156112,
- "left_knee": 86.47715736618562,
- "left_ankle": 108.27418676120591,
- "right_wrist": 131.39193970453832,
- "right_elbow": 66.99422869847527,
- "right_shoulder": 65.40369990695048,
- "left_shoulder": 55.13806147273237,
- "left_elbow": 72.08841055625088,
- "left_wrist": 133.12596328060803,
- "neck": 55.34843376032666,
- "head": 105.27566304979041
- }
- },
- {
- "joint_set": "J12",
- "num_joints": 12,
- "num_samples": 24103,
- "MPJPE_mm": 128.1254365008071,
- "PA_MPJPE_mm": 83.45991771534707,
- "PCK3D_50mm": 0.4018206585625579,
- "PCK3D_100mm": 0.7634111936273493,
- "AUC_0_150mm": 0.5270369681371926,
- "per_joint_PA_MPJPE_mm": {
- "right_ankle": 108.2210577739729,
- "right_knee": 77.55027929654213,
- "right_hip": 56.76159622502863,
- "left_hip": 55.128970396144354,
- "left_knee": 86.3975925162686,
- "left_ankle": 106.61321908556592,
- "right_wrist": 128.25350289371067,
- "right_elbow": 63.56421772013803,
- "right_shoulder": 63.30385673394593,
- "left_shoulder": 58.01321225872592,
- "left_elbow": 68.1198069406846,
- "left_wrist": 129.59170074343703
- }
- }
- ],
- "per_sequence_SMPL24_adapter": {
- "P1_14_outdoor_climb": {
- "n": 1270,
- "PA_MPJPE_mm": 59.96084782121447
- },
- "P2_23_outdoor_hug_tree": {
- "n": 1086,
- "PA_MPJPE_mm": 70.5604715190046
- },
- "P3_31_outdoor_workout": {
- "n": 1216,
- "PA_MPJPE_mm": 70.47761945749234
- },
- "P3_32_outdoor_soccer_warmup_a": {
- "n": 1084,
- "PA_MPJPE_mm": 68.34712277884026
- },
- "P3_33_outdoor_soccer_warmup_b": {
- "n": 1433,
- "PA_MPJPE_mm": 57.934833634318636
- },
- "P5_42_indoor_dancing": {
- "n": 1291,
- "PA_MPJPE_mm": 118.20184658637902
- },
- "P5_44_indoor_rom": {
- "n": 1381,
- "PA_MPJPE_mm": 76.91203318112069
- },
- "P6_49_outdoor_big_stairs_down": {
- "n": 1559,
- "PA_MPJPE_mm": 84.59367128883652
- },
- "P6_50_outdoor_workout": {
- "n": 1532,
- "PA_MPJPE_mm": 78.0931628334193
- },
- "P6_51_outdoor_dancing": {
- "n": 1427,
- "PA_MPJPE_mm": 63.6953970017694
- },
- "P7_57_outdoor_rock_chair": {
- "n": 1558,
- "PA_MPJPE_mm": 92.70131767949314
- },
- "P7_59_outdoor_rom": {
- "n": 1839,
- "PA_MPJPE_mm": 84.57222965135817
- },
- "P7_60_outdoor_workout": {
- "n": 1693,
- "PA_MPJPE_mm": 70.72311772175966
- },
- "P8_64_outdoor_skateboard": {
- "n": 1704,
- "PA_MPJPE_mm": 68.29277764547261
- },
- "P8_68_outdoor_handstand": {
- "n": 1606,
- "PA_MPJPE_mm": 119.19548259740635
- },
- "P8_69_outdoor_cartwheel": {
- "n": 656,
- "PA_MPJPE_mm": 94.3408367550109
- },
- "P9_76_outdoor_sitting": {
- "n": 1768,
- "PA_MPJPE_mm": 91.77105294977648
- }
- }
- }
-}
\ No newline at end of file
diff --git a/benchmark/run_gen10.sh b/benchmark/run_gen10.sh
deleted file mode 100755
index 418a882..0000000
--- a/benchmark/run_gen10.sh
+++ /dev/null
@@ -1,22 +0,0 @@
-#!/bin/bash
-# Generation 10 (all 288 px): the baseline family, the standard-crop-probability ablation and
-# the commercial-data run. 3DPW test and EMDB-1, every frame, published / GT-joint / RF-DETR boxes.
-set -u
-cd "$(dirname "$0")/.."
-PY=${PY:-python}
-R=instanthmr_distill_train/runs
-DPW_SEQ=${DPW_SEQ:-$HOME/Downloads/sequenceFiles/sequenceFiles}
-DPW_IMG=${DPW_IMG:-$HOME/Downloads/imageFiles}
-EMDB=${EMDB:-$HOME/Downloads/EMDB_root}
-OUT=benchmark/results
-CK="$(ls $R/g10*_b*_288_s0/best_student_model_v3.pth)"
-for B in gt-joints detector; do
- $PY benchmark/eval_3dpw_ckpt.py --ckpt $CK --sequence-dir "$DPW_SEQ" --image-root "$DPW_IMG" \
- --split test --gt h36m --adapter $OUT/adapter_j14_h36m_teacher.npz --bbox $B \
- --out $OUT/gen10_3dpw_test_${B}_288.json
-done
-for B in annotated gt-joints detector; do
- $PY benchmark/eval_emdb_ckpt.py --ckpt $CK --emdb-root "$EMDB" --bbox $B \
- --out $OUT/gen10_emdb1_${B}_288.json
-done
-echo ALL DONE
diff --git a/benchmark/run_gen10_resume.sh b/benchmark/run_gen10_resume.sh
deleted file mode 100755
index b9f25f7..0000000
--- a/benchmark/run_gen10_resume.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-#!/bin/bash
-# The two passes of run_gen10.sh still missing after the 2026-10-29 interruption.
-set -u
-cd "$(dirname "$0")/.."
-PY=${PY:-python}
-CK="$(ls instanthmr_distill_train/runs/g10*_b*_288_s0/best_student_model_v3.pth)"
-for B in gt-joints detector; do
- $PY benchmark/eval_emdb_ckpt.py --ckpt $CK --emdb-root "${EMDB:-$HOME/Downloads/EMDB_root}" --bbox $B \
- --out benchmark/results/gen10_emdb1_${B}_288.json
-done
-echo ALL DONE
diff --git a/benchmark/run_gen9w.sh b/benchmark/run_gen9w.sh
deleted file mode 100755
index 7d51598..0000000
--- a/benchmark/run_gen9w.sh
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/bin/bash
-# Generation-9 v4 family + g8h reference: 3DPW test and EMDB-1, full protocol (stride 1),
-# published boxes, ground-truth-joint boxes and RF-DETR boxes. One invocation per input size.
-set -u
-cd "$(dirname "$0")/.."
-PY=${PY:-python}
-R=instanthmr_distill_train/runs
-DPW_SEQ=${DPW_SEQ:-$HOME/Downloads/sequenceFiles/sequenceFiles}
-DPW_IMG=${DPW_IMG:-$HOME/Downloads/imageFiles}
-EMDB=${EMDB:-$HOME/Downloads/EMDB_root}
-OUT=benchmark/results
-C224="$R/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth $(ls $R/g9w_b*_224_s0/best_student_model_v3.pth)"
-C288="$(ls $R/g9w_b*_288_s0/best_student_model_v3.pth)"
-for S in 224 288; do
- eval CK=\$C$S
- for B in gt-joints detector; do
- $PY benchmark/eval_3dpw_ckpt.py --ckpt $CK --sequence-dir "$DPW_SEQ" --image-root "$DPW_IMG" \
- --split test --gt h36m --adapter $OUT/adapter_j14_h36m_teacher.npz --bbox $B \
- --out $OUT/gen9w_3dpw_test_${B}_$S.json
- done
- for B in annotated gt-joints detector; do
- $PY benchmark/eval_emdb_ckpt.py --ckpt $CK --emdb-root "$EMDB" --bbox $B \
- --out $OUT/gen9w_emdb1_${B}_$S.json
- done
-done
-echo ALL DONE
diff --git a/datasets_pipeline/README.md b/datasets_pipeline/README.md
index 7d55bc1..400dc4b 100644
--- a/datasets_pipeline/README.md
+++ b/datasets_pipeline/README.md
@@ -135,7 +135,7 @@ $THREEDPW_IMG_DIR/outdoors_climbing_01/image_00386.jpg
You need `imageFiles.zip` only. `sequenceFiles.zip` holds the original SMPL
ground truth, which this pipeline never reads — the MHR fits come from the
-parquet. Keep it for the 3DPW *benchmark* (`benchmark/eval_3dpw.py`), not for
+parquet. Keep it for the 3DPW ground truth (`benchmark/make_3dpw_gt.py`), not for
training data.
Same command, `--split 3dpw_train`. Both are small enough for
@@ -224,7 +224,7 @@ predicted hand keypoint to the top-left of the frame.
a hand crop unless all 21 of that hand's keypoints are visible (true for 0.3% of
SA-1B hands and 100% of every other split).
-`train_distill_mhr_only.py` consumes the mask: `loss_2d_native`,
+`train.py` consumes the mask: `loss_2d_native`,
`loss_2d_simcc` and `loss_reproj` are weighted by `joints_2d_vis`, which
defaults to all-ones when a folder does not carry it — so every existing
`data/sam3d_gt_*` folder trains exactly as before (verified to float32
diff --git a/datasets_pipeline/build_split.py b/datasets_pipeline/build_split.py
index 2805fda..2377636 100755
--- a/datasets_pipeline/build_split.py
+++ b/datasets_pipeline/build_split.py
@@ -48,8 +48,8 @@ A non-default --context requires an explicit --output-dir, and the builder
refuses to add crops of one kind to a folder holding the other, so a context
corpus can never silently mix with a standard one.
-Verified by `tools/verify_context_crops.py` on 200 AIC rows: default output
-byte-identical to HEAD; the 1.2x window cut back out of a 2.0x / 448 crop
+Verified on 200 AIC rows: default output byte-identical to the standard
+crop; the 1.2x window cut back out of a 2.0x / 448 crop
matches today's 224 crop at 42.2 dB mean (sigma-1 blur, padding masked) with
a 0.09 px shift. At 2.0x, 195 of 200 AIC squares reach past the frame edge.
@@ -57,7 +57,7 @@ a 0.09 px shift. At 2.0x, 195 of 200 AIC squares reach past the frame edge.
v4 preset): K square windows per person, each cut from the full-resolution
photo with ONE resize -- the same operation as the runtime crop, so the training
texture is the deployment texture (a 448 px context crop plus a second warp
-was not: STATUS.md 2026-10-21). Window 0 is the 1.2x square of the tight box
+was not). Window 0 is the 1.2x square of the tight box
with the RUNTIME's rounding (floor / ceil, instanthmr/inference.py), i.e. the
crop the phone cuts for a perfect detector box -- within 1 px of bbox_square,
whose int() edges the runtime does not use; windows 1..K-1 are v3 framings (side log-uniform in
@@ -109,7 +109,7 @@ from tools.parquet_to_npz import ( # noqa: E402
# MHR70 keypoint blocks. Index 41 is right_wrist, 62 is left_wrist; each hand
# block is its 20 finger joints plus that wrist. Taken from JOINT_NAMES in
-# instanthmr_distill_train/train_distill_mhr_only.py.
+# training/train.py.
RIGHT_HAND_KPS = list(range(21, 42))
LEFT_HAND_KPS = list(range(42, 63))
@@ -330,7 +330,7 @@ def main() -> None:
sys.exit("--windows needs K >= 2, the default --context and --no-hands")
if args.output_dir is None or "{size}" not in str(args.output_dir):
sys.exit("--windows needs an --output-dir containing {size}, e.g. "
- "$SCRATCH/instanthmr/data_win{size}/sam3d_gt_aic")
+ "data_win{size}/sam3d_gt_aic")
out_dirs = [Path(str(args.output_dir).format(size=s)) for s in win_sizes]
else:
out_dirs = [out_dir]
@@ -430,7 +430,7 @@ def main() -> None:
# a few entries keep every photo decoded once. Unbounded, the
# cache held every decoded frame of the shard -- ~10 MB each on
# SA-1B -- and 10 builder processes per job ran out of memory
- # after ~40 min (Jean Zay 2026-10-04, OUT_OF_MEMORY).
+ # after ~40 min.
if len(img_cache) >= 8:
img_cache.clear()
bgr = cv2.imread(str(args.image_dir / relpath))
diff --git a/datasets_pipeline/coco_license_subset.py b/datasets_pipeline/coco_license_subset.py
index 2014da4..e000bb7 100644
--- a/datasets_pipeline/coco_license_subset.py
+++ b/datasets_pipeline/coco_license_subset.py
@@ -13,8 +13,8 @@ Of the 39,665 people in sam3d_gt_coco: 4 -> 6,396, 5 -> 3,537, 6 -> 1,980,
legal choice; --licenses makes it explicit and it is recorded in the output.
python datasets_pipeline/coco_license_subset.py \
- --src $SCRATCH/instanthmr/data_win288/sam3d_gt_coco \
- --out $SCRATCH/instanthmr/data_win288_com/sam3d_gt_cococom --licenses 4,7
+ --src data_win288/sam3d_gt_coco \
+ --out data_win288_com/sam3d_gt_cococom --licenses 4,7
The output is a normal split folder (annotations/, body_crops/, images ->
body_crops), so it sits in a data root beside other splits and needs its own
@@ -58,7 +58,7 @@ def main() -> None:
(args.out / "body_crops").mkdir()
os.symlink("body_crops", args.out / "images")
- # One directory listing, not one glob per person: on Lustre each glob re-reads
+ # One directory listing, not one glob per person: on a network filesystem each glob re-reads
# the whole 40k-entry folder.
crop_of = {e.name.rsplit(".", 1)[0]: e.path for e in os.scandir(crops)}
kept = dropped = 0
diff --git a/datasets_pipeline/probe_split.py b/datasets_pipeline/probe_split.py
index 6a15012..c7d681a 100755
--- a/datasets_pipeline/probe_split.py
+++ b/datasets_pipeline/probe_split.py
@@ -205,7 +205,7 @@ def main() -> None:
"whether a hand clears --hand-min-px")
p.add_argument("--mhr", type=Path, default=REPO / "checkpoints" / "mhr_model.pt")
p.add_argument("--regressor", type=Path,
- default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")
+ default=REPO / "training" / "assets" / "mhr_j127_to_kp70.npy")
args = p.parse_args()
for d in args.annotation_dir:
if args.list_groups:
diff --git a/datasets_pipeline/registry.py b/datasets_pipeline/registry.py
index 3aecbb4..8e25cc3 100755
--- a/datasets_pipeline/registry.py
+++ b/datasets_pipeline/registry.py
@@ -129,14 +129,6 @@ DATASETS: dict[str, Dataset] = {
"download imageFiles.zip and unzip so that sequence folders "
"(courtyard_arguing_00/, ...) sit directly under the image dir.",
),
- "semelle": Dataset(
- name="semelle",
- splits=("semelle_train",),
- method="cli",
- notes="Private: ../semelle-kahina videos labelled by "
- "datasets_pipeline/semelle/label.py (teacher pose, insole contact, "
- "mocap feet). Its rows and undistorted frames are local.",
- ),
"harmony4d": Dataset(
name="harmony4d",
splits=("harmony4d_train", "harmony4d_test"),
diff --git a/datasets_pipeline/verify_split.py b/datasets_pipeline/verify_split.py
index ad18caa..47726d0 100755
--- a/datasets_pipeline/verify_split.py
+++ b/datasets_pipeline/verify_split.py
@@ -81,7 +81,7 @@ def structure(d: Path, workers: int = 16, hand_scan_limit: int = 300_000,
orphan_body = sorted(body - stems)[:5]
# Reading every npz is the expensive half of this check: single-threaded it
- # runs at ~100/s on Lustre, so a 3.4M-crop split would take 9 h. Sample
+ # runs at ~100/s on a network filesystem, so a 3.4M-crop split would take 9 h. Sample
# above hand_scan_limit and scale, and thread the reads either way.
scan = anns
scaled = False
@@ -207,7 +207,7 @@ def main() -> None:
p.add_argument("--mhr", type=Path,
default=REPO / "checkpoints" / "mhr_model.pt")
p.add_argument("--regressor", type=Path,
- default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")
+ default=REPO / "training" / "assets" / "mhr_j127_to_kp70.npy")
args = p.parse_args()
if not (args.dir / "annotations").is_dir():
diff --git a/demo.py b/demo.py
index b4d4be7..5c8fd4a 100755
--- a/demo.py
+++ b/demo.py
@@ -14,7 +14,7 @@ unpacked asset folder (``--mhr-assets``). See ``instanthmr/mhr_renderer.py``.
MHR-only checkpoints
--------------------
-A model exported from ``train_distill_mhr_only.py`` has FOUR ONNX outputs — it
+A model exported from ``train.py`` has FOUR ONNX outputs — it
has no 3D coordinate head, so its 70 keypoints are derived from the MHR forward
pass, exactly like the SAM3D teacher does. Such a model therefore *requires* an
MHR rig; when none is requested explicitly the demo falls back to
@@ -140,6 +140,13 @@ def parse_args() -> argparse.Namespace:
"(default: 5, as in the benchmarks; the token was trained on pairs "
"5-20 frames apart).",
)
+ p.add_argument(
+ "--contact-threshold", type=float, default=0.5, metavar="P",
+ help="Prev-token models only: a foot is drawn planted when its contact "
+ "probability exceeds P (default: 0.5). On held-out jumps, "
+ "0.8 gets both feet off in 92%% of flight frames instead of 83%%, "
+ "and keeps 98%% (instead of 99%%) of both-planted frames.",
+ )
p.add_argument(
"--smoothing-filter", action="store_true",
help="Temporally smooth per-person predictions with a moving-average "
@@ -297,7 +304,7 @@ def build_mhr(args: argparse.Namespace, required: bool = False):
sys.stderr.write(
f"\n[error] MHR TorchScript rig not found: {path}\n"
" This is the ~700 MB mhr_model.pt the distillation loads with\n"
- " torch.jit.load; see instanthmr_distill_train/checkpoints/README.md.\n"
+ " torch.jit.load; see training/checkpoints/README.md.\n"
" Alternatively use --mhr-assets with the unpacked MHR asset folder.\n\n"
)
sys.exit(1)
@@ -365,6 +372,7 @@ def build_all(args: argparse.Namespace):
mhr_renderer=mhr,
mesh_alpha=args.mesh_alpha,
reference=build_reference(args),
+ contact_threshold=args.contact_threshold,
)
return pipeline, viz, mhr
diff --git a/docs/accuracy_audit_2026-09-06.md b/docs/accuracy_audit_2026-09-06.md
deleted file mode 100644
index 596b888..0000000
--- a/docs/accuracy_audit_2026-09-06.md
+++ /dev/null
@@ -1,104 +0,0 @@
-Audit of the working tree on 6 September 2026
-
-> **Historical record — kept as written, not updated.** Read it as a snapshot of
-> 6 September 2026; several findings below were acted on or superseded later.
-> One terminology note: where this audit says "teacher" or "distillation" it
-> means the training *targets*, which are the released ground-truth annotations
-> of `facebook/sam-3d-body-dataset`, not inference from the
-> `sam-3d-body-dinov3` model. See `CLAUDE.md`, "What the training labels are".
-
-The best opportunity is to improve what the student learns from its supervision before increasing the backbone. There are also confirmed evaluation/export defects that must be fixed before measuring a new method. Three research directions below could support separate papers, but their accuracy gains and novelty remain experimental questions; none is a guaranteed CVPR contribution.
-
-Scope: inspected the current MHR-only trainer, DDP driver, augmentation and conversion code, evaluation, inference/export, available run configurations/results, and relevant primary research. Existing uncommitted changes were included in the audit and preserved. Numerical probes ran on CPU with the local TorchScript rig, local annotations, and the epoch-137 EMA checkpoint b3_s1. CUDA was unavailable to this process, so there was no new GPU timing, distributed reproduction, or training sweep. The small probes are diagnostics, not new benchmark results.
-
-**What the current baseline actually is.**
-
-The active default architecture has 39,961,412 trainable parameters, verified by instantiating it: RepViT-M2.3, a four-layer decoder of width 512, and 71 queries. One query supplies all 204 MHR parameters, 45 surface-shape parameters, and three camera translations; 70 queries supply the auxiliary/deployed 2D output. Self-attention allows those queries to help the global query, so their presence does not prove wasted capacity. The released model's 17M/9-query description and approximately 5 ms timing are not measurements of this current pipeline.
-
-The stored report `benchmark/results/3dpw_test_b3_s1_teacher_adapter.json` records 41.825 mm J14 PA-MPJPE and 69.030 mm root-relative MPJPE on 35,463 test person-frames, with zero dropped predictions. These are existing results, not rerun results. Training used rebalanced losses, so loss rebalance is already part of that baseline. The README's later 41.71 claim is not the value in this report; identify exact checkpoint, adapter and preprocessing hashes before quoting a baseline.
-
-**Confirmed implementation problems, ordered by priority.**
-
-1. **Standalone export silently removes scale bounds.** `tools/pth_to_onnx.py:116` constructs the default configuration, where bounds are off; line 128 loads with `strict=False`, and unexpected keys only print a message. A bounded model's `scale_lo` and `scale_hi` are therefore ignored. Reproducing this exact reload pattern produced no missing keys, two unexpected keys, and a maximum MHR-output difference of 0.288254 on the same input, before any ONNX conversion. This used newly initialized models with identical loaded weights, isolating the configuration defect; it is not a measured deployed checkpoint error. Restore the saved architecture/configuration, reject unexpected model keys, and compare all exported outputs with the original configured model on real crops.
-
-2. **Focal conditioning is not carried through evaluation and deployment.** `benchmark/eval_3dpw_ckpt.py:97` creates a default config; its dataset construction at line 109 does not enable `cliff_focal`. Bounded checkpoints instead fail strict loading because the evaluator has no corresponding buffers. `instanthmr/pipeline.py:98` does not pass focal calibration into `InstantHMR`. Even direct use of `InstantHMR(focal=...)` returns the diagonal-image heuristic in `HMRPrediction.focal_length` at `instanthmr/inference.py:256` and in the batch path. Training validation does pass the new flag. Consequently the documented equality of all evaluation paths only holds for compatible older configurations. Store preprocessing mode independently of whether intrinsics are known, serialize intrinsics/config metadata, and test training-validation/checkpoint/ONNX parity.
-
-3. **The v2 crop-center update omits division by zoom.** `train_distill_mhr_only.py:678` uses `cx -= dx * crop_w / 2` and similarly for y. With zero roll, the image transform is `u' = s(u-center) + center + translation`; its inverse puts the crop center at `center - translation/s`. The conditioning update requires that same `/s`. On a real MPII crop of width 996 pixels, scale 2 and dx 0.20, the code supplies center 1101.4 instead of 1151.2: a 49.8-pixel discrepancy. This is a controlled augmentation probe, not an estimate of mean dataset error. Derive conditioning from the complete inverse affine transform, including rotation and flip, instead of separate manual corrections.
-
-4. **The new camera representation remains approximate for unequal fx/fy.** `train_distill_mhr_only.py:704` uses fy for both axes. A camera with diagonal focal matrix F does not in general satisfy `R F = F R`; rotating pixels and rotating 3D with an unchanged anisotropic camera are different operations. A synthetic fx=1000, fy=500, 30-degree roll example gives 44.88 pixels of disagreement. This does not establish an error of that size in the actual corpus: sampled COCO/AIC/MPII focals had equal axes, and a separate 128-sample Harmony4D probe found fx/fy between 0.99393 and 1.03757. This is a smaller, conditional concern in the available data. Carry full transformed intrinsics/rays or rectify to square pixels. Existing counter-rotation in the reprojection loss avoids claiming that its forward/reverse algebra itself is broken; the issue is the physical consistency and information content of the input representation.
-
-5. **Validation can reward a model that fails on hard frames.** `val3dpw.py:179` removes non-finite predictions before calculating accuracy. Logging a drop count does not prevent checkpoint selection on the easier surviving subset. Mark such a checkpoint invalid for selection, or explicitly define a fixed-population failure metric. `train_distill_jz.py:120` also averages rank means and counts rather than reducing error sums and valid counts. That is slightly inaccurate for uneven shards and potentially worse when failure counts differ. The existing b3_s1 report has zero drops, so this is a vulnerability in selection, not evidence its stored result was inflated this way.
-
-6. **Recovery bookkeeping overstates successful updates.** In both regular and fallback paths, the driver calls the scheduler and EMA after `GradScaler.step` without checking whether gradient overflow skipped the optimizer update (`train_distill_jz.py:715`, `:793`). A finite loss does not imply finite gradients. Likewise, finite non-FK losses are not mathematically bounded: squared parameter error and cross-entropy can grow without bound. Make successful-update counting explicit and inspect gradient finiteness. The scale bounds and fallback remain useful safeguards; they do not by themselves certify recovery. This was established by control-flow inspection, not a GPU overflow reproduction.
-
-**Remaining learning bottlenecks and limits on the evidence.**
-
-- **Incomplete direct parameter supervision.** At the default geometric probabilities, only 20% of samples receive direct root-parameter loss. With `pose_split` enabled, local pose receives it on 60%; without it, only 20%. Flipped examples still have 3D geometric supervision, so saying they have no 3D training signal would be wrong. Correct mirrored local-transform targets and valid rotated root targets can restore direct supervision without another image encoder pass. Validate the rig's mirror mapping instead of assuming its Euler channels have SMPL's permutation/sign convention.
-- **Redundant target coordinates.** The actual 889-by-204 rig transform has rank 192. A parameter displacement of norm 10 along a null direction changed skeleton positions by at most 0.000153 mm across 192 sampled annotations. Thus coordinate losses distinguish some targets that are identical for the skeleton. This verifies redundancy, not that those directions contain substantial teacher variance or explain several millimeters of error. Measure that variance and its trunk gradients before treating this as the dominant cause. Skeleton equivalence must also be checked against mesh corrective behavior before replacing exported parameters.
-- **The mesh is poorly assessed by the training objective.** `MHRForwardPass.get_joints` explicitly replaces the 45 identity parameters with zeros. Perturbing supplied shape by +2 changed its skeleton by exactly zero in the probe. Shape still receives direct parameter loss, but there is no surface reconstruction loss or surface metric to establish mesh fidelity. PA-MPJPE improvement can coexist with worse body shape or hands. Add sparse surface supervision and a surface evaluation using a valid cross-topology correspondence where needed.
-- **Substantial student/label discrepancy remains.** On 32 unaugmented crops from each local source, b3_s1 had body-30 PA error against annotations of 33.57 mm on COCO, 27.21 on AIC, and 33.09 on MPII. These are training-source samples, not verified individual members of the cluster's train partition. They support investigating fit/representation quality, without establishing a generalization score or proving the backbone is the cause. On a separate 192-annotation probe, the skeleton-to-keypoint approximation alone contributed 1.35 mm over 70 points and 3.14 mm over the 30 body points. It is not the whole approximately 30 mm discrepancy, but the overall 70-point average hides its larger body residual.
-- **Auxiliary objectives compete for shared features.** The README reports that 2D terms account for roughly 56% of trunk gradient norms in one study and that detaching them gave a small pilot improvement. Those measurements were not independently rerun here. Critically, Gaussian-label entropy divided by total cross-entropy is not the fraction of the task that is learned: `CE=H(target)+KL(target||prediction)`, and H is a constant with zero model gradient. The quoted entropy floor does not prove 95% convergence or justify detachment. Gradient direction, variance, and matched-seed task results are the relevant evidence. Nor does one global output token alone prove a capacity bottleneck.
-- **Data quantity is not useful supervision quantity.** Mixture weights depend on dataset counts. Per-keypoint observation confidence is used for 2D; the 3D objective uses fixed body/finger weights rather than sample-specific fit reliability or ambiguity. Many occluded labels are complete optimization fits, which does not mean they are visually identifiable. Do not simply remove all invisible 3D supervision: it also supplies useful pose priors. Measure error by visibility, pose rarity and label provenance, and separate uncertainty in fitted labels from uncertainty in image evidence.
-- **The data loader uses stored person crops.** Original frames and optional hand crops exist in the pipeline but are not consumed by this trainer. This limits realistic crop jitter and hand detail; interpolating a 224-pixel body crop cannot restore unseen detail. Stronger augmentation should be tested against access to actual surrounding image content.
-
-**Evaluation issues that matter for a submission.**
-
-The local corpus is documented as containing all 3DPW sequences in `sam3d_distill_mix`; cluster-trained checkpoints are documented as using a different, clean corpus. The loader itself does not enforce this separation. Use a saved source-image/sequence manifest and explicit exclusions for validation and test, including any offline regressor/feature-cache fitting. The skeleton-to-keypoint fitter scans every supplied dataset folder, so its provenance also matters.
-
-The fixed teacher-derived J14 adapter is preferable to fitting a different adapter per student, but it is still learned using paired 3DPW-train ground truth. Disclose that target-dataset supervision. Rotation/translation equivariance does not prove that a linear map cannot absorb systematic pose error. Test the map on held-out sequences and an external dataset or derive it from body-model correspondences. The fitting script's halfway split is not an explicit sequence grouping. Never refit on the reported split.
-
-The harness uses GT-derived boxes, further crop expansion, a 60%-inside-image filter and its own person-frame list. Using the standard SMPL/H36M reference alone does not establish an identical published protocol. Reevaluate comparison models on the same manifest, or implement their standard protocol faithfully. Report GT boxes versus detected boxes and known versus estimated intrinsics separately. Disable recurring test-set monitoring while choosing methods: human design decisions can overfit it even if checkpoint-selection code ignores it.
-
-PA-MPJPE removes global translation, rotation and scale. Root-relative MPJPE removes translation but still penalizes scale and orientation. Neither is a temporal consistency metric; both can detect pose errors that also cause jitter. Report camera-space accuracy, surface error, hand error, temporal acceleration error and bone-length variation when making those claims. Low acceleration alone can result from oversmoothing: compare against GT motion and report lag. The repository's documented 34–49 mm seed spread is for a different 70-keypoint metric, not a measured standard deviation of the current J14+adapter result.
-
-**Three candidate contributions, each intended as an independent research program.**
-
-**1. Distillation over identifiable articulated geometry.**
-
-Hypothesis: a compact student wastes learning capacity on arbitrary choices made by fitting solvers, and receives inconsistent or missing parameter supervision under image transformations. Define an equivalence class of rig parameters that produce the same relevant geometry; supervise that class and physically meaningful local transforms instead of blindly reproducing each solver coordinate.
-
-Start with the verified linear redundancy: for the transform A and parameter prediction p, use a normalized robust loss on `A(p_student-p_target)`, or predict its independent coordinates. Add exact transformation rules for roll and mirroring, periodic/geodesic rotation losses, and confidence that distinguishes image-observable constraints from teacher priors. Keep surface-specific information in a separately supervised residual if skeleton-equivalent parameters are not mesh-equivalent. Recover a valid bounded representative for export; naive projection followed by clipping can break equivalence.
-
-The inexpensive baseline is transformed-coordinate loss with the deployed architecture unchanged. A 192-coordinate head plus fixed reconstruction costs about 39,000 multiply-accumulates per person; it can be fused with adjacent linear operations where there is no intervening nonlinear constraint. Target less than 10% extra training time and approximately unchanged inference; verify both rather than assuming them.
-
-The research claim must be broader than removing 12 dimensions: demonstrate invariance to equivalent relabelings, better gradient conditioning, and improved sample efficiency across more than one rig/annotation source. Compare against raw parameter loss, geometry-only loss, normalized joint-transform loss, a pose-prior baseline and augmentation fixes alone. [MHR](https://arxiv.org/abs/2511.15586), [HybrIK](https://openaccess.thecvf.com/content/CVPR2021/html/Li_HybrIK_A_Hybrid_Analytical-Neural_Inverse_Kinematics_Solution_for_3D_Human_CVPR_2021_paper.html) and [TokenHMR](https://arxiv.org/abs/2404.16752) already cover expressive rigs, geometric inverse kinematics and pose representations; those are not new claims here.
-
-Main risk: the redundant directions may have little actual target variance, so removing them may produce negligible accuracy gains. A cheap first experiment should compare projected and raw parameter losses with identical geometric losses and seeds. If only numerical stability improves, this remains an engineering fix rather than an independent accuracy paper.
-
-**2. Distill the teacher features that resolve 3D ambiguity, under a fixed teacher-compute budget.**
-
-Hypothesis: final fitted labels leave the compact encoder to rediscover difficult cues such as front/back orientation, foreshortening and occluded-limb context. The current loss contains no intermediate teacher-feature transfer. Transfer a small, body-part representation chosen for its influence on 3D pose rather than generic feature similarity or 2D localization.
-
-On a predeclared 5–10% training subset, extract clean-image teacher part descriptors. Estimate which feature directions affect depth/articulation using the teacher decoder's output sensitivities, while separating camera and surface-shape responses. Cache a compressed descriptor and reliability information. Train small student-side projectors on corresponding part queries/features, including degraded views with correctly matched body-part identities. Retain the ordinary supervision on the full corpus. Do not assume spatially warping cached DINO features is equivalent to reencoding a rotated/occluded image.
-
-A possible cache format is 16 descriptors × 64 channels × 2 bytes = 2,048 bytes/image: about 7.6 GB for 3.7M images, or 0.38 GB for 5%, before metadata. This is a proposed compression format, not a measured sufficient representation. Generating teacher features and sensitivities has a real offline GPU cost; report it and enforce a budget. Do not run the teacher every minibatch. Remove projectors at inference, giving exactly zero additional deployed network operations if the student architecture is kept fixed. Target no more than 20% additional total training-plus-preprocessing GPU-hours.
-
-The publishable claim would be that pose-sensitive feature selection transfers more 3D accuracy per teacher GPU-hour than generic feature distillation. Compare equal-budget output-only transfer, ordinary feature matching, random projections, uniform versus selected examples, and simple 2D-loss attenuation/detachment. Require effects beyond a single backbone and on difficult poses in an external dataset. [DUNE](https://arxiv.org/abs/2503.14405) already distills heterogeneous 2D/3D/human features, and [BinParam](https://www.sciencedirect.com/science/article/pii/S0925231226009574) includes feature distillation for efficient parametric reconstruction. Adding feature MSE alone is therefore insufficient novelty.
-
-Main risks: fitting privileged annotations does not guarantee the monocular teacher knows the same answer, and sensitivity selection may retain unstable teacher directions. Teacher accuracy, descriptor compression and reliability calibration must be measured. This is my first choice for improving pose accuracy at fixed deployed architecture, provided the offline budget is acceptable.
-
-**3. A single inexpensive correction that uses 2D evidence only where it constrains 3D.**
-
-Hypothesis: the model predicts both 2D locations and a body model, but the deployed 2D evidence never explicitly corrects the body. A generic reprojection correction can improve pixels while worsening depth. Learn a correction restricted by which pose directions the available observations can actually determine.
-
-Use calibrated 2D uncertainty, residuals between predicted image joints and projected MHR joints, the current pose, and full camera information. During training, the projection/kinematic Jacobian identifies well-observed versus ambiguous directions. A damped small correction should use image residuals in the former and preserve the learned prior in the latter. Distill that operation into a compact module predicting, for example, 16–32 correction coefficients. Runtime uses one encoder pass, one correction module and the necessary rig evaluation; no test-time autograd or iterative diffusion. Calibrate uncertainty separately: SimCC's fixed-width Gaussian training targets do not automatically provide calibrated uncertainty.
-
-The Jacobian and damped-update formulation are training tools here, not an assumption that a full 204-parameter solver is free. Profile the actual correction plus FK. Set a design cap of 0.5M extra parameters and 10% additional median and p95 person-pipeline latency, including geometry decoding. Stop or redesign if the cost exceeds that cap. Numerical stability and ONNX compatibility must be demonstrated.
-
-The claim is safe, calibrated use of available evidence under occlusion and camera error at a tightly bounded compute cost. Compare against an equal-size unconstrained residual MLP, one ungated correction step, reprojection-loss tuning, and an existing refinement method. Evaluate PA-MPJPE and absolute error together; merely improving reprojection is insufficient. [PyMAF](https://openaccess.thecvf.com/content/ICCV2021/papers/Zhang_PyMAF_3D_Human_Pose_and_Shape_Regression_With_Pyramidal_Mesh_ICCV_2021_paper.pdf) already feeds alignment evidence back into regression; [TokenHMR](https://arxiv.org/abs/2404.16752) studies the 2D/3D tradeoff; [Multi-HMR 2](https://arxiv.org/abs/2606.14841) addresses camera-centric recovery. A residual head, uncertainty weighting or camera conditioning alone is not sufficient novelty.
-
-Main risk: monocular depth ambiguity cannot be solved by algebra or confidence alone. The correction must fall back to its prior when evidence is insufficient, and a lower 2D error cannot guarantee a lower true 3D error. Demonstrate failure detection and robustness rather than claiming a universal guarantee.
-
-**How to decide whether any direction deserves a full paper.**
-
-First establish a corrected baseline with config/export parity, stable bounds, correct crop geometry, clean sequence manifests and fixed adapter provenance. Preserve an old-compatible configuration for historical comparisons. Run ordinary fixes as baseline work; do not count their gains as the proposed method's novelty.
-
-The documented new-run safeguards `--cliff-focal`, `--bound-scales` and `--anomaly-safe-fallback` still default to off. Neither preset automatically enables all of them. Explicit run configuration is necessary, and the export/evaluation defects above must be repaired before a bounded, focal-aware run can be judged through all entry points.
-
-Screen each direction independently on matched data, initial weights, sample counts and optimizer-update budgets using validation only. Use paired seeds for early elimination, then at least three complete seeds for a promising direction and more if variance remains high. Bootstrap by sequence/person rather than treating adjacent frames as independent. Compare against the corrected baseline and the simplest non-novel alternative.
-
-As a project decision threshold—not a CVPR rule or a predicted outcome—seek about 10% lower primary 3D error (roughly 4 mm if the corrected baseline remains near 42 mm), replicated on at least one external benchmark, with hand/surface quality preserved and the compute caps met. A smaller aggregate gain can still be meaningful if the paper establishes a substantial, general failure-mode improvement with strong evidence. Fix the threshold on validation before inspecting final test results.
-
-For public comparisons include [SAM 3D Body](https://arxiv.org/abs/2602.15989), [Fast SAM 3D Body](https://arxiv.org/abs/2603.15603), [FastHMR](https://arxiv.org/abs/2510.10868), and relevant strong pose baselines such as [NLF](https://arxiv.org/abs/2407.07532), subject to matched data and evaluation protocols. Fast SAM 3D Body already reports up to 10.9× end-to-end acceleration; a lightweight SAM student by itself is not a sufficient novelty argument. Separate network-only timing, rig decoding and detector-inclusive timing on identical hardware and precision.
-
-Reproduction artifacts from this audit are in `/tmp/instanthmr-audit/`: `probe.py`, `probe_results.json`, `checkpoint_probe.py`, and `checkpoint_results.json`. They use the existing `sam3d_video_312` Python environment and should be run from the repository root. The probes do not modify model weights or training data. The report is the only repository file added by this audit.
diff --git a/docs/agents/domain.md b/docs/agents/domain.md
deleted file mode 100644
index 3524904..0000000
--- a/docs/agents/domain.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# Domain Docs
-
-How the engineering skills should consume this repo's domain documentation when exploring the codebase.
-
-## Before exploring, read these
-
-- **`CONTEXT.md`** at the repo root, or
-- **`CONTEXT-MAP.md`** at the repo root if it exists: it points at one `CONTEXT.md` per context. Read each one relevant to the topic.
-- **`docs/adr/`**: read ADRs that touch the area you're about to work in. In multi-context repos, also check `src/<context>/docs/adr/` for context-scoped decisions.
-
-If any of these files don't exist, **proceed silently**. Don't flag their absence; don't suggest creating them upfront. The `/domain-modeling` skill (reached via `/grill-with-docs` and `/improve-codebase-architecture`) creates them lazily when terms or decisions actually get resolved.
-
-## File structure
-
-Single-context repo (most repos):
-
-```
-/
-├── CONTEXT.md
-├── docs/adr/
-│ ├── 0001-event-sourced-orders.md
-│ └── 0002-postgres-for-write-model.md
-└── src/
-```
-
-Multi-context repo (presence of `CONTEXT-MAP.md` at the root):
-
-```
-/
-├── CONTEXT-MAP.md
-├── docs/adr/ ← system-wide decisions
-└── src/
- ├── ordering/
- │ ├── CONTEXT.md
- │ └── docs/adr/ ← context-specific decisions
- └── billing/
- ├── CONTEXT.md
- └── docs/adr/
-```
-
-## Use the glossary's vocabulary
-
-When your output names a domain concept (in an issue title, a refactor proposal, a hypothesis, a test name), use the term as defined in `CONTEXT.md`. Don't drift to synonyms the glossary explicitly avoids.
-
-If the concept you need isn't in the glossary yet, that's a signal: either you're inventing language the project doesn't use (reconsider) or there's a real gap (note it for `/domain-modeling`).
-
-## Flag ADR conflicts
-
-If your output contradicts an existing ADR, surface it explicitly rather than silently overriding:
-
-> _Contradicts ADR-0007 (event-sourced orders), but worth reopening because…_
diff --git a/docs/agents/issue-tracker.md b/docs/agents/issue-tracker.md
deleted file mode 100644
index 0209a19..0000000
--- a/docs/agents/issue-tracker.md
+++ /dev/null
@@ -1,30 +0,0 @@
-# Issue tracker: Local Markdown
-
-Issues and specs for this repo live as markdown files in `.scratch/`.
-
-## Conventions
-
-- One feature per directory: `.scratch/<feature-slug>/`
-- The spec is `.scratch/<feature-slug>/spec.md`
-- Implementation issues are one file per ticket at `.scratch/<feature-slug>/issues/<NN>-<slug>.md`, numbered from `01`, never a single combined tickets file
-- Triage state is recorded as a `Status:` line near the top of each issue file (see `triage-labels.md` for the role strings)
-- Comments and conversation history append to the bottom of the file under a `## Comments` heading
-
-## When a skill says "publish to the issue tracker"
-
-Create a new file under `.scratch/<feature-slug>/` (creating the directory if needed).
-
-## When a skill says "fetch the relevant ticket"
-
-Read the file at the referenced path. The user will normally pass the path or the issue number directly.
-
-## Wayfinding operations
-
-Used by `/wayfinder`. The **map** is a file with one **child** file per ticket.
-
-- **Map**: `.scratch/<effort>/map.md` (the Notes / Decisions-so-far / Fog body).
-- **Child ticket**: `.scratch/<effort>/issues/NN-<slug>.md`, numbered from `01`, with the question in the body. A `Type:` line records the ticket type (`research`/`prototype`/`grilling`/`task`); a `Status:` line records `claimed`/`resolved`.
-- **Blocking**: a `Blocked by: NN, NN` line near the top. A ticket is unblocked when every file it lists is `resolved`.
-- **Frontier**: scan `.scratch/<effort>/issues/` for files that are open, unblocked, and unclaimed; first by number wins.
-- **Claim**: set `Status: claimed` and save before any work.
-- **Resolve**: append the answer under an `## Answer` heading, set `Status: resolved`, then append a context pointer (gist + link) to the map's Decisions-so-far in `map.md`.
diff --git a/docs/agents/triage-labels.md b/docs/agents/triage-labels.md
deleted file mode 100644
index b716855..0000000
--- a/docs/agents/triage-labels.md
+++ /dev/null
@@ -1,15 +0,0 @@
-# Triage Labels
-
-The skills speak in terms of five canonical triage roles. This file maps those roles to the actual label strings used in this repo's issue tracker.
-
-| Label in mattpocock/skills | Label in our tracker | Meaning |
-| -------------------------- | -------------------- | ---------------------------------------- |
-| `needs-triage` | `needs-triage` | Maintainer needs to evaluate this issue |
-| `needs-info` | `needs-info` | Waiting on reporter for more information |
-| `ready-for-agent` | `ready-for-agent` | Fully specified, ready for an AFK agent |
-| `ready-for-human` | `ready-for-human` | Requires human implementation |
-| `wontfix` | `wontfix` | Will not be actioned |
-
-When a skill mentions a role (e.g. "apply the AFK-ready triage label"), use the corresponding label string from this table.
-
-Edit the right-hand column to match whatever vocabulary you actually use.
diff --git a/docs/architecture.md b/docs/architecture.md
deleted file mode 100644
index 7c629b8..0000000
--- a/docs/architecture.md
+++ /dev/null
@@ -1,115 +0,0 @@
-# Model architecture
-
-InstantHMR mimics `facebook/sam-3d-body-dinov3` on the **per-person 3D pose
-estimation** task, running ~80× faster on a single GPU and exporting cleanly to
-ONNX / TFLite / QNN for mobile and edge deployment.
-
-> **This page describes the released 17 M checkpoint, not what the trainer
-> builds today.** `train_distill_mhr_only.py`'s `DistillConfig` defaults to
-> `repvit_m2_3` at `d_model=512`, 8 heads, 4 decoder layers, and **71** query
-> tokens (1 global + 70 for the 2D SimCC head) — **39.96 M parameters**:
-> 22.40 M backbone, 16.82 M decoder. The per-part joint queries in the table
-> below are gone; every parameter output now comes from one
-> `Linear(512 → 252)` on the single global token, and the 3D keypoints are
-> derived from the MHR forward pass rather than predicted. Read the config, not
-> this page, before quoting a parameter count.
-
-## High level
-
-```
- RF-DETR ──► square 1.2× crop ──► RepViT backbone ──► cross-attn decoder
- │
- CLIFF cond [cx, cy, scale] ──┘
- ▼
- mhr_params (204), shape (45), cam_trans (3),
- joints_2d (70, 2), joints_3d (70, 3)
-```
-
-## Backbone — RepViT-M1.5
-
-`timm.create_model("repvit_m1_5", pretrained=False, features_only=True)`
-gives a `(B, 512, 7, 7)` feature map at 224 input. Re-parameterised
-inverted-residual blocks make this fast on both desktop and mobile CPUs.
-
-## Decoder — 9-query cross-attention
-
-Two `nn.TransformerDecoderLayer` blocks (`d_model=256`, 4 heads, GELU,
-pre-LN) attend the flattened backbone features (49 tokens) with 9 learnable
-query tokens:
-
-| Query | Output | Head shape |
-|-------|------------------------------|-----------:|
-| 0 | camera translation (CLIFF) | (3,) |
-| 1 | pose (axis-angle joints) | (136,) |
-| 2 | scales | (68,) |
-| 3 | identity shape blendshapes | (45,) |
-| 4–8 | torso / L arm / R arm / L leg / R leg → joints | (5×256 → 70×3) |
-
-A fixed 2D sin-cos positional embedding is added to the 7×7 token grid.
-
-## CLIFF camera conditioning
-
-The camera-translation head is the only one that consumes the CLIFF
-conditioning vector — `[cx_norm, cy_norm, b_scale]` of the detector bbox
-(RF-DETR at runtime, YOLO during annotation) in **full-frame** coordinates.
-This is what lets InstantHMR predict translation consistent with the
-original perspective of the source image, without ever seeing pixels
-outside the 224×224 crop.
-
-```
-cam_token (B, 256)
-cliff_cond (B, 3)
- │ │
- └─────cat─────┘
- │
- LayerNorm + MLP
- │
- ▼
- cam_trans (B, 3)
-```
-
-## Output coordinate frame
-
-Both `joints_3d` and `cam_trans` are in metres in the SAM3D camera frame:
-
-- **X right**, **Y down**, **Z forward** (right-handed, Y-down).
-- `joints_3d` are body-centred — add `cam_trans` to land in camera space.
-- `joints_2d` come from a dedicated 2D head, in **normalised crop coords**
- `[-1, 1]`; the demo's `instanthmr/inference.py` re-projects them to
- full-frame pixels using the bbox-derived crop transform.
-
-## Why a small model at all?
-
-SAM 3D Body is excellent but heavy: its DINOv3 tokenizer + decoder runs at
-<20 FPS on a desktop GPU and doesn't fit comfortably on edge hardware. Training
-a compact student on the **same ground-truth MHR annotations Meta used to build
-it** (`facebook/sam-3d-body-dataset`) gives us InstantHMR — a model that:
-
-- runs at ~200 FPS on a single RTX 4070,
-- exports to ONNX / TFLite / QNN, and
-- preserves the SAM3D 70-keypoint & MHR outputs, so any downstream code that
- consumed `pred_keypoints_3d` keeps working.
-
-Distilling from the `sam-3d-body-dinov3` model instead is an option
-(`tools/annotate_dataset.py`) and is useful for imagery the released dataset
-does not cover — but it caps the student at the teacher's accuracy, which
-training on the ground truth does not.
-
-## Detection: RF-DETR
-
-The demo pipeline uses **RF-DETR** (Roboflow's transformer-based detector)
-for the upstream person detection step. We picked it over YOLO because:
-
-- it has **stronger recall on small / partially-occluded persons**, which
- matters a lot once the bbox is tight enough to drive a pose model;
-- its single-stage transformer head produces calibrated confidence scores,
- so the `--det-confidence` knob behaves predictably; and
-- it ships with a `optimize_for_inference()` pass (TorchScript +
- fused ops) that reliably halves detector latency on a desktop GPU.
-
-At inference the demo reports detector and HMR latency **separately**, both
-in the console and as scalar plots in the Rerun viewer, so it's always
-clear whether RF-DETR or InstantHMR is the bottleneck on a given machine.
-The default `--detector-variant medium` is tuned for ~480p–1080p input.
-Drop to `nano` / `small` for higher FPS at lower resolution, or step up to
-`base` / `large` for crowded scenes.
diff --git a/docs/architecture_audit_2026-09-12.md b/docs/architecture_audit_2026-09-12.md
deleted file mode 100644
index 683c05a..0000000
--- a/docs/architecture_audit_2026-09-12.md
+++ /dev/null
@@ -1,174 +0,0 @@
-# Generation 7 architecture audit — 12 September 2026
-
-**The regression-space mismatch is real and worth testing. It does not establish a particular accuracy gain. The largest overlooked architectural difference is the teacher's repeated geometry-to-image feedback inside its decoder.** A full reproduction of the output head alone will leave that difference intact.
-
-This is an assessment, not a training change. No trainer, deployment code, existing TODO, or checkpoint was modified. The quoted generation-7 results are validation statistics supplied by the user; no generation-7 checkpoint was available in the local `runs/` directory. These probes establish representation compatibility and approximation errors, not improvements from retraining.
-
-The wrapper in `../sam3-biomechanics/video_to_pose_pipeline` loads the implementation in `../sam3-biomechanics/sam-3d-body`. I inspected that implementation and the actual `video_to_pose_pipeline/checkpoints/sam-3d-body-dinov3/model_config.yaml` and `model.ckpt`, rather than relying on generic architectural descriptions. Its model files have no local git modifications. The wrapper also has optional encoder replacement/precision/quantization paths; they are separate from the reference architecture discussed here.
-
-Evidence: [main measurements](audit_2026-09-12/measurements.json), [follow-up measurements](audit_2026-09-12/followup_measurements.json), [representation/geometry probe](audit_2026-09-12/probe_pathway.py), [additional geometry probe](audit_2026-09-12/probe_geometry.py). Sampling used seed 20260912: 6,000 COCO annotations and 1,000 each from AIC, MPII, Harmony4D and 3DPW, selected without replacement from sorted filenames. Full mesh probes used the first 256 selected samples per dataset, in batches of 16. All computations ran on CPU; these are not GPU latency measurements. SA-1B is not available locally and was not sampled. The 3DPW probe is a representation check on annotations, not a benchmark evaluation.
-
-**1. What reproducing the regression pathway actually means.**
-
-The current student emits 204 MHR parameters, 45 shape coefficients and 3 camera coordinates from one linear layer on a single token. The teacher also reads pose from one token. A single global token is therefore not itself a departure from the teacher.
-
-| Component | Current student | Actual local DINOv3 teacher |
-|---|---|---|
-| Pose regression head | Linear(512, 252), including shape and camera | Two linear layers with ReLU, 1024 → 1024 → 519; camera has a separate MLP |
-| Initial pose | Near-zero raw parameters | Learned initial estimate; continuous neutral-pose initialization and residual prediction |
-| Global orientation | Three raw Euler values | Six values → orthogonalized rotation matrix → Euler triple |
-| Local pose | Raw `6:136` | 260 continuous values → 133 values, with hand/jaw masking and slicing |
-| Skeleton scales | 68 freely predicted values, then bounds | 28 coefficients → fixed affine map to 68 scales |
-| Finger poses | Part of the raw 204-vector | Two separate 54-value continuous blocks, converted and inserted by joint index |
-| Expression | No expression head | 72 predicted values explicitly multiplied by zero |
-
-Source: [student model](../instanthmr_distill_train/train_distill_mhr_only.py#L886), [teacher MHR head](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/heads/mhr_head.py#L39), [actual configuration](../../sam3-biomechanics/video_to_pose_pipeline/checkpoints/sam-3d-body-dinov3/model_config.yaml).
-
-The TODO's decomposition of the 260-dimensional representation is correct: 23 triples represented with 6D, 58 scalar angles represented with sin/cos, and six raw translation-like size channels. However, it includes finger channels that the head subsequently overwrites and a jaw triple that is discarded. The six size channels reach MHR indices `130:136`; genuine local rotation parameters occupy `6:130`, which is **124 values**, not 130.
-
-The measured case for changing representation is:
-
-| Probe | Result | What it establishes |
-|---|---|---|
-| COCO root components with absolute angle > 2.8 rad | 12.84% | Large root angles are common |
-| COCO poses with an axis within 0.34 rad of ±π | 21.23% | A substantial subset is near an Euler wrap boundary |
-| COCO middle root angle within 0.1 rad of ±π/2 | 2.17% | Near-singular Euler configurations occur |
-| COCO local rotation absolute angle, p99 | 1.188 rad | Local angles usually do not approach wrap boundaries |
-| Continuous body round-trip, all 10,000 annotations | Maximum parameter difference 2.38e-7 | Existing labels can supply continuous body targets |
-| Separate continuous hand round-trip | Maximum parameter difference 1.19e-7 | Existing labels also support the teacher's distinct hand parameterization |
-| GT scales projected into checkpoint's scale basis | Maximum residual < 2.2e-8 across sampled datasets | The scale subspace does not exclude these GT skeletons |
-
-These percentages differ slightly from the TODO because the sample differs. They support its qualitative observation. They are **not** the percentage of current predictions suffering a wrap error. That requires predictions paired with targets.
-
-A prediction just below −π and a target just below +π can represent nearby orientations while incurring a large raw Euler loss. Predicting continuous rotations and using an appropriate rotation loss addresses that mechanism. [Zhou et al.](https://arxiv.org/abs/1812.07035) provides the underlying continuity result, not a promised gain for this model.
-
-The local-pose case is less directly established by the angle histogram: those angles mostly remain away from wraps. The full pathway is still a sensible controlled experiment, especially because the scale basis and distinct hand mapping are compatible with the GT. It would be unjustified to claim that a head port alone will close the benchmark gap. Global rotation is removed by PA alignment; improvements to it principally affect orientation-sensitive errors, with only indirect benefits to PA-MPJPE.
-
-**Corrections required before implementing TODO item 1:**
-
-- **Fix the root inverse convention.** The teacher calls `roma.rotmat_to_euler("ZYX", R)`. Its inverse must therefore start with `R = roma.euler_to_rotmat("ZYX", p[:, 3:6])`, then concatenate the first and second **columns** of `R`. Do not pass the stored triple directly to `batch6DFromXYZ`. A simple input `[0.3, 0.5, 0.7]` returns `[0.7, 0.5, 0.3]` through that proposed inverse/forward pair. The correct construction preserves rig geometry, including sampled noncanonical Euler targets. For a physical rotation loss, respect the rig's actual extrinsic-XYZ convention; do not silently identify a teacher latent rotation matrix with the rig rotation.
-- **Pad the body target correctly.** The inverse expects 133 values: use MHR `6:136` plus three zero jaw values. Passing the 204-vector or only 130 values is wrong. Reconstruct hands using `hand_joint_idxs_left/right`, the hand conversion, and the checkpoint's hand buffers. A 339-value head that simply retains the body representation's finger channels is a different pathway from the teacher's hand overwrite.
-- **Separate inactive outputs from useful behavior.** 519 pose outputs reproduce the literal layout. 447 pose outputs omit the identically disabled face block while retaining both hand blocks; camera still needs its own three outputs. Neither choice requires a hand-crop encoder. Predicting 72 values that are always zeroed cannot improve accuracy.
-- **Include the MLP, initial estimate and valid continuous initialization.** All-zero 6D vectors and all-zero sin/cos pairs are degenerate. Neutral rotation initialization is not the current all-zero raw-Euler initialization. The teacher also normalizes decoder outputs before its head.
-- **Keep bounds compatible with the scale subspace.** PCA coefficients are not inherently bounded. Applying the existing coordinatewise tanh to the *expanded* 68 scales generally moves the result outside the teacher's affine subspace. A faithful port needs stability constraints that respect that subspace, plus separate handling of the six flexible size channels and zero root translation. Blindly retaining the old scale remap is not an exact reproduction.
-- **Use a numerically stable scale inverse.** The 28×68 checkpoint matrix has effective rank 24 in float32, three exactly zero singular values, and one tiny singular value (~1.29e-7). Do not assume an orthonormal, full-rank PCA basis or use transpose as its inverse. In the 256-sample COCO follow-up, a pseudoinverse with relative cutoff 1e-5 reconstructs scales to 1.50e-8 and the mesh to <0.00016 mm maximum vertex difference. Coefficients in null directions are not identifiable; expanded-scale supervision avoids inventing targets for them.
-- **Test equivalence in geometry, not only raw angles.** Euler canonicalization can change an equivalent angle triple. All sampled local-body triples round-tripped numerically here, but that is not a universal identity over arbitrary angles. Check rotations, FK and gradients near chart singularities. Returning through Euler for FK does not magically remove all numerical singularities.
-- **Treat loss changes as separate experiments.** Rotation-matrix or stable angular losses avoid wrap-dependent root penalties. Canonical 6D L2 supervision is not universally invalid, as the TODO implies, but its metric differs from physical rotation error. Likewise, SmoothL1 or unsquared Euclidean losses are not bugs merely because the paper describes other losses.
-
-**2. The most consequential missing decoder mechanism: geometry guides the next image lookup.**
-
-The teacher's actual configuration enables intermediate predictions, intermediate supervision, 2D keypoint tokens, 3D keypoint tokens, and keypoint updates. Its decoder does the following between layers:
-
-1. Predict an MHR mesh and camera from the current pose token.
-2. Obtain the mesh keypoints and project them into the crop.
-3. Bilinearly sample image features at those projected locations and add them to the corresponding 2D keypoint tokens. Out-of-image or behind-camera points are masked.
-4. Embed the predicted 2D coordinates into token positional information.
-5. Subtract the pelvis midpoint from predicted 3D keypoints and embed those coordinates into 70 separate 3D keypoint tokens.
-6. Let the next decoder layer use those updated tokens to revise the pose.
-
-Your four-layer decoder only exchanges generic query features and globally attends the image memory. It predicts MHR once, after the last layer. Its SimCC coordinates are never fed back into feature sampling or the pose decoder.
-
-This provides a concrete mechanism for improving **local body articulation**: after a provisional knee or elbow estimate, the next layer can inspect the corresponding image evidence. It is my highest-priority missing architectural mechanism beyond the parameter head. The recommendation is an inference from the implementation, not an ablation result on generation 7.
-
-There is an important distinction: the teacher has no independent freely regressed XYZ pose head in the inspected inference code, but **it does have 70 auxiliary 3D tokens**. The student's comment explaining their removal conflates these two things. Restoring an unconstrained 70×3 head alone would not reproduce their function.
-
-Also, the inspected callback predicts against the initial pose estimate at each layer; it does **not** add each new regression to the previous layer's predicted pose. Do not substitute conventional cumulative iterative regression and call it identical. Intermediate outputs are observable in released inference code; the configuration requests intermediate supervision, but the actual training loss implementation is not released.
-
-Sources: [intermediate prediction callback](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/meta_arch/sam3d_body.py#L463), [decoder loop](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/decoders/promptable_decoder.py#L153), [2D and 3D token updates](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/meta_arch/sam3d_body.py#L1759).
-
-The reference decoder also uses six layers at width 1024, versus four at width 512 in the student, repeats positional information within its layers, and applies a final normalization. Attention and feed-forward dimensions differ too, so width alone does not describe the cost. Its two-way image-to-token attention is explicitly disabled: adding that is not necessary for fidelity. The MLP/final-normalization changes are inexpensive candidates; none has a measured generation-7 gain yet.
-
-Implementation is more substantial than replacing a linear head: expose decoder stages, run differentiable FK and projection between them, update query features, and supervise useful intermediate predictions. Reuse the existing image encoding. The current deploy graph returns parameters and runs rig geometry outside that graph; moving FK and feature sampling inside decoding changes that export boundary. Validate the intended ONNX/mobile runtime before committing to the full design. A smaller two-stage prototype is a useful cost/accuracy experiment, but is not full teacher reproduction.
-
-**3. Image features are much coarser, beyond the crop-resolution issue already noted.**
-
-The actual DINOv3 checkpoint uses 512×512 crops and 16-pixel patches: a 32×32 image feature grid. Both current student backbones produce a **7×7** final feature grid at 224 input; I verified their forward outputs. This is 1,024 versus 49 spatial tokens. These are memory-grid sizes, not FLOP ratios.
-
-A stored-image resolution increase and a feature-stride change solve different problems. Rebuilding crops at higher resolution restores source detail. Using a stride-16 feature map or fusing an earlier backbone stage preserves more of that detail for the decoder. At 224 input, stride 16 gives a 14×14 grid without increasing input pixels. That is a particularly relevant experiment if adding joint-local feature sampling: sampling a 7×7 map remains coarse even with a sophisticated decoder.
-
-The teacher also has large-scale DINOv3 visual pretraining and substantially greater model capacity. Exact head mechanics cannot establish that this representation advantage has disappeared. Pose-oriented pretraining or feature distillation into the compact encoder is a plausible separate experiment. Ground-truth supervision remains the current training source; feature distillation would be an additional signal.
-
-One correction to TODO item 2: upsampling a 224 crop does not add image information, but that does not make training a larger input **necessarily pointless**. It can change effective feature stride and optimization. The clean experiment separates source resolution from network sampling density. Also, the current dataset resizes to the network input **before** its affine augmentation; merely storing 320-pixel crops would not preserve their detail through a zoom. Warp/crop from the higher-resolution source directly into the final network input.
-
-**4. The skeleton-only landmark approximation hides body-surface information.**
-
-The teacher forms its 70 landmarks from both mesh vertices and skeletal joints. The student uses an affine 70×127 skeleton regressor. This approximation is accurate for most fingers but less accurate for several body-surface landmarks.
-
-| Dataset, 256 samples each | All 70 landmarks: mean error | 30 nonfinger landmarks: mean error |
-|---|---:|---:|
-| COCO | 1.263 mm | 2.933 mm |
-| AIC | 1.322 mm | 3.071 mm |
-| MPII | 1.408 mm | 3.273 mm |
-| Harmony4D | 1.616 mm | 3.750 mm |
-| 3DPW annotations | 1.421 mm | 3.300 mm |
-
-These compare two landmark operators applied to the **same GT parameters**, using the local rig and teacher checkpoint mapping. No learned student prediction enters the comparison. In COCO, the worst individual discrepancy is 38.82 mm; mean discrepancies include small-toe tips at 8.8–9.1 mm and acromion landmarks at 6.7–6.8 mm. The exact operator is also closer to the stored COCO annotations: 0.175 mm mean versus the approximation's 1.242 mm. The small nonzero exact residual means the local rig/mapping is not perfectly identical to the annotation generator.
-
-An additional distinction matters for shape: changing the 45 shape coefficients cannot move the skeleton, but **can** move the teacher's mesh-derived landmarks. Zeroing shape on these COCO samples moves the nonfinger exact landmarks by 1.05 mm mean and up to 11.56 mm. Thus “shape has no geometric gradient” is a property of the student's skeleton-only landmark path, not of the teacher's full geometry. Generation 7's vertex loss already provides a shape gradient; this proposal adds accurate landmark locations and their gradients.
-
-The checkpoint's first 70 keypoints reference only **468 distinct mesh vertices**. You could extend the existing subset-skinning mechanism to include those exact indices, plus any vertices needed by the current vertex loss, and apply the sparse teacher mapping. This avoids making all 18,439 vertices mandatory at every decoder stage. Implementation feasibility follows from the current subset-skinning code; its combined runtime was not benchmarked here.
-
-This is a measurable modeling approximation worth testing. Its error is neither a guaranteed benchmark gain nor a quantity that should be subtracted from PA-MPJPE.
-
-**5. Camera conditioning and loss definitions still differ.**
-
-The teacher supplies per-location camera-ray embeddings to the image memory, as well as CLIFF information to the pose token. The student adds one learned projection of three CLIFF values to all queries. Dense ray conditioning gives each image feature its viewing direction explicitly; it is a plausible improvement for perspective-sensitive poses. For ideal square crops with known centered intrinsics, much of that information is already derivable from CLIFF plus grid position, so it is an inductive bias, not automatically extra information.
-
-The teacher separately regresses a crop camera `(scale, tx, ty)` and converts it using focal length and bbox geometry to full-camera translation. Your model directly regresses translation in metres. That is another regression-pathway difference omitted from TODO item 1. Its most direct benefits would concern depth and placement rather than PA-MPJPE. Source: [camera head](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/heads/camera_head.py#L83), [ray conditioning](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/meta_arch/sam3d_body.py#L1027).
-
-For the specific generation-7 command, `--crop-centre-fix` is a **no-op under baseline** because `cliff_follows_aug` remains false. The affine zoom/translation therefore does not get the same conditioning update as the v2 path. This can be tested independently of adopting every stronger v2 augmentation.
-
-The [paper's training section](https://arxiv.org/html/2602.15989v1#S4) describes L1 keypoint losses with learned uncertainty, pelvis/wrist normalization, annotation-dependent hand weights, parameter L2 and joint-limit penalties. The released code cannot establish their full formulas or weights. Your generation-7 configuration already uses **unsquared Euclidean** 3D error, not the default SmoothL1 listed in the TODO; shape weight is 0.03 and finger weight 0.2.
-
-Two verified loss differences deserve attention:
-
-- **No explicit pelvis/wrist centering in the main 3D loss.** It compares `pred_kp3d - tgt_3d` directly. `fk_scale` is only a training warm-up multiplier. Rig-origin coordinates are not equivalent to subtracting each sample's predicted and target hip midpoint. In the COCO probe, the GT hip midpoint lies about 907 mm from the rig coordinate origin. Separate wrist-relative hand losses would prevent arm-placement error from dominating finger articulation supervision. Retain an additional placement-sensitive term if adding centered losses.
-- **A naive joint-limit penalty conflicts with the annotations.** All 6,000 COCO samples violate at least one stored limit among the 124 local rotation parameters, using a tolerance of 1e-5 rad. About 20.23% of sampled parameter entries violate their limits; the maximum violation is 0.817 rad. There are 24 zero-width local-rotation limits. For example, `spine0_rx_flexible`, `l_talocrural_rx_flexible` and `r_subtalar_rz_flexible` have `[0,0]` limits with positive penalty weights, yet nonzero GT values. Do not apply all these limits unmodified. Use an annotation-compatible soft prior, or exclude the conflicting constraints and assess the others.
-
-Parameter supervision also remains masked by augmentation: with the generation-7 defaults, the root parameter loss sees about 20% of samples, and local-pose, scale, shape and vertex terms see about 60%. Geometry still supervises the other samples, so this is not a complete loss of their training signal. Correctly transforming parameter targets under rotations/flips could increase coverage, but requires rig-aware transformations and is not proven to match the unreleased teacher training pipeline.
-
-Learned uncertainty and hand-availability weighting are sensible experiments, particularly on ambiguous annotations. They need a proper probabilistic objective or regularizer; simply allowing a learned weight to fall toward zero can evade the task. Matching the teacher's stated norm is a lower priority than fixing demonstrated mismatches, and should not be bundled with the head ablation.
-
-**6. A tiny hand encoder is feasible; body gains need a feedback mechanism.**
-
-The teacher **shares one image encoder** across body and hand crops and uses separate body and hand decoders. Full inference runs body estimation, recrops each hand from the original image, estimates the hands, prompts the body decoder with hand wrists and body elbows, and then merges hand parameters with wrist-frame conversion and validity checks. It reuses cached body image embeddings for the prompted body refinement. It also updates selected hand scale and shape components. Source: [full inference and fusion](../../sam3-biomechanics/sam-3d-body/sam_3d_body/models/meta_arch/sam3d_body.py#L1197).
-
-Your proposed separate HGNetV2-B0 encoder is a reasonable compact alternative, though it is not the teacher's weight-sharing design. The installed timm provides B0 through B6; B0 has **1,850,396 parameters** after replacing its classifier head with Identity, and yields `(B,1024,7,7)` features at 224 input. A 256-wide, two-layer hand decoder shared by left and right crops is a reasonable initial experiment, not a measured optimum. Batch the two hands together and normalize left/right conventions consistently.
-
-The main implementation work is:
-
-1. Obtain original-resolution hand crops and labels. In the sampled COCO body crops, the projected hand extent averages only **15.4 pixels** at 224 input; Harmony4D averages **10.5 pixels**. Enlarging those saved patches cannot recover their original detail. Extents come from annotation coordinates, not a guarantee of visibility. Use original frames or store separate hand crops, and retain mappings needed to recreate their camera coordinates.
-2. Add crop jitter and annotation/visibility gates so training does not assume perfect hand boxes. The existing 2D joints can bootstrap boxes; a learned detector and confidence output are an additional option.
-3. Predict hand articulation plus wrist orientation in a consistent frame, with wrist-relative geometry supervision. Existing MHR GT supports articulation targets; hand-only normalization, flips and wrist transforms still need to be implemented.
-4. Convert predicted global wrist orientation into the body's local wrist frame, gate unreliable estimates, and preserve the body's fallback prediction.
-5. To improve the **body pose**, feed hand evidence back into body estimation. Reproduce the teacher's wrist/elbow prompting or train a smaller body-refinement stage. Merely overwriting finger parameters does not improve hip, knee or torso estimates by kinematics.
-
-Measured check: zeroing all 54 finger articulation parameters on 256 GT COCO samples changes the teacher's 30 nonfinger landmarks by only **0.102 mm mean**, while strongly moving fingers. The student's approximate landmark operator reports 1.298 mm body movement for the same intervention, showing some artificial finger-to-body coupling in that operator. Neither number is a measured gain from a trained hand network. SMPL conversion and body-metric adapters can also introduce small indirect effects.
-
-Expect the strongest benefit in hand articulation and full-mesh quality. A body-J14 gain is plausible through better wrists, arm refinement or shared training, but is not guaranteed by adding a hand encoder. Matching the complete teacher fusion is substantial work; adding a basic finger-only refinement branch is much simpler. Hand-specific training diversity also matters, beyond branch size.
-
-**7. Suggested order of experiments.**
-
-| Order | Experiment | Main quantity it should improve or clarify |
-|---|---|---|
-| 1 | Correct full continuous head, distinct hand blocks, MLP/init and compatible scale constraints | Parameter fitting, orientation, scale consistency; establishes pathway baseline |
-| 2 | Exact mesh-derived 70 landmarks using supported vertex subset | Removes a measured landmark approximation and restores surface-sensitive gradients |
-| 3 | Geometry feedback between decoder stages, including 3D token updates | Local body articulation; strongest missing body-specific mechanism |
-| 4 | Denser image features and source-aware cropping, tested separately | Local image evidence and spatial localization |
-| 5 | Camera-ray conditioning and explicit pelvis/wrist-relative terms | Perspective handling and separation of articulation from placement |
-| 6 | Dedicated hand crops/branch, then body refinement from wrists | Hands first; body improvement depends on refinement |
-
-These are priorities based on verified differences and mechanisms, not a ranking measured by training. Keeping the head experiment separate from decoder feedback is compatible with fully porting the head; it makes the result interpretable. Preserve generation 7 as the control and compare 2–3 matched seeds at comparable optimization budgets. The head and exact-landmark tests can use existing annotations; high-detail recropping requires original imagery. Measure body and hand errors separately, plus mesh error and absolute placement where relevant.
-
-The teacher's broader training mix includes additional multiview, synthetic and hand-specific datasets. Your five-source mix does not reproduce that coverage. The provided mix weights total 0.70 and the sampler renormalizes them: **SA-1B 42.86%, AIC 25.71%, Harmony4D 14.29%, COCO 12.86%, MPII 4.29%**, assuming all five are staged. Those are the actual probabilities, not 30/18/10/9/3 percent. Unlisted folders receive zero sampling weight. Source: [mixture implementation](../instanthmr_distill_train/train_distill_jz.py#L277).
-
-Finally, the supplied numbers are 3DPW validation J14+adapter records, not published test results. The summarizer also takes independent minima across epochs/RAW/EMA, so the displayed PA/MPJPE pair may not belong to one checkpoint. Establish proximity to published methods using one selected checkpoint, the matching test split, joint/conversion convention and bbox protocol; evaluate EMDB independently. Representation round-trips and the older 62.90-versus-31.84 fitting audit do not establish the achievable improvement from this head or a numerical gap attributable to it.
-
-To rerun the two main probes in the existing environment:
-
-```sh
-python docs/audit_2026-09-12/probe_pathway.py
-python docs/audit_2026-09-12/probe_geometry.py
-```
-
-The first writes `measurements.json`; the second prints the focused follow-up results. `scale_rank` in the first JSON uses float64's default tolerance (25); the effective float32 rank (24) and stable truncated-inverse check are recorded separately. Raw parameter maxima and geometric maxima use different sample counts, as specified above. The extra limit and backbone observations are recorded in `followup_measurements.json`.
diff --git a/docs/audit_2026-09-12/followup_measurements.json b/docs/audit_2026-09-12/followup_measurements.json
deleted file mode 100644
index 3edaea8..0000000
--- a/docs/audit_2026-09-12/followup_measurements.json
+++ /dev/null
@@ -1,64 +0,0 @@
-{
- "geometry_samples": 256,
- "dataset": "coco",
- "seed": 20260912,
- "scale_basis_rank_float32": 24,
- "scale_basis_rank_float64": 25,
- "scale_basis_pinv_rtol": 1e-05,
- "stable_scale_projection_max_abs": 1.4955233162972092e-08,
- "stable_scale_projection_max_abs_coeff": 3.551532238553479,
- "stable_scale_projection_pve_mm": {
- "mean": 1.431666341034088e-08,
- "max": 0.00015275544137693942
- },
- "hand_zero_exact_nonfinger_keypoint_mm": {
- "mean": 0.10175210982561111,
- "max": 1.5162975788116455
- },
- "hand_zero_approx_nonfinger_keypoint_mm": {
- "mean": 1.2984461784362793,
- "max": 6.655439376831055
- },
- "shape_zero_exact_nonfinger_keypoint_mm": {
- "mean": 1.049141526222229,
- "max": 11.555269241333008
- },
- "shape_zero_pve_mm": 2.7963504791259766,
- "coco_pose_limits": {
- "n": 6000,
- "indices": "6:130",
- "tolerance": 1e-05,
- "parameter_violation_fraction": 0.20225538313388824,
- "sample_violation_fraction": 1.0,
- "max_violation_rad": 0.8167469501495361,
- "zero_width_limits": 24,
- "total_local_rotation_limits": 124
- },
- "hgnetv2_b0": {
- "backbone_parameters_without_head": 1850396,
- "features_at_224": [
- 1,
- 1024,
- 7,
- 7
- ]
- },
- "hgnetv2_b4": {
- "backbone_parameters_without_head": 13553376,
- "features_at_224": [
- 1,
- 2048,
- 7,
- 7
- ]
- },
- "repvit_m2_3": {
- "backbone_parameters_without_head": 22404232,
- "features_at_224": [
- 1,
- 640,
- 7,
- 7
- ]
- }
-}
diff --git a/docs/audit_2026-09-12/measurements.json b/docs/audit_2026-09-12/measurements.json
deleted file mode 100644
index 7caa68f..0000000
--- a/docs/audit_2026-09-12/measurements.json
+++ /dev/null
@@ -1,506 +0,0 @@
-{
- "seed": 20260912,
- "scale_rank": 25,
- "scale_singular_values": [
- 0.4216929972171783,
- 0.3619723954052365,
- 0.3444689810276031,
- 0.3400789604916061,
- 0.3044403195381164,
- 0.26572715914133743,
- 0.2176144868135452,
- 0.20565041106231255,
- 0.1840951988319296,
- 0.17085281014442444,
- 0.148983675210781,
- 0.14221401512622833,
- 0.13806758695652796,
- 0.12133844196796417,
- 0.11143232759453013,
- 0.10651327160577374,
- 0.1045430526137352,
- 0.09462177180850188,
- 0.05311582610011101,
- 0.04739458113908767,
- 0.04616745188832282,
- 0.04183576628565788,
- 0.009205676615238188,
- 0.006077288184314966,
- 1.2910352609196704e-07,
- 0.0,
- 0.0,
- 0.0
- ],
- "datasets": {
- "coco": {
- "n": 6000,
- "root_fraction_abs_gt_2_8": 0.12844444811344147,
- "root_near_pi_sample_fraction": 0.2123333364725113,
- "root_near_gimbal_0_1_fraction": 0.021666666492819786,
- "body_roundtrip_param_abs": {
- "mean": 4.847996730237163e-09,
- "p95": 2.9802322387695312e-08,
- "max": 2.384185791015625e-07
- },
- "body_roundtrip_samples_gt_1e_4": 0,
- "scale_projection_abs": {
- "mean": 1.7601321293363869e-09,
- "p95": 5.676704272622457e-09,
- "max": 1.6244783407337593e-08
- },
- "root_roundtrip_rig_matrix_abs": {
- "mean": 4.301777355269425e-08,
- "p95": 1.7881393432617188e-07,
- "max": 6.48200511932373e-07
- },
- "root_naive_parameter_abs": {
- "mean": 0.6891651266078213,
- "p95": 5.875875496864319,
- "max": 6.276391506195068
- },
- "hand_roundtrip_param_abs": {
- "mean": 1.5643703233787295e-09,
- "p95": 7.450580596923828e-09,
- "max": 1.1920928955078125e-07
- },
- "geometry_n": 256,
- "body_roundtrip_skel_mm": {
- "mean": 2.0324994928157182e-05,
- "p95": 8.294915496662693e-05,
- "max": 0.0002288818359375
- },
- "scale_projection_skel_mm": {
- "mean": 1.95310167898661e-08,
- "p95": 0.0,
- "max": 7.631490007042885e-05
- },
- "scale_projection_pve_mm": {
- "mean": 1.4316663921023e-08,
- "p95": 0.0,
- "max": 0.00015275544137693942
- },
- "hand_only_nonhand_kp_mm": {
- "mean": 1.2984461276149117,
- "p95": 4.054974603652953,
- "max": 6.655439376831055
- },
- "skeleton_kp_vs_exact_mm": {
- "mean": 1.262880131083348,
- "p95": 6.897165751457212,
- "max": 38.820594787597656
- },
- "skeleton_body_kp_vs_exact_mm": {
- "mean": 2.9331194676983565,
- "p95": 9.563913536071777,
- "max": 38.820594787597656
- },
- "exact_kp_vs_annotation_mm": {
- "mean": 0.1747331356528752,
- "p95": 1.8100950360298131,
- "max": 9.01367473602295
- },
- "skeleton_kp_vs_annotation_mm": {
- "mean": 1.2424783247384557,
- "p95": 6.975071215629569,
- "max": 37.93433380126953
- },
- "pelvis_midhip_mm": {
- "mean": 907.3023588657379,
- "p95": 914.8213043212891,
- "max": 920.49072265625
- },
- "root_canonical_skel_mm": {
- "mean": 3.776603092655997e-05,
- "p95": 0.0001314105553319671,
- "max": 0.00038194627268239856
- },
- "hand_span_in_224_crop_px": {
- "mean": 15.433155534744262,
- "p95": 28.704643535614004,
- "max": 90.4273452758789
- },
- "sampled_image_sizes": [
- [
- 224,
- 224
- ]
- ]
- },
- "aic": {
- "n": 1000,
- "root_fraction_abs_gt_2_8": 0.04633333161473274,
- "root_near_pi_sample_fraction": 0.07699999958276749,
- "root_near_gimbal_0_1_fraction": 0.017999999225139618,
- "body_roundtrip_param_abs": {
- "mean": 4.803188577833974e-09,
- "p95": 2.9802322387695312e-08,
- "max": 2.384185791015625e-07
- },
- "body_roundtrip_samples_gt_1e_4": 0,
- "scale_projection_abs": {
- "mean": 1.7736721108909556e-09,
- "p95": 5.722091468013386e-09,
- "max": 1.5501413652696527e-08
- },
- "root_roundtrip_rig_matrix_abs": {
- "mean": 3.3997358211005726e-08,
- "p95": 1.341104507446289e-07,
- "max": 4.172325134277344e-07
- },
- "root_naive_parameter_abs": {
- "mean": 0.31238043495535384,
- "p95": 0.924203491210937,
- "max": 6.295416831970215
- },
- "hand_roundtrip_param_abs": {
- "mean": 1.5494886858558697e-09,
- "p95": 7.450580596923828e-09,
- "max": 1.1920928955078125e-07
- },
- "geometry_n": 256,
- "body_roundtrip_skel_mm": {
- "mean": 1.9502637364221968e-05,
- "p95": 7.867813110351562e-05,
- "max": 0.00018688123964238912
- },
- "scale_projection_skel_mm": {
- "mean": 1.6573496143283542e-08,
- "p95": 0.0,
- "max": 0.0001537753560114652
- },
- "scale_projection_pve_mm": {
- "mean": 1.0493360439691971e-08,
- "p95": 0.0,
- "max": 0.00030577124562114477
- },
- "hand_only_nonhand_kp_mm": {
- "mean": 1.3133377586958617,
- "p95": 4.055889153480527,
- "max": 7.546623229980469
- },
- "skeleton_kp_vs_exact_mm": {
- "mean": 1.3220521317657032,
- "p95": 7.123849296569824,
- "max": 46.387332916259766
- },
- "skeleton_body_kp_vs_exact_mm": {
- "mean": 3.0706044515078403,
- "p95": 9.829323673248286,
- "max": 46.387332916259766
- },
- "exact_kp_vs_annotation_mm": {
- "mean": 0.15112333263938502,
- "p95": 1.4468539297580705,
- "max": 9.727354049682617
- },
- "skeleton_kp_vs_annotation_mm": {
- "mean": 1.2894970125568872,
- "p95": 7.168204927444458,
- "max": 46.3869743347168
- },
- "pelvis_midhip_mm": {
- "mean": 908.8520421981812,
- "p95": 915.5803833007812,
- "max": 941.2371826171875
- },
- "root_canonical_skel_mm": {
- "mean": 3.112303484497241e-05,
- "p95": 0.0001027138641802594,
- "max": 0.0003773421049118042
- },
- "hand_span_in_224_crop_px": {
- "mean": 14.884918254375458,
- "p95": 26.578073787689206,
- "max": 50.65440368652344
- },
- "sampled_image_sizes": [
- [
- 224,
- 224
- ]
- ]
- },
- "mpii": {
- "n": 1000,
- "root_fraction_abs_gt_2_8": 0.1326666623353958,
- "root_near_pi_sample_fraction": 0.2150000035762787,
- "root_near_gimbal_0_1_fraction": 0.024000000208616257,
- "body_roundtrip_param_abs": {
- "mean": 4.869290418546849e-09,
- "p95": 2.9802322387695312e-08,
- "max": 1.1920928955078125e-07
- },
- "body_roundtrip_samples_gt_1e_4": 0,
- "scale_projection_abs": {
- "mean": 1.727188326217252e-09,
- "p95": 5.578587583543446e-09,
- "max": 1.5026239752469195e-08
- },
- "root_roundtrip_rig_matrix_abs": {
- "mean": 4.29708905155874e-08,
- "p95": 1.7881393432617188e-07,
- "max": 5.364418029785156e-07
- },
- "root_naive_parameter_abs": {
- "mean": 0.6591208255854047,
- "p95": 5.836613655090332,
- "max": 6.284758567810059
- },
- "hand_roundtrip_param_abs": {
- "mean": 1.5445617368284586e-09,
- "p95": 7.450580596923828e-09,
- "max": 1.1920928955078125e-07
- },
- "geometry_n": 256,
- "body_roundtrip_skel_mm": {
- "mean": 2.0649441278268837e-05,
- "p95": 8.529922342859209e-05,
- "max": 0.0002288818359375
- },
- "scale_projection_skel_mm": {
- "mean": 1.4238418234473052e-08,
- "p95": 0.0,
- "max": 7.864199869800359e-05
- },
- "scale_projection_pve_mm": {
- "mean": 8.68299566776343e-09,
- "p95": 0.0,
- "max": 0.000152587890625
- },
- "hand_only_nonhand_kp_mm": {
- "mean": 1.303009856767555,
- "p95": 4.113340353965758,
- "max": 7.0834760665893555
- },
- "skeleton_kp_vs_exact_mm": {
- "mean": 1.408206587886578,
- "p95": 7.6829523801803585,
- "max": 51.369140625
- },
- "skeleton_body_kp_vs_exact_mm": {
- "mean": 3.272702324564984,
- "p95": 10.91020331382751,
- "max": 51.369140625
- },
- "exact_kp_vs_annotation_mm": {
- "mean": 0.19452631403964796,
- "p95": 2.0200669407844525,
- "max": 10.951011657714844
- },
- "skeleton_kp_vs_annotation_mm": {
- "mean": 1.3965330520900545,
- "p95": 7.800119304656979,
- "max": 46.11688232421875
- },
- "pelvis_midhip_mm": {
- "mean": 905.5695853233337,
- "p95": 913.4701995849609,
- "max": 932.9898071289062
- },
- "root_canonical_skel_mm": {
- "mean": 4.161703200372286e-05,
- "p95": 0.0001537753560114652,
- "max": 0.00032259715953841805
- },
- "hand_span_in_224_crop_px": {
- "mean": 13.570606416463852,
- "p95": 20.74840297698974,
- "max": 33.275596618652344
- },
- "sampled_image_sizes": [
- [
- 224,
- 224
- ]
- ]
- },
- "harmony4d": {
- "n": 1000,
- "root_fraction_abs_gt_2_8": 0.20666666328907013,
- "root_near_pi_sample_fraction": 0.36899998784065247,
- "root_near_gimbal_0_1_fraction": 0.02199999988079071,
- "body_roundtrip_param_abs": {
- "mean": 4.859422403169584e-09,
- "p95": 2.9802322387695312e-08,
- "max": 1.1920928955078125e-07
- },
- "body_roundtrip_samples_gt_1e_4": 0,
- "scale_projection_abs": {
- "mean": 2.6347890349725476e-09,
- "p95": 8.493963740696664e-09,
- "max": 2.1240262909616447e-08
- },
- "root_roundtrip_rig_matrix_abs": {
- "mean": 5.056530547638734e-08,
- "p95": 1.9371509552001953e-07,
- "max": 5.662441253662109e-07
- },
- "root_naive_parameter_abs": {
- "mean": 1.0471845632053447,
- "p95": 5.9726653575897215,
- "max": 6.24089241027832
- },
- "hand_roundtrip_param_abs": {
- "mean": 1.4260116559478989e-09,
- "p95": 7.450580596923828e-09,
- "max": 1.1920928955078125e-07
- },
- "geometry_n": 256,
- "body_roundtrip_skel_mm": {
- "mean": 2.2661963085494054e-05,
- "p95": 8.529922342859209e-05,
- "max": 0.00023515372595284134
- },
- "scale_projection_skel_mm": {
- "mean": 2.287333476430838e-08,
- "p95": 0.0,
- "max": 7.630325853824615e-05
- },
- "scale_projection_pve_mm": {
- "mean": 1.970640635590727e-08,
- "p95": 0.0,
- "max": 0.000152587890625
- },
- "hand_only_nonhand_kp_mm": {
- "mean": 1.7946685821541275,
- "p95": 5.6304475307464585,
- "max": 17.68724250793457
- },
- "skeleton_kp_vs_exact_mm": {
- "mean": 1.6157738262989496,
- "p95": 8.519428968429562,
- "max": 44.049713134765625
- },
- "skeleton_body_kp_vs_exact_mm": {
- "mean": 3.749637672129704,
- "p95": 12.776697397232054,
- "max": 44.049713134765625
- },
- "exact_kp_vs_annotation_mm": {
- "mean": 0.10912871654061149,
- "p95": 0.0007656355737708509,
- "max": 4.679005146026611
- },
- "skeleton_kp_vs_annotation_mm": {
- "mean": 1.5388319493067095,
- "p95": 8.519144773483273,
- "max": 44.04958724975586
- },
- "pelvis_midhip_mm": {
- "mean": 912.6007525920868,
- "p95": 921.6853790283203,
- "max": 930.41455078125
- },
- "root_canonical_skel_mm": {
- "mean": 5.116117408168774e-05,
- "p95": 0.00017625244800001383,
- "max": 0.0003417296102270484
- },
- "hand_span_in_224_crop_px": {
- "mean": 10.482691010951996,
- "p95": 15.397425985336302,
- "max": 29.435840606689453
- },
- "sampled_image_sizes": [
- [
- 224,
- 224
- ]
- ]
- },
- "3dpw": {
- "n": 1000,
- "root_fraction_abs_gt_2_8": 0.11433333158493042,
- "root_near_pi_sample_fraction": 0.2070000022649765,
- "root_near_gimbal_0_1_fraction": 0.00800000037997961,
- "body_roundtrip_param_abs": {
- "mean": 4.315976630643039e-09,
- "p95": 2.9802322387695312e-08,
- "max": 2.384185791015625e-07
- },
- "body_roundtrip_samples_gt_1e_4": 0,
- "scale_projection_abs": {
- "mean": 2.0754229214372135e-09,
- "p95": 6.65673839800318e-09,
- "max": 2.031111703826305e-08
- },
- "root_roundtrip_rig_matrix_abs": {
- "mean": 4.3235324281138474e-08,
- "p95": 1.7881393432617188e-07,
- "max": 6.258487701416016e-07
- },
- "root_naive_parameter_abs": {
- "mean": 0.7765007006836385,
- "p95": 5.773379302024838,
- "max": 6.353257179260254
- },
- "hand_roundtrip_param_abs": {
- "mean": 1.2745278679943238e-09,
- "p95": 7.450580596923828e-09,
- "max": 1.1920928955078125e-07
- },
- "geometry_n": 256,
- "body_roundtrip_skel_mm": {
- "mean": 2.151847284910666e-05,
- "p95": 8.529922342859209e-05,
- "max": 0.0002157918643206358
- },
- "scale_projection_skel_mm": {
- "mean": 1.4343623989342038e-08,
- "p95": 0.0,
- "max": 7.630325853824615e-05
- },
- "scale_projection_pve_mm": {
- "mean": 9.343029776613385e-09,
- "p95": 0.0,
- "max": 0.000152587890625
- },
- "hand_only_nonhand_kp_mm": {
- "mean": 1.1839595396388778,
- "p95": 3.4480524182319616,
- "max": 7.466592311859131
- },
- "skeleton_kp_vs_exact_mm": {
- "mean": 1.420894855516573,
- "p95": 7.527710986137388,
- "max": 40.37519454956055
- },
- "skeleton_body_kp_vs_exact_mm": {
- "mean": 3.2999284368693274,
- "p95": 10.43739891052244,
- "max": 40.37519454956055
- },
- "exact_kp_vs_annotation_mm": {
- "mean": 0.3037412910945271,
- "p95": 2.8356644988059996,
- "max": 9.90914535522461
- },
- "skeleton_kp_vs_annotation_mm": {
- "mean": 1.4251954034570269,
- "p95": 7.732362079620358,
- "max": 40.375099182128906
- },
- "pelvis_midhip_mm": {
- "mean": 910.3145744800568,
- "p95": 920.9322509765625,
- "max": 930.5852661132812
- },
- "root_canonical_skel_mm": {
- "mean": 3.8249140468245345e-05,
- "p95": 0.00015843627625145018,
- "max": 0.0004565700073726475
- },
- "hand_span_in_224_crop_px": {
- "mean": 14.849500558376313,
- "p95": 20.30525197982788,
- "max": 35.22533416748047
- },
- "sampled_image_sizes": [
- [
- 224,
- 224
- ]
- ]
- }
- }
-}
diff --git a/docs/audit_2026-09-12/probe_geometry.py b/docs/audit_2026-09-12/probe_geometry.py
deleted file mode 100644
index 8b097ab..0000000
--- a/docs/audit_2026-09-12/probe_geometry.py
+++ /dev/null
@@ -1,51 +0,0 @@
-"""Read-only CPU audit; run from any directory in the existing sam3d_video_312 environment.
-Requires the adjacent sam3-biomechanics checkout and its DINOv3 checkpoint.
-"""
-import torch,numpy as np,json
-from pathlib import Path
-import importlib.util
-
-torch.set_num_threads(4)
-root=Path(__file__).resolve().parents[2]
-s=torch.load(root.parent/'sam3-biomechanics/video_to_pose_pipeline/checkpoints/sam-3d-body-dinov3/model.ckpt',map_location='cpu',weights_only=False,mmap=True)
-rig=torch.jit.load(str(root/'checkpoints/mhr_model.pt'),map_location='cpu').eval()
-for param in rig.parameters(): param.requires_grad_(False)
-ct=rig.character_torch
-print('limits repr',ct.parameter_limits)
-print('limits attributes',ct.parameter_limits._c.dump_to_str(True,False,False)[:5000])
-paths=sorted((root/'data/sam3d_gt_coco/annotations').glob('*.npz'))
-rng=np.random.default_rng(20260912)
-paths=[paths[i] for i in rng.choice(len(paths),6000,replace=False)[:256]]
-r=[]
-for path in paths:
- with np.load(path) as z:r.append((z['mhr_model_params'],z['shape_params']))
-p,h=[torch.from_numpy(np.stack([a[i] for a in r])).float() for i in range(2)]
-C=s['head_pose.scale_comps'].double(); mu=s['head_pose.scale_mean'].double()
-coeff=(p[:,136:].double()-mu)@torch.linalg.pinv(C,rtol=1e-5)
-print('stable_pca_max_residual',float((coeff@C+mu-p[:,136:]).abs().max()),'coeff_max',float(coeff.abs().max()))
-W=torch.from_numpy(np.load(root/'instanthmr_distill_train/assets/mhr_j127_to_kp70.npy')).float()
-K=s['head_pose.keypoint_mapping'][:70].float()
-hand=torch.cat([s['head_pose.hand_joint_idxs_left'],s['head_pose.hand_joint_idxs_right']]).long()
-body=list(range(21))+[41]+list(range(62,70))
-err=[]; exacterr=[]; approxerr=[]; shapeerr=[]; shapekp=[]; pcerr=[]; pcent=[]
-for start in range(0,len(p),16):
- pp=p[start:start+16]; hh=h[start:start+16]
- with torch.no_grad():
- vv,ss=rig(hh,pp,torch.zeros(len(pp),72)); ss=ss[...,:3]; [email protected]([vv,ss],1); ka=W@ss
- err.append((ka-kp).norm(dim=-1)*10)
- ph=pp.clone(); ph[:,hand]=0
- vh,sh=rig(hh,ph,torch.zeros(len(pp),72)); [email protected]([vh,sh[...,:3]],1)
- exacterr.append((kh-kp).norm(dim=-1)*10); approxerr.append(((W@sh[...,:3])-ka).norm(dim=-1)*10)
- v0,s0=rig(torch.zeros_like(hh),pp,torch.zeros(len(pp),72)); [email protected]([v0,s0[...,:3]],1)
- shapeerr.append((v0-vv).norm(dim=-1)*10);shapekp.append((k0-kp).norm(dim=-1)*10)
- pc=pp.clone();pc[:,136:]=(coeff[start:start+16]@C+mu).float()
- vc,sc=rig(hh,pc,torch.zeros(len(pp),72));pcerr.append((vc-vv).norm(dim=-1)*10)
- # Rig origin is not hip-normalized: measure change of midhip induced by scale only.
- [email protected]([vc,sc[...,:3]],1)
- pcent.append((((kp[:,9]+kp[:,10])-(kc[:,9]+kc[:,10]))/2).norm(dim=-1)*10)
-for name,vals in [('regressor error',err),('hand zero exact',exacterr),('hand zero approx',approxerr),('shape zero exactkp',shapekp)]:
- e=torch.cat(vals);print(name,'body mean/max',e[:,body].mean().item(),e[:,body].max().item(),'top means',[(i,e[:,i].mean().item()) for i in e.mean(0).argsort(descending=True)[:12].tolist()])
-print('pca24 PVE mean/max',torch.cat(pcerr).mean().item(),torch.cat(pcerr).max().item())
-print('pca24 midhip movement max',torch.cat(pcent).max().item())
-print('shapezero pve',torch.cat(shapeerr).mean().item())
-print('student regressor row sum range',W.sum(1).min().item(),W.sum(1).max().item())
diff --git a/docs/audit_2026-09-12/probe_pathway.py b/docs/audit_2026-09-12/probe_pathway.py
deleted file mode 100644
index f257674..0000000
--- a/docs/audit_2026-09-12/probe_pathway.py
+++ /dev/null
@@ -1,102 +0,0 @@
-"""Read-only CPU audit; run from any directory in the existing sam3d_video_312 environment.
-Requires the adjacent sam3-biomechanics checkout and its DINOv3 checkpoint.
-"""
-import importlib.util, json, sys
-from pathlib import Path
-import numpy as np
-import torch
-import roma
-from PIL import Image
-
-torch.set_num_threads(4)
-root=Path(__file__).resolve().parents[2]
-teacher=root.parent/'sam3-biomechanics/sam-3d-body'
-modpath=teacher/'sam_3d_body/models/modules/mhr_utils.py'
-spec=importlib.util.spec_from_file_location('mhr_utils',modpath)
-u=importlib.util.module_from_spec(spec); spec.loader.exec_module(u)
-s=torch.load(root.parent/'sam3-biomechanics/video_to_pose_pipeline/checkpoints/sam-3d-body-dinov3/model.ckpt',map_location='cpu',weights_only=False,mmap=True)
-C=s['head_pose.scale_comps'].double(); mu=s['head_pose.scale_mean'].double(); pinv=torch.linalg.pinv(C)
-print('scale singular values',torch.linalg.svdvals(C).tolist(),flush=True)
-rig=torch.jit.load(str(root/'checkpoints/mhr_model.pt'),map_location='cpu').eval()
-for p in rig.parameters(): p.requires_grad_(False)
-W=torch.from_numpy(np.load(root/'instanthmr_distill_train/assets/mhr_j127_to_kp70.npy')).float()
-K=s['head_pose.keypoint_mapping'][:70].float()
-hand_idx=torch.cat([s['head_pose.hand_joint_idxs_left'],s['head_pose.hand_joint_idxs_right']]).long()
-body_idx=[i for i in range(70) if i not in list(range(21,41))+list(range(42,62))]
-print('hand rig idx',hand_idx.tolist(),flush=True)
-print('keypoint vertex support',int((K[:,:18439].abs().sum(0)>0).sum()),flush=True)
-
-def stat(x):
- x=torch.as_tensor(x).double().reshape(-1)
- return {k:float(v) for k,v in [('mean',x.mean()),('p95',torch.quantile(x,.95)),('max',x.max())]}
-
-def skel(p,h):
- ct=rig.character_torch
- return ct.joint_parameters_to_skeleton_state(ct.model_parameters_to_joint_parameters(torch.cat([p,h],1)))[...,:3]
-
-out={'seed':20260912,'scale_rank':int(torch.linalg.matrix_rank(C)), 'scale_singular_values':torch.linalg.svdvals(C).tolist(),'datasets':{}}
-for split,n in [('coco',6000),('aic',1000),('mpii',1000),('harmony4d',1000),('3dpw',1000)]:
- paths=sorted((root/f'data/sam3d_gt_{split}/annotations').glob('*.npz'))
- rng=np.random.default_rng(20260912)
- chosen=[paths[i] for i in rng.choice(len(paths),min(n,len(paths)),replace=False)]
- records=[]
- for path in chosen:
- with np.load(path) as z: records.append([z[k].copy() for k in ['mhr_model_params','shape_params','joints_3d','joints_2d','bbox_square']])
- p,h,j,j2,bbox=[torch.from_numpy(np.stack([r[i] for r in records])).float() for i in range(5)]
- b=torch.cat([p[:,6:136],torch.zeros(len(p),3)],1)
- bc=u.compact_model_params_to_cont_body(b); br=u.compact_cont_to_model_params_body(bc)
- pr=p.clone(); pr[:,6:136]=br[:,:130]
- scales=((p[:,136:].double()-mu)@pinv)@C+mu
- rootR=roma.euler_to_rotmat('ZYX',p[:,3:6])
- root_e=roma.rotmat_to_euler('ZYX',rootR)
- root_rig=roma.euler_to_rotmat('xyz',p[:,3:6])
- round_rig=roma.euler_to_rotmat('xyz',root_e)
- naive_e=roma.rotmat_to_euler('ZYX',u.batch6DFromXYZ(p[:,3:6],True))
- d={'n':len(p),'root_fraction_abs_gt_2_8':float((p[:,3:6].abs()>2.8).float().mean()),'root_near_pi_sample_fraction':float(((p[:,3:6].abs()-torch.pi).abs()<.34).any(1).float().mean()),'root_near_gimbal_0_1_fraction':float(((p[:,4].abs()-torch.pi/2).abs()<.1).float().mean()),'body_roundtrip_param_abs':stat((br-b).abs()),'body_roundtrip_samples_gt_1e_4':int(((br-b).abs().amax(1)>1e-4).sum()),'scale_projection_abs':stat((scales-p[:,136:]).abs()),'root_roundtrip_rig_matrix_abs':stat((round_rig-root_rig).abs()),'root_naive_parameter_abs':stat((naive_e-p[:,3:6]).abs())}
- # Hand roundtrip through the distinct, joint-ordered conversion.
- hr=p.clone()
- for ids in [s['head_pose.hand_joint_idxs_left'].long(),s['head_pose.hand_joint_idxs_right'].long()]:
- hr[:,ids]=u.compact_cont_to_model_params_hand(u.compact_model_params_to_cont_hand(p[:,ids]))
- d['hand_roundtrip_param_abs']=stat((hr-p).abs())
- # Full FK probes on a fixed subset; cm -> mm. Keep runtime and memory bounded.
- errs={k:[] for k in ['body_roundtrip_skel_mm','scale_projection_skel_mm','scale_projection_pve_mm','hand_only_nonhand_kp_mm','skeleton_kp_vs_exact_mm','skeleton_body_kp_vs_exact_mm','exact_kp_vs_annotation_mm','skeleton_kp_vs_annotation_mm','pelvis_midhip_mm','root_canonical_skel_mm']}
- for start in range(0,min(256,len(p)),16):
- pp=p[start:start+16]; hh=h[start:start+16]
- with torch.no_grad():
- vv,ss=rig(hh,pp,torch.zeros(len(pp),72)); ss=ss[...,:3]
- [email protected]([vv,ss],1)
- kp_s=W@ss
- errs['skeleton_kp_vs_exact_mm'].append((kp-kp_s).norm(dim=-1)*10)
- errs['skeleton_body_kp_vs_exact_mm'].append((kp[:,body_idx]-kp_s[:,body_idx]).norm(dim=-1)*10)
- sign=torch.tensor([1.,-1.,-1.])
- errs['exact_kp_vs_annotation_mm'].append((kp*sign/100-j[start:start+16]).norm(dim=-1)*1000)
- errs['skeleton_kp_vs_annotation_mm'].append((kp_s*sign/100-j[start:start+16]).norm(dim=-1)*1000)
- errs['pelvis_midhip_mm'].append(((kp[:,9]+kp[:,10])/2).norm(dim=-1)*10)
- errs['body_roundtrip_skel_mm'].append((skel(pr[start:start+16],hh)-ss).norm(dim=-1)*10)
- pc=pp.clone(); pc[:,136:]=scales[start:start+16].float()
- vc,sc=rig(hh,pc,torch.zeros(len(pp),72))
- errs['scale_projection_skel_mm'].append((sc[...,:3]-ss).norm(dim=-1)*10)
- errs['scale_projection_pve_mm'].append((vc-vv).norm(dim=-1)*10)
- ph=pp.clone(); ph[:,hand_idx]=0
- sh=skel(ph,hh)
- errs['hand_only_nonhand_kp_mm'].append(((W@sh)[:,body_idx]-kp_s[:,body_idx]).norm(dim=-1)*10)
- proot=pp.clone(); proot[:,3:6]=root_e[start:start+16]
- errs['root_canonical_skel_mm'].append((skel(proot,hh)-ss).norm(dim=-1)*10)
- d['geometry_n']=min(256,len(p))
- d.update({k:stat(torch.cat(v).flatten()) for k,v in errs.items()})
- # Visible hand extents within the stored 224px body crop, without claiming visibility labels exist.
- ext=[]
- for ids in [list(range(21,42)),list(range(42,63))]:
- xy=j2[:,ids]; side=(xy.amax(1)-xy.amin(1)).amax(1)
- ext.append(side/(bbox[:,2]-bbox[:,0])*224)
- d['hand_span_in_224_crop_px']=stat(torch.cat(ext))
- imgs=[]
- for path in chosen[:16]:
- im=root/f'data/sam3d_gt_{split}/images/{path.stem}.png'
- if not im.exists(): im=im.with_suffix('.jpg')
- if im.exists():
- with Image.open(im) as image: imgs.append(list(image.size))
- d['sampled_image_sizes']=sorted({tuple(i) for i in imgs})
- out['datasets'][split]=d
- print(split,json.dumps(d),flush=True)
-Path(__file__).with_name('measurements.json').write_text(json.dumps(out,indent=2)+'\n')
diff --git a/docs/todo.md b/docs/todo.md
deleted file mode 100644
index 3856a11..0000000
--- a/docs/todo.md
+++ /dev/null
@@ -1,603 +0,0 @@
-# Open work
-
-Cleared on 12 September 2026. The previous 884 lines were the 2026-09-06 audit
-backlog; most items had been implemented, superseded, or disproved, and the file
-had stopped being a reliable statement of what is open. Recover it with
-`git show HEAD~1:docs/todo.md` if you need an old entry.
-
-Two rules that kept the old file honest and are worth keeping:
-
-- **State the measurement, not the intuition.** An item is worth writing down
- once there is a number attached to it — what was measured, on what data, and
- what the result was.
-- **Delete an item when it lands or dies.** A file of stale entries is worse
- than an empty one, because it looks authoritative.
-
-Reference material: `benchmark/README.md` (evaluation protocol, conversion
-routes, measured floors, gen-6/7 tables), `instanthmr_distill_train/README.md`
-(label source, presets, size bounds, vertex term), `CLAUDE.md` (repo traps),
-`docs/accuracy_audit_2026-09-06.md` (the original audit, as a dated record).
-
-**Architecture constraint, confirmed 2026-09-12:** real-time inference on
-smartphones is a primary contribution. Keep efficient encoders such as HGNetV2
-and RepViT; DINOv3 is a reference, not a proposed deployment backbone. Evaluate
-architecture changes against the current model's latency, memory and operator
-support on the intended phone runtime. No acceptable slowdown has been
-quantified yet; do not assume a larger model or a repeated decoder is affordable
-from desktop GPU timings alone. Training-only changes should be distinguished
-from additions to the deployed inference path.
-
-The [2026-09-12 architecture audit](architecture_audit_2026-09-12.md) records
-code references and GT probes for the items below. Its accuracy opportunities
-are hypotheses unless accompanied by a matched training ablation.
-
----
-
-# 0. Next run: a baseline-preset model family (agreed 2026-10-25)
-
-Train `--preset baseline` at B0 / B1 / B2 / B4, 288 px (B4-224 already exists:
-g8h_lr1x_e600), same options and 600-epoch schedule as g8h_lr1x_e600.
-
-**Why, measured 2026-10-24** (`benchmark/results/gen9w_*.json`, every frame, best
-checkpoints): with the deployed RF-DETR boxes, g8h B4-224 (baseline) beats every
-v4 model, including v4 B4-288. On 3DPW test (J14+adapter PA) it scores 41.02
-against 41.27; on EMDB-1 (SMPL24+adapter PA) 51.27 against 52.67. v4 wins only on
-EMDB's annotated boxes (54.22 vs 62.87), which are looser than any detector's.
-288 beats 224 at every v4 size by 0.7-2.6 mm on 3DPW and 1.8-3.9 mm on EMDB
-(RF-DETR boxes), for +1.8-6 ms on desktop CPU.
-
-**Plan (2026-10-25), 288 only:** two arms per backbone, both with baseline losses
-and the baseline occlusion, no JPEG, on `data_win288`
-(`datasets_pipeline/jeanzay/28_dev_driver_g10.sh`):
-
-- `g10b_*`: `--preset baseline --windows standard`, i.e. the baseline framing.
-- `g10f_*`: `--preset baseline --windows random`, i.e. v4's framings.
-
-This also separates v4's two changes: g10f vs g9w isolates occlusion + JPEG, and
-g10b vs g10f isolates the framings. Judge on RF-DETR boxes and on the published
-protocols. Delete this item when the table is in `benchmark/results/`.
-
----
-
-# 1. Reproduce the teacher's regression pathway exactly
-
-**Status: implemented 2026-09-12, launched as generation 8.** Behind
-`--cont-head`, default off; the default path is verified bit-identical (1544
-state_dict keys, same sha256, forward and all 12 loss terms 0.000e+00 apart on
-48 real augmented samples under both `--losses legacy` and `rebalanced`). Code
-in `instanthmr_distill_train/mhr_cont.py` (the vendored conversions) and
-`ContMHRHead` in `train_distill_mhr_only.py`; asset built by
-`tools/build_cont_head_assets.py`; gated by `tools/verify_cont_head.py`.
-Measurements are in `datasets_pipeline/jeanzay/STATUS.md`, "Generation 8".
-
-**One deviation from §1.3 below, and it is deliberate.** That section says to
-build the root inverse with `roma`'s `"ZYX"`, matching the teacher's literal
-call. Measured against the rig -- `checkpoints/mhr_model.pt`, md5-identical to
-the teacher's own `assets/mhr_model.pt` -- driving `model_params[3:6]` and
-rigid-fitting the 125 joints it moves recovers **extrinsic XYZ**
-(`batch6DFromXYZ`) to 0.00000-0.014 deg, while `roma.euler_to_rotmat("ZYX", .)`
-of the same triple is **36-92 deg away**: roma's uppercase `"ZYX"` returns the
-triple Z-first, so it is the reverse of the rig's. Either convention
-round-trips, so both would train; this one is used because only then is the
-intermediate rotation matrix the body's actual root rotation, which is what
-makes the new chordal loss a physical angular error instead of a distance in a
-permuted space. §1.3's own instruction -- "the two conventions must not be
-silently conflated" -- is the reason.
-
-**Decision taken 2026-09-12:** reproduce the
-teacher's pathway *in full*, not a measurement-guided subset. The point is to
-remove the pathway as a confound, so a partial port would leave exactly the
-question it is meant to settle.
-
-## 1.1 The problem, stated precisely
-
-The student and SAM 3D Body feed MHR the same thing — the rig-native 204-vector —
-but they **regress in different spaces**, and the student's is the harder one.
-
-The teacher's head emits **519** numbers in a continuous space and converts them
-to the 204-vector *inside the head*, before MHR sees anything. Verified in
-`sam_3d_body/models/heads/mhr_head.py`, `forward()` (lines ~284-326) and
-`mhr_forward()` (lines ~198-215):
-
-| teacher head emits | dims | conversion applied inside the head | reaches MHR as |
-|---|---|---|---|
-| global rotation, 6D | 6 | `rot6d_to_rotmat` → `roma.rotmat_to_euler("ZYX", ·)` | 3 Euler angles |
-| body pose, continuous | 260 | `compact_cont_to_model_params_body` → 133, then `[:130]` | 130 |
-| shape | 45 | passes through raw | 45 |
-| bone scales, PCA | 28 | `scale_mean + coeffs @ scale_comps` | 68 bone scales |
-| hands (2 x 54) | 108 | `replace_hands_in_pose` into the body block | — |
-| face | 72 | multiplied by 0 in `forward()` | — |
-
-The 260-dim body space decomposes as **23 three-DoF joints x 6D (138)** +
-**58 one-DoF hinges x (sin, cos) (116)** + **6 one-DoF translation channels, raw
-(6)**. Those 6 raw channels are the `*_length/_width_flexible` parameters and
-land at `130:136` of the 204-vector — i.e. **the continuous representation does
-not bound them**, so they still need separate size constraints. Bounds on the
-expanded PCA scales must respect the scale subspace; see §1.3.
-
-The student instead emits the 204-vector directly from one `nn.Linear`:
-`head_global = nn.Linear(d_model, model_params_dim + shape_dim + cam_dim)` =
-204 + 45 + 3 = 252 numbers, all off a single query token
-(`shared_feats[:, 0, :]`). So it regresses raw Euler angles and 68 free bone
-scales where the teacher regresses 6D, sin/cos and 28 PCA coefficients.
-
-**The teacher uses a two-layer MLP and an initial pose estimate**, whereas the
-student uses a single linear layer. Reproduce the MLP and prediction relative
-to that estimate at a width compatible with the mobile budget. Valid continuous
-initialization matters: all-zero 6D vectors and all-zero sin/cos pairs are
-degenerate. Use a valid neutral-pose encoding, including identity rotations.
-
-## 1.2 Why this is not merely cosmetic — measured
-
-Over 6,000 `data/sam3d_gt_coco/annotations/*.npz` ground-truth samples:
-
-| parameter block | range | p99 \|angle\| | fraction beyond 2.8 rad |
-|---|---|---|---|
-| root rotation `3:6` | spans ±π: [−3.187, +3.361] | 3.13 rad | **13.59 %** |
-| local joints `6:130` | max 2.908 | **1.18 rad** | **0.0016 %** |
-
-- **22.32 %** of samples have at least one root axis within 0.34 rad of ±π,
- near a wrap boundary of the target.
-- The root's middle Euler axis spans the **±π/2 singularity** (max 1.6226 vs
- π/2 = 1.5708). At gimbal lock the first and third angles become degenerate;
- the range alone does not prove a sample lies exactly there. **2.25 %** are
- within 0.1 rad of it, **6.42 %** within 0.2 rad.
-- The 124 local rotation values in `6:130` essentially never wrap
- (p99 1.18 rad). This measures target angles, not an accuracy gain from changing
- their representation. The decision remains full reproduction.
-
-**Near-boundary targets do not prove current predictions cross that boundary.**
-Nor do they establish an expected millimetre improvement. That requires paired
-predictions and targets, followed by matched training ablations. Root-orientation
-improvements mostly affect MPJPE and orientation quality; PA alignment removes
-global rotation. Any improvement in PA-MPJPE from fixing root regression is
-indirect, not a guaranteed consequence of the angle histogram.
-
-Zhou et al., *On the Continuity of Rotation Representations* (CVPR 2019) is the
-reference for why a discontinuous target costs accuracy.
-
-## 1.3 What to implement
-
-Head emits 519 pose dims, or 447 if only the identically disabled 72-dim face
-block is omitted (**record that choice**), plus a separate 3-dim camera output.
-It then converts to the 204-vector with the
-teacher's own functions so the two pathways are identical:
-
-- `sam_3d_body.models.modules.rot6d_to_rotmat` + `roma.rotmat_to_euler("ZYX")`
-- `sam_3d_body.models.modules.mhr_utils.compact_cont_to_model_params_body`
-- `scale_mean + coeffs @ scale_comps` (both buffers are already in the
- teacher's state dict; they must be copied into the student or loaded from the
- rig)
-- the separate hand conversion and `replace_hands_in_pose`, using the teacher's
- hand buffers and joint-index mappings
-
-**The separate hand parameter blocks matter even without a hand encoder.** They
-overwrite the body representation's finger channels. Omitting that overwrite is
-not the same pathway. A 339-dim head retaining fingers only through the body
-representation would therefore be a partial port; a dedicated hand-crop encoder
-is not required to reproduce the full head.
-
-**Correct root inverse:** do not apply `batch6DFromXYZ` directly to the stored
-root triple. The teacher decodes with `roma.rotmat_to_euler("ZYX", R)`, so build
-the inverse with the matching convention and concatenate matrix **columns**:
-
-```python
-R = roma.euler_to_rotmat("ZYX", mhr_model_params[:, 3:6])
-root_6d = torch.cat([R[:, :, 0], R[:, :, 1]], dim=-1)
-```
-
-The rejected inverse returns `[0.7, 0.5, 0.3]` from `[0.3, 0.5, 0.7]` through
-the teacher's forward conversion. For the body inverse, pass `6:136` plus three
-zero jaw values to `compact_model_params_to_cont_body` (133 inputs). Hand
-targets use their own joint ordering and inverse. These targets all come from
-the existing GT 204-vector; no new annotation pass is needed. Test root rotation
-losses against the rig's actual rotation convention as well as this head-space
-round-trip; the two conventions must not be silently conflated.
-
-**Existing scale bounds cannot simply be applied after PCA expansion.**
-Coordinatewise tanh generally moves the expanded scales outside the teacher's
-scale subspace. Preserve that subspace when designing bounds, for example by
-constraining coefficients with validated limits; still constrain the six
-flexible size channels separately and preserve zero root translation. PCA
-coefficients are not inherently bounded. The checkpoint's 28x68 scale matrix
-has effective float32 rank 24, so use a stable truncated pseudoinverse if
-constructing coefficient targets, not its transpose. The audit's GT projection
-residual is <2.2e-8; retaining the subspace is compatible with sampled GT scales.
-
-Every FK / geometric loss stays where it is and operates on the converted 204,
-with gradients flowing back through the conversion. Prefer a rotation-space
-metric over raw Euler differences for root supervision. Canonical 6D L2 targets
-are possible, but their metric is not physical angular error: multiple 6D
-vectors encode the same rotation after orthogonalization. Changing the output
-representation and changing other loss norms are distinct experiments.
-
-Gate it behind a config flag whose default reproduces today's behaviour, and
-prove bit-identity of the old path (`--preset baseline` and `--losses legacy`
-are controls for runs in flight).
-
-## 1.4 Loss and regularisation differences — configuration matters
-
-The public SAM 3D Body release is **inference-only** — no training code, no loss
-definitions. The following is from the paper (§ *Model Training*), so treat it as
-the teacher's stated design, not as verified code:
-
-> *2D/3D Keypoint Loss: We supervise 2D/3D joint locations using an L1 loss,
-> incorporating learnable per-joint uncertainty to modulate the loss based on
-> prediction confidence. For 3D body and hand keypoints, we normalize them with
-> their respective pelvis and wrist locations before computing the loss.*
->
-> *Parameter Losses: MHR parameters (pose, shape) are supervised with L2
-> regression losses, and joint limit penalties are imposed to discourage
-> anatomically implausible poses.*
-
-**Already implemented in `--losses rebalanced`, including generation 7:**
-`apply_rebalanced_losses` sets `kp3d_loss="euclid"` and
-`w_keypoints3d=0.35`. This is unsquared Euclidean distance in metres, not
-SmoothL1 and not squared L2. **Do not re-fix the default SmoothL1 behavior as if
-generation 7 still used it.** The legacy/default configuration keeps SmoothL1
-for reproducibility. Switching the rebalanced path to the paper's coordinatewise
-L1 would be a new ablation, not completion of a missing fix.
-
-Deltas against the student today (deferred items are not implementation tasks):
-
-| item | teacher (paper) | student now | action |
-|---|---|---|---|
-| parameter loss type | **L2** on MHR parameters | `SmoothL1`, `pose_beta` 0.05 rad (`_m_beta`) | decide; SmoothL1 at beta 0.05 is nearly L1 in the operating range, not L2 |
-| **joint limit penalty** | present, on the pose angles | absent; existing size bounds are not a pose penalty | do not apply stored pose limits blindly: all 6,000 audited COCO GT samples violate at least one, largely from flexible joints with `[0,0]` limits; any future prior must be annotation-compatible |
-| keypoint loss type | L1 | legacy: SmoothL1; **rebalanced/gen-7: unsquared Euclidean**, already implemented | no correction pending for rebalanced |
-| per-joint weighting | **learnable per-joint uncertainty** | fixed weights only (`finger_weight`, default 1.0, 0.2 in gen-6/7) | deferred with annotation-aware weighting; not current work |
-| 3D keypoint normalisation | pelvis (body) / wrist (hands) before the loss | direct difference in rig-origin coordinates; `fk_scale` is only a warm-up multiplier | deferred by user on 2026-09-12; do not implement now |
-| warm-up on the 3D term | yes | `kp3d_warmup_steps` = 2000 | already present |
-| root vs local split | one continuous body space | `pose_split` separates `loss_pose_root` (beta 1.0) from `loss_pose` (beta 0.05) | revisit once the root is 6D |
-
-**The loss metric is a separate bug from the output space.** Converting fixes the
-network's output manifold; it does not fix the metric. A SmoothL1 in radians on
-the root still scores a prediction at −π+ε against a target at π−ε as a ~2π
-error when it is a tiny rotation. The 13.59 % statistic measures large GT angles,
-not the frequency of this prediction error. Put the root's parameter loss on
-the rotation matrix (Frobenius or a numerically stable geodesic formulation)
-rather than on the Euler triple. Test finite gradients near identity and π.
-
-Current student weight defaults, for the record: `w_pose` 1.0, `w_scale` 0.1,
-`w_shape` 1.0, `w_cam` 0.1, `w_keypoints3d` 2.0, `w_keypoints2d` 10.0,
-`w_3d_joints` 1e-3, `w_reproj` 0.01, `w_simcc` 1.0, `w_verts` 0.0. The teacher's
-λ weights are not published.
-
-For **generation 7**, override that defaults list with `w_shape=0.03`,
-`w_keypoints3d=0.35`, `pose_beta=0.05`, `pose_split=True`,
-`finger_weight=0.2`, and the explicitly requested `w_verts=0.35`.
-
-## 1.5 Verification
-
-1. **Bit-identity of the old path** with the flag off — diff a few dozen
- augmented samples against `git show HEAD:<file>`, comparing *alternating*
- calls (the TorchScript MHR forward differs by ~5e-10 on its first invocation
- in a process).
-2. **Round-trip correctness**: test padded body targets, separate hand targets
- and the corrected root inverse on real GT. Compare rotations and FK geometry
- as well as parameters: equivalent Euler triples need not be numerically
- identical after canonicalization. Verify finite gradients and valid neutral
- initialization before training.
-3. **The progress metric is the fitting gap, not 3DPW.** The student is roughly
- twice as far from its labels as it needs to be *on images it has already
- trained on*, and that — not split, conversion or selection — is where the
- ~23 mm gap to SOTA lives. Track it directly while changing the regression
- space.
-
- **Build the measurement; it does not exist today.** The number quoted in the
- old todo (**62.90 mm** with the student's predicted MHR params against
- **31.84 mm** with the GT params, held-out vertex error over 8,000 crops on
- five splits) came from a throwaway MLP ablation whose code is gone — no
- `z_body` / `head_SMPL` / `smpl_head` remains in the tree. It was also a
- by-product of a different question (whether an SMPL head should read the
- trunk latent or the predicted parameters; answer: 0.07 mm, irrelevant), it
- carries the MLP's own 31.84 mm floor, and it used `SMPL_MALE` throughout.
- Do not quote it next to a benchmark number.
-
- **Prefer a floor-free version:** run the student over a few thousand
- *training* crops, push the predicted 204-vector and the GT 204-vector through
- `MHRForwardPass` separately, and compare the 70 keypoints **root-relative,
- not Procrustes-aligned**. No learned head means no floor; no Procrustes means
- body-size and proportion errors stay visible. ~20 lines.
-
- **What exists today and its trap.** The epoch log's `held-in` line is a 2 %
- slice of the training corpus scored against the GT labels — the right family
- of metric. But `train_distill_jz.py:900-903` **overwrites** `Mesh_PA_MPJPE`
- and `Mesh_MPJPE` with the 3DPW numbers before they reach `history.jsonl`
- (deliberately, so selection keys off 3DPW), so those two fields carry no
- held-in information. Only `Mesh_PA_MPJPE_body` survives. Current values:
- g7r_s0 ep169 **34.15** (3DPW-val 39.93), g7h_s0 ep111 **40.52** (43.65),
- g6vv_s ep252 **43.44** (42.22), g6bv_s ep299 **41.82** (42.24) — note that
- two of the four runs are *worse* on their own training data than on 3DPW,
- which is worth explaining on its own.
-4. `python tools/ddp_smoke.py --backbone <name> --w-verts <w>` before submitting
- — a changed head shape is exactly the class of change that passes every
- single-process check and then aborts DDP on step 2.
-5. Seed spread is wide; anything under ~2 mm on 3DPW needs 2-3 seeds.
-
----
-
-# 2. Crop resolution — the corpus is stored at the network input size
-
-**Status:** open.
-
-`datasets_pipeline/build_split.py --body-size` defaults to **224**, and the
-stored crops are 224x224 (`data/sam3d_gt_*/images/*.png`, verified). The network
-input is also 224. Two consequences:
-
-1. **Upsampling existing crops adds no source detail.** Training at 256 or 288
- may still change feature sampling density, but evaluate that separately from
- rebuilding crops with `build_split.py --body-size <N>` from original frames.
-2. **`--preset v2`'s zoom augmentation currently upsamples.** Its zoom ceiling is
- 2.0, so a zoomed-in sample is interpolated from the 224 crop: up to 2x
- upsampling. The model is being trained to expect blur that a real crop at that
- scale would not have. Storing crops at 288-320 and letting augmentation crop
- *down* into 224 can address this **without changing the network at all**.
- The current dataset resizes to 224 **before** affine augmentation, so the
- implementation must instead sample the affine crop directly from the stored
- higher-resolution source. Changing stored resolution alone is insufficient.
-
-**Keep the efficient encoder and mobile speed constraint.** Test modest input
-increases (256/288) on the current HGNetV2/RepViT, or earlier-stage/stride-16
-features at the same 224 input, as separate options. The current encoders emit
-7x7 final features at 224; stride-16 features would give 14x14. More image tokens
-also cost decoder time and memory. Benchmark the complete exported path on the
-intended phone runtime before promoting either option. Do not replace the
-encoder with DINOv3 to reproduce the teacher's 32x32 feature grid.
-
-For scale: NLF uses EfficientNetV2-S at **256 px** and EfficientNetV2-L at
-**384 px** (NLF paper, *Implementation details*). 384² is 2.94x the pixels of
-224². The student is 40.1 M parameters total / 22.6 M backbone (`g7r_s0`,
-`repvit_m2_3`); `g7h_s0` (`hgnetv2_b4`) is 31.9 M / 13.6 M. NLF-L is therefore
-~3x the student in both parameters and pixel budget — the fair size comparison
-is NLF-**S**, not NLF-L.
-
-Cost to rebuild: the corpus is 4.2 M crops, and `sam3d_gt_harmony4d` has no
-originals on the cluster (staged from the laptop), so a rebuild is not free.
-Decide the target size once and do it in one pass.
-
-## Verification
-
-- Confirm stored crop size and that `build_split.py` writes what was asked.
-- Re-measure 3DPW J14+adapter PA-MPJPE at matched epochs, 2-3 seeds.
-- Check separately whether removing the augmentation upsampling helps at the
- *same* 224 input — that isolates the blur effect from the resolution effect.
-
----
-
-# 3. Model selection
-
-**Status:** open, low priority relative to items 1 and 2.
-
-Selection is `min` over epochs of 3DPW **validation** J14 PA-MPJPE with the
-adapter applied, from `train_distill_jz.py` via `val3dpw.py` (validation split,
-12 sequences, stride 5, 1,993 person-frames, H36M GT).
-
-**The split choice is correct — keep it.** Selecting on 3DPW test or EMDB would
-turn the only clean numbers in the repo into training metrics.
-
-**Selection noise is not a problem, measured.** The EMA validation curve has std
-**0.08-0.17 mm** over the last 30 epochs, so `min` over ~300 epochs is only
-0.2-0.6 mm optimistic. The val→test gap (g7r_s0: 39.93 → 41.55, **+1.62 mm**) is
-genuine split difference, not a winner's curse. Older runs showing test
-*better* than val by 6.5-7.2 mm (b2_s0, b3_s1, v3_s1) had their validation
-scored by a pre-2026-09-05 `val3dpw.py`; those gaps are not comparable.
-
-Two real defects:
-
-1. **One metric, blind to most of what matters.** PA-MPJPE removes translation,
- rotation and scale; root-relative MPJPE removes translation. So `cam_trans`
- drift, body-size error, temporal jitter and identity are all invisible to the
- selector — and they are what the vid1/vid4/vid5 visual check actually
- judges. The ranking is not stable across benchmarks either: `g6vv_s` is best
- on 3DPW and under projected-GT-joint EMDB boxes, and **fourth** under EMDB's
- own boxes. Add an EMDB column and a jitter number (frame-to-frame
- acceleration; rigid bone-length CV, GT floor 1.06 %) to the selection.
-2. **`summarize_runs.py` reports a pair no checkpoint achieves.**
- `datasets_pipeline/jeanzay/summarize_runs.py` reduces every METRICS column
- with `min` *independently* across epochs, and `_dpw` additionally takes
- `min(raw, ema)` per epoch. So the `g7r_s0` row "39.93 PA / 69.27 MPJPE" is PA
- from epoch 169's EMA weights and MPJPE from a different epoch, possibly
- different weights. Report the pair at the selected epoch; keep per-column
- records in a separate row if they are wanted.
-
----
-
-# 4. Decoder feedback within the smartphone inference budget
-
-**Status:** open experiment, conditional on preserving real-time smartphone
-inference. User accepts an adapted version if full teacher feedback costs too
-much. Do not copy the teacher's six-layer, 1024-wide decoder wholesale.
-
-**Verified difference:** the student predicts MHR once after four decoder
-layers. The teacher predicts provisional geometry between layers, projects
-keypoints, samples image features at their 2D locations, and supplies updated
-2D/3D coordinate embeddings to subsequent layers. It has 70 auxiliary 3D
-tokens, although it has no independent XYZ regression head. Restoring a freely
-regressed XYZ head alone would not reproduce this mechanism.
-
-The expected benefit is better local articulation from inspecting image
-evidence near provisional joints. No generation-7 accuracy gain or phone
-latency has been measured for this change.
-
-## Implementation and verification
-
-- Start with one feedback update between existing decoder layers; reuse the
- encoder features and keep decoder width/depth fixed for the first ablation.
- Try a small body-joint subset and the existing cheap skeleton FK before
- considering full mesh geometry or additional 3D tokens at every layer.
-- A cheaper alternative is intermediate SimCC coordinates driving feature
- sampling, without FK inside decoding. Label this as a 2D-guided approximation
- and measure it separately; it is not the teacher's geometry-derived feedback.
-- Intermediate supervision can be training-only, but feature updates used by
- later layers belong to inference too. Do not assume they can be removed at
- export without testing the resulting accuracy change.
-- The current parameter-output graph leaves FK outside the neural model.
- Putting FK or bilinear feature sampling inside decoding changes the export
- requirements. Check actual phone-backend support and fallbacks, end-to-end
- latency, peak memory and sustained throughput, alongside matched accuracy.
- If the change costs too much, simplify or leave it unselected; desktop FPS
- and FLOP estimates do not establish smartphone feasibility.
-
----
-
-# 5. Exact 70-landmark readout using a small vertex subset
-
-**Status: implemented 2026-09-13** behind `--exact-landmarks`, default off, as
-a training-loss change only. Asset built by `tools/build_landmark_assets.py`
-(the first 70 rows of `head_pose.keypoint_mapping`, vertex columns restricted to
-their 468-vertex support); `MHRForwardPass._build_vertex_subset` now takes the
-union of those 468 with the vertex-loss subset; `loss_3d_native` and
-`loss_reproj` read through `regress_keypoints_exact`. Gated by
-`tools/verify_exact_landmarks.py`. Measurements in
-`datasets_pipeline/jeanzay/STATUS.md`, "Generation 8b".
-
-Verified: **2.4e-04 mm** against full 18,439-vertex skinning with the full
-teacher mapping, on GT, on pose perturbed by N(0, 0.25) rad, and under resampled
-identity; gradients agree to 2e-07 relative in `model_params` and 2e-09 absolute
-in `shape_params`. The 468 and the 595 vertex-loss samples **overlap in only
-10**, confirming the warning below. Union skinning is 1053 vertices and costs
-nothing measurable (181.2 ms for 595 alone vs 180.9 ms for the union, batch 64,
-forward+backward). `loss_verts` still averages over exactly its own 595 with
-uniform weighting.
-
-The disagreement this removes, measured against the teacher's readout on
-identical GT geometry: the fitted `(70, 127)` matrix is **3.037 mm mean over the
-30 non-finger landmarks** (1.306 mm over all 70, 18.05 mm worst), in the
-2.93-3.75 mm band recorded below. The exact readout is 0.000 mm by construction.
-It also gives the 70 landmarks a gradient into `shape_params` -- they move
-0.821 mm mean when identity is resampled, against exactly 0.000 mm for the
-fitted matrix, which is blind to identity by construction.
-
-**Scoring stays on the fitted readout by default, and 2026-09-22 measured what
-that costs.** `val3dpw.py`, `benchmark/eval_3dpw_ckpt.py` and
-`benchmark/eval_emdb_ckpt.py` all take a `landmarks` / `--landmarks` switch now;
-`fitted` is the default everywhere and reproduces every recorded number to the
-decimal. The switch is worth **0.13 mm** of 3DPW-test J14+adapter PA-MPJPE
-(g8h_s0 40.80 -> 40.66, g8h_s1 40.89 -> 40.77, g8r_s0 41.27 -> 41.13, stride 1)
-and 0.09-0.11 mm of EMDB-1 SMPL24+adapter PA-MPJPE, and it is worth the same
-0.13 mm on g7h_s0, which was trained WITHOUT `--exact-landmarks` (42.90 ->
-42.77). So it is a property of the operator, not of the training flag, and it is
-an offset rather than a re-ranking: every checkpoint moves the same way by the
-same amount, far inside the 34-49 mm seed spread.
-
-The mismatch it removes is real, though: the adapters
-(`adapter_j14_h36m_teacher.npz`, `adapter_smpl24_teacher.npz`) are fitted from
-the stored annotation `joints_3d`, which came out of the teacher's exact mapping
--- so `fitted` feeds an adapter one operator's keypoints when it was fitted on
-another's. Nothing on the GT side needs changing: 3DPW's H36M joints and EMDB's
-SMPL joints are SMPL-derived and know nothing about the MHR readout.
-
-`fitted` stays the default for two reasons: it is what the exported graph's
-consumer computes, so the published number keeps describing the deployed path;
-and flipping it would shift every recorded row by 0.13 mm, which is exactly the
-kind of quiet offset that makes two generations uncomparable in one table.
-
-Original scoping, kept for context: preserve MHR's
-direct pose-parameter prediction and rig/mesh separation. First evaluate as a
-training-loss change; a deployed landmark-readout change is a separate decision
-subject to the smartphone speed constraint.
-
-**What these landmarks are:** the 70 annotation keypoints supervised by
-`loss_3d_native` and used for reprojection, including nose, shoulders, elbows,
-knees, wrists, fingers, toes and surface landmarks such as the acromion. They
-are not the rig's 127 kinematic joints or its joint-angle parameters.
-
-The teacher first predicts pose, scale and shape parameters. MHR computes
-skeletal joints by forward kinematics and skins the mesh using those joints.
-Only then does a fixed checkpoint matrix read out landmark positions:
-
-```text
-predicted parameters → FK joints J → skinning → vertices V
- J + V → fixed mapping → 70 landmark positions
-
-teacher: K = W_joint @ J + W_vertex @ V
-student: K ≈ W_fitted @ J
-```
-
-This is **not reconstructing joint angles from mesh vertices**. Angles remain
-network predictions, and joints remain FK outputs. Mesh-surface landmarks
-depend on shape and skinning as well as skeletal pose; replacing them with a
-skeleton-only fit introduces an approximation.
-
-Verified examples from the actual DINOv3 checkpoint's mapping:
-
-| landmark | contributing mesh vertices | contributing skeletal joints |
-|---|---:|---:|
-| nose | 8 | 0 |
-| left elbow | 21 | 0 |
-| left knee | 0 | 1 |
-| right wrist | 0 | 1 |
-| left small-toe tip | 24 | 0 |
-
-On 256 GT poses per split, the student's approximation differs from the
-teacher's mapping by **2.93–3.75 mm mean over the 30 nonfinger landmarks**.
-Those are operator discrepancies on identical GT geometry, not predicted
-benchmark gains. The full audit includes errors against the stored annotations.
-
-**Feasible reuse:** the teacher's first 70 mapping rows reference only **468
-distinct vertices**. `MHRForwardPass._build_vertex_subset` already restricts
-skinning to selected vertices, but currently chooses a farthest-point subset.
-Extend it to accept the exact mapping-supported indices, or their union with
-the current 595 vertex-loss indices. Do not assume the 595 already contain the
-468, or that changing the subset size alone selects the right vertices.
-
-## Implementation and verification
-
-- Load the teacher's fixed mapping and select its first 70 rows. Split its
- vertex and joint columns, restrict vertex columns to their nonzero support,
- and remap them to the compact skinning indices. Preserve units and axes.
-- Reuse one FK pass and the existing shape-dependent subset skinning; avoid
- constructing full vertices solely for this landmark readout. The existing
- shape-blend work and any other full-topology operations must still be
- accounted for when profiling; 468 indices are not an end-to-end speed claim.
-- Prove landmark and gradient equivalence against the full reference mapping
- on real GT and perturbed predictions, including shape changes. Measure the
- union-subset cost at the actual training batch size.
-- Gate the new loss readout so current runs remain reproducible. A training-only
- implementation adds no deployed operators, but deployment would retain the
- approximate landmarks; evaluate accuracy through that retained deployment
- path as well. If exact deployed landmarks are wanted, measure their phone
- runtime cost separately and document the changed readout in comparisons.
-- Reuse this subset in decoder feedback only if item 4's latency gate passes;
- it is not a requirement to run skinning between all decoder layers.
-
----
-
-# 6. Camera-ray conditioning and crop-camera parameterization
-
-**Status:** open, focused on absolute pose and subject placement; keep the
-efficient encoder and benchmark any inference overhead.
-
-**Focal-aware conditioning is already present in generation 7** via
-`--cliff-focal`. A learned projection already embeds the three conditioning
-values, including crop size and offset normalized by focal length, into the
-queries. Do not describe this task as simply adding focal information or adding
-a learnable focal token; neither captures the remaining teacher difference.
-
-Two distinct experiments:
-
-1. **Spatial camera rays.** For each feature-grid position, recover its original
- image coordinate `(u, v)` through the crop transform and compute its viewing
- direction, proportional to `((u-cx)/fx, (v-cy)/fy, 1)`. Here `cx, cy` denote
- the camera principal point, not the person-box centre. The teacher encodes
- these rays and fuses them with image features. Test a lightweight fusion at
- the student's existing feature resolution; do not blindly copy the wide
- teacher module. Rays are derived from camera/crop metadata, not learned
- focal estimates. Keep augmentations, intrinsics and inference fallback
- calibration consistent. With ideal centred square crops, much of this is
- derivable from existing CLIFF plus grid position, so gains are not automatic.
-2. **Separate crop-camera head.** The student directly predicts translation in
- metres; the teacher predicts crop scale and lateral offsets, then converts
- them to full-camera translation using focal length and box geometry. Assess
- that parameterization separately from ray embeddings. Check coordinate/sign
- conventions, depth stability and train/export/inference consistency.
-
-Measure absolute camera-space error, depth/placement stability and reprojection
-as well as body MPJPE. PA-MPJPE removes translation, rotation and scale and is
-not the main success metric for these changes. Keep the current conditioning
-and camera path available as the matched control.
-
-**Deferred by user, 2026-09-12:** pelvis/wrist-relative supervision and
-annotation-aware weighting (including the related uncertainty-weighting
-proposal) are not current implementation tasks.
diff --git a/docs/training.md b/docs/training.md
deleted file mode 100644
index 918fef5..0000000
--- a/docs/training.md
+++ /dev/null
@@ -1,129 +0,0 @@
-# Training
-
-**Label source.** Training is supervised by the released ground-truth
-annotations of
-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
-— the human MHR fits used to build SAM 3D Body — not by teacher inference.
-Distilling from `facebook/sam-3d-body-dinov3` with `tools/annotate_dataset.py`
-is an option for imagery the dataset does not cover; see
-[`docs/annotation.md`](annotation.md). The `train_distill_*.py` filenames are
-historical and say nothing about which labels you are using.
-
-The InstantHMR training notebook is
-[`notebooks/distill_transformer_decoder.ipynb`](../notebooks/distill_transformer_decoder.ipynb).
-It is the source of truth for the notebook workflow — this page is a tour, not a
-re-implementation.
-
-**For cluster training, the notebook is not the source of truth.** That is
-`instanthmr_distill_train/train_distill_mhr_only.py` (dataset, losses, model)
-driven by `train_distill_jz.py` (DDP, mixture sampling, resume, 3DPW
-validation), and the two have diverged from the notebook. See
-[`instanthmr_distill_train/README.md`](../instanthmr_distill_train/README.md)
-for which script to run, what `--preset v2` changes, and the pair-index cache.
-
-## Setup
-
-```bash
-pip install torch timm onnx tqdm loguru
-jupyter notebook notebooks/distill_transformer_decoder.ipynb
-```
-
-You'll need the annotations produced by
-[`docs/annotation.md`](annotation.md). The notebook expects the directory
-layout:
-
-```
-<dataset_root>/
-├── images/ *.png (224×224 crops)
-└── annotations/ *.npz (teacher outputs)
-```
-
-## What the notebook does
-
-1. **Dataset** — loads the `(image, annotation)` pairs, applies light
- augmentation (colour jitter, horizontal flip with paired-joint remapping),
- and yields the 4-tuple `(image, cliff_cond, gt_targets, mask)`.
-2. **Architecture** — defines InstantHMR (RepViT-M1.5 backbone + 9-token
- cross-attention decoder) exactly as it ships in this repo's
- `instanthmr/inference.py` (the runtime path is parameter-compatible
- with the trained `.pth`).
-3. **Loss** — weighted L1 on each output head:
- - mhr_params, shape_params, cam_trans (parameter regression),
- - joints_3d (mean per-joint position error in camera frame),
- - joints_2d (after re-projection back to full-frame pixels).
-4. **Training loop** — AdamW with cosine schedule + warm-up, mixed precision
- on CUDA, gradient clipping at 1.0.
-5. **Validation** — held-out split, MPJPE on `joints_3d` reported.
-6. **Export** — at the end of training, the model is exported to
- `instanthmr.onnx` with dynamic batch axis using the same input schema
- the demo expects (see [`models/README.md`](../models/README.md)).
-
-## Hyperparameters at a glance
-
-| | |
-|---|---|
-| Backbone | RepViT-M1.5 (`timm`) |
-| Decoder layers | 2 |
-| `d_model` | 256 |
-| Heads | 4 |
-| Dropout | 0.1 |
-| Image size | 224 |
-| Optimiser | AdamW (β=0.9/0.999, wd=1e-2) |
-| LR schedule | cosine + linear warm-up |
-| Precision | mixed (fp16 forward, fp32 master) |
-
-The exact values for the released checkpoint live in the notebook's config
-cell.
-
-## Re-exporting to ONNX
-
-The notebook's export cell is reproduced below for reference:
-
-```python
-dummy_image = torch.randn(1, 3, 224, 224, device=device)
-dummy_cliff = torch.randn(1, 3, device=device)
-
-torch.onnx.export(
- model.eval(),
- (dummy_image, dummy_cliff),
- "instanthmr.onnx",
- input_names=["image", "cliff_cond"],
- output_names=[
- "mhr_params", "shape_params", "cam_trans",
- "joints_2d", "joints_3d",
- ],
- dynamic_axes={
- "image": {0: "batch"},
- "cliff_cond": {0: "batch"},
- "mhr_params": {0: "batch"},
- "shape_params": {0: "batch"},
- "cam_trans": {0: "batch"},
- "joints_2d": {0: "batch"},
- "joints_3d": {0: "batch"},
- },
- opset_version=17,
-)
-```
-
-Verify with `onnxruntime`:
-
-```python
-import onnxruntime as ort
-s = ort.InferenceSession("instanthmr.onnx",
- providers=["CPUExecutionProvider"])
-print([(i.name, i.shape) for i in s.get_inputs()])
-print([(o.name, o.shape) for o in s.get_outputs()])
-```
-
-## Tips
-
-- **Don't drift the preprocessing.** The demo's
- `InstantHMR._preprocess` (`instanthmr/inference.py`) is the exact
- inverse of the annotation crop: 1.2× square padded crop → 224 → ImageNet
- norm. Any change to either side has to be mirrored on the other.
-- **Joints are body-centred.** Don't add `cam_trans` to `joints_3d` before
- computing the loss; the teacher stores them as body-centred, and
- InstantHMR's last layer is trained against that.
-- **CLIFF conditioning lives in full-frame coords**, not crop coords.
- `cx_norm = 2·cx/W − 1` etc. Recomputing this from the crop will silently
- degrade the camera head.
diff --git a/instanthmr/inference.py b/instanthmr/inference.py
index 36cde81..493cea4 100644
--- a/instanthmr/inference.py
+++ b/instanthmr/inference.py
@@ -13,10 +13,10 @@ into full-frame pixels.
Two graph layouts are supported, and the session tells them apart by itself:
* **5 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d, joints_3d)``
- — the student has its own 3D coordinate head (``train_distill.py`` and the
- optimized / correctives scripts).
+ — older students with their own 3D coordinate head (earlier trainers,
+ not in this repository).
* **4 outputs** ``(mhr_params, shape_params, cam_trans, joints_2d)`` — the
- MHR-only student (``train_distill_mhr_only.py``) has no 3D head, exactly
+ MHR-only student (``train.py``) has no 3D head, exactly
like the SAM3D teacher. Its 70 keypoints are derived here by running the
MHR skeleton forward on the predicted parameters, so such a model needs an
MHR backend (see :mod:`instanthmr.mhr_renderer`) passed as ``mhr=``.
@@ -268,7 +268,7 @@ class InstantHMR:
def _joints_3d(self, mhr_params: np.ndarray, shape_params: np.ndarray) -> np.ndarray:
"""Regress the 70 keypoints from predicted MHR parameters.
- Mirrors ``train_distill_mhr_only.MHRForwardPass.get_native_keypoints``:
+ Mirrors ``train.MHRForwardPass.get_native_keypoints``:
skeleton forward, cm -> m, Y/Z flip, then the (70, 127) affine map. The
student was trained through exactly this operator, so any other
keypoint definition would silently shift the predictions.
@@ -276,7 +276,7 @@ class InstantHMR:
if self._mhr is None:
raise RuntimeError(
"this ONNX graph has no 'joints_3d' output — it comes from "
- "train_distill_mhr_only.py, where the 3D keypoints are derived from "
+ "train.py, where the 3D keypoints are derived from "
"the MHR forward pass.\n"
"Pass an MHR backend, e.g.\n"
" from instanthmr.mhr_renderer import build_mhr\n"
@@ -498,7 +498,7 @@ class InstantHMR:
# CLIFF conditioning vector (full-frame coords). Must match the variant
# the checkpoint was TRAINED with -- see `cliff_focal` in
- # instanthmr_distill_train/train_distill_mhr_only.py. A model trained
+ # training/train.py. A model trained
# with the focal-aware form and run with the pixel form (or the other
# way round) does not error; it silently mis-places the person in depth.
if self.cliff_focal:
diff --git a/instanthmr/mhr_renderer.py b/instanthmr/mhr_renderer.py
index b08a5e9..97b3bb4 100644
--- a/instanthmr/mhr_renderer.py
+++ b/instanthmr/mhr_renderer.py
@@ -6,11 +6,11 @@ InstantHMR and runs a forward pass through Meta's MHR body model to obtain
* the full body **mesh** (skinned vertices), and
* the 70 native **keypoints**, regressed from the 127 skeleton joints.
-The keypoints matter because ``train_distill_mhr_only.py`` deletes the
+The keypoints matter because ``train.py`` deletes the
student's 3D coordinate head: its ONNX graph has four outputs instead of five
and the 3D pose must be derived here, exactly the way the teacher derives it
from the mesh. The (70, 127) affine map lives in
-``instanthmr_distill_train/assets/mhr_j127_to_kp70.npy`` and was fitted against
+``training/assets/mhr_j127_to_kp70.npy`` and was fitted against
the teacher's own labels (held-out residual: 1.36 mm mean).
Two interchangeable backends
@@ -57,7 +57,7 @@ import numpy as np
_REPO_ROOT = Path(__file__).resolve().parent.parent
#: (70, 127) affine map from MHR skeleton joints to the 70 native keypoints.
-DEFAULT_KP_REGRESSOR = _REPO_ROOT / "instanthmr_distill_train/assets/mhr_j127_to_kp70.npy"
+DEFAULT_KP_REGRESSOR = _REPO_ROOT / "training/assets/mhr_j127_to_kp70.npy"
#: TorchScript rig shipped with the training code.
DEFAULT_MHR_SCRIPT = _REPO_ROOT / "checkpoints/mhr_model.pt"
@@ -141,7 +141,7 @@ class _MHRBase:
"""Raw MHR (centimetres, Y-up / Z-back) -> InstantHMR (metres, Y-down / Z-forward).
Negating both Y and Z keeps the frame right-handed after the Y flip.
- This is the exact transform ``train_distill_mhr_only.MHRForwardPass``
+ This is the exact transform ``train.MHRForwardPass``
applies before the keypoint regression, so student and demo agree.
"""
out = xyz / 100.0
@@ -180,7 +180,7 @@ class _MHRBase:
raise RuntimeError(
f"no MHR keypoint regressor loaded (looked for {DEFAULT_KP_REGRESSOR}).\n"
"It ships with the training code; regenerate it with\n"
- " python instanthmr_distill_train/train_distill_mhr_only.py --fit-regressor"
+ " python training/train.py --fit-regressor"
)
torch = self._torch
pose, squeeze = self._as_batch(mhr_params, NUM_MODEL_PARAMS)
diff --git a/instanthmr/visualizer.py b/instanthmr/visualizer.py
index 99fc469..41c7562 100644
--- a/instanthmr/visualizer.py
+++ b/instanthmr/visualizer.py
@@ -18,7 +18,7 @@ Logs, per frame:
- ``timing/total_ms`` : scalar plot of full pipeline latency.
- ``timing/fps`` : scalar plot of effective frames-per-second.
- ``contact/person_N/{left,right}``: per-foot contact probability, and the
- feet drawn red while planted (> 0.5), for a
+ feet drawn red while planted (> contact_threshold), for a
prev-token graph that has a contact head.
``rr.init(..., spawn=True)`` is called automatically so the viewer opens
@@ -88,6 +88,7 @@ class RerunVisualizer:
mhr_renderer: "MHRRenderer | None" = None,
mesh_alpha: float = MESH_ALPHA,
reference=None,
+ contact_threshold: float = 0.5,
):
import rerun as rr
@@ -103,6 +104,7 @@ class RerunVisualizer:
int(round(255 * min(max(mesh_alpha, 0.0), 1.0))),
)
self._prev_num_ref = 0
+ self.contact_threshold = contact_threshold
rr.init(application_id, spawn=spawn_viewer)
# SAM3D / MHR convention: right-handed, Y-down camera frame.
@@ -196,6 +198,7 @@ class RerunVisualizer:
annotated = self._draw_overlay(
image_rgb, persons, detector_ms, hmr_ms, render_ms, total_ms,
+ self.contact_threshold,
)
rr.log("camera/image", rr.Image(annotated))
@@ -257,7 +260,7 @@ class RerunVisualizer:
if person.contact_prob is not None:
for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):
- planted = prob > 0.5
+ planted = prob > self.contact_threshold
rr.log(f"{path}/feet/{name}", rr.Points3D(
positions=joints[list(kp)], radii=0.03 if planted else 0.015,
colors=[PLANTED_COLOR if planted else FREE_COLOR]))
@@ -343,6 +346,7 @@ class RerunVisualizer:
hmr_ms: Optional[float],
render_ms: float,
total_ms: Optional[float],
+ contact_threshold: float = 0.5,
) -> np.ndarray:
"""Draw per-person bbox + 2D skeleton + a timing HUD onto the frame."""
out = image_rgb.copy()
@@ -378,7 +382,7 @@ class RerunVisualizer:
if person.contact_prob is not None:
r = max(4, int(round(0.012 * max(h, w))))
for name, kp, prob in zip(FOOT_NAMES, FOOT_KP, person.contact_prob):
- planted = prob > 0.5
+ planted = prob > contact_threshold
for k in kp:
if valid[k]:
cv2.circle(out, tuple(j2d[k].astype(int)), r,
diff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.md
deleted file mode 100644
index 9014060..0000000
--- a/instanthmr_distill_train/README.md
+++ /dev/null
@@ -1,924 +0,0 @@
-# InstantHMR — training scripts
-
-## What the labels are — read this first
-
-**Training is supervised by the released ground-truth annotations of
-[`facebook/sam-3d-body-dataset`](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)**
-— the human MHR fits Meta used to build SAM 3D Body. They are **not** teacher
-inference. Every run in this repo's result tables, and the whole cluster corpus
-(`sam3d_gt_sa1b` / `_aic` / `_harmony4d` / `_coco` / `_mpii`), is of this kind.
-
-Distilling from the `facebook/sam-3d-body-dinov3` model is a supported
-**option**, not the default: `tools/annotate_dataset.py` runs it over your own
-images and is the only thing that writes `data/sam3d_distill_mix/`. Reach for it
-to label imagery the released dataset does not cover, and note the cost — a
-student trained on teacher output cannot exceed the teacher, while one trained on
-the ground truth can.
-
-Two naming traps that follow from the history:
-
-- **"distill" in a filename means nothing about the labels.**
- `train_distill_mhr_only.py`, `train_distill_jz.py` and the
- `data/sam3d_distill_mix/` directory keep the name from when distillation was
- the only path. Only `annotate_dataset.py` produces teacher labels.
-- **"teacher" further down this file usually means "the training target".** Where
- a number was measured against a label — parameter ranges, oracle ablations,
- the `adapter_j14_h36m_teacher.npz` fit — the label is the dataset's ground
- truth unless the text says `annotate_dataset.py` explicitly.
-
-## Which script
-
-Four files, and the difference matters.
-
-| file | what it is |
-|---|---|
-| `train_distill_mhr_only.py` | **the trainer.** Dataset, augmentation, student architecture, MHR forward pass, all losses, metrics, the single-GPU loop. Everything of substance lives here. |
-| `train_distill_jz.py` | **the multi-GPU entry point.** `import train_distill_mhr_only as T` plus what a cluster needs: DDP, mixture sampling, resumable checkpointing, held-out 3DPW validation, the pair-index cache. Run this one on a cluster. |
-| `val3dpw.py` | 3DPW held-out validation used by `train_distill_jz.py`. Cross-validated against `benchmark/eval_3dpw.py` — they agree exactly. |
-| `train_distill.py` | **legacy.** The original notebook port. See the warning below before using it. |
-
-They are one trainer in two layers, not two trainers. A change to the dataset or
-a loss goes in `train_distill_mhr_only.py` and both entry points pick it up; only
-cluster plumbing and the CLI belong in `train_distill_jz.py`. Do not fork the
-base file — that is how the ONNX bug below happened.
-
-> **`train_distill.py` predates the SimCC 2D head.** Exporting a SimCC
-> checkpoint through its ONNX path silently zeroes the 2D head, and the symptom
-> is keypoints collapsing to the bounding-box centre with no error raised. Use it
-> for reference, not for exporting current checkpoints.
-
-## The `--preset` switch
-
-`train_distill_jz.py --preset {baseline,v2}` selects the model generation.
-`baseline` is the default and is bit-for-bit identical to the pre-v2 trainer, so
-the two are directly comparable on the same corpus and the same seeds.
-
-`v2` (see `apply_v2_preset`) bundles four changes from MeTRAbs (2007.07227) and
-NLF (2407.07532):
-
-| item | change | why |
-|---|---|---|
-| truncation | zoom ceiling 2.0, `geom_trans` 0.20 | `bbox_square` is exactly 1.2x the tight box on every split, so the old symmetric ±0.25 shaved ~2% off each side and essentially never truncated. Measured effect: images with a truncated keypoint go from 17.2% to 57.0%. |
-| prerequisite A | `mask_oob_2d` | SimCC spans ±`kp2d_range` and `pred_2d` is a soft-argmax over the same bins, so a target at 1.67 is unreachable. Clamping asserts it sits at the edge and leaves a permanent saturating gradient. Mask instead; the 3D loss still supervises those joints. Do **not** widen `kp2d_range` — `bin_w = 2*range/(bins-1)`, so that degrades every in-frame joint. |
-| prerequisite B | `cliff_follows_aug` | CLIFF describes the box the crop came from. Flip and rotation already moved it; `dx/dy/scale` did not, which was harmless at ±0.08 and 1.25 and is not at truncation strength. |
-| occlusion | `RandomErasing` p 0.2 → 0.7 | MeTRAbs Table 7: 52.8 → 49.3 mm on top of an already-strong colour pipeline. |
-| JPEG | `RandomJPEG` p 0.3, q 30–90 | The one item on NLF's list that was missing, and the most deployment-relevant — phone camera stacks emit JPEG. |
-| absolute pose | `w_reproj` 0.01 → 0.5, flipped half un-masked, `loss_cam` unsquared Euclidean | MeTRAbs Table 8 and NLF §3.2 both insist on supervising the camera-frame pose. At 0.01 the term was decorative and half the batch was masked out of it. |
-
-Two things to know when reading v2 results:
-
-- **PA-MPJPE and root-relative MPJPE are both blind to the translation half of
- the absolute-pose change.** Procrustes removes translation; root-relative
- alignment removes it too. The reprojection and flip-unmasking parts should
- still move them. An absolute metric needs 3DPW's real intrinsics and a
- `f_true/f_assumed` correction — the corpus uses a synthetic
- `f = sqrt(H² + W²)`, ~12% off 3DPW's real focal.
-- **Truncation augmentation is not expected to show on a full-body benchmark.**
- MeTRAbs's 124.7 → 77.8 mm is measured on deliberately truncated crops. It is
- in v2 because the papers already established it works, not to be re-proved.
-
-`w_reproj = 0.5` is the least-grounded number in the preset (a 50x jump, chosen
-so the term stops being decorative). Sweep it first if v2 underperforms:
-`--preset v2 --w_reproj 0.1`.
-
-## The `--losses` switch
-
-**Orthogonal to `--preset`.** `--preset` controls the input pipeline and the
-absolute-pose terms; `--losses {legacy,rebalanced}` controls the loss budget.
-`legacy` is the default and is bit-identical to the pre-rebalance trainer, so
-b2 / v2 runs already in flight stay valid controls. The four arms are
-`baseline`, `v2`, `baseline + rebalanced`, `v2 + rebalanced`.
-
-`rebalanced` (see `apply_rebalanced_losses` in `train_distill_mhr_only.py`)
-comes from two measurements on a trained checkpoint, both reproducible from the
-repo root against `data/sam3d_gt_coco`:
-
-- **an oracle ablation** — replace one predicted group with the teacher's value
- and re-measure. Of the 37.8 mm of removable PA-MPJPE, `pose[6:136]` (the 130
- local joint angles) accounts for **34.3 mm**; the bone scales, the root and
- the 45 identity blendshapes account for **0.00 mm each**.
-- **a per-term gradient norm** — back-propagate each term alone. `loss_shape`
- owned 27% of the update direction and `loss_cam` 15%, against `loss_pose`'s
- 4.7% on 21% of the batch.
-
-| item | change | why |
-|---|---|---|
-| shape | `w_shape` 1.0 -> 0.03 | Driving MHR params 204–248 to ±2σ moves the 127-joint skeleton by exactly `0.00e+00` cm, and `MHRForwardPass.get_joints` substitutes zeros for `shape_params` anyway. The blendshapes drive the mesh only. Kept non-zero so the deployed mesh identity stays supervised. |
-| pose mask | `pose_split` | `loss_pose` was masked to `m_ident` (21% of the batch) because the dataset rotates the labels. Model params 3/4/5 map to joint-parameter slots 10/11/12 and nothing else, so `pose[6:136]` is exactly invariant to the in-plane rotation and only needs the flip mask — 21% -> 60% coverage on the parameters that carry all of the error. Root stays on `m_ident`; the two halves keep their per-parameter weight, so this changes the mask and not the root-vs-local balance. |
-| pose beta | `pose_beta` 1.0 -> 0.05 | SmoothL1's beta is in radians here, so 1.0 is ~10x the p95 error and the term is a scaled MSE. |
-| 3D loss | `kp3d_loss` `euclid`, `w_keypoints3d` 2.0 -> 0.35 | SmoothL1's beta of 1.0 is one **metre**, so at a 37.7 mm mean / 119 mm p95 error every sample is in the quadratic branch: 16x less gradient than L1 and 31x less than NLF's unsquared Euclidean at the same weight. The weight drops because the form under it changed. |
-| fingers | `finger_weight` 0.2 | 40 of the 70 keypoints are finger joints and they carry 61% of `loss_3d_native`'s mass, while J12/J14 contain none of them and SA-1B — 55% of the batch under `--mix sqrt` — records hands as 8% observed. Applied to `loss_3d_native` and `loss_reproj`, the two FK-derived geometric terms; the 2D head's own terms are untouched. |
-
-**The unsquared Euclidean is gentler where divergence happens, not harsher.**
-`d‖x‖/dx` is a unit vector, so its gradient is flat in the error, where
-SmoothL1(beta=1) grows with it. Measured against the form it replaces, at the
-weights above: **0.62x** the gradient on a freshly initialised model (515 mm
-error) and 6.4x once trained (60 mm). That is the opposite of the shape that
-produced the LR blow-ups in `datasets_pipeline/jeanzay/STATUS.md`.
-
-Validation also gains `Mesh_PA_MPJPE_body` / `Mesh_MPJPE_body` over the 30
-non-finger keypoints, reported next to the existing metrics. **Selection is
-unchanged** — and with `--val-3dpw` on, both are replaced by the 3DPW J12
-numbers anyway.
-
-## `--val-3dpw-gt` (added 2026-09-05)
-
-3DPW validation now scores against the published-protocol ground truth — SMPL
-run forward on the sequence's `poses`/`betas`, then the Human3.6M regressor —
-instead of the pickles' raw `jointPositions`. Two things follow.
-
-`ThreeDPWValSet` needs the precomputed reference next to `sequenceFiles/`, built
-once by `benchmark/make_3dpw_gt.py` and rsynced to the cluster as
-`$SCRATCH/instanthmr/3dpw/gt_h36m/`. It is 14 MB for all three splits. Without
-it the job raises at startup rather than silently falling back.
-
-The two references are ~13 mm apart on J12 PA-MPJPE, so **a selection metric
-from before this change cannot be compared with one from after**. A resumed run
-must keep the metric it started on: `--val-3dpw-gt jointpositions`, or
-`DPW_GT=jointpositions` for `52_train_ddp.slurm`. `run_config.json` records
-which was used.
-
-## `--cliff-focal` (added 2026-09-06, mandatory)
-
-**Use it on every new run.** It changes the CLIFF conditioning from
-image-normalised pixels to a perspective-correct form:
-
-```
-[ atan((cx - W/2) / f), atan((cy - H/2) / f), box_px / f ]
-```
-
-— where the box sits in the field of view, and how large it is *angularly*.
-`f` is `cam_focal_length[1]` (fy), a single scalar because the rotation
-augmentation mixes the two image axes and `fx != fy` on the calibrated splits.
-
-**Why it is not optional any more.** Under a *fixed* focal the two forms are
-interchangeable, and the corpus this trainer was written for had exactly that:
-every crop from `tools/annotate_dataset.py` carries the synthetic
-`f = sqrt(H^2+W^2)`, i.e. `f/diag = 1.000` with **zero** spread. The rebuilt
-corpus carries the datasets' own focals instead:
-
-| split | f / diag | sd |
-|---|---|---|
-| `sam3d_distill_mix` (old corpus) | 1.000 | **0.000** |
-| `sam3d_gt_aic` | 1.184 | 0.371 |
-| `sam3d_gt_coco` | 1.049 | 0.344 |
-| `sam3d_gt_mpii` | 0.961 | 0.266 |
-| `sam3d_gt_harmony4d` | 0.285 | 0.010 |
-
-With that spread the pixel form is **ambiguous**: of 4,000 COCO crops, 124 have
-a near-twin under it (conditioning distance < 0.005) whose true depths differ by
-**1.34 m**. The network can only predict the conditional mean, and the residual
-leaves as unstable `cam_trans` and body scale. Measured on 3DPW test, models
-trained on the new corpus jitter at 220-234 mm/frame² of translation
-acceleration (98% of it in depth) against 148 for one trained on the
-constant-focal corpus, and a ground-truth floor of 7.
-
-**PA-MPJPE cannot see any of this** — Procrustes removes translation and scale,
-and the metric moves by ~0.1 mm either way. Judge the change on `cam_trans`
-stability and bone-length variance, never on the benchmark.
-
-**Three places compute this vector and all three must agree**:
-`SAM3DStudentDataset.__getitem__`, `val3dpw._preprocess` (which takes the real
-`fy` from 3DPW's `cam_intrinsics`), and `instanthmr.inference.InstantHMR`.
-Mismatching them raises nothing; it silently mis-places the person in depth.
-
-### Carrying the flag past training (fixed 2026-09-06)
-
-Until 2026-09-06 nothing outside the trainer knew about it: the checkpoint
-evaluator and the ONNX exporter both built a default `DistillConfig`, so a
-focal-aware model was scored and deployed with the pixel form. Two lookups
-close that, and neither needs a flag in the normal case:
-
-| consumer | how it learns the form |
-|---|---|
-| `train_distill_jz.py` validation | `cfg.cliff_focal` directly |
-| `benchmark/eval_3dpw_ckpt.py` | `run_config.json` beside the checkpoint |
-| `instanthmr.inference.InstantHMR` | `cliff_focal` in the ONNX metadata, stamped by `tools/pth_to_onnx.py` |
-
-`T.config_from_checkpoint()` in `train_distill_mhr_only.py` is the single
-implementation. It also restores `bound_scales` — from the weights, since
-`--bound-scales` registers `scale_lo`/`scale_hi` buffers and their presence
-*is* the flag. Both scripts print what they inferred and where it came from;
-`--cliff-focal` / `--no-cliff-focal` override it for a checkpoint whose
-`run_config.json` is missing or predates the flag.
-
-**The value of `f`, once the form is known, is a separate question.** Training
-and both 3DPW evaluators use the dataset's own focal — `cam_focal_length[1]`
-per annotation, and 3DPW's calibrated `K[1,1]` (1969.23 px on 10 of the 12
-validation sequences, 1961.85 on the two landscape ones). Deployment has no
-annotation, so `instanthmr.inference` falls back to
-`FOCAL_FALLBACK_DIAG * sqrt(H^2+W^2)` with `FOCAL_FALLBACK_DIAG = 1.05`, and
-`demo.py --focal <px>` supplies the real one. The fallback applies to
-focal-aware graphs **only**: a pixel-conditioned checkpoint was trained on a
-corpus where `cam_focal_length = sqrt(H^2+W^2)` exactly, so `1.00 x diag` is
-that model's camera and is left untouched.
-
-Measured cost of getting the form wrong: **0.13 mm** J14+adapter PA-MPJPE on a
-170-frame 3DPW validation probe (47.46 vs 47.33) — consistent with the ~0.1 mm
-above. Like the flag itself, this matters for absolute placement and jitter,
-not for the headline number.
-
-The default is `False` and is bit-identical to the pre-flag trainer, verified
-over augmented samples tensor by tensor.
-
-## `--bound-scales` / `--anomaly-safe-fallback` (added 2026-09-06, mandatory)
-
-**Use both on every new run.** They fix the divergence that killed all four
-runs of the 2026-09-06 sweep at epochs 24-32 (`fno_s*`, `fsa_s*`), and every
-earlier sweep that was "fixed" by lowering the peak LR.
-
-### What was actually wrong
-
-`head_global` is a plain `nn.Linear`. 77 of its outputs are **body-size**
-parameters that the MHR forward kinematics multiplies along the kinematic
-chain, and nothing bounded them:
-
-| block | what it is | unbounded behaviour (measured on `checkpoints/mhr_model.pt`) |
-|---|---|---|
-| `0:3` | root translation | linear; +100 -> 22 m skeleton. Teacher value is **exactly 0.000e+00** in all 24,000 sampled annotations across all six splits — the person is placed by `cam_trans`. |
-| `130:136` | `*_length/_width_flexible` | linear, transform weights up to 10.0; +100 -> 32 m |
-| `136:204` | the 68 bone scales | **multiplicative**: +10 -> 336 m, +20 -> 703 km, +25 -> 28,300 km |
-
-Note `130:136`. Those six sit at the end of the *pose* block but they drive
-joint **translation** channels, not rotations, so they are size parameters.
-Bounding only `136:204` still let a blown-up head reach a 375 m skeleton.
-`6:130` are genuine rotations and stay at ~1.5 m at any magnitude — leave them
-alone.
-
-Loss on real teacher targets, with the head drifted by `d` on the size block:
-
-| drift | total loss, unbounded | total loss, `--bound-scales` |
-|---|---|---|
-| 0 | 6.5 | 6.5 |
-| 10 | **140** (over `anomaly_loss_threshold=100`) | 15.6 |
-| 20 | **2.2e5** | 25.1 |
-
-That is where the `4.06e7` / `8.76e13` losses in the dead logs come from.
-
-### Why the guard made it permanent rather than catching it
-
-An anomalous batch was discarded whole. But `loss_scale` / `loss_pose` in that
-same batch are small and point straight back at the teacher — they are the
-**only** force pulling the sizes out of the runaway region — so skipping
-removed the fix along with the fault and quarantined those samples for good.
-Two amplifiers turned that into a dead job:
-
-* `bad` is all-reduced with `MAX`, so **one** bad sample out of 4x64 = 256
- vetoes the step for every rank. At 1% runaway samples 92% of steps are
- vetoed; at 2%, 99.4%. The run aborts when 2% of the data is bad, not the model.
-* The EMA rollback (`anomaly_skip_patience`, `max_ema_rollbacks`) existed only
- in `train_distill_mhr_only`'s single-GPU loop. The DDP path had the abort and
- **no recovery at all**.
-
-`--anomaly-safe-fallback` steps on `safe_loss_subset(losses)` — everything
-except `FK_LOSS_KEYS` — instead of discarding the batch. Verified: with the
-scales sitting +20 off the teacher, that gradient has cosine **+1.0000** with
-`teacher - prediction` and moves **100%** of the 68 scales the right way, while
-the full loss is 2.9e5. The rollback is now in the DDP loop too.
-
-### Why lowering the LR never worked
-
-`OneCycleLR(pct_start=0.1)` puts the peak at 10% of the run — epoch 30 of 300.
-The four dead runs died at epochs 24-32. Lowering `max_lr` does not remove the
-peak, it just slows the random walk toward the runaway region.
-
-### The bound
-
-`assets/mhr_size_bounds.npz` holds, per parameter, the union of the rig's own
-`character_torch.parameter_limits` and the observed teacher range; the model
-widens each by `--scale-bound-margin` (0.5) and squashes with `tanh`. The rig
-bounds every one of these to `[-1.1, 1.1]`; the teacher respects that for the
-scales but **not** for `130:136`, which the rig pins to `[0,0]` while the
-teacher uses them up to 1.64 (`leg_length_flexible`) — hence the union.
-
-Measured over 24,000 annotations from all six splits: **0.000000%** of teacher
-targets fall outside the bound and the worst sits at `|atanh| = 0.80`, well
-inside tanh's linear region, so the bound costs nothing. With the flag on, a
-head deliberately blown up to `bias ~ N(0, 5000)` still yields a 1.58 m
-skeleton. Exports to ONNX (matches torch to 1.2e-7).
-
-Regenerate the asset from the rig plus the corpus if either changes; the file
-also stores `rig_lo` / `rig_hi` unmodified for reference.
-
-Both flags default to **off** and are bit-identical to the pre-flag trainer,
-verified tensor by tensor. `run_config.json` records `bound_scales`,
-`scale_bound_margin` and `anomaly_safe_fallback`.
-
-### The 12-dimensional null space (not fixed, know it exists)
-
-`parameter_transform[:, :204]` has **rank 192**. Twelve directions move the
-parameters and leave the skeleton *exactly* unchanged — they pair each
-`*_flexible` parameter against its `scale_*` partner (`scale_spine_length` vs
-`spine_length_flexible`, `scale_hip_width` vs `hip_width_flexible`, and the
-spine rotations). Along those, every geometric loss has zero gradient and only
-`loss_pose` / `loss_scale` act, so the model is being asked to regress 12
-dimensions of solver-arbitrary teacher noise that no image can determine.
-Confirmed directly: pushing `scale_spine_length` to 20.3 along a null direction
-moves the skeleton by 1.2e-7 m. The bound now keeps that wandering finite;
-collapsing the redundancy properly would be a separate change.
-
-## `--w-verts` — the surface (PVE) term (added 2026-09-07)
-
-Every geometric loss in this trainer is blind to body shape. `get_joints` zeroes
-the 45 `shape_params` before the forward pass, and that is not a shortcut — the
-identity coefficients move the 127-joint skeleton by **exactly** `0.000e+00 cm`.
-They are mesh-only. So `w_shape` in parameter space was the only thing
-supervising them, and at `--losses rebalanced` that weight is 0.03, carrying
-0.2% of the trunk gradient.
-
-`--w-verts W` adds a term that skins the rig and compares vertices, which is the
-only route by which a geometric gradient can reach shape. Measured
-`d(loss)/d(shape_params)`: **8.9e-03** through vertices, **exactly 0.000e+00**
-through joints.
-
-**What it costs to ignore shape.** Running the rig on real teacher parameters
-with the identity zeroed against the true identity: **5.95 mm mean, 22.68 mm max
-per-vertex error**. That is a floor a shape-blind model cannot perceive, let
-alone fix.
-
-### It uses the rig's own mesh, not a separate LOD
-
-The obvious way to make this cheap is to load `lod6.fbx` (595 vertices against
-lod0's 18,439). Don't — it needs `pymomentum` to parse, which segfaults against
-the torch in this environment, and lod6 is a *different topology*, so the
-teacher correspondence would have to be fitted rather than being exact.
-
-Instead `n_verts` picks a farthest-point subset of the rig's **own** 18,439
-vertices and rewrites the three flattened influence tables of the rig's own
-`linear_blend_skinning` — `(vertex, bone, weight)` triples — to reference only
-those, under a compacted index. The skinning op is untouched, so the result is
-the full mesh's answer restricted to those vertices, **verified bit-exact to
-4.6e-05 cm**. Same topology, same vertex ids as the teacher: the correspondence
-is exact, not fitted. The default `n_verts=595` is lod6's vertex count, chosen
-so the sampling density matches what the LOD would have given.
-
-Measured at batch 64 on an RTX 4070:
-
-| path | fwd+bwd | vs skeleton-only |
-|---|---|---|
-| skeleton FK only (what the trainer already did) | 4.85 ms | — |
-| + 595-vertex subset | 5.44 ms | **+0.6 ms** |
-| + all 18,439 vertices | 21.6 ms | +16.7 ms |
-
-Against a 207 ms step that is **+0.3%** for the subset and +8.1% for the full
-mesh. On a real training step, end to end: **+0.9% wall time and +0 MiB peak
-GPU** — the backbone activations dominate and the vertex tensors fit in slack.
-
-### Choosing the weight
-
-The term is an unsquared Euclidean distance in **metres**, the same form as
-`loss_3d_native` under `--losses rebalanced`, so the weights are directly
-comparable and `loss_verts / w_verts * 1000` reads as mean PVE in millimetres.
-Per-term gradient norm on the trunk, pretrained backbone, `v2 + rebalanced`:
-
-| term | \|grad\| | share |
-|---|---|---|
-| `loss_reproj` | 33.97 | 61.3% |
-| `loss_verts` (at w=1.0) | 7.21 | 13.0% |
-| `loss_cam` | 4.51 | 8.1% |
-| `loss_3d_native` | 3.74 | 6.7% |
-| `loss_pose` | 2.93 | 5.3% |
-| `loss_shape` | 0.12 | **0.2%** |
-
-**`--w-verts 0.35` is the recommended setting**: `|grad| = 2.52`, 5.0% of the
-update, just under `loss_3d_native` and nowhere near `loss_reproj`. At 1.0 it
-would be the second-largest term in the budget, which is not what a first trial
-should do.
-
-`loss_verts` is in `FK_LOSS_KEYS`, so `--anomaly-safe-fallback` drops it along
-with the other forward-kinematics terms on an anomalous batch — a blown-up bone
-scale explodes the vertices exactly as it explodes the joints. It is also
-multiplied by `fk_scale`, so it ramps over `kp3d_warmup_steps` (2000) like
-`loss_3d_native` and `loss_reproj`; a PVE read from the first epoch is
-suppressed by that ramp, not genuinely small.
-
-`w_verts = 0.0` is the default and disables the term **and every code path it
-touches** — `MHRForwardPass` is then constructed with `n_verts=0` and builds no
-subset at all. Verified: every loss term matches `git show HEAD` to the last
-printed digit on 48 real augmented samples, under both `baseline/legacy` and
-`v2/rebalanced`.
-
-**There is no PVE metric yet, only the loss.** See `docs/todo.md` item 13.
-
-## `--crop-centre-fix` (added 2026-09-07)
-
-`--preset v2` sets `cliff_follows_aug`, which moves the CLIFF conditioning to
-describe the *augmented* box. The crop-centre half of that correction was
-missing a division by the zoom: the warp is `u' = s*R*(u - c) + c + t`, so
-inverting it puts the visible window's centre at `-t/s`, not `-t`.
-
-Verified against the exact preimage of the crop centre under the real
-`M_total` that `cv2.warpAffine` was called with, over 400 augmented COCO crops
-at v2 strength, rotation disabled so the zoom is the only variable:
-
-| | mean | p95 | max |
-|---|---|---|---|
-| off (the gen-5 behaviour) | 7.73 px | 21.00 px | 34.93 px |
-| **on** | **0.68 px** | 1.12 px | 1.50 px |
-
-The residual is the bbox-rounding floor. **Default off** so generation 5 and
-`--preset baseline` reproduce bit-for-bit; it is a no-op under `--preset
-baseline`, which does not set `cliff_follows_aug` at all. Pass it for every new
-`v2` run.
-
-This is a wrong **input**, not a wrong label — the reprojection chain is exact
-to 0.000 px — and it only affects lateral placement, which PA-MPJPE cannot see.
-Do not expect the benchmark to move; expect the conditioning to stop lying.
-
-The rotation half of the same correction is **not** settled: see `docs/todo.md`
-item 11.
-
-## `--cont-head` — the teacher's continuous regression space (added 2026-09-12)
-
-**What it changes.** Without it, the student emits the rig-native 204-vector
-straight out of one `nn.Linear`: raw Euler angles and 68 free bone scales.
-SAM 3D Body does not. Its head emits a *continuous* parameterisation and
-converts to the 204-vector inside the head, before MHR sees anything. This flag
-reproduces that pathway.
-
-| the head emits | dims | converted, inside the model, to |
-|---|---|---|
-| root rotation, 6D | 6 | 3 extrinsic-XYZ Euler angles |
-| body pose, continuous | 260 | 130 joint angles |
-| identity coefficients | 45 | passed through unchanged |
-| bone-scale PCA coefficients | 28 | 68 bone scales |
-| two hand blocks | 2 x 54 | the 54 finger channels, overwritten |
-
-447 numbers total — the teacher's 519 minus the 72-dim face block its own
-`forward()` multiplies by zero. They come from a two-layer MLP at the student's
-own 512 width on LayerNorm'd pose-token features (`nn.TransformerDecoder` is
-built here with `norm_first=True` layers and no final norm, so its output is
-unnormalised), plus a learned initial estimate. Camera translation moves to its
-own 3-dim head, unchanged in parameterisation.
-
-**Nothing downstream changes.** The model still returns 204 MHR parameters, 45
-identity coefficients, camera translation and 70 2D keypoints; every loss, the
-forward kinematics, `tools/pth_to_onnx.py` and `instanthmr.inference` see the
-contract they already saw. `config_from_checkpoint` reads the flag back out of
-the weights (the `cont_head.*` buffers ARE the flag), so an export needs no
-argument.
-
-**Why 260 for the body.** 23 three-DoF joints x 6D (138) + 58 one-DoF hinges x
-(sin, cos) (116) + 6 raw translation channels (6). Those last six are the
-`*_length/_width_flexible` parameters at `130:136` — body SIZE, not rotation —
-and **the continuous space does not bound them**, so they keep their own tanh
-against the rig limits exactly as under `--bound-scales`.
-
-**The root convention is the rig's, not the teacher's literal call.**
-`mhr_head.forward()` decodes the root with `roma.rotmat_to_euler("ZYX", R)`.
-Measured against `checkpoints/mhr_model.pt` (md5-identical to the teacher's own
-`assets/mhr_model.pt`): driving `model_params[3:6]` and rigid-fitting the 125
-joints it moves recovers **extrinsic XYZ** to 0.00000–0.014 deg, while roma's
-`"ZYX"` of the same triple is **36–92 deg away** — roma returns the triple
-Z-first, so it is the reverse of the rig's. Either convention round-trips and
-both would train; the rig's is used because only then is the intermediate
-rotation matrix the body's *actual* root rotation, which is what makes the
-accompanying chordal loss a physical angular error rather than a distance in a
-permuted space. All of this lives in `mhr_cont.py`, which is a transcription of
-the teacher's own functions verified at **0.000e+00**.
-
-**Bone scales are bounded as coefficients, and that ends the runaway.**
-`--bound-scales` clamps the 68 expanded scales coordinatewise, which is wrong
-for a PCA head: an arbitrary point of the 68-dim box is generally not in the
-24-dimensional column space `scale_comps` spans, so the clamp silently leaves
-the subspace. The bound moves onto the 28 coefficients, symmetric about zero so
-an untrained head sits exactly on `scale_mean`. Measured: feeding the head
-`N(0, 25)` — the regime that produced a 28,300 km skeleton and killed four runs
-at epochs 24–32 — now gives a **2.66 m** skeleton, scales still in-subspace to
-1.2e-07.
-
-**The root loss changes with it** (`root_rot_loss`, set by the same flag) from
-SmoothL1 on the Euler triple to the squared chordal distance
-`||Rp - Rt||_F^2 / 4`, on the rotation the head built rather than one rebuilt
-from the decoded Euler triple — `atan2` in the backward path is unstable, and
-2.25% of corpus roots sit within 0.1 rad of the ±pi/2 gimbal. Divided by the six
-root parameters the old term averaged over, so it is budget-neutral: 0.00003 vs
-0.00003 at 1 deg, 0.00063 vs 0.00063 at 5 deg, and 0.333 vs 0.437 at 179 deg —
-matched where it matters, saturating instead of growing.
-
-**Cost: none worth measuring.** +0.37 M parameters, and ONNX CPU single-thread
-batch 1 goes 102.03 -> 102.49 ms on `repvit_m2_3` and 64.55 -> 64.19 ms on
-`hgnetv2_b4`. The 980 extra graph nodes are all elementwise on 447 numbers.
-
-**Before using it:**
-
-```sh
-python tools/verify_cont_head.py --data_root data # 24 gates, exits non-zero on any failure
-python tools/ddp_smoke.py --backbone <name> --w-verts 0.35 --cont-head --data_root data/sam3d_gt_mpii
-```
-
-`instanthmr_distill_train/assets/mhr_cont_head.npz` (30 KB) carries the
-teacher's scale and hand bases plus the measured coefficient bounds;
-`tools/build_cont_head_assets.py` rebuilds it from the teacher checkpoint and
-the corpus. **It is a new file and must be rsynced**, or the job dies at
-startup.
-
-Default off. With it off every tensor in the trainer is bit-identical to the
-previous commit — verified: 1544 state_dict keys, same sha256, forward and all
-12 loss terms 0.000e+00 apart on 48 real augmented samples, under both
-`--losses legacy` and `--losses rebalanced`.
-
-## `--exact-landmarks` — the teacher's own 70-landmark readout (added 2026-09-13)
-
-**What it changes.** The student derives its 70 annotation keypoints from the
-127 skeleton joints alone, through a fitted `(70, 127)` matrix. SAM 3D Body does
-not — it reads them off the *skinned mesh* and the joints together, with a fixed
-matrix from its checkpoint:
-
-```
-teacher: K = W_joint @ J + W_vertex @ V
-student: K ≈ W_fitted @ J
-```
-
-**21 of the 70 landmarks have no joint contribution at all** — nose, elbows, toe
-tips, acromion and other surface points — so a skeleton-only matrix cannot
-represent them, and none of the 70 can move with body shape. Measured against
-the teacher's readout on identical GT geometry:
-
-| readout | all 70 | 30 non-finger | worst |
-|---|---|---|---|
-| fitted `(70, 127)` | 1.306 mm mean | **3.037 mm mean** | 18.05 mm |
-| exact subset readout | 0.000 mm | 0.000 mm | 0.000 mm |
-
-Those targets came out of the teacher's mapping, so this removes an operator
-mismatch between the prediction and its own label. It affects `loss_3d_native`
-and `loss_reproj` only.
-
-**Why it is affordable.** The first 70 mapping rows reference just **468
-distinct vertices**. `MHRForwardPass` already skins an arbitrary subset exactly
-(not approximately — it rewrites the rig's three flattened influence tables and
-leaves the op untouched), so this takes the **union** of those 468 with the
-`--w-verts` subset. Do not assume the 595 already contain the 468: measured,
-they overlap in **10**. Union is 1053 vertices, and the cost is unmeasurable —
-181.2 ms for 595 alone against 180.9 ms for the union, batch 64, forward+backward.
-The skinning is dominated by the forward-kinematics pass, not the vertex count.
-
-`loss_verts` keeps averaging over exactly its own 595 farthest-point samples
-with uniform weighting, via `verts_loss_pos`; averaging over the union instead
-would silently reweight the term and break comparability with generations 6-8.
-
-**It gives identity a second geometric gradient.** Resample `shape_params` and
-the exact landmarks move **0.821 mm mean**; the fitted ones move exactly
-**0.000 mm**. Before this, `--w-verts` was the only route by which the 45
-identity coefficients saw any geometry at all.
-
-**Verified** (`python tools/verify_exact_landmarks.py --data_root data`) against
-full 18,439-vertex skinning with the full `(70, 18566)` teacher mapping:
-**2.4e-04 mm** on real GT, on pose perturbed by N(0, 0.25) rad, and under
-resampled identity; gradients agree to 2e-07 relative in `model_params` and
-2e-09 absolute in `shape_params`. Every row of the mapping sums to exactly 1.0,
-so the readout is translation-equivariant and commutes with `to_vision` — which
-is why the mapping is applied after the unit/axis conversion.
-
-**Scoring stays on the fitted readout by default, and it is now a switch.**
-`val3dpw.evaluate(..., exact_landmarks=)`, `--val-landmarks {fitted,exact}` on
-the trainer, and `--landmarks {fitted,exact}` on `benchmark/eval_3dpw_ckpt.py`
-and `benchmark/eval_emdb_ckpt.py` all default to `fitted`, which reproduces
-every recorded number to the decimal.
-
-Measured 2026-09-22, 3DPW test stride 1, J14+adapter PA-MPJPE:
-
-| run | trained with `--exact-landmarks` | fitted | exact |
-|---|---|---|---|
-| g8h_s0 | yes | 40.80 | **40.66** |
-| g8h_s1 | yes | 40.89 | **40.77** |
-| g8r_s0 | yes | 41.27 | **41.13** |
-| g7h_s0 | **no** | 42.90 | **42.77** |
-
-0.13 mm, the same on the run that never saw the flag — the switch is a property
-of the operator, not of the training. EMDB-1 SMPL24+adapter moves 0.09-0.11 mm
-the same way. It is an offset, not a re-ranking, and it is far inside the
-34-49 mm seed spread.
-
-What it removes is nonetheless a genuine mismatch: the MHR70 -> J14 and
-MHR70 -> SMPL24 adapters are fitted by `benchmark/fit_adapter_mhr.py` from the
-stored annotation `joints_3d`, which is the teacher's exact mesh+joint readout,
-so `fitted` hands the adapter keypoints from a different operator than the one
-it was fitted on. The GT side needs nothing: 3DPW's H36M joints and EMDB's SMPL
-joints are SMPL-derived and independent of any MHR readout.
-
-`fitted` remains the default because it is what the exported graph's consumer
-computes, so the published number keeps describing the path that ships, and
-because flipping it would shift every recorded row by 0.13 mm. **Deployment is
-unchanged**: the exported graph never emitted these 3D keypoints — the consumer
-derives them — so no operator is added to the phone runtime. Making the deployed
-readout exact is a separate decision with its own latency measurement.
-
-`instanthmr_distill_train/assets/mhr_landmarks70.npz` (167 KB) is **a new file
-and must be rsynced**, or the job dies at startup. Default off; with it off
-every loss term is bit-identical, verified across `legacy`, `rebalanced`,
-`w_verts` on/off and `cont_head` on/off.
-
-## `--preset v3` — framing robustness from context crops (added 2026-09-25)
-
-v2, plus: every crop is cut from a **context crop** (`build_split.py --context
-2.0 --body-size 448`, its own `--data_root`) through a window drawn in frame
-pixels -- side log-uniform in 0.6-2.0x the tight box (1.2 = today's crop, 0.6 =
-v2's deepest truncation), centre shifted up to +/-15% of the side, v2's
-rotation and flip. It is one resampling step from the stored 448 px crop, so
-zoom-in no longer upsamples a 224 crop. 20% of samples (`geom_p` 0.8, as v2)
-keep the identity window, i.e. today's crop, which also keeps root-rotation
-supervision on the un-augmented subset. Validation uses the identity window.
-
-**Why.** g8h loses 12-15 mm on EMDB under EMDB's own boxes, and 16 mm once our
-own boxes are enlarged to 1.6x -- the loosest framing a 1.2x-cropped corpus can
-show, and then only with black margins. v2 cannot help: it only zooms in.
-
-**The window is an ordinary `aug_M`** relative to `bbox_square`, so labels, the
-reprojection loss and the model are untouched; the CLIFF vector is computed
-from the exact window (centre, side / 1.2) and then flipped and rotated with
-the labels. `apply_v3_preset` forces `cliff_focal`.
-
-**The clamp.** The rotated window is kept inside the stored square only on
-sides where that square ends inside the photo. Where it already runs past the
-photo's edge, further out is outside the photo too, so the black read there is
-the real frame edge -- what a deployment crop shows when its box leaves the
-frame. Clamping every side instead thinned windows looser than 1.6x from 18.5%
-to 10.5% and centred them; this rule keeps 17.0%, with their shift intact.
-
-**Expect black.** In AIC / COCO the person usually nearly fills the photo: 95%
-of 2.0x squares run past the frame, and windows looser than 1.5x are on average
-38% black (frame edge). That is representative of deployment, but it caps how
-much loose-and-real-scene framing this corpus can teach.
-
-Verified by `tools/verify_v3_preset.py` on real AIC rows built both ways:
-baseline / v2 bit-identical; identity window = today's labels and CLIFF
-exactly; reprojection chain vs labels 2e-4 px; CLIFF vs the recovered window
-0.000 px; no read past an inside-photo stored edge; standard or mixed roots
-refused (also through `PairIndex`).
-
-## `--preset v4` — v3's framings, cut like the runtime crop (added 2026-10-21)
-
-**Why.** v3 models learned the texture of their own crop pipeline: a 448 px
-context crop, then a second warp. On the same 997 3DPW-val people, g9_b4_224
-(epoch 314) scores 49.3 mm J14 PA on crops cut straight from the frame (what
-`val3dpw` and the phone do) against 43.6 mm on crops built v3's way; g8h scores
-40.0 / 39.4, i.e. it does not care. Lanczos made it worse, not better.
-
-**What.** `build_split.py --windows 4 --window-sizes 224,288 --no-hands
---output-dir .../data_win{size}/sam3d_gt_<split>` stores, per person, one tile
-of four windows side by side (K*S x S px), each cut from the full-resolution
-photo with one `INTER_LINEAR` resize. Window 0 is the 1.2x box with the
-runtime's own floor / ceil rounding; windows 1-3 are v3 framings (side
-log-uniform in 0.6-2.0x the tight box, centre shift up to +/-15% of the side,
-black only where the window leaves the photo). Boxes are in `win_boxes` and
-are seeded by the person's name, so both sizes and every resume agree.
-
-The dataset (`crop_windows`) draws one window per sample (window 0 when not
-augmenting), uses its box as `bbox_square`, and builds CLIFF from the window
-(its centre, side / 1.2) -- what a detector box framing the person that way
-gives at runtime. Then the ordinary baseline geometry (rotation, flip, mild
-zoom / shift) and v2's occlusion and JPEG, with `cliff_follows_aug` and
-`mask_oob_2d`. **Loss weights are the baseline ones** (no v2 item 3).
-`--image-size` must equal the corpus's window size; anything else is refused.
-
-Verified by `tools/verify_windows.py` on 64 AIC people: default and `--context`
-builds byte-identical to HEAD; every window pixel-equal to
-`InstantHMR._preprocess` for the matching detector box (PNG), CLIFF equal to
-1e-7, labels 6e-5 px; JPEG q95 tiles 37.6-38.6 dB (224) / 39.1-40.2 dB (288)
-against the runtime crop, i.e. JPEG loss only; reprojection chain 1.6e-4 px;
-baseline / v2 samples bit-identical with the flag off. Storage ~83 KB (224) +
-~121 KB (288) per person, ~0.9 TB for the 4.2 M-person corpus.
-
-**`--windows {standard,random}` (added 2026-10-25)** reads the same window corpus
-under any preset but v3, so framing and photometrics can be varied apart.
-`standard` always takes window 0 (the runtime crop) and changes nothing else:
-that is the baseline preset at any `--image-size`, since `data/` exists only at
-224. `random` adds v4's framings plus their two prerequisites (`cliff_follows_aug`,
-`mask_oob_2d`). So `--preset baseline --windows random` is v4 without its
-occlusion and JPEG changes, and `--preset v4` is unchanged (it implies `random`).
-Verified in `tools/verify_windows.py` [8]-[9]: v4 samples are bit-identical to
-before. `standard` draws the same augmentation and CLIFF as baseline on `data/`,
-with labels within 1.1 px (window 0 uses the runtime's box rounding), and its
-reprojection chain closes to 1e-4 px at 288.
-
-## `--image-size` — the network input side (added 2026-09-25)
-
-Default 224, i.e. unchanged. Must be a multiple of 32, because the decoder's
-positional grid is `image_size // 32` (7x7 at 224, 9x9 at 288). One setting
-drives every place that builds the input: the training dataset, `val3dpw`
-(and so the 3DPW / EMDB / SMPL-fit harnesses), `tools/pth_to_onnx.py` and
-`instanthmr.inference`.
-
-**It is recoverable from the weights.** `mem_pos_embed` is a saved buffer with
-one row per patch, so `config_from_checkpoint` reads the grid off it; the exact
-value comes from `run_config.json`, and a `run_config.json` that contradicts
-the weights raises instead of silently resizing. The export stamps
-`image_size` into the ONNX metadata, and `instanthmr.inference` reads it back
-(falling back to the graph's static input shape, then 224). The harnesses
-refuse to score checkpoints of different sizes in one invocation, like
-`--cliff-focal`.
-
-On today's 224 corpus a larger size only upsamples the stored crop; the real
-test of resolution needs the context-crop corpus (`build_split.py --context`).
-
-Verified by `tools/verify_input_size.py`: baseline and v2 samples bit-identical
-at 224; a 288 checkpoint round-trips through the config, the export metadata,
-the inference package and both harnesses.
-
-## Tried and rejected: detaching the 2D head (2026-09-06 / 09-07)
-
-**Do not retry this without reading the failure mode.** The flag is gone from
-the code; this section is the record of why.
-
-The idea was to stop the 2D SimCC head from training the **shared trunk** —
-backbone plus decoder — with one line in `InstantHMRStudent.forward`,
-`feat_2d = feat_2d.detach()`. The head kept its full loss and stayed a deployed
-output; only its gradient into the trunk was cut. The motivation was a real
-measurement: on the converged `b3_s1` checkpoint, of the gradient reaching
-backbone + decoder, `loss_2d_simcc` owned 47.6% and `loss_2d_native` 8.6%
-against 20.7% for all seven 3D body-pose terms together, and the 2D gradient had
-the worst signal-to-noise of any term (`|mean g| / mean |g|` 0.28 against the
-pose group's 0.39) at only +0.21 per-batch cosine with it.
-
-**It failed on both axes.** Two matched seeds each (`dt` = detached, `bno` =
-control), identical mixture, LR and flags, scored at epoch 83-87 of 300 on 3DPW
-test (35,463 person-frames, published-protocol GT, adapter applied) and COCO
-val2017 (6,352 persons, GT boxes):
-
-| arm | 3DPW PA-MPJPE | 3DPW MPJPE | COCO OKS AP | COCO [email protected] |
-|---|---|---|---|---|
-| `bno` (control) | **44.45 ± 0.90** | **71.37 ± 1.04** | **39.4** | **73.5%** |
-| `dt` (detached) | 45.53 ± 0.30 | 79.61 ± 3.86 | **0.00** | 3.1% |
-
-**The 2D head does not degrade — it collapses.** Measured over 48 COCO crops,
-the spread of the 70 predicted keypoints *within* one crop:
-
-| arm | within-crop spread (x, y) | across-crop centroid spread (x, y) |
-|---|---|---|
-| `bno_s0` / `bno_s1` | 0.226 / 0.311, 0.251 / 0.310 | 0.123 / 0.271, 0.129 / 0.287 |
-| `dt_s0` / `dt_s1` | **0.013 / 0.011**, **0.020 / 0.023** | 0.140 / 0.231, 0.155 / 0.232 |
-
-A 15-25x collapse within the crop, on both seeds, while the across-crop spread
-is unchanged — the head still tracks *where the person is*, but predicts all 70
-keypoints at the same point. Hence OKS AP of exactly 0.00: no prediction ever
-clears the lowest threshold.
-
-**Why, and this is the part worth remembering.** `head_2d_feat` and
-`head_2d_logits` are **shared across all 70 queries**, applied per token. For
-the output to differ per keypoint, query *k*'s feature must encode keypoint *k*
-— and the only thing that ever forced that was the 2D loss. Detach it and the
-70 query features are shaped solely by what helps the *global* token through
-self-attention, where per-keypoint identity is worth nothing. They converge on
-each other and the shared head has nothing left to distinguish. Nothing in the
-objective pushes it back, so more epochs do not recover it.
-
-An earlier note in this file claimed the queries "stay alive and become free
-capacity for the token the parameters are read from". Alive, yes. Still
-keypoint-specific, no.
-
-**Two secondary lessons.**
-
-*Early epochs lied.* At epoch 9 `dt` led by 5.6 mm J14 PA. By epoch 84 it
-trailed. `OneCycleLR(pct_start=0.1)` peaks at epoch 30 of 300, and an arm
-spending less on the fast-converging 2D task looks good before the LR anneals.
-Do not read a 300-epoch A/B before the peak.
-
-*The entropy argument behind it was wrong.* `loss_2d_simcc` sits at 2.227 nats
-against an irreducible floor of 2.112 — the entropy of its own Gaussian soft
-label at `sigma = 2 * bin_w`, computed directly. That was read as "only 5.2% of
-the term is reducible, so the task is 95% converged". It does not follow:
-`CE = H(target) + KL(target || prediction)` and `H` is a constant with **zero
-gradient**, so the ratio says nothing about how much of the *learnable* part is
-left. The learnable part is the whole of `KL = 0.115` nats. Use the floor only
-as a scale for reading the `simcc` column — subtract 2.112 to get what the model
-is actually graded on. The gradient measurements above are independent of this
-and still stand.
-
-**If you want the trunk's capacity back, the cost is not the head.** It is
-0.025 of ~10.8 GFLOPs, 0.23%. The 15.7% is the **70 decoder queries** that feed
-it, and removing those is a different architecture that no run has tested. See
-`docs/todo.md`.
-
-## The reported number
-
-One number everywhere: **J14 PA-MPJPE, adapter-applied, against the
-published-protocol 3DPW ground truth.** The training log, `summarize_runs.py`,
-`benchmark/eval_3dpw_ckpt.py` and a paper results table all print the same
-quantity — cross-checked to the decimal (41.71 mm for `b3_s1` on 3DPW test).
-J14 MPJPE rides alongside it. J12 is gone: it was the rig-clean row against the
-old `jointPositions` GT, and against the H36M one it is not (those hips sit
-292 mm apart against the MHR rig's ~120).
-
-The adapter is `benchmark/results/adapter_j14_h36m_teacher.npz`, a fixed 14x70
-map fitted from the teacher's labels and independent of any student, so it does
-not move between checkpoints. `--val-3dpw-adapter none` scores raw MHR joints
-instead; a missing file is a hard error, not a silent fallback.
-
-The epoch log prints two lines because there are **two different datasets**:
-
-```
-held-in (teacher agreement) RAW loss 5.09 body PA 153.5 | EMA ...
-3DPW-validation RAW J14 PA 126.0 MPJPE 283.0 | EMA J14 PA 125.7 MPJPE 284.0 mm
-```
-
-`held-in` is a 2% slice of the **training corpus** scored against the
-**teacher's** labels — agreement, not generalisation, and only a training-health
-signal. `3DPW-validation` is 12 held-out sequences against real **MoCap**.
-Before 2026-09-06 both appeared on one line under a single "val" label, because
-`Mesh_PA_MPJPE` is overwritten with the 3DPW number when `--val-3dpw` is on.
-
-**Runs started before 2026-09-06 are not comparable**: their `J14_PA_MPJPE` in
-`history.jsonl` is the raw MHR joint set, ~26 mm higher than the adapter-applied
-one. `summarize_runs.py` still renders them, in the same column.
-
-## The pair index
-
-`SAM3DStudentDataset` enumerates the corpus with one `glob` plus a `stat()` per
-crop. At 3.7M crops that is ~7.5M metadata operations, and `build_jz_loaders`
-builds two dataset objects per rank — eight concurrent scans on a 4-GPU job, 78
-minutes on Lustre before a single training step.
-
-`build_pair_index()` caches that enumeration to
-`<data_root>/.pair_index_v2.pkl`, rank 0 builds it behind a barrier while the
-others wait, and train/val share one list. Build it ahead of time with
-`--warm-cache` (which also populates `$TORCH_HOME` with the timm backbone, since
-compute nodes have no internet). Cache validity is keyed on sub-folder mtimes;
-`--rebuild-index` forces a rescan.
-
-## Legacy: notebook port for `pfcalcul`
-
-Everything below describes `train_distill.py`, the standalone version of
-`notebooks/distill_transformer_decoder.ipynb`. It bundles notebook cells
-**0, 1, 3, 7, 8, 10, 11 and 15** into a single script: setup, config, dataset,
-student architecture, distillation loss, HMR metrics, the EMA + early-stopping
-training loop, and ONNX export/quantization.
-
-## Folder layout (on the cluster)
-
-Project root: `/pfcalcul/work/kchalabi/envs/lstm/instanthmr_distill_train/`
-
-```
-instanthmr_distill_train/
-├── train_distill.py # the standalone training script
-├── submit_train_distill.sh # SLURM launcher (sbatch)
-├── requirements.txt # pip deps (auto-installed by the platform)
-├── checkpoints/
-│ └── mhr_model.pt # <-- YOU must place this here (≈700 MB, not in git)
-└── runs/ # created at runtime: checkpoints, logs, exports
-```
-
-## Dataset layout
-
-The script takes a **single entry-point folder** (`--data_root`) that contains
-one or more sub-folders, each with its own `annotations/` and `images/`:
-
-```
-instanthmr_data/
-├── sam3d_distill_coco/
-│ ├── annotations/ *.npz
-│ └── images/ *.jpg | *.png
-├── sam3d_gt_coco/
-│ ├── annotations/
-│ └── images/
-└── ...
-```
-
-All sub-folders are loaded and concatenated. File names may collide across
-sub-folders — that is fine, each `(image, npz)` pair is stored as a full path,
-so identically-named files in different sub-folders are kept as distinct
-samples.
-
-## Before launching
-
-1. Copy `mhr_model.pt` into `checkpoints/` (or pass `--mhr_model_path`).
-2. Make sure the dataset is registered as `/datasets/instanthmr_data` so that
- `datasynch_perso` can sync it onto the node.
-
-## Launch
-
-From `/pfcalcul/work/kchalabi/envs/lstm/`:
-
-```sh
-sh instanthmr_distill_train/submit_train_distill.sh
-```
-
-The launcher syncs `/datasets/instanthmr_data`, `cd`s into the project folder,
-and runs `python3 train_distill.py --data_root ../instanthmr_data`. If
-`datasynch_perso` places the data somewhere else, edit the `--data_root` value
-(and the `datasynch_perso` line) in `submit_train_distill.sh`.
-
-## Useful flags
-
-```
-python3 train_distill.py \
- --data_root ../instanthmr_data \
- --output_dir runs/distill_repvit_cliff_v2 \
- --epochs 400 --batch_size 64 --lr 3e-4 --num_workers 8 \
- --self-test # run arch + perfect-student loss sanity checks first
- --no-resume # ignore any existing checkpoint, train from scratch
- --no-export # skip ONNX export/quantization after training
-```
-
-On the cluster, flags go through `EXTRA_TRAIN_ARGS` on
-`datasets_pipeline/jeanzay/52_train_ddp.slurm`, not on a bare command line. The
-generation-6 set is:
-
-```
---preset {baseline|v2} --losses rebalanced \
---cliff-focal --bound-scales --anomaly-safe-fallback --crop-centre-fix \
---w-verts {0|0.35}
-```
-
-Training auto-resumes from `runs/<name>/best_student_model_v3.pth` if present.
-Outputs (best checkpoints + `export/*.onnx`) land under `--output_dir`.
diff --git a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh b/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh
deleted file mode 100755
index fc57210..0000000
--- a/instanthmr_distill_train/submit_train_distill_mhr_only_dolcino.sh
+++ /dev/null
@@ -1,48 +0,0 @@
-#!/bin/sh -e
-
-echo "example: $0 --gpu 1"
-
-# if `requirements.txt` is present, virtual environment is installed and used
-EXEC="python3 train_distill_mhr_only.py"
-
-GPU_REQUIRED=true
-
-. /pfcalcul/tools/sbatchHelpers5.sh
-
-# --data_root points at the dataset synced by datasynch_perso (see below).
-# It is synced next to this project folder, i.e. ../instanthmr_data once we
-# cd into instanthmr_distill_train.
-EXEC="python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8 --batch_size 128 --w_keypoints3d 1.0 $useropt"
-
-# SBATCH --output/--error paths are relative to the submission directory, so
-# create the log folder before submitting.
-mkdir -p instanthmr_distill_mhr_only
-
-jobMessage=$(
-######################### SBATCH launcher
-######################### #SBATCH --option # <= this is an enabled paramater
-######################### ##SBATCH --option # <= this is a disabled parameter
-sbatch ${sbatchopt} << eof
-#!/bin/bash
-#SBATCH --job-name="instanthmr_mhr_only"
-#SBATCH --mail-type=ALL
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=8
-#SBATCH --output=instanthmr_distill_mhr_only/stdout_perelha.txt
-#SBATCH --error=instanthmr_distill_mhr_only/stderr_perelha.txt
-#SBATCH --partition=gpu
-#SBATCH --gres=gpu:1
-#SBATCH --mem-per-gpu=50000
-#SBATCH --time=4-00:00:00
-
-. /pfcalcul/tools/sbatchHelpers5.sh
-
-/pfcalcul/tools/datasync /datasets/instanthmr_data
-
-
-time $EXEC
-
-eof
-)
-
-showSubmitted
diff --git a/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh b/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh
deleted file mode 100755
index 538d592..0000000
--- a/instanthmr_distill_train/submit_train_levjepa_cloud_dolcino.sh
+++ /dev/null
@@ -1,77 +0,0 @@
-#!/bin/sh -e
-
-echo "example: $0 --gpu 1"
-
-# if `requirements.txt` is present, virtual environment is installed and used
-EXEC="python3 train_levjepa_cloud.py"
-
-GPU_REQUIRED=true
-
-. /pfcalcul/tools/sbatchHelpers5.sh
-
-# ---------------------------------------------------------------------------
-# BEFORE THE FIRST SUBMIT, stage the encoder ONCE from a machine with internet:
-#
-# python3 -c "from huggingface_hub import snapshot_download; \
-# print(snapshot_download('galilai-group/LeVJEPA-VideoMix-Large'))"
-# rsync -avPL <printed path>/ <cluster>:<ENCODER_DIR>/
-#
-# -L (--copy-links) is MANDATORY. A HuggingFace snapshot dir is nothing but
-# symlinks into ../../blobs/, so plain `rsync -av` copies six dangling links
-# (~650 bytes) instead of the 1.2 GB of weights, and the job then fails to load
-# the model. The trailing slash on the source flattens the snapshot into
-# ENCODER_DIR rather than nesting it under a commit-hash folder.
-#
-# Compute nodes generally cannot reach huggingface.co, so --encoder_path points
-# at that staged copy and the loader runs with local_files_only=True. Without
-# it the job dies on the first AutoModel.from_pretrained call.
-#
-# requirements.txt must contain `transformers` + `safetensors` (the weights ship
-# their own modeling code, hence trust_remote_code=True). A venv built before
-# that line was added will NOT have it -- delete venv/ so it is rebuilt, or the
-# job fails with "ModuleNotFoundError: No module named 'transformers'".
-# ---------------------------------------------------------------------------
-# NOTE the LEADING SLASH: without it this is a relative path and resolves
-# against the submission directory, so from_pretrained gets a path that does
-# not exist. Point it at the flattened snapshot (config.json + *.py +
-# model.safetensors directly inside), not at a nested commit-hash folder.
-ENCODER_DIR="/pfcalcul/work/kchalabi/envs/lstm/instanthmr_distill_train/levjepa_encoder"
-
-# batch_size is in CLIPS (16 frames each). Stills are encoded as T=1 -- 197
-# tokens instead of 3137 -- so they run at static_batch_mult x that batch.
-# The encoder is FROZEN: no backward through 305 M params, no LoRA state.
-# --max_hours 92 checkpoints and stops cleanly before SLURM's 4-day SIGKILL.
-EXEC="python3 -u train_levjepa_cloud.py --data_root /datasets/instanthmr_data --encoder_path $ENCODER_DIR --num_workers 8 --batch_size 8 --static_batch_mult 8 --epochs 30 --max_hours 92 $useropt"
-
-# SBATCH --output/--error paths are relative to the submission directory, so
-# create the log folder before submitting.
-mkdir -p instanthmr_levjepa_cloud
-
-jobMessage=$(
-######################### SBATCH launcher
-######################### #SBATCH --option # <= this is an enabled paramater
-######################### ##SBATCH --option # <= this is a disabled parameter
-sbatch ${sbatchopt} << eof
-#!/bin/bash
-#SBATCH --job-name="instanthmr_levjepa_cloud"
-#SBATCH --mail-type=ALL
-#SBATCH --ntasks=1
-#SBATCH --cpus-per-task=8
-#SBATCH --output=instanthmr_levjepa_cloud/stdout_perelha.txt
-#SBATCH --error=instanthmr_levjepa_cloud/stderr_perelha.txt
-#SBATCH --partition=gpu
-#SBATCH --gres=gpu:1
-#SBATCH --mem-per-gpu=50000
-#SBATCH --time=4-00:00:00
-
-. /pfcalcul/tools/sbatchHelpers5.sh
-
-/pfcalcul/tools/datasync /datasets/instanthmr_data
-
-
-time $EXEC
-
-eof
-)
-
-showSubmitted
diff --git a/instanthmr_distill_train/train_distill.py b/instanthmr_distill_train/train_distill.py
deleted file mode 100644
index 3275216..0000000
--- a/instanthmr_distill_train/train_distill.py
+++ /dev/null
@@ -1,1466 +0,0 @@
-#!/usr/bin/env python3
-# ============================================================
-# Standalone Distillation Training for the InstantHMR Student
-# ------------------------------------------------------------
-# Derived from notebooks/distill_transformer_decoder.ipynb.
-# Cells included (notebook labels): 0 (setup) | 1 (config) |
-# 3 (dataset) | 7 (student architecture) | 8 (distillation
-# loss) | 10 (HMR metrics) | 11 (training loop) |
-# 15 (export & static quantization).
-#
-# KEY CHANGE vs. the notebook
-# ---------------------------
-# The dataset now takes a SINGLE `data_root` folder as entry
-# point. That folder contains one or more sub-folders, each with
-# its own `annotations/` and `images/` directory (same layout as
-# the repo's `data/` folder). Every sub-folder is loaded and
-# concatenated. Image / npz file names may collide across
-# sub-folders: that is fine because each (image, npz) pair is
-# stored as a full path, and pairing is done per sub-folder.
-#
-# Usage (single GPU):
-# python3 train_distill.py --data_root ../instanthmr_data
-# ============================================================
-import os
-
-# Must be set before the first CUDA allocation (read lazily by the allocator).
-os.environ.setdefault("PYTORCH_CUDA_ALLOC_CONF", "expandable_segments:True")
-
-import argparse
-import math
-import random
-import gc
-from dataclasses import dataclass, field
-from pathlib import Path
-
-import numpy as np
-from PIL import Image
-
-import torch
-import torch.nn as nn
-import torch.nn.functional as F
-import torch.optim as optim
-import torch.optim.lr_scheduler as lr_scheduler
-from torch.utils.data import Dataset, DataLoader, random_split
-from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn
-
-import torchvision.transforms as transforms
-import torchvision.transforms.functional as F_t
-from torchvision.transforms import InterpolationMode
-
-import timm
-from tqdm import tqdm
-
-# --- Notebook Cell 0 setup (headless-safe) ---
-import matplotlib
-matplotlib.use("Agg")
-import matplotlib.pyplot as plt
-import cv2
-import onnxruntime as ort
-
-print(ort.get_available_providers())
-
-device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-print(f"Using device: {device}")
-
-# The project root is wherever this script lives, e.g.
-# /pfcalcul/work/kchalabi/envs/lstm/instanthmr_distill_train/
-PROJECT_ROOT = Path(__file__).resolve().parent
-# datasynch_perso syncs /datasets/instanthmr_data next to the project folder.
-DEFAULT_DATA_ROOT = "/pfcalcul/datasets/instanthmr_data/"
-
-
-# ============================================================
-# Cell 1 — Configuration
-# ============================================================
-@dataclass
-class DistillConfig:
- """All hyperparameters in one place."""
- # --- Paths ---
- # Single entry-point folder containing one or more sub-folders,
- # each with its own annotations/ and images/ directories.
- data_root: str = str(DEFAULT_DATA_ROOT)
- mhr_model_path: str = str(PROJECT_ROOT / "checkpoints/mhr_model.pt") # path to MHR model
- log_dir: str = str(PROJECT_ROOT / "runs/distill_repvit_cliff_v3") # v3: full balanced dataset (Harmony4D capped to 150k)
-
- # --- Data ---
- image_size: int = 224
- max_images: int | None = None # None = use every crop; a number truncates (path-biased)
- # Per-dataset crop caps: sub-folder name -> max crops kept. The subset is
- # RANDOMLY sampled (seed 42), not the first N, so it stays representative.
- # Harmony4D is ~53% of the corpus and a narrow domain -> cap it for balance.
- per_dataset_caps: dict = field(default_factory=lambda: {"sam3d_gt_harmony4d": 300_000})
- val_split: float = 0.1
- num_workers: int = 4
- augment: bool = True
-
- # --- Model (Transformer + CLIFF) ---
- backbone: str = "repvit_m2_3"
- backbone_feat_dim: int = 640 # RepViT m2_3 usually outputs 640
- d_model: int = 512
- n_heads: int = 8
- n_decoder_layers: int = 4
- dropout: float = 0.1
-
- # --- Output dimensions ---
- pose_dim: int = 136
- scale_dim: int = 68
- shape_dim: int = 45
- cam_dim: int = 3
- cliff_dim: int = 3
- num_joints: int = 70 # We track 70 joints
-
- @property
- def model_params_dim(self) -> int:
- return self.pose_dim + self.scale_dim # 204
-
- # --- Training ---
- batch_size: int = 64
- lr: float = 3e-4
- weight_decay: float = 1e-4
- epochs: int = 400
- warmup_epochs: int = 3
- grad_clip: float = 1.0 # tighter clip: 5.0 let a bad step blow up the MHR loss
- anomaly_loss_threshold: float = 100.0 # skip a batch whose loss exceeds this (normal is <2)
- use_amp: bool = True
- ema_decay: float = 0.9998 # EMA shadow-weights decay (Cell 11)
- early_stop_patience: int = 150 # stop after N epochs where NEITHER raw nor ema improved (Cell 11)
- resume: bool = True # resume from best_student_model_v3.pth if present
-
- # --- Loss weights (UPDATED for HMR Balance) ---
- w_pose: float = 1.0
- w_scale: float = 0.1
- w_shape: float = 1.0
- w_cam: float = 0.1
- w_keypoints3d: float = 10.0 # Bumped up: 3D joints are the strongest spatial anchor
- w_keypoints2d: float = 10.0 # Bumped up: Forces pixel-perfect alignment
- w_3d_joints: float = 1e-3
- w_structural: float = 0.01
- w_reproj_3d: float = 0.01 # High weight to enforce strict scale/camera alignment
- w_reproj_mhr: float = 0.01 # High weight to enforce strict scale/camera alignment
-
- # ==========================================================
- # Feet Barycenter Anchors
- # ==========================================================
- w_feet = 0.01 # Soft prior weight, matching w_structural
-
- # Native 70-joint format indices
- native_left_foot = [15, 16, 17]
- native_right_foot = [18, 19, 20]
-
- # MHR 127-joint format indices
- mhr_left_foot = [6, 7, 8]
- mhr_right_foot = [22, 23, 24]
-
- # Data-Driven Topology Mapping (Native 70 <-> MHR 127)
- native_mapping_ids = [ 1, 2, 5, 6, 9, 10, 11, 12, 13, 14, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62 ]
- mhr_mapping_ids = [ 125, 123, 75, 39, 2, 18, 3, 19, 5, 21, 64, 63, 62, 61, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 41, 100, 99, 98, 97, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 77 ]
-
-
-# ============================================================
-# Cell 3 — Dataset and Data Augmentation
-# ============================================================
-class RandomPixelate:
- """Simulates low-quality webcams or low-bandwidth video."""
- def __init__(self, p=0.2, min_res=32, max_res=112):
- self.p = p
- self.min_res = min_res
- self.max_res = max_res
-
- def __call__(self, img):
- if random.random() < self.p:
- orig_w, orig_h = img.size
- degraded_h = random.randint(self.min_res, self.max_res)
- degraded_w = int(orig_w * (degraded_h / orig_h))
- # Downsample then upsample to create pixelation effect
- img = F_t.resize(img, [degraded_h, degraded_w], interpolation=InterpolationMode.NEAREST)
- img = F_t.resize(img, [orig_h, orig_w], interpolation=InterpolationMode.NEAREST)
- return img
-
-
-class SAM3DStudentDataset(Dataset):
- def __init__(self, data_root: str,
- image_size: int = 224, max_images: int | None = None,
- augment: bool = True, per_dataset_caps: dict | None = None):
- super().__init__()
- self.image_size = image_size
- self.data_root = Path(data_root)
-
- # --- 1. Build Transform Pipeline ---
- # The images on disk are already 224x224, but Resize guarantees safety.
- tfm_list = [transforms.Resize((image_size, image_size))]
-
- if augment:
- tfm_list.extend([
- # ColorJitter (esp. hue -> RGB/HSV round-trip) is the priciest CPU op in
- # the pipeline. Applying it on EVERY image was starving the GPU, so run it
- # half the time via RandomApply — still solid colour regularisation.
- transforms.RandomApply([transforms.ColorJitter(
- brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1)], p=0.5),
- RandomPixelate(p=0.2, min_res=48, max_res=112),
- transforms.RandomApply([transforms.GaussianBlur(kernel_size=5)], p=0.2),
- ])
-
- tfm_list.extend([
- transforms.ToTensor(),
- # Standard ImageNet Normalization
- transforms.Normalize(mean=[0.485, 0.456, 0.406],
- std=[0.229, 0.224, 0.225]),
- ])
-
- if augment:
- tfm_list.append(transforms.RandomErasing(p=0.2, scale=(0.02, 0.15)))
-
- self.transform = transforms.Compose(tfm_list)
-
- # --- 2. Pair Discovery Across Sub-folders ---
- # `data_root` contains one or more sub-folders, each with its own
- # annotations/ and images/ directories. We pair the npz with its image
- # WITHIN each sub-folder, so identical file names in different
- # sub-folders never collide (pairs are stored as full paths).
- self.pairs = []
- dataset_dirs = self._find_dataset_dirs(self.data_root)
- if not dataset_dirs:
- raise FileNotFoundError(
- f"No annotations/images sub-folders found under '{self.data_root}'. "
- f"Expected <data_root>/<subset>/annotations/*.npz and "
- f"<data_root>/<subset>/images/*.{{jpg,png}}."
- )
-
- # Seeded RNG so the per-dataset random subset is IDENTICAL across the
- # augmented and clean dataset instances -> the train/val split stays
- # consistent between them.
- cap_rng = random.Random(42)
- caps = per_dataset_caps or {}
-
- for ann_dir, img_dir in dataset_dirs:
- sub_name = ann_dir.parent.name
- sub_pairs = []
- for npz_path in sorted(ann_dir.glob("*.npz")):
- stem = npz_path.stem
- # Check for jpg or png
- img_path = img_dir / f"{stem}.jpg"
- if not img_path.exists():
- img_path = img_dir / f"{stem}.png"
-
- if img_path.exists():
- sub_pairs.append((img_path, npz_path))
-
- # Per-dataset cap: randomly sample a REPRESENTATIVE subset (not the
- # first N, which would bias toward alphabetically-early sequences).
- cap = caps.get(sub_name)
- if cap is not None and len(sub_pairs) > cap:
- n_before = len(sub_pairs)
- sub_pairs.sort(key=lambda p: (str(p[1]), str(p[0]))) # deterministic pool
- sub_pairs = cap_rng.sample(sub_pairs, cap)
- print(f" [cap] {sub_name}: sampled {cap:,} of {n_before:,} crops")
- self.pairs.extend(sub_pairs)
-
- # Deterministic global order so the train/val split is reproducible.
- self.pairs.sort(key=lambda p: (str(p[1]), str(p[0])))
-
- if max_images is not None:
- self.pairs = self.pairs[:max_images]
-
- print(f" Found {len(dataset_dirs)} sub-folder(s) under {self.data_root}; "
- f"{len(self.pairs)} valid (image, npz) pairs (augment={augment}).")
-
- @staticmethod
- def _find_dataset_dirs(root: Path):
- """Return [(annotations_dir, images_dir), ...] for every dataset folder.
-
- Handles both layouts:
- * root itself directly contains annotations/ and images/
- * root contains sub-folders, each with annotations/ and images/
- """
- dirs = []
- if (root / "annotations").is_dir() and (root / "images").is_dir():
- dirs.append((root / "annotations", root / "images"))
-
- if root.is_dir():
- for sub in sorted(root.iterdir()):
- if not sub.is_dir():
- continue
- ann, img = sub / "annotations", sub / "images"
- if ann.is_dir() and img.is_dir():
- dirs.append((ann, img))
- return dirs
-
- def __len__(self):
- return len(self.pairs)
-
- def __getitem__(self, idx):
- img_path, npz_path = self.pairs[idx]
-
- # --- 3. Load and Augment Image ---
- image = Image.open(img_path).convert("RGB")
- image = self.transform(image) # Returns float tensor [3, 224, 224]
-
- # --- 4. Load Annotations ---
- ann = np.load(npz_path)
-
- orig_h, orig_w = ann["orig_shape"]
-
- # --- 5. Correct Mathematical Rescaling for 2D Joints ---
- sq_bbox = ann["bbox_square"]
- sq_x1, sq_y1 = sq_bbox[0], sq_bbox[1]
- orig_crop_size = sq_bbox[2] - sq_bbox[0]
-
- # Scale factor from the original crop square to the 224x224 tensor
- true_scale = self.image_size / max(orig_crop_size, 1.0)
-
- # Shift full-frame 2D joints relative to crop top-left, then scale
- joints_2d = ann["joints_2d"].copy()
- joints_2d[:, 0] = (joints_2d[:, 0] - sq_x1) * true_scale
- joints_2d[:, 1] = (joints_2d[:, 1] - sq_y1) * true_scale
-
- # NEW: Normalize to [-1, 1] for neural network stability!
- joints_2d[:, 0] = (joints_2d[:, 0] / self.image_size) * 2.0 - 1.0
- joints_2d[:, 1] = (joints_2d[:, 1] / self.image_size) * 2.0 - 1.0
-
- joints_2d = torch.from_numpy(joints_2d).float()
-
- # --- 6. CLIFF Conditioning Vectors ---
- # Using the tighter YOLO bbox to tell the network where the person actually is
- tight_bbox = ann["bbox"]
- cx = (tight_bbox[0] + tight_bbox[2]) / 2.0
- cy = (tight_bbox[1] + tight_bbox[3]) / 2.0
-
- # Normalize to [-1, 1] space for better network stability
- cx_norm = 2.0 * (cx / orig_w) - 1.0
- cy_norm = 2.0 * (cy / orig_h) - 1.0
-
- # Scale relative to the longest edge of the original image
- b_size = max(tight_bbox[2] - tight_bbox[0], tight_bbox[3] - tight_bbox[1])
- b_scale = b_size / max(orig_w, orig_h)
- cliff_cond = torch.tensor([cx_norm, cy_norm, b_scale], dtype=torch.float32)
-
- return {
- "image": image, # [3, 224, 224]
- "cliff_cond": cliff_cond, # [3]
- "mhr_model_params": torch.from_numpy(ann["mhr_model_params"]).float(), # [204]
- "shape_params": torch.from_numpy(ann["shape_params"]).float(), # [45]
- "cam_trans": torch.from_numpy(ann["cam_trans"]).float(), # [3]
- "cam_focal": torch.from_numpy(ann["cam_focal_length"]).float(), # [2]
- "joints_2d": joints_2d, # [70, 2] Crop Space
- "joints_3d": torch.from_numpy(ann["joints_3d"]).float(), # [70, 3] Root-centered
- "orig_shape": torch.tensor([orig_h, orig_w], dtype=torch.float32), # [2]
- "bbox_square": sq_bbox
- }
-
-
-def build_dataloaders(cfg):
- """Build train/val datasets and loaders (Cell 3 bottom)."""
- full_dataset = SAM3DStudentDataset(
- cfg.data_root,
- augment=cfg.augment,
- max_images=cfg.max_images,
- per_dataset_caps=cfg.per_dataset_caps,
- )
-
- val_dataset_clean = SAM3DStudentDataset(
- cfg.data_root,
- augment=False,
- max_images=cfg.max_images,
- per_dataset_caps=cfg.per_dataset_caps,
- )
-
- n_val = int(len(full_dataset) * cfg.val_split)
- n_train = len(full_dataset) - n_val
-
- # Use a generator for reproducible splits
- generator = torch.Generator().manual_seed(42)
- train_dataset, _ = random_split(full_dataset, [n_train, n_val], generator=generator)
-
- generator = torch.Generator().manual_seed(42)
- _, val_dataset = random_split(val_dataset_clean, [n_train, n_val], generator=generator)
-
- # Keep workers alive across epochs and prefetch more batches so the GPU
- # stays fed (helps the ~40% idle time seen in nvidia-smi).
- loader_kwargs = dict(num_workers=cfg.num_workers, pin_memory=True)
- if cfg.num_workers > 0:
- loader_kwargs["persistent_workers"] = True
- loader_kwargs["prefetch_factor"] = 4
-
- train_loader = DataLoader(
- train_dataset,
- batch_size=cfg.batch_size,
- shuffle=True,
- drop_last=True,
- **loader_kwargs,
- )
-
- val_loader = DataLoader(
- val_dataset,
- batch_size=cfg.batch_size,
- shuffle=False,
- **loader_kwargs,
- )
-
- print(f"Dataset Loaded! Train batches: {len(train_loader)} | Val batches: {len(val_loader)}")
- return train_loader, val_loader, full_dataset
-
-
-# ============================================================
-# Cell 7 — Student Architecture (Mini-SAM3D with PE)
-# ============================================================
-def get_2d_sincos_pos_embed(embed_dim, grid_size):
- """Generate 2D sine-cosine positional embedding for preserving spatial topology."""
- grid_h, grid_w = grid_size, grid_size
- grid_y, grid_x = torch.meshgrid(torch.arange(grid_h), torch.arange(grid_w), indexing='ij')
-
- omega = torch.arange(embed_dim // 4).float() / (embed_dim // 4)
- omega = 1.0 / (10000 ** omega)
-
- out_y = grid_y.flatten().unsqueeze(1) * omega.unsqueeze(0)
- out_x = grid_x.flatten().unsqueeze(1) * omega.unsqueeze(0)
-
- pe_y = torch.cat([torch.sin(out_y), torch.cos(out_y)], dim=1)
- pe_x = torch.cat([torch.sin(out_x), torch.cos(out_x)], dim=1)
-
- pos_embed = torch.cat([pe_y, pe_x], dim=1) # (H*W, embed_dim)
- return pos_embed.unsqueeze(0) # (1, H*W, embed_dim)
-
-
-class InstantHMRStudent(nn.Module):
- def __init__(self, cfg, pretrained=True):
- super().__init__()
- self.cfg = cfg
-
- # 1. The Vision Backbone (RepViT)
- self.backbone = timm.create_model(cfg.backbone, pretrained=pretrained, num_classes=0)
- embed_dim = self.backbone.num_features
- self.feat_proj = nn.Linear(embed_dim, cfg.d_model)
-
- # --- NEW: Positional Embedding for Transformer Memory ---
- # RepViT naturally downsamples by 32x. For a 224x224 image, the grid is 7x7.
- self.grid_size = cfg.image_size // 32
- pos_embed = get_2d_sincos_pos_embed(cfg.d_model, self.grid_size)
- # register_buffer ensures it moves to GPU automatically but isn't updated by the optimizer
- self.register_buffer("mem_pos_embed", pos_embed)
-
- # 2. Learnable Query Tokens (1 Global + 70 2D + 70 3D = 141 Queries)
- self.num_global = 1
- self.num_2d = cfg.num_joints
- self.num_3d = cfg.num_joints
- self.total_queries = self.num_global + self.num_2d + self.num_3d
-
- # self.query_embed = nn.Parameter(torch.randn(1, self.total_queries, cfg.d_model))
- self.query_embed = nn.Parameter(torch.randn(1, self.total_queries, cfg.d_model) * 0.02)
-
- # CLIFF Conditioning Injector
- self.cond_proj = nn.Sequential(
- nn.Linear(cfg.cliff_dim, 128),
- nn.GELU(),
- nn.Linear(128, cfg.d_model)
- )
-
- # 3. Transformer Cross-Attention Decoder
- decoder_layer = nn.TransformerDecoderLayer(
- d_model=cfg.d_model,
- nhead=cfg.n_heads,
- dim_feedforward=cfg.d_model * 4,
- dropout=cfg.dropout,
- activation="gelu",
- batch_first=True,
- norm_first=True
- )
- self.transformer = nn.TransformerDecoder(decoder_layer, num_layers=cfg.n_decoder_layers)
-
- # 4. The Output Heads
-
- # Head A: Global Head (Token 0 -> MHR Params + Shape Params + Camera)
- self.global_out_dim = cfg.model_params_dim + cfg.shape_dim + cfg.cam_dim
- self.head_global = nn.Linear(cfg.d_model, self.global_out_dim)
-
- # --- Safely Split 2D Head ---
- self.head_2d_feat = nn.Sequential(
- nn.Linear(cfg.d_model, 256),
- nn.GELU()
- )
- self.head_2d_xy = nn.Sequential(
- nn.Linear(256, 2)
- # nn.Tanh() # Bounds predictions to [-1, 1]!
- )
- if getattr(cfg, 'learn_uncertainty', False):
- # Uncertainty must remain unbounded!
- self.head_2d_unc = nn.Linear(256, 1)
-
- # Head C: 3D Keypoint Head (No Tanh here, 3D meters can exceed [-1, 1])
- self.dim_3d = 4 if getattr(cfg, 'learn_uncertainty', False) else 3
- self.head_3d = nn.Sequential(
- nn.Linear(cfg.d_model, 256),
- nn.GELU(),
- nn.Linear(256, self.dim_3d)
- )
-
- # Initialize camera Z translation
- nn.init.constant_(self.head_global.bias[-1], 2.0)
- nn.init.constant_(self.head_global.bias[-2], 0.0)
- nn.init.constant_(self.head_global.bias[-3], 0.0)
-
- # ==========================================================
- # HMR-Standard Zero-Initialization for Stability
- # ==========================================================
- # 1. Global Head (MHR Params) - Must start near neutral pose!
- nn.init.normal_(self.head_global.weight, mean=0.0, std=1e-4)
- # Zero out the biases (except the last 3 camera params we just set)
- nn.init.constant_(self.head_global.bias[:-3], 0.0)
-
- # 2. 2D Spatial Head - Start with tiny weights
- nn.init.normal_(self.head_2d_xy[0].weight, mean=0.0, std=1e-4)
- nn.init.constant_(self.head_2d_xy[0].bias, 0.0)
-
- # 3. 3D Spatial Head - Start with tiny weights
- nn.init.normal_(self.head_3d[-1].weight, mean=0.0, std=1e-4)
- nn.init.constant_(self.head_3d[-1].bias, 0.0)
-
- def forward(self, images, cliff_cond):
- B = images.shape[0]
-
- # 1. Extract spatial image features
- img_feats = self.backbone.forward_features(images)
-
- # Defensive flattening
- if img_feats.dim() == 4:
- img_feats = img_feats.flatten(2).transpose(1, 2) # [B, H*W, C]
-
- memory = self.feat_proj(img_feats)
-
- # --- NEW: Inject Spatial Topology into Memory ---
- # Broadcasting [1, H*W, d_model] across Batch size
- memory = memory + self.mem_pos_embed
-
- # 2. Prepare Query Tokens [B, 141, d_model]
- queries = self.query_embed.expand(B, -1, -1)
-
- cond = self.cond_proj(cliff_cond).unsqueeze(1)
- queries = queries + cond
-
- # 3. Cross Attention
- shared_feats = self.transformer(tgt=queries, memory=memory)
-
- # 4. Route tokens to their dedicated heads
- feat_global = shared_feats[:, 0, :]
- feat_2d = shared_feats[:, 1 : 1 + self.num_2d, :]
- feat_3d = shared_feats[:, 1 + self.num_2d :, :]
-
- # --- Decode Global Token ---
- global_preds = self.head_global(feat_global)
-
- idx_mhr = self.cfg.model_params_dim
- idx_shape = idx_mhr + self.cfg.shape_dim
-
- pred_mhr_params = global_preds[:, :idx_mhr]
- pred_shape_params = global_preds[:, idx_mhr : idx_shape]
- pred_cam_trans = global_preds[:, idx_shape :]
-
- # --- Decode 2D Spatial Tokens (Safely!) ---
- feat_2d_processed = self.head_2d_feat(feat_2d)
- pred_xy = self.head_2d_xy(feat_2d_processed) # [B, 70, 2] bounded by Tanh
-
- if getattr(self.cfg, 'learn_uncertainty', False):
- pred_unc = self.head_2d_unc(feat_2d_processed) # [B, 70, 1] unbounded
- pred_joints_2d = torch.cat([pred_xy, pred_unc], dim=-1) # [B, 70, 3]
- else:
- pred_joints_2d = pred_xy
-
- # --- Decode 3D Spatial Tokens ---
- pred_joints_3d = self.head_3d(feat_3d)
-
- return {
- "mhr_params": pred_mhr_params,
- "shape_params": pred_shape_params,
- "cam_trans": pred_cam_trans,
- "joints_2d": pred_joints_2d,
- "joints_3d": pred_joints_3d
- }
-
-
-# ============================================================
-# Cell 8 — The Distillation Loss Module (Foolproof Edition)
-# ============================================================
-class MHRForwardPass:
- """Modified, differentiable forward pass over the MHR rig."""
- def __init__(self, mhr_path, device):
- self.device = device
- self.mhr = torch.jit.load(mhr_path, map_location=device).eval()
- for p in self.mhr.parameters():
- p.requires_grad_(False)
-
- def get_joints(self, model_params, shape_params):
- B = model_params.shape[0]
- dummy_identity = torch.zeros(B, 45, device=model_params.device, dtype=model_params.dtype)
- concat_params = torch.cat([model_params, dummy_identity], dim=1)
-
- joint_params = self.mhr.character_torch.model_parameters_to_joint_parameters(concat_params.to(self.device))
- skel_state = self.mhr.character_torch.joint_parameters_to_skeleton_state(joint_params)
-
- return skel_state.to(model_params.device)
-
-
-class DistillationLoss(nn.Module):
- def __init__(self, cfg, mhr_module):
- super().__init__()
- self.cfg = cfg
- self.mhr_module = mhr_module
- self.mse = nn.MSELoss()
- self.l1 = nn.SmoothL1Loss()
- self.strict_l1 = nn.L1Loss()
-
- def _project_3d_to_norm_2d(self, j3d, pred_cam_trans, targets):
- """Helper to project 3D points (in meters) to normalized [-1, 1] 2D crop space."""
- # A. Move to Camera Space
- j3d_cam = j3d + pred_cam_trans.unsqueeze(1)
-
- # Prevent 1/Z^2 Gradient Explosions
- # 0.01 allows a gradient multiplier of 10,000.
- # 0.4 meters (40cm) limits the multiplier to a safe limit, protecting the kinematic tree.
- z_cam = torch.clamp(j3d_cam[..., 2], min=0.4)
-
- # B. Extract Intrinsics
- fx = targets["cam_focal"][:, 0:1] # Shape: [B, 1]
- fy = targets["cam_focal"][:, 1:2] # Shape: [B, 1]
- cx = targets["orig_shape"][:, 1:2] / 2.0 # Shape: [B, 1]
- cy = targets["orig_shape"][:, 0:1] / 2.0 # Shape: [B, 1]
-
- # C. Full-Frame Projection
- u_full = (fx * j3d_cam[..., 0] / z_cam) + cx
- v_full = (fy * j3d_cam[..., 1] / z_cam) + cy
-
- # D. Map to Crop Space
- bbox = targets["bbox_square"]
- sq_x1 = bbox[:, 0:1] # [B, 1]
- sq_y1 = bbox[:, 1:2] # [B, 1]
- crop_w = torch.clamp(bbox[:, 2:3] - bbox[:, 0:1], min=1.0) # [B, 1]
-
- true_scale = self.cfg.image_size / crop_w
- u_crop = (u_full - sq_x1) * true_scale
- v_crop = (v_full - sq_y1) * true_scale
-
- # E. Normalize to [-1, 1]
- u_norm = (u_crop / self.cfg.image_size) * 2.0 - 1.0
- v_norm = (v_crop / self.cfg.image_size) * 2.0 - 1.0
-
- return torch.stack([u_norm, v_norm], dim=-1) # Shape: [B, J, 2]
-
- def forward(self, preds, targets):
- losses = {}
- idx_pose = self.cfg.pose_dim
-
- pred_pose = preds["mhr_params"][:, :idx_pose]
- pred_scale = preds["mhr_params"][:, idx_pose:]
- tgt_pose = targets["mhr_model_params"][:, :idx_pose]
- tgt_scale = targets["mhr_model_params"][:, idx_pose:]
-
- # --- 1. Base Parameters ---
- losses['loss_pose'] = self.l1(pred_pose, tgt_pose) * self.cfg.w_pose
- losses['loss_scale'] = self.mse(pred_scale, tgt_scale) * self.cfg.w_scale
- losses['loss_shape'] = self.mse(preds["shape_params"], targets["shape_params"]) * self.cfg.w_shape
- losses['loss_cam'] = self.mse(preds["cam_trans"], targets["cam_trans"]) * self.cfg.w_cam
-
- # --- 2. Differentiable 3D Joint Alignment (Centimeter space corrected!) ---
- pred_mhr_joints = self.mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]
- with torch.no_grad():
- tgt_mhr_joints = self.mhr_module.get_joints(targets["mhr_model_params"], targets["shape_params"])[..., :3]
-
- losses['loss_mhr_joints'] = self.l1(pred_mhr_joints, tgt_mhr_joints.detach()) * self.cfg.w_3d_joints
-
- # --- Base MHR Vision Coordinates (Needed for Structural & Reprojection) ---
- pred_mhr_vision = pred_mhr_joints.clone() / 100.0
- pred_mhr_vision[..., 1] = -pred_mhr_vision[..., 1]
- pred_mhr_vision[..., 2] = -pred_mhr_vision[..., 2]
-
- # ==============================================================================
- # 2.5 Structural Consistency Loss (MHR mesh vs Native Keypoints)
- # ==============================================================================
- if hasattr(self.cfg, 'native_mapping_ids') and hasattr(self.cfg, 'mhr_mapping_ids'):
- pred_native_anchors = preds["joints_3d"][:, self.cfg.native_mapping_ids, :3]
- pred_mhr_anchors = pred_mhr_vision[:, self.cfg.mhr_mapping_ids, :3]
- losses['loss_structural'] = self.l1(pred_native_anchors, pred_mhr_anchors) * self.cfg.w_structural
-
- # ==============================================================================
- # Feet Barycenter Alignment Loss
- # ==============================================================================
- if hasattr(self.cfg, 'native_left_foot') and hasattr(self.cfg, 'w_feet'):
- nat_lf_pts = preds["joints_3d"][:, self.cfg.native_left_foot, :3]
- nat_rf_pts = preds["joints_3d"][:, self.cfg.native_right_foot, :3]
-
- mhr_lf_pts = pred_mhr_vision[:, self.cfg.mhr_left_foot, :3]
- mhr_rf_pts = pred_mhr_vision[:, self.cfg.mhr_right_foot, :3]
-
- nat_lf_bary = nat_lf_pts.mean(dim=1)
- nat_rf_bary = nat_rf_pts.mean(dim=1)
- mhr_lf_bary = mhr_lf_pts.mean(dim=1)
- mhr_rf_bary = mhr_rf_pts.mean(dim=1)
-
- losses['loss_feet'] = (self.strict_l1(nat_lf_bary, mhr_lf_bary) + self.strict_l1(nat_rf_bary, mhr_rf_bary)) * self.cfg.w_feet
-
- # --- 3. DYNAMIC Native Keypoints ---
- pred_2d = preds["joints_2d"]
- tgt_2d = targets["joints_2d"]
- losses['loss_2d_native'] = self.l1(pred_2d[..., :2], tgt_2d) * self.cfg.w_keypoints2d
-
- pred_3d = preds["joints_3d"]
- tgt_3d = targets["joints_3d"]
- losses['loss_3d_native'] = self.l1(pred_3d[..., :3], tgt_3d) * self.cfg.w_keypoints3d
-
- # ==============================================================================
- # 4. 2D Reprojection Loss (Camera-Pose Consistency)
- # ==============================================================================
- # A. Native 3D Head Reprojection (All 70 Joints)
- pred_reproj_native = self._project_3d_to_norm_2d(preds["joints_3d"][..., :3], preds["cam_trans"], targets)
- losses['loss_reproj_native'] = self.strict_l1(pred_reproj_native, tgt_2d) * self.cfg.w_reproj_3d
-
- # B. MHR Mesh Anchors Reprojection (52 Anchors)
- if hasattr(self.cfg, 'native_mapping_ids') and hasattr(self.cfg, 'mhr_mapping_ids'):
- # Only project the 52 matched anchors from the MHR mesh
- pred_mhr_anchors_vision = pred_mhr_vision[:, self.cfg.mhr_mapping_ids, :3]
- pred_reproj_mhr = self._project_3d_to_norm_2d(pred_mhr_anchors_vision, preds["cam_trans"], targets)
-
- # Compare against the corresponding 52 ground truth 2D targets
- tgt_2d_anchors = tgt_2d[:, self.cfg.native_mapping_ids, :]
- losses['loss_reproj_mhr'] = self.strict_l1(pred_reproj_mhr, tgt_2d_anchors) * self.cfg.w_reproj_mhr
-
- losses['total_loss'] = sum(losses.values())
- return losses
-
-
-# ============================================================
-# Cell 10 — HMR Evaluation Metrics (MPJPE & PA-MPJPE)
-# ============================================================
-def batched_procrustes_alignment(pred_pts, gt_pts):
- """
- Computes the optimal Similarity Transform (Scale, Rotation, Translation)
- to align pred_pts to gt_pts using Batched SVD.
- """
- B, N, D = pred_pts.shape
-
- # 1. Mean center the point clouds
- mu_pred = pred_pts.mean(dim=1, keepdim=True)
- mu_gt = gt_pts.mean(dim=1, keepdim=True)
-
- pred_centered = pred_pts - mu_pred
- gt_centered = gt_pts - mu_gt
-
- # 2. Calculate scale (Frobenius norm)
- norm_pred = torch.linalg.norm(pred_centered, dim=(1, 2), keepdim=True)
- norm_gt = torch.linalg.norm(gt_centered, dim=(1, 2), keepdim=True)
-
- pred_normalized = pred_centered / torch.clamp(norm_pred, min=1e-8)
- gt_normalized = gt_centered / torch.clamp(norm_gt, min=1e-8)
-
- # 3. Compute covariance matrix H (X^T Y)
- H = torch.bmm(pred_normalized.transpose(1, 2), gt_normalized)
-
- # 4. Singular Value Decomposition (SVD)
- # PyTorch returns U, S, and Vh (where Vh is V^T)
- U, S, Vh = torch.linalg.svd(H)
-
- # 5. Compute Rotation matrix R (U * V^T) - FIXED!
- R = torch.bmm(U, Vh)
-
- # 6. Fix reflection (Strict SO(3) rotation)
- det = torch.linalg.det(R)
-
- # If det < 0, we must flip the sign of the 3rd column of U
- # Shape of det_sign is [B, 1]
- det_sign = torch.where(det < 0, torch.tensor(-1.0, device=pred_pts.device), torch.tensor(1.0, device=pred_pts.device)).unsqueeze(-1)
-
- U_fixed = U.clone()
- U_fixed[:, :, 2] *= det_sign
-
- # Recompute R with the reflection fix
- R_fixed = torch.bmm(U_fixed, Vh)
-
- # 7. Optimal Scale
- # We must apply the same sign fix to the 3rd singular value!
- S_fixed = S.clone()
- S_fixed[:, 2] *= det_sign.squeeze(-1)
-
- scale = S_fixed.sum(dim=-1, keepdim=True).unsqueeze(-1) * (norm_gt / torch.clamp(norm_pred, min=1e-8))
-
- # 8. Apply the transformation
- pred_aligned = scale * torch.bmm(pred_centered, R_fixed) + mu_gt
-
- return pred_aligned
-
-
-def evaluate_hmr_batch(preds, targets, cfg, mhr_module):
- """
- Calculates MPJPE and PA-MPJPE for both the 3D Keypoints Head and the MHR Mesh.
- Returns errors in MILLIMETERS (mm).
- """
- metrics = {}
-
- # ---------------------------------------------------------
- # 1. Evaluate the Native 3D Spatial Head (70 Joints)
- # ---------------------------------------------------------
- pred_3d_native = preds["joints_3d"][..., :3]
- gt_3d_native = targets["joints_3d"][..., :3]
-
- # MPJPE requires root-relative alignment (usually joint 0 is the root/pelvis)
- pred_3d_root_rel = pred_3d_native - pred_3d_native[:, 0:1, :]
- gt_3d_root_rel = gt_3d_native - gt_3d_native[:, 0:1, :]
-
- # MPJPE in mm (* 1000)
- err_native = torch.linalg.norm(pred_3d_root_rel - gt_3d_root_rel, dim=-1)
- metrics['Head_3D_MPJPE'] = err_native.mean().item() * 1000.0
-
- # PA-MPJPE in mm
- pred_3d_aligned = batched_procrustes_alignment(pred_3d_native, gt_3d_native)
- err_native_pa = torch.linalg.norm(pred_3d_aligned - gt_3d_native, dim=-1)
- metrics['Head_3D_PA_MPJPE'] = err_native_pa.mean().item() * 1000.0
-
- # ---------------------------------------------------------
- # 2. Evaluate the MHR Mesh Head (Using the 52 Anchors)
- # ---------------------------------------------------------
- if hasattr(cfg, 'native_mapping_ids') and hasattr(cfg, 'mhr_mapping_ids'):
- # Generate 3D mesh joints in cm
- pred_mhr_raw = mhr_module.get_joints(preds["mhr_params"], preds["shape_params"])[..., :3]
-
- # Convert to Meters and align axes
- pred_mhr_meters = pred_mhr_raw / 100.0
- pred_mhr_meters[..., 1] = -pred_mhr_meters[..., 1]
- pred_mhr_meters[..., 2] = -pred_mhr_meters[..., 2]
-
- # Extract the 52 matching anchors
- pred_mesh_anchors = pred_mhr_meters[:, cfg.mhr_mapping_ids, :]
- gt_mesh_anchors = gt_3d_native[:, cfg.native_mapping_ids, :]
-
- # Mean-Center for MPJPE (To bypass differing origin definitions between rigs)
- pred_mesh_centered = pred_mesh_anchors - pred_mesh_anchors.mean(dim=1, keepdim=True)
- gt_mesh_centered = gt_mesh_anchors - gt_mesh_anchors.mean(dim=1, keepdim=True)
-
- # Mesh MPJPE in mm
- err_mesh = torch.linalg.norm(pred_mesh_centered - gt_mesh_centered, dim=-1)
- metrics['Mesh_MPJPE'] = err_mesh.mean().item() * 1000.0
-
- # Mesh PA-MPJPE in mm (Procrustes naturally handles translation offsets!)
- pred_mesh_aligned = batched_procrustes_alignment(pred_mesh_anchors, gt_mesh_anchors)
- err_mesh_pa = torch.linalg.norm(pred_mesh_aligned - gt_mesh_anchors, dim=-1)
- metrics['Mesh_PA_MPJPE'] = err_mesh_pa.mean().item() * 1000.0
-
- return metrics
-
-
-# ============================================================
-# Cell 11 — Full Distillation Training Loop (EMA + Early-Stopping)
-# ============================================================
-def make_ema_avg_fn(max_decay):
- """EMA with a decay WARMUP: decay = min(max_decay, (1+n)/(10+n)).
- Early on (small n) decay is small so the shadow TRACKS the model instead of
- sitting near random init -> removes the early 'mean-pose mush' that gamed PA-MPJPE."""
- @torch.no_grad()
- def avg_fn(ema_params, model_params, num_averaged):
- # Integer buffers (e.g. BatchNorm num_batches_tracked) can't be lerp'd -> copy them,
- # mirroring torch's get_ema_multi_avg_fn. Without this, _foreach_lerp_ errors on Long.
- if not (torch.is_floating_point(ema_params[0]) or torch.is_complex(ema_params[0])):
- for e, m in zip(ema_params, model_params):
- e.copy_(m)
- return
- n = num_averaged.item() if torch.is_tensor(num_averaged) else float(num_averaged)
- decay = min(max_decay, (1.0 + n) / (10.0 + n))
- torch._foreach_lerp_(ema_params, model_params, 1.0 - decay)
- return avg_fn
-
-
-def train_instant_hmr():
- EMA_DECAY = getattr(cfg, "ema_decay", 0.9998)
- EARLY_STOP_PATIENCE = getattr(cfg, "early_stop_patience", 50) # epochs where NEITHER raw nor ema improved
-
- print(f"--- Training {cfg.backbone} | EMA decay={EMA_DECAY} | early-stop patience={EARLY_STOP_PATIENCE} ---")
- os.makedirs(cfg.log_dir, exist_ok=True)
-
- # Set to True to enable resuming
- RESUME_FROM_CHECKPOINT = getattr(cfg, "resume", True)
-
- # 1. Model + EMA shadow + loss
- model = InstantHMRStudent(cfg, pretrained=True).to(device)
- criterion = DistillationLoss(cfg, mhr_module)
-
- # 2. Optimizer & AMP
- optimizer = optim.AdamW(model.parameters(), lr=cfg.lr, weight_decay=cfg.weight_decay)
- scaler = torch.amp.GradScaler(device='cuda', enabled=cfg.use_amp)
-
- # 3. LR schedule
- steps_per_epoch = len(train_loader)
- scheduler = lr_scheduler.OneCycleLR(
- optimizer, max_lr=cfg.lr, epochs=cfg.epochs,
- steps_per_epoch=steps_per_epoch, pct_start=0.1,
- )
-
- best_raw_path = os.path.join(cfg.log_dir, "best_student_model_raw.pth")
- best_ema_path = os.path.join(cfg.log_dir, "best_student_model_ema.pth")
- best_ckpt_path = os.path.join(cfg.log_dir, "best_student_model_v3.pth") # best overall
-
- # Tracking variables
- start_epoch = 0
- best_raw_pa = float('inf')
- best_ema_pa = float('inf')
- best_overall_pa = float('inf')
- epochs_no_improve = 0
-
- # 4. Resume from Checkpoint Logic
- if RESUME_FROM_CHECKPOINT and os.path.exists(best_ckpt_path):
- print(f"🔄 Resuming from checkpoint: {best_ckpt_path}")
- # weights_only=False is often required to load optimizer/scheduler dicts safely in newer PyTorch versions
- ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=False)
-
- # Load states
- model.load_state_dict(ckpt["model_state_dict"], strict=False)
- if "optimizer_state_dict" in ckpt:
- optimizer.load_state_dict(ckpt["optimizer_state_dict"])
- if "scaler_state_dict" in ckpt:
- scaler.load_state_dict(ckpt["scaler_state_dict"])
- if "scheduler_state_dict" in ckpt:
- scheduler.load_state_dict(ckpt["scheduler_state_dict"])
-
- start_epoch = ckpt.get("epoch", -1) + 1
- best_overall_pa = ckpt.get("val_pa_mpjpe", float('inf'))
- best_raw_pa = best_overall_pa
- best_ema_pa = best_overall_pa
- print(f"✅ Resumed successfully from Epoch {start_epoch}. Best PA-MPJPE was {best_overall_pa:.1f}mm.")
- else:
- print("🚀 Starting training from scratch with fresh weights.")
-
- # Initialize EMA *after* potential checkpoint loading so it inherits the resumed weights
- ema_model = AveragedModel(model, multi_avg_fn=make_ema_avg_fn(EMA_DECAY), use_buffers=True)
-
- # --- validation helper: evaluate ANY model over val_loader (closes over criterion/loaders) ---
- @torch.no_grad()
- def run_validation(eval_model, tag, epoch):
- eval_model.eval()
- vloss = {'total': 0.0, '2d': 0.0, '3d': 0.0, 'mhr': 0.0, 'struct': 0.0}
- hmr = {'Head_3D_MPJPE': 0.0, 'Head_3D_PA_MPJPE': 0.0, 'Mesh_MPJPE': 0.0, 'Mesh_PA_MPJPE': 0.0}
- n = 0
- vbar = tqdm(val_loader, desc=f"Epoch {epoch+1}/{cfg.epochs} [Val:{tag}]", leave=False)
- for batch in vbar:
- batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}
- with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):
- preds_fp16 = eval_model(batch["image"], batch["cliff_cond"])
- preds = {k: v.float() for k, v in preds_fp16.items()}
- losses = criterion(preds, batch)
- if math.isnan(losses['total_loss'].item()) or math.isinf(losses['total_loss'].item()):
- continue
- vloss['total'] += losses['total_loss'].item()
- vloss['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()
- vloss['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()
- vloss['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()
- vloss['struct'] += losses.get('loss_structural', torch.tensor(0)).item()
- bm = evaluate_hmr_batch(preds, batch, cfg, mhr_module)
- for k in hmr:
- hmr[k] += bm.get(k, 0.0)
- n += 1
- vbar.set_postfix({'PA-MPJPE': f"{hmr['Mesh_PA_MPJPE']/max(n,1):.1f}mm"})
- vbar.close()
- vloss = {k: v / max(n, 1) for k, v in vloss.items()}
- hmr = {k: v / max(n, 1) for k, v in hmr.items()}
- return vloss, hmr, n
-
- # Note the loop now starts at `start_epoch`
- for epoch in range(start_epoch, cfg.epochs):
- # ---------------- TRAIN ----------------
- model.train()
- tm = {'total': 0.0, '2d': 0.0, '3d': 0.0, 'pose': 0.0, 'scale': 0.0, 'shape': 0.0, 'mhr': 0.0, 'struct': 0.0}
- nb = 0
- pbar = tqdm(train_loader, desc=f"Epoch {epoch+1}/{cfg.epochs} [Train]")
- for batch in pbar:
- batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}
- optimizer.zero_grad(set_to_none=True)
- with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):
- preds_fp16 = model(batch["image"], batch["cliff_cond"])
- preds_fp32 = {k: v.float() for k, v in preds_fp16.items()}
- losses = criterion(preds_fp32, batch)
- loss = losses['total_loss']
- loss_val = loss.item()
- # Skip NaN/Inf AND anomalously large (but finite) batches — a single
- # ~1e14 MHR-loss batch would otherwise take a step and diverge training.
- if math.isnan(loss_val) or math.isinf(loss_val):
- print("⚠️ WARNING: NaN/Inf loss detected! Skipping step.")
- continue
- if loss_val > cfg.anomaly_loss_threshold:
- print(f"⚠️ WARNING: anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold} — skipping step.")
- continue
- scaler.scale(loss).backward()
- scaler.unscale_(optimizer)
- torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
- scale_before = scaler.get_scale()
- scaler.step(optimizer)
- scaler.update()
- scale_after = scaler.get_scale()
- if scale_before <= scale_after:
- scheduler.step()
- ema_model.update_parameters(model)
- tm['total'] += loss.item()
- tm['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()
- tm['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()
- tm['pose'] += losses.get('loss_pose', torch.tensor(0)).item()
- tm['scale'] += losses.get('loss_scale', torch.tensor(0)).item()
- tm['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()
- tm['struct'] += losses.get('loss_structural', torch.tensor(0)).item()
- nb += 1
- pbar.set_postfix({'Tot': f"{loss.item():.3f}",
- 'MHR': f"{losses.get('loss_mhr_joints', torch.tensor(0)).item():.3f}"})
- pbar.close()
- avg_train = {k: v / max(nb, 1) for k, v in tm.items()}
-
- # ---------------- VALIDATE: raw model AND EMA model ----------------
- raw_val, raw_hmr, n_raw = run_validation(model, "raw", epoch)
- ema_val, ema_hmr, n_ema = run_validation(ema_model, "ema", epoch)
-
- # ---------------- SUMMARY ----------------
- print(f"\n📈 Epoch {epoch+1} Summary | LR: {scheduler.get_last_lr()[0]:.2e}")
- print(f" [Train] Tot: {avg_train['total']:.4f} | MHR: {avg_train['mhr']:.4f} | 2D: {avg_train['2d']:.4f} | 3D: {avg_train['3d']:.4f}")
- print(f" [Val RAW] Tot: {raw_val['total']:.4f} | Mesh PA-MPJPE: {raw_hmr['Mesh_PA_MPJPE']:.1f} | Mesh MPJPE: {raw_hmr['Mesh_MPJPE']:.1f} | Head PA: {raw_hmr['Head_3D_PA_MPJPE']:.1f} mm")
- print(f" [Val EMA] Tot: {ema_val['total']:.4f} | Mesh PA-MPJPE: {ema_hmr['Mesh_PA_MPJPE']:.1f} | Mesh MPJPE: {ema_hmr['Mesh_MPJPE']:.1f} | Head PA: {ema_hmr['Head_3D_PA_MPJPE']:.1f} mm")
- delta = raw_hmr['Mesh_PA_MPJPE'] - ema_hmr['Mesh_PA_MPJPE']
- print(f" 🔬 EMA vs RAW (Mesh PA-MPJPE): {'EMA better' if delta > 0 else 'RAW better'} by {abs(delta):.1f} mm")
-
- # ---------------- CHECKPOINTS ----------------
-
- # Save standard states dict for all checkpoints to allow resuming from any of them
- def get_save_dict(model_to_save, val_metric, source_name):
- return {
- 'epoch': epoch,
- 'model_state_dict': model_to_save.state_dict(),
- 'optimizer_state_dict': optimizer.state_dict(),
- 'scaler_state_dict': scaler.state_dict(),
- 'scheduler_state_dict': scheduler.state_dict(),
- 'val_pa_mpjpe': val_metric,
- 'source': source_name
- }
-
- # ---------------- CHECKPOINTS + EARLY-STOP BOOKKEEPING ----------------
- # Did each model improve its OWN best this epoch?
- improved_raw = n_raw > 0 and raw_hmr['Mesh_PA_MPJPE'] < best_raw_pa
- improved_ema = n_ema > 0 and ema_hmr['Mesh_PA_MPJPE'] < best_ema_pa
-
- if improved_raw:
- best_raw_pa = raw_hmr['Mesh_PA_MPJPE']
- torch.save(get_save_dict(model, best_raw_pa, 'raw'), best_raw_path)
- if improved_ema:
- best_ema_pa = ema_hmr['Mesh_PA_MPJPE']
- torch.save(get_save_dict(ema_model.module, best_ema_pa, 'ema'), best_ema_path)
-
- # Canonical "best overall" checkpoint (loaded by the export/eval cells)
- current = min(raw_hmr['Mesh_PA_MPJPE'], ema_hmr['Mesh_PA_MPJPE'])
- if current < best_overall_pa - 1e-4:
- best_overall_pa = current
- use_ema = ema_hmr['Mesh_PA_MPJPE'] <= raw_hmr['Mesh_PA_MPJPE']
- better = ema_model.module if use_ema else model
- torch.save(get_save_dict(better, best_overall_pa, 'ema' if use_ema else 'raw'), best_ckpt_path)
- print(f"\U0001f4be New best overall: {best_overall_pa:.1f} mm ({'EMA' if use_ema else 'RAW'}) -> {os.path.basename(best_ckpt_path)}")
-
- # ---------------- EARLY STOPPING ----------------
- # Stop ONLY after EARLY_STOP_PATIENCE epochs where NEITHER raw NOR ema improved its own best.
- if improved_raw or improved_ema:
- epochs_no_improve = 0
- else:
- epochs_no_improve += 1
- print(f" ⏳ Neither RAW nor EMA improved: {epochs_no_improve}/{EARLY_STOP_PATIENCE} (best raw {best_raw_pa:.1f} / ema {best_ema_pa:.1f} mm)")
- if epochs_no_improve >= EARLY_STOP_PATIENCE:
- print(f"\n\U0001f6d1 Early stopping at epoch {epoch+1}: neither RAW nor EMA improved for {EARLY_STOP_PATIENCE} epochs.")
- break
-
- print(f"\n🎉 Training complete! Best overall Mesh PA-MPJPE: {best_overall_pa:.1f} mm")
- print(f" best RAW: {best_raw_pa:.1f} mm -> {best_raw_path}")
- print(f" best EMA: {best_ema_pa:.1f} mm -> {best_ema_path}")
-
-
-# ============================================================
-# Cell 15 — Export & Static Quantization (FP32, FP16, INT8)
-# ============================================================
-class HMRDeployWrapper(nn.Module):
- """Deploy wrapper that returns the full output tuple."""
- def __init__(self, model):
- super().__init__()
- self.model = model
-
- def forward(self, image, cliff_cond):
- out = self.model(image, cliff_cond)
- return (out["mhr_params"], out["shape_params"], out["cam_trans"],
- out["joints_2d"], out["joints_3d"])
-
-
-def topological_sort_onnx(model):
- """Sort ONNX graph nodes in topological order.
-
- The ORT float16 converter with keep_io_types=True inserts Cast nodes at graph
- inputs but doesn't re-sort the graph, breaking topological order. This fixes it.
- """
- available = set()
- for inp in model.graph.input:
- available.add(inp.name)
- for init in model.graph.initializer:
- available.add(init.name)
-
- remaining = list(model.graph.node)
- sorted_nodes = []
-
- while remaining:
- progress = False
- for node in remaining:
- if all(inp in available or inp == '' for inp in node.input):
- sorted_nodes.append(node)
- remaining.remove(node)
- for out in node.output:
- available.add(out)
- progress = True
- break
- if not progress:
- sorted_nodes.extend(remaining)
- break
-
- del model.graph.node[:]
- model.graph.node.extend(sorted_nodes)
- return model
-
-
-def export_and_quantize():
- print("--- Exporting and Quantizing Full Model for Deployment ---")
-
- # --- INT8 Calibration Data Reader ---
- try:
- from onnxruntime.quantization import (
- CalibrationDataReader, quantize_static,
- QuantType, QuantFormat, CalibrationMethod,
- )
-
- class HMRCalibrationDataReader(CalibrationDataReader):
- def __init__(self, dataloader, max_samples=100):
- self.data_list = []
-
- print(f"Extracting {max_samples} individual images for INT8 calibration...")
- for batch in dataloader:
- images = batch["image"].numpy().astype(np.float32)
- cliffs = batch["cliff_cond"].numpy().astype(np.float32)
-
- for b_idx in range(images.shape[0]):
- if len(self.data_list) >= max_samples:
- break
- self.data_list.append({
- "image": images[b_idx:b_idx + 1],
- "cliff_cond": cliffs[b_idx:b_idx + 1]
- })
- if len(self.data_list) >= max_samples:
- break
-
- self.enum_data = iter(self.data_list)
-
- def get_next(self):
- return next(self.enum_data, None)
-
- except ImportError:
- print("⚠️ 'onnxruntime' is not installed. INT8 quantization will fail.")
- HMRCalibrationDataReader = None
-
- # --- Export & Quantize ---
- export_dir = Path(cfg.log_dir) / "export"
- export_dir.mkdir(parents=True, exist_ok=True)
-
- onnx_fp32_path = export_dir / "sam3d_full_fp32.onnx"
- onnx_fp16_path = export_dir / "sam3d_full_fp16.onnx"
- onnx_int8_path = export_dir / "sam3d_full_int8.onnx"
-
- output_keys = ["mhr_params", "shape_params", "cam_trans", "joints_2d", "joints_3d"]
-
- export_target_model = InstantHMRStudent(cfg, pretrained=False).to(device)
-
- best_ckpt_path = os.path.join(cfg.log_dir, "best_student_model_v3.pth")
-
- if os.path.exists(best_ckpt_path):
- ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=True)
- clean_state_dict = {k.replace('_orig_mod.', ''): v for k, v in ckpt['model_state_dict'].items()}
- export_target_model.load_state_dict(clean_state_dict, strict=False)
- print(f"✅ Loaded best model from epoch {ckpt.get('epoch', 0)+1}.\n")
-
- deploy_model = HMRDeployWrapper(export_target_model)
- deploy_model.eval()
-
- dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)
- dummy_cliff = torch.randn(1, 3, device=device)
-
- dynamic_axes = {"image": {0: "batch"}, "cliff_cond": {0: "batch"}}
- for k in output_keys:
- dynamic_axes[k] = {0: "batch"}
-
- # A. Export FP32
- try:
- import onnx
- torch.onnx.export(
- deploy_model, (dummy_img, dummy_cliff), str(onnx_fp32_path),
- input_names=["image", "cliff_cond"], output_names=output_keys,
- dynamic_axes=dynamic_axes, opset_version=17
- )
- print(f"✅ FP32 saved: {onnx_fp32_path.name} | Size: {onnx_fp32_path.stat().st_size / 1e6:.1f} MB")
- except Exception as e:
- print(f"❌ FP32 export failed: {e}")
-
- # B. Export FP16 — uses ORT's converter which correctly updates Cast node `to` attributes
- # + topological sort to fix node ordering after keep_io_types Cast insertion
- try:
- from onnxruntime.transformers.float16 import convert_float_to_float16
- import warnings
- with warnings.catch_warnings():
- warnings.simplefilter("ignore")
- model_fp32 = onnx.load(str(onnx_fp32_path))
- model_fp16 = convert_float_to_float16(model_fp32, keep_io_types=True)
- model_fp16 = topological_sort_onnx(model_fp16)
- onnx.save(model_fp16, str(onnx_fp16_path))
- onnx.checker.check_model(onnx.load(str(onnx_fp16_path)), full_check=True)
- print(f"✅ FP16 saved: {onnx_fp16_path.name} | Size: {onnx_fp16_path.stat().st_size / 1e6:.1f} MB")
- except ImportError:
- print("⚠️ FP16 skipped (onnxruntime.transformers not available)")
- except Exception as e:
- print(f"❌ FP16 conversion failed: {e}")
-
- # C. Export INT8 — selective: only quantize Conv ops (backbone), skip transformer/heads
- # Naive quantization of all ops destroys accuracy (Softmax, LayerNorm, attention MatMuls
- # need FP32 precision) and is slower due to unfused QDQ overhead blocking NCHWc optimization.
- try:
- calib_reader = HMRCalibrationDataReader(train_loader, max_samples=100)
- calib_reader.enum_data = iter(calib_reader.data_list)
-
- # Collect transformer + head + projection node names to exclude from quantization
- # (defense-in-depth: op_types_to_quantize=['Conv'] already excludes them since they
- # have zero Conv ops, but explicit exclusion prevents issues if the model changes)
- fp32_model = onnx.load(str(onnx_fp32_path))
- nodes_to_exclude = []
- for node in fp32_model.graph.node:
- name = node.name or ''
- if any(kw in name for kw in ['/model/transformer/', '/model/head_',
- '/model/feat_proj', '/model/cond_proj']):
- nodes_to_exclude.append(name)
- del fp32_model
- print(f" Excluding {len(nodes_to_exclude)} transformer/head nodes from quantization")
-
- quantize_static(
- model_input=str(onnx_fp32_path),
- model_output=str(onnx_int8_path),
- calibration_data_reader=calib_reader,
- quant_format=QuantFormat.QDQ,
- op_types_to_quantize=['Conv'], # Only quantize Conv (backbone)
- per_channel=True, # Better accuracy for Conv weights
- reduce_range=False, # Not needed with VNNI
- activation_type=QuantType.QInt8,
- weight_type=QuantType.QInt8,
- nodes_to_exclude=nodes_to_exclude,
- calibrate_method=CalibrationMethod.Percentile,
- extra_options={
- 'CalibTensorRangeSymmetric': True,
- },
- )
- print(f"✅ INT8 saved: {onnx_int8_path.name} | Size: {onnx_int8_path.stat().st_size / 1e6:.1f} MB")
- except NameError:
- print("⚠️ INT8 skipped (train_loader not available)")
- except Exception as e:
- import traceback
- print(f"❌ INT8 conversion failed: {e}")
- traceback.print_exc()
-
- # D. Save PyTorch state dict
- sd_path = export_dir / "sam3d_student_state_dict.pth"
- torch.save(export_target_model.state_dict(), sd_path)
- print(f"\n✅ PyTorch state dict saved: {sd_path.name}")
-
- else:
- print("⚠️ No checkpoint found at:", best_ckpt_path)
-
-
-# ============================================================
-# Optional self-tests (notebook Cell 7 / Cell 8 sanity checks)
-# ============================================================
-def run_self_tests():
- print("--- Architecture Output Shapes ---")
- test_model = InstantHMRStudent(cfg, pretrained=False)
- dummy_img = torch.randn(2, 3, 224, 224)
- dummy_cliff = torch.randn(2, 3)
- out = test_model(dummy_img, dummy_cliff)
- for k, v in out.items():
- print(f"{k}: {v.shape}")
-
- print("\n--- Running Loss Function Sanity Check (Perfect Student) ---")
- criterion = DistillationLoss(cfg, mhr_module)
- sample_batch_cpu = next(iter(train_loader))
- sample_batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in sample_batch_cpu.items()}
- mock_preds = {
- "mhr_params": sample_batch["mhr_model_params"].clone(),
- "shape_params": sample_batch["shape_params"].clone(),
- "cam_trans": sample_batch["cam_trans"].clone(),
- "joints_2d": sample_batch["joints_2d"].clone(),
- "joints_3d": sample_batch["joints_3d"].clone(),
- }
- losses = criterion(mock_preds, sample_batch)
- for k, v in losses.items():
- print(f" {k:<20}: {v.item():.6f}")
- if abs(losses['total_loss'].item()) < 1e-4:
- print("✅ SUCCESS: The Perfect Student achieved a loss of 0! Math verified.")
- else:
- print("❌ ERROR: Loss is not zero. Check the math.")
-
-
-# ============================================================
-# Entry point
-# ============================================================
-# Module-level globals populated by main() so the cell code above
-# (which references them directly) keeps working unchanged.
-cfg = None
-train_loader = None
-val_loader = None
-full_dataset = None
-mhr_module = None
-
-def run_overfit_test(steps=1000, subset=8, lr=5e-4):
- print(f"--- 1-Batch Overfit Test ({subset} images, {steps} steps) ---")
- model = InstantHMRStudent(cfg, pretrained=True).to(device)
- criterion = DistillationLoss(cfg, mhr_module)
- optimizer = optim.AdamW(model.parameters(), lr=lr, weight_decay=0.0)
- scaler = torch.amp.GradScaler(device='cuda', enabled=cfg.use_amp)
-
- batch = next(iter(train_loader))
- batch = {k: (v[:subset].to(device) if isinstance(v, torch.Tensor) else v) for k, v in batch.items()}
-
- def reg(ls):
- return (ls['loss_2d_native'].item() + ls['loss_3d_native'].item()
- + ls['loss_cam'].item() + ls['loss_pose'].item() + ls.get('loss_bone_length', torch.tensor(0)).item())
-
- first_reg = last_reg = last_pa = None
- n_skipped = 0
- model.train()
-
- for step in range(steps):
- optimizer.zero_grad(set_to_none=True)
- with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):
- preds = model(batch["image"], batch["cliff_cond"])
- preds = {k: v.float() for k, v in preds.items()}
- losses = criterion(preds, batch)
-
- lv = losses['total_loss'].item()
- if math.isnan(lv) or math.isinf(lv) or lv > cfg.anomaly_loss_threshold:
- n_skipped += 1
- continue
-
- scaler.scale(losses['total_loss']).backward()
- scaler.unscale_(optimizer)
- torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)
- scaler.step(optimizer)
- scaler.update()
-
- if first_reg is None:
- first_reg = reg(losses)
- last_reg = reg(losses)
-
- if step % 50 == 0 or step == steps - 1:
- model.eval()
- with torch.no_grad():
- p = {k: v.float() for k, v in model(batch["image"], batch["cliff_cond"]).items()}
- m = evaluate_hmr_batch(p, batch, cfg, mhr_module)
- model.train()
- last_pa = m['Head_3D_PA_MPJPE']
-
- # UPDATED PRINT STATEMENT: Tracking Shape and Raw MPJPE
- print(f" step {step:04d} | tot {losses['total_loss'].item():.3f} "
- f"| 3d {losses['loss_3d_native'].item():.4f} "
- f"| pose {losses['loss_pose'].item():.4f} "
- f"| shape {losses.get('loss_shape', torch.tensor(0)).item():.4f} "
- f"| mhr {losses['loss_mhr_joints'].item():.4f} "
- f"| Head [MPJPE: {m['Head_3D_MPJPE']:.1f} | PA: {m['Head_3D_PA_MPJPE']:.1f}] mm "
- f"| Mesh [MPJPE: {m['Mesh_MPJPE']:.1f} | PA: {m['Mesh_PA_MPJPE']:.1f}] mm")
-
- ok = (last_reg < first_reg * 0.10) and (last_pa is not None and last_pa < 40.0)
- print(f"\n{'✅ SUCCESS' if ok else '❌ FAIL'}: regression loss {first_reg:.4f} -> {last_reg:.5f} "
- f"| final Head PA-MPJPE {last_pa:.1f} mm | anomalous steps skipped: {n_skipped}")
- return ok
-
-def parse_args():
- p = argparse.ArgumentParser(description="Standalone InstantHMR distillation training.")
- p.add_argument("--data_root", type=str, default=None,
- help="Entry-point folder containing sub-folders with annotations/ + images/.")
- p.add_argument("--output_dir", type=str, default=None,
- help="Where checkpoints/logs/exports are written (cfg.log_dir).")
- p.add_argument("--mhr_model_path", type=str, default=None,
- help="Path to the TorchScript MHR model (mhr_model.pt).")
- p.add_argument("--epochs", type=int, default=None)
- p.add_argument("--batch_size", type=int, default=None)
- p.add_argument("--lr", type=float, default=None)
- p.add_argument("--num_workers", type=int, default=None)
- p.add_argument("--max_images", type=int, default=None)
- p.add_argument("--harmony4d_cap", type=int, default=None,
- help="Max Harmony4D crops kept (randomly sampled). 0 = uncapped/use all.")
- p.add_argument("--no-resume", dest="no_resume", action="store_true",
- help="Ignore any existing checkpoint and train from scratch.")
- p.add_argument("--self-test", dest="self_test", action="store_true",
- help="Run the geometry/MHR-consistency checks and exit.")
- p.add_argument("--overfit-test", dest="overfit_test", action="store_true",
- help="Run the 1-batch overfit sanity check and exit.")
- p.add_argument("--no-export", dest="no_export", action="store_true",
- help="Skip the ONNX export / quantization step after training.")
- p.add_argument("--gpu", type=int, default=None, help="GPU count (informational).")
- args, unknown = p.parse_known_args()
- if unknown:
- print(f"⚠️ Ignoring unrecognized arguments: {unknown}")
- return args
-
-
-def main():
- global cfg, train_loader, val_loader, full_dataset, mhr_module
-
- args = parse_args()
-
- cfg = DistillConfig()
- if args.data_root is not None: cfg.data_root = args.data_root
- if args.output_dir is not None: cfg.log_dir = args.output_dir
- if args.mhr_model_path is not None: cfg.mhr_model_path = args.mhr_model_path
- if args.epochs is not None: cfg.epochs = args.epochs
- if args.batch_size is not None: cfg.batch_size = args.batch_size
- if args.lr is not None: cfg.lr = args.lr
- if args.num_workers is not None: cfg.num_workers = args.num_workers
- if args.max_images is not None: cfg.max_images = args.max_images
- if args.harmony4d_cap is not None:
- if args.harmony4d_cap <= 0:
- cfg.per_dataset_caps.pop("sam3d_gt_harmony4d", None) # 0 = uncapped
- else:
- cfg.per_dataset_caps["sam3d_gt_harmony4d"] = args.harmony4d_cap
- if args.no_resume: cfg.resume = False
-
- print("=" * 60)
- print(f"Project root : {PROJECT_ROOT}")
- print(f"Data root : {cfg.data_root}")
- print(f"Output dir : {cfg.log_dir}")
- print(f"MHR model : {cfg.mhr_model_path}")
- print(f"Backbone : {cfg.backbone} | epochs={cfg.epochs} | batch={cfg.batch_size} | lr={cfg.lr}")
- print(f"max_images : {cfg.max_images if cfg.max_images is not None else 'unlimited (all crops)'}")
- print(f"dataset caps : {cfg.per_dataset_caps if cfg.per_dataset_caps else 'none'}")
- print("=" * 60)
-
- if not Path(cfg.mhr_model_path).exists():
- raise FileNotFoundError(
- f"MHR model not found at '{cfg.mhr_model_path}'. "
- f"Place mhr_model.pt in the checkpoints/ folder or pass --mhr_model_path."
- )
-
- # Build data (Cell 3), MHR module (Cell 8)
- train_loader, val_loader, full_dataset = build_dataloaders(cfg)
- mhr_module = MHRForwardPass(cfg.mhr_model_path, device)
-
- if args.self_test:
- run_self_tests()
-
- if args.overfit_test:
- run_overfit_test()
- return
- # Train (Cell 11)
- gc.collect()
- if torch.cuda.is_available():
- torch.cuda.empty_cache()
- train_instant_hmr()
-
- # Export + quantize (Cell 15)
- if not args.no_export:
- export_and_quantize()
-
-
-if __name__ == "__main__":
- main()
diff --git a/models/README.md b/models/README.md
index 721c67f..e9541ce 100644
--- a/models/README.md
+++ b/models/README.md
@@ -41,8 +41,7 @@ repository: <https://github.com/mohamdev/InstantHMR>.
Most users only need `instanthmr.onnx`. The `.pth` is included for people
who want to continue training, swap the head, or re-export with different
-options — see [`docs/training.md`](https://github.com/mohamdev/InstantHMR/blob/main/docs/training.md)
-in the main repo.
+options — see [`training/README.md`](../training/README.md).
## ONNX I/O
diff --git a/notebooks/distill_transformer_decoder.ipynb b/notebooks/distill_transformer_decoder.ipynb
deleted file mode 100644
index 54bfc94..0000000
--- a/notebooks/distill_transformer_decoder.ipynb
+++ /dev/null
@@ -1,2501 +0,0 @@
-{
- "cells": [
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ad5fd1be",
- "metadata": {},
- "outputs": [],
- "source": [
- "import torch\n",
- "import torch.nn as nn\n",
- "import torch.nn.functional as F\n",
- "import torch.optim as optim\n",
- "import torchvision.transforms as T\n",
- "import matplotlib.pyplot as plt\n",
- "import timm\n",
- "import cv2\n",
- "from pathlib import Path\n",
- "from tqdm.notebook import tqdm\n",
- "import onnxruntime as ort\n",
- "print(ort.get_available_providers())\n",
- "\n",
- "\n",
- "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- "import os; os.environ[\"PYTORCH_CUDA_ALLOC_CONF\"] = \"expandable_segments:True\"\n",
- "print(f\"Using device: {device}\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "7e2ae437",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 1 — Configuration\n",
- "# ============================================================\n",
- "from dataclasses import dataclass \n",
- "from pathlib import Path\n",
- "\n",
- "PROJECT_ROOT = Path(\"/home/madjel/Projects/gitpackages/sam3d-student-release/\") #set your path\n",
- "\n",
- "@dataclass\n",
- "class DistillConfig:\n",
- " \"\"\"All hyperparameters in one place.\"\"\"\n",
- " # --- Paths ---\n",
- " images_dir: str = str(PROJECT_ROOT / \"data/sam3d_gt_harmony4d/images\") #path to cropped images from tools/annotate_dataset.py\n",
- " annotations_dir: str = str(PROJECT_ROOT / \"data/sam3d_gt_harmony4d/annotations\") #path to annotations from tools/annotate_dataset.py\n",
- " # images_dir: str = str(\"/home/madjel/Projects/gitpackages/sam3-biomechanics/video_to_pose_pipeline/micro_dataset_cliff/images\") #path to cropped images from tools/annotate_dataset.py\n",
- " # annotations_dir: str = str(\"/home/madjel/Projects/gitpackages/sam3-biomechanics/video_to_pose_pipeline/micro_dataset_cliff/annotations\") #path to annotations from tools/annotate_dataset.py\n",
- " mhr_model_path: str = str(PROJECT_ROOT / \"checkpoints/mhr_model.pt\") #path to MHR model\n",
- " log_dir: str = str(PROJECT_ROOT / \"runs/distill_repvit_cliff_v3\") # Bumped to v3!\n",
- "\n",
- " print(f\"Project root: {PROJECT_ROOT}\")\n",
- " print(f\"Images dir: {images_dir}\")\n",
- " print(f\"Annotations dir: {annotations_dir}\")\n",
- " print(f\"MHR model path: {mhr_model_path}\")\n",
- " # --- Data ---\n",
- " image_size: int = 224\n",
- " max_images: int = 100_000 \n",
- " val_split: float = 0.1\n",
- " num_workers: int = 4\n",
- " augment: bool = True\n",
- "\n",
- " # --- Model (Transformer + CLIFF) ---\n",
- " backbone: str = \"repvit_m2_3\" \n",
- " backbone_feat_dim: int = 640 # RepViT m2_3 usually outputs 640\n",
- " d_model: int = 512 \n",
- " n_heads: int = 8 \n",
- " n_decoder_layers: int = 4 \n",
- " dropout: float = 0.1\n",
- "\n",
- " # --- Output dimensions ---\n",
- " pose_dim: int = 136 \n",
- " scale_dim: int = 68 \n",
- " shape_dim: int = 45 \n",
- " cam_dim: int = 3 \n",
- " cliff_dim: int = 3 \n",
- " num_joints: int = 70 # We track 70 joints\n",
- "\n",
- " @property\n",
- " def model_params_dim(self) -> int:\n",
- " return self.pose_dim + self.scale_dim # 204\n",
- "\n",
- " # --- Training ---\n",
- " batch_size: int = 64\n",
- " lr: float = 3e-4\n",
- " weight_decay: float = 1e-4\n",
- " epochs: int = 400\n",
- " warmup_epochs: int = 3\n",
- " grad_clip: float = 5.0\n",
- " use_amp: bool = True \n",
- " ema_decay: float = 0.9998 # EMA shadow-weights decay (Cell 11)\n",
- " early_stop_patience: int = 150 # stop after N epochs where NEITHER raw nor ema improved (Cell 11)\n",
- "\n",
- " # --- Loss weights (UPDATED for HMR Balance) ---\n",
- " w_pose: float = 1.0\n",
- " w_scale: float = 0.1\n",
- " w_shape: float = 1.0\n",
- " w_cam: float = 0.1 \n",
- " w_keypoints3d: float = 10.0 # Bumped up: 3D joints are the strongest spatial anchor\n",
- " w_keypoints2d: float = 10.0 # Bumped up: Forces pixel-perfect alignment\n",
- " w_3d_joints: float = 1e-3\n",
- " w_structural: float = 0.01\n",
- " w_reproj_3d: float = 0.01 # High weight to enforce strict scale/camera alignment\n",
- " w_reproj_mhr: float = 0.01 # High weight to enforce strict scale/camera alignment\n",
- "\n",
- " # ==========================================================\n",
- " # Feet Barycenter Anchors\n",
- " # ==========================================================\n",
- " w_feet = 0.01 # Soft prior weight, matching w_structural\n",
- " \n",
- " # Native 70-joint format indices\n",
- " native_left_foot = [15, 16, 17]\n",
- " native_right_foot = [18, 19, 20]\n",
- " \n",
- " # MHR 127-joint format indices\n",
- " mhr_left_foot = [6, 7, 8] \n",
- " mhr_right_foot = [22, 23, 24]\n",
- " \n",
- " # Data-Driven Topology Mapping (Native 70 <-> MHR 127)\n",
- " native_mapping_ids = [ 1, 2, 5, 6, 9, 10, 11, 12, 13, 14, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62 ]\n",
- " mhr_mapping_ids = [ 125, 123, 75, 39, 2, 18, 3, 19, 5, 21, 64, 63, 62, 61, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 41, 100, 99, 98, 97, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 77 ]\n",
- "\n",
- "cfg = DistillConfig()\n",
- "print(f\"Config loaded. Backbone: {cfg.backbone}. Ready for V3 Training.\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "561a65f3",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ==============================================================================\n",
- "# Cell 2 — X-Ray Dataset Visualizer (Math Verification)\n",
- "# ==============================================================================\n",
- "import numpy as np\n",
- "import cv2\n",
- "import torch\n",
- "import matplotlib.pyplot as plt\n",
- "from pathlib import Path\n",
- "import random\n",
- "\n",
- "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- "\n",
- "\n",
- "# --- 1. Load Lightweight MHR Wrapper ---\n",
- "class MHRForwardPass:\n",
- " def __init__(self, mhr_path, device):\n",
- " self.device = device\n",
- " self.mhr = torch.jit.load(mhr_path, map_location=device).eval()\n",
- " for p in self.mhr.parameters():\n",
- " p.requires_grad_(False)\n",
- "\n",
- " @torch.no_grad()\n",
- " def get_joints(self, model_params, shape_params):\n",
- " B = model_params.shape[0]\n",
- " dummy_identity = torch.zeros(B, 45, device=self.device, dtype=model_params.dtype)\n",
- " concat_params = torch.cat([model_params, dummy_identity], dim=1)\n",
- " joint_params = self.mhr.character_torch.model_parameters_to_joint_parameters(concat_params)\n",
- " skel_state = self.mhr.character_torch.joint_parameters_to_skeleton_state(joint_params)\n",
- " return skel_state\n",
- "\n",
- "mhr_module = MHRForwardPass(cfg.mhr_model_path, device)\n",
- "\n",
- "# --- 2. Load a Random Sample ---\n",
- "npz_files = list(Path(cfg.annotations_dir).glob(\"*.npz\"))\n",
- "sample_npz = random.choice(npz_files)\n",
- "sample_stem = sample_npz.stem\n",
- "\n",
- "img_path = Path(cfg.images_dir) / f\"{sample_stem}.jpg\"\n",
- "if not img_path.exists():\n",
- " img_path = Path(cfg.images_dir) / f\"{sample_stem}.png\"\n",
- "\n",
- "ann = np.load(str(sample_npz))\n",
- "img_bgr = cv2.imread(str(img_path))\n",
- "img_rgb = cv2.cvtColor(img_bgr, cv2.COLOR_BGR2RGB)\n",
- "\n",
- "# --- 3. Extract Ground Truths ---\n",
- "gt_2d_full = ann[\"joints_2d\"] \n",
- "gt_3d_raw = ann[\"joints_3d\"] \n",
- "model_params = ann[\"mhr_model_params\"] \n",
- "shape_params = ann[\"shape_params\"] \n",
- "cam_trans = ann[\"cam_trans\"] \n",
- "focal_length = ann[\"cam_focal_length\"] \n",
- "orig_shape = ann[\"orig_shape\"] \n",
- "bbox_square = ann[\"bbox_square\"] \n",
- "\n",
- "# --- 4. Forward Pass through MHR with 6D Math Injection ---\n",
- "mp_tensor = torch.from_numpy(model_params).unsqueeze(0).to(device)\n",
- "sp_tensor = torch.from_numpy(shape_params).unsqueeze(0).to(device)\n",
- "raw_mhr_joints = mhr_module.get_joints(mp_tensor, sp_tensor)[0, :, :3].cpu().numpy()\n",
- "\n",
- "# ------------------------------------------------------------------------------\n",
- "# THE HOLY GRAIL MATH: Coordinate Transformations\n",
- "# ------------------------------------------------------------------------------\n",
- "# A. MHR -> Vision Coordinates\n",
- "raw_mhr_joints = raw_mhr_joints / 100.0 \n",
- "raw_mhr_joints[:, 1] = -raw_mhr_joints[:, 1] \n",
- "raw_mhr_joints[:, 2] = -raw_mhr_joints[:, 2] \n",
- "\n",
- "# B. SAM3D 70-Joints -> Camera Space\n",
- "j3d_70_cam = gt_3d_raw + cam_trans \n",
- "\n",
- "# --- 5. Project 3D to Full-Frame 2D ---\n",
- "orig_h, orig_w = orig_shape\n",
- "principal_point = np.array([orig_w / 2.0, orig_h / 2.0])\n",
- "\n",
- "mhr_cam = raw_mhr_joints + cam_trans\n",
- "z_mhr = np.maximum(mhr_cam[:, 2], 0.01)\n",
- "u_mhr_full = (focal_length[0] * mhr_cam[:, 0] / z_mhr) + principal_point[0]\n",
- "v_mhr_full = (focal_length[1] * mhr_cam[:, 1] / z_mhr) + principal_point[1]\n",
- "\n",
- "z_70 = np.maximum(j3d_70_cam[:, 2], 0.01)\n",
- "u_70_full = (focal_length[0] * j3d_70_cam[:, 0] / z_70) + principal_point[0]\n",
- "v_70_full = (focal_length[1] * j3d_70_cam[:, 1] / z_70) + principal_point[1]\n",
- "\n",
- "# --- 6. Map Full-Frame 2D to 224x224 Crop Space ---\n",
- "x1, y1, x2, y2 = bbox_square\n",
- "crop_w, crop_h = x2 - x1, y2 - y1\n",
- "\n",
- "u_mhr_crop = (u_mhr_full - x1) / max(crop_w, 1) * cfg.image_size\n",
- "v_mhr_crop = (v_mhr_full - y1) / max(crop_h, 1) * cfg.image_size\n",
- "\n",
- "u_70_crop = (u_70_full - x1) / max(crop_w, 1) * cfg.image_size\n",
- "v_70_crop = (v_70_full - y1) / max(crop_h, 1) * cfg.image_size\n",
- "\n",
- "gt_u_crop = (gt_2d_full[:, 0] - x1) / max(crop_w, 1) * cfg.image_size\n",
- "gt_v_crop = (gt_2d_full[:, 1] - y1) / max(crop_h, 1) * cfg.image_size\n",
- "\n",
- "# --- 7. Visualization ---\n",
- "fig, axes = plt.subplots(1, 2, figsize=(12, 6))\n",
- "\n",
- "axes[0].imshow(img_rgb)\n",
- "axes[0].scatter(gt_u_crop, gt_v_crop, s=15, c='lime', edgecolors='black', label='Native 2D')\n",
- "axes[0].scatter(u_70_crop, v_70_crop, s=40, facecolors='none', edgecolors='cyan', linewidths=1.5, label='Projected 3D (70pts)')\n",
- "axes[0].set_title(f\"Target 1: Native 2D vs Projected 3D Head\\n{sample_stem}\", fontsize=10)\n",
- "axes[0].legend(loc='upper right')\n",
- "axes[0].axis('off')\n",
- "\n",
- "axes[1].imshow(img_rgb)\n",
- "axes[1].scatter(u_mhr_crop, v_mhr_crop, s=15, c='red', edgecolors='black', marker='X')\n",
- "axes[1].set_title(f\"Target 2: Projected MHR Params\\nMath verification (6D Cycle)\", fontsize=10)\n",
- "axes[1].axis('off')\n",
- "\n",
- "plt.tight_layout()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "1d5e32ef",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 3 — Dataset and Data Augmentation\n",
- "# ============================================================\n",
- "import os\n",
- "import glob\n",
- "import random\n",
- "import numpy as np\n",
- "from pathlib import Path\n",
- "from PIL import Image\n",
- "\n",
- "import torch\n",
- "from torch.utils.data import Dataset, DataLoader, random_split\n",
- "import torchvision.transforms as transforms\n",
- "import torchvision.transforms.functional as F_t\n",
- "from torchvision.transforms import InterpolationMode\n",
- "\n",
- "class RandomPixelate:\n",
- " \"\"\"Simulates low-quality webcams or low-bandwidth video.\"\"\"\n",
- " def __init__(self, p=0.2, min_res=32, max_res=112):\n",
- " self.p = p\n",
- " self.min_res = min_res \n",
- " self.max_res = max_res\n",
- "\n",
- " def __call__(self, img):\n",
- " if random.random() < self.p:\n",
- " orig_w, orig_h = img.size\n",
- " degraded_h = random.randint(self.min_res, self.max_res)\n",
- " degraded_w = int(orig_w * (degraded_h / orig_h))\n",
- " # Downsample then upsample to create pixelation effect\n",
- " img = F_t.resize(img, [degraded_h, degraded_w], interpolation=InterpolationMode.NEAREST)\n",
- " img = F_t.resize(img, [orig_h, orig_w], interpolation=InterpolationMode.NEAREST)\n",
- " return img\n",
- " \n",
- "\n",
- "class SAM3DStudentDataset(Dataset):\n",
- " def __init__(self, images_dir: str, annotations_dir: str,\n",
- " image_size: int = 224, max_images: int | None = None,\n",
- " augment: bool = True):\n",
- " super().__init__()\n",
- " self.image_size = image_size\n",
- " self.images_dir = Path(images_dir)\n",
- " self.annotations_dir = Path(annotations_dir)\n",
- "\n",
- " # --- 1. Build Transform Pipeline ---\n",
- " # The images on disk are already 224x224, but Resize guarantees safety.\n",
- " tfm_list = [transforms.Resize((image_size, image_size))]\n",
- " \n",
- " if augment:\n",
- " tfm_list.extend([\n",
- " transforms.ColorJitter(brightness=0.2, contrast=0.2, saturation=0.2, hue=0.1),\n",
- " RandomPixelate(p=0.2, min_res=48, max_res=112), \n",
- " transforms.RandomApply([transforms.GaussianBlur(kernel_size=5)], p=0.2), \n",
- " ])\n",
- " \n",
- " tfm_list.extend([\n",
- " transforms.ToTensor(),\n",
- " # Standard ImageNet Normalization\n",
- " transforms.Normalize(mean=[0.485, 0.456, 0.406],\n",
- " std=[0.229, 0.224, 0.225]),\n",
- " ])\n",
- " \n",
- " if augment:\n",
- " tfm_list.append(transforms.RandomErasing(p=0.2, scale=(0.02, 0.15)))\n",
- " \n",
- " self.transform = transforms.Compose(tfm_list)\n",
- "\n",
- " # --- 2. Pair Discovery ---\n",
- " # Fast indexing of available NPZ files and corresponding images\n",
- " ann_files = list(self.annotations_dir.glob(\"*.npz\"))\n",
- " \n",
- " self.pairs = []\n",
- " for npz_path in sorted(ann_files):\n",
- " stem = npz_path.stem\n",
- " # Check for jpg or png\n",
- " img_path = self.images_dir / f\"{stem}.jpg\"\n",
- " if not img_path.exists():\n",
- " img_path = self.images_dir / f\"{stem}.png\"\n",
- " \n",
- " if img_path.exists():\n",
- " self.pairs.append((img_path, npz_path))\n",
- "\n",
- " if max_images is not None:\n",
- " self.pairs = self.pairs[:max_images]\n",
- "\n",
- " def __len__(self):\n",
- " return len(self.pairs)\n",
- "\n",
- " def __getitem__(self, idx):\n",
- " img_path, npz_path = self.pairs[idx]\n",
- "\n",
- " # --- 3. Load and Augment Image ---\n",
- " image = Image.open(img_path).convert(\"RGB\")\n",
- " image = self.transform(image) # Returns float tensor [3, 224, 224]\n",
- "\n",
- " # --- 4. Load Annotations ---\n",
- " ann = np.load(npz_path)\n",
- " \n",
- " orig_h, orig_w = ann[\"orig_shape\"]\n",
- " \n",
- " # --- 5. Correct Mathematical Rescaling for 2D Joints ---\n",
- " sq_bbox = ann[\"bbox_square\"] \n",
- " sq_x1, sq_y1 = sq_bbox[0], sq_bbox[1]\n",
- " orig_crop_size = sq_bbox[2] - sq_bbox[0]\n",
- " \n",
- " # Scale factor from the original crop square to the 224x224 tensor\n",
- " true_scale = self.image_size / max(orig_crop_size, 1.0)\n",
- "\n",
- " # Shift full-frame 2D joints relative to crop top-left, then scale\n",
- " joints_2d = ann[\"joints_2d\"].copy()\n",
- " joints_2d[:, 0] = (joints_2d[:, 0] - sq_x1) * true_scale\n",
- " joints_2d[:, 1] = (joints_2d[:, 1] - sq_y1) * true_scale\n",
- "\n",
- " # NEW: Normalize to [-1, 1] for neural network stability!\n",
- " joints_2d[:, 0] = (joints_2d[:, 0] / self.image_size) * 2.0 - 1.0\n",
- " joints_2d[:, 1] = (joints_2d[:, 1] / self.image_size) * 2.0 - 1.0\n",
- "\n",
- " joints_2d = torch.from_numpy(joints_2d).float()\n",
- "\n",
- " # --- 6. CLIFF Conditioning Vectors ---\n",
- " # Using the tighter YOLO bbox to tell the network where the person actually is\n",
- " tight_bbox = ann[\"bbox\"] \n",
- " cx = (tight_bbox[0] + tight_bbox[2]) / 2.0\n",
- " cy = (tight_bbox[1] + tight_bbox[3]) / 2.0\n",
- " \n",
- " # Normalize to [-1, 1] space for better network stability\n",
- " cx_norm = 2.0 * (cx / orig_w) - 1.0\n",
- " cy_norm = 2.0 * (cy / orig_h) - 1.0\n",
- " \n",
- " # Scale relative to the longest edge of the original image\n",
- " b_size = max(tight_bbox[2] - tight_bbox[0], tight_bbox[3] - tight_bbox[1])\n",
- " b_scale = b_size / max(orig_w, orig_h)\n",
- " cliff_cond = torch.tensor([cx_norm, cy_norm, b_scale], dtype=torch.float32)\n",
- "\n",
- " return {\n",
- " \"image\": image, # [3, 224, 224]\n",
- " \"cliff_cond\": cliff_cond, # [3]\n",
- " \"mhr_model_params\": torch.from_numpy(ann[\"mhr_model_params\"]).float(), # [204]\n",
- " \"shape_params\": torch.from_numpy(ann[\"shape_params\"]).float(), # [45]\n",
- " \"cam_trans\": torch.from_numpy(ann[\"cam_trans\"]).float(), # [3]\n",
- " \"cam_focal\": torch.from_numpy(ann[\"cam_focal_length\"]).float(), # [2]\n",
- " \"joints_2d\": joints_2d, # [70, 2] Crop Space\n",
- " \"joints_3d\": torch.from_numpy(ann[\"joints_3d\"]).float(), # [70, 3] Root-centered\n",
- " \"orig_shape\": torch.tensor([orig_h, orig_w], dtype=torch.float32), # [2]\n",
- " \"bbox_square\": sq_bbox\n",
- " }\n",
- "\n",
- "# --- Initialization & Dataloaders ---\n",
- "full_dataset = SAM3DStudentDataset(\n",
- " cfg.images_dir, \n",
- " cfg.annotations_dir, \n",
- " augment=cfg.augment, \n",
- " max_images=cfg.max_images\n",
- ")\n",
- "\n",
- "val_dataset_clean = SAM3DStudentDataset(\n",
- " cfg.images_dir, \n",
- " cfg.annotations_dir, \n",
- " augment=False, \n",
- " max_images=cfg.max_images\n",
- ")\n",
- "\n",
- "n_val = int(len(full_dataset) * cfg.val_split)\n",
- "n_train = len(full_dataset) - n_val\n",
- "\n",
- "# Use a generator for reproducible splits\n",
- "generator = torch.Generator().manual_seed(42)\n",
- "train_dataset, _ = random_split(full_dataset, [n_train, n_val], generator=generator)\n",
- "\n",
- "generator = torch.Generator().manual_seed(42)\n",
- "_, val_dataset = random_split(val_dataset_clean, [n_train, n_val], generator=generator)\n",
- "\n",
- "train_loader = DataLoader(\n",
- " train_dataset, \n",
- " batch_size=cfg.batch_size, \n",
- " shuffle=True, \n",
- " num_workers=cfg.num_workers, \n",
- " pin_memory=True, \n",
- " drop_last=True\n",
- ")\n",
- "\n",
- "val_loader = DataLoader(\n",
- " val_dataset, \n",
- " batch_size=cfg.batch_size, \n",
- " shuffle=False, \n",
- " num_workers=cfg.num_workers, \n",
- " pin_memory=True\n",
- ")\n",
- "\n",
- "\n",
- "print(f\"Dataset Loaded! Train batches: {len(train_loader)} | Val batches: {len(val_loader)}\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "0f003098",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ==============================================================================\n",
- "# Cell 4 — CLIFF & Augmentation Sanity Check\n",
- "# ==============================================================================\n",
- "import matplotlib.pyplot as plt\n",
- "import cv2\n",
- "\n",
- "# 1. Grab a single batch from the DataLoader\n",
- "sample_batch = next(iter(train_loader))\n",
- "\n",
- "# Extract the first item in the batch\n",
- "idx = 0\n",
- "img_tensor = sample_batch[\"image\"][idx]\n",
- "# Un-normalize the 2D joints for visualization ---\n",
- "joints_2d_norm = sample_batch[\"joints_2d\"][idx].numpy() # Currently in [-1, 1] space\n",
- "joints_2d_crop = ((joints_2d_norm + 1.0) / 2.0) * cfg.image_size # Map back to [0, 224] pixel space\n",
- "\n",
- "# Get MHR parameters and Camera data\n",
- "mhr_params = sample_batch[\"mhr_model_params\"][idx:idx+1].to(device)\n",
- "shape_params = sample_batch[\"shape_params\"][idx:idx+1].to(device)\n",
- "cam_trans = sample_batch[\"cam_trans\"][idx].numpy()\n",
- "cam_focal = sample_batch[\"cam_focal\"][idx].numpy() # Original UNTOUCHED focal length\n",
- "\n",
- "# Get CLIFF Conditioning & Geometry data\n",
- "cliff_cond = sample_batch[\"cliff_cond\"][idx].numpy()\n",
- "orig_shape = sample_batch[\"orig_shape\"][idx].numpy() # [H, W]\n",
- "bbox_square = sample_batch[\"bbox_square\"][idx].numpy() # [x1, y1, x2, y2]\n",
- "\n",
- "orig_h, orig_w = orig_shape[0], orig_shape[1]\n",
- "\n",
- "# ------------------------------------------------------------------------------\n",
- "# SANITY CHECK 1: Reverse-Engineer CLIFF Conditioning\n",
- "# ------------------------------------------------------------------------------\n",
- "# cliff_cond = [cx_norm, cy_norm, scale]\n",
- "cx_full = ((cliff_cond[0] + 1.0) / 2.0) * orig_w\n",
- "cy_full = ((cliff_cond[1] + 1.0) / 2.0) * orig_h\n",
- "box_size_full = cliff_cond[2] * max(orig_h, orig_w)\n",
- "\n",
- "print(\"--- CLIFF Conditioning Verification ---\")\n",
- "print(f\"Original Image Size: {orig_w}x{orig_h}\")\n",
- "print(f\"BBox Center in Full Image: X={cx_full:.1f}, Y={cy_full:.1f}\")\n",
- "print(f\"BBox Max Dimension: {box_size_full:.1f} pixels\")\n",
- "print(f\"Does this look like a human bounding box? (Yes, if center is within image bounds)\")\n",
- "print(\"---------------------------------------\")\n",
- "\n",
- "# 2. Un-normalize the image tensor for viewing\n",
- "mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)\n",
- "std = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)\n",
- "img_unnorm = img_tensor * std + mean\n",
- "img_np = img_unnorm.permute(1, 2, 0).numpy().clip(0, 1)\n",
- "\n",
- "# 3. Generate 3D Joints using MHR\n",
- "raw_3d_joints = mhr_module.get_joints(mhr_params, shape_params)[0, :, :3].cpu().numpy()\n",
- "raw_3d_joints = raw_3d_joints / 100.0 # cm to meters\n",
- "raw_3d_joints[:, 1] = -raw_3d_joints[:, 1] # Y-down convention\n",
- "raw_3d_joints[:, 2] = -raw_3d_joints[:, 2] # Z-forward convention\n",
- "\n",
- "# ------------------------------------------------------------------------------\n",
- "# SANITY CHECK 2: The CLIFF Projection Math\n",
- "# ------------------------------------------------------------------------------\n",
- "# A. Project natively into the FULL FRAME space\n",
- "j3d_cam = raw_3d_joints + cam_trans\n",
- "z = np.maximum(j3d_cam[:, 2], 0.01)\n",
- "\n",
- "principal_point_x = orig_w / 2.0\n",
- "principal_point_y = orig_h / 2.0\n",
- "\n",
- "u_full = (cam_focal[0] * j3d_cam[:, 0] / z) + principal_point_x\n",
- "v_full = (cam_focal[1] * j3d_cam[:, 1] / z) + principal_point_y\n",
- "\n",
- "# B. Map the full-frame 2D points into the 224x224 crop space\n",
- "sq_x1, sq_y1, sq_x2, sq_y2 = bbox_square\n",
- "crop_w = sq_x2 - sq_x1\n",
- "crop_h = sq_y2 - sq_y1\n",
- "true_scale = 224.0 / max(crop_w, 1.0)\n",
- "\n",
- "u_crop_proj = (u_full - sq_x1) * true_scale\n",
- "v_crop_proj = (v_full - sq_y1) * true_scale\n",
- "\n",
- "# 4. Visualize\n",
- "fig, axes = plt.subplots(1, 2, figsize=(12, 6))\n",
- "\n",
- "# Plot 1: The Native 2D labels from the dataset (Checks augmentation/mapping)\n",
- "axes[0].imshow(img_np)\n",
- "axes[0].scatter(joints_2d_crop[:, 0], joints_2d_crop[:, 1], s=15, c='lime', edgecolors='black', label='Native 2D')\n",
- "axes[0].set_title(f\"Target 1: Native 2D Joints\\n(Verifies Augmentations & Crop mapping)\", fontsize=10)\n",
- "axes[0].legend(loc='upper right')\n",
- "axes[0].axis('off')\n",
- "\n",
- "# Plot 2: The 3D -> 2D CLIFF Projection (Checks focal length & cam_trans)\n",
- "axes[1].imshow(img_np)\n",
- "axes[1].scatter(u_crop_proj, v_crop_proj, s=15, c='red', edgecolors='black', marker='X')\n",
- "axes[1].set_title(f\"Target 2: CLIFF Projected 3D MHR\\n(Verifies Full-Frame Camera Math)\", fontsize=10)\n",
- "axes[1].axis('off')\n",
- "\n",
- "plt.tight_layout()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "47dea531",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ==============================================================================\n",
- "# Cell 5 — Data-Driven Topology Discovery (Pixel-Space Edition - FIXED)\n",
- "# ==============================================================================\n",
- "import torch\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "import matplotlib.lines as mlines\n",
- "from tqdm import tqdm\n",
- "\n",
- "print(\"--- Starting 2D Pixel-Space Topology Discovery ---\")\n",
- "\n",
- "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- "mhr_module = MHRForwardPass(cfg.mhr_model_path, device)\n",
- "\n",
- "# --- Configuration ---\n",
- "NUM_BATCHES_TO_SCAN = 100 \n",
- "PIXEL_THRESHOLD = 1.0 # 15 pixels is a great sweet spot for 224x224 anatomical matches\n",
- "\n",
- "cumulative_dist_matrix = torch.zeros((cfg.num_joints, 127), device=device)\n",
- "total_samples = 0\n",
- "\n",
- "print(f\"Scanning {NUM_BATCHES_TO_SCAN} batches to compute average 2D pixel distances...\")\n",
- "train_iter = iter(train_loader)\n",
- "\n",
- "for _ in tqdm(range(NUM_BATCHES_TO_SCAN)):\n",
- " batch = next(train_iter)\n",
- " batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}\n",
- " B = batch[\"mhr_model_params\"].shape[0]\n",
- " \n",
- " # 1. Un-normalize Native 2D to Crop Space [0, 224]\n",
- " native_2d_norm = batch[\"joints_2d\"][..., :2] \n",
- " native_2d_crop = ((native_2d_norm + 1.0) / 2.0) * cfg.image_size \n",
- " \n",
- " # 2. Extract 3D MHR Joints\n",
- " with torch.no_grad():\n",
- " mhr_3d_raw = mhr_module.get_joints(batch[\"mhr_model_params\"], batch[\"shape_params\"])[..., :3]\n",
- " \n",
- " mhr_3d = mhr_3d_raw / 100.0 \n",
- " mhr_3d[..., 1] = -mhr_3d[..., 1] \n",
- " mhr_3d[..., 2] = -mhr_3d[..., 2] \n",
- " \n",
- " # 3. Apply CLIFF Projection Math (Batched properly!)\n",
- " cam_trans = batch[\"cam_trans\"].unsqueeze(1) # [B, 1, 3]\n",
- " j3d_cam = mhr_3d + cam_trans # [B, 127, 3]\n",
- " z = torch.clamp(j3d_cam[..., 2], min=0.01) # [B, 127]\n",
- " \n",
- " # FIXED: Removed the buggy .unsqueeze(1) so shapes match [B, 1] * [B, 127]\n",
- " orig_shape = batch[\"orig_shape\"]\n",
- " orig_w, orig_h = orig_shape[:, 1:2], orig_shape[:, 0:1] # [B, 1]\n",
- " cx, cy = orig_w / 2.0, orig_h / 2.0 # [B, 1]\n",
- " \n",
- " focal_key = \"cam_focal\" if \"cam_focal\" in batch else \"cam_focal_length\"\n",
- " cam_focal = batch[focal_key]\n",
- " fx, fy = cam_focal[:, 0:1], cam_focal[:, 1:2] # [B, 1]\n",
- " \n",
- " u_full = (fx * j3d_cam[..., 0] / z) + cx # [B, 127]\n",
- " v_full = (fy * j3d_cam[..., 1] / z) + cy # [B, 127]\n",
- " \n",
- " bbox = batch[\"bbox_square\"]\n",
- " sq_x1, sq_y1, sq_x2 = bbox[:, 0:1], bbox[:, 1:2], bbox[:, 2:3] # [B, 1]\n",
- " crop_w = torch.clamp(sq_x2 - sq_x1, min=1.0) # [B, 1]\n",
- " true_scale = cfg.image_size / crop_w # [B, 1]\n",
- " \n",
- " u_crop = (u_full - sq_x1) * true_scale # [B, 127]\n",
- " v_crop = (v_full - sq_y1) * true_scale # [B, 127]\n",
- " mhr_2d_crop = torch.stack([u_crop, v_crop], dim=-1) # [B, 127, 2]\n",
- " \n",
- " # 4. Compute Pairwise Distance in Pixel Space!\n",
- " dist_matrix = torch.cdist(native_2d_crop, mhr_2d_crop) # [B, 70, 127]\n",
- " cumulative_dist_matrix += dist_matrix.sum(dim=0)\n",
- " total_samples += B\n",
- "\n",
- "# Average the distance matrix\n",
- "avg_dist_matrix = cumulative_dist_matrix / total_samples\n",
- "\n",
- "# --- Extract the Mapping ---\n",
- "min_dists, closest_mhr_indices = avg_dist_matrix.min(dim=1)\n",
- "\n",
- "valid_mask = min_dists < PIXEL_THRESHOLD\n",
- "native_mapping_ids = torch.arange(70, device=device)[valid_mask].cpu().numpy()\n",
- "mhr_mapping_ids = closest_mhr_indices[valid_mask].cpu().numpy()\n",
- "matched_distances = min_dists[valid_mask].cpu().numpy()\n",
- "\n",
- "native_ids_2d, mhr_ids_2d = native_mapping_ids, mhr_mapping_ids \n",
- "print(\"\\n--- Discovery Complete ---\")\n",
- "print(f\"Found {len(native_mapping_ids)} valid anchor pairs with average distance < {PIXEL_THRESHOLD} pixels.\")\n",
- "\n",
- "# ==============================================================================\n",
- "# VISUALIZATION: Plotting the Anchors on a Sample\n",
- "# ==============================================================================\n",
- "sample_idx = 0\n",
- "nat_2d_viz = native_2d_crop[sample_idx].cpu().numpy()\n",
- "mhr_2d_viz = mhr_2d_crop[sample_idx].cpu().numpy()\n",
- "\n",
- "# Un-normalize image\n",
- "img_tensor = batch[\"image\"][sample_idx].cpu()\n",
- "mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)\n",
- "std = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)\n",
- "img_unnorm = img_tensor * std + mean\n",
- "img_rgb = img_unnorm.permute(1, 2, 0).numpy().clip(0, 1)\n",
- "\n",
- "fig, ax = plt.subplots(figsize=(8, 8))\n",
- "ax.imshow(img_rgb)\n",
- "ax.set_title(f\"Data-Driven Anchor Mapping (Thresh: {PIXEL_THRESHOLD} px)\", fontsize=14)\n",
- "\n",
- "# Plot all joints faintly\n",
- "ax.scatter(nat_2d_viz[:, 0], nat_2d_viz[:, 1], s=20, c='cyan', alpha=0.3, label='All Native')\n",
- "ax.scatter(mhr_2d_viz[:, 0], mhr_2d_viz[:, 1], s=20, c='red', alpha=0.3, label='All MHR')\n",
- "\n",
- "# Draw connections for the valid mapped anchors\n",
- "for n_idx, m_idx in zip(native_mapping_ids, mhr_mapping_ids):\n",
- " ax.scatter(nat_2d_viz[n_idx, 0], nat_2d_viz[n_idx, 1], s=50, c='lime', edgecolors='black', zorder=5)\n",
- " ax.scatter(mhr_2d_viz[m_idx, 0], mhr_2d_viz[m_idx, 1], s=50, c='yellow', marker='X', edgecolors='black', zorder=5)\n",
- " \n",
- " line = mlines.Line2D([nat_2d_viz[n_idx, 0], mhr_2d_viz[m_idx, 0]], \n",
- " [nat_2d_viz[n_idx, 1], mhr_2d_viz[m_idx, 1]], \n",
- " color='magenta', linewidth=1.5, zorder=4)\n",
- " ax.add_line(line)\n",
- "\n",
- "# Clean up legend\n",
- "handles, labels = ax.get_legend_handles_labels()\n",
- "match_marker = mlines.Line2D([], [], color='magenta', marker='o', markersize=8, markerfacecolor='lime', label='Discovered Anchors')\n",
- "ax.legend(handles=[handles[0], handles[1], match_marker], loc='upper right')\n",
- "ax.axis('off')\n",
- "plt.tight_layout()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "af8c0277",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ==============================================================================\n",
- "# Cell 6 — Data-Driven Topology Discovery (3D Meter Space - FIXED ALIGNMENT)\n",
- "# ==============================================================================\n",
- "import torch\n",
- "import numpy as np\n",
- "import matplotlib.pyplot as plt\n",
- "from mpl_toolkits.mplot3d import Axes3D\n",
- "import matplotlib.lines as mlines\n",
- "from tqdm import tqdm\n",
- "\n",
- "print(\"--- Starting 3D Topology Discovery ---\")\n",
- "\n",
- "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- "mhr_module = MHRForwardPass(cfg.mhr_model_path, device)\n",
- "\n",
- "# --- Configuration ---\n",
- "NUM_BATCHES_TO_SCAN = 100 \n",
- "# 4 cm is a rigorous but realistic threshold for overlapping anatomical joints\n",
- "DISTANCE_THRESHOLD_METERS = 0.01 \n",
- "\n",
- "cumulative_dist_matrix = torch.zeros((cfg.num_joints, 127), device=device)\n",
- "total_samples = 0\n",
- "\n",
- "print(f\"Scanning {NUM_BATCHES_TO_SCAN} batches to compute true 3D distances...\")\n",
- "train_iter = iter(train_loader)\n",
- "\n",
- "for _ in tqdm(range(NUM_BATCHES_TO_SCAN)):\n",
- " batch = next(train_iter)\n",
- " batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}\n",
- " B = batch[\"mhr_model_params\"].shape[0]\n",
- " \n",
- " # 1. Extract 3D Native Joints (Body-Centered, Meters, Y-Down, Z-Forward)\n",
- " native_3d = batch[\"joints_3d\"][..., :3] \n",
- " \n",
- " # 2. Extract 3D MHR Joints (Internal coords, cm)\n",
- " with torch.no_grad():\n",
- " mhr_3d_raw = mhr_module.get_joints(batch[\"mhr_model_params\"], batch[\"shape_params\"])[..., :3]\n",
- " \n",
- " # Convert MHR to the exact same frame as Native 3D!\n",
- " mhr_3d = mhr_3d_raw / 100.0 \n",
- " mhr_3d[..., 1] = -mhr_3d[..., 1] \n",
- " mhr_3d[..., 2] = -mhr_3d[..., 2] \n",
- " \n",
- " # NO MEAN CENTERING! They are already in the same coordinate space!\n",
- " \n",
- " # 3. Compute Pairwise Distance in 3D METER Space [B, 70, 127]\n",
- " dist_matrix = torch.cdist(native_3d, mhr_3d) \n",
- " cumulative_dist_matrix += dist_matrix.sum(dim=0)\n",
- " total_samples += B\n",
- "\n",
- "# Average the distance matrix\n",
- "avg_dist_matrix = cumulative_dist_matrix / total_samples\n",
- "\n",
- "# --- Extract the Mapping ---\n",
- "min_dists, closest_mhr_indices = avg_dist_matrix.min(dim=1)\n",
- "\n",
- "valid_mask = min_dists < DISTANCE_THRESHOLD_METERS\n",
- "native_mapping_ids = torch.arange(70, device=device)[valid_mask].cpu().numpy()\n",
- "mhr_mapping_ids = closest_mhr_indices[valid_mask].cpu().numpy()\n",
- "matched_distances = min_dists[valid_mask].cpu().numpy()\n",
- "\n",
- "native_ids_3d, mhr_ids_3d = native_mapping_ids, mhr_mapping_ids \n",
- "print(\"\\n--- Discovery Complete ---\")\n",
- "print(f\"Found {len(native_mapping_ids)} valid anchor pairs with average distance < {DISTANCE_THRESHOLD_METERS} meters.\")\n",
- "\n",
- "# ==============================================================================\n",
- "# VISUALIZATION: Interactive 3D Plot with Plotly\n",
- "# ==============================================================================\n",
- "import plotly.graph_objects as go\n",
- "import numpy as np\n",
- "\n",
- "sample_idx = 0\n",
- "# Extract raw coordinates\n",
- "nat_3d_viz = native_3d[sample_idx].cpu().numpy()\n",
- "mhr_3d_viz = mhr_3d[sample_idx].cpu().numpy()\n",
- "\n",
- "# To make the person stand upright in Plotly (where Z is usually up):\n",
- "# Data X -> Plotly X\n",
- "# Data Z (Forward) -> Plotly Y (Depth)\n",
- "# Data -Y (Down) -> Plotly Z (Height)\n",
- "def map_coords(pts):\n",
- " return pts[:, 0], pts[:, 2], -pts[:, 1]\n",
- "\n",
- "nx, ny, nz = map_coords(nat_3d_viz)\n",
- "mx, my, mz = map_coords(mhr_3d_viz)\n",
- "\n",
- "fig = go.Figure()\n",
- "\n",
- "# 1. Plot Unmatched Native Joints (Cyan)\n",
- "fig.add_trace(go.Scatter3d(\n",
- " x=nx, y=ny, z=nz,\n",
- " mode='markers',\n",
- " marker=dict(size=4, color='cyan', opacity=0.4),\n",
- " name='All Native (70)',\n",
- " text=[f\"Native Joint: {i}\" for i in range(70)],\n",
- " hoverinfo='text'\n",
- "))\n",
- "\n",
- "# 2. Plot Unmatched MHR Joints (Red)\n",
- "fig.add_trace(go.Scatter3d(\n",
- " x=mx, y=my, z=mz,\n",
- " mode='markers',\n",
- " marker=dict(size=4, color='red', opacity=0.4),\n",
- " name='All MHR (127)',\n",
- " text=[f\"MHR Joint: {i}\" for i in range(127)],\n",
- " hoverinfo='text'\n",
- "))\n",
- "\n",
- "# 3. Plot Matched Anchors\n",
- "# Extract coordinates for the matched pairs\n",
- "nat_anchors_x, nat_anchors_y, nat_anchors_z = nx[native_mapping_ids], ny[native_mapping_ids], nz[native_mapping_ids]\n",
- "mhr_anchors_x, mhr_anchors_y, mhr_anchors_z = mx[mhr_mapping_ids], my[mhr_mapping_ids], mz[mhr_mapping_ids]\n",
- "\n",
- "# Native Anchors (Green)\n",
- "fig.add_trace(go.Scatter3d(\n",
- " x=nat_anchors_x, y=nat_anchors_y, z=nat_anchors_z,\n",
- " mode='markers',\n",
- " marker=dict(size=7, color='lime', line=dict(color='black', width=1)),\n",
- " name='Native Anchors',\n",
- " text=[f\"Matched Native: {i}\" for i in native_mapping_ids],\n",
- " hoverinfo='text'\n",
- "))\n",
- "\n",
- "# MHR Anchors (Yellow)\n",
- "fig.add_trace(go.Scatter3d(\n",
- " x=mhr_anchors_x, y=mhr_anchors_y, z=mhr_anchors_z,\n",
- " mode='markers',\n",
- " marker=dict(size=7, color='yellow', symbol='cross', line=dict(color='black', width=1)),\n",
- " name='MHR Anchors',\n",
- " text=[f\"Matched MHR: {i}\" for i in mhr_mapping_ids],\n",
- " hoverinfo='text'\n",
- "))\n",
- "\n",
- "# 4. Draw Connection Lines (Magenta)\n",
- "# To draw multiple disconnected lines efficiently in Plotly, we interleave with None\n",
- "line_x, line_y, line_z = [], [], []\n",
- "for i in range(len(native_mapping_ids)):\n",
- " line_x.extend([nat_anchors_x[i], mhr_anchors_x[i], None])\n",
- " line_y.extend([nat_anchors_y[i], mhr_anchors_y[i], None])\n",
- " line_z.extend([nat_anchors_z[i], mhr_anchors_z[i], None])\n",
- "\n",
- "fig.add_trace(go.Scatter3d(\n",
- " x=line_x, y=line_y, z=line_z,\n",
- " mode='lines',\n",
- " line=dict(color='magenta', width=4),\n",
- " name='Physical Distance',\n",
- " hoverinfo='none'\n",
- "))\n",
- "\n",
- "# 5. Layout and Camera Setup\n",
- "fig.update_layout(\n",
- " title=f\"Interactive 3D Anchor Mapping (Thresh: {DISTANCE_THRESHOLD_METERS*100:.1f} cm)\",\n",
- " scene=dict(\n",
- " xaxis_title='X (Meters)',\n",
- " yaxis_title='Z Depth (Meters)',\n",
- " zaxis_title='Y Height (Meters)',\n",
- " aspectmode='data' # THIS IS CRITICAL: Keeps the human proportions physically accurate!\n",
- " ),\n",
- " margin=dict(l=0, r=0, b=0, t=40),\n",
- " legend=dict(x=0.8, y=0.9)\n",
- ")\n",
- "\n",
- "fig.show()\n",
- "\n",
- "print(\"--- Anchor Verification: 2D vs 3D ---\")\n",
- "\n",
- "# 1. Create sets of tuples for easy comparison\n",
- "mapping_2d = set(zip(native_ids_2d, mhr_ids_2d))\n",
- "mapping_3d = set(zip(native_ids_3d, mhr_ids_3d))\n",
- "\n",
- "# 2. Check for perfect agreement\n",
- "if mapping_2d == mapping_3d:\n",
- " print(\"✅ PERFECT MATCH! Both 2D and 3D discovery found the exact same 52 anchor pairs.\")\n",
- "else:\n",
- " print(\"⚠️ MISMATCH DETECTED. The lists differ slightly.\")\n",
- " \n",
- " # Find what's unique to 2D\n",
- " only_in_2d = mapping_2d - mapping_3d\n",
- " if only_in_2d:\n",
- " print(f\"\\nPairs found ONLY in 2D ({len(only_in_2d)}):\")\n",
- " for nat, mhr in sorted(only_in_2d):\n",
- " print(f\" Native [{nat:02d}] <--> MHR [{mhr:03d}]\")\n",
- " \n",
- " # Find what's unique to 3D\n",
- " only_in_3d = mapping_3d - mapping_2d\n",
- " if only_in_3d:\n",
- " print(f\"\\nPairs found ONLY in 3D ({len(only_in_3d)}):\")\n",
- " for nat, mhr in sorted(only_in_3d):\n",
- " print(f\" Native [{nat:02d}] <--> MHR [{mhr:03d}]\")\n",
- "\n",
- "# 3. Print the final agreed-upon list to copy into your config!\n",
- "agreed_mapping = sorted(list(mapping_2d.intersection(mapping_3d)))\n",
- "\n",
- "print(\"\\n--- Final Verified Anchor Dictionary ---\")\n",
- "print(\"Copy this into your DistillConfig:\")\n",
- "print(\"native_mapping_ids = [\", \", \".join(str(pair[0]) for pair in agreed_mapping), \"]\")\n",
- "print(\"mhr_mapping_ids = [\", \", \".join(str(pair[1]) for pair in agreed_mapping), \"]\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "10f2586d",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 7 — Student Architecture (Mini-SAM3D with PE)\n",
- "# ============================================================\n",
- "import torch\n",
- "import torch.nn as nn\n",
- "import timm\n",
- "\n",
- "def get_2d_sincos_pos_embed(embed_dim, grid_size):\n",
- " \"\"\"Generate 2D sine-cosine positional embedding for preserving spatial topology.\"\"\"\n",
- " grid_h, grid_w = grid_size, grid_size\n",
- " grid_y, grid_x = torch.meshgrid(torch.arange(grid_h), torch.arange(grid_w), indexing='ij')\n",
- " \n",
- " omega = torch.arange(embed_dim // 4).float() / (embed_dim // 4)\n",
- " omega = 1.0 / (10000 ** omega)\n",
- " \n",
- " out_y = grid_y.flatten().unsqueeze(1) * omega.unsqueeze(0)\n",
- " out_x = grid_x.flatten().unsqueeze(1) * omega.unsqueeze(0)\n",
- " \n",
- " pe_y = torch.cat([torch.sin(out_y), torch.cos(out_y)], dim=1)\n",
- " pe_x = torch.cat([torch.sin(out_x), torch.cos(out_x)], dim=1)\n",
- " \n",
- " pos_embed = torch.cat([pe_y, pe_x], dim=1) # (H*W, embed_dim)\n",
- " return pos_embed.unsqueeze(0) # (1, H*W, embed_dim)\n",
- "\n",
- "\n",
- "class InstantHMRStudent(nn.Module):\n",
- " def __init__(self, cfg, pretrained=True):\n",
- " super().__init__()\n",
- " self.cfg = cfg\n",
- " \n",
- " # 1. The Vision Backbone (RepViT)\n",
- " self.backbone = timm.create_model(cfg.backbone, pretrained=pretrained, num_classes=0)\n",
- " embed_dim = self.backbone.num_features \n",
- " self.feat_proj = nn.Linear(embed_dim, cfg.d_model)\n",
- " \n",
- " # --- NEW: Positional Embedding for Transformer Memory ---\n",
- " # RepViT naturally downsamples by 32x. For a 224x224 image, the grid is 7x7.\n",
- " self.grid_size = cfg.image_size // 32\n",
- " pos_embed = get_2d_sincos_pos_embed(cfg.d_model, self.grid_size)\n",
- " # register_buffer ensures it moves to GPU automatically but isn't updated by the optimizer\n",
- " self.register_buffer(\"mem_pos_embed\", pos_embed) \n",
- " \n",
- " # 2. Learnable Query Tokens (1 Global + 70 2D + 70 3D = 141 Queries)\n",
- " self.num_global = 1\n",
- " self.num_2d = cfg.num_joints\n",
- " self.num_3d = cfg.num_joints\n",
- " self.total_queries = self.num_global + self.num_2d + self.num_3d\n",
- " \n",
- " # self.query_embed = nn.Parameter(torch.randn(1, self.total_queries, cfg.d_model))\n",
- " self.query_embed = nn.Parameter(torch.randn(1, self.total_queries, cfg.d_model) * 0.02)\n",
- " \n",
- " # CLIFF Conditioning Injector\n",
- " self.cond_proj = nn.Sequential(\n",
- " nn.Linear(cfg.cliff_dim, 128),\n",
- " nn.GELU(),\n",
- " nn.Linear(128, cfg.d_model)\n",
- " )\n",
- " \n",
- " # 3. Transformer Cross-Attention Decoder\n",
- " decoder_layer = nn.TransformerDecoderLayer(\n",
- " d_model=cfg.d_model,\n",
- " nhead=cfg.n_heads,\n",
- " dim_feedforward=cfg.d_model * 4,\n",
- " dropout=cfg.dropout,\n",
- " activation=\"gelu\",\n",
- " batch_first=True,\n",
- " norm_first=True\n",
- " )\n",
- " self.transformer = nn.TransformerDecoder(decoder_layer, num_layers=cfg.n_decoder_layers)\n",
- " \n",
- " # 4. The Output Heads\n",
- " \n",
- " # Head A: Global Head (Token 0 -> MHR Params + Shape Params + Camera)\n",
- " self.global_out_dim = cfg.model_params_dim + cfg.shape_dim + cfg.cam_dim\n",
- " self.head_global = nn.Linear(cfg.d_model, self.global_out_dim)\n",
- " \n",
- " # --- Safely Split 2D Head ---\n",
- " self.head_2d_feat = nn.Sequential(\n",
- " nn.Linear(cfg.d_model, 256),\n",
- " nn.GELU()\n",
- " )\n",
- " self.head_2d_xy = nn.Sequential(\n",
- " nn.Linear(256, 2)\n",
- " # nn.Tanh() # Bounds predictions to [-1, 1]!\n",
- " )\n",
- " if getattr(cfg, 'learn_uncertainty', False):\n",
- " # Uncertainty must remain unbounded!\n",
- " self.head_2d_unc = nn.Linear(256, 1) \n",
- " \n",
- " # Head C: 3D Keypoint Head (No Tanh here, 3D meters can exceed [-1, 1])\n",
- " self.dim_3d = 4 if getattr(cfg, 'learn_uncertainty', False) else 3\n",
- " self.head_3d = nn.Sequential(\n",
- " nn.Linear(cfg.d_model, 256),\n",
- " nn.GELU(),\n",
- " nn.Linear(256, self.dim_3d)\n",
- " )\n",
- " \n",
- " # Initialize camera Z translation\n",
- " nn.init.constant_(self.head_global.bias[-1], 2.0)\n",
- " nn.init.constant_(self.head_global.bias[-2], 0.0)\n",
- " nn.init.constant_(self.head_global.bias[-3], 0.0)\n",
- "\n",
- " # ==========================================================\n",
- " # HMR-Standard Zero-Initialization for Stability\n",
- " # ==========================================================\n",
- " # 1. Global Head (MHR Params) - Must start near neutral pose!\n",
- " nn.init.normal_(self.head_global.weight, mean=0.0, std=1e-4)\n",
- " # Zero out the biases (except the last 3 camera params we just set)\n",
- " nn.init.constant_(self.head_global.bias[:-3], 0.0)\n",
- " \n",
- " # 2. 2D Spatial Head - Start with tiny weights\n",
- " nn.init.normal_(self.head_2d_xy[0].weight, mean=0.0, std=1e-4)\n",
- " nn.init.constant_(self.head_2d_xy[0].bias, 0.0)\n",
- " \n",
- " # 3. 3D Spatial Head - Start with tiny weights\n",
- " nn.init.normal_(self.head_3d[-1].weight, mean=0.0, std=1e-4)\n",
- " nn.init.constant_(self.head_3d[-1].bias, 0.0)\n",
- "\n",
- " def forward(self, images, cliff_cond):\n",
- " B = images.shape[0]\n",
- " \n",
- " # 1. Extract spatial image features\n",
- " img_feats = self.backbone.forward_features(images)\n",
- " \n",
- " # Defensive flattening \n",
- " if img_feats.dim() == 4:\n",
- " img_feats = img_feats.flatten(2).transpose(1, 2) # [B, H*W, C]\n",
- " \n",
- " memory = self.feat_proj(img_feats) \n",
- " \n",
- " # --- NEW: Inject Spatial Topology into Memory ---\n",
- " # Broadcasting [1, H*W, d_model] across Batch size\n",
- " memory = memory + self.mem_pos_embed \n",
- " \n",
- " # 2. Prepare Query Tokens [B, 141, d_model]\n",
- " queries = self.query_embed.expand(B, -1, -1)\n",
- " \n",
- " cond = self.cond_proj(cliff_cond).unsqueeze(1)\n",
- " queries = queries + cond\n",
- " \n",
- " # 3. Cross Attention\n",
- " shared_feats = self.transformer(tgt=queries, memory=memory)\n",
- " \n",
- " # 4. Route tokens to their dedicated heads\n",
- " feat_global = shared_feats[:, 0, :] \n",
- " feat_2d = shared_feats[:, 1 : 1 + self.num_2d, :] \n",
- " feat_3d = shared_feats[:, 1 + self.num_2d :, :] \n",
- " \n",
- " # --- Decode Global Token ---\n",
- " global_preds = self.head_global(feat_global) \n",
- " \n",
- " idx_mhr = self.cfg.model_params_dim\n",
- " idx_shape = idx_mhr + self.cfg.shape_dim\n",
- " \n",
- " pred_mhr_params = global_preds[:, :idx_mhr]\n",
- " pred_shape_params = global_preds[:, idx_mhr : idx_shape]\n",
- " pred_cam_trans = global_preds[:, idx_shape :]\n",
- " \n",
- " # --- Decode 2D Spatial Tokens (Safely!) ---\n",
- " feat_2d_processed = self.head_2d_feat(feat_2d)\n",
- " pred_xy = self.head_2d_xy(feat_2d_processed) # [B, 70, 2] bounded by Tanh\n",
- " \n",
- " if getattr(self.cfg, 'learn_uncertainty', False):\n",
- " pred_unc = self.head_2d_unc(feat_2d_processed) # [B, 70, 1] unbounded\n",
- " pred_joints_2d = torch.cat([pred_xy, pred_unc], dim=-1) # [B, 70, 3]\n",
- " else:\n",
- " pred_joints_2d = pred_xy\n",
- " \n",
- " # --- Decode 3D Spatial Tokens ---\n",
- " pred_joints_3d = self.head_3d(feat_3d) \n",
- " \n",
- " return {\n",
- " \"mhr_params\": pred_mhr_params,\n",
- " \"shape_params\": pred_shape_params,\n",
- " \"cam_trans\": pred_cam_trans,\n",
- " \"joints_2d\": pred_joints_2d,\n",
- " \"joints_3d\": pred_joints_3d\n",
- " }\n",
- "\n",
- "# --- Quick Architecture Sanity Check ---\n",
- "if __name__ == \"__main__\":\n",
- " test_model = InstantHMRStudent(cfg, pretrained=False)\n",
- " dummy_img = torch.randn(2, 3, 224, 224)\n",
- " dummy_cliff = torch.randn(2, 3)\n",
- " \n",
- " out = test_model(dummy_img, dummy_cliff)\n",
- " print(\"--- Architecture Output Shapes ---\")\n",
- " for k, v in out.items():\n",
- " print(f\"{k}: {v.shape}\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "9239175a",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 8 — The Distillation Loss Module (Foolproof Edition)\n",
- "# ============================================================\n",
- "import torch\n",
- "import torch.nn as nn\n",
- "\n",
- "# --- Modified Forward Pass (Differentiable) ---\n",
- "class MHRForwardPass:\n",
- " def __init__(self, mhr_path, device):\n",
- " self.device = device\n",
- " self.mhr = torch.jit.load(mhr_path, map_location=device).eval()\n",
- " for p in self.mhr.parameters():\n",
- " p.requires_grad_(False)\n",
- "\n",
- " def get_joints(self, model_params, shape_params):\n",
- " B = model_params.shape[0]\n",
- " dummy_identity = torch.zeros(B, 45, device=model_params.device, dtype=model_params.dtype)\n",
- " concat_params = torch.cat([model_params, dummy_identity], dim=1)\n",
- " \n",
- " joint_params = self.mhr.character_torch.model_parameters_to_joint_parameters(concat_params.to(self.device))\n",
- " skel_state = self.mhr.character_torch.joint_parameters_to_skeleton_state(joint_params)\n",
- " \n",
- " return skel_state.to(model_params.device)\n",
- "\n",
- "class DistillationLoss(nn.Module):\n",
- " def __init__(self, cfg, mhr_module):\n",
- " super().__init__()\n",
- " self.cfg = cfg\n",
- " self.mhr_module = mhr_module \n",
- " self.mse = nn.MSELoss()\n",
- " self.l1 = nn.SmoothL1Loss()\n",
- " self.strict_l1 = nn.L1Loss()\n",
- "\n",
- " def _project_3d_to_norm_2d(self, j3d, pred_cam_trans, targets):\n",
- " \"\"\"Helper to project 3D points (in meters) to normalized [-1, 1] 2D crop space.\"\"\"\n",
- " # A. Move to Camera Space\n",
- " j3d_cam = j3d + pred_cam_trans.unsqueeze(1)\n",
- " \n",
- " # Prevent 1/Z^2 Gradient Explosions\n",
- " # 0.01 allows a gradient multiplier of 10,000. \n",
- " # 0.4 meters (40cm) limits the multiplier to a safe limit, protecting the kinematic tree.\n",
- " z_cam = torch.clamp(j3d_cam[..., 2], min=0.4)\n",
- " \n",
- " # B. Extract Intrinsics\n",
- " fx = targets[\"cam_focal\"][:, 0:1] # Shape: [B, 1]\n",
- " fy = targets[\"cam_focal\"][:, 1:2] # Shape: [B, 1]\n",
- " cx = targets[\"orig_shape\"][:, 1:2] / 2.0 # Shape: [B, 1]\n",
- " cy = targets[\"orig_shape\"][:, 0:1] / 2.0 # Shape: [B, 1]\n",
- " \n",
- " # C. Full-Frame Projection\n",
- " u_full = (fx * j3d_cam[..., 0] / z_cam) + cx\n",
- " v_full = (fy * j3d_cam[..., 1] / z_cam) + cy\n",
- " \n",
- " # D. Map to Crop Space\n",
- " bbox = targets[\"bbox_square\"]\n",
- " sq_x1 = bbox[:, 0:1] # [B, 1]\n",
- " sq_y1 = bbox[:, 1:2] # [B, 1]\n",
- " crop_w = torch.clamp(bbox[:, 2:3] - bbox[:, 0:1], min=1.0) # [B, 1]\n",
- " \n",
- " true_scale = self.cfg.image_size / crop_w\n",
- " u_crop = (u_full - sq_x1) * true_scale\n",
- " v_crop = (v_full - sq_y1) * true_scale\n",
- " \n",
- " # E. Normalize to [-1, 1]\n",
- " u_norm = (u_crop / self.cfg.image_size) * 2.0 - 1.0\n",
- " v_norm = (v_crop / self.cfg.image_size) * 2.0 - 1.0\n",
- " \n",
- " return torch.stack([u_norm, v_norm], dim=-1) # Shape: [B, J, 2]\n",
- "\n",
- " def forward(self, preds, targets):\n",
- " losses = {}\n",
- " idx_pose = self.cfg.pose_dim\n",
- " \n",
- " pred_pose = preds[\"mhr_params\"][:, :idx_pose]\n",
- " pred_scale = preds[\"mhr_params\"][:, idx_pose:]\n",
- " tgt_pose = targets[\"mhr_model_params\"][:, :idx_pose]\n",
- " tgt_scale = targets[\"mhr_model_params\"][:, idx_pose:]\n",
- " \n",
- " # --- 1. Base Parameters ---\n",
- " losses['loss_pose'] = self.l1(pred_pose, tgt_pose) * self.cfg.w_pose\n",
- " losses['loss_scale'] = self.mse(pred_scale, tgt_scale) * self.cfg.w_scale\n",
- " losses['loss_shape'] = self.mse(preds[\"shape_params\"], targets[\"shape_params\"]) * self.cfg.w_shape\n",
- " losses['loss_cam'] = self.mse(preds[\"cam_trans\"], targets[\"cam_trans\"]) * self.cfg.w_cam\n",
- " \n",
- " # --- 2. Differentiable 3D Joint Alignment (Centimeter space corrected!) ---\n",
- " pred_mhr_joints = self.mhr_module.get_joints(preds[\"mhr_params\"], preds[\"shape_params\"])[..., :3]\n",
- " with torch.no_grad():\n",
- " tgt_mhr_joints = self.mhr_module.get_joints(targets[\"mhr_model_params\"], targets[\"shape_params\"])[..., :3]\n",
- " \n",
- " losses['loss_mhr_joints'] = self.l1(pred_mhr_joints, tgt_mhr_joints.detach()) * self.cfg.w_3d_joints\n",
- "\n",
- " # --- Base MHR Vision Coordinates (Needed for Structural & Reprojection) ---\n",
- " pred_mhr_vision = pred_mhr_joints.clone() / 100.0\n",
- " pred_mhr_vision[..., 1] = -pred_mhr_vision[..., 1]\n",
- " pred_mhr_vision[..., 2] = -pred_mhr_vision[..., 2]\n",
- "\n",
- " # ==============================================================================\n",
- " # 2.5 Structural Consistency Loss (MHR mesh vs Native Keypoints)\n",
- " # ==============================================================================\n",
- " if hasattr(self.cfg, 'native_mapping_ids') and hasattr(self.cfg, 'mhr_mapping_ids'):\n",
- " pred_native_anchors = preds[\"joints_3d\"][:, self.cfg.native_mapping_ids, :3]\n",
- " pred_mhr_anchors = pred_mhr_vision[:, self.cfg.mhr_mapping_ids, :3]\n",
- " losses['loss_structural'] = self.l1(pred_native_anchors, pred_mhr_anchors) * self.cfg.w_structural\n",
- "\n",
- " # ==============================================================================\n",
- " # Feet Barycenter Alignment Loss\n",
- " # ==============================================================================\n",
- " if hasattr(self.cfg, 'native_left_foot') and hasattr(self.cfg, 'w_feet'):\n",
- " nat_lf_pts = preds[\"joints_3d\"][:, self.cfg.native_left_foot, :3]\n",
- " nat_rf_pts = preds[\"joints_3d\"][:, self.cfg.native_right_foot, :3]\n",
- " \n",
- " mhr_lf_pts = pred_mhr_vision[:, self.cfg.mhr_left_foot, :3]\n",
- " mhr_rf_pts = pred_mhr_vision[:, self.cfg.mhr_right_foot, :3]\n",
- " \n",
- " nat_lf_bary = nat_lf_pts.mean(dim=1)\n",
- " nat_rf_bary = nat_rf_pts.mean(dim=1)\n",
- " mhr_lf_bary = mhr_lf_pts.mean(dim=1)\n",
- " mhr_rf_bary = mhr_rf_pts.mean(dim=1)\n",
- " \n",
- " losses['loss_feet'] = (self.strict_l1(nat_lf_bary, mhr_lf_bary) + self.strict_l1(nat_rf_bary, mhr_rf_bary)) * self.cfg.w_feet\n",
- "\n",
- " # --- 3. DYNAMIC Native Keypoints ---\n",
- " pred_2d = preds[\"joints_2d\"]\n",
- " tgt_2d = targets[\"joints_2d\"]\n",
- " losses['loss_2d_native'] = self.l1(pred_2d[..., :2], tgt_2d) * self.cfg.w_keypoints2d\n",
- " \n",
- " pred_3d = preds[\"joints_3d\"]\n",
- " tgt_3d = targets[\"joints_3d\"]\n",
- " losses['loss_3d_native'] = self.l1(pred_3d[..., :3], tgt_3d) * self.cfg.w_keypoints3d\n",
- "\n",
- " # ==============================================================================\n",
- " # 4. 2D Reprojection Loss (Camera-Pose Consistency)\n",
- " # ==============================================================================\n",
- " # A. Native 3D Head Reprojection (All 70 Joints)\n",
- " pred_reproj_native = self._project_3d_to_norm_2d(preds[\"joints_3d\"][..., :3], preds[\"cam_trans\"], targets)\n",
- " losses['loss_reproj_native'] = self.strict_l1(pred_reproj_native, tgt_2d) * self.cfg.w_reproj_3d\n",
- "\n",
- " # B. MHR Mesh Anchors Reprojection (52 Anchors)\n",
- " if hasattr(self.cfg, 'native_mapping_ids') and hasattr(self.cfg, 'mhr_mapping_ids'):\n",
- " # Only project the 52 matched anchors from the MHR mesh\n",
- " pred_mhr_anchors_vision = pred_mhr_vision[:, self.cfg.mhr_mapping_ids, :3]\n",
- " pred_reproj_mhr = self._project_3d_to_norm_2d(pred_mhr_anchors_vision, preds[\"cam_trans\"], targets) #detach the MHR anchors to prevent gradients from flowing back through the projection during this loss calculation\n",
- " # pred_reproj_mhr = self._project_3d_to_norm_2d(pred_mhr_anchors_vision, preds[\"cam_trans\"], targets)\n",
- " \n",
- " # Compare against the corresponding 52 ground truth 2D targets\n",
- " tgt_2d_anchors = tgt_2d[:, self.cfg.native_mapping_ids, :]\n",
- " losses['loss_reproj_mhr'] = self.strict_l1(pred_reproj_mhr, tgt_2d_anchors) * self.cfg.w_reproj_mhr\n",
- " \n",
- " losses['total_loss'] = sum(losses.values())\n",
- " return losses\n",
- "\n",
- "# ==============================================================================\n",
- "# SANITY CHECK: The Perfect Student\n",
- "# ==============================================================================\n",
- "if __name__ == \"__main__\":\n",
- " print(\"--- Running Loss Function Sanity Check ---\")\n",
- " device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- " mhr_module = MHRForwardPass(cfg.mhr_model_path, device)\n",
- " criterion = DistillationLoss(cfg, mhr_module)\n",
- " \n",
- " sample_batch_cpu = next(iter(train_loader))\n",
- " sample_batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in sample_batch_cpu.items()}\n",
- " \n",
- " mock_preds = {\n",
- " \"mhr_params\": sample_batch[\"mhr_model_params\"].clone(),\n",
- " \"shape_params\": sample_batch[\"shape_params\"].clone(),\n",
- " \"cam_trans\": sample_batch[\"cam_trans\"].clone(),\n",
- " \"joints_2d\": sample_batch[\"joints_2d\"].clone(),\n",
- " \"joints_3d\": sample_batch[\"joints_3d\"].clone()\n",
- " }\n",
- " \n",
- " losses = criterion(mock_preds, sample_batch)\n",
- " \n",
- " print(\"Losses for the Perfect Student:\")\n",
- " for k, v in losses.items():\n",
- " print(f\" {k:<20}: {v.item():.6f}\")\n",
- " \n",
- " if abs(losses['total_loss'].item()) < 1e-4:\n",
- " print(\"\\n✅ SUCCESS: The Perfect Student achieved a loss of 0! Math is fully verified.\")\n",
- " else:\n",
- " print(\"\\n❌ ERROR: Loss is not zero. Check the math.\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "ba2129c1",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 9 — The 1-Batch Overfit Test (VRAM Optimized)\n",
- "# ============================================================\n",
- "import torch\n",
- "import torch.optim as optim\n",
- "import matplotlib.pyplot as plt\n",
- "import gc\n",
- "\n",
- "print(\"--- Starting 1-Batch Overfit Test ---\")\n",
- "\n",
- "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
- "\n",
- "# 1. Flush Notebook Ghost Memory\n",
- "gc.collect()\n",
- "torch.cuda.empty_cache()\n",
- "\n",
- "# 2. Instantiate Model, Loss, and Optimizer\n",
- "overfit_model = InstantHMRStudent(cfg, pretrained=True).to(device)\n",
- "criterion = DistillationLoss(cfg, mhr_module)\n",
- "\n",
- "optimizer = optim.AdamW(overfit_model.parameters(), lr=1e-3, weight_decay=0.0)\n",
- "\n",
- "# Setup Automatic Mixed Precision (AMP) Scaler to save VRAM\n",
- "scaler = torch.amp.GradScaler(device='cuda', enabled=cfg.use_amp)\n",
- "\n",
- "# 3. Grab exactly ONE batch and SLICE it to fit in VRAM\n",
- "overfit_batch_cpu = next(iter(train_loader))\n",
- "\n",
- "# We only need 8 images to prove the network can learn!\n",
- "def slice_and_move(batch_dict, dev, subset_size=8):\n",
- " return {k: v[:subset_size].to(dev) if isinstance(v, torch.Tensor) else v for k, v in batch_dict.items()}\n",
- "\n",
- "overfit_batch = slice_and_move(overfit_batch_cpu, device, subset_size=8)\n",
- "\n",
- "# 4. Training Loop (150 epochs on the exact same 8 images)\n",
- "epochs = 1000\n",
- "loss_history = []\n",
- "\n",
- "overfit_model.train()\n",
- "for epoch in range(epochs):\n",
- " optimizer.zero_grad(set_to_none=True) # slightly more memory efficient than zero_grad()\n",
- " \n",
- " # Forward pass with Mixed Precision\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " images = overfit_batch[\"image\"]\n",
- " cliff_cond = overfit_batch[\"cliff_cond\"]\n",
- " \n",
- " preds = overfit_model(images, cliff_cond)\n",
- " \n",
- " # Compute loss\n",
- " losses = criterion(preds, overfit_batch)\n",
- " total_loss = losses['total_loss']\n",
- " \n",
- " # Backward pass with scaled gradients\n",
- " scaler.scale(total_loss).backward()\n",
- " \n",
- " # Unscale before gradient clipping\n",
- " scaler.unscale_(optimizer)\n",
- " torch.nn.utils.clip_grad_norm_(overfit_model.parameters(), cfg.grad_clip)\n",
- " \n",
- " # Optimizer step\n",
- " scaler.step(optimizer)\n",
- " scaler.update()\n",
- " \n",
- " loss_history.append(total_loss.item())\n",
- " \n",
- " # Print progress every 10 epochs\n",
- " if epoch % 10 == 0 or epoch == epochs - 1:\n",
- " print(f\"Epoch {epoch:03d} | Total Loss: {total_loss.item():.4f} | \"\n",
- " f\"2D Loss: {losses['loss_2d_native'].item():.4f} | \"\n",
- " f\"3D Loss: {losses['loss_3d_native'].item():.4f} | \"\n",
- " f\"Loss pose: {losses['loss_pose'].item():.4f} | \"\n",
- " f\"Scale Loss: {losses['loss_scale'].item():.4f} | \"\n",
- " f\"Shape Loss: {losses['loss_shape'].item():.4f} | \"\n",
- " f\"Cam Loss: {losses['loss_cam'].item():.4f}\"\n",
- " f\"Mhr joints Loss: {losses['loss_mhr_joints'].item():.4f}\",\n",
- " f\"Structural Loss: {losses['loss_structural'].item():.4f}\")\n",
- "\n",
- "# 5. Plot the results\n",
- "plt.figure(figsize=(10, 5))\n",
- "plt.plot(loss_history, label='Total Loss', color='blue', linewidth=2)\n",
- "plt.title('1-Batch Overfit Curve (8 Images)', fontsize=14)\n",
- "plt.xlabel('Epoch', fontsize=12)\n",
- "plt.ylabel('Total Loss', fontsize=12)\n",
- "plt.yscale('log') # Log scale helps visualize the exponential decay\n",
- "plt.grid(True, which=\"both\", ls=\"--\", alpha=0.5)\n",
- "plt.legend()\n",
- "plt.tight_layout()\n",
- "plt.show()\n",
- "\n",
- "if loss_history[-1] < loss_history[0] * 0.05:\n",
- " print(\"\\n✅ SUCCESS: The model successfully overfit the batch! Gradients are flowing.\")\n",
- "else:\n",
- " print(\"\\n❌ WARNING: The loss did not decrease significantly. Check for detached tensors or learning rate issues.\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "27616862",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 10 — HMR Evaluation Metrics (MPJPE & PA-MPJPE)\n",
- "# ============================================================\n",
- "import torch\n",
- "import numpy as np\n",
- "\n",
- "def batched_procrustes_alignment(pred_pts, gt_pts):\n",
- " \"\"\"\n",
- " Computes the optimal Similarity Transform (Scale, Rotation, Translation) \n",
- " to align pred_pts to gt_pts using Batched SVD.\n",
- " \"\"\"\n",
- " B, N, D = pred_pts.shape\n",
- " \n",
- " # 1. Mean center the point clouds\n",
- " mu_pred = pred_pts.mean(dim=1, keepdim=True)\n",
- " mu_gt = gt_pts.mean(dim=1, keepdim=True)\n",
- " \n",
- " pred_centered = pred_pts - mu_pred\n",
- " gt_centered = gt_pts - mu_gt\n",
- " \n",
- " # 2. Calculate scale (Frobenius norm)\n",
- " norm_pred = torch.linalg.norm(pred_centered, dim=(1,2), keepdim=True)\n",
- " norm_gt = torch.linalg.norm(gt_centered, dim=(1,2), keepdim=True)\n",
- " \n",
- " pred_normalized = pred_centered / torch.clamp(norm_pred, min=1e-8)\n",
- " gt_normalized = gt_centered / torch.clamp(norm_gt, min=1e-8)\n",
- " \n",
- " # 3. Compute covariance matrix H (X^T Y)\n",
- " H = torch.bmm(pred_normalized.transpose(1, 2), gt_normalized)\n",
- " \n",
- " # 4. Singular Value Decomposition (SVD)\n",
- " # PyTorch returns U, S, and Vh (where Vh is V^T)\n",
- " U, S, Vh = torch.linalg.svd(H)\n",
- " \n",
- " # 5. Compute Rotation matrix R (U * V^T) - FIXED!\n",
- " R = torch.bmm(U, Vh)\n",
- " \n",
- " # 6. Fix reflection (Strict SO(3) rotation)\n",
- " det = torch.linalg.det(R)\n",
- " \n",
- " # If det < 0, we must flip the sign of the 3rd column of U\n",
- " # Shape of det_sign is [B, 1]\n",
- " det_sign = torch.where(det < 0, torch.tensor(-1.0, device=pred_pts.device), torch.tensor(1.0, device=pred_pts.device)).unsqueeze(-1)\n",
- " \n",
- " U_fixed = U.clone()\n",
- " U_fixed[:, :, 2] *= det_sign\n",
- " \n",
- " # Recompute R with the reflection fix\n",
- " R_fixed = torch.bmm(U_fixed, Vh)\n",
- " \n",
- " # 7. Optimal Scale\n",
- " # We must apply the same sign fix to the 3rd singular value!\n",
- " S_fixed = S.clone()\n",
- " S_fixed[:, 2] *= det_sign.squeeze(-1)\n",
- " \n",
- " scale = S_fixed.sum(dim=-1, keepdim=True).unsqueeze(-1) * (norm_gt / torch.clamp(norm_pred, min=1e-8))\n",
- " \n",
- " # 8. Apply the transformation\n",
- " pred_aligned = scale * torch.bmm(pred_centered, R_fixed) + mu_gt\n",
- " \n",
- " return pred_aligned\n",
- "\n",
- "@torch.no_grad()\n",
- "def evaluate_hmr_batch(preds, targets, cfg, mhr_module):\n",
- " \"\"\"\n",
- " Calculates MPJPE and PA-MPJPE for both the 3D Keypoints Head and the MHR Mesh.\n",
- " Returns errors in MILLIMETERS (mm).\n",
- " \"\"\"\n",
- " metrics = {}\n",
- " \n",
- " # ---------------------------------------------------------\n",
- " # 1. Evaluate the Native 3D Spatial Head (70 Joints)\n",
- " # ---------------------------------------------------------\n",
- " pred_3d_native = preds[\"joints_3d\"][..., :3]\n",
- " gt_3d_native = targets[\"joints_3d\"][..., :3]\n",
- " \n",
- " # MPJPE requires root-relative alignment (usually joint 0 is the root/pelvis)\n",
- " pred_3d_root_rel = pred_3d_native - pred_3d_native[:, 0:1, :]\n",
- " gt_3d_root_rel = gt_3d_native - gt_3d_native[:, 0:1, :]\n",
- " \n",
- " # MPJPE in mm (* 1000)\n",
- " err_native = torch.linalg.norm(pred_3d_root_rel - gt_3d_root_rel, dim=-1)\n",
- " metrics['Head_3D_MPJPE'] = err_native.mean().item() * 1000.0\n",
- " \n",
- " # PA-MPJPE in mm\n",
- " pred_3d_aligned = batched_procrustes_alignment(pred_3d_native, gt_3d_native)\n",
- " err_native_pa = torch.linalg.norm(pred_3d_aligned - gt_3d_native, dim=-1)\n",
- " metrics['Head_3D_PA_MPJPE'] = err_native_pa.mean().item() * 1000.0\n",
- "\n",
- " # ---------------------------------------------------------\n",
- " # 2. Evaluate the MHR Mesh Head (Using the 52 Anchors)\n",
- " # ---------------------------------------------------------\n",
- " if hasattr(cfg, 'native_mapping_ids') and hasattr(cfg, 'mhr_mapping_ids'):\n",
- " # Generate 3D mesh joints in cm\n",
- " pred_mhr_raw = mhr_module.get_joints(preds[\"mhr_params\"], preds[\"shape_params\"])[..., :3]\n",
- " \n",
- " # Convert to Meters and align axes\n",
- " pred_mhr_meters = pred_mhr_raw / 100.0\n",
- " pred_mhr_meters[..., 1] = -pred_mhr_meters[..., 1]\n",
- " pred_mhr_meters[..., 2] = -pred_mhr_meters[..., 2]\n",
- " \n",
- " # Extract the 52 matching anchors\n",
- " pred_mesh_anchors = pred_mhr_meters[:, cfg.mhr_mapping_ids, :]\n",
- " gt_mesh_anchors = gt_3d_native[:, cfg.native_mapping_ids, :]\n",
- " \n",
- " # Mean-Center for MPJPE (To bypass differing origin definitions between rigs)\n",
- " pred_mesh_centered = pred_mesh_anchors - pred_mesh_anchors.mean(dim=1, keepdim=True)\n",
- " gt_mesh_centered = gt_mesh_anchors - gt_mesh_anchors.mean(dim=1, keepdim=True)\n",
- " \n",
- " # Mesh MPJPE in mm\n",
- " err_mesh = torch.linalg.norm(pred_mesh_centered - gt_mesh_centered, dim=-1)\n",
- " metrics['Mesh_MPJPE'] = err_mesh.mean().item() * 1000.0\n",
- " \n",
- " # Mesh PA-MPJPE in mm (Procrustes naturally handles translation offsets!)\n",
- " pred_mesh_aligned = batched_procrustes_alignment(pred_mesh_anchors, gt_mesh_anchors)\n",
- " err_mesh_pa = torch.linalg.norm(pred_mesh_aligned - gt_mesh_anchors, dim=-1)\n",
- " metrics['Mesh_PA_MPJPE'] = err_mesh_pa.mean().item() * 1000.0\n",
- " \n",
- " return metrics\n",
- "\n",
- "import gc, torch\n",
- "for _n in [\"overfit_model\", \"optimizer\", \"scaler\", \"overfit_batch\", \"overfit_batch_cpu\"]:\n",
- " globals().pop(_n, None)\n",
- "gc.collect(); torch.cuda.empty_cache()\n",
- "print(f\"GPU allocated now: {torch.cuda.memory_allocated()/1e9:.2f} GB\") # expect ~1.5 GB (just mhr_module)\n",
- "\n",
- "\n",
- "# # ==============================================================================\n",
- "# # SANITY CHECK: Evaluating Your Overfit Model\n",
- "# # ==============================================================================\n",
- "# if __name__ == \"__main__\":\n",
- "# print(\"--- Running Evaluation on Overfit Model ---\")\n",
- "# overfit_model.eval()\n",
- " \n",
- "# with torch.no_grad():\n",
- "# with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- "# eval_preds = overfit_model(overfit_batch[\"image\"], overfit_batch[\"cliff_cond\"])\n",
- " \n",
- "# # Upcast to float32 for rigorous geometric math\n",
- "# eval_preds_fp32 = {k: v.float() for k, v in eval_preds.items()}\n",
- " \n",
- "# metrics = evaluate_hmr_batch(eval_preds_fp32, overfit_batch, cfg, mhr_module)\n",
- " \n",
- "# print(\"\\n--- Final Metrics on 1-Batch Overfit ---\")\n",
- "# print(f\"3D Keypoint Head MPJPE: {metrics['Head_3D_MPJPE']:.2f} mm\")\n",
- "# print(f\"3D Keypoint Head PA-MPJPE: {metrics['Head_3D_PA_MPJPE']:.2f} mm\")\n",
- "# print(\"-\" * 40)\n",
- "# print(f\"MHR Mesh Anchors MPJPE: {metrics['Mesh_MPJPE']:.2f} mm\")\n",
- "# print(f\"MHR Mesh Anchors PA-MPJPE: {metrics['Mesh_PA_MPJPE']:.2f} mm\")\n",
- " \n",
- "# if metrics['Mesh_PA_MPJPE'] < 10.0:\n",
- "# print(\"\\n✅ OUTSTANDING: Mesh PA-MPJPE is under 10mm! Your model has achieved sub-centimeter physical accuracy.\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "d44aeb4a",
- "metadata": {},
- "outputs": [
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "5db7042569984e51a6f4b47a35739df5",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Epoch 123/400 [Val:raw]: 0%| | 0/4 [00:00<?, ?it/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "cb1bb10421eb494faa69e9465ddc0d94",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Epoch 123/400 [Val:ema]: 0%| | 0/4 [00:00<?, ?it/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- },
- {
- "name": "stdout",
- "output_type": "stream",
- "text": [
- "\n",
- "📈 Epoch 123 Summary | LR: 2.62e-04\n",
- " [Train] Tot: 0.0193 | MHR: 0.0027 | 2D: 0.0035 | 3D: 0.0045\n",
- " [Val RAW] Tot: 0.0640 | Mesh PA-MPJPE: 35.0 | Mesh MPJPE: 68.5 | Head PA: 38.1 mm\n",
- " [Val EMA] Tot: 0.0599 | Mesh PA-MPJPE: 22.9 | Mesh MPJPE: 48.4 | Head PA: 38.1 mm\n",
- " 🔬 EMA vs RAW (Mesh PA-MPJPE): EMA better by 12.1 mm\n",
- " ⏳ Neither RAW nor EMA improved: 1/150 (best raw 29.1 / ema 22.9 mm)\n"
- ]
- },
- {
- "data": {
- "application/vnd.jupyter.widget-view+json": {
- "model_id": "fb1ac92369754f959286325daadd5426",
- "version_major": 2,
- "version_minor": 0
- },
- "text/plain": [
- "Epoch 124/400 [Train]: 0%| | 0/33 [00:00<?, ?it/s]"
- ]
- },
- "metadata": {},
- "output_type": "display_data"
- }
- ],
- "source": [
- "# ============================================================\n",
- "# Cell 11 — Full Distillation Training Loop (EMA + Early-Stopping)\n",
- "# ============================================================\n",
- "import torch\n",
- "import torch.optim as optim\n",
- "import torch.optim.lr_scheduler as lr_scheduler\n",
- "import os\n",
- "import math\n",
- "from tqdm.notebook import tqdm\n",
- "from torch.optim.swa_utils import AveragedModel, get_ema_multi_avg_fn\n",
- "\n",
- "import gc, torch\n",
- "for _n in [\"overfit_model\", \"optimizer\", \"scaler\", \"overfit_batch\", \"overfit_batch_cpu\"]:\n",
- " globals().pop(_n, None)\n",
- "gc.collect(); torch.cuda.empty_cache()\n",
- "print(f\"GPU allocated now: {torch.cuda.memory_allocated()/1e9:.2f} GB\") # expect ~1.5 GB (just mhr_module)\n",
- "\n",
- "# EMA / early-stopping knobs (override in cfg if you want; sane defaults otherwise)\n",
- "EMA_DECAY = getattr(cfg, \"ema_decay\", 0.9998)\n",
- "EARLY_STOP_PATIENCE = getattr(cfg, \"early_stop_patience\", 50) # epochs where NEITHER raw nor ema improved\n",
- "\n",
- "\n",
- "def make_ema_avg_fn(max_decay):\n",
- " \"\"\"EMA with a decay WARMUP: decay = min(max_decay, (1+n)/(10+n)).\n",
- " Early on (small n) decay is small so the shadow TRACKS the model instead of\n",
- " sitting near random init -> removes the early 'mean-pose mush' that gamed PA-MPJPE.\"\"\"\n",
- " @torch.no_grad()\n",
- " def avg_fn(ema_params, model_params, num_averaged):\n",
- " # Integer buffers (e.g. BatchNorm num_batches_tracked) can't be lerp'd -> copy them,\n",
- " # mirroring torch's get_ema_multi_avg_fn. Without this, _foreach_lerp_ errors on Long.\n",
- " if not (torch.is_floating_point(ema_params[0]) or torch.is_complex(ema_params[0])):\n",
- " for e, m in zip(ema_params, model_params):\n",
- " e.copy_(m)\n",
- " return\n",
- " n = num_averaged.item() if torch.is_tensor(num_averaged) else float(num_averaged)\n",
- " decay = min(max_decay, (1.0 + n) / (10.0 + n))\n",
- " torch._foreach_lerp_(ema_params, model_params, 1.0 - decay)\n",
- " return avg_fn\n",
- "\n",
- "def train_instant_hmr():\n",
- " print(f\"--- Training {cfg.backbone} | EMA decay={EMA_DECAY} | early-stop patience={EARLY_STOP_PATIENCE} ---\")\n",
- " os.makedirs(cfg.log_dir, exist_ok=True)\n",
- "\n",
- " # Set to True to enable resuming\n",
- " RESUME_FROM_CHECKPOINT = True \n",
- "\n",
- " # 1. Model + EMA shadow + loss\n",
- " model = InstantHMRStudent(cfg, pretrained=True).to(device)\n",
- " criterion = DistillationLoss(cfg, mhr_module)\n",
- "\n",
- " # 2. Optimizer & AMP\n",
- " optimizer = optim.AdamW(model.parameters(), lr=cfg.lr, weight_decay=cfg.weight_decay)\n",
- " scaler = torch.amp.GradScaler(device='cuda', enabled=cfg.use_amp)\n",
- "\n",
- " # 3. LR schedule\n",
- " steps_per_epoch = len(train_loader)\n",
- " scheduler = lr_scheduler.OneCycleLR(\n",
- " optimizer, max_lr=cfg.lr, epochs=cfg.epochs,\n",
- " steps_per_epoch=steps_per_epoch, pct_start=0.1,\n",
- " )\n",
- "\n",
- " best_raw_path = os.path.join(cfg.log_dir, \"best_student_model_raw.pth\")\n",
- " best_ema_path = os.path.join(cfg.log_dir, \"best_student_model_ema.pth\")\n",
- " best_ckpt_path = os.path.join(cfg.log_dir, \"best_student_model_v3.pth\") # best overall\n",
- "\n",
- " # Tracking variables\n",
- " start_epoch = 0\n",
- " best_raw_pa = float('inf')\n",
- " best_ema_pa = float('inf')\n",
- " best_overall_pa = float('inf')\n",
- " epochs_no_improve = 0\n",
- "\n",
- " # 4. Resume from Checkpoint Logic\n",
- " if RESUME_FROM_CHECKPOINT and os.path.exists(best_ckpt_path):\n",
- " print(f\"🔄 Resuming from checkpoint: {best_ckpt_path}\")\n",
- " # weights_only=False is often required to load optimizer/scheduler dicts safely in newer PyTorch versions\n",
- " ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=False)\n",
- " \n",
- " # Load states\n",
- " model.load_state_dict(ckpt[\"model_state_dict\"], strict=False)\n",
- " if \"optimizer_state_dict\" in ckpt:\n",
- " optimizer.load_state_dict(ckpt[\"optimizer_state_dict\"])\n",
- " if \"scaler_state_dict\" in ckpt:\n",
- " scaler.load_state_dict(ckpt[\"scaler_state_dict\"])\n",
- " if \"scheduler_state_dict\" in ckpt:\n",
- " scheduler.load_state_dict(ckpt[\"scheduler_state_dict\"])\n",
- " \n",
- " start_epoch = ckpt.get(\"epoch\", -1) + 1\n",
- " best_overall_pa = ckpt.get(\"val_pa_mpjpe\", float('inf'))\n",
- " best_raw_pa = best_overall_pa \n",
- " best_ema_pa = best_overall_pa\n",
- " print(f\"✅ Resumed successfully from Epoch {start_epoch}. Best PA-MPJPE was {best_overall_pa:.1f}mm.\")\n",
- " else:\n",
- " print(\"🚀 Starting training from scratch with fresh weights.\")\n",
- "\n",
- " # Initialize EMA *after* potential checkpoint loading so it inherits the resumed weights\n",
- " ema_model = AveragedModel(model, multi_avg_fn=make_ema_avg_fn(EMA_DECAY), use_buffers=True)\n",
- "\n",
- " # --- validation helper: evaluate ANY model over val_loader (closes over criterion/loaders) ---\n",
- " @torch.no_grad()\n",
- " def run_validation(eval_model, tag, epoch):\n",
- " eval_model.eval()\n",
- " vloss = {'total': 0.0, '2d': 0.0, '3d': 0.0, 'mhr': 0.0, 'struct': 0.0}\n",
- " hmr = {'Head_3D_MPJPE': 0.0, 'Head_3D_PA_MPJPE': 0.0, 'Mesh_MPJPE': 0.0, 'Mesh_PA_MPJPE': 0.0}\n",
- " n = 0\n",
- " vbar = tqdm(val_loader, desc=f\"Epoch {epoch+1}/{cfg.epochs} [Val:{tag}]\", leave=False)\n",
- " for batch in vbar:\n",
- " batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " preds_fp16 = eval_model(batch[\"image\"], batch[\"cliff_cond\"])\n",
- " preds = {k: v.float() for k, v in preds_fp16.items()}\n",
- " losses = criterion(preds, batch)\n",
- " if math.isnan(losses['total_loss'].item()) or math.isinf(losses['total_loss'].item()):\n",
- " continue\n",
- " vloss['total'] += losses['total_loss'].item()\n",
- " vloss['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()\n",
- " vloss['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()\n",
- " vloss['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()\n",
- " vloss['struct'] += losses.get('loss_structural', torch.tensor(0)).item()\n",
- " bm = evaluate_hmr_batch(preds, batch, cfg, mhr_module)\n",
- " for k in hmr:\n",
- " hmr[k] += bm.get(k, 0.0)\n",
- " n += 1\n",
- " vbar.set_postfix({'PA-MPJPE': f\"{hmr['Mesh_PA_MPJPE']/max(n,1):.1f}mm\"})\n",
- " vbar.close()\n",
- " vloss = {k: v / max(n, 1) for k, v in vloss.items()}\n",
- " hmr = {k: v / max(n, 1) for k, v in hmr.items()}\n",
- " return vloss, hmr, n\n",
- "\n",
- " # Note the loop now starts at `start_epoch`\n",
- " for epoch in range(start_epoch, cfg.epochs):\n",
- " # ---------------- TRAIN ----------------\n",
- " model.train()\n",
- " tm = {'total': 0.0, '2d': 0.0, '3d': 0.0, 'pose': 0.0, 'scale': 0.0, 'shape': 0.0, 'mhr': 0.0, 'struct': 0.0}\n",
- " nb = 0\n",
- " pbar = tqdm(train_loader, desc=f\"Epoch {epoch+1}/{cfg.epochs} [Train]\")\n",
- " for batch in pbar:\n",
- " batch = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}\n",
- " optimizer.zero_grad(set_to_none=True)\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " preds_fp16 = model(batch[\"image\"], batch[\"cliff_cond\"])\n",
- " preds_fp32 = {k: v.float() for k, v in preds_fp16.items()}\n",
- " losses = criterion(preds_fp32, batch)\n",
- " loss = losses['total_loss']\n",
- " if math.isnan(loss.item()) or math.isinf(loss.item()):\n",
- " print(\"⚠️ WARNING: NaN/Inf loss detected! Skipping step.\")\n",
- " continue\n",
- " scaler.scale(loss).backward()\n",
- " scaler.unscale_(optimizer)\n",
- " torch.nn.utils.clip_grad_norm_(model.parameters(), cfg.grad_clip)\n",
- " scale_before = scaler.get_scale()\n",
- " scaler.step(optimizer)\n",
- " scaler.update()\n",
- " scale_after = scaler.get_scale()\n",
- " if scale_before <= scale_after: \n",
- " scheduler.step()\n",
- " ema_model.update_parameters(model) \n",
- " tm['total'] += loss.item()\n",
- " tm['2d'] += losses.get('loss_2d_native', torch.tensor(0)).item()\n",
- " tm['3d'] += losses.get('loss_3d_native', torch.tensor(0)).item()\n",
- " tm['pose'] += losses.get('loss_pose', torch.tensor(0)).item()\n",
- " tm['scale'] += losses.get('loss_scale', torch.tensor(0)).item()\n",
- " tm['mhr'] += losses.get('loss_mhr_joints', torch.tensor(0)).item()\n",
- " tm['struct'] += losses.get('loss_structural', torch.tensor(0)).item()\n",
- " nb += 1\n",
- " pbar.set_postfix({'Tot': f\"{loss.item():.3f}\",\n",
- " 'MHR': f\"{losses.get('loss_mhr_joints', torch.tensor(0)).item():.3f}\"})\n",
- " pbar.close()\n",
- " avg_train = {k: v / max(nb, 1) for k, v in tm.items()}\n",
- "\n",
- " # ---------------- VALIDATE: raw model AND EMA model ----------------\n",
- " raw_val, raw_hmr, n_raw = run_validation(model, \"raw\", epoch)\n",
- " ema_val, ema_hmr, n_ema = run_validation(ema_model, \"ema\", epoch)\n",
- "\n",
- " # ---------------- SUMMARY ----------------\n",
- " print(f\"\\n📈 Epoch {epoch+1} Summary | LR: {scheduler.get_last_lr()[0]:.2e}\")\n",
- " print(f\" [Train] Tot: {avg_train['total']:.4f} | MHR: {avg_train['mhr']:.4f} | 2D: {avg_train['2d']:.4f} | 3D: {avg_train['3d']:.4f}\")\n",
- " print(f\" [Val RAW] Tot: {raw_val['total']:.4f} | Mesh PA-MPJPE: {raw_hmr['Mesh_PA_MPJPE']:.1f} | Mesh MPJPE: {raw_hmr['Mesh_MPJPE']:.1f} | Head PA: {raw_hmr['Head_3D_PA_MPJPE']:.1f} mm\")\n",
- " print(f\" [Val EMA] Tot: {ema_val['total']:.4f} | Mesh PA-MPJPE: {ema_hmr['Mesh_PA_MPJPE']:.1f} | Mesh MPJPE: {ema_hmr['Mesh_MPJPE']:.1f} | Head PA: {ema_hmr['Head_3D_PA_MPJPE']:.1f} mm\")\n",
- " delta = raw_hmr['Mesh_PA_MPJPE'] - ema_hmr['Mesh_PA_MPJPE']\n",
- " print(f\" 🔬 EMA vs RAW (Mesh PA-MPJPE): {'EMA better' if delta > 0 else 'RAW better'} by {abs(delta):.1f} mm\")\n",
- "\n",
- " # ---------------- CHECKPOINTS ----------------\n",
- " \n",
- " # Save standard states dict for all checkpoints to allow resuming from any of them\n",
- " def get_save_dict(model_to_save, val_metric, source_name):\n",
- " return {\n",
- " 'epoch': epoch,\n",
- " 'model_state_dict': model_to_save.state_dict(),\n",
- " 'optimizer_state_dict': optimizer.state_dict(),\n",
- " 'scaler_state_dict': scaler.state_dict(),\n",
- " 'scheduler_state_dict': scheduler.state_dict(),\n",
- " 'val_pa_mpjpe': val_metric,\n",
- " 'source': source_name\n",
- " }\n",
- "\n",
- " # ---------------- CHECKPOINTS + EARLY-STOP BOOKKEEPING ----------------\n",
- " # Did each model improve its OWN best this epoch?\n",
- " improved_raw = n_raw > 0 and raw_hmr['Mesh_PA_MPJPE'] < best_raw_pa\n",
- " improved_ema = n_ema > 0 and ema_hmr['Mesh_PA_MPJPE'] < best_ema_pa\n",
- "\n",
- " if improved_raw:\n",
- " best_raw_pa = raw_hmr['Mesh_PA_MPJPE']\n",
- " torch.save(get_save_dict(model, best_raw_pa, 'raw'), best_raw_path)\n",
- " if improved_ema:\n",
- " best_ema_pa = ema_hmr['Mesh_PA_MPJPE']\n",
- " torch.save(get_save_dict(ema_model.module, best_ema_pa, 'ema'), best_ema_path)\n",
- "\n",
- " # Canonical \"best overall\" checkpoint (loaded by the export/eval cells)\n",
- " current = min(raw_hmr['Mesh_PA_MPJPE'], ema_hmr['Mesh_PA_MPJPE'])\n",
- " if current < best_overall_pa - 1e-4:\n",
- " best_overall_pa = current\n",
- " use_ema = ema_hmr['Mesh_PA_MPJPE'] <= raw_hmr['Mesh_PA_MPJPE']\n",
- " better = ema_model.module if use_ema else model\n",
- " torch.save(get_save_dict(better, best_overall_pa, 'ema' if use_ema else 'raw'), best_ckpt_path)\n",
- " print(f\"\\U0001f4be New best overall: {best_overall_pa:.1f} mm ({'EMA' if use_ema else 'RAW'}) -> {os.path.basename(best_ckpt_path)}\")\n",
- "\n",
- " # ---------------- EARLY STOPPING ----------------\n",
- " # Stop ONLY after EARLY_STOP_PATIENCE epochs where NEITHER raw NOR ema improved its own best.\n",
- " if improved_raw or improved_ema:\n",
- " epochs_no_improve = 0\n",
- " else:\n",
- " epochs_no_improve += 1\n",
- " print(f\" \\u23f3 Neither RAW nor EMA improved: {epochs_no_improve}/{EARLY_STOP_PATIENCE} (best raw {best_raw_pa:.1f} / ema {best_ema_pa:.1f} mm)\")\n",
- " if epochs_no_improve >= EARLY_STOP_PATIENCE:\n",
- " print(f\"\\n\\U0001f6d1 Early stopping at epoch {epoch+1}: neither RAW nor EMA improved for {EARLY_STOP_PATIENCE} epochs.\")\n",
- " break\n",
- "\n",
- " print(f\"\\n🎉 Training complete! Best overall Mesh PA-MPJPE: {best_overall_pa:.1f} mm\")\n",
- " print(f\" best RAW: {best_raw_pa:.1f} mm -> {best_raw_path}\")\n",
- " print(f\" best EMA: {best_ema_pa:.1f} mm -> {best_ema_path}\")\n",
- "\n",
- "# Kick off the training!\n",
- "if __name__ == \"__main__\":\n",
- " import gc\n",
- " gc.collect()\n",
- " torch.cuda.empty_cache()\n",
- " train_instant_hmr()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "e12c985b",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 13 — Validation Visualization (The Reveal!)\n",
- "# ============================================================\n",
- "import torch\n",
- "import matplotlib.pyplot as plt\n",
- "import os\n",
- "\n",
- "print(\"--- Running Inference Visualization ---\")\n",
- "\n",
- "# 1. Load the Best Saved Model\n",
- "best_model = InstantHMRStudent(cfg, pretrained=False).to(device)\n",
- "checkpoint_path = os.path.join(cfg.log_dir, \"best_student_model_v3.pth\")\n",
- "\n",
- "if os.path.exists(checkpoint_path):\n",
- " checkpoint = torch.load(checkpoint_path, map_location=device, weights_only=True)\n",
- " \n",
- " # --- THE FIX: Clean the torch.compile() prefix ---\n",
- " clean_state_dict = {}\n",
- " for k, v in checkpoint['model_state_dict'].items():\n",
- " # Remove '_orig_mod.' prefix if it exists\n",
- " new_key = k.replace('_orig_mod.', '')\n",
- " clean_state_dict[new_key] = v\n",
- "\n",
- " # Load the cleaned dictionary\n",
- " best_model.load_state_dict(clean_state_dict, strict=False)\n",
- " print(f\"✅ Loaded Best Model from Epoch {checkpoint.get('epoch', 0)+1} with PA-MPJPE: {checkpoint.get('val_pa_mpjpe', 0):.1f} mm\")\n",
- "else:\n",
- " print(\"⚠️ No checkpoint found! Using current model in memory.\")\n",
- " best_model = model\n",
- "\n",
- "best_model.eval()\n",
- "\n",
- "# 2. Grab a Validation Batch\n",
- "val_batch = next(iter(val_loader))\n",
- "images = val_batch[\"image\"].to(device)\n",
- "cliff_cond = val_batch[\"cliff_cond\"].to(device)\n",
- "gt_2d = val_batch[\"joints_2d\"].cpu()\n",
- "\n",
- "# 3. Run Inference\n",
- "with torch.no_grad():\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " preds = best_model(images, cliff_cond)\n",
- "\n",
- "# Extract only the X,Y coordinates\n",
- "pred_2d = preds[\"joints_2d\"][..., :2].cpu()\n",
- "\n",
- "# 4. Math Helper: Un-normalize coordinates from [-1, 1] back to [0, 224] pixels\n",
- "def unnorm_coords(coords, size=224.0):\n",
- " return ((coords + 1.0) / 2.0) * size\n",
- "\n",
- "gt_2d_px = unnorm_coords(gt_2d[..., :2], cfg.image_size)\n",
- "pred_2d_px = unnorm_coords(pred_2d, cfg.image_size)\n",
- "\n",
- "# 5. Image Helper: Un-normalize ImageNet pixels for matplotlib\n",
- "mean = torch.tensor([0.485, 0.456, 0.406]).view(3, 1, 1)\n",
- "std = torch.tensor([0.229, 0.224, 0.225]).view(3, 1, 1)\n",
- "\n",
- "# 6. Plot a 2x2 Grid of Predictions\n",
- "fig, axes = plt.subplots(2, 2, figsize=(10, 10))\n",
- "axes = axes.flatten()\n",
- "\n",
- "for i in range(4): # Plot the first 4 images in the batch\n",
- " # Prepare image\n",
- " img_unnorm = images[i].cpu() * std + mean\n",
- " img_np = img_unnorm.permute(1, 2, 0).numpy().clip(0, 1)\n",
- " \n",
- " # Plot Image\n",
- " axes[i].imshow(img_np)\n",
- " \n",
- " # Plot Ground Truth (Teacher) - Lime Green Dots\n",
- " axes[i].scatter(gt_2d_px[i, :, 0], gt_2d_px[i, :, 1], \n",
- " c='lime', s=25, edgecolors='black', label='Teacher (GT)' if i==0 else \"\")\n",
- " \n",
- " # Plot Predictions (Student) - Red Crosses\n",
- " axes[i].scatter(pred_2d_px[i, :, 0], pred_2d_px[i, :, 1], \n",
- " c='red', s=35, marker='X', edgecolors='black', linewidths=0.5, label='Student' if i==0 else \"\")\n",
- " \n",
- " axes[i].axis('off')\n",
- " axes[i].set_title(f\"Validation Sample {i+1}\", fontsize=12)\n",
- "\n",
- "fig.legend(loc='upper center', ncol=2, fontsize=14, bbox_to_anchor=(0.5, 1.05))\n",
- "plt.tight_layout()\n",
- "plt.show()"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "2cd9f6d4",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 14 — Inference Speed Benchmark (Fixed Loader)\n",
- "# ============================================================\n",
- "import time\n",
- "import os\n",
- "import torch\n",
- "\n",
- "print(\"--- Running Inference Speed Benchmark ---\")\n",
- "\n",
- "@torch.no_grad()\n",
- "def benchmark_fps(model, image_size=224, batch_size=1, n_warmup=50, n_iters=200):\n",
- " \"\"\"Benchmark model throughput on GPU.\"\"\"\n",
- " model.eval()\n",
- " # Dummy inputs matching our architecture\n",
- " dummy_img = torch.randn(batch_size, 3, image_size, image_size, device=device)\n",
- " dummy_cliff = torch.randn(batch_size, 3, device=device) # [cx_norm, cy_norm, b_scale]\n",
- "\n",
- " # Warmup (Wraps the forward pass in AMP for real-world float16 speed)\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " for _ in range(n_warmup):\n",
- " _ = model(dummy_img, dummy_cliff)\n",
- " \n",
- " if torch.cuda.is_available():\n",
- " torch.cuda.synchronize()\n",
- "\n",
- " # Timed run\n",
- " t0 = time.perf_counter()\n",
- " with torch.amp.autocast(device_type='cuda', dtype=torch.float16, enabled=cfg.use_amp):\n",
- " for _ in range(n_iters):\n",
- " _ = model(dummy_img, dummy_cliff)\n",
- " \n",
- " if torch.cuda.is_available():\n",
- " torch.cuda.synchronize()\n",
- " \n",
- " elapsed = time.perf_counter() - t0\n",
- "\n",
- " fps = n_iters * batch_size / elapsed\n",
- " ms_per_image = elapsed / (n_iters * batch_size) * 1000\n",
- " return fps, ms_per_image\n",
- "\n",
- "# 1. Instantiate the Model\n",
- "benchmark_model = InstantHMRStudent(cfg, pretrained=False).to(device)\n",
- "\n",
- "# 2. Load best checkpoint with the compile fix!\n",
- "best_ckpt_path = os.path.join(cfg.log_dir, \"best_student_model_v3.pth\")\n",
- "if os.path.exists(best_ckpt_path):\n",
- " ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=True)\n",
- " \n",
- " # --- Clean the torch.compile() prefix ---\n",
- " clean_state_dict = {}\n",
- " for k, v in ckpt['model_state_dict'].items():\n",
- " new_key = k.replace('_orig_mod.', '')\n",
- " clean_state_dict[new_key] = v\n",
- " \n",
- " benchmark_model.load_state_dict(clean_state_dict, strict=False)\n",
- " print(f\"✅ Loaded best model from epoch {ckpt.get('epoch', 0)+1} (PA-MPJPE={ckpt.get('val_pa_mpjpe', 0):.1f} mm)\")\n",
- "else:\n",
- " print(\"⚠️ No checkpoint found! Benchmarking with random weights.\")\n",
- "\n",
- "# 3. Benchmark at different batch sizes\n",
- "fps_b1, ms_b1 = benchmark_fps(benchmark_model, batch_size=1)\n",
- "fps_b16, ms_b16 = benchmark_fps(benchmark_model, batch_size=16)\n",
- "fps_b64, ms_b64 = benchmark_fps(benchmark_model, batch_size=64)\n",
- "\n",
- "# 4. Compute total parameters\n",
- "total_params = sum(p.numel() for p in benchmark_model.parameters())\n",
- "\n",
- "print(f\"\\n🚀 Inference Speed (RTX {torch.cuda.get_device_name(0) if torch.cuda.is_available() else 'CPU'}):\")\n",
- "print(f\" Batch=1: {fps_b1:.0f} FPS ({ms_b1:.1f} ms/image)\")\n",
- "print(f\" Batch=16: {fps_b16:.0f} FPS ({ms_b16:.1f} ms/image)\")\n",
- "print(f\" Batch=64: {fps_b64:.0f} FPS ({ms_b64:.1f} ms/image)\")\n",
- "print(f\"\\n📊 Comparison:\")\n",
- "print(f\" SAM3D Teacher: ~3 FPS (670M params)\")\n",
- "print(f\" Student: {fps_b1:.0f} FPS ({total_params/1e6:.1f}M params) → {fps_b1/3:.0f}x faster\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "334859a9",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 15 — Export & Static Quantization (FP32, FP16, INT8)\n",
- "# ============================================================\n",
- "import os\n",
- "import torch\n",
- "import torch.nn as nn\n",
- "import numpy as np\n",
- "from pathlib import Path\n",
- "\n",
- "print(\"--- Exporting and Quantizing Full Model for Deployment ---\")\n",
- "\n",
- "# --- 1. Deploy Wrapper (full outputs) ---\n",
- "class HMRDeployWrapper(nn.Module):\n",
- " def __init__(self, model):\n",
- " super().__init__()\n",
- " self.model = model\n",
- " def forward(self, image, cliff_cond):\n",
- " out = self.model(image, cliff_cond)\n",
- " return (out[\"mhr_params\"], out[\"shape_params\"], out[\"cam_trans\"], \n",
- " out[\"joints_2d\"], out[\"joints_3d\"])\n",
- "\n",
- "# --- 2. INT8 Calibration Data Reader ---\n",
- "try:\n",
- " from onnxruntime.quantization import (\n",
- " CalibrationDataReader, quantize_static,\n",
- " QuantType, QuantFormat, CalibrationMethod,\n",
- " )\n",
- " \n",
- " class HMRCalibrationDataReader(CalibrationDataReader):\n",
- " def __init__(self, dataloader, max_samples=100):\n",
- " self.data_list = []\n",
- " \n",
- " print(f\"Extracting {max_samples} individual images for INT8 calibration...\")\n",
- " for batch in dataloader:\n",
- " images = batch[\"image\"].numpy().astype(np.float32)\n",
- " cliffs = batch[\"cliff_cond\"].numpy().astype(np.float32)\n",
- " \n",
- " for b_idx in range(images.shape[0]):\n",
- " if len(self.data_list) >= max_samples:\n",
- " break\n",
- " self.data_list.append({\n",
- " \"image\": images[b_idx:b_idx+1],\n",
- " \"cliff_cond\": cliffs[b_idx:b_idx+1]\n",
- " })\n",
- " if len(self.data_list) >= max_samples:\n",
- " break\n",
- " \n",
- " self.enum_data = iter(self.data_list)\n",
- "\n",
- " def get_next(self):\n",
- " return next(self.enum_data, None)\n",
- "\n",
- "except ImportError:\n",
- " print(\"⚠️ 'onnxruntime' is not installed. INT8 quantization will fail.\")\n",
- "\n",
- "\n",
- "def topological_sort_onnx(model):\n",
- " \"\"\"Sort ONNX graph nodes in topological order.\n",
- " \n",
- " The ORT float16 converter with keep_io_types=True inserts Cast nodes at graph\n",
- " inputs but doesn't re-sort the graph, breaking topological order. This fixes it.\n",
- " \"\"\"\n",
- " available = set()\n",
- " for inp in model.graph.input:\n",
- " available.add(inp.name)\n",
- " for init in model.graph.initializer:\n",
- " available.add(init.name)\n",
- " \n",
- " remaining = list(model.graph.node)\n",
- " sorted_nodes = []\n",
- " \n",
- " while remaining:\n",
- " progress = False\n",
- " for node in remaining:\n",
- " if all(inp in available or inp == '' for inp in node.input):\n",
- " sorted_nodes.append(node)\n",
- " remaining.remove(node)\n",
- " for out in node.output:\n",
- " available.add(out)\n",
- " progress = True\n",
- " break\n",
- " if not progress:\n",
- " sorted_nodes.extend(remaining)\n",
- " break\n",
- " \n",
- " del model.graph.node[:]\n",
- " model.graph.node.extend(sorted_nodes)\n",
- " return model\n",
- "\n",
- "\n",
- "# --- 3. Export & Quantize ---\n",
- "export_dir = Path(cfg.log_dir) / \"export\"\n",
- "export_dir.mkdir(parents=True, exist_ok=True)\n",
- "\n",
- "onnx_fp32_path = export_dir / \"sam3d_full_fp32.onnx\"\n",
- "onnx_fp16_path = export_dir / \"sam3d_full_fp16.onnx\"\n",
- "onnx_int8_path = export_dir / \"sam3d_full_int8.onnx\"\n",
- "\n",
- "output_keys = [\"mhr_params\", \"shape_params\", \"cam_trans\", \"joints_2d\", \"joints_3d\"]\n",
- "\n",
- "export_target_model = InstantHMRStudent(cfg, pretrained=False).to(device)\n",
- "\n",
- "best_ckpt_path = os.path.join(cfg.log_dir, \"best_student_model_v3.pth\")\n",
- "\n",
- "if os.path.exists(best_ckpt_path):\n",
- " ckpt = torch.load(best_ckpt_path, map_location=device, weights_only=True)\n",
- " clean_state_dict = {k.replace('_orig_mod.', ''): v for k, v in ckpt['model_state_dict'].items()}\n",
- " export_target_model.load_state_dict(clean_state_dict, strict=False)\n",
- " print(f\"✅ Loaded best model from epoch {ckpt.get('epoch', 0)+1}.\\n\")\n",
- " \n",
- " deploy_model = HMRDeployWrapper(export_target_model)\n",
- " deploy_model.eval()\n",
- " \n",
- " dummy_img = torch.randn(1, 3, cfg.image_size, cfg.image_size, device=device)\n",
- " dummy_cliff = torch.randn(1, 3, device=device)\n",
- " \n",
- " dynamic_axes = {\"image\": {0: \"batch\"}, \"cliff_cond\": {0: \"batch\"}}\n",
- " for k in output_keys:\n",
- " dynamic_axes[k] = {0: \"batch\"}\n",
- "\n",
- " # A. Export FP32\n",
- " try:\n",
- " import onnx\n",
- " torch.onnx.export(\n",
- " deploy_model, (dummy_img, dummy_cliff), str(onnx_fp32_path),\n",
- " input_names=[\"image\", \"cliff_cond\"], output_names=output_keys,\n",
- " dynamic_axes=dynamic_axes, opset_version=17\n",
- " )\n",
- " print(f\"✅ FP32 saved: {onnx_fp32_path.name} | Size: {onnx_fp32_path.stat().st_size / 1e6:.1f} MB\")\n",
- " except Exception as e:\n",
- " print(f\"❌ FP32 export failed: {e}\")\n",
- "\n",
- " # B. Export FP16 — uses ORT's converter which correctly updates Cast node `to` attributes\n",
- " # + topological sort to fix node ordering after keep_io_types Cast insertion\n",
- " try:\n",
- " from onnxruntime.transformers.float16 import convert_float_to_float16\n",
- " import warnings\n",
- " with warnings.catch_warnings():\n",
- " warnings.simplefilter(\"ignore\")\n",
- " model_fp32 = onnx.load(str(onnx_fp32_path))\n",
- " model_fp16 = convert_float_to_float16(model_fp32, keep_io_types=True)\n",
- " model_fp16 = topological_sort_onnx(model_fp16)\n",
- " onnx.save(model_fp16, str(onnx_fp16_path))\n",
- " onnx.checker.check_model(onnx.load(str(onnx_fp16_path)), full_check=True)\n",
- " print(f\"✅ FP16 saved: {onnx_fp16_path.name} | Size: {onnx_fp16_path.stat().st_size / 1e6:.1f} MB\")\n",
- " except ImportError:\n",
- " print(\"⚠️ FP16 skipped (onnxruntime.transformers not available)\")\n",
- " except Exception as e:\n",
- " print(f\"❌ FP16 conversion failed: {e}\")\n",
- "\n",
- " # C. Export INT8 — selective: only quantize Conv ops (backbone), skip transformer/heads\n",
- " # Naive quantization of all ops destroys accuracy (Softmax, LayerNorm, attention MatMuls\n",
- " # need FP32 precision) and is slower due to unfused QDQ overhead blocking NCHWc optimization.\n",
- " try:\n",
- " calib_reader = HMRCalibrationDataReader(train_loader, max_samples=100)\n",
- " calib_reader.enum_data = iter(calib_reader.data_list)\n",
- " \n",
- " # Collect transformer + head + projection node names to exclude from quantization\n",
- " # (defense-in-depth: op_types_to_quantize=['Conv'] already excludes them since they\n",
- " # have zero Conv ops, but explicit exclusion prevents issues if the model changes)\n",
- " fp32_model = onnx.load(str(onnx_fp32_path))\n",
- " nodes_to_exclude = []\n",
- " for node in fp32_model.graph.node:\n",
- " name = node.name or ''\n",
- " if any(kw in name for kw in ['/model/transformer/', '/model/head_',\n",
- " '/model/feat_proj', '/model/cond_proj']):\n",
- " nodes_to_exclude.append(name)\n",
- " del fp32_model\n",
- " print(f\" Excluding {len(nodes_to_exclude)} transformer/head nodes from quantization\")\n",
- " \n",
- " quantize_static(\n",
- " model_input=str(onnx_fp32_path),\n",
- " model_output=str(onnx_int8_path),\n",
- " calibration_data_reader=calib_reader,\n",
- " quant_format=QuantFormat.QDQ,\n",
- " op_types_to_quantize=['Conv'], # Only quantize Conv (backbone)\n",
- " per_channel=True, # Better accuracy for Conv weights\n",
- " reduce_range=False, # Not needed with VNNI\n",
- " activation_type=QuantType.QInt8,\n",
- " weight_type=QuantType.QInt8,\n",
- " nodes_to_exclude=nodes_to_exclude,\n",
- " calibrate_method=CalibrationMethod.Percentile,\n",
- " extra_options={\n",
- " 'CalibTensorRangeSymmetric': True,\n",
- " },\n",
- " )\n",
- " print(f\"✅ INT8 saved: {onnx_int8_path.name} | Size: {onnx_int8_path.stat().st_size / 1e6:.1f} MB\")\n",
- " except NameError:\n",
- " print(\"⚠️ INT8 skipped (train_loader not available)\")\n",
- " except Exception as e:\n",
- " import traceback\n",
- " print(f\"❌ INT8 conversion failed: {e}\")\n",
- " traceback.print_exc()\n",
- "\n",
- " # D. Save PyTorch state dict\n",
- " sd_path = export_dir / \"sam3d_student_state_dict.pth\"\n",
- " torch.save(export_target_model.state_dict(), sd_path)\n",
- " print(f\"\\n✅ PyTorch state dict saved: {sd_path.name}\")\n",
- " \n",
- "else:\n",
- " print(\"⚠️ No checkpoint found at:\", best_ckpt_path)"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "11d916bf",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 17 — ONNX Profiler (FP32, FP16, INT8) x (CPU, GPU)\n",
- "# ============================================================\n",
- "import time\n",
- "import numpy as np\n",
- "from pathlib import Path\n",
- "\n",
- "print(\"--- ONNX Hardware Profiler ---\")\n",
- "\n",
- "try:\n",
- " import onnxruntime as ort\n",
- " ONNX_AVAILABLE = True\n",
- "except ImportError:\n",
- " ONNX_AVAILABLE = False\n",
- " print(\"⚠️ Missing: pip install onnxruntime-gpu\")\n",
- "\n",
- "if ONNX_AVAILABLE:\n",
- " def profile_onnx_model(onnx_path, provider, batch_size=1, warmup_iters=20, benchmark_iters=100):\n",
- " \"\"\"Profile latency for a specific ONNX model and execution provider.\"\"\"\n",
- " \n",
- " sess_options = ort.SessionOptions()\n",
- " sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL\n",
- " sess_options.log_severity_level = 3\n",
- " \n",
- " try:\n",
- " session = ort.InferenceSession(str(onnx_path), sess_options=sess_options, providers=[provider])\n",
- " except Exception as e:\n",
- " return None, None, f\"Error: {e}\"\n",
- " \n",
- " active_provider = session.get_providers()[0]\n",
- " if active_provider != provider:\n",
- " return None, None, f\"Fallback! Requested {provider} but got {active_provider}\"\n",
- "\n",
- " ort_inputs = {}\n",
- " dtype_map = {'tensor(float)': np.float32, 'tensor(float16)': np.float16}\n",
- " \n",
- " for inp in session.get_inputs():\n",
- " shape = [batch_size if isinstance(dim, str) else dim for dim in inp.shape]\n",
- " np_dtype = dtype_map.get(inp.type, np.float32)\n",
- " ort_inputs[inp.name] = np.random.randn(*shape).astype(np_dtype)\n",
- "\n",
- " for _ in range(warmup_iters):\n",
- " session.run(None, ort_inputs)\n",
- "\n",
- " t0 = time.perf_counter()\n",
- " for _ in range(benchmark_iters):\n",
- " session.run(None, ort_inputs)\n",
- " t1 = time.perf_counter()\n",
- "\n",
- " total_time = t1 - t0\n",
- " avg_ms = (total_time / benchmark_iters) * 1000.0\n",
- " fps = (benchmark_iters * batch_size) / total_time\n",
- " return avg_ms, fps, \"OK\"\n",
- "\n",
- " # --- Setup ---\n",
- " export_dir = Path(cfg.log_dir) / \"export\"\n",
- " precisions = [\"fp32\", \"fp16\", \"int8\"]\n",
- "\n",
- " available_providers = ort.get_available_providers()\n",
- " providers_to_test = ['CPUExecutionProvider']\n",
- " if 'CUDAExecutionProvider' in available_providers:\n",
- " providers_to_test.append('CUDAExecutionProvider')\n",
- " else:\n",
- " print(\"⚠️ CUDAExecutionProvider not found. Profiling CPU only.\")\n",
- " print(\" Fix: pip uninstall onnxruntime onnxruntime-gpu -y && pip install onnxruntime-gpu\")\n",
- "\n",
- " print(f\"\\n{'Precision':<6} | {'Size':<7} | {'HW':<4} | {'Latency':<12} | {'FPS':<8} | Notes\")\n",
- " print(\"-\" * 75)\n",
- "\n",
- " for prec in precisions:\n",
- " model_path = export_dir / f\"sam3d_full_{prec}.onnx\"\n",
- " \n",
- " if not model_path.exists():\n",
- " print(f\"{prec:<6} | --- | --- | {'Not Found':<12} | ---\")\n",
- " continue\n",
- " \n",
- " size_mb = f\"{model_path.stat().st_size / 1e6:.1f}MB\"\n",
- " \n",
- " for provider in providers_to_test:\n",
- " hw_tag = \"GPU\" if \"CUDA\" in provider else \"CPU\"\n",
- " \n",
- " # FP16 on CPU: not supported (no native x86 FP16 compute)\n",
- " if prec == \"fp16\" and provider == \"CPUExecutionProvider\":\n",
- " print(f\"{prec:<6} | {size_mb:<7} | {hw_tag:<4} | {'N/A':<12} | --- | x86 has no FP16 ALU\")\n",
- " continue\n",
- " \n",
- " note = \"\"\n",
- " if prec == \"fp16\" and \"CUDA\" in provider:\n",
- " note = \"Best for GPU deploy\"\n",
- " elif prec == \"int8\" and \"CPU\" in provider:\n",
- " note = \"Conv-only quant, for edge/NPU\"\n",
- " elif prec == \"int8\" and \"CUDA\" in provider:\n",
- " note = \"Needs TensorRT EP for gains\"\n",
- " \n",
- " ms, fps, status = profile_onnx_model(model_path, provider)\n",
- " \n",
- " if ms is not None:\n",
- " print(f\"{prec:<6} | {size_mb:<7} | {hw_tag:<4} | {ms:>7.2f} ms | {fps:>6.1f} | {note}\")\n",
- " else:\n",
- " print(f\"{prec:<6} | {size_mb:<7} | {hw_tag:<4} | {'Failed':<12} | --- | {status}\")\n",
- "\n",
- " print(\"-\" * 75)\n",
- " print(\"\\nDeployment guide:\")\n",
- " print(\" GPU (CUDA EP) → FP16 (best latency + half memory)\")\n",
- " print(\" GPU (TensorRT EP) → INT8 (if TRT available, even faster)\")\n",
- " print(\" CPU (x86 VNNI) → FP32 or INT8 (Conv-only quant, test both)\")\n",
- " print(\" Mobile NPU → INT8 with NNAPI/QNN EP (native INT8 HW)\")"
- ]
- },
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "eikwijqvez7",
- "metadata": {},
- "outputs": [],
- "source": [
- "# ============================================================\n",
- "# Cell 18 — ONNX Accuracy Comparison (FP32 vs FP16 vs INT8)\n",
- "# ============================================================\n",
- "import numpy as np\n",
- "import torch\n",
- "from pathlib import Path\n",
- "from tqdm.notebook import tqdm\n",
- "\n",
- "print(\"--- ONNX Quantization Accuracy Comparison ---\")\n",
- "\n",
- "try:\n",
- " import onnxruntime as ort\n",
- " ONNX_AVAILABLE = True\n",
- "except ImportError:\n",
- " ONNX_AVAILABLE = False\n",
- " print(\"⚠️ Missing: pip install onnxruntime-gpu\")\n",
- "\n",
- "if ONNX_AVAILABLE:\n",
- " export_dir = Path(cfg.log_dir) / \"export\"\n",
- " output_names = [\"mhr_params\", \"shape_params\", \"cam_trans\", \"joints_2d\", \"joints_3d\"]\n",
- " \n",
- " models_to_eval = {\n",
- " \"FP32\": export_dir / \"sam3d_full_fp32.onnx\",\n",
- " \"FP16\": export_dir / \"sam3d_full_fp16.onnx\",\n",
- " \"INT8\": export_dir / \"sam3d_full_int8.onnx\",\n",
- " }\n",
- " \n",
- " # Filter to existing models only\n",
- " models_to_eval = {k: v for k, v in models_to_eval.items() if v.exists()}\n",
- " print(f\"Found models: {', '.join(models_to_eval.keys())}\")\n",
- " \n",
- " # Use GPU if available, otherwise CPU (FP16 needs GPU)\n",
- " available_providers = ort.get_available_providers()\n",
- " providers = ['CUDAExecutionProvider', 'CPUExecutionProvider'] if 'CUDAExecutionProvider' in available_providers else ['CPUExecutionProvider']\n",
- " \n",
- " results = {}\n",
- " \n",
- " for model_name, model_path in models_to_eval.items():\n",
- " print(f\"\\nEvaluating {model_name} ({model_path.stat().st_size / 1e6:.1f} MB)...\")\n",
- " \n",
- " sess_options = ort.SessionOptions()\n",
- " sess_options.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL\n",
- " sess_options.log_severity_level = 3\n",
- " \n",
- " try:\n",
- " session = ort.InferenceSession(str(model_path), sess_options=sess_options, providers=providers)\n",
- " except Exception as e:\n",
- " print(f\" ❌ Failed to load: {e}\")\n",
- " continue\n",
- " \n",
- " active_provider = session.get_providers()[0]\n",
- " print(f\" Running on: {active_provider}\")\n",
- " \n",
- " # Accumulate metrics across all val batches\n",
- " hmr_trackers = {'Head_3D_MPJPE': 0.0, 'Head_3D_PA_MPJPE': 0.0, 'Mesh_MPJPE': 0.0, 'Mesh_PA_MPJPE': 0.0}\n",
- " valid_batches = 0\n",
- " \n",
- " for batch in tqdm(val_loader, desc=f\" {model_name}\", leave=False):\n",
- " images_np = batch[\"image\"].numpy().astype(np.float32)\n",
- " cliff_np = batch[\"cliff_cond\"].numpy().astype(np.float32)\n",
- " \n",
- " ort_inputs = {\"image\": images_np, \"cliff_cond\": cliff_np}\n",
- " ort_outputs = session.run(None, ort_inputs)\n",
- " \n",
- " # Convert ONNX outputs to PyTorch tensor dict\n",
- " preds = {}\n",
- " for name, arr in zip(output_names, ort_outputs):\n",
- " preds[name] = torch.from_numpy(arr.astype(np.float32)).to(device)\n",
- " \n",
- " # Move GT targets to device\n",
- " targets = {k: v.to(device) if isinstance(v, torch.Tensor) else v for k, v in batch.items()}\n",
- " \n",
- " batch_metrics = evaluate_hmr_batch(preds, targets, cfg, mhr_module)\n",
- " for k in hmr_trackers:\n",
- " hmr_trackers[k] += batch_metrics.get(k, 0.0)\n",
- " valid_batches += 1\n",
- " \n",
- " avg_metrics = {k: v / max(valid_batches, 1) for k, v in hmr_trackers.items()}\n",
- " results[model_name] = avg_metrics\n",
- " print(f\" ✅ {valid_batches} batches evaluated\")\n",
- " \n",
- " # --- Print Comparison Table ---\n",
- " print(f\"\\n{'='*72}\")\n",
- " print(f\"{'Metric':<22} |\", end=\"\")\n",
- " for name in results:\n",
- " print(f\" {name:>10}\", end=\" |\")\n",
- " \n",
- " # Show degradation vs FP32\n",
- " if \"FP32\" in results and len(results) > 1:\n",
- " print(f\" {'vs FP32':>10}\", end=\"\")\n",
- " print()\n",
- " print(\"-\" * 72)\n",
- " \n",
- " metric_labels = {\n",
- " 'Head_3D_MPJPE': 'Head 3D MPJPE (mm)',\n",
- " 'Head_3D_PA_MPJPE': 'Head 3D PA-MPJPE (mm)',\n",
- " 'Mesh_MPJPE': 'Mesh MPJPE (mm)',\n",
- " 'Mesh_PA_MPJPE': 'Mesh PA-MPJPE (mm)',\n",
- " }\n",
- " \n",
- " for key, label in metric_labels.items():\n",
- " print(f\"{label:<22} |\", end=\"\")\n",
- " for name in results:\n",
- " print(f\" {results[name][key]:>10.2f}\", end=\" |\")\n",
- " \n",
- " # Show delta vs FP32 for non-FP32 models\n",
- " if \"FP32\" in results and len(results) > 1:\n",
- " deltas = []\n",
- " for name in results:\n",
- " if name != \"FP32\":\n",
- " delta = results[name][key] - results[\"FP32\"][key]\n",
- " deltas.append(f\"{delta:+.2f}\")\n",
- " print(f\" {', '.join(deltas):>10}\", end=\"\")\n",
- " print()\n",
- " \n",
- " print(f\"{'='*72}\")"
- ]
- }
- ],
- "metadata": {
- "kernelspec": {
- "display_name": "sam3d_video_312",
- "language": "python",
- "name": "python3"
- },
- "language_info": {
- "codemirror_mode": {
- "name": "ipython",
- "version": 3
- },
- "file_extension": ".py",
- "mimetype": "text/x-python",
- "name": "python",
- "nbconvert_exporter": "python",
- "pygments_lexer": "ipython3",
- "version": "3.12.12"
- }
- },
- "nbformat": 4,
- "nbformat_minor": 5
-}
diff --git a/tools/bench_backbone.py b/tools/bench_backbone.py
deleted file mode 100644
index 74ec540..0000000
--- a/tools/bench_backbone.py
+++ /dev/null
@@ -1,261 +0,0 @@
-#!/usr/bin/env python3
-"""Backbone A/B for the MHR-only InstantHMR student: RepViT vs MobileNetV4.
-
-The student is backbone-agnostic -- InstantHMRStudent reads the feature width
-from `backbone.num_features` and assumes a stride-32 (B,C,H,W) map, which every
-MobileNetV4 variant satisfies -- so this measures the cost side of the swap on
-the *whole* student, not on the backbone in isolation.
-
- python tools/bench_backbone.py [--batches 1 8 32] [--iters 50] [--no-onnx]
-"""
-from __future__ import annotations
-
-import argparse
-import contextlib
-import io
-import sys
-import time
-from pathlib import Path
-
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-TRAIN_DIR = ROOT / "instanthmr_distill_train"
-sys.path.insert(0, str(TRAIN_DIR))
-
-DEV = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-CANDIDATES = [
- "repvit_m1_5",
- "repvit_m2_3",
- "mobilenetv4_conv_medium",
- "mobilenetv4_hybrid_medium",
- "mobilenetv4_conv_large",
-]
-
-
-def timeit(fn, iters, warmup=10):
- for _ in range(warmup):
- fn()
- if DEV.type == "cuda":
- torch.cuda.synchronize()
- t0 = time.perf_counter()
- for _ in range(iters):
- fn()
- if DEV.type == "cuda":
- torch.cuda.synchronize()
- return (time.perf_counter() - t0) / iters * 1000.0
-
-
-def load_module():
- buf = io.StringIO()
- with contextlib.redirect_stdout(buf):
- import train_distill_mhr_only as M
- return M
-
-
-def build(M, name, pretrained=False):
- cfg = M.DistillConfig()
- cfg.backbone = name
- model = M.InstantHMRStudent(cfg, pretrained=pretrained).to(DEV).eval()
- return cfg, model
-
-
-def bench_torch(model, cfg, batch, iters, amp):
- img = torch.randn(batch, 3, cfg.image_size, cfg.image_size, device=DEV)
- cliff = torch.randn(batch, 3, device=DEV)
-
- def step():
- with torch.no_grad(), torch.amp.autocast(
- device_type="cuda", dtype=torch.float16, enabled=amp):
- model(img, cliff)
-
- def bb():
- with torch.no_grad(), torch.amp.autocast(
- device_type="cuda", dtype=torch.float16, enabled=amp):
- model.backbone.forward_features(img)
-
- return timeit(step, iters), timeit(bb, iters)
-
-
-def bench_train_step(M, name, batch, iters):
- """fwd+bwd on the regression heads only (no data, no MHR) + peak memory."""
- cfg, model = build(M, name, pretrained=False)
- model.train()
- opt = torch.optim.AdamW(model.parameters(), lr=1e-4)
- scaler = torch.amp.GradScaler(device="cuda")
- img = torch.randn(batch, 3, cfg.image_size, cfg.image_size, device=DEV)
- cliff = torch.randn(batch, 3, device=DEV)
- torch.cuda.reset_peak_memory_stats()
-
- def step():
- opt.zero_grad(set_to_none=True)
- with torch.amp.autocast(device_type="cuda", dtype=torch.float16):
- out = model(img, cliff)
- loss = sum(v.float().square().mean() for v in out.values())
- scaler.scale(loss).backward()
- scaler.step(opt)
- scaler.update()
-
- ms = timeit(step, iters, warmup=5)
- peak = torch.cuda.max_memory_allocated() / 2**20
- del model, opt, img, cliff
- torch.cuda.empty_cache()
- return ms, peak
-
-
-def export_onnx(M, name, cfg, model, out_dir):
- """Export + fp16-convert the student. Returns the fp16 .onnx path."""
- import onnx
- # Same converter the training script's export_and_quantize() uses.
- # onnxconverter_common's version mistypes the decoder's attention Cast nodes
- # and the resulting graph fails to load in ORT.
- from onnxruntime.transformers.float16 import convert_float_to_float16
-
- class Wrap(torch.nn.Module):
- def __init__(self, m, c):
- super().__init__()
- self.m, self.c = m, c
-
- def forward(self, image, cliff_cond):
- o = self.m(image, cliff_cond)
- return (o["mhr_params"], o["shape_params"],
- o["cam_trans"], o["joints_2d"])
-
- w = Wrap(model, cfg).eval().to(DEV)
- f32 = out_dir / f"{name}_fp32.onnx"
- f16 = out_dir / f"{name}_fp16.onnx"
- dummy = (torch.randn(1, 3, cfg.image_size, cfg.image_size, device=DEV),
- torch.randn(1, 3, device=DEV))
- with contextlib.redirect_stdout(io.StringIO()):
- torch.onnx.export(
- w, dummy, str(f32), input_names=["image", "cliff_cond"],
- output_names=["mhr_params", "shape_params", "cam_trans", "joints_2d"],
- dynamic_axes={"image": {0: "B"}, "cliff_cond": {0: "B"},
- "mhr_params": {0: "B"}, "shape_params": {0: "B"},
- "cam_trans": {0: "B"}, "joints_2d": {0: "B"}},
- opset_version=17, do_constant_folding=True)
- m16 = convert_float_to_float16(onnx.load(str(f32)), keep_io_types=True)
- onnx.save(m16, str(f16))
- return f16
-
-
-def bench_ort(path, cfg, batches, iters):
- import onnxruntime as ort
- so = ort.SessionOptions()
- so.log_severity_level = 3
- sess = ort.InferenceSession(str(path), so, providers=["CUDAExecutionProvider"])
- assert "CUDAExecutionProvider" in sess.get_providers(), "ORT fell back to CPU"
- res = {}
- for b in batches:
- feed = {"image": np.random.randn(b, 3, cfg.image_size, cfg.image_size).astype(np.float32),
- "cliff_cond": np.random.randn(b, 3).astype(np.float32)}
- for _ in range(10):
- sess.run(None, feed)
- t0 = time.perf_counter()
- for _ in range(iters):
- sess.run(None, feed)
- res[b] = (time.perf_counter() - t0) / iters * 1000.0
- del sess
- return res
-
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--batches", type=int, nargs="+", default=[1, 8, 32])
- ap.add_argument("--iters", type=int, default=50)
- ap.add_argument("--train-batch", type=int, default=64)
- ap.add_argument("--no-onnx", action="store_true")
- ap.add_argument("--models", type=str, nargs="+", default=CANDIDATES)
- args = ap.parse_args()
-
- M = load_module()
- out_dir = Path("/tmp/claude-1000/bench_backbone")
- out_dir.mkdir(parents=True, exist_ok=True)
-
- if DEV.type == "cuda":
- print(f"device: {torch.cuda.get_device_name(0)} torch {torch.__version__}")
- print()
-
- rows = []
- for name in args.models:
- cfg, model = build(M, name)
- nb = sum(p.numel() for p in model.backbone.parameters()) / 1e6
- nt = sum(p.numel() for p in model.parameters()) / 1e6
- with torch.no_grad():
- f = model.backbone.forward_features(
- torch.randn(1, 3, cfg.image_size, cfg.image_size, device=DEV))
- r = {"name": name, "bb_params": nb, "tot_params": nt,
- "feat": tuple(f.shape[1:]), "torch": {}, "bb": {}}
- for b in args.batches:
- r["torch"][b], r["bb"][b] = bench_torch(model, cfg, b, args.iters, amp=True)
- del model
- torch.cuda.empty_cache()
- r["train_ms"], r["train_mem"] = bench_train_step(M, name, args.train_batch, 20)
- rows.append(r)
-
- if not args.no_onnx:
- for r in rows:
- cfg, model = build(M, r["name"])
- try:
- p = export_onnx(M, r["name"], cfg, model, out_dir)
- del model
- torch.cuda.empty_cache()
- r["ort"] = bench_ort(p, cfg, args.batches, args.iters)
- except Exception as e:
- r["ort"] = None
- r["ort_err"] = f"{type(e).__name__}: {e}"
- finally:
- torch.cuda.empty_cache()
-
- bs = args.batches
- print("=" * 118)
- print("PARAMETERS / FEATURE MAP")
- print("-" * 118)
- print(f"{'backbone':30s} {'bb params(M)':>13s} {'total(M)':>10s} {'feat map':>16s}")
- for r in rows:
- print(f"{r['name']:30s} {r['bb_params']:13.2f} {r['tot_params']:10.2f} {str(r['feat']):>16s}")
-
- print()
- print("=" * 118)
- print(f"TORCH fp16 autocast, FULL student forward (ms/iter) -- backbone-only in ()")
- print("-" * 118)
- hdr = "".join(f"{'B=' + str(b):>22s}" for b in bs)
- print(f"{'backbone':30s}{hdr}")
- for r in rows:
- cells = "".join(f"{r['torch'][b]:>13.2f} ({r['bb'][b]:5.2f})" for b in bs)
- print(f"{r['name']:30s}{cells}")
-
- print()
- print("=" * 118)
- print(f"TRAINING STEP fwd+bwd+AdamW @ batch {args.train_batch} (ms) / peak GPU mem (MiB)")
- print("-" * 118)
- print(f"{'backbone':30s} {'ms/step':>10s} {'peak MiB':>10s}")
- for r in rows:
- print(f"{r['name']:30s} {r['train_ms']:10.1f} {r['train_mem']:10.0f}")
-
- if not args.no_onnx:
- print()
- print("=" * 118)
- print("ONNX fp16 / onnxruntime CUDAExecutionProvider (ms/iter)")
- print("-" * 118)
- print(f"{'backbone':30s}" + "".join(f"{'B=' + str(b):>12s}" for b in bs))
- for r in rows:
- if r.get("ort") is None:
- print(f"{r['name']:30s} EXPORT FAILED: {r.get('ort_err','')[:70]}")
- else:
- print(f"{r['name']:30s}" + "".join(f"{r['ort'][b]:12.2f}" for b in bs))
- base = next((r for r in rows if r["name"] == "repvit_m2_3" and r.get("ort")), None)
- if base:
- print()
- print(f"{'speedup vs repvit_m2_3':30s}" + "".join(f"{'B=' + str(b):>12s}" for b in bs))
- for r in rows:
- if r.get("ort"):
- print(f"{r['name']:30s}"
- + "".join(f"{base['ort'][b] / r['ort'][b]:11.2f}x" for b in bs))
- print("=" * 118)
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/bench_student_arch.py b/tools/bench_student_arch.py
deleted file mode 100644
index 1a43549..0000000
--- a/tools/bench_student_arch.py
+++ /dev/null
@@ -1,256 +0,0 @@
-#!/usr/bin/env python3
-"""Component-level benchmark: 3D-head student vs MHR-only student.
-
-Measures backbone / transformer-decoder / heads separately, plus the MHR
-skeleton forward that the MHR-only model needs at runtime to produce its 70
-3D keypoints, and the end-to-end ONNX latency of both exported graphs.
-
- python tools/bench_student_arch.py [--batches 1 8] [--iters 60] [--no-onnx]
-"""
-from __future__ import annotations
-
-import argparse
-import contextlib
-import io
-import sys
-import time
-from pathlib import Path
-
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-TRAIN_DIR = ROOT / "instanthmr_distill_train"
-sys.path.insert(0, str(TRAIN_DIR))
-
-DEV = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-
-
-def timeit(fn, iters, warmup):
- for _ in range(warmup):
- fn()
- if DEV.type == "cuda":
- torch.cuda.synchronize()
- t0 = time.perf_counter()
- for _ in range(iters):
- fn()
- if DEV.type == "cuda":
- torch.cuda.synchronize()
- return (time.perf_counter() - t0) / iters * 1000.0
-
-
-class Staged:
- """Re-implements InstantHMRStudent.forward split into timeable stages."""
-
- def __init__(self, model, cfg, batch):
- self.m, self.cfg, self.B = model, cfg, batch
- self.img = torch.randn(batch, 3, cfg.image_size, cfg.image_size, device=DEV)
- self.cliff = torch.randn(batch, 3, device=DEV)
- with torch.no_grad():
- f = model.backbone.forward_features(self.img)
- if f.dim() == 4:
- f = f.flatten(2).transpose(1, 2)
- self.feats = f
- mem = model.feat_proj(f) + model.mem_pos_embed
- self.mem = mem
- q = model.query_embed.expand(batch, -1, -1) + model.cond_proj(self.cliff).unsqueeze(1)
- self.q = q
- self.shared = model.transformer(tgt=q, memory=mem)
-
- def backbone(self):
- f = self.m.backbone.forward_features(self.img)
- return f.flatten(2).transpose(1, 2) if f.dim() == 4 else f
-
- def project(self):
- mem = self.m.feat_proj(self.feats) + self.m.mem_pos_embed
- q = self.m.query_embed.expand(self.B, -1, -1) + self.m.cond_proj(self.cliff).unsqueeze(1)
- return mem, q
-
- def decoder(self):
- return self.m.transformer(tgt=self.q, memory=self.mem)
-
- def heads(self):
- s = self.shared
- g = self.m.head_global(s[:, 0, :])
- f2 = self.m.head_2d_feat(s[:, 1:1 + self.m.num_2d, :])
- lg = self.m.head_2d_logits(f2).unflatten(-1, (2, self.m.kp2d_bins))
- p = torch.softmax(lg.float(), dim=-1)
- out = (p * self.m.kp2d_bin_centers.float()).sum(dim=-1)
- if hasattr(self.m, "head_3d"):
- out = out.sum() + self.m.head_3d(s[:, 1 + self.m.num_2d:, :]).sum()
- return g, out
-
- def full(self):
- return self.m(self.img, self.cliff)
-
-
-def bench_torch(name, model, cfg, batches, iters, warmup, amp):
- model = model.to(DEV).eval()
- rows = []
- for B in batches:
- st = Staged(model, cfg, B)
- ctx = (lambda: torch.amp.autocast("cuda", dtype=torch.float16)) if amp else contextlib.nullcontext
- def wrap(fn):
- def g():
- with torch.no_grad(), ctx():
- fn()
- return g
- r = dict(
- B=B,
- backbone=timeit(wrap(st.backbone), iters, warmup),
- project=timeit(wrap(st.project), iters, warmup),
- decoder=timeit(wrap(st.decoder), iters, warmup),
- heads=timeit(wrap(st.heads), iters, warmup),
- full=timeit(wrap(st.full), iters, warmup),
- queries=model.total_queries,
- )
- rows.append(r)
- del st
- torch.cuda.empty_cache()
- print(f"\n=== {name} (torch eager, {'fp16 autocast' if amp else 'fp32'}) ===")
- print(f"{'B':>3} {'queries':>8} {'backbone':>10} {'proj':>8} {'decoder':>9} {'heads':>8} {'full fwd':>10}")
- for r in rows:
- print(f"{r['B']:>3} {r['queries']:>8} {r['backbone']:>9.3f}m {r['project']:>7.3f}m "
- f"{r['decoder']:>8.3f}m {r['heads']:>7.3f}m {r['full']:>9.3f}m")
- return rows
-
-
-def bench_mhr(batches, iters, warmup):
- W = np.load(TRAIN_DIR / "assets/mhr_j127_to_kp70.npy")
- Wt = torch.from_numpy(W).to(DEV)
- mhr = torch.jit.load(str(TRAIN_DIR / "checkpoints/mhr_model.pt"), map_location=DEV).eval()
- ct = mhr.character_torch
- out = {}
- for B in batches:
- mp = torch.zeros(B, 204, device=DEV)
- z = torch.zeros(B, 45, device=DEV)
- fe = torch.zeros(B, 72, device=DEV)
- sh = torch.zeros(B, 45, device=DEV)
-
- def skel():
- with torch.no_grad():
- jp = ct.model_parameters_to_joint_parameters(torch.cat([mp, z], 1))
- ss = ct.joint_parameters_to_skeleton_state(jp)[..., :3] / 100.0
- v = torch.stack([ss[..., 0], -ss[..., 1], -ss[..., 2]], -1)
- return torch.einsum('kj,bjc->bkc', Wt, v)
-
- def mesh():
- with torch.no_grad():
- return mhr(sh, mp, fe)
-
- out[B] = (timeit(skel, iters, warmup), timeit(mesh, iters, warmup))
- print("\n=== MHR forward (runtime, no_grad) ===")
- print(f"{'B':>3} {'skeleton+W (70 kp3d)':>22} {'full mesh (LOD1 verts)':>24}")
- for B, (a, b) in out.items():
- print(f"{B:>3} {a:>21.3f}m {b:>23.3f}m")
- return out
-
-
-def export_onnx(model, cfg, path, five_outputs):
- import warnings
- model = model.to(DEV).eval()
-
- class Wrap(torch.nn.Module):
- def __init__(self, m):
- super().__init__()
- self.m = m
-
- def forward(self, image, cliff):
- o = self.m(image, cliff)
- if five_outputs:
- return (o["mhr_params"], o["shape_params"], o["cam_trans"],
- o["joints_2d"], o["joints_3d"])
- return (o["mhr_params"], o["shape_params"], o["cam_trans"], o["joints_2d"])
-
- names = ["mhr_params", "shape_params", "cam_trans", "joints_2d"] + (["joints_3d"] if five_outputs else [])
- dyn = {"image": {0: "b"}, "cliff_cond": {0: "b"}}
- for n in names:
- dyn[n] = {0: "b"}
- with warnings.catch_warnings():
- warnings.simplefilter("ignore")
- torch.onnx.export(Wrap(model),
- (torch.randn(1, 3, cfg.image_size, cfg.image_size, device=DEV),
- torch.randn(1, 3, device=DEV)),
- str(path), input_names=["image", "cliff_cond"],
- output_names=names, dynamic_axes=dyn, opset_version=17)
- import onnx
- from onnxruntime.transformers.float16 import convert_float_to_float16
- m16 = convert_float_to_float16(onnx.load(str(path)), keep_io_types=True)
- p16 = path.with_name(path.stem + "_fp16.onnx")
- onnx.save(m16, str(p16))
- return path, p16
-
-
-def bench_onnx(name, path, cfg, batches, iters, warmup):
- import onnxruntime as ort
- so = ort.SessionOptions()
- so.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALL
- so.log_severity_level = 3
- sess = ort.InferenceSession(str(path), so, providers=["CUDAExecutionProvider", "CPUExecutionProvider"])
- ep = sess.get_providers()[0]
- res = {}
- for B in batches:
- feed = {"image": np.random.randn(B, 3, cfg.image_size, cfg.image_size).astype(np.float32),
- "cliff_cond": np.random.randn(B, 3).astype(np.float32)}
- res[B] = timeit(lambda: sess.run(None, feed), iters, warmup)
- print(f"\n=== ONNX {name} [{ep}] {path.stat().st_size/1e6:.1f} MB ===")
- for B, t in res.items():
- print(f" B={B:<3} {t:8.3f} ms")
- return res
-
-
-def main():
- ap = argparse.ArgumentParser()
- ap.add_argument("--batches", type=int, nargs="+", default=[1, 8])
- ap.add_argument("--iters", type=int, default=60)
- ap.add_argument("--warmup", type=int, default=20)
- ap.add_argument("--no-onnx", action="store_true")
- args = ap.parse_args()
-
- buf = io.StringIO()
- with contextlib.redirect_stdout(buf):
- import train_distill_optimized_correctives as OLD
- import train_distill_mhr_only as NEW
-
- print(f"device: {torch.cuda.get_device_name(0) if DEV.type=='cuda' else 'cpu'} "
- f"| torch {torch.__version__} | iters={args.iters} warmup={args.warmup}")
-
- cfg_old, cfg_new = OLD.DistillConfig(), NEW.DistillConfig()
- m_old = OLD.InstantHMRStudent(cfg_old, pretrained=False)
- m_new = NEW.InstantHMRStudent(cfg_new, pretrained=False)
- n_old = sum(p.numel() for p in m_old.parameters())
- n_new = sum(p.numel() for p in m_new.parameters())
- print(f"params: 3D-head {n_old/1e6:.2f} M ({m_old.total_queries} queries) | "
- f"MHR-only {n_new/1e6:.2f} M ({m_new.total_queries} queries) | "
- f"delta {(n_new-n_old)/1e6:+.2f} M")
-
- for amp in (False, True):
- bench_torch("3D-head student", m_old, cfg_old, args.batches, args.iters, args.warmup, amp)
- bench_torch("MHR-only student", m_new, cfg_new, args.batches, args.iters, args.warmup, amp)
-
- mhr = bench_mhr(args.batches, args.iters, args.warmup)
-
- if args.no_onnx:
- return
- out = Path("/tmp/claude-1000/-home-madjel-Projects-gitpackages-sam3d-student-release/"
- "9cf624e4-b910-464f-9a5c-70fc1323c5a3/scratchpad")
- out.mkdir(parents=True, exist_ok=True)
- p_old32, p_old16 = export_onnx(m_old, cfg_old, out / "student_3dhead.onnx", True)
- p_new32, p_new16 = export_onnx(m_new, cfg_new, out / "student_mhronly.onnx", False)
- r_old = bench_onnx("3D-head fp16", p_old16, cfg_old, args.batches, args.iters, args.warmup)
- r_new = bench_onnx("MHR-only fp16", p_new16, cfg_new, args.batches, args.iters, args.warmup)
-
- print("\n=== END-TO-END, MHR cost included ===")
- print(f"{'B':>3} {'3D-head onnx':>13} {'MHR-only onnx':>14} {'+ skeleton':>11} "
- f"{'new total':>10} {'speedup':>9}")
- for B in args.batches:
- tot = r_new[B] + mhr[B][0]
- print(f"{B:>3} {r_old[B]:>12.3f}m {r_new[B]:>13.3f}m {mhr[B][0]:>10.3f}m "
- f"{tot:>9.3f}m {r_old[B]/tot:>8.2f}x")
- print("\nnote: the 'full mesh' column above is what a mesh-rendering pipeline already")
- print("pays; where the mesh is drawn, the skeleton forward is strictly redundant work")
- print("that can be read off the same call, i.e. the 3D keypoints become free.")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/build_cont_head_assets.py b/tools/build_cont_head_assets.py
index 357bbe7..0571ceb 100644
--- a/tools/build_cont_head_assets.py
+++ b/tools/build_cont_head_assets.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-"""Build `instanthmr_distill_train/assets/mhr_cont_head.npz` for `--cont-head`.
+"""Build `training/assets/mhr_cont_head.npz` for `--cont-head`.
python tools/build_cont_head_assets.py --data_root data
@@ -81,7 +81,7 @@ def main():
ap.add_argument("--teacher-ckpt", type=Path, default=None)
ap.add_argument("--limit", type=int, default=20000)
ap.add_argument("--out", type=Path,
- default=Path("instanthmr_distill_train/assets/mhr_cont_head.npz"))
+ default=Path("training/assets/mhr_cont_head.npz"))
args = ap.parse_args()
buf = teacher_buffers(args.teacher_ckpt)
diff --git a/tools/build_contact_floor.py b/tools/build_contact_floor.py
new file mode 100644
index 0000000..d4215cb
--- /dev/null
+++ b/tools/build_contact_floor.py
@@ -0,0 +1,105 @@
+#!/usr/bin/env python3
+"""Floor plane per video camera, for the height-based contact labels of --contact-floor.
+
+ python tools/build_contact_floor.py --split-dir <root>/sam3d_gt_harmony4d --out <file>.npz
+
+The trainer's motion rule (a foot that moved < contact_thr between paired
+frames is planted, > noncontact_thr is not) fails on Harmony4D: a foot that
+stays within 1 cm of the floor still moves a median 3.1 cm over 5 frames there
+(pivots, slides, label jitter), so 42% of its pairs come out "both feet in the
+air" against ~4% with both feet more than 5 cm up. Height above the floor is
+what separates them, and the floor of a static camera is the plane most foot
+keypoints lie on.
+
+For every camera track (the annotation stem without `_<frame>_p<person>`, the
+key `train.PAIR_NAME` pairs on), RANSAC fits a plane through
+the 8 foot keypoints (joints_3d + cam_trans, camera frame) of all its frames:
+score = points within 2 cm minus 5x points more than 5 cm below it, normal
+pointing to the side most points lie on. Calibrated against insole-measured
+contact (368k foot labels): planted = lowest foot point < 2 cm above the plane
+and moved < 5 cm, in the air = > 5 cm above it agrees with the insoles 97.3%,
+and no insole-loaded foot is ever > 5 cm up.
+
+Output: keys (K,) str, normal (K, 3), point (K, 3), inlier (K,) = fraction of
+foot points within 2 cm, n (K,) = frames. Fits with inlier < --min-inlier are
+dropped, so their samples keep the motion rule.
+"""
+from __future__ import annotations
+
+import argparse
+import collections
+import re
+import sys
+from pathlib import Path
+
+import numpy as np
+
+FOOT = [13, 15, 16, 17, 14, 18, 19, 20]
+PAIR_NAME = re.compile(r"^(.*)_(\d+)_p(\d+)$") # = train.PAIR_NAME
+
+
+def fit_floor(P: np.ndarray, iters: int = 400, seed: int = 0):
+ rng = np.random.default_rng(seed)
+ best = None
+ for _ in range(iters):
+ a, b, c = P[rng.choice(len(P), 3, replace=False)]
+ n = np.cross(b - a, c - a)
+ nn = np.linalg.norm(n)
+ if nn < 1e-6:
+ continue
+ n /= nn
+ d = (P - a) @ n
+ if np.median(d) < 0:
+ n, d = -n, -d
+ score = ((d > -0.02) & (d < 0.02)).sum() - 5 * (d < -0.05).sum()
+ if best is None or score > best[0]:
+ best = (score, n, a, float(((d > -0.02) & (d < 0.02)).mean()))
+ return best[1:]
+
+
+def main() -> int:
+ p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--split-dir", required=True, help="a sam3d_gt_<name> folder (has annotations/)")
+ p.add_argument("--out", required=True)
+ p.add_argument("--min-frames", type=int, default=30)
+ p.add_argument("--min-inlier", type=float, default=0.25)
+ args = p.parse_args()
+
+ groups = collections.defaultdict(list)
+ for f in (Path(args.split_dir) / "annotations").iterdir():
+ m = PAIR_NAME.match(f.stem)
+ if m:
+ groups[m.group(1)].append(f)
+ print(f"{len(groups)} camera tracks in {args.split_dir}", flush=True)
+
+ keys, normals, points, inliers, ns = [], [], [], [], []
+ dropped = 0
+ for i, (k, files) in enumerate(sorted(groups.items())):
+ if len(files) < args.min_frames:
+ dropped += 1
+ continue
+ P = []
+ for f in files:
+ with np.load(f) as z:
+ P.append(z["joints_3d"][FOOT] + z["cam_trans"])
+ P = np.concatenate(P).astype(np.float64)
+ P = P[np.isfinite(P).all(1)]
+ n, a, inl = fit_floor(P)
+ if inl < args.min_inlier:
+ dropped += 1
+ continue
+ keys.append(k); normals.append(n); points.append(a); inliers.append(inl); ns.append(len(files))
+ if (i + 1) % 200 == 0:
+ print(f" {i + 1}/{len(groups)}", flush=True)
+ np.savez(args.out, keys=np.array(keys), normal=np.array(normals, np.float32),
+ point=np.array(points, np.float32), inlier=np.array(inliers, np.float32),
+ n=np.array(ns))
+ inl = np.array(inliers)
+ print(f"wrote {args.out}: {len(keys)} floors ({dropped} tracks dropped: < {args.min_frames} frames "
+ f"or inlier < {args.min_inlier}); inlier fraction pct 10/50/90 = "
+ f"{np.round(np.percentile(inl, [10, 50, 90]), 2) if len(inl) else '-'}")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/tools/build_landmark_assets.py b/tools/build_landmark_assets.py
index 9d5236d..989e77c 100644
--- a/tools/build_landmark_assets.py
+++ b/tools/build_landmark_assets.py
@@ -1,5 +1,5 @@
#!/usr/bin/env python3
-"""Build `instanthmr_distill_train/assets/mhr_landmarks70.npz` for `--exact-landmarks`.
+"""Build `training/assets/mhr_landmarks70.npz` for `--exact-landmarks`.
python tools/build_landmark_assets.py
@@ -48,7 +48,7 @@ def main():
formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("--teacher-ckpt", type=Path, default=None)
ap.add_argument("--out", type=Path,
- default=Path("instanthmr_distill_train/assets/mhr_landmarks70.npz"))
+ default=Path("training/assets/mhr_landmarks70.npz"))
args = ap.parse_args()
ckpt = args.teacher_ckpt
diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.py
index 7e54443..091ae9f 100644
--- a/tools/ddp_smoke.py
+++ b/tools/ddp_smoke.py
@@ -5,10 +5,10 @@
python tools/ddp_smoke.py --backbone repvit_m2_3 # the control
python tools/ddp_smoke.py --image-size 288 # a non-224 input
-WHY THIS EXISTS. A plain single-process smoke run -- `train_distill_jz.py` on a
+WHY THIS EXISTS. A plain single-process smoke run -- `train_ddp.py` on a
laptop, `--self-test`, `--overfit-test` -- builds no DDP Reducer at all, so it
-cannot see the one class of failure that only exists under `srun`: a parameter
-that never receives a gradient. `52_train_ddp.slurm` runs
+cannot see the one class of failure that only exists in a multi-GPU run: a
+parameter that never receives a gradient. `train_ddp.py` under torchrun runs
`DistributedDataParallel` with the default `find_unused_parameters=False`, which
aborts on the SECOND optimiser step with
@@ -44,8 +44,8 @@ import numpy as np
import torch
import torch.distributed as dist
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))
-import train_distill_mhr_only as T # noqa: E402
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "training"))
+import train as T # noqa: E402
def main() -> int:
@@ -63,8 +63,8 @@ def main() -> int:
p.add_argument("--data_root", default="data")
p.add_argument("--batch_size", type=int, default=2,
help="Small on purpose. Which parameters receive a gradient "
- "does not depend on the batch size, and a Jean Zay "
- "LOGIN node kills a process around 3-4 GB: measured peak "
+ "does not depend on the batch size, and a shared "
+ "login node may kill a process around 3-4 GB: measured peak "
"RSS is 4.49 GB for repvit_m2_3 at batch 8 (killed) "
"against 2.45 GB at batch 2. Raise it only if you are on "
"a compute node.")
@@ -83,6 +83,9 @@ def main() -> int:
p.add_argument("--prev-token", dest="prev_token", action="store_true",
help="Foot lock: the previous-pose query, the contact head and "
"the three foot-lock losses, all at weight 1.")
+ p.add_argument("--contact-floor", dest="contact_floor", default="",
+ help="With --prev-token: Harmony4D contact by height (tools/build_contact_floor.py).")
+ p.add_argument("--contact-flight-weight", dest="contact_flight_weight", type=float, default=1.0)
p.add_argument("--pretrained", action="store_true",
help="Load the timm weights too. Off by default: the head "
"geometry this checks does not depend on them.")
@@ -111,6 +114,8 @@ def main() -> int:
if args.prev_token:
cfg.prev_token = cfg.contact_head = True
cfg.w_contact = cfg.w_zero_vel = cfg.w_foot_mocap = 1.0
+ cfg.contact_floor = args.contact_floor
+ cfg.contact_flight_weight = args.contact_flight_weight
if args.backbone:
cfg.backbone = args.backbone
if args.image_size is not None:
diff --git a/tools/parquet_to_npz.py b/tools/parquet_to_npz.py
index 19cc9cf..a7e5c09 100644
--- a/tools/parquet_to_npz.py
+++ b/tools/parquet_to_npz.py
@@ -2,7 +2,7 @@
"""
Convert the official ``facebook/sam-3d-body-dataset`` parquet annotations into the
per-crop ``images/*.png`` + ``annotations/*.npz`` layout consumed by
-``notebooks/distill_transformer_decoder.ipynb``.
+``training/train.py``.
This lets you train InstantHMR directly on the original SAM 3D Body ground-truth
MHR fits, instead of (or alongside) the teacher-distilled pseudo-labels produced
diff --git a/tools/plot_history.py b/tools/plot_history.py
index f3f14ad..91ba753 100644
--- a/tools/plot_history.py
+++ b/tools/plot_history.py
@@ -1,13 +1,12 @@
#!/usr/bin/env python3
"""Plot the per-epoch curves in one or more runs' `history.jsonl`.
- python tools/plot_history.py instanthmr_distill_train/runs/g6bv_s
+ python tools/plot_history.py training/runs/g6bv_s
python tools/plot_history.py .../g6bv_s .../g6vv_s -o /tmp/gen6.png
python tools/plot_history.py $RUNS_DIR/g6*_s --tail 60
-`summarize_runs.py` answers "which run is ahead". This answers "is it still
-going down, and is it healthy" — the question you have to settle before
-resubmitting a chained run for more epochs.
+This answers "is it still going down, and is it healthy" — the question you
+have to settle before relaunching a run for more epochs.
Six panels:
@@ -29,7 +28,7 @@ Six panels:
Ratio below 1 means the average is ahead of the raw weights,
which is the normal state. A spike above 1 is the cheapest
detector there is for an intra-epoch excursion the loss guard
- never saw — see docs/todo.md item 5b.
+ never saw.
"""
from __future__ import annotations
@@ -50,8 +49,8 @@ ROLLBACK_SKIPS = 50
def find_history(p: Path) -> Path | None:
"""history.jsonl for a run, whether given the run dir or its parent.
- 52_train_ddp.slurm writes to $RUNS_DIR/<name>, and train_distill_jz.py
- creates <name>/ again underneath, so the file usually sits one level deeper
+ A launcher that writes to $RUNS_DIR/<name> while train_ddp.py
+ creates <name>/ again underneath leaves so the file usually sits one level deeper
than the directory you name on the command line.
"""
if p.is_file():
diff --git a/tools/profiling/export_family.py b/tools/profiling/export_family.py
deleted file mode 100644
index ebd9edb..0000000
--- a/tools/profiling/export_family.py
+++ /dev/null
@@ -1,77 +0,0 @@
-"""ONNX exports of the g8h architecture with the backbone and input size swapped.
-
- python tools/profiling/export_family.py --out /tmp/fam # ticket 03
- python tools/profiling/export_family.py --out /tmp/fam288 --image-size 288 \
- --backbones b4 --no-trained # ticket 06
-
-Random weights (latency does not depend on them; ticket 03 measured random B4
-within 2-6% of the trained g8h on every phone), plus optionally the trained
-g8h. Network only, no FK / mesh: that is what differs between members.
-Then profile with tools/profiling/profile_family.py.
-"""
-import argparse, copy, io, json, sys
-from pathlib import Path
-import torch, onnx, onnxsim
-REPO = Path(__file__).resolve().parents[2]
-sys.path.insert(0, str(REPO / "instanthmr_distill_train"))
-import train_distill_mhr_only as T
-
-CKPT = REPO / "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_ema.pth"
-
-
-class Deploy(torch.nn.Module):
- def __init__(self, m): super().__init__(); self.m = m
- def forward(self, image, cliff_cond):
- o = self.m(image, cliff_cond)
- return o["mhr_params"], o["shape_params"], o["cam_trans"], o["joints_2d"]
-
-
-def export(model, path: Path, size: int):
- model.eval()
- buf = io.BytesIO()
- torch.onnx.export(Deploy(model), (torch.zeros(1, 3, size, size), torch.zeros(1, 3)), buf,
- input_names=["image", "cliff_cond"],
- output_names=["mhr_params", "shape_params", "cam_trans", "joints_2d"],
- opset_version=17)
- m, ok = onnxsim.simplify(onnx.load_from_string(buf.getvalue()))
- assert ok
- onnx.save(m, path)
- n = sum(x.numel() for x in model.parameters())
- nb = sum(x.numel() for x in model.backbone.parameters())
- print(json.dumps(dict(name=path.stem, image_size=size, params_M=round(n / 1e6, 2),
- backbone_M=round(nb / 1e6, 2))), flush=True)
-
-
-def main():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--out", type=Path, required=True)
- p.add_argument("--image-size", type=int, default=224)
- p.add_argument("--backbones", default="b4,b1,b0", help="hgnetv2 sizes, comma-separated")
- p.add_argument("--no-trained", action="store_true", help="skip the trained g8h export")
- args = p.parse_args()
- args.out.mkdir(parents=True, exist_ok=True)
-
- blob = torch.load(CKPT, map_location="cpu", weights_only=False)
- state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}
- cfg, _ = T.config_from_checkpoint(state, CKPT)
- cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2
- cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])
-
- if not args.no_trained:
- if args.image_size != cfg.image_size:
- sys.exit("the trained g8h is a 224 model; pass --no-trained with --image-size")
- trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)
- missing, unexpected = trained.load_state_dict(state, strict=False)
- assert not missing and not unexpected, (missing[:5], unexpected[:5])
- export(trained, args.out / "g8h_trained_b4.onnx", args.image_size)
- for b in args.backbones.split(","):
- c = copy.deepcopy(cfg)
- c.backbone, c.image_size = f"hgnetv2_{b}", args.image_size
- torch.manual_seed(0)
- export(T.InstantHMRStudent(c, pretrained=False),
- args.out / f"random_{b}_{args.image_size}.onnx", args.image_size)
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/profiling/profile_family.py b/tools/profiling/profile_family.py
deleted file mode 100644
index c6b166a..0000000
--- a/tools/profiling/profile_family.py
+++ /dev/null
@@ -1,64 +0,0 @@
-"""Compile every .onnx in a directory for the S23 (ORT runtime) and profile it on
-AI Hub: NPU on the Snapdragon 8-series phones, CPU-only on the reference phones.
-
- python tools/profiling/profile_family.py --dir /tmp/fam288 --image-size 288
-
-Writes <dir>/profile.json (one row per model x device x unit) and prints rows.
-"""
-import argparse, json
-from pathlib import Path
-import qai_hub as hub
-
-CPU = ["Samsung Galaxy S23", "Samsung Galaxy A73 5G", "Samsung Galaxy A53 5G",
- "Samsung Galaxy A14 5G", "Google Pixel 8"]
-NPU = ["Samsung Galaxy S22 5G", "Samsung Galaxy S23", "Samsung Galaxy S24", "Samsung Galaxy S25"]
-
-
-def main():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--dir", type=Path, required=True)
- p.add_argument("--image-size", type=int, default=224)
- p.add_argument("--npu", default="S23", help="'S23' or 'all' Snapdragon 8-series phones")
- args = p.parse_args()
- s = args.image_size
- specs = dict(image=(1, 3, s, s), cliff_cond=(1, 3))
- npu = NPU if args.npu == "all" else ["Samsung Galaxy S23"]
-
- comp = {f.stem: hub.submit_compile_job(model=str(f), device=hub.Device("Samsung Galaxy S23"),
- input_specs=specs, options="--target_runtime onnx",
- name=f"fam-{f.stem}")
- for f in sorted(args.dir.glob("*.onnx"))}
- jobs = []
- for n, cj in comp.items():
- m = cj.get_target_model()
- if m is None:
- print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True)
- continue
- for d in npu:
- jobs.append((n, "npu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
- name=f"fam-{n}-npu")))
- for d in CPU:
- jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),
- name=f"fam-{n}-cpu", options="--compute_unit cpu")))
- out = []
- for n, u, d, j in jobs:
- row = dict(model=n, image_size=s, unit=u, device=d, job=j.job_id)
- if j.wait().success:
- prof = j.download_profile()
- summ = prof["execution_summary"]
- units = {}
- for layer in prof.get("execution_detail", []):
- units[layer.get("compute_unit")] = units.get(layer.get("compute_unit"), 0) + 1
- row.update(ms=round(summ["estimated_inference_time"] / 1000, 2),
- peak_mb=round(summ.get("estimated_inference_peak_memory", 0) / 2**20, 1),
- layers=units)
- else:
- row["error"] = j.get_status().message
- print(json.dumps(row), flush=True)
- out.append(row)
- json.dump(out, open(args.dir / "profile.json", "w"), indent=1)
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/profiling/time_onnx_local.py b/tools/profiling/time_onnx_local.py
deleted file mode 100644
index 54dfc9e..0000000
--- a/tools/profiling/time_onnx_local.py
+++ /dev/null
@@ -1,78 +0,0 @@
-#!/usr/bin/env python3
-"""Model-only latency of exported InstantHMR graphs on this machine.
-
-Batch 1, the graph alone (no detector, no MHR forward pass, no crop), ONNX
-Runtime with the CPU provider (all cores, then one thread) and the CUDA
-provider. Median and p90 over --iters runs after --warmup. The input size is
-read from the ONNX metadata, as instanthmr.inference does. Run it on an idle
-machine: anything else on the GPU or the cores shows up in the numbers.
-
- python tools/profiling/time_onnx_local.py models/family/*.onnx --out bench.json
-"""
-
-from __future__ import annotations
-
-import argparse
-import json
-import platform
-import time
-from pathlib import Path
-
-import numpy as np
-import torch # noqa: F401 loads the cuDNN onnxruntime's CUDA provider needs; without it
-import onnxruntime as ort # the provider fails to load and ORT silently runs on the CPU
-
-
-def time_session(path: str, providers, threads: int | None, warmup: int, iters: int):
- so = ort.SessionOptions()
- if threads:
- so.intra_op_num_threads = threads
- sess = ort.InferenceSession(path, so, providers=providers)
- if sess.get_providers()[0] != providers[0]:
- raise SystemExit(f"{providers[0]} did not load (got {sess.get_providers()}); "
- "refusing to report a CPU fallback as a GPU number")
- size = int(sess.get_modelmeta().custom_metadata_map.get("image_size", 224))
- feeds = {"image": np.random.randn(1, 3, size, size).astype(np.float32),
- "cliff_cond": np.zeros((1, 3), dtype=np.float32)}
- for _ in range(warmup):
- sess.run(None, feeds)
- t = []
- for _ in range(iters):
- t0 = time.perf_counter()
- sess.run(None, feeds)
- t.append((time.perf_counter() - t0) * 1e3)
- return size, float(np.median(t)), float(np.percentile(t, 90))
-
-
-def main():
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("onnx", nargs="+")
- p.add_argument("--warmup", type=int, default=30)
- p.add_argument("--iters", type=int, default=200)
- p.add_argument("--out", default=None)
- args = p.parse_args()
-
- modes = [("cpu_all", ["CPUExecutionProvider"], None),
- ("cpu_1t", ["CPUExecutionProvider"], 1),
- ("cuda", ["CUDAExecutionProvider", "CPUExecutionProvider"], None)]
- rows = []
- print(f"{'model':<20s} {'in':>4s}" + "".join(f" {m:>16s}" for m, _, _ in modes))
- for path in args.onnx:
- row = {"model": Path(path).stem}
- for m, prov, th in modes:
- size, med, p90 = time_session(path, prov, th, args.warmup, args.iters)
- row["image_size"], row[m] = size, {"median_ms": med, "p90_ms": p90}
- rows.append(row)
- print(f"{row['model']:<20s} {row['image_size']:>4d}"
- + "".join(f" {row[m]['median_ms']:8.2f} ({row[m]['p90_ms']:5.1f})"
- for m, _, _ in modes), flush=True)
- if args.out:
- Path(args.out).write_text(json.dumps(dict(
- machine=dict(cpu=platform.processor() or platform.machine(),
- onnxruntime=ort.__version__),
- rows=rows), indent=2))
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/pth_to_onnx.py b/tools/pth_to_onnx.py
index b287c48..e6ad1f4 100644
--- a/tools/pth_to_onnx.py
+++ b/tools/pth_to_onnx.py
@@ -11,33 +11,20 @@ Cell 15, so the graph I/O matches ``instanthmr/inference.py``:
size (224 unless trained with --image-size), stamped as image_size
outputs: mhr_params, shape_params, cam_trans, joints_2d[, joints_3d]
-Architecture selection
-----------------------
-The student head changed twice across training scripts:
-
- * ``train_distill.py`` regresses 2D joints directly (``head_2d_xy``);
- * the optimized / correctives scripts predict them with a SimCC
- classification head (``head_2d_logits`` + ``kp2d_bin_centers``);
- * ``train_distill_mhr_only.py`` additionally **deletes the 3D head**
- (``head_3d``) — the 70 keypoints are derived from the MHR forward pass by
- the consumer, so its graph has FOUR outputs instead of five.
-
-Exporting a SimCC checkpoint through the old architecture silently leaves
-``head_2d_xy`` at its near-zero init, which collapses every 2D keypoint onto
-the centre of the bbox. ``--arch auto`` (the default) picks the right module by
-inspecting the checkpoint, and a missing-key check makes any future mismatch a
-hard error instead of a silent one. The output list is read off the deploy
-wrapper itself, so a 4- vs 5-output graph never has to be configured by hand.
+Architecture
+------------
+Only checkpoints of ``training/train.py`` are supported: a SimCC 2D head
+(``head_2d_logits``) and no 3D head, since the 70 keypoints come from the MHR
+forward pass, so the graph has FOUR outputs. Checkpoints of the earlier
+trainers (a ``head_3d`` or a ``head_2d_xy`` head) are rejected with an error:
+exporting them through this architecture would silently leave a head at its
+initial weights. A missing-key check makes any other mismatch a hard error.
Usage
-----
- # default: convert the EMA checkpoint -> models/instanthmr.onnx
- python tools/pth_to_onnx.py
-
- # or point at a specific checkpoint / output
- python ./tools/pth_to_onnx.py \
- --ckpt ./instanthmr_distill_train/runs/g8h_lr1x_e600/g8h_lr1x_e600/best_student_model_v3.pth \
- --output ./models/instanthmr_g8h_lr1x_e600.onnx
+ python tools/pth_to_onnx.py \
+ --ckpt training/runs/<run>/best_student_model_v3.pth \
+ --output models/instanthmr_<run>.onnx
"""
import argparse
import importlib
@@ -47,17 +34,14 @@ from pathlib import Path
import torch
REPO = Path(__file__).resolve().parent.parent
-TRAIN_DIR = REPO / "instanthmr_distill_train"
+TRAIN_DIR = REPO / "training"
# The model class + deploy wrapper live in the standalone training scripts.
sys.path.insert(0, str(TRAIN_DIR))
# arch name -> training module that defines the matching InstantHMRStudent.
ARCH_MODULES = {
- "mhr_only": "train_distill_mhr_only", # SimCC 2D head, NO 3D head
- "correctives": "train_distill_optimized_correctives", # SimCC 2D head (head_2d_logits)
- "optimized": "train_distill_optimized", # SimCC 2D head, identical weights
- "legacy": "train_distill", # direct 2D regression (head_2d_xy)
+ "mhr_only": "train", # SimCC 2D head, NO 3D head
}
# Canonical ONNX output order. The MHR-only graph stops after `joints_2d`.
@@ -70,32 +54,24 @@ def detect_arch(state: dict) -> str:
has_simcc = "head_2d_logits.weight" in state
if has_simcc and not has_3d:
- # train_distill_mhr_only.py: the 3D coordinate head was removed and the
+ # train.py: the 3D coordinate head was removed and the
# decoder shrank from 141 to 71 queries.
return "mhr_only"
- if has_simcc:
- # `optimized` and `correctives` share identical parameters (they differ
- # only by an fp32 softmax cast), so either module loads such a
- # checkpoint; prefer the newer one.
- return "correctives"
- if "head_2d_xy.0.weight" in state:
- return "legacy"
+ if has_3d or "head_2d_xy.0.weight" in state:
+ raise SystemExit(
+ "[error] this checkpoint comes from an earlier trainer (it has a 'head_3d' or "
+ "'head_2d_xy' head), which this release does not include.")
raise SystemExit(
- "[error] cannot infer architecture: checkpoint has neither "
- "'head_2d_logits.weight' nor 'head_2d_xy.0.weight'. Pass --arch explicitly."
- )
+ "[error] cannot infer architecture: checkpoint has no 'head_2d_logits.weight'.")
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__,
formatter_class=argparse.RawDescriptionHelpFormatter)
- ap.add_argument("--ckpt", type=Path,
- default=TRAIN_DIR / "runs/distill_optimized_v2/best_student_model_ema.pth",
- help="Path to the .pth checkpoint (default: the EMA checkpoint).")
+ ap.add_argument("--ckpt", type=Path, required=True,
+ help="Path to the .pth checkpoint written by training/train.py or train_ddp.py.")
ap.add_argument("--output", type=Path, default=REPO / "models" / "instanthmr.onnx",
help="Where to write the ONNX (default: models/instanthmr.onnx).")
- ap.add_argument("--arch", choices=["auto", *ARCH_MODULES], default="auto",
- help="Student architecture / training module (default: auto-detect).")
ap.add_argument("--allow-missing", action="store_true",
help="Export even if checkpoint weights are missing (NOT recommended).")
ap.add_argument("--opset", type=int, default=17)
@@ -117,7 +93,7 @@ def main() -> None:
state = ckpt.get("model_state_dict", ckpt)
state = {k.replace("_orig_mod.", ""): v for k, v in state.items()} # strip torch.compile prefix
- arch = detect_arch(state) if args.arch == "auto" else args.arch
+ arch = detect_arch(state)
T = importlib.import_module(ARCH_MODULES[arch])
print(f"Architecture: {arch} (from {ARCH_MODULES[arch]}.py)")
@@ -126,11 +102,7 @@ def main() -> None:
# parameters inside forward(); exporting without it leaves the head's raw
# pre-tanh numbers to be read as bone scales, which is 21.0 mm MPJPE /
# 14.4 mm PA-MPJPE of silent corruption. See T.config_from_checkpoint.
- if hasattr(T, "config_from_checkpoint"):
- cfg, prov = T.config_from_checkpoint(
- state, args.ckpt, cliff_focal=args.cliff_focal)
- else: # older training modules predate the helper
- cfg, prov = T.DistillConfig(), {}
+ cfg, prov = T.config_from_checkpoint(state, args.ckpt, cliff_focal=args.cliff_focal)
print(f" bound_scales : {getattr(cfg, 'bound_scales', False)} "
f"({prov.get('bound_scales', 'default')})")
print(f" cliff_focal : {getattr(cfg, 'cliff_focal', False)} "
@@ -167,9 +139,8 @@ def main() -> None:
sys.exit(
f"\n[error] {len(missing)} parameter(s) had no weights in the checkpoint, so they "
f"would be exported at random init.\n"
- f" This usually means --arch is wrong for this checkpoint "
- f"(detected: {arch}).\n"
- f" Re-run with the right --arch, or --allow-missing to force it."
+ f" This usually means the checkpoint does not match training/train.py's\n"
+ f" model. --allow-missing forces the export."
)
deploy = T.HMRDeployWrapper(model).eval()
@@ -187,7 +158,7 @@ def main() -> None:
print(f" prev_token : {prev_token}")
# Ask the wrapper how many tensors it returns instead of hard-coding five:
- # train_distill_mhr_only.py drops `joints_3d` from the tuple.
+ # train.py drops `joints_3d` from the tuple.
with torch.no_grad():
n_out = len(deploy(*inputs))
if n_out not in (4, 5):
@@ -263,7 +234,7 @@ def main() -> None:
if j2d.std() < 0.01:
print(f"\n[warning] joints_2d std is {j2d.std():.5f} — the 2D keypoints are collapsed "
f"onto the crop centre.\n The exported 2D head is almost certainly "
- f"untrained (wrong --arch?).")
+ f"untrained.")
except Exception as e: # noqa: BLE001
print(f" (onnxruntime sanity check skipped: {e})")
diff --git a/tools/verify_cont_head.py b/tools/verify_cont_head.py
index 0744b67..3ae24a2 100644
--- a/tools/verify_cont_head.py
+++ b/tools/verify_cont_head.py
@@ -21,8 +21,7 @@ Six checks, all on real ground truth and the real rig:
random poses, and at the two places the geometry is singular: identity and a
half turn.
5. **Root-loss magnitude.** The chordal rotation loss replaces a SmoothL1 on the
- Euler triple. `docs/todo.md` asks for the gradient comparison explicitly, so
- it is printed rather than assumed.
+ Euler triple. The gradient comparison is printed rather than assumed.
6. **Scale stability** under a large random head output: the failure mode that
killed four runs at epochs 24-32 was a bone scale running away, and the PCA
parameterisation changes how that can happen.
@@ -40,9 +39,9 @@ from pathlib import Path
import numpy as np
import torch
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "training"))
import mhr_cont as MC # noqa: E402
-import train_distill_mhr_only as T # noqa: E402
+import train as T # noqa: E402
OK, BAD = " ok ", "*FAIL*"
_fails = 0
@@ -224,7 +223,7 @@ def main() -> int:
f"loss {e.mean():.4f}, max |grad| {six.grad.abs().max():.3e}")
# --- 5. root-loss magnitude vs the SmoothL1 it replaces -----------------
- print("\nroot-loss magnitude (docs/todo.md asks for this explicitly):")
+ print("\nroot-loss magnitude:")
err_deg = [1.0, 5.0, 30.0, 90.0, 179.0]
print(f" {'error':>10} | {'chordal /6':>10} | {'SmoothL1(beta=1) on the Euler triple':>38}")
for d in err_deg:
diff --git a/tools/verify_context_crops.py b/tools/verify_context_crops.py
deleted file mode 100644
index b4787c0..0000000
--- a/tools/verify_context_crops.py
+++ /dev/null
@@ -1,246 +0,0 @@
-#!/usr/bin/env python3
-"""Verify the crop builder's --context option on a real split.
-
-Builds the same --max-samples rows three times, into a scratch directory:
-
- head the builder as committed at HEAD, default flags
- default the working-tree builder, default flags
- context the working-tree builder, --context 2.0 --body-size 448
-
-and checks, from the outside (files on disk only):
-
-1. default is byte-identical to head: crops, and every npz array.
-2. context records its geometry: bbox_context is the --context square around the
- same person, and bbox_square is still today's 1.2x square.
-3. The 1.2x window cut out of the context crop and resized to 224 shows the
- same pixels as today's 224 crop (PSNR on every row; the black frame-edge
- padding both builders produce is masked out).
-4. That window is not shifted against today's crop (sub-pixel, by phase
- correlation), so the recorded geometry is the geometry the pixels were cut
- with. The keypoints themselves are unchanged from HEAD and land inside the
- context crop when mapped through bbox_context.
-5. context_padded is set exactly when that square leaves the frame, and
- the padding it produced is black.
-
- python tools/verify_context_crops.py --split aic_train \
- --annotation-dir ~/sam3d_data/annotations/aic_train \
- --image-dir ~/sam3d_data/images/aic --n 64
-"""
-
-from __future__ import annotations
-
-import argparse
-import os
-import subprocess
-import sys
-import tempfile
-from pathlib import Path
-
-import cv2
-import numpy as np
-
-REPO = Path(__file__).resolve().parent.parent
-BUILDER = REPO / "datasets_pipeline" / "build_split.py"
-
-
-def build(builder: Path, out: Path, args, extra: list[str]) -> None:
- env = dict(os.environ, PYTHONPATH=f"{REPO / 'datasets_pipeline'}:{REPO}")
- cmd = [sys.executable, str(builder), "--split", args.split,
- "--annotation-dir", str(args.annotation_dir),
- "--image-dir", str(args.image_dir), "--output-dir", str(out),
- "--max-samples", str(args.n), "--originals", "none", "--crop-format", "png",
- *extra]
- subprocess.run(cmd, check=True, env=env, stdout=subprocess.DEVNULL)
-
-
-def crops(d: Path) -> dict[str, Path]:
- return {p.stem: p for p in (d / "body_crops").glob("*.png")}
-
-
-def load(d: Path, name: str) -> dict:
- with np.load(d / "annotations" / f"{name}.npz", allow_pickle=True) as z:
- return {k: z[k] for k in z.files}
-
-
-def to_px(xy: np.ndarray, box: np.ndarray, size: int) -> np.ndarray:
- """Full-frame pixels -> pixels of a crop of `box` resized to `size`."""
- x1, y1, x2, y2 = box.astype(np.float64)
- return np.stack([(xy[:, 0] - x1) * size / (x2 - x1),
- (xy[:, 1] - y1) * size / (y2 - y1)], axis=1)
-
-
-def same(a: np.ndarray, b: np.ndarray) -> bool:
- """Array equality where NaN == NaN (unwritten hand crops are NaN boxes)."""
- if a.dtype.kind == "f" and b.dtype.kind == "f":
- return np.array_equal(a, b, equal_nan=True)
- return np.array_equal(a, b)
-
-
-def window_warp(sq: np.ndarray, ctx: np.ndarray, src: int, dst: int) -> np.ndarray:
- """Inverse affine (dst pixel -> context-crop pixel) that cuts the `sq` window
- out of a `src`-px crop of `ctx` at `dst` px. Pixel CENTRES, as cv2.resize
- maps them: frame x = box_x1 + (i + 0.5) * side / size - 0.5 on both sides."""
- s1 = (sq[2:] - sq[:2]) / dst
- s2 = (ctx[2:] - ctx[:2]) / src
- a = s1 / s2
- b = (sq[:2] - ctx[:2] + 0.5 * s1) / s2 - 0.5
- return np.float32([[a[0], 0, b[0]], [0, a[1], b[1]]])
-
-
-def shift(a: np.ndarray, b: np.ndarray) -> float:
- """Sub-pixel translation between two images. cv2.phaseCorrelate reports a
- constant (0.5, 0.5) for an image against itself, so measure relative to that."""
- ga = cv2.cvtColor(a, cv2.COLOR_BGR2GRAY).astype(np.float64)
- gb = cv2.cvtColor(b, cv2.COLOR_BGR2GRAY).astype(np.float64)
- (dx, dy), _ = cv2.phaseCorrelate(ga, gb)
- (bx, by), _ = cv2.phaseCorrelate(gb, gb)
- return float(np.hypot(dx - bx, dy - by))
-
-
-def psnr(a: np.ndarray, b: np.ndarray) -> float:
- mse = np.mean((a.astype(np.float64) - b.astype(np.float64)) ** 2)
- return float("inf") if mse == 0 else 10 * np.log10(255.0 ** 2 / mse)
-
-
-def main() -> None:
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--split", default="aic_train")
- p.add_argument("--annotation-dir", type=Path, required=True)
- p.add_argument("--image-dir", type=Path, required=True)
- p.add_argument("--n", type=int, default=64)
- p.add_argument("--context", type=float, default=2.0)
- p.add_argument("--size", type=int, default=448)
- args = p.parse_args()
-
- tmp = Path(tempfile.mkdtemp(prefix="verify_context_"))
- head_builder = tmp / "build_split_head.py"
- head_builder.write_text(subprocess.run(
- ["git", "-C", str(REPO), "show", "HEAD:datasets_pipeline/build_split.py"],
- check=True, capture_output=True, text=True).stdout)
-
- d_head, d_def, d_ctx = tmp / "head", tmp / "default", tmp / "context"
- build(head_builder, d_head, args, [])
- build(BUILDER, d_def, args, [])
- build(BUILDER, d_ctx, args,
- ["--context", str(args.context), "--body-size", str(args.size)])
-
- fails: list[str] = []
-
- # 1. default == HEAD
- ch, cd = crops(d_head), crops(d_def)
- if ch.keys() != cd.keys() or not ch:
- fails.append(f"default: crop set differs from HEAD ({len(ch)} vs {len(cd)})")
- n_diff = sum(ch[k].read_bytes() != cd[k].read_bytes() for k in ch.keys() & cd.keys())
- hands_h = sorted(p.name for p in (d_head / "hands_crops").glob("*"))
- hands_d = sorted(p.name for p in (d_def / "hands_crops").glob("*"))
- n_diff += sum((d_head / "hands_crops" / f).read_bytes()
- != (d_def / "hands_crops" / f).read_bytes()
- for f in set(hands_h) & set(hands_d))
- npz_diff = 0
- for k in ch.keys() & cd.keys():
- a, b = load(d_head, k), load(d_def, k)
- if a.keys() != b.keys() or any(not same(a[x], b[x]) for x in a):
- npz_diff += 1
- print(f"[1] default vs HEAD: {len(ch)} crops, {n_diff} differing image files, "
- f"{npz_diff} differing npz, hand crops {len(hands_h)} vs {len(hands_d)}")
- if n_diff or npz_diff or hands_h != hands_d:
- fails.append("default output is not byte-identical to HEAD")
-
- # 2-5. context
- cc = crops(d_ctx)
- if cc.keys() != ch.keys():
- fails.append(f"context: crop set differs ({len(cc)} vs {len(ch)})")
- geo_err, max_shift, ps, pad_bad, pad_rows = 0.0, 0.0, [], 0, 0
- kp_moved, kp_outside, kp_outside_today = 0, 0, 0
- for k in sorted(cc.keys() & ch.keys()):
- old, new = load(d_head, k), load(d_ctx, k)
- if "bbox_context" not in new or "context_padded" not in new:
- fails.append(f"{k}: bbox_context / context_padded missing")
- break
- sq, ctx = old["bbox_square"].astype(np.float64), new["bbox_context"].astype(np.float64)
- geo_err = max(geo_err, float(np.abs(new["bbox_square"] - old["bbox_square"]).max()))
- # Against the tight person box both squares are built from. The helper
- # int()-truncates each edge, so every edge may sit up to 1 px inside.
- tb = new["bbox"].astype(np.float64)
- half = max(tb[2] - tb[0], tb[3] - tb[1]) * args.context / 2
- exact = np.array([(tb[0] + tb[2]) / 2 - half, (tb[1] + tb[3]) / 2 - half,
- (tb[0] + tb[2]) / 2 + half, (tb[1] + tb[3]) / 2 + half])
- geo_err = max(geo_err, float(np.abs(ctx - exact).max()))
-
- img_old = cv2.imread(str(ch[k]))
- img_ctx = cv2.imread(str(cc[k]))
- s = img_ctx.shape[0]
-
- # keypoints: unchanged from HEAD; every visible keypoint inside today's
- # 1.2x crop is inside the context crop. (Some annotation boxes do not
- # enclose all keypoints, so a few already fall outside today's crop;
- # that is reported, not failed.)
- if not np.array_equal(old["joints_2d"], new["joints_2d"]):
- kp_moved += 1
- jf = new["joints_2d"].astype(np.float64)
- vis = new["joints_2d_vis"] > 0
- j_ctx, j_sq = to_px(jf, ctx, s), to_px(jf, sq, 224)
- in_sq = vis & ((j_sq >= 0) & (j_sq <= 224)).all(axis=1)
- kp_outside += int(((j_ctx[in_sq] < 0) | (j_ctx[in_sq] > s)).any())
- kp_outside_today += int((vis & ~in_sq).any())
-
- h, w = (int(v) for v in new["orig_shape"][:2])
- leaves = ctx[0] < 0 or ctx[1] < 0 or ctx[2] > w or ctx[3] > h
- pad_rows += bool(leaves)
- if bool(new["context_padded"]) != leaves:
- pad_bad += 1
- if leaves:
- # every pixel mapped from outside the frame must be black
- u = (np.arange(s) + 0.5) * (ctx[2] - ctx[0]) / s + ctx[0]
- v = (np.arange(s) + 0.5) * (ctx[3] - ctx[1]) / s + ctx[1]
- out = ((u[None, :] < -1) | (u[None, :] > w + 1)
- | (v[:, None] < -1) | (v[:, None] > h + 1))
- if out.any() and img_ctx[out].max() > 0:
- pad_bad += 1
- M = window_warp(sq, ctx, s, img_old.shape[0])
- cut = cv2.warpAffine(img_ctx, M, img_old.shape[1::-1],
- flags=cv2.INTER_AREA | cv2.WARP_INVERSE_MAP)
- # compare real pixels only: drop the frame-edge padding (and a 4-px
- # band around it, where resampling blends image and black)
- real = cv2.erode((img_old.max(axis=2) > 0).astype(np.uint8), np.ones((9, 9)))
- real[:4], real[-4:], real[:, :4], real[:, -4:] = 0, 0, 0, 0
- # Blur both first: shrinking a large person to 224 with INTER_LINEAR
- # aliases, and the two routes alias differently. Alignment, not
- # resampling texture, is what this check is about.
- if real.sum() > 0.25 * real.size:
- b_cut = cv2.GaussianBlur(cut, (0, 0), 1.0)
- b_old = cv2.GaussianBlur(img_old, (0, 0), 1.0)
- ps.append(psnr(b_cut[real > 0], b_old[real > 0]))
- max_shift = max(max_shift, shift(cut, img_old))
-
- n_old = cv2.imread(str(next(iter(ch.values())))).shape[0]
- print(f"[2] geometry: max error {geo_err:.3f} px (1.2x square unchanged; "
- f"{args.context}x square vs exact, to int() rounding)")
- print(f"[3] 1.2x window vs today's crop: PSNR mean {np.mean(ps):.1f} dB, "
- f"min {np.min(ps):.1f} dB over {len(ps)} of {len(cc)} rows (sigma-1 blur, padding masked)")
- print(f"[4] window shift vs today's crop: max {max_shift:.3f} px ({n_old}-px units); "
- f"keypoints changed in {kp_moved} rows; in-crop keypoints lost by the context "
- f"crop in {kp_outside}; rows with keypoints already outside today's crop: "
- f"{kp_outside_today}")
- print(f"[5] context_padded: {pad_rows} of {len(cc)} rows leave the frame, {pad_bad} wrong")
- if geo_err > 1.0 + 1e-6:
- fails.append(f"geometry error {geo_err:.2f} px")
- if not ps or np.min(ps) < 30.0:
- fails.append(f"window PSNR too low ({np.min(ps) if ps else 'n/a'})")
- if max_shift > 0.25:
- fails.append(f"window shifted by {max_shift:.3f} px against today's crop")
- if kp_moved or kp_outside:
- fails.append(f"keypoints changed in {kp_moved} rows, lost by the context crop in {kp_outside}")
- if pad_bad:
- fails.append(f"{pad_bad} rows with wrong context_padded / non-black padding")
-
- print(f"scratch: {tmp}")
- if fails:
- print("FAIL\n " + "\n ".join(fails))
- sys.exit(1)
- print("PASS")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/verify_exact_landmarks.py b/tools/verify_exact_landmarks.py
index 4ebfbd9..3ec0430 100644
--- a/tools/verify_exact_landmarks.py
+++ b/tools/verify_exact_landmarks.py
@@ -37,8 +37,8 @@ from pathlib import Path
import numpy as np
import torch
-sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "instanthmr_distill_train"))
-import train_distill_mhr_only as T # noqa: E402
+sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "training"))
+import train as T # noqa: E402
OK, BAD = " ok ", "*FAIL*"
_fails = 0
diff --git a/tools/verify_input_size.py b/tools/verify_input_size.py
deleted file mode 100644
index 88fed6b..0000000
--- a/tools/verify_input_size.py
+++ /dev/null
@@ -1,234 +0,0 @@
-#!/usr/bin/env python3
-"""Verify that the network input size is one checkpoint setting, end to end.
-
- python tools/verify_input_size.py --ref <reference train_distill_mhr_only.py> \
- --data_root data/sam3d_gt_coco --emdb-root ~/Downloads/EMDB_root \
- --dpw-seq ~/Downloads/sequenceFiles/sequenceFiles --dpw-img ~/Downloads/imageFiles
-
---ref is the trainer as it was before the change (default: HEAD). Checks:
-
-1. baseline and v2 augmented samples are bit-identical to --ref at 224,
- compared on alternating calls (the rig's first call differs by ~5e-10).
-2. A 288 checkpoint round-trips: config_from_checkpoint reads 288 from the
- weights (the positional-embedding grid) with or without run_config.json,
- and refuses a run_config.json that contradicts the weights.
-3. The 3DPW / EMDB crop builder emits the configured size, and at 224 is
- byte-identical to --ref's.
-4. The ONNX export of a 288 checkpoint carries image_size=288 in its metadata,
- and the inference package builds 288 crops from it on a real frame.
-5. The EMDB and 3DPW harnesses run a 288 checkpoint end to end.
-"""
-
-from __future__ import annotations
-
-import argparse
-import importlib.util
-import json
-import random
-import subprocess
-import sys
-import tempfile
-from pathlib import Path
-
-import cv2
-import numpy as np
-import torch
-
-REPO = Path(__file__).resolve().parent.parent
-TRAIN = REPO / "instanthmr_distill_train"
-sys.path.insert(0, str(TRAIN))
-sys.path.insert(0, str(REPO))
-import train_distill_mhr_only as T # noqa: E402
-import val3dpw # noqa: E402
-
-G8H = TRAIN / "runs/g8h_s0/g8h_s0/best_student_model_v3.pth"
-fails: list[str] = []
-
-
-def check(ok: bool, msg: str) -> None:
- print(f" {'OK ' if ok else 'BAD'} {msg}")
- if not ok:
- fails.append(msg)
-
-
-def load_module(path: Path, name: str):
- spec = importlib.util.spec_from_file_location(name, path)
- mod = importlib.util.module_from_spec(spec)
- spec.loader.exec_module(mod)
- return mod
-
-
-def dataset(mod, root: str, v2: bool, augment: bool = True, **size):
- cfg = mod.DistillConfig()
- cfg.cliff_focal = True
- cfg.crop_centre_fix = True
- if v2:
- jz = load_module(TRAIN / "train_distill_jz.py", "jz_for_preset")
- jz.apply_v2_preset(cfg)
- return mod.SAM3DStudentDataset(
- root, augment=augment, max_images=64, geom_p=cfg.geom_p,
- geom_rot_deg=cfg.geom_rot_deg, geom_scale_range=cfg.geom_scale_range,
- geom_trans=cfg.geom_trans, geom_flip_p=cfg.geom_flip_p,
- geom_scale_max=cfg.geom_scale_max, cliff_follows_aug=cfg.cliff_follows_aug,
- crop_centre_fix=cfg.crop_centre_fix, cliff_focal=cfg.cliff_focal,
- occl_p=cfg.occl_p, occl_scale=cfg.occl_scale, jpeg_p=cfg.jpeg_p,
- jpeg_quality=cfg.jpeg_quality, **size)
-
-
-def sample(ds, i: int, seed: int) -> dict:
- random.seed(seed)
- np.random.seed(seed)
- torch.manual_seed(seed)
- return ds[i]
-
-
-def same(a, b) -> bool:
- if isinstance(a, torch.Tensor):
- return torch.equal(a, b)
- if isinstance(a, np.ndarray):
- return np.array_equal(a, b, equal_nan=a.dtype.kind == "f")
- if isinstance(a, dict):
- return a.keys() == b.keys() and all(same(a[k], b[k]) for k in a)
- return a == b
-
-
-def g8h_cfg():
- st = torch.load(G8H, map_location="cpu", weights_only=False)["model_state_dict"]
- cfg, _ = T.config_from_checkpoint(st, G8H)
- return cfg
-
-
-def main() -> None:
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--ref", type=Path, default=None)
- p.add_argument("--data_root", default=str(REPO / "data/sam3d_gt_coco"))
- p.add_argument("--emdb-root", type=Path, default=Path.home() / "Downloads/EMDB_root")
- p.add_argument("--dpw-seq", type=Path,
- default=Path.home() / "Downloads/sequenceFiles/sequenceFiles")
- p.add_argument("--dpw-img", type=Path, default=Path.home() / "Downloads/imageFiles")
- p.add_argument("--n", type=int, default=24)
- args = p.parse_args()
- tmp = Path(tempfile.mkdtemp(prefix="verify_input_size_"))
-
- ref_path = args.ref
- if ref_path is None:
- ref_path = tmp / "train_distill_mhr_only_ref.py"
- ref_path.write_text(subprocess.run(
- ["git", "-C", str(REPO), "show",
- "HEAD:instanthmr_distill_train/train_distill_mhr_only.py"],
- check=True, capture_output=True, text=True).stdout)
- R = load_module(ref_path, "train_distill_mhr_only_ref")
-
- print("[1] augmented samples vs reference, 224")
- for v2 in (False, True):
- # the new code as both trainers now call it (image_size passed), the
- # reference as it was called before (no such argument)
- new = dataset(T, args.data_root, v2, image_size=224)
- old = dataset(R, args.data_root, v2)
- diff = 0
- for k in range(args.n):
- i = k % len(new)
- a = sample(new, i, 1000 + k) # alternate: new, old, new, old ...
- b = sample(old, i, 1000 + k)
- diff += not same(a, b)
- check(diff == 0, f"{'v2' if v2 else 'baseline'}: {diff} of {args.n} samples differ")
-
- print("[1b] labels do not depend on the input size (no augmentation)")
- d224 = dataset(T, args.data_root, False, augment=False, image_size=224)
- d288 = dataset(T, args.data_root, False, augment=False, image_size=288)
- bad = 0
- for i in range(min(args.n, len(d224))):
- a, b = d224[i], d288[i]
- bad += not (a["image"].shape[-1] == 224 and b["image"].shape[-1] == 288
- and torch.allclose(a["joints_2d"], b["joints_2d"], atol=1e-6)
- and torch.equal(a["cliff_cond"], b["cliff_cond"]))
- check(bad == 0, f"224 vs 288: {bad} samples with a different label or CLIFF vector")
-
- print("[2] a 288 checkpoint round-trips through config_from_checkpoint")
- cfg288 = g8h_cfg()
- cfg288.image_size = 288
- torch.manual_seed(0)
- m288 = T.InstantHMRStudent(cfg288, pretrained=False).eval()
- ck_dir = tmp / "r288"
- ck_dir.mkdir()
- ck = ck_dir / "best_student_model_v3.pth"
- torch.save({"model_state_dict": m288.state_dict(), "epoch": 0}, ck)
- st = m288.state_dict()
- c, prov = T.config_from_checkpoint(st, ck)
- check(c.image_size == 288, f"no run_config.json: image_size={c.image_size} "
- f"[{prov.get('image_size')}]")
- rc = {"image_size": 288, "cliff_focal": True, "backbone": cfg288.backbone}
- (ck_dir / "run_config.json").write_text(json.dumps(rc))
- c, prov = T.config_from_checkpoint(st, ck)
- check(c.image_size == 288, f"with run_config.json: image_size={c.image_size}")
- (ck_dir / "bad").mkdir()
- (ck_dir / "bad" / "run_config.json").write_text(json.dumps({**rc, "image_size": 224}))
- try:
- T.config_from_checkpoint(st, ck_dir / "bad" / "x.pth")
- check(False, "contradicting run_config.json was accepted")
- except ValueError as e:
- check(True, f"contradicting run_config.json refused: {e}")
- c224 = g8h_cfg()
- check(c224.image_size == 224, f"g8h_s0 reads back as {c224.image_size}")
-
- print("[3] crop builder (3DPW / EMDB validation)")
- frame = next(iter(sorted(args.dpw_img.rglob("*.jpg"))))
- rgb = cv2.cvtColor(cv2.imread(str(frame)), cv2.COLOR_BGR2RGB)
- h, w = rgb.shape[:2]
- box = np.array([w * 0.3, h * 0.2, w * 0.6, h * 0.9])
- Rv = load_module(args.ref.parent / "val3dpw.py", "val3dpw_ref") if args.ref else None
- a, ca = val3dpw._preprocess(rgb, box, h, w, 1500.0)
- if Rv is not None:
- b, cb = Rv._preprocess(rgb, box, h, w, 1500.0)
- check(np.array_equal(a, b) and np.array_equal(ca, cb), "224 crop byte-identical to reference")
- else:
- print(" SKIP 224 crop vs reference (pass --ref <dir>/train_distill_mhr_only.py with val3dpw.py beside it)")
- check(a.shape == (3, 224, 224), f"default crop {a.shape}")
- a288, _ = val3dpw._preprocess(rgb, box, h, w, 1500.0, size=288)
- check(a288.shape == (3, 288, 288), f"size=288 crop {a288.shape}")
-
- print("[4] ONNX export and the inference package")
- onnx_path = tmp / "r288.onnx"
- r = subprocess.run([sys.executable, str(REPO / "tools/pth_to_onnx.py"), "--ckpt", str(ck),
- "--output", str(onnx_path)], capture_output=True, text=True)
- check(r.returncode == 0, f"pth_to_onnx exit {r.returncode}"
- + ("" if r.returncode == 0 else f": {r.stderr[-400:]}"))
- if onnx_path.exists():
- import onnx
- meta = {q.key: q.value for q in onnx.load(str(onnx_path)).metadata_props}
- check(meta.get("image_size") == "288", f"ONNX metadata image_size={meta.get('image_size')}")
- from instanthmr.inference import InstantHMR
- from instanthmr.mhr_renderer import build_mhr
- net = InstantHMR(onnx_path, device="cpu",
- mhr=build_mhr(script_path=str(REPO / "checkpoints/mhr_model.pt")))
- check(net.input_size == 288, f"InstantHMR input_size={net.input_size}")
- out = net.predict(rgb, box)
- j2d = np.asarray(out.joints_2d)
- check(np.isfinite(j2d).all() and j2d.shape[-1] == 2, f"predict on a real frame: joints_2d {j2d.shape}")
-
- print("[5] harnesses run a 288 checkpoint")
- for script, extra in (
- ("benchmark/eval_emdb_ckpt.py",
- ["--emdb-root", str(args.emdb_root), "--stride", "400",
- "--adapter", str(REPO / "benchmark/results/adapter_smpl24_teacher.npz")]),
- ("benchmark/eval_3dpw_ckpt.py",
- ["--sequence-dir", str(args.dpw_seq), "--image-root", str(args.dpw_img),
- "--split", "test", "--stride", "400"])):
- out_json = tmp / (Path(script).stem + ".json")
- r = subprocess.run([sys.executable, str(REPO / script), "--ckpt", str(ck),
- *extra, "--num-workers", "2", "--out", str(out_json)],
- capture_output=True, text=True, cwd=REPO)
- check(r.returncode == 0 and out_json.exists(),
- f"{Path(script).name}: exit {r.returncode}"
- + ("" if r.returncode == 0 else f": {r.stderr[-400:]}"))
-
- print(f"scratch: {tmp}")
- if fails:
- print(f"FAIL ({len(fails)})")
- sys.exit(1)
- print("PASS")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/verify_prev_token.py b/tools/verify_prev_token.py
deleted file mode 100644
index db5dd05..0000000
--- a/tools/verify_prev_token.py
+++ /dev/null
@@ -1,144 +0,0 @@
-#!/usr/bin/env python3
-"""Checks for --prev-token (foot lock) on real data.
-
- python tools/verify_prev_token.py --data_root data
-
-1. OFF IS BIT-IDENTICAL. With the flag off, dataset samples, the loss and the
- model output equal `git show HEAD`'s, on augmented samples drawn under the
- same seeds. The window corpus is not exercised (HEAD predates window0_p);
- the change does not touch that branch.
-2. PAIR GEOMETRY. With the noise at zero, on Harmony4D pairs under flip and
- rotation: previous-pose feet + true displacement == the current GT feet,
- in the augmented frame, to float precision. This is what the zero-velocity
- loss relies on.
-3. Contact labels derived on Harmony4D: how many pairs are found, and the
- contact / non-contact / masked split.
-
-Exit status 0 when 1 and 2 pass.
-"""
-from __future__ import annotations
-
-import argparse
-import importlib
-import random
-import subprocess
-import sys
-from pathlib import Path
-
-import numpy as np
-import torch
-
-ROOT = Path(__file__).resolve().parent.parent
-TRAIN = ROOT / "instanthmr_distill_train"
-sys.path.insert(0, str(TRAIN))
-
-
-def seeded(fn, seed):
- random.seed(seed); np.random.seed(seed); torch.manual_seed(seed)
- return fn()
-
-
-def same(a, b):
- if torch.is_tensor(a):
- return torch.equal(a, b)
- return bool(np.array_equal(np.asarray(a), np.asarray(b)))
-
-
-def main() -> int:
- p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--data_root", default=str(ROOT / "data"))
- p.add_argument("--n", type=int, default=48)
- args = p.parse_args()
-
- head_path = TRAIN / "_head_train_distill_mhr_only.py"
- head_path.write_text(subprocess.run(
- ["git", "show", "HEAD:instanthmr_distill_train/train_distill_mhr_only.py"],
- cwd=ROOT, capture_output=True, text=True, check=True).stdout)
- try:
- H = importlib.import_module("_head_train_distill_mhr_only")
- finally:
- head_path.unlink()
- import train_distill_mhr_only as T
- ok = True
-
- # ---- 1. off == HEAD ----------------------------------------------------
- kw = dict(augment=True, per_dataset_caps={"*": 64}, cliff_focal=True,
- crop_centre_fix=True, cliff_follows_aug=True, image_size=224)
- ds_h = H.SAM3DStudentDataset(args.data_root, **kw)
- ds_n = T.SAM3DStudentDataset(args.data_root, temporal=None, **kw)
- idx = random.Random(0).sample(range(len(ds_n)), args.n)
- bad = 0
- for k, i in enumerate(idx):
- a = seeded(lambda: ds_h[i], k)
- b = seeded(lambda: ds_n[i], k)
- if set(a) != set(b) or not all(same(a[x], b[x]) for x in a):
- bad += 1
- print(f"[1] {args.n} augmented samples, flag off vs HEAD: {args.n - bad} identical")
- ok &= bad == 0
-
- cfg_h, cfg_n = H.DistillConfig(), T.DistillConfig()
- for c, M in ((cfg_h, H), (cfg_n, T)):
- M.apply_rebalanced_losses(c)
- c.cont_head = c.root_rot_loss = True
- c.bound_scales = c.cliff_focal = True
- c.backbone = "hgnetv2_b4"
- dev = T.device
- W = np.load(cfg_n.kp_regressor_path)
- mhr = T.MHRForwardPass(cfg_n.mhr_model_path, dev, kp_regressor=W)
- batch = torch.utils.data.default_collate([seeded(lambda: ds_n[i], 100 + j) for j, i in enumerate(idx[:8])])
- batch = {k: (v.to(dev) if torch.is_tensor(v) else v) for k, v in batch.items()}
- torch.manual_seed(0); m_h = H.InstantHMRStudent(cfg_h, pretrained=False).to(dev).eval()
- m_n = T.InstantHMRStudent(cfg_n, pretrained=False).to(dev).eval()
- m_n.load_state_dict(m_h.state_dict())
- print(f" state_dict keys: HEAD {len(m_h.state_dict())}, now {len(m_n.state_dict())}")
- ok &= len(m_h.state_dict()) == len(m_n.state_dict())
- with torch.no_grad():
- o_h = m_h(batch["image"], batch["cliff_cond"])
- o_n = m_n(batch["image"], batch["cliff_cond"])
- o_n2 = m_n(batch["image"], batch["cliff_cond"])
- out_same = all(torch.equal(o_n2[k], o_h[k]) for k in o_h) and set(o_h) == set(o_n)
- print(f" model outputs identical: {out_same}")
- ok &= out_same
- l_h, l_n = H.DistillationLoss(cfg_h, mhr), T.DistillationLoss(cfg_n, mhr)
- with torch.no_grad():
- for _ in range(2): # alternate: the first MHR call in a process differs ~5e-10
- a = l_h({k: v.clone() for k, v in o_h.items()}, batch)
- b = l_n({k: v.clone() for k, v in o_h.items()}, batch)
- loss_same = set(a) == set(b) and all(torch.equal(a[k], b[k]) for k in a)
- print(f" loss terms identical: {loss_same} ({len(a)} terms)")
- ok &= loss_same
-
- # ---- 2./3. pairs on Harmony4D -----------------------------------------
- cfg = T.DistillConfig()
- cfg.prev_token = True
- cfg.prev_mask_p = 0.0
- cfg.prev_noise_joint = cfg.prev_noise_xy = cfg.prev_noise_logz = 0.0
- tk = T.temporal_kwargs(cfg)
- h4d = Path(args.data_root) / "sam3d_gt_harmony4d"
- ds = T.SAM3DStudentDataset(str(h4d.parent), temporal=tk, geom_p=1.0, geom_flip_p=0.5,
- per_dataset_caps={"*": 0, "sam3d_gt_harmony4d": 3000}, **{
- k: v for k, v in kw.items() if k not in ("augment", "per_dataset_caps")})
- worst, n_pair, n_flip, cstat = 0.0, 0, 0, np.zeros(3)
- for k in range(400):
- s = seeded(lambda: ds[k], 1000 + k)
- if not s["has_pair"]:
- continue
- n_pair += 1
- n_flip += int(s["aug_flip"] > 0.5)
- cur = (s["joints_3d"] + s["cam_trans"])[T.FOOT_KP]
- worst = max(worst, float((s["prev_feet"] + s["feet_disp"] - cur).abs().max()))
- cv, c = s["contact_valid"].numpy(), s["contact"].numpy()
- cstat += [(c * cv).sum(), ((1 - c) * cv).sum(), (1 - cv).sum()]
- print(f"[2] {n_pair}/400 Harmony4D samples found their previous frame "
- f"({n_flip} flipped); max |prev + disp - current| = {worst:.2e} m")
- ok &= n_pair > 0 and worst < 1e-5
- tot = cstat.sum()
- print(f"[3] per-foot labels: contact {cstat[0]/tot:.0%}, moving {cstat[1]/tot:.0%}, "
- f"masked {cstat[2]/tot:.0%} (thresholds {cfg.contact_thr*1000:.0f} / "
- f"{cfg.noncontact_thr*1000:.0f} mm per pair)")
- print("PASS" if ok else "FAIL")
- return 0 if ok else 1
-
-
-if __name__ == "__main__":
- raise SystemExit(main())
diff --git a/tools/verify_v3_preset.py b/tools/verify_v3_preset.py
deleted file mode 100644
index 81b71b3..0000000
--- a/tools/verify_v3_preset.py
+++ /dev/null
@@ -1,290 +0,0 @@
-#!/usr/bin/env python3
-"""Verify the v3 preset on real context crops.
-
-Needs the same rows built twice with datasets_pipeline/build_split.py:
-standard (default flags) and context (--context 2.0 --body-size 448).
-
- python tools/verify_v3_preset.py --std <root with sam3d_gt_aic> --ctx <root> \
- --ref <train_distill_mhr_only.py before the change>
-
-Checks, all from the dataset's outputs plus the npz geometry:
-
-1. baseline and v2 samples are bit-identical to --ref (alternating calls).
-2. Un-augmented v3 (the identity window) reproduces today's crop: labels and
- CLIFF equal the standard split's, image within resampling noise.
-3. Reprojection chain: joints_3d + cam_trans projected through bbox_square and
- aug_M land on the joints_2d labels, exactly as the loss computes them.
-4. CLIFF describes the window actually cut: the preimage of the crop centre
- under aug_M, flipped and rotated like the labels, is the CLIFF centre, and
- the CLIFF size is the window side / 1.2.
-5. No pixel is read from beyond a stored-square edge that lies INSIDE the
- photo (that would be a synthetic black margin). Past an edge that is
- already outside the photo, black is the real frame edge and allowed.
- Also reports the loose tail the clamp leaves: P(>1.6x), P(>1.8x) and the
- mean shift of loose windows.
-6. Box-scale draws cover the configured range (log-uniform 0.6-2.0x).
-7. Output size follows image_size (224 and 288).
-8. A standard split is refused by the v3 dataset, naming the split.
-"""
-
-from __future__ import annotations
-
-import argparse
-import importlib.util
-import math
-import random
-import sys
-from pathlib import Path
-
-import numpy as np
-import torch
-from PIL import Image
-
-REPO = Path(__file__).resolve().parent.parent
-TRAIN = REPO / "instanthmr_distill_train"
-sys.path.insert(0, str(TRAIN))
-import train_distill_mhr_only as T # noqa: E402
-import train_distill_jz as JZ # noqa: E402
-
-MEAN = np.array([0.485, 0.456, 0.406])
-STD = np.array([0.229, 0.224, 0.225])
-fails: list[str] = []
-
-
-def check(ok: bool, msg: str) -> None:
- print(f" {'OK ' if ok else 'BAD'} {msg}")
- if not ok:
- fails.append(msg)
-
-
-def load_module(path: Path, name: str):
- spec = importlib.util.spec_from_file_location(name, path)
- mod = importlib.util.module_from_spec(spec)
- spec.loader.exec_module(mod)
- return mod
-
-
-def cfg_for(preset: str, image_size: int = 224):
- cfg = T.DistillConfig()
- cfg.cliff_focal = True
- cfg.crop_centre_fix = True
- cfg.image_size = image_size
- if preset in ("v2", "v3"):
- JZ.apply_v2_preset(cfg)
- if preset == "v3":
- JZ.apply_v3_preset(cfg)
- return cfg
-
-
-def dataset(mod, root, cfg, augment=True, v3=False):
- kw = dict(geom_p=cfg.geom_p, geom_rot_deg=cfg.geom_rot_deg,
- geom_scale_range=cfg.geom_scale_range, geom_trans=cfg.geom_trans,
- geom_flip_p=cfg.geom_flip_p, geom_scale_max=cfg.geom_scale_max,
- cliff_follows_aug=cfg.cliff_follows_aug, crop_centre_fix=cfg.crop_centre_fix,
- cliff_focal=cfg.cliff_focal, occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
- jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality, image_size=cfg.image_size)
- if v3:
- kw.update(context_crops=True, v3_box_min=cfg.v3_box_min,
- v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)
- return mod.SAM3DStudentDataset(str(root), augment=augment, **kw)
-
-
-def seeded(ds, i, seed):
- random.seed(seed)
- np.random.seed(seed)
- torch.manual_seed(seed)
- return ds[i]
-
-
-def same(a, b) -> bool:
- if isinstance(a, torch.Tensor):
- return torch.equal(a, b)
- if isinstance(a, np.ndarray):
- return np.array_equal(a, b, equal_nan=a.dtype.kind == "f")
- return a == b
-
-
-def to_uint8(img: torch.Tensor) -> np.ndarray:
- x = img.numpy().transpose(1, 2, 0) * STD + MEAN
- return np.clip(x * 255.0, 0, 255)
-
-
-def reproject(s: dict, size: int) -> np.ndarray:
- """The loss's own chain: undo the in-plane rotation and the flip on the 3D
- labels (the dataset applied flip then rotation), project in the original
- camera, map into the 1.2x crop, then let aug_M re-apply both."""
- c, sn = float(s["aug_cos"]), float(s["aug_sin"])
- R = np.array([[c, sn], [-sn, c]])
- j3 = s["joints_3d"].numpy().astype(np.float64).copy()
- cam = s["cam_trans"].numpy().astype(np.float64).copy()
- j3[:, :2] = j3[:, :2] @ R # inverse of @ R.T
- cam[:2] = R.T @ cam[:2] # inverse of R @
- if float(s["aug_flip"]) > 0.5:
- j3[:, 0], cam[0] = -j3[:, 0], -cam[0]
- j = j3 + cam
- fx, fy = s["cam_focal"].numpy()
- h, w = s["orig_shape"].numpy()
- u = fx * j[:, 0] / j[:, 2] + w / 2.0
- v = fy * j[:, 1] / j[:, 2] + h / 2.0
- sq = np.asarray(s["bbox_square"], dtype=np.float64)
- k = size / max(sq[2] - sq[0], 1.0)
- uc, vc = (u - sq[0]) * k, (v - sq[1]) * k
- M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)
- ua = M[0, 0] * uc + M[0, 1] * vc + M[0, 2]
- va = M[1, 0] * uc + M[1, 1] * vc + M[1, 2]
- return np.stack([ua / size * 2 - 1, va / size * 2 - 1], axis=1)
-
-
-def window_from_sample(s: dict, ann: dict, size: int, sc: int): # noqa: C901
- """Recover the window from aug_M + npz geometry, independently of the
- dataset's own bookkeeping. Returns (centre in frame px, side in frame px,
- context-crop px of the 4 output corners)."""
- M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)
- A, t = M[:, :2], M[:, 2]
- Ainv = np.linalg.inv(A)
- sq = ann["bbox_square"].astype(np.float64)
- ks = (sq[2:] - sq[:2]) / size # std px -> frame px
- to_frame = lambda o: sq[:2] + (Ainv @ (np.asarray(o, float) - t)) * ks # noqa: E731
- centre = to_frame([size / 2.0, size / 2.0])
- side = size / abs(np.linalg.det(A)) ** 0.5 * (ks[0] * ks[1]) ** 0.5
- ctx = ann["bbox_context"].astype(np.float64)
- corners = [to_frame(c) for c in ([0, 0], [size, 0], [0, size], [size, size])]
- return centre, side, corners
-
-
-def main() -> None:
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--std", type=Path, required=True)
- p.add_argument("--ctx", type=Path, required=True)
- p.add_argument("--ref", type=Path, required=True)
- p.add_argument("--n", type=int, default=400)
- args = p.parse_args()
- R = load_module(args.ref, "train_distill_mhr_only_ref")
-
- print("[1] baseline and v2 vs reference")
- for preset in ("baseline", "v2"):
- cfg = cfg_for(preset)
- new, old = dataset(T, args.std, cfg), dataset(R, args.std, cfg)
- diff = 0
- for k in range(24):
- a, b = seeded(new, k, 7 + k), seeded(old, k, 7 + k)
- diff += not all(same(a[x], b[x]) for x in a)
- check(diff == 0, f"{preset}: {diff} of 24 samples differ")
-
- cfg = cfg_for("v3")
- std0 = dataset(T, args.std, cfg_for("baseline"), augment=False)
- ctx0 = dataset(T, args.ctx, cfg, augment=False, v3=True)
- print("[2] identity window reproduces today's crop")
- lab, cl, ps = 0.0, 0.0, []
- for i in range(min(64, len(std0))):
- a, b = std0[i], ctx0[i]
- lab = max(lab, float((a["joints_2d"] - b["joints_2d"]).abs().max()))
- cl = max(cl, float((a["cliff_cond"] - b["cliff_cond"]).abs().max()))
- ia, ib = to_uint8(a["image"]), to_uint8(b["image"])
- mse = np.mean((ia[8:-8, 8:-8] - ib[8:-8, 8:-8]) ** 2)
- ps.append(10 * math.log10(255.0 ** 2 / max(mse, 1e-9)))
- check(lab < 1e-5, f"joints_2d max diff {lab:.2e}")
- check(cl < 1e-6, f"cliff_cond max diff {cl:.2e}")
- check(min(ps) > 25.0, f"image PSNR mean {np.mean(ps):.1f} dB, min {min(ps):.1f} dB "
- f"(448 context crop vs 224 crop, resampling noise)")
-
- for size in (224, 288):
- cfg = cfg_for("v3", size)
- ds = dataset(T, args.ctx, cfg, augment=True, v3=True)
- print(f"[3-7] augmented v3 at {size} px, {args.n} samples")
- rep, cc, cs, outside, shapes, ms, shifts = 0.0, 0.0, 0.0, 0, set(), [], []
- for k in range(args.n):
- i = k % len(ds)
- s = seeded(ds, i, 100 + k)
- ann = dict(np.load(ds.pairs[i][1]))
- shapes.add(tuple(s["image"].shape))
- # 3: reprojection chain vs labels (visible keypoints, output px)
- vis = s["joints_2d_vis"].numpy() > 0
- err = np.abs(reproject(s, size) - s["joints_2d"].numpy())[vis] * size / 2
- rep = max(rep, float(err.max()))
- # 4: CLIFF vs the window recovered from aug_M. Augmented samples
- # only: an un-augmented one keeps today's CLIFF (the tight-box
- # centre), which is 0.5 px off the int()-rounded 1.2x square.
- sc = Image.open(ds.pairs[i][0]).size[0]
- centre, side, corners = window_from_sample(s, ann, size, sc)
- h, w = ann["orig_shape"][:2]
- flip, cos_t, sin_t = float(s["aug_flip"]), float(s["aug_cos"]), float(s["aug_sin"])
- cx, cy = centre
- if flip > 0.5:
- cx = w - cx
- d = np.array([[cos_t, sin_t], [-sin_t, cos_t]]) @ np.array([cx - w / 2, cy - h / 2])
- f = float(ann["cam_focal_length"][1])
- want = np.array([math.atan(d[0] / f), math.atan(d[1] / f), side / 1.2 / f])
- got = s["cliff_cond"].numpy().astype(np.float64)
- if float(s["aug_active"]) > 0.5:
- cc = max(cc, float(np.abs(np.tan(got[:2]) - np.tan(want[:2])).max() * f))
- cs = max(cs, abs(got[2] - want[2]) / want[2])
- # 5: no corner past a stored-square edge that is inside the photo
- ctx = ann["bbox_context"].astype(np.float64)
- tol = 0.5 * (ctx[2] - ctx[0]) / sc # half a stored pixel
- for c in corners:
- outside += int((ctx[0] >= 0 and c[0] < ctx[0] - tol) or
- (ctx[2] <= w and c[0] > ctx[2] + tol) or
- (ctx[1] >= 0 and c[1] < ctx[1] - tol) or
- (ctx[3] <= h and c[1] > ctx[3] + tol)) > 0
- tight = ann["bbox"].astype(np.float64)
- ts = max(tight[2] - tight[0], tight[3] - tight[1])
- ms.append(side / ts)
- tc = (tight[:2] + tight[2:]) / 2
- shifts.append(float(np.abs(np.asarray(centre) - tc).max() / side))
- ms, shifts = np.array(ms), np.array(shifts)
- on = np.abs(ms - 1.2) > 1e-3
- active = ms[on]
- loose = on & (ms > 1.6)
- p16, p18 = np.mean(active > 1.6), np.mean(active > 1.8)
- ideal16 = math.log(2.0 / 1.6) / math.log(2.0 / 0.6)
- ideal18 = math.log(2.0 / 1.8) / math.log(2.0 / 0.6)
- check(p16 > 0.8 * ideal16,
- f"loose tail: P(>1.6x) {p16:.3f} (unclamped {ideal16:.3f}), P(>1.8x) {p18:.3f} "
- f"(unclamped {ideal18:.3f}); mean shift of >1.6x windows "
- f"{shifts[loose].mean():.3f} of the side (unclamped 0.100)")
- check(rep < 1e-3, f"reprojection chain vs labels: max {rep:.2e} px")
- check(cc < 0.05 and cs < 1e-4, f"CLIFF vs recovered window: centre {cc:.3f} px, "
- f"size rel {cs:.1e}")
- check(outside == 0, f"{outside} samples read past a stored-square edge inside the photo")
- lo, hi = active.min(), active.max()
- frac = np.mean(active < 1.1)
- expect = math.log(1.1 / 0.6) / math.log(2.0 / 0.6)
- check(lo < 0.65 and hi > 1.8 and abs(frac - expect) < 0.08,
- f"box multiple drawn {lo:.2f}-{hi:.2f}x; P(<1.1x) {frac:.2f} vs "
- f"log-uniform {expect:.2f} ({len(active)} active draws)")
- check(shapes == {(3, size, size)}, f"output shapes {shapes}")
-
- print("[8] a standard split is refused")
- try:
- dataset(T, args.std, cfg_for("v3"), v3=True)
- check(False, "standard split accepted by the v3 dataset")
- except ValueError as e:
- check("sam3d_gt_aic" in str(e), f"refused: {e}")
-
- # a root mixing a context split (sorted first) with a standard one
- import tempfile
- mixed = Path(tempfile.mkdtemp(prefix="v3_mixed_"))
- (mixed / "sam3d_gt_aic").symlink_to((args.ctx / "sam3d_gt_aic").resolve())
- (mixed / "sam3d_gt_zstd").symlink_to((args.std / "sam3d_gt_aic").resolve())
- try:
- dataset(T, mixed, cfg_for("v3"), v3=True)
- check(False, "mixed context + standard root accepted")
- except ValueError as e:
- check("sam3d_gt_zstd" in str(e), f"mixed root refused: {str(e)[:60]}...")
- idx = JZ.build_pair_index(str(mixed), mixed / "idx.npz")
- try:
- T.SAM3DStudentDataset(str(mixed), augment=True, pairs=idx, context_crops=True)
- check(False, "mixed root accepted through a PairIndex")
- except ValueError as e:
- check("sam3d_gt_zstd" in str(e), "mixed root refused through a PairIndex")
-
- if fails:
- print(f"FAIL ({len(fails)})")
- sys.exit(1)
- print("PASS")
-
-
-if __name__ == "__main__":
- main()
diff --git a/tools/verify_windows.py b/tools/verify_windows.py
deleted file mode 100644
index 5a983e7..0000000
--- a/tools/verify_windows.py
+++ /dev/null
@@ -1,413 +0,0 @@
-#!/usr/bin/env python3
-"""Verify the precomputed-window corpus (build_split.py --windows) and the
---preset v4 dataset on real rows.
-
- python tools/verify_windows.py --split aic_train \
- --annotation-dir ~/sam3d_data/annotations/aic_train \
- --image-dir ~/sam3d_data/images/aic \
- --ref <train_distill_mhr_only.py before the change> --n 64
-
-Checks:
-
-1. Builder: the default and --context 2.0 outputs are byte-identical to HEAD's.
-2. Window build: both size folders hold the same people and the same npz, which
- is the standard npz plus win_boxes; row 0 is within 1 px of bbox_square,
- rows 1.. are integer exact squares; each tile is K*S x S.
-3. Every window is the RUNTIME crop: instanthmr.inference's own _preprocess,
- given the detector box that window stands for (the tight box itself for
- window 0), returns the same pixels (PNG build: exact) and the same CLIFF
- vector, and for windows 1.. the labels land where the runtime crop puts the
- annotated keypoints (window 0's box can be 1 px off square, and the label
- normalisation, unchanged since g8, uses its width for both axes). Also reports PSNR against the
- runtime crop for a JPEG q95 build (the cluster corpus).
-4. Augmented v4 (baseline geometry on top): the loss's reprojection chain lands
- on the joints_2d labels.
-5. The window framings follow the v3 range: side log-uniform 0.6-2.0x the
- tight box, centre shift within +/-15% of the side.
-6. With crop_windows off, baseline and v2 samples are bit-identical to --ref.
-7. A standard split, and tiles of the wrong size, are refused.
-"""
-
-from __future__ import annotations
-
-import argparse
-import importlib.util
-import math
-import os
-import random
-import subprocess
-import sys
-import tempfile
-import types
-from pathlib import Path
-
-import cv2
-import numpy as np
-import torch
-
-REPO = Path(__file__).resolve().parent.parent
-BUILDER = REPO / "datasets_pipeline" / "build_split.py"
-sys.path.insert(0, str(REPO / "instanthmr_distill_train"))
-sys.path.insert(0, str(REPO))
-sys.path.insert(0, str(REPO / "tools"))
-import train_distill_mhr_only as T # noqa: E402
-import train_distill_jz as JZ # noqa: E402
-from instanthmr.inference import InstantHMR # noqa: E402
-from verify_v3_preset import reproject, same, seeded # noqa: E402
-
-MEAN = np.array([0.485, 0.456, 0.406])
-STD = np.array([0.229, 0.224, 0.225])
-K = 4
-SIZES = (224, 288)
-fails: list[str] = []
-
-
-def check(ok: bool, msg: str) -> None:
- print(f" {'OK ' if ok else 'BAD'} {msg}")
- if not ok:
- fails.append(msg)
-
-
-def build(builder: Path, out: Path, args, extra: list[str], fmt: str = "png") -> None:
- env = dict(os.environ, PYTHONPATH=f"{REPO / 'datasets_pipeline'}:{REPO}")
- cmd = [sys.executable, str(builder), "--split", args.split,
- "--annotation-dir", str(args.annotation_dir),
- "--image-dir", str(args.image_dir), "--output-dir", str(out),
- "--max-samples", str(args.n), "--originals", "none", "--crop-format", fmt,
- *extra]
- subprocess.run(cmd, check=True, env=env, stdout=subprocess.DEVNULL)
-
-
-def tree_diff(a: Path, b: Path) -> int:
- fa = {p.relative_to(a) for p in a.rglob("*") if p.is_file()}
- fb = {p.relative_to(b) for p in b.rglob("*") if p.is_file()}
- if fa != fb or not fa:
- return max(1, len(fa ^ fb))
- return sum((a / f).read_bytes() != (b / f).read_bytes() for f in fa)
-
-
-def load(d: Path, name: str) -> dict:
- with np.load(d / "annotations" / f"{name}.npz", allow_pickle=True) as z:
- return {k: z[k] for k in z.files}
-
-
-def psnr(a, b) -> float:
- mse = np.mean((np.asarray(a, np.float64) - np.asarray(b, np.float64)) ** 2)
- return float("inf") if mse == 0 else 10 * math.log10(255.0 ** 2 / mse)
-
-
-def runtime(img_rgb, box, size, focal):
- """instanthmr.inference's crop + CLIFF for detector box `box`."""
- stub = types.SimpleNamespace(cliff_focal=True, input_size=size,
- _focal_px=lambda h, w, f: f)
- h, w = img_rgb.shape[:2]
- crop, x1, y1, side, cliff = InstantHMR._preprocess(stub, img_rgb, box, h, w, focal)
- u8 = np.clip((crop.transpose(1, 2, 0) * STD + MEAN) * 255.0, 0, 255)
- return u8, np.array([x1, y1, side]), cliff
-
-
-def detector_box(win: np.ndarray, size: int, img_rgb, focal):
- """The detector box whose 1.2x square is exactly `win` at the runtime:
- side / 1.2 in float lands a hair off the integer and the runtime's floor /
- ceil then cut one pixel more, so nudge it by ulps until it does not."""
- c, half = (win[:2] + win[2:]) / 2, (win[2] - win[0]) / 2.4
- for t in range(64):
- det = np.concatenate([c - half, c + half])
- out = runtime(img_rgb, det, size, focal)
- x1, y1, side = out[1] # the runtime cuts floor(x1) .. ceil(x1 + side)
- if [math.floor(x1), math.floor(y1), math.ceil(x1 + side), math.ceil(y1 + side)] \
- == [int(v) for v in win]:
- return det, out
- half = np.nextafter(half, np.inf if t % 2 else -np.inf)
- raise RuntimeError(f"no detector box reproduces {win}")
-
-
-def to_uint8(img: torch.Tensor) -> np.ndarray:
- return np.clip((img.numpy().transpose(1, 2, 0) * STD + MEAN) * 255.0, 0, 255)
-
-
-def cfg_for(preset: str, size: int = 224):
- cfg = T.DistillConfig()
- cfg.cliff_focal = True
- cfg.crop_centre_fix = True
- cfg.image_size = size
- if preset == "v2":
- JZ.apply_v2_preset(cfg)
- if preset == "v4":
- JZ.apply_v4_preset(cfg)
- return cfg
-
-
-def dataset(mod, root, cfg, augment=True, **over):
- kw = dict(geom_p=cfg.geom_p, geom_rot_deg=cfg.geom_rot_deg,
- geom_scale_range=cfg.geom_scale_range, geom_trans=cfg.geom_trans,
- geom_flip_p=cfg.geom_flip_p, geom_scale_max=cfg.geom_scale_max,
- cliff_follows_aug=cfg.cliff_follows_aug, crop_centre_fix=cfg.crop_centre_fix,
- cliff_focal=cfg.cliff_focal, occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,
- jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality, image_size=cfg.image_size)
- if cfg.crop_windows:
- kw["crop_windows"] = True
- kw.update(over)
- return mod.SAM3DStudentDataset(str(root), augment=augment, **kw)
-
-
-def main() -> None: # noqa: C901
- p = argparse.ArgumentParser(description=__doc__,
- formatter_class=argparse.RawDescriptionHelpFormatter)
- p.add_argument("--split", default="aic_train")
- p.add_argument("--annotation-dir", type=Path, required=True)
- p.add_argument("--image-dir", type=Path, required=True)
- p.add_argument("--ref", type=Path, required=True)
- p.add_argument("--n", type=int, default=64)
- args = p.parse_args()
-
- tmp = Path(tempfile.mkdtemp(prefix="verify_windows_"))
- head = tmp / "build_split_head.py"
- head.write_text(subprocess.run(
- ["git", "-C", str(REPO), "show", "HEAD:datasets_pipeline/build_split.py"],
- check=True, capture_output=True, text=True).stdout)
-
- print("[1] builder defaults vs HEAD")
- build(head, tmp / "head" / "sam3d_gt_aic", args, [])
- build(BUILDER, tmp / "std" / "sam3d_gt_aic", args, [])
- build(BUILDER, tmp / "std_nh" / "sam3d_gt_aic", args, ["--no-hands"])
- check(tree_diff(tmp / "head", tmp / "std") == 0, "default build byte-identical to HEAD")
- ctx = ["--context", "2.0", "--body-size", "448", "--no-hands"]
- build(head, tmp / "head_ctx", args, ctx)
- build(BUILDER, tmp / "ctx", args, ctx)
- check(tree_diff(tmp / "head_ctx", tmp / "ctx") == 0, "--context 2.0 build byte-identical to HEAD")
-
- win = ["--windows", str(K), "--window-sizes", ",".join(map(str, SIZES)), "--no-hands"]
- build(BUILDER, tmp / "win{size}" / "sam3d_gt_aic", args, win)
- build(BUILDER, tmp / "winjpg{size}" / "sam3d_gt_aic", args, win, fmt="jpg")
- std = tmp / "std" / "sam3d_gt_aic"
- wd = {s: tmp / f"win{s}" / "sam3d_gt_aic" for s in SIZES}
- wj = {s: tmp / f"winjpg{s}" / "sam3d_gt_aic" for s in SIZES}
-
- print("[2] window build layout")
- names = sorted(q.stem for q in (std / "annotations").glob("*.npz"))
- sets = [sorted(q.stem for q in (wd[s] / "annotations").glob("*.npz")) for s in SIZES]
- check(all(x == names for x in sets) and names, f"{len(names)} people in every folder")
- bad_npz = bad_box = bad_tile = 0
- for nm in names:
- a, b, c = (load(tmp / "std_nh" / "sam3d_gt_aic", nm), load(wd[SIZES[0]], nm),
- load(wd[SIZES[1]], nm))
- wb = b.pop("win_boxes")
- bad_npz += (b.keys() != a.keys() or not all(same(a[k], b[k]) for k in a)
- or not np.array_equal(wb, c.pop("win_boxes"))
- or not all(same(b[k], c[k]) for k in b))
- sides = wb[1:, 2:] - wb[1:, :2]
- bad_box += (wb.shape != (K, 4) or np.abs(wb[0] - a["bbox_square"]).max() > 1
- or not np.array_equal(wb[1:], np.round(wb[1:]))
- or not np.array_equal(sides[:, 0], sides[:, 1]))
- for s in SIZES:
- tile = cv2.imread(str(wd[s] / "body_crops" / f"{nm}.png"))
- bad_tile += tile.shape != (s, K * s, 3)
- check(bad_npz == 0, f"npz = standard --no-hands npz + win_boxes, same in both folders "
- f"({bad_npz} bad)")
- check(bad_box == 0, f"win_boxes: row 0 within 1 px of bbox_square, rows 1.. integer "
- f"squares ({bad_box} bad)")
- check(bad_tile == 0, f"tiles K*S x S ({bad_tile} bad)")
-
- print("[3] each window is the runtime crop")
- for s in SIZES:
- cfg = cfg_for("v4", s)
- plain = dataset(T, wd[s].parent, cfg, augment=False).transform
- ds = dataset(T, wd[s].parent, cfg, augment=True, geom_p=0.0)
- ds.transform = plain # no colour / occlusion noise
- dsj = dataset(T, wj[s].parent, cfg, augment=True, geom_p=0.0)
- dsj.transform = plain
- exact = cl = lab = 0.0
- ps = {k: [] for k in range(K)}
- n_k = np.zeros(K, int)
- for i in range(len(ds)):
- nm = Path(ds.pairs[i][1]).stem
- a = load(wd[s], nm)
- img = cv2.cvtColor(cv2.imread(str(args.image_dir / a["image_relpath"].item())),
- cv2.COLOR_BGR2RGB)
- f = float(a["cam_focal_length"][1])
- for k in range(K):
- # the draw inside the dataset is random.randrange(K): pick k
- for seed in range(10_000):
- random.seed(seed)
- if random.randrange(K) == k:
- break
- smp = seeded(ds, i, seed)
- smj = seeded(dsj, i, seed)
- box = a["win_boxes"][k].astype(np.float64)
- if k == 0:
- rt, (x1, y1, side), cliff = runtime(img, a["bbox"].astype(np.float64), s, f)
- else:
- _, (rt, (x1, y1, side), cliff) = detector_box(box, s, img, f)
- exact = max(exact, float(np.abs(to_uint8(smp["image"]) - rt).max()))
- ps[k].append(psnr(to_uint8(smj["image"]), rt))
- cl = max(cl, float(np.abs(smp["cliff_cond"].numpy() - cliff).max()))
- j = a["joints_2d"].astype(np.float64)
- want = np.stack([(j[:, 0] - x1) / side, (j[:, 1] - y1) / side], 1) * 2 - 1
- vis = smp["joints_2d_vis"].numpy() > 0
- err = np.abs(smp["joints_2d"].numpy() - want)[vis] * s / 2
- if k:
- lab = max(lab, float(err.max()))
- n_k[k] += 1
- check(exact < 0.01, f"{s} px: every window pixel-equal to the runtime crop "
- f"(max |diff| {exact:.4f} /255, PNG build)")
- check(cl < 1e-5, f"{s} px: CLIFF = runtime's for that box (max diff {cl:.1e})")
- check(lab < 1e-3, f"{s} px: labels = keypoints in the runtime crop (max {lab:.1e} px)")
- msg = ", ".join(f"w{k} {np.mean(ps[k]):.1f}" for k in range(K))
- check(min(np.mean(ps[k]) for k in range(K)) > 36.0,
- f"{s} px JPEG q95 build vs runtime crop, PSNR dB: {msg}")
-
- print("[4] augmented v4: reprojection chain")
- for s in SIZES:
- ds = dataset(T, wd[s].parent, cfg_for("v4", s), augment=True, geom_p=1.0)
- rep, shapes, aug = 0.0, set(), 0
- for t in range(200):
- smp = seeded(ds, t % len(ds), 500 + t)
- vis = smp["joints_2d_vis"].numpy() > 0
- err = np.abs(reproject(smp, s) - smp["joints_2d"].numpy())[vis] * s / 2
- rep = max(rep, float(err.max()))
- shapes.add(tuple(smp["image"].shape))
- aug += int(float(smp["aug_active"]) > 0.5)
- check(rep < 1e-3 and aug == 200, f"{s} px: max {rep:.2e} px over 200 augmented samples")
- check(shapes == {(3, s, s)}, f"{s} px: output shapes {shapes}")
-
- print("[5] framing range")
- ms, sh = [], []
- for nm in names:
- a = load(wd[224], nm)
- t = a["bbox"].astype(np.float64)
- b = max(t[2] - t[0], t[3] - t[1])
- for w in a["win_boxes"][1:].astype(np.float64):
- side = w[2] - w[0]
- ms.append(side / b)
- sh.append(np.abs((w[:2] + w[2:]) / 2 - (t[:2] + t[2:]) / 2).max() / side)
- ms, sh = np.array(ms), np.array(sh)
- expect = math.log(1.1 / 0.6) / math.log(2.0 / 0.6)
- check(ms.min() > 0.59 and ms.max() < 2.01 and abs(np.mean(ms < 1.1) - expect) < 0.08,
- f"side {ms.min():.2f}-{ms.max():.2f}x, P(<1.1x) {np.mean(ms < 1.1):.2f} vs "
- f"log-uniform {expect:.2f} ({len(ms)} windows)")
- check(sh.max() < 0.15 + 0.01, f"centre shift max {sh.max():.3f} of the side")
-
- print("[6] crop_windows off: bit-identical to --ref")
- spec = importlib.util.spec_from_file_location("ref_trainer", args.ref)
- R = importlib.util.module_from_spec(spec)
- spec.loader.exec_module(R)
- for preset in ("baseline", "v2"):
- cfg = cfg_for(preset)
- new, old = dataset(T, std.parent, cfg), dataset(R, std.parent, cfg)
- diff = 0
- for t in range(24):
- for _ in range(2): # alternating calls
- a, b = seeded(new, t, 7 + t), seeded(old, t, 7 + t)
- diff += not all(same(a[x], b[x]) for x in a)
- check(diff == 0, f"{preset}: {diff} of 48 samples differ")
-
- print("[7] refusals")
- for root, size, want in ((std.parent, 224, "win_boxes"), (wd[224].parent, 288, "tile")):
- try:
- dataset(T, root, cfg_for("v4", size))
- check(False, f"{root.name} accepted at {size} px")
- except ValueError as e:
- check(want in str(e), f"refused: {str(e)[:90]}")
-
- print("[8] window_mode: v4 unchanged, 'standard' = the baseline framing")
- cfg = cfg_for("v4")
- new, old = dataset(T, wd[224].parent, cfg), dataset(R, wd[224].parent, cfg)
- diff = sum(not all(same(a[x], b[x]) for x in a)
- for t in range(24) for _ in range(2)
- for a, b in [(seeded(new, t, 7 + t), seeded(old, t, 7 + t))])
- check(diff == 0, f"v4 (window_mode random): {diff} of 48 samples differ from --ref")
- # Same seeds, baseline augmentation: on the standard corpus vs window 0 of
- # the window corpus. The standard draw consumes no randomness, so every
- # augmentation must match; only the crop box differs (int() vs the runtime's
- # floor / ceil rounding, <= 1 px), so the labels may move by about that.
- base = cfg_for("baseline")
- a_ds = dataset(T, std.parent, base)
- b_ds = dataset(T, wd[224].parent, base, crop_windows=True, window_mode="standard")
- names_a = [Path(p[1]).stem for p in a_ds.pairs]
- cl = lab = 0.0
- same_aug = box0 = True
- for t in range(48):
- i = t % len(b_ds)
- j = names_a.index(Path(b_ds.pairs[i][1]).stem)
- a, b = seeded(a_ds, j, 300 + t), seeded(b_ds, i, 300 + t)
- win0 = load(wd[224], Path(b_ds.pairs[i][1]).stem)["win_boxes"][0]
- box0 &= np.array_equal(np.asarray(b["bbox_square"]), win0)
- same_aug &= all(torch.equal(a[k], b[k]) for k in ("aug_active", "aug_flip", "aug_cos", "aug_sin"))
- cl = max(cl, float((a["cliff_cond"] - b["cliff_cond"]).abs().max()))
- lab = max(lab, float((a["joints_2d"] - b["joints_2d"]).abs().max()) * 112)
- check(box0, "standard: every augmented sample uses window 0")
- check(same_aug and cl < 1e-6, f"standard: same augmentation draws and CLIFF as baseline "
- f"on the standard corpus (CLIFF max diff {cl:.1e})")
- check(lab < 3.0, f"standard: labels within {lab:.2f} px of baseline's (box rounding only)")
- ds = dataset(T, wd[288].parent, cfg_for("baseline", 288), crop_windows=True,
- window_mode="standard", geom_p=1.0)
- rep = 0.0
- for t in range(100):
- smp = seeded(ds, t % len(ds), 900 + t)
- vis = smp["joints_2d_vis"].numpy() > 0
- rep = max(rep, float((np.abs(reproject(smp, 288) - smp["joints_2d"].numpy())[vis]
- * 144).max()))
- check(rep < 1e-3, f"standard at 288: reprojection chain max {rep:.2e} px")
-
- print("[9] --windows on the trainer's command line")
- def cfg_cli(*flags):
- old_argv = sys.argv
- sys.argv = ["train_distill_jz.py", "--cliff-focal", *flags]
- try:
- return JZ.build_cfg(JZ.parse_args())
- finally:
- sys.argv = old_argv
- c = cfg_cli("--preset", "baseline", "--windows", "standard")
- check(c.crop_windows and c.window_mode == "standard" and not c.cliff_follows_aug
- and not c.mask_oob_2d and c.occl_p == 0.2 and c.jpeg_p == 0.0 and c.w_reproj == 0.01,
- "baseline + --windows standard: window 0, baseline augmentation and losses")
- c = cfg_cli("--preset", "baseline", "--windows", "random")
- check(c.crop_windows and c.window_mode == "random" and c.cliff_follows_aug
- and c.mask_oob_2d and c.occl_p == 0.2 and c.occl_scale == (0.02, 0.15)
- and c.jpeg_p == 0.0 and c.w_reproj == 0.01,
- "baseline + --windows random: v4 framings, baseline occlusion, no JPEG")
- c4 = cfg_cli("--preset", "v4")
- check(c4.window_mode == "random" and c4.occl_p == 0.7 and c4.jpeg_p == 0.3,
- "--preset v4 unchanged")
- try:
- cfg_cli("--preset", "v3", "--windows", "standard")
- check(False, "--preset v3 --windows accepted")
- except SystemExit as e:
- check("v3" in str(e), "--preset v3 --windows refused")
-
- print("[10] --window0-p: how often the standard crop is drawn")
- rcfg = cfg_for("baseline")
- for p0 in (0.5, 0.75):
- ds = dataset(T, wd[224].parent, rcfg, crop_windows=True, window0_p=p0)
- counts = np.zeros(K, int)
- for t in range(2000):
- random.seed(5000 + t)
- i = t % len(ds)
- box = np.asarray(ds[i]["bbox_square"])
- wb = load(wd[224], Path(ds.pairs[i][1]).stem)["win_boxes"]
- counts[int(np.argmax([np.array_equal(box, w) for w in wb]))] += 1
- f = counts / counts.sum()
- rest = (1 - p0) / (K - 1)
- check(abs(f[0] - p0) < 0.04 and np.all(np.abs(f[1:] - rest) < 0.04),
- f"p0={p0}: window frequencies {np.round(f, 3).tolist()} "
- f"(expect {p0} then {rest:.3f} each)")
- c = cfg_cli("--preset", "baseline", "--windows", "random", "--window0-p", "0.5")
- check(c.window0_p == 0.5 and c.window_mode == "random", "--window0-p 0.5 reaches the config")
- for bad in (("--preset", "baseline", "--windows", "standard", "--window0-p", "0.5"),
- ("--preset", "baseline", "--window0-p", "0.5"),
- ("--preset", "baseline", "--windows", "random", "--window0-p", "1.5")):
- try:
- cfg_cli(*bad)
- check(False, f"accepted: {' '.join(bad)}")
- except SystemExit:
- check(True, f"refused: {' '.join(bad[2:])}")
-
- if fails:
- print(f"FAIL ({len(fails)})")
- sys.exit(1)
- print("PASS")
-
-
-if __name__ == "__main__":
- main()
diff --git a/training/README.md b/training/README.md
new file mode 100644
index 0000000..a8b21bf
--- /dev/null
+++ b/training/README.md
@@ -0,0 +1,127 @@
+# Training InstantHMR
+
+This folder trains the InstantHMR models from the released ground-truth
+annotations of [SAM 3D Body](https://huggingface.co/datasets/facebook/sam-3d-body-dataset)
+(the MHR fits Meta used to build SAM 3D Body). It is not a distillation setup:
+the labels are the dataset's, not a teacher model's output.
+
+| file | what it is |
+|---|---|
+| `train.py` | The trainer: dataset, augmentation, model, MHR forward pass, losses, metrics, and a single-GPU loop. It also holds the self-checks (`--self-test`, `--overfit-test`). |
+| `train_ddp.py` | The entry point for real runs. It imports `train.py` and adds multi-GPU training (torchrun or Slurm), per-dataset mixture sampling, resume, and 3DPW validation. Without a launcher it runs as one process on one GPU. |
+| `val3dpw.py` | 3DPW validation used by `train_ddp.py --val-3dpw`. |
+| `mhr_cont.py` | The continuous MHR head (`--cont-head`). |
+| `assets/` | Small files the trainer reads: the keypoint regressor, the body-size bounds, the continuous-head and landmark tables, and the MHR-to-J14 adapter for 3DPW validation. |
+
+A change to the model or a loss goes in `train.py`, so both entry points get it.
+
+## Setup
+
+```bash
+pip install -r training/requirements.txt
+```
+
+The MHR rig is not in git. Copy `assets/mhr_model.pt` from the
+[SAM 3D Body checkpoint](https://huggingface.co/facebook/sam-3d-body-dinov3) to
+`training/checkpoints/mhr_model.pt`, or pass `--mhr_model_path`.
+
+## Data
+
+`datasets_pipeline/` downloads the annotations and images and writes one folder
+per dataset, `sam3d_gt_<dataset>/{images,annotations}`; see
+[`datasets_pipeline/README.md`](../datasets_pipeline/README.md).
+
+The released models read a **window corpus**: for each person, four square
+crops cut from the full photo (the standard 1.2x crop and three other
+framings), stored side by side at 288 px. Build it per dataset with:
+
+```bash
+python datasets_pipeline/build_split.py \
+ --split coco_train --annotation-dir <annotations>/coco_train --image-dir <coco images> \
+ --windows 4 --window-sizes 288 --no-hands \
+ --output-dir 'data_win{size}/sam3d_gt_coco'
+```
+
+The released mix uses SA-1B, AI Challenger, Harmony4D, COCO and MPII.
+
+## Training the released models
+
+The four sizes differ only in `--backbone` (`hgnetv2_b0`, `_b1`, `_b2`, `_b4`
+for nano, tiny, small, medium). They were trained on 4 GPUs with 64 crops each:
+
+```bash
+torchrun --standalone --nproc_per_node=4 training/train_ddp.py \
+ --data_root data_win288 --output_dir runs/medium \
+ --backbone hgnetv2_b4 --image-size 288 \
+ --preset baseline --losses rebalanced \
+ --windows random --window0-p 0.75 \
+ --mix sa1b=0.30,aic=0.18,harmony4d=0.10,coco=0.09,mpii=0.03 \
+ --epochs 600 --samples-per-epoch 400000 --batch_size 64 \
+ --lr 3e-4 --lr-scaling sqrt --seed 0 \
+ --w_reproj 0.01 --w_keypoints3d 0.35 --w-verts 0.35 --n-verts 595 \
+ --cont-head --exact-landmarks --cliff-focal --crop-centre-fix \
+ --bound-scales --anomaly-safe-fallback \
+ --val-3dpw <3DPW>/sequenceFiles --val-3dpw-images <3DPW>/imageFiles --val-3dpw-gt h36m
+```
+
+- `--batch_size` is per GPU, and `--lr-scaling sqrt` multiplies `--lr` by the
+ square root of the number of GPUs (6e-4 on 4 GPUs). On another number of
+ GPUs, the effective batch and the learning rate change with it.
+- `--windows random --window0-p 0.75` uses the standard crop 75% of the time
+ and one of the three other framings otherwise.
+- **Medium-commercial** (M-C) is the medium recipe with
+ `--windows standard` (no `--window0-p`) and
+ `--mix harmony4d=0.75,coco=0.25`, on commercially usable images only:
+ Harmony4D (MIT) and the 6,742 COCO people whose Flickr licence is CC BY or
+ "no known copyright". Build that COCO split with
+ `datasets_pipeline/coco_license_subset.py --licenses 4,7` and put it in the
+ `--data_root` beside Harmony4D. The released M-C was stopped at epoch 532 of
+ 600.
+
+These commands rebuild the exact settings recorded in each released model's
+`run_config.json`.
+
+A run writes `last.pth` every epoch and resumes from it when relaunched with the
+same command. `best_student_model_v3.pth` is the checkpoint with the best 3DPW
+validation score.
+
+### 3DPW validation
+
+`--val-3dpw` scores the 3DPW validation split every epoch, on 14 joints from
+the H36M regressor, as in the 3DPW literature. It needs that ground truth built
+once next to `sequenceFiles`:
+
+```bash
+python benchmark/make_3dpw_gt.py --sequence-dir <3DPW>/sequenceFiles \
+ --smpl-dir <folder with SMPL_MALE.pkl, SMPL_FEMALE.pkl> \
+ --regressor <J_regressor_h36m.npy> --out <3DPW>/gt_h36m
+```
+
+Leave `--val-3dpw-split` at `validation`: choosing a checkpoint on the test
+split makes its test score no longer held out. Without `--val-3dpw`, the run
+selects on a 2% slice of the training corpus set aside (`--val-frac`), which
+tells you whether training is healthy, not how well the model generalises.
+
+## Checks before a long run
+
+```bash
+# Every parameter must get a gradient under DistributedDataParallel; a missing
+# one passes every single-GPU run and only aborts a multi-GPU job.
+python tools/ddp_smoke.py --backbone hgnetv2_b4 --image-size 288 \
+ --w-verts 0.35 --cont-head --exact-landmarks --data_root <data>
+
+# The model must be able to fit 8 images.
+python training/train.py --overfit-test --data_root <data>
+
+python -m pytest -q training/test_train_ddp.py
+```
+
+## Export to ONNX
+
+```bash
+python tools/pth_to_onnx.py --ckpt runs/medium/best_student_model_v3.pth \
+ --output models/instanthmr_medium.onnx
+```
+
+The exporter reads the training settings from the `run_config.json` beside the
+checkpoint, so keep the two together.
diff --git a/benchmark/results/adapter_j14_h36m_teacher.npz b/training/assets/adapter_j14_h36m.npz
similarity index 100%
rename from benchmark/results/adapter_j14_h36m_teacher.npz
rename to training/assets/adapter_j14_h36m.npz
diff --git a/instanthmr_distill_train/assets/mhr_cont_head.npz b/training/assets/mhr_cont_head.npz
similarity index 100%
rename from instanthmr_distill_train/assets/mhr_cont_head.npz
rename to training/assets/mhr_cont_head.npz
diff --git a/instanthmr_distill_train/assets/mhr_j127_to_kp70.npy b/training/assets/mhr_j127_to_kp70.npy
similarity index 100%
rename from instanthmr_distill_train/assets/mhr_j127_to_kp70.npy
rename to training/assets/mhr_j127_to_kp70.npy
diff --git a/instanthmr_distill_train/assets/mhr_landmarks70.npz b/training/assets/mhr_landmarks70.npz
similarity index 100%
rename from instanthmr_distill_train/assets/mhr_landmarks70.npz
rename to training/assets/mhr_landmarks70.npz
diff --git a/instanthmr_distill_train/assets/mhr_size_bounds.npz b/training/assets/mhr_size_bounds.npz
similarity index 100%
rename from instanthmr_distill_train/assets/mhr_size_bounds.npz
rename to training/assets/mhr_size_bounds.npz
diff --git a/instanthmr_distill_train/mhr_cont.py b/training/mhr_cont.py
similarity index 100%
rename from instanthmr_distill_train/mhr_cont.py
rename to training/mhr_cont.py
diff --git a/instanthmr_distill_train/requirements.txt b/training/requirements.txt
similarity index 87%
rename from instanthmr_distill_train/requirements.txt
rename to training/requirements.txt
index 0d83e24..a779db2 100644
--- a/instanthmr_distill_train/requirements.txt
+++ b/training/requirements.txt
@@ -1,8 +1,4 @@
# InstantHMR distillation training — pip dependencies.
-# The compute platform builds a virtual environment from this file
-# automatically when submit_train_distill.sh runs (see the sbatchHelpers
-# "if requirements.txt is present" note).
-#
# On Linux, the default PyPI torch wheel already bundles CUDA, so plain
# `torch`/`torchvision` give GPU-enabled builds. If the node needs a specific
# CUDA version, install torch separately from https://pytorch.org first.
diff --git a/training/test_train_ddp.py b/training/test_train_ddp.py
new file mode 100644
index 0000000..f72364f
--- /dev/null
+++ b/training/test_train_ddp.py
@@ -0,0 +1,43 @@
+"""Where train_ddp.py takes its process ranks from: torchrun, Slurm, or neither.
+
+ python -m pytest -q training/test_train_ddp.py
+"""
+import sys
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from train_ddp import ranks_from_env # noqa: E402
+
+
+def test_torchrun_ranks():
+ env = {"RANK": "5", "LOCAL_RANK": "1", "WORLD_SIZE": "8", "MASTER_ADDR": "node0", "MASTER_PORT": "29500"}
+ assert ranks_from_env(env) == (5, 1, 8)
+
+
+def test_slurm_ranks():
+ env = {"SLURM_PROCID": "3", "SLURM_LOCALID": "3", "SLURM_NTASKS": "4", "SLURM_JOB_NODELIST": "n[1-2]"}
+ assert ranks_from_env(env) == (3, 3, 4)
+
+
+def test_single_process_without_a_launcher():
+ assert ranks_from_env({}) == (0, 0, 1)
+
+
+def test_one_process_launches_stay_single_process():
+ # torchrun --nproc_per_node=1 and a one-task Slurm step need no process group
+ assert ranks_from_env({"RANK": "0", "LOCAL_RANK": "0", "WORLD_SIZE": "1"}) == (0, 0, 1)
+ assert ranks_from_env({"SLURM_PROCID": "0", "SLURM_LOCALID": "0", "SLURM_NTASKS": "1"}) == (0, 0, 1)
+
+
+def test_one_torchrun_process_inside_a_multi_task_slurm_job_is_single_process():
+ # sbatch --ntasks=4 + torchrun --nproc_per_node=1: torchrun's world of one decides, not Slurm's 4 tasks
+ env = {"SLURM_PROCID": "0", "SLURM_LOCALID": "0", "SLURM_NTASKS": "4",
+ "RANK": "0", "LOCAL_RANK": "0", "WORLD_SIZE": "1"}
+ assert ranks_from_env(env) == (0, 0, 1)
+
+
+def test_torchrun_wins_when_run_inside_a_slurm_allocation():
+ # torchrun started by hand inside an salloc: Slurm's variables describe the allocation, not the ranks
+ env = {"SLURM_PROCID": "0", "SLURM_LOCALID": "0", "SLURM_NTASKS": "1",
+ "RANK": "2", "LOCAL_RANK": "2", "WORLD_SIZE": "4"}
+ assert ranks_from_env(env) == (2, 2, 4)
diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/training/train.py
similarity index 97%
rename from instanthmr_distill_train/train_distill_mhr_only.py
rename to training/train.py
index ea7c76c..f68d9b8 100644
--- a/instanthmr_distill_train/train_distill_mhr_only.py
+++ b/training/train.py
@@ -33,10 +33,10 @@
# patched before this model can be used there; see the note on
# HMRDeployWrapper below.
#
-# Sanity: python3 train_distill_mhr_only.py --self-test --data_root ../data/sam3d_gt_mpii
-# python3 train_distill_mhr_only.py --overfit-test --data_root ../data/sam3d_gt_mpii
-# Refit W: python3 train_distill_mhr_only.py --fit-regressor --data_root ../data
-# Full: python3 -u train_distill_mhr_only.py --data_root /datasets/instanthmr_data --num_workers 8
+# Sanity: python3 train.py --self-test --data_root ../data/sam3d_gt_mpii
+# python3 train.py --overfit-test --data_root ../data/sam3d_gt_mpii
+# Refit W: python3 train.py --fit-regressor --data_root ../data
+# Full: python3 -u train.py --data_root /datasets/instanthmr_data --num_workers 8
# ============================================================
import os
@@ -70,7 +70,7 @@ import torchvision.transforms as transforms
try:
import mhr_cont as MC
except ImportError: # imported as a package rather than a script
- from instanthmr_distill_train import mhr_cont as MC
+ from training import mhr_cont as MC
import torchvision.transforms.functional as F_t
from torchvision.transforms import InterpolationMode
@@ -90,7 +90,7 @@ device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
print(f"Using device: {device}")
PROJECT_ROOT = Path(__file__).resolve().parent
-DEFAULT_DATA_ROOT = "/pfcalcul/datasets/instanthmr_data/"
+DEFAULT_DATA_ROOT = "data"
# MHR70 joint ordering
JOINT_NAMES = [
@@ -236,7 +236,7 @@ class DistillConfig:
reproj_all_samples: bool = False # include flipped samples in loss_reproj
cam_loss: str = "mse" # "mse" | "euclid" (unsquared L2 on xyz)
- # --- loss-rebalance knobs (`--losses rebalanced` in train_distill_jz.py).
+ # --- loss-rebalance knobs (`--losses rebalanced` in train_ddp.py).
# --- Every default below reproduces the pre-rebalance behaviour bit for bit,
# --- so `--preset baseline` and `--preset v2` are unaffected. All four come
# --- from the measured error/gradient budget:
@@ -342,7 +342,7 @@ class DistillConfig:
# continuous body block, 28 bone-scale PCA coefficients and two 54-dim hand
# blocks -- converted to the 204-vector inside the model, so nothing
# downstream (losses, FK, export, deployment) sees a different contract.
- # See instanthmr_distill_train/mhr_cont.py and docs/todo.md item 1.
+ # See training/mhr_cont.py.
cont_head: bool = False
cont_head_path: str = str(PROJECT_ROOT / "assets/mhr_cont_head.npz")
# Rotation-matrix loss on the root instead of SmoothL1 on its Euler triple.
@@ -372,7 +372,7 @@ class DistillConfig:
w_scale: float = 0.1
w_shape: float = 1.0
w_cam: float = 0.1
- # 2.0, NOT the 10.0 of train_distill_optimized_correctives.py. There, w=10
+ # 2.0, NOT the 10.0 of the earlier trainer. There, w=10
# fed a free 3D coordinate head that had no other supervision. Here the same
# parameters are ALREADY supervised directly by loss_pose / loss_scale, and
# this term back-propagates through the MHR forward kinematics. Measured
@@ -437,32 +437,46 @@ class DistillConfig:
# Splits whose crops are named <...>_<frame>_p<id> with consecutive
# annotations `pair_frame_step` frames apart, and the largest gap (in
# annotation steps) drawn for each. Stills use their own GT as "previous".
- pair_gaps: dict = field(default_factory=lambda: {"harmony4d": 2, "semelle": 4})
+ pair_gaps: dict = field(default_factory=lambda: {"harmony4d": 2})
pair_frame_step: int = 5
# Contact from GT foot motion where no measured contact exists (Harmony4D):
# the mean displacement of a foot's 4 keypoints between the pair, under
# contact_thr is contact, over noncontact_thr is not, in between masked.
contact_thr: float = 0.020
noncontact_thr: float = 0.050
+ # That rule fails on Harmony4D: a foot within 1 cm of the floor still moves
+ # a median 3.1 cm over 5 frames there, so 42% of its pairs come out "both
+ # feet in the air" against ~4% really off the floor. With a floor table
+ # (tools/build_contact_floor.py), a track that has a fitted floor is
+ # labelled by height instead: lowest foot point < floor_contact_h above it
+ # and moved < floor_contact_d is contact, > floor_air_h is not (pair or
+ # not), else masked; 97.3% agreement with insole-measured contact. "" = off.
+ contact_floor: str = ""
+ floor_contact_h: float = 0.020
+ floor_contact_d: float = 0.050
+ floor_air_h: float = 0.050
+ # Contact BCE multiplier on samples with BOTH feet validly off the ground
+ # (flight), the rarest label: ~3 in 10,000 samples.
+ contact_flight_weight: float = 1.0
contact_head: bool = False
w_contact: float = 0.0 # BCE on the per-foot contact logits
w_zero_vel: float = 0.0 # foot displacement vs the previous-pose input, in contact
- w_foot_mocap: float = 0.0 # semelle: mocap foot keypoints (camera frame), replacing
+ w_foot_mocap: float = 0.0 # mocap foot keypoints (camera frame) where a split has them, replacing
# the teacher's in loss_3d_native
- # REMOVED w.r.t. train_distill_optimized_correctives.py:
+ # REMOVED w.r.t. the earlier trainer:
# w_structural / w_feet — they glued the 3D head to the mesh; no head, no glue.
# w_bone_length — MHR enforces bone lengths through its scale params.
# w_reproj_3d/w_reproj_mhr — merged into the single w_reproj above.
# Kept only for the legacy 52-anchor Mesh52_* metrics, so that runs of this
- # script stay comparable with train_distill_optimized_correctives.py.
+ # script stay comparable with the earlier trainer.
# No loss uses them any more.
native_mapping_ids = [ 1, 2, 5, 6, 9, 10, 11, 12, 13, 14, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62 ]
mhr_mapping_ids = [ 125, 123, 75, 39, 2, 18, 3, 19, 5, 21, 64, 63, 62, 61, 59, 58, 57, 56, 55, 54, 53, 52, 51, 50, 49, 48, 47, 46, 45, 44, 41, 100, 99, 98, 97, 95, 94, 93, 92, 91, 90, 89, 88, 87, 86, 85, 84, 83, 82, 81, 80, 77 ]
def apply_rebalanced_losses(cfg) -> None:
"""The four loss-budget fixes, as one switch. Lives here, not in
- train_distill_jz.py, so the single-GPU trainer and the DDP driver cannot
+ train_ddp.py, so the single-GPU trainer and the DDP driver cannot
drift apart.
Every number is chosen from a measured unit-weight gradient norm on real
@@ -513,7 +527,7 @@ def config_from_checkpoint(state, ckpt_path=None, **overrides):
* The INPUT-PIPELINE flags are not. ``--cliff-focal`` changes what the
conditioning vector MEANS, not the graph, so nothing in the weights
records it; it is read from the ``run_config.json`` that
- train_distill_jz.py writes beside the checkpoint. Feeding a focal-aware
+ train_ddp.py writes beside the checkpoint. Feeding a focal-aware
model the pixel form (or the reverse) raises nothing -- it silently
mis-places the person in depth.
@@ -702,7 +716,20 @@ def temporal_kwargs(cfg):
noise_xy=cfg.prev_noise_xy, noise_logz=cfg.prev_noise_logz,
pair_gaps=dict(cfg.pair_gaps), frame_step=cfg.pair_frame_step,
contact_thr=cfg.contact_thr, noncontact_thr=cfg.noncontact_thr,
- pairs_only=getattr(cfg, "prev_pairs_only", False))
+ pairs_only=getattr(cfg, "prev_pairs_only", False),
+ floor=_load_floors(getattr(cfg, "contact_floor", "")),
+ floor_contact_h=getattr(cfg, "floor_contact_h", 0.02),
+ floor_contact_d=getattr(cfg, "floor_contact_d", 0.05),
+ floor_air_h=getattr(cfg, "floor_air_h", 0.05))
+
+
+def _load_floors(path):
+ """tools/build_contact_floor.py output -> {camera track: (normal, point)}."""
+ if not path:
+ return None
+ z = np.load(path)
+ return {str(k): (n.astype(np.float32), a.astype(np.float32))
+ for k, n, a in zip(z["keys"], z["normal"], z["point"])}
class SAM3DStudentDataset(Dataset):
@@ -773,13 +800,13 @@ class SAM3DStudentDataset(Dataset):
# is one glob plus a stat() per crop -- ~7.5 M metadata operations on a
# 3.7 M-crop corpus -- and on a parallel filesystem it dominates job
# startup, so having every DDP rank redo it is the largest fixed cost of
- # a cluster run. See build_pair_index() in train_distill_jz.py.
+ # a cluster run. See build_pair_index() in train_ddp.py.
if pairs is not None:
- # Keep a compact index (train_distill_jz.PairIndex) AS IS. Calling
+ # Keep a compact index (train_ddp.PairIndex) AS IS. Calling
# list() on it would materialise 4.2 M Path pairs -- ~2 GB of Python
# objects that 36 forked dataloader workers then copy page by page
- # through refcount writes, which is what OOM-killed four Jean Zay
- # jobs at 74-76 GiB. Only copy a real list, which is the local /
+ # through refcount writes, which OOM-killed four 4-GPU
+ # jobs at 74-76 GiB per node. Only copy a real list, which is the local /
# single-process path and small.
self.pairs = list(pairs) if isinstance(pairs, (list, tuple)) else pairs
print(f" Prebuilt index: {len(self.pairs)} (image, npz) pairs "
@@ -952,8 +979,7 @@ class SAM3DStudentDataset(Dataset):
M_total = np.hstack([A2 @ A1, (A2 @ t1 + t2)[:, None]])
# context-crop pixel index -> standard-crop pixel index (pixel centres,
- # as cv2.resize maps them on both sides; verified against today's crops
- # in tools/verify_context_crops.py)
+ # as cv2.resize maps them on both sides; verified against standard crops)
ctx_np = np.asarray(img_pil)
sc_y, sc_x = ctx_np.shape[:2]
ax, ay = (ctx[2] - ctx[0]) / sc_x, (ctx[3] - ctx[1]) / sc_y
@@ -998,9 +1024,23 @@ class SAM3DStudentDataset(Dataset):
fm_prev = None
contact, c_valid = np.zeros(2, np.float32), np.zeros(2, np.float32)
+ fl = tc.get("floor")
+ fl = fl.get(PAIR_NAME.match(npz_path.stem).group(1)) if (
+ fl and PAIR_NAME.match(npz_path.stem)) else None
if "contact" in ann:
contact = ann["contact"].astype(np.float32)
c_valid = ann["contact_valid"].astype(np.float32)
+ elif fl is not None:
+ h = (((j_cur + t_cur)[FOOT_KP] - fl[1]) @ fl[0]).reshape(2, 4).min(1)
+ air = h > tc["floor_air_h"]
+ on = h < tc["floor_contact_h"]
+ if has_pair:
+ d = np.linalg.norm((j_cur + t_cur)[FOOT_KP] - (j_prev + t_prev)[FOOT_KP], axis=1)
+ on &= d.reshape(2, 4).mean(1) < tc["floor_contact_d"]
+ else:
+ on[:] = False # no motion to check: only "off" is certain
+ contact = on.astype(np.float32)
+ c_valid = (on | air).astype(np.float32)
elif has_pair:
d = np.linalg.norm((j_cur + t_cur)[FOOT_KP] - (j_prev + t_prev)[FOOT_KP], axis=1)
d = d.reshape(2, 4).mean(1)
@@ -1911,7 +1951,7 @@ class DistillationLoss(nn.Module):
# Built on the MHR module's device, NOT the CPU default: every call site
# in this repo constructs DistillationLoss without .to(device)
# (run_overfit_test, run_self_tests, train_instant_hmr and
- # train_distill_jz.train all do), so a CPU buffer would raise on the
+ # train_ddp.train all do), so a CPU buffer would raise on the
# first step. register_buffer still lets an explicit .to() move it.
w = torch.ones(cfg.num_joints, device=getattr(mhr_module, "device", "cpu"))
w[torch.from_numpy(FINGER_KP).to(w.device)] = float(getattr(cfg, "finger_weight", 1.0))
@@ -2218,7 +2258,7 @@ class DistillationLoss(nn.Module):
# "euclid" also has a FLAT gradient (d||x||/dx is a unit vector), so it is
# gentler than the current form where the error is large -- 0.62x on a
# freshly initialised model -- and only stronger once the model is close.
- # Semelle frames with mocap feet: the teacher's foot keypoints leave this
+ # Frames with mocap feet: the teacher's foot keypoints leave this
# term and loss_foot_mocap supervises them instead (the teacher's feet
# slide from frame to frame; the markers do not).
mocap_feet = (float(getattr(self.cfg, "w_foot_mocap", 0.0)) > 0.0
@@ -2307,7 +2347,11 @@ class DistillationLoss(nn.Module):
if "contact_logits" in preds and cfg.w_contact > 0:
bce = F.binary_cross_entropy_with_logits(
preds["contact_logits"].float(), targets["contact"], reduction="none")
- losses['loss_contact'] = (bce * cv).sum() / cv.sum().clamp_min(1.0) * cfg.w_contact
+ w = cv
+ if getattr(cfg, "contact_flight_weight", 1.0) != 1.0:
+ flight = ((cv > 0).all(1) & (targets["contact"] == 0).all(1)).float()
+ w = cv * (1.0 + (cfg.contact_flight_weight - 1.0) * flight).unsqueeze(1)
+ losses['loss_contact'] = (bce * w).sum() / cv.sum().clamp_min(1.0) * cfg.w_contact
if cfg.w_zero_vel > 0:
feet = torch.as_tensor(FOOT_KP, device=pred_kp3d.device)
e = ((pred_abs[:, feet] - targets["prev_feet"]) - targets["feet_disp"]).norm(dim=-1)
@@ -2412,7 +2456,7 @@ def evaluate_hmr_batch(preds, targets, cfg, mhr_module):
The Head_3D_* metrics are gone with the 3D head. Mesh52_* replicates the
old 52-anchor formula verbatim so that these runs remain comparable with
- train_distill_optimized_correctives.py.
+ the earlier trainer.
"""
metrics = {}
gt_3d_native = targets["joints_3d"][..., :3]
@@ -2644,7 +2688,7 @@ def train_instant_hmr():
skip_reason = f"anomalous loss {loss_val:.1f} > {cfg.anomaly_loss_threshold}"
if skip_reason is not None:
- # Same policy as train_distill_jz: an anomalous batch still
+ # Same policy as train_ddp: an anomalous batch still
# carries the restoring gradient in its non-FK terms, so step
# on those rather than discarding it. Kept in both loops on
# purpose -- the DDP path having a recovery the single-GPU path
@@ -2782,7 +2826,7 @@ def train_instant_hmr():
class HMRDeployWrapper(nn.Module):
"""FOUR outputs: (mhr_params, shape_params, cam_trans, joints_2d).
- BREAKING CHANGE vs train_distill_optimized_correctives.py, which exported a
+ BREAKING CHANGE vs the earlier trainer, which exported a
fifth tensor `joints_3d` straight from the (now deleted) 3D head.
The MHR skeleton block cannot be folded into the ONNX graph: the TorchScript
@@ -2797,7 +2841,7 @@ class HMRDeployWrapper(nn.Module):
* __init__ (~l.119): the comment "Output order from export: ...,
joints_3d" is now wrong; assert len(out_names) == 4 and load the MHR
- TorchScript model + instanthmr_distill_train/assets/mhr_j127_to_kp70.npy.
+ TorchScript model + training/assets/mhr_j127_to_kp70.npy.
* predict (~l.157-162): `joints_3d_local = outs[4][0]` must become
joints_3d_local = self._mhr_keypoints(mhr_params[None])[0]
@@ -3229,7 +3273,7 @@ def run_overfit_test(steps=3000, subset=8, lr=5e-4):
# forward kinematics.
# On the IDENTICAL 52-anchor mesh metric and the same 1000 steps, this script
# reaches 23.8 / 26.6 / 32.4 mm across three seeds where
- # train_distill_optimized_correctives.py reaches 34.5 mm — i.e. the mesh got
+ # the earlier trainer reaches 34.5 mm — i.e. the mesh got
# BETTER, the bar just now measures the mesh instead of a bypass head.
# Observed 70-keypoint spread across seeds is 34-49 mm; 55 mm keeps headroom
# without being vacuous.
diff --git a/instanthmr_distill_train/train_distill_jz.py b/training/train_ddp.py
similarity index 93%
rename from instanthmr_distill_train/train_distill_jz.py
rename to training/train_ddp.py
index 6515316..e04fdf3 100755
--- a/instanthmr_distill_train/train_distill_jz.py
+++ b/training/train_ddp.py
@@ -1,46 +1,43 @@
#!/usr/bin/env python3
-"""Multi-GPU (DDP) training of the InstantHMR baseline on Jean Zay.
+"""Multi-GPU (DDP) training of InstantHMR.
-This is a thin driver around ``train_distill_mhr_only.py``: it imports the
-config, dataset, model, loss and metrics from there rather than copying them,
-so the changes queued in the change plan only ever have to be made once. What
-lives here is the four things the single-GPU script cannot do on a cluster.
+This is a thin driver around ``train.py``: it imports the config, dataset,
+model, loss and metrics from there rather than copying them, so a change to the
+model or a loss is made once. What lives here is what the single-GPU script
+cannot do.
-**1. DistributedDataParallel.** Ranks come from Slurm (``SLURM_PROCID`` /
-``SLURM_LOCALID`` / ``SLURM_NTASKS``) with ``MASTER_ADDR`` taken from the first
-node in the allocation, so one ``srun`` line launches the whole job.
+**1. DistributedDataParallel.** Ranks come from torchrun (``RANK`` /
+``LOCAL_RANK`` / ``WORLD_SIZE``) or, under Slurm's ``srun``, from
+``SLURM_PROCID`` / ``SLURM_LOCALID`` / ``SLURM_NTASKS``. Without a launcher it
+runs as a single process.
**2. Mixture control instead of caps.** SA-1B is 3.40 M of the 3.74 M crops —
-91% of the corpus — so uniform sampling trains almost exclusively on it. The
-existing ``per_dataset_caps`` is a ``random.sample`` applied once at dataset
-construction, which *deletes* crops for the whole run. Here a
+91% of the corpus — so uniform sampling trains almost exclusively on it. A
:class:`DistributedWeightedSampler` draws with replacement from per-dataset
-weights, so every crop stays reachable and the mixture is a knob (``--mix``),
-not a demolition. Default is weight proportional to sqrt(count), which pulls
-SA-1B from 91% to ~69% while keeping COCO and MPII visible.
+weights, so every crop stays reachable and the mixture is a knob (``--mix``).
+Default is weight proportional to sqrt(count), which pulls SA-1B from 91% to
+~69% while keeping COCO and MPII visible.
**3. Epochs decoupled from corpus size.** ``--samples-per-epoch`` fixes the
epoch length, so "epoch" means a checkpoint interval rather than "one pass over
-3.74 M crops" (~58 k steps, ~4 h on one V100). Size it so an epoch is 30-60 min
-and a preemption costs at most that.
+3.74 M crops". Size it so an epoch is 30-60 min and a preemption costs at most
+that.
-**4. Resume by resubmitting.** Every epoch writes ``last.pth`` atomically with
+**4. Resume by relaunching.** Every epoch writes ``last.pth`` atomically with
the *full* state — model, EMA, optimiser, scaler, scheduler, RNG, epoch, best
-metrics. Relaunching the same sbatch picks it up. Note this differs from the
+metrics. Relaunching the same command picks it up. Note this differs from the
single-GPU script, which resumes from ``best_student_model_v3.pth`` and so
-silently discards every epoch since the last improvement.
+discards every epoch since the last improvement.
-Offline caveat: Jean Zay compute nodes have no internet, and
-``InstantHMRStudent(pretrained=True)`` pulls timm weights from HuggingFace. Run
-``--warm-cache`` once on a login or ``prepost`` node to populate
-``$TORCH_HOME``/``$HF_HOME`` before the first training job.
+``InstantHMRStudent(pretrained=True)`` pulls timm weights from Hugging Face. On
+machines without internet access, run ``--warm-cache`` once where there is
+access to populate ``$TORCH_HOME`` / ``$HF_HOME``.
- # once, on a node with the proxy
- python instanthmr_distill_train/train_distill_jz.py --warm-cache
+ # 4 GPUs on one machine
+ torchrun --standalone --nproc_per_node=4 training/train_ddp.py \
+ --data_root data --output_dir runs/my_run
- # then, under srun
- srun python instanthmr_distill_train/train_distill_jz.py \
- --data_root $DATA_ROOT --output_dir $RUNS_DIR/baseline
+See ``training/README.md`` for the settings of the released models.
"""
from __future__ import annotations
@@ -70,22 +67,38 @@ from torch.optim.swa_utils import AveragedModel
from torch.utils.data import DataLoader, Sampler
sys.path.insert(0, str(Path(__file__).resolve().parent))
-import train_distill_mhr_only as T # noqa: E402 (sets up matplotlib/ort at import)
+import train as T # noqa: E402 (sets up matplotlib/ort at import)
# ============================================================
# Distributed plumbing
# ============================================================
+def ranks_from_env(env) -> tuple[int, int, int]:
+ """``(rank, local_rank, world_size)`` from the launcher's environment.
+
+ torchrun sets ``RANK`` / ``LOCAL_RANK`` / ``WORLD_SIZE``; Slurm's ``srun``
+ sets ``SLURM_PROCID`` / ``SLURM_LOCALID`` / ``SLURM_NTASKS``. torchrun is
+ read first, because when it runs inside a Slurm allocation the Slurm
+ variables describe the allocation, not the ranks. A world of one, or no
+ launcher at all, is a single process.
+ """
+ if "WORLD_SIZE" in env:
+ if int(env["WORLD_SIZE"]) > 1:
+ return int(env["RANK"]), int(env["LOCAL_RANK"]), int(env["WORLD_SIZE"])
+ return 0, 0, 1
+ if int(env.get("SLURM_NTASKS", 1)) > 1 and "SLURM_PROCID" in env:
+ return int(env["SLURM_PROCID"]), int(env["SLURM_LOCALID"]), int(env["SLURM_NTASKS"])
+ return 0, 0, 1
+
+
def setup_distributed() -> tuple[int, int, int]:
- """Initialise the process group from Slurm's environment.
+ """Initialise the process group from torchrun's or Slurm's environment.
Returns ``(rank, local_rank, world_size)``. Falls back to a single-process
- group when not launched under ``srun``, so the same file runs on a laptop.
+ group without a launcher, so the same file runs on a laptop.
"""
- if "SLURM_PROCID" in os.environ and int(os.environ.get("SLURM_NTASKS", 1)) > 1:
- rank = int(os.environ["SLURM_PROCID"])
- local_rank = int(os.environ["SLURM_LOCALID"])
- world_size = int(os.environ["SLURM_NTASKS"])
+ rank, local_rank, world_size = ranks_from_env(os.environ)
+ if world_size > 1 and "WORLD_SIZE" not in os.environ: # srun: torchrun sets the rest itself
if "MASTER_ADDR" not in os.environ:
nodelist = os.environ["SLURM_JOB_NODELIST"]
head = subprocess.check_output(
@@ -93,15 +106,13 @@ def setup_distributed() -> tuple[int, int, int]:
).split()[0]
os.environ["MASTER_ADDR"] = head
os.environ.setdefault("MASTER_PORT", "29500")
- else:
- rank, local_rank, world_size = 0, 0, 1
torch.cuda.set_device(local_rank)
if world_size > 1:
dist.init_process_group(
backend="nccl", rank=rank, world_size=world_size,
- # A cold Lustre cache can make the first epoch's data loading slow
- # enough to trip the 30-minute default on the initial barrier.
+ # A cold cache on a shared filesystem can make the first epoch's data
+ # loading slow enough to trip the 30-minute default on the initial barrier.
timeout=_dt.timedelta(minutes=90),
)
return rank, local_rank, world_size
@@ -147,7 +158,7 @@ class PairIndex:
9 workers per rank -- 36 forked processes per node. Fork gives them
copy-on-write pages, but CPython writes a refcount into an object's header
just to *read* it, so every worker steadily converts the shared index into
- a private copy of itself. Measured on Jean Zay: 74-76 GiB resident per node
+ a private copy of itself. Measured on 4-GPU nodes: 74-76 GiB resident per node
against a 160 GB cgroup, and the OOM-killer firing mid-epoch at epochs
14-33 with no Python traceback.
@@ -216,7 +227,7 @@ def build_pair_index(data_root: str, cache: Path | None = None,
rebuild: bool = False, log_fn=print) -> list:
"""The (image, npz) pair list, cached to disk.
- Scanning 3.7 M crops takes over an hour on Lustre and every rank of every
+ Scanning 3.7 M crops takes over an hour on a network filesystem and every rank of every
job in a chained run would otherwise repeat it. The cache stores
``(sub_folder, stem, image_extension)`` rather than full paths -- an order
of magnitude smaller than pickling Path objects -- and paths are rebuilt on
@@ -569,11 +580,11 @@ def train(args, cfg, rank, local_rank, world):
if not ap.is_file():
raise SystemExit(
f"3DPW adapter not found: {ap}\nIt ships in the repo as "
- "benchmark/results/adapter_j14_h36m_teacher.npz; rsync it, "
+ "training/assets/adapter_j14_h36m.npz; "
"or pass --val-3dpw-adapter none to score raw MHR joints.")
dpw_adapter = np.load(ap)["W"]
log(f"3DPW adapter: {ap.name} {tuple(dpw_adapter.shape)} "
- f"-> J14 numbers match benchmark/eval_3dpw_ckpt.py")
+ f"-> J14 PA-MPJPE through the MHR70->J14 adapter")
dpw = ThreeDPWValSet(args.val_3dpw, args.val_3dpw_images,
split=args.val_3dpw_split, stride=args.val_3dpw_stride,
gt=args.val_3dpw_gt, cliff_focal=cfg.cliff_focal,
@@ -713,8 +724,8 @@ def train(args, cfg, rank, local_rank, world):
f"effective batch {cfg.batch_size * world} | peak lr {cfg.lr:.2e}")
if is_main():
- # summarize_runs.py reads this for the epoch total, the preset and the
- # seed; without it every run shows "?/?" and no preset.
+ # The settings of the run, read back by T.config_from_checkpoint (and by
+ # tools/pth_to_onnx.py) to rebuild the model exactly.
atomic_save_json(out_dir / "run_config.json", {
"preset": args.preset, "seed": args.seed, "epochs": args.epochs,
"samples_per_epoch": args.samples_per_epoch,
@@ -748,6 +759,8 @@ def train(args, cfg, rank, local_rank, world):
"prev_mask_p": cfg.prev_mask_p, "prev_pairs_only": cfg.prev_pairs_only,
"w_contact": cfg.w_contact,
"w_zero_vel": cfg.w_zero_vel, "w_foot_mocap": cfg.w_foot_mocap,
+ "contact_floor": cfg.contact_floor,
+ "contact_flight_weight": cfg.contact_flight_weight,
"started": time.strftime("%Y-%m-%d %H:%M")})
ema_rollbacks = 0
@@ -845,9 +858,9 @@ def train(args, cfg, rank, local_rank, world):
log(f"⚠️ skipping step {i} (loss {lv:.1f}) "
f"[{n_skip_run} consecutive]")
- # EMA rollback. This existed only in train_distill_mhr_only's
+ # EMA rollback. This existed only in train's
# single-GPU loop; the DDP path had the abort and no recovery,
- # so every Jean Zay divergence burned the whole allocation.
+ # so every divergence burned the whole job.
# Both branches are driven by the already-all-reduced skip
# count, so all ranks roll back on the same step, and the EMA
# is bit-identical across ranks (DDP keeps the weights in sync
@@ -948,11 +961,9 @@ def train(args, cfg, rank, local_rank, world):
dpw_raw["J14_MPJPE"], dpw_em["J14_MPJPE"])
# Peak GPU across ALL ranks, not just rank 0. Rank 0 is not the
- # hungriest rank: on 2026-09-08 four gen-6 jobs died of CUDA OOM on a
- # 16 GB V100 reporting 14.99 GiB allocated, while this line -- rank 0
- # only -- printed a flat 9.3 GiB for 32 straight epochs. The "measured
- # peak use is 9.3 GiB" note that justified running without
- # --constraint=v100-32g came from exactly this blind number.
+ # hungriest rank: four jobs once died of CUDA OOM on a 16 GB GPU
+ # reporting 14.99 GiB allocated, while this line -- rank 0 only --
+ # printed a flat 9.3 GiB for 32 straight epochs.
_pk = torch.cuda.max_memory_allocated() / 2**30
pk_max, pk_rank = _pk, 0
if dist.is_initialized() and world > 1:
@@ -1057,7 +1068,7 @@ def train(args, cfg, rank, local_rank, world):
log(f"\n🛑 early stop: no improvement for {cfg.early_stop_patience} epochs")
break
- # The sbatch resubmits itself unless this file exists, so it must be written
+ # A self-resubmitting job script can stop on this file, so it must be written
# on normal completion only — never in an exception path.
if is_main():
(out_dir / "DONE").write_text(
@@ -1115,7 +1126,7 @@ def parse_args():
p.add_argument("--val-3dpw-split", default="validation",
choices=("validation", "test", "train"),
help="Keep this at 'validation'. Selecting on 'test' spends "
- "the split benchmark/eval_3dpw.py reports.")
+ "the split a paper reports.")
p.add_argument("--val-3dpw-test", action="store_true",
help="ALSO evaluate the 24-sequence test split each epoch and "
"log it, so you can watch the number the paper will "
@@ -1124,11 +1135,10 @@ def parse_args():
p.add_argument("--val-3dpw-stride", type=int, default=5,
help="3DPW is 30 fps; neighbouring frames are near-duplicates.")
p.add_argument("--val-3dpw-adapter",
- default="benchmark/results/adapter_j14_h36m_teacher.npz",
- help="Fixed MHR70->J14 map applied before scoring, so the "
- "training log reports the same number as "
- "benchmark/eval_3dpw_ckpt.py and the paper table. "
- "'none' scores the raw MHR joints instead.")
+ default="training/assets/adapter_j14_h36m.npz",
+ help="Fixed MHR70->J14 map, fitted on the training annotations, "
+ "applied before scoring. 'none' scores the raw MHR joints "
+ "instead.")
p.add_argument("--cliff-focal", action="store_true",
help="Perspective-correct CLIFF conditioning: angles off the "
"optical axis and angular box size instead of "
@@ -1230,13 +1240,22 @@ def parse_args():
p.add_argument("--prev-token", dest="prev_token", action="store_true",
help="Foot lock: an extra decoder query carrying the previous "
"frame's 70 keypoints + cam_trans (withheld with "
- "--prev-mask-p), real pairs on Harmony4D / semelle, GT + "
+ "--prev-mask-p), real pairs on Harmony4D, GT + "
"noise elsewhere. Also turns on the per-foot contact head.")
p.add_argument("--prev-mask-p", dest="prev_mask_p", type=float, default=None)
p.add_argument("--prev-pairs-only", dest="prev_pairs_only", action="store_true",
help="With --prev-token: withhold the previous pose on stills "
"instead of feeding their own GT + noise, so it only ever "
- "comes from a real earlier frame (Harmony4D / semelle).")
+ "comes from a real earlier frame (Harmony4D).")
+ p.add_argument("--contact-floor", dest="contact_floor", default=None, metavar="NPZ",
+ help="With --prev-token: label Harmony4D contact by height above "
+ "the per-camera floor of this table (tools/build_contact_floor.py) "
+ "instead of by foot motion alone, which calls 42%% of its "
+ "pairs 'both feet in the air'. Tracks without a floor keep "
+ "the motion rule.")
+ p.add_argument("--contact-flight-weight", dest="contact_flight_weight", type=float,
+ default=None, help="With --prev-token: contact-loss multiplier on "
+ "samples with both feet off the ground (default 1).")
p.add_argument("--w-contact", dest="w_contact", type=float, default=None)
p.add_argument("--w-zero-vel", dest="w_zero_vel", type=float, default=None)
p.add_argument("--w-foot-mocap", dest="w_foot_mocap", type=float, default=None)
@@ -1291,7 +1310,7 @@ def parse_args():
help="Rescan the corpus even if the cache looks current.")
p.add_argument("--warm-cache", action="store_true",
help="Instantiate the backbone to populate $TORCH_HOME / "
- "$HF_HOME, then exit. Run on a login or prepost node.")
+ "$HF_HOME, then exit. Run where there is internet access.")
p.add_argument("--dry-run", action="store_true",
help="Build config, dataset and sampler, print the mixture, "
"then exit without training.")
@@ -1331,7 +1350,10 @@ def build_cfg(args):
if not args.prev_token:
raise SystemExit("--prev-pairs-only needs --prev-token")
cfg.prev_pairs_only = True
- for k in ("prev_mask_p", "w_contact", "w_zero_vel", "w_foot_mocap"):
+ if args.contact_floor and not Path(args.contact_floor).is_file():
+ raise SystemExit(f"--contact-floor: no such file {args.contact_floor}")
+ for k in ("prev_mask_p", "w_contact", "w_zero_vel", "w_foot_mocap",
+ "contact_floor", "contact_flight_weight"):
if getattr(args, k) is not None:
if not args.prev_token:
raise SystemExit(f"--{k.replace('_', '-')} needs --prev-token")
@@ -1446,7 +1468,7 @@ def apply_v4_preset(cfg) -> None:
is the standard 1.2x crop, the others v3 framings. v3 cut its windows from a
448 px context crop with a second warp, and its models learned that texture
(3DPW val J14 PA, g9_b4_224 ep314: 49.3 on crops cut from the frame vs 43.6
- on crops built its own way; g8h 40.0 vs 39.4, STATUS.md 2026-10-21).
+ on crops built its own way; g8h 40.0 vs 39.4).
On top: v2's occlusion and JPEG with the two prerequisites truncated
framings need, and baseline geometry (the windows already supply the zoom
@@ -1472,7 +1494,7 @@ def main():
f"params under TORCH_HOME={os.environ.get('TORCH_HOME')} "
f"HF_HOME={os.environ.get('HF_HOME')}")
# The pair index too: this is the other thing a compute node should
- # never have to build, and a prepost node has the walltime for it.
+ # never have to build.
build_pair_index(cfg.data_root, args.index_cache,
rebuild=args.rebuild_index)
return
@@ -1499,7 +1521,7 @@ def main():
raise SystemExit(
f"keypoint regressor missing: {cfg.kp_regressor_path}\n"
f"It ships with the repo; if absent, fit it once with\n"
- f" python instanthmr_distill_train/train_distill_mhr_only.py "
+ f" python training/train.py "
f"--fit-regressor --data_root {cfg.data_root}")
if not Path(cfg.mhr_model_path).exists():
raise SystemExit(f"MHR TorchScript rig missing: {cfg.mhr_model_path}")
diff --git a/instanthmr_distill_train/val3dpw.py b/training/val3dpw.py
similarity index 91%
rename from instanthmr_distill_train/val3dpw.py
rename to training/val3dpw.py
index 222305d..2106d1d 100644
--- a/instanthmr_distill_train/val3dpw.py
+++ b/training/val3dpw.py
@@ -9,28 +9,25 @@ different and more interesting question.
Two rules that matter more than the code:
**Use the ``validation`` split (12 sequences), not ``test``.** The 24 test
-sequences are what ``benchmark/eval_3dpw.py`` reports for publication. Select a
+sequences are what papers report. Select a
checkpoint on them and that number stops being held-out — it becomes a number
you optimised against. The validation split exists precisely so you don't have
to spend the test split on model selection.
**Never build ``3dpw_train`` into the training corpus.** Neighbouring frames of
-the same person in the same scene make frame-level holdout meaningless. This is
-already the situation locally, where ``sam3d_distill_mix`` contains crops from
-all 60 sequences (see ``benchmark/README.md``). On Jean Zay it is not: the built
-corpus is coco/mpii/aic/sa1b only, so 3DPW there is genuinely unseen — which is
-what makes this worth doing on the cluster and not on the laptop.
+the same person in the same scene make frame-level holdout meaningless. The
+released training mix (SA-1B, AIC, Harmony4D, COCO, MPII) contains no 3DPW
+image, so 3DPW is genuinely unseen.
GT is the published protocol by default — SMPL forward + the H36M regressor,
-precomputed by ``benchmark/make_3dpw_gt.py`` into ``<3DPW root>/gt_h36m/`` and
-rsynced next to ``sequenceFiles``. ``gt="jointpositions"`` restores the raw
-pickle field the harness used before; the two are several mm apart, so numbers
+precomputed by ``benchmark/make_3dpw_gt.py`` into ``<3DPW root>/gt_h36m/``
+next to ``sequenceFiles``. ``gt="jointpositions"`` uses the raw pickle field
+instead; the two are several mm apart, so numbers
from the two sources must never be compared to each other.
Selection metric is **J14 PA-MPJPE with the MHR70 -> J14 adapter applied** —
-the same quantity `benchmark/eval_3dpw_ckpt.py` reports and a paper quotes, so
-the training log, the run summary and the results table all carry one number.
-The adapter (`benchmark/results/adapter_j14_h36m_teacher.npz`) is a fixed 14x70
+the 3DPW protocol of the literature (14 joints from the H36M regressor).
+The adapter (`training/assets/adapter_j14_h36m.npz`) is a fixed 14x70
map fitted from the teacher's MHR labels, independent of any student, so it
does not move between checkpoints or between runs.
"""
@@ -69,7 +66,7 @@ def _preprocess(image_rgb: np.ndarray, bbox: np.ndarray, h: int, w: int,
duplicated rather than imported because that module pulls in onnxruntime,
and this one runs inside the training loop on every rank. If the crop
convention there ever changes, this must follow or the validation numbers
- will silently stop matching ``benchmark/eval_3dpw.py``.
+ will silently stop matching the deployed model.
"""
x1, y1, x2, y2 = np.asarray(bbox, dtype=np.float64)
bw, bh = x2 - x1, y2 - y1
@@ -196,7 +193,7 @@ def evaluate(model, loader, mhr_module, device, use_amp: bool = True,
preds.append(kp[keep.to(kp.device)].float().cpu().numpy())
gts.append(b["gt"][keep].numpy())
- # Always return the SAME key set. train_distill_jz.all_reduce_mean builds a
+ # Always return the SAME key set. train_ddp.all_reduce_mean builds a
# tensor from sorted(keys) and all-reduces it, so a rank returning a short
# dict while the others return a full one is a size mismatch -> NCCL hang.
res: dict[str, float] = {"n": 0.0, "n_dropped": float(n_dropped),