Python
feat: --preset v3 samples crops from context crops over a wide framing range; 288 px phone latencies
mohamdev2cf4b455805f
diff --git a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md b/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.mdindex 6bcc3f6..dd691d3 100644--- a/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md+++ b/.scratch/v3-framing-robustness/issues/01-archive-aic-mpii-originals.md@@ -4,8 +4,12 @@**Blocked by:** None (can start immediately; already queued as job 189727).-**Status:** ready-for-human+**Status:** done (2026-09-26)-- [ ] `hmr_archive_<jobid>.out` ends with an `ls -lh` listing five tars: aic, coco, mpii, sa1b, harmony4d-- [ ] The aic and mpii tars hold real frames (hard links/copies), not links: `tar tvf` shows regular files under `original_images/`-- [ ] If the job timed out on sa1b, the partial sa1b tar was deleted before resubmitting (the script skips any existing tar, including a half-written one)+- [x] `hmr_archive_<jobid>.out` ends with an `ls -lh` listing five tars: aic, coco, mpii, sa1b, harmony4d+- [x] The aic and mpii tars hold real frames (hard links/copies), not links: `tar tvf` shows regular files under `original_images/`+- [x] If the job timed out on sa1b, the partial sa1b tar was deleted before resubmitting (the script skips any existing tar, including a half-written one)++## Answer++Job 189727 wrote five split tars, but `40_archive.slurm`'s unanchored `--exclude=images` also dropped every nested `images/` folder -- i.e. all AIC and MPII original photos (0 in either tar; mpii.tar 228 MB). Fixed in the template (exclude only a top-level `images` symlink, anchored; tested on a fake tree) and rsynced. The photos were archived separately by job 226766: `sam3d_gt_aic_originals.tar` 13 GB, `sam3d_gt_mpii_originals.tar` 2.6 GB. `sam3d_gt_harmony4d.tar` verified to hold all 496,036 crops. STORE total 155 GB.diff --git a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md b/.scratch/v3-framing-robustness/issues/06-profile-288-input.mdindex 49872c1..9ffc66c 100644--- a/.scratch/v3-framing-robustness/issues/06-profile-288-input.md+++ b/.scratch/v3-framing-robustness/issues/06-profile-288-input.md@@ -4,8 +4,22 @@**Blocked by:** 05-**Status:** ready-for-agent+**Status:** done (2026-09-25), uncommitted-- [ ] NPU latency on Galaxy S22, S23, S24 and S25; CPU latency on S23, A73, A53, A14 and Pixel 8-- [ ] 100% NPU placement and fp16 parity on the S23-- [ ] Result recorded next to ticket 03's table, with the 224-vs-288 latency ratio per phone+- [x] NPU latency on Galaxy S22, S23, S24 and S25; CPU latency on S23, A73, A53, A14 and Pixel 8+- [~] 100% NPU placement on all four Snapdragon phones (yes); fp16 parity deferred to trained weights (ticket 11), as in ticket 03+- [x] Result recorded next to ticket 03's table, with the 224-vs-288 latency ratio per phone++## Answer++Random-weight exports at 288 px (`tools/profiling/export_family.py --image-size 288`), same method as ticket 03. Raw rows: `benchmark/results/family_latency_aihub_288.json`.++| model @ 288 | S22 NPU | S23 NPU | S24 NPU | S25 NPU | S23 CPU | A73 CPU | Pixel 8 CPU | A53 CPU | A14 CPU |+|---|---|---|---|---|---|---|---|---|---|+| B4 | 5.5 | 3.27 | 2.31 | 1.88 | 93.3 | 119.0 | 115.0 | 288.7 | 294.2 |+| B0 | 3.7 | 2.05 | 1.31 | 1.00 | 29.3 | 38.2 | 32.1 | 93.5 | 79.0 |++Cost of 288 over 224 (same backbone): B4 x1.32 on the S23 NPU, x1.46-1.55 on CPUs; B0 x1.28 NPU, x1.07-1.23 CPU. All exports 100% on the NPU.++- **The 288 arm is deployable on the NPU tier:** B4 at 288 is 1.9-5.5 ms across Snapdragon 8 Gen 1-Elite, far inside 33 ms. Ticket 10 may train it.+- **Not for the CPU tier with B4** (93-294 ms). B0 at 288 stays near budget on mid-range CPUs (Pixel 8 32.1, A73 38.2) because below B4 the decoder, not the image, dominates cost -- a 288 CPU-tier member is plausible if 288 proves worth it.diff --git a/.scratch/v3-framing-robustness/issues/07-v3-preset.md b/.scratch/v3-framing-robustness/issues/07-v3-preset.mdindex f1f6590..72da005 100644--- a/.scratch/v3-framing-robustness/issues/07-v3-preset.md+++ b/.scratch/v3-framing-robustness/issues/07-v3-preset.md@@ -4,11 +4,24 @@**Blocked by:** 04, 05-**Status:** ready-for-agent--- [ ] A small local split rebuilt with ticket 04's builder (context 2.0 / 448) trains under v3 at 224 and at 288-- [ ] Verification on real samples: 2D labels land on the right pixels; CLIFF crop-centre error at the rounding floor (mean / p95 / max px, as in the crop-centre-fix check); no black pixel inside the stored context; box-scale draws cover 0.6–2.0x; output size equals the configured input size-- [ ] baseline and v2 samples bit-identical against HEAD (alternating calls)-- [ ] The v3 dataset refuses a split without context geometry, with a message naming the split-- [ ] `ddp_smoke` passes for v3 at 224 and at 288-- [ ] Files that must be rsynced to the cluster listed in the status doc+**Status:** done (2026-09-25), uncommitted++- [x] A small local split rebuilt with ticket 04's builder (context 2.0 / 448) trains under v3 at 224 and at 288+- [x] Verification on real samples: 2D labels land on the right pixels; CLIFF crop-centre error at the rounding floor (mean / p95 / max px, as in the crop-centre-fix check); no black pixel inside the stored context; box-scale draws cover 0.6–2.0x; output size equals the configured input size+- [x] baseline and v2 samples bit-identical against HEAD (alternating calls)+- [x] The v3 dataset refuses a split without context geometry, with a message naming the split+- [x] `ddp_smoke` passes for v3 at 224 and at 288+- [x] Files that must be rsynced to the cluster listed in the status doc++## Answer++Implemented as specified, with two decisions taken with the user (2026-09-25):+- **Clamp (Q22, option b):** keep the rotated window inside the stored square only on sides where the square ends inside the photo. Tail restored: P(>1.6x) 17.0% (unclamped 18.5%, all-sides clamp 10.5%), P(>1.8x) 6.6% (8.8% / 1.9%), loose windows keep their shift.+- **geom_p (Q23):** 80/20 as v2; the 20% identity windows keep root-rotation supervision. Revisit only if v3 helps.++Evidence: `tools/verify_v3_preset.py` PASS (see README section `--preset v3`). `ddp_smoke --context-crops` passes at 224 and 288. A 1-epoch `train_distill_jz.py --preset v3 --image-size 288` run on the local context split completed and recorded preset / image_size / v3 fields in run_config.json. Rsync list and cluster checks: STATUS.md, "Code ready, not launched".++Findings:+- 95% of 2.0x context squares (AIC, COCO) run past the frame; windows >1.5x are ~38% black on average (frame edge, representative of deployment). Harmony4D / SA-1B may differ; measure after the rebuild.+- Naming: `--preset v3` is unrelated to the old `v3_s{0,1,2}` runs (v2 + rebalance). Noted in STATUS.+- `apply_v3_preset` forces `cliff_focal`.diff --git a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.mdindex def5f0e..22b9f6d 100644--- a/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md+++ b/.scratch/v3-framing-robustness/issues/09-rebuild-harmony4d.md@@ -2,10 +2,14 @@**What to build:** Harmony4D rejoins the corpus as context crops. Its original frames exist nowhere, so its zips are streamed scene by scene: download, undistort, build context crops, delete. The route depends on the `prepost` download test: HTTP 206 means the streaming job runs on Jean Zay (keeping no 4K originals); otherwise the same script runs on the laptop and the crops are rsynced as one tar.-**Blocked by:** 04, and the `prepost` curl result (job 190182)+**Blocked by:** None (04 done; route decided)-**Status:** ready-for-human+**Status:** ready-for-human -- route 2 (laptop)-- [ ] Route chosen from the `prepost` result and recorded+- [x] Route chosen from the `prepost` result and recorded- [ ] Undistortion blockers in the undistortion notes handled before the first full scene- [ ] 496,036 train crops (± source-missing), verified and archived to `$STORE`++## Route decided (2026-09-27): laptop++`prepost` job 226661 returned `HTTP 302, 0 bytes`: Hugging Face's redirect to its file host `us.aws.cdn.hf.co` cannot be followed from Jean Zay compute nodes (the login node could, 2026-09-25; same failure as STATUS 7.1 on 2026-09-03). So stream on the laptop, scene by scene (65 GB free: download one zip, undistort, build context crops with `--context 2.0 --body-size 448 --output-dir`, delete), then rsync one tar of crops to Jean Zay's context data root.diff --git a/benchmark/results/family_latency_aihub_288.json b/benchmark/results/family_latency_aihub_288.jsonnew file mode 100644index 0000000..f9fe6ec--- /dev/null+++ b/benchmark/results/family_latency_aihub_288.json@@ -0,0 +1,218 @@+[+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S22 5G",+ "job": "jg9zoorwp",+ "ms": 3.7,+ "peak_mb": 150.2,+ "layers": {+ "NPU": 827+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S23",+ "job": "jp1noo98g",+ "ms": 2.05,+ "peak_mb": 90.1,+ "layers": {+ "NPU": 827+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S24",+ "job": "jgdd66krg",+ "ms": 1.31,+ "peak_mb": 169.7,+ "layers": {+ "NPU": 827+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S25",+ "job": "j57eoomvp",+ "ms": 1.0,+ "peak_mb": 167.3,+ "layers": {+ "NPU": 827+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy S23",+ "job": "jp4yee78p",+ "ms": 29.25,+ "peak_mb": 139.7,+ "layers": {+ "CPU": 722+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A73 5G",+ "job": "jpxl00q3p",+ "ms": 38.2,+ "peak_mb": 125.0,+ "layers": {+ "CPU": 722+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A53 5G",+ "job": "j5m0997dg",+ "ms": 93.51,+ "peak_mb": 125.8,+ "layers": {+ "CPU": 722+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A14 5G",+ "job": "jgnz114kg",+ "ms": 78.97,+ "peak_mb": 182.6,+ "layers": {+ "CPU": 722+ }+ },+ {+ "model": "random_b0_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Google Pixel 8",+ "job": "jprlxxr0p",+ "ms": 32.06,+ "peak_mb": 142.9,+ "layers": {+ "CPU": 722+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S22 5G",+ "job": "jp2roo1rg",+ "ms": 5.5,+ "peak_mb": 174.3,+ "layers": {+ "NPU": 890+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S23",+ "job": "jpyo88l85",+ "ms": 3.27,+ "peak_mb": 159.2,+ "layers": {+ "NPU": 890+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S24",+ "job": "jp0moow9g",+ "ms": 2.31,+ "peak_mb": 232.4,+ "layers": {+ "NPU": 890+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "npu",+ "device": "Samsung Galaxy S25",+ "job": "jp8ejjnkp",+ "ms": 1.88,+ "peak_mb": 141.0,+ "layers": {+ "NPU": 890+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy S23",+ "job": "jgk2661wg",+ "ms": 93.32,+ "peak_mb": 173.5,+ "layers": {+ "CPU": 762+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A73 5G",+ "job": "j5ql44nnp",+ "ms": 119.02,+ "peak_mb": 160.3,+ "layers": {+ "CPU": 762+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A53 5G",+ "job": "jglywwdj5",+ "ms": 288.74,+ "peak_mb": 146.3,+ "layers": {+ "CPU": 762+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Samsung Galaxy A14 5G",+ "job": "j568oox6g",+ "ms": 294.21,+ "peak_mb": 147.4,+ "layers": {+ "CPU": 762+ }+ },+ {+ "model": "random_b4_288",+ "image_size": 288,+ "unit": "cpu",+ "device": "Google Pixel 8",+ "job": "jp3zood35",+ "ms": 115.04,+ "peak_mb": 198.5,+ "layers": {+ "CPU": 762+ }+ }+]\ No newline at end of filediff --git a/instanthmr_distill_train/README.md b/instanthmr_distill_train/README.mdindex c522325..9251cf8 100644--- a/instanthmr_distill_train/README.md+++ b/instanthmr_distill_train/README.md@@ -609,6 +609,44 @@ and must be rsynced**, or the job dies at startup. Default off; with it offevery loss term is bit-identical, verified across `legacy`, `rebalanced`,`w_verts` on/off and `cont_head` on/off.+## `--preset v3` — framing robustness from context crops (added 2026-09-25)++v2, plus: every crop is cut from a **context crop** (`build_split.py --context+2.0 --body-size 448`, its own `--data_root`) through a window drawn in frame+pixels -- side log-uniform in 0.6-2.0x the tight box (1.2 = today's crop, 0.6 =+v2's deepest truncation), centre shifted up to +/-15% of the side, v2's+rotation and flip. It is one resampling step from the stored 448 px crop, so+zoom-in no longer upsamples a 224 crop. 20% of samples (`geom_p` 0.8, as v2)+keep the identity window, i.e. today's crop, which also keeps root-rotation+supervision on the un-augmented subset. Validation uses the identity window.++**Why.** g8h loses 12-15 mm on EMDB under EMDB's own boxes, and 16 mm once our+own boxes are enlarged to 1.6x -- the loosest framing a 1.2x-cropped corpus can+show, and then only with black margins. v2 cannot help: it only zooms in.++**The window is an ordinary `aug_M`** relative to `bbox_square`, so labels, the+reprojection loss and the model are untouched; the CLIFF vector is computed+from the exact window (centre, side / 1.2) and then flipped and rotated with+the labels. `apply_v3_preset` forces `cliff_focal`.++**The clamp.** The rotated window is kept inside the stored square only on+sides where that square ends inside the photo. Where it already runs past the+photo's edge, further out is outside the photo too, so the black read there is+the real frame edge -- what a deployment crop shows when its box leaves the+frame. Clamping every side instead thinned windows looser than 1.6x from 18.5%+to 10.5% and centred them; this rule keeps 17.0%, with their shift intact.++**Expect black.** In AIC / COCO the person usually nearly fills the photo: 95%+of 2.0x squares run past the frame, and windows looser than 1.5x are on average+38% black (frame edge). That is representative of deployment, but it caps how+much loose-and-real-scene framing this corpus can teach.++Verified by `tools/verify_v3_preset.py` on real AIC rows built both ways:+baseline / v2 bit-identical; identity window = today's labels and CLIFF+exactly; reprojection chain vs labels 2e-4 px; CLIFF vs the recovered window+0.000 px; no read past an inside-photo stored edge; standard or mixed roots+refused (also through `PairIndex`).+## `--image-size` — the network input side (added 2026-09-25)Default 224, i.e. unchanged. Must be a multiple of 32, because the decoder'sdiff --git a/instanthmr_distill_train/train_distill_jz.py b/instanthmr_distill_train/train_distill_jz.pyindex 4e38c99..49f55f2 100755--- a/instanthmr_distill_train/train_distill_jz.py+++ b/instanthmr_distill_train/train_distill_jz.py@@ -171,6 +171,11 @@ class PairIndex:def __len__(self):return len(self._stem)+ def first_per_subfolder(self):+ """Index of the first pair in each sub-folder (one np.unique over the+ byte array; no Path objects)."""+ return [int(k) for k in np.unique(self._sub, return_index=True)[1]]+def __getitem__(self, i):if isinstance(i, slice):# Stay compact through --max_images too, or the one configuration@@ -390,7 +395,9 @@ def build_jz_loaders(cfg, args, rank: int, world: int):cliff_focal=cfg.cliff_focal,occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,- image_size=cfg.image_size)+ image_size=cfg.image_size,+ context_crops=cfg.context_crops, v3_box_min=cfg.v3_box_min,+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)# Rank 0 builds the index (or reads the cache) while the others wait, then# everyone loads the same cache. Building on all ranks at once would put@@ -701,7 +708,8 @@ def train(args, cfg, rank, local_rank, world):"lr": cfg.lr, "lr_arg": args.lr, "lr_scaling": args.lr_scaling,"mix": args.mix, "steps_per_epoch": steps_per_epoch,"backbone": cfg.backbone, "w_reproj": cfg.w_reproj,- "image_size": cfg.image_size,+ "image_size": cfg.image_size, "context_crops": cfg.context_crops,+ "v3_box": [cfg.v3_box_min, cfg.v3_box_max], "v3_shift": cfg.v3_shift,"cont_head": cfg.cont_head, "root_rot_loss": cfg.root_rot_loss,"exact_landmarks": cfg.exact_landmarks,"geom_scale_max": cfg.geom_scale_max, "occl_p": cfg.occl_p,@@ -1154,10 +1162,13 @@ def parse_args():help="Skip the timm download. Compute nodes have no internet, ""so run --warm-cache once on a login node instead.")p.add_argument("--log-every", type=int, default=100)- p.add_argument("--preset", choices=["baseline", "v2"], default="baseline",+ p.add_argument("--preset", choices=["baseline", "v2", "v3"], default="baseline",help="'v2' enables the wave-1 bundle: truncation, occlusion ""and JPEG augmentation with the SimCC-mask and CLIFF ""prerequisites, plus absolute-pose supervision. "+ "'v3' is v2 with every crop sampled from a context-crop "+ "split over a wide range of framings (needs --data_root "+ "of build_split.py --context splits). ""'baseline' reproduces the pre-v2 trainer exactly.")p.add_argument("--w_reproj", type=float, default=None)p.add_argument("--crop-centre-fix", dest="crop_centre_fix", action="store_true",@@ -1257,8 +1268,10 @@ def build_cfg(args):raise SystemExit(f"--image-size {args.image_size} is not a multiple of 32")cfg.image_size = args.image_size- if args.preset == "v2":+ if args.preset in ("v2", "v3"):apply_v2_preset(cfg)+ if args.preset == "v3":+ apply_v3_preset(cfg)# Orthogonal axis: --preset is the input pipeline + absolute-pose terms,# --losses is the loss budget. Applied second so it owns w_keypoints3d.if args.losses == "rebalanced":@@ -1300,6 +1313,26 @@ def apply_v2_preset(cfg) -> None:cfg.cam_loss = "euclid"+def apply_v3_preset(cfg) -> None:+ """v2 plus framing robustness: every crop is sampled from a context crop+ (build_split.py --context 2.0 --body-size 448) over a wide range of+ framings -- loose and off-centre like EMDB's and a phone detector's boxes,+ tight and truncated like a person close to the camera -- always surrounded+ by real scene. Applied on top of apply_v2_preset: v2's occlusion, JPEG,+ rotation, flip and absolute-pose terms are kept; its zoom/shift (which can+ only zoom INTO a 1.2x crop) are replaced by the context window.++ Measured motivation (2026-09-25): g8h loses 12-15 mm on EMDB under EMDB's+ own boxes, and 16 mm when our boxes are enlarged to 1.6x -- the loosest+ framing the 1.2x corpus can show, and only with black margins."""+ cfg.context_crops = True+ cfg.v3_box_min, cfg.v3_box_max = 0.6, 2.0+ cfg.v3_shift = 0.15+ # The window's CLIFF vector is built in the perspective-correct form; the+ # pixel form is ambiguous on this corpus's real focals. Mandatory, not a flag.+ cfg.cliff_focal = True++def main():args = parse_args()diff --git a/instanthmr_distill_train/train_distill_mhr_only.py b/instanthmr_distill_train/train_distill_mhr_only.pyindex 75395d1..f13ffce 100644--- a/instanthmr_distill_train/train_distill_mhr_only.py+++ b/instanthmr_distill_train/train_distill_mhr_only.py@@ -170,6 +170,14 @@ class DistillConfig:log_dir: str = str(PROJECT_ROOT / "runs/distill_mhr_only")image_size: int = 224+ # --preset v3: sample each crop from a context crop (build_split.py+ # --context) over a wide range of framings. The box multiple is drawn+ # log-uniformly in [v3_box_min, v3_box_max] x the tight box (1.2 = today's+ # crop), the centre shifted by up to +/- v3_shift of the window side.+ context_crops: bool = False+ v3_box_min: float = 0.6+ v3_box_max: float = 2.0+ v3_shift: float = 0.15max_images: int | None = Noneper_dataset_caps: dict = field(default_factory=lambda: {"sam3d_gt_harmony4d": 300_000})val_split: float = 0.1@@ -556,6 +564,43 @@ class RandomPixelate:img = F_t.resize(img, [orig_h, orig_w], interpolation=InterpolationMode.NEAREST)return img+def _apply_label_geometry(M_total, aug_flip, angle, joints_2d, joints_2d_vis,+ joints_3d, cam_trans, cx, cy, orig_w, orig_h, W, H):+ """Move every label with an augmentation: joints_2d through M_total (crop+ px -> augmented px), the left/right swap for a flip, the in-plane rotation+ of joints_3d / cam_trans, and the CLIFF box centre with both. Shared by the+ standard and the v3 path so the two can never disagree."""+ px = (joints_2d[:, 0] + 1.0) * 0.5 * W+ py = (joints_2d[:, 1] + 1.0) * 0.5 * H+ px2 = M_total[0, 0] * px + M_total[0, 1] * py + M_total[0, 2]+ py2 = M_total[1, 0] * px + M_total[1, 1] * py + M_total[1, 2]+ joints_2d[:, 0] = (px2 / (0.5 * W) - 1.0).astype(np.float32)+ joints_2d[:, 1] = (py2 / (0.5 * H) - 1.0).astype(np.float32)++ if aug_flip > 0.5:+ joints_3d[:, 0] = -joints_3d[:, 0]+ joints_3d = joints_3d[FLIP_PERM]+ # joints_2d was already mirrored geometrically by M_total, but a+ # mirrored image LOOKS like the opposite-handed person, so the+ # left/right labels must be swapped too - exactly as for joints_3d.+ # Without this the 2D head gets contradictory left/right supervision+ # and collapses symmetric joints toward the body midline.+ joints_2d = joints_2d[FLIP_PERM]+ joints_2d_vis = joints_2d_vis[FLIP_PERM]+ cam_trans[0] = -cam_trans[0]+ cx = orig_w - cx+ rad = math.radians(angle)+ cos_t, sin_t = math.cos(rad), math.sin(rad)+ R_v = np.array([[cos_t, sin_t], [-sin_t, cos_t]], dtype=np.float32)+ joints_3d[:, :2] = joints_3d[:, :2] @ R_v.T+ cam_trans[:2] = R_v @ cam_trans[:2]++ dxy = np.array([cx - orig_w / 2.0, cy - orig_h / 2.0], dtype=np.float64)+ dxy = R_v.astype(np.float64) @ dxy+ cx, cy = orig_w / 2.0 + dxy[0], orig_h / 2.0 + dxy[1]+ return joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy, cos_t, sin_t++class SAM3DStudentDataset(Dataset):def __init__(self, data_root: str,image_size: int = 224, max_images: int | None = None,@@ -569,9 +614,13 @@ class SAM3DStudentDataset(Dataset):crop_centre_fix: bool = False,cliff_focal: bool = False,occl_p: float = 0.2, occl_scale: tuple = (0.02, 0.15),- jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90)):+ jpeg_p: float = 0.0, jpeg_quality: tuple = (30, 90),+ context_crops: bool = False, v3_box_min: float = 0.6,+ v3_box_max: float = 2.0, v3_shift: float = 0.15):super().__init__()self.image_size = image_size+ self.context_crops = context_crops+ self.v3_box_min, self.v3_box_max, self.v3_shift = v3_box_min, v3_box_max, v3_shiftself.data_root = Path(data_root)self.augment = augmentself.geom_p = geom_p@@ -623,6 +672,7 @@ class SAM3DStudentDataset(Dataset):self.pairs = list(pairs) if isinstance(pairs, (list, tuple)) else pairsprint(f" Prebuilt index: {len(self.pairs)} (image, npz) pairs "f"(augment={augment}).")+ self._check_context_split()returnself.pairs = []@@ -666,6 +716,32 @@ class SAM3DStudentDataset(Dataset):print(f" Found {len(dataset_dirs)} sub-folder(s) under {self.data_root}; "f"{len(self.pairs)} valid (image, npz) pairs (augment={augment}).")+ self._check_context_split()++ def _check_context_split(self):+ """v3 samples from context crops. A standard 224 split would still+ 'work' -- with the 1.2x crop read as if it were the context, i.e. every+ box silently 1.67x too large -- so refuse it up front."""+ if not self.context_crops or len(self.pairs) == 0:+ return+ # One probe per sub-folder, so a --data_root mixing context and+ # standard splits fails here and not hours in, inside a worker.+ firsts = getattr(self.pairs, "first_per_subfolder", None)+ if firsts is not None:+ probes = firsts()+ else:+ seen = {}+ for k, (_, npz) in enumerate(self.pairs):+ seen.setdefault(Path(npz).parent, k)+ probes = list(seen.values())+ for k in probes:+ npz = Path(self.pairs[k][1])+ with np.load(npz) as z:+ if "bbox_context" not in z.files:+ raise ValueError(+ f"{npz.parent.parent.name}: {npz.name} has no bbox_context. "+ f"--preset v3 needs a context-crop split, built with "+ f"build_split.py --context 2.0 --body-size 448 --output-dir ...")@staticmethoddef _find_dataset_dirs(root: Path):@@ -682,6 +758,88 @@ class SAM3DStudentDataset(Dataset):dirs.append((ann, img))return dirs+ def _context_window(self, img_pil, ann, W, H, b_size, cx, cy):+ """v3: cut the network input out of a context crop in ONE resampling.++ The window is drawn in full-frame px: side = m x the tight box, m+ log-uniform in [v3_box_min, v3_box_max]; centre = box centre + a shift of+ up to +/- v3_shift x side; plus v2's flip and rotation. It is clamped so+ the ROTATED window stays inside the stored context square, shrinking+ the side first if it cannot fit -- so no synthetic black margin ever+ appears, only the real frame edge the builder padded.++ The window is expressed as an ordinary aug_M relative to the standard+ 1.2x crop (bbox_square at W px), so labels, the reprojection loss and+ everything downstream are unchanged. The image is warped from the+ context crop with aug_M composed with the context -> standard-crop map.+ Un-augmented samples use the identity window, i.e. today's crop.++ Returns (image, M_total, aug_active, flip, angle, window centre (frame+ px, before flip/rotation), window side / 1.2 in frame px).+ """+ sq = ann["bbox_square"].astype(np.float64)+ ctx = ann["bbox_context"].astype(np.float64)+ kx, ky = W / max(sq[2] - sq[0], 1.0), H / max(sq[3] - sq[1], 1.0)+ M_total = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float64)+ aug_active, aug_flip, angle, centre, bpx = 0.0, 0.0, 0.0, None, None++ if self.augment and random.random() < self.geom_p:+ aug_active = 1.0+ aug_flip = 1.0 if random.random() < self.geom_flip_p else 0.0+ angle = random.uniform(-self.geom_rot_deg, self.geom_rot_deg)+ m = math.exp(random.uniform(math.log(self.v3_box_min), math.log(self.v3_box_max)))+ side = m * b_size+ wx = cx + random.uniform(-self.v3_shift, self.v3_shift) * side+ wy = cy + random.uniform(-self.v3_shift, self.v3_shift) * side+ # Keep the rotated window inside the stored square -- but only on+ # sides where that square ends INSIDE the photo. Where it already+ # runs past the photo's edge, anything further out is outside the+ # photo too, so the black it reads IS the real frame edge (what a+ # deployment crop shows when its box leaves the frame); clamping+ # there would only thin out the loose, off-centre framings v3 is for.+ rad = math.radians(angle)+ reach = abs(math.cos(rad)) + abs(math.sin(rad)) # rotated square's extent / side+ oh, ow = (float(v) for v in ann["orig_shape"][:2])+ lo_x = ctx[0] if ctx[0] >= 0 else -math.inf+ hi_x = ctx[2] if ctx[2] <= ow else math.inf+ lo_y = ctx[1] if ctx[1] >= 0 else -math.inf+ hi_y = ctx[3] if ctx[3] <= oh else math.inf+ side = min(side, min(hi_x - lo_x, hi_y - lo_y) / reach)+ e = side * reach / 2.0+ wx = min(max(wx, lo_x + e), hi_x - e)+ wy = min(max(wy, lo_y + e), hi_y - e)+ centre, bpx = (wx, wy), side / 1.2++ # standard-crop px: flip, then zoom and rotate about the window+ # centre and move it to the output centre -- v2's M_total form. The+ # zoom is per axis: bbox_square's int() edges can make it 198x200+ # px, and only per-axis zoom keeps the window SQUARE in the frame,+ # which the clamp above and the 3D label rotation both assume.+ c = np.array([(wx - sq[0]) * kx, (wy - sq[1]) * ky])+ M_flip = (np.array([[-1.0, 0.0, float(W)], [0.0, 1.0, 0.0]]) if aug_flip+ else np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]]))+ fc = M_flip[:, :2] @ c + M_flip[:, 2]+ rot = cv2.getRotationMatrix2D((0.0, 0.0), angle, 1.0)[:, :2]+ lin = rot @ np.diag([(sq[2] - sq[0]) / side, (sq[3] - sq[1]) / side])+ M_rst = np.hstack([lin, (np.array([W / 2.0, H / 2.0]) - lin @ fc)[:, None]])+ A2, t2 = M_rst[:, :2], M_rst[:, 2]+ A1, t1 = M_flip[:, :2], M_flip[:, 2]+ M_total = np.hstack([A2 @ A1, (A2 @ t1 + t2)[:, None]])++ # context-crop pixel index -> standard-crop pixel index (pixel centres,+ # as cv2.resize maps them on both sides; verified against today's crops+ # in tools/verify_context_crops.py)+ ctx_np = np.asarray(img_pil)+ sc_y, sc_x = ctx_np.shape[:2]+ ax, ay = (ctx[2] - ctx[0]) / sc_x, (ctx[3] - ctx[1]) / sc_y+ C = np.array([[ax * kx, 0.0, (0.5 * ax + ctx[0] - sq[0]) * kx - 0.5],+ [0.0, ay * ky, (0.5 * ay + ctx[1] - sq[1]) * ky - 0.5]])+ A, t = M_total[:, :2], M_total[:, 2]+ M_img = np.hstack([A @ C[:, :2], (A @ C[:, 2] + t)[:, None]])+ img_np = cv2.warpAffine(ctx_np, M_img, (W, H), flags=cv2.INTER_LINEAR,+ borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0))+ return Image.fromarray(img_np), M_total, aug_active, aug_flip, angle, centre, bpx+def __len__(self):return len(self.pairs)@@ -732,7 +890,21 @@ class SAM3DStudentDataset(Dataset):aug_flip, cos_t, sin_t = 0.0, 1.0, 0.0M_total = np.array([[1.0, 0.0, 0.0], [0.0, 1.0, 0.0]], dtype=np.float64)aug_active = 0.0- if self.augment and random.random() < self.geom_p:+ if self.context_crops:+ (img_pil, M_total, aug_active, aug_flip, angle,+ v3_centre, v3_bpx) = self._context_window(img_pil, ann, W, H, b_size, cx, cy)+ if aug_active:+ # CLIFF describes the window actually cut: its centre, and the+ # tight box that would give it at the standard 1.2x expansion.+ # Both are exact (no crop-space approximation), then flipped and+ # rotated with the labels below.+ cx, cy = v3_centre+ b_scale = v3_bpx / max(orig_w, orig_h)+ (joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy,+ cos_t, sin_t) = _apply_label_geometry(+ M_total, aug_flip, angle, joints_2d, joints_2d_vis, joints_3d,+ cam_trans, cx, cy, orig_w, orig_h, W, H)+ elif self.augment and random.random() < self.geom_p:aug_active = 1.0img_np = np.asarray(img_pil.resize((W, H)))@@ -767,34 +939,10 @@ class SAM3DStudentDataset(Dataset):borderMode=cv2.BORDER_CONSTANT, borderValue=(0, 0, 0))img_pil = Image.fromarray(img_np)- px = (joints_2d[:, 0] + 1.0) * 0.5 * W- py = (joints_2d[:, 1] + 1.0) * 0.5 * H- px2 = M_total[0, 0] * px + M_total[0, 1] * py + M_total[0, 2]- py2 = M_total[1, 0] * px + M_total[1, 1] * py + M_total[1, 2]- joints_2d[:, 0] = (px2 / (0.5 * W) - 1.0).astype(np.float32)- joints_2d[:, 1] = (py2 / (0.5 * H) - 1.0).astype(np.float32)-- if aug_flip > 0.5:- joints_3d[:, 0] = -joints_3d[:, 0]- joints_3d = joints_3d[FLIP_PERM]- # joints_2d was already mirrored geometrically by M_total, but a- # mirrored image LOOKS like the opposite-handed person, so the- # left/right labels must be swapped too - exactly as for joints_3d.- # Without this the 2D head gets contradictory left/right supervision- # and collapses symmetric joints toward the body midline.- joints_2d = joints_2d[FLIP_PERM]- joints_2d_vis = joints_2d_vis[FLIP_PERM]- cam_trans[0] = -cam_trans[0]- cx = orig_w - cx- rad = math.radians(angle)- cos_t, sin_t = math.cos(rad), math.sin(rad)- R_v = np.array([[cos_t, sin_t], [-sin_t, cos_t]], dtype=np.float32)- joints_3d[:, :2] = joints_3d[:, :2] @ R_v.T- cam_trans[:2] = R_v @ cam_trans[:2]-- dxy = np.array([cx - orig_w / 2.0, cy - orig_h / 2.0], dtype=np.float64)- dxy = R_v.astype(np.float64) @ dxy- cx, cy = orig_w / 2.0 + dxy[0], orig_h / 2.0 + dxy[1]+ (joints_2d, joints_2d_vis, joints_3d, cam_trans, cx, cy,+ cos_t, sin_t) = _apply_label_geometry(+ M_total, aug_flip, angle, joints_2d, joints_2d_vis, joints_3d,+ cam_trans, cx, cy, orig_w, orig_h, W, H)# CLIFF conditioning describes the box this crop came from. The flip# and rotation above already move it; dx/dy/scale did not, which was@@ -885,7 +1033,9 @@ def build_dataloaders(cfg):cliff_focal=cfg.cliff_focal,occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality,- image_size=cfg.image_size)+ image_size=cfg.image_size,+ context_crops=cfg.context_crops, v3_box_min=cfg.v3_box_min,+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)full_dataset = SAM3DStudentDataset(cfg.data_root,augment=cfg.augment,diff --git a/tools/ddp_smoke.py b/tools/ddp_smoke.pyindex 5e4c52d..acd8b2f 100644--- a/tools/ddp_smoke.py+++ b/tools/ddp_smoke.py@@ -52,6 +52,9 @@ def main() -> int:p = argparse.ArgumentParser(description=__doc__,formatter_class=argparse.RawDescriptionHelpFormatter)p.add_argument("--backbone", default=None, help="timm name; default is the config's")+ p.add_argument("--context-crops", dest="context_crops", action="store_true",+ help="--preset v3's dataset; --data_root must be a context-crop "+ "split (build_split.py --context 2.0 --body-size 448)")p.add_argument("--image-size", dest="image_size", type=int, default=None,help="network input side; the positional grid and every crop follow it")p.add_argument("--data_root", default="data")@@ -109,7 +112,8 @@ def main() -> int:ds = T.SAM3DStudentDataset(cfg.data_root, augment=True, max_images=args.batch_size,per_dataset_caps={}, cliff_focal=cfg.cliff_focal,crop_centre_fix=cfg.crop_centre_fix,- image_size=cfg.image_size)+ image_size=cfg.image_size,+ context_crops=args.context_crops)loader = torch.utils.data.DataLoader(ds, batch_size=args.batch_size, num_workers=0)batch = {k: (v.to(T.device) if torch.is_tensor(v) else v)for k, v in next(iter(loader)).items()}diff --git a/tools/profiling/export_family.py b/tools/profiling/export_family.pyindex 04f0bf7..ebd9edb 100644--- a/tools/profiling/export_family.py+++ b/tools/profiling/export_family.py@@ -1,7 +1,15 @@-"""Usage: python tools/profiling/export_family.py <out_dir> then python tools/profiling/profile_family.py <out_dir>-Ticket 03: ONNX exports of the g8h architecture with the backbone swapped.-Random weights for B0/B1/B4, plus the trained g8h, all through one path."""-import io, sys, copy, json+"""ONNX exports of the g8h architecture with the backbone and input size swapped.++ python tools/profiling/export_family.py --out /tmp/fam # ticket 03+ python tools/profiling/export_family.py --out /tmp/fam288 --image-size 288 \+ --backbones b4 --no-trained # ticket 06++Random weights (latency does not depend on them; ticket 03 measured random B4+within 2-6% of the trained g8h on every phone), plus optionally the trained+g8h. Network only, no FK / mesh: that is what differs between members.+Then profile with tools/profiling/profile_family.py.+"""+import argparse, copy, io, json, sysfrom pathlib import Pathimport torch, onnx, onnxsimREPO = Path(__file__).resolve().parents[2]@@ -9,12 +17,7 @@ sys.path.insert(0, str(REPO / "instanthmr_distill_train"))import train_distill_mhr_only as TCKPT = REPO / "instanthmr_distill_train/runs/g8h_s0/g8h_s0/best_student_model_ema.pth"-OUT = Path(sys.argv[1])-blob = torch.load(CKPT, map_location="cpu", weights_only=False)-state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}-cfg, _ = T.config_from_checkpoint(state, CKPT)-cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2-cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])+class Deploy(torch.nn.Module):def __init__(self, m): super().__init__(); self.m = m@@ -22,26 +25,53 @@ class Deploy(torch.nn.Module):o = self.m(image, cliff_cond)return o["mhr_params"], o["shape_params"], o["cam_trans"], o["joints_2d"]-def export(model, name):++def export(model, path: Path, size: int):model.eval()buf = io.BytesIO()- torch.onnx.export(Deploy(model), (torch.zeros(1, 3, 224, 224), torch.zeros(1, 3)), buf,+ torch.onnx.export(Deploy(model), (torch.zeros(1, 3, size, size), torch.zeros(1, 3)), buf,input_names=["image", "cliff_cond"],output_names=["mhr_params", "shape_params", "cam_trans", "joints_2d"],opset_version=17)m, ok = onnxsim.simplify(onnx.load_from_string(buf.getvalue()))assert ok- p = OUT / f"{name}.onnx"; onnx.save(m, p)+ onnx.save(m, path)n = sum(x.numel() for x in model.parameters())nb = sum(x.numel() for x in model.backbone.parameters())- print(json.dumps(dict(name=name, params_M=round(n/1e6, 2), backbone_M=round(nb/1e6, 2),- mb=round(p.stat().st_size/2**20, 1))), flush=True)--trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)-missing, unexpected = trained.load_state_dict(state, strict=False)-assert not missing and not unexpected, (missing[:5], unexpected[:5])-export(trained, "g8h_trained_b4")-for bb in ("hgnetv2_b4", "hgnetv2_b1", "hgnetv2_b0"):- c = copy.deepcopy(cfg); c.backbone = bb- torch.manual_seed(0)- export(T.InstantHMRStudent(c, pretrained=False), f"random_{bb.split('_')[1]}")+ print(json.dumps(dict(name=path.stem, image_size=size, params_M=round(n / 1e6, 2),+ backbone_M=round(nb / 1e6, 2))), flush=True)+++def main():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--out", type=Path, required=True)+ p.add_argument("--image-size", type=int, default=224)+ p.add_argument("--backbones", default="b4,b1,b0", help="hgnetv2 sizes, comma-separated")+ p.add_argument("--no-trained", action="store_true", help="skip the trained g8h export")+ args = p.parse_args()+ args.out.mkdir(parents=True, exist_ok=True)++ blob = torch.load(CKPT, map_location="cpu", weights_only=False)+ state = {k.replace("_orig_mod.", ""): v for k, v in blob["model_state_dict"].items()}+ cfg, _ = T.config_from_checkpoint(state, CKPT)+ cfg.kp2d_bins = state["head_2d_logits.bias"].shape[0] // 2+ cfg.kp2d_range = float(state["kp2d_bin_centers"][-1])++ if not args.no_trained:+ if args.image_size != cfg.image_size:+ sys.exit("the trained g8h is a 224 model; pass --no-trained with --image-size")+ trained = T.InstantHMRStudent(copy.deepcopy(cfg), pretrained=False)+ missing, unexpected = trained.load_state_dict(state, strict=False)+ assert not missing and not unexpected, (missing[:5], unexpected[:5])+ export(trained, args.out / "g8h_trained_b4.onnx", args.image_size)+ for b in args.backbones.split(","):+ c = copy.deepcopy(cfg)+ c.backbone, c.image_size = f"hgnetv2_{b}", args.image_size+ torch.manual_seed(0)+ export(T.InstantHMRStudent(c, pretrained=False),+ args.out / f"random_{b}_{args.image_size}.onnx", args.image_size)+++if __name__ == "__main__":+ main()diff --git a/tools/profiling/profile_family.py b/tools/profiling/profile_family.pyindex 85bca8c..c6b166a 100644--- a/tools/profiling/profile_family.py+++ b/tools/profiling/profile_family.py@@ -1,35 +1,64 @@-import json, sys+"""Compile every .onnx in a directory for the S23 (ORT runtime) and profile it on+AI Hub: NPU on the Snapdragon 8-series phones, CPU-only on the reference phones.++ python tools/profiling/profile_family.py --dir /tmp/fam288 --image-size 288++Writes <dir>/profile.json (one row per model x device x unit) and prints rows.+"""+import argparse, jsonfrom pathlib import Pathimport qai_hub as hub-D = Path(sys.argv[1])+CPU = ["Samsung Galaxy S23", "Samsung Galaxy A73 5G", "Samsung Galaxy A53 5G","Samsung Galaxy A14 5G", "Google Pixel 8"]-specs = dict(image=(1, 3, 224, 224), cliff_cond=(1, 3))-names = ["g8h_trained_b4", "random_b4", "random_b1", "random_b0"]-comp = {n: hub.submit_compile_job(model=str(D / f"{n}.onnx"), device=hub.Device("Samsung Galaxy S23"),- input_specs=specs, options="--target_runtime onnx", name=f"t03-{n}")- for n in names}-jobs = []-for n, cj in comp.items():- m = cj.get_target_model()- if m is None:- print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True); continue- jobs.append((n, "npu", "Samsung Galaxy S23",- hub.submit_profile_job(model=m, device=hub.Device("Samsung Galaxy S23"), name=f"t03-{n}-npu")))- for d in CPU:- jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),- name=f"t03-{n}-cpu", options="--compute_unit cpu")))-out = []-for n, u, d, j in jobs:- row = dict(model=n, unit=u, device=d, job=j.job_id)- if j.wait().success:- p = j.download_profile(); s = p["execution_summary"]- units = {}- for l in p.get("execution_detail", []):- units[l.get("compute_unit")] = units.get(l.get("compute_unit"), 0) + 1- row.update(ms=round(s["estimated_inference_time"] / 1000, 2),- peak_mb=round(s.get("estimated_inference_peak_memory", 0) / 2**20, 1), layers=units)- else:- row["error"] = j.get_status().message- print(json.dumps(row), flush=True); out.append(row)-json.dump(out, open(D / "profile.json", "w"), indent=1)+NPU = ["Samsung Galaxy S22 5G", "Samsung Galaxy S23", "Samsung Galaxy S24", "Samsung Galaxy S25"]+++def main():+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--dir", type=Path, required=True)+ p.add_argument("--image-size", type=int, default=224)+ p.add_argument("--npu", default="S23", help="'S23' or 'all' Snapdragon 8-series phones")+ args = p.parse_args()+ s = args.image_size+ specs = dict(image=(1, 3, s, s), cliff_cond=(1, 3))+ npu = NPU if args.npu == "all" else ["Samsung Galaxy S23"]++ comp = {f.stem: hub.submit_compile_job(model=str(f), device=hub.Device("Samsung Galaxy S23"),+ input_specs=specs, options="--target_runtime onnx",+ name=f"fam-{f.stem}")+ for f in sorted(args.dir.glob("*.onnx"))}+ jobs = []+ for n, cj in comp.items():+ m = cj.get_target_model()+ if m is None:+ print(json.dumps(dict(model=n, compile_failed=cj.get_status().message)), flush=True)+ continue+ for d in npu:+ jobs.append((n, "npu", d, hub.submit_profile_job(model=m, device=hub.Device(d),+ name=f"fam-{n}-npu")))+ for d in CPU:+ jobs.append((n, "cpu", d, hub.submit_profile_job(model=m, device=hub.Device(d),+ name=f"fam-{n}-cpu", options="--compute_unit cpu")))+ out = []+ for n, u, d, j in jobs:+ row = dict(model=n, image_size=s, unit=u, device=d, job=j.job_id)+ if j.wait().success:+ prof = j.download_profile()+ summ = prof["execution_summary"]+ units = {}+ for layer in prof.get("execution_detail", []):+ units[layer.get("compute_unit")] = units.get(layer.get("compute_unit"), 0) + 1+ row.update(ms=round(summ["estimated_inference_time"] / 1000, 2),+ peak_mb=round(summ.get("estimated_inference_peak_memory", 0) / 2**20, 1),+ layers=units)+ else:+ row["error"] = j.get_status().message+ print(json.dumps(row), flush=True)+ out.append(row)+ json.dump(out, open(args.dir / "profile.json", "w"), indent=1)+++if __name__ == "__main__":+ main()diff --git a/tools/verify_v3_preset.py b/tools/verify_v3_preset.pynew file mode 100644index 0000000..81b71b3--- /dev/null+++ b/tools/verify_v3_preset.py@@ -0,0 +1,290 @@+#!/usr/bin/env python3+"""Verify the v3 preset on real context crops.++Needs the same rows built twice with datasets_pipeline/build_split.py:+standard (default flags) and context (--context 2.0 --body-size 448).++ python tools/verify_v3_preset.py --std <root with sam3d_gt_aic> --ctx <root> \+ --ref <train_distill_mhr_only.py before the change>++Checks, all from the dataset's outputs plus the npz geometry:++1. baseline and v2 samples are bit-identical to --ref (alternating calls).+2. Un-augmented v3 (the identity window) reproduces today's crop: labels and+ CLIFF equal the standard split's, image within resampling noise.+3. Reprojection chain: joints_3d + cam_trans projected through bbox_square and+ aug_M land on the joints_2d labels, exactly as the loss computes them.+4. CLIFF describes the window actually cut: the preimage of the crop centre+ under aug_M, flipped and rotated like the labels, is the CLIFF centre, and+ the CLIFF size is the window side / 1.2.+5. No pixel is read from beyond a stored-square edge that lies INSIDE the+ photo (that would be a synthetic black margin). Past an edge that is+ already outside the photo, black is the real frame edge and allowed.+ Also reports the loose tail the clamp leaves: P(>1.6x), P(>1.8x) and the+ mean shift of loose windows.+6. Box-scale draws cover the configured range (log-uniform 0.6-2.0x).+7. Output size follows image_size (224 and 288).+8. A standard split is refused by the v3 dataset, naming the split.+"""++from __future__ import annotations++import argparse+import importlib.util+import math+import random+import sys+from pathlib import Path++import numpy as np+import torch+from PIL import Image++REPO = Path(__file__).resolve().parent.parent+TRAIN = REPO / "instanthmr_distill_train"+sys.path.insert(0, str(TRAIN))+import train_distill_mhr_only as T # noqa: E402+import train_distill_jz as JZ # noqa: E402++MEAN = np.array([0.485, 0.456, 0.406])+STD = np.array([0.229, 0.224, 0.225])+fails: list[str] = []+++def check(ok: bool, msg: str) -> None:+ print(f" {'OK ' if ok else 'BAD'} {msg}")+ if not ok:+ fails.append(msg)+++def load_module(path: Path, name: str):+ spec = importlib.util.spec_from_file_location(name, path)+ mod = importlib.util.module_from_spec(spec)+ spec.loader.exec_module(mod)+ return mod+++def cfg_for(preset: str, image_size: int = 224):+ cfg = T.DistillConfig()+ cfg.cliff_focal = True+ cfg.crop_centre_fix = True+ cfg.image_size = image_size+ if preset in ("v2", "v3"):+ JZ.apply_v2_preset(cfg)+ if preset == "v3":+ JZ.apply_v3_preset(cfg)+ return cfg+++def dataset(mod, root, cfg, augment=True, v3=False):+ kw = dict(geom_p=cfg.geom_p, geom_rot_deg=cfg.geom_rot_deg,+ geom_scale_range=cfg.geom_scale_range, geom_trans=cfg.geom_trans,+ geom_flip_p=cfg.geom_flip_p, geom_scale_max=cfg.geom_scale_max,+ cliff_follows_aug=cfg.cliff_follows_aug, crop_centre_fix=cfg.crop_centre_fix,+ cliff_focal=cfg.cliff_focal, occl_p=cfg.occl_p, occl_scale=cfg.occl_scale,+ jpeg_p=cfg.jpeg_p, jpeg_quality=cfg.jpeg_quality, image_size=cfg.image_size)+ if v3:+ kw.update(context_crops=True, v3_box_min=cfg.v3_box_min,+ v3_box_max=cfg.v3_box_max, v3_shift=cfg.v3_shift)+ return mod.SAM3DStudentDataset(str(root), augment=augment, **kw)+++def seeded(ds, i, seed):+ random.seed(seed)+ np.random.seed(seed)+ torch.manual_seed(seed)+ return ds[i]+++def same(a, b) -> bool:+ if isinstance(a, torch.Tensor):+ return torch.equal(a, b)+ if isinstance(a, np.ndarray):+ return np.array_equal(a, b, equal_nan=a.dtype.kind == "f")+ return a == b+++def to_uint8(img: torch.Tensor) -> np.ndarray:+ x = img.numpy().transpose(1, 2, 0) * STD + MEAN+ return np.clip(x * 255.0, 0, 255)+++def reproject(s: dict, size: int) -> np.ndarray:+ """The loss's own chain: undo the in-plane rotation and the flip on the 3D+ labels (the dataset applied flip then rotation), project in the original+ camera, map into the 1.2x crop, then let aug_M re-apply both."""+ c, sn = float(s["aug_cos"]), float(s["aug_sin"])+ R = np.array([[c, sn], [-sn, c]])+ j3 = s["joints_3d"].numpy().astype(np.float64).copy()+ cam = s["cam_trans"].numpy().astype(np.float64).copy()+ j3[:, :2] = j3[:, :2] @ R # inverse of @ R.T+ cam[:2] = R.T @ cam[:2] # inverse of R @+ if float(s["aug_flip"]) > 0.5:+ j3[:, 0], cam[0] = -j3[:, 0], -cam[0]+ j = j3 + cam+ fx, fy = s["cam_focal"].numpy()+ h, w = s["orig_shape"].numpy()+ u = fx * j[:, 0] / j[:, 2] + w / 2.0+ v = fy * j[:, 1] / j[:, 2] + h / 2.0+ sq = np.asarray(s["bbox_square"], dtype=np.float64)+ k = size / max(sq[2] - sq[0], 1.0)+ uc, vc = (u - sq[0]) * k, (v - sq[1]) * k+ M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)+ ua = M[0, 0] * uc + M[0, 1] * vc + M[0, 2]+ va = M[1, 0] * uc + M[1, 1] * vc + M[1, 2]+ return np.stack([ua / size * 2 - 1, va / size * 2 - 1], axis=1)+++def window_from_sample(s: dict, ann: dict, size: int, sc: int): # noqa: C901+ """Recover the window from aug_M + npz geometry, independently of the+ dataset's own bookkeeping. Returns (centre in frame px, side in frame px,+ context-crop px of the 4 output corners)."""+ M = s["aug_M"].numpy().astype(np.float64).reshape(2, 3)+ A, t = M[:, :2], M[:, 2]+ Ainv = np.linalg.inv(A)+ sq = ann["bbox_square"].astype(np.float64)+ ks = (sq[2:] - sq[:2]) / size # std px -> frame px+ to_frame = lambda o: sq[:2] + (Ainv @ (np.asarray(o, float) - t)) * ks # noqa: E731+ centre = to_frame([size / 2.0, size / 2.0])+ side = size / abs(np.linalg.det(A)) ** 0.5 * (ks[0] * ks[1]) ** 0.5+ ctx = ann["bbox_context"].astype(np.float64)+ corners = [to_frame(c) for c in ([0, 0], [size, 0], [0, size], [size, size])]+ return centre, side, corners+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--std", type=Path, required=True)+ p.add_argument("--ctx", type=Path, required=True)+ p.add_argument("--ref", type=Path, required=True)+ p.add_argument("--n", type=int, default=400)+ args = p.parse_args()+ R = load_module(args.ref, "train_distill_mhr_only_ref")++ print("[1] baseline and v2 vs reference")+ for preset in ("baseline", "v2"):+ cfg = cfg_for(preset)+ new, old = dataset(T, args.std, cfg), dataset(R, args.std, cfg)+ diff = 0+ for k in range(24):+ a, b = seeded(new, k, 7 + k), seeded(old, k, 7 + k)+ diff += not all(same(a[x], b[x]) for x in a)+ check(diff == 0, f"{preset}: {diff} of 24 samples differ")++ cfg = cfg_for("v3")+ std0 = dataset(T, args.std, cfg_for("baseline"), augment=False)+ ctx0 = dataset(T, args.ctx, cfg, augment=False, v3=True)+ print("[2] identity window reproduces today's crop")+ lab, cl, ps = 0.0, 0.0, []+ for i in range(min(64, len(std0))):+ a, b = std0[i], ctx0[i]+ lab = max(lab, float((a["joints_2d"] - b["joints_2d"]).abs().max()))+ cl = max(cl, float((a["cliff_cond"] - b["cliff_cond"]).abs().max()))+ ia, ib = to_uint8(a["image"]), to_uint8(b["image"])+ mse = np.mean((ia[8:-8, 8:-8] - ib[8:-8, 8:-8]) ** 2)+ ps.append(10 * math.log10(255.0 ** 2 / max(mse, 1e-9)))+ check(lab < 1e-5, f"joints_2d max diff {lab:.2e}")+ check(cl < 1e-6, f"cliff_cond max diff {cl:.2e}")+ check(min(ps) > 25.0, f"image PSNR mean {np.mean(ps):.1f} dB, min {min(ps):.1f} dB "+ f"(448 context crop vs 224 crop, resampling noise)")++ for size in (224, 288):+ cfg = cfg_for("v3", size)+ ds = dataset(T, args.ctx, cfg, augment=True, v3=True)+ print(f"[3-7] augmented v3 at {size} px, {args.n} samples")+ rep, cc, cs, outside, shapes, ms, shifts = 0.0, 0.0, 0.0, 0, set(), [], []+ for k in range(args.n):+ i = k % len(ds)+ s = seeded(ds, i, 100 + k)+ ann = dict(np.load(ds.pairs[i][1]))+ shapes.add(tuple(s["image"].shape))+ # 3: reprojection chain vs labels (visible keypoints, output px)+ vis = s["joints_2d_vis"].numpy() > 0+ err = np.abs(reproject(s, size) - s["joints_2d"].numpy())[vis] * size / 2+ rep = max(rep, float(err.max()))+ # 4: CLIFF vs the window recovered from aug_M. Augmented samples+ # only: an un-augmented one keeps today's CLIFF (the tight-box+ # centre), which is 0.5 px off the int()-rounded 1.2x square.+ sc = Image.open(ds.pairs[i][0]).size[0]+ centre, side, corners = window_from_sample(s, ann, size, sc)+ h, w = ann["orig_shape"][:2]+ flip, cos_t, sin_t = float(s["aug_flip"]), float(s["aug_cos"]), float(s["aug_sin"])+ cx, cy = centre+ if flip > 0.5:+ cx = w - cx+ d = np.array([[cos_t, sin_t], [-sin_t, cos_t]]) @ np.array([cx - w / 2, cy - h / 2])+ f = float(ann["cam_focal_length"][1])+ want = np.array([math.atan(d[0] / f), math.atan(d[1] / f), side / 1.2 / f])+ got = s["cliff_cond"].numpy().astype(np.float64)+ if float(s["aug_active"]) > 0.5:+ cc = max(cc, float(np.abs(np.tan(got[:2]) - np.tan(want[:2])).max() * f))+ cs = max(cs, abs(got[2] - want[2]) / want[2])+ # 5: no corner past a stored-square edge that is inside the photo+ ctx = ann["bbox_context"].astype(np.float64)+ tol = 0.5 * (ctx[2] - ctx[0]) / sc # half a stored pixel+ for c in corners:+ outside += int((ctx[0] >= 0 and c[0] < ctx[0] - tol) or+ (ctx[2] <= w and c[0] > ctx[2] + tol) or+ (ctx[1] >= 0 and c[1] < ctx[1] - tol) or+ (ctx[3] <= h and c[1] > ctx[3] + tol)) > 0+ tight = ann["bbox"].astype(np.float64)+ ts = max(tight[2] - tight[0], tight[3] - tight[1])+ ms.append(side / ts)+ tc = (tight[:2] + tight[2:]) / 2+ shifts.append(float(np.abs(np.asarray(centre) - tc).max() / side))+ ms, shifts = np.array(ms), np.array(shifts)+ on = np.abs(ms - 1.2) > 1e-3+ active = ms[on]+ loose = on & (ms > 1.6)+ p16, p18 = np.mean(active > 1.6), np.mean(active > 1.8)+ ideal16 = math.log(2.0 / 1.6) / math.log(2.0 / 0.6)+ ideal18 = math.log(2.0 / 1.8) / math.log(2.0 / 0.6)+ check(p16 > 0.8 * ideal16,+ f"loose tail: P(>1.6x) {p16:.3f} (unclamped {ideal16:.3f}), P(>1.8x) {p18:.3f} "+ f"(unclamped {ideal18:.3f}); mean shift of >1.6x windows "+ f"{shifts[loose].mean():.3f} of the side (unclamped 0.100)")+ check(rep < 1e-3, f"reprojection chain vs labels: max {rep:.2e} px")+ check(cc < 0.05 and cs < 1e-4, f"CLIFF vs recovered window: centre {cc:.3f} px, "+ f"size rel {cs:.1e}")+ check(outside == 0, f"{outside} samples read past a stored-square edge inside the photo")+ lo, hi = active.min(), active.max()+ frac = np.mean(active < 1.1)+ expect = math.log(1.1 / 0.6) / math.log(2.0 / 0.6)+ check(lo < 0.65 and hi > 1.8 and abs(frac - expect) < 0.08,+ f"box multiple drawn {lo:.2f}-{hi:.2f}x; P(<1.1x) {frac:.2f} vs "+ f"log-uniform {expect:.2f} ({len(active)} active draws)")+ check(shapes == {(3, size, size)}, f"output shapes {shapes}")++ print("[8] a standard split is refused")+ try:+ dataset(T, args.std, cfg_for("v3"), v3=True)+ check(False, "standard split accepted by the v3 dataset")+ except ValueError as e:+ check("sam3d_gt_aic" in str(e), f"refused: {e}")++ # a root mixing a context split (sorted first) with a standard one+ import tempfile+ mixed = Path(tempfile.mkdtemp(prefix="v3_mixed_"))+ (mixed / "sam3d_gt_aic").symlink_to((args.ctx / "sam3d_gt_aic").resolve())+ (mixed / "sam3d_gt_zstd").symlink_to((args.std / "sam3d_gt_aic").resolve())+ try:+ dataset(T, mixed, cfg_for("v3"), v3=True)+ check(False, "mixed context + standard root accepted")+ except ValueError as e:+ check("sam3d_gt_zstd" in str(e), f"mixed root refused: {str(e)[:60]}...")+ idx = JZ.build_pair_index(str(mixed), mixed / "idx.npz")+ try:+ T.SAM3DStudentDataset(str(mixed), augment=True, pairs=idx, context_crops=True)+ check(False, "mixed root accepted through a PairIndex")+ except ValueError as e:+ check("sam3d_gt_zstd" in str(e), "mixed root refused through a PairIndex")++ if fails:+ print(f"FAIL ({len(fails)})")+ sys.exit(1)+ print("PASS")+++if __name__ == "__main__":+ main()