Python
feat: added datasets pipeline
mohamdev8a3b5d9c9da1
diff --git a/datasets_pipeline/README.md b/datasets_pipeline/README.mdnew file mode 100644index 0000000..dc5f556--- /dev/null+++ b/datasets_pipeline/README.md@@ -0,0 +1,282 @@+# datasets_pipeline++Rebuild any split of the SAM-3D-Body corpus from scratch, keeping the original+frames this time, into the layout the training code already understands.++```+data/sam3d_gt_<dataset>/+├── original_images/ referenced source frames, native relative paths+├── annotations/ <name>.npz+├── body_crops/ <name>.png 224 square person crop+├── hands_crops/ <name>_l.png <name>_r.png+└── images -> body_crops compat symlink+```++`body_crops` is **byte-identical** to what `tools/parquet_to_npz.py` writes into+`images/` — verified md5-for-md5 on 120 AIC crops against the existing+`data/sam3d_gt_aic/`. `build_split.py` imports that module's crop and annotation+builders rather than re-implementing them, so the two can never drift. The+`images` symlink means `SAM3DStudentDataset` finds these folders with **no code+change**; that path is tested, not assumed.++## The scripts++| script | does |+|---|---|+| `fetch_annotations.py` | parquet shards from the gated HF repo |+| `probe_split.py` | **is this split worth its download?** — from annotations alone |+| `fetch_images.py` | source frames, per dataset |+| `build_split.py` | parquet + frames → the folder above |+| `verify_split.py` | integrity, reprojection, and how much hand signal a split carries |++Plus `plan_disk.py` (cost a split before downloading it), `stream_archives.py`+(build a dataset larger than your disk), `registry.py` (every URL and gotcha in+one place), and `undistort.md`.++Annotations are ~1 GB per split; frames are tens to thousands of GB. So the+order is always: fetch annotations → `probe_split.py` → decide → fetch frames.++## Quick start++```bash+export ANN_DIR=/scratch/sam3d_ann+export IMG_ROOT=/scratch/sam3d_img++python datasets_pipeline/fetch_annotations.py --list+python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR+python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/coco_train++python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco+python datasets_pipeline/build_split.py \+ --split coco_train \+ --annotation-dir $ANN_DIR/coco_train \+ --image-dir $IMG_ROOT/coco \+ --data-root data --validate++python datasets_pipeline/verify_split.py --dir data/sam3d_gt_coco+```++Everything is resume-safe: re-running skips crops already on disk.++## What each dataset costs, and how to get it++| dataset | images | route | undistort |+|---|---|---|---|+| coco | 13.5 GB | `wget` train2014.zip — **only train2014 is referenced** | no |+| mpii | 12.1 GB | `wget` mpii_human_pose_v1.tar.gz | no |+| aic | 36.4 GB | OpenDataLab CLI (login) — where this corpus came from before | no |+| 3dpw | 4.6 GB | licence form → imageFiles.zip | no |+| harmony4d | 352 GB | HF `Jyun-Ting/Harmony4D`, ungated, per-scene zips | **yes** |+| egohumans | 107 GB referenced (550 GB advertised) | Google Drive, per-take tar.gz → `gdown --folder` | **yes** |+| egoexo4d | ~8 GB (procedure) | `egoexo -o $DEST --parts sam_3d_body` (signed licence) | no |+| sa1b | ~11 TB | terms → `links.txt` → `--links-file` | no |++`fetch_images.py` handles the first two and harmony4d directly. For the rest it+prints the exact commands and exits 2 rather than pretending.++Measured output sizes, from `plan_disk.py` over the shards already on this+machine (PNG crops, 20 % hand yield):++```+split persons frames body hands originals TOTAL+3dpw_train 22,405 17,410 1.7G 0.7G 4.4G 6.7G+aic_train 287,391 169,923 21.6G 8.6G 11.9G 42.1G+harmony4d_train 496,036 248,018 37.2G 14.9G 347.2G 399.3G+mpii_train 7,135 5,441 0.5G 0.2G 1.1G 1.9G+```++Two things fall out of that table.++**`original_images` is the whole cost.** Everything except Harmony4D fits+comfortably: coco + mpii + aic + 3dpw is roughly 60 GB of originals and 40 GB of+crops. Harmony4D alone is 347 GB of 4K frames and does not fit in 172 GB at any+setting. Build it with `--originals none` — `body_crops` and `hands_crops` are+self-contained, and the npz records `image_relpath`, so the originals for any+subset can be re-fetched later.++**`--crop-format jpg` cuts the crops 3×** (75 KB → 24 KB per 224 crop) and+decodes ~4× faster, which also speeds up training. It is not the default only+because PNG keeps byte-identity with the existing `data/` tree. If you are+rebuilding from scratch anyway, use jpg.++## Datasets you fetch by hand (AIC, 3DPW)++Both are one archive behind a login, so you download and `rsync` them yourself.+The only thing that matters afterwards is **what `--image-dir` points at**: it+is the root that the parquet `image` strings are relative to, and that root+differs per dataset because `image_relpath()` prepends a different prefix.++**AIC** — `image` is a bare `<hash>.jpg`, and the resolver prepends+`train/images/`. So unpack until this exists:++```+$AIC_IMG_DIR/train/images/0000252aea98840a550dac9a78c476ecb9f47ffa.jpg+```++which is exactly the layout already on this machine at+`~/sam3d_data/images/aic`. Then:++```bash+python datasets_pipeline/build_split.py --split aic_train \+ --annotation-dir $ANN_DIR/aic_train --image-dir $AIC_IMG_DIR \+ --data-root data --validate+```++`imageFiles.zip`-style archives often unpack one level deeper than you expect;+if the build reports `missing_image` for everything, the root is wrong by one+directory. A single `--max-samples 20` run tells you in seconds.++**3DPW** — `image` is `<sequence>/image_00386.jpg` and the resolver prepends+nothing, so sequence folders sit directly under the root:++```+$THREEDPW_IMG_DIR/outdoors_climbing_01/image_00386.jpg+```++You need `imageFiles.zip` only. `sequenceFiles.zip` holds the original SMPL+ground truth, which this pipeline never reads — the MHR fits come from the+parquet. Keep it for the 3DPW *benchmark* (`benchmark/eval_3dpw.py`), not for+training data.++Same command, `--split 3dpw_train`. Both are small enough for+`--originals copy` (4.4 GB and 11.9 GB of referenced frames).++## EgoHumans++Google Drive, one `.tar.gz` per take, and the untar needs+`--strip-components=8` or you get eight nested empty parents:++```bash+pip install gdown+gdown --folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR -O $EGOHUMANS_DATA_DIR+cd $EGOHUMANS_DATA_DIR/01_tagging && tar xzf 001_tagging.tar.gz --strip-components=8+```++The dataset advertises 550 GB, but `egohumans_train` references only **89 takes+across 7 sequences** — 272k persons over 268k 4K frames, ~107 GB of referenced+frames. Get the exact shopping list before downloading anything:++```bash+python datasets_pipeline/probe_split.py \+ --annotation-dir $ANN_DIR/egohumans_train --list-groups 2+```++Only the `exo/` cameras are referenced; `ego/`, `colmap/` and `processed_data/`+inside each tarball are dead weight you can delete after extraction. Frames+must be undistorted (`undistort.md`), then build per take with+`--image-prefix 01_tagging/001_tagging/`, or let `stream_archives.py` do it.++Worth knowing before you spend the 107 GB: it is good body data (multi-view,+272k persons) but useless for hands — sports at distance, so only 21.5% of+hands clear 64 px and the finger articulation is 6.3 mm, the lowest in the+corpus.++## SA-1B++Accept the terms at <https://ai.meta.com/datasets/segment-anything-downloads/>+to get `links.txt` (one `sa_NNNNNN.tar <signed-url>` per line), then:++```bash+python datasets_pipeline/stream_archives.py --dataset sa1b \+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \+ --work-dir /scratch/sa1b --data-root data \+ --links-file ~/links.txt --max-archives 50 --build-every 10 \+ -- --originals none --crop-format jpg+```++Any prefix of the tar list is a valid dataset, so you can stop whenever. But+the measured economics are poor, and I would not start here:++- References are spread **uniformly over all 1000 tars** — 994 of 1000 id+ blocks are hit by just 3% of the annotations — so there is no dense subset to+ cherry-pick.+- Only **11.4%** of SA-1B is referenced. Each ~10 GB tar yields ~1,270 useful+ frames ≈ 3,400 crops, i.e. **~2.3 MB of download per crop** against AIC's+ 0.13 MB. The full 3.4M crops cost ~11 TB.+- It is the one split with real visibility flags: a mean of **44 of 70**+ keypoints are literal `(0, 0)` placeholders, and only **0.3%** of hands are+ fully visible.++That last point is a trap for any converter. `tools/parquet_to_npz.py` drops the+visibility column (`joints_2d = kp2[:, :2]`), so a naive SA-1B build writes+those placeholders as if they were observations at the image corner — poisoning+the 2D loss and blowing every hand box out to the frame edge. `build_split.py`+stores `joints_2d_vis` / `joints_3d_conf` and refuses to cut a hand crop unless+all 21 of its keypoints are visible. Every other split is 100% visible, so this+only bites on SA-1B.++## Datasets that do not fit: `stream_archives.py`++Never holds more than one archive: download → extract → undistort → build →+delete. Any prefix of the archive list is a valid dataset, so you can stop when+the disk or the clock says so and resume later.++```bash+python datasets_pipeline/stream_archives.py --dataset harmony4d \+ --split harmony4d_train --annotation-dir $ANN_DIR/harmony4d_train \+ --work-dir /scratch/h4d --data-root data \+ --undistort-python ~/anaconda3/envs/sam_3d_body_data/bin/python \+ -- --originals none --crop-format jpg++python datasets_pipeline/stream_archives.py --dataset sa1b \+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \+ --work-dir /scratch/sa1b --data-root data \+ --links-file ~/sa1b_links.txt --max-archives 200 --build-every 20 \+ -- --originals none+```++`--min-free-gb` (default 25) aborts before an archive that would fill the disk.+Read `undistort.md` before the first Harmony4D run — Meta's script has three+blockers that a fresh clone hits.++## Hand crops++The hand box is free: `joints_2d` is already in full-frame pixel coordinates, so+the 21 keypoints of each hand give a tight box with no extra annotation.++- `--hand-min-px 64` (default) is SAM3D's own validity gate, applied to the raw+ keypoint span. On AIC this keeps ~17 % of hands, at a median 145 source pixels+ upsampled 1.5× to 224.+- `--hand-expand 2.0` stores margin around the hand so you can jitter the box+ *inside* the stored crop at train time. That is what makes a model tolerant of+ a real detector's boxes, and it is why cropping dynamically from originals+ buys nothing.+- Crops are stored **unflipped**. SAM3D trains one right-hand model and mirrors+ the left; flipping a loaded crop is a free numpy slice, so baking the mirror+ into disk would only remove a choice.+- `hand_valid_l` / `hand_valid_r` in the npz say whether a crop exists;+ `hand_box_*` (tight) and `hand_crop_*` (the square actually written) let you+ re-derive any tighter crop without touching the originals.++## Which split to spend money on++`probe_split.py` measured every split from its annotations. *Articulation* is+how far the labels' finger poses sit from ONE constant hand pose, in mm at the+fingertips — an upper bound on what a hand branch could learn from them.+*≥96 px* is the share of hands whose crop would be at least 96 source pixels.++| split | persons | articulation | hands ≥96 px | download |+|---|---|---|---|---|+| **egoexo4d_procedure** | ~350k | **16.1 mm** | **85.5%** | licence + CLI |+| egoexo4d_physical | ~700k | 14.5 mm | 19.3% | licence + CLI |+| harmony4d | 496k | 12.6 mm | 3.4% | 352 GB |+| sa1b | 3.4M | 8.5 mm | 0.3% visible at all | ~11 TB |+| 3dpw | 22k | 7.5 mm | 71.4% | 4.6 GB |+| mpii | 7k | 7.4 mm | 11.5% | 12.1 GB |+| aic | 287k | 6.9 mm | 16.6% | 36.4 GB |+| coco | ~42k | 6.5 mm | 3.0% | 13.5 GB |+| egohumans | 272k | 6.3 mm | 6.4% | 107 GB |++COCO, MPII and AIC have no finger keypoints and 3DPW is SMPL, so the official+MHR fits fall back on the prior — a hand branch trained on them learns a+constant. **EgoExo4D procedure** (cooking, bike repair, piano — hands doing+things close to the camera) is the only split in the corpus where both terms+are good at once: 2.5× the articulation of COCO/AIC and two thirds of hands+above 128 px. It is also cheap, ~8 GB of referenced frames for 350k persons.+That is where the licence you are waiting on pays off.++Re-run the check on your own build at any time:++```bash+python datasets_pipeline/verify_split.py --dir data/sam3d_gt_egoexo4d --hand-signal+```diff --git a/datasets_pipeline/build_split.py b/datasets_pipeline/build_split.pynew file mode 100755index 0000000..e359360--- /dev/null+++ b/datasets_pipeline/build_split.py@@ -0,0 +1,331 @@+#!/usr/bin/env python3+"""Turn one parquet split + its source frames into a trainable dataset folder.++Output layout, per source dataset (train and test merge into one folder, which+is what data/ already does and what the training script's discovery expects):++ <data_root>/sam3d_gt_<dataset>/+ original_images/ the referenced source frames, native relative paths+ annotations/ <name>.npz+ body_crops/ <name>.png 224 square person crop+ hands_crops/ <name>_l.png <name>_r.png+ images -> body_crops compat symlink, see below++`body_crops` is byte-identical to what `tools/parquet_to_npz.py` writes into+`images/`: this script imports that module's crop and annotation builders+rather than re-implementing them, so a folder built here and a folder built+there are interchangeable. The `images` symlink means the current+`SAM3DStudentDataset._find_dataset_dirs` finds the folder with no code change.++The npz keeps every key the training script already reads and adds five that+it ignores today and that a hand branch will need:++ image_relpath (str) exact source path — the existing `<dataset>_<stem>`+ naming flattens "/" to "_" and is NOT invertible for+ harmony4d / egohumans / egoexo4d / 3dpw+ joints_2d_vis (70,) per-keypoint visibility, dropped by parquet_to_npz.+ It is all-ones on coco/mpii/aic/3dpw/harmony4d/+ egohumans/egoexo4d, but on SA-1B a mean of 44 of 70+ keypoints are exactly (0, 0) placeholders — writing+ those as if they were observations poisons the 2D+ loss and inflates every hand box.+ joints_3d_conf (70,) the matching confidence column of keypoints_3d+ hand_box_l/_r (4,) tight xyxy over the 21 hand keypoints, full-frame px+ hand_crop_l/_r (4,) the square crop actually written (NaN if skipped)+ hand_valid_l/_r () bool: was a crop written++Hand crops are stored UNFLIPPED. SAM3D trains a single right-hand model and+mirrors the left at inference; flipping a loaded crop is a free numpy slice, so+baking the mirror into disk would only remove a choice.++ python datasets_pipeline/build_split.py \+ --split coco_train --annotation-dir $ANN_DIR/coco_train \+ --image-dir $COCO_IMG_DIR --data-root data+"""++from __future__ import annotations++import argparse+import os+import shutil+import sys+from pathlib import Path++import cv2+import numpy as np+import pandas as pd+from tqdm import tqdm++HERE = Path(__file__).resolve().parent+sys.path.insert(0, str(HERE))+sys.path.insert(0, str(HERE.parent))++from registry import dataset_for_split, image_relpath, output_dir_for_split # noqa: E402+from tools.parquet_to_npz import ( # noqa: E402+ build_annotation, get_square_crop_padded, reproj_error,+)++# MHR70 keypoint blocks. Index 41 is right_wrist, 62 is left_wrist; each hand+# block is its 20 finger joints plus that wrist. Taken from JOINT_NAMES in+# instanthmr_distill_train/train_distill_mhr_only.py.+RIGHT_HAND_KPS = list(range(21, 42))+LEFT_HAND_KPS = list(range(42, 63))+++def hand_boxes(joints_2d: np.ndarray,+ vis: np.ndarray | None = None) -> dict[str, np.ndarray | None]:+ """Tight xyxy box over each hand's 21 keypoints, in full-frame pixels.++ Returns None for a hand that is not fully visible. A partially visible hand+ cannot give a usable box: the invisible keypoints are stored as (0, 0), so+ including even one drags the box to the image corner. On SA-1B that is the+ normal case — only 0.3% of hands have all 21 keypoints marked visible.+ """+ out: dict[str, np.ndarray | None] = {}+ for side, idx in (("l", LEFT_HAND_KPS), ("r", RIGHT_HAND_KPS)):+ if vis is not None and not (vis[idx] > 0).all():+ out[side] = None+ continue+ p = joints_2d[idx]+ out[side] = np.array([p[:, 0].min(), p[:, 1].min(),+ p[:, 0].max(), p[:, 1].max()], dtype=np.float32)+ return out+++def square_box(box: np.ndarray, expand: float) -> np.ndarray:+ cx, cy = (box[0] + box[2]) / 2.0, (box[1] + box[3]) / 2.0+ half = max(box[2] - box[0], box[3] - box[1]) * expand / 2.0+ return np.array([cx - half, cy - half, cx + half, cy + half], dtype=np.float32)+++def enough_inside(sq: np.ndarray, h: int, w: int, min_frac: float) -> bool:+ """Reject hands that are mostly outside the frame — the crop would be padding."""+ ix1, iy1 = max(0.0, sq[0]), max(0.0, sq[1])+ ix2, iy2 = min(float(w), sq[2]), min(float(h), sq[3])+ if ix2 <= ix1 or iy2 <= iy1:+ return False+ area = (sq[2] - sq[0]) * (sq[3] - sq[1])+ return ((ix2 - ix1) * (iy2 - iy1)) / max(area, 1e-6) >= min_frac+++def write_crop(img_rgb, box, size, path: Path) -> None:+ crop, _ = get_square_crop_padded(img_rgb, box, expand=1.0)+ crop = cv2.resize(crop, (size, size), interpolation=cv2.INTER_LINEAR)+ cv2.imwrite(str(path), cv2.cvtColor(crop, cv2.COLOR_RGB2BGR))+++def place_original(src: Path, dst: Path, mode: str) -> None:+ if mode == "none" or dst.exists():+ return+ dst.parent.mkdir(parents=True, exist_ok=True)+ if mode == "copy":+ shutil.copy2(src, dst)+ elif mode == "link":+ os.symlink(os.path.relpath(src.resolve(), dst.parent), dst)+ elif mode == "hardlink":+ try:+ os.link(src, dst)+ except OSError:+ shutil.copy2(src, dst)+++def parse_args() -> argparse.Namespace:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--split", required=True, help="e.g. coco_train, harmony4d_train")+ p.add_argument("--annotation-dir", required=True, type=Path)+ p.add_argument("--image-dir", required=True, type=Path,+ help="root the parquet `image` paths are relative to")+ p.add_argument("--data-root", type=Path, default=Path("data"))+ p.add_argument("--output-dir", type=Path, default=None,+ help="override the sam3d_gt_<dataset> destination")++ p.add_argument("--body-size", type=int, default=224)+ p.add_argument("--crop-format", choices=("png", "jpg"), default="png",+ help="png keeps body_crops byte-identical to the existing "+ "data/ tree; jpg is ~5x smaller and ~4x faster to decode")+ p.add_argument("--jpg-quality", type=int, default=95)++ p.add_argument("--hands", action=argparse.BooleanOptionalAction, default=True)+ p.add_argument("--hand-size", type=int, default=224)+ p.add_argument("--hand-expand", type=float, default=2.0,+ help="square hand box = 21-keypoint span x this. 2.0 leaves "+ "room to jitter the box inside the stored crop at "+ "train time, which is what makes the model tolerant of "+ "a real detector's boxes.")+ p.add_argument("--hand-min-px", type=float, default=64.0,+ help="skip hands whose SPAN (before --hand-expand) is smaller "+ "than this in source pixels. 64 is SAM3D's own gate.")+ p.add_argument("--hand-min-inside", type=float, default=0.6,+ help="skip hands whose square crop is less than this fraction "+ "inside the frame")++ p.add_argument("--originals", choices=("copy", "link", "hardlink", "none"),+ default="copy",+ help="copy: self-contained, prunes unreferenced source images. "+ "link/hardlink: free, but the source root must survive.")++ p.add_argument("--image-prefix", default=None,+ help="only rows whose `image` starts with this — the hook "+ "stream_archives.py uses to build one archive at a time")+ p.add_argument("--max-samples", type=int, default=None)+ p.add_argument("--keep-invalid", action="store_true",+ help="keep rows with mhr_valid == False (dropped by default)")+ p.add_argument("--validate", action="store_true",+ help="assert < --reproj-tol px reprojection error per person")+ p.add_argument("--reproj-tol", type=float, default=1.0)+ return p.parse_args()+++def main() -> None:+ args = parse_args()+ ds = dataset_for_split(args.split)+ out_dir = args.output_dir or output_dir_for_split(args.split, args.data_root)++ dirs = {k: out_dir / k for k in+ ("original_images", "annotations", "body_crops", "hands_crops")}+ for k, d in dirs.items():+ if k == "original_images" and args.originals == "none":+ continue+ if k == "hands_crops" and not args.hands:+ continue+ d.mkdir(parents=True, exist_ok=True)++ # Let the unmodified training script discover this folder.+ compat = out_dir / "images"+ if not compat.exists():+ os.symlink("body_crops", compat)++ parquets = sorted(args.annotation_dir.glob("*.parquet"))+ if not parquets:+ sys.exit(f"no parquet files in {args.annotation_dir}")++ ext = args.crop_format+ imwrite_params = ([int(cv2.IMWRITE_JPEG_QUALITY), args.jpg_quality]+ if ext == "jpg" else [])+ existing = {p.stem for p in dirs["annotations"].glob("*.npz")}+ total = len(existing)+ st = {k: 0 for k in ("crops", "hands_l", "hands_r", "hand_too_small",+ "hand_out_of_frame", "hand_occluded", "skipped_existing",+ "skipped_invalid", "skipped_prefix", "missing_image",+ "failed")}++ pbar = tqdm(parquets, desc=f"{args.split} shards", unit="shard")+ for pq_path in pbar:+ df = pd.read_parquet(pq_path)+ img_cache: dict[str, np.ndarray | None] = {}++ for _, row in df.iterrows():+ dataset, image = row["dataset"], row["image"]+ if args.image_prefix and not image.startswith(args.image_prefix):+ st["skipped_prefix"] += 1+ continue++ stem = (os.path.splitext(image)[0]+ .replace("/", "_").replace(os.sep, "_").replace(" ", "_"))+ name = f"{dataset}_{stem}_p{int(row['person_id'])}"+ if name in existing:+ st["skipped_existing"] += 1+ continue+ if not args.keep_invalid and not bool(row["mhr_valid"]):+ st["skipped_invalid"] += 1+ continue++ relpath = image_relpath(dataset, image)+ if image not in img_cache:+ bgr = cv2.imread(str(args.image_dir / relpath))+ img_cache[image] = (cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)+ if bgr is not None else None)+ img_rgb = img_cache[image]+ if img_rgb is None:+ st["missing_image"] += 1+ continue++ h, w = img_rgb.shape[:2]+ try:+ ann = build_annotation(row, h, w)+ if args.validate:+ err = reproj_error(ann["joints_3d"], ann["cam_trans"],+ ann["joints_2d"], ann["cam_focal_length"],+ ann["orig_shape"])+ if err > args.reproj_tol:+ raise ValueError(f"reproj err {err:.2f}px > {args.reproj_tol}px")++ body_crop, sq_bbox = get_square_crop_padded(img_rgb, ann["bbox"],+ expand=1.2)+ body_crop = cv2.resize(body_crop, (args.body_size, args.body_size),+ interpolation=cv2.INTER_LINEAR)+ cv2.imwrite(str(dirs["body_crops"] / f"{name}.{ext}"),+ cv2.cvtColor(body_crop, cv2.COLOR_RGB2BGR),+ imwrite_params)++ ann["bbox_square"] = sq_bbox+ ann["image_relpath"] = relpath+ ann["dataset"] = dataset+ kp2 = np.asarray(row["keypoints_2d"].tolist(), dtype=np.float32)+ kp3 = np.asarray(row["keypoints_3d"].tolist(), dtype=np.float32)+ vis = kp2.reshape(-1, 3)[:, 2].copy()+ ann["joints_2d_vis"] = vis+ ann["joints_3d_conf"] = kp3.reshape(-1, 4)[:, 3].copy()++ boxes = hand_boxes(ann["joints_2d"], vis)+ for side in ("l", "r"):+ tight = boxes[side]+ nan4 = np.full(4, np.nan, dtype=np.float32)+ ann[f"hand_box_{side}"] = tight if tight is not None else nan4+ ann[f"hand_crop_{side}"] = nan4+ ann[f"hand_valid_{side}"] = np.array(False)+ if tight is None:+ st["hand_occluded"] += 1+ continue+ span = max(tight[2] - tight[0], tight[3] - tight[1])+ sq = square_box(tight, args.hand_expand)+ if not args.hands:+ continue+ if span < args.hand_min_px:+ st["hand_too_small"] += 1+ continue+ if not enough_inside(sq, h, w, args.hand_min_inside):+ st["hand_out_of_frame"] += 1+ continue+ write_crop(img_rgb, sq, args.hand_size,+ dirs["hands_crops"] / f"{name}_{side}.{ext}")+ ann[f"hand_crop_{side}"] = sq+ ann[f"hand_valid_{side}"] = np.array(True)+ st[f"hands_{side}"] += 1++ np.savez(dirs["annotations"] / f"{name}.npz", **ann)+ if args.originals != "none":+ place_original(args.image_dir / relpath,+ dirs["original_images"] / relpath,+ args.originals)+ except Exception as e: # noqa: BLE001+ st["failed"] += 1+ tqdm.write(f"[fail] {name}: {e}")+ continue++ existing.add(name)+ st["crops"] += 1+ total += 1+ if args.max_samples and total >= args.max_samples:+ pbar.close()+ report(st, total, out_dir)+ return++ pbar.set_postfix(crops=st["crops"], hands=st["hands_l"] + st["hands_r"],+ missing=st["missing_image"])++ report(st, total, out_dir)+++def report(st: dict, total: int, out_dir: Path) -> None:+ print("=" * 62)+ print(f"BUILD COMPLETE -> {out_dir}")+ for k, v in st.items():+ print(f" {k:20s}: {v:,}")+ print(f" {'total on disk':20s}: {total:,}")+ print("=" * 62)+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/fetch_annotations.py b/datasets_pipeline/fetch_annotations.pynew file mode 100755index 0000000..ea27311--- /dev/null+++ b/datasets_pipeline/fetch_annotations.py@@ -0,0 +1,110 @@+#!/usr/bin/env python3+"""Download parquet annotation shards from facebook/sam-3d-body-dataset.++The gated HF repo needs an accepted licence and a token (`hf auth login`, or+HF_TOKEN in the environment). Downloads are resumable and skip shards already+on disk, so re-running after a killed job is free.++ # everything for one split+ python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR++ # a slice, for a cheap first pass on a huge split+ python datasets_pipeline/fetch_annotations.py --splits sa1b_train --out $ANN_DIR \+ --max-shards 50++ # just tell me what is available and how big it is+ python datasets_pipeline/fetch_annotations.py --list+"""++from __future__ import annotations++import argparse+import sys+from concurrent.futures import ThreadPoolExecutor, as_completed+from pathlib import Path++from huggingface_hub import HfApi, hf_hub_download+from tqdm import tqdm++sys.path.insert(0, str(Path(__file__).resolve().parent))+from registry import HF_ANNOTATION_REPO, SPLIT_TO_DATASET # noqa: E402+++def shard_index(api: HfApi) -> dict[str, list[str]]:+ """split -> sorted list of repo-relative parquet paths."""+ out: dict[str, list[str]] = {}+ for f in api.list_repo_files(HF_ANNOTATION_REPO, repo_type="dataset"):+ if not f.endswith(".parquet"):+ continue+ parts = f.split("/")+ if len(parts) < 3:+ continue+ out.setdefault(parts[1], []).append(f)+ return {k: sorted(v) for k, v in sorted(out.items())}+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--splits", nargs="+", help="e.g. coco_train harmony4d_train")+ p.add_argument("--out", type=Path, help="destination root; one folder per split")+ p.add_argument("--max-shards", type=int, default=None,+ help="take only the first N shards of each split")+ p.add_argument("--workers", type=int, default=8)+ p.add_argument("--list", action="store_true", help="print available splits and exit")+ args = p.parse_args()++ api = HfApi()+ index = shard_index(api)++ if args.list:+ print(f"{'split':26s} {'shards':>7s} ~persons")+ for split, files in index.items():+ # ~1,750 rows/shard, measured across coco/aic/sa1b/egoexo4d/egohumans.+ print(f"{split:26s} {len(files):7d} {len(files) * 1750:>9,d}")+ return++ if not args.splits or not args.out:+ p.error("--splits and --out are required unless --list is given")++ unknown = [s for s in args.splits if s not in index]+ if unknown:+ p.error(f"unknown split(s): {unknown}. Run --list for the full set.")+ for s in args.splits:+ if s not in SPLIT_TO_DATASET:+ print(f"[warn] {s} has no entry in registry.py; build_split.py will "+ f"not know where its images live.", file=sys.stderr)++ for split in args.splits:+ files = index[split]+ if args.max_shards:+ files = files[: args.max_shards]+ dest = args.out / split+ dest.mkdir(parents=True, exist_ok=True)+ todo = [f for f in files if not (dest / Path(f).name).exists()]+ print(f"{split}: {len(files)} shards, {len(todo)} to download -> {dest}")+ if not todo:+ continue++ def grab(repo_file: str) -> None:+ hf_hub_download(HF_ANNOTATION_REPO, repo_file, repo_type="dataset",+ local_dir=str(dest.parent / ".hf_staging"))+ src = dest.parent / ".hf_staging" / repo_file+ src.replace(dest / src.name)++ with ThreadPoolExecutor(max_workers=args.workers) as ex:+ futs = {ex.submit(grab, f): f for f in todo}+ for fut in tqdm(as_completed(futs), total=len(futs), unit="shard"):+ try:+ fut.result()+ except Exception as e: # noqa: BLE001+ tqdm.write(f"[fail] {futs[fut]}: {e}")++ staging = args.out / ".hf_staging"+ if staging.exists():+ import shutil+ shutil.rmtree(staging, ignore_errors=True)+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/fetch_images.py b/datasets_pipeline/fetch_images.pynew file mode 100755index 0000000..d95dec8--- /dev/null+++ b/datasets_pipeline/fetch_images.py@@ -0,0 +1,183 @@+#!/usr/bin/env python3+"""Download and extract the source frames for one SAM-3D-Body dataset.++Every dataset that CAN be fetched with a plain URL is fetched with a plain URL+(`wget -c`, so a killed job resumes). The four that cannot — AIC, 3DPW,+EgoExo4D, SA-1B — need a login or a signed licence, and this script prints the+exact commands instead of pretending.++ python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco+ python datasets_pipeline/fetch_images.py --dataset mpii --dest $IMG_ROOT/mpii+ python datasets_pipeline/fetch_images.py --dataset harmony4d --dest $IMG_ROOT/harmony4d_raw \+ --archives train/09_karate.zip --no-keep-archive+ python datasets_pipeline/fetch_images.py --dataset sa1b --dest $IMG_ROOT/sa1b \+ --links-file ~/sa1b_links.txt --max-archives 20++`--dry-run` prints the plan and the byte counts without downloading anything.+"""++from __future__ import annotations++import argparse+import shutil+import subprocess+import sys+import tarfile+import zipfile+from pathlib import Path++sys.path.insert(0, str(Path(__file__).resolve().parent))+from registry import DATASETS # noqa: E402++HF_RESOLVE = "https://huggingface.co/datasets/{repo}/resolve/main/{path}"+++def run(cmd: list[str]) -> None:+ print("+", " ".join(cmd), flush=True)+ subprocess.run(cmd, check=True)+++def wget(url: str, dest: Path) -> Path:+ dest.mkdir(parents=True, exist_ok=True)+ out = dest / url.split("/")[-1].split("?")[0]+ run(["wget", "-c", "-O", str(out), url])+ return out+++def extract(archive: Path, into: Path, strip_top: bool = False) -> None:+ into.mkdir(parents=True, exist_ok=True)+ print(f"extracting {archive.name} -> {into}", flush=True)+ if archive.suffix == ".zip":+ with zipfile.ZipFile(archive) as z:+ z.extractall(into)+ elif archive.name.endswith((".tar.gz", ".tgz", ".tar")):+ mode = "r:gz" if archive.name.endswith((".tar.gz", ".tgz")) else "r"+ with tarfile.open(archive, mode) as t:+ t.extractall(into, filter="data")+ else:+ raise SystemExit(f"don't know how to extract {archive.name}")+++def manual(ds_name: str, notes: str) -> None:+ print("=" * 70)+ print(f"{ds_name}: needs a credential or a signed licence — cannot wget it.")+ print("=" * 70)+ print(notes)+ print()+ print("Once the frames are on disk, point build_split.py --image-dir at the "+ "root that the parquet `image` paths are relative to and carry on.")+ sys.exit(2)+++def fetch_direct(ds, dest: Path, keep: bool, dry: bool) -> None:+ for url in ds.urls:+ if dry:+ print(f"[dry-run] wget {url}")+ continue+ arc = wget(url, dest)+ extract(arc, dest)+ if not keep:+ arc.unlink()+++def fetch_hf_zips(ds, dest: Path, wanted: list[str] | None,+ max_archives: int | None, keep: bool, dry: bool) -> None:+ archives = list(wanted or ds.archives)+ unknown = [a for a in archives if a not in ds.archives]+ if unknown:+ raise SystemExit(f"unknown archive(s) {unknown}\nknown:\n "+ + "\n ".join(ds.archives))+ if max_archives:+ archives = archives[:max_archives]+ for rel in archives:+ url = HF_RESOLVE.format(repo=ds.hf_repo, path=rel)+ if dry:+ print(f"[dry-run] wget {url}")+ continue+ # Harmony4D zips already carry the train/<scene>/ prefix internally.+ arc = wget(url, dest / "_archives")+ extract(arc, dest)+ if not keep:+ arc.unlink()+++def fetch_links_file(ds, dest: Path, links_file: Path,+ max_archives: int | None, keep: bool, dry: bool) -> None:+ """SA-1B: Meta hands you a two-column `name<TAB>signed-url` file."""+ lines = [ln.strip() for ln in links_file.read_text().splitlines() if ln.strip()]+ entries = []+ for ln in lines:+ parts = ln.split()+ if len(parts) == 2 and parts[1].startswith("http"):+ entries.append((parts[0], parts[1]))+ if not entries:+ raise SystemExit(f"no `name url` pairs found in {links_file}")+ if max_archives:+ entries = entries[:max_archives]+ print(f"{len(entries)} SA-1B shards to fetch")+ for name, url in entries:+ if dry:+ print(f"[dry-run] {name}")+ continue+ arc = dest / "_archives" / name+ arc.parent.mkdir(parents=True, exist_ok=True)+ run(["wget", "-c", "-O", str(arc), url])+ # Each tar holds sa_*.jpg plus per-image json masks we do not need.+ with tarfile.open(arc) as t:+ members = [m for m in t.getmembers() if m.name.endswith(".jpg")]+ t.extractall(dest, members=members, filter="data")+ if not keep:+ arc.unlink()+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--dataset", required=True, choices=sorted(DATASETS))+ p.add_argument("--dest", type=Path, required=True)+ p.add_argument("--archives", nargs="*", default=None,+ help="subset of archives (hf_zips only); default is all")+ p.add_argument("--max-archives", type=int, default=None)+ p.add_argument("--links-file", type=Path, default=None, help="SA-1B links.txt")+ p.add_argument("--keep-archive", action=argparse.BooleanOptionalAction,+ default=False, help="keep the downloaded zip/tar after extracting")+ p.add_argument("--dry-run", action="store_true")+ p.add_argument("--list-archives", action="store_true")+ args = p.parse_args()++ ds = DATASETS[args.dataset]+ if args.list_archives:+ for a in ds.archives or ("(single blob)",):+ print(a)+ return++ print(f"{ds.name}: method={ds.method} ~{ds.approx_gb or '?'} GB "+ f"undistort={'YES' if ds.needs_undistort else 'no'}")+ if ds.notes:+ print(" " + ds.notes.replace("\n", "\n "))+ print()++ if ds.method == "direct":+ fetch_direct(ds, args.dest, args.keep_archive, args.dry_run)+ elif ds.method == "hf_zips":+ fetch_hf_zips(ds, args.dest, args.archives, args.max_archives,+ args.keep_archive, args.dry_run)+ elif ds.method == "links_file":+ if not args.links_file:+ manual(ds.name, ds.notes)+ fetch_links_file(ds, args.dest, args.links_file, args.max_archives,+ args.keep_archive, args.dry_run)+ else:+ manual(ds.name, ds.notes)++ if ds.needs_undistort:+ print()+ print("!! These frames are DISTORTED. Run the undistortion step before "+ "build_split.py — see datasets_pipeline/undistort.md.")+ free = (shutil.disk_usage(args.dest).free / 1e9+ if args.dest.exists() else shutil.disk_usage(".").free / 1e9)+ print(f"done -> {args.dest} ({free:.0f} GB free)")+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/plan_disk.py b/datasets_pipeline/plan_disk.pynew file mode 100755index 0000000..9d64db0--- /dev/null+++ b/datasets_pipeline/plan_disk.py@@ -0,0 +1,91 @@+#!/usr/bin/env python3+"""Cost a split before you download it.++Reads only the `image` and `mhr_valid` columns of the parquets — cheap, no+frames needed — and reports how many person crops and unique source frames a+split will produce, plus what each output folder will weigh.++ python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/aic_train+ python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/* --brief++Per-file sizes are measured on the existing data/ tree, not assumed:+ 224 PNG body crop ~75 KB, 224 JPEG q95 ~24 KB.+Source-frame sizes come from registry.py's measured download sizes divided by+the frame count, except where the dataset is a known fixed resolution.+"""++from __future__ import annotations++import argparse+from pathlib import Path++import pyarrow.parquet as pq++# KB per stored file, measured on data/sam3d_gt_*.+CROP_KB = {"png": 75.0, "jpg": 24.0}+# KB per source frame, measured (aic/coco/mpii/3dpw) or estimated from the+# 4K zips (harmony4d).+FRAME_KB = {"coco": 160.0, "mpii": 210.0, "aic": 70.0, "3dpw": 250.0,+ "harmony4d": 1400.0, "egohumans": 400.0, "egoexo4d": 400.0,+ "sa1b": 700.0}+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--annotation-dir", nargs="+", type=Path, required=True)+ p.add_argument("--crop-format", choices=("png", "jpg"), default="png")+ p.add_argument("--hand-yield", type=float, default=0.20,+ help="fraction of the 2 hands per person that clear "+ "--hand-min-px; 0.20 is what AIC measures at 64 px")+ p.add_argument("--brief", action="store_true")+ args = p.parse_args()++ ckb = CROP_KB[args.crop_format]+ tot = {"rows": 0, "frames": 0, "body": 0.0, "hand": 0.0, "orig": 0.0}+ hdr = (f"{'split':26s} {'persons':>10s} {'frames':>10s} "+ f"{'body':>8s} {'hands':>8s} {'originals':>10s} {'TOTAL':>9s}")+ print(hdr)+ print("-" * len(hdr))++ for d in args.annotation_dir:+ shards = sorted(d.glob("*.parquet"))+ if not shards:+ continue+ split = d.name+ dataset = split.split("_")[0]+ rows = 0+ frames: set[str] = set()+ for s in shards:+ t = pq.read_table(s, columns=["image", "mhr_valid"])+ im = t.column("image").to_pylist()+ mv = t.column("mhr_valid").to_pylist()+ rows += sum(mv)+ frames.update(i for i, v in zip(im, mv) if v)++ body = rows * ckb / 1e6 # GB+ hand = rows * 2 * args.hand_yield * ckb / 1e6+ orig = len(frames) * FRAME_KB.get(dataset, 400.0) / 1e6+ print(f"{split:26s} {rows:10,d} {len(frames):10,d} "+ f"{body:7.1f}G {hand:7.1f}G {orig:9.1f}G {body+hand+orig:8.1f}G")+ tot["rows"] += rows+ tot["frames"] += len(frames)+ tot["body"] += body+ tot["hand"] += hand+ tot["orig"] += orig++ if not args.brief:+ print("-" * len(hdr))+ print(f"{'TOTAL':26s} {tot['rows']:10,d} {tot['frames']:10,d} "+ f"{tot['body']:7.1f}G {tot['hand']:7.1f}G {tot['orig']:9.1f}G "+ f"{tot['body']+tot['hand']+tot['orig']:8.1f}G")+ print()+ print("`originals` is the dominant term everywhere except COCO. If it "+ "does not fit, build with --originals none: body_crops and "+ "hands_crops are self-contained, and the npz `image_relpath` "+ "field records exactly which frame each crop came from so the "+ "originals can be re-fetched later for the rows you still want.")+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/probe_split.py b/datasets_pipeline/probe_split.pynew file mode 100755index 0000000..6eb5be3--- /dev/null+++ b/datasets_pipeline/probe_split.py@@ -0,0 +1,207 @@+#!/usr/bin/env python3+"""Decide whether a split is worth its download — from the parquets alone.++Annotations are ~1 GB per split; frames are tens to thousands of GB. This+reads only the annotations and answers, before you spend the bandwidth:++ size persons, unique frames, and how many GB of frames that implies.+ Totals extrapolate over the shards ON DISK, so fetch the whole+ split's annotations first if you want the true count.+ labels keypoint visibility health. NOT cosmetic: SA-1B stores invisible+ keypoints as literal (0, 0) with a visibility column that+ parquet_to_npz.py discards, so a naive build writes 44 of 70+ keypoints per person as fake observations at the image corner.+ hands how many hands are fully visible and how big their crop would be+ signal how far the finger poses sit from ONE constant hand pose, in mm at+ the fingertips — an upper bound on what a hand branch could learn++ python datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/*+ python datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/sa1b_train --shards 60+"""++from __future__ import annotations++import argparse+import sys+from pathlib import Path++import numpy as np+import pyarrow.parquet as pq++HERE = Path(__file__).resolve().parent+REPO = HERE.parent+sys.path.insert(0, str(HERE))+from plan_disk import FRAME_KB # noqa: E402++# Fraction of the frames you must actually download that this corpus+# references. 1.0 where one archive holds everything and unreferenced images+# cost nothing extra to skip. SA-1B is the exception: its references are spread+# uniformly over all 1000 tars (994 of 1000 id-blocks are hit by just 3% of the+# annotations), so reaching 1.27M referenced images means pulling ~11 TB.+ARCHIVE_COVERAGE = {"sa1b": 0.114}++FINGER_PARAMS = list(range(68, 122))+RIGHT_HAND_KPS = list(range(21, 42))+LEFT_HAND_KPS = list(range(42, 63))+RIGHT_WRIST, LEFT_WRIST = 41, 62+++def articulation_mm(model_params: np.ndarray, mhr_path: Path,+ reg_path: Path) -> tuple[float, float] | None:+ import torch++ if not mhr_path.exists() or not reg_path.exists():+ return None+ mhr = torch.jit.load(str(mhr_path), map_location="cpu").eval()+ W = torch.tensor(np.load(reg_path), dtype=torch.float32)++ def kp70(mp: torch.Tensor) -> torch.Tensor:+ cp = torch.cat([mp, torch.zeros(mp.shape[0], 45)], 1)+ jp = mhr.character_torch.model_parameters_to_joint_parameters(cp)+ ss = mhr.character_torch.joint_parameters_to_skeleton_state(jp)[..., :3] / 100.0+ v = torch.stack([ss[..., 0], -ss[..., 1], -ss[..., 2]], -1)+ return torch.einsum("kj,bjc->bkc", W, v)++ X = torch.tensor(model_params)+ Xm = X.clone()+ Xm[:, FINGER_PARAMS] = X[:, FINGER_PARAMS].mean(0, keepdim=True)+ with torch.no_grad():+ a, b = kp70(X), kp70(Xm)+ e = torch.cat([+ ((a[:, k] - a[:, [w]]) - (b[:, k] - b[:, [w]])).norm(dim=-1) * 1000+ for k, w in ((RIGHT_HAND_KPS, RIGHT_WRIST), (LEFT_HAND_KPS, LEFT_WRIST))+ ], 1).numpy()+ return float(e.mean()), float(np.percentile(e, 90))+++def probe(d: Path, shards: int, hand_expand: float, args) -> None:+ files = sorted(d.glob("*.parquet"))+ if not files:+ return+ used = files[:shards]++ rows = 0+ frames: set[str] = set()+ all_vis = 0+ zero_kps = []+ params: list[np.ndarray] = []+ spans: list[float] = []+ hands_seen = hands_full = 0++ for f in used:+ t = pq.read_table(f, columns=["image", "mhr_valid", "keypoints_2d",+ "model_params"])+ for im, mv, k, mp in zip(t.column("image").to_pylist(),+ t.column("mhr_valid").to_pylist(),+ t.column("keypoints_2d").to_pylist(),+ t.column("model_params").to_pylist()):+ if not mv:+ continue+ rows += 1+ frames.add(im)+ k = np.asarray(k, dtype=np.float32).reshape(70, 3)+ all_vis += bool((k[:, 2] > 0).all())+ zero_kps.append(int(((k[:, 0] == 0) & (k[:, 1] == 0)).sum()))+ params.append(np.asarray(mp, dtype=np.float32))+ for idx in (RIGHT_HAND_KPS, LEFT_HAND_KPS):+ hands_seen += 1+ h = k[idx]+ if not (h[:, 2] > 0).all():+ continue+ hands_full += 1+ spans.append(max(np.ptp(h[:, 0]), np.ptp(h[:, 1])) * hand_expand)++ scale = len(files) / len(used)+ dataset = d.name.split("_")[0]+ cov = ARCHIVE_COVERAGE.get(dataset, 1.0)+ gb = len(frames) * scale * FRAME_KB.get(dataset, 400.0) / 1e6 / cov++ print(f"\n== {d.name} ({len(used)}/{len(files)} shards read) ==")+ print(f" persons : {rows:,} sampled -> ~{rows * scale:,.0f} "+ f"over the {len(files)} shards ON DISK")+ print(f" unique frames : {len(frames):,} sampled -> ~{len(frames) * scale:,.0f}")+ note = "" if cov == 1.0 else f" [only {cov:.0%} of what you must pull is referenced]"+ print(f" frames to download : ~{gb:,.0f} GB "+ f"({gb * 1000 / max(rows * scale, 1):.2f} MB per person crop){note}")++ pct = 100.0 * all_vis / max(rows, 1)+ flag = "" if pct > 99 else " <-- gate hand boxes on joints_2d_vis"+ print(f" all 70 kps visible : {pct:5.1f}% "+ f"mean (0,0) placeholders {np.mean(zero_kps):.1f}/70{flag}")++ if spans:+ s = np.array(spans)+ print(f" hands fully visible: {100 * hands_full / max(hands_seen, 1):5.1f}%")+ print(f" hand crop, px : p50 {np.percentile(s, 50):4.0f} "+ f">=64px {100 * (s >= 64).mean():5.1f}% "+ f">=96px {100 * (s >= 96).mean():5.1f}% "+ f">=128px {100 * (s >= 128).mean():5.1f}%")+ else:+ print(f" hands fully visible: 0.0% -- no usable hand crops")++ art = articulation_mm(np.stack(params), args.mhr, args.regressor)+ if art is None:+ print(" finger articulation: (needs checkpoints/mhr_model.pt + the regressor)")+ return+ mean, p90 = art+ verdict = ("worth a hand branch" if mean > 20 else+ "promising — the best you will find in this corpus" if mean > 12 else+ "NOT worth a hand branch: labels are ~a constant hand pose")+ print(f" finger articulation: {mean:.1f} mm mean, {p90:.1f} mm p90 -> {verdict}")+++def list_groups(d: Path, depth: int) -> None:+ """Referenced source groups at `depth` path components, with person counts."""+ import collections++ files = sorted(d.glob("*.parquet"))+ groups: collections.Counter = collections.Counter()+ frames: dict[str, set[str]] = collections.defaultdict(set)+ for f in files:+ t = pq.read_table(f, columns=["image", "mhr_valid"])+ for im, mv in zip(t.column("image").to_pylist(),+ t.column("mhr_valid").to_pylist()):+ if not mv:+ continue+ parts = im.split("/")+ g = "/".join(parts[:depth]) if len(parts) > depth else im+ groups[g] += 1+ frames[g].add(im)+ print(f"\n== {d.name}: {len(groups)} referenced group(s) at depth {depth}, "+ f"{sum(groups.values()):,} persons ({len(files)} shards) ==")+ for g, n in sorted(groups.items()):+ print(f" {g:44s} {n:8,d} persons {len(frames[g]):8,d} frames")+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--annotation-dir", nargs="+", type=Path, required=True)+ p.add_argument("--list-groups", type=int, default=0, metavar="DEPTH",+ help="instead of probing, print the referenced source groups "+ "at this path depth, with person counts. DEPTH 2 gives "+ "the exact EgoHumans take list (<seq>/<take>) and "+ "EgoExo4D take list; DEPTH 1 gives Harmony4D scenes and "+ "3DPW sequences. This is the download shopping list — "+ "EgoHumans advertises 550 GB but only 89 takes are "+ "referenced.")+ p.add_argument("--shards", type=int, default=12,+ help="shards to read per split; 12 is plenty for the "+ "distributions, more only tightens the person count")+ p.add_argument("--hand-expand", type=float, default=1.5,+ help="matches build_split.py's square expansion when judging "+ "whether a hand clears --hand-min-px")+ p.add_argument("--mhr", type=Path, default=REPO / "checkpoints" / "mhr_model.pt")+ p.add_argument("--regressor", type=Path,+ default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")+ args = p.parse_args()+ for d in args.annotation_dir:+ if args.list_groups:+ list_groups(d, args.list_groups)+ else:+ probe(d, args.shards, args.hand_expand, args)+ print()+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/registry.py b/datasets_pipeline/registry.pynew file mode 100755index 0000000..fd90f4a--- /dev/null+++ b/datasets_pipeline/registry.py@@ -0,0 +1,220 @@+#!/usr/bin/env python3+"""Per-dataset acquisition specs for the SAM-3D-Body corpus.++One place that answers, for every split of ``facebook/sam-3d-body-dataset``:++ * how the images are obtained (a plain URL, a Hugging Face repo, a CLI that+ needs credentials, or a licence form),+ * where a parquet ``image`` string lands inside the extracted tree,+ * whether the frames need undistorting first,+ * how big it all is.++``image_relpath`` is re-exported from ``tools/parquet_to_npz.py`` rather than+re-implemented, so this module can never drift from the converter that the+existing ``data/sam3d_gt_*`` folders were built with.++Sizes are measured, not guessed:+ * COCO train2014 / MPII: HTTP Content-Length, checked 2026-08-31.+ * Harmony4D: the Hugging Face tree API for ``Jyun-Ting/Harmony4D``.+ * AIC: the OpenDataLab tarball already on this machine.+"""++from __future__ import annotations++import sys+from dataclasses import dataclass, field+from pathlib import Path++REPO_ROOT = Path(__file__).resolve().parent.parent+if str(REPO_ROOT) not in sys.path:+ sys.path.insert(0, str(REPO_ROOT))++from tools.parquet_to_npz import image_relpath # noqa: E402,F401 (re-export)++HF_ANNOTATION_REPO = "facebook/sam-3d-body-dataset"+++@dataclass(frozen=True)+class Dataset:+ """Everything needed to go from a bare machine to usable frames."""++ name: str+ splits: tuple[str, ...]+ # How to get the images. "direct" = plain URLs; everything else needs a+ # human somewhere in the loop and fetch_images.py prints instructions.+ method: str # direct | hf_zips | opendatalab | gdrive | cli | links_file+ urls: tuple[str, ...] = () # for method == "direct"+ hf_repo: str = "" # for method == "hf_zips"+ # Where image_relpath() is rooted, relative to the extraction dir. The+ # archives usually already create this, so it is normally "".+ image_root_suffix: str = ""+ needs_undistort: bool = False+ approx_gb: float = 0.0 # download size+ notes: str = ""+ # Per-scene/shard archives, for streaming one at a time. Empty means the+ # dataset arrives as one blob.+ archives: tuple[str, ...] = field(default_factory=tuple)+++# --------------------------------------------------------------------------+# The eight source datasets.+# --------------------------------------------------------------------------+HARMONY4D_TRAIN_ZIPS = (+ "01_hugging", "02_grappling", "03_grappling2", "04_sword_part1",+ "04_sword_part2", "05_sword2", "07_ballroom", "08_ballroom2",+ "09_karate", "10_karate2", "11_karate3", "12_mma", "13_mma2",+ "14_mma3", "15_mma4",+)+HARMONY4D_TEST_ZIPS = (+ "01_hugging", "03_grappling2", "05_sword2", "06_sword3",+ "08_ballroom2", "15_mma4", "16_mma5",+)++DATASETS: dict[str, Dataset] = {+ "coco": Dataset(+ name="coco",+ splits=("coco_train",),+ method="direct",+ urls=("http://images.cocodataset.org/zips/train2014.zip",),+ approx_gb=13.5,+ notes="Only train2014 is referenced by coco_train — val2014/test2014 are "+ "a waste of 9 GB. Verified over every image string in shard 0.",+ ),+ "mpii": Dataset(+ name="mpii",+ splits=("mpii_train",),+ method="direct",+ urls=("https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/"+ "mpii_human_pose_v1.tar.gz",),+ approx_gb=12.1,+ notes="Extracts to images/. Only 5,441 of ~25k images are referenced, "+ "so --originals copy reclaims ~11 GB.",+ ),+ "aic": Dataset(+ name="aic",+ splits=("aic_train",),+ method="opendatalab",+ approx_gb=36.4,+ notes="AI Challenger 2017. The original challenger.ai host is dead; the "+ "working route is OpenDataLab, which is where this corpus came "+ "from last time (see ~/sam3d_data/aic_download).\n"+ " pip install opendatalab && odl login\n"+ " odl get OpenDataLab/AI_Challenger -d $DEST\n"+ " tar xzf $DEST/OpenDataLab___AI_Challenger/raw/AI_Challenger.tar.gz\n"+ "Needs the train/images/ subtree only.",+ ),+ "3dpw": Dataset(+ name="3dpw",+ splits=("3dpw_train",),+ method="cli",+ approx_gb=4.6,+ notes="Accept the licence at "+ "https://virtualhumans.mpi-inf.mpg.de/3DPW/license.html, then "+ "download imageFiles.zip and unzip so that sequence folders "+ "(courtyard_arguing_00/, ...) sit directly under the image dir.",+ ),+ "harmony4d": Dataset(+ name="harmony4d",+ splits=("harmony4d_train", "harmony4d_test"),+ method="hf_zips",+ hf_repo="Jyun-Ting/Harmony4D",+ needs_undistort=True,+ approx_gb=351.7,+ archives=tuple(f"train/{s}.zip" for s in HARMONY4D_TRAIN_ZIPS)+ + tuple(f"test/{s}.zip" for s in HARMONY4D_TEST_ZIPS),+ notes="287 GB train + 65 GB test of 4K frames, ungated on Hugging Face. "+ "Far past any sane disk budget as a whole, so run it through "+ "stream_archives.py, which does one scene zip at a time and "+ "deletes it after the crops are written. Frames MUST be "+ "undistorted first — see undistort.md for the three known "+ "blockers in Meta's script.",+ ),+ "egohumans": Dataset(+ name="egohumans",+ splits=("egohumans_train",),+ method="gdrive",+ needs_undistort=True,+ approx_gb=107.0,+ notes="Google Drive folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR, one "+ ".tar.gz per TAKE under <sequence>/<take>.tar.gz.\n"+ " pip install gdown\n"+ " gdown --folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR -O $DEST\n"+ " tar xzf 001_tagging.tar.gz --strip-components=8 # NOT optional\n"+ "The dataset advertises 550 GB but egohumans_train references "+ "only 89 takes across 7 sequences (tagging, lego, fencing, "+ "basketball, volleyball, badminton, tennis) = 272k persons over "+ "268k 4K frames, ~107 GB of referenced frames. Get the exact "+ "shopping list with:\n"+ " probe_split.py --annotation-dir $ANN/egohumans_train --list-groups 2\n"+ "Only the exo/ cameras are referenced; ego/, colmap/ and "+ "processed_data/ in each tarball are dead weight. Undistort with "+ "sam-3d-body/data/scripts/egohumans/undistort_images.py — see "+ "undistort.md.",+ ),+ "egoexo4d": Dataset(+ name="egoexo4d",+ splits=("egoexo4d_physical_train", "egoexo4d_physical_test",+ "egoexo4d_procedure_train", "egoexo4d_procedure_test"),+ method="cli",+ approx_gb=0.0,+ notes="Needs the signed Ego-Exo4D licence (you have one pending). Then:\n"+ " pip install ego4d && egoexo -o $DEST --parts sam_3d_body\n"+ "The sam_3d_body part is already undistorted. 610 train shards "+ "~= 1.1M person crops, and it is the one split in this corpus "+ "where hands are routinely large enough to learn from.",+ ),+ "sa1b": Dataset(+ name="sa1b",+ splits=("sa1b_train",),+ method="links_file",+ approx_gb=0.0,+ notes="1,850 annotation shards ~= 3.4M person crops — bigger than every "+ "other split combined. Accept the terms at "+ "https://ai.meta.com/datasets/segment-anything-downloads/ to get "+ "links.txt (one `sa_NNNNNN.tar <signed-url>` per line), then feed "+ "it to fetch_images.py --links-file.\n"+ "MEASURED, and it is bad news: the referenced images are spread "+ "UNIFORMLY over all 1000 tars — 994 of 1000 id-blocks are hit by "+ "just 3% of the annotations — and only 11.4% of SA-1B is "+ "referenced at all. So there is no dense subset: you pull a whole "+ "~10 GB tar to get ~1,270 useful frames (~3,400 crops), i.e. "+ "~2.3 MB of download per crop against AIC's 0.13 MB. Worse, "+ "SA-1B is the ONE split whose keypoints carry real visibility "+ "flags: a mean of 44 of 70 are literal (0,0) placeholders and "+ "only 0.3% of hands are fully visible.\n"+ "Any prefix of the tar list is a valid subset — each tar is "+ "self-contained and the builder skips rows whose image is absent "+ "— so stream it and stop whenever. But spend the bandwidth on "+ "egoexo4d first.",+ ),+}++# Which parquet split belongs to which source dataset.+SPLIT_TO_DATASET = {s: d.name for d in DATASETS.values() for s in d.splits}+++def dataset_for_split(split: str) -> Dataset:+ try:+ return DATASETS[SPLIT_TO_DATASET[split]]+ except KeyError:+ raise SystemExit(+ f"unknown split {split!r}. Known splits:\n "+ + "\n ".join(sorted(SPLIT_TO_DATASET))+ ) from None+++def output_dir_for_split(split: str, data_root: Path) -> Path:+ """`coco_train` -> `<data_root>/sam3d_gt_coco`.++ Train and test of the same dataset deliberately land in the same folder:+ that is the convention the existing data/ tree already uses, and the+ training script's dataset discovery walks one level of sub-folders.+ """+ return data_root / f"sam3d_gt_{SPLIT_TO_DATASET[split]}"+++if __name__ == "__main__":+ print(f"{'dataset':12s} {'method':12s} {'GB':>7s} splits")+ for d in DATASETS.values():+ gb = f"{d.approx_gb:.1f}" if d.approx_gb else "?"+ print(f"{d.name:12s} {d.method:12s} {gb:>7s} {', '.join(d.splits)}")diff --git a/datasets_pipeline/requirements.txt b/datasets_pipeline/requirements.txtnew file mode 100644index 0000000..3af6ee7--- /dev/null+++ b/datasets_pipeline/requirements.txt@@ -0,0 +1,8 @@+# On top of the training env (numpy, opencv-python, tqdm).+pyarrow>=14+pandas>=2+huggingface_hub>=0.23 # fetch_annotations.py; needs an accepted licence + token+# Optional, per dataset:+# gdown egohumans (Google Drive folder)+# opendatalab aic (odl login && odl get OpenDataLab/AI_Challenger)+# ego4d egoexo4d (egoexo -o $DEST --parts sam_3d_body)diff --git a/datasets_pipeline/stream_archives.py b/datasets_pipeline/stream_archives.pynew file mode 100755index 0000000..26e7692--- /dev/null+++ b/datasets_pipeline/stream_archives.py@@ -0,0 +1,177 @@+#!/usr/bin/env python3+"""Build a dataset that does not fit on disk, one archive at a time.++Harmony4D is 352 GB of 4K zips and SA-1B is ~11 TB; neither can be staged+whole on a 172 GB volume. The loop below never holds more than one archive:++ download -> extract -> [undistort] -> build crops -> delete everything raw++Because build_split.py is resume-safe and silently skips rows whose image is+absent, any prefix of the archive list yields a valid, self-consistent dataset.+Stop whenever you have enough, or when the disk says so.++ # Harmony4D, scene by scene, keeping nothing but the crops+ python datasets_pipeline/stream_archives.py --dataset harmony4d \+ --split harmony4d_train --annotation-dir $ANN_DIR/harmony4d_train \+ --work-dir /scratch/h4d --data-root data++ # SA-1B, 20 tars at a time, stop after 200+ python datasets_pipeline/stream_archives.py --dataset sa1b \+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \+ --work-dir /scratch/sa1b --data-root data \+ --links-file ~/sa1b_links.txt --max-archives 200 --build-every 20++Anything passed after `--` goes straight to build_split.py, e.g.+`-- --crop-format jpg --hand-min-px 48 --originals none`.+"""++from __future__ import annotations++import argparse+import shutil+import subprocess+import sys+from pathlib import Path++HERE = Path(__file__).resolve().parent+sys.path.insert(0, str(HERE))+from registry import DATASETS, dataset_for_split # noqa: E402+++def sh(cmd: list, cwd: Path | None = None) -> None:+ print("\n+ " + " ".join(str(c) for c in cmd)+ + (f" (cwd={cwd})" if cwd else ""), flush=True)+ subprocess.run([str(c) for c in cmd], check=True,+ cwd=str(cwd) if cwd else None)+++def free_gb(p: Path) -> float:+ while not p.exists():+ p = p.parent+ return shutil.disk_usage(p).free / 1e9+++def undistort_harmony4d(raw: Path, undist: Path, scene_rel: str,+ sam3d_repo: Path, python_bin: str) -> None:+ """Run Meta's undistortion over every take of one scene.++ `--seqs` wants the TAKE folder (train/09_karate/007_karate), not the scene:+ the script appends `exo/` to whatever it is given. See undistort.md.+ """+ scene_dir = raw / scene_rel+ takes = sorted(d.name for d in scene_dir.iterdir() if d.is_dir())+ if not takes:+ raise SystemExit(f"no takes under {scene_dir}")+ seqs = ",".join(f"{scene_rel}/{t}" for t in takes)+ scripts = sam3d_repo / "data" / "scripts"+ if not (scripts / "lib").is_dir():+ raise SystemExit(+ f"{scripts}/lib is missing — Meta's undistort script imports it but "+ f"the clone does not ship it. See datasets_pipeline/undistort.md.")+ sh([python_bin, "harmony4d/undistort_images.py",+ "--src_dir", raw.resolve(), "--dst_dir", undist.resolve(),+ "--seqs", seqs], cwd=scripts)+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--dataset", required=True, choices=sorted(DATASETS))+ p.add_argument("--split", required=True)+ p.add_argument("--annotation-dir", required=True, type=Path)+ p.add_argument("--work-dir", required=True, type=Path,+ help="scratch space for one archive at a time")+ p.add_argument("--data-root", type=Path, default=Path("data"))+ p.add_argument("--archives", nargs="*", default=None)+ p.add_argument("--max-archives", type=int, default=None)+ p.add_argument("--build-every", type=int, default=1,+ help="build after this many archives (SA-1B has flat, "+ "unpredictable filenames so batching amortises the "+ "parquet scan)")+ p.add_argument("--links-file", type=Path, default=None)+ p.add_argument("--min-free-gb", type=float, default=25.0,+ help="abort before an archive that would leave less than this")+ p.add_argument("--keep-frames", action="store_true",+ help="do not delete the extracted frames after building")+ p.add_argument("--sam3d-repo", type=Path,+ default=Path("/home/madjel/Projects/gitpackages/sam3-biomechanics/sam-3d-body"),+ help="for the undistortion scripts")+ p.add_argument("--undistort-python", default=sys.executable,+ help="interpreter with pycolmap available (the "+ "sam_3d_body_data env, not the training env)")+ p.add_argument("build_args", nargs="*",+ help="extra args for build_split.py, after --")+ args = p.parse_args()++ ds = DATASETS[args.dataset]+ if dataset_for_split(args.split).name != ds.name:+ raise SystemExit(f"split {args.split} does not belong to {ds.name}")++ raw = args.work_dir / "raw"+ undist = args.work_dir / "undistorted"+ image_dir = undist if ds.needs_undistort else raw++ if ds.method == "hf_zips":+ archives = list(args.archives or ds.archives)+ elif ds.method == "links_file":+ if not args.links_file:+ raise SystemExit("--links-file is required for sa1b")+ archives = [ln.split()[0] for ln in args.links_file.read_text().splitlines()+ if len(ln.split()) == 2]+ else:+ raise SystemExit(+ f"{ds.name} arrives as one blob, not per-archive — fetch it with "+ f"fetch_images.py and run build_split.py directly.")+ if args.max_archives:+ archives = archives[: args.max_archives]++ print(f"{ds.name}: {len(archives)} archive(s), work dir {args.work_dir}")+ pending: list[str] = []++ for i, arc in enumerate(archives, 1):+ if free_gb(args.work_dir) < args.min_free_gb:+ print(f"\nSTOP: only {free_gb(args.work_dir):.0f} GB free, below "+ f"--min-free-gb {args.min_free_gb}. "+ f"{len(archives) - i + 1} archive(s) left; re-run to resume.")+ break++ print(f"\n===== [{i}/{len(archives)}] {arc} "+ f"({free_gb(args.work_dir):.0f} GB free) =====")+ fetch = [sys.executable, HERE / "fetch_images.py",+ "--dataset", ds.name, "--dest", raw, "--no-keep-archive"]+ if ds.method == "hf_zips":+ fetch += ["--archives", arc]+ else:+ fetch += ["--links-file", args.links_file, "--max-archives", "1"]+ sh(fetch)++ scene_rel = arc[:-4] if arc.endswith(".zip") else None+ if ds.needs_undistort and scene_rel:+ undistort_harmony4d(raw, undist, scene_rel, args.sam3d_repo,+ args.undistort_python)++ pending.append(arc)+ if len(pending) < args.build_every and i < len(archives):+ continue++ build = [sys.executable, HERE / "build_split.py",+ "--split", args.split,+ "--annotation-dir", args.annotation_dir,+ "--image-dir", image_dir,+ "--data-root", args.data_root]+ if ds.method == "hf_zips" and len(pending) == 1 and scene_rel:+ build += ["--image-prefix", f"{scene_rel}/"]+ build += args.build_args+ sh(build)+ pending.clear()++ if not args.keep_frames:+ for d in (raw, undist):+ if d.exists():+ shutil.rmtree(d, ignore_errors=True)++ print(f"\nstreaming done; {free_gb(args.work_dir):.0f} GB free")+++if __name__ == "__main__":+ main()diff --git a/datasets_pipeline/undistort.md b/datasets_pipeline/undistort.mdnew file mode 100644index 0000000..b0718c5--- /dev/null+++ b/datasets_pipeline/undistort.md@@ -0,0 +1,55 @@+# Undistorting Harmony4D and EgoHumans++Both datasets ship **distorted** frames plus COLMAP intrinsics. The SAM-3D-Body+annotations are expressed against the *undistorted* frames, so skipping this+step silently misaligns every keypoint. EgoExo4D's `sam_3d_body` part is already+undistorted — nothing to do there.++Meta's scripts live in `sam-3d-body/data/scripts/{harmony4d,egohumans}/undistort_images.py`.+Three things block a fresh clone. All three were hit and fixed on 2026-06-30.++### 1. `ModuleNotFoundError: No module named 'lib'`++The script does `from lib.datasets.ego_exo_scene import EgoExoSceneVis`, and+`_init_paths.py` expects `lib/` as a sibling of `harmony4d/` — but the clone+only ships `config/`. Vendor the seven files of `lib/` (`datasets/` + `utils/`)+from <https://github.com/jyuntins/harmony4d> (`main`) into `data/scripts/lib/`.++`stream_archives.py` checks for this directory and fails loudly rather than+part-way through a 45 GB scene.++### 2. `mmcv` / `smplx` missing++`lib/datasets/ego_exo_scene.py` imports both at module level, but only uses+them for SMPL/keypoint *visualisation* — never for undistortion. Make those two+imports lazy (`try/except ImportError → None`) instead of installing two large+packages you will not call. `pycolmap` genuinely is required.++### 3. `--seqs` wants the TAKE, not the scene++The script appends `exo/` to whatever `--seqs` names, so `train/01_hugging` is+wrong (there is no `exo/` there) — it must be `train/01_hugging/001_hugging`.+One scene can hold several takes; pass them comma-separated:++```bash+ls -d $HARMONY4D_DATA_DIR/train/09_karate/*/ \+ | sed 's#.*/train/#train/#; s#/$##' | paste -sd,+```++`stream_archives.py` enumerates the takes of each scene and builds this list for+you, so the manual form is only needed for a one-off.++### Running it++Use the `sam_3d_body_data` env (it has `pycolmap`), not the training env:++```bash+cd $SAM3D_REPO/data/scripts+python harmony4d/undistort_images.py \+ --src_dir $HARMONY4D_DATA_DIR \+ --dst_dir $HARMONY4D_IMG_DIR \+ --seqs train/09_karate/007_karate+```++Validated on `01_hugging/001_hugging` (6.6k frames, 22 exo cameras).+Pass the interpreter to the streamer with `--undistort-python`.diff --git a/datasets_pipeline/verify_split.py b/datasets_pipeline/verify_split.pynew file mode 100755index 0000000..2eba390--- /dev/null+++ b/datasets_pipeline/verify_split.py@@ -0,0 +1,175 @@+#!/usr/bin/env python3+"""Check a built dataset folder, and measure how much hand signal it carries.++Three independent reports:++ structure every npz has its body crop; every hand crop has an npz saying it+ should exist; no orphans either way.+ geometry RMS reprojection error of (joints_3d + cam_trans) onto joints_2d,+ and the source-pixel size of the stored hand crops.+ signal --hand-signal: how far the labels' finger poses sit from a single+ constant mean hand, in millimetres at the fingertips.++The last one is the number that decides whether a hand branch is worth+building on a split. It is an upper bound on what any model could learn about+finger articulation from these labels: if the whole dataset is 7 mm away from+one fixed hand pose, a perfect hand branch buys at most 7 mm. Measured on the+current corpus: coco 6.5, aic 6.9, mpii 7.4, 3dpw 7.5, harmony4d 12.6 mm —+those splits have no finger observations in their source annotations, so the+MHR fits fall back on the prior. Run this on egoexo4d before committing to a+second encoder.++ python datasets_pipeline/verify_split.py --dir data/sam3d_gt_aic+ python datasets_pipeline/verify_split.py --dir data/sam3d_gt_egoexo4d --hand-signal+"""++from __future__ import annotations++import argparse+import random+import sys+from pathlib import Path++import numpy as np++HERE = Path(__file__).resolve().parent+REPO = HERE.parent+sys.path.insert(0, str(REPO))+from tools.parquet_to_npz import reproj_error # noqa: E402++# MHR model-parameter indices of the 54 finger DOFs (27 per hand), derived from+# the rig's parameter_transform: these are the only pose params that move the+# finger joints and nothing else.+FINGER_PARAMS = list(range(68, 122))+RIGHT_HAND_KPS = list(range(21, 42))+LEFT_HAND_KPS = list(range(42, 63))+RIGHT_WRIST, LEFT_WRIST = 41, 62+++def structure(d: Path) -> None:+ anns = sorted((d / "annotations").glob("*.npz"))+ body = {p.stem for p in (d / "body_crops").iterdir()} if (d / "body_crops").exists() else set()+ hands = {p.stem for p in (d / "hands_crops").iterdir()} if (d / "hands_crops").exists() else set()+ stems = {p.stem for p in anns}++ missing_body = sorted(stems - body)[:5]+ orphan_body = sorted(body - stems)[:5]+ want_hands, have_hands, missing_hands = 0, 0, []+ for p in anns:+ a = np.load(p, allow_pickle=False)+ for side in ("l", "r"):+ key = f"hand_valid_{side}"+ if key not in a.files or not bool(a[key]):+ continue+ want_hands += 1+ if f"{p.stem}_{side}" in hands:+ have_hands += 1+ elif len(missing_hands) < 5:+ missing_hands.append(f"{p.stem}_{side}")++ print(f" annotations : {len(anns):,}")+ print(f" body_crops : {len(body):,}"+ f" (missing {len(stems - body):,}, orphan {len(body - stems):,})")+ print(f" hands_crops : {len(hands):,}"+ f" (declared {want_hands:,}, present {have_hands:,})")+ if (d / "original_images").exists():+ n = sum(1 for _ in (d / "original_images").rglob("*") if _.is_file())+ print(f" original_images : {n:,} files")+ if missing_body:+ print(f" !! body crop missing for: {missing_body}")+ if orphan_body:+ print(f" !! body crop with no npz: {orphan_body}")+ if missing_hands:+ print(f" !! hand crop declared but absent: {missing_hands}")+++def geometry(d: Path, n: int, seed: int) -> None:+ anns = sorted((d / "annotations").glob("*.npz"))+ random.Random(seed).shuffle(anns)+ errs, hand_px = [], []+ for p in anns[:n]:+ a = np.load(p, allow_pickle=False)+ errs.append(reproj_error(a["joints_3d"], a["cam_trans"], a["joints_2d"],+ a["cam_focal_length"], a["orig_shape"]))+ for side in ("l", "r"):+ k = f"hand_crop_{side}"+ if k in a.files and np.isfinite(a[k]).all():+ hand_px.append(float(a[k][2] - a[k][0]))+ errs = np.array(errs)+ print(f" reprojection RMS : mean {errs.mean():.3f} px, "+ f"p99 {np.percentile(errs, 99):.3f} px, max {errs.max():.3f} px")+ if hand_px:+ h = np.array(hand_px)+ print(f" hand crop source px: p10 {np.percentile(h, 10):.0f} "+ f"p50 {np.percentile(h, 50):.0f} p90 {np.percentile(h, 90):.0f}"+ f" ({len(h):,} hands in the sample)")+ else:+ print(" hand crop source px: no hand crops in this sample")+++def hand_signal(d: Path, n: int, seed: int, mhr_path: Path, reg_path: Path) -> None:+ import torch++ if not mhr_path.exists() or not reg_path.exists():+ print(f" !! need {mhr_path} and {reg_path}; skipping")+ return+ mhr = torch.jit.load(str(mhr_path), map_location="cpu").eval()+ W = torch.tensor(np.load(reg_path), dtype=torch.float32)++ def kp70(mp: torch.Tensor) -> torch.Tensor:+ cp = torch.cat([mp, torch.zeros(mp.shape[0], 45)], 1)+ jp = mhr.character_torch.model_parameters_to_joint_parameters(cp)+ ss = mhr.character_torch.joint_parameters_to_skeleton_state(jp)[..., :3] / 100.0+ v = torch.stack([ss[..., 0], -ss[..., 1], -ss[..., 2]], -1)+ return torch.einsum("kj,bjc->bkc", W, v)++ anns = sorted((d / "annotations").glob("*.npz"))+ random.Random(seed).shuffle(anns)+ X = torch.tensor(np.stack([np.load(p)["mhr_model_params"] for p in anns[:n]]))+ Xm = X.clone()+ Xm[:, FINGER_PARAMS] = X[:, FINGER_PARAMS].mean(0, keepdim=True)+ with torch.no_grad():+ a, b = kp70(X), kp70(Xm)+ e = torch.cat([+ ((a[:, k] - a[:, [w]]) - (b[:, k] - b[:, [w]])).norm(dim=-1) * 1000+ for k, w in ((RIGHT_HAND_KPS, RIGHT_WRIST), (LEFT_HAND_KPS, LEFT_WRIST))+ ], 1).numpy()+ print(f" finger articulation: {e.mean():.1f} mm mean, "+ f"{np.percentile(e, 90):.1f} mm p90 (over {len(X):,} labels)")+ print(f" finger-param spread: {X[:, FINGER_PARAMS].std(0).mean():.4f} rad mean std")+ verdict = ("worth a hand branch" if e.mean() > 20+ else "marginal — check against your target error"+ if e.mean() > 12 else+ "NOT worth a hand branch: the labels are ~a constant hand pose")+ print(f" verdict : {verdict}")+++def main() -> None:+ p = argparse.ArgumentParser(description=__doc__,+ formatter_class=argparse.RawDescriptionHelpFormatter)+ p.add_argument("--dir", type=Path, required=True, help="data/sam3d_gt_<dataset>")+ p.add_argument("--sample", type=int, default=500)+ p.add_argument("--seed", type=int, default=0)+ p.add_argument("--hand-signal", action="store_true")+ p.add_argument("--mhr", type=Path,+ default=REPO / "checkpoints" / "mhr_model.pt")+ p.add_argument("--regressor", type=Path,+ default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")+ args = p.parse_args()++ if not (args.dir / "annotations").is_dir():+ sys.exit(f"{args.dir} has no annotations/ — is that a built split?")++ print(f"\n== {args.dir} ==")+ print("structure")+ structure(args.dir)+ print("geometry")+ geometry(args.dir, args.sample, args.seed)+ if args.hand_signal:+ print("hand signal")+ hand_signal(args.dir, args.sample, args.seed, args.mhr, args.regressor)+ print()+++if __name__ == "__main__":+ main()