Skip to content
Python

feat: added datasets pipeline

mohamdev8a3b5d9c9da1
diff --git a/datasets_pipeline/README.md b/datasets_pipeline/README.md
new file mode 100644
index 0000000..dc5f556
--- /dev/null
+++ b/datasets_pipeline/README.md
@@ -0,0 +1,282 @@
+# datasets_pipeline
+
+Rebuild any split of the SAM-3D-Body corpus from scratch, keeping the original
+frames this time, into the layout the training code already understands.
+
+```
+data/sam3d_gt_<dataset>/
+├── original_images/ referenced source frames, native relative paths
+├── annotations/ <name>.npz
+├── body_crops/ <name>.png 224 square person crop
+├── hands_crops/ <name>_l.png <name>_r.png
+└── images -> body_crops compat symlink
+```
+
+`body_crops` is **byte-identical** to what `tools/parquet_to_npz.py` writes into
+`images/` — verified md5-for-md5 on 120 AIC crops against the existing
+`data/sam3d_gt_aic/`. `build_split.py` imports that module's crop and annotation
+builders rather than re-implementing them, so the two can never drift. The
+`images` symlink means `SAM3DStudentDataset` finds these folders with **no code
+change**; that path is tested, not assumed.
+
+## The scripts
+
+| script | does |
+|---|---|
+| `fetch_annotations.py` | parquet shards from the gated HF repo |
+| `probe_split.py` | **is this split worth its download?** — from annotations alone |
+| `fetch_images.py` | source frames, per dataset |
+| `build_split.py` | parquet + frames → the folder above |
+| `verify_split.py` | integrity, reprojection, and how much hand signal a split carries |
+
+Plus `plan_disk.py` (cost a split before downloading it), `stream_archives.py`
+(build a dataset larger than your disk), `registry.py` (every URL and gotcha in
+one place), and `undistort.md`.
+
+Annotations are ~1 GB per split; frames are tens to thousands of GB. So the
+order is always: fetch annotations → `probe_split.py` → decide → fetch frames.
+
+## Quick start
+
+```bash
+export ANN_DIR=/scratch/sam3d_ann
+export IMG_ROOT=/scratch/sam3d_img
+
+python datasets_pipeline/fetch_annotations.py --list
+python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR
+python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/coco_train
+
+python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco
+python datasets_pipeline/build_split.py \
+ --split coco_train \
+ --annotation-dir $ANN_DIR/coco_train \
+ --image-dir $IMG_ROOT/coco \
+ --data-root data --validate
+
+python datasets_pipeline/verify_split.py --dir data/sam3d_gt_coco
+```
+
+Everything is resume-safe: re-running skips crops already on disk.
+
+## What each dataset costs, and how to get it
+
+| dataset | images | route | undistort |
+|---|---|---|---|
+| coco | 13.5 GB | `wget` train2014.zip — **only train2014 is referenced** | no |
+| mpii | 12.1 GB | `wget` mpii_human_pose_v1.tar.gz | no |
+| aic | 36.4 GB | OpenDataLab CLI (login) — where this corpus came from before | no |
+| 3dpw | 4.6 GB | licence form → imageFiles.zip | no |
+| harmony4d | 352 GB | HF `Jyun-Ting/Harmony4D`, ungated, per-scene zips | **yes** |
+| egohumans | 107 GB referenced (550 GB advertised) | Google Drive, per-take tar.gz → `gdown --folder` | **yes** |
+| egoexo4d | ~8 GB (procedure) | `egoexo -o $DEST --parts sam_3d_body` (signed licence) | no |
+| sa1b | ~11 TB | terms → `links.txt` → `--links-file` | no |
+
+`fetch_images.py` handles the first two and harmony4d directly. For the rest it
+prints the exact commands and exits 2 rather than pretending.
+
+Measured output sizes, from `plan_disk.py` over the shards already on this
+machine (PNG crops, 20 % hand yield):
+
+```
+split persons frames body hands originals TOTAL
+3dpw_train 22,405 17,410 1.7G 0.7G 4.4G 6.7G
+aic_train 287,391 169,923 21.6G 8.6G 11.9G 42.1G
+harmony4d_train 496,036 248,018 37.2G 14.9G 347.2G 399.3G
+mpii_train 7,135 5,441 0.5G 0.2G 1.1G 1.9G
+```
+
+Two things fall out of that table.
+
+**`original_images` is the whole cost.** Everything except Harmony4D fits
+comfortably: coco + mpii + aic + 3dpw is roughly 60 GB of originals and 40 GB of
+crops. Harmony4D alone is 347 GB of 4K frames and does not fit in 172 GB at any
+setting. Build it with `--originals none` — `body_crops` and `hands_crops` are
+self-contained, and the npz records `image_relpath`, so the originals for any
+subset can be re-fetched later.
+
+**`--crop-format jpg` cuts the crops 3×** (75 KB → 24 KB per 224 crop) and
+decodes ~4× faster, which also speeds up training. It is not the default only
+because PNG keeps byte-identity with the existing `data/` tree. If you are
+rebuilding from scratch anyway, use jpg.
+
+## Datasets you fetch by hand (AIC, 3DPW)
+
+Both are one archive behind a login, so you download and `rsync` them yourself.
+The only thing that matters afterwards is **what `--image-dir` points at**: it
+is the root that the parquet `image` strings are relative to, and that root
+differs per dataset because `image_relpath()` prepends a different prefix.
+
+**AIC** — `image` is a bare `<hash>.jpg`, and the resolver prepends
+`train/images/`. So unpack until this exists:
+
+```
+$AIC_IMG_DIR/train/images/0000252aea98840a550dac9a78c476ecb9f47ffa.jpg
+```
+
+which is exactly the layout already on this machine at
+`~/sam3d_data/images/aic`. Then:
+
+```bash
+python datasets_pipeline/build_split.py --split aic_train \
+ --annotation-dir $ANN_DIR/aic_train --image-dir $AIC_IMG_DIR \
+ --data-root data --validate
+```
+
+`imageFiles.zip`-style archives often unpack one level deeper than you expect;
+if the build reports `missing_image` for everything, the root is wrong by one
+directory. A single `--max-samples 20` run tells you in seconds.
+
+**3DPW** — `image` is `<sequence>/image_00386.jpg` and the resolver prepends
+nothing, so sequence folders sit directly under the root:
+
+```
+$THREEDPW_IMG_DIR/outdoors_climbing_01/image_00386.jpg
+```
+
+You need `imageFiles.zip` only. `sequenceFiles.zip` holds the original SMPL
+ground truth, which this pipeline never reads — the MHR fits come from the
+parquet. Keep it for the 3DPW *benchmark* (`benchmark/eval_3dpw.py`), not for
+training data.
+
+Same command, `--split 3dpw_train`. Both are small enough for
+`--originals copy` (4.4 GB and 11.9 GB of referenced frames).
+
+## EgoHumans
+
+Google Drive, one `.tar.gz` per take, and the untar needs
+`--strip-components=8` or you get eight nested empty parents:
+
+```bash
+pip install gdown
+gdown --folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR -O $EGOHUMANS_DATA_DIR
+cd $EGOHUMANS_DATA_DIR/01_tagging && tar xzf 001_tagging.tar.gz --strip-components=8
+```
+
+The dataset advertises 550 GB, but `egohumans_train` references only **89 takes
+across 7 sequences** — 272k persons over 268k 4K frames, ~107 GB of referenced
+frames. Get the exact shopping list before downloading anything:
+
+```bash
+python datasets_pipeline/probe_split.py \
+ --annotation-dir $ANN_DIR/egohumans_train --list-groups 2
+```
+
+Only the `exo/` cameras are referenced; `ego/`, `colmap/` and `processed_data/`
+inside each tarball are dead weight you can delete after extraction. Frames
+must be undistorted (`undistort.md`), then build per take with
+`--image-prefix 01_tagging/001_tagging/`, or let `stream_archives.py` do it.
+
+Worth knowing before you spend the 107 GB: it is good body data (multi-view,
+272k persons) but useless for hands — sports at distance, so only 21.5% of
+hands clear 64 px and the finger articulation is 6.3 mm, the lowest in the
+corpus.
+
+## SA-1B
+
+Accept the terms at <https://ai.meta.com/datasets/segment-anything-downloads/>
+to get `links.txt` (one `sa_NNNNNN.tar <signed-url>` per line), then:
+
+```bash
+python datasets_pipeline/stream_archives.py --dataset sa1b \
+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \
+ --work-dir /scratch/sa1b --data-root data \
+ --links-file ~/links.txt --max-archives 50 --build-every 10 \
+ -- --originals none --crop-format jpg
+```
+
+Any prefix of the tar list is a valid dataset, so you can stop whenever. But
+the measured economics are poor, and I would not start here:
+
+- References are spread **uniformly over all 1000 tars** — 994 of 1000 id
+ blocks are hit by just 3% of the annotations — so there is no dense subset to
+ cherry-pick.
+- Only **11.4%** of SA-1B is referenced. Each ~10 GB tar yields ~1,270 useful
+ frames ≈ 3,400 crops, i.e. **~2.3 MB of download per crop** against AIC's
+ 0.13 MB. The full 3.4M crops cost ~11 TB.
+- It is the one split with real visibility flags: a mean of **44 of 70**
+ keypoints are literal `(0, 0)` placeholders, and only **0.3%** of hands are
+ fully visible.
+
+That last point is a trap for any converter. `tools/parquet_to_npz.py` drops the
+visibility column (`joints_2d = kp2[:, :2]`), so a naive SA-1B build writes
+those placeholders as if they were observations at the image corner — poisoning
+the 2D loss and blowing every hand box out to the frame edge. `build_split.py`
+stores `joints_2d_vis` / `joints_3d_conf` and refuses to cut a hand crop unless
+all 21 of its keypoints are visible. Every other split is 100% visible, so this
+only bites on SA-1B.
+
+## Datasets that do not fit: `stream_archives.py`
+
+Never holds more than one archive: download → extract → undistort → build →
+delete. Any prefix of the archive list is a valid dataset, so you can stop when
+the disk or the clock says so and resume later.
+
+```bash
+python datasets_pipeline/stream_archives.py --dataset harmony4d \
+ --split harmony4d_train --annotation-dir $ANN_DIR/harmony4d_train \
+ --work-dir /scratch/h4d --data-root data \
+ --undistort-python ~/anaconda3/envs/sam_3d_body_data/bin/python \
+ -- --originals none --crop-format jpg
+
+python datasets_pipeline/stream_archives.py --dataset sa1b \
+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \
+ --work-dir /scratch/sa1b --data-root data \
+ --links-file ~/sa1b_links.txt --max-archives 200 --build-every 20 \
+ -- --originals none
+```
+
+`--min-free-gb` (default 25) aborts before an archive that would fill the disk.
+Read `undistort.md` before the first Harmony4D run — Meta's script has three
+blockers that a fresh clone hits.
+
+## Hand crops
+
+The hand box is free: `joints_2d` is already in full-frame pixel coordinates, so
+the 21 keypoints of each hand give a tight box with no extra annotation.
+
+- `--hand-min-px 64` (default) is SAM3D's own validity gate, applied to the raw
+ keypoint span. On AIC this keeps ~17 % of hands, at a median 145 source pixels
+ upsampled 1.5× to 224.
+- `--hand-expand 2.0` stores margin around the hand so you can jitter the box
+ *inside* the stored crop at train time. That is what makes a model tolerant of
+ a real detector's boxes, and it is why cropping dynamically from originals
+ buys nothing.
+- Crops are stored **unflipped**. SAM3D trains one right-hand model and mirrors
+ the left; flipping a loaded crop is a free numpy slice, so baking the mirror
+ into disk would only remove a choice.
+- `hand_valid_l` / `hand_valid_r` in the npz say whether a crop exists;
+ `hand_box_*` (tight) and `hand_crop_*` (the square actually written) let you
+ re-derive any tighter crop without touching the originals.
+
+## Which split to spend money on
+
+`probe_split.py` measured every split from its annotations. *Articulation* is
+how far the labels' finger poses sit from ONE constant hand pose, in mm at the
+fingertips — an upper bound on what a hand branch could learn from them.
+*≥96 px* is the share of hands whose crop would be at least 96 source pixels.
+
+| split | persons | articulation | hands ≥96 px | download |
+|---|---|---|---|---|
+| **egoexo4d_procedure** | ~350k | **16.1 mm** | **85.5%** | licence + CLI |
+| egoexo4d_physical | ~700k | 14.5 mm | 19.3% | licence + CLI |
+| harmony4d | 496k | 12.6 mm | 3.4% | 352 GB |
+| sa1b | 3.4M | 8.5 mm | 0.3% visible at all | ~11 TB |
+| 3dpw | 22k | 7.5 mm | 71.4% | 4.6 GB |
+| mpii | 7k | 7.4 mm | 11.5% | 12.1 GB |
+| aic | 287k | 6.9 mm | 16.6% | 36.4 GB |
+| coco | ~42k | 6.5 mm | 3.0% | 13.5 GB |
+| egohumans | 272k | 6.3 mm | 6.4% | 107 GB |
+
+COCO, MPII and AIC have no finger keypoints and 3DPW is SMPL, so the official
+MHR fits fall back on the prior — a hand branch trained on them learns a
+constant. **EgoExo4D procedure** (cooking, bike repair, piano — hands doing
+things close to the camera) is the only split in the corpus where both terms
+are good at once: 2.5× the articulation of COCO/AIC and two thirds of hands
+above 128 px. It is also cheap, ~8 GB of referenced frames for 350k persons.
+That is where the licence you are waiting on pays off.
+
+Re-run the check on your own build at any time:
+
+```bash
+python datasets_pipeline/verify_split.py --dir data/sam3d_gt_egoexo4d --hand-signal
+```
diff --git a/datasets_pipeline/build_split.py b/datasets_pipeline/build_split.py
new file mode 100755
index 0000000..e359360
--- /dev/null
+++ b/datasets_pipeline/build_split.py
@@ -0,0 +1,331 @@
+#!/usr/bin/env python3
+"""Turn one parquet split + its source frames into a trainable dataset folder.
+
+Output layout, per source dataset (train and test merge into one folder, which
+is what data/ already does and what the training script's discovery expects):
+
+ <data_root>/sam3d_gt_<dataset>/
+ original_images/ the referenced source frames, native relative paths
+ annotations/ <name>.npz
+ body_crops/ <name>.png 224 square person crop
+ hands_crops/ <name>_l.png <name>_r.png
+ images -> body_crops compat symlink, see below
+
+`body_crops` is byte-identical to what `tools/parquet_to_npz.py` writes into
+`images/`: this script imports that module's crop and annotation builders
+rather than re-implementing them, so a folder built here and a folder built
+there are interchangeable. The `images` symlink means the current
+`SAM3DStudentDataset._find_dataset_dirs` finds the folder with no code change.
+
+The npz keeps every key the training script already reads and adds five that
+it ignores today and that a hand branch will need:
+
+ image_relpath (str) exact source path — the existing `<dataset>_<stem>`
+ naming flattens "/" to "_" and is NOT invertible for
+ harmony4d / egohumans / egoexo4d / 3dpw
+ joints_2d_vis (70,) per-keypoint visibility, dropped by parquet_to_npz.
+ It is all-ones on coco/mpii/aic/3dpw/harmony4d/
+ egohumans/egoexo4d, but on SA-1B a mean of 44 of 70
+ keypoints are exactly (0, 0) placeholders — writing
+ those as if they were observations poisons the 2D
+ loss and inflates every hand box.
+ joints_3d_conf (70,) the matching confidence column of keypoints_3d
+ hand_box_l/_r (4,) tight xyxy over the 21 hand keypoints, full-frame px
+ hand_crop_l/_r (4,) the square crop actually written (NaN if skipped)
+ hand_valid_l/_r () bool: was a crop written
+
+Hand crops are stored UNFLIPPED. SAM3D trains a single right-hand model and
+mirrors the left at inference; flipping a loaded crop is a free numpy slice, so
+baking the mirror into disk would only remove a choice.
+
+ python datasets_pipeline/build_split.py \
+ --split coco_train --annotation-dir $ANN_DIR/coco_train \
+ --image-dir $COCO_IMG_DIR --data-root data
+"""
+
+from __future__ import annotations
+
+import argparse
+import os
+import shutil
+import sys
+from pathlib import Path
+
+import cv2
+import numpy as np
+import pandas as pd
+from tqdm import tqdm
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+sys.path.insert(0, str(HERE.parent))
+
+from registry import dataset_for_split, image_relpath, output_dir_for_split # noqa: E402
+from tools.parquet_to_npz import ( # noqa: E402
+ build_annotation, get_square_crop_padded, reproj_error,
+)
+
+# MHR70 keypoint blocks. Index 41 is right_wrist, 62 is left_wrist; each hand
+# block is its 20 finger joints plus that wrist. Taken from JOINT_NAMES in
+# instanthmr_distill_train/train_distill_mhr_only.py.
+RIGHT_HAND_KPS = list(range(21, 42))
+LEFT_HAND_KPS = list(range(42, 63))
+
+
+def hand_boxes(joints_2d: np.ndarray,
+ vis: np.ndarray | None = None) -> dict[str, np.ndarray | None]:
+ """Tight xyxy box over each hand's 21 keypoints, in full-frame pixels.
+
+ Returns None for a hand that is not fully visible. A partially visible hand
+ cannot give a usable box: the invisible keypoints are stored as (0, 0), so
+ including even one drags the box to the image corner. On SA-1B that is the
+ normal case — only 0.3% of hands have all 21 keypoints marked visible.
+ """
+ out: dict[str, np.ndarray | None] = {}
+ for side, idx in (("l", LEFT_HAND_KPS), ("r", RIGHT_HAND_KPS)):
+ if vis is not None and not (vis[idx] > 0).all():
+ out[side] = None
+ continue
+ p = joints_2d[idx]
+ out[side] = np.array([p[:, 0].min(), p[:, 1].min(),
+ p[:, 0].max(), p[:, 1].max()], dtype=np.float32)
+ return out
+
+
+def square_box(box: np.ndarray, expand: float) -> np.ndarray:
+ cx, cy = (box[0] + box[2]) / 2.0, (box[1] + box[3]) / 2.0
+ half = max(box[2] - box[0], box[3] - box[1]) * expand / 2.0
+ return np.array([cx - half, cy - half, cx + half, cy + half], dtype=np.float32)
+
+
+def enough_inside(sq: np.ndarray, h: int, w: int, min_frac: float) -> bool:
+ """Reject hands that are mostly outside the frame — the crop would be padding."""
+ ix1, iy1 = max(0.0, sq[0]), max(0.0, sq[1])
+ ix2, iy2 = min(float(w), sq[2]), min(float(h), sq[3])
+ if ix2 <= ix1 or iy2 <= iy1:
+ return False
+ area = (sq[2] - sq[0]) * (sq[3] - sq[1])
+ return ((ix2 - ix1) * (iy2 - iy1)) / max(area, 1e-6) >= min_frac
+
+
+def write_crop(img_rgb, box, size, path: Path) -> None:
+ crop, _ = get_square_crop_padded(img_rgb, box, expand=1.0)
+ crop = cv2.resize(crop, (size, size), interpolation=cv2.INTER_LINEAR)
+ cv2.imwrite(str(path), cv2.cvtColor(crop, cv2.COLOR_RGB2BGR))
+
+
+def place_original(src: Path, dst: Path, mode: str) -> None:
+ if mode == "none" or dst.exists():
+ return
+ dst.parent.mkdir(parents=True, exist_ok=True)
+ if mode == "copy":
+ shutil.copy2(src, dst)
+ elif mode == "link":
+ os.symlink(os.path.relpath(src.resolve(), dst.parent), dst)
+ elif mode == "hardlink":
+ try:
+ os.link(src, dst)
+ except OSError:
+ shutil.copy2(src, dst)
+
+
+def parse_args() -> argparse.Namespace:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--split", required=True, help="e.g. coco_train, harmony4d_train")
+ p.add_argument("--annotation-dir", required=True, type=Path)
+ p.add_argument("--image-dir", required=True, type=Path,
+ help="root the parquet `image` paths are relative to")
+ p.add_argument("--data-root", type=Path, default=Path("data"))
+ p.add_argument("--output-dir", type=Path, default=None,
+ help="override the sam3d_gt_<dataset> destination")
+
+ p.add_argument("--body-size", type=int, default=224)
+ p.add_argument("--crop-format", choices=("png", "jpg"), default="png",
+ help="png keeps body_crops byte-identical to the existing "
+ "data/ tree; jpg is ~5x smaller and ~4x faster to decode")
+ p.add_argument("--jpg-quality", type=int, default=95)
+
+ p.add_argument("--hands", action=argparse.BooleanOptionalAction, default=True)
+ p.add_argument("--hand-size", type=int, default=224)
+ p.add_argument("--hand-expand", type=float, default=2.0,
+ help="square hand box = 21-keypoint span x this. 2.0 leaves "
+ "room to jitter the box inside the stored crop at "
+ "train time, which is what makes the model tolerant of "
+ "a real detector's boxes.")
+ p.add_argument("--hand-min-px", type=float, default=64.0,
+ help="skip hands whose SPAN (before --hand-expand) is smaller "
+ "than this in source pixels. 64 is SAM3D's own gate.")
+ p.add_argument("--hand-min-inside", type=float, default=0.6,
+ help="skip hands whose square crop is less than this fraction "
+ "inside the frame")
+
+ p.add_argument("--originals", choices=("copy", "link", "hardlink", "none"),
+ default="copy",
+ help="copy: self-contained, prunes unreferenced source images. "
+ "link/hardlink: free, but the source root must survive.")
+
+ p.add_argument("--image-prefix", default=None,
+ help="only rows whose `image` starts with this — the hook "
+ "stream_archives.py uses to build one archive at a time")
+ p.add_argument("--max-samples", type=int, default=None)
+ p.add_argument("--keep-invalid", action="store_true",
+ help="keep rows with mhr_valid == False (dropped by default)")
+ p.add_argument("--validate", action="store_true",
+ help="assert < --reproj-tol px reprojection error per person")
+ p.add_argument("--reproj-tol", type=float, default=1.0)
+ return p.parse_args()
+
+
+def main() -> None:
+ args = parse_args()
+ ds = dataset_for_split(args.split)
+ out_dir = args.output_dir or output_dir_for_split(args.split, args.data_root)
+
+ dirs = {k: out_dir / k for k in
+ ("original_images", "annotations", "body_crops", "hands_crops")}
+ for k, d in dirs.items():
+ if k == "original_images" and args.originals == "none":
+ continue
+ if k == "hands_crops" and not args.hands:
+ continue
+ d.mkdir(parents=True, exist_ok=True)
+
+ # Let the unmodified training script discover this folder.
+ compat = out_dir / "images"
+ if not compat.exists():
+ os.symlink("body_crops", compat)
+
+ parquets = sorted(args.annotation_dir.glob("*.parquet"))
+ if not parquets:
+ sys.exit(f"no parquet files in {args.annotation_dir}")
+
+ ext = args.crop_format
+ imwrite_params = ([int(cv2.IMWRITE_JPEG_QUALITY), args.jpg_quality]
+ if ext == "jpg" else [])
+ existing = {p.stem for p in dirs["annotations"].glob("*.npz")}
+ total = len(existing)
+ st = {k: 0 for k in ("crops", "hands_l", "hands_r", "hand_too_small",
+ "hand_out_of_frame", "hand_occluded", "skipped_existing",
+ "skipped_invalid", "skipped_prefix", "missing_image",
+ "failed")}
+
+ pbar = tqdm(parquets, desc=f"{args.split} shards", unit="shard")
+ for pq_path in pbar:
+ df = pd.read_parquet(pq_path)
+ img_cache: dict[str, np.ndarray | None] = {}
+
+ for _, row in df.iterrows():
+ dataset, image = row["dataset"], row["image"]
+ if args.image_prefix and not image.startswith(args.image_prefix):
+ st["skipped_prefix"] += 1
+ continue
+
+ stem = (os.path.splitext(image)[0]
+ .replace("/", "_").replace(os.sep, "_").replace(" ", "_"))
+ name = f"{dataset}_{stem}_p{int(row['person_id'])}"
+ if name in existing:
+ st["skipped_existing"] += 1
+ continue
+ if not args.keep_invalid and not bool(row["mhr_valid"]):
+ st["skipped_invalid"] += 1
+ continue
+
+ relpath = image_relpath(dataset, image)
+ if image not in img_cache:
+ bgr = cv2.imread(str(args.image_dir / relpath))
+ img_cache[image] = (cv2.cvtColor(bgr, cv2.COLOR_BGR2RGB)
+ if bgr is not None else None)
+ img_rgb = img_cache[image]
+ if img_rgb is None:
+ st["missing_image"] += 1
+ continue
+
+ h, w = img_rgb.shape[:2]
+ try:
+ ann = build_annotation(row, h, w)
+ if args.validate:
+ err = reproj_error(ann["joints_3d"], ann["cam_trans"],
+ ann["joints_2d"], ann["cam_focal_length"],
+ ann["orig_shape"])
+ if err > args.reproj_tol:
+ raise ValueError(f"reproj err {err:.2f}px > {args.reproj_tol}px")
+
+ body_crop, sq_bbox = get_square_crop_padded(img_rgb, ann["bbox"],
+ expand=1.2)
+ body_crop = cv2.resize(body_crop, (args.body_size, args.body_size),
+ interpolation=cv2.INTER_LINEAR)
+ cv2.imwrite(str(dirs["body_crops"] / f"{name}.{ext}"),
+ cv2.cvtColor(body_crop, cv2.COLOR_RGB2BGR),
+ imwrite_params)
+
+ ann["bbox_square"] = sq_bbox
+ ann["image_relpath"] = relpath
+ ann["dataset"] = dataset
+ kp2 = np.asarray(row["keypoints_2d"].tolist(), dtype=np.float32)
+ kp3 = np.asarray(row["keypoints_3d"].tolist(), dtype=np.float32)
+ vis = kp2.reshape(-1, 3)[:, 2].copy()
+ ann["joints_2d_vis"] = vis
+ ann["joints_3d_conf"] = kp3.reshape(-1, 4)[:, 3].copy()
+
+ boxes = hand_boxes(ann["joints_2d"], vis)
+ for side in ("l", "r"):
+ tight = boxes[side]
+ nan4 = np.full(4, np.nan, dtype=np.float32)
+ ann[f"hand_box_{side}"] = tight if tight is not None else nan4
+ ann[f"hand_crop_{side}"] = nan4
+ ann[f"hand_valid_{side}"] = np.array(False)
+ if tight is None:
+ st["hand_occluded"] += 1
+ continue
+ span = max(tight[2] - tight[0], tight[3] - tight[1])
+ sq = square_box(tight, args.hand_expand)
+ if not args.hands:
+ continue
+ if span < args.hand_min_px:
+ st["hand_too_small"] += 1
+ continue
+ if not enough_inside(sq, h, w, args.hand_min_inside):
+ st["hand_out_of_frame"] += 1
+ continue
+ write_crop(img_rgb, sq, args.hand_size,
+ dirs["hands_crops"] / f"{name}_{side}.{ext}")
+ ann[f"hand_crop_{side}"] = sq
+ ann[f"hand_valid_{side}"] = np.array(True)
+ st[f"hands_{side}"] += 1
+
+ np.savez(dirs["annotations"] / f"{name}.npz", **ann)
+ if args.originals != "none":
+ place_original(args.image_dir / relpath,
+ dirs["original_images"] / relpath,
+ args.originals)
+ except Exception as e: # noqa: BLE001
+ st["failed"] += 1
+ tqdm.write(f"[fail] {name}: {e}")
+ continue
+
+ existing.add(name)
+ st["crops"] += 1
+ total += 1
+ if args.max_samples and total >= args.max_samples:
+ pbar.close()
+ report(st, total, out_dir)
+ return
+
+ pbar.set_postfix(crops=st["crops"], hands=st["hands_l"] + st["hands_r"],
+ missing=st["missing_image"])
+
+ report(st, total, out_dir)
+
+
+def report(st: dict, total: int, out_dir: Path) -> None:
+ print("=" * 62)
+ print(f"BUILD COMPLETE -> {out_dir}")
+ for k, v in st.items():
+ print(f" {k:20s}: {v:,}")
+ print(f" {'total on disk':20s}: {total:,}")
+ print("=" * 62)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/fetch_annotations.py b/datasets_pipeline/fetch_annotations.py
new file mode 100755
index 0000000..ea27311
--- /dev/null
+++ b/datasets_pipeline/fetch_annotations.py
@@ -0,0 +1,110 @@
+#!/usr/bin/env python3
+"""Download parquet annotation shards from facebook/sam-3d-body-dataset.
+
+The gated HF repo needs an accepted licence and a token (`hf auth login`, or
+HF_TOKEN in the environment). Downloads are resumable and skip shards already
+on disk, so re-running after a killed job is free.
+
+ # everything for one split
+ python datasets_pipeline/fetch_annotations.py --splits coco_train --out $ANN_DIR
+
+ # a slice, for a cheap first pass on a huge split
+ python datasets_pipeline/fetch_annotations.py --splits sa1b_train --out $ANN_DIR \
+ --max-shards 50
+
+ # just tell me what is available and how big it is
+ python datasets_pipeline/fetch_annotations.py --list
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from pathlib import Path
+
+from huggingface_hub import HfApi, hf_hub_download
+from tqdm import tqdm
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from registry import HF_ANNOTATION_REPO, SPLIT_TO_DATASET # noqa: E402
+
+
+def shard_index(api: HfApi) -> dict[str, list[str]]:
+ """split -> sorted list of repo-relative parquet paths."""
+ out: dict[str, list[str]] = {}
+ for f in api.list_repo_files(HF_ANNOTATION_REPO, repo_type="dataset"):
+ if not f.endswith(".parquet"):
+ continue
+ parts = f.split("/")
+ if len(parts) < 3:
+ continue
+ out.setdefault(parts[1], []).append(f)
+ return {k: sorted(v) for k, v in sorted(out.items())}
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--splits", nargs="+", help="e.g. coco_train harmony4d_train")
+ p.add_argument("--out", type=Path, help="destination root; one folder per split")
+ p.add_argument("--max-shards", type=int, default=None,
+ help="take only the first N shards of each split")
+ p.add_argument("--workers", type=int, default=8)
+ p.add_argument("--list", action="store_true", help="print available splits and exit")
+ args = p.parse_args()
+
+ api = HfApi()
+ index = shard_index(api)
+
+ if args.list:
+ print(f"{'split':26s} {'shards':>7s} ~persons")
+ for split, files in index.items():
+ # ~1,750 rows/shard, measured across coco/aic/sa1b/egoexo4d/egohumans.
+ print(f"{split:26s} {len(files):7d} {len(files) * 1750:>9,d}")
+ return
+
+ if not args.splits or not args.out:
+ p.error("--splits and --out are required unless --list is given")
+
+ unknown = [s for s in args.splits if s not in index]
+ if unknown:
+ p.error(f"unknown split(s): {unknown}. Run --list for the full set.")
+ for s in args.splits:
+ if s not in SPLIT_TO_DATASET:
+ print(f"[warn] {s} has no entry in registry.py; build_split.py will "
+ f"not know where its images live.", file=sys.stderr)
+
+ for split in args.splits:
+ files = index[split]
+ if args.max_shards:
+ files = files[: args.max_shards]
+ dest = args.out / split
+ dest.mkdir(parents=True, exist_ok=True)
+ todo = [f for f in files if not (dest / Path(f).name).exists()]
+ print(f"{split}: {len(files)} shards, {len(todo)} to download -> {dest}")
+ if not todo:
+ continue
+
+ def grab(repo_file: str) -> None:
+ hf_hub_download(HF_ANNOTATION_REPO, repo_file, repo_type="dataset",
+ local_dir=str(dest.parent / ".hf_staging"))
+ src = dest.parent / ".hf_staging" / repo_file
+ src.replace(dest / src.name)
+
+ with ThreadPoolExecutor(max_workers=args.workers) as ex:
+ futs = {ex.submit(grab, f): f for f in todo}
+ for fut in tqdm(as_completed(futs), total=len(futs), unit="shard"):
+ try:
+ fut.result()
+ except Exception as e: # noqa: BLE001
+ tqdm.write(f"[fail] {futs[fut]}: {e}")
+
+ staging = args.out / ".hf_staging"
+ if staging.exists():
+ import shutil
+ shutil.rmtree(staging, ignore_errors=True)
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/fetch_images.py b/datasets_pipeline/fetch_images.py
new file mode 100755
index 0000000..d95dec8
--- /dev/null
+++ b/datasets_pipeline/fetch_images.py
@@ -0,0 +1,183 @@
+#!/usr/bin/env python3
+"""Download and extract the source frames for one SAM-3D-Body dataset.
+
+Every dataset that CAN be fetched with a plain URL is fetched with a plain URL
+(`wget -c`, so a killed job resumes). The four that cannot — AIC, 3DPW,
+EgoExo4D, SA-1B — need a login or a signed licence, and this script prints the
+exact commands instead of pretending.
+
+ python datasets_pipeline/fetch_images.py --dataset coco --dest $IMG_ROOT/coco
+ python datasets_pipeline/fetch_images.py --dataset mpii --dest $IMG_ROOT/mpii
+ python datasets_pipeline/fetch_images.py --dataset harmony4d --dest $IMG_ROOT/harmony4d_raw \
+ --archives train/09_karate.zip --no-keep-archive
+ python datasets_pipeline/fetch_images.py --dataset sa1b --dest $IMG_ROOT/sa1b \
+ --links-file ~/sa1b_links.txt --max-archives 20
+
+`--dry-run` prints the plan and the byte counts without downloading anything.
+"""
+
+from __future__ import annotations
+
+import argparse
+import shutil
+import subprocess
+import sys
+import tarfile
+import zipfile
+from pathlib import Path
+
+sys.path.insert(0, str(Path(__file__).resolve().parent))
+from registry import DATASETS # noqa: E402
+
+HF_RESOLVE = "https://huggingface.co/datasets/{repo}/resolve/main/{path}"
+
+
+def run(cmd: list[str]) -> None:
+ print("+", " ".join(cmd), flush=True)
+ subprocess.run(cmd, check=True)
+
+
+def wget(url: str, dest: Path) -> Path:
+ dest.mkdir(parents=True, exist_ok=True)
+ out = dest / url.split("/")[-1].split("?")[0]
+ run(["wget", "-c", "-O", str(out), url])
+ return out
+
+
+def extract(archive: Path, into: Path, strip_top: bool = False) -> None:
+ into.mkdir(parents=True, exist_ok=True)
+ print(f"extracting {archive.name} -> {into}", flush=True)
+ if archive.suffix == ".zip":
+ with zipfile.ZipFile(archive) as z:
+ z.extractall(into)
+ elif archive.name.endswith((".tar.gz", ".tgz", ".tar")):
+ mode = "r:gz" if archive.name.endswith((".tar.gz", ".tgz")) else "r"
+ with tarfile.open(archive, mode) as t:
+ t.extractall(into, filter="data")
+ else:
+ raise SystemExit(f"don't know how to extract {archive.name}")
+
+
+def manual(ds_name: str, notes: str) -> None:
+ print("=" * 70)
+ print(f"{ds_name}: needs a credential or a signed licence — cannot wget it.")
+ print("=" * 70)
+ print(notes)
+ print()
+ print("Once the frames are on disk, point build_split.py --image-dir at the "
+ "root that the parquet `image` paths are relative to and carry on.")
+ sys.exit(2)
+
+
+def fetch_direct(ds, dest: Path, keep: bool, dry: bool) -> None:
+ for url in ds.urls:
+ if dry:
+ print(f"[dry-run] wget {url}")
+ continue
+ arc = wget(url, dest)
+ extract(arc, dest)
+ if not keep:
+ arc.unlink()
+
+
+def fetch_hf_zips(ds, dest: Path, wanted: list[str] | None,
+ max_archives: int | None, keep: bool, dry: bool) -> None:
+ archives = list(wanted or ds.archives)
+ unknown = [a for a in archives if a not in ds.archives]
+ if unknown:
+ raise SystemExit(f"unknown archive(s) {unknown}\nknown:\n "
+ + "\n ".join(ds.archives))
+ if max_archives:
+ archives = archives[:max_archives]
+ for rel in archives:
+ url = HF_RESOLVE.format(repo=ds.hf_repo, path=rel)
+ if dry:
+ print(f"[dry-run] wget {url}")
+ continue
+ # Harmony4D zips already carry the train/<scene>/ prefix internally.
+ arc = wget(url, dest / "_archives")
+ extract(arc, dest)
+ if not keep:
+ arc.unlink()
+
+
+def fetch_links_file(ds, dest: Path, links_file: Path,
+ max_archives: int | None, keep: bool, dry: bool) -> None:
+ """SA-1B: Meta hands you a two-column `name<TAB>signed-url` file."""
+ lines = [ln.strip() for ln in links_file.read_text().splitlines() if ln.strip()]
+ entries = []
+ for ln in lines:
+ parts = ln.split()
+ if len(parts) == 2 and parts[1].startswith("http"):
+ entries.append((parts[0], parts[1]))
+ if not entries:
+ raise SystemExit(f"no `name url` pairs found in {links_file}")
+ if max_archives:
+ entries = entries[:max_archives]
+ print(f"{len(entries)} SA-1B shards to fetch")
+ for name, url in entries:
+ if dry:
+ print(f"[dry-run] {name}")
+ continue
+ arc = dest / "_archives" / name
+ arc.parent.mkdir(parents=True, exist_ok=True)
+ run(["wget", "-c", "-O", str(arc), url])
+ # Each tar holds sa_*.jpg plus per-image json masks we do not need.
+ with tarfile.open(arc) as t:
+ members = [m for m in t.getmembers() if m.name.endswith(".jpg")]
+ t.extractall(dest, members=members, filter="data")
+ if not keep:
+ arc.unlink()
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--dataset", required=True, choices=sorted(DATASETS))
+ p.add_argument("--dest", type=Path, required=True)
+ p.add_argument("--archives", nargs="*", default=None,
+ help="subset of archives (hf_zips only); default is all")
+ p.add_argument("--max-archives", type=int, default=None)
+ p.add_argument("--links-file", type=Path, default=None, help="SA-1B links.txt")
+ p.add_argument("--keep-archive", action=argparse.BooleanOptionalAction,
+ default=False, help="keep the downloaded zip/tar after extracting")
+ p.add_argument("--dry-run", action="store_true")
+ p.add_argument("--list-archives", action="store_true")
+ args = p.parse_args()
+
+ ds = DATASETS[args.dataset]
+ if args.list_archives:
+ for a in ds.archives or ("(single blob)",):
+ print(a)
+ return
+
+ print(f"{ds.name}: method={ds.method} ~{ds.approx_gb or '?'} GB "
+ f"undistort={'YES' if ds.needs_undistort else 'no'}")
+ if ds.notes:
+ print(" " + ds.notes.replace("\n", "\n "))
+ print()
+
+ if ds.method == "direct":
+ fetch_direct(ds, args.dest, args.keep_archive, args.dry_run)
+ elif ds.method == "hf_zips":
+ fetch_hf_zips(ds, args.dest, args.archives, args.max_archives,
+ args.keep_archive, args.dry_run)
+ elif ds.method == "links_file":
+ if not args.links_file:
+ manual(ds.name, ds.notes)
+ fetch_links_file(ds, args.dest, args.links_file, args.max_archives,
+ args.keep_archive, args.dry_run)
+ else:
+ manual(ds.name, ds.notes)
+
+ if ds.needs_undistort:
+ print()
+ print("!! These frames are DISTORTED. Run the undistortion step before "
+ "build_split.py — see datasets_pipeline/undistort.md.")
+ free = (shutil.disk_usage(args.dest).free / 1e9
+ if args.dest.exists() else shutil.disk_usage(".").free / 1e9)
+ print(f"done -> {args.dest} ({free:.0f} GB free)")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/plan_disk.py b/datasets_pipeline/plan_disk.py
new file mode 100755
index 0000000..9d64db0
--- /dev/null
+++ b/datasets_pipeline/plan_disk.py
@@ -0,0 +1,91 @@
+#!/usr/bin/env python3
+"""Cost a split before you download it.
+
+Reads only the `image` and `mhr_valid` columns of the parquets — cheap, no
+frames needed — and reports how many person crops and unique source frames a
+split will produce, plus what each output folder will weigh.
+
+ python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/aic_train
+ python datasets_pipeline/plan_disk.py --annotation-dir $ANN_DIR/* --brief
+
+Per-file sizes are measured on the existing data/ tree, not assumed:
+ 224 PNG body crop ~75 KB, 224 JPEG q95 ~24 KB.
+Source-frame sizes come from registry.py's measured download sizes divided by
+the frame count, except where the dataset is a known fixed resolution.
+"""
+
+from __future__ import annotations
+
+import argparse
+from pathlib import Path
+
+import pyarrow.parquet as pq
+
+# KB per stored file, measured on data/sam3d_gt_*.
+CROP_KB = {"png": 75.0, "jpg": 24.0}
+# KB per source frame, measured (aic/coco/mpii/3dpw) or estimated from the
+# 4K zips (harmony4d).
+FRAME_KB = {"coco": 160.0, "mpii": 210.0, "aic": 70.0, "3dpw": 250.0,
+ "harmony4d": 1400.0, "egohumans": 400.0, "egoexo4d": 400.0,
+ "sa1b": 700.0}
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--annotation-dir", nargs="+", type=Path, required=True)
+ p.add_argument("--crop-format", choices=("png", "jpg"), default="png")
+ p.add_argument("--hand-yield", type=float, default=0.20,
+ help="fraction of the 2 hands per person that clear "
+ "--hand-min-px; 0.20 is what AIC measures at 64 px")
+ p.add_argument("--brief", action="store_true")
+ args = p.parse_args()
+
+ ckb = CROP_KB[args.crop_format]
+ tot = {"rows": 0, "frames": 0, "body": 0.0, "hand": 0.0, "orig": 0.0}
+ hdr = (f"{'split':26s} {'persons':>10s} {'frames':>10s} "
+ f"{'body':>8s} {'hands':>8s} {'originals':>10s} {'TOTAL':>9s}")
+ print(hdr)
+ print("-" * len(hdr))
+
+ for d in args.annotation_dir:
+ shards = sorted(d.glob("*.parquet"))
+ if not shards:
+ continue
+ split = d.name
+ dataset = split.split("_")[0]
+ rows = 0
+ frames: set[str] = set()
+ for s in shards:
+ t = pq.read_table(s, columns=["image", "mhr_valid"])
+ im = t.column("image").to_pylist()
+ mv = t.column("mhr_valid").to_pylist()
+ rows += sum(mv)
+ frames.update(i for i, v in zip(im, mv) if v)
+
+ body = rows * ckb / 1e6 # GB
+ hand = rows * 2 * args.hand_yield * ckb / 1e6
+ orig = len(frames) * FRAME_KB.get(dataset, 400.0) / 1e6
+ print(f"{split:26s} {rows:10,d} {len(frames):10,d} "
+ f"{body:7.1f}G {hand:7.1f}G {orig:9.1f}G {body+hand+orig:8.1f}G")
+ tot["rows"] += rows
+ tot["frames"] += len(frames)
+ tot["body"] += body
+ tot["hand"] += hand
+ tot["orig"] += orig
+
+ if not args.brief:
+ print("-" * len(hdr))
+ print(f"{'TOTAL':26s} {tot['rows']:10,d} {tot['frames']:10,d} "
+ f"{tot['body']:7.1f}G {tot['hand']:7.1f}G {tot['orig']:9.1f}G "
+ f"{tot['body']+tot['hand']+tot['orig']:8.1f}G")
+ print()
+ print("`originals` is the dominant term everywhere except COCO. If it "
+ "does not fit, build with --originals none: body_crops and "
+ "hands_crops are self-contained, and the npz `image_relpath` "
+ "field records exactly which frame each crop came from so the "
+ "originals can be re-fetched later for the rows you still want.")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/probe_split.py b/datasets_pipeline/probe_split.py
new file mode 100755
index 0000000..6eb5be3
--- /dev/null
+++ b/datasets_pipeline/probe_split.py
@@ -0,0 +1,207 @@
+#!/usr/bin/env python3
+"""Decide whether a split is worth its download — from the parquets alone.
+
+Annotations are ~1 GB per split; frames are tens to thousands of GB. This
+reads only the annotations and answers, before you spend the bandwidth:
+
+ size persons, unique frames, and how many GB of frames that implies.
+ Totals extrapolate over the shards ON DISK, so fetch the whole
+ split's annotations first if you want the true count.
+ labels keypoint visibility health. NOT cosmetic: SA-1B stores invisible
+ keypoints as literal (0, 0) with a visibility column that
+ parquet_to_npz.py discards, so a naive build writes 44 of 70
+ keypoints per person as fake observations at the image corner.
+ hands how many hands are fully visible and how big their crop would be
+ signal how far the finger poses sit from ONE constant hand pose, in mm at
+ the fingertips — an upper bound on what a hand branch could learn
+
+ python datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/*
+ python datasets_pipeline/probe_split.py --annotation-dir $ANN_DIR/sa1b_train --shards 60
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+import numpy as np
+import pyarrow.parquet as pq
+
+HERE = Path(__file__).resolve().parent
+REPO = HERE.parent
+sys.path.insert(0, str(HERE))
+from plan_disk import FRAME_KB # noqa: E402
+
+# Fraction of the frames you must actually download that this corpus
+# references. 1.0 where one archive holds everything and unreferenced images
+# cost nothing extra to skip. SA-1B is the exception: its references are spread
+# uniformly over all 1000 tars (994 of 1000 id-blocks are hit by just 3% of the
+# annotations), so reaching 1.27M referenced images means pulling ~11 TB.
+ARCHIVE_COVERAGE = {"sa1b": 0.114}
+
+FINGER_PARAMS = list(range(68, 122))
+RIGHT_HAND_KPS = list(range(21, 42))
+LEFT_HAND_KPS = list(range(42, 63))
+RIGHT_WRIST, LEFT_WRIST = 41, 62
+
+
+def articulation_mm(model_params: np.ndarray, mhr_path: Path,
+ reg_path: Path) -> tuple[float, float] | None:
+ import torch
+
+ if not mhr_path.exists() or not reg_path.exists():
+ return None
+ mhr = torch.jit.load(str(mhr_path), map_location="cpu").eval()
+ W = torch.tensor(np.load(reg_path), dtype=torch.float32)
+
+ def kp70(mp: torch.Tensor) -> torch.Tensor:
+ cp = torch.cat([mp, torch.zeros(mp.shape[0], 45)], 1)
+ jp = mhr.character_torch.model_parameters_to_joint_parameters(cp)
+ ss = mhr.character_torch.joint_parameters_to_skeleton_state(jp)[..., :3] / 100.0
+ v = torch.stack([ss[..., 0], -ss[..., 1], -ss[..., 2]], -1)
+ return torch.einsum("kj,bjc->bkc", W, v)
+
+ X = torch.tensor(model_params)
+ Xm = X.clone()
+ Xm[:, FINGER_PARAMS] = X[:, FINGER_PARAMS].mean(0, keepdim=True)
+ with torch.no_grad():
+ a, b = kp70(X), kp70(Xm)
+ e = torch.cat([
+ ((a[:, k] - a[:, [w]]) - (b[:, k] - b[:, [w]])).norm(dim=-1) * 1000
+ for k, w in ((RIGHT_HAND_KPS, RIGHT_WRIST), (LEFT_HAND_KPS, LEFT_WRIST))
+ ], 1).numpy()
+ return float(e.mean()), float(np.percentile(e, 90))
+
+
+def probe(d: Path, shards: int, hand_expand: float, args) -> None:
+ files = sorted(d.glob("*.parquet"))
+ if not files:
+ return
+ used = files[:shards]
+
+ rows = 0
+ frames: set[str] = set()
+ all_vis = 0
+ zero_kps = []
+ params: list[np.ndarray] = []
+ spans: list[float] = []
+ hands_seen = hands_full = 0
+
+ for f in used:
+ t = pq.read_table(f, columns=["image", "mhr_valid", "keypoints_2d",
+ "model_params"])
+ for im, mv, k, mp in zip(t.column("image").to_pylist(),
+ t.column("mhr_valid").to_pylist(),
+ t.column("keypoints_2d").to_pylist(),
+ t.column("model_params").to_pylist()):
+ if not mv:
+ continue
+ rows += 1
+ frames.add(im)
+ k = np.asarray(k, dtype=np.float32).reshape(70, 3)
+ all_vis += bool((k[:, 2] > 0).all())
+ zero_kps.append(int(((k[:, 0] == 0) & (k[:, 1] == 0)).sum()))
+ params.append(np.asarray(mp, dtype=np.float32))
+ for idx in (RIGHT_HAND_KPS, LEFT_HAND_KPS):
+ hands_seen += 1
+ h = k[idx]
+ if not (h[:, 2] > 0).all():
+ continue
+ hands_full += 1
+ spans.append(max(np.ptp(h[:, 0]), np.ptp(h[:, 1])) * hand_expand)
+
+ scale = len(files) / len(used)
+ dataset = d.name.split("_")[0]
+ cov = ARCHIVE_COVERAGE.get(dataset, 1.0)
+ gb = len(frames) * scale * FRAME_KB.get(dataset, 400.0) / 1e6 / cov
+
+ print(f"\n== {d.name} ({len(used)}/{len(files)} shards read) ==")
+ print(f" persons : {rows:,} sampled -> ~{rows * scale:,.0f} "
+ f"over the {len(files)} shards ON DISK")
+ print(f" unique frames : {len(frames):,} sampled -> ~{len(frames) * scale:,.0f}")
+ note = "" if cov == 1.0 else f" [only {cov:.0%} of what you must pull is referenced]"
+ print(f" frames to download : ~{gb:,.0f} GB "
+ f"({gb * 1000 / max(rows * scale, 1):.2f} MB per person crop){note}")
+
+ pct = 100.0 * all_vis / max(rows, 1)
+ flag = "" if pct > 99 else " <-- gate hand boxes on joints_2d_vis"
+ print(f" all 70 kps visible : {pct:5.1f}% "
+ f"mean (0,0) placeholders {np.mean(zero_kps):.1f}/70{flag}")
+
+ if spans:
+ s = np.array(spans)
+ print(f" hands fully visible: {100 * hands_full / max(hands_seen, 1):5.1f}%")
+ print(f" hand crop, px : p50 {np.percentile(s, 50):4.0f} "
+ f">=64px {100 * (s >= 64).mean():5.1f}% "
+ f">=96px {100 * (s >= 96).mean():5.1f}% "
+ f">=128px {100 * (s >= 128).mean():5.1f}%")
+ else:
+ print(f" hands fully visible: 0.0% -- no usable hand crops")
+
+ art = articulation_mm(np.stack(params), args.mhr, args.regressor)
+ if art is None:
+ print(" finger articulation: (needs checkpoints/mhr_model.pt + the regressor)")
+ return
+ mean, p90 = art
+ verdict = ("worth a hand branch" if mean > 20 else
+ "promising — the best you will find in this corpus" if mean > 12 else
+ "NOT worth a hand branch: labels are ~a constant hand pose")
+ print(f" finger articulation: {mean:.1f} mm mean, {p90:.1f} mm p90 -> {verdict}")
+
+
+def list_groups(d: Path, depth: int) -> None:
+ """Referenced source groups at `depth` path components, with person counts."""
+ import collections
+
+ files = sorted(d.glob("*.parquet"))
+ groups: collections.Counter = collections.Counter()
+ frames: dict[str, set[str]] = collections.defaultdict(set)
+ for f in files:
+ t = pq.read_table(f, columns=["image", "mhr_valid"])
+ for im, mv in zip(t.column("image").to_pylist(),
+ t.column("mhr_valid").to_pylist()):
+ if not mv:
+ continue
+ parts = im.split("/")
+ g = "/".join(parts[:depth]) if len(parts) > depth else im
+ groups[g] += 1
+ frames[g].add(im)
+ print(f"\n== {d.name}: {len(groups)} referenced group(s) at depth {depth}, "
+ f"{sum(groups.values()):,} persons ({len(files)} shards) ==")
+ for g, n in sorted(groups.items()):
+ print(f" {g:44s} {n:8,d} persons {len(frames[g]):8,d} frames")
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--annotation-dir", nargs="+", type=Path, required=True)
+ p.add_argument("--list-groups", type=int, default=0, metavar="DEPTH",
+ help="instead of probing, print the referenced source groups "
+ "at this path depth, with person counts. DEPTH 2 gives "
+ "the exact EgoHumans take list (<seq>/<take>) and "
+ "EgoExo4D take list; DEPTH 1 gives Harmony4D scenes and "
+ "3DPW sequences. This is the download shopping list — "
+ "EgoHumans advertises 550 GB but only 89 takes are "
+ "referenced.")
+ p.add_argument("--shards", type=int, default=12,
+ help="shards to read per split; 12 is plenty for the "
+ "distributions, more only tightens the person count")
+ p.add_argument("--hand-expand", type=float, default=1.5,
+ help="matches build_split.py's square expansion when judging "
+ "whether a hand clears --hand-min-px")
+ p.add_argument("--mhr", type=Path, default=REPO / "checkpoints" / "mhr_model.pt")
+ p.add_argument("--regressor", type=Path,
+ default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")
+ args = p.parse_args()
+ for d in args.annotation_dir:
+ if args.list_groups:
+ list_groups(d, args.list_groups)
+ else:
+ probe(d, args.shards, args.hand_expand, args)
+ print()
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/registry.py b/datasets_pipeline/registry.py
new file mode 100755
index 0000000..fd90f4a
--- /dev/null
+++ b/datasets_pipeline/registry.py
@@ -0,0 +1,220 @@
+#!/usr/bin/env python3
+"""Per-dataset acquisition specs for the SAM-3D-Body corpus.
+
+One place that answers, for every split of ``facebook/sam-3d-body-dataset``:
+
+ * how the images are obtained (a plain URL, a Hugging Face repo, a CLI that
+ needs credentials, or a licence form),
+ * where a parquet ``image`` string lands inside the extracted tree,
+ * whether the frames need undistorting first,
+ * how big it all is.
+
+``image_relpath`` is re-exported from ``tools/parquet_to_npz.py`` rather than
+re-implemented, so this module can never drift from the converter that the
+existing ``data/sam3d_gt_*`` folders were built with.
+
+Sizes are measured, not guessed:
+ * COCO train2014 / MPII: HTTP Content-Length, checked 2026-08-31.
+ * Harmony4D: the Hugging Face tree API for ``Jyun-Ting/Harmony4D``.
+ * AIC: the OpenDataLab tarball already on this machine.
+"""
+
+from __future__ import annotations
+
+import sys
+from dataclasses import dataclass, field
+from pathlib import Path
+
+REPO_ROOT = Path(__file__).resolve().parent.parent
+if str(REPO_ROOT) not in sys.path:
+ sys.path.insert(0, str(REPO_ROOT))
+
+from tools.parquet_to_npz import image_relpath # noqa: E402,F401 (re-export)
+
+HF_ANNOTATION_REPO = "facebook/sam-3d-body-dataset"
+
+
+@dataclass(frozen=True)
+class Dataset:
+ """Everything needed to go from a bare machine to usable frames."""
+
+ name: str
+ splits: tuple[str, ...]
+ # How to get the images. "direct" = plain URLs; everything else needs a
+ # human somewhere in the loop and fetch_images.py prints instructions.
+ method: str # direct | hf_zips | opendatalab | gdrive | cli | links_file
+ urls: tuple[str, ...] = () # for method == "direct"
+ hf_repo: str = "" # for method == "hf_zips"
+ # Where image_relpath() is rooted, relative to the extraction dir. The
+ # archives usually already create this, so it is normally "".
+ image_root_suffix: str = ""
+ needs_undistort: bool = False
+ approx_gb: float = 0.0 # download size
+ notes: str = ""
+ # Per-scene/shard archives, for streaming one at a time. Empty means the
+ # dataset arrives as one blob.
+ archives: tuple[str, ...] = field(default_factory=tuple)
+
+
+# --------------------------------------------------------------------------
+# The eight source datasets.
+# --------------------------------------------------------------------------
+HARMONY4D_TRAIN_ZIPS = (
+ "01_hugging", "02_grappling", "03_grappling2", "04_sword_part1",
+ "04_sword_part2", "05_sword2", "07_ballroom", "08_ballroom2",
+ "09_karate", "10_karate2", "11_karate3", "12_mma", "13_mma2",
+ "14_mma3", "15_mma4",
+)
+HARMONY4D_TEST_ZIPS = (
+ "01_hugging", "03_grappling2", "05_sword2", "06_sword3",
+ "08_ballroom2", "15_mma4", "16_mma5",
+)
+
+DATASETS: dict[str, Dataset] = {
+ "coco": Dataset(
+ name="coco",
+ splits=("coco_train",),
+ method="direct",
+ urls=("http://images.cocodataset.org/zips/train2014.zip",),
+ approx_gb=13.5,
+ notes="Only train2014 is referenced by coco_train — val2014/test2014 are "
+ "a waste of 9 GB. Verified over every image string in shard 0.",
+ ),
+ "mpii": Dataset(
+ name="mpii",
+ splits=("mpii_train",),
+ method="direct",
+ urls=("https://datasets.d2.mpi-inf.mpg.de/andriluka14cvpr/"
+ "mpii_human_pose_v1.tar.gz",),
+ approx_gb=12.1,
+ notes="Extracts to images/. Only 5,441 of ~25k images are referenced, "
+ "so --originals copy reclaims ~11 GB.",
+ ),
+ "aic": Dataset(
+ name="aic",
+ splits=("aic_train",),
+ method="opendatalab",
+ approx_gb=36.4,
+ notes="AI Challenger 2017. The original challenger.ai host is dead; the "
+ "working route is OpenDataLab, which is where this corpus came "
+ "from last time (see ~/sam3d_data/aic_download).\n"
+ " pip install opendatalab && odl login\n"
+ " odl get OpenDataLab/AI_Challenger -d $DEST\n"
+ " tar xzf $DEST/OpenDataLab___AI_Challenger/raw/AI_Challenger.tar.gz\n"
+ "Needs the train/images/ subtree only.",
+ ),
+ "3dpw": Dataset(
+ name="3dpw",
+ splits=("3dpw_train",),
+ method="cli",
+ approx_gb=4.6,
+ notes="Accept the licence at "
+ "https://virtualhumans.mpi-inf.mpg.de/3DPW/license.html, then "
+ "download imageFiles.zip and unzip so that sequence folders "
+ "(courtyard_arguing_00/, ...) sit directly under the image dir.",
+ ),
+ "harmony4d": Dataset(
+ name="harmony4d",
+ splits=("harmony4d_train", "harmony4d_test"),
+ method="hf_zips",
+ hf_repo="Jyun-Ting/Harmony4D",
+ needs_undistort=True,
+ approx_gb=351.7,
+ archives=tuple(f"train/{s}.zip" for s in HARMONY4D_TRAIN_ZIPS)
+ + tuple(f"test/{s}.zip" for s in HARMONY4D_TEST_ZIPS),
+ notes="287 GB train + 65 GB test of 4K frames, ungated on Hugging Face. "
+ "Far past any sane disk budget as a whole, so run it through "
+ "stream_archives.py, which does one scene zip at a time and "
+ "deletes it after the crops are written. Frames MUST be "
+ "undistorted first — see undistort.md for the three known "
+ "blockers in Meta's script.",
+ ),
+ "egohumans": Dataset(
+ name="egohumans",
+ splits=("egohumans_train",),
+ method="gdrive",
+ needs_undistort=True,
+ approx_gb=107.0,
+ notes="Google Drive folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR, one "
+ ".tar.gz per TAKE under <sequence>/<take>.tar.gz.\n"
+ " pip install gdown\n"
+ " gdown --folder 1JD963urzuzV_R_6FOVOtlx8UupwUuknR -O $DEST\n"
+ " tar xzf 001_tagging.tar.gz --strip-components=8 # NOT optional\n"
+ "The dataset advertises 550 GB but egohumans_train references "
+ "only 89 takes across 7 sequences (tagging, lego, fencing, "
+ "basketball, volleyball, badminton, tennis) = 272k persons over "
+ "268k 4K frames, ~107 GB of referenced frames. Get the exact "
+ "shopping list with:\n"
+ " probe_split.py --annotation-dir $ANN/egohumans_train --list-groups 2\n"
+ "Only the exo/ cameras are referenced; ego/, colmap/ and "
+ "processed_data/ in each tarball are dead weight. Undistort with "
+ "sam-3d-body/data/scripts/egohumans/undistort_images.py — see "
+ "undistort.md.",
+ ),
+ "egoexo4d": Dataset(
+ name="egoexo4d",
+ splits=("egoexo4d_physical_train", "egoexo4d_physical_test",
+ "egoexo4d_procedure_train", "egoexo4d_procedure_test"),
+ method="cli",
+ approx_gb=0.0,
+ notes="Needs the signed Ego-Exo4D licence (you have one pending). Then:\n"
+ " pip install ego4d && egoexo -o $DEST --parts sam_3d_body\n"
+ "The sam_3d_body part is already undistorted. 610 train shards "
+ "~= 1.1M person crops, and it is the one split in this corpus "
+ "where hands are routinely large enough to learn from.",
+ ),
+ "sa1b": Dataset(
+ name="sa1b",
+ splits=("sa1b_train",),
+ method="links_file",
+ approx_gb=0.0,
+ notes="1,850 annotation shards ~= 3.4M person crops — bigger than every "
+ "other split combined. Accept the terms at "
+ "https://ai.meta.com/datasets/segment-anything-downloads/ to get "
+ "links.txt (one `sa_NNNNNN.tar <signed-url>` per line), then feed "
+ "it to fetch_images.py --links-file.\n"
+ "MEASURED, and it is bad news: the referenced images are spread "
+ "UNIFORMLY over all 1000 tars — 994 of 1000 id-blocks are hit by "
+ "just 3% of the annotations — and only 11.4% of SA-1B is "
+ "referenced at all. So there is no dense subset: you pull a whole "
+ "~10 GB tar to get ~1,270 useful frames (~3,400 crops), i.e. "
+ "~2.3 MB of download per crop against AIC's 0.13 MB. Worse, "
+ "SA-1B is the ONE split whose keypoints carry real visibility "
+ "flags: a mean of 44 of 70 are literal (0,0) placeholders and "
+ "only 0.3% of hands are fully visible.\n"
+ "Any prefix of the tar list is a valid subset — each tar is "
+ "self-contained and the builder skips rows whose image is absent "
+ "— so stream it and stop whenever. But spend the bandwidth on "
+ "egoexo4d first.",
+ ),
+}
+
+# Which parquet split belongs to which source dataset.
+SPLIT_TO_DATASET = {s: d.name for d in DATASETS.values() for s in d.splits}
+
+
+def dataset_for_split(split: str) -> Dataset:
+ try:
+ return DATASETS[SPLIT_TO_DATASET[split]]
+ except KeyError:
+ raise SystemExit(
+ f"unknown split {split!r}. Known splits:\n "
+ + "\n ".join(sorted(SPLIT_TO_DATASET))
+ ) from None
+
+
+def output_dir_for_split(split: str, data_root: Path) -> Path:
+ """`coco_train` -> `<data_root>/sam3d_gt_coco`.
+
+ Train and test of the same dataset deliberately land in the same folder:
+ that is the convention the existing data/ tree already uses, and the
+ training script's dataset discovery walks one level of sub-folders.
+ """
+ return data_root / f"sam3d_gt_{SPLIT_TO_DATASET[split]}"
+
+
+if __name__ == "__main__":
+ print(f"{'dataset':12s} {'method':12s} {'GB':>7s} splits")
+ for d in DATASETS.values():
+ gb = f"{d.approx_gb:.1f}" if d.approx_gb else "?"
+ print(f"{d.name:12s} {d.method:12s} {gb:>7s} {', '.join(d.splits)}")
diff --git a/datasets_pipeline/requirements.txt b/datasets_pipeline/requirements.txt
new file mode 100644
index 0000000..3af6ee7
--- /dev/null
+++ b/datasets_pipeline/requirements.txt
@@ -0,0 +1,8 @@
+# On top of the training env (numpy, opencv-python, tqdm).
+pyarrow>=14
+pandas>=2
+huggingface_hub>=0.23 # fetch_annotations.py; needs an accepted licence + token
+# Optional, per dataset:
+# gdown egohumans (Google Drive folder)
+# opendatalab aic (odl login && odl get OpenDataLab/AI_Challenger)
+# ego4d egoexo4d (egoexo -o $DEST --parts sam_3d_body)
diff --git a/datasets_pipeline/stream_archives.py b/datasets_pipeline/stream_archives.py
new file mode 100755
index 0000000..26e7692
--- /dev/null
+++ b/datasets_pipeline/stream_archives.py
@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+"""Build a dataset that does not fit on disk, one archive at a time.
+
+Harmony4D is 352 GB of 4K zips and SA-1B is ~11 TB; neither can be staged
+whole on a 172 GB volume. The loop below never holds more than one archive:
+
+ download -> extract -> [undistort] -> build crops -> delete everything raw
+
+Because build_split.py is resume-safe and silently skips rows whose image is
+absent, any prefix of the archive list yields a valid, self-consistent dataset.
+Stop whenever you have enough, or when the disk says so.
+
+ # Harmony4D, scene by scene, keeping nothing but the crops
+ python datasets_pipeline/stream_archives.py --dataset harmony4d \
+ --split harmony4d_train --annotation-dir $ANN_DIR/harmony4d_train \
+ --work-dir /scratch/h4d --data-root data
+
+ # SA-1B, 20 tars at a time, stop after 200
+ python datasets_pipeline/stream_archives.py --dataset sa1b \
+ --split sa1b_train --annotation-dir $ANN_DIR/sa1b_train \
+ --work-dir /scratch/sa1b --data-root data \
+ --links-file ~/sa1b_links.txt --max-archives 200 --build-every 20
+
+Anything passed after `--` goes straight to build_split.py, e.g.
+`-- --crop-format jpg --hand-min-px 48 --originals none`.
+"""
+
+from __future__ import annotations
+
+import argparse
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+sys.path.insert(0, str(HERE))
+from registry import DATASETS, dataset_for_split # noqa: E402
+
+
+def sh(cmd: list, cwd: Path | None = None) -> None:
+ print("\n+ " + " ".join(str(c) for c in cmd)
+ + (f" (cwd={cwd})" if cwd else ""), flush=True)
+ subprocess.run([str(c) for c in cmd], check=True,
+ cwd=str(cwd) if cwd else None)
+
+
+def free_gb(p: Path) -> float:
+ while not p.exists():
+ p = p.parent
+ return shutil.disk_usage(p).free / 1e9
+
+
+def undistort_harmony4d(raw: Path, undist: Path, scene_rel: str,
+ sam3d_repo: Path, python_bin: str) -> None:
+ """Run Meta's undistortion over every take of one scene.
+
+ `--seqs` wants the TAKE folder (train/09_karate/007_karate), not the scene:
+ the script appends `exo/` to whatever it is given. See undistort.md.
+ """
+ scene_dir = raw / scene_rel
+ takes = sorted(d.name for d in scene_dir.iterdir() if d.is_dir())
+ if not takes:
+ raise SystemExit(f"no takes under {scene_dir}")
+ seqs = ",".join(f"{scene_rel}/{t}" for t in takes)
+ scripts = sam3d_repo / "data" / "scripts"
+ if not (scripts / "lib").is_dir():
+ raise SystemExit(
+ f"{scripts}/lib is missing — Meta's undistort script imports it but "
+ f"the clone does not ship it. See datasets_pipeline/undistort.md.")
+ sh([python_bin, "harmony4d/undistort_images.py",
+ "--src_dir", raw.resolve(), "--dst_dir", undist.resolve(),
+ "--seqs", seqs], cwd=scripts)
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--dataset", required=True, choices=sorted(DATASETS))
+ p.add_argument("--split", required=True)
+ p.add_argument("--annotation-dir", required=True, type=Path)
+ p.add_argument("--work-dir", required=True, type=Path,
+ help="scratch space for one archive at a time")
+ p.add_argument("--data-root", type=Path, default=Path("data"))
+ p.add_argument("--archives", nargs="*", default=None)
+ p.add_argument("--max-archives", type=int, default=None)
+ p.add_argument("--build-every", type=int, default=1,
+ help="build after this many archives (SA-1B has flat, "
+ "unpredictable filenames so batching amortises the "
+ "parquet scan)")
+ p.add_argument("--links-file", type=Path, default=None)
+ p.add_argument("--min-free-gb", type=float, default=25.0,
+ help="abort before an archive that would leave less than this")
+ p.add_argument("--keep-frames", action="store_true",
+ help="do not delete the extracted frames after building")
+ p.add_argument("--sam3d-repo", type=Path,
+ default=Path("/home/madjel/Projects/gitpackages/sam3-biomechanics/sam-3d-body"),
+ help="for the undistortion scripts")
+ p.add_argument("--undistort-python", default=sys.executable,
+ help="interpreter with pycolmap available (the "
+ "sam_3d_body_data env, not the training env)")
+ p.add_argument("build_args", nargs="*",
+ help="extra args for build_split.py, after --")
+ args = p.parse_args()
+
+ ds = DATASETS[args.dataset]
+ if dataset_for_split(args.split).name != ds.name:
+ raise SystemExit(f"split {args.split} does not belong to {ds.name}")
+
+ raw = args.work_dir / "raw"
+ undist = args.work_dir / "undistorted"
+ image_dir = undist if ds.needs_undistort else raw
+
+ if ds.method == "hf_zips":
+ archives = list(args.archives or ds.archives)
+ elif ds.method == "links_file":
+ if not args.links_file:
+ raise SystemExit("--links-file is required for sa1b")
+ archives = [ln.split()[0] for ln in args.links_file.read_text().splitlines()
+ if len(ln.split()) == 2]
+ else:
+ raise SystemExit(
+ f"{ds.name} arrives as one blob, not per-archive — fetch it with "
+ f"fetch_images.py and run build_split.py directly.")
+ if args.max_archives:
+ archives = archives[: args.max_archives]
+
+ print(f"{ds.name}: {len(archives)} archive(s), work dir {args.work_dir}")
+ pending: list[str] = []
+
+ for i, arc in enumerate(archives, 1):
+ if free_gb(args.work_dir) < args.min_free_gb:
+ print(f"\nSTOP: only {free_gb(args.work_dir):.0f} GB free, below "
+ f"--min-free-gb {args.min_free_gb}. "
+ f"{len(archives) - i + 1} archive(s) left; re-run to resume.")
+ break
+
+ print(f"\n===== [{i}/{len(archives)}] {arc} "
+ f"({free_gb(args.work_dir):.0f} GB free) =====")
+ fetch = [sys.executable, HERE / "fetch_images.py",
+ "--dataset", ds.name, "--dest", raw, "--no-keep-archive"]
+ if ds.method == "hf_zips":
+ fetch += ["--archives", arc]
+ else:
+ fetch += ["--links-file", args.links_file, "--max-archives", "1"]
+ sh(fetch)
+
+ scene_rel = arc[:-4] if arc.endswith(".zip") else None
+ if ds.needs_undistort and scene_rel:
+ undistort_harmony4d(raw, undist, scene_rel, args.sam3d_repo,
+ args.undistort_python)
+
+ pending.append(arc)
+ if len(pending) < args.build_every and i < len(archives):
+ continue
+
+ build = [sys.executable, HERE / "build_split.py",
+ "--split", args.split,
+ "--annotation-dir", args.annotation_dir,
+ "--image-dir", image_dir,
+ "--data-root", args.data_root]
+ if ds.method == "hf_zips" and len(pending) == 1 and scene_rel:
+ build += ["--image-prefix", f"{scene_rel}/"]
+ build += args.build_args
+ sh(build)
+ pending.clear()
+
+ if not args.keep_frames:
+ for d in (raw, undist):
+ if d.exists():
+ shutil.rmtree(d, ignore_errors=True)
+
+ print(f"\nstreaming done; {free_gb(args.work_dir):.0f} GB free")
+
+
+if __name__ == "__main__":
+ main()
diff --git a/datasets_pipeline/undistort.md b/datasets_pipeline/undistort.md
new file mode 100644
index 0000000..b0718c5
--- /dev/null
+++ b/datasets_pipeline/undistort.md
@@ -0,0 +1,55 @@
+# Undistorting Harmony4D and EgoHumans
+
+Both datasets ship **distorted** frames plus COLMAP intrinsics. The SAM-3D-Body
+annotations are expressed against the *undistorted* frames, so skipping this
+step silently misaligns every keypoint. EgoExo4D's `sam_3d_body` part is already
+undistorted — nothing to do there.
+
+Meta's scripts live in `sam-3d-body/data/scripts/{harmony4d,egohumans}/undistort_images.py`.
+Three things block a fresh clone. All three were hit and fixed on 2026-06-30.
+
+### 1. `ModuleNotFoundError: No module named 'lib'`
+
+The script does `from lib.datasets.ego_exo_scene import EgoExoSceneVis`, and
+`_init_paths.py` expects `lib/` as a sibling of `harmony4d/` — but the clone
+only ships `config/`. Vendor the seven files of `lib/` (`datasets/` + `utils/`)
+from <https://github.com/jyuntins/harmony4d> (`main`) into `data/scripts/lib/`.
+
+`stream_archives.py` checks for this directory and fails loudly rather than
+part-way through a 45 GB scene.
+
+### 2. `mmcv` / `smplx` missing
+
+`lib/datasets/ego_exo_scene.py` imports both at module level, but only uses
+them for SMPL/keypoint *visualisation* — never for undistortion. Make those two
+imports lazy (`try/except ImportError → None`) instead of installing two large
+packages you will not call. `pycolmap` genuinely is required.
+
+### 3. `--seqs` wants the TAKE, not the scene
+
+The script appends `exo/` to whatever `--seqs` names, so `train/01_hugging` is
+wrong (there is no `exo/` there) — it must be `train/01_hugging/001_hugging`.
+One scene can hold several takes; pass them comma-separated:
+
+```bash
+ls -d $HARMONY4D_DATA_DIR/train/09_karate/*/ \
+ | sed 's#.*/train/#train/#; s#/$##' | paste -sd,
+```
+
+`stream_archives.py` enumerates the takes of each scene and builds this list for
+you, so the manual form is only needed for a one-off.
+
+### Running it
+
+Use the `sam_3d_body_data` env (it has `pycolmap`), not the training env:
+
+```bash
+cd $SAM3D_REPO/data/scripts
+python harmony4d/undistort_images.py \
+ --src_dir $HARMONY4D_DATA_DIR \
+ --dst_dir $HARMONY4D_IMG_DIR \
+ --seqs train/09_karate/007_karate
+```
+
+Validated on `01_hugging/001_hugging` (6.6k frames, 22 exo cameras).
+Pass the interpreter to the streamer with `--undistort-python`.
diff --git a/datasets_pipeline/verify_split.py b/datasets_pipeline/verify_split.py
new file mode 100755
index 0000000..2eba390
--- /dev/null
+++ b/datasets_pipeline/verify_split.py
@@ -0,0 +1,175 @@
+#!/usr/bin/env python3
+"""Check a built dataset folder, and measure how much hand signal it carries.
+
+Three independent reports:
+
+ structure every npz has its body crop; every hand crop has an npz saying it
+ should exist; no orphans either way.
+ geometry RMS reprojection error of (joints_3d + cam_trans) onto joints_2d,
+ and the source-pixel size of the stored hand crops.
+ signal --hand-signal: how far the labels' finger poses sit from a single
+ constant mean hand, in millimetres at the fingertips.
+
+The last one is the number that decides whether a hand branch is worth
+building on a split. It is an upper bound on what any model could learn about
+finger articulation from these labels: if the whole dataset is 7 mm away from
+one fixed hand pose, a perfect hand branch buys at most 7 mm. Measured on the
+current corpus: coco 6.5, aic 6.9, mpii 7.4, 3dpw 7.5, harmony4d 12.6 mm —
+those splits have no finger observations in their source annotations, so the
+MHR fits fall back on the prior. Run this on egoexo4d before committing to a
+second encoder.
+
+ python datasets_pipeline/verify_split.py --dir data/sam3d_gt_aic
+ python datasets_pipeline/verify_split.py --dir data/sam3d_gt_egoexo4d --hand-signal
+"""
+
+from __future__ import annotations
+
+import argparse
+import random
+import sys
+from pathlib import Path
+
+import numpy as np
+
+HERE = Path(__file__).resolve().parent
+REPO = HERE.parent
+sys.path.insert(0, str(REPO))
+from tools.parquet_to_npz import reproj_error # noqa: E402
+
+# MHR model-parameter indices of the 54 finger DOFs (27 per hand), derived from
+# the rig's parameter_transform: these are the only pose params that move the
+# finger joints and nothing else.
+FINGER_PARAMS = list(range(68, 122))
+RIGHT_HAND_KPS = list(range(21, 42))
+LEFT_HAND_KPS = list(range(42, 63))
+RIGHT_WRIST, LEFT_WRIST = 41, 62
+
+
+def structure(d: Path) -> None:
+ anns = sorted((d / "annotations").glob("*.npz"))
+ body = {p.stem for p in (d / "body_crops").iterdir()} if (d / "body_crops").exists() else set()
+ hands = {p.stem for p in (d / "hands_crops").iterdir()} if (d / "hands_crops").exists() else set()
+ stems = {p.stem for p in anns}
+
+ missing_body = sorted(stems - body)[:5]
+ orphan_body = sorted(body - stems)[:5]
+ want_hands, have_hands, missing_hands = 0, 0, []
+ for p in anns:
+ a = np.load(p, allow_pickle=False)
+ for side in ("l", "r"):
+ key = f"hand_valid_{side}"
+ if key not in a.files or not bool(a[key]):
+ continue
+ want_hands += 1
+ if f"{p.stem}_{side}" in hands:
+ have_hands += 1
+ elif len(missing_hands) < 5:
+ missing_hands.append(f"{p.stem}_{side}")
+
+ print(f" annotations : {len(anns):,}")
+ print(f" body_crops : {len(body):,}"
+ f" (missing {len(stems - body):,}, orphan {len(body - stems):,})")
+ print(f" hands_crops : {len(hands):,}"
+ f" (declared {want_hands:,}, present {have_hands:,})")
+ if (d / "original_images").exists():
+ n = sum(1 for _ in (d / "original_images").rglob("*") if _.is_file())
+ print(f" original_images : {n:,} files")
+ if missing_body:
+ print(f" !! body crop missing for: {missing_body}")
+ if orphan_body:
+ print(f" !! body crop with no npz: {orphan_body}")
+ if missing_hands:
+ print(f" !! hand crop declared but absent: {missing_hands}")
+
+
+def geometry(d: Path, n: int, seed: int) -> None:
+ anns = sorted((d / "annotations").glob("*.npz"))
+ random.Random(seed).shuffle(anns)
+ errs, hand_px = [], []
+ for p in anns[:n]:
+ a = np.load(p, allow_pickle=False)
+ errs.append(reproj_error(a["joints_3d"], a["cam_trans"], a["joints_2d"],
+ a["cam_focal_length"], a["orig_shape"]))
+ for side in ("l", "r"):
+ k = f"hand_crop_{side}"
+ if k in a.files and np.isfinite(a[k]).all():
+ hand_px.append(float(a[k][2] - a[k][0]))
+ errs = np.array(errs)
+ print(f" reprojection RMS : mean {errs.mean():.3f} px, "
+ f"p99 {np.percentile(errs, 99):.3f} px, max {errs.max():.3f} px")
+ if hand_px:
+ h = np.array(hand_px)
+ print(f" hand crop source px: p10 {np.percentile(h, 10):.0f} "
+ f"p50 {np.percentile(h, 50):.0f} p90 {np.percentile(h, 90):.0f}"
+ f" ({len(h):,} hands in the sample)")
+ else:
+ print(" hand crop source px: no hand crops in this sample")
+
+
+def hand_signal(d: Path, n: int, seed: int, mhr_path: Path, reg_path: Path) -> None:
+ import torch
+
+ if not mhr_path.exists() or not reg_path.exists():
+ print(f" !! need {mhr_path} and {reg_path}; skipping")
+ return
+ mhr = torch.jit.load(str(mhr_path), map_location="cpu").eval()
+ W = torch.tensor(np.load(reg_path), dtype=torch.float32)
+
+ def kp70(mp: torch.Tensor) -> torch.Tensor:
+ cp = torch.cat([mp, torch.zeros(mp.shape[0], 45)], 1)
+ jp = mhr.character_torch.model_parameters_to_joint_parameters(cp)
+ ss = mhr.character_torch.joint_parameters_to_skeleton_state(jp)[..., :3] / 100.0
+ v = torch.stack([ss[..., 0], -ss[..., 1], -ss[..., 2]], -1)
+ return torch.einsum("kj,bjc->bkc", W, v)
+
+ anns = sorted((d / "annotations").glob("*.npz"))
+ random.Random(seed).shuffle(anns)
+ X = torch.tensor(np.stack([np.load(p)["mhr_model_params"] for p in anns[:n]]))
+ Xm = X.clone()
+ Xm[:, FINGER_PARAMS] = X[:, FINGER_PARAMS].mean(0, keepdim=True)
+ with torch.no_grad():
+ a, b = kp70(X), kp70(Xm)
+ e = torch.cat([
+ ((a[:, k] - a[:, [w]]) - (b[:, k] - b[:, [w]])).norm(dim=-1) * 1000
+ for k, w in ((RIGHT_HAND_KPS, RIGHT_WRIST), (LEFT_HAND_KPS, LEFT_WRIST))
+ ], 1).numpy()
+ print(f" finger articulation: {e.mean():.1f} mm mean, "
+ f"{np.percentile(e, 90):.1f} mm p90 (over {len(X):,} labels)")
+ print(f" finger-param spread: {X[:, FINGER_PARAMS].std(0).mean():.4f} rad mean std")
+ verdict = ("worth a hand branch" if e.mean() > 20
+ else "marginal — check against your target error"
+ if e.mean() > 12 else
+ "NOT worth a hand branch: the labels are ~a constant hand pose")
+ print(f" verdict : {verdict}")
+
+
+def main() -> None:
+ p = argparse.ArgumentParser(description=__doc__,
+ formatter_class=argparse.RawDescriptionHelpFormatter)
+ p.add_argument("--dir", type=Path, required=True, help="data/sam3d_gt_<dataset>")
+ p.add_argument("--sample", type=int, default=500)
+ p.add_argument("--seed", type=int, default=0)
+ p.add_argument("--hand-signal", action="store_true")
+ p.add_argument("--mhr", type=Path,
+ default=REPO / "checkpoints" / "mhr_model.pt")
+ p.add_argument("--regressor", type=Path,
+ default=REPO / "instanthmr_distill_train" / "assets" / "mhr_j127_to_kp70.npy")
+ args = p.parse_args()
+
+ if not (args.dir / "annotations").is_dir():
+ sys.exit(f"{args.dir} has no annotations/ — is that a built split?")
+
+ print(f"\n== {args.dir} ==")
+ print("structure")
+ structure(args.dir)
+ print("geometry")
+ geometry(args.dir, args.sample, args.seed)
+ if args.hand_signal:
+ print("hand signal")
+ hand_signal(args.dir, args.sample, args.seed, args.mhr, args.regressor)
+ print()
+
+
+if __name__ == "__main__":
+ main()