Skip to content

Motion capture for any creature from one AI video: two cameras in one frame, a DINOv2 joint reader, the rig fitted in both views, a GLB out.

Load the released wolf LoRA from its URL

rehan-remadeb24df9fc77a7
diff --git a/mocap/fal.py b/mocap/fal.py
index b6a6b6b..8df330e 100644
--- a/mocap/fal.py
+++ b/mocap/fal.py
@@ -76,7 +76,7 @@ def run(endpoint, payload, **kw):
TYPES = {"png": "image/png", "jpg": "image/jpeg", "jpeg": "image/jpeg", "webp": "image/webp", "mp4": "video/mp4",
- "zip": "application/zip"}
+ "zip": "application/zip", "safetensors": "application/octet-stream"}
def upload(path):
diff --git a/mocap/lora.py b/mocap/lora.py
index ff72b79..1315baf 100644
--- a/mocap/lora.py
+++ b/mocap/lora.py
@@ -52,14 +52,21 @@ def resume(run_dir):
print("done", json.dumps(res)[:600], info, flush=True)
+RELEASE_LORA = "https://github.com/Patch-gg/diptych/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors"
+
+
def lora_path(src):
- """The LoRA for fal to load: a training run's directory (the URL in its lora.json), a URL, or a local .safetensors
- (uploaded to fal first, e.g. the wolf LoRA from the examples release)."""
+ """The LoRA for fal to load: a training run's directory (the URL in its lora.json), a URL (such as the wolf LoRA in
+ the examples release, RELEASE_LORA), or a small local .safetensors (uploaded to fal first)."""
if str(src).startswith(("http://", "https://")):
return str(src)
p = pathlib.Path(src)
if p.suffix == ".safetensors":
- return fal.upload(p)
+ try:
+ return fal.upload(p)
+ except Exception as e: # fal's simple upload refuses big files (the wolf LoRA is 131 MB): use a URL instead
+ raise SystemExit(f"could not upload {p.name} to fal ({e.__class__.__name__}); pass the LoRA's URL instead, "
+ "e.g. " + RELEASE_LORA) from None
return json.loads((p / "lora.json").read_text())["response"]["lora_file"]["url"]
diff --git a/skills/diptych-lora/SKILL.md b/skills/diptych-lora/SKILL.md
index 308aaba..da88096 100644
--- a/skills/diptych-lora/SKILL.md
+++ b/skills/diptych-lora/SKILL.md
@@ -45,12 +45,15 @@ its licence is non-commercial; say which you're using.
```bash
.venv/bin/python -m mocap.lora generate runs/<c>/lora/v1 runs/<c>/ref/ref.png prompts/<c>/pounce.txt runs/<c>/videos/pounce/lora.mp4 1.0 42
```
- The wolf's trained LoRA is in the examples release, so it can be tried without training (about $0.38 a clip):
+ The wolf's trained LoRA is in the examples release, so it can be tried without training (about $0.38 a clip);
+ fal loads it straight from the release:
```bash
- tools/fetch_examples.sh wolf --clips --lora
- .venv/bin/python -m mocap.lora generate runs/wolf/lora/twocam_wolf_h3_lora.safetensors runs/wolf/ref/ref.png \
- prompts/wolf/heldout/pounce.txt runs/wolf/videos/pounce/lora.mp4 1.0 42
+ tools/fetch_examples.sh wolf --clips
+ .venv/bin/python -m mocap.lora generate \
+ https://github.com/Patch-gg/diptych/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors \
+ runs/wolf/ref/ref.png prompts/wolf/heldout/pounce.txt runs/wolf/videos/pounce/lora.mp4 1.0 42
```
+ `tools/fetch_examples.sh wolf --lora` downloads the file itself (131 MB), to run H3 with it yourself.
6. A/B test on held-out moves against plain H3 with the same seed, then capture both and compare paw agreement and
reprojection error (diptych-capture).