Skip to content

Motion capture for any creature from one AI video: two cameras in one frame, a DINOv2 joint reader, the rig fitted in both views, a GLB out.

Rename to anymove

rehan-remade1281201948aa
diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json
index 46a3267..5f603a0 100644
--- a/.claude-plugin/marketplace.json
+++ b/.claude-plugin/marketplace.json
@@ -1,13 +1,13 @@
{
- "name": "diptych",
- "description": "diptych by Patch: motion capture for any creature from one AI video.",
+ "name": "anymove",
+ "description": "anymove by Patch: motion capture for any creature from one AI video.",
"owner": {
"name": "Patch",
"url": "https://patch.gg"
},
"plugins": [
{
- "name": "diptych",
+ "name": "anymove",
"source": "./",
"description": "Skills to rig a creature, film it with two cameras in one AI clip, train its joint reader, capture the move as a GLB, and train the MiniMax H3 LoRA.",
"category": "gamedev",
diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json
index 9d22013..558d5e5 100644
--- a/.claude-plugin/plugin.json
+++ b/.claude-plugin/plugin.json
@@ -1,14 +1,14 @@
{
- "name": "diptych",
- "displayName": "diptych",
+ "name": "anymove",
+ "displayName": "anymove",
"version": "0.1.0",
"description": "Motion capture for any creature from one AI video: two cameras in one frame, a DINOv2 joint reader trained on the creature, the rig fitted in both views, a GLB out. Skills for rigging, filming, training, capturing and the H3 LoRA.",
"author": {
"name": "Patch",
"url": "https://patch.gg"
},
- "homepage": "https://github.com/Patch-gg/diptych",
- "repository": "https://github.com/Patch-gg/diptych",
+ "homepage": "https://github.com/Patch-gg/anymove",
+ "repository": "https://github.com/Patch-gg/anymove",
"license": "Apache-2.0",
"keywords": ["animation", "motion-capture", "mocap", "creature", "rigging", "blender", "dinov2", "video-generation", "gamedev"]
}
diff --git a/README.md b/README.md
index 826e78c..094dd57 100644
--- a/README.md
+++ b/README.md
@@ -1,16 +1,16 @@
<picture>
<source media="(prefers-color-scheme: dark)" srcset="docs/readme/header-evening.png">
- <img alt="patch. diptych. any creature. any move." src="docs/readme/header-day.png" width="800">
+ <img alt="patch. anymove. motion capture for any creature, from one AI video." src="docs/readme/header-day.png" width="800">
</picture>
<br><br>
-<img alt="A dragon roars; an AI video shows it from the front and the side at once; every joint is found in both views; the two views stand up as walls and pin each joint in 3D; the rig lands on the points; a fire demon, a spider and a wolf move the same way." src="docs/diptych.gif" width="800">
+<img alt="A dragon roars; an AI video shows it from the front and the side at once; every joint is found in both views; the two views stand up as walls and pin each joint in 3D; the rig lands on the points; a fire demon, a spider and a wolf move the same way." src="docs/anymove.gif" width="800">
**@patch_gg** · dragon, giant spider, fire demon, wolf · one AI clip per move, $0.26 each, read and fitted in under
a minute.
-diptych turns one AI video into an animation for any rigged creature, wings, tails and eight legs included. A video
+anymove turns one AI video into an animation for any rigged creature, wings, tails and eight legs included. A video
model films the creature from two cameras in one frame, front on the left and side on the right. A joint reader
trained on that one creature finds every joint in both views. A joint sits at the same height in both, so the pair
pins it in 3D, and the rig is fitted to those joints frame by frame and exported as a GLB.
@@ -41,7 +41,7 @@ You need an NVIDIA GPU with CUDA (24 GB to train a reader; reading and fitting n
to make new clips.
```bash
-git clone https://github.com/Patch-gg/diptych && cd diptych
+git clone https://github.com/Patch-gg/anymove && cd anymove
python3.11 -m venv .venv
.venv/bin/pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128
.venv/bin/pip install -r requirements.txt
@@ -58,7 +58,7 @@ MOCAP_CREATURE=assets/dragon .venv/bin/python -m mocap.solve examples/dragon_roa
```
The whole capture, from the AI clip to a GLB, with an example creature's mesh, trained reader and clip from the
-[release](https://github.com/Patch-gg/diptych/releases/tag/examples-v1) (13 MB mesh, 355 MB reader, 4 MB clip):
+[release](https://github.com/Patch-gg/anymove/releases/tag/examples-v1) (13 MB mesh, 355 MB reader, 4 MB clip):
```bash
tools/fetch_examples.sh dragon --readers --clips
@@ -94,21 +94,21 @@ The skills below walk an agent (or you) through each step, with the checks that
## agent skills.
-diptych is a [Claude Code](https://code.claude.com) plugin and its own marketplace:
+anymove is a [Claude Code](https://code.claude.com) plugin and its own marketplace:
```
-/plugin marketplace add Patch-gg/diptych
-/plugin install diptych@diptych
+/plugin marketplace add Patch-gg/anymove
+/plugin install anymove@anymove
```
| skill | for |
|---|---|
-| `diptych` | the method, setup, which step comes next, costs |
-| `diptych-rig` | mesh, facing and scale, placing joints in rig.json, building and painting the rig, framing |
-| `diptych-film` | writing the prompt, generating clips on fal, checking them before capture |
-| `diptych-reader` | rendering the dataset, training the reader, the exam in centimetres |
-| `diptych-capture` | reading, fitting, the quality gates, fixing bad fits, showcase renders |
-| `diptych-lora` | training and testing the MiniMax H3 LoRA on fal |
+| `anymove` | the method, setup, which step comes next, costs |
+| `anymove-rig` | mesh, facing and scale, placing joints in rig.json, building and painting the rig, framing |
+| `anymove-film` | writing the prompt, generating clips on fal, checking them before capture |
+| `anymove-reader` | rendering the dataset, training the reader, the exam in centimetres |
+| `anymove-capture` | reading, fitting, the quality gates, fixing bad fits, showcase renders |
+| `anymove-lora` | training and testing the MiniMax H3 LoRA on fal |
They're plain [Agent Skills](https://agentskills.io) (`skills/<name>/SKILL.md`), so other agents can load them from that
folder too.
diff --git a/docs/anymove.gif b/docs/anymove.gif
new file mode 100644
index 0000000..3d3f22f
Binary files /dev/null and b/docs/anymove.gif differ
diff --git a/docs/diptych.gif b/docs/diptych.gif
deleted file mode 100644
index fe2fcc8..0000000
Binary files a/docs/diptych.gif and /dev/null differ
diff --git a/docs/readme/header-day.png b/docs/readme/header-day.png
index 324d8a3..96c3b18 100644
Binary files a/docs/readme/header-day.png and b/docs/readme/header-day.png differ
diff --git a/docs/readme/header-evening.png b/docs/readme/header-evening.png
index 1b6ea19..25ff0cc 100644
Binary files a/docs/readme/header-evening.png and b/docs/readme/header-evening.png differ
diff --git a/docs/readme/make_art.py b/docs/readme/make_art.py
index 777804d..d34a387 100644
--- a/docs/readme/make_art.py
+++ b/docs/readme/make_art.py
@@ -58,8 +58,8 @@ def header(style):
img = Image.new('RGBA', (1600, 400), (0, 0, 0, 0))
d = ImageDraw.Draw(img)
draw_rects(d, logo('patch-lockup-white', ink, quiet), 0, 0, 6) # 3 px cells on screen
- d.text((0, 254), 'diptych.', font=pixel(144), fill=ink, anchor='ls') # 8 px cells: the headline size
- d.text((0, 352), 'any creature. any move.', font=funnel(52), fill=quiet, anchor='ls')
+ d.text((0, 254), 'anymove.', font=pixel(144), fill=ink, anchor='ls') # 8 px cells: the headline size
+ d.text((0, 352), 'motion capture for any creature, from one AI video.', font=funnel(52), fill=quiet, anchor='ls')
img.save(os.path.join(HERE, f'header-{style}.png'), optimize=True)
diff --git a/mocap/lora.py b/mocap/lora.py
index 1315baf..281cdc2 100644
--- a/mocap/lora.py
+++ b/mocap/lora.py
@@ -52,7 +52,7 @@ def resume(run_dir):
print("done", json.dumps(res)[:600], info, flush=True)
-RELEASE_LORA = "https://github.com/Patch-gg/diptych/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors"
+RELEASE_LORA = "https://github.com/Patch-gg/anymove/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors"
def lora_path(src):
diff --git a/skills/diptych-capture/SKILL.md b/skills/anymove-capture/SKILL.md
similarity index 92%
rename from skills/diptych-capture/SKILL.md
rename to skills/anymove-capture/SKILL.md
index c304580..d636cdb 100644
--- a/skills/diptych-capture/SKILL.md
+++ b/skills/anymove-capture/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych-capture
-description: Turn a two-camera AI clip into a game-ready animation with diptych - read every joint with the creature's trained reader, check that the two views agree, fit the rig in both views at once, and export a GLB with renders and a side-by-side comparison. Covers the quality gates (paw agreement, reprojection error), what to do when a fit fails, and three-quarter showcase renders. Use when someone has a diptych clip to capture, asks whether a capture is good, wants the GLB or renders of a move, or a fit looks wrong.
+name: anymove-capture
+description: Turn a two-camera AI clip into a game-ready animation with anymove - read every joint with the creature's trained reader, check that the two views agree, fit the rig in both views at once, and export a GLB with renders and a side-by-side comparison. Covers the quality gates (paw agreement, reprojection error), what to do when a fit fails, and three-quarter showcase renders. Use when someone has an anymove clip to capture, asks whether a capture is good, wants the GLB or renders of a move, or a fit looks wrong.
license: Apache-2.0
-compatibility: NVIDIA GPU for reading (a few GB), CPU or GPU for the fit, Blender 5.x for the GLB and renders, Python 3.11 venv from requirements.txt. Needs runs/<c>/reader.pt (diptych-reader) and runs/<c>/ref/ref.png (diptych-rig).
+compatibility: NVIDIA GPU for reading (a few GB), CPU or GPU for the fit, Blender 5.x for the GLB and renders, Python 3.11 venv from requirements.txt. Needs runs/<c>/reader.pt (anymove-reader) and runs/<c>/ref/ref.png (anymove-rig).
---
-# diptych-capture: clip in, animation out
+# anymove-capture: clip in, animation out
```bash
tools/capture.sh <c> runs/<c>/videos/roar/s1/h3max.mp4 roar
@@ -41,7 +41,7 @@ reprojection error px: {'front': 5.92, 'front_x_only': 2.55, 'side': 7.99}
```
- **Paw agreement score:** 0.8 or more, the views tell one story: keep it. 0.5-0.8: look at `showcase.mp4`, often one
- limb is off. Under 0.5: the video model animated the views differently; reroll the clip (diptych-film). `None` for
+ limb is off. Under 0.5: the video model animated the views differently; reroll the clip (anymove-film). `None` for
a foot means it barely moved (planted feet in a roar); judge that clip by the reprojection error instead.
- **Reprojection error** (layout pixels, how far the fitted rig's joints land from the readings): front_x_only of 1-3
px and side of 2-8 px were typical on good clips. Scale by the creature's size in the frame.
diff --git a/skills/diptych-film/SKILL.md b/skills/anymove-film/SKILL.md
similarity index 80%
rename from skills/diptych-film/SKILL.md
rename to skills/anymove-film/SKILL.md
index 52c7dae..9d65801 100644
--- a/skills/diptych-film/SKILL.md
+++ b/skills/anymove-film/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych-film
-description: Make the two-camera AI clip for a diptych creature - write the move's prompt from creature.json, send the reference frame to MiniMax H3 Max (or H3, Seedance, FLUX 3 Max) on fal, and check the clip kept the split screen, the limb colours and both views in step before capture. Use when someone wants a new move for a rigged diptych creature, when clips come back with swapped colours, a moving camera or a broken split screen, or when choosing a video model for two-camera clips.
+name: anymove-film
+description: Make the two-camera AI clip for an anymove creature - write the move's prompt from creature.json, send the reference frame to MiniMax H3 Max (or H3, Seedance, FLUX 3 Max) on fal, and check the clip kept the split screen, the limb colours and both views in step before capture. Use when someone wants a new move for a rigged anymove creature, when clips come back with swapped colours, a moving camera or a broken split screen, or when choosing a video model for two-camera clips.
license: Apache-2.0
-compatibility: Needs a fal.ai key (FAL_KEY in .env or the environment) and a creature already set up with diptych-rig (runs/<c>/ref/ref.png).
+compatibility: Needs a fal.ai key (FAL_KEY in .env or the environment) and a creature already set up with anymove-rig (runs/<c>/ref/ref.png).
---
-# diptych-film: one move, two cameras, one clip
+# anymove-film: one move, two cameras, one clip
The video model gets one image, `runs/<c>/ref/ref.png` (front view left, side view right, limbs painted), and a prompt
that pins everything except the action. Both views come from one generation, which is what keeps them in step.
@@ -45,11 +45,11 @@ Look at a few frames (or the whole clip) and reject it if:
- the split screen breaks: a view zooms, pans or cuts, the divider moves, the creature walks out of a view;
- a limb's colour changes or swaps sides (plain H3 swapped them on 2 of 5 wolf clips; the reader then labels the
- wrong legs); the LoRA (diptych-lora) kept them on 5 of 5;
+ wrong legs); the LoRA (anymove-lora) kept them on 5 of 5;
- the body count changes (an extra leg, a lost wing), or the creature turns to face another way;
- fire, smoke, dust or a weapon covers joints.
-The real gate is numeric and comes in capture: the paw agreement between the views (diptych-capture). What the
+The real gate is numeric and comes in capture: the paw agreement between the views (anymove-capture). What the
clips taught so far:
- Views disagree in the legs, not in time. Seedance's trot played the front legs in anti-phase between the views
diff --git a/skills/diptych-lora/SKILL.md b/skills/anymove-lora/SKILL.md
similarity index 92%
rename from skills/diptych-lora/SKILL.md
rename to skills/anymove-lora/SKILL.md
index da88096..eac84c6 100644
--- a/skills/diptych-lora/SKILL.md
+++ b/skills/anymove-lora/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych-lora
-description: Train and use a MiniMax H3 image-to-video LoRA on fal that keeps diptych's two-camera layout - split screen held, limb colours kept, both views in step. Builds the dataset by re-rendering clean captured motions (and their mirrors) from both cameras, packs it, runs a cheap debug pass, trains, and A/B tests against plain H3 with the same seeds. Use when plain video models keep swapping limb colours or breaking the split screen for a creature, or when someone asks about @Graalitoo's LoRA step, its cost or its training data.
+name: anymove-lora
+description: Train and use a MiniMax H3 image-to-video LoRA on fal that keeps anymove's two-camera layout - split screen held, limb colours kept, both views in step. Builds the dataset by re-rendering clean captured motions (and their mirrors) from both cameras, packs it, runs a cheap debug pass, trains, and A/B tests against plain H3 with the same seeds. Use when plain video models keep swapping limb colours or breaking the split screen for a creature, or when someone asks about @Graalitoo's LoRA step, its cost or its training data.
license: Apache-2.0
-compatibility: Needs a fal.ai key (FAL_KEY), Blender 5.x, and a creature with several good captures (diptych-capture). Training runs on fal, not locally.
+compatibility: Needs a fal.ai key (FAL_KEY), Blender 5.x, and a creature with several good captures (anymove-capture). Training runs on fal, not locally.
---
-# diptych-lora: teach the video model the layout
+# anymove-lora: teach the video model the layout
Graalitoo's addition to the method: a LoRA on MiniMax H3 (open weights) so two-camera clips come out right more
often. fal hosts both the trainer (`minimax/h3/i2v/trainer`, $0.01 a step) and LoRA inference
@@ -22,7 +22,7 @@ its licence is non-commercial; say which you're using.
## Steps
-1. Capture 15-20 good moves (diptych-film, diptych-capture). Prompts live in `prompts/<c>/<move>.txt` or
+1. Capture 15-20 good moves (anymove-film, anymove-capture). Prompts live in `prompts/<c>/<move>.txt` or
`prompts/<c>/train/<move>.txt`; the wolf has 14 training and 4 held-out moves already written.
2. Build the set (Blender renders each clip and its mirror, 5 at a time):
```bash
@@ -50,12 +50,12 @@ its licence is non-commercial; say which you're using.
```bash
tools/fetch_examples.sh wolf --clips
.venv/bin/python -m mocap.lora generate \
- https://github.com/Patch-gg/diptych/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors \
+ https://github.com/Patch-gg/anymove/releases/download/examples-v1/lora-wolf-twocam-h3.safetensors \
runs/wolf/ref/ref.png prompts/wolf/heldout/pounce.txt runs/wolf/videos/pounce/lora.mp4 1.0 42
```
`tools/fetch_examples.sh wolf --lora` downloads the file itself (131 MB), to run H3 with it yourself.
6. A/B test on held-out moves against plain H3 with the same seed, then capture both and compare paw agreement and
- reprojection error (diptych-capture).
+ reprojection error (anymove-capture).
## What it did for the wolf (one seed per move: a first read, not a benchmark)
diff --git a/skills/diptych-reader/SKILL.md b/skills/anymove-reader/SKILL.md
similarity index 92%
rename from skills/diptych-reader/SKILL.md
rename to skills/anymove-reader/SKILL.md
index e120896..1f567e1 100644
--- a/skills/diptych-reader/SKILL.md
+++ b/skills/anymove-reader/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych-reader
-description: Train the joint reader for one diptych creature - render about 2,200 random poses from both layout cameras with every joint's true position, fine-tune DINOv2 (ViT-B/14 with registers) plus a heatmap head on them, check the held-out error, and optionally run the exam (a made-up motion with known joints, read back and scored in centimetres). Use after diptych-rig and before diptych-capture, when a reader misses or swaps joints on real clips, or when planning GPU time for dataset rendering and training.
+name: anymove-reader
+description: Train the joint reader for one anymove creature - render about 2,200 random poses from both layout cameras with every joint's true position, fine-tune DINOv2 (ViT-B/14 with registers) plus a heatmap head on them, check the held-out error, and optionally run the exam (a made-up motion with known joints, read back and scored in centimetres). Use after anymove-rig and before anymove-capture, when a reader misses or swaps joints on real clips, or when planning GPU time for dataset rendering and training.
license: Apache-2.0
compatibility: NVIDIA GPU with CUDA (about 12 GB at batch 20, so a 24 GB card), Blender 5.x, Python 3.11 venv from requirements.txt. Downloads facebook/dinov2-with-registers-base on first use.
---
-# diptych-reader: teach DINOv2 one creature
+# anymove-reader: teach DINOv2 one creature
The reader is trained on the creature itself, so it knows every joint name and every limb colour. Data is free:
Blender poses the rig at random inside its joint limits and records where each joint lands in both views.
@@ -59,7 +59,7 @@ The wolf scored 0.94 cm mean 3D joint error on a 1.6 m body (median 0.63 cm, paw
## When the reader struggles on real clips
-- Joints jump between limbs: the limb colours in the clip drifted from the paint (diptych-film), or two limbs share
+- Joints jump between limbs: the limb colours in the clip drifted from the paint (anymove-film), or two limbs share
similar colours. Repaint far-apart colours and retrain.
- A pose the clip shows never appeared in training (lying on its side, a full curl): widen that bone's limits in
rig.json, rebuild the rig, render and train again. Random poses don't cover lying-down; captured motions would.
diff --git a/skills/diptych-rig/SKILL.md b/skills/anymove-rig/SKILL.md
similarity index 93%
rename from skills/diptych-rig/SKILL.md
rename to skills/anymove-rig/SKILL.md
index 1ec46df..d8b502f 100644
--- a/skills/diptych-rig/SKILL.md
+++ b/skills/anymove-rig/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych-rig
-description: Set up a new creature for diptych - get a textured mesh, scale and face it, place its skeleton in rig.json with grid views, build and skin the rig in Blender, paint each limb its own colour, choose the two-camera framing, and render the reference frame the video model starts from. Use when adding a creature to diptych, when a rig tears or deforms badly, when the creature is cut off or tiny in the two-camera frame, or when someone asks how to rig a non-humanoid for diptych.
+name: anymove-rig
+description: Set up a new creature for anymove - get a textured mesh, scale and face it, place its skeleton in rig.json with grid views, build and skin the rig in Blender, paint each limb its own colour, choose the two-camera framing, and render the reference frame the video model starts from. Use when adding a creature to anymove, when a rig tears or deforms badly, when the creature is cut off or tiny in the two-camera frame, or when someone asks how to rig a non-humanoid for anymove.
license: Apache-2.0
compatibility: Blender 5.x on PATH (or BLENDER=...), Python 3.11 venv from requirements.txt. A fal.ai key only if the mesh is generated with Tripo.
---
-# diptych-rig: one creature, ready to film
+# anymove-rig: one creature, ready to film
Everything for creature `<c>` lives in `assets/<c>/`: `model.glb` (the mesh), `creature.json` (framing, paint, words
for the prompt), `rig.json` (the skeleton). Copy a close example to start: `assets/wolf` (quadruped), `assets/dragon`
@@ -13,7 +13,7 @@ for the prompt), `rig.json` (the skeleton). Copy a close example to start: `asse
## 1. The mesh
-- Any textured GLB works. diptych's examples are Tripo H3.1 (`tripo3d/h3.1/image-to-3d` on fal, $0.40 with HD
+- Any textured GLB works. anymove's examples are Tripo H3.1 (`tripo3d/h3.1/image-to-3d` on fal, $0.40 with HD
texture) from one FLUX 3 concept image: a plain 3/4 view of the creature on a neutral background, whole body in
frame, limbs apart and clearly separate.
- Prefer a design where every limb reads separately. Fire, smoke, capes, weapons and long hair hide joints from the
@@ -94,10 +94,10 @@ with wings up needs more headroom than one standing. Re-run `tools/rig.sh <c>` a
Also in `creature.json`: `name` ("spider"), `noun` ("giant spider"), `describe` ("a black armoured giant spider with
bone-coloured markings"), `colour_code` ("first left leg pink, first right leg blue; ..."), `count` ("Exactly eight
legs, two fangs, one body, one round abdomen."), `flank` ("flank" or "side") and `limbs` ("legs" or "limbs").
-`tools/make_prompt.py` builds every prompt from these (diptych-film).
+`tools/make_prompt.py` builds every prompt from these (anymove-film).
Last, `"camera": {"cam": "-5.4,-6.0,1.4", "target": "0,-0.3,1.5", "lens": 32}` is the 3/4 view that
-`tools/capture.sh` renders each move from (diptych-capture says how to frame it). Without it, the wolf's applies.
+`tools/capture.sh` renders each move from (anymove-capture says how to frame it). Without it, the wolf's applies.
## Failure modes seen
diff --git a/skills/diptych/SKILL.md b/skills/anymove/SKILL.md
similarity index 74%
rename from skills/diptych/SKILL.md
rename to skills/anymove/SKILL.md
index 46cd6b3..b77a6ab 100644
--- a/skills/diptych/SKILL.md
+++ b/skills/anymove/SKILL.md
@@ -1,11 +1,11 @@
---
-name: diptych
-description: Motion capture for any creature from one AI video, with the diptych repo. A video model films a rigged 3D creature from two cameras in one frame, a DINOv2 joint reader trained on that creature finds every joint in both views, and the rig is fitted to them and exported as a GLB. Use when someone wants to animate a non-human creature (dragon, spider, wolf, monster) from a text prompt or an AI video, asks how diptych works, or needs to know which step comes next; it routes to diptych-rig, diptych-film, diptych-reader, diptych-capture and diptych-lora.
+name: anymove
+description: Motion capture for any creature from one AI video, with the anymove repo. A video model films a rigged 3D creature from two cameras in one frame, a DINOv2 joint reader trained on that creature finds every joint in both views, and the rig is fitted to them and exported as a GLB. Use when someone wants to animate a non-human creature (dragon, spider, wolf, monster) from a text prompt or an AI video, asks how anymove works, or needs to know which step comes next; it routes to anymove-rig, anymove-film, anymove-reader, anymove-capture and anymove-lora.
license: Apache-2.0
compatibility: Needs an NVIDIA GPU with CUDA (24 GB to train a reader at batch 20), Blender 5.x, Python 3.11, and a fal.ai key for video generation.
---
-# diptych
+# anymove
Animate any rigged creature from one AI video. The trick is the frame: one image holds two orthographic views of the
creature at the same scale, front on the left and side on the right, sharing the vertical axis. A joint sits on the
@@ -16,20 +16,20 @@ height, and together they pin the joint in 3D. The method is @CanItCode's; the L
| step | once per | what happens | skill |
|---|---|---|---|
-| 1. mesh | creature | a textured 3D model (Tripo H3.1 from one image, or your own GLB), facing -Y, real size | diptych-rig |
-| 2. skeleton | creature | joints placed in `assets/<c>/rig.json`, rig built and skinned in Blender | diptych-rig |
-| 3. limb colours | creature | each limb painted its own colour so left and right never swap | diptych-rig |
-| 4. two cameras | move | the painted reference frame plus a strict prompt go to MiniMax H3 Max on fal | diptych-film |
-| 5. teach a reader | creature | 2,200 random poses rendered with known joints; DINOv2 learns this creature | diptych-reader |
-| 6. read every joint | clip | the reader finds every joint in both views, every frame | diptych-capture |
-| 7. two views, 3D | clip | shared heights tie the views; the pair gives each joint in 3D | diptych-capture |
-| 8. the rig follows | clip | forward kinematics in torch fitted to both views; GLB out | diptych-capture |
-| optional | creature | a MiniMax H3 LoRA that keeps the two-camera layout and colours more reliably | diptych-lora |
+| 1. mesh | creature | a textured 3D model (Tripo H3.1 from one image, or your own GLB), facing -Y, real size | anymove-rig |
+| 2. skeleton | creature | joints placed in `assets/<c>/rig.json`, rig built and skinned in Blender | anymove-rig |
+| 3. limb colours | creature | each limb painted its own colour so left and right never swap | anymove-rig |
+| 4. two cameras | move | the painted reference frame plus a strict prompt go to MiniMax H3 Max on fal | anymove-film |
+| 5. teach a reader | creature | 2,200 random poses rendered with known joints; DINOv2 learns this creature | anymove-reader |
+| 6. read every joint | clip | the reader finds every joint in both views, every frame | anymove-capture |
+| 7. two views, 3D | clip | shared heights tie the views; the pair gives each joint in 3D | anymove-capture |
+| 8. the rig follows | clip | forward kinematics in torch fitted to both views; GLB out | anymove-capture |
+| optional | creature | a MiniMax H3 LoRA that keeps the two-camera layout and colours more reliably | anymove-lora |
## Before anything
-1. Work in a diptych checkout. If the working directory has no `tools/capture.sh`, clone it:
- `git clone https://github.com/Patch-gg/diptych && cd diptych`.
+1. Work in an anymove checkout. If the working directory has no `tools/capture.sh`, clone it:
+ `git clone https://github.com/Patch-gg/anymove && cd anymove`.
2. Set up once: `python3.11 -m venv .venv`, then
`.venv/bin/pip install torch==2.11.0 --index-url https://download.pytorch.org/whl/cu128` and
`.venv/bin/pip install -r requirements.txt`. Blender 5.x must be on PATH, or set `BLENDER=/path/to/blender`.
@@ -41,13 +41,13 @@ height, and together they pin the joint in 3D. The method is @CanItCode's; the L
## Which step next
- No `assets/<c>/model.glb`: make or fetch the mesh. Examples: `tools/fetch_examples.sh <c>` (wolf, dragon, spider,
- demon). New creature: diptych-rig.
+ demon). New creature: anymove-rig.
- No `assets/<c>/rigged/rigged.blend` or `runs/<c>/ref/ref.png`: `tools/prep.sh <c>` then `tools/rig.sh <c>`
- (diptych-rig).
-- No `runs/<c>/reader.pt`: diptych-reader (about 45 min on one RTX 4090), or `tools/fetch_examples.sh <c> --readers`.
-- No clip for the move: diptych-film (about $0.26 a 5 s clip).
-- A clip: diptych-capture, `tools/capture.sh <c> <video.mp4> <name>`.
-- Clips keep swapping limb colours or the halves drift apart: diptych-lora.
+ (anymove-rig).
+- No `runs/<c>/reader.pt`: anymove-reader (about 45 min on one RTX 4090), or `tools/fetch_examples.sh <c> --readers`.
+- No clip for the move: anymove-film (about $0.26 a 5 s clip).
+- A clip: anymove-capture, `tools/capture.sh <c> <video.mp4> <name>`.
+- Clips keep swapping limb colours or the halves drift apart: anymove-lora.
## Try it without a GPU
diff --git a/tools/capture.sh b/tools/capture.sh
index 226e234..415ab61 100755
--- a/tools/capture.sh
+++ b/tools/capture.sh
@@ -8,7 +8,7 @@ source "$(dirname "$0")/_env.sh"
need_creature "${1:-}"
c=$1; video=${2:?video}; name=${3:?name}; views=${4:-persp,side}
rig=assets/$c/rigged; out=runs/$c/clips/$name
-# the 3/4 showcase camera: creature.json "camera" (frame it to the creature's biggest move; see diptych-capture)
+# the 3/4 showcase camera: creature.json "camera" (frame it to the creature's biggest move; see anymove-capture)
read -r cam target lens < <($PY -c "import json; c = json.load(open('assets/$c/creature.json')).get('camera', {}); print(c.get('cam', '-3.2,-3.0,1.6'), c.get('target', '0,0,0.5'), c.get('lens', 50))")
mkdir -p "$out"
$PY -m mocap.detect "$video" "runs/$c/reader.pt" "runs/$c/ref/ref.png" "$out/det.npz" "$out/overlay.mp4" | tail -1
diff --git a/tools/examples.txt b/tools/examples.txt
index 3deed5a..97658ba 100644
--- a/tools/examples.txt
+++ b/tools/examples.txt
@@ -1,5 +1,5 @@
# Release assets for the example creatures: sha256, bytes, asset name on the release, where it goes in the repo.
-# Release: https://github.com/Patch-gg/diptych/releases/tag/examples-v1 (tools/fetch_examples.sh downloads and checks these)
+# Release: https://github.com/Patch-gg/anymove/releases/tag/examples-v1 (tools/fetch_examples.sh downloads and checks these)
# mesh: the Tripo H3.1 model each assets/<creature>/rig.json was placed on
7d25b0f75c03f84d11bef0cde823d2039637115fd2250eba23e1d3ff99ffdce4 10726072 wolf.glb assets/wolf/model.glb
bf2f97d4220cad1c208580fde139990da7b9552717f7962c9e9f081ce79d8128 13256908 dragon.glb assets/dragon/model.glb
@@ -19,5 +19,5 @@ e06ebcaf4eb5fad98939c95ff682e9988390626a14670a61b158cb162a8e049f 245855 ref-wolf
f852e1386fcd67fc4fc9e438c2b1c9e713392a05603f61f384b0ec98ecd90842 3444393 clip-spider-rear.mp4 runs/spider/videos/rear/s1/h3max.mp4
264493c12adecc09069b5e5d740abecfc51169b5cc57484a92121b7a4543fdf7 282439 ref-demon.png runs/demon/ref/ref.png
2062ef188701f01b11c38dab8a768ce321a0e42a5375e2a1eb9cd390be7ef951 4762986 clip-demon-flap.mp4 runs/demon/videos/flap/s1/h3max.mp4
-# lora: the wolf's two-camera LoRA on MiniMax H3 (1,200 steps; skills/diptych-lora). MiniMax H3's community licence applies, not Apache-2.0
+# lora: the wolf's two-camera LoRA on MiniMax H3 (1,200 steps; skills/anymove-lora). MiniMax H3's community licence applies, not Apache-2.0
2eaa8a8e558d6468443422719a48575be9182c03fe4deae249322d07be81eded 131229792 lora-wolf-twocam-h3.safetensors runs/wolf/lora/twocam_wolf_h3_lora.safetensors
diff --git a/tools/fetch_examples.sh b/tools/fetch_examples.sh
index 6ab4167..e53894f 100755
--- a/tools/fetch_examples.sh
+++ b/tools/fetch_examples.sh
@@ -7,7 +7,7 @@
# --lora: the wolf's MiniMax H3 LoRA (131 MB; MiniMax H3's community licence), runs/wolf/lora/
set -euo pipefail
cd "$(dirname "$0")/.."
-URL=${DIPTYCH_RELEASE_URL:-https://github.com/Patch-gg/diptych/releases/download/examples-v1}
+URL=${ANYMOVE_RELEASE_URL:-https://github.com/Patch-gg/anymove/releases/download/examples-v1}
kinds=(glb); creatures=()
for a in "$@"; do
case $a in