ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.
ReShot — user guide
Install
Requirements: Python ≥ 3.10, PyTorch ≥ 2.1 (CPU, CUDA or Apple MPS), ffmpeg.
# with your own ffmpeg on PATH
pip install reshot
# no ffmpeg? this extra bundles a static binary
pip install "reshot[ffmpeg]"
# skeletons (--control pose) need ONNX Runtime; on an NVIDIA card use onnxruntime-gpu instead
pip install "reshot[pose]"
PyTorch is not pinned to a CUDA version; install the build for your machine first if
pip picks the wrong one (see https://pytorch.org/get-started/locally/).
Model weights download from Hugging Face on first run into ~/.cache/huggingface: 111 MB for
depth, and for --control pose two ONNX files (yolox_l.onnx + dw-ll_ucoco_384.onnx, ~340 MB)
from yzd-v/DWPose. Offline machines: copy that cache, or pass --checkpoint /path/video_depth_anything_vits.pth (depth) / --checkpoint-dir /folder/with/both.onnx (pose).
In China: export HF_ENDPOINT=https://hf-mirror.com.
Web page
reshot with no arguments starts a local web page and opens it in your browser. It is the
same pipeline behind a form: drop a clip, choose what to make (depth map / skeleton / lines),
the target model and quality, run, preview reference and result side by side, download. Nothing
leaves your machine; the server listens on 127.0.0.1 only. Results are saved to ~/ReShot with
<clip>_<control>_<target>.mp4 names (never overwriting an earlier result), a .json of metrics
next to each, and for skeletons a .keypoints.json with the pose data (a download button too).
reshot # http://127.0.0.1:8765 (next free port if taken)
reshot web --port 9000 --out ./depth # choose port and folder
reshot web --no-browser # print the URL only, e.g. over SSH
Each model loads once when its first job runs and stays loaded; later runs start immediately. Jobs queue one after another (one GPU).
Command line
reshot INPUT -o OUTPUT [options]
Control type
| option | default | notes |
|---|---|---|
--control depth|pose|canny |
depth |
What the output shows. Several at once with commas (depth,pose), then -o must be a directory and files are named <name>_<control>.mp4. |
--no-hands |
off | pose: body only, skip the 21-point hands. |
--face |
off | pose: also draw the 68 face points. Off because the face shape is what ReShot is meant to throw away. |
--no-smooth |
off | pose: raw per-frame output — no identity tracking, no visibility hysteresis, no One-Euro smoothing. For debugging the estimator. |
--pose-detect-every N |
3 |
pose: run the person detector every N frames and follow the skeletons in between; cuts and unreliable skeletons re-detect at once. 1 = every frame, ~1.5× slower on CPU. Measured N=3 vs 1 on the 289-frame demo: 280 frames identical people count, 80 % of joints within 2 px, 91 % within 5 px. |
--keypoints PATH |
– | pose: also write the skeletons as JSON, format reshot-pose/1 (see below). |
--checkpoint-dir DIR |
– | pose: folder with yolox_l.onnx and dw-ll_ucoco_384.onnx, skips the download. |
--canny LOW,HIGH |
100,200 |
canny: hysteresis thresholds. Lower LOW = more lines; higher HIGH = only strong edges. |
Depth model
| option | default | notes |
|---|---|---|
--model small|base|large |
small |
Only small is Apache-2.0. base/large are CC-BY-NC-4.0: allowed for research, a warning is printed. |
--device auto|cuda|mps|cpu |
auto |
fp16 on CUDA, fp32 elsewhere (fp16 on MPS is unusably slow). |
--quality Q |
fast |
Resolution the model works at. fast = 644×364 for 16:9 / 364×644 for 9:16 (full = 924×518 / 518×924 ( |
--input-size N |
— | Expert: exact model short side, multiple of 14; overrides --quality. |
--checkpoint PATH |
– | Local .pth, skips the download. |
Output
| option | default | notes |
|---|---|---|
--target none|seedance|h3|wan |
none |
fps + frame-size preset, see Targets. |
--fps N |
preset or source | Frames are chosen by timestamp, so 30 → 24 yields 24, not 30. Never upsamples. |
--max-res N |
1280 |
Cap on the output's longer side. Depth inference always runs at model resolution regardless; pose and canny read frames at the output size. |
--max-frames N |
all | Stop after N source frames. |
--invert |
off | far = white. Default is near = white (what depth ControlNets expect). |
--clip PCT |
0 |
Trim PCT % from each tail before scaling to 0–255, so one hot pixel can't crush contrast. 0 = exact min/max. |
--gamma G |
1.0 |
> 1 darkens mid-tones: more separation close to the camera. |
--crf N |
12 |
x264 quality. Generous on purpose — banding in a control video becomes jitter in the generated clip. |
--npz PATH |
– | depth: raw float32 depth at processing resolution (depths[T,H,W], fps). |
--metrics PATH |
– | JSON with config, plan, timings, peak RSS, output size. |
Safety
| option | notes |
|---|---|
--force |
Run even when the host-RAM estimate exceeds half of physical RAM. |
Exit codes
| code | meaning | typical fix |
|---|---|---|
| 0 | done | |
| 1 | unexpected error (a bug — please report with -v output) |
|
| 2 | bad input or arguments | check the path / the option value in the message |
| 3 | RAM estimate over budget | use the suggested --max-frames, split the clip, or --force |
| 4 | ffmpeg missing | pip install imageio-ffmpeg or install ffmpeg |
| 5 | weights could not be obtained | set HF_ENDPOINT to a mirror, or --checkpoint |
Targets
--target |
fps | frame size | max length | notes |
|---|---|---|---|---|
none |
source | even | – | |
seedance |
24 | multiple of 16 (centre crop), ≥ 407,696 px (warns) | 15 s (warns) | Seedance 2.0 / 2.5 reference video (@视频N motion + camera) |
h3 |
24 | multiple of 32 (centre crop) | 15 s (warns) | MiniMax-H3-Fun-Controlnet-Union depth / pose / canny input, or reference video |
wan |
16 | multiple of 16 | – | Wan 2.1 VACE control video; values from the docs, not yet verified end-to-end |
Frame size is cropped, never padded: a padded black border reads to the generator as a far wall.
Python API
from pathlib import Path
from reshot import RunConfig, run, run_many, plan
cfg = RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3", metrics=Path("m.json"))
print(plan(cfg)) # sizes, frame counts, RAM estimate — no decoding yet
result = run(cfg) # same code path as the CLI
print(result.frames, result.fps, result.ms_per_frame, result.peak_rss_bytes)
Lower-level pieces, for your own post-processing:
from reshot import extract, to_gray, upsample_frames, write_gray_video
depths, fps = extract("in.mp4") # float32 [T, H, W], larger = closer
gray = to_gray(depths, clip_percent=0.5, gamma=1.2) # uint8 [T, H, W], whole-clip normalised
write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 736))
Errors that are the user's to fix are subclasses of reshot.ReshotError and carry an
exit_code; anything else is a bug.
RunConfig(..., backend="fake") runs the entire pipeline with a synthetic depth field (or, for
control="pose", two walking stick figures) and no model — handy for testing your own integration
in milliseconds.
Pose
from reshot import RunConfig, run
run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), target="h3", control="pose", keypoints=Path("pose.json")))
The pieces, for your own pipeline:
from reshot import read_video, write_rgb_video
from reshot.backends import get_backend
from reshot.pose.tracking import stabilise, track, smooth, apply_hysteresis
from reshot.pose.skeleton import render_frame, LAYOUT
frames, fps = read_video("in.mp4") # uint8 RGB [T, H, W, 3]
clip = get_backend("dwpose").infer(frames, fps) # PoseClip: keypoints[t] = [N, 134, 2] px, scores[t] = [N, 134]
stabilise(clip) # = track → apply_hysteresis → smooth, in place
rgb = [render_frame(k, s, 480, 864, src_w=clip.width, src_h=clip.height) for k, s in zip(clip.keypoints, clip.scores)]
write_rgb_video(np.stack(rgb), "pose.mp4", fps)
clip.to_json_dict() # the same structure as --keypoints
Keypoint layout (LAYOUT): 0–17 body in OpenPose-18 order (nose, neck, R-shoulder, R-elbow,
R-wrist, L-shoulder, L-elbow, L-wrist, R-hip, R-knee, R-ankle, L-hip, L-knee, L-ankle, R-eye,
L-eye, R-ear, L-ear), 18–23 feet, 24–91 face (68), 92–112 left hand (21), 113–133 right hand (21).
Scores are 0–1; a joint below 0.3 is not drawn.
--keypoints JSON ("format": "reshot-pose/1"): width, height, fps, layout, then
frames[t].people[n] with id (stable across frames), keypoints (134 × [x, y] in pixels of the
processing frame — width × height — not the cropped output) and scores (134). Written after
tracking / hysteresis / smoothing, so it matches the video exactly.
Canny
from reshot.edges import canny_frame
edges = canny_frame(rgb_frame, out_h, out_w, low=100, high=200) # uint8 [out_h, out_w], 0 or 255
Recipes
Seedance 2.0 / 2.5 (reference video)
reshot ref.mp4 -o ref_depth.mp4 --target seedance(24 fps, ×16, ≤ 15 s).Submit it as a reference video (API:
role: "reference_video"; in the console, attach it as a video reference) and address it in the prompt as@视频1, asking for its motion and camera:参考@视频1的动作与运镜。两名武者在雨夜屋顶对决,黑色劲装,冷蓝月光,电影感。Describe only the look — cast, wardrobe, lighting, style. Staging comes from the video.
What the official reference-video spec asks for, and what the preset does about it:
official requirement preset mp4 / mov, H.264 or H.265 H.264 mp4, yuv420p 24 – 60 fps 24 fps 2.0: 2 – 15 s per clip, ≤ 15 s total · 2.5: 2 – 30 s per clip, ≤ 30 s total warns above 15 s width × height ≥ 407,696 px, each side 300 – 6,000 px, aspect 0.4 – 2.5 warns below the pixel floor — raise --max-resor use a larger source≤ 200 MB CRF 12 keeps a 15 s 1080p clip well under Seedance 2.5 allows up to 30 s; pass
--fps 24without--targetand check the size yourself if you need the longer window.
MiniMax H3 Fun ControlNet (ComfyUI)
reshot ref.mp4 -o ref_depth.mp4 --target h3(24 fps, ×32, ≤ 15 s) — or--control pose/--control cannyfor the other two conditions the Union weights accept.- In ComfyUI with the Fun ControlNet Union weights, load the control video, choose the matching condition (depth / pose / canny), write your prompt for the look (cast, wardrobe, lighting, style). Staging and camera come from the control video. Or skip the files: the ComfyUI-ReShot pack has ReShot Depth / Pose / Canny Video nodes that take a Load Video node directly.
- Keep the generator's resolution equal to the control video's, or let the node resize — but do not pad.
Wan 2.1 VACE
reshot ref.mp4 -o ref_depth.mp4 --target wan, then use it as the depth control video in your
VACE workflow. Report back if the fps/size values need adjusting — this preset has not been
verified end-to-end yet.
Batch
Give reshot several inputs and a directory; the model loads once for all of them and
each clip comes out as <name>_depth.mp4. --metrics / --npz become directories too
(<name>.json / <name>.npz). A clip that fails is reported and skipped; the exit code
is that of the first failure.
reshot clips/*.mp4 -o depth/ --target seedance --metrics depth/metrics/
From Python, run_many() is the same thing:
from pathlib import Path
from reshot import RunConfig, run_many
cfgs = [RunConfig(input=p, output=Path("depth") / f"{p.stem}_depth.mp4", target="seedance") for p in Path("clips").glob("*.mp4")]
for r in run_many(cfgs): # RunResult, or the ReshotError for a clip that failed
print(r)
Long videos
Everything is held in RAM for the whole clip (frames + depth). reshot estimates the need and
refuses beyond half of physical RAM with a --max-frames suggestion. For a 3-minute clip, split
it into ≤ 15 s pieces first (ffmpeg -ss … -t 15 …); the generators want short control videos anyway.
Apple Silicon
Works out of the box (--device auto picks MPS). Two things measured on an M2 Max:
torch 2.9.1 is ~3.5× faster than 2.6.0, and fp16 is disabled on MPS because it does not finish.
Expect ~0.5 s/frame at 736×1280; a 12 s clip takes ~2.5 minutes.
Reproducible depth for research
--npz writes the raw float depth at processing resolution. Combined with --metrics (which
records the exact config and plan) a run is fully reproducible.
How it works
input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode
depth: at model resolution ─▶ Video Depth Anything (32-frame windows, 10-frame overlap, alignment)
─▶ whole-clip normalisation → uint8, near = white ─▶ per-frame upscale to output size
pose: at output size ─▶ YOLOX-L person boxes (every 3rd frame; skeleton-tracked boxes between)
─▶ RTMPose whole-body, 133 points per person ─▶ OpenPose-18 layout + neck
─▶ track ids (IoU) ─▶ visibility hysteresis (on > 0.3, off < 0.2) ─▶ One-Euro per joint
─▶ OpenPose render: limbs as ellipses at 60 %, joints on top, hands as rainbow edges
canny: at output size ─▶ grey ─▶ 3×3 Gaussian ─▶ cv2.Canny(low, high)
─▶ centre-crop to target multiple ─▶ x264 (grey for depth / canny, rgb24 for pose)
- Model: Video Depth Anything
(CVPR 2025). Its temporal module is what keeps the depth stable across frames; single-image
models flicker. The model code is vendored unmodified (two import lines) under
third_party/. - Resolution: the model works at
--qualitysize — 644×364 (fast) or 924×518 (full) for a 16:9 clip. Feeding larger frames only inflates the float depth kept per frame; reshot decodes at that size and upscales the 8-bit result to the source size. - Normalisation:
(d − min) / (max − min)over all frames; optional percentile clip. - Pose model: DWPose (ICCV 2023 workshop), the
estimator behind
controlnet_aux's pose images — so the skeletons look like what the video ControlNets were trained on. Apache-2.0 code and weights. The detector gets BGR, the estimator RGB with ImageNet normalisation, matching their mmdet / mmpose training configs. - Encoding:
libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart.
Troubleshooting
| symptom | cause | fix |
|---|---|---|
ModuleNotFoundError: torchvision |
torch installed without torchvision | pip install torchvision (matching your torch build) |
| exit 4 / "ffmpeg not found" | no ffmpeg | pip install imageio-ffmpeg or install ffmpeg |
| exit 5 / weights download fails | no route to huggingface.co | export HF_ENDPOINT=https://hf-mirror.com, or --checkpoint |
| exit 3 | clip too long for RAM | use the suggested --max-frames, split the clip |
| very slow on Mac | torch < 2.9, or fp16 forced | pip install -U torch; never force fp16 on MPS |
output is 30 fps although --fps 24 |
source fps < 24 | reshot never upsamples |
| output smaller than source | --target crop or --max-res |
expected; see Targets |
| banding in the depth map | --crf raised |
keep ≤ 14 |
| people look flat / merged with background | scene has little depth range | try --clip 0.5 --gamma 1.3 |
pose needs ONNX Runtime |
the pose extra is not installed |
pip install "reshot[pose]" (or onnxruntime-gpu) |
skeleton video is black / no person found |
the detector saw nobody: people too small, or a drawing / animal the person detector does not know | try a tighter crop of the source; for non-humans use depth |
| a limb pops in and out | the estimator is unsure of it | that is the hysteresis already; --no-smooth to see the raw output, or accept — the model reads a missing limb better than a jumping one |
| skeleton lags a fast move | One-Euro smoothing | --no-smooth; lag is small by design (beta 0.5) |
FAQ
Is Small good enough? For a control video, yes: the generator reads silhouette, size and motion, not fine geometry, and Small's silhouettes are clean (see the demo GIF). We have not benchmarked Base/Large against Small ourselves; they are CC-BY-NC, so Small is also the only choice for commercial work.
Why near = white? Depth ControlNets (H3 Fun, Wan VACE, SD depth) were trained on inverse-depth
maps where closer is brighter. --invert if a tool wants the opposite.
Can it output pose / canny / normals? Pose and canny since 0.5.0 (--control pose, --control canny). Normals, HED and MLSD are not planned unless someone asks.
Does it remove the background / keep only people? Not yet (--people-only is on the roadmap).
The full-frame depth already carries no identity or style.
GPU memory? Small in fp16 needs ~2 GB at the default input size; a 4090 is idle most of the time.