Skip to content

ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.

17.6 KB
Raw Blame

ReShot — user guide

中文版

  1. Install
  2. Command line
  3. Python API
  4. Recipes
  5. How it works
  6. Troubleshooting
  7. FAQ

Install

Requirements: Python ≥ 3.10, PyTorch ≥ 2.1 (CPU, CUDA or Apple MPS), ffmpeg.

# with your own ffmpeg on PATH
pip install reshot

# no ffmpeg? this extra bundles a static binary
pip install "reshot[ffmpeg]"

# skeletons (--control pose) need ONNX Runtime; on an NVIDIA card use onnxruntime-gpu instead
pip install "reshot[pose]"

PyTorch is not pinned to a CUDA version; install the build for your machine first if pip picks the wrong one (see https://pytorch.org/get-started/locally/).

Model weights download from Hugging Face on first run into ~/.cache/huggingface: 111 MB for depth, and for --control pose two ONNX files (yolox_l.onnx + dw-ll_ucoco_384.onnx, ~340 MB) from yzd-v/DWPose. Offline machines: copy that cache, or pass --checkpoint /path/video_depth_anything_vits.pth (depth) / --checkpoint-dir /folder/with/both.onnx (pose). In China: export HF_ENDPOINT=https://hf-mirror.com.

Web page

reshot with no arguments starts a local web page and opens it in your browser. It is the same pipeline behind a form: drop a clip, choose what to make (depth map / skeleton / lines), the target model and quality, run, preview reference and result side by side, download. Nothing leaves your machine; the server listens on 127.0.0.1 only. Results are saved to ~/ReShot with <clip>_<control>_<target>.mp4 names (never overwriting an earlier result), a .json of metrics next to each, and for skeletons a .keypoints.json with the pose data (a download button too).

reshot                                   # http://127.0.0.1:8765 (next free port if taken)
reshot web --port 9000 --out ./depth     # choose port and folder
reshot web --no-browser                  # print the URL only, e.g. over SSH

Each model loads once when its first job runs and stays loaded; later runs start immediately. Jobs queue one after another (one GPU).

Command line

reshot INPUT -o OUTPUT [options]

Control type

option default notes
--control depth|pose|canny depth What the output shows. Several at once with commas (depth,pose), then -o must be a directory and files are named <name>_<control>.mp4.
--no-hands off pose: body only, skip the 21-point hands.
--face off pose: also draw the 68 face points. Off because the face shape is what ReShot is meant to throw away.
--no-smooth off pose: raw per-frame output — no identity tracking, no visibility hysteresis, no One-Euro smoothing. For debugging the estimator.
--pose-detect-every N 3 pose: run the person detector every N frames and follow the skeletons in between; cuts and unreliable skeletons re-detect at once. 1 = every frame, ~1.5× slower on CPU. Measured N=3 vs 1 on the 289-frame demo: 280 frames identical people count, 80 % of joints within 2 px, 91 % within 5 px.
--keypoints PATH – pose: also write the skeletons as JSON, format reshot-pose/1 (see below).
--checkpoint-dir DIR – pose: folder with yolox_l.onnx and dw-ll_ucoco_384.onnx, skips the download.
--canny LOW,HIGH 100,200 canny: hysteresis thresholds. Lower LOW = more lines; higher HIGH = only strong edges.

Depth model

option default notes
--model small|base|large small Only small is Apache-2.0. base/large are CC-BY-NC-4.0: allowed for research, a warning is printed.
--device auto|cuda|mps|cpu auto fp16 on CUDA, fp32 elsewhere (fp16 on MPS is unusably slow).
--quality Q fast Resolution the model works at. fast = 644×364 for 16:9 / 364×644 for 9:16 (3 GB VRAM); full = 924×518 / 518×924 (11 GB, 2.5× slower, sharper fine detail: mean Δ 5.3/255, edges −4.5 %). Output size is unaffected.
--input-size N — Expert: exact model short side, multiple of 14; overrides --quality.
--checkpoint PATH – Local .pth, skips the download.

Output

option default notes
--target none|seedance|h3|wan none fps + frame-size preset, see Targets.
--fps N preset or source Frames are chosen by timestamp, so 30 → 24 yields 24, not 30. Never upsamples.
--max-res N 1280 Cap on the output's longer side. Depth inference always runs at model resolution regardless; pose and canny read frames at the output size.
--max-frames N all Stop after N source frames.
--invert off far = white. Default is near = white (what depth ControlNets expect).
--clip PCT 0 Trim PCT % from each tail before scaling to 0–255, so one hot pixel can't crush contrast. 0 = exact min/max.
--gamma G 1.0 > 1 darkens mid-tones: more separation close to the camera.
--crf N 12 x264 quality. Generous on purpose — banding in a control video becomes jitter in the generated clip.
--npz PATH – depth: raw float32 depth at processing resolution (depths[T,H,W], fps).
--metrics PATH – JSON with config, plan, timings, peak RSS, output size.

Safety

option notes
--force Run even when the host-RAM estimate exceeds half of physical RAM.

Exit codes

code meaning typical fix
0 done
1 unexpected error (a bug — please report with -v output)
2 bad input or arguments check the path / the option value in the message
3 RAM estimate over budget use the suggested --max-frames, split the clip, or --force
4 ffmpeg missing pip install imageio-ffmpeg or install ffmpeg
5 weights could not be obtained set HF_ENDPOINT to a mirror, or --checkpoint

Targets

--target fps frame size max length notes
none source even –
seedance 24 multiple of 16 (centre crop), ≥ 407,696 px (warns) 15 s (warns) Seedance 2.0 / 2.5 reference video (@视频N motion + camera)
h3 24 multiple of 32 (centre crop) 15 s (warns) MiniMax-H3-Fun-Controlnet-Union depth / pose / canny input, or reference video
wan 16 multiple of 16 – Wan 2.1 VACE control video; values from the docs, not yet verified end-to-end

Frame size is cropped, never padded: a padded black border reads to the generator as a far wall.

Python API

from pathlib import Path
from reshot import RunConfig, run, run_many, plan

cfg = RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3", metrics=Path("m.json"))
print(plan(cfg))          # sizes, frame counts, RAM estimate — no decoding yet
result = run(cfg)         # same code path as the CLI
print(result.frames, result.fps, result.ms_per_frame, result.peak_rss_bytes)

Lower-level pieces, for your own post-processing:

from reshot import extract, to_gray, upsample_frames, write_gray_video

depths, fps = extract("in.mp4")                     # float32 [T, H, W], larger = closer
gray = to_gray(depths, clip_percent=0.5, gamma=1.2)  # uint8 [T, H, W], whole-clip normalised
write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 736))

Errors that are the user's to fix are subclasses of reshot.ReshotError and carry an exit_code; anything else is a bug.

RunConfig(..., backend="fake") runs the entire pipeline with a synthetic depth field (or, for control="pose", two walking stick figures) and no model — handy for testing your own integration in milliseconds.

Pose

from reshot import RunConfig, run
run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), target="h3", control="pose", keypoints=Path("pose.json")))

The pieces, for your own pipeline:

from reshot import read_video, write_rgb_video
from reshot.backends import get_backend
from reshot.pose.tracking import stabilise, track, smooth, apply_hysteresis
from reshot.pose.skeleton import render_frame, LAYOUT

frames, fps = read_video("in.mp4")                    # uint8 RGB [T, H, W, 3]
clip = get_backend("dwpose").infer(frames, fps)       # PoseClip: keypoints[t] = [N, 134, 2] px, scores[t] = [N, 134]
stabilise(clip)                                       # = track → apply_hysteresis → smooth, in place
rgb = [render_frame(k, s, 480, 864, src_w=clip.width, src_h=clip.height) for k, s in zip(clip.keypoints, clip.scores)]
write_rgb_video(np.stack(rgb), "pose.mp4", fps)
clip.to_json_dict()                                   # the same structure as --keypoints

Keypoint layout (LAYOUT): 0–17 body in OpenPose-18 order (nose, neck, R-shoulder, R-elbow, R-wrist, L-shoulder, L-elbow, L-wrist, R-hip, R-knee, R-ankle, L-hip, L-knee, L-ankle, R-eye, L-eye, R-ear, L-ear), 18–23 feet, 24–91 face (68), 92–112 left hand (21), 113–133 right hand (21). Scores are 0–1; a joint below 0.3 is not drawn.

--keypoints JSON ("format": "reshot-pose/1"): width, height, fps, layout, then frames[t].people[n] with id (stable across frames), keypoints (134 × [x, y] in pixels of the processing frame — width × height — not the cropped output) and scores (134). Written after tracking / hysteresis / smoothing, so it matches the video exactly.

Canny

from reshot.edges import canny_frame
edges = canny_frame(rgb_frame, out_h, out_w, low=100, high=200)   # uint8 [out_h, out_w], 0 or 255

Recipes

Seedance 2.0 / 2.5 (reference video)

  1. reshot ref.mp4 -o ref_depth.mp4 --target seedance (24 fps, ×16, ≤ 15 s).

  2. Submit it as a reference video (API: role: "reference_video"; in the console, attach it as a video reference) and address it in the prompt as @视频1, asking for its motion and camera:

    参考@视频1的动作与运镜。两名武者在雨夜屋顶对决,黑色劲装,冷蓝月光,电影感。
    

    Describe only the look — cast, wardrobe, lighting, style. Staging comes from the video.

  3. What the official reference-video spec asks for, and what the preset does about it:

    official requirement preset
    mp4 / mov, H.264 or H.265 H.264 mp4, yuv420p
    24 – 60 fps 24 fps
    2.0: 2 – 15 s per clip, ≤ 15 s total · 2.5: 2 – 30 s per clip, ≤ 30 s total warns above 15 s
    width × height ≥ 407,696 px, each side 300 – 6,000 px, aspect 0.4 – 2.5 warns below the pixel floor — raise --max-res or use a larger source
    ≤ 200 MB CRF 12 keeps a 15 s 1080p clip well under

    Seedance 2.5 allows up to 30 s; pass --fps 24 without --target and check the size yourself if you need the longer window.

MiniMax H3 Fun ControlNet (ComfyUI)

  1. reshot ref.mp4 -o ref_depth.mp4 --target h3 (24 fps, ×32, ≤ 15 s) — or --control pose / --control canny for the other two conditions the Union weights accept.
  2. In ComfyUI with the Fun ControlNet Union weights, load the control video, choose the matching condition (depth / pose / canny), write your prompt for the look (cast, wardrobe, lighting, style). Staging and camera come from the control video. Or skip the files: the ComfyUI-ReShot pack has ReShot Depth / Pose / Canny Video nodes that take a Load Video node directly.
  3. Keep the generator's resolution equal to the control video's, or let the node resize — but do not pad.

Wan 2.1 VACE

reshot ref.mp4 -o ref_depth.mp4 --target wan, then use it as the depth control video in your VACE workflow. Report back if the fps/size values need adjusting — this preset has not been verified end-to-end yet.

Batch

Give reshot several inputs and a directory; the model loads once for all of them and each clip comes out as <name>_depth.mp4. --metrics / --npz become directories too (<name>.json / <name>.npz). A clip that fails is reported and skipped; the exit code is that of the first failure.

reshot clips/*.mp4 -o depth/ --target seedance --metrics depth/metrics/

From Python, run_many() is the same thing:

from pathlib import Path
from reshot import RunConfig, run_many

cfgs = [RunConfig(input=p, output=Path("depth") / f"{p.stem}_depth.mp4", target="seedance") for p in Path("clips").glob("*.mp4")]
for r in run_many(cfgs):        # RunResult, or the ReshotError for a clip that failed
    print(r)

Long videos

Everything is held in RAM for the whole clip (frames + depth). reshot estimates the need and refuses beyond half of physical RAM with a --max-frames suggestion. For a 3-minute clip, split it into ≤ 15 s pieces first (ffmpeg -ss … -t 15 …); the generators want short control videos anyway.

Apple Silicon

Works out of the box (--device auto picks MPS). Two things measured on an M2 Max: torch 2.9.1 is ~3.5× faster than 2.6.0, and fp16 is disabled on MPS because it does not finish. Expect ~0.5 s/frame at 736×1280; a 12 s clip takes ~2.5 minutes.

Reproducible depth for research

--npz writes the raw float depth at processing resolution. Combined with --metrics (which records the exact config and plan) a run is fully reproducible.

How it works

input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode
  depth:    at model resolution ─▶ Video Depth Anything (32-frame windows, 10-frame overlap, alignment)
            ─▶ whole-clip normalisation → uint8, near = white ─▶ per-frame upscale to output size
  pose:     at output size ─▶ YOLOX-L person boxes (every 3rd frame; skeleton-tracked boxes between)
            ─▶ RTMPose whole-body, 133 points per person ─▶ OpenPose-18 layout + neck
            ─▶ track ids (IoU) ─▶ visibility hysteresis (on > 0.3, off < 0.2) ─▶ One-Euro per joint
            ─▶ OpenPose render: limbs as ellipses at 60 %, joints on top, hands as rainbow edges
  canny:    at output size ─▶ grey ─▶ 3×3 Gaussian ─▶ cv2.Canny(low, high)
            ─▶ centre-crop to target multiple ─▶ x264 (grey for depth / canny, rgb24 for pose)
  • Model: Video Depth Anything (CVPR 2025). Its temporal module is what keeps the depth stable across frames; single-image models flicker. The model code is vendored unmodified (two import lines) under third_party/.
  • Resolution: the model works at --quality size — 644×364 (fast) or 924×518 (full) for a 16:9 clip. Feeding larger frames only inflates the float depth kept per frame; reshot decodes at that size and upscales the 8-bit result to the source size.
  • Normalisation: (d − min) / (max − min) over all frames; optional percentile clip.
  • Pose model: DWPose (ICCV 2023 workshop), the estimator behind controlnet_aux's pose images — so the skeletons look like what the video ControlNets were trained on. Apache-2.0 code and weights. The detector gets BGR, the estimator RGB with ImageNet normalisation, matching their mmdet / mmpose training configs.
  • Encoding: libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart.

Troubleshooting

symptom cause fix
ModuleNotFoundError: torchvision torch installed without torchvision pip install torchvision (matching your torch build)
exit 4 / "ffmpeg not found" no ffmpeg pip install imageio-ffmpeg or install ffmpeg
exit 5 / weights download fails no route to huggingface.co export HF_ENDPOINT=https://hf-mirror.com, or --checkpoint
exit 3 clip too long for RAM use the suggested --max-frames, split the clip
very slow on Mac torch < 2.9, or fp16 forced pip install -U torch; never force fp16 on MPS
output is 30 fps although --fps 24 source fps < 24 reshot never upsamples
output smaller than source --target crop or --max-res expected; see Targets
banding in the depth map --crf raised keep ≤ 14
people look flat / merged with background scene has little depth range try --clip 0.5 --gamma 1.3
pose needs ONNX Runtime the pose extra is not installed pip install "reshot[pose]" (or onnxruntime-gpu)
skeleton video is black / no person found the detector saw nobody: people too small, or a drawing / animal the person detector does not know try a tighter crop of the source; for non-humans use depth
a limb pops in and out the estimator is unsure of it that is the hysteresis already; --no-smooth to see the raw output, or accept — the model reads a missing limb better than a jumping one
skeleton lags a fast move One-Euro smoothing --no-smooth; lag is small by design (beta 0.5)

FAQ

Is Small good enough? For a control video, yes: the generator reads silhouette, size and motion, not fine geometry, and Small's silhouettes are clean (see the demo GIF). We have not benchmarked Base/Large against Small ourselves; they are CC-BY-NC, so Small is also the only choice for commercial work.

Why near = white? Depth ControlNets (H3 Fun, Wan VACE, SD depth) were trained on inverse-depth maps where closer is brighter. --invert if a tool wants the opposite.

Can it output pose / canny / normals? Pose and canny since 0.5.0 (--control pose, --control canny). Normals, HED and MLSD are not planned unless someone asks.

Does it remove the background / keep only people? Not yet (--people-only is on the roadmap). The full-frame depth already carries no identity or style.

GPU memory? Small in fp16 needs ~2 GB at the default input size; a 4090 is idle most of the time.