Skip to content

ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.

8.6 KB
Raw Blame
{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "# ReShot — copy the shot, not the actors\n\nTurn a reference video into a **depth map, skeleton or line video** on a free Colab GPU, then give that grey clip to\nSeedance or MiniMax H3 as the reference: it repeats the choreography and the camera moves with\nyour own characters in it.\n\n**Before you run:** menu *Runtime → Change runtime type → T4 GPU*. Then *Runtime → Run all*.\nFour steps: check the GPU · install · pick a clip · run and download.\n\nCode and docs: [github.com/maosika-ai/reshot](https://github.com/maosika-ai/reshot) · Apache-2.0 ·\nmade at [Maosika 猫斯卡](https://www.maosika.com), the AI short-drama production system.\n\n在免费的 Colab GPU 上把参考视频变成深度图视频,再交给 Seedance / MiniMax H3 当参考。\n先在菜单 *代码执行程序 → 更改运行时类型* 选 **T4 GPU**,然后 *全部运行*。"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1 · GPU"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": [
    "#@title Check that this runtime has a GPU\n",
    "import torch, subprocess\n",
    "print(subprocess.run([\"nvidia-smi\", \"-L\"], capture_output=True, text=True).stdout.strip() or \"no nvidia-smi\")\n",
    "print(\"torch\", torch.__version__, \"· CUDA available:\", torch.cuda.is_available())\n",
    "assert torch.cuda.is_available(), \"No GPU. Runtime → Change runtime type → T4 GPU, then run again.\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2 · Install"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": "#@title Install ReShot (about a minute; Colab already has a CUDA build of PyTorch)\nin_china = False  #@param {type:\"boolean\"}\nimport os\nif in_china:\n    os.environ[\"HF_ENDPOINT\"] = \"https://hf-mirror.com\"   # the 111 MB weights come from Hugging Face\n%pip install -q \"reshot[ffmpeg,pose]\"\nimport reshot\nprint(\"reshot\", reshot.__version__)"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3 · Pick a clip"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": [
    "#@title Use the demo clip, or upload your own (mp4 / mov / webm, best ≤ 15 s)\n",
    "source = \"demo clip (12 s corridor fight)\"  #@param [\"demo clip (12 s corridor fight)\", \"upload my own\"]\n",
    "import urllib.request, os\n",
    "if source.startswith(\"upload\"):\n",
    "    from google.colab import files\n",
    "    up = files.upload()\n",
    "    assert up, \"nothing uploaded\"\n",
    "    INPUT = next(iter(up))\n",
    "else:\n",
    "    INPUT = \"city_night.mp4\"\n",
    "    if not os.path.exists(INPUT):\n",
    "        urllib.request.urlretrieve(\"https://github.com/maosika-ai/reshot/raw/main/examples/city_night.mp4\", INPUT)\n",
    "import cv2\n",
    "cap = cv2.VideoCapture(INPUT); n, fps = cap.get(cv2.CAP_PROP_FRAME_COUNT), cap.get(cv2.CAP_PROP_FPS)\n",
    "print(f\"{INPUT}: {int(cap.get(3))}×{int(cap.get(4))} · {fps:.0f} fps · {n/fps:.1f} s\"); cap.release()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": "## 4 · Make the control video"
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": "#@title Run (measured on a free T4: the 12 s demo clip takes about a minute at `fast`; `full` is ~2× that; `pose` is CPU-bound, a few minutes)\ncontrol = \"depth\"  #@param [\"depth\", \"pose\", \"canny\"]\ntarget = \"seedance\"  #@param [\"seedance\", \"h3\", \"wan\", \"none\"]\nquality = \"fast\"  #@param [\"fast\", \"full\"]\nimport json, os, shutil, subprocess, time\nt0 = time.time()\nOUTPUT = f\"{control}.mp4\"\nr = subprocess.run([shutil.which(\"reshot\") or \"reshot\", INPUT, \"-o\", OUTPUT, \"--control\", control,\n                    \"--target\", target, \"--quality\", quality, \"--metrics\", \"metrics.json\", \"--keypoints\", \"pose.json\"],\n                   capture_output=True, text=True)\nprint(r.stderr[-3000:])\nassert r.returncode == 0 and os.path.exists(OUTPUT), \"run failed — read the message above\"\nm = json.load(open(\"metrics.json\"))\nprint(f\"done in {time.time()-t0:.0f} s · {m['frames']} frames @ {m['fps']} fps · \"\n      f\"{m['output_size'][0]}×{m['output_size'][1]} · {m['ms_per_frame']:.0f} ms/frame on {m['device']}\")"
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": "#@title Look at it: reference and result side by side, then the two clips\nimport base64, cv2\nfrom IPython.display import HTML, display\ndef frame(path, idx=0):\n    cap = cv2.VideoCapture(path); cap.set(cv2.CAP_PROP_POS_FRAMES, idx); ok, bgr = cap.read(); cap.release()\n    h = 360; bgr = cv2.resize(bgr, (int(bgr.shape[1] * h / bgr.shape[0]), h))\n    return base64.b64encode(cv2.imencode(\".jpg\", bgr)[1]).decode()\ndef video(path):\n    return f'<video controls loop muted autoplay width=\"360\" src=\"data:video/mp4;base64,{base64.b64encode(open(path,\"rb\").read()).decode()}\"></video>'\ndisplay(HTML(f'<div style=\"display:flex;gap:12px\"><img src=\"data:image/jpeg;base64,{frame(INPUT)}\"><img src=\"data:image/jpeg;base64,{frame(OUTPUT)}\"></div>'\n             f'<div style=\"display:flex;gap:12px;margin-top:12px\">{video(INPUT)}{video(OUTPUT)}</div>'))"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5 · Download, and what to do with it"
   ]
  },
  {
   "cell_type": "code",
   "metadata": {
    "cellView": "form"
   },
   "execution_count": null,
   "outputs": [],
   "source": "#@title Download the result and print the prompt line for your video model\nfrom google.colab import files\nLOOK = {\n  \"depth\": (\"a grey depth map in which near objects are white and far objects are black\", \"its grey depth look is not transferred\"),\n  \"pose\": (\"an OpenPose skeleton video — coloured stick figures on black, one per person\", \"the stick-figure look is not transferred\"),\n  \"canny\": (\"a white-line edge drawing on black\", \"the line-drawing look is not transferred\"),\n}[control]\nSEEDANCE = {\n  \"depth\": \"参考@视频1的动作与运镜,顺序与视频保持一致。\",\n  \"pose\": \"参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。\",\n  \"canny\": \"参考@视频1的构图、动作与运镜,顺序与视频保持一致;只取构图和动作,不要画成线稿。\",\n}[control]\nNEXT = {\n  \"seedance\": (f\"Upload {OUTPUT} to Seedance 2.0 / 2.5 as a reference video and start the prompt with:\",\n               SEEDANCE + \"\\n(then describe your people, wardrobe, set and look)\"),\n  \"h3\": (f\"In MiniMax H3 attach {OUTPUT} as <Video 1> (a character sheet as <Picture 1>); define it and say its look is not to be copied:\",\n         f\"<Subject 3> is the choreography and camera movement shown in <Video 1>, {LOOK[0]}: (one sentence on what happens in the clip)\\n\"\n         f\"<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; {LOOK[1]}.\"),\n  \"wan\": (f\"Connect {OUTPUT} to WanVaceToVideo → control_video in ComfyUI.\", \"\"),\n  \"none\": (f\"Hand {OUTPUT} to any model that reads a {control} video.\", \"\"),\n}\nhow, prompt = NEXT[target]\nprint(how); print(); print(prompt)\nfiles.download(OUTPUT)\nif control == \"pose\" and os.path.exists(\"pose.json\"):\n    files.download(\"pose.json\")   # the skeletons as JSON, in case you want to edit or retarget them"
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "Made something? Post the reference, the depth map, the take and your prompt line in\n",
    "[Show and tell](https://github.com/maosika-ai/reshot/discussions/2). Something broke?\n",
    "[Open an issue](https://github.com/maosika-ai/reshot/issues/new/choose) with the message from step 4.\n",
    "\n",
    "For whole folders, longer clips and scripting, run it on your own machine: `pip install reshot`,\n",
    "then `reshot` for the local web page or `reshot clip.mp4 -o depth.mp4 --target seedance`."
   ]
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "name": "reshot_colab.ipynb",
   "provenance": [],
   "gpuType": "T4"
  },
  "kernelspec": {
   "display_name": "Python 3",
   "name": "python3"
  },
  "language_info": {
   "name": "python"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}