ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Apache-2.0. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system.
Python ai-short-dramaai-videocannycomfyuicontrolnetdepth-anythingdepth-estimationdwposemaosikaminimaxopenposepose-controlnetpose-estimationseedanceshort-dramavideo-generationvideo-to-videowan
0.5.0: --control pose (DWPose skeletons, tracked + smoothed, keypoints JSON) and --control canny; web page picks depth / skeleton / lines; --pose-detect-every; docs, Colab, CI, demo assets
wangfeng8ebc883b5730
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex fe5b58c..8b701ed 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -32,7 +32,7 @@ jobs:- run: python -m pytest -q testsmodel-smoke:- name: real model on CPU (8 frames)+ name: real models on CPU (8 frames: depth + pose)runs-on: ubuntu-lateststeps:- uses: actions/checkout@v4@@ -41,7 +41,7 @@ jobs:- uses: actions/cache@v4with:path: ~/.cache/huggingface- key: hf-vda-small-v1+ key: hf-vda-small-dwpose-v2- run: |python -m pip install -U pippython -m pip install torch torchvision --index-url https://download.pytorch.org/whl/cpudiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 6d08e5f..a52ecde 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -1,5 +1,29 @@# Changelog+## 0.5.0 — 2026-09-14+- **Skeleton output — `--control pose`** (closes #6). DWPose (YOLOX-L person detector + RTMPose+ whole-body estimator, Apache-2.0 code and weights) in ONNX Runtime, drawn in the OpenPose+ layout the video ControlNets were trained on: 18 body joints + 21-point hands, faces off by+ default. Across frames: identity tracking (IoU), visibility hysteresis (on > 0.3, off < 0.2)+ and a One-Euro filter per joint, so the skeleton does not flicker. `--keypoints x.json` writes+ every skeleton (`reshot-pose/1`: stable id per person, 134 points + scores per frame).+ `pip install "reshot[pose]"`; weights (~340 MB) download once from `yzd-v/DWPose`, or+ `--checkpoint-dir`. Measured: 306 ms/frame on an M2 Max CPU, the 12 s demo in 90 s.+- **`--pose-detect-every N` (default 3).** The detector is 80 % of the pose cost, so it runs every+ third frame and the previous skeletons supply the boxes in between; a cut or an unreliable+ skeleton re-detects at once. Against every-frame detection on the 289-frame demo: 280 frames+ identical people count, 80 % of joints within 2 px, 91 % within 5 px, 35 % faster.+- **Line output — `--control canny`**, thresholds `--canny LOW,HIGH`. No model.+- `--control depth,pose,canny` writes all three in one go (`-o` a directory); one model load per kind.+- **Web page**: a third choice, *What to make* — depth map / skeleton / lines; quality and the+ advanced fields show only for the type they apply to; the result panel names the file, offers+ the keypoints JSON for skeletons, and prints the prompt line for your target × type.+- Python: `write_rgb_video`, `reshot.pose` (`PoseClip`, `skeleton.render_frame`,+ `tracking.stabilise`), `reshot.edges.canny_frame`; `RunConfig.control` and the pose / canny fields.+- Colab notebook and `docs/AGENT_INSTALL.md` cover the new types.+- Pose and canny are read at the output size (no "model resolution"), with their own, much+ smaller host-RAM estimate.+## 0.4.1 — 2026-09-14- **Colab notebook.** `examples/reshot_colab.ipynb` — a free GPU, upload a clip, download`depth.mp4`, no install. "Open in Colab" badge in both READMEs.diff --git a/NOTICE b/NOTICEindex a699ab2..a6884de 100644--- a/NOTICE+++ b/NOTICE@@ -6,7 +6,13 @@ Video Depth Anything (https://github.com/DepthAnything/Video-Depth-Anything),licensed under the Apache License 2.0. It is vendored, unmodified except fortwo import lines, under reshot/third_party/video_depth_anything/.+The pose pre/post-processing in reshot/pose/dwpose.py is a NumPy rewrite of the ONNX+reference implementation in DWPose (IDEA-Research, https://github.com/IDEA-Research/DWPose),+licensed under the Apache License 2.0. The OpenPose-style renderer in reshot/pose/skeleton.py+follows the drawing conventions of the same project (colours, limb table, hand edges).+Model weights are downloaded at runtime from Hugging Face and carry their own licences:depth-anything/Video-Depth-Anything-Small Apache-2.0 (default; commercial use OK)depth-anything/Video-Depth-Anything-Base CC-BY-NC-4.0 (opt-in; non-commercial)depth-anything/Video-Depth-Anything-Large CC-BY-NC-4.0 (opt-in; non-commercial)+ yzd-v/DWPose yolox_l.onnx + dw-ll_ucoco_384.onnx Apache-2.0 (--control pose; commercial use OK)diff --git a/README.md b/README.mdindex b2279f7..a9b0cae 100644--- a/README.md+++ b/README.md@@ -1,6 +1,6 @@<h1 align="center">ReShot</h1><p align="center"><b>Copy the shot, not the actors.</b></p>-<p align="center">ReShot turns a reference video into a depth map, so Seedance or MiniMax H3 can repeat its choreography and camera moves — with your own characters in it.</p>+<p align="center">ReShot turns a reference video into a depth map, a skeleton or a line video, so Seedance or MiniMax H3 can repeat its choreography and camera moves — with your own characters in it.</p><p align="center"><img src="docs/demo-fight.gif" width="720" alt="a fight scene, its depth map, and three new takes generated from it"></p><p align="center"><sub>Top: the reference and its depth map. Bottom: three takes generated from that depth map — two women, one rabbit. Same moves, same camera. <a href="docs/demo-fight.mp4">Full-resolution clip</a>.</sub></p>@@ -23,13 +23,21 @@ You have a clip whose fight, dance or camera move is exactly what you want in yo## What ReShot does-ReShot takes an `.mp4` in and writes an `.mp4` out. The output is a **depth map video**: every frame is grey, near things are white, far things are black. It keeps where everyone stands, how big they are, how they move and how the camera moves. It throws away faces, clothes, lighting and style.+ReShot takes an `.mp4` in and writes an `.mp4` out. The output is a control video — one of three kinds:-<p align="center"><img src="docs/img/step1_reference.jpg" width="360" alt="reference frame"> <img src="docs/img/step2_depth.jpg" width="360" alt="the same frame as a depth map"></p>+| `--control` | what comes out | keeps | throws away | pick it for |+|---|---|---|---|---|+| **`depth`** (default) | grey video, near = white, far = black | where everyone stands, how big they are, every move, the camera, the set's shape | faces, clothes, lighting, style | anything: fights, walks, camera moves, crowds |+| **`pose`** | OpenPose-style skeletons — coloured stick figures on black, hands included, faces not | every limb and hand, exactly | everything else, including body shape and the set | dance, martial arts, anything where the limbs are the point; the `pose` input of MiniMax H3 Fun ControlNet |+| **`canny`** | white edge lines on black | composition and outlines | colour and shading — but *not* the outline of a face or a costume | when you want the picture's layout, not only the people |++<p align="center"><img src="docs/img/step1_reference.jpg" width="300" alt="reference frame"> <img src="docs/img/step2_depth.jpg" width="300" alt="the same frame as a depth map"> <img src="docs/img/step2_pose.jpg" width="300" alt="the same frame as a skeleton"></p>+<p align="center"><img src="docs/demo-controls.gif" width="864" alt="the corridor fight as reference, depth map and skeleton, playing side by side"></p>+<p align="center"><sub>Reference · depth map · skeleton, the same 12 seconds. Four stills at full size: <a href="docs/img/controls_strip.jpg">controls_strip.jpg</a>.</sub></p>-You give that grey video to your video model as the reference and describe the people and the look in the prompt. The model takes the moves from the video and everything else from your words.+You give that video to your video model as the reference and describe the people and the look in the prompt. The model takes the moves from the video and everything else from your words.-Technically: monocular video depth estimation. The model predicts relative inverse depth for every frame; ReShot normalises it once over the whole clip to 8-bit grey (near = white) and encodes it as a standard depth-map video.+Technically: depth is monocular video depth estimation (Video Depth Anything), normalised once over the whole clip to 8-bit grey; pose is DWPose (a YOLOX person detector plus an RTMPose whole-body estimator) with identity tracking, visibility hysteresis and One-Euro smoothing across frames, drawn in the OpenPose layout the video ControlNets were trained on; canny is OpenCV, no model at all.## How to use it@@ -57,11 +65,11 @@ set HF_ENDPOINT=https://hf-mirror.com if I'm in China, run the fake-backend smokon a short clip with --metrics and show me the numbers. Don't say it's done until depth.mp4 exists.```-### 2. Make the depth map+### 2. Make the control video-**Easiest — the web page.** Type `reshot` with nothing after it: a page opens in your browser (everything stays on your computer). Drop the clip in, pick which model it is for, click **Make depth map**. You get the reference and the depth map side by side, the numbers, a download button, and the prompt line to paste into Seedance or MiniMax H3.+**Easiest — the web page.** Type `reshot` with nothing after it: a page opens in your browser (everything stays on your computer). Drop the clip in, pick **what to make** (depth map / skeleton / lines) and which model it is for, click the button. You get the reference and the result side by side, the numbers, a download button, and the prompt line to paste into Seedance or MiniMax H3.-<p align="center"><img src="docs/img/web_ui_en.jpg" width="720" alt="the ReShot web page: reference clip and depth map side by side, target and quality on the right"></p>+<p align="center"><img src="docs/img/web_ui_en.jpg" width="720" alt="the ReShot web page: reference clip and skeleton side by side; what to make, target and quality on the right"></p>```bashreshot # opens http://127.0.0.1:8765 — results land in ~/ReShot@@ -71,10 +79,14 @@ reshot web --port 9000 --out ./depth --no-browser # options, if you want the**Or the command line**, for scripts and folders:```bash-reshot reference.mp4 -o depth.mp4 --target seedance+reshot reference.mp4 -o depth.mp4 --target seedance # depth map (default)+reshot reference.mp4 -o pose.mp4 --target h3 --control pose # skeletons; pip install "reshot[pose]" first+reshot reference.mp4 -o out/ --control depth,pose,canny # all three, out/reference_<control>.mp4```-`--target seedance` sets 24 fps, H.264, a frame size that is a multiple of 16 and at least 407,696 pixels, up to 15 seconds — the reference-video rules of the Seedance API. For MiniMax H3 use `--target h3` (multiples of 32). On an RTX 4090 a 12-second clip takes about 20 seconds; on a MacBook a few minutes. A whole folder at once, one model load: `reshot clips/*.mp4 -o depth/ --target seedance`.+`--target seedance` sets 24 fps, H.264, a frame size that is a multiple of 16 and at least 407,696 pixels, up to 15 seconds — the reference-video rules of the Seedance API. For MiniMax H3 use `--target h3` (multiples of 32). On an RTX 4090 a 12-second clip takes about 20 seconds for depth; on a MacBook a few minutes for depth and about 1.5 minutes for skeletons (the pose models run on the CPU there). A whole folder at once, one model load: `reshot clips/*.mp4 -o depth/ --target seedance`.++`--control pose` also writes the skeletons as JSON with `--keypoints pose.json` (every person, every frame, 134 points with scores, a stable id per person) — so you can edit, retarget or analyse them; the web page offers the same file as a download.### 3. Give it to the video model@@ -85,6 +97,8 @@ reshot reference.mp4 -o depth.mp4 --target seedance一名穿深绿色丝绒旗袍的女子在狭窄的金属走廊里与三名黑衣守卫搏斗,冷蓝走廊光,红色警示灯,电影感。```+With a skeleton video say so, and say the lines must not appear: `参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。`+**ComfyUI.** Install [ComfyUI-ReShot](https://github.com/maosika-ai/ComfyUI-ReShot) and drop the *ReShot Depth Video* node between Load Video and your model — no command line at all.**MiniMax H3.** Attach `depth.mp4` as `<Video 1>`. If you want a specific face, attach a character sheet as `<Picture 1>`. These are the three sheets used for the demo:@@ -106,6 +120,8 @@ is transferred onto <Subject 1> and <Subject 2>; its grey depth look is not tranTwo things in there matter. **Say in words what happens in the grey clip** — the model reads the depth map far better when the prompt tells it what the blobs are doing. **Say that the grey look is not to be copied**, or you may get a grey film back.+For a skeleton video the same two sentences become `<Video 1>, an OpenPose skeleton video — coloured stick figures on black, one per person: …` and `… the stick-figure look is not transferred.` With the Fun ControlNet weights in ComfyUI, `pose.mp4` goes straight into the ControlNet's **pose** input instead — no prompt gymnastics needed. The web page prints the right lines for whichever you made.+### 4. What comes out<p align="center"><img src="docs/img/step4_takes.jpg" width="720" alt="the three takes"></p>@@ -118,6 +134,8 @@ Left to right: [Jiang Xue](docs/prompts/take1_jiangxue_armor.txt) in bronze armo**Doesn't:** faces (use a character sheet), clothes, lighting, colour, props in detail, and anything smaller than a hand. Those come from your prompt and your reference images.+**Depth or pose?** Depth carries the whole picture — where the walls are, who is bigger, a pipe being torn off — and works on anything, people or not. Pose carries only the people, but every limb and finger exactly, and nothing of their body shape or the set. Use pose for dance and martial arts, depth for everything else; when in doubt make both (`--control depth,pose`) and see which one the model follows better. One case where pose is the wrong tool: a tight close-up. The estimator still guesses where the shoulders and hips are when they are off screen, and draws them along the frame edge — depth carries a close-up faithfully. Canny carries the outline of clothes and faces too, so "copy the shot, not the actors" is only half true for it.+Things we learned making the demo:- **Keep the reference under 15 seconds** (Seedance's limit) and cut it to the shots you want before running ReShot. Everything in the clip gets copied, including the boring part at the end.@@ -134,6 +152,8 @@ Things we learned making the demo:| **Apple Silicon** | Yes | default `fast`: M2 Max 46 ms/frame, a 12 s clip in about 18 s || **CPU only** | Yes, slow | ~1.8 s/frame || **Host RAM** | 16 GB covers clips up to ~27 s at 720p | peak = 2 GB + 224 MB per second of 720p; the tool refuses before starting if a clip won't fit |+| **`--control pose`** | any of the above; needs `pip install "reshot[pose]"` (ONNX Runtime; `onnxruntime-gpu` for CUDA) | M2 Max on the CPU: 306 ms/frame, a 12 s clip in 90 s; two weight files, 340 MB, once. Host RAM ≈ 1.4 GB |+| **`--control canny`** | anything | no model; the 12 s clip in under a second |**`--quality` — the resolution the model works at.** The output video is always the size of your source; this sets the size of the picture the depth model looks at.@@ -142,7 +162,9 @@ Things we learned making the demo:| `fast` (default) | 644×364 · 364×644 · 490×364 | ~3 GB | 34 ms/frame on a 3080 Ti | large shapes identical to full; a loose strand of hair merges into the cheek || `full` | 924×518 · 518×924 · 686×518 | ~11 GB | 83 ms/frame on a 3080 Ti | sharper fine silhouettes |-The web page has the same two choices under **Quality**; the command line takes `--quality`.+The web page has the same two choices under **Quality**; the command line takes `--quality`. Quality applies to depth only — the pose models have fixed working sizes (640 px for the detector, 288×384 per person for the estimator) and canny is per pixel at the output size.++**Pose speed knob — `--pose-detect-every N` (default 3).** The person detector is 80 % of the pose cost on a CPU, so it runs on every third frame and the skeletons of the previous frame supply the boxes in between; a cut in the clip or a skeleton that has gone unreliable brings the detector back at once. Measured on the 289-frame corridor clip against detecting every frame: 280 of 289 frames have the same people, 80 % of joints land within 2 px and 91 % within 5 px; the 9 differing frames are a person entering the shot and getting their skeleton one or two frames late. It is 35 % faster (306 vs 469 ms/frame on an M2 Max). `--pose-detect-every 1` turns it off.We recommend `fast`: for copying blocking and camera it is all a video model needs — the demo takes were made from a depth map shrunk to 320×176 before it even reached MiniMax H3, far below either setting. Measured full against fast on the same 294-frame clip: mean difference 5.3 of 255 grey levels, 95 % of pixels within 15, edge energy −4.5 %. Use `full` for close-ups where thin silhouettes matter and you have the VRAM. The run prints the exact resolution it uses (`model fast: the model sees 644x364`) and records it in `--metrics`. Experts can set any short side with `--input-size`.@@ -168,12 +190,27 @@ from reshot import RunConfig, runrun(RunConfig(input=Path("reference.mp4"), output=Path("depth.mp4"), target="seedance"))```-Three details make the output something a video model will actually follow:+Four details make the output something a video model will actually follow:- **One scale for the whole clip.** Depth is normalised once over all frames, never per frame, so a wall keeps the same grey when someone walks past it. Per-frame normalisation makes the scene "breathe".+- **Skeletons that don't flicker.** A per-frame pose estimator swaps people, jitters joints and blinks limbs at the visibility threshold; a video model reads all three as motion. ReShot tracks each person across frames (IoU of body boxes), holds a joint until its score drops well below the draw threshold (hysteresis), and runs a One-Euro filter per joint — still at rest, quick to follow a punch.- **Frames picked by timestamp.** 30 fps → 24 fps really is 24; nothing is duplicated or dropped in a pattern the model could learn.- **Cropped, never padded.** Frame size is trimmed to the model's grid. A black border would read as a far wall.+Pose from Python:++```python+from reshot import RunConfig, run+run(RunConfig(input=Path("reference.mp4"), output=Path("pose.mp4"), target="h3", control="pose", keypoints=Path("pose.json")))++from reshot.backends import get_backend+from reshot.pose.tracking import stabilise+from reshot.pose.skeleton import render_frame+clip = get_backend("dwpose").infer(frames, fps) # frames: uint8 RGB [T, H, W, 3] → PoseClip+stabilise(clip) # track → hysteresis → smooth, in place+rgb = render_frame(clip.keypoints[0], clip.scores[0], h, w, hands=True, face=False)+```+The model is Video Depth Anything Small (ByteDance, CVPR 2025). It works on overlapping 32-frame windows and aligns them, so depth doesn't jitter between frames. A 12-second 720p clip peaks at 3.9 GB of host RAM and 11 GB of VRAM (3 GB with `--input-size 364`) (measured; the estimate the tool shows before starting is a fitted line, within 0.05 GB of measurements) and it refuses up front if a clip won't fit. A `fake` backend runs the whole pipeline without a model for your own tests. Errors that are yours to fix are `ReshotError` subclasses with an exit code and a concrete fix in the message.## Community@@ -181,11 +218,11 @@ The model is Video Depth Anything Small (ByteDance, CVPR 2025). It works on over- **Made something?** Post the reference, the depth map, the take and your prompt line in [Show and tell](https://github.com/maosika-ai/reshot/discussions/2). The best ones get linked here.- **The model didn't follow the depth map?** Ask in [Q&A](https://github.com/maosika-ai/reshot/discussions/categories/q-a) with one frame and the prompt — it is usually the prompt.- **Something broke?** [Open an issue](https://github.com/maosika-ai/reshot/issues/new/choose). We answer within a day.-- **Want to help?** Start with a [good first issue](https://github.com/maosika-ai/reshot/labels/good%20first%20issue): a preset for another model, a Colab notebook, a README in your language. Pose output is the big one — see [#6](https://github.com/maosika-ai/reshot/issues/6).+- **Want to help?** Start with a [good first issue](https://github.com/maosika-ai/reshot/labels/good%20first%20issue): a preset for another model, a README in your language, HED / MLSD outputs for the other two slots of the H3 Fun ControlNet. Pose landed in 0.5.0 ([#6](https://github.com/maosika-ai/reshot/issues/6)).## License-Apache-2.0, and so is the default model. The vendored model code is under `reshot/third_party/`. Use it in a product, a pipeline, a service. The larger research-only weights (Base, Large) are CC-BY-NC and never load unless you ask for them.+Apache-2.0, and so are the default depth model and the pose models (DWPose, both code and weights). The vendored model code is under `reshot/third_party/`; the pose pre/post-processing is a NumPy rewrite of DWPose's ONNX reference, credited in [NOTICE](NOTICE). Use it in a product, a pipeline, a service. The larger research-only depth weights (Base, Large) are CC-BY-NC and never load unless you ask for them.Issues and pull requests are welcome; see [CONTRIBUTING.md](CONTRIBUTING.md).diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 1534ee0..fc53ae2 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -1,6 +1,6 @@<h1 align="center">ReShot</h1><p align="center"><b>复制走位,不复制演员。</b></p>-<p align="center">ReShot 把一段参考视频变成深度图,让 Seedance 或 MiniMax H3 照着它的动作和运镜再拍一遍——人物换成你的。</p>+<p align="center">ReShot 把一段参考视频变成深度图、骨架或线稿视频,让 Seedance 或 MiniMax H3 照着它的动作和运镜再拍一遍——人物换成你的。</p><p align="center"><img src="docs/demo-fight.gif" width="720" alt="一段武打参考片、它的深度图、用深度图生成的三段新武打片"></p><p align="center"><sub>上排:参考片和它的深度图。下排:用这张深度图生成的三段片——两个女人、一只兔子。动作一样,镜头一样。<a href="docs/demo-fight.mp4">高清原片</a>。</sub></p>@@ -23,13 +23,21 @@## ReShot 做什么-ReShot 吃进一个 `.mp4`,吐出一个 `.mp4`。输出是一段**深度图视频**:每一帧都是灰的,近的地方白,远的地方黑。它留下的是谁站在哪、谁大谁小、怎么动、镜头怎么走;扔掉的是脸、衣服、光线和画风。+ReShot 吃进一个 `.mp4`,吐出一个 `.mp4`。输出是控制视频,三种任选:-<p align="center"><img src="docs/img/step1_reference.jpg" width="360" alt="参考片的一帧"> <img src="docs/img/step2_depth.jpg" width="360" alt="同一帧的深度图"></p>+| `--control` | 出来的是什么 | 留下 | 扔掉 | 什么时候选它 |+|---|---|---|---|---|+| **`depth`**(默认) | 灰片,近白远黑 | 谁站在哪、谁大谁小、每个动作、运镜、场景的形状 | 脸、衣服、光线、画风 | 什么都行:打戏、走位、运镜、群戏 |+| **`pose`** | OpenPose 画法的骨架——黑底彩色火柴人,带手、不画脸 | 每一根肢体、每一只手,分毫不差 | 其余全部,包括体型和场景 | 跳舞、武打,凡是靠肢体的;MiniMax H3 Fun ControlNet 的 `pose` 口 |+| **`canny`** | 黑底白线的边缘图 | 构图和轮廓 | 颜色和明暗——但脸型和服装的轮廓**留着** | 想要整个画面的布局、不只是人的时候 |++<p align="center"><img src="docs/img/step1_reference.jpg" width="300" alt="参考片的一帧"> <img src="docs/img/step2_depth.jpg" width="300" alt="同一帧的深度图"> <img src="docs/img/step2_pose.jpg" width="300" alt="同一帧的骨架"></p>+<p align="center"><img src="docs/demo-controls.gif" width="864" alt="走廊武打的参考片、深度图和骨架并排播放"></p>+<p align="center"><sub>参考片 · 深度图 · 骨架,同一段 12 秒。四张全尺寸静帧:<a href="docs/img/controls_strip.jpg">controls_strip.jpg</a>。</sub></p>-你把这段灰片当参考视频交给视频模型,提示词里写人物和画风。动作和镜头模型从灰片里读,其余全听你的。+你把这段视频当参考交给视频模型,提示词里写人物和画风。动作和镜头模型从片里读,其余全听你的。-严格地说,这是单目视频深度估计:模型对每一帧预测相对逆深度,ReShot 对整段视频做一次归一化,变成 8 位灰度(近白远黑),再编码成标准的深度图视频。+严格地说:深度图是单目视频深度估计(Video Depth Anything),对整段视频做一次归一化变成 8 位灰度;骨架是 DWPose(YOLOX 检人 + RTMPose 全身估点),跨帧做身份跟踪、可见性迟滞和 One-Euro 平滑,再按视频 ControlNet 训练时见过的 OpenPose 画法画出来;线稿就是 OpenCV,不用模型。## 怎么用@@ -57,11 +65,11 @@ pip install reshot # 需要 PATH 里有 ffmpegdepth.mp4 没生成出来之前不要说完成。```-### 2. 抽深度图+### 2. 出控制视频-**最省事——网页。** 只敲 `reshot`,后面什么都不加,浏览器会打开一个页面(所有处理都在你自己电脑上)。把片子拖进去,选给哪家模型用,点**生成深度图**。参考片和深度图并排预览,数字、下载按钮、要粘到 Seedance 或 MiniMax H3 的提示词都在同一屏。+**最省事——网页。** 只敲 `reshot`,后面什么都不加,浏览器会打开一个页面(所有处理都在你自己电脑上)。把片子拖进去,选**出什么**(深度图 / 骨架 / 线稿)和给哪家模型用,点按钮。参考片和结果并排预览,数字、下载按钮、要粘到 Seedance 或 MiniMax H3 的提示词都在同一屏。-<p align="center"><img src="docs/img/web_ui_zh.jpg" width="720" alt="ReShot 网页:左边参考片与深度图并排,右边选模型和画质"></p>+<p align="center"><img src="docs/img/web_ui_zh.jpg" width="720" alt="ReShot 网页:左边参考片与骨架并排,右边选出什么、给谁用和画质"></p>```bashreshot # 打开 http://127.0.0.1:8765,结果存到 ~/ReShot@@ -71,10 +79,14 @@ reshot web --port 9000 --out ./depth --no-browser # 想改端口 / 目录 /**或者命令行**,适合脚本和批量:```bash-reshot 参考片.mp4 -o 深度图.mp4 --target seedance+reshot 参考片.mp4 -o 深度图.mp4 --target seedance # 深度图(默认)+reshot 参考片.mp4 -o 骨架.mp4 --target h3 --control pose # 骨架;先 pip install "reshot[pose]"+reshot 参考片.mp4 -o out/ --control depth,pose,canny # 三种一起出,out/参考片_<control>.mp4```-`--target seedance` 会把输出设成 24 fps、H.264、边长 16 的倍数、不少于 407,696 像素、最长 15 秒——这就是 Seedance 接口对参考视频的要求。给 MiniMax H3 用 `--target h3`(边长 32 的倍数)。RTX 4090 上 12 秒的片约 20 秒跑完,MacBook 要几分钟。整个文件夹一起来、模型只加载一次:`reshot clips/*.mp4 -o depth/ --target seedance`。+`--target seedance` 会把输出设成 24 fps、H.264、边长 16 的倍数、不少于 407,696 像素、最长 15 秒——这就是 Seedance 接口对参考视频的要求。给 MiniMax H3 用 `--target h3`(边长 32 的倍数)。RTX 4090 上 12 秒的片深度图约 20 秒跑完;MacBook 深度图要几分钟、骨架约一分半(骨架模型在 Mac 上走 CPU)。整个文件夹一起来、模型只加载一次:`reshot clips/*.mp4 -o depth/ --target seedance`。++`--control pose` 加 `--keypoints 骨架.json` 会把骨架数据也存成 JSON(每帧每个人 134 个点带分数、每个人一个稳定的 id),可以拿去改、重定向、做分析;网页上同一份文件有下载按钮。### 3. 交给视频模型@@ -85,6 +97,8 @@ reshot 参考片.mp4 -o 深度图.mp4 --target seedance一名穿深绿色丝绒旗袍的女子在狭窄的金属走廊里与三名黑衣守卫搏斗,冷蓝走廊光,红色警示灯,电影感。```+挂的是骨架视频就明说,并且说清线条不要出现:`参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。`+**ComfyUI。** 装 [ComfyUI-ReShot](https://github.com/maosika-ai/ComfyUI-ReShot),在 Load Video 和你的模型之间放一个 *ReShot Depth Video* 节点就行,完全不用命令行。**MiniMax H3。** 把 `深度图.mp4` 挂成 `<Video 1>`。想要固定的脸,再把定妆图挂成 `<Picture 1>`。演示用的三张定妆图:@@ -106,6 +120,8 @@ is transferred onto <Subject 1> and <Subject 2>; its grey depth look is not tran两个要点。**用文字把灰片里发生的事写一遍**——提示词告诉它那些灰影在干什么,模型读深度图会准得多。**明说灰色外观不要抄**,不然可能给你出一部灰片。+换成骨架视频,这两句就写成 `<Video 1>, an OpenPose skeleton video — coloured stick figures on black, one per person: …` 和 `… the stick-figure look is not transferred.`。在 ComfyUI 里配 Fun ControlNet 权重时更省事:`骨架.mp4` 直接接 ControlNet 的 **pose** 口,不用在提示词里绕。网页会按你出的是哪种,打印对应的那几句。+### 4. 出来的是什么<p align="center"><img src="docs/img/step4_takes.jpg" width="720" alt="三段生成片"></p>@@ -118,6 +134,8 @@ is transferred onto <Subject 1> and <Subject 2>; its grey depth look is not tran**不能:** 脸(用定妆图)、衣服、光线、颜色、道具细节,以及比手还小的东西。这些全靠你的提示词和参考图。+**深度图还是骨架?** 深度图带走整个画面——墙在哪、谁更大、一根管子被扯下来——人和非人都管用。骨架只带人,但每根肢体每根手指都准,体型和场景一点不带。跳舞、武打选骨架,其余选深度图;拿不准就两个都出(`--control depth,pose`),看模型跟哪个跟得更好。有一种情况骨架不合适:大特写。肩和髋不在画面里时估计器照样会猜,把它们画在画面边上——特写老老实实用深度图。线稿会把服装和脸的轮廓也带过去,「只复制走位」对它只成立一半。+做演示时踩出来的几条:- **参考片控制在 15 秒以内**(Seedance 的上限),跑 ReShot 之前先把片子剪到你要的那几个镜头。片子里有什么就会复刻什么,包括结尾没用的那段。@@ -134,6 +152,8 @@ is transferred onto <Subject 1> and <Subject 2>; its grey depth look is not tran| **Apple 芯片** | 能 | 默认 `fast`:M2 Max 上 46 ms/帧,12 秒的片约 18 秒 || **纯 CPU** | 能,很慢 | 约 1.8 秒/帧 || **内存** | 16 GB 能跑 720p 约 27 秒 | 峰值 = 2 GB + 每秒 720p 约 224 MB;放不下会在开始之前拒绝 |+| **`--control pose`** | 以上都行;要先 `pip install "reshot[pose]"`(ONNX Runtime;N 卡装 `onnxruntime-gpu`) | M2 Max 走 CPU:306 ms/帧,12 秒的片 90 秒;两个权重文件共 340 MB,只下一次。内存约 1.4 GB |+| **`--control canny`** | 什么都行 | 不用模型;12 秒的片不到一秒 |**`--quality`——模型工作的分辨率。** 输出视频永远和原片一样大;这个选项定的是深度模型看到的那张图有多大。@@ -142,7 +162,9 @@ is transferred onto <Subject 1> and <Subject 2>; its grey depth look is not tran| `fast`(默认) | 644×364 · 364×644 · 490×364 | 约 3 GB | 3080 Ti 上 34 ms/帧 | 大形状与 full 一模一样;脸颊旁一缕散发会糊进脸里 || `full` | 924×518 · 518×924 · 686×518 | 约 11 GB | 3080 Ti 上 83 ms/帧 | 细轮廓更锐 |-网页上「画质」就是这两档;命令行用 `--quality`。+网页上「画质」就是这两档;命令行用 `--quality`。画质只管深度图——骨架模型的工作尺寸是固定的(检人 640 px,估点每人 288×384),线稿按输出尺寸逐像素算。++**骨架的速度旋钮——`--pose-detect-every N`(默认 3)。** CPU 上检人占骨架总耗时的 80%,所以检测器每三帧跑一次,中间两帧的人框由上一帧的骨架推出来;片子里出现剪切、或某个骨架分数掉到不可信,检测器立刻回来。在 289 帧的走廊片上和逐帧检测对比:280 帧人数完全一致,80% 的关节误差在 2 px 以内、91% 在 5 px 以内;差的 9 帧是有人进画面时骨架晚了一两帧出现。快 35%(M2 Max 上 306 对 469 ms/帧)。`--pose-detect-every 1` 关掉它。我们推荐 `fast`:复制走位和运镜,视频模型要的它全有——演示里的三段片,深度图交给 MiniMax H3 之前还缩到了 320×176,比这两档都小得多。同一段 294 帧的片上 full 对 fast 实测:平均差 5.3 个灰阶(满量程 255),95% 的像素差在 15 以内,边缘锐度低 4.5%。特写里在意细轮廓、显存又够,再用 `full`。运行时会打印实际用的分辨率(`model fast: the model sees 644x364`),也写进 `--metrics`。要指定任意短边用 `--input-size`。@@ -168,24 +190,39 @@ from reshot import RunConfig, runrun(RunConfig(input=Path("参考片.mp4"), output=Path("深度图.mp4"), target="seedance"))```-三个细节决定了视频模型愿不愿意跟着这段灰片走:+四个细节决定了视频模型愿不愿意跟着这段片走:- **整段只用一把尺子。** 深度对全部帧归一化一次,绝不逐帧归一化,有人从墙前走过,墙的灰度不变。逐帧归一化会让整个场景「呼吸」。+- **骨架不闪。** 逐帧的姿态估计会换人、抖关节、在阈值附近让肢体忽有忽无,视频模型把这三样全读成动作。ReShot 跨帧跟踪每个人(身体框 IoU 配对),关节一旦画上就要分数掉到远低于阈值才撤(迟滞),每个关节各过一个 One-Euro 滤波——静止时稳,出拳时跟得上。- **按时间戳选帧。** 30 fps 转 24 fps 就真的是 24,不会以某种规律重复或丢帧让模型学了去。- **只裁不补。** 尺寸裁到模型要的网格上,绝不补黑边——黑边会被当成远处的墙。-模型是 Video Depth Anything Small(字节跳动,CVPR 2025),以 32 帧为一个窗口重叠推理并对齐,深度不会一帧一帧地跳。12 秒 720p 一段片峰值占 3.9 GB 内存、11 GB 显存(`--input-size 364` 时 3 GB)(实测;开跑前显示的估算是拟合出来的直线,与实测误差 0.05 GB 以内),放不下会在开始之前拒绝。`fake` 后端不加载模型就能跑完整条流水线,方便你写测试。属于「用户该修」的错误都是 `ReshotError` 的子类,带退出码和具体的修法。+骨架从 Python 调:++```python+from reshot import RunConfig, run+run(RunConfig(input=Path("参考片.mp4"), output=Path("骨架.mp4"), target="h3", control="pose", keypoints=Path("骨架.json")))++from reshot.backends import get_backend+from reshot.pose.tracking import stabilise+from reshot.pose.skeleton import render_frame+clip = get_backend("dwpose").infer(frames, fps) # frames: uint8 RGB [T, H, W, 3] → PoseClip+stabilise(clip) # 跟踪 → 迟滞 → 平滑,原地改+rgb = render_frame(clip.keypoints[0], clip.scores[0], h, w, hands=True, face=False)+```++深度模型是 Video Depth Anything Small(字节跳动,CVPR 2025),以 32 帧为一个窗口重叠推理并对齐,深度不会一帧一帧地跳。12 秒 720p 一段片峰值占 3.9 GB 内存、11 GB 显存(`--input-size 364` 时 3 GB)(实测;开跑前显示的估算是拟合出来的直线,与实测误差 0.05 GB 以内),放不下会在开始之前拒绝。`fake` 后端不加载模型就能跑完整条流水线,方便你写测试。属于「用户该修」的错误都是 `ReshotError` 的子类,带退出码和具体的修法。## 社区- **做出片了?** 把参考片、深度图、成片和你的提示词那一行发到 [作品分享](https://github.com/maosika-ai/reshot/discussions/2),好的会链到这里。- **模型没照深度图走?** 到 [问答区](https://github.com/maosika-ai/reshot/discussions/categories/q-a) 贴一帧画面和提示词——多半是提示词的事。- **出错了?** [提 issue](https://github.com/maosika-ai/reshot/issues/new/choose),一天内回复。-- **想搭把手?** 从 [good first issue](https://github.com/maosika-ai/reshot/labels/good%20first%20issue) 开始:给别的模型加预设、写个 Colab、翻译 README。骨骼输出是最大的一件,见 [#6](https://github.com/maosika-ai/reshot/issues/6)。+- **想搭把手?** 从 [good first issue](https://github.com/maosika-ai/reshot/labels/good%20first%20issue) 开始:给别的模型加预设、翻译 README、给 H3 Fun ControlNet 剩下两个口做 HED / MLSD 输出。骨架已在 0.5.0 落地([#6](https://github.com/maosika-ai/reshot/issues/6))。## 协议-Apache-2.0,默认模型也是。模型代码 vendored 在 `reshot/third_party/`。用在产品、流水线、服务里都行。更大的研究用权重(Base、Large)是 CC-BY-NC,不主动指定不会加载。+Apache-2.0,默认深度模型和骨架模型(DWPose,代码和权重)也都是。模型代码 vendored 在 `reshot/third_party/`;骨架的前后处理是照 DWPose 的 ONNX 参考实现用 NumPy 重写的,[NOTICE](NOTICE) 里有署名。用在产品、流水线、服务里都行。更大的研究用深度权重(Base、Large)是 CC-BY-NC,不主动指定不会加载。欢迎 issue 和 PR,见 [CONTRIBUTING.md](CONTRIBUTING.md)。diff --git a/docs/AGENT_INSTALL.md b/docs/AGENT_INSTALL.mdindex 4452387..57ac9a3 100644--- a/docs/AGENT_INSTALL.md+++ b/docs/AGENT_INSTALL.md@@ -39,10 +39,13 @@ Prefer a virtual environment (`python3 -m venv .venv && source .venv/bin/activat## 3. Install ReShot```bash-pip install "reshot[ffmpeg]" # or just: pip install reshot (when ffmpeg is on PATH)-reshot --version # prints e.g. "reshot 0.3.7"+pip install "reshot[ffmpeg,pose]" # pose = ONNX Runtime for --control pose; or just: pip install reshot+reshot --version # prints e.g. "reshot 0.5.0"```+On a machine with an NVIDIA card, `pip install onnxruntime-gpu` (instead of the `pose` extra) makes+the skeleton output run on the GPU too; the CPU build works everywhere and is fine for short clips.+Verify torch sees the GPU:```bash@@ -64,14 +67,18 @@ Offline machine: fetch `video_depth_anything_vits.pth` from`https://huggingface.co/depth-anything/Video-Depth-Anything-Small` elsewhere and pass`--checkpoint /path/to/video_depth_anything_vits.pth`.+`--control pose` downloads two more files on its first run (`yolox_l.onnx`, `dw-ll_ucoco_384.onnx`,+~340 MB together, from `https://huggingface.co/yzd-v/DWPose`); offline, put both in one folder and+pass `--checkpoint-dir /that/folder`.+## 5. Smoke test without the model (seconds, no download)```bash-RESHOT_FAKE_BACKEND=1 reshot <any short .mp4> -o /tmp/reshot_smoke.mp4 --max-frames 8 --target seedance+RESHOT_FAKE_BACKEND=1 reshot <any short .mp4> -o /tmp/reshot_smoke/ --max-frames 8 --target seedance --control depth,pose,canny```(Windows PowerShell: `$env:RESHOT_FAKE_BACKEND=1; reshot …`.) This exercises decoding, presets and-encoding with a fake depth model. It must print a `wrote` line. If it fails, the problem is ffmpeg+encoding with fake depth and pose models (canny needs none). It must print three `wrote` lines. If it fails, the problem is ffmpegor the input file, not the model.## 6. Real run@@ -90,6 +97,17 @@ Defaults are right for almost everyone: `--quality fast` (the model works at 644~3 GB VRAM). Only suggest `--quality full` (924×518, ~11 GB VRAM) if the card has ≥ 12 GB and theuser cares about fine silhouettes.+If the user's clip is a dance or a fight, also make the skeleton version — it is what pose+ControlNets want and it is the more precise of the two for limbs:++```bash+reshot reference.mp4 -o pose.mp4 --target h3 --control pose --keypoints pose.json --metrics pose_run.json+```++`pose.mp4` must show coloured stick figures on black that follow the people; `people_per_frame_max`+in the metrics says how many it found. If it is 0 for a clip with people, the detector never fired —+report it as a bug with the clip's resolution and fps.+## 7. What to tell the user at the end- The exact command that worked on their machine, ready to reuse.@@ -97,6 +115,8 @@ user cares about fine silhouettes.prompt like `参考@视频1的动作与运镜,顺序与视频保持一致。` + their own description of people and look —or attach it as `<Video 1>` for MiniMax H3 (see the README's step 3 for the prompt lines that matter).- For MiniMax H3 reference videos add `--target h3 --max-res 320`.+- `--control pose` (skeleton) and `--control canny` (lines) exist too; `--control depth,pose -o out/`+ writes both in one go.- Batch: `reshot clips/*.mp4 -o depth/ --target seedance` runs a folder on one model load.- ComfyUI users: install the node pack https://github.com/maosika-ai/ComfyUI-ReShot instead of the CLI.diff --git a/docs/PYPI.md b/docs/PYPI.mdindex 3ccef33..fc8980d 100644--- a/docs/PYPI.md+++ b/docs/PYPI.md@@ -2,15 +2,18 @@**Copy the shot, not the actors.**-ReShot turns a reference video into a depth-map video (near = white, far = black), so Seedance, MiniMax H3 or Wan can repeat its choreography and camera moves with your own characters.+ReShot turns a reference video into a depth-map video (near = white, far = black), an OpenPose-style skeleton video or a canny line video, so Seedance, MiniMax H3 or Wan can repeat its choreography and camera moves with your own characters.```bash-pip install reshot+pip install "reshot[pose]"reshot # web page: opens in your browser, drop the clip inreshot reference.mp4 -o depth.mp4 --target seedance # or the command line+reshot reference.mp4 -o pose.mp4 --target h3 --control pose # skeletons (DWPose), hands included+reshot reference.mp4 -o out/ --control depth,pose,canny # all three at once```-- **Web page** (`reshot` with nothing after it): everything runs on your computer; reference and depth map side by side, download, and the prompt line for Seedance / MiniMax H3 to copy.+- **Web page** (`reshot` with nothing after it): everything runs on your computer; pick depth map / skeleton / lines, reference and result side by side, download, and the prompt line for Seedance / MiniMax H3 to copy.+- **Skeletons** tracked across frames (identity, visibility hysteresis, One-Euro smoothing) so they don't flicker; `--keypoints` dumps them as JSON.- Presets for **Seedance 2.0 / 2.5** reference video (`--target seedance`), **MiniMax H3** (`h3`), **Wan 2.1 VACE** (`wan`).- `--quality fast` (default; the model sees 644×364 for 16:9, ~3 GB VRAM) or `full` (924×518, ~11 GB).- Depth normalised once over the whole clip — no flicker; frames picked by timestamp; sizes cropped to the model's grid, never padded.diff --git a/docs/USAGE.md b/docs/USAGE.mdindex 4641c6c..19127b4 100644--- a/docs/USAGE.md+++ b/docs/USAGE.md@@ -20,22 +20,28 @@ pip install reshot# no ffmpeg? this extra bundles a static binarypip install "reshot[ffmpeg]"++# skeletons (--control pose) need ONNX Runtime; on an NVIDIA card use onnxruntime-gpu instead+pip install "reshot[pose]"```PyTorch is not pinned to a CUDA version; install the build for your machine first if`pip` picks the wrong one (see https://pytorch.org/get-started/locally/).-Model weights (111 MB) download from Hugging Face on first run into `~/.cache/huggingface`.-Offline machines: copy that cache, or pass `--checkpoint /path/video_depth_anything_vits.pth`.+Model weights download from Hugging Face on first run into `~/.cache/huggingface`: 111 MB for+depth, and for `--control pose` two ONNX files (`yolox_l.onnx` + `dw-ll_ucoco_384.onnx`, ~340 MB)+from `yzd-v/DWPose`. Offline machines: copy that cache, or pass `--checkpoint+/path/video_depth_anything_vits.pth` (depth) / `--checkpoint-dir /folder/with/both.onnx` (pose).In China: `export HF_ENDPOINT=https://hf-mirror.com`.## Web page`reshot` with no arguments starts a local web page and opens it in your browser. It is the-same pipeline behind a form: drop a clip, choose the target model and quality, run, preview-reference and depth side by side, download. Nothing leaves your machine; the server listens on-127.0.0.1 only. Results are saved to `~/ReShot` with `<clip>_depth_<target>.mp4` names (never-overwriting an earlier result) and a `.json` of metrics next to each.+same pipeline behind a form: drop a clip, choose what to make (depth map / skeleton / lines),+the target model and quality, run, preview reference and result side by side, download. Nothing+leaves your machine; the server listens on 127.0.0.1 only. Results are saved to `~/ReShot` with+`<clip>_<control>_<target>.mp4` names (never overwriting an earlier result), a `.json` of metrics+next to each, and for skeletons a `.keypoints.json` with the pose data (a download button too).```bashreshot # http://127.0.0.1:8765 (next free port if taken)@@ -43,7 +49,7 @@ reshot web --port 9000 --out ./depth # choose port and folderreshot web --no-browser # print the URL only, e.g. over SSH```-The model loads once when the first job runs and stays loaded; later runs start immediately.+Each model loads once when its first job runs and stays loaded; later runs start immediately.Jobs queue one after another (one GPU).## Command line@@ -52,7 +58,20 @@ Jobs queue one after another (one GPU).reshot INPUT -o OUTPUT [options]```-### Model+### Control type++| option | default | notes |+|---|---|---|+| `--control depth\|pose\|canny` | `depth` | What the output shows. Several at once with commas (`depth,pose`), then `-o` must be a directory and files are named `<name>_<control>.mp4`. |+| `--no-hands` | off | pose: body only, skip the 21-point hands. |+| `--face` | off | pose: also draw the 68 face points. Off because the face shape is what ReShot is meant to throw away. |+| `--no-smooth` | off | pose: raw per-frame output — no identity tracking, no visibility hysteresis, no One-Euro smoothing. For debugging the estimator. |+| `--pose-detect-every N` | `3` | pose: run the person detector every N frames and follow the skeletons in between; cuts and unreliable skeletons re-detect at once. `1` = every frame, ~1.5× slower on CPU. Measured N=3 vs 1 on the 289-frame demo: 280 frames identical people count, 80 % of joints within 2 px, 91 % within 5 px. |+| `--keypoints PATH` | – | pose: also write the skeletons as JSON, format `reshot-pose/1` (see below). |+| `--checkpoint-dir DIR` | – | pose: folder with `yolox_l.onnx` and `dw-ll_ucoco_384.onnx`, skips the download. |+| `--canny LOW,HIGH` | `100,200` | canny: hysteresis thresholds. Lower LOW = more lines; higher HIGH = only strong edges. |++### Depth model| option | default | notes ||---|---|---|@@ -68,13 +87,13 @@ reshot INPUT -o OUTPUT [options]|---|---|---|| `--target none\|seedance\|h3\|wan` | `none` | fps + frame-size preset, see [Targets](#targets). || `--fps N` | preset or source | Frames are chosen by timestamp, so 30 → 24 yields 24, not 30. Never upsamples. |-| `--max-res N` | `1280` | Cap on the output's longer side. Inference always runs at model resolution regardless. |+| `--max-res N` | `1280` | Cap on the output's longer side. Depth inference always runs at model resolution regardless; pose and canny read frames at the output size. || `--max-frames N` | all | Stop after N source frames. || `--invert` | off | far = white. Default is near = white (what depth ControlNets expect). || `--clip PCT` | `0` | Trim PCT % from each tail before scaling to 0–255, so one hot pixel can't crush contrast. `0` = exact min/max. || `--gamma G` | `1.0` | > 1 darkens mid-tones: more separation close to the camera. || `--crf N` | `12` | x264 quality. Generous on purpose — banding in a control video becomes jitter in the generated clip. |-| `--npz PATH` | – | Raw float32 depth at processing resolution (`depths[T,H,W]`, `fps`). |+| `--npz PATH` | – | depth: raw float32 depth at processing resolution (`depths[T,H,W]`, `fps`). || `--metrics PATH` | – | JSON with config, plan, timings, peak RSS, output size. |### Safety@@ -100,7 +119,7 @@ reshot INPUT -o OUTPUT [options]|---|---|---|---|---|| `none` | source | even | – | || `seedance` | 24 | multiple of 16 (centre crop), ≥ 407,696 px (warns) | 15 s (warns) | Seedance 2.0 / 2.5 reference video (`@视频N` motion + camera) |-| `h3` | 24 | multiple of 32 (centre crop) | 15 s (warns) | MiniMax-H3-Fun-Controlnet-Union depth input |+| `h3` | 24 | multiple of 32 (centre crop) | 15 s (warns) | MiniMax-H3-Fun-Controlnet-Union depth / pose / canny input, or reference video || `wan` | 16 | multiple of 16 | – | Wan 2.1 VACE control video; values from the docs, not yet verified end-to-end |Frame size is **cropped**, never padded: a padded black border reads to the generator as a far wall.@@ -130,8 +149,49 @@ write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 7Errors that are the *user's* to fix are subclasses of `reshot.ReshotError` and carry an`exit_code`; anything else is a bug.-`RunConfig(..., backend="fake")` runs the entire pipeline with a synthetic depth field and no-model — handy for testing your own integration in milliseconds.+`RunConfig(..., backend="fake")` runs the entire pipeline with a synthetic depth field (or, for+`control="pose"`, two walking stick figures) and no model — handy for testing your own integration+in milliseconds.++### Pose++```python+from reshot import RunConfig, run+run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), target="h3", control="pose", keypoints=Path("pose.json")))+```++The pieces, for your own pipeline:++```python+from reshot import read_video, write_rgb_video+from reshot.backends import get_backend+from reshot.pose.tracking import stabilise, track, smooth, apply_hysteresis+from reshot.pose.skeleton import render_frame, LAYOUT++frames, fps = read_video("in.mp4") # uint8 RGB [T, H, W, 3]+clip = get_backend("dwpose").infer(frames, fps) # PoseClip: keypoints[t] = [N, 134, 2] px, scores[t] = [N, 134]+stabilise(clip) # = track → apply_hysteresis → smooth, in place+rgb = [render_frame(k, s, 480, 864, src_w=clip.width, src_h=clip.height) for k, s in zip(clip.keypoints, clip.scores)]+write_rgb_video(np.stack(rgb), "pose.mp4", fps)+clip.to_json_dict() # the same structure as --keypoints+```++**Keypoint layout** (`LAYOUT`): 0–17 body in OpenPose-18 order (nose, neck, R-shoulder, R-elbow,+R-wrist, L-shoulder, L-elbow, L-wrist, R-hip, R-knee, R-ankle, L-hip, L-knee, L-ankle, R-eye,+L-eye, R-ear, L-ear), 18–23 feet, 24–91 face (68), 92–112 left hand (21), 113–133 right hand (21).+Scores are 0–1; a joint below 0.3 is not drawn.++**`--keypoints` JSON** (`"format": "reshot-pose/1"`): `width`, `height`, `fps`, `layout`, then+`frames[t].people[n]` with `id` (stable across frames), `keypoints` (134 × [x, y] in pixels of the+*processing* frame — `width × height` — not the cropped output) and `scores` (134). Written after+tracking / hysteresis / smoothing, so it matches the video exactly.++### Canny++```python+from reshot.edges import canny_frame+edges = canny_frame(rgb_frame, out_h, out_w, low=100, high=200) # uint8 [out_h, out_w], 0 or 255+```## Recipes@@ -161,10 +221,13 @@ model — handy for testing your own integration in milliseconds.### MiniMax H3 Fun ControlNet (ComfyUI)-1. `reshot ref.mp4 -o ref_depth.mp4 --target h3` (24 fps, ×32, ≤ 15 s).+1. `reshot ref.mp4 -o ref_depth.mp4 --target h3` (24 fps, ×32, ≤ 15 s) — or `--control pose` /+ `--control canny` for the other two conditions the Union weights accept.2. In ComfyUI with the [Fun ControlNet Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union)- weights, load `ref_depth.mp4` as the control video, choose the **depth** condition, write your+ weights, load the control video, choose the matching condition (**depth** / **pose** / **canny**), write yourprompt for the *look* (cast, wardrobe, lighting, style). Staging and camera come from the control video.+ Or skip the files: the [ComfyUI-ReShot](https://github.com/maosika-ai/ComfyUI-ReShot) pack has+ *ReShot Depth / Pose / Canny Video* nodes that take a Load Video node directly.3. Keep the generator's resolution equal to the control video's, or let the node resize — but do notpad.@@ -216,10 +279,15 @@ records the exact config and plan) a run is fully reproducible.## How it works```-input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode at model resolution- ─▶ Video Depth Anything (32-frame windows, 10-frame overlap, cross-window alignment)- ─▶ whole-clip normalisation → uint8, near = white- ─▶ per-frame upscale to output size ─▶ centre-crop to target multiple ─▶ x264+input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode+ depth: at model resolution ─▶ Video Depth Anything (32-frame windows, 10-frame overlap, alignment)+ ─▶ whole-clip normalisation → uint8, near = white ─▶ per-frame upscale to output size+ pose: at output size ─▶ YOLOX-L person boxes (every 3rd frame; skeleton-tracked boxes between)+ ─▶ RTMPose whole-body, 133 points per person ─▶ OpenPose-18 layout + neck+ ─▶ track ids (IoU) ─▶ visibility hysteresis (on > 0.3, off < 0.2) ─▶ One-Euro per joint+ ─▶ OpenPose render: limbs as ellipses at 60 %, joints on top, hands as rainbow edges+ canny: at output size ─▶ grey ─▶ 3×3 Gaussian ─▶ cv2.Canny(low, high)+ ─▶ centre-crop to target multiple ─▶ x264 (grey for depth / canny, rgb24 for pose)```- **Model**: [Video Depth Anything](https://github.com/DepthAnything/Video-Depth-Anything)@@ -229,6 +297,10 @@ input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode at modelclip. Feeding larger frames only inflates the float depth kept per frame; reshot decodes at thatsize and upscales the 8-bit result to the source size.- **Normalisation**: `(d − min) / (max − min)` over *all* frames; optional percentile clip.+- **Pose model**: [DWPose](https://github.com/IDEA-Research/DWPose) (ICCV 2023 workshop), the+ estimator behind `controlnet_aux`'s pose images — so the skeletons look like what the video+ ControlNets were trained on. Apache-2.0 code and weights. The detector gets BGR, the estimator RGB+ with ImageNet normalisation, matching their mmdet / mmpose training configs.- **Encoding**: `libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart`.## Troubleshooting@@ -244,6 +316,10 @@ input video ─▶ probe ─▶ plan (sizes, frames, RAM) ─▶ decode at model| output smaller than source | `--target` crop or `--max-res` | expected; see Targets || banding in the depth map | `--crf` raised | keep ≤ 14 || people look flat / merged with background | scene has little depth range | try `--clip 0.5 --gamma 1.3` |+| `pose needs ONNX Runtime` | the `pose` extra is not installed | `pip install "reshot[pose]"` (or `onnxruntime-gpu`) |+| skeleton video is black / `no person found` | the detector saw nobody: people too small, or a drawing / animal the person detector does not know | try a tighter crop of the source; for non-humans use depth |+| a limb pops in and out | the estimator is unsure of it | that is the hysteresis already; `--no-smooth` to see the raw output, or accept — the model reads a missing limb better than a jumping one |+| skeleton lags a fast move | One-Euro smoothing | `--no-smooth`; lag is small by design (beta 0.5) |## FAQ@@ -255,7 +331,7 @@ choice for commercial work.**Why near = white?** Depth ControlNets (H3 Fun, Wan VACE, SD depth) were trained on inverse-depthmaps where closer is brighter. `--invert` if a tool wants the opposite.-**Can it output pose / canny / normals?** Not yet — on the roadmap.+**Can it output pose / canny / normals?** Pose and canny since 0.5.0 (`--control pose`, `--control canny`). Normals, HED and MLSD are not planned unless someone asks.**Does it remove the background / keep only people?** Not yet (`--people-only` is on the roadmap).The full-frame depth already carries no identity or style.diff --git a/docs/USAGE.zh-CN.md b/docs/USAGE.zh-CN.mdindex 884ee90..f8e8d9b 100644--- a/docs/USAGE.zh-CN.md+++ b/docs/USAGE.zh-CN.md@@ -20,19 +20,23 @@ pip install reshot# 没有 ffmpeg?这个 extra 自带一个静态二进制pip install "reshot[ffmpeg]"++# 骨架(--control pose)要 ONNX Runtime;N 卡改装 onnxruntime-gpu+pip install "reshot[pose]"```PyTorch 不锁 CUDA 版本;如果 `pip` 装错了,先按 https://pytorch.org/get-started/locally/ 装好再装 reshot。-模型权重(111 MB)首次运行时从 Hugging Face 下到 `~/.cache/huggingface`。-离线机器:把这个缓存目录拷过去,或用 `--checkpoint /路径/video_depth_anything_vits.pth`。-国内:`export HF_ENDPOINT=https://hf-mirror.com`。+模型权重首次运行时从 Hugging Face 下到 `~/.cache/huggingface`:深度 111 MB;`--control pose`+另下两个 ONNX 文件(`yolox_l.onnx` + `dw-ll_ucoco_384.onnx`,约 340 MB,仓库 `yzd-v/DWPose`)。+离线机器:把这个缓存目录拷过去,或用 `--checkpoint /路径/video_depth_anything_vits.pth`(深度)/+`--checkpoint-dir /放两个onnx的目录`(骨架)。国内:`export HF_ENDPOINT=https://hf-mirror.com`。## 网页只敲 `reshot` 不带参数,会在本机起一个网页并用浏览器打开。它和命令行是同一条流水线,只是套了个表单:-拖入片子、选给哪家模型用和画质、运行、参考片与深度图并排预览、下载。什么都不会离开你的电脑,服务只监听-127.0.0.1。结果存在 `~/ReShot`,文件名 `<片名>_depth_<目标>.mp4`(不会覆盖之前的结果),旁边有一份同名 `.json` 指标。+拖入片子、选出什么(深度图 / 骨架 / 线稿)、给哪家模型用和画质、运行、参考片与结果并排预览、下载。什么都不会离开你的电脑,服务只监听+127.0.0.1。结果存在 `~/ReShot`,文件名 `<片名>_<类型>_<目标>.mp4`(不会覆盖之前的结果),旁边有一份同名 `.json` 指标;骨架另有一份 `.keypoints.json` 骨架数据(页面上也有下载按钮)。```bashreshot # http://127.0.0.1:8765(被占就用下一个空闲端口)@@ -40,7 +44,7 @@ reshot web --port 9000 --out ./depth # 指定端口和目录reshot web --no-browser # 只打印地址不开浏览器,比如通过 SSH 用```-模型在第一次运行时加载一次,之后常驻;后面的任务立刻开始。任务排队一个个跑(一块显卡)。+每种模型在第一次用到时加载一次,之后常驻;后面的任务立刻开始。任务排队一个个跑(一块显卡)。## 命令行@@ -48,7 +52,20 @@ reshot web --no-browser # 只打印地址不开浏览器,比reshot 输入 -o 输出 [选项]```-### 模型+### 出什么++| 选项 | 默认 | 说明 |+|---|---|---|+| `--control depth\|pose\|canny` | `depth` | 输出哪种控制视频。逗号隔开可以一次出几种(`depth,pose`),这时 `-o` 必须是目录,文件名 `<片名>_<类型>.mp4`。 |+| `--no-hands` | 关 | 骨架:只画身体,不画 21 点的手。 |+| `--face` | 关 | 骨架:连 68 个脸点一起画。默认不画——脸型正是 ReShot 要扔掉的东西。 |+| `--no-smooth` | 关 | 骨架:逐帧原始结果——不跟踪身份、不做可见性迟滞、不做 One-Euro 平滑。排查估计器时用。 |+| `--pose-detect-every N` | `3` | 骨架:每 N 帧跑一次检人,中间帧用上一帧的骨架推框;剪切和骨架不可信时立刻重检。`1` = 每帧都检,CPU 上慢约 1.5 倍。289 帧演示片上 N=3 对 1 实测:280 帧人数一致、80% 关节误差 <2 px、91% <5 px。 |+| `--keypoints 路径` | – | 骨架:同时把骨架数据存成 JSON,格式 `reshot-pose/1`(见下)。 |+| `--checkpoint-dir 目录` | – | 骨架:放着 `yolox_l.onnx` 和 `dw-ll_ucoco_384.onnx` 的目录,跳过下载。 |+| `--canny 低,高` | `100,200` | 线稿:迟滞阈值。低阈值越低线越多;高阈值越高只留强边。 |++### 深度模型| 选项 | 默认 | 说明 ||---|---|---|@@ -64,7 +81,7 @@ reshot 输入 -o 输出 [选项]|---|---|---|| `--target none\|seedance\|h3\|wan` | `none` | fps + 尺寸预设,见[目标预设](#目标预设)。 || `--fps N` | 预设或原片 | 按**时间戳**选帧,30→24 真的是 24。绝不向上插帧。 |-| `--max-res N` | `1280` | 输出长边上限。推理永远在模型分辨率进行,与此无关。 |+| `--max-res N` | `1280` | 输出长边上限。深度推理永远在模型分辨率进行,与此无关;骨架和线稿按输出尺寸读帧。 || `--max-frames N` | 全部 | 只处理前 N 帧。 || `--invert` | 关 | 改成远白近黑。默认近白远黑(深度 ControlNet 要的就是这个)。 || `--clip PCT` | `0` | 两端各裁 PCT% 再拉到 0–255,防一个热点像素压暗全片。`0` 为精确 min/max。 |@@ -96,7 +113,7 @@ reshot 输入 -o 输出 [选项]|---|---|---|---|---|| `none` | 原片 | 偶数 | – | || `seedance` | 24 | 16 的倍数(居中裁),≥ 407,696 像素(不够警告) | 15 秒(超了警告) | Seedance 2.0 / 2.5 参考视频(`@视频N` 继承动作与运镜) |-| `h3` | 24 | 32 的倍数(居中裁) | 15 秒(超了警告) | MiniMax-H3-Fun-Controlnet-Union 深度输入 |+| `h3` | 24 | 32 的倍数(居中裁) | 15 秒(超了警告) | MiniMax-H3-Fun-Controlnet-Union 的 depth / pose / canny 输入,或参考视频 || `wan` | 16 | 16 的倍数 | – | Wan 2.1 VACE 控制视频;按文档取值,尚未端到端实测 |尺寸用**裁**不用补:补上去的黑边在生成模型眼里是一堵远墙。@@ -125,7 +142,41 @@ write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 7属于"用户该修"的错误都是 `reshot.ReshotError` 的子类、带 `exit_code`;其余都是 bug。-`RunConfig(..., backend="fake")` 用合成深度场跑完整条流水线、不载模型——几毫秒验证你自己的集成。+`RunConfig(..., backend="fake")` 用合成深度场(`control="pose"` 时是两个走路的火柴人)跑完整条流水线、不载模型——几毫秒验证你自己的集成。++### 骨架++```python+from reshot import RunConfig, run+run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), target="h3", control="pose", keypoints=Path("pose.json")))+```++拆开用:++```python+from reshot import read_video, write_rgb_video+from reshot.backends import get_backend+from reshot.pose.tracking import stabilise, track, smooth, apply_hysteresis+from reshot.pose.skeleton import render_frame, LAYOUT++frames, fps = read_video("in.mp4") # uint8 RGB [T, H, W, 3]+clip = get_backend("dwpose").infer(frames, fps) # PoseClip:keypoints[t] = [N, 134, 2] 像素,scores[t] = [N, 134]+stabilise(clip) # = track → apply_hysteresis → smooth,原地改+rgb = [render_frame(k, s, 480, 864, src_w=clip.width, src_h=clip.height) for k, s in zip(clip.keypoints, clip.scores)]+write_rgb_video(np.stack(rgb), "pose.mp4", fps)+clip.to_json_dict() # 和 --keypoints 同一结构+```++**关键点布局**(`LAYOUT`):0–17 身体,OpenPose-18 顺序(鼻、颈、右肩、右肘、右腕、左肩、左肘、左腕、右髋、右膝、右踝、左髋、左膝、左踝、右眼、左眼、右耳、左耳),18–23 脚,24–91 脸(68),92–112 左手(21),113–133 右手(21)。分数 0–1,低于 0.3 不画。++**`--keypoints` 的 JSON**(`"format": "reshot-pose/1"`):`width`、`height`、`fps`、`layout`,然后 `frames[t].people[n]` 带 `id`(跨帧稳定)、`keypoints`(134 × [x, y],是**处理帧** `width × height` 的像素坐标,不是裁剪后的输出)、`scores`(134)。在跟踪 / 迟滞 / 平滑之后写出,和视频严格一致。++### 线稿++```python+from reshot.edges import canny_frame+edges = canny_frame(rgb_frame, out_h, out_w, low=100, high=200) # uint8 [out_h, out_w],0 或 255+```## 实战配方@@ -154,10 +205,11 @@ write_gray_video(upsample_frames(gray, 1280, 736), "out.mp4", fps, size=(1280, 7### MiniMax H3 Fun ControlNet(ComfyUI)-1. `reshot 参考.mp4 -o 参考_depth.mp4 --target h3`(24 fps、32 倍数、≤15 秒)。+1. `reshot 参考.mp4 -o 参考_depth.mp4 --target h3`(24 fps、32 倍数、≤15 秒)——Union 权重另外两个口用 `--control pose` / `--control canny`。2. ComfyUI 里装 [Fun ControlNet Union](https://huggingface.co/alibaba-pai/MiniMax-H3-Fun-Controlnet-Union) 权重,- 把 `参考_depth.mp4` 作为控制视频、条件选 **depth**,提示词只写**长相**(人物、服装、光线、画风)。- 走位和镜头由控制视频决定。+ 把控制视频接进去、条件选对应的 **depth** / **pose** / **canny**,提示词只写**长相**(人物、服装、光线、画风)。+ 走位和镜头由控制视频决定。不想倒文件:[ComfyUI-ReShot](https://github.com/maosika-ai/ComfyUI-ReShot) 节点包有+ *ReShot Depth / Pose / Canny Video* 三个节点,直接接 Load Video。3. 生成分辨率和控制视频保持一致,或让节点缩放——但别补边。### Wan 2.1 VACE@@ -203,16 +255,22 @@ MPS 上 fp16 跑不完,所以强制 fp32。736×1280 约 0.5 秒/帧,12 秒## 原理```-输入视频 ─▶ 探测 ─▶ 计划(尺寸、帧数、内存)─▶ 按模型分辨率解码- ─▶ Video Depth Anything(32 帧窗口、10 帧重叠、跨窗口对齐)- ─▶ 整段归一化 → uint8,近白远黑- ─▶ 逐帧放大到输出尺寸 ─▶ 居中裁到目标倍数 ─▶ x264+输入视频 ─▶ 探测 ─▶ 计划(尺寸、帧数、内存)─▶ 解码+ 深度: 按模型分辨率 ─▶ Video Depth Anything(32 帧窗口、10 帧重叠、跨窗口对齐)+ ─▶ 整段归一化 → uint8,近白远黑 ─▶ 逐帧放大到输出尺寸+ 骨架: 按输出尺寸 ─▶ YOLOX-L 检人框(每 3 帧一次;中间帧用骨架推框)+ ─▶ RTMPose 全身 133 点 ─▶ 转 OpenPose-18 布局 + 颈点+ ─▶ 跨帧配 id(IoU)─▶ 可见性迟滞(>0.3 开、<0.2 关)─▶ 每关节 One-Euro+ ─▶ OpenPose 画法:肢体椭圆压到 60%、关节点盖在上面、手用彩虹色连线+ 线稿: 按输出尺寸 ─▶ 灰度 ─▶ 3×3 高斯 ─▶ cv2.Canny(低, 高)+ ─▶ 居中裁到目标倍数 ─▶ x264(深度 / 线稿走 gray,骨架走 rgb24)```- **模型**:[Video Depth Anything](https://github.com/DepthAnything/Video-Depth-Anything)(CVPR 2025)。它的时序模块是深度不闪的原因;单图模型逐帧跑必闪。模型代码原样 vendored 在 `third_party/`(只改两行 import)。- **分辨率**:模型工作在 `--quality` 定的尺寸——16:9 的片 fast 是 644×364、full 是 924×518。喂更大的帧只会让每帧存的浮点深度变大;reshot 按这个尺寸解码,最后把 8 位结果放大回原片尺寸。- **归一化**:`(d − min) / (max − min)` 对**全部帧**做一次;可选百分位裁剪。+- **骨架模型**:[DWPose](https://github.com/IDEA-Research/DWPose)(ICCV 2023 workshop),就是 `controlnet_aux` 出 pose 图用的那个估计器——所以我们的骨架长得和视频 ControlNet 训练时看到的一样。代码和权重都是 Apache-2.0。检测器喂 BGR、估计器喂 ImageNet 归一化的 RGB,和它们 mmdet / mmpose 的训练配置一致。- **编码**:`libx264 -crf 12 -pix_fmt yuv420p -g 2·fps -movflags +faststart`。## 排错@@ -228,6 +286,10 @@ MPS 上 fp16 跑不完,所以强制 fp32。736×1280 约 0.5 秒/帧,12 秒| 输出比原片小 | `--target` 裁剪或 `--max-res` | 正常,见目标预设 || 深度图有色带 | 调高了 `--crf` | 保持 ≤14 || 人和背景糊在一起 | 场景纵深本身很小 | 试 `--clip 0.5 --gamma 1.3` |+| `pose needs ONNX Runtime` | 没装 `pose` extra | `pip install "reshot[pose]"`(或 `onnxruntime-gpu`) |+| 骨架视频全黑 / `no person found` | 检测器没看到人:人太小,或是画 / 动物这类检人器不认的东西 | 把原片裁得更紧;非人类用深度图 |+| 某根肢体忽有忽无 | 估计器对它拿不准 | 迟滞已经在起作用;`--no-smooth` 看原始输出,或者接受——模型读「缺一根」比读「乱跳」强 |+| 骨架跟不上快动作 | One-Euro 平滑 | `--no-smooth`;滞后按设计很小(beta 0.5) |## 常见问题@@ -237,7 +299,7 @@ Base/Large 和 Small 的差距我们没有自己测过;它们是 CC-BY-NC,**为什么近白远黑?** 深度 ControlNet(H3 Fun、Wan VACE、SD depth)训练用的都是"越近越亮"的逆深度图。要反过来用 `--invert`。-**能出骨架 / 边缘 / 法线吗?** 还不能,在路线图上。+**能出骨架 / 边缘 / 法线吗?** 骨架和边缘 0.5.0 起有了(`--control pose`、`--control canny`)。法线、HED、MLSD 没人要就不做。**能去背景、只留人吗?** 还不能(`--people-only` 在路线图上)。全画面深度本身已经不含任何身份和画风信息。diff --git a/docs/demo-controls.gif b/docs/demo-controls.gifnew file mode 100644index 0000000..4a8f8c5Binary files /dev/null and b/docs/demo-controls.gif differdiff --git a/docs/img/controls_strip.jpg b/docs/img/controls_strip.jpgnew file mode 100644index 0000000..42db1e1Binary files /dev/null and b/docs/img/controls_strip.jpg differdiff --git a/docs/img/step2_canny.jpg b/docs/img/step2_canny.jpgnew file mode 100644index 0000000..356ac38Binary files /dev/null and b/docs/img/step2_canny.jpg differdiff --git a/docs/img/step2_pose.jpg b/docs/img/step2_pose.jpgnew file mode 100644index 0000000..476ad61Binary files /dev/null and b/docs/img/step2_pose.jpg differdiff --git a/docs/img/web_ui_en.jpg b/docs/img/web_ui_en.jpgindex b4c40c7..55b9ac6 100644Binary files a/docs/img/web_ui_en.jpg and b/docs/img/web_ui_en.jpg differdiff --git a/docs/img/web_ui_zh.jpg b/docs/img/web_ui_zh.jpgindex fb82a1b..2aa27eb 100644Binary files a/docs/img/web_ui_zh.jpg and b/docs/img/web_ui_zh.jpg differdiff --git a/examples/reshot_colab.ipynb b/examples/reshot_colab.ipynbindex 10269c7..6798bfd 100644--- a/examples/reshot_colab.ipynb+++ b/examples/reshot_colab.ipynb@@ -3,22 +3,7 @@{"cell_type": "markdown","metadata": {},- "source": [- "# ReShot — copy the shot, not the actors\n",- "\n",- "Turn a reference video into a **depth map video** on a free Colab GPU, then give that grey clip to\n",- "Seedance or MiniMax H3 as the reference: it repeats the choreography and the camera moves with\n",- "your own characters in it.\n",- "\n",- "**Before you run:** menu *Runtime → Change runtime type → T4 GPU*. Then *Runtime → Run all*.\n",- "Four steps: check the GPU · install · pick a clip · run and download.\n",- "\n",- "Code and docs: [github.com/maosika-ai/reshot](https://github.com/maosika-ai/reshot) · Apache-2.0 ·\n",- "made at [Maosika 猫斯卡](https://www.maosika.com), the AI short-drama production system.\n",- "\n",- "在免费的 Colab GPU 上把参考视频变成深度图视频,再交给 Seedance / MiniMax H3 当参考。\n",- "先在菜单 *代码执行程序 → 更改运行时类型* 选 **T4 GPU**,然后 *全部运行*。"- ]+ "source": "# ReShot — copy the shot, not the actors\n\nTurn a reference video into a **depth map, skeleton or line video** on a free Colab GPU, then give that grey clip to\nSeedance or MiniMax H3 as the reference: it repeats the choreography and the camera moves with\nyour own characters in it.\n\n**Before you run:** menu *Runtime → Change runtime type → T4 GPU*. Then *Runtime → Run all*.\nFour steps: check the GPU · install · pick a clip · run and download.\n\nCode and docs: [github.com/maosika-ai/reshot](https://github.com/maosika-ai/reshot) · Apache-2.0 ·\nmade at [Maosika 猫斯卡](https://www.maosika.com), the AI short-drama production system.\n\n在免费的 Colab GPU 上把参考视频变成深度图视频,再交给 Seedance / MiniMax H3 当参考。\n先在菜单 *代码执行程序 → 更改运行时类型* 选 **T4 GPU**,然后 *全部运行*。"},{"cell_type": "markdown",@@ -56,16 +41,7 @@},"execution_count": null,"outputs": [],- "source": [- "#@title Install ReShot (about a minute; Colab already has a CUDA build of PyTorch)\n",- "in_china = False #@param {type:\"boolean\"}\n",- "import os\n",- "if in_china:\n",- " os.environ[\"HF_ENDPOINT\"] = \"https://hf-mirror.com\" # the 111 MB weights come from Hugging Face\n",- "%pip install -q \"reshot[ffmpeg]\"\n",- "import reshot\n",- "print(\"reshot\", reshot.__version__)"- ]+ "source": "#@title Install ReShot (about a minute; Colab already has a CUDA build of PyTorch)\nin_china = False #@param {type:\"boolean\"}\nimport os\nif in_china:\n os.environ[\"HF_ENDPOINT\"] = \"https://hf-mirror.com\" # the 111 MB weights come from Hugging Face\n%pip install -q \"reshot[ffmpeg,pose]\"\nimport reshot\nprint(\"reshot\", reshot.__version__)"},{"cell_type": "markdown",@@ -102,9 +78,7 @@{"cell_type": "markdown","metadata": {},- "source": [- "## 4 · Make the depth map"- ]+ "source": "## 4 · Make the control video"},{"cell_type": "code",@@ -113,21 +87,7 @@},"execution_count": null,"outputs": [],- "source": [- "#@title Run (measured on a free T4: the 12 s demo clip takes about a minute at `fast`; `full` is ~2× that)\n",- "target = \"seedance\" #@param [\"seedance\", \"h3\", \"wan\", \"none\"]\n",- "quality = \"fast\" #@param [\"fast\", \"full\"]\n",- "import json, os, shutil, subprocess, time\n",- "t0 = time.time()\n",- "r = subprocess.run([shutil.which(\"reshot\") or \"reshot\", INPUT, \"-o\", \"depth.mp4\",\n",- " \"--target\", target, \"--quality\", quality, \"--metrics\", \"metrics.json\"],\n",- " capture_output=True, text=True)\n",- "print(r.stderr[-3000:])\n",- "assert r.returncode == 0 and os.path.exists(\"depth.mp4\"), \"run failed — read the message above\"\n",- "m = json.load(open(\"metrics.json\"))\n",- "print(f\"done in {time.time()-t0:.0f} s · {m['frames']} frames @ {m['fps']} fps · \"\n",- " f\"{m['output_size'][0]}×{m['output_size'][1]} · {m['ms_per_frame']:.0f} ms/frame on {m['device']}\")"- ]+ "source": "#@title Run (measured on a free T4: the 12 s demo clip takes about a minute at `fast`; `full` is ~2× that; `pose` is CPU-bound, a few minutes)\ncontrol = \"depth\" #@param [\"depth\", \"pose\", \"canny\"]\ntarget = \"seedance\" #@param [\"seedance\", \"h3\", \"wan\", \"none\"]\nquality = \"fast\" #@param [\"fast\", \"full\"]\nimport json, os, shutil, subprocess, time\nt0 = time.time()\nOUTPUT = f\"{control}.mp4\"\nr = subprocess.run([shutil.which(\"reshot\") or \"reshot\", INPUT, \"-o\", OUTPUT, \"--control\", control,\n \"--target\", target, \"--quality\", quality, \"--metrics\", \"metrics.json\", \"--keypoints\", \"pose.json\"],\n capture_output=True, text=True)\nprint(r.stderr[-3000:])\nassert r.returncode == 0 and os.path.exists(OUTPUT), \"run failed — read the message above\"\nm = json.load(open(\"metrics.json\"))\nprint(f\"done in {time.time()-t0:.0f} s · {m['frames']} frames @ {m['fps']} fps · \"\n f\"{m['output_size'][0]}×{m['output_size'][1]} · {m['ms_per_frame']:.0f} ms/frame on {m['device']}\")"},{"cell_type": "code",@@ -136,19 +96,7 @@},"execution_count": null,"outputs": [],- "source": [- "#@title Look at it: reference and depth side by side, then the two clips\n",- "import base64, cv2\n",- "from IPython.display import HTML, display\n",- "def frame(path, idx=0):\n",- " cap = cv2.VideoCapture(path); cap.set(cv2.CAP_PROP_POS_FRAMES, idx); ok, bgr = cap.read(); cap.release()\n",- " h = 360; bgr = cv2.resize(bgr, (int(bgr.shape[1] * h / bgr.shape[0]), h))\n",- " return base64.b64encode(cv2.imencode(\".jpg\", bgr)[1]).decode()\n",- "def video(path):\n",- " return f'<video controls loop muted autoplay width=\"360\" src=\"data:video/mp4;base64,{base64.b64encode(open(path,\"rb\").read()).decode()}\"></video>'\n",- "display(HTML(f'<div style=\"display:flex;gap:12px\"><img src=\"data:image/jpeg;base64,{frame(INPUT)}\"><img src=\"data:image/jpeg;base64,{frame(\"depth.mp4\")}\"></div>'\n",- " f'<div style=\"display:flex;gap:12px;margin-top:12px\">{video(INPUT)}{video(\"depth.mp4\")}</div>'))"- ]+ "source": "#@title Look at it: reference and result side by side, then the two clips\nimport base64, cv2\nfrom IPython.display import HTML, display\ndef frame(path, idx=0):\n cap = cv2.VideoCapture(path); cap.set(cv2.CAP_PROP_POS_FRAMES, idx); ok, bgr = cap.read(); cap.release()\n h = 360; bgr = cv2.resize(bgr, (int(bgr.shape[1] * h / bgr.shape[0]), h))\n return base64.b64encode(cv2.imencode(\".jpg\", bgr)[1]).decode()\ndef video(path):\n return f'<video controls loop muted autoplay width=\"360\" src=\"data:video/mp4;base64,{base64.b64encode(open(path,\"rb\").read()).decode()}\"></video>'\ndisplay(HTML(f'<div style=\"display:flex;gap:12px\"><img src=\"data:image/jpeg;base64,{frame(INPUT)}\"><img src=\"data:image/jpeg;base64,{frame(OUTPUT)}\"></div>'\n f'<div style=\"display:flex;gap:12px;margin-top:12px\">{video(INPUT)}{video(OUTPUT)}</div>'))"},{"cell_type": "markdown",@@ -164,22 +112,7 @@},"execution_count": null,"outputs": [],- "source": [- "#@title Download depth.mp4 and print the prompt line for your video model\n",- "from google.colab import files\n",- "NEXT = {\n",- " \"seedance\": (\"Upload depth.mp4 to Seedance 2.0 / 2.5 as a reference video and start the prompt with:\",\n",- " \"参考@视频1的动作与运镜,顺序与视频保持一致。\\n(then describe your people, wardrobe, set and look)\"),\n",- " \"h3\": (\"In MiniMax H3 attach depth.mp4 as <Video 1> (a character sheet as <Picture 1>); define it and say the grey look is not to be copied:\",\n",- " \"<Subject 3> is the choreography and camera movement shown in <Video 1>, a grey depth map in which near objects are white and far objects are black: (one sentence on what happens in the clip)\\n\"\n",- " \"<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; its grey depth look is not transferred.\"),\n",- " \"wan\": (\"Connect depth.mp4 to WanVaceToVideo → control_video in ComfyUI.\", \"\"),\n",- " \"none\": (\"Hand depth.mp4 to any model that reads a depth video.\", \"\"),\n",- "}\n",- "how, prompt = NEXT[target]\n",- "print(how); print(); print(prompt)\n",- "files.download(\"depth.mp4\")"- ]+ "source": "#@title Download the result and print the prompt line for your video model\nfrom google.colab import files\nLOOK = {\n \"depth\": (\"a grey depth map in which near objects are white and far objects are black\", \"its grey depth look is not transferred\"),\n \"pose\": (\"an OpenPose skeleton video — coloured stick figures on black, one per person\", \"the stick-figure look is not transferred\"),\n \"canny\": (\"a white-line edge drawing on black\", \"the line-drawing look is not transferred\"),\n}[control]\nSEEDANCE = {\n \"depth\": \"参考@视频1的动作与运镜,顺序与视频保持一致。\",\n \"pose\": \"参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。\",\n \"canny\": \"参考@视频1的构图、动作与运镜,顺序与视频保持一致;只取构图和动作,不要画成线稿。\",\n}[control]\nNEXT = {\n \"seedance\": (f\"Upload {OUTPUT} to Seedance 2.0 / 2.5 as a reference video and start the prompt with:\",\n SEEDANCE + \"\\n(then describe your people, wardrobe, set and look)\"),\n \"h3\": (f\"In MiniMax H3 attach {OUTPUT} as <Video 1> (a character sheet as <Picture 1>); define it and say its look is not to be copied:\",\n f\"<Subject 3> is the choreography and camera movement shown in <Video 1>, {LOOK[0]}: (one sentence on what happens in the clip)\\n\"\n f\"<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; {LOOK[1]}.\"),\n \"wan\": (f\"Connect {OUTPUT} to WanVaceToVideo → control_video in ComfyUI.\", \"\"),\n \"none\": (f\"Hand {OUTPUT} to any model that reads a {control} video.\", \"\"),\n}\nhow, prompt = NEXT[target]\nprint(how); print(); print(prompt)\nfiles.download(OUTPUT)\nif control == \"pose\" and os.path.exists(\"pose.json\"):\n files.download(\"pose.json\") # the skeletons as JSON, in case you want to edit or retarget them"},{"cell_type": "markdown",@@ -212,4 +145,4 @@},"nbformat": 4,"nbformat_minor": 0-}\ No newline at end of file+}diff --git a/pyproject.toml b/pyproject.tomlindex ce7dec9..a1b3575 100644--- a/pyproject.toml+++ b/pyproject.toml@@ -5,12 +5,12 @@ build-backend = "setuptools.build_meta"[project]name = "reshot"dynamic = ["version"]-description = "ReShot — copy the shot, not the actors. Video → depth map for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system."+description = "ReShot — copy the shot, not the actors. Video → depth map, OpenPose skeleton or canny lines for Seedance 2.0/2.5 reference video, MiniMax H3 Fun ControlNet and Wan VACE. Open-sourced by Maosika 猫斯卡 (www.maosika.com), the AI short-drama production system."readme = "docs/PYPI.md"license = { text = "Apache-2.0" }requires-python = ">=3.10"authors = [{ name = "Maosika 猫斯卡" }]-keywords = ["video", "depth", "controlnet", "video-generation", "depth-map", "depth-anything", "seedance", "minimax-h3", "ai-short-drama", "maosika"]+keywords = ["video", "depth", "pose", "openpose", "dwpose", "canny", "controlnet", "video-generation", "depth-map", "depth-anything", "seedance", "minimax-h3", "ai-short-drama", "maosika"]classifiers = ["License :: OSI Approved :: Apache Software License","Programming Language :: Python :: 3",@@ -35,7 +35,8 @@ dependencies = [[project.optional-dependencies]ffmpeg = ["imageio-ffmpeg>=0.4.7"] # bundled ffmpeg if none on PATH-dev = ["pytest>=8", "ruff"]+pose = ["onnxruntime>=1.16"] # --control pose (DWPose); on CUDA install onnxruntime-gpu instead+dev = ["pytest>=8", "ruff", "onnxruntime>=1.16"][project.scripts]reshot = "reshot.cli:main"diff --git a/reshot/__init__.py b/reshot/__init__.pyindex c86994a..c5a0172 100644--- a/reshot/__init__.py+++ b/reshot/__init__.py@@ -1,12 +1,16 @@-"""ReShot — turn any video into a depth-map video for video-generation control.+"""ReShot — turn any video into a depth-map, skeleton or line video for video-generation control.from pathlib import Pathfrom reshot import RunConfig, runrun(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))+run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), target="h3", control="pose"))from reshot import extract, to_gray, write_gray_videodepths, fps = extract("in.mp4") # float32 [T, H, W], larger = closerwrite_gray_video(to_gray(depths), "out.mp4", fps)++Pose building blocks live in `reshot.pose` (PoseClip, skeleton.render_frame, tracking.stabilise);+canny in `reshot.edges`."""from __future__ import annotations@@ -25,7 +29,7 @@ os.environ.setdefault("PYTORCH_MPS_LOW_WATERMARK_RATIO", "0.5")from ._version import __version__from .config import RunConfigfrom .errors import BackendError, InputError, RamBudgetError, ReshotError, ToolMissingError-from .io import probe_video, read_video, write_gray_video+from .io import probe_video, read_video, write_gray_video, write_rgb_videofrom .pipeline import Plan, RunResult, extract, model_input_resolution, plan, resolve_input_size, run, run_manyfrom .planning import memory_verdict, processing_max_resfrom .postprocess import to_gray, upsample_frames@@ -58,4 +62,5 @@ __all__ = ["to_gray","upsample_frames","write_gray_video",+ "write_rgb_video",]diff --git a/reshot/_version.py b/reshot/_version.pyindex 3d26edf..3d18726 100644--- a/reshot/_version.py+++ b/reshot/_version.py@@ -1 +1 @@-__version__ = "0.4.1"+__version__ = "0.5.0"diff --git a/reshot/backends/__init__.py b/reshot/backends/__init__.pyindex c518008..cdc6e1e 100644--- a/reshot/backends/__init__.py+++ b/reshot/backends/__init__.py@@ -1,4 +1,5 @@-"""Backend registry. `get_backend("vda", model="small", device="auto")`."""+"""Backend registry. `get_backend("vda", model="small", device="auto")` for depth,+`get_backend("dwpose", device="auto")` for pose; `"fake"` / `"fake-pose"` load nothing."""from __future__ import annotations@@ -6,10 +7,11 @@ from typing import Anyfrom .base import BackendInfo, DepthBackendfrom .fake import FakeBackend+from .fake_pose import FakePoseBackendfrom .vda import VDABackend, pick_device-def get_backend(name: str, **kwargs: Any) -> DepthBackend:+def get_backend(name: str, **kwargs: Any) -> Any:"""Instantiate a backend by name. Unknown names raise `BackendError`."""from ..errors import BackendError@@ -17,7 +19,13 @@ def get_backend(name: str, **kwargs: Any) -> DepthBackend:return VDABackend(**kwargs)if name == "fake":return FakeBackend()+ if name == "dwpose":+ from ..pose.dwpose import DWPoseBackend++ return DWPoseBackend(**kwargs)+ if name == "fake-pose":+ return FakePoseBackend()raise BackendError(f"unknown backend {name!r}")-__all__ = ["BackendInfo", "DepthBackend", "FakeBackend", "VDABackend", "get_backend", "pick_device"]+__all__ = ["BackendInfo", "DepthBackend", "FakeBackend", "FakePoseBackend", "VDABackend", "get_backend", "pick_device"]diff --git a/reshot/backends/fake_pose.py b/reshot/backends/fake_pose.pynew file mode 100644index 0000000..dddfd22--- /dev/null+++ b/reshot/backends/fake_pose.py@@ -0,0 +1,84 @@+"""A pose backend that draws two walking stick figures without loading a model.++Same purpose as :class:`reshot.backends.fake.FakeBackend`: exercise everything around+inference (tracking, rendering, encoding, the CLI, the web UI) in CI, in milliseconds.+Deterministic: frame `t` of a `w × h` clip always yields the same skeletons, and every+joint scores 0.9 except the right wrist, which dips to 0.1 every 7th frame so the+hysteresis path is exercised too."""++from __future__ import annotations++import numpy as np++from ..pose import PoseClip+from .base import BackendInfo+++def _figure(cx: float, cy: float, size: float, phase: float) -> np.ndarray:+ """134 points of a front-facing figure; limbs swing with `phase` (radians)."""+ s = size+ sw = 0.3 * s * np.sin(phase)+ p = np.zeros((134, 2), dtype=np.float32)+ p[0] = (cx, cy - 0.9 * s) # nose+ p[1] = (cx, cy - 0.7 * s) # neck+ p[2] = (cx - 0.25 * s, cy - 0.7 * s) # r shoulder+ p[3] = (cx - 0.3 * s, cy - 0.35 * s + sw) # r elbow+ p[4] = (cx - 0.3 * s, cy + sw) # r wrist+ p[5] = (cx + 0.25 * s, cy - 0.7 * s)+ p[6] = (cx + 0.3 * s, cy - 0.35 * s - sw)+ p[7] = (cx + 0.3 * s, cy - sw)+ p[8] = (cx - 0.15 * s, cy) # r hip+ p[9] = (cx - 0.15 * s, cy + 0.45 * s - sw)+ p[10] = (cx - 0.15 * s, cy + 0.9 * s)+ p[11] = (cx + 0.15 * s, cy)+ p[12] = (cx + 0.15 * s, cy + 0.45 * s + sw)+ p[13] = (cx + 0.15 * s, cy + 0.9 * s)+ p[14] = (cx - 0.05 * s, cy - 0.95 * s)+ p[15] = (cx + 0.05 * s, cy - 0.95 * s)+ p[16] = (cx - 0.1 * s, cy - 0.9 * s)+ p[17] = (cx + 0.1 * s, cy - 0.9 * s)+ # hands: a small fan of 21 points around each wrist+ for base, wrist in ((92, p[4]), (113, p[7])):+ ang = np.linspace(-0.6, 0.6, 5)+ p[base] = wrist+ k = base + 1+ for a in ang:+ for r in (1, 2, 3, 4):+ p[k] = wrist + 0.03 * s * r * np.array([np.sin(a), np.cos(a)])+ k += 1+ # face: a ring of 68 points around the nose (not rendered by default)+ ang = np.linspace(0, 2 * np.pi, 68, endpoint=False)+ p[24:92] = p[0] + 0.08 * s * np.stack([np.cos(ang), np.sin(ang)], 1)+ # feet+ p[18:24] = p[[10, 10, 10, 13, 13, 13]] + np.array([[0, 0.02 * s]] * 6)+ return p+++class FakePoseBackend:+ info = BackendInfo(name="fake-pose", variant="none", license="n/a", commercial_ok=True)+ fp32 = True+ device = "cpu"++ def infer(self, frames: np.ndarray, fps: float, *, progress=None) -> PoseClip:+ t, h, w = frames.shape[:3]+ clip = PoseClip(width=w, height=h, fps=fps)+ size = 0.4 * min(h, w)+ for i in range(t):+ ph = 2 * np.pi * i / max(fps, 1) * 1.5+ a = _figure(w * (0.3 + 0.1 * i / max(t, 1)), h * 0.5, size, ph)+ b = _figure(w * 0.7, h * 0.5, size * 0.8, ph + np.pi)+ kps = np.stack([a, b])+ scs = np.full((2, 134), 0.9, dtype=np.float32)+ if i % 7 == 3:+ scs[:, 4] = 0.1 # right wrist drops out+ clip.keypoints.append(kps)+ clip.scores.append(scs)+ if progress:+ progress(i + 1, t)+ return clip++ def peak_memory_bytes(self) -> dict[str, int]:+ return {}+++__all__ = ["FakePoseBackend"]diff --git a/reshot/cli.py b/reshot/cli.pyindex dd602e8..1644b23 100644--- a/reshot/cli.py+++ b/reshot/cli.py@@ -2,6 +2,8 @@reshot in.mp4 -o out.mp4 # Apache-2.0 Small model, auto devicereshot in.mp4 -o out.mp4 --target h3 # 24 fps, ×32 dims, ≤15 s for MiniMax H3+ reshot in.mp4 -o pose.mp4 --control pose # OpenPose-style skeleton video instead of depth+ reshot in.mp4 -o out/ --control depth,pose,canny # all three, out/<name>_<control>.mp4reshot in.mp4 -o out.mp4 --npz d.npz --metrics run.jsonreshot clips/*.mp4 -o depth/ --target seedance # batch: one model load, out/<name>_depth.mp4reshot # no arguments: local web UI, opens in the browser@@ -23,8 +25,8 @@ import sysfrom pathlib import Pathfrom ._version import __version__-from .config import BACKENDS, MODEL_VARIANTS, QUALITIES, RunConfig-from .errors import ReshotError+from .config import BACKENDS, CONTROLS, MODEL_VARIANTS, QUALITIES, RunConfig+from .errors import InputError, ReshotErrorfrom .pipeline import run, run_manyfrom .reporter import StderrReporterfrom .targets import TARGETS@@ -33,14 +35,42 @@ from .targets import TARGETSdef build_parser() -> argparse.ArgumentParser:p = argparse.ArgumentParser(prog="reshot",- description="Video → depth-map video for video-generation ControlNets.",+ description="Video → depth-map / skeleton / line video for video-generation ControlNets.",epilog="Docs: https://github.com/maosika-ai/reshot",)p.add_argument("input", type=Path, nargs="+", help="input video(s) (anything ffmpeg/OpenCV can read)")p.add_argument("-o", "--output", type=Path, required=True, help="output .mp4, or a directory when there are several inputs")- g = p.add_argument_group("model")+ g = p.add_argument_group("control")+ g.add_argument(+ "--control",+ default="depth",+ metavar="KIND[,KIND]",+ help="what to output: depth (default), pose (OpenPose-style skeletons via DWPose), canny (lines); "+ "several at once with commas — then -o must be a directory",+ )+ g.add_argument("--no-hands", action="store_true", help="pose: body only, no 21-point hands")+ g.add_argument("--face", action="store_true", help="pose: also draw the 68 face points (off by default)")+ g.add_argument("--no-smooth", action="store_true", help="pose: raw per-frame output, no tracking / smoothing")+ g.add_argument("--keypoints", type=Path, help="pose: also write the skeletons as JSON (reshot-pose/1)")+ g.add_argument(+ "--checkpoint-dir",+ type=Path,+ help="pose: folder with yolox_l.onnx + dw-ll_ucoco_384.onnx instead of the download",+ )+ g.add_argument(+ "--pose-detect-every",+ type=int,+ default=3,+ metavar="N",+ help="pose: run the person detector every N frames and follow the skeletons in between "+ "(cuts always re-detect); 1 = every frame, ~2.5x slower on CPU (default 3)",+ )+ g.add_argument(+ "--canny", default="100,200", metavar="LOW,HIGH", help="canny: hysteresis thresholds (default 100,200)"+ )+ g = p.add_argument_group("depth model")g.add_argument("--model",default="small",@@ -93,10 +123,18 @@ def build_parser() -> argparse.ArgumentParser:return p+def _controls(args: argparse.Namespace) -> list[str]:+ kinds = [k.strip() for k in str(args.control).split(",") if k.strip()]+ for k in kinds:+ if k not in CONTROLS:+ raise InputError(f"unknown --control {k!r}; choose from {', '.join(CONTROLS)}")+ return list(dict.fromkeys(kinds)) or ["depth"]++def _is_batch(args: argparse.Namespace) -> bool:- """Several inputs, or an output that is (or is spelled like) a directory."""+ """Several inputs or controls, or an output that is (or is spelled like) a directory."""o = args.output- return len(args.input) > 1 or o.is_dir() or str(o).endswith(("/", "\\"))+ return len(args.input) > 1 or len(_controls(args)) > 1 or o.is_dir() or str(o).endswith(("/", "\\"))def configs_from_args(args: argparse.Namespace) -> list[RunConfig]:@@ -108,33 +146,51 @@ def configs_from_args(args: argparse.Namespace) -> list[RunConfig]:return Nonereturn option / f"{stem}{ext}" if batch else option+ try:+ low, high = (int(x) for x in str(args.canny).split(","))+ except ValueError as exc:+ raise InputError("--canny must be LOW,HIGH, e.g. 100,200") from exc+cfgs = []for src in args.input:stem = src.stem- cfgs.append(- RunConfig(- input=src,- output=args.output / f"{stem}_depth.mp4" if batch else args.output,- model=args.model,- backend=args.backend,- device=args.device,- target=args.target,- fps=args.fps,- max_res=args.max_res,- max_frames=args.max_frames,- quality=args.quality,- input_size=args.input_size,- invert=args.invert,- clip_percent=args.clip,- gamma=args.gamma,- crf=args.crf,- npz=per_clip(args.npz, stem, ".npz"),- metrics=per_clip(args.metrics, stem, ".json"),- checkpoint=args.checkpoint,- force=args.force,- cuda_memory_fraction=args.cuda_memory_fraction,+ for control in _controls(args):+ # several controls → several output files; the per-clip npz/metrics/keypoints+ # names carry the control too so they do not overwrite each other+ tag = f"{stem}_{control}" if len(_controls(args)) > 1 else stem+ cfgs.append(+ RunConfig(+ input=src,+ output=args.output / f"{stem}_{control}.mp4" if batch else args.output,+ model=args.model,+ backend=args.backend,+ device=args.device,+ target=args.target,+ control=control,+ fps=args.fps,+ max_res=args.max_res,+ max_frames=args.max_frames,+ quality=args.quality,+ input_size=args.input_size,+ invert=args.invert,+ clip_percent=args.clip,+ gamma=args.gamma,+ crf=args.crf,+ npz=per_clip(args.npz, tag, ".npz"),+ metrics=per_clip(args.metrics, tag, ".json"),+ checkpoint=args.checkpoint,+ force=args.force,+ cuda_memory_fraction=args.cuda_memory_fraction,+ pose_hands=not args.no_hands,+ pose_face=args.face,+ pose_smooth=not args.no_smooth,+ keypoints=per_clip(args.keypoints, tag, ".keypoints.json"),+ checkpoint_dir=args.checkpoint_dir,+ pose_detect_every=args.pose_detect_every,+ canny_low=low,+ canny_high=high,+ ))- )return cfgsdiff --git a/reshot/config.py b/reshot/config.pyindex 345b655..b260245 100644--- a/reshot/config.py+++ b/reshot/config.py@@ -12,7 +12,8 @@ from .targets import TARGETSMODEL_VARIANTS = ("small", "base", "large")QUALITIES = ("fast", "full") # model working size: fast = 644×364 for 16:9 (default), full = 924×518-BACKENDS = ("vda", "fake")+BACKENDS = ("vda", "fake") # "fake" also selects the fake pose backend when control == "pose"+CONTROLS = ("depth", "pose", "canny") # what the output video shows; see pipeline.py@dataclass(frozen=True)@@ -23,6 +24,7 @@ class RunConfig:backend: str = "vda"device: str = "auto"target: str = "none"+ control: str = "depth" # depth | pose | canny — see CONTROLSfps: float | None = None # None = preset or sourcemax_res: int = 1280 # cap on the OUTPUT's longer sidemax_frames: int | None = None@@ -36,6 +38,16 @@ class RunConfig:metrics: Path | None = Nonecheckpoint: Path | None = Noneforce: bool = False # ignore the RAM budget+ # pose only+ pose_hands: bool = True # draw the 21-point hands+ pose_face: bool = False # draw the 68 face points (off: faces are what ReShot throws away)+ pose_smooth: bool = True # track + hysteresis + One-Euro; off = raw per-frame output+ keypoints: Path | None = None # also write the skeletons as JSON (reshot-pose/1)+ checkpoint_dir: Path | None = None # local folder with the two DWPose .onnx files+ pose_detect_every: int = 3 # detector cadence in frames; 1 = every frame (slowest, no tracking assumptions)+ # canny only+ canny_low: int = 100+ canny_high: int = 200cuda_memory_fraction: float | None = None # cap VRAM to this share of the card (verification aid)extra: dict = field(default_factory=dict) # free-form, echoed into metrics@@ -46,6 +58,12 @@ class RunConfig:raise InputError(f"unknown --model {self.model!r}; choose from {', '.join(MODEL_VARIANTS)}")if self.backend not in BACKENDS:raise InputError(f"unknown backend {self.backend!r}; choose from {', '.join(BACKENDS)}")+ if self.control not in CONTROLS:+ raise InputError(f"unknown --control {self.control!r}; choose from {', '.join(CONTROLS)}")+ if self.pose_detect_every < 1:+ raise InputError("--pose-detect-every must be ≥ 1")+ if not 0 <= self.canny_low <= self.canny_high <= 1000:+ raise InputError("--canny must be LOW,HIGH with 0 <= LOW <= HIGH")if self.quality not in QUALITIES:raise InputError(f"unknown --quality {self.quality!r}; choose from {', '.join(QUALITIES)}")if self.input_size is not None and (self.input_size < 14 or self.input_size % 14):diff --git a/reshot/edges.py b/reshot/edges.pynew file mode 100644index 0000000..b5a415e--- /dev/null+++ b/reshot/edges.py@@ -0,0 +1,37 @@+"""Canny line video: the third control type, and the only one without a model.++Per frame: resize to the output size, grey, Gaussian blur, `cv2.Canny`. White lines on+black, which is what the canny ControlNets (MiniMax H3 Fun ControlNet's `canny` slot+included) expect. Blur first: video compression noise otherwise turns into a crawl of+short edges that the generator reads as texture.++Caveat for users, stated in the docs: unlike depth, lines carry the outline of clothes,+hair and a face. "Copy the shot, not the actors" is only half true for canny.+"""++from __future__ import annotations++from collections.abc import Iterator++import cv2+import numpy as np+++def canny_frame(rgb: np.ndarray, h: int, w: int, *, low: int = 100, high: int = 200, blur: int = 3) -> np.ndarray:+ """One RGB frame → uint8 `[h, w]` edge map."""+ if rgb.shape[0] != h or rgb.shape[1] != w:+ rgb = cv2.resize(rgb, (w, h), interpolation=cv2.INTER_AREA)+ grey = cv2.cvtColor(rgb, cv2.COLOR_RGB2GRAY)+ if blur and blur > 1:+ k = blur if blur % 2 else blur + 1+ grey = cv2.GaussianBlur(grey, (k, k), 0)+ return cv2.Canny(grey, low, high)+++def canny_frames(frames: np.ndarray, h: int, w: int, *, low: int = 100, high: int = 200) -> Iterator[np.ndarray]:+ """Lazy per-frame edges for the writer, so a whole edge clip is never held in RAM."""+ for f in frames:+ yield canny_frame(f, h, w, low=low, high=high)+++__all__ = ["canny_frame", "canny_frames"]diff --git a/reshot/io.py b/reshot/io.pyindex 1778222..2338c79 100644--- a/reshot/io.py+++ b/reshot/io.py@@ -1,7 +1,7 @@"""Video in / video out.Reading uses OpenCV (no decord dependency — decord has no arm64 macOS wheels).-Writing pipes raw 8-bit grey frames into ffmpeg. We deliberately do not go through+Writing pipes raw 8-bit grey (depth, canny) or RGB (skeleton) frames into ffmpeg. We deliberately do not go throughimageio's writer: we need explicit control over pixel format, GOP length and CRF,because the output is a *control signal*, not a video for humans."""@@ -127,20 +127,50 @@ def write_gray_video(becomes jitter in the generated clip. Measured cost is small (~0.5–1 Mbps at 720p).A key frame every `keyint_seconds` lets downstream tools cut the clip anywhere."""- # Accept either a whole [T, H, W] array or a lazy frame iterator (with `size`), so- # the caller can upsample frame by frame instead of materialising the full-res clip.- if isinstance(gray, np.ndarray):- if gray.ndim != 3 or gray.dtype != np.uint8:- raise ValueError(f"expected uint8 [T, H, W], got {gray.shape} {gray.dtype}")- h, w = gray.shape[1], gray.shape[2]- frames: Iterable[np.ndarray] = gray+ return _write_video(gray, path, fps, channels=1, crf=crf, keyint_seconds=keyint_seconds, size=size)+++def write_rgb_video(+ rgb: np.ndarray | Iterable[np.ndarray],+ path: str | Path,+ fps: float,+ *,+ crf: int = 12,+ keyint_seconds: float = 2.0,+ size: tuple[int, int] | None = None,+) -> Path:+ """Encode uint8 `[T, H, W, 3]` RGB as H.264 yuv420p — the skeleton video. Same+ encoder settings as the grey writer; the thin coloured limbs on black are the content+ x264 smears first at a stingy CRF, so 12 stays."""+ return _write_video(rgb, path, fps, channels=3, crf=crf, keyint_seconds=keyint_seconds, size=size)+++def _write_video(+ data: np.ndarray | Iterable[np.ndarray],+ path: str | Path,+ fps: float,+ *,+ channels: int,+ crf: int,+ keyint_seconds: float,+ size: tuple[int, int] | None,+) -> Path:+ # Accept either a whole array or a lazy frame iterator (with `size`), so the caller+ # can produce frames one at a time instead of materialising the full-res clip.+ nd = 3 if channels == 1 else 4+ if isinstance(data, np.ndarray):+ if data.ndim != nd or data.dtype != np.uint8 or (channels == 3 and data.shape[-1] != 3):+ raise ValueError(f"expected uint8 [T, H, W{', 3' if channels == 3 else ''}], got {data.shape} {data.dtype}")+ h, w = data.shape[1], data.shape[2]+ frames: Iterable[np.ndarray] = dataelse:if size is None:raise ValueError("size=(h, w) is required when passing a frame iterator")h, w = size- frames = gray+ frames = dataif h % 2 or w % 2:raise ValueError("frame size must be even for yuv420p; use targets.fit_dimensions first")+ expect = (h, w) if channels == 1 else (h, w, 3)path = Path(path)path.parent.mkdir(parents=True, exist_ok=True)cmd = [@@ -151,7 +181,7 @@ def write_gray_video("-f","rawvideo","-pix_fmt",- "gray",+ "gray" if channels == 1 else "rgb24","-s",f"{w}x{h}","-r",@@ -182,8 +212,8 @@ def write_gray_video(# intermediate bytes object. BrokenPipe means ffmpeg died early; its stderr# (read below) carries the real reason, so swallow the pipe error here.for frame in frames:- if frame.shape != (h, w) or frame.dtype != np.uint8:- raise ValueError(f"frame {frame.shape} {frame.dtype} does not match {(h, w)} uint8")+ if frame.shape != expect or frame.dtype != np.uint8:+ raise ValueError(f"frame {frame.shape} {frame.dtype} does not match {expect} uint8")proc.stdin.write(np.ascontiguousarray(frame).tobytes())except BrokenPipeError:passdiff --git a/reshot/pipeline.py b/reshot/pipeline.pyindex 77e8768..accf7db 100644--- a/reshot/pipeline.py+++ b/reshot/pipeline.py@@ -1,12 +1,19 @@-"""The pipeline: plan → read → depth → grey → encode. This is the only module that wires-the steps together; the CLI and the Python API both call `run()`.+"""The pipeline: plan → read → model → encode. This is the only module that wires the+steps together; the CLI and the Python API both call `run()`.from reshot import RunConfig, runresult = run(RunConfig(input=Path("in.mp4"), output=Path("out.mp4"), target="h3"))+ result = run(RunConfig(input=Path("in.mp4"), output=Path("pose.mp4"), control="pose"))++Three control types share the plan/read/encode stages and differ in the middle:++ depth frames → Video Depth Anything → float depth → whole-clip grey → grey video+ pose frames → DWPose (detect + estimate) → track/smooth → OpenPose render → RGB video+ canny frames → per-frame Canny → grey video (no model at all)Design notes-------------* Inference always runs at the model's working resolution (short side `input_size`).+* Depth inference always runs at the model's working resolution (short side `input_size`).The 8-bit result is scaled to the output size frame by frame during encoding, so a12 s 720p clip needs ~4 GB of host RAM instead of 8–20 GB (see planning.py).* Normalisation is computed once over the whole clip; per-frame normalisation makes@@ -30,7 +37,7 @@ from ._version import __version__from .backends import get_backend, pick_devicefrom .config import RunConfigfrom .errors import InputError, RamBudgetError, ReshotError-from .io import probe_video, read_video, write_gray_video+from .io import probe_video, read_video, write_gray_video, write_rgb_videofrom .planning import gib, memory_verdict, processing_max_resfrom .postprocess import to_gray, upsample_framesfrom .reporter import NullReporter, Reporter@@ -105,14 +112,20 @@ def plan(cfg: RunConfig) -> Plan:# from a thumbnail. The three demo takes were made from full-resolution depth# downscaled afterwards, which is what this now does.cap = cfg.max_res if cfg.max_res > 0 else 10**9- proc_max_res = processing_max_res(src_w, src_h, _planning_input_size(cfg))- scale = min(1.0, proc_max_res / max(src_w, src_h))- proc_w, proc_h = (int(round(src_w * scale / 2) * 2), int(round(src_h * scale / 2) * 2))-out_scale = min(1.0, cap / max(src_w, src_h))out_h, out_w = fit_dimensions(int(src_h * out_scale), int(src_w * out_scale), target.multiple)- verdict = memory_verdict(frames, proc_w, proc_h)+ if cfg.control == "depth":+ proc_max_res = processing_max_res(src_w, src_h, _planning_input_size(cfg))+ else:+ # Pose and canny have no "model resolution": the detector letterboxes to 640 on+ # its own and canny is per pixel, so frames are read at the output size and every+ # keypoint / edge lands exactly where it is drawn.+ proc_max_res = max(out_w, out_h)+ scale = min(1.0, proc_max_res / max(src_w, src_h))+ proc_w, proc_h = (int(round(src_w * scale / 2) * 2), int(round(src_h * scale / 2) * 2))++ verdict = memory_verdict(frames, proc_w, proc_h, control=cfg.control)return Plan(src_w,src_h,@@ -131,10 +144,25 @@ def plan(cfg: RunConfig) -> Plan:def _make_backend(cfg: RunConfig):- """Build the depth backend a config asks for (the fake one when the env var is set)."""- backend_name = "fake" if os.environ.get("RESHOT_FAKE_BACKEND") else cfg.backend- if backend_name == "fake":- return get_backend("fake"), backend_name+ """Build the backend a config's `control` asks for (a fake one when the env var is+ set). Canny needs none and returns `(None, "none")`."""+ fake = bool(os.environ.get("RESHOT_FAKE_BACKEND")) or cfg.backend == "fake"+ if cfg.control == "canny":+ return None, "none"+ if cfg.control == "pose":+ if fake:+ return get_backend("fake-pose"), "fake-pose"+ return (+ get_backend(+ "dwpose",+ device=cfg.device,+ checkpoint_dir=str(cfg.checkpoint_dir) if cfg.checkpoint_dir else None,+ detect_every=cfg.pose_detect_every,+ ),+ "dwpose",+ )+ if fake:+ return get_backend("fake"), "fake"device = pick_device(cfg.device)backend = get_backend("vda",@@ -143,7 +171,7 @@ def _make_backend(cfg: RunConfig):checkpoint=str(cfg.checkpoint) if cfg.checkpoint else None,cuda_memory_fraction=cfg.cuda_memory_fraction,)- return backend, backend_name+ return backend, "vda"def run_many(cfgs: list[RunConfig], reporter: Reporter | None = None) -> list[RunResult | ReshotError]:@@ -160,20 +188,25 @@ def run_many(cfgs: list[RunConfig], reporter: Reporter | None = None) -> list[Rukey = {(c.model, c.backend, c.device, c.checkpoint) for c in cfgs}if len(key) != 1:raise InputError("batch: every input must use the same --model / --backend / --device / --checkpoint")- backend, backend_name = _make_backend(cfgs[0])- results: list[RunResult | ReshotError] = []- try:- for i, cfg in enumerate(cfgs, 1):- rep.step("file", f"[{i}/{len(cfgs)}] {cfg.input} → {cfg.output}")- try:- results.append(_run_one(cfg, rep, backend, backend_name))- except ReshotError as exc:- rep.step("error", f"{cfg.input}: {exc}")- results.append(exc)- finally:- _release_accelerator(backend.device)- del backend- return results+ # One backend per control type (`--control depth,pose` is two groups), each loaded+ # once and released before the next group so depth and pose never share the GPU.+ results: dict[int, RunResult | ReshotError] = {}+ for control in dict.fromkeys(c.control for c in cfgs):+ group = [(i, c) for i, c in enumerate(cfgs) if c.control == control]+ backend, backend_name = _make_backend(group[0][1])+ try:+ for i, cfg in group:+ rep.step("file", f"[{i + 1}/{len(cfgs)}] {cfg.input} → {cfg.output}")+ try:+ results[i] = _run_one(cfg, rep, backend, backend_name)+ except ReshotError as exc:+ rep.step("error", f"{cfg.input}: {exc}")+ results[i] = exc+ finally:+ if backend is not None:+ _release_accelerator(backend.device)+ del backend+ return [results[i] for i in range(len(cfgs))]def run(cfg: RunConfig, reporter: Reporter | None = None, *, backend=None) -> RunResult:@@ -190,12 +223,13 @@ def run(cfg: RunConfig, reporter: Reporter | None = None, *, backend=None) -> Rutry:return _run_one(cfg, reporter or NullReporter(), backend, backend_name)finally:- _release_accelerator(backend.device)+ if backend is not None:+ _release_accelerator(backend.device)del backenddef _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunResult:- """One clip through plan → read → depth → grey → encode on an already-built backend."""+ """One clip through plan → read → model → encode on an already-built backend (None for canny)."""t0 = time.time()target = TARGETS[cfg.target]@@ -209,7 +243,9 @@ def _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunRe)if p.ram_estimate > p.ram_budget and not cfg.force:raise RamBudgetError(- p.ram_estimate, p.ram_budget, memory_verdict(p.frames, p.proc_w, p.proc_h)["max_frames_ok"]+ p.ram_estimate,+ p.ram_budget,+ memory_verdict(p.frames, p.proc_w, p.proc_h, control=cfg.control)["max_frames_ok"],)# ── read ──────────────────────────────────────────────────────────────────@@ -220,7 +256,54 @@ def _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunReif target.max_seconds and t / fps > target.max_seconds + 1e-6:rep.step("warn", f"clip is {t / fps:.1f}s; {target.key} accepts ≤ {target.max_seconds:.0f}s — trim it")- # ── depth ─────────────────────────────────────────────────────────────────+ # ── model → encode (per control type) ─────────────────────────────────────+ if cfg.control == "depth":+ stage = _depth_stage(cfg, rep, backend, frames, fps, p, target)+ elif cfg.control == "pose":+ stage = _pose_stage(cfg, rep, backend, frames, fps, p, target)+ else:+ stage = _canny_stage(cfg, rep, frames, fps, p, target)+ del frames+ out, device, precision, model_seconds, extra_metrics = stage+ total = time.time() - t0+ peak = _peak_rss_bytes()+ rep.step("wrote", f"{out} ({total:.1f}s total, peak RSS {gib(peak)})")++ metrics = {+ "version": __version__,+ "control": cfg.control,+ "config": {k: (str(v) if isinstance(v, Path) else v) for k, v in asdict(cfg).items()},+ "plan": asdict(p),+ "device": device,+ "precision": precision,+ "frames": t,+ "fps": fps,+ **extra_metrics,+ "depth_seconds": round(model_seconds, 3), # historical name: model time for any control+ "ms_per_frame": round(model_seconds / t * 1000, 1),+ "total_seconds": round(total, 2),+ "peak_rss_bytes": peak,+ "output": str(out),+ "output_size": [p.out_w, p.out_h],+ "output_bytes": os.path.getsize(out),+ }+ if cfg.metrics:+ Path(cfg.metrics).parent.mkdir(parents=True, exist_ok=True)+ with open(cfg.metrics, "w", encoding="utf-8") as fh:+ json.dump(metrics, fh, ensure_ascii=False, indent=2)+ rep.step("metrics", str(cfg.metrics))+ if cfg.control == "depth" and not _license_ok(metrics):+ rep.step("note", "weights are non-commercial; use --model small for commercial work")++ return RunResult(+ out, t, fps, p.out_w, p.out_h, device, precision, backend_name, model_seconds, total, peak, metrics+ )+++def _depth_stage(cfg: RunConfig, rep: Reporter, backend, frames: np.ndarray, fps: float, p: Plan, target):+ """frames → depth → whole-clip grey → grey video. Returns+ `(output, device, precision, model_seconds, extra_metrics)`."""+ t, h, w = frames.shape[:3]device = backend.deviceprecision = "fp32" if backend.fp32 else "fp16"rep.step(@@ -233,7 +316,6 @@ def _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunRet1 = time.time()depths = backend.infer(frames, fps, input_size=input_size)depth_seconds = time.time() - t1- del framesgpu_peak = backend.peak_memory_bytes()_release_accelerator(backend.device) # the model stays; only cached activations gorep.step("depth", f"{depth_seconds:.1f}s = {depth_seconds / t * 1000:.0f} ms/frame")@@ -248,9 +330,84 @@ def _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunRenp.savez_compressed(cfg.npz, depths=depths, fps=fps)rep.step("npz", str(cfg.npz))- # ── grey → encode ─────────────────────────────────────────────────────────gray = to_gray(depths, invert=cfg.invert, clip_percent=cfg.clip_percent, gamma=cfg.gamma)del depths+ _report_size(rep, p, target, w, h)+ out = write_gray_video(+ upsample_frames(gray, p.out_h, p.out_w), cfg.output, fps, crf=cfg.crf, size=(p.out_h, p.out_w)+ )+ extra = {"input_size_used": input_size, "model_input_resolution": [mw, mh], **gpu_peak}+ return out, device, precision, depth_seconds, extra+++def _pose_stage(cfg: RunConfig, rep: Reporter, backend, frames: np.ndarray, fps: float, p: Plan, target):+ """frames → skeletons → track/smooth → OpenPose render → RGB video."""+ from .pose.skeleton import render_frame+ from .pose.tracking import stabilise++ t, h, w = frames.shape[:3]+ device = backend.device+ rep.step("model", f"{backend.info.name}/{backend.info.variant} on {device}, licence {backend.info.license}")+ t1 = time.time()+ last = {"pct": -1}++ def progress(i: int, n: int) -> None: # one line per 10 %, not per frame+ pct = i * 10 // n+ if pct != last["pct"]:+ last["pct"] = pct+ rep.step("pose", f"{i}/{n} frames")++ clip = backend.infer(frames, fps, progress=progress)+ model_seconds = time.time() - t1+ people = [k.shape[0] for k in clip.keypoints]+ rep.step(+ "pose",+ f"{model_seconds:.1f}s = {model_seconds / t * 1000:.0f} ms/frame; "+ f"people per frame: min {min(people)}, max {max(people)}",+ )+ if max(people) == 0:+ rep.step("warn", "no person found in any frame — the skeleton video will be black")+ if cfg.pose_smooth:+ stabilise(clip)+ rep.step("pose", "tracked, hysteresis on visibility, One-Euro smoothed")+ if cfg.keypoints:+ Path(cfg.keypoints).parent.mkdir(parents=True, exist_ok=True)+ with open(cfg.keypoints, "w", encoding="utf-8") as fh:+ json.dump(clip.to_json_dict(), fh, ensure_ascii=False, separators=(",", ":"))+ rep.step("keypoints", str(cfg.keypoints))++ _report_size(rep, p, target, w, h)+ parts = "body" + (" + hands" if cfg.pose_hands else "") + (" + face" if cfg.pose_face else "")+ rep.step("render", f"OpenPose style, {parts}")++ def rendered():+ for i in range(clip.frames):+ yield render_frame(+ clip.keypoints[i], clip.scores[i], p.out_h, p.out_w,+ src_w=w, src_h=h, hands=cfg.pose_hands, face=cfg.pose_face,+ ) # fmt: skip++ out = write_rgb_video(rendered(), cfg.output, fps, crf=cfg.crf, size=(p.out_h, p.out_w))+ extra = {"people_per_frame_max": max(people), "people_per_frame_min": min(people)}+ return out, device, "fp32", model_seconds, extra+++def _canny_stage(cfg: RunConfig, rep: Reporter, frames: np.ndarray, fps: float, p: Plan, target):+ """frames → per-frame Canny → grey video. No model."""+ from .edges import canny_frames++ h, w = frames.shape[1:3]+ _report_size(rep, p, target, w, h)+ rep.step("canny", f"thresholds {cfg.canny_low}/{cfg.canny_high}")+ t1 = time.time()+ out = write_gray_video(+ canny_frames(frames, p.out_h, p.out_w, low=cfg.canny_low, high=cfg.canny_high),+ cfg.output, fps, crf=cfg.crf, size=(p.out_h, p.out_w),+ ) # fmt: skip+ return out, "cpu", "n/a", time.time() - t1, {"canny": [cfg.canny_low, cfg.canny_high]}+++def _report_size(rep: Reporter, p: Plan, target, w: int, h: int) -> None:if (p.out_h, p.out_w) != (h, w):rep.step("size", f"{w}x{h} → {p.out_w}x{p.out_h} (multiple of {target.multiple})")if target.min_pixels and p.out_w * p.out_h < target.min_pixels:@@ -259,43 +416,6 @@ def _run_one(cfg: RunConfig, rep: Reporter, backend, backend_name: str) -> RunRef"{p.out_w}x{p.out_h} is below {target.key}'s minimum of {target.min_pixels:,} pixels ""— the API will reject it; use a larger source or raise --max-res",)- out = write_gray_video(- upsample_frames(gray, p.out_h, p.out_w), cfg.output, fps, crf=cfg.crf, size=(p.out_h, p.out_w)- )- total = time.time() - t0- peak = _peak_rss_bytes()- rep.step("wrote", f"{out} ({total:.1f}s total, peak RSS {gib(peak)})")-- metrics = {- "version": __version__,- "config": {k: (str(v) if isinstance(v, Path) else v) for k, v in asdict(cfg).items()},- "plan": asdict(p),- "device": device,- "precision": precision,- "frames": t,- "fps": fps,- "input_size_used": input_size,- "model_input_resolution": [mw, mh],- "depth_seconds": round(depth_seconds, 3),- "ms_per_frame": round(depth_seconds / t * 1000, 1),- "total_seconds": round(total, 2),- "peak_rss_bytes": peak,- **gpu_peak,- "output": str(out),- "output_size": [p.out_w, p.out_h],- "output_bytes": os.path.getsize(out),- }- if cfg.metrics:- Path(cfg.metrics).parent.mkdir(parents=True, exist_ok=True)- with open(cfg.metrics, "w", encoding="utf-8") as fh:- json.dump(metrics, fh, ensure_ascii=False, indent=2)- rep.step("metrics", str(cfg.metrics))- if not _license_ok(metrics):- rep.step("note", "weights are non-commercial; use --model small for commercial work")-- return RunResult(- out, t, fps, p.out_w, p.out_h, device, precision, backend_name, depth_seconds, total, peak, metrics- )def extract(diff --git a/reshot/planning.py b/reshot/planning.pyindex 836fd6f..88c7afd 100644--- a/reshot/planning.py+++ b/reshot/planning.py@@ -22,6 +22,12 @@ import osHOST_BASE_BYTES = int(2.0 * 2**30)HOST_BYTES_PER_PIXEL_FRAME = 20.0+#: Pose and canny keep only the uint8 frames plus one working copy and load ONNX Runtime+#: instead of torch: a much smaller base and slope. Not fitted — an upper bound by+#: construction (3 B RGB frame + 1 B output/working row), see `estimate_host_bytes`.+HOST_BASE_BYTES_LIGHT = int(0.75 * 2**30)+HOST_BYTES_PER_PIXEL_FRAME_LIGHT = 4.0+#: Refuse when the estimate exceeds this share of physical RAM. Half, not 90 %: the#: OS, the browser and the GPU driver (unified memory on Apple Silicon) all need room.HOST_RAM_BUDGET_RATIO = 0.5@@ -40,9 +46,11 @@ def processing_max_res(width: int, height: int, input_size: int = 518) -> int:return int(round(scaled / 2) * 2)-def estimate_host_bytes(frames: int, proc_w: int, proc_h: int) -> int:+def estimate_host_bytes(frames: int, proc_w: int, proc_h: int, control: str = "depth") -> int:"""Peak host RSS for a run: fixed base plus a per-pixel-frame slope (see constants)."""- return HOST_BASE_BYTES + int(frames * proc_w * proc_h * HOST_BYTES_PER_PIXEL_FRAME)+ if control == "depth":+ return HOST_BASE_BYTES + int(frames * proc_w * proc_h * HOST_BYTES_PER_PIXEL_FRAME)+ return HOST_BASE_BYTES_LIGHT + int(frames * proc_w * proc_h * HOST_BYTES_PER_PIXEL_FRAME_LIGHT)def physical_ram_bytes() -> int:@@ -63,17 +71,18 @@ def physical_ram_bytes() -> int:return 16 * 2**30-def memory_verdict(frames: int, proc_w: int, proc_h: int, ram_bytes: int | None = None) -> dict:+def memory_verdict(frames: int, proc_w: int, proc_h: int, ram_bytes: int | None = None, control: str = "depth") -> dict:"""`{"estimate", "budget", "ok", "max_frames_ok"}` — the CLI turns this into text."""ram = ram_bytes if ram_bytes is not None else physical_ram_bytes()budget = int(ram * HOST_RAM_BUDGET_RATIO)- est = estimate_host_bytes(frames, proc_w, proc_h)- per_frame = max(1, estimate_host_bytes(1, proc_w, proc_h) - HOST_BASE_BYTES)+ base = HOST_BASE_BYTES if control == "depth" else HOST_BASE_BYTES_LIGHT+ est = estimate_host_bytes(frames, proc_w, proc_h, control)+ per_frame = max(1, estimate_host_bytes(1, proc_w, proc_h, control) - base)return {"estimate": est,"budget": budget,"ok": est <= budget,- "max_frames_ok": max(1, (budget - HOST_BASE_BYTES) // per_frame),+ "max_frames_ok": max(1, (budget - base) // per_frame),}diff --git a/reshot/pose/__init__.py b/reshot/pose/__init__.pynew file mode 100644index 0000000..5d89f3e--- /dev/null+++ b/reshot/pose/__init__.py@@ -0,0 +1,67 @@+"""Pose: video → per-frame 2-D skeletons → an OpenPose-style skeleton video.++The parts, in the order a run uses them:++* :mod:`.dwpose` — DWPose in ONNX Runtime: a YOLOX-L person detector and an RTMPose+ whole-body estimator (133 COCO-WholeBody keypoints per person).+* :mod:`.skeleton` — COCO-WholeBody → OpenPose-18 body layout, and the renderer that+ draws the coloured skeleton the video ControlNets were trained on.+* :mod:`.tracking` — identity across frames, One-Euro smoothing, visibility hysteresis:+ what keeps a per-frame estimator from flickering on video.++Keypoints travel between the parts as `PoseClip`: for every frame an `[N, 134, 2]` float32+array of pixel coordinates and an `[N, 134]` score array, N people, 134 = 18 OpenPose body+joints + 6 feet + 68 face + 21 + 21 hand joints (see :mod:`.skeleton`).+"""++from __future__ import annotations++from dataclasses import dataclass, field++import numpy as np+++@dataclass+class PoseClip:+ """Skeletons for a whole clip. `keypoints[t]` is `[N_t, 134, 2]` pixels in a+ `width × height` frame, `scores[t]` is `[N_t, 134]` in [0, 1]. N_t may differ per+ frame. `track_ids[t]` (`[N_t]` ints) is filled by :func:`reshot.pose.tracking.track`."""++ width: int+ height: int+ fps: float+ keypoints: list[np.ndarray] = field(default_factory=list)+ scores: list[np.ndarray] = field(default_factory=list)+ track_ids: list[np.ndarray] = field(default_factory=list)++ @property+ def frames(self) -> int:+ return len(self.keypoints)++ def to_json_dict(self) -> dict:+ """Compact, self-describing JSON: coordinates rounded to 0.1 px, scores to 0.01."""+ from .skeleton import LAYOUT++ return {+ "format": "reshot-pose/1",+ "layout": LAYOUT,+ "width": self.width,+ "height": self.height,+ "fps": self.fps,+ "frames": [+ {+ "people": [+ {+ "id": int(self.track_ids[t][n]) if self.track_ids else n,+ "keypoints": np.round(self.keypoints[t][n], 1).tolist(),+ "scores": np.round(self.scores[t][n], 2).tolist(),+ }+ for n in range(len(self.keypoints[t]))+ ]+ }+ for t in range(self.frames)+ ],+ }+++__all__ = ["PoseClip"]diff --git a/reshot/pose/dwpose.py b/reshot/pose/dwpose.pynew file mode 100644index 0000000..4d30cd3--- /dev/null+++ b/reshot/pose/dwpose.py@@ -0,0 +1,328 @@+"""DWPose (YOLOX-L detector + RTMPose-L whole-body) in ONNX Runtime.++Why DWPose: it is what the community ControlNet preprocessors (`controlnet_aux`) use to+make the pose images that the video ControlNets — MiniMax H3 Fun ControlNet among them —+were trained on. Producing the same 18-joint OpenPose layout, hands included, from the+same estimator means our skeleton video looks like the model's training data, which+matters more than a point of keypoint accuracy. Both the code (IDEA-Research/DWPose) and+the weights (`yzd-v/DWPose` on Hugging Face) are Apache-2.0, so commercial use is fine.++What was ruled out and why: OpenPose itself (CMU licence, non-commercial); Ultralytics+YOLO-pose (AGPL, incompatible with an Apache package); Sapiens (non-commercial);+MediaPipe (single person — useless for a fight).++Pre/post-processing follows the reference implementation in DWPose's ControlNet folder+(`annotator/dwpose/onnxdet.py`, `onnxpose.py`), rewritten here in NumPy with two+corrections: the detector gets BGR (YOLOX was trained on raw BGR pixels) and the pose+model gets RGB normalised with ImageNet mean/std (its mmpose config sets+`bgr_to_rgb=True`); the reference feeds the same array to both.++Speed: on a CPU the detector is ~80 % of the time (368 ms vs 75 ms per person for the pose+model, M2 Max, 864×496). So the detector runs every `detect_every` frames and in between+the boxes come from the previous frame's skeleton (padded) — standard pose tracking. Two+guards keep that honest: a cut (large frame-to-frame difference on a thumbnail) or a+person whose skeleton has gone unreliable forces a detection on that frame. Measured on+the 289-frame corridor demo, every-3 vs every-1: see the module test / changelog.++Runtime: ONNX Runtime on CPU, or its CUDA provider when `onnxruntime-gpu` is installed.+CoreML is not requested: RTMPose's SimCC heads are known to fall back op by op under the+CoreML provider, so plain CPU is the predictable choice on a Mac (not benchmarked by us).+"""++from __future__ import annotations++import logging+import time++import cv2+import numpy as np++from . import PoseClip++log = logging.getLogger(__name__)++HF_REPO = "yzd-v/DWPose"+DET_FILE = "yolox_l.onnx"+POSE_FILE = "dw-ll_ucoco_384.onnx"+DET_INPUT = (640, 640) # (h, w) the YOLOX-L export was traced at+DET_SCORE = 0.3 # keep a person box above this (after class score × objectness)+DET_NMS = 0.45+DET_PRESCORE = 0.1+POSE_BBOX_PADDING = 1.25 # mmpose top-down convention: enlarge the box before cropping+DETECT_EVERY = 3 # detector cadence; between detections boxes follow the previous skeleton+TRACK_BOX_PAD = 1.2 # a box derived from a skeleton is grown by this before cropping+CUT_THRESHOLD = 28.0 # mean |Δ| on a 64-px grey thumbnail above this = a cut → detect now+TRACK_MIN_SCORE = 0.5 # mean body-joint score below this = the skeleton is not to be trusted+IMAGENET_MEAN = np.array([123.675, 116.28, 103.53], dtype=np.float32)+IMAGENET_STD = np.array([58.395, 57.12, 57.375], dtype=np.float32)+++def _weights(checkpoint_dir: str | None) -> tuple[str, str]:+ """Paths of the two ONNX files: from `checkpoint_dir` if given, else the HF download+ (which honours `HF_ENDPOINT`, so a mirror works without special-casing)."""+ if checkpoint_dir:+ from pathlib import Path++ d = Path(checkpoint_dir)+ return str(d / DET_FILE), str(d / POSE_FILE)+ from huggingface_hub import hf_hub_download++ return hf_hub_download(HF_REPO, DET_FILE), hf_hub_download(HF_REPO, POSE_FILE)+++def _session(path: str, device: str):+ import onnxruntime as ort++ available = ort.get_available_providers()+ providers = ["CPUExecutionProvider"]+ if device == "cuda" and "CUDAExecutionProvider" in available:+ providers = ["CUDAExecutionProvider", "CPUExecutionProvider"]+ opts = ort.SessionOptions()+ opts.log_severity_level = 3+ return ort.InferenceSession(path, sess_options=opts, providers=providers)+++# ── detector ──────────────────────────────────────────────────────────────────────+++def _det_preprocess(img_bgr: np.ndarray) -> tuple[np.ndarray, float]:+ """Letterbox onto a grey 640×640 canvas (top-left aligned, as YOLOX does). Returns the+ CHW float32 tensor and the scale that maps canvas pixels back to the source."""+ h, w = img_bgr.shape[:2]+ r = min(DET_INPUT[0] / h, DET_INPUT[1] / w)+ nh, nw = int(h * r), int(w * r)+ canvas = np.full((DET_INPUT[0], DET_INPUT[1], 3), 114, dtype=np.uint8)+ canvas[:nh, :nw] = cv2.resize(img_bgr, (nw, nh), interpolation=cv2.INTER_LINEAR)+ return np.ascontiguousarray(canvas.transpose(2, 0, 1), dtype=np.float32), r+++def _det_grid_decode(out: np.ndarray) -> np.ndarray:+ """YOLOX raw head output `[1, A, 85]` → boxes in canvas pixels (cx, cy, w, h) + scores."""+ grids, strides_all = [], []+ for stride in (8, 16, 32):+ hs, ws = DET_INPUT[0] // stride, DET_INPUT[1] // stride+ xv, yv = np.meshgrid(np.arange(ws), np.arange(hs))+ grid = np.stack((xv, yv), 2).reshape(1, -1, 2)+ grids.append(grid)+ strides_all.append(np.full((1, grid.shape[1], 1), stride))+ grid = np.concatenate(grids, 1)+ stride = np.concatenate(strides_all, 1)+ out = out.copy()+ out[..., :2] = (out[..., :2] + grid) * stride+ out[..., 2:4] = np.exp(out[..., 2:4]) * stride+ return out[0]+++def _nms(boxes: np.ndarray, scores: np.ndarray, thr: float) -> list[int]:+ x1, y1, x2, y2 = boxes.T+ areas = (x2 - x1 + 1) * (y2 - y1 + 1)+ order = scores.argsort()[::-1]+ keep = []+ while order.size:+ i = order[0]+ keep.append(int(i))+ xx1 = np.maximum(x1[i], x1[order[1:]])+ yy1 = np.maximum(y1[i], y1[order[1:]])+ xx2 = np.minimum(x2[i], x2[order[1:]])+ yy2 = np.minimum(y2[i], y2[order[1:]])+ inter = np.maximum(0.0, xx2 - xx1 + 1) * np.maximum(0.0, yy2 - yy1 + 1)+ iou = inter / (areas[i] + areas[order[1:]] - inter)+ order = order[1:][iou <= thr]+ return keep+++def detect_people(session, img_bgr: np.ndarray) -> np.ndarray:+ """Person boxes `[N, 4]` (x1, y1, x2, y2) in source pixels, best first. COCO class 0 only."""+ tensor, r = _det_preprocess(img_bgr)+ raw = session.run(None, {session.get_inputs()[0].name: tensor[None]})[0]+ pred = _det_grid_decode(raw)+ boxes = pred[:, :4]+ scores = pred[:, 4:5] * pred[:, 5:] # objectness × class prob+ person = scores[:, 0]+ xyxy = np.empty_like(boxes)+ xyxy[:, 0] = boxes[:, 0] - boxes[:, 2] / 2+ xyxy[:, 1] = boxes[:, 1] - boxes[:, 3] / 2+ xyxy[:, 2] = boxes[:, 0] + boxes[:, 2] / 2+ xyxy[:, 3] = boxes[:, 1] + boxes[:, 3] / 2+ xyxy /= r+ mask = person > DET_PRESCORE+ if not mask.any():+ return np.zeros((0, 4), dtype=np.float32)+ cand, cand_s = xyxy[mask], person[mask]+ keep = _nms(cand, cand_s, DET_NMS)+ boxes_k, scores_k = cand[keep], cand_s[keep]+ good = scores_k > DET_SCORE+ return boxes_k[good].astype(np.float32)+++# ── pose (top-down, SimCC) ─────────────────────────────────────────────────────+++def _box_to_center_scale(box: np.ndarray, aspect: float) -> tuple[np.ndarray, np.ndarray]:+ """mmpose `bbox_xyxy2cs` + `_fix_aspect_ratio`: centre and padded, aspect-fixed size."""+ x1, y1, x2, y2 = box+ center = np.array([(x1 + x2) / 2, (y1 + y2) / 2], dtype=np.float32)+ w, h = (x2 - x1) * POSE_BBOX_PADDING, (y2 - y1) * POSE_BBOX_PADDING+ if w > h * aspect:+ h = w / aspect+ else:+ w = h * aspect+ return center, np.array([w, h], dtype=np.float32)+++def _warp_matrix(center: np.ndarray, scale: np.ndarray, out_w: int, out_h: int) -> np.ndarray:+ """Affine that maps the padded box to the model's input rectangle (no rotation)."""+ src = np.zeros((3, 2), dtype=np.float32)+ src[0] = center+ src[1] = center + np.array([0.0, -scale[0] * 0.5])+ d = src[0] - src[1]+ src[2] = src[1] + np.array([-d[1], d[0]])+ dst = np.zeros((3, 2), dtype=np.float32)+ dst[0] = (out_w * 0.5, out_h * 0.5)+ dst[1] = dst[0] + np.array([0.0, -out_w * 0.5])+ d = dst[0] - dst[1]+ dst[2] = dst[1] + np.array([-d[1], d[0]])+ return cv2.getAffineTransform(src, dst)+++def _simcc_decode(simcc_x: np.ndarray, simcc_y: np.ndarray, split_ratio: float = 2.0):+ """SimCC heads `[1, K, W·r]`, `[1, K, H·r]` → `(xy [K, 2] in input px, score [K])`."""+ sx, sy = simcc_x[0], simcc_y[0]+ x = sx.argmax(axis=1).astype(np.float32)+ y = sy.argmax(axis=1).astype(np.float32)+ score = np.minimum(sx.max(axis=1), sy.max(axis=1))+ xy = np.stack((x, y), axis=-1) / split_ratio+ xy[score <= 0] = -1+ return xy, score+++def estimate_pose(session, img_rgb: np.ndarray, boxes: np.ndarray) -> tuple[np.ndarray, np.ndarray]:+ """Whole-body keypoints for every box: `([N, 133, 2] pixels, [N, 133] scores)`."""+ _, _, in_h, in_w = session.get_inputs()[0].shape+ in_h, in_w = int(in_h), int(in_w)+ name = session.get_inputs()[0].name+ kps, scs = [], []+ for box in boxes:+ center, scale = _box_to_center_scale(box, in_w / in_h)+ m = _warp_matrix(center, scale, in_w, in_h)+ crop = cv2.warpAffine(img_rgb, m, (in_w, in_h), flags=cv2.INTER_LINEAR)+ tensor = ((crop.astype(np.float32) - IMAGENET_MEAN) / IMAGENET_STD).transpose(2, 0, 1)+ simcc_x, simcc_y = session.run(None, {name: tensor[None]})+ xy, score = _simcc_decode(simcc_x, simcc_y)+ xy = xy / np.array([in_w, in_h], dtype=np.float32) * scale + center - scale / 2+ kps.append(xy)+ scs.append(score)+ if not kps:+ return np.zeros((0, 133, 2), dtype=np.float32), np.zeros((0, 133), dtype=np.float32)+ return np.stack(kps).astype(np.float32), np.stack(scs).astype(np.float32)+++# ── backend ───────────────────────────────────────────────────────────────────────+++class DWPoseBackend:+ """`infer(frames, fps) → PoseClip` with 134-point layout (see :mod:`.skeleton`)."""++ device: str+ fp32 = True++ def __init__(+ self, device: str = "auto", checkpoint_dir: str | None = None, detect_every: int = DETECT_EVERY+ ) -> None:+ from ..backends.base import BackendInfo+ from ..errors import BackendError++ try:+ import onnxruntime # noqa: F401+ except ImportError as exc:+ raise BackendError(+ 'pose needs ONNX Runtime: pip install "reshot[pose]" (or onnxruntime-gpu for CUDA)'+ ) from exc+ try:+ det_path, pose_path = _weights(checkpoint_dir)+ except Exception as exc:+ raise BackendError(+ f"could not obtain DWPose weights: {exc}. "+ "Set HF_ENDPOINT to a mirror, or pass --checkpoint-dir with yolox_l.onnx + dw-ll_ucoco_384.onnx."+ ) from exc+ self.device = "cuda" if device in ("auto", "cuda") and _cuda_provider_available() else "cpu"+ self.detect_every = max(1, int(detect_every))+ t0 = time.time()+ self.det = _session(det_path, self.device)+ self.pose = _session(pose_path, self.device)+ log.info("DWPose loaded on %s in %.1fs", self.device, time.time() - t0)+ self.info = BackendInfo(+ name="dwpose", variant="yolox_l+dw-ll_ucoco_384", license="Apache-2.0", commercial_ok=True+ )++ def infer(self, frames: np.ndarray, fps: float, *, progress=None) -> PoseClip:+ """`frames`: uint8 RGB `[T, H, W, 3]`. `progress(i, t)` is called per frame if given."""+ from .skeleton import wholebody_to_openpose++ if frames.ndim != 4 or frames.shape[-1] != 3:+ raise ValueError(f"expected [T, H, W, 3] uint8 RGB, got {frames.shape}")+ t, h, w = frames.shape[:3]+ clip = PoseClip(width=w, height=h, fps=fps)+ prev_thumb = None+ prev_kps = prev_scs = None+ detections = 0+ for i in range(t):+ rgb = frames[i]+ thumb = cv2.resize(rgb, (64, 36), interpolation=cv2.INTER_AREA).mean(axis=2)+ cut = prev_thumb is None or float(np.abs(thumb - prev_thumb).mean()) > CUT_THRESHOLD+ prev_thumb = thumb+ boxes = None if cut or i % self.detect_every == 0 else boxes_from_skeletons(prev_kps, prev_scs, w, h)+ if boxes is None:+ boxes = detect_people(self.det, np.ascontiguousarray(rgb[..., ::-1]))+ detections += 1+ kps, scs = estimate_pose(self.pose, rgb, boxes)+ kps, scs = wholebody_to_openpose(kps, scs)+ clip.keypoints.append(kps)+ clip.scores.append(scs)+ prev_kps, prev_scs = kps, scs+ if progress:+ progress(i + 1, t)+ log.info("detector ran on %d of %d frames", detections, t)+ return clip++ def peak_memory_bytes(self) -> dict[str, int]:+ return {}+++def boxes_from_skeletons(kps: np.ndarray | None, scs: np.ndarray | None, w: int, h: int) -> np.ndarray | None:+ """Person boxes for the next frame from this frame's OpenPose skeletons, or None when+ any skeleton is missing / unreliable (→ the caller runs the detector instead).++ Nobody in the frame is *not* a reason to detect every frame: an empty frame stays+ empty until the cadence or a cut brings the detector back."""+ if kps is None or scs is None:+ return None+ from .skeleton import VISIBLE_SCORE++ boxes = []+ for n in range(kps.shape[0]):+ body_s = scs[n, :18]+ vis = body_s > VISIBLE_SCORE+ if vis.sum() < 4 or float(body_s[vis].mean()) < TRACK_MIN_SCORE:+ return None+ pts = kps[n, :18][vis]+ x1, y1 = pts.min(axis=0)+ x2, y2 = pts.max(axis=0)+ cx, cy = (x1 + x2) / 2, (y1 + y2) / 2+ bw, bh = max(x2 - x1, 32.0) * TRACK_BOX_PAD, max(y2 - y1, 32.0) * TRACK_BOX_PAD+ boxes.append(+ [max(0.0, cx - bw / 2), max(0.0, cy - bh / 2), min(float(w), cx + bw / 2), min(float(h), cy + bh / 2)]+ )+ return np.array(boxes, dtype=np.float32).reshape(-1, 4)+++def _cuda_provider_available() -> bool:+ try:+ import onnxruntime as ort++ return "CUDAExecutionProvider" in ort.get_available_providers()+ except Exception:+ return False+++__all__ = ["DWPoseBackend", "boxes_from_skeletons", "detect_people", "estimate_pose"]diff --git a/reshot/pose/skeleton.py b/reshot/pose/skeleton.pynew file mode 100644index 0000000..f113bb0--- /dev/null+++ b/reshot/pose/skeleton.py@@ -0,0 +1,177 @@+"""Keypoint layout and the OpenPose-style renderer.++Layout (134 points per person, index → meaning), in the order DWPose's ControlNet+annotator produces it, so a skeleton video from ReShot matches what `controlnet_aux`+draws and what the video ControlNets were trained on:++ 0–17 body, OpenPose-18 order: nose, neck, R-shoulder, R-elbow, R-wrist, L-shoulder,+ L-elbow, L-wrist, R-hip, R-knee, R-ankle, L-hip, L-knee, L-ankle, R-eye, L-eye,+ R-ear, L-ear+ 18–23 feet (COCO-WholeBody, unused by the renderer)+ 24–91 face, 68 points+ 92–112 left hand, 21 points+ 113–133 right hand, 21 points++The estimator emits 133 COCO-WholeBody points (17 COCO body + 6 feet + 68 face + 42 hand);+`wholebody_to_openpose` inserts the synthetic neck (mid-shoulders) and reorders the body.++Rendering follows OpenPose's `draw_bodypose` / `draw_handpose` (via DWPose's util.py):+18 joint colours, 17 limb ellipses at 60 % brightness with the joints on top, hands as+rainbow-coloured edges with blue joints. Stroke sizes were made for ~512-px images; they+are scaled with the frame so a 1280-px skeleton is not a hairline.+"""++from __future__ import annotations++import colorsys++import cv2+import numpy as np++LAYOUT = {+ "body": [0, 18],+ "feet": [18, 24],+ "face": [24, 92],+ "hand_left": [92, 113],+ "hand_right": [113, 134],+ "body_names": [+ "nose", "neck", "r_shoulder", "r_elbow", "r_wrist", "l_shoulder", "l_elbow", "l_wrist",+ "r_hip", "r_knee", "r_ankle", "l_hip", "l_knee", "l_ankle", "r_eye", "l_eye", "r_ear", "l_ear",+ ],+} # fmt: skip++#: COCO-WholeBody (with the neck inserted at 17 → 18 body points) → OpenPose order.+_MMPOSE_IDX = [17, 6, 8, 10, 7, 9, 12, 14, 16, 13, 15, 2, 1, 4, 3]+_OPENPOSE_IDX = [1, 2, 3, 4, 6, 7, 8, 9, 10, 12, 13, 14, 15, 16, 17]++#: Limbs as 1-based OpenPose pairs (the classic table); only the first 17 are drawn.+LIMBS = [+ [2, 3], [2, 6], [3, 4], [4, 5], [6, 7], [7, 8], [2, 9], [9, 10], [10, 11], [2, 12],+ [12, 13], [13, 14], [2, 1], [1, 15], [15, 17], [1, 16], [16, 18], [3, 17], [6, 18],+] # fmt: skip+BODY_COLORS = [+ (255, 0, 0), (255, 85, 0), (255, 170, 0), (255, 255, 0), (170, 255, 0), (85, 255, 0),+ (0, 255, 0), (0, 255, 85), (0, 255, 170), (0, 255, 255), (0, 170, 255), (0, 85, 255),+ (0, 0, 255), (85, 0, 255), (170, 0, 255), (255, 0, 255), (255, 0, 170), (255, 0, 85),+] # fmt: skip+HAND_EDGES = [+ [0, 1], [1, 2], [2, 3], [3, 4], [0, 5], [5, 6], [6, 7], [7, 8], [0, 9], [9, 10], [10, 11],+ [11, 12], [0, 13], [13, 14], [14, 15], [15, 16], [0, 17], [17, 18], [18, 19], [19, 20],+] # fmt: skip+_HAND_COLORS = [+ tuple(int(c * 255) for c in colorsys.hsv_to_rgb(i / len(HAND_EDGES), 1.0, 1.0)) for i in range(len(HAND_EDGES))+]++#: Below this a joint is not drawn (OpenPose/DWPose convention).+VISIBLE_SCORE = 0.3+#: Reference frame size the stroke widths were designed for.+_REF_SIZE = 512.0+++def wholebody_to_openpose(kps: np.ndarray, scores: np.ndarray) -> tuple[np.ndarray, np.ndarray]:+ """`[N, 133, 2]`, `[N, 133]` → `[N, 134, 2]`, `[N, 134]` in the layout above.++ The neck is the shoulder midpoint and is visible only when both shoulders are."""+ if kps.shape[0] == 0:+ return np.zeros((0, 134, 2), np.float32), np.zeros((0, 134), np.float32)+ neck = kps[:, [5, 6]].mean(axis=1, keepdims=True) # [N, 1, 2]+ both = (scores[:, 5] > VISIBLE_SCORE) & (scores[:, 6] > VISIBLE_SCORE)+ neck_s = np.where(both, np.minimum(scores[:, 5], scores[:, 6]), 0.0)[:, None] # [N, 1]+ kps2 = np.concatenate([kps[:, :17], neck, kps[:, 17:]], axis=1)+ sc2 = np.concatenate([scores[:, :17], neck_s, scores[:, 17:]], axis=1)+ out_k, out_s = kps2.copy(), sc2.copy()+ out_k[:, _OPENPOSE_IDX] = kps2[:, _MMPOSE_IDX]+ out_s[:, _OPENPOSE_IDX] = sc2[:, _MMPOSE_IDX]+ return out_k.astype(np.float32), out_s.astype(np.float32)+++def stroke_sizes(h: int, w: int) -> tuple[int, int, int, int]:+ """(limb half-width, body joint radius, hand line width, hand joint radius) for a frame."""+ k = max(0.5, min(h, w) / _REF_SIZE)+ return max(1, round(4 * k)), max(1, round(4 * k)), max(1, round(2 * k)), max(1, round(4 * k))+++def render_frame(+ kps: np.ndarray,+ scores: np.ndarray,+ h: int,+ w: int,+ *,+ src_w: int | None = None,+ src_h: int | None = None,+ hands: bool = True,+ face: bool = False,+) -> np.ndarray:+ """Draw every person of one frame onto black. `kps` are pixels in a `src_w × src_h`+ frame (defaults to `w × h`); the output is `[h, w, 3]` uint8 RGB."""+ canvas = np.zeros((h, w, 3), dtype=np.uint8)+ if kps.shape[0] == 0:+ return canvas+ sx = w / (src_w or w)+ sy = h / (src_h or h)+ pts = kps * np.array([sx, sy], dtype=np.float32)+ vis = scores > VISIBLE_SCORE+ limb_w, joint_r, hand_w, hand_r = stroke_sizes(h, w)++ # body limbs (ellipses), then dim, then joints — the OpenPose look+ for n in range(pts.shape[0]):+ for i in range(17):+ a, b = LIMBS[i][0] - 1, LIMBS[i][1] - 1+ if not (vis[n, a] and vis[n, b]):+ continue+ (xa, ya), (xb, yb) = pts[n, a], pts[n, b]+ mx, my = (xa + xb) / 2, (ya + yb) / 2+ length = float(np.hypot(xa - xb, ya - yb))+ angle = float(np.degrees(np.arctan2(ya - yb, xa - xb)))+ poly = cv2.ellipse2Poly((int(mx), int(my)), (max(1, int(length / 2)), limb_w), int(angle), 0, 360, 1)+ cv2.fillConvexPoly(canvas, poly, BODY_COLORS[i])+ canvas = (canvas * 0.6).astype(np.uint8)+ for n in range(pts.shape[0]):+ for i in range(18):+ if vis[n, i]:+ cv2.circle(canvas, (int(pts[n, i, 0]), int(pts[n, i, 1])), joint_r, BODY_COLORS[i], thickness=-1)++ if hands:+ for n in range(pts.shape[0]):+ for lo, hi in (LAYOUT["hand_left"], LAYOUT["hand_right"]):+ hp, hv = pts[n, lo:hi], vis[n, lo:hi]+ for ie, (a, b) in enumerate(HAND_EDGES):+ if hv[a] and hv[b]:+ cv2.line(canvas, _pt(hp[a]), _pt(hp[b]), _HAND_COLORS[ie], thickness=hand_w)+ for j in range(hi - lo):+ if hv[j]:+ cv2.circle(canvas, _pt(hp[j]), hand_r, (0, 0, 255), thickness=-1)++ if face:+ lo, hi = LAYOUT["face"]+ for n in range(pts.shape[0]):+ for j in range(lo, hi):+ if vis[n, j]:+ cv2.circle(canvas, _pt(pts[n, j]), max(1, hand_r // 2), (255, 255, 255), thickness=-1)+ return canvas+++def _pt(p: np.ndarray) -> tuple[int, int]:+ return int(p[0]), int(p[1])+++def body_bbox(kps: np.ndarray, scores: np.ndarray) -> np.ndarray | None:+ """Axis-aligned box of the visible body joints of one person, or None if fewer than two."""+ vis = scores[:18] > VISIBLE_SCORE+ if vis.sum() < 2:+ return None+ p = kps[:18][vis]+ return np.array([p[:, 0].min(), p[:, 1].min(), p[:, 0].max(), p[:, 1].max()], dtype=np.float32)+++__all__ = [+ "BODY_COLORS",+ "HAND_EDGES",+ "LAYOUT",+ "LIMBS",+ "VISIBLE_SCORE",+ "body_bbox",+ "render_frame",+ "stroke_sizes",+ "wholebody_to_openpose",+]diff --git a/reshot/pose/tracking.py b/reshot/pose/tracking.pynew file mode 100644index 0000000..051c963--- /dev/null+++ b/reshot/pose/tracking.py@@ -0,0 +1,169 @@+"""Turn per-frame detections into stable skeletons over time.++A per-frame estimator has three kinds of flicker, and a video model reads every one of+them as motion:++1. **Identity** — person A and person B swap slots between frames. Colours in the+ OpenPose render are per joint, not per person, so this is invisible in the video,+ but smoothing needs to know who is who. `track` assigns ids by greedy IoU of body+ boxes against the previous frame; a track that is unmatched for `max_gap` frames ends.+2. **Position jitter** — sub-pixel noise on every joint. `smooth` runs a One-Euro filter+ (Casiez et al., 2012) per joint per track: it removes jitter at rest and follows fast+ moves with little lag, which is exactly the trade-off a fight needs.+3. **Visibility** — a joint hovering around the 0.3 threshold appears and disappears.+ `apply_hysteresis` needs `on_score` to switch a joint on and only drops it below+ `off_score`, so a limb at 0.28–0.33 stays drawn.++The same idea as depth's whole-clip normalisation: decide once, over time, not per frame.+"""++from __future__ import annotations++import math++import numpy as np++from . import PoseClip+from .skeleton import VISIBLE_SCORE, body_bbox+++def _iou(a: np.ndarray, b: np.ndarray) -> float:+ x1, y1 = max(a[0], b[0]), max(a[1], b[1])+ x2, y2 = min(a[2], b[2]), min(a[3], b[3])+ inter = max(0.0, x2 - x1) * max(0.0, y2 - y1)+ area = (a[2] - a[0]) * (a[3] - a[1]) + (b[2] - b[0]) * (b[3] - b[1]) - inter+ return float(inter / area) if area > 0 else 0.0+++def track(clip: PoseClip, *, iou_threshold: float = 0.3, max_gap: int = 8) -> PoseClip:+ """Fill `clip.track_ids`. Greedy: best IoU pair first, each side used once."""+ live: dict[int, tuple[np.ndarray, int]] = {} # id → (last body box, last frame index)+ next_id = 0+ clip.track_ids = []+ for t in range(clip.frames):+ kps, scs = clip.keypoints[t], clip.scores[t]+ n = kps.shape[0]+ ids = np.full(n, -1, dtype=np.int64)+ boxes = [body_bbox(kps[i], scs[i]) for i in range(n)]+ pairs = []+ for i, box in enumerate(boxes):+ if box is None:+ continue+ for tid, (prev_box, _) in live.items():+ iou = _iou(box, prev_box)+ if iou >= iou_threshold:+ pairs.append((iou, i, tid))+ pairs.sort(reverse=True)+ used_i, used_t = set(), set()+ for _, i, tid in pairs:+ if i in used_i or tid in used_t:+ continue+ ids[i] = tid+ used_i.add(i)+ used_t.add(tid)+ for i in range(n):+ if ids[i] < 0:+ ids[i] = next_id+ next_id += 1+ if boxes[i] is not None:+ live[int(ids[i])] = (boxes[i], t)+ # forget tracks that have not been seen for a while+ live = {tid: v for tid, v in live.items() if t - v[1] <= max_gap}+ clip.track_ids.append(ids)+ return clip+++class OneEuro:+ """One-Euro filter for a vector signal. `min_cutoff` sets jitter removal at rest,+ `beta` how quickly the cutoff opens up when the signal moves fast."""++ def __init__(self, freq: float, min_cutoff: float = 1.0, beta: float = 0.5, d_cutoff: float = 1.0) -> None:+ self.freq = freq+ self.min_cutoff = min_cutoff+ self.beta = beta+ self.d_cutoff = d_cutoff+ self.x_prev: np.ndarray | None = None+ self.dx_prev: np.ndarray | None = None++ @staticmethod+ def _alpha(cutoff: np.ndarray | float, freq: float) -> np.ndarray | float:+ tau = 1.0 / (2 * math.pi * cutoff)+ te = 1.0 / freq+ return 1.0 / (1.0 + tau / te)++ def __call__(self, x: np.ndarray) -> np.ndarray:+ if self.x_prev is None:+ self.x_prev = x.copy()+ self.dx_prev = np.zeros_like(x)+ return x+ dx = (x - self.x_prev) * self.freq+ a_d = self._alpha(self.d_cutoff, self.freq)+ dx_hat = a_d * dx + (1 - a_d) * self.dx_prev+ cutoff = self.min_cutoff + self.beta * np.abs(dx_hat)+ a = self._alpha(cutoff, self.freq)+ x_hat = a * x + (1 - a) * self.x_prev+ self.x_prev, self.dx_prev = x_hat, dx_hat+ return x_hat+++def smooth(clip: PoseClip, *, min_cutoff: float = 1.0, beta: float = 0.5) -> PoseClip:+ """One-Euro per track per joint, in place. Joints below `VISIBLE_SCORE` are not fed to+ the filter (their coordinates are noise) and reset it when they come back."""+ if not clip.track_ids:+ track(clip)+ filters: dict[int, OneEuro] = {}+ seen_vis: dict[int, np.ndarray] = {}+ for t in range(clip.frames):+ kps, scs, ids = clip.keypoints[t], clip.scores[t], clip.track_ids[t]+ for n in range(kps.shape[0]):+ tid = int(ids[n])+ vis = scs[n] > VISIBLE_SCORE+ f = filters.get(tid)+ if f is None:+ f = filters[tid] = OneEuro(clip.fps, min_cutoff, beta)+ seen_vis[tid] = vis.copy()+ x = kps[n].copy()+ # a joint that was invisible last frame restarts from its raw position+ if f.x_prev is not None:+ fresh = vis & ~seen_vis[tid]+ f.x_prev[fresh] = x[fresh]+ f.dx_prev[fresh] = 0.0+ hidden = ~vis+ x[hidden] = f.x_prev[hidden] # hold, so the filter state does not drift on noise+ kps[n] = f(x)+ seen_vis[tid] = vis+ return clip+++def apply_hysteresis(clip: PoseClip, *, on_score: float = VISIBLE_SCORE, off_score: float = 0.2) -> PoseClip:+ """Rewrite scores so a joint, once on, only turns off below `off_score`. In place.++ Implemented by lifting the score of a held joint to `on_score + ε` so the renderer's+ single threshold still works; the raw score is not preserved (the JSON dump comes+ after this, deliberately: what you see is what you get)."""+ if not clip.track_ids:+ track(clip)+ state: dict[int, np.ndarray] = {}+ for t in range(clip.frames):+ scs, ids = clip.scores[t], clip.track_ids[t]+ for n in range(scs.shape[0]):+ tid = int(ids[n])+ prev = state.get(tid)+ on = scs[n] > on_score+ if prev is not None:+ hold = prev & (scs[n] >= off_score) & ~on+ scs[n][hold] = on_score + 1e-3+ on = on | hold+ state[tid] = on+ return clip+++def stabilise(clip: PoseClip) -> PoseClip:+ """The default chain: track → hysteresis → smooth. In place; returns the clip."""+ track(clip)+ apply_hysteresis(clip)+ smooth(clip)+ return clip+++__all__ = ["OneEuro", "apply_hysteresis", "smooth", "stabilise", "track"]diff --git a/reshot/web/__init__.py b/reshot/web/__init__.pyindex f90f0a2..09dcb2d 100644--- a/reshot/web/__init__.py+++ b/reshot/web/__init__.py@@ -9,13 +9,15 @@ APIPOST /api/upload multipart "file" → source info (id, size, fps, frames, seconds)GET /api/source/<id> the uploaded clip, for the preview playerPOST /api/jobs JSON options → {"id"}; runs in the single worker thread+ ``control`` picks depth (default) / pose / cannyGET /api/jobs/<id> status, progress steps, result / error- GET /api/output/<id> the depth video (``?download=1`` for an attachment)- GET /api/info version, device, VRAM, output folder+ GET /api/output/<id> the control video (``?download=1`` for an attachment)+ GET /api/keypoints/<id> the pose JSON of a pose job (``?download=1`` likewise)+ GET /api/info version, device, VRAM, output folder, which controls can run-Why one worker thread with a cached model: the depth model is loaded once per server-process (3–6 s) and every job reuses it; two jobs at once would fight for the GPU, so they-queue. Why polling instead of SSE: 500 ms polls are plenty for a 20-second job and keep+Why one worker thread with cached models: a model is loaded once per server process+(3–6 s for depth, ~1 s for pose) and every job of that control reuses it; two jobs at once+would fight for the GPU, so they queue. Why polling instead of SSE: 500 ms polls are plenty for a 20-second job and keepthe server a plain ``http.server``."""@@ -40,7 +42,7 @@ from pathlib import Pathfrom urllib.parse import parse_qs, urlparsefrom .._version import __version__-from ..config import RunConfig+from ..config import CONTROLS, RunConfigfrom ..errors import ReshotErrorfrom ..io import probe_videofrom ..pipeline import _cuda_total_bytes, _make_backend, _run_one@@ -76,6 +78,7 @@ class _Job:"id": self.id,"status": self.status,"source_name": self.source_name,+ "control": self.cfg.control,"target": self.cfg.target,"quality": self.cfg.quality,"steps": self.steps,@@ -94,25 +97,35 @@ class _State:self.sources: dict[str, dict] = {}self.jobs: dict[str, _Job] = {}self.queue: queue.Queue[_Job] = queue.Queue()- self.backend = None- self.backend_name = ""+ self.backends: dict[str, tuple] = {} # control → (backend, name); each loaded onceself.lock = threading.Lock()self.device = "?"+ @property+ def backend(self):+ """The depth backend, if loaded (kept for callers that predate pose / canny)."""+ return self.backends.get("depth", (None, ""))[0]+def worker(self) -> None:while True:job = self.queue.get()job.status = "running"job.started = time.time()try:- if self.backend is None:- job.step("model", "loading the depth model (once per session)")- self.backend, self.backend_name = _make_backend(job.cfg)- self.device = self.backend.device- res = _run_one(job.cfg, job, self.backend, self.backend_name)+ control = job.cfg.control+ if control not in self.backends:+ if control != "canny":+ job.step("model", f"loading the {control} model (once per session)")+ self.backends[control] = _make_backend(job.cfg)+ if self.backends[control][0] is not None:+ self.device = self.backends[control][0].device+ backend, backend_name = self.backends[control]+ res = _run_one(job.cfg, job, backend, backend_name)job.result = {+ "control": control,"output": str(res.output),"output_url": f"/api/output/{job.id}",+ "keypoints_url": f"/api/keypoints/{job.id}" if job.cfg.keypoints else None,"frames": res.frames,"fps": res.fps,"width": res.width,@@ -125,6 +138,7 @@ class _State:"model_input_resolution": res.metrics.get("model_input_resolution"),"gpu_peak_reserved_bytes": res.metrics.get("gpu_peak_reserved_bytes"),"output_bytes": res.metrics.get("output_bytes"),+ "people_per_frame_max": res.metrics.get("people_per_frame_max"),}job.status = "done"except ReshotError as exc:@@ -139,6 +153,15 @@ class _State:self.queue.task_done()+def _onnxruntime_installed() -> bool:+ try:+ import onnxruntime # noqa: F401++ return True+ except ImportError:+ return False++def _safe_stem(name: str) -> str:stem = Path(name).stem or "clip"return "".join(c if c.isalnum() or c in "-_ ." else "_" for c in stem)[:80].strip() or "clip"@@ -223,6 +246,12 @@ class _Handler(BaseHTTPRequestHandler):return self._json({"error": "no output yet"}, HTTPStatus.NOT_FOUND)q = parse_qs(url.query)self._file(Path(job.result["output"]), Path(job.result["output"]).name if q.get("download") else None)+ elif parts[:2] == ["api", "keypoints"] and len(parts) == 3:+ job = self.state.jobs.get(parts[2])+ if not job or not job.result or not job.cfg.keypoints or not Path(job.cfg.keypoints).exists():+ return self._json({"error": "no keypoints for this job"}, HTTPStatus.NOT_FOUND)+ q = parse_qs(url.query)+ self._file(Path(job.cfg.keypoints), Path(job.cfg.keypoints).name if q.get("download") else None)else:self._json({"error": "not found"}, HTTPStatus.NOT_FOUND)except (BrokenPipeError, ConnectionResetError):@@ -247,7 +276,10 @@ class _Handler(BaseHTTPRequestHandler):"targets": {k: {"fps": v.fps, "multiple": v.multiple, "max_seconds": v.max_seconds} for k, v in TARGETS.items()},+ "controls": list(CONTROLS),+ "pose_available": _onnxruntime_installed(),"model_loaded": st.backend is not None,+ "models_loaded": sorted(st.backends),}def _upload(self) -> None:@@ -303,17 +335,25 @@ class _Handler(BaseHTTPRequestHandler):if not src:return self._json({"error": "upload a video first"}, HTTPStatus.BAD_REQUEST)target = str(body.get("target", "seedance"))+ control = str(body.get("control", "depth"))+ if control == "pose" and not _onnxruntime_installed():+ return self._json(+ {"error": 'pose needs ONNX Runtime: pip install "reshot[pose]", then restart reshot'},+ HTTPStatus.BAD_REQUEST,+ )stem = _safe_stem(src["name"])- out = self.state.out_dir / f"{stem}_depth_{target}.mp4"+ out = self.state.out_dir / f"{stem}_{control}_{target}.mp4"n = 2while out.exists(): # never overwrite an earlier result- out = self.state.out_dir / f"{stem}_depth_{target}_{n}.mp4"+ out = self.state.out_dir / f"{stem}_{control}_{target}_{n}.mp4"n += 1try:+ canny = body.get("canny") or [100, 200]cfg = RunConfig(input=Path(src["path"]),output=out,target=target,+ control=control,quality=str(body.get("quality", "fast")),max_res=int(body.get("max_side") or 1280) if body.get("max_side") else 1280,max_frames=int(body["max_frames"]) if body.get("max_frames") else None,@@ -322,8 +362,14 @@ class _Handler(BaseHTTPRequestHandler):gamma=float(body.get("gamma", 1.0) or 1.0),metrics=out.with_suffix(".json"),force=bool(body.get("force", False)),+ pose_hands=bool(body.get("pose_hands", True)),+ pose_face=bool(body.get("pose_face", False)),+ pose_smooth=bool(body.get("pose_smooth", True)),+ keypoints=out.with_name(out.stem + ".keypoints.json") if control == "pose" else None,+ canny_low=int(canny[0]),+ canny_high=int(canny[1]),)- except (ReshotError, ValueError) as exc:+ except (ReshotError, ValueError, TypeError, IndexError) as exc:return self._json({"error": str(exc)}, HTTPStatus.BAD_REQUEST)job = _Job(cfg, src["name"])self.state.jobs[job.id] = jobdiff --git a/reshot/web/index.html b/reshot/web/index.htmlindex 3a4610f..2750134 100644--- a/reshot/web/index.html+++ b/reshot/web/index.html@@ -149,7 +149,7 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}<main class="wrap"><div class="intro"><h1 data-i18n="intro.h1">复制走位,不复制演员。</h1>- <p data-i18n="intro.p">把一段参考视频变成深度图视频,交给 Seedance 或 MiniMax H3 当参考,它就会照着这段的动作和运镜再拍一遍——人物换成你的。所有处理都在这台电脑上完成。</p>+ <p data-i18n="intro.p">把一段参考视频变成深度图、骨架或线稿视频,交给 Seedance 或 MiniMax H3 当参考,它就会照着这段的动作和运镜再拍一遍——人物换成你的。所有处理都在这台电脑上完成。</p></div><section class="card">@@ -164,7 +164,7 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}</div><div class="players" id="players" hidden><div class="player" id="srcPlayer"><span class="tag" data-i18n="tag.src">参考片</span><video id="srcVideo" controls muted loop playsinline></video></div>- <div class="player" id="outPlayer" hidden><span class="tag depth" data-i18n="tag.depth">深度图</span><video id="outVideo" controls muted loop playsinline></video></div>+ <div class="player" id="outPlayer" hidden><span class="tag depth" id="outTag">深度图</span><video id="outVideo" controls muted loop playsinline></video></div></div><div class="meta" id="meta" hidden></div><div class="stage-foot" id="stageFoot" hidden>@@ -175,6 +175,17 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}<!-- right: options --><div class="opts">+ <div class="opt">+ <h3 data-i18n="opt.control">出什么</h3>+ <div class="seg" id="control">+ <button type="button" data-v="depth" class="on" data-i18n="c.depth">深度图</button>+ <button type="button" data-v="pose" data-i18n="c.pose">骨架</button>+ <button type="button" data-v="canny" data-i18n="c.canny">线稿</button>+ </div>+ <div class="note" id="controlNote"></div>+ <div class="warn" id="poseWarn" hidden></div>+ </div>+<div class="opt"><h3 data-i18n="opt.target">给谁用</h3><div class="seg" id="target">@@ -187,7 +198,7 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}<div class="warn" id="lenWarn" hidden></div></div>- <div class="opt">+ <div class="opt" id="qualityOpt"><h3 data-i18n="opt.quality">画质</h3><div class="seg" id="quality"><button type="button" data-v="fast" class="on" data-i18n="q.fast">快速(推荐)</button>@@ -201,9 +212,14 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}<div class="adv-body"><div class="field"><label for="maxSide" data-i18n="adv.maxSide">输出长边上限</label><input type="number" id="maxSide" min="0" step="2" placeholder="0"><div class="tip" data-i18n="adv.maxSideTip">0 = 原片尺寸。给 MiniMax H3 当参考视频建议 320。</div></div><div class="field"><label for="maxFrames" data-i18n="adv.maxFrames">只处理前 N 帧</label><input type="number" id="maxFrames" min="1" step="1" placeholder="全部"><div class="tip" data-i18n="adv.maxFramesTip">试效果时用,比如 48。</div></div>- <div class="field"><label for="clip" data-i18n="adv.clip">裁掉两端 %</label><input type="number" id="clip" min="0" max="10" step="0.1" value="0"><div class="tip" data-i18n="adv.clipTip">有东西贴着镜头把其他全压黑时用 0.5–1。</div></div>- <div class="field"><label for="gamma">Gamma</label><input type="number" id="gamma" min="0.2" max="3" step="0.05" value="1"><div class="tip" data-i18n="adv.gammaTip">>1 压暗中间调,把近处拉开。一般不动。</div></div>- <label class="check"><input type="checkbox" id="invert"><span data-i18n="adv.invert">反转:远白近黑(默认近白远黑,Seedance / H3 / ControlNet 都要这个)</span></label>+ <div class="field" data-for="depth"><label for="clip" data-i18n="adv.clip">裁掉两端 %</label><input type="number" id="clip" min="0" max="10" step="0.1" value="0"><div class="tip" data-i18n="adv.clipTip">有东西贴着镜头把其他全压黑时用 0.5–1。</div></div>+ <div class="field" data-for="depth"><label for="gamma">Gamma</label><input type="number" id="gamma" min="0.2" max="3" step="0.05" value="1"><div class="tip" data-i18n="adv.gammaTip">>1 压暗中间调,把近处拉开。一般不动。</div></div>+ <label class="check" data-for="depth"><input type="checkbox" id="invert"><span data-i18n="adv.invert">反转:远白近黑(默认近白远黑,Seedance / H3 / ControlNet 都要这个)</span></label>+ <label class="check" data-for="pose"><input type="checkbox" id="poseHands" checked><span data-i18n="adv.poseHands">画手(21 点,抓握与手势靠它)</span></label>+ <label class="check" data-for="pose"><input type="checkbox" id="poseFace"><span data-i18n="adv.poseFace">画脸(68 点;默认不画——脸型会被带过去)</span></label>+ <label class="check" data-for="pose"><input type="checkbox" id="poseSmooth" checked><span data-i18n="adv.poseSmooth">跨帧稳定:跟踪 + 平滑 + 关节迟滞(关掉 = 逐帧原始结果)</span></label>+ <div class="field" data-for="canny"><label for="cannyLow" data-i18n="adv.cannyLow">线稿低阈值</label><input type="number" id="cannyLow" min="0" max="1000" step="10" value="100"><div class="tip" data-i18n="adv.cannyLowTip">越低线越多。</div></div>+ <div class="field" data-for="canny"><label for="cannyHigh" data-i18n="adv.cannyHigh">线稿高阈值</label><input type="number" id="cannyHigh" min="0" max="1000" step="10" value="200"><div class="tip" data-i18n="adv.cannyHighTip">越高只留强边。默认 100 / 200。</div></div></div></details>@@ -211,7 +227,7 @@ footer a{color:var(--text-2)}footer a:hover{color:var(--accent-bright)}<div class="err" id="err" hidden></div><div class="result" id="result" hidden></div>- <button class="cta" id="run" type="button" disabled data-i18n="cta">生成深度图</button>+ <button class="cta" id="run" type="button" disabled>生成深度图</button></div></section>@@ -230,14 +246,21 @@ const I18N = {zh: {brand:"猫斯卡", "hdr.maosika":"去猫斯卡做 AI 短剧 ↗","intro.h1":"复制走位,不复制演员。",- "intro.p":"把一段参考视频变成深度图视频,交给 Seedance 或 MiniMax H3 当参考,它就会照着这段的动作和运镜再拍一遍——人物换成你的。所有处理都在这台电脑上完成。",+ "intro.p":"把一段参考视频变成深度图、骨架或线稿视频,交给 Seedance 或 MiniMax H3 当参考,它就会照着这段的动作和运镜再拍一遍——人物换成你的。所有处理都在这台电脑上完成。","drop.big":"拖入参考视频,或点击选择","drop.sub":"mp4 / mov / webm · 15 秒以内效果最好","drop.hint":"视频不会上传到任何服务器",- "tag.src":"参考片","tag.depth":"深度图","stage.change":"换一段视频","stage.sync":"同步播放",+ "tag.src":"参考片","tag.depth":"深度图","tag.pose":"骨架","tag.canny":"线稿","stage.change":"换一段视频","stage.sync":"同步播放",+ "opt.control":"出什么","c.depth":"深度图","c.pose":"骨架","c.canny":"线稿",+ c_depth:"<b>深度图</b>:近白远黑的灰片。带走站位、体量、动作、运镜;扔掉脸、衣服、光。默认选它。",+ c_pose:"<b>骨架</b>:黑底彩色火柴人(OpenPose 画法,DWPose 估计,含手、不画脸)。跳舞、武打这类靠肢体的最准;MiniMax H3 ControlNet 的 pose 口直接收。第一次用会下载两个模型文件(约 340 MB)。",+ c_canny:"<b>线稿</b>:黑底白线的边缘图,不用模型、秒出。注意它会把衣服轮廓和脸型带过去——「只复制走位」对线稿只成立一半。",+ poseWarn:"骨架需要 ONNX Runtime:在终端里执行 <code>pip install \"reshot[pose]\"</code>,然后重启 reshot。","opt.target":"给谁用","opt.none":"通用","opt.quality":"画质","q.fast":"快速(推荐)","q.full":"完整","adv.title":"高级选项","adv.maxSide":"输出长边上限","adv.maxSideTip":"0 = 原片尺寸。给 MiniMax H3 当参考视频建议 320。","adv.maxFrames":"只处理前 N 帧","adv.maxFramesTip":"试效果时用,比如 48。","adv.clip":"裁掉两端 %","adv.clipTip":"有东西贴着镜头把其他全压黑时用 0.5–1。","adv.gammaTip":">1 压暗中间调,把近处拉开。一般不动。","adv.invert":"反转:远白近黑(默认近白远黑,Seedance / H3 / ControlNet 都要这个)",- cta:"生成深度图", "cta.running":"生成中…", "cta.again":"再生成一次",+ "adv.poseHands":"画手(21 点,抓握与手势靠它)","adv.poseFace":"画脸(68 点;默认不画——脸型会被带过去)","adv.poseSmooth":"跨帧稳定:跟踪 + 平滑 + 关节迟滞(关掉 = 逐帧原始结果)",+ "adv.cannyLow":"线稿低阈值","adv.cannyLowTip":"越低线越多。","adv.cannyHigh":"线稿高阈值","adv.cannyHighTip":"越高只留强边。默认 100 / 200。",+ cta_depth:"生成深度图", cta_pose:"生成骨架", cta_canny:"生成线稿", "cta.running":"生成中…", "cta.again":"再生成一次","foot.by":"出品","foot.what":"AI 短剧 / AI 短视频自动生产系统","foot.comfy":"ComfyUI 节点",t_seedance:"<b>Seedance 2.0 / 2.5 参考视频</b>:24 fps,宽高 16 的倍数,≤ 15 秒。上传后提示词写「参考@视频1的动作与运镜」。",t_h3:"<b>MiniMax H3</b>:24 fps,宽高 32 的倍数,≤ 15 秒。当参考视频挂 <Video 1> 时,高级选项里把长边限到 320 效果最好。",@@ -248,28 +271,39 @@ const I18N = {lenWarn:"这段 {s} 秒,{t} 只收 ≤ {m} 秒的参考视频——先剪短,或在高级选项里限制帧数。",meta_fmt:"{w}×{h} · {fps} fps · {n} 帧 · {s} 秒 · {mb} MB",uploading:"读取视频…", queued:"排队中", loading_model:"加载模型(每次启动只一次)",- step_plan:"计划", step_read:"解码", step_model:"模型", step_depth:"深度", step_size:"尺寸", step_wrote:"写出", step_warn:"提醒", step_vram:"显存", step_quality:"画质", step_metrics:"指标", step_npz:"npz", step_note:"注",- done:"完成", chips:{frames:"帧",fps:"fps",size:"尺寸",speed:"ms/帧",device:"设备",model:"模型看到",vram:"显存峰值",file:"文件"},- download:"下载深度图", openFolder:"保存在", next_title:"下一步",+ step_plan:"计划", step_read:"解码", step_model:"模型", step_depth:"深度", step_pose:"骨架", step_render:"绘制", step_canny:"线稿", step_keypoints:"关键点", step_size:"尺寸", step_wrote:"写出", step_warn:"提醒", step_vram:"显存", step_quality:"画质", step_metrics:"指标", step_npz:"npz", step_note:"注",+ done:"完成", chips:{frames:"帧",fps:"fps",size:"尺寸",speed:"ms/帧",device:"设备",model:"模型看到",vram:"显存峰值",file:"文件",people:"最多同框"},+ download_depth:"下载深度图", download_pose:"下载骨架视频", download_canny:"下载线稿", download_kp:"下载关键点 JSON", openFolder:"保存在", next_title:"下一步",next_seedance:"到 Seedance 把这个文件当「参考视频」上传,提示词开头写:",- next_seedance_p:"参考@视频1的动作与运镜,顺序与视频保持一致。\n(接着写你的人物、服装、场景和画风)",- next_h3:"在 MiniMax H3 里把它挂成 <Video 1>(定妆图挂 <Picture 1>),提示词里定义它并说明灰色外观不要抄:",- next_h3_p:"<Subject 3> is the fight choreography and camera movement shown in <Video 1>, a grey depth map in which near objects are white and far objects are black: (用一句话写清灰片里发生了什么)\n<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; its grey depth look is not transferred.",- next_wan:"在 ComfyUI 里接到 WanVaceToVideo 的 control_video。", next_none:"交给任何认深度视频的模型。",+ next_seedance_p_depth:"参考@视频1的动作与运镜,顺序与视频保持一致。\n(接着写你的人物、服装、场景和画风)",+ next_seedance_p_pose:"参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。\n(接着写你的人物、服装、场景和画风)",+ next_seedance_p_canny:"参考@视频1的构图、动作与运镜,顺序与视频保持一致;只取构图和动作,不要画成线稿。\n(接着写你的人物、服装、场景和画风)",+ next_h3:"在 MiniMax H3 里把它挂成 <Video 1>(定妆图挂 <Picture 1>),提示词里定义它并说明它的外观不要抄:",+ next_h3_p_depth:"<Subject 3> is the fight choreography and camera movement shown in <Video 1>, a grey depth map in which near objects are white and far objects are black: (用一句话写清灰片里发生了什么)\n<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; its grey depth look is not transferred.",+ next_h3_p_pose:"<Subject 3> is the choreography and camera movement shown in <Video 1>, an OpenPose skeleton video — coloured stick figures on black, one per person: (用一句话写清火柴人在做什么)\n<Subject 3>: attribute_transfer - every pose, action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; the stick-figure look is not transferred.",+ next_h3_p_canny:"<Subject 3> is the composition, action and camera movement shown in <Video 1>, a white-line edge drawing on black: (用一句话写清线稿里发生了什么)\n<Subject 3>: attribute_transfer - the composition, every action, timing and camera move of <Video 1> are transferred onto <Subject 1>; the line-drawing look is not transferred.",+ next_wan:"在 ComfyUI 里接到 WanVaceToVideo 的 control_video。", next_none_depth:"交给任何认深度视频的模型。", next_none_pose:"交给任何认 OpenPose 骨架视频的模型(ControlNet pose / VACE)。", next_none_canny:"交给任何认 canny 线稿视频的模型。",copy:"复制提示词", copied:"已复制 ✓", err_title:"没成功","info.device":"本机:{d}",},en: {brand:"Maosika", "hdr.maosika":"Make AI short drama at Maosika ↗","intro.h1":"Copy the shot, not the actors.",- "intro.p":"Turn a reference video into a depth-map video, give it to Seedance or MiniMax H3 as the reference, and it repeats the moves and camera with your own characters. Everything runs on this computer.",+ "intro.p":"Turn a reference video into a depth-map, skeleton or line video, give it to Seedance or MiniMax H3 as the reference, and it repeats the moves and camera with your own characters. Everything runs on this computer.","drop.big":"Drop a reference video, or click to choose","drop.sub":"mp4 / mov / webm · under 15 s works best","drop.hint":"Nothing is uploaded to any server",- "tag.src":"Reference","tag.depth":"Depth map","stage.change":"Change video","stage.sync":"Play both",+ "tag.src":"Reference","tag.depth":"Depth map","tag.pose":"Skeleton","tag.canny":"Lines","stage.change":"Change video","stage.sync":"Play both",+ "opt.control":"What to make","c.depth":"Depth map","c.pose":"Skeleton","c.canny":"Lines",+ c_depth:"<b>Depth map</b>: grey, near = white. Keeps blocking, scale, moves and camera; drops faces, clothes and light. The default.",+ c_pose:"<b>Skeleton</b>: coloured stick figures on black (OpenPose style, estimated by DWPose, hands yes, face no). Most precise for dance and fights; MiniMax H3 ControlNet takes it on its pose input. The first run downloads two model files (~340 MB).",+ c_canny:"<b>Lines</b>: white edges on black — no model, instant. It does carry the outline of clothes and a face: “copy the shot, not the actors” is only half true here.",+ poseWarn:"Skeletons need ONNX Runtime: run <code>pip install \"reshot[pose]\"</code> in a terminal, then restart reshot.","opt.target":"For which model","opt.none":"Generic","opt.quality":"Quality","q.fast":"Fast (recommended)","q.full":"Full","adv.title":"Advanced","adv.maxSide":"Max output long side","adv.maxSideTip":"0 = source size. 320 recommended for MiniMax H3 reference videos.","adv.maxFrames":"Only the first N frames","adv.maxFramesTip":"For a quick test, e.g. 48.","adv.clip":"Clip both ends %","adv.clipTip":"0.5–1 when something at the lens crushes everything else to black.","adv.gammaTip":">1 darkens mid-tones, spreading the near range. Rarely needed.","adv.invert":"Invert: far = white (default near = white, what Seedance / H3 / ControlNets expect)",- cta:"Make depth map", "cta.running":"Working…", "cta.again":"Run again",+ "adv.poseHands":"Draw hands (21 points — grips and gestures)","adv.poseFace":"Draw the face (68 points; off by default — it carries the face shape)","adv.poseSmooth":"Stabilise over time: tracking + smoothing + joint hysteresis (off = raw per-frame output)",+ "adv.cannyLow":"Lines: low threshold","adv.cannyLowTip":"Lower = more lines.","adv.cannyHigh":"Lines: high threshold","adv.cannyHighTip":"Higher = only strong edges. Default 100 / 200.",+ cta_depth:"Make depth map", cta_pose:"Make skeleton", cta_canny:"Make lines", "cta.running":"Working…", "cta.again":"Run again","foot.by":"Made by","foot.what":"AI short drama & short video production system","foot.comfy":"ComfyUI nodes",t_seedance:"<b>Seedance 2.0 / 2.5 reference video</b>: 24 fps, sides ×16, ≤ 15 s. Prompt with “参考@视频1的动作与运镜”.",t_h3:"<b>MiniMax H3</b>: 24 fps, sides ×32, ≤ 15 s. As a reference video (<Video 1>) it works best small — set max long side 320 in Advanced.",@@ -280,14 +314,18 @@ const I18N = {lenWarn:"This clip is {s} s; {t} accepts reference videos ≤ {m} s — trim it first, or limit frames in Advanced.",meta_fmt:"{w}×{h} · {fps} fps · {n} frames · {s} s · {mb} MB",uploading:"Reading video…", queued:"Queued", loading_model:"Loading the model (once per session)",- step_plan:"plan", step_read:"decode", step_model:"model", step_depth:"depth", step_size:"size", step_wrote:"wrote", step_warn:"note", step_vram:"vram", step_quality:"quality", step_metrics:"metrics", step_npz:"npz", step_note:"note",- done:"Done", chips:{frames:"frames",fps:"fps",size:"size",speed:"ms/frame",device:"device",model:"model saw",vram:"VRAM peak",file:"file"},- download:"Download depth map", openFolder:"Saved in", next_title:"Next",+ step_plan:"plan", step_read:"decode", step_model:"model", step_depth:"depth", step_pose:"pose", step_render:"render", step_canny:"canny", step_keypoints:"keypoints", step_size:"size", step_wrote:"wrote", step_warn:"note", step_vram:"vram", step_quality:"quality", step_metrics:"metrics", step_npz:"npz", step_note:"note",+ done:"Done", chips:{frames:"frames",fps:"fps",size:"size",speed:"ms/frame",device:"device",model:"model saw",vram:"VRAM peak",file:"file",people:"most people in a frame"},+ download_depth:"Download depth map", download_pose:"Download skeleton video", download_canny:"Download lines", download_kp:"Download keypoints JSON", openFolder:"Saved in", next_title:"Next",next_seedance:"Upload this file to Seedance as a reference video and start the prompt with:",- next_seedance_p:"参考@视频1的动作与运镜,顺序与视频保持一致。\n(then describe your people, wardrobe, set and look)",- next_h3:"In MiniMax H3 attach it as <Video 1> (character sheet as <Picture 1>); define it in the prompt and say the grey look is not to be copied:",- next_h3_p:"<Subject 3> is the fight choreography and camera movement shown in <Video 1>, a grey depth map in which near objects are white and far objects are black: (one sentence on what happens in the clip)\n<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; its grey depth look is not transferred.",- next_wan:"Connect it to WanVaceToVideo → control_video in ComfyUI.", next_none:"Hand it to any model that reads a depth video.",+ next_seedance_p_depth:"参考@视频1的动作与运镜,顺序与视频保持一致。\n(then describe your people, wardrobe, set and look)",+ next_seedance_p_pose:"参考@视频1里火柴人的动作与运镜,顺序与视频保持一致;只取动作和机位,画面里不要出现骨架线条。\n(then describe your people, wardrobe, set and look)",+ next_seedance_p_canny:"参考@视频1的构图、动作与运镜,顺序与视频保持一致;只取构图和动作,不要画成线稿。\n(then describe your people, wardrobe, set and look)",+ next_h3:"In MiniMax H3 attach it as <Video 1> (character sheet as <Picture 1>); define it in the prompt and say its look is not to be copied:",+ next_h3_p_depth:"<Subject 3> is the fight choreography and camera movement shown in <Video 1>, a grey depth map in which near objects are white and far objects are black: (one sentence on what happens in the clip)\n<Subject 3>: attribute_transfer - every action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; its grey depth look is not transferred.",+ next_h3_p_pose:"<Subject 3> is the choreography and camera movement shown in <Video 1>, an OpenPose skeleton video — coloured stick figures on black, one per person: (one sentence on what the figures do)\n<Subject 3>: attribute_transfer - every pose, action, position, timing and camera move of <Video 1> is transferred onto <Subject 1>; the stick-figure look is not transferred.",+ next_h3_p_canny:"<Subject 3> is the composition, action and camera movement shown in <Video 1>, a white-line edge drawing on black: (one sentence on what happens in the clip)\n<Subject 3>: attribute_transfer - the composition, every action, timing and camera move of <Video 1> are transferred onto <Subject 1>; the line-drawing look is not transferred.",+ next_wan:"Connect it to WanVaceToVideo → control_video in ComfyUI.", next_none_depth:"Hand it to any model that reads a depth video.", next_none_pose:"Hand it to any model that reads an OpenPose skeleton video (ControlNet pose / VACE).", next_none_canny:"Hand it to any model that reads a canny line video.",copy:"Copy prompt", copied:"Copied ✓", err_title:"That didn't work","info.device":"This machine: {d}",}@@ -297,7 +335,7 @@ if (localStorage.getItem("reshot.lang")) lang = localStorage.getItem("reshot.lanconst t = (k, vars) => { let s = (I18N[lang][k] ?? I18N.zh[k] ?? k); if (typeof s === "string" && vars) for (const [a,b] of Object.entries(vars)) s = s.replaceAll(`{${a}}`, b); return s; };const $ = id => document.getElementById(id);-const state = { source:null, target:"seedance", quality:"fast", job:null, info:null, poll:null };+const state = { source:null, control:"depth", target:"seedance", quality:"fast", job:null, info:null, poll:null };function applyLang() {document.documentElement.lang = lang;@@ -310,8 +348,14 @@ function applyLang() {$("lang").onclick = () => { lang = lang === "zh" ? "en" : "zh"; localStorage.setItem("reshot.lang", lang); applyLang(); };function renderNotes() {+ $("controlNote").innerHTML = t("c_" + state.control);$("targetNote").innerHTML = t("t_" + state.target);$("qualityNote").innerHTML = t("q_" + state.quality);+ // quality is a depth-model setting; each advanced field belongs to one control type+ $("qualityOpt").hidden = state.control !== "depth";+ document.querySelectorAll("[data-for]").forEach(el => { el.hidden = el.dataset.for !== state.control; });+ const pw = $("poseWarn"), noPose = state.control === "pose" && state.info && state.info.pose_available === false;+ pw.innerHTML = t("poseWarn"); pw.hidden = !noPose;const info = state.info, tg = info && info.targets[state.target];const w = $("lenWarn");if (state.source && tg && tg.max_seconds && state.source.seconds > tg.max_seconds + 0.05 && !$("maxFrames").value) {@@ -324,10 +368,10 @@ function seg(id, key) {$(id).querySelectorAll("button").forEach(x => x.classList.toggle("on", x === b));state[key] = b.dataset.v;if (key === "target" && b.dataset.v === "h3" && !$("maxSide").value) $("maxSide").placeholder = "320";- renderNotes();+ renderNotes(); renderRunBtn(); // the button's label and enabled state follow the control type});}-seg("target", "target"); seg("quality", "quality");+seg("control", "control"); seg("target", "target"); seg("quality", "quality");$("maxFrames").addEventListener("input", renderNotes);// ---- source ----@@ -370,8 +414,9 @@ function renderMeta() {}function renderRunBtn() {const b = $("run"), running = state.job && (state.job.status === "queued" || state.job.status === "running");- b.disabled = !state.source || !!running;- b.textContent = running ? t("cta.running") : (state.job && state.job.status === "done" ? t("cta.again") : t("cta"));+ const noPose = state.control === "pose" && state.info && state.info.pose_available === false;+ b.disabled = !state.source || !!running || noPose;+ b.textContent = running ? t("cta.running") : (state.job && state.job.status === "done" ? t("cta.again") : t("cta_" + state.control));}// ---- run ----@@ -379,9 +424,11 @@ $("run").onclick = async () => {if (!state.source) return;hideErr(); clearResult();const body = {- source_id: state.source.id, target: state.target, quality: state.quality,+ source_id: state.source.id, control: state.control, target: state.target, quality: state.quality,max_side: +$("maxSide").value || 0, max_frames: +$("maxFrames").value || 0,clip_percent: +$("clip").value || 0, gamma: +$("gamma").value || 1, invert: $("invert").checked,+ pose_hands: $("poseHands").checked, pose_face: $("poseFace").checked, pose_smooth: $("poseSmooth").checked,+ canny: [+$("cannyLow").value || 0, +$("cannyHigh").value || 0],};try {const r = await fetch("/api/jobs", { method: "POST", headers: { "Content-Type": "application/json" }, body: JSON.stringify(body) });@@ -408,18 +455,24 @@ function renderProg(j) {function renderResult(j) {const r = j.result; if (!r) return;$("prog").hidden = true;+ const control = j.control || r.control || "depth";$("outVideo").src = r.output_url + "?t=" + Date.now(); $("outPlayer").hidden = false; $("players").classList.add("two"); $("syncBtn").hidden = false;+ $("outTag").textContent = t("tag." + control);const c = t("chips"), chips = [- [c.frames, r.frames], [c.fps, r.fps], [c.size, `${r.width}×${r.height}`], [c.speed, r.ms_per_frame], [c.device, `${r.device} ${r.precision}`],+ [c.frames, r.frames], [c.fps, r.fps], [c.size, `${r.width}×${r.height}`], [c.speed, r.ms_per_frame], [c.device, control === "canny" ? r.device : `${r.device} ${r.precision}`],r.model_input_resolution ? [c.model, r.model_input_resolution.join("×")] : null,+ r.people_per_frame_max != null ? [c.people, r.people_per_frame_max] : null,r.gpu_peak_reserved_bytes ? [c.vram, (r.gpu_peak_reserved_bytes / 2**30).toFixed(1) + " GB"] : null,r.output_bytes ? [c.file, (r.output_bytes / 1048576).toFixed(2) + " MB"] : null,].filter(Boolean).map(([k, v]) => `<span class="chip">${k} <b>${v}</b></span>`).join("");- const nextKey = "next_" + j.target, prompt = I18N[lang][nextKey + "_p"];+ // the "next step" text depends on the target; the prompt line on target × control+ const nextKey = j.target === "none" || j.target === "wan" ? "next_" + j.target + (j.target === "none" ? "_" + control : "") : "next_" + j.target;+ const prompt = I18N[lang]["next_" + j.target + "_p_" + control];+ const kpBtn = r.keypoints_url ? `<a class="btn" href="${r.keypoints_url}?download=1" download>${t("download_kp")}</a>` : "";$("result").innerHTML = `<h4>${t("done")} · ${j.elapsed}s</h4><div class="chips">${chips}</div>- <div class="btns"><a class="btn primary" href="${r.output_url}?download=1" download>${t("download")}</a></div>+ <div class="btns"><a class="btn primary" href="${r.output_url}?download=1" download>${t("download_" + control)}</a>${kpBtn}</div><div class="path">${t("openFolder")} ${esc(r.output)}</div><div class="next"><b>${t("next_title")}</b> · ${t(nextKey)}${prompt ? `<code id="promptText">${esc(prompt)}</code><div class="btns" style="margin-top:8px"><button class="btn" id="copyBtn" type="button">${t("copy")}</button></div>` : ""}</div>`;$("result").hidden = false;diff --git a/scripts/shoot_web_ui.py b/scripts/shoot_web_ui.pyindex dda7ecb..c3cd783 100644--- a/scripts/shoot_web_ui.py+++ b/scripts/shoot_web_ui.py@@ -3,7 +3,7 @@Why not Playwright: it downloads its own browser; Chrome is already installed and`websockets` is enough to talk to it. Usage (server already running on --url):- python scripts/shoot_web_ui.py --video 走廊武打.mp4 --out docs/img/web_ui.jpg+ python scripts/shoot_web_ui.py --video 走廊武打.mp4 --out docs/img/web_ui.jpg --control pose"""from __future__ import annotations@@ -22,7 +22,7 @@ import websocketsCHROME = "/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"-async def shoot(url: str, video: Path, out: Path, width: int, height: int, lang: str) -> None:+async def shoot(url: str, video: Path, out: Path, width: int, height: int, lang: str, control: str) -> None:port = 9333proc = subprocess.Popen([@@ -86,6 +86,8 @@ async def shoot(url: str, video: Path, out: Path, width: int, height: int, lang:if await js("!document.getElementById('players').hidden"):breakawait asyncio.sleep(0.2)+ await js(f"document.querySelector('#control button[data-v=\"{control}\"]').click()")+ await asyncio.sleep(0.3)await js("document.getElementById('run').click()")for _ in range(600):if await js("!document.getElementById('result').hidden"):@@ -110,8 +112,9 @@ def main() -> None:ap.add_argument("--width", type=int, default=1360)ap.add_argument("--height", type=int, default=980)ap.add_argument("--lang", choices=["zh", "en"], default="zh")+ ap.add_argument("--control", choices=["depth", "pose", "canny"], default="depth")a = ap.parse_args()- asyncio.run(shoot(a.url, a.video, a.out, a.width, a.height, a.lang))+ asyncio.run(shoot(a.url, a.video, a.out, a.width, a.height, a.lang, a.control))if __name__ == "__main__":diff --git a/tests/test_model_smoke.py b/tests/test_model_smoke.pyindex dc12fa5..09d2565 100644--- a/tests/test_model_smoke.py+++ b/tests/test_model_smoke.py@@ -47,3 +47,28 @@ def test_cli_end_to_end(tmp_path):metrics = tmp_path / "m.json"assert main([str(src), "-o", str(out), "--device", "cpu", "--target", "h3", "--metrics", str(metrics)]) == 0assert out.exists() and metrics.exists()+++def test_pose_cli_end_to_end_finds_a_person(tmp_path):+ """Real DWPose on the CPU over the first 8 frames of the repo's demo clip (its top-left+ quadrant is the reference footage, a person in a corridor). Downloads ~340 MB once."""+ import json+ from pathlib import Path++ from reshot.cli import main++ src = Path(__file__).resolve().parents[1] / "docs" / "demo-fight.mp4"+ out, metrics, kp = tmp_path / "pose.mp4", tmp_path / "m.json", tmp_path / "kp.json"+ rc = main(+ [str(src), "-o", str(out), "--control", "pose", "--device", "cpu", "--target", "h3",+ "--max-frames", "8", "--max-res", "720", "--metrics", str(metrics), "--keypoints", str(kp)]+ ) # fmt: skip+ assert rc == 0 and out.exists()+ m = json.loads(metrics.read_text())+ assert m["control"] == "pose" and m["people_per_frame_max"] >= 1, m+ frames = json.loads(kp.read_text())["frames"]+ assert len(frames) == 8 and any(p["scores"][1] > 0.3 for p in frames[0]["people"]) # a neck was seen+ cap = cv2.VideoCapture(str(out))+ ok, frame = cap.read()+ cap.release()+ assert ok and frame.max() > 100 # coloured skeleton, not a black videodiff --git a/tests/test_pipeline.py b/tests/test_pipeline.pyindex 778fa5a..8c980ab 100644--- a/tests/test_pipeline.py+++ b/tests/test_pipeline.py@@ -54,7 +54,9 @@ def test_ram_budget_refuses_then_force_allows(tmp_path, monkeypatch):import reshot.pipeline as plsrc = _clip(tmp_path / "src.mp4")- monkeypatch.setattr(pl, "memory_verdict", lambda *_: {"estimate": 10, "budget": 1, "ok": False, "max_frames_ok": 1})+ monkeypatch.setattr(+ pl, "memory_verdict", lambda *_, **__: {"estimate": 10, "budget": 1, "ok": False, "max_frames_ok": 1}+ )with pytest.raises(RamBudgetError) as ei:run(RunConfig(input=src, output=tmp_path / "o.mp4", backend="fake"))assert ei.value.exit_code == 3 and "--max-frames 1" in str(ei.value)diff --git a/tests/test_pose.py b/tests/test_pose.pynew file mode 100644index 0000000..387e9fa--- /dev/null+++ b/tests/test_pose.py@@ -0,0 +1,251 @@+"""Pose and canny: the layout conversion, the renderer, tracking / smoothing / hysteresis,+and the whole pipeline with the fake pose backend — no model, no download."""++import json+from pathlib import Path++import cv2+import numpy as np+import pytest++from reshot import RunConfig, run+from reshot.backends.fake_pose import FakePoseBackend+from reshot.edges import canny_frame+from reshot.pose import PoseClip+from reshot.pose.skeleton import LAYOUT, VISIBLE_SCORE, body_bbox, render_frame, wholebody_to_openpose+from reshot.pose.tracking import OneEuro, apply_hysteresis, smooth, track+++def _clip(path: Path, frames=12, w=200, h=120, fps=30):+ vw = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), fps, (w, h))+ for i in range(frames):+ vw.write(np.full((h, w, 3), (i * 10) % 256, np.uint8))+ vw.release()+ return path+++# ── layout ────────────────────────────────────────────────────────────────────────+++def test_wholebody_to_openpose_inserts_neck_and_reorders():+ kps = np.arange(133 * 2, dtype=np.float32).reshape(1, 133, 2)+ scs = np.full((1, 133), 0.9, np.float32)+ out_k, out_s = wholebody_to_openpose(kps, scs)+ assert out_k.shape == (1, 134, 2) and out_s.shape == (1, 134)+ # neck = mean of COCO shoulders 5 and 6, lands at OpenPose index 1+ assert np.allclose(out_k[0, 1], (kps[0, 5] + kps[0, 6]) / 2)+ # OpenPose 2 = right shoulder = COCO 6; OpenPose 8 = right hip = COCO 12+ assert np.allclose(out_k[0, 2], kps[0, 6]) and np.allclose(out_k[0, 8], kps[0, 12])+ # nose stays at 0; hands are untouched (shifted by one for the neck)+ assert np.allclose(out_k[0, 0], kps[0, 0]) and np.allclose(out_k[0, 92], kps[0, 91])+ assert LAYOUT["hand_right"] == [113, 134]+++def test_neck_invisible_when_a_shoulder_is():+ kps = np.zeros((1, 133, 2), np.float32)+ scs = np.full((1, 133), 0.9, np.float32)+ scs[0, 5] = 0.1+ _, out_s = wholebody_to_openpose(kps, scs)+ assert out_s[0, 1] == 0.0+++def test_empty_frame_round_trips():+ k, s = wholebody_to_openpose(np.zeros((0, 133, 2), np.float32), np.zeros((0, 133), np.float32))+ assert k.shape == (0, 134, 2) and s.shape == (0, 134)+ assert render_frame(k, s, 64, 96).sum() == 0+++# ── renderer ──────────────────────────────────────────────────────────────────────+++def test_render_draws_body_hands_and_optionally_face():+ clip = FakePoseBackend().infer(np.zeros((1, 240, 320, 3), np.uint8), 24.0)+ k, s = clip.keypoints[0], clip.scores[0]+ body_only = render_frame(k, s, 240, 320, hands=False, face=False)+ with_hands = render_frame(k, s, 240, 320, hands=True, face=False)+ with_face = render_frame(k, s, 240, 320, hands=True, face=True)+ assert body_only.dtype == np.uint8 and body_only.shape == (240, 320, 3)+ assert body_only.any() # something was drawn+ assert (with_hands != body_only).any() and (with_face != with_hands).any()+ # OpenPose colour scheme: the neck→right-shoulder limb (index 0) is red+ assert (body_only[..., 0] > body_only[..., 1]).any()+++def test_render_scales_keypoints_from_source_to_output_size():+ clip = FakePoseBackend().infer(np.zeros((1, 240, 320, 3), np.uint8), 24.0)+ k, s = clip.keypoints[0], clip.scores[0]+ small = render_frame(k, s, 120, 160, src_w=320, src_h=240)+ big = render_frame(k, s, 240, 320)++ # same drawing at half the size: the centroid of the drawn pixels lands at half the position+ def centroid(img):+ ys, xs = np.nonzero(img.max(-1))+ return xs.mean(), ys.mean()++ (bx, by), (sx, sy) = centroid(big), centroid(small)+ assert abs(bx / 2 - sx) < 3 and abs(by / 2 - sy) < 3+++def test_hidden_joints_are_not_drawn():+ clip = FakePoseBackend().infer(np.zeros((1, 240, 320, 3), np.uint8), 24.0)+ k, s = clip.keypoints[0], clip.scores[0].copy()+ full = render_frame(k, s, 240, 320, hands=False)+ s[:, 8:14] = 0.0 # hide both legs+ legless = render_frame(k, s, 240, 320, hands=False)+ assert (legless > 0).sum() < (full > 0).sum()+ assert body_bbox(k[0], s[0]) is not None and body_bbox(k[0], np.zeros(134)) is None+++# ── tracking / smoothing / hysteresis ─────────────────────────────────────────────+++def test_track_keeps_ids_across_frames_and_swapped_order():+ clip = FakePoseBackend().infer(np.zeros((6, 240, 320, 3), np.uint8), 24.0)+ # swap the two people in frame 3: ids must follow the bodies, not the slots+ clip.keypoints[3] = clip.keypoints[3][::-1].copy()+ clip.scores[3] = clip.scores[3][::-1].copy()+ track(clip)+ assert [list(i) for i in clip.track_ids[:3]] == [[0, 1]] * 3+ assert list(clip.track_ids[3]) == [1, 0]+ assert list(clip.track_ids[4]) == [0, 1]+++def test_track_starts_a_new_id_after_a_gap():+ clip = PoseClip(width=320, height=240, fps=24.0)+ person = FakePoseBackend().infer(np.zeros((1, 240, 320, 3), np.uint8), 24.0)+ k, s = person.keypoints[0][:1], person.scores[0][:1]+ for t in range(20):+ present = t < 5 or t >= 15 # gone for 10 frames > max_gap+ clip.keypoints.append(k if present else np.zeros((0, 134, 2), np.float32))+ clip.scores.append(s if present else np.zeros((0, 134), np.float32))+ track(clip, max_gap=8)+ assert clip.track_ids[0][0] == 0 and clip.track_ids[15][0] == 1+++def test_one_euro_removes_jitter_and_follows_motion():+ rng = np.random.default_rng(0)+ f = OneEuro(24.0, min_cutoff=1.0, beta=0.5)+ still = np.array([100.0, 100.0])+ outs = [f(still + rng.normal(0, 2, 2)) for _ in range(48)]+ assert np.std(np.array(outs[24:]), axis=0).max() < 2.0 # less than the input noise+ f2 = OneEuro(24.0)+ xs = [f2(np.array([float(i * 20), 0.0])) for i in range(24)] # 20 px/frame sprint+ assert abs(xs[-1][0] - 23 * 20) < 60 # follows fast motion with bounded lag+++def test_smooth_keeps_shapes_and_changes_little():+ clip = FakePoseBackend().infer(np.zeros((12, 240, 320, 3), np.uint8), 24.0)+ raw = [k.copy() for k in clip.keypoints]+ smooth(clip)+ assert all(a.shape == b.shape for a, b in zip(raw, clip.keypoints, strict=True))+ assert clip.keypoints[0].tolist() == raw[0].tolist() # first frame passes through+ assert np.abs(clip.keypoints[6] - raw[6]).max() < 0.2 * min(240, 320)+++def test_hysteresis_holds_a_joint_that_dips_briefly():+ clip = FakePoseBackend().infer(np.zeros((8, 240, 320, 3), np.uint8), 24.0)+ assert clip.scores[3][0, 4] == pytest.approx(0.1) # the fake's dropout frame+ clip.scores[3][0, 4] = 0.25 # between off (0.2) and on (0.3): should be held+ apply_hysteresis(clip)+ assert clip.scores[3][0, 4] > VISIBLE_SCORE+ clip2 = FakePoseBackend().infer(np.zeros((8, 240, 320, 3), np.uint8), 24.0)+ apply_hysteresis(clip2)+ assert clip2.scores[3][0, 4] == pytest.approx(0.1) # 0.1 < off: really gone+++# ── canny ─────────────────────────────────────────────────────────────────────────+++def test_canny_frame_finds_an_edge_and_resizes():+ img = np.zeros((120, 200, 3), np.uint8)+ img[:, 100:] = 255+ e = canny_frame(img, 60, 96)+ assert e.shape == (60, 96) and e.dtype == np.uint8+ col = e.sum(axis=0)+ assert col.argmax() in range(44, 52) # the edge sits at the middle column+ assert set(np.unique(e)) <= {0, 255}+++# ── pipeline with the fake backends ───────────────────────────────────────────────+++def test_run_pose_end_to_end(tmp_path):+ src = _clip(tmp_path / "src.mp4", frames=12, w=200, h=120)+ cfg = RunConfig(+ input=src,+ output=tmp_path / "pose.mp4",+ backend="fake",+ control="pose",+ target="h3",+ keypoints=tmp_path / "kp.json",+ metrics=tmp_path / "m.json",+ )+ res = run(cfg)+ assert (res.width, res.height) == (192, 96) and res.fps == 24+ assert res.output.exists() and res.output.stat().st_size > 500+ cap = cv2.VideoCapture(str(res.output))+ ok, frame = cap.read()+ cap.release()+ assert ok and frame.shape == (96, 192, 3) and frame.max() > 100 # coloured skeleton, not black+ kp = json.loads((tmp_path / "kp.json").read_text())+ assert kp["format"] == "reshot-pose/1" and len(kp["frames"]) == res.frames+ assert len(kp["frames"][0]["people"]) == 2 and len(kp["frames"][0]["people"][0]["keypoints"]) == 134+ assert {p["id"] for p in kp["frames"][5]["people"]} == {0, 1}+ m = json.loads((tmp_path / "m.json").read_text())+ assert m["control"] == "pose" and m["people_per_frame_max"] == 2+++def test_run_canny_end_to_end(tmp_path):+ src = _clip(tmp_path / "src.mp4", frames=6, w=200, h=120)+ res = run(RunConfig(input=src, output=tmp_path / "canny.mp4", control="canny", target="seedance"))+ assert (res.width, res.height) == (192, 112) and res.output.exists()+ assert res.metrics["control"] == "canny" and res.metrics["canny"] == [100, 200]+++def test_plan_reads_at_output_size_for_pose_and_canny(tmp_path):+ from reshot import plan++ src = _clip(tmp_path / "src.mp4", frames=6, w=1920, h=1080)+ p_depth = plan(RunConfig(input=src, output=tmp_path / "o.mp4", backend="fake"))+ p_pose = plan(RunConfig(input=src, output=tmp_path / "o.mp4", backend="fake", control="pose"))+ assert p_depth.proc_h < 720 # depth: model resolution+ assert (p_pose.proc_w, p_pose.proc_h) == (1280, 720) # pose: the output size+ assert p_pose.ram_estimate < p_depth.ram_estimate+++def test_cli_multi_control_writes_one_file_each(tmp_path):+ from reshot.cli import main++ src = _clip(tmp_path / "src.mp4")+ out = tmp_path / "out"+ assert main([str(src), "-o", str(out), "--control", "depth,pose,canny", "--backend", "fake"]) == 0+ assert sorted(p.name for p in out.glob("*.mp4")) == ["src_canny.mp4", "src_depth.mp4", "src_pose.mp4"]+++def test_cli_rejects_bad_control_and_canny(tmp_path):+ from reshot.cli import main++ src = _clip(tmp_path / "src.mp4")+ assert main([str(src), "-o", str(tmp_path / "o.mp4"), "--control", "hed", "--backend", "fake"]) == 2+ assert main([str(src), "-o", str(tmp_path / "o.mp4"), "--control", "canny", "--canny", "x"]) == 2+ from reshot.errors import InputError++ with pytest.raises(InputError):+ RunConfig(input=src, output=tmp_path / "o.mp4", control="canny", canny_low=300, canny_high=200)+++def test_pose_detect_every_reaches_the_backend(tmp_path, monkeypatch):+ """The CLI flag must land in the backend constructor — it silently did not once."""+ import reshot.pipeline as pl++ seen = {}++ def fake_get_backend(name, **kw):+ seen.update(name=name, **kw)+ return FakePoseBackend()++ monkeypatch.setattr(pl, "get_backend", fake_get_backend)+ monkeypatch.delenv("RESHOT_FAKE_BACKEND", raising=False) # test_web sets it at import time+ src = _clip(tmp_path / "src.mp4", frames=3)+ run(RunConfig(input=src, output=tmp_path / "o.mp4", control="pose", pose_detect_every=5))+ assert seen["name"] == "dwpose" and seen["detect_every"] == 5diff --git a/tests/test_web.py b/tests/test_web.pyindex 4bb3d5e..85ff0a8 100644--- a/tests/test_web.py+++ b/tests/test_web.py@@ -130,3 +130,85 @@ def test_bad_inputs(server):assert st == 400st, j = _post(base + "/api/jobs", b"{", "application/json")assert st == 400+++def _wait(base, job_id):+ for _ in range(200):+ _, body, _ = _get(base + f"/api/jobs/{job_id}")+ job = json.loads(body)+ if job["status"] in ("done", "error"):+ return job+ time.sleep(0.1)+ raise AssertionError("job did not finish")+++def test_info_lists_controls(server):+ base, _ = server+ _, body, _ = _get(base + "/api/info")+ info = json.loads(body)+ assert info["controls"] == ["depth", "pose", "canny"]+ assert isinstance(info["pose_available"], bool) and info["models_loaded"] == []+++def test_pose_job_with_keypoints_download(server, tmp_path):+ base, _ = server+ clip = _clip(tmp_path / "dance.mp4", frames=12)+ body, ctype = _multipart("file", "dance.mp4", clip.read_bytes())+ _, src = _post(base + "/api/upload", body, ctype)+ st, j = _post(+ base + "/api/jobs",+ json.dumps({"source_id": src["id"], "control": "pose", "target": "h3", "pose_face": True}).encode(),+ "application/json",+ )+ assert st == 200+ job = _wait(base, j["id"])+ assert job["status"] == "done", job+ assert job["control"] == "pose" and job["result"]["control"] == "pose"+ assert job["result"]["people_per_frame_max"] == 2 # the fake pose backend draws two figures+ assert {s["tag"] for s in job["steps"]} >= {"pose", "render", "keypoints", "wrote"}+ out = Path(job["result"]["output"])+ assert out.name == "dance_pose_h3.mp4" and out.exists()+ st, raw, hdr = _get(base + job["result"]["keypoints_url"] + "?download=1")+ kp = json.loads(raw)+ assert st == 200 and kp["format"] == "reshot-pose/1" and "attachment" in hdr["Content-Disposition"]+ assert len(kp["frames"]) == job["result"]["frames"]+ # the pose backend is cached for the session; depth is untouched+ _, body, _ = _get(base + "/api/info")+ assert json.loads(body)["models_loaded"] == ["pose"]+++def test_canny_job_needs_no_model(server, tmp_path):+ base, _ = server+ clip = _clip(tmp_path / "c.mp4", frames=6)+ body, ctype = _multipart("file", "c.mp4", clip.read_bytes())+ _, src = _post(base + "/api/upload", body, ctype)+ st, j = _post(+ base + "/api/jobs",+ json.dumps({"source_id": src["id"], "control": "canny", "target": "seedance", "canny": [50, 150]}).encode(),+ "application/json",+ )+ assert st == 200+ job = _wait(base, j["id"])+ assert job["status"] == "done", job+ assert job["result"]["keypoints_url"] is None and Path(job["result"]["output"]).name == "c_canny_seedance.mp4"+ assert not any(s["tag"] == "model" for s in job["steps"])+ with pytest.raises(urllib.error.HTTPError) as ei:+ _get(base + f"/api/keypoints/{j['id']}")+ assert ei.value.code == 404+++def test_bad_control_and_canny_values(server, tmp_path):+ base, _ = server+ clip = _clip(tmp_path / "c.mp4", frames=6)+ body, ctype = _multipart("file", "c.mp4", clip.read_bytes())+ _, src = _post(base + "/api/upload", body, ctype)+ st, j = _post(+ base + "/api/jobs", json.dumps({"source_id": src["id"], "control": "hed"}).encode(), "application/json"+ )+ assert st == 400 and "control" in j["error"]+ st, j = _post(+ base + "/api/jobs",+ json.dumps({"source_id": src["id"], "control": "canny", "canny": [300, 100]}).encode(),+ "application/json",+ )+ assert st == 400