Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

feat(rife): a torch backend runs rife-v4.6 in PyTorch from the binary's own model, as one call of the binary does; the binary stays the default

Soohong Kim6b30d243e31f
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 3e904c9..760794d 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,14 @@
All notable public changes to `sprite-gen` are recorded here. Versions track the `version:` field in `SKILL.md` and `pyproject.toml`.
+## Unreleased - The new RIFE torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan; rife-ncnn-vulkan stays the default
+
+**Upgrading from 2.45.2: Defaults stay unchanged, cache keys update once, reports show the backend, install checks model hashes, and schemas are unchanged.** With `SPRITE_GEN_RIFE_BACKEND` unset (the default), RIFE is the `rife-ncnn-vulkan` binary as before. Its frames are the same bytes as 2.45.2. Measured: the binary's frames and sixteen-pass readings through the changed engine equal the old engine's, byte for byte. A `--between-cache` directory kept by 2.45.2 is made again once. The key now names the backend (`backend: "ncnn"`). The engine's `rife.py` changed, and its sha256 is part of the key anyway. Reports take `interpolator.kind` from what made the frames: `rife-ncnn-vulkan` as before for the binary, or `rife-torch` for the new backend. The overlapped wrap's record counts `binary_calls` for the binary as before, or `runs` for the torch backend. `sprite-gen rife install` now checks the model's two files (`rife-v4.6/flownet.param`, `flownet.bin`) against pinned SHA-256 after unpacking, for either backend. A model that differs installs nothing. The line a missing RIFE is reported with (`install` in records, warnings) is `sprite-gen rife install` for the binary. Where the torch backend is chosen, that line is: `pip install "sprite-gen[torch]" && sprite-gen rife install --backend torch`. No command-line flags of `video-loop` / `video-cycle-align` / `video-set` change. No restoration schemas change: `source-restoration-v5`, repair report `schema_version` 4, comparison `schema_version` 6.
+
+- **New: RIFE's torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan.** `SPRITE_GEN_RIFE_BACKEND=torch` runs the same rife-v4.6 in PyTorch (`sprite_gen/video/rife_torch.py`). It serves a GPU the binary cannot run on: rife-ncnn-vulkan needs Vulkan, and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. The network is rife-v4.6's IFNet written again from Practical-RIFE's `IFNet_HDv3` (MIT, notice in NOTICE). Its weights are read out of the binary's own model files (SHA-256 pinned, the binary's float16 weights), with no other download. One run does what one call of the binary does with `-x -z`. The pair is turned eight ways and run both ways in time, with flows averaged at each of four levels. The loop follows rife-ncnn-vulkan's `RIFE::process_v4` (MIT, notice in NOTICE). `SPRITE_GEN_RIFE_DEVICE` names `cuda` (default) or `cpu`. `SPRITE_GEN_RIFE_TTA` means the same for both backends. The default augmentation stays `-x -z`. To install: `pip install "sprite-gen[torch]"` (new extra: `torch>=2.13.0,<3`). Then run: `sprite-gen rife install --backend torch` (new flag; defaults to `SPRITE_GEN_RIFE_BACKEND`). This unpacks the model, and makes a check frame through PyTorch on `SPRITE_GEN_RIFE_DEVICE`. It reads the model only, so a Linux ARM machine the binary does not run on is not refused. A CPU-only host installs PyTorch's CPU build first: `pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu` (192 MB, against the CUDA build's 527 MB plus its CUDA libraries). Nothing falls back. With torch chosen the binary is never looked for or run. `cuda` where PyTorch finds no CUDA is refused naming why (a CPU-only PyTorch, or a CUDA that finds no GPU) and does not run on the CPU. `mps` (Apple's GPU) is refused (no lane measures it; on a Mac the binary runs on the GPU already). A backend or device named wrong is refused. PyTorch not installed, or no model where the backend looks (`SPRITE_GEN_RIFE_MODEL`, else the install root), is RIFE not installed, as the binary missing is. Under `--repair auto` the loop is cut as filmed with the warning, and its record names the torch install line. On CUDA each frame runs with cuDNN deterministic, its benchmark off and convolutions in IEEE float32 (no TF32). The same frame twice is the same bytes on one GPU, driver and PyTorch. What made a frame is the RIFE's identity: backend, model bytes loaded, network code, PyTorch, on CUDA the GPU, driver, CUDA, cuDNN and those settings; on the CPU its instruction set. Because of this identity, `--between-cache` never hands one backend's or GPU's frame to another. Measured on synthetic walkers: Seven walker pairs (clean, outline lost, ghost, torn, a cross-fade to look at) get the same verdicts from the binary on an Apple GPU, the binary on Mesa llvmpipe and the torch backend on a CPU. The torch frame lies 0.005 to 0.052 of 255 on average from the binary's Apple GPU frame. The binary's own llvmpipe frame lies 0.007 to 0.25. A 960 x 960 frame of a synthetic walker with `-x -z`: On an Apple M4 Max CPU (12 threads), 3.3 to 3.6 s; the binary on that Mac's GPU takes 1.27 to 1.37 s. In a Modal container of two CPUs (the app's job size), 22.6 to 28.5 s on two threads; the binary on Mesa llvmpipe in the same container takes 127.7 to 131.5 s. The torch backend's frames are not the binary's bytes, which is one reason the binary stays the default. [loop-repair](docs/loop-repair.md) section 1
+- **Lane: `scripts/linux_lane.sh` tests PyTorch 2.13.0 CPU after the suite.** After the suite, PyTorch 2.13.0 CPU is installed into the same environment. It runs `sprite-gen rife install --backend torch`'s check frame. It runs every test marked `torch_rife` with `SPRITE_GEN_RIFE_DEVICE=cpu`, with none skipped. `--rife-only` runs the same. `--rife-unmeasured` lets `torch_rife` tests skip as it lets `real_rife` ones. CUDA is not measured by this lane.
+- **Tests**: `tests/video/test_rife_torch.py` (the eight ways undo themselves and are the single passes' eight turns; flows of a textured patch run alone each of the sixteen ways and turned back point alike, where a wrong sign, a transpose's x/y, or a backward swap puts one elsewhere; `-z` and `-x` give the same frame the other way in time and flipped; the install check frame's disc lands half way; the model loads strict and other bytes are refused by SHA-256; the same frame twice is the same bytes and the identity names what makes it; seven recorded synthetic walker pairs in `tests/fixtures/rife-walkers` are judged as the binary judged them with no binary, the frame within twice the binary's own GPU-to-GPU gap, and alike to this machine's binary where both run; without PyTorch: torch chosen and not installed is RIFE not installed and the binary is never run, a loop's record names it, a backend or device named wrong is refused before anything is looked for, cuda where there is none is refused, a model not the pinned one is refused; with a runner stood in: reports name rife-torch and count runs, the loop and the alignment name the torch backend, and a frame kept by one runner is not taken for another); `tests/video/test_rife_install.py` (a zip whose model is not the pinned one installs nothing for both backends; `--backend torch` installs on a machine the binary does not run on and checks its frame through torch); `tests/release/test_lane_rife_collection.py` (a test RIFE decides carries `real_rife` or `torch_rife`); new pytest mark `torch_rife` (skipped where PyTorch, the model or the named device is missing).
+
## v2.45.2 - The new `video-loop --overlap-wrap off` turns the overlapped wrap off on its own; the motion anchor's fine search is faster and gives the same values
**Upgrading from 2.45.1: By default a run delivers the same bytes as 2.45.1, sooner, and only `--overlap-wrap off` changes what is delivered.** With the default `--overlap-wrap on`, the loop files, the frames and the report are the same bytes as 2.45.1 (the report's `producer` field names the new version), and `video-loop --anchor motion-auto` finishes sooner. With `--overlap-wrap off`, the overlapped wrap is not tried (the report's `wrap_overlap.skipped` reads `turned off (--overlap-wrap off)`, no RIFE calls), a `warning:` line is printed, and the seam gate refuses the cut as filmed, so the run fails on its seam instead of spending minutes of CPU on RIFE. No restoration schemas change: `source-restoration-v5`, repair report `schema_version` 4, comparison `schema_version` 6.
diff --git a/NOTICE b/NOTICE
index 83e09ec..a142b10 100644
--- a/NOTICE
+++ b/NOTICE
@@ -27,3 +27,14 @@ The estimation-only `estimate_pixel_grid_runlen` pitch estimator
block-size estimation from perfectpixel-studio
(https://github.com/gykim80/perfectpixel-studio, internal/sprite/pixelize.go),
Copyright Andrew Kim (gykim80), MIT License.
+
+RIFE's torch backend (sprite_gen/video/rife_torch.py) writes again the IFNet of
+rife-v4.6 — its structure and parameter names, and its warp — from Practical-RIFE
+(https://github.com/hzwer/Practical-RIFE, commit f6b5132, train_log/IFNet_HDv3.py
+and model/warplayer.py), Copyright (c) 2021 hzwer, MIT License.
+
+Its test-time augmentation — the pair turned eight ways and run both ways in time,
+the flows averaged at every level — follows rife-ncnn-vulkan 20221029
+(https://github.com/nihui/rife-ncnn-vulkan, src/rife.cpp RIFE::process_v4 and its
+rife_*_tta shaders), Copyright (c) 2020 nihui, MIT License. It reads its weights
+from that release's rife-v4.6 model, which sprite-gen does not ship.
diff --git a/SKILL.md b/SKILL.md
index cc2f285..6e2fe02 100644
--- a/SKILL.md
+++ b/SKILL.md
@@ -12,7 +12,7 @@ depends_on:
- name: img2webp
why: "video-loop WebP with exact alpha (libwebp); Pillow's animated writer drops -exact"
- name: rife-ncnn-vulkan
- why: "video-loop jump repair and video-cycle-align (RIFE v4.6 in-betweens with the -x -z test-time augmentation, release 20221029). Install once with `sprite-gen rife install` (sha256-checked, into the user data directory); SPRITE_GEN_RIFE or PATH override it; Linux without a GPU also needs libvulkan1 mesa-vulkan-drivers. Without it, walk and run loops are cut as filmed with a warning"
+ why: "video-loop jump repair and video-cycle-align (RIFE v4.6 in-betweens with the -x -z test-time augmentation, release 20221029). Install once with `sprite-gen rife install` (sha256-checked, into the user data directory); SPRITE_GEN_RIFE or PATH override it; Linux without a GPU also needs libvulkan1 mesa-vulkan-drivers. Without it, walk and run loops are cut as filmed with a warning. SPRITE_GEN_RIFE_BACKEND=torch runs the same model in PyTorch instead, for a GPU with no Vulkan (`pip install \"sprite-gen[torch]\"`, then `sprite-gen rife install --backend torch`; SPRITE_GEN_RIFE_DEVICE=cuda or cpu)"
required_scripts:
- scripts/prepare_sprite_run.py
- scripts/generate_sprite_image.py
diff --git a/docs/loop-repair.md b/docs/loop-repair.md
index b21f006..e67e0cb 100644
--- a/docs/loop-repair.md
+++ b/docs/loop-repair.md
@@ -1,6 +1,6 @@
# Loop repair — RIFE in-betweens for jump frames, the jolt index, one cycle per set
-> Owns: Where RIFE runs and what it costs, which model and augmentation it runs with, how it is installed and what runs without it, jump-frame repair in `video-loop`, the flow reading (frames off a walk's way), the eased retime (a part that rushes on its way), the jolt index and its gate, cycle alignment across a direction set · Index: [docs/README.md](README.md)
+> Owns: Where RIFE runs and what it costs, which model and augmentation it runs with, its two backends (the binary and PyTorch), how it is installed and what runs without it, jump-frame repair in `video-loop`, the flow reading (frames off a walk's way), the eased retime (a part that rushes on its way), the jolt index and its gate, cycle alignment across a direction set · Index: [docs/README.md](README.md)
A walk loop cut from a generated clip can look right in every still and still hitch when it
plays: the video model redraws thin hair a little differently every frame, and now and then
@@ -30,6 +30,53 @@ the binary's test-time augmentation, spatial and temporal (`-x -z`, `rife.TTA`;
| `rife-ncnn-vulkan-20221029-macos.zip` | `4a63a1f3c9c715773c57d2ee51df1b315ed20cd6c63103e45c483ecc4400b595` |
| `rife-ncnn-vulkan-20221029-windows.zip` | `d8e4d772d26cd8006ef0ad0bc82eb191b53c68677d1ae2f42506d74cbbbea606` |
+| Model file (the macOS and Ubuntu zips hold the same, measured 2026-10-21) | SHA-256 |
+|---|---|
+| `rife-v4.6/flownet.param` | `724569596bcd1e7b9fa50455c604777ebed99746d2ef40aa86e31b5725f1053c` |
+| `rife-v4.6/flownet.bin` | `f334ed2260149ce0188a6dcf049844e8b0cdd912e01cbcfb63553157d2508958` |
+
+#### The torch backend: `SPRITE_GEN_RIFE_BACKEND=torch`
+
+The binary is the default. `SPRITE_GEN_RIFE_BACKEND=torch` runs the same model in PyTorch instead
+(`sprite_gen/video/rife_torch.py`), for a GPU the binary cannot run on: rife-ncnn-vulkan needs Vulkan,
+and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. It is the same RIFE:
+
+- the network is rife-v4.6's IFNet (Practical-RIFE `IFNet_HDv3`), written again with its parameter
+ names;
+- its weights are read out of the binary's own model files above, checked against those SHA-256 first
+ (the binary's float16 weights widened as the binary widens them, no other download);
+- one run does what one call of the binary does with `-x -z`: the pair turned its eight ways and run
+ both ways in time, every way's flows turned back and averaged at each of the network's four levels,
+ the frames averaged and rounded as the binary rounds them; `SPRITE_GEN_RIFE_TTA` means the same;
+- a frame is the same two runs (colour, coverage) and a read the same sixteen passes (section 4), the
+ engine's part of them done by the same code for either backend (`rife.frame_runs`, `rife.pass_runs`).
+
+| Variable | Values | Default |
+|---|---|---|
+| `SPRITE_GEN_RIFE_BACKEND` | `ncnn` (the binary), `torch` | `ncnn` |
+| `SPRITE_GEN_RIFE_DEVICE` | `cuda`, `cpu` (the torch backend's device) | `cuda` |
+| `SPRITE_GEN_RIFE_MODEL` | a `rife-v4.6` directory | beside the binary (`ncnn`); the install root's (`torch`) |
+| `SPRITE_GEN_RIFE_TTA` | `-x`, `-z`, both, or empty | `-x -z`, for both backends |
+
+Nothing falls back. With torch chosen the engine neither looks for the binary nor runs it. A device is
+not chosen for you and not traded for another: `cuda` where PyTorch finds no CUDA is refused, naming
+why (a PyTorch built for the CPU only, or a CUDA that finds no GPU), and does not run on the CPU.
+Apple's GPU (`mps`) is refused: no release lane measures it, and on a Mac the binary already runs on
+the GPU. PyTorch not installed, or no model where it looks, is RIFE not installed ("Without RIFE"
+below); a backend or device named wrong, a model whose files are not the pinned ones, or a run that
+fails on the device, is a RIFE that fails.
+
+On CUDA every frame runs with cuDNN deterministic, its benchmark off and its convolutions in IEEE
+float32 (`torch.backends.cudnn.conv.fp32_precision = "ieee"`: PyTorch turns TF32 on by default), so the
+same frame twice is the same bytes on one GPU, driver and PyTorch. On the CPU it is the same bytes
+whatever the thread count. PyTorch does not promise the same bytes across GPUs or releases, so what
+made a frame (the backend, the model bytes it loaded, its network code, PyTorch, on CUDA the GPU,
+driver, CUDA, cuDNN and those settings as they read back, on the CPU its instruction set) is the RIFE's
+identity, and a frame kept for one is never taken for another (section 4, `--between-cache`). The
+engine's reports name what made their frames: `interpolator.kind` is `rife-ncnn-vulkan` or
+`rife-torch`, and the overlapped wrap counts the binary's calls (`binary_calls`) or the torch backend's
+runs (`runs`).
+
### Install — `sprite-gen rife install`
```bash
@@ -55,11 +102,28 @@ finds out at install time (`--no-check` skips it). `--zip FILE` installs from a
downloaded (behind a proxy, in an image build), checked the same way; `--dir DIR` installs
elsewhere and prints the `SPRITE_GEN_RIFE=…` line the engine then needs. Other platforms (Linux
on ARM, for one) have no release build: build it from the repository and set `SPRITE_GEN_RIFE`.
+Either backend's install checks the model's two files against their SHA-256 above once the zip is
+unpacked, and installs nothing where they differ.
The engine finds the binary by, in order: `SPRITE_GEN_RIFE` (the binary's path), then
-`rife-ncnn-vulkan` on `PATH`, then the install root above. The model is `SPRITE_GEN_RIFE_MODEL`
-or `rife-v4.6/` next to the binary. When an earlier place holds another RIFE, the install says
-which one the engine runs.
+`rife-ncnn-vulkan` on `PATH`, then the install root above. The model is `SPRITE_GEN_RIFE_MODEL` or
+`rife-v4.6/` next to the binary. When an earlier place holds another RIFE, the install says which one
+the engine runs.
+
+For the torch backend:
+
+```bash
+pip install "sprite-gen[torch]" # PyTorch 2.13 or later
+sprite-gen rife install --backend torch # the model, and a check frame through PyTorch
+```
+
+`--backend torch` (the default where `SPRITE_GEN_RIFE_BACKEND=torch`) installs the same zip and makes
+the check frame through PyTorch on `SPRITE_GEN_RIFE_DEVICE` (name `cpu` for an image built without a
+GPU). It reads the model only, so a Linux machine the binary does not run on (ARM) is not refused. The
+backend reads the model from `SPRITE_GEN_RIFE_MODEL`, else the install root's `rife-v4.6/`, never from
+beside a binary on `PATH`. PyPI's Linux wheel of PyTorch is its CUDA build (527 MB, and its CUDA
+libraries); a host that runs the backend on the CPU only installs PyTorch's CPU build first (192 MB):
+`pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu`.
### Without RIFE
@@ -72,6 +136,11 @@ cut before RIFE existed still cuts — and nothing is left unsaid:
| `video-loop` jump repair (section 2) | `--repair auto`: a jump is cut as filmed, and each filmed ghost is given way to the clean frame beside it (its own `warning:` line). Where a jump is left, `jump_repair` reads `why`, `rife` (what was not found), `install` and `unmade` (the frames, neighbours and `off` of the group the flow search stopped at), and one `video-loop: warning:` line on stderr says the jump frame was not repaired. `applied` is false only where nothing was replaced | `--repair on`: fails with the install line |
| `video-set` cycle alignment (section 4) | `--align-cycles auto`: the state is skipped and every loop keeps its own length; `cycle_align.<state>` reads `applied: false` with `why`, `rife`, `install`; the set report lists it under `warnings`; one `video-set: warning:` line | `video-cycle-align`: fails with the install line |
+With `SPRITE_GEN_RIFE_BACKEND=torch`, PyTorch not installed or no model where the backend looks is RIFE
+not installed in the same way: the records' `rife` names what was missing and `install` gives the torch
+backend's line (`pip install "sprite-gen[torch]" && sprite-gen rife install --backend torch`); the
+binary is not run in its place.
+
A RIFE that is found and then fails (a binary that cannot reach Vulkan, say) is an error in every
mode: the install is there and broken, which a warning would hide. A jolt gate passed to a loop
whose repair could not run (section 3) reads the loop as filmed, and its failure says the
@@ -198,8 +267,37 @@ minutes. A deployment that aligns on a CPU names a lighter augmentation for that
`SPRITE_GEN_RIFE_TTA=` (none) is 4 to 6 s a 960 × 960 frame, `-z` 7 to 10 s. On a Mac it is under
two seconds either way. That is the reason section 4 keeps every frame that lands on a source frame.
-A PyTorch RIFE was not chosen: it would add a framework of hundreds of megabytes to the image
-for the same model, and llvmpipe already gives the GPU's frame.
+**The torch backend is for a GPU the binary cannot use; the binary stays the default.** A GPU offered
+for compute only cannot run the binary at all, and there the torch backend runs the same model and
+augmentation. Elsewhere it adds PyTorch, hundreds of megabytes, for the same model: on a Mac the binary
+already runs on the GPU. On a CPU host the torch backend is faster than the binary on llvmpipe (the
+table below), but its frames are not the binary's bytes; a deployment that wants it there names it, and
+the default does not move. Measured on synthetic walkers, 2026-10-21:
+
+- **The same verdicts.** The seven pairs of `tests/fixtures/rife-walkers` (clean steps, an outline lost
+ where legs cross far, a ghost band, parts torn, a cross-fade to look at) get the same faults and
+ looks from the binary on an Apple GPU, the binary on llvmpipe and the torch backend on a CPU, each
+ reading at least a tenth of its bound away from it (and three more pairs besides, one left out for a
+ torn reading 8 % from its bound on llvmpipe). The tests keep it: the torch backend judges the seven
+ as the binary did on a machine with no binary, and as this machine's binary does where both run.
+- **Frames as close as two GPUs'.** Against the binary's frame on the Apple GPU, the torch backend's
+ lies 0.005 to 0.052 of 255 apart on average and 11 at most (premultiplied RGBA in the box either
+ covers), and the binary's own frame on llvmpipe 0.007 to 0.25 apart and 76 at most: closer on six of
+ the seven pairs, the seventh 7 % further.
+- **Time** (a 960 × 960 frame of the walker drawn 512 × 768, legs crossing far and the short step, t
+ 0.5, after a warm call):
+
+| | `-x -z` frame | plain frame | read (`-x -z` frame and sixteen passes) |
+|---|---|---|---|
+| binary, Apple M4 Max GPU | 1.27 to 1.37 s | 0.77 to 0.82 s | 12.6 to 13.2 s |
+| torch backend, Apple M4 Max CPU (12 threads) | 3.3 to 3.6 s | 0.38 to 0.40 s | 8.8 to 9.0 s |
+| binary, Modal `cpu=2` (llvmpipe), one cold call each | 127.7 to 131.5 s | 9.3 to 9.5 s | not run |
+| torch backend, the same Modal container, 2 threads | 22.6 to 28.5 s | 1.5 to 1.7 s | 45.1 to 46.1 s |
+
+The torch backend's frame on an x86 CPU is not the same bytes as on an Apple CPU (its torn reading is),
+which the CPU's instruction set in its identity keeps apart. PyTorch sizes its threads by the host's
+cores, not a container's share (the container above saw 18): a container names its share
+(`OMP_NUM_THREADS`). On a CUDA GPU the release lane measures it (docs/release.md).
### Licences
@@ -210,8 +308,12 @@ for the same model, and llvmpipe already gives the GPU's frame.
| RIFE v4.x weights (Practical-RIFE) | MIT — "The content of these links is under the same MIT license as this project." | github.com/hzwer/Practical-RIFE |
| ncnn | BSD 3-Clause | github.com/Tencent/ncnn |
-sprite-gen ships none of these. A deployment that bakes the release zip into an image ships
-the zip's `LICENSE` with it.
+| sprite-gen's torch backend (`sprite_gen/video/rife_torch.py`) | MIT notices in `NOTICE`: the IFNet written again from Practical-RIFE `IFNet_HDv3.py` and `warplayer.py`, © 2021 hzwer; the augmentation from rife-ncnn-vulkan's `RIFE::process_v4`, © 2020 nihui | Practical-RIFE f6b5132 (2022-09-26); rife-ncnn-vulkan 20221029 |
+| PyTorch (the `torch` extra) | BSD 3-Clause | github.com/pytorch/pytorch |
+
+sprite-gen ships none of the binaries, weights or PyTorch: its torch backend is code written again from
+the two MIT sources above, their notices in `NOTICE`, and reads the weights from the release zip's
+model. A deployment that bakes the release zip into an image ships the zip's `LICENSE` with it.
## 2. Jump frames — `video-loop --repair auto`
@@ -1560,22 +1662,28 @@ sprite-gen video-cycle-align --loop-dir set/front-walk/loop --loop-dir set/side-
remembered: the RIFE binary file and each file in the model directory, by sha256; the model directory's
path as given to the binary (the binary reads the model's version from the directory name); the
augmentation every call asks for at that moment (`rife.tta()`: `SPRITE_GEN_RIFE_TTA`, or the default
- `-x -z` when it is not set — section 1, "Which setting"), as another augmentation makes another frame; the
- engine code as loaded, which is the sha256 of `sprite_gen/video/rife.py` read once at import (`rife.py` written
- over on disk during a run changes no code that is running, so a frame made by the loaded code is kept
- under that code, and a later process that loads other code gets another key and makes the frames again);
- the NumPy and Pillow versions; and the host as `os.uname()` reports it at that moment (host name,
- OS release, build, and architecture), with the OS platform. A binary or model file that cannot be read
- fails the alignment, as a RIFE that fails does. Any change in cycle length, `--cycles`, source pixels,
- binary, model, augmentation, engine code, or host makes the frame again. The key does not read the GPU driver or its
- environment; a caller that changes those between passes gives each pass its own directory. Reading the
- identity hashes 36.5 MB of binary and model files in about 17 ms on an Apple M1: once for a frame taken,
- twice for a frame made (making a frame with RIFE takes on the order of a second). After a frame is made,
- its key is read again; if anything it is made from changed while it was being made, the frame is handed
- on but not kept, and counted as `not_kept`. Each kept frame is a PNG file named by its key, with its
- pixel sha256 in a text chunk, written to a temporary file and renamed into place. A kept file that does
- not read back as that frame is made again, written over, and counted as `unreadable`. A taken frame is
- a new image of exactly the kept pixels, so the aligned loop is the same, byte for byte, as an alignment
+ `-x -z` when it is not set — section 1, "Which setting"), as another augmentation makes another frame;
+ the engine code as loaded, which is the sha256 of `sprite_gen/video/rife.py` read once at import
+ (`rife.py` written over on disk during a run changes no code that is running, so a frame made by the
+ loaded code is kept under that code, and a later process that loads other code gets another key and
+ makes the frames again); the NumPy and Pillow versions; and the host as `os.uname()` reports it at that
+ moment (host name, OS release, build, and architecture), with the OS platform. With the torch backend
+ (section 1) the binary's and the model's files give way to what the backend loaded and runs on, read
+ once when it loads and kept for its life: the sha256 of the model bytes it read, its network code
+ (`rife_torch.py`, read at import), PyTorch's version, the device, and on CUDA the GPU's name and
+ capability, the driver, CUDA's and cuDNN's versions and the determinism settings as they read back, on
+ the CPU its instruction set; the key names the backend either way, so a frame one backend made is never
+ taken for the other's. A binary or model file that cannot be read fails the alignment, as a RIFE that
+ fails does. Any change in cycle length, `--cycles`, source pixels, binary, model, augmentation, engine
+ code, or host makes the frame again. The key does not read the GPU driver or its environment; a caller
+ that changes those between passes gives each pass its own directory. Reading the identity hashes 36.5
+ MB of binary and model files in about 17 ms on an Apple M1: once for a frame taken, twice for a frame
+ made (making a frame with RIFE takes on the order of a second). After a frame is made, its key is read
+ again; if anything it is made from changed while it was being made, the frame is handed on but not
+ kept, and counted as `not_kept`. Each kept frame is a PNG file named by its key, with its pixel sha256
+ in a text chunk, written to a temporary file and renamed into place. A kept file that does not read
+ back as that frame is made again, written over, and counted as `unreadable`. A taken frame is a new
+ image of exactly the kept pixels, so the aligned loop is the same, byte for byte, as an alignment
without the directory. Nothing is kept without `--between-cache` (there is no default directory). The
caller gives each alignment its own directory and removes it when that alignment is done. The engine
creates `DIR` mode 0700 and files mode 0600, never deletes `DIR`, and records `dir`, `hits`, `misses`,
diff --git a/docs/release.md b/docs/release.md
index db8b0d3..6fea27f 100644
--- a/docs/release.md
+++ b/docs/release.md
@@ -24,6 +24,19 @@ RIFE 가 닿는 곳이 하나도 없는 상태로 스위트 전체를 돌려 그
시험(`pytest -m real_rife`)을 돌린다 — 하나라도 건너뛰거나, 표시된 시험이 없으면 실패다. 그래서 새
실제-RIFE 시험은 표시만 붙이면 `--rife-only` 가 저절로 걷는다.
+RIFE 의 torch 백엔드(`SPRITE_GEN_RIFE_BACKEND=torch`, docs/loop-repair.md 1절)를 돌리는 시험은
+`torch_rife` 표시를 단다(`pyproject.toml` 이 등록한다). `tests/conftest.py` 는 PyTorch 가 import 되지
+않거나, `rife.locate_model()` 이 모델을 못 찾거나, `SPRITE_GEN_RIFE_DEVICE` 가 이름 댄 장치가
+없으면(기본 cuda 인데 CUDA 가 없으면) 그 표시가 붙은 시험을 건너뛴다. 레인은 스위트를 다 돈 뒤 같은
+환경에 PyTorch 2.13.0 의 CPU 판을 PyTorch CPU 인덱스에서 깔고, `sprite-gen rife install --backend torch`
+의 확인 프레임(PyTorch 로 만든다)과 `torch_rife` 가 붙은 시험 전부(`pytest -m torch_rife`)를
+`SPRITE_GEN_RIFE_DEVICE=cpu` 로 돌린다. 하나라도 건너뛰거나 표시된 시험이 없으면 실패다. `--rife-only`
+도 같은 단계를 돈다. `real_rife` 와 `torch_rife` 가 함께 붙은 시험(같은 기계의 바이너리와 torch 판의
+판정 비교)은 `real_rife` 단계에서 빼고 이 단계에서 돈다. `--rife-unmeasured` 는 이 단계를 돌지 않고, 그
+실행에서는 `torch_rife` 가 붙은 시험도 RIFE 때문에 건너뛸 수 있는 것으로 친다.
+`tests/release/test_lane_rife_collection.py` 의 표시 검사도 두 표시를 같이 받는다. CUDA 위의 torch 판은
+이 레인이 재지 않는다.
+
표시 없이 RIFE 가 없을 때 건너뛰거나 모이지 않는 시험은 두 실행 어느 쪽도 재지 않는다. 그래서
`--rife-unmeasured` 는 시험을 `pytest --rife-unmeasured` 로 돌리고, 그 실행에서 건너뛴 것마다 — skip
표시로든, fixture 나 시험 본문에서든, 파일째 import 에서든 — `tests/conftest.py` 가 `real_rife` 표시를
diff --git a/pyproject.toml b/pyproject.toml
index 3ef60a7..8586665 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -45,6 +45,11 @@ dev = [
# subprocess.wait(), which is exactly the case that hangs here.
"pytest-timeout>=2.3",
]
+# RIFE's torch backend (SPRITE_GEN_RIFE_BACKEND=torch, docs/loop-repair.md section 1): rife-v4.6 in PyTorch, for a
+# GPU host the binary cannot run on (Vulkan-less CUDA). 2.13.0 is the release it was measured on. PyPI's Linux x86-64
+# wheel is the CUDA 13 build; a host that runs it on the CPU only takes PyTorch's CPU index
+# (`pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu`) before the extra.
+torch = ["torch>=2.13.0,<3"]
# The console script is what removes the "which python?" question: pip writes it
# with a shebang naming the interpreter of the environment it was installed
diff --git a/scripts/linux_lane.sh b/scripts/linux_lane.sh
index ada08cb..99e7c81 100755
--- a/scripts/linux_lane.sh
+++ b/scripts/linux_lane.sh
@@ -5,7 +5,10 @@
# What the CI job ran, in the same order: a clean virtual environment installed the way the
# quickstart does (`pip install -e ".[dev]"`), RIFE installed through the user's own path
# (`sprite-gen rife install`, its check frame on the host's Vulkan driver), the quickstart smoke
-# (prepare -> extract -> compose) and the whole test suite.
+# (prepare -> extract -> compose) and the whole test suite. Then RIFE's torch backend on the CPU:
+# PyTorch (TORCH_VERSION, from PyTorch's CPU index) into the same environment, its install check
+# frame (`sprite-gen rife install --backend torch`) and every test marked torch_rife, none skipped
+# (SPRITE_GEN_RIFE_DEVICE=cpu). Its CUDA run is the T4 lane's (docs/release.md).
#
# Run it from a clean clone of the commit being measured, on x86-64 Linux with:
# - Python 3.14 (`python3.14` on PATH, or named by SPRITE_GEN_LANE_PYTHON),
@@ -29,9 +32,10 @@
# tests/conftest.py names it as not RIFE's. The run ends by naming
# what it did not measure and exits 3, not 0.
# --rife-only the RIFE gates alone, on a host where Vulkan works: `rife install`
-# (its check frame) and every test of the suite marked `real_rife`,
-# none of it skipped — the tests `--rife-unmeasured` let skip for RIFE
-# (docs/release.md).
+# (its check frame), every test of the suite marked `real_rife`, and
+# the torch backend's gates above, none of it skipped — the tests
+# `--rife-unmeasured` let skip for RIFE (docs/release.md). The tests
+# marked both run with the torch backend's.
# A commit has passed the lane when one run exits 0, or when a `--rife-unmeasured` run (exit 3)
# and a `--rife-only` run (exit 0) of the same commit both did.
set -euo pipefail
@@ -69,6 +73,9 @@ fi
PINNED_PYTHON=3.14
PYTHON=${SPRITE_GEN_LANE_PYTHON:-python$PINNED_PYTHON}
+# The PyTorch the torch backend was measured on; its CPU build, as a host without a GPU installs it.
+TORCH_VERSION=2.13.0
+TORCH_CPU_INDEX=https://download.pytorch.org/whl/cpu
fail() {
echo "linux_lane: $*" >&2
@@ -77,6 +84,20 @@ fail() {
step() {
echo "== $*"
}
+# RIFE's torch backend on the CPU: PyTorch into .venv, the install's check frame through it (the model
+# `sprite-gen rife install` already put in place), and every test marked torch_rife, none skipped.
+torch_gates() {
+ step "RIFE torch backend (PyTorch $TORCH_VERSION, CPU)"
+ .venv/bin/pip install --quiet --index-url "$TORCH_CPU_INDEX" "torch==$TORCH_VERSION"
+ .venv/bin/python -c 'import torch; print("torch", torch.__version__, "cuda", torch.version.cuda)'
+ SPRITE_GEN_RIFE_BACKEND=torch SPRITE_GEN_RIFE_DEVICE=cpu .venv/bin/sprite-gen rife install
+ step "RIFE torch tests (every test marked torch_rife, none skipped)"
+ torch_log=$(mktemp)
+ # No test marked: pytest exits 5 ("no tests ran"), and the lane with it.
+ SPRITE_GEN_RIFE_DEVICE=cpu .venv/bin/python -m pytest -q -rs -m torch_rife | tee "$torch_log"
+ ! grep -E '^SKIPPED' "$torch_log" >/dev/null || fail "a torch_rife test skipped; this run is meant to measure them"
+ rm -f "$torch_log"
+}
cd "$(git rev-parse --show-toplevel)"
root=$(pwd -P)
@@ -121,9 +142,11 @@ if [ "$rife_only" = 1 ]; then
step "RIFE tests (every test marked real_rife, none skipped)"
rife_log=$(mktemp)
trap 'rm -f "$rife_log"' EXIT
- # No test marked: pytest exits 5 ("no tests ran"), and the lane with it.
- .venv/bin/python -m pytest -q -rs -m real_rife | tee "$rife_log"
+ # No test marked: pytest exits 5 ("no tests ran"), and the lane with it. Those marked torch_rife as
+ # well run with the torch backend's.
+ .venv/bin/python -m pytest -q -rs -m "real_rife and not torch_rife" | tee "$rife_log"
! grep -E '^SKIPPED' "$rife_log" >/dev/null || fail "a RIFE test skipped; this run is meant to measure them"
+ torch_gates
echo "linux_lane: the RIFE gates passed at $sha"
exit 0
fi
@@ -152,9 +175,14 @@ test -f "$smoke/run/sprite-sheet-alpha.png"
test -f "$smoke/run/manifest.json"
step "tests"
-# Without RIFE, a skip of a test not marked real_rife fails unless tests/conftest.py names it.
+# Without RIFE, a skip of a test not marked real_rife or torch_rife fails unless tests/conftest.py names it.
+# PyTorch is not installed yet: the tests marked torch_rife skip here and run in torch_gates.
.venv/bin/python -m pytest -q ${rife_unmeasured:+--rife-unmeasured}
+if [ -z "$rife_unmeasured" ]; then
+ torch_gates
+fi
+
if [ -n "$rife_unmeasured" ]; then
echo "linux_lane: every gate but RIFE passed at $sha"
echo "linux_lane: RIFE NOT MEASURED here ($rife_unmeasured): run \`scripts/linux_lane.sh --rife-only\` at $sha on a host where Vulkan works"
diff --git a/sprite_gen/video/align.py b/sprite_gen/video/align.py
index 056190e..0f94c16 100644
--- a/sprite_gen/video/align.py
+++ b/sprite_gen/video/align.py
@@ -988,7 +988,7 @@ def align_set(loop_dirs: list[Path], *, length: int | None = None, interpolate:
"cycles_given": {str(loops[i][0]): k for i, k in given.items()},
"feet_given": {str(loops[i][0]): f for i, f in told.items()},
"fps": round(fps, 4), "cycle_seconds": round(target / fps, 4),
- "interpolator": ({"kind": "rife-ncnn-vulkan", **located[0]} if located else
+ "interpolator": (dict(located[0]) if located else
{"kind": "injected"} if any(r.get("smear") for r in rows) else None),
"between_cache": cache.record() if cache is not None else None,
"made_by_rife": sum(r["made_by_rife"] for r in rows),
@@ -1074,7 +1074,7 @@ def run(**kwargs: object) -> int:
except rife_mod.RifeNotInstalled as exc:
# Asked for by name, so no RIFE is a failure, never a quiet skip (video-set skips with a warning).
raise SystemExit(f"video-cycle-align: {exc}; frames between source frames are made by RIFE — "
- f"`{rife_mod.INSTALL_COMMAND}` (docs/loop-repair.md)") from exc
+ f"`{rife_mod.install_command()}` (docs/loop-repair.md)") from exc
for line in report["warnings"]:
print(f"video-cycle-align: warning: {line}", file=sys.stderr)
print(json.dumps({k: report[k] for k in ("length", "lengths", "cycles_given", "feet_given", "between", "made_by_rife", "replaced", "retake",
diff --git a/sprite_gen/video/batch.py b/sprite_gen/video/batch.py
index 68b6544..0e966e4 100644
--- a/sprite_gen/video/batch.py
+++ b/sprite_gen/video/batch.py
@@ -972,9 +972,9 @@ def align_gaits(results: list[dict[str, Any]], root: Path, mode: str, *, interpo
views=[_align_view(r) for r in rows], state=state, feet=told)
except rife_mod.RifeNotInstalled as exc:
out[state] = {"ok": True, "applied": False, "reason": "rife-not-installed", "why": "RIFE not installed — each loop keeps its own length",
- "rife": str(exc), "install": rife_mod.INSTALL_COMMAND}
+ "rife": str(exc), "install": rife_mod.install_command()}
print(f"video-set: warning: {state}: cycles not aligned — RIFE is not installed, so each loop keeps its own "
- f"length; run `{rife_mod.INSTALL_COMMAND}`, then `sprite-gen video-cycle-align` (docs/loop-repair.md)",
+ f"length; run `{rife_mod.install_command()}`, then `sprite-gen video-cycle-align` (docs/loop-repair.md)",
file=sys.stderr)
continue
except align_mod.CycleSuspects as exc:
@@ -1154,7 +1154,7 @@ def run_set(
warnings = ([f"{r['item']}: jump frame not repaired — {r['loop']['jump_repair']['why']}" for r in results if r.get("ok") and "jump_repair" in r["loop"]]
+ [f"cycle-align:{st}: not aligned — {a['why']}" for st, a in aligned.items() if a.get("reason") == "rife-not-installed"])
if warnings:
- warnings.append(f"install RIFE with `{rife_mod.INSTALL_COMMAND}` (docs/loop-repair.md)")
+ warnings.append(f"install RIFE with `{rife_mod.install_command()}` (docs/loop-repair.md)")
# What the set left for someone to count — never a failure, never silent.
warnings += [f"cycle-align:{st}: not aligned — {a['why']}; count them, then `sprite-gen video-cycle-align --cycles <loop>=<k>` "
f"(the loops under `suspects` in {a['report']}, docs/loop-repair.md section 4)"
diff --git a/sprite_gen/video/loop.py b/sprite_gen/video/loop.py
index 499ad3c..982a114 100644
--- a/sprite_gen/video/loop.py
+++ b/sprite_gen/video/loop.py
@@ -975,11 +975,10 @@ def _repair_jumps(frames: list[Image.Image], interpolate: rife_mod.Interpolate |
if required:
raise
frames, overlap = asked(frames, read=None)
- overlap.update(why="RIFE not installed — the wrap cut as filmed", rife=str(exc), install=rife_mod.INSTALL_COMMAND)
+ overlap.update(why="RIFE not installed — the wrap cut as filmed", rife=str(exc), install=rife_mod.install_command())
overlapped = overlap["frames"]
if overlap["calls"]:
- overlap["interpolator"] = ({"kind": "rife-ncnn-vulkan", **made_by[0], "binary_calls": interpolate.calls}
- if made_by else {"kind": "injected"})
+ overlap["interpolator"] = {**made_by[0], **interpolate.counts()} if made_by else {"kind": "injected"}
wrap_calls = calls
try:
repaired, record = repair_mod.repair_jumps(frames, lazy, facing=facing, screen=screen, overlapped=overlapped,
@@ -995,17 +994,17 @@ def _repair_jumps(frames: list[Image.Image], interpolate: rife_mod.Interpolate |
if "unmade" in record:
record.update(why="RIFE not installed — cut as filmed" if not record["replaced"] else
"RIFE not installed — the jump cut as filmed, each filmed ghost given way to the clean frame beside it",
- rife=str(exc), install=rife_mod.INSTALL_COMMAND)
+ rife=str(exc), install=rife_mod.install_command())
elif record.get("ease_skipped") == "no interpolator":
# no jump, but a part that rushes: the frames either side of it were to be made again at eased times
record.update(why="RIFE not installed — the rush cut as filmed" if not record["replaced"] else
"RIFE not installed — the rush cut as filmed, each filmed ghost given way to the clean frame beside it",
- rife=str(exc), install=rife_mod.INSTALL_COMMAND)
+ rife=str(exc), install=rife_mod.install_command())
return repaired, record, overlap
frames = repaired
record["applied"] = bool(record["replaced"])
if calls > wrap_calls:
- record["interpolator"] = {"kind": "rife-ncnn-vulkan", **made_by[0]} if made_by else {"kind": "injected"}
+ record["interpolator"] = dict(made_by[0]) if made_by else {"kind": "injected"}
return frames, record, overlap
@@ -1029,7 +1028,7 @@ def _ghost_lines(jump_repair: dict[str, Any]) -> list[str]:
if r["proposal"] is not None:
why = f"RIFE's frame between its neighbours has a fault ({', '.join(r['faults'])})"
elif r["no_proposal"] == "no interpolator":
- why = f"RIFE is not installed, so no frame between its neighbours was made (`{rife_mod.INSTALL_COMMAND}`)"
+ why = f"RIFE is not installed, so no frame between its neighbours was made (`{rife_mod.install_command()}`)"
else:
why = r["no_proposal"]
lines.append(line + f"given way to frame {r['taken']['source']} beside it, shown twice in a row (cycle/frame-{j:03d}.png): "
@@ -1048,7 +1047,7 @@ def _overlap_lines(overlap: dict[str, Any]) -> list[str]:
f"two drawings inside its outline (crossfade {r['proposal']['crossfade']:.4f}); kept, a reading to look at "
"(docs/loop-repair.md section 2)" for r in taken["made"] if r["look"]]
if "rife" in overlap:
- return [f"{said}, and RIFE is not installed, so the overlapped wrap was not made — run `{rife_mod.INSTALL_COMMAND}` "
+ return [f"{said}, and RIFE is not installed, so the overlapped wrap was not made — run `{rife_mod.install_command()}` "
"and cut it again (docs/loop-repair.md section 2)"]
if "skipped" in overlap:
return [f"{said}; the overlapped wrap was not made: {overlap['skipped']} (docs/loop-repair.md section 2)"]
@@ -1692,7 +1691,7 @@ def run_loop(
frames, report_base["jump_repair"], overlap = _repair_jumps(frames, interpolate, facing=facing, required=repair == "on",
screen=report_base["ghost_screen"], wrap=wrap)
except (ValueError, rife_mod.RifeUnavailable) as exc:
- fix = (f"install RIFE (`{rife_mod.INSTALL_COMMAND}`), or pass --repair auto to cut it as filmed with a warning"
+ fix = (f"install RIFE (`{rife_mod.install_command()}`), or pass --repair auto to cut it as filmed with a warning"
if isinstance(exc, rife_mod.RifeNotInstalled) else "fix RIFE, or pass --repair off to cut it as filmed")
error = (f"video-loop: {exc}; the loop has a frame to make (after a jump, for a part that rushes, at a wrap a part "
f"on its own beat does not close, or for a filmed ghost between two clean frames) — {fix} (docs/loop-repair.md)")
@@ -1734,12 +1733,12 @@ def run_loop(
unmade = report_base["jump_repair"]["unmade"]
print(f"video-loop: warning: a jump frame was not repaired (frames {unmade['frames']}, {unmade['off']:.2f}x its "
"region's usual step off the line between its neighbours) — RIFE is not installed, so the loop is cut as filmed; "
- f"run `{rife_mod.INSTALL_COMMAND}` and cut it again (docs/loop-repair.md)", file=sys.stderr)
+ f"run `{rife_mod.install_command()}` and cut it again (docs/loop-repair.md)", file=sys.stderr)
elif "rife" in report_base["jump_repair"]:
rush = report_base["jump_repair"]["surge"]
print(f"video-loop: warning: a part of the loop rushes (frames {rush['frames']}, {rush['ratio']:.1f}x its steps "
"either side) and was not eased — RIFE is not installed, so the loop is cut as filmed; "
- f"run `{rife_mod.INSTALL_COMMAND}` and cut it again (docs/loop-repair.md section 2)", file=sys.stderr)
+ f"run `{rife_mod.install_command()}` and cut it again (docs/loop-repair.md section 2)", file=sys.stderr)
else:
report_base["jump_repair"] = {"applied": False, "why": "--repair off" if repair == "off" else f"not a gait state ({state})"}
if prof.gait:
@@ -1766,7 +1765,7 @@ def run_loop(
error = (f"video-loop: loop jolts — {'; '.join(over)} ({cycle.get('kind', 'periodic')} length {L} frames from {i}"
f"{', after repairing frames ' + str(report_base['jump_repair']['replaced']) if report_base['jump_repair'].get('replaced') else ''}); "
"the motion does not play smoothly — regenerate the clip"
- f"{' (the jump repair did not run: RIFE is not installed — ' + rife_mod.INSTALL_COMMAND + ')' if 'rife' in report_base['jump_repair'] else ''}")
+ f"{' (the jump repair did not run: RIFE is not installed — ' + rife_mod.install_command() + ')' if 'rife' in report_base['jump_repair'] else ''}")
shutil.rmtree(out_dir / FALLBACK_FRAMES_DIR, ignore_errors=True)
write_loop_report(target, {**report_base, "status": "failed", "error": error, "cycle": cycle})
raise SystemExit(error)
diff --git a/sprite_gen/video/rife.py b/sprite_gen/video/rife.py
index 9c60ab5..4cad205 100644
--- a/sprite_gen/video/rife.py
+++ b/sprite_gen/video/rife.py
@@ -1,5 +1,6 @@
# SPDX-License-Identifier: Apache-2.0
-"""RIFE in-betweens for RGBA sprite frames, through the external `rife-ncnn-vulkan` binary.
+"""RIFE in-betweens for RGBA sprite frames, through the external `rife-ncnn-vulkan` binary or, where SPRITE_GEN_RIFE_BACKEND
+names it, the same rife-v4.6 in PyTorch (`rife_torch`).
Where it runs, what it costs and why this build: docs/loop-repair.md section 1. RIFE reads
three colour channels and no alpha, so a frame is interpolated as two images — its colour and
@@ -29,6 +30,16 @@ puts it (`install_root()`, docs/loop-repair.md section 1). A RIFE that cannot be
`RifeNotInstalled`, which a caller may answer by cutting the loop as filmed with a warning; a
RIFE that is found and then fails is `RifeUnavailable` and always an error.
+SPRITE_GEN_RIFE_BACKEND=torch runs the frame in PyTorch instead (`backend()`, `rife_torch`), on the device
+SPRITE_GEN_RIFE_DEVICE names (`device()`: cuda, the default, or cpu), from the same model files the binary runs
+(`locate_model()`, checked against MODEL_SHA256). Nothing falls back: with torch chosen the binary is neither looked for
+nor run, and a device that cannot be had is not traded for another. PyTorch or the model not installed is
+`RifeNotInstalled`, as the binary not installed is; the line that installs it is `install_command()`.
+
+Wherever it runs, a frame is the same two runs of RIFE and the read the same sixteen passes (`frame_runs`, `pass_runs`):
+what turns a frame into RIFE's input (`inputs`) and its two runs back into a frame (`_joined`) is done here, and the runs
+are a runner's (`Binary`, `rife_torch.TorchRunner`), which `Rife` hands each frame whole.
+
The interpolator is a plain callable `(a, b, t) -> frame`, so a caller can be handed a stand-in
(`Interpolate`), and a test needs no binary. A caller that reads the passes takes a callable
`(a, b, t) -> (frame, disagreement)` (`Read`, `Rife.read`).
@@ -44,6 +55,7 @@ again would give the same bytes.
from __future__ import annotations
import hashlib
+import importlib
import json
import os
import platform
@@ -62,6 +74,18 @@ from sprite_gen._deps import np
BINARY = "rife-ncnn-vulkan"
MODEL = "rife-v4.6"
RELEASE = "20221029"
+# What runs RIFE (`backend()`): the binary, the default, or rife-v4.6 in PyTorch (`rife_torch`); and the torch devices
+# it runs on (`device()`), the first the default.
+BACKENDS = ("ncnn", "torch")
+DEVICES = ("cuda", "cpu")
+# The model files of rife-v4.6 as the release zips ship them, the macOS and the Ubuntu zip alike: `sprite-gen rife install`
+# checks them, and the torch backend reads its weights out of exactly these (`model_bytes`), as it reads them by their
+# layout (docs/loop-repair.md section 1).
+MODEL_SHA256 = {
+ "flownet.param": "724569596bcd1e7b9fa50455c604777ebed99746d2ef40aa86e31b5725f1053c",
+ "flownet.bin": "f334ed2260149ce0188a6dcf049844e8b0cdd912e01cbcfb63553157d2508958",
+}
+TORCH_EXTRA = 'pip install "sprite-gen[torch]"'
# The binary's test-time augmentation, passed on every call: `-x` runs the pair as its eight flips and
# transposes, `-z` runs it both ways in time as well, and what comes back is averaged — so a frame made
# from a to b is the one made from b to a (docs/loop-repair.md section 1, "Which setting"). On a machine
@@ -218,6 +242,72 @@ def tta() -> tuple[str, ...]:
return modes
+def backend() -> str:
+ """What runs RIFE: SPRITE_GEN_RIFE_BACKEND, `ncnn` (the binary) where it is unset or empty, or `torch` (rife-v4.6 in
+ PyTorch, `rife_torch`). Anything else is refused, not read as the default."""
+ named = os.environ.get("SPRITE_GEN_RIFE_BACKEND") or "ncnn"
+ if named not in BACKENDS:
+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_BACKEND={named!r}: name ncnn ({BINARY}, the default) or torch (rife-v4.6 in "
+ f"PyTorch), or leave it unset")
+ return named
+
+
+def device() -> str:
+ """The torch device the torch backend runs on: SPRITE_GEN_RIFE_DEVICE, `cuda` where it is unset or empty, or `cpu`.
+ Nothing is chosen for it: a CUDA that is not there is refused (`rife_torch.TorchRunner`), never traded for the CPU.
+ Apple's GPU (`mps`) is not opened: no release lane measures it, and on a Mac the binary already runs on the GPU."""
+ named = os.environ.get("SPRITE_GEN_RIFE_DEVICE") or DEVICES[0]
+ if named == "mps":
+ raise RifeUnavailable("SPRITE_GEN_RIFE_DEVICE=mps: the torch backend does not run on Apple's GPU — no release lane "
+ f"measures it there; on a Mac the default backend ({BINARY}) runs on the GPU, or name cpu")
+ if named not in DEVICES:
+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_DEVICE={named!r}: name cuda (the default) or cpu")
+ return named
+
+
+def install_command() -> str:
+ """The line that installs the RIFE `backend()` names, quoted wherever a missing RIFE is reported: INSTALL_COMMAND for the
+ binary; for torch the extra and the model, `sprite-gen rife install --backend torch`. A backend refused names the
+ binary's, as `Rife()` refuses it before anything is looked for."""
+ try:
+ chosen = backend()
+ except RifeUnavailable:
+ chosen = "ncnn"
+ return INSTALL_COMMAND if chosen == "ncnn" else f"{TORCH_EXTRA} && {INSTALL_COMMAND} --backend torch"
+
+
+def locate_model() -> Path:
+ """The model directory the torch backend reads, with no binary looked for: SPRITE_GEN_RIFE_MODEL, else where
+ `sprite-gen rife install` puts the model (`<install_root()>/rife-ncnn-vulkan-<release>-<platform>/rife-v4.6`)."""
+ named = os.environ.get("SPRITE_GEN_RIFE_MODEL")
+ model = Path(named).expanduser() if named else installed_binary().parent / MODEL
+ if not (model / "flownet.param").is_file() or not (model / "flownet.bin").is_file():
+ where = "SPRITE_GEN_RIFE_MODEL" if named else "SPRITE_GEN_RIFE_MODEL is not set, and the install root"
+ raise RifeNotInstalled(f"RIFE model {model} has no flownet.param / flownet.bin ({where}); run `{install_command()}` "
+ f"(the model {MODEL}, sha256-checked, into the user data directory), or set "
+ f"SPRITE_GEN_RIFE_MODEL to a {MODEL} directory; see docs/loop-repair.md")
+ return model.resolve()
+
+
+def model_bytes(model: Path) -> dict[str, bytes]:
+ """The two files of the model in `model`, each read once and checked against MODEL_SHA256, so the bytes handed on are
+ the bytes checked: another model, or one cut short or written over, is refused by name, never read
+ (`RifeUnavailable`)."""
+ files = {}
+ for name, pinned in MODEL_SHA256.items():
+ try:
+ data = (model / name).read_bytes()
+ except OSError as exc:
+ raise RifeUnavailable(f"RIFE model {model}: {name} could not be read: {exc}") from exc
+ got = hashlib.sha256(data).hexdigest()
+ if got != pinned:
+ raise RifeUnavailable(f"RIFE model {model}: {name} has sha256 {got}, expected {pinned} ({MODEL} of "
+ f"{BINARY} {RELEASE}); install it again (`{INSTALL_COMMAND} --force`), or point "
+ f"SPRITE_GEN_RIFE_MODEL at that {MODEL}")
+ files[name] = data
+ return files
+
+
def _call(binary: Path, model: Path, a: Image.Image, b: Image.Image, t: float, tmp: Path,
modes: tuple[str, ...] | None = None) -> Image.Image:
"""One run of the binary from `a` to `b` at `t`, with the augmentation `tta()` names, or `modes` (`passes`: none)."""
@@ -279,21 +369,70 @@ def bleed(frame: np.ndarray) -> np.ndarray:
return np.where(frame[..., 3:] > 0, frame[..., :3], 0.0 if fill is None else fill)
-def between(a: Image.Image, b: Image.Image, t: float, *, binary: Path, model: Path, tmp: Path) -> Image.Image:
- """The RGBA frame at fraction `t` (0..1) of the way from `a` to `b`."""
+# One run of RIFE, wherever it runs: from the first image to the second (H x W x 3, uint8) at a fraction, with the
+# augmentation named (None: `tta()`'s, read for the run), to the image it makes (H x W x 3, uint8).
+Run = Callable[[np.ndarray, np.ndarray, float, tuple[str, ...] | None], np.ndarray]
+
+
+def inputs(a: Image.Image, b: Image.Image) -> tuple[list[np.ndarray], list[np.ndarray]]:
+ """What RIFE is given for a frame between `a` and `b`: their colour images as `bleed` makes them (H x W x 3) and their
+ coverage (H x W), uint8 — the same wherever RIFE runs."""
if a.size != b.size:
raise ValueError(f"rife: frames differ in size ({a.size} vs {b.size})")
fa, fb = (np.asarray(f.convert("RGBA"), dtype=np.float32) / 255.0 for f in (a, b))
- colour = [Image.fromarray(np.uint8(np.clip(bleed(x), 0, 1) * 255 + 0.5), "RGB") for x in (fa, fb)]
- cov = [Image.fromarray(np.uint8(x[..., 3] * 255 + 0.5), "L").convert("RGB") for x in (fa, fb)]
- rgb = np.asarray(_call(binary, model, colour[0], colour[1], t, tmp), dtype=np.float32) / 255.0
- alpha = np.asarray(_call(binary, model, cov[0], cov[1], t, tmp).convert("L"), dtype=np.float32) / 255.0
- return _joined(rgb, alpha)
+ return ([np.uint8(np.clip(bleed(x), 0, 1) * 255 + 0.5) for x in (fa, fb)],
+ [np.uint8(x[..., 3] * 255 + 0.5) for x in (fa, fb)])
+
+
+def _grey(cover: np.ndarray) -> np.ndarray:
+ """Coverage (H x W) as the grey image RIFE runs on."""
+ return np.repeat(cover[..., None], 3, axis=-1)
+
+
+def _coverage(made: np.ndarray) -> np.ndarray:
+ """The coverage RIFE made from two grey images, read back to one channel as Pillow reads a grey (`convert("L")`)."""
+ return np.asarray(Image.fromarray(made, "RGB").convert("L"))
+
+
+def frame_runs(run: Run, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray]:
+ """A frame's two runs (`inputs`): the colour, then the coverage, from the first to the second at `t` with the
+ augmentation every call asks for — (colour H x W x 3, coverage H x W), uint8, which `_joined` makes a frame."""
+ return run(colour[0], colour[1], t, None), _coverage(run(_grey(cover[0]), _grey(cover[1]), t, None))
+
+
+def pass_runs(run: Run, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> np.ndarray:
+ """How far the sixteen single passes of a frame (`inputs`) disagree, per pixel (`passes`): PASS_CALLS runs."""
+ grey = [_grey(c) for c in cover]
+ rgbs, alphas = [], []
+ for geometry in PASS_GEOMETRIES:
+ for first, second, at in ((0, 1, t), (1, 0, 1 - t)):
+ made = [run(_turned(pair[first], geometry), _turned(pair[second], geometry), at, ()) for pair in (colour, grey)]
+ rgbs.append(_turned_back(made[0].astype(np.float32) / 255.0, geometry))
+ alphas.append(_turned_back(_coverage(made[1]).astype(np.float32)[..., None] / 255.0, geometry)[..., 0])
+ rgb, alpha = np.stack(rgbs), np.stack(alphas)
+ pre = np.concatenate([rgb * alpha[..., None], alpha[..., None]], axis=-1)
+ apart = np.sqrt(((pre[0::2].mean(axis=0) - pre[1::2].mean(axis=0)) ** 2).sum(axis=-1) / 4.0)
+ return apart.astype(np.float32)
+
+
+def _binary_run(binary: Path, model: Path, tmp: Path) -> Run:
+ """A run of the binary (`_call`, its images written in `tmp`)."""
+ def run(x: np.ndarray, y: np.ndarray, t: float, modes: tuple[str, ...] | None) -> np.ndarray:
+ a, b = Image.fromarray(x, "RGB"), Image.fromarray(y, "RGB")
+ return np.asarray(_call(binary, model, a, b, t, tmp) if modes is None else _call(binary, model, a, b, t, tmp, modes))
+ return run
+
+
+def between(a: Image.Image, b: Image.Image, t: float, *, binary: Path, model: Path, tmp: Path) -> Image.Image:
+ """The RGBA frame at fraction `t` (0..1) of the way from `a` to `b`, through the binary."""
+ colour, cover = inputs(a, b)
+ return _joined(*frame_runs(_binary_run(binary, model, tmp), colour, cover, t))
def _joined(rgb: np.ndarray, alpha: np.ndarray) -> Image.Image:
- """The RGBA frame of a colour run and a coverage run (0..1): coverage under ALPHA_FLOOR dropped, the colour kept
+ """The RGBA frame of a colour run and a coverage run (uint8): coverage under ALPHA_FLOOR dropped, the colour kept
wherever there is coverage."""
+ rgb, alpha = rgb.astype(np.float32) / 255.0, alpha.astype(np.float32) / 255.0
alpha = np.where(alpha < ALPHA_FLOOR, 0.0, alpha)
out = np.dstack([np.where(alpha[..., None] > 0, rgb, 0.0), alpha])
return Image.fromarray(np.uint8(np.clip(out, 0, 1) * 255 + 0.5), "RGBA")
@@ -332,22 +471,8 @@ def passes(a: Image.Image, b: Image.Image, t: float, *, binary: Path, model: Pat
apart the mean of the eight forward passes and the mean of the eight backward ones put the premultiplied colour and
coverage (0..1, `torn` reads it). Their mean is not the frame: where they disagree it blends their guesses part-covered,
which the augmentation, combining them inside the binary, does not. PASS_CALLS calls of the binary."""
- if a.size != b.size:
- raise ValueError(f"rife: frames differ in size ({a.size} vs {b.size})")
- fa, fb = (np.asarray(f.convert("RGBA"), dtype=np.float32) / 255.0 for f in (a, b))
- colour = [np.uint8(np.clip(bleed(x), 0, 1) * 255 + 0.5) for x in (fa, fb)]
- cover = [np.repeat(np.uint8(x[..., 3:] * 255 + 0.5), 3, axis=-1) for x in (fa, fb)]
- rgbs, alphas = [], []
- for geometry in PASS_GEOMETRIES:
- for first, second, at in ((0, 1, t), (1, 0, 1 - t)):
- runs = [_call(binary, model, Image.fromarray(_turned(pair[first], geometry), "RGB"),
- Image.fromarray(_turned(pair[second], geometry), "RGB"), at, tmp, ()) for pair in (colour, cover)]
- rgbs.append(_turned_back(np.asarray(runs[0], dtype=np.float32) / 255.0, geometry))
- alphas.append(_turned_back(np.asarray(runs[1].convert("L"), dtype=np.float32)[..., None] / 255.0, geometry)[..., 0])
- rgb, alpha = np.stack(rgbs), np.stack(alphas)
- pre = np.concatenate([rgb * alpha[..., None], alpha[..., None]], axis=-1)
- apart = np.sqrt(((pre[0::2].mean(axis=0) - pre[1::2].mean(axis=0)) ** 2).sum(axis=-1) / 4.0)
- return apart.astype(np.float32)
+ colour, cover = inputs(a, b)
+ return pass_runs(_binary_run(binary, model, tmp), colour, cover, t)
def _mask(mask: np.ndarray, size: int, op: type) -> np.ndarray:
@@ -549,57 +674,115 @@ def _file_sha256(path: Path) -> str:
return h.hexdigest()
-class Rife:
- """A located RIFE as an `Interpolate` callable, and as a `Read` (`read`). Counts the frames it made (`made`) and the
- times it ran the binary (`calls`)."""
+class Binary:
+ """rife-ncnn-vulkan as the runner of a `Rife`: each run one call of the binary, its images written in a directory of
+ its own for the frame. What its frames depend on is read anew every time it is asked (`identity`): the binary runs
+ from its files on disk, and reads its model from them, every call."""
- def __init__(self, binary: Path | None = None, model: Path | None = None):
- if binary is None or model is None:
- binary, model = locate()
+ def __init__(self, binary: Path, model: Path):
self.binary, self.model = binary, model
+
+ def frame(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray]:
+ with tempfile.TemporaryDirectory(prefix="sprite-gen-rife-") as td:
+ return frame_runs(_binary_run(self.binary, self.model, Path(td)), colour, cover, t)
+
+ def read(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ with tempfile.TemporaryDirectory(prefix="sprite-gen-rife-") as td:
+ run = _binary_run(self.binary, self.model, Path(td))
+ return (*frame_runs(run, colour, cover, t), pass_runs(run, colour, cover, t))
+
+ def describe(self) -> dict[str, str]:
+ return {"kind": BINARY, "binary": str(self.binary), "model": self.model.name}
+
+ def identity(self) -> dict[str, object]:
+ """The binary's and each of the model's files by sha256, and the model's path as the binary is handed it (the
+ binary reads the model's version from its name), read now. Files that cannot be read are a RIFE that fails
+ (`RifeUnavailable`)."""
+ try:
+ model = sorted(p for p in self.model.iterdir() if p.is_file())
+ return {"backend": "ncnn", "binary": _file_sha256(self.binary), "model": str(self.model),
+ "model_files": {p.name: _file_sha256(p) for p in model}}
+ except OSError as exc:
+ raise RifeUnavailable(f"{self.binary.name}: the binary or model it runs from could not be read: {exc}") from exc
+
+
+def torch_runner(model: Path | None = None) -> object:
+ """The torch backend's runner (`rife_torch.TorchRunner`) on `device()`, from `model` (else `locate_model()`): the device
+ named is refused before anything is imported, PyTorch not installed is `RifeNotInstalled` with the extra that installs
+ it."""
+ named = device()
+ try:
+ rife_torch = importlib.import_module("sprite_gen.video.rife_torch")
+ except ImportError as exc:
+ raise RifeNotInstalled(f"SPRITE_GEN_RIFE_BACKEND=torch, and PyTorch cannot be imported ({exc}); install it and the "
+ f"model {MODEL} with `{install_command()}`; see docs/loop-repair.md") from exc
+ except OSError as exc: # installed, and its libraries do not load: a RIFE found that fails
+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_BACKEND=torch, and PyTorch failed to load: {exc}") from exc
+ return rife_torch.TorchRunner(model or locate_model(), named)
+
+
+class Rife:
+ """A located RIFE as an `Interpolate` callable, and as a `Read` (`read`). Counts the frames it made (`made`) and the
+ times it ran RIFE (`calls`): the binary's calls, or the torch backend's runs.
+
+ What runs it is its runner, which makes a frame whole from what `inputs` gives it (`frame`: two runs; `read`: two
+ runs and the sixteen passes) and names what its frames depend on (`identity`): `Binary` for the binary, or
+ `rife_torch.TorchRunner`. `Rife()` takes the one `backend()` names; `Rife(binary, model)` is the binary at those
+ paths; `Rife(runner=...)` any other."""
+
+ def __init__(self, binary: Path | None = None, model: Path | None = None, *, runner: object | None = None):
+ if runner is None:
+ if binary is not None and model is not None:
+ runner = Binary(binary, model)
+ elif backend() == "ncnn":
+ runner = Binary(*locate())
+ else:
+ runner = torch_runner()
+ self.runner = runner
+ self.binary, self.model = getattr(runner, "binary", None), getattr(runner, "model", None)
self.tta = tta() # refused here, before a frame is asked for
self.made = 0
self.calls = 0
def __call__(self, a: Image.Image, b: Image.Image, t: float) -> Image.Image:
- with tempfile.TemporaryDirectory(prefix="sprite-gen-rife-") as td:
- frame = between(a, b, t, binary=self.binary, model=self.model, tmp=Path(td))
+ frame = _joined(*self.runner.frame(*inputs(a, b), t)) # type: ignore[attr-defined]
self.made += 1
self.calls += 2
return frame
def read(self, a: Image.Image, b: Image.Image, t: float) -> tuple[Image.Image, np.ndarray]:
"""The frame at `t`, made as every call makes it (`between`), and how far the sixteen single passes disagree
- there (`passes`): 2 + PASS_CALLS calls of the binary."""
- with tempfile.TemporaryDirectory(prefix="sprite-gen-rife-") as td:
- frame = between(a, b, t, binary=self.binary, model=self.model, tmp=Path(td))
- apart = passes(a, b, t, binary=self.binary, model=self.model, tmp=Path(td))
+ there (`passes`): 2 + PASS_CALLS runs of RIFE."""
+ rgb, alpha, apart = self.runner.read(*inputs(a, b), t) # type: ignore[attr-defined]
self.made += 1
self.calls += 2 + PASS_CALLS
- return frame, apart
+ return _joined(rgb, alpha), apart
def describe(self) -> dict[str, str]:
- return {"binary": str(self.binary), "model": self.model.name, "tta": " ".join(self.tta)}
+ """What made the frames, for a report: its `kind` (`rife-ncnn-vulkan`, `rife-torch`), where it runs from, and the
+ augmentation."""
+ return {**self.runner.describe(), "tta": " ".join(self.tta)} # type: ignore[attr-defined]
+
+ def counts(self) -> dict[str, int]:
+ """The runs it made, as a report names them: the binary's calls (`binary_calls`), or the torch backend's runs
+ (`runs`)."""
+ return {"binary_calls" if isinstance(self.runner, Binary) else "runs": self.calls}
def identity(self) -> str:
"""What a frame this RIFE makes depends on besides its two frames and t, as one string (`Cache` keys
- on it), read as it is now on every call — nothing of it is kept, so a binary, model or machine changed
- under this very RIFE is another identity: the binary's and each of the model's files by sha256, and the
- model's path as the binary is handed it (the binary reads the model's version from its name); the
- augmentation every call asks for now (`tta()`: SPRITE_GEN_RIFE_TTA is read on every call, and another is
- another frame); the engine code this process loaded (`CODE_SHA256`: it turns a frame into RIFE's two runs and back) and NumPy's and
+ on it): its runner's (`identity()` of the runner — for the binary its files, read again on every call, so a
+ binary or model changed under this very RIFE is another identity; for the torch backend what it loaded and runs
+ on, the same for its life) and this host's, read as it is now on every call: the augmentation every call asks
+ for now (`tta()`: SPRITE_GEN_RIFE_TTA is read on every call, and another is another frame); the engine code this
+ process loaded (`CODE_SHA256`: it turns a frame into RIFE's two runs and back) and NumPy's and
Pillow's versions; and the machine — the OS, and `uname`'s host name, OS release and build and
architecture (platform.uname() keeps its first answer) — as the Apple GPU and llvmpipe are only measured
to agree to the third decimal of an error (docs/loop-repair.md section 1), so a frame made on one is not
- taken on another. Files that cannot be read are a RIFE that fails (`RifeUnavailable`)."""
- try:
- model = sorted(p for p in self.model.iterdir() if p.is_file())
- files = {"binary": _file_sha256(self.binary), "model": str(self.model), "model_files": {p.name: _file_sha256(p) for p in model}}
- except OSError as exc:
- raise RifeUnavailable(f"{self.binary.name}: the binary or model it runs from could not be read: {exc}") from exc
+ taken on another."""
return json.dumps({
- **files, "tta": list(tta()), "code": CODE_SHA256, "numpy": np.__version__, "pillow": PIL.__version__,
- "machine": [sys.platform, *(os.uname() if hasattr(os, "uname") else platform.uname()[:5])]}, sort_keys=True)
+ **self.runner.identity(), "tta": list(tta()), "code": CODE_SHA256, "numpy": np.__version__, # type: ignore[attr-defined]
+ "pillow": PIL.__version__, "machine": [sys.platform, *(os.uname() if hasattr(os, "uname") else platform.uname()[:5])]},
+ sort_keys=True)
def _kept(path: Path, size: tuple[int, int]) -> Image.Image | None:
diff --git a/sprite_gen/video/rife_install.py b/sprite_gen/video/rife_install.py
index 59c6ab6..c69c5be 100644
--- a/sprite_gen/video/rife_install.py
+++ b/sprite_gen/video/rife_install.py
@@ -9,9 +9,15 @@ directory — which `rife.locate()` searches after SPRITE_GEN_RIFE and PATH. `--
somewhere else and prints the SPRITE_GEN_RIFE line that points the engine at it.
Unpacked into a staging directory and renamed into place, so an interrupted install never leaves
-a half-written RIFE where the engine looks. Running it again over a finished install of the same
+a half-written RIFE where the engine looks; the model's two files are checked against their pinned
+SHA-256 (`rife.MODEL_SHA256`) before it is. Running it again over a finished install of the same
zip does nothing but the check. The check makes one frame between two discs through the
installed binary, so a machine that has the files but no working Vulkan finds out here.
+
+`--backend torch` (the default where SPRITE_GEN_RIFE_BACKEND names torch) installs the same zip for the
+torch backend, which reads the model only (`rife.locate_model()`): a Linux machine the binary does not run
+on (aarch64) is not refused, and the check frame is made through PyTorch on SPRITE_GEN_RIFE_DEVICE
+(`pip install "sprite-gen[torch]"` first; an image built without a GPU checks with SPRITE_GEN_RIFE_DEVICE=cpu).
"""
from __future__ import annotations
@@ -53,14 +59,14 @@ def zip_name(platform: str) -> str:
return f"{rife.BINARY}-{rife.RELEASE}-{platform}.zip"
-def _platform(platform: str | None) -> str:
+def _platform(platform: str | None, backend: str = "ncnn") -> str:
name = platform or rife.platform_release()
if name not in ZIP_SHA256:
raise SystemExit(f"rife install: rife-ncnn-vulkan {rife.RELEASE} ships no build for {sys.platform}; build it from "
f"github.com/nihui/rife-ncnn-vulkan and set SPRITE_GEN_RIFE to the binary (docs/loop-repair.md)")
machines = ZIP_MACHINES[name]
machine = platform_mod.machine().lower()
- if platform is None and machines is not None and machine not in machines:
+ if backend == "ncnn" and platform is None and machines is not None and machine not in machines:
raise SystemExit(f"rife install: the {name} build of rife-ncnn-vulkan {rife.RELEASE} is x86-64 only and this machine "
f"is {machine}; build it from github.com/nihui/rife-ncnn-vulkan and set SPRITE_GEN_RIFE to the binary")
return name
@@ -140,12 +146,15 @@ def _disc(cx: int) -> Image.Image:
return Image.fromarray(a, "RGBA")
-def check(binary: Path, model: Path) -> dict[str, Any]:
- """One frame half way between two discs through `binary`: the disc must land in the middle."""
+def check(binary: Path, model: Path, *, backend: str = "ncnn") -> dict[str, Any]:
+ """One frame half way between two discs through `binary`, or through the torch backend from `model`: the disc must
+ land in the middle."""
try:
- mid = rife.Rife(binary, model)(_disc(30), _disc(66), 0.5)
+ made = rife.Rife(binary, model) if backend == "ncnn" else rife.Rife(runner=rife.torch_runner(model))
+ mid = made(_disc(30), _disc(66), 0.5)
except rife.RifeUnavailable as exc:
- linux = " — on Linux without a GPU install `libvulkan1 mesa-vulkan-drivers`" if sys.platform.startswith("linux") else ""
+ linux = (" — on Linux without a GPU install `libvulkan1 mesa-vulkan-drivers`"
+ if backend == "ncnn" and sys.platform.startswith("linux") else "")
raise SystemExit(f"rife install: installed, but the check frame failed: {exc}{linux}") from exc
xs = np.nonzero(np.asarray(mid)[..., 3] > 127)[1]
centre = float(xs.mean()) if xs.size else None
@@ -155,14 +164,20 @@ def check(binary: Path, model: Path) -> dict[str, Any]:
def install(*, dest: Path | None = None, zip_path: Path | None = None, platform: str | None = None,
- force: bool = False, run_check: bool = True) -> dict[str, Any]:
- name = _platform(platform)
+ force: bool = False, run_check: bool = True, backend: str | None = None) -> dict[str, Any]:
+ try:
+ backend = backend or rife.backend()
+ except rife.RifeUnavailable as exc:
+ raise SystemExit(f"rife install: {exc}") from exc
+ if backend not in rife.BACKENDS:
+ raise SystemExit(f"rife install: backend {backend!r}: name ncnn or torch")
+ name = _platform(platform, backend)
sha = ZIP_SHA256[name]
root = (dest or rife.install_root()).expanduser().resolve()
binary = rife.installed_binary(root, name)
target = binary.parent
- report: dict[str, Any] = {"release": rife.RELEASE, "model": rife.MODEL, "platform": name, "zip": zip_name(name),
- "sha256": sha, "dir": str(target), "binary": str(binary)}
+ report: dict[str, Any] = {"backend": backend, "release": rife.RELEASE, "model": rife.MODEL, "platform": name,
+ "zip": zip_name(name), "sha256": sha, "dir": str(target), "binary": str(binary)}
if not force and _complete(target, name, sha):
report["installed"] = "already"
else:
@@ -182,6 +197,10 @@ def install(*, dest: Path | None = None, zip_path: Path | None = None, platform:
raise SystemExit(f"rife install: {zip_path.name} has SHA-256 {got}, expected {sha}; nothing was installed")
staging = work / target.name
report["files"] = _unpack(zip_path, staging, name)
+ try:
+ rife.model_bytes(staging / rife.MODEL)
+ except rife.RifeUnavailable as exc:
+ raise SystemExit(f"rife install: {exc}; nothing was installed") from exc
(staging / MARKER).write_text(json.dumps({"release": rife.RELEASE, "zip": zip_name(name), "sha256": sha}) + "\n",
encoding="utf-8")
if target.exists():
@@ -189,7 +208,18 @@ def install(*, dest: Path | None = None, zip_path: Path | None = None, platform:
staging.rename(target)
report["installed"] = "now"
if run_check:
- report["check"] = check(binary, target / rife.MODEL)
+ report["check"] = check(binary, target / rife.MODEL, backend=backend)
+ if backend == "torch":
+ # What the torch backend will read: SPRITE_GEN_RIFE_MODEL before the install root.
+ try:
+ found = str(rife.locate_model())
+ except rife.RifeUnavailable:
+ found = None
+ report["engine_finds"] = found
+ if found != str((target / rife.MODEL).resolve()):
+ report["note"] = (f"the torch backend reads SPRITE_GEN_RIFE_MODEL before {rife.install_root()}; to read this "
+ f"install set SPRITE_GEN_RIFE_MODEL={target / rife.MODEL}" + (f" (it now reads {found})" if found else ""))
+ return report
# What the engine will run: SPRITE_GEN_RIFE and PATH are searched before the install root.
try:
found = str(rife.locate()[0])
@@ -211,13 +241,16 @@ def add_arguments(parser: argparse.ArgumentParser) -> None:
"its SHA-256 is checked all the same")
p.add_argument("--force", action="store_true", help="unpack again over a finished install")
p.add_argument("--no-check", action="store_true", help="skip the check frame made through the installed binary")
+ p.add_argument("--backend", choices=rife.BACKENDS, help="what will run RIFE: ncnn (the binary) or torch (the model only, "
+ "its check frame through PyTorch on SPRITE_GEN_RIFE_DEVICE); "
+ "default SPRITE_GEN_RIFE_BACKEND, else ncnn")
def run(**kwargs: object) -> int:
if kwargs.get("action") != "install":
raise SystemExit(f"rife: unknown action {kwargs.get('action')!r}; expected install")
report = install(dest=kwargs.get("dir"), zip_path=kwargs.get("zip"), force=bool(kwargs.get("force")), # type: ignore[arg-type]
- run_check=not kwargs.get("no_check"))
+ run_check=not kwargs.get("no_check"), backend=kwargs.get("backend")) # type: ignore[arg-type]
if report.get("note"):
print(f"rife install: note: {report['note']}", file=sys.stderr)
print(json.dumps({k: v for k, v in report.items() if k != "files"}, ensure_ascii=False, indent=2))
diff --git a/sprite_gen/video/rife_torch.py b/sprite_gen/video/rife_torch.py
new file mode 100644
index 0000000..d67ed75
--- /dev/null
+++ b/sprite_gen/video/rife_torch.py
@@ -0,0 +1,358 @@
+# SPDX-License-Identifier: Apache-2.0
+# The IFNet below follows Practical-RIFE's IFNet_HDv3 and warplayer (MIT, Copyright (c) 2021 hzwer; NOTICE).
+# The augmentation loop follows rife-ncnn-vulkan 20221029's RIFE::process_v4 (MIT, Copyright (c) 2020 nihui; NOTICE).
+"""rife-v4.6 in PyTorch: the torch backend of `rife.Rife` (SPRITE_GEN_RIFE_BACKEND=torch, docs/loop-repair.md section 1).
+
+The one module of the engine that imports torch, and it is imported only where the torch backend is chosen
+(`rife.Rife()`), so the engine runs without PyTorch and is refused for it only there (`rife.RifeNotInstalled`).
+
+The network is rife-v4.6's IFNet (Practical-RIFE f6b5132, `train_log/IFNet_HDv3.py`, `model/warplayer.py`) written
+again here with the same parameter names — four IFBlocks of two strided 3x3 convolutions, eight residual convolutions and
+a transposed convolution, the warp a bilinear `grid_sample` at the border — rather than imported: the official module
+makes its warp grid on a device it picks itself at import.
+
+Its weights are the binary's own model (`rife.MODEL_SHA256`), read without ncnn (`state_from_ncnn`): the 44
+convolutions of `flownet.param` in file order, their float16 weights widened to float32 and their float32 biases; a
+residual convolution's scale (`beta`) is folded into its convolution there, so it is 1 here; a transposed convolution's
+weights are stored (out, in, ...) and read (in, out, ...). The same weights as the binary, as rounded for it.
+
+What the binary's augmentation does inside one call is done here the same way (rife-ncnn-vulkan 20221029
+`RIFE::process_v4`, `-x` and `-z`): the pair, padded with zeros to a multiple of 32, is turned its eight ways (`_geo`: 0
+as is, 1 left-right, 2 both, 3 up-down, 4 transposed, 5-7 transposed then 1-3) and run both ways in time (b to a at
+1 - t); at each of the four levels every way's flows and mask are turned back to the pair's own frame (`_to_canon`; the
+backward ones swapped, `_swap`), averaged, and handed back to every way (`_from_canon`) before the next level; the eight
+ways' frames, each the mean of its forward and backward one, are turned back and averaged; the frame is rounded as the
+binary rounds it, floor(v * 255 + 0.5). From the upload to the uint8 frame everything runs on the device.
+
+On CUDA every frame runs with cuDNN deterministic, its benchmark off and its convolutions in IEEE float32 (no TF32,
+which PyTorch turns on by default): the same frame twice is the same bytes on one GPU, driver and PyTorch, and what
+they are is in the runner's identity. On the CPU it is the same bytes whatever the thread count.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import math
+import platform
+import re
+import shutil
+import subprocess
+from pathlib import Path
+
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+
+from sprite_gen._deps import np
+from sprite_gen.video import rife
+
+# The network code this process loaded (the runner's identity, `net`), read at import as rife.CODE_SHA256 is.
+CODE_SHA256 = hashlib.sha256(Path(__file__).read_bytes()).hexdigest()
+SCALES = (8, 4, 2, 1)
+BLOCKS = ((7, 192), (12, 128), (12, 96), (12, 64)) # (input planes, width) of each IFBlock
+# The official state dict's convolutions in the order the ncnn model holds them: each block's two strided ones, its
+# eight residual ones and its transposed one.
+LAYERS = tuple(f"block{i}.{name}" for i in range(len(BLOCKS))
+ for name in ("conv0.0.0", "conv0.1.0", *(f"convblock.{j}.conv" for j in range(8)), "lastconv.0"))
+NCNN_FLOAT16 = 0x01306B47 # ncnn's tag before a float16 weight blob; a float32 one is tagged 0
+NVIDIA_VERSION = Path("/proc/driver/nvidia/version")
+
+
+def _conv(cin: int, cout: int) -> nn.Sequential:
+ return nn.Sequential(nn.Conv2d(cin, cout, 3, 2, 1, bias=True), nn.LeakyReLU(0.2, True))
+
+
+class ResConv(nn.Module):
+ def __init__(self, c: int):
+ super().__init__()
+ self.conv = nn.Conv2d(c, c, 3, 1, 1)
+ self.beta = nn.Parameter(torch.ones((1, c, 1, 1)))
+ self.relu = nn.LeakyReLU(0.2, True)
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ return self.relu(self.conv(x) * self.beta + x)
+
+
+class IFBlock(nn.Module):
+ def __init__(self, in_planes: int, c: int):
+ super().__init__()
+ self.conv0 = nn.Sequential(_conv(in_planes, c // 2), _conv(c // 2, c))
+ self.convblock = nn.Sequential(*(ResConv(c) for _ in range(8)))
+ self.lastconv = nn.Sequential(nn.ConvTranspose2d(c, 4 * 6, 4, 2, 1), nn.PixelShuffle(2))
+
+ def raw(self, x: torch.Tensor, flow: torch.Tensor | None, scale: int) -> torch.Tensor:
+ """The official IFBlock.forward up to its output at the block's own scale, before it is scaled back up."""
+ x = F.interpolate(x, scale_factor=1.0 / scale, mode="bilinear", align_corners=False)
+ if flow is not None:
+ flow = F.interpolate(flow, scale_factor=1.0 / scale, mode="bilinear", align_corners=False) * 1.0 / scale
+ x = torch.cat((x, flow), 1)
+ return self.lastconv(self.convblock(self.conv0(x)))
+
+
+class IFNet(nn.Module):
+ def __init__(self):
+ super().__init__()
+ for i, (cin, c) in enumerate(BLOCKS):
+ setattr(self, f"block{i}", IFBlock(cin, c))
+
+
+def _ncnn_layers(param: bytes, weights: bytes) -> list[tuple[str, np.ndarray, np.ndarray]]:
+ """The convolutions of an ncnn model in file order: (layer type, weights, biases) as float32."""
+ layers, pos = [], 0
+ for line in param.decode("utf-8").split("\n")[2:]:
+ parts = line.split()
+ if len(parts) < 4 or parts[0] not in ("Convolution", "Deconvolution"):
+ continue
+ n_in, n_out = int(parts[2]), int(parts[3])
+ kv = dict(p.split("=", 1) for p in parts[4 + n_in + n_out:])
+ num_output, size = int(kv["0"]), int(kv["6"])
+ tag = int.from_bytes(weights[pos:pos + 4], "little")
+ pos += 4
+ if tag == NCNN_FLOAT16:
+ w = np.frombuffer(weights, np.float16, size, pos).astype(np.float32)
+ pos += (size * 2 + 3) // 4 * 4
+ elif tag == 0:
+ w = np.frombuffer(weights, np.float32, size, pos).copy()
+ pos += size * 4
+ else:
+ raise ValueError(f"{parts[1]}: weight tag {tag:#x}")
+ if int(kv.get("5", 0)) != 1:
+ raise ValueError(f"{parts[1]}: no bias")
+ b = np.frombuffer(weights, np.float32, num_output, pos).copy()
+ pos += num_output * 4
+ layers.append((parts[0], w, b))
+ if pos != len(weights):
+ raise ValueError(f"read {pos} of the model's {len(weights)} bytes")
+ return layers
+
+
+def state_from_ncnn(files: dict[str, bytes]) -> dict[str, torch.Tensor]:
+ """The official state dict of IFNet, every name and shape, filled from the ncnn model's files (`rife.model_bytes`)."""
+ shapes = {k: v.shape for k, v in IFNet().state_dict().items()}
+ layers = _ncnn_layers(files["flownet.param"], files["flownet.bin"])
+ if len(layers) != len(LAYERS):
+ raise ValueError(f"{len(layers)} convolutions in the model, {len(LAYERS)} in the network")
+ state = {}
+ for name, (kind, w, b) in zip(LAYERS, layers):
+ shape = shapes[name + ".weight"]
+ if kind == "Deconvolution":
+ weight = torch.from_numpy(w).reshape(shape[1], shape[0], *shape[2:]).transpose(0, 1).contiguous()
+ else:
+ weight = torch.from_numpy(w).reshape(shape)
+ state[name + ".weight"], state[name + ".bias"] = weight, torch.from_numpy(b).reshape(shapes[name + ".bias"])
+ if ".convblock." in name:
+ beta = name.removesuffix(".conv") + ".beta"
+ state[beta] = torch.ones(shapes[beta])
+ return state
+
+
+def _geo(x: torch.Tensor, g: int) -> torch.Tensor:
+ """An (N, C, H, W) tensor turned the binary's way `g` (rife_preproc_tta)."""
+ if g >= 4:
+ x = x.transpose(2, 3)
+ f = g % 4
+ return x.flip(3) if f == 1 else x.flip(2, 3) if f == 2 else x.flip(2) if f == 3 else x
+
+
+def _geo_back(x: torch.Tensor, g: int) -> torch.Tensor:
+ """`_geo` undone."""
+ f = g % 4
+ x = x.flip(3) if f == 1 else x.flip(2, 3) if f == 2 else x.flip(2) if f == 3 else x
+ return x.transpose(2, 3) if g >= 4 else x
+
+
+def _signs(g: int) -> tuple[int, int]:
+ """The signs way `g` puts on a flow's x and y: a flip left-right turns x, up-down turns y."""
+ f = g % 4
+ return (-1 if f in (1, 2) else 1), (-1 if f in (2, 3) else 1)
+
+
+def _to_canon(t: torch.Tensor, g: int) -> torch.Tensor:
+ """A way's flows and mask (N, 5, h, w: flow 0 x y, flow 1 x y, mask) in the pair's own frame (rife_v4_flow_tta_avg)."""
+ t = _geo_back(t, g)
+ sx, sy = _signs(g)
+ x0, y0, x1, y1, m = t.unbind(1)
+ v = (sx * x0, sy * y0, sx * x1, sy * y1, m) if g < 4 else (sy * y0, sx * x0, sy * y1, sx * x1, m) # transposed: x is y
+ return torch.stack(v, 1)
+
+
+def _from_canon(c: torch.Tensor, g: int) -> torch.Tensor:
+ """`_to_canon` undone: flows and mask in the pair's own frame, as way `g` sees them."""
+ sx, sy = _signs(g)
+ dx0, dy0, dx1, dy1, m = c.unbind(1)
+ v = (sx * dx0, sy * dy0, sx * dx1, sy * dy1, m) if g < 4 else (sx * dy0, sy * dx0, sx * dy1, sy * dx1, m)
+ return _geo(torch.stack(v, 1), g)
+
+
+def _swap(c: torch.Tensor) -> torch.Tensor:
+ """A backward run's flows and mask as the forward run's (rife_v4_flow_tta_temporal_avg): flows swapped, mask negated."""
+ return torch.cat([c[:, 2:4], c[:, 0:2], -c[:, 4:5]], 1)
+
+
+def _driver() -> str:
+ """The NVIDIA driver's version: from /proc/driver/nvidia/version, else nvidia-smi's; '' where neither reads."""
+ try:
+ found = re.search(r"Kernel Module(?: for \S+)?\s+(\d+(?:\.\d+)+)", NVIDIA_VERSION.read_text().splitlines()[0])
+ if found:
+ return found.group(1)
+ except (OSError, IndexError):
+ pass
+ smi = shutil.which("nvidia-smi")
+ if smi:
+ try:
+ out = subprocess.run([smi, "--query-gpu=driver_version", "--format=csv,noheader"], capture_output=True,
+ text=True, timeout=30)
+ if out.returncode == 0 and out.stdout.strip():
+ return out.stdout.strip().splitlines()[0].strip()
+ except (OSError, subprocess.TimeoutExpired):
+ pass
+ return ""
+
+
+def _deterministic() -> dict[str, object]:
+ """cuDNN set to make the same frame twice the same bytes, through PyTorch's newer precision setting only (the two
+ kinds of setting do not mix), and read back as it now stands."""
+ cudnn = torch.backends.cudnn
+ cudnn.deterministic, cudnn.benchmark = True, False
+ cudnn.conv.fp32_precision = "ieee"
+ return {"cudnn_deterministic": cudnn.deterministic, "cudnn_benchmark": cudnn.benchmark,
+ "cudnn_conv_fp32_precision": cudnn.conv.fp32_precision}
+
+
+class TorchRunner:
+ """rife-v4.6 on a torch device, as the runner of a `rife.Rife`: the model in `model` (`rife.model_bytes`, checked
+ before it is read) loaded once, a frame its two runs and a read its sixteen passes as well (`rife.frame_runs`,
+ `rife.pass_runs`). What its frames depend on is read once, as it loaded and set them, and stays for its life
+ (`identity`): the weights' bytes, this network's code, PyTorch's version, and on CUDA its CUDA and cuDNN, the GPU,
+ the driver and the settings that make a frame the same bytes; on the CPU the instructions PyTorch runs on.
+ `cuda` that is not there is refused, by why (`rife.RifeUnavailable`)."""
+
+ def __init__(self, model: Path, device: str):
+ if device not in rife.DEVICES:
+ raise rife.RifeUnavailable(f"rife-torch: device {device!r}: name cuda or cpu")
+ if device == "cuda" and not torch.cuda.is_available():
+ wheel = ("a PyTorch built for the CPU only (torch.version.cuda is None)" if torch.version.cuda is None else
+ f"PyTorch {torch.__version__} built for CUDA {torch.version.cuda}, which finds no GPU it can use")
+ raise rife.RifeUnavailable(f"SPRITE_GEN_RIFE_DEVICE=cuda, and CUDA is not available here: {wheel}; nothing runs "
+ "on another device in its place — name SPRITE_GEN_RIFE_DEVICE=cpu to run on the CPU")
+ files = rife.model_bytes(model)
+ net = IFNet()
+ try:
+ net.load_state_dict(state_from_ncnn(files), strict=True)
+ except (ValueError, RuntimeError) as exc:
+ raise rife.RifeUnavailable(f"rife-torch: the model {model} does not load: {exc}") from exc
+ self.model, self.device = model, device
+ self.net = net.to(device).eval()
+ self.blocks = tuple(getattr(self.net, f"block{i}") for i in range(len(BLOCKS)))
+ self._grids: dict[tuple, torch.Tensor] = {}
+ ident: dict[str, object] = {"backend": "torch", "device": device, "net": CODE_SHA256, "torch": torch.__version__,
+ "weights": {name: hashlib.sha256(data).hexdigest() for name, data in files.items()}}
+ if device == "cuda":
+ ident.update(cuda=torch.version.cuda, cudnn=torch.backends.cudnn.version(), gpu=torch.cuda.get_device_name(),
+ capability=list(torch.cuda.get_device_capability()), driver=_driver(), determinism=_deterministic())
+ else: # a frame made on the CPU is made by its instructions; one made on a GPU is not, and its host's CPU is no part of it
+ ident.update(cpu=torch.backends.cpu.get_cpu_capability(), arch=platform.machine())
+ self._identity = ident
+
+ def identity(self) -> dict[str, object]:
+ return dict(self._identity)
+
+ def describe(self) -> dict[str, str]:
+ return {"kind": "rife-torch", "device": self.device, "model": self.model.name}
+
+ def frame(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray]:
+ self._settle()
+ return rife.frame_runs(self.run, colour, cover, t)
+
+ def read(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray, np.ndarray]:
+ self._settle()
+ return (*rife.frame_runs(self.run, colour, cover, t), rife.pass_runs(self.run, colour, cover, t))
+
+ def _settle(self) -> None:
+ """The settings a frame on CUDA is made under, set again for each (other code in the process may set them)."""
+ if self.device == "cuda":
+ _deterministic()
+
+ def run(self, x: np.ndarray, y: np.ndarray, t: float, modes: tuple[str, ...] | None) -> np.ndarray:
+ """One run, as one call of the binary makes it (`rife.Run`): `t` as the binary is handed it (`-s`, four
+ decimals), the augmentation `modes` (None: `rife.tta()`). A run that fails on the device (out of its memory) is
+ a RIFE that fails (`rife.RifeUnavailable`)."""
+ modes = rife.tta() if modes is None else modes
+ try:
+ return self._run(x, y, float(f"{t:.4f}"), tuple(modes))
+ except RuntimeError as exc:
+ raise rife.RifeUnavailable(f"rife-torch on {self.device} failed: {exc}") from exc
+
+ @torch.inference_mode()
+ def _run(self, x: np.ndarray, y: np.ndarray, t: float, modes: tuple[str, ...]) -> np.ndarray:
+ h, w = x.shape[:2]
+ pads = [self._pad(im) for im in (x, y)]
+ ways = [(g, d) for g in (range(8) if "-x" in modes else (0,)) for d in ((0, 1) if "-z" in modes else (0,))]
+ frames, _ = self._infer(pads, t, ways)
+ v = torch.stack(frames).mean(0)[0, :, :h, :w]
+ v = torch.clamp(torch.floor(v * 255.0 + 0.5), 0, 255).to(torch.uint8)
+ return v.permute(1, 2, 0).contiguous().cpu().numpy()
+
+ def _pad(self, im: np.ndarray) -> torch.Tensor:
+ """An H x W x 3 uint8 image on the device, 0..1, padded with zeros at the bottom and right to a multiple of 32."""
+ h, w = im.shape[:2]
+ x = torch.zeros((1, 3, math.ceil(h / 32) * 32, math.ceil(w / 32) * 32), dtype=torch.float32, device=self.device)
+ x[0, :, :h, :w] = torch.from_numpy(np.ascontiguousarray(im.transpose(2, 0, 1))).to(self.device).float() / 255.0
+ return x
+
+ def _warp(self, img: torch.Tensor, flow: torch.Tensor) -> torch.Tensor:
+ """The official warp: `img` sampled where `flow` points (bilinear, the border repeated), on the flow's device."""
+ k = (str(flow.device), flow.dtype, tuple(flow.shape))
+ if k not in self._grids:
+ n, _, h, w = flow.shape
+ gx = torch.linspace(-1.0, 1.0, w, device=flow.device, dtype=flow.dtype).view(1, 1, 1, w).expand(n, -1, h, -1)
+ gy = torch.linspace(-1.0, 1.0, h, device=flow.device, dtype=flow.dtype).view(1, 1, h, 1).expand(n, -1, -1, w)
+ self._grids[k] = torch.cat([gx, gy], 1)
+ flow = torch.cat([flow[:, 0:1] / ((img.shape[3] - 1.0) / 2.0), flow[:, 1:2] / ((img.shape[2] - 1.0) / 2.0)], 1)
+ return F.grid_sample(img, (self._grids[k] + flow).permute(0, 2, 3, 1), mode="bilinear", padding_mode="border",
+ align_corners=True)
+
+ def _infer(self, pads: list[torch.Tensor], t: float, ways: list[tuple[int, int]], *,
+ flows: bool = False) -> tuple[list[torch.Tensor], list[torch.Tensor] | None]:
+ """The four levels over `ways` — (g, d): turned way g, forward (d 0: the first to the second at t) or backward
+ (d 1: the second to the first at 1 - t) — of a padded pair, their flows averaged at every level where there is
+ more than one: each way's frame turned back (1 x 3 x H x W) and, with `flows`, its last flows and mask in the
+ pair's own frame as the forward run sees them (1 x 5 x H x W)."""
+ groups = []
+ for transposed in (False, True): # the ways a transpose turns are another shape: a batch of their own
+ items = [(g, d) for g, d in ways if (g >= 4) == transposed]
+ if items:
+ img0 = torch.cat([_geo(pads[d], g) for g, d in items])
+ img1 = torch.cat([_geo(pads[1 - d], g) for g, d in items])
+ ts = torch.cat([torch.full_like(img0[k:k + 1, :1], t if d == 0 else 1.0 - t) for k, (g, d) in enumerate(items)])
+ groups.append({"items": items, "img0": img0, "img1": img1, "ts": ts, "flow": None, "mask": None})
+ many = len(ways) > 1
+ for block, scale in zip(self.blocks, SCALES):
+ raws = []
+ for gr in groups:
+ if gr["flow"] is None:
+ raws.append(block.raw(torch.cat((gr["img0"], gr["img1"], gr["ts"]), 1), None, scale)[:, :5].float())
+ else:
+ x = torch.cat((gr["w0"], gr["w1"], gr["ts"], gr["mask"]), 1)
+ raws.append(block.raw(x, gr["flow"], scale)[:, :5].float())
+ if many:
+ canon = [(_swap if d else (lambda c: c))(_to_canon(raw[k:k + 1], g))
+ for gr, raw in zip(groups, raws) for k, (g, d) in enumerate(gr["items"])]
+ avg = torch.stack(canon).mean(0)
+ raws = [torch.cat([_from_canon(_swap(avg) if d else avg, g) for g, d in gr["items"]]) for gr in groups]
+ for gr, raw in zip(groups, raws):
+ up = F.interpolate(raw, scale_factor=scale, mode="bilinear", align_corners=False)
+ f0, m0 = up[:, :4] * scale, up[:, 4:5]
+ gr["flow"] = f0 if gr["flow"] is None else gr["flow"] + f0
+ gr["mask"] = m0 if gr["mask"] is None else gr["mask"] + m0
+ gr["w0"] = self._warp(gr["img0"], gr["flow"][:, :2])
+ gr["w1"] = self._warp(gr["img1"], gr["flow"][:, 2:4])
+ frames = []
+ for gr in groups:
+ m = torch.sigmoid(gr["mask"])
+ merged = (gr["w0"] * m + gr["w1"] * (1 - m)).float()
+ frames += [_geo_back(merged[k:k + 1], g) for k, (g, d) in enumerate(gr["items"])]
+ if not flows:
+ return frames, None
+ return frames, [(_swap if d else (lambda c: c))(_to_canon(torch.cat([gr["flow"][k:k + 1], gr["mask"][k:k + 1]], 1), g))
+ for gr in groups for k, (g, d) in enumerate(gr["items"])]
diff --git a/tests/conftest.py b/tests/conftest.py
index ad35071..0347d10 100644
--- a/tests/conftest.py
+++ b/tests/conftest.py
@@ -61,7 +61,7 @@ def symlink_or_skip(link: Path, target: Path) -> None:
# Tests that skip where no RIFE is reachable for a reason other than RIFE: an input the lane does not
# give. Named without their parameters. In the lane's RIFE-less run (`pytest --rife-unmeasured`) these
-# and the tests marked real_rife may skip; any other skip fails (docs/release.md).
+# and the tests marked real_rife or torch_rife may skip; any other skip fails (docs/release.md).
NOT_RIFE_SKIPS = {
# SPRITE_GEN_CHROMA_REAL_STILLS names model stills, which the repository does not hold.
"tests/frames/test_chroma_key_relative.py::test_real_still_keys_out_its_painted_border",
@@ -73,38 +73,64 @@ def pytest_addoption(parser: pytest.Parser) -> None:
"--rife-unmeasured",
action="store_true",
help="the RIFE-less run of scripts/linux_lane.sh: a skip fails unless its test is marked real_rife "
- "or named in NOT_RIFE_SKIPS (tests/conftest.py)",
+ "or torch_rife, or named in NOT_RIFE_SKIPS (tests/conftest.py)",
)
+def _torch_rife_missing() -> str | None:
+ """Why the torch backend cannot run here, None where it can: PyTorch, the rife-v4.6 model it reads
+ (`rife.locate_model()`) and the device SPRITE_GEN_RIFE_DEVICE names."""
+ from sprite_gen.video import rife
+
+ try:
+ import torch
+ except ImportError as exc:
+ return f'PyTorch not installed (pip install "sprite-gen[torch]"): {exc}'
+ try:
+ rife.locate_model()
+ named = rife.device()
+ except rife.RifeUnavailable as exc:
+ return f"RIFE's torch backend cannot run here: {exc}"
+ if named == "cuda" and not torch.cuda.is_available():
+ return "SPRITE_GEN_RIFE_DEVICE is cuda (the default) and PyTorch finds no CUDA here; name SPRITE_GEN_RIFE_DEVICE=cpu"
+ return None
+
+
def pytest_collection_modifyitems(items: list[pytest.Item]) -> None:
"""A test marked `real_rife` runs the real binary: skipped where `rife.locate()` finds none, and run
- by `scripts/linux_lane.sh --rife-only` (`-m real_rife`)."""
+ by `scripts/linux_lane.sh --rife-only` (`-m real_rife`). A test marked `torch_rife` runs RIFE's torch
+ backend: skipped where PyTorch, the model or the device is missing (`_torch_rife_missing`), and run by
+ the lanes with none skipped (`-m torch_rife`, docs/release.md)."""
marked = [item for item in items if item.get_closest_marker("real_rife")]
- if not marked:
- return
- from sprite_gen.video import rife
+ torched = [item for item in items if item.get_closest_marker("torch_rife")]
+ if marked:
+ from sprite_gen.video import rife
- try:
- rife.locate()
- except rife.RifeUnavailable:
- for item in marked:
- item.add_marker(pytest.mark.skip(reason="rife-ncnn-vulkan not installed (SPRITE_GEN_RIFE / PATH / sprite-gen rife install)"))
+ try:
+ rife.locate()
+ except rife.RifeUnavailable:
+ for item in marked:
+ item.add_marker(pytest.mark.skip(reason="rife-ncnn-vulkan not installed (SPRITE_GEN_RIFE / PATH / sprite-gen rife install)"))
+ why = _torch_rife_missing() if torched else None
+ if why:
+ for item in torched:
+ item.add_marker(pytest.mark.skip(reason=why))
def _fail_unmeasured_skip(report: pytest.TestReport | pytest.CollectReport, node: pytest.Item | pytest.Collector) -> None:
"""In the lane's RIFE-less run a skip that `--rife-only` will not run fails, by name: neither run would
- measure it. `--rife-only` runs the tests marked real_rife; a skip of any other test, or of a whole
- file, stands only where NOT_RIFE_SKIPS names it. Whatever skipped it: a skip mark, a fixture, the
+ measure it. `--rife-only` runs the tests marked real_rife or torch_rife; a skip of any other test, or of
+ a whole file, stands only where NOT_RIFE_SKIPS names it. Whatever skipped it: a skip mark, a fixture, the
test body, the file at import."""
if not report.skipped or hasattr(report, "wasxfail") or not node.config.getoption("rife_unmeasured", False):
return
- if node.get_closest_marker("real_rife") or report.nodeid.partition("[")[0] in NOT_RIFE_SKIPS:
+ if (node.get_closest_marker("real_rife") or node.get_closest_marker("torch_rife")
+ or report.nodeid.partition("[")[0] in NOT_RIFE_SKIPS):
return
path, line, reason = report.longrepr
report.outcome = "failed"
report.longrepr = (
- "skipped where no RIFE is reachable, but not marked real_rife: `scripts/linux_lane.sh --rife-only` "
+ "skipped where no RIFE is reachable, but not marked real_rife or torch_rife: `scripts/linux_lane.sh --rife-only` "
f"runs only the marked tests, so neither run measures it\n{path}:{line}: {reason}\n"
"A test that needs the real rife-ncnn-vulkan is marked `real_rife`; one that skips for another "
"reason is named in NOT_RIFE_SKIPS (tests/conftest.py)."
@@ -120,7 +146,6 @@ def _engine_tta(monkeypatch: pytest.MonkeyPatch) -> None:
monkeypatch.delenv("SPRITE_GEN_RIFE_BACKEND", raising=False)
-
@pytest.hookimpl(wrapper=True)
def pytest_runtest_makereport(item: pytest.Item) -> Generator[None, pytest.TestReport, pytest.TestReport]:
report = yield
diff --git a/tests/release/test_lane_rife_collection.py b/tests/release/test_lane_rife_collection.py
index 434e337..a3a4a67 100644
--- a/tests/release/test_lane_rife_collection.py
+++ b/tests/release/test_lane_rife_collection.py
@@ -1,11 +1,12 @@
# SPDX-License-Identifier: Apache-2.0
"""`scripts/linux_lane.sh --rife-unmeasured` runs the suite where no RIFE is reachable, and the tests
-that need one skip; `--rife-only` runs the tests marked `real_rife` (`pytest -m real_rife`), none
-skipped. A test that skips without RIFE and is not marked is measured by neither run.
+that need one skip; `--rife-only` runs the tests marked `real_rife` or `torch_rife` (`pytest -m real_rife`,
+`-m torch_rife`), none skipped. A test that skips without RIFE and is not marked is measured by neither run.
Its collection: the suite is collected twice, with nothing else changed, once with a stand-in RIFE
-everywhere `rife.locate()` looks (SPRITE_GEN_RIFE, PATH, the install root) and once with none. Every
-test the two collections skip, or collect, differently is decided by RIFE, and must carry the mark.
+everywhere `rife.locate()` and `rife.locate_model()` look (SPRITE_GEN_RIFE, PATH, the install root) and
+once with none. Every test the two collections skip, or collect, differently is decided by RIFE, and must
+carry one of the marks.
Its run: a fixture or the test body can skip it, which no collection reads, so the RIFE-less run itself
(`pytest --rife-unmeasured`, tests/conftest.py) fails each skip of a test that is not marked, unless it
@@ -26,7 +27,8 @@ from sprite_gen.video import rife
ROOT = Path(__file__).resolve().parents[2]
# Loaded into each collection (-p): per collected test, whether its skip and skipif marks skip it —
-# pytest's own reading, the mark tests/conftest.py adds included — and whether it is marked real_rife.
+# pytest's own reading, the mark tests/conftest.py adds included — and whether it is marked real_rife or
+# torch_rife.
PROBE = '''
import json, os
from _pytest.skipping import evaluate_skip_marks
@@ -34,7 +36,8 @@ from _pytest.skipping import evaluate_skip_marks
def pytest_collection_finish(session):
with open(os.environ["RIFE_PROBE_OUT"], "w") as out:
- json.dump({item.nodeid: [evaluate_skip_marks(item) is not None, item.get_closest_marker("real_rife") is not None]
+ json.dump({item.nodeid: [evaluate_skip_marks(item) is not None,
+ any(item.get_closest_marker(m) is not None for m in ("real_rife", "torch_rife"))]
for item in session.items}, out)
'''
@@ -104,7 +107,7 @@ def test_every_test_rife_decides_is_marked_real_rife(tmp_path):
decided = (present.keys() ^ absent.keys()) | {n for n in present.keys() & absent.keys() if present[n][0] != absent[n][0]}
unmarked = sorted(n for n in decided if not (present.get(n) or absent[n])[1])
assert not unmarked, (
- "skipped or collected by whether RIFE is reachable, but not marked real_rife — "
+ "skipped or collected by whether RIFE is reachable, but not marked real_rife or torch_rife — "
"`scripts/linux_lane.sh --rife-only` would not run them:\n" + "\n".join(unmarked))
assert decided, "no test is decided by RIFE: the stand-in or the conftest skip no longer reaches the marked tests"