Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
docs(rife): the torch backend on an NVIDIA T4, in its process and from a host without PyTorch, on the synthetic 960 walkers
Soohong Kime44ed7e05bae
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex a4e0e77..bcbbe99 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -6,8 +6,8 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the**Upgrading from 2.45.2: Defaults stay unchanged, cache keys update once, reports show the backend, install checks model hashes, and schemas are unchanged.** With `SPRITE_GEN_RIFE_BACKEND` unset (the default), RIFE is the `rife-ncnn-vulkan` binary as before. Its frames are the same bytes as 2.45.2. Measured: the binary's frames and sixteen-pass readings through the changed engine equal the old engine's, byte for byte. A `--between-cache` directory kept by 2.45.2 is made again once. The key now names the backend (`backend: "ncnn"`). The engine's `rife.py` changed, and its sha256 is part of the key anyway. Reports take `interpolator.kind` from what made the frames: `rife-ncnn-vulkan` as before for the binary, or `rife-torch` for the new backend. The overlapped wrap's record counts `binary_calls` for the binary as before, or `runs` for the torch backend. `SPRITE_GEN_RIFE_EXECUTOR` (new) is unset by default, and RIFE runs in this process. `sprite-gen rife install` now checks the model's two files (`rife-v4.6/flownet.param`, `flownet.bin`) against pinned SHA-256 after unpacking, for either backend. A model that differs installs nothing. The line a missing RIFE is reported with (`install` in records, warnings) is `sprite-gen rife install` for the binary. Where the torch backend is chosen, that line is: `pip install "sprite-gen[torch]" && sprite-gen rife install --backend torch`. No command-line flags of `video-loop` / `video-cycle-align` / `video-set` change. No restoration schemas change: `source-restoration-v5`, repair report `schema_version` 4, comparison `schema_version` 6.-- **New: RIFE's torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan.** `SPRITE_GEN_RIFE_BACKEND=torch` runs the same rife-v4.6 in PyTorch (`sprite_gen/video/rife_torch.py`). It serves a GPU the binary cannot run on: rife-ncnn-vulkan needs Vulkan, and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. The network is rife-v4.6's IFNet written again from Practical-RIFE's `IFNet_HDv3` (MIT, notice in NOTICE). Its weights are read out of the binary's own model files (SHA-256 pinned, the binary's float16 weights), with no other download. One run does what one call of the binary does with `-x -z`. The pair is turned eight ways and run both ways in time, with flows averaged at each of four levels. The loop follows rife-ncnn-vulkan's `RIFE::process_v4` (MIT, notice in NOTICE). `SPRITE_GEN_RIFE_DEVICE` names `cuda` (default) or `cpu`. `SPRITE_GEN_RIFE_TTA` means the same for both backends. The default augmentation stays `-x -z`. To install: `pip install "sprite-gen[torch]"` (new extra: `torch>=2.13.0,<3`). Then run: `sprite-gen rife install --backend torch` (new flag; defaults to `SPRITE_GEN_RIFE_BACKEND`). This unpacks the model, and makes a check frame through PyTorch on `SPRITE_GEN_RIFE_DEVICE`. It reads the model only, so a Linux ARM machine the binary does not run on is not refused. A CPU-only host installs PyTorch's CPU build first: `pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu` (192 MB, against the CUDA build's 527 MB plus its CUDA libraries). Nothing falls back. With torch chosen the binary is never looked for or run. `cuda` where PyTorch finds no CUDA is refused naming why (a CPU-only PyTorch, or a CUDA that finds no GPU) and does not run on the CPU. `mps` (Apple's GPU) is refused (no lane measures it; on a Mac the binary runs on the GPU already). A backend or device named wrong is refused. A device that fails as the network is put on it or as a frame runs (out of its memory, its driver or CUDA failing) is a RIFE that fails, in PyTorch's own words: the loop fails with its record, and the install's check frame fails. PyTorch not installed, or no model where the backend looks (`SPRITE_GEN_RIFE_MODEL`, else the install root), is RIFE not installed, as the binary missing is. Under `--repair auto` the loop is cut as filmed with the warning, and its record names the torch install line. On CUDA each frame runs with cuDNN deterministic, its benchmark off and convolutions in IEEE float32 (no TF32). The same frame twice is the same bytes on one GPU, driver and PyTorch. What made a frame is the RIFE's identity: backend, model bytes loaded, network code, PyTorch, on CUDA the GPU, driver, CUDA, cuDNN and those settings; on the CPU its instruction set. Because of this identity, `--between-cache` never hands one backend's or GPU's frame to another. Measured on synthetic walkers: Seven walker pairs (clean, outline lost, ghost, torn, a cross-fade to look at) get the same verdicts from the binary on an Apple GPU, the binary on Mesa llvmpipe and the torch backend on a CPU. The torch frame lies 0.005 to 0.052 of 255 on average from the binary's Apple GPU frame. The binary's own llvmpipe frame lies 0.007 to 0.25. A 960 x 960 frame of a synthetic walker with `-x -z`: On an Apple M4 Max CPU (12 threads), 3.3 to 3.6 s; the binary on that Mac's GPU takes 1.27 to 1.37 s. In a Modal container of two CPUs (the app's job size), 22.6 to 28.5 s on two threads; the binary on Mesa llvmpipe in the same container takes 127.7 to 131.5 s. The torch backend's frames are not the binary's bytes, which is one reason the binary stays the default. [loop-repair](docs/loop-repair.md) section 1-- **New: a host with no GPU sends the torch backend's frames to a GPU elsewhere (`SPRITE_GEN_RIFE_EXECUTOR`).** The host names a send of its own, `SPRITE_GEN_RIFE_EXECUTOR=module:attr`: a function from a request's bytes to its answer's. It reaches `sprite_gen.video.rife_torch.serve` on the GPU's machine, the same sprite-gen with PyTorch and the model installed there. The engine knows nothing of what carries the bytes, and the host needs no PyTorch. A frame is one call (its two runs made there), and a read is one call (its two runs and sixteen passes, the passes reduced there to how far they disagree). A RIFE asks the executor what it is once, as it is made (the probe, which takes the cold start), so a `video-loop` makes at most 53 calls. The RIFE holds that answer for its life, and every request carries it. An executor that is not it any more (another GPU, driver or image) makes nothing, and the RIFE fails naming what differs (`gpu: Tesla T4 -> NVIDIA L4`). An answer from another identity is not handed on. So one RIFE never hands on frames of two GPUs, and a frame taken from `--between-cache` is the frame it would make; the next RIFE asks again and keys its frames anew. Refused: an executor named with the binary's backend or with `SPRITE_GEN_RIFE_DEVICE` (the device is the executor's), a send that cannot be imported, and an executor whose `rife.py` or protocol is another (run the same sprite-gen on both sides). A send that raises, an answer that does not read (not the format, damaged or cut short on the way, or not the frame's size) and the executor's own failure are each a RIFE that fails, in its words; nothing is asked again or run elsewhere. The host names one GPU type for the executor (no fallback list) and does not deploy it anew while a run goes on. Requests and answers are a compressed NumPy archive, read with `allow_pickle=False`, each member whole so its CRC-32 is checked before its array is read: an answer damaged on the way hands on no other pixels. Reports name `executor` and the executor's `device` and `gpu`; the overlapped wrap counts `runs` and `executor_calls`. [loop-repair](docs/loop-repair.md) section 1+- **New: RIFE's torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan.** `SPRITE_GEN_RIFE_BACKEND=torch` runs the same rife-v4.6 in PyTorch (`sprite_gen/video/rife_torch.py`). It serves a GPU the binary cannot run on: rife-ncnn-vulkan needs Vulkan, and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. The network is rife-v4.6's IFNet written again from Practical-RIFE's `IFNet_HDv3` (MIT, notice in NOTICE). Its weights are read out of the binary's own model files (SHA-256 pinned, the binary's float16 weights), with no other download. One run does what one call of the binary does with `-x -z`. The pair is turned eight ways and run both ways in time, with flows averaged at each of four levels. The loop follows rife-ncnn-vulkan's `RIFE::process_v4` (MIT, notice in NOTICE). `SPRITE_GEN_RIFE_DEVICE` names `cuda` (default) or `cpu`. `SPRITE_GEN_RIFE_TTA` means the same for both backends. The default augmentation stays `-x -z`. To install: `pip install "sprite-gen[torch]"` (new extra: `torch>=2.13.0,<3`). Then run: `sprite-gen rife install --backend torch` (new flag; defaults to `SPRITE_GEN_RIFE_BACKEND`). This unpacks the model, and makes a check frame through PyTorch on `SPRITE_GEN_RIFE_DEVICE`. It reads the model only, so a Linux ARM machine the binary does not run on is not refused. A CPU-only host installs PyTorch's CPU build first: `pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu` (192 MB, against the CUDA build's 527 MB plus its CUDA libraries). Nothing falls back. With torch chosen the binary is never looked for or run. `cuda` where PyTorch finds no CUDA is refused naming why (a CPU-only PyTorch, or a CUDA that finds no GPU) and does not run on the CPU. `mps` (Apple's GPU) is refused (no lane measures it; on a Mac the binary runs on the GPU already). A backend or device named wrong is refused. A device that fails as the network is put on it or as a frame runs (out of its memory, its driver or CUDA failing) is a RIFE that fails, in PyTorch's own words: the loop fails with its record, and the install's check frame fails. PyTorch not installed, or no model where the backend looks (`SPRITE_GEN_RIFE_MODEL`, else the install root), is RIFE not installed, as the binary missing is. Under `--repair auto` the loop is cut as filmed with the warning, and its record names the torch install line. On CUDA each frame runs with cuDNN deterministic, its benchmark off and convolutions in IEEE float32 (no TF32). The same frame twice is the same bytes on one GPU, driver and PyTorch. What made a frame is the RIFE's identity: backend, model bytes loaded, network code, PyTorch, on CUDA the GPU, driver, CUDA, cuDNN and those settings; on the CPU its instruction set. Because of this identity, `--between-cache` never hands one backend's or GPU's frame to another. Measured on synthetic walkers: Seven walker pairs (clean, outline lost, ghost, torn, a cross-fade to look at) get the same verdicts from the binary on an Apple GPU, the binary on Mesa llvmpipe and the torch backend on a CPU. The torch frame lies 0.005 to 0.052 of 255 on average from the binary's Apple GPU frame. The binary's own llvmpipe frame lies 0.007 to 0.25. A 960 x 960 frame of a synthetic walker with `-x -z`: On an Apple M4 Max CPU (12 threads), 3.3 to 3.6 s; the binary on that Mac's GPU takes 1.27 to 1.37 s. In a Modal container of two CPUs (the app's job size), 22.6 to 28.5 s on two threads; the binary on Mesa llvmpipe in the same container takes 127.7 to 131.5 s. On an NVIDIA T4 rented from Modal (CUDA), 1.36 to 1.41 s. The torch backend's frames are not the binary's bytes, which is one reason the binary stays the default. [loop-repair](docs/loop-repair.md) section 1+- **New: a host with no GPU sends the torch backend's frames to a GPU elsewhere (`SPRITE_GEN_RIFE_EXECUTOR`).** The host names a send of its own, `SPRITE_GEN_RIFE_EXECUTOR=module:attr`: a function from a request's bytes to its answer's. It reaches `sprite_gen.video.rife_torch.serve` on the GPU's machine, the same sprite-gen with PyTorch and the model installed there. The engine knows nothing of what carries the bytes, and the host needs no PyTorch. A frame is one call (its two runs made there), and a read is one call (its two runs and sixteen passes, the passes reduced there to how far they disagree). A RIFE asks the executor what it is once, as it is made (the probe, which takes the cold start), so a `video-loop` makes at most 53 calls. The RIFE holds that answer for its life, and every request carries it. An executor that is not it any more (another GPU, driver or image) makes nothing, and the RIFE fails naming what differs (`gpu: Tesla T4 -> NVIDIA L4`). An answer from another identity is not handed on. So one RIFE never hands on frames of two GPUs, and a frame taken from `--between-cache` is the frame it would make; the next RIFE asks again and keys its frames anew. Refused: an executor named with the binary's backend or with `SPRITE_GEN_RIFE_DEVICE` (the device is the executor's), a send that cannot be imported, and an executor whose `rife.py` or protocol is another (run the same sprite-gen on both sides). A send that raises, an answer that does not read (not the format, damaged or cut short on the way, or not the frame's size) and the executor's own failure are each a RIFE that fails, in its words; nothing is asked again or run elsewhere. The host names one GPU type for the executor (no fallback list) and does not deploy it anew while a run goes on. Requests and answers are a compressed NumPy archive, read with `allow_pickle=False`, each member whole so its CRC-32 is checked before its array is read: an answer damaged on the way hands on no other pixels. Reports name `executor` and the executor's `device` and `gpu`; the overlapped wrap counts `runs` and `executor_calls`. Measured on an NVIDIA T4 rented from Modal, from a host of two CPUs without PyTorch: a 960 x 960 `-x -z` frame of a synthetic walker takes 1.49 to 1.52 s, 0.1 to 0.2 s more than in the T4's own process, a read as long, and the frames and readings are the same bytes as there. [loop-repair](docs/loop-repair.md) section 1- **Lane: `scripts/linux_lane.sh` tests PyTorch 2.13.0 CPU after the suite.** After the suite, PyTorch 2.13.0 CPU is installed into the same environment. It runs `sprite-gen rife install --backend torch`'s check frame. It runs every test marked `torch_rife` with `SPRITE_GEN_RIFE_DEVICE=cpu`, with none skipped. `--rife-only` runs the same. `--rife-unmeasured` lets `torch_rife` tests skip as it lets `real_rife` ones. CUDA is not measured by this lane.- **Lane: `scripts/t4_lane.py` tests the torch backend on an NVIDIA T4, rented from Modal.** `modal run scripts/t4_lane.py` builds the checked-out commit's tree into an image as the Linux lane installs a clean clone: Python 3.14, PyTorch 2.13.0 from PyPI (its Linux wheel is the CUDA 13 build), the tree with `.[dev]`, and the model by `sprite-gen rife install --backend torch`, where the engine looks for no binary. On one T4 it makes the install's check frame on CUDA and runs every test marked `torch_rife` and not `real_rife` on CUDA, with none skipped. `--whole-suite` first runs the whole suite in that environment, PyTorch in it, as the Linux lane's RIFE-less run does. A commit passes the lanes where the Linux lane passed at it and one T4 lane run at it exited 0. [release](docs/release.md)- **Tests**: `tests/video/test_rife_torch.py` (the eight ways undo themselves and are the single passes' eight turns; flows of a textured patch run alone each of the sixteen ways and turned back point alike, where a wrong sign, a transpose's x/y, or a backward swap puts one elsewhere; `-z` and `-x` give the same frame the other way in time and flipped; the install check frame's disc lands half way; the model loads strict and other bytes are refused by SHA-256; the same frame twice is the same bytes and the identity names what makes it; seven recorded synthetic walker pairs in `tests/fixtures/rife-walkers` are judged as the binary judged them with no binary, the frame within twice the binary's own GPU-to-GPU gap, and alike to this machine's binary where both run; a device that fails as the network is put on it or as a frame runs (out of its memory, its driver, CUDA not starting, a CUDA call deferred to its start) is a RIFE that fails in PyTorch's words, a TypeError there is raised as it is, a loop whose device fails as the network is put on it fails with its record, and the install's check frame fails; without PyTorch: torch chosen and not installed is RIFE not installed and the binary is never run, a loop's record names it, a backend or device named wrong is refused before anything is looked for, cuda where there is none is refused, a model not the pinned one is refused; with a runner stood in: reports name rife-torch and count runs, the loop and the alignment name the torch backend, and a frame kept by one runner is not taken for another); `tests/video/test_rife_executor.py` (a frame is one call and a read one call, and the RIFE probes once as it is made, their bytes the same device's in this process with this host's augmentation; every request names the engine code and the identity held; the host runs without PyTorch; the identity is the executor's as the RIFE was made; a T4 frame kept is the RIFE's own after the executor turns L4, which it then refuses to make anew, and the next RIFE misses and keeps the L4 frame under its own key; an executor whose GPU, driver, network code, weights, PyTorch, cuDNN or determinism changed makes nothing, the field named; an answer from another identity is neither handed on nor kept; another container of the same identity runs on, and the next RIFE takes every frame with its probe alone; a T4 and L4 pool is refused at the first L4 request; a frame made while this host's augmentation changed is not kept; other engine code or another protocol is refused before anything loads; a request that does not read, is pickled or was damaged on the way is answered so, in the reading's words; a send, an answer or an executor that fails is a RIFE that fails, in its words; an answer damaged in a member's local header, a compressed block, an array's header or its own bytes, or its end record, or cut short, is a RIFE that fails in the words of what reading it raised, which is its cause, and undamaged is the frame made here; each byte of the probe's, a frame's and a read's answer damaged in turn fails the RIFE or reads as it was made, nothing else raised and no other pixels handed on; a reading that cannot read what `pack` writes raises as it is, a fault of the code; an answer that is not bytes does not read; an executor named with the binary, with a device, or not there is refused and the binary never runs; a loop names the executor, and a loop whose executor's answer was damaged on the way fails with its record (`failed`, in the decompressor's words, the gait fallback's frames removed); marked `torch_rife`: `rife_torch.serve` makes the frame and the read of the backend in this process, loaded once); `tests/video/test_rife_install.py` (a zip whose model is not the pinned one installs nothing for both backends; `--backend torch` installs on a machine the binary does not run on and checks its frame through torch); `tests/release/test_lane_rife_collection.py` (a test RIFE decides carries `real_rife` or `torch_rife`); `tests/release/test_t4_lane.py` (the T4 lane installs the PyTorch and the Python the Linux lane does); new pytest mark `torch_rife` (skipped where PyTorch, the model or the named device is missing).diff --git a/docs/loop-repair.md b/docs/loop-repair.mdindex 4e93e78..7c83628 100644--- a/docs/loop-repair.md+++ b/docs/loop-repair.md@@ -352,11 +352,19 @@ the default does not move. Measured on synthetic walkers, 2026-10-21:| torch backend, Apple M4 Max CPU (12 threads) | 3.3 to 3.6 s | 0.38 to 0.40 s | 8.8 to 9.0 s || binary, Modal `cpu=2` (llvmpipe), one cold call each | 127.7 to 131.5 s | 9.3 to 9.5 s | not run || torch backend, the same Modal container, 2 threads | 22.6 to 28.5 s | 1.5 to 1.7 s | 45.1 to 46.1 s |+| torch backend, Modal NVIDIA T4 (CUDA), in its process | 1.36 to 1.41 s | 0.32 to 0.36 s | 3.9 to 4.1 s |+| torch backend on that T4, from a `cpu=2` host without PyTorch (`SPRITE_GEN_RIFE_EXECUTOR`, a Modal call each) | 1.49 to 1.52 s | 0.46 to 0.49 s | 4.0 to 4.1 s |The torch backend's frame on an x86 CPU is not the same bytes as on an Apple CPU (its torn reading is),which the CPU's instruction set in its identity keeps apart. PyTorch sizes its threads by the host'scores, not a container's share (the container above saw 18): a container names its share-(`OMP_NUM_THREADS`). On a CUDA GPU the release lane measures it (docs/release.md).+(`OMP_NUM_THREADS`).++On the T4 (driver 610.57.04, PyTorch 2.13.0 for CUDA 13.0, cuDNN 9.20, 2026-10-22) the frames made+through the executor, from another container of the same image, are the bytes made in the T4's own+process: both walkers' frames, plain frames, and reads with their disagreement. The executor adds 0.1 to+0.2 s to a frame, a read takes as long as in the process, and the probe, a T4 container's cold start,+took 8.4 s. The T4 lane runs the torch tests on CUDA for a release (docs/release.md).### Licences