Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
feat(rife): the torch backend on a GPU elsewhere — a host's send carries a frame or a read in one call, and a RIFE holds the executor it was made with
Soohong Kim5535879539cd
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex e8cf841..b40b149 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,11 +4,12 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the## Unreleased - The new RIFE torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan; rife-ncnn-vulkan stays the default-**Upgrading from 2.45.2: Defaults stay unchanged, cache keys update once, reports show the backend, install checks model hashes, and schemas are unchanged.** With `SPRITE_GEN_RIFE_BACKEND` unset (the default), RIFE is the `rife-ncnn-vulkan` binary as before. Its frames are the same bytes as 2.45.2. Measured: the binary's frames and sixteen-pass readings through the changed engine equal the old engine's, byte for byte. A `--between-cache` directory kept by 2.45.2 is made again once. The key now names the backend (`backend: "ncnn"`). The engine's `rife.py` changed, and its sha256 is part of the key anyway. Reports take `interpolator.kind` from what made the frames: `rife-ncnn-vulkan` as before for the binary, or `rife-torch` for the new backend. The overlapped wrap's record counts `binary_calls` for the binary as before, or `runs` for the torch backend. `sprite-gen rife install` now checks the model's two files (`rife-v4.6/flownet.param`, `flownet.bin`) against pinned SHA-256 after unpacking, for either backend. A model that differs installs nothing. The line a missing RIFE is reported with (`install` in records, warnings) is `sprite-gen rife install` for the binary. Where the torch backend is chosen, that line is: `pip install "sprite-gen[torch]" && sprite-gen rife install --backend torch`. No command-line flags of `video-loop` / `video-cycle-align` / `video-set` change. No restoration schemas change: `source-restoration-v5`, repair report `schema_version` 4, comparison `schema_version` 6.+**Upgrading from 2.45.2: Defaults stay unchanged, cache keys update once, reports show the backend, install checks model hashes, and schemas are unchanged.** With `SPRITE_GEN_RIFE_BACKEND` unset (the default), RIFE is the `rife-ncnn-vulkan` binary as before. Its frames are the same bytes as 2.45.2. Measured: the binary's frames and sixteen-pass readings through the changed engine equal the old engine's, byte for byte. A `--between-cache` directory kept by 2.45.2 is made again once. The key now names the backend (`backend: "ncnn"`). The engine's `rife.py` changed, and its sha256 is part of the key anyway. Reports take `interpolator.kind` from what made the frames: `rife-ncnn-vulkan` as before for the binary, or `rife-torch` for the new backend. The overlapped wrap's record counts `binary_calls` for the binary as before, or `runs` for the torch backend. `SPRITE_GEN_RIFE_EXECUTOR` (new) is unset by default, and RIFE runs in this process. `sprite-gen rife install` now checks the model's two files (`rife-v4.6/flownet.param`, `flownet.bin`) against pinned SHA-256 after unpacking, for either backend. A model that differs installs nothing. The line a missing RIFE is reported with (`install` in records, warnings) is `sprite-gen rife install` for the binary. Where the torch backend is chosen, that line is: `pip install "sprite-gen[torch]" && sprite-gen rife install --backend torch`. No command-line flags of `video-loop` / `video-cycle-align` / `video-set` change. No restoration schemas change: `source-restoration-v5`, repair report `schema_version` 4, comparison `schema_version` 6.- **New: RIFE's torch backend runs rife-v4.6 in PyTorch for compute GPUs without Vulkan.** `SPRITE_GEN_RIFE_BACKEND=torch` runs the same rife-v4.6 in PyTorch (`sprite_gen/video/rife_torch.py`). It serves a GPU the binary cannot run on: rife-ncnn-vulkan needs Vulkan, and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. The network is rife-v4.6's IFNet written again from Practical-RIFE's `IFNet_HDv3` (MIT, notice in NOTICE). Its weights are read out of the binary's own model files (SHA-256 pinned, the binary's float16 weights), with no other download. One run does what one call of the binary does with `-x -z`. The pair is turned eight ways and run both ways in time, with flows averaged at each of four levels. The loop follows rife-ncnn-vulkan's `RIFE::process_v4` (MIT, notice in NOTICE). `SPRITE_GEN_RIFE_DEVICE` names `cuda` (default) or `cpu`. `SPRITE_GEN_RIFE_TTA` means the same for both backends. The default augmentation stays `-x -z`. To install: `pip install "sprite-gen[torch]"` (new extra: `torch>=2.13.0,<3`). Then run: `sprite-gen rife install --backend torch` (new flag; defaults to `SPRITE_GEN_RIFE_BACKEND`). This unpacks the model, and makes a check frame through PyTorch on `SPRITE_GEN_RIFE_DEVICE`. It reads the model only, so a Linux ARM machine the binary does not run on is not refused. A CPU-only host installs PyTorch's CPU build first: `pip install torch==2.13.0 --index-url https://download.pytorch.org/whl/cpu` (192 MB, against the CUDA build's 527 MB plus its CUDA libraries). Nothing falls back. With torch chosen the binary is never looked for or run. `cuda` where PyTorch finds no CUDA is refused naming why (a CPU-only PyTorch, or a CUDA that finds no GPU) and does not run on the CPU. `mps` (Apple's GPU) is refused (no lane measures it; on a Mac the binary runs on the GPU already). A backend or device named wrong is refused. A device that fails as the network is put on it or as a frame runs (out of its memory, its driver or CUDA failing) is a RIFE that fails, in PyTorch's own words: the loop fails with its record, and the install's check frame fails. PyTorch not installed, or no model where the backend looks (`SPRITE_GEN_RIFE_MODEL`, else the install root), is RIFE not installed, as the binary missing is. Under `--repair auto` the loop is cut as filmed with the warning, and its record names the torch install line. On CUDA each frame runs with cuDNN deterministic, its benchmark off and convolutions in IEEE float32 (no TF32). The same frame twice is the same bytes on one GPU, driver and PyTorch. What made a frame is the RIFE's identity: backend, model bytes loaded, network code, PyTorch, on CUDA the GPU, driver, CUDA, cuDNN and those settings; on the CPU its instruction set. Because of this identity, `--between-cache` never hands one backend's or GPU's frame to another. Measured on synthetic walkers: Seven walker pairs (clean, outline lost, ghost, torn, a cross-fade to look at) get the same verdicts from the binary on an Apple GPU, the binary on Mesa llvmpipe and the torch backend on a CPU. The torch frame lies 0.005 to 0.052 of 255 on average from the binary's Apple GPU frame. The binary's own llvmpipe frame lies 0.007 to 0.25. A 960 x 960 frame of a synthetic walker with `-x -z`: On an Apple M4 Max CPU (12 threads), 3.3 to 3.6 s; the binary on that Mac's GPU takes 1.27 to 1.37 s. In a Modal container of two CPUs (the app's job size), 22.6 to 28.5 s on two threads; the binary on Mesa llvmpipe in the same container takes 127.7 to 131.5 s. The torch backend's frames are not the binary's bytes, which is one reason the binary stays the default. [loop-repair](docs/loop-repair.md) section 1+- **New: a host with no GPU sends the torch backend's frames to a GPU elsewhere (`SPRITE_GEN_RIFE_EXECUTOR`).** The host names a send of its own, `SPRITE_GEN_RIFE_EXECUTOR=module:attr`: a function from a request's bytes to its answer's. It reaches `sprite_gen.video.rife_torch.serve` on the GPU's machine, the same sprite-gen with PyTorch and the model installed there. The engine knows nothing of what carries the bytes, and the host needs no PyTorch. A frame is one call (its two runs made there), and a read is one call (its two runs and sixteen passes, the passes reduced there to how far they disagree). A RIFE asks the executor what it is once, as it is made (the probe, which takes the cold start), so a `video-loop` makes at most 53 calls. The RIFE holds that answer for its life, and every request carries it. An executor that is not it any more (another GPU, driver or image) makes nothing, and the RIFE fails naming what differs (`gpu: Tesla T4 -> NVIDIA L4`). An answer from another identity is not handed on. So one RIFE never hands on frames of two GPUs, and a frame taken from `--between-cache` is the frame it would make; the next RIFE asks again and keys its frames anew. Refused: an executor named with the binary's backend or with `SPRITE_GEN_RIFE_DEVICE` (the device is the executor's), a send that cannot be imported, and an executor whose `rife.py` or protocol is another (run the same sprite-gen on both sides). A send that raises, an answer that does not read and the executor's own failure are each a RIFE that fails, in its words; nothing is asked again or run elsewhere. The host names one GPU type for the executor (no fallback list) and does not deploy it anew while a run goes on. Requests and answers are a compressed NumPy archive, read with `allow_pickle=False`. Reports name `executor` and the executor's `device` and `gpu`; the overlapped wrap counts `runs` and `executor_calls`. [loop-repair](docs/loop-repair.md) section 1- **Lane: `scripts/linux_lane.sh` tests PyTorch 2.13.0 CPU after the suite.** After the suite, PyTorch 2.13.0 CPU is installed into the same environment. It runs `sprite-gen rife install --backend torch`'s check frame. It runs every test marked `torch_rife` with `SPRITE_GEN_RIFE_DEVICE=cpu`, with none skipped. `--rife-only` runs the same. `--rife-unmeasured` lets `torch_rife` tests skip as it lets `real_rife` ones. CUDA is not measured by this lane.-- **Tests**: `tests/video/test_rife_torch.py` (the eight ways undo themselves and are the single passes' eight turns; flows of a textured patch run alone each of the sixteen ways and turned back point alike, where a wrong sign, a transpose's x/y, or a backward swap puts one elsewhere; `-z` and `-x` give the same frame the other way in time and flipped; the install check frame's disc lands half way; the model loads strict and other bytes are refused by SHA-256; the same frame twice is the same bytes and the identity names what makes it; seven recorded synthetic walker pairs in `tests/fixtures/rife-walkers` are judged as the binary judged them with no binary, the frame within twice the binary's own GPU-to-GPU gap, and alike to this machine's binary where both run; a device that fails as the network is put on it or as a frame runs (out of its memory, its driver, CUDA not starting, a CUDA call deferred to its start) is a RIFE that fails in PyTorch's words, a TypeError there is raised as it is, a loop whose device fails as the network is put on it fails with its record, and the install's check frame fails; without PyTorch: torch chosen and not installed is RIFE not installed and the binary is never run, a loop's record names it, a backend or device named wrong is refused before anything is looked for, cuda where there is none is refused, a model not the pinned one is refused; with a runner stood in: reports name rife-torch and count runs, the loop and the alignment name the torch backend, and a frame kept by one runner is not taken for another); `tests/video/test_rife_install.py` (a zip whose model is not the pinned one installs nothing for both backends; `--backend torch` installs on a machine the binary does not run on and checks its frame through torch); `tests/release/test_lane_rife_collection.py` (a test RIFE decides carries `real_rife` or `torch_rife`); new pytest mark `torch_rife` (skipped where PyTorch, the model or the named device is missing).+- **Tests**: `tests/video/test_rife_torch.py` (the eight ways undo themselves and are the single passes' eight turns; flows of a textured patch run alone each of the sixteen ways and turned back point alike, where a wrong sign, a transpose's x/y, or a backward swap puts one elsewhere; `-z` and `-x` give the same frame the other way in time and flipped; the install check frame's disc lands half way; the model loads strict and other bytes are refused by SHA-256; the same frame twice is the same bytes and the identity names what makes it; seven recorded synthetic walker pairs in `tests/fixtures/rife-walkers` are judged as the binary judged them with no binary, the frame within twice the binary's own GPU-to-GPU gap, and alike to this machine's binary where both run; a device that fails as the network is put on it or as a frame runs (out of its memory, its driver, CUDA not starting, a CUDA call deferred to its start) is a RIFE that fails in PyTorch's words, a TypeError there is raised as it is, a loop whose device fails as the network is put on it fails with its record, and the install's check frame fails; without PyTorch: torch chosen and not installed is RIFE not installed and the binary is never run, a loop's record names it, a backend or device named wrong is refused before anything is looked for, cuda where there is none is refused, a model not the pinned one is refused; with a runner stood in: reports name rife-torch and count runs, the loop and the alignment name the torch backend, and a frame kept by one runner is not taken for another); `tests/video/test_rife_executor.py` (a frame is one call and a read one call, and the RIFE probes once as it is made, their bytes the same device's in this process with this host's augmentation; every request names the engine code and the identity held; the host runs without PyTorch; the identity is the executor's as the RIFE was made; a T4 frame kept is the RIFE's own after the executor turns L4, which it then refuses to make anew, and the next RIFE misses and keeps the L4 frame under its own key; an executor whose GPU, driver, network code, weights, PyTorch, cuDNN or determinism changed makes nothing, the field named; an answer from another identity is neither handed on nor kept; another container of the same identity runs on, and the next RIFE takes every frame with its probe alone; a T4 and L4 pool is refused at the first L4 request; a frame made while this host's augmentation changed is not kept; other engine code or another protocol is refused before anything loads; a request that does not read, or is pickled, is answered so; a send, an answer or an executor that fails is a RIFE that fails, in its words; an executor named with the binary, with a device, or not there is refused and the binary never runs; a loop names the executor; marked `torch_rife`: `rife_torch.serve` makes the frame and the read of the backend in this process, loaded once); `tests/video/test_rife_install.py` (a zip whose model is not the pinned one installs nothing for both backends; `--backend torch` installs on a machine the binary does not run on and checks its frame through torch); `tests/release/test_lane_rife_collection.py` (a test RIFE decides carries `real_rife` or `torch_rife`); new pytest mark `torch_rife` (skipped where PyTorch, the model or the named device is missing).## v2.45.2 - The new `video-loop --overlap-wrap off` turns the overlapped wrap off on its own; the motion anchor's fine search is faster and gives the same valuesdiff --git a/SKILL.md b/SKILL.mdindex 6e2fe02..3239cf2 100644--- a/SKILL.md+++ b/SKILL.md@@ -12,7 +12,7 @@ depends_on:- name: img2webpwhy: "video-loop WebP with exact alpha (libwebp); Pillow's animated writer drops -exact"- name: rife-ncnn-vulkan- why: "video-loop jump repair and video-cycle-align (RIFE v4.6 in-betweens with the -x -z test-time augmentation, release 20221029). Install once with `sprite-gen rife install` (sha256-checked, into the user data directory); SPRITE_GEN_RIFE or PATH override it; Linux without a GPU also needs libvulkan1 mesa-vulkan-drivers. Without it, walk and run loops are cut as filmed with a warning. SPRITE_GEN_RIFE_BACKEND=torch runs the same model in PyTorch instead, for a GPU with no Vulkan (`pip install \"sprite-gen[torch]\"`, then `sprite-gen rife install --backend torch`; SPRITE_GEN_RIFE_DEVICE=cuda or cpu)"+ why: "video-loop jump repair and video-cycle-align (RIFE v4.6 in-betweens with the -x -z test-time augmentation, release 20221029). Install once with `sprite-gen rife install` (sha256-checked, into the user data directory); SPRITE_GEN_RIFE or PATH override it; Linux without a GPU also needs libvulkan1 mesa-vulkan-drivers. Without it, walk and run loops are cut as filmed with a warning. SPRITE_GEN_RIFE_BACKEND=torch runs the same model in PyTorch instead, for a GPU with no Vulkan (`pip install \"sprite-gen[torch]\"`, then `sprite-gen rife install --backend torch`; SPRITE_GEN_RIFE_DEVICE=cuda or cpu; SPRITE_GEN_RIFE_EXECUTOR=module:attr sends its frames to a GPU elsewhere, rife_torch.serve there)"required_scripts:- scripts/prepare_sprite_run.py- scripts/generate_sprite_image.pydiff --git a/docs/loop-repair.md b/docs/loop-repair.mdindex 14a2e72..db0ce6b 100644--- a/docs/loop-repair.md+++ b/docs/loop-repair.md@@ -1,6 +1,6 @@# Loop repair — RIFE in-betweens for jump frames, the jolt index, one cycle per set-> Owns: Where RIFE runs and what it costs, which model and augmentation it runs with, its two backends (the binary and PyTorch), how it is installed and what runs without it, jump-frame repair in `video-loop`, the flow reading (frames off a walk's way), the eased retime (a part that rushes on its way), the jolt index and its gate, cycle alignment across a direction set · Index: [docs/README.md](README.md)+> Owns: Where RIFE runs and what it costs, which model and augmentation it runs with, its two backends (the binary and PyTorch, the latter in this process or on a GPU elsewhere), how it is installed and what runs without it, jump-frame repair in `video-loop`, the flow reading (frames off a walk's way), the eased retime (a part that rushes on its way), the jolt index and its gate, cycle alignment across a direction set · Index: [docs/README.md](README.md)A walk loop cut from a generated clip can look right in every still and still hitch when itplays: the video model redraws thin hair a little differently every frame, and now and then@@ -57,6 +57,7 @@ and a GPU offered for compute alone (CUDA with no Vulkan driver) has none. It is| `SPRITE_GEN_RIFE_DEVICE` | `cuda`, `cpu` (the torch backend's device) | `cuda` || `SPRITE_GEN_RIFE_MODEL` | a `rife-v4.6` directory | beside the binary (`ncnn`); the install root's (`torch`) || `SPRITE_GEN_RIFE_TTA` | `-x`, `-z`, both, or empty | `-x -z`, for both backends |+| `SPRITE_GEN_RIFE_EXECUTOR` | `module:attr`, the host's send to a GPU elsewhere (torch backend; below) | unset: in this process |Nothing falls back. With torch chosen the engine neither looks for the binary nor runs it. A device isnot chosen for you and not traded for another: `cuda` where PyTorch finds no CUDA is refused, naming@@ -78,6 +79,60 @@ engine's reports name what made their frames: `interpolator.kind` is `rife-ncnn-`rife-torch`, and the overlapped wrap counts the binary's calls (`binary_calls`) or the torch backend'sruns (`runs`).+#### A GPU elsewhere: `SPRITE_GEN_RIFE_EXECUTOR`++A host with no GPU (a job container, say) can have the torch backend's frames made on a machine that+has one. The engine knows nothing of what carries them. The host names a function of its own,+`SPRITE_GEN_RIFE_EXECUTOR=module:attr`, from a request's bytes to its answer's, and that function+reaches `sprite_gen.video.rife_torch.serve` on the GPU's machine: the same sprite-gen, with PyTorch+and the model installed there (`sprite-gen rife install --backend torch`). The host needs neither.+It sets `SPRITE_GEN_RIFE_BACKEND=torch` and no `SPRITE_GEN_RIFE_DEVICE`: the device is the+executor's, and naming one here as well is refused. So is an executor named with the binary's+backend.++| Asked for | Runs on the executor | Calls of the send |+|---|---|---|+| a RIFE made (the probe) | none: the executor says what it is, loading the backend (the cold start) | 1 |+| a frame | 2 (colour and coverage), with the augmentation this host names | 1 |+| a read (the overlapped wrap) | 2, and the 32 runs of the sixteen passes, reduced there to how far they disagree | 1 |++Frames are not batched: the repairs make and judge one frame at a time. A `video-loop` makes at+most 53 calls: the probe, three flow repairs, nine eased frames and forty frames of the overlapped+wrap. A `video-cycle-align` makes one call for each frame it makes, and the probe.++A RIFE asks the executor what it is once, as it is made, and holds the answer for its life. The+answer is the torch backend's identity there (above: the model bytes, its network code, PyTorch,+on CUDA the GPU, its driver, CUDA, cuDNN and the determinism settings), with the protocol. The+RIFE's identity, which `--between-cache` keys frames on (section 4), is that answer, the+executor's name and this host's own part. Every frame and read carries the identity held. An+executor that is not it any more (another GPU, driver or image) makes nothing, and the RIFE fails,+naming what differs (`gpu: Tesla T4 -> NVIDIA L4`). An answer from another identity is not handed+on. So one RIFE never hands on frames of two GPUs, and a frame it takes from the cache is the frame+it would make. The next loop or alignment makes a new RIFE, which asks again and keys its frames+anew. Containers of one image, GPU and driver are one executor: their hosts' CPU and name are no+part of the identity, so a cache directory serves every one of them.++The host keeps to three things:++- Name one GPU type for the executor, with no list to fall back on. A request a GPU of another type+ takes fails the RIFE.+- Run the same sprite-gen on both sides. A request names this host's `rife.py` by SHA-256, and an+ executor of other code is refused as the RIFE is made.+- Do not deploy the executor anew while a loop or an alignment runs: that run fails, by why.++A send that raises, an answer that does not read, or the executor's own failure (its model not+installed, its GPU out of memory) is a RIFE that fails, in its words. Nothing is asked again, and+nothing runs here or on another device in its place.++A request and an answer are each one compressed NumPy archive (`np.savez_compressed`, read with+`allow_pickle=False`). Its `header` is JSON: `protocol` (1) and `code` (the `rife.py` SHA-256),+then `probe`, or `expect`, `t`, `modes` and `read`. Its images are RIFE's input as `rife.inputs`+makes it: the colour (H × W × 3) and the coverage (H × W) of the two frames, uint8. The answer+holds the identity, and the made colour and coverage (uint8) and, for a read, the disagreement+(float32); or an error (`protocol`, `code`, `identity` or `failed`, with what it names). Reports+name the executor: `interpolator` reads `kind` `rife-torch`, `executor`, and the executor's+`device` and `gpu`; the overlapped wrap counts `runs` and `executor_calls`.+### Install — `sprite-gen rife install````bash@@ -1674,7 +1729,9 @@ sprite-gen video-cycle-align --loop-dir set/front-walk/loop --loop-dir set/side-(`rife_torch.py`, read at import), PyTorch's version, the device, and on CUDA the GPU's name andcapability, the driver, CUDA's and cuDNN's versions and the determinism settings as they read back, onthe CPU its instruction set; the key names the backend either way, so a frame one backend made is never- taken for the other's. A binary or model file that cannot be read fails the alignment, as a RIFE that+ taken for the other's. On an executor (section 1, "A GPU elsewhere") that part is what the executor+ answered as the RIFE was made, held for its life, and the executor's name: an executor that changes+ under the RIFE fails it rather than make a frame of another identity. A binary or model file that cannot be read fails the alignment, as a RIFE thatfails does. Any change in cycle length, `--cycles`, source pixels, binary, model, augmentation, enginecode, or host makes the frame again. The key does not read the GPU driver or its environment; a callerthat changes those between passes gives each pass its own directory. Reading the identity hashes 36.5diff --git a/sprite_gen/video/rife.py b/sprite_gen/video/rife.pyindex 4cad205..8023f61 100644--- a/sprite_gen/video/rife.py+++ b/sprite_gen/video/rife.py@@ -36,6 +36,11 @@ SPRITE_GEN_RIFE_DEVICE names (`device()`: cuda, the default, or cpu), from the snor run, and a device that cannot be had is not traded for another. PyTorch or the model not installed is`RifeNotInstalled`, as the binary not installed is; the line that installs it is `install_command()`.+SPRITE_GEN_RIFE_EXECUTOR runs the torch backend on a GPU elsewhere (`executor()`, `RemoteRunner`): it names the host's send,+a function from a request's bytes to its answer's, which reaches the same sprite-gen there (`rife_torch.serve`, `answer`).+A frame is one call, a read one call, and a RIFE asks the executor what it is once, as it is made, holds that for its life+and refuses an executor that is not it — so a frame it hands on or keeps is that executor's. The host needs no PyTorch.+Wherever it runs, a frame is the same two runs of RIFE and the read the same sixteen passes (`frame_runs`, `pass_runs`):what turns a frame into RIFE's input (`inputs`) and its two runs back into a frame (`_joined`) is done here, and the runsare a runner's (`Binary`, `rife_torch.TorchRunner`), which `Rife` hands each frame whole.@@ -56,6 +61,7 @@ from __future__ import annotationsimport hashlibimport importlib+import ioimport jsonimport osimport platform@@ -63,6 +69,7 @@ import shutilimport subprocessimport sysimport tempfile+import zipfilefrom pathlib import Pathfrom typing import Callable@@ -86,6 +93,8 @@ MODEL_SHA256 = {"flownet.bin": "f334ed2260149ce0188a6dcf049844e8b0cdd912e01cbcfb63553157d2508958",}TORCH_EXTRA = 'pip install "sprite-gen[torch]"'+# `RemoteRunner`: the version of the requests and answers a host and its executor exchange (`pack`); another is refused.+PROTOCOL = 1# The binary's test-time augmentation, passed on every call: `-x` runs the pair as its eight flips and# transposes, `-z` runs it both ways in time as well, and what comes back is averaged — so a frame made# from a to b is the one made from b to a (docs/loop-repair.md section 1, "Which setting"). On a machine@@ -265,6 +274,41 @@ def device() -> str:return named+def executor() -> str | None:+ """The send that carries the torch backend's frames to a GPU elsewhere (`RemoteRunner`): SPRITE_GEN_RIFE_EXECUTOR,+ `module:attr` of the host's function from a request's bytes to its answer's; None where it is unset or empty. It runs+ the torch backend only, on the executor's own device, which its identity names: named with another backend, or with+ SPRITE_GEN_RIFE_DEVICE here as well, it is refused."""+ named = os.environ.get("SPRITE_GEN_RIFE_EXECUTOR")+ if not named:+ return None+ if backend() != "torch":+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_EXECUTOR={named!r}: an executor runs the torch backend; name "+ "SPRITE_GEN_RIFE_BACKEND=torch with it, or unset it")+ on = os.environ.get("SPRITE_GEN_RIFE_DEVICE")+ if on:+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_DEVICE={on!r} with SPRITE_GEN_RIFE_EXECUTOR={named!r}: the device is the "+ "executor's own, named where it runs and by its identity; unset SPRITE_GEN_RIFE_DEVICE here")+ return named+++def executor_send(named: str) -> Callable[[bytes], bytes]:+ """The send `executor()` names, imported: refused by name where it is not `module:attr`, its module cannot be imported+ or it has no such function."""+ module, colon, attr = named.partition(":")+ if not (module and colon and attr):+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_EXECUTOR={named!r}: name the host's send as module:attr, a function from a "+ "request's bytes to its answer's")+ try:+ found = importlib.import_module(module)+ except ImportError as exc:+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_EXECUTOR={named!r}: {module} could not be imported: {exc}") from exc+ send = getattr(found, attr, None)+ if not callable(send):+ raise RifeUnavailable(f"SPRITE_GEN_RIFE_EXECUTOR={named!r}: {module} has no function {attr}")+ return send++def install_command() -> str:"""The line that installs the RIFE `backend()` names, quoted wherever a missing RIFE is reported: INSTALL_COMMAND for thebinary; for torch the extra and the model, `sprite-gen rife install --backend torch`. A backend refused names the@@ -706,6 +750,190 @@ class Binary:raise RifeUnavailable(f"{self.binary.name}: the binary or model it runs from could not be read: {exc}") from exc+def pack(header: dict, arrays: dict[str, np.ndarray] | None = None) -> bytes:+ """A request or an answer between a `RemoteRunner` and its executor as bytes: `header` as UTF-8 JSON and `arrays`, in one+ compressed NumPy archive."""+ out = io.BytesIO()+ np.savez_compressed(out, header=np.frombuffer(json.dumps(header).encode(), np.uint8), **(arrays or {}))+ return out.getvalue()+++def unpack(data: bytes) -> tuple[dict, dict[str, np.ndarray]]:+ """`pack` read back: (header, arrays). Anything else is a ValueError naming why — pickled objects among them, which are+ never loaded."""+ if not isinstance(data, (bytes, bytearray, memoryview)):+ raise ValueError(f"{type(data).__name__}, not bytes")+ try:+ archive = np.load(io.BytesIO(bytes(data)), allow_pickle=False)+ if not isinstance(archive, np.lib.npyio.NpzFile):+ raise ValueError("an array, not an archive of them")+ with archive:+ arrays = {name: archive[name] for name in archive.files}+ header = json.loads(arrays.pop("header").tobytes().decode("utf-8"))+ except (OSError, ValueError, EOFError, KeyError, zipfile.BadZipFile) as exc:+ raise ValueError(f"{type(exc).__name__}: {exc}") from exc+ if not isinstance(header, dict):+ raise ValueError("its header is not an object")+ return header, arrays+++def _differs(held: dict, other: object) -> str:+ """Where another identity is not the one held, field by field (`gpu: Tesla T4 -> NVIDIA L4`)."""+ other = other if isinstance(other, dict) else {}+ return ", ".join(f"{k}: {held.get(k)} -> {other.get(k)}" for k in sorted(set(held) | set(other)) if held.get(k) != other.get(k))+++def _asked(header: dict, arrays: dict[str, np.ndarray]) -> tuple[list[np.ndarray], list[np.ndarray], float, tuple[str, ...], bool]:+ """A frame's or a read's request, as `RemoteRunner` sends it: (colour, coverage, t, the augmentation, whether the passes+ are read), or a ValueError naming what is not so."""+ t, modes, read = header.get("t"), header.get("modes"), header.get("read")+ if isinstance(t, bool) or not isinstance(t, (int, float)):+ raise ValueError(f"t is {t!r}, not a number")+ if (not isinstance(modes, list) or any(not isinstance(m, str) or m not in TTA_MODES for m in modes)+ or len(set(modes)) < len(modes)):+ raise ValueError(f"the augmentation is {modes!r}, not a list of {' or '.join(TTA_MODES)}, each once")+ if not isinstance(read, bool):+ raise ValueError(f"read is {read!r}, not true or false")+ cover = arrays.get("cover_a")+ if cover is None or cover.ndim != 2:+ raise ValueError("cover_a is not an image of one channel")+ h, w = cover.shape+ for name, shape in (("colour_a", (h, w, 3)), ("colour_b", (h, w, 3)), ("cover_a", (h, w)), ("cover_b", (h, w))):+ got = arrays.get(name)+ if got is None or got.shape != shape or got.dtype != np.uint8:+ raise ValueError(f"{name} is {'missing' if got is None else f'{got.shape} {got.dtype}'}, not {shape} uint8")+ return [arrays["colour_a"], arrays["colour_b"]], [arrays["cover_a"], arrays["cover_b"]], float(t), tuple(modes), read+++def answer(request: bytes, runner: Callable[[], object]) -> bytes:+ """The executor's answer to a request of a `RemoteRunner`, as bytes (`rife_torch.serve`): the frame or the read made by+ the runner that `runner()` loads, the torch backend. In order, before anything runs: a request that does not read,+ another protocol or another engine code (this rife.py, by sha256) is answered so, nothing loaded; the probe is answered+ by the runner's identity and the protocol, as it is loaded; a frame or a read that does not `expect` that identity is+ answered by it, nothing made. Then the frame's two runs, and a read's sixteen passes and their disagreement as well+ (`frame_runs`, `pass_runs`), with the augmentation the request names. A RIFE that fails here, loading or running, is+ answered in its words; anything else is raised, a fault of the code."""+ try:+ header, arrays = unpack(request)+ except ValueError as exc:+ return pack({"error": "failed", "message": f"the request does not read: {exc}"})+ if header.get("protocol") != PROTOCOL:+ return pack({"error": "protocol", "protocol": PROTOCOL})+ if header.get("code") != CODE_SHA256:+ return pack({"error": "code", "code": CODE_SHA256})+ try:+ loaded = runner()+ own = json.loads(json.dumps({"protocol": PROTOCOL, **loaded.identity()})) # type: ignore[attr-defined]+ except RifeUnavailable as exc:+ return pack({"error": "failed", "message": str(exc)})+ if header.get("probe") is True:+ return pack({"identity": own})+ if header.get("expect") != own:+ return pack({"error": "identity", "identity": own})+ try:+ colour, cover, t, modes, read = _asked(header, arrays)+ except ValueError as exc:+ return pack({"error": "failed", "message": f"the request does not read: {exc}"})++ def run(x: np.ndarray, y: np.ndarray, at: float, named: tuple[str, ...] | None) -> np.ndarray:+ return loaded.run(x, y, at, modes if named is None else named) # type: ignore[attr-defined]++ try:+ rgb, alpha = frame_runs(run, colour, cover, t)+ made = {"rgb": rgb, "alpha": alpha}+ if read:+ made["apart"] = pass_runs(run, colour, cover, t)+ except RifeUnavailable as exc:+ return pack({"error": "failed", "message": str(exc)})+ return pack({"identity": own}, made)+++class RemoteRunner:+ """The torch backend on a GPU elsewhere, as the runner of a `Rife` (SPRITE_GEN_RIFE_EXECUTOR): `send`, the host's, from a+ request's bytes to its answer's (`pack`), reaches the same sprite-gen there (`rife_torch.serve`, `answer`). A frame is+ one call, its two runs made there; a read is one call, its two runs and sixteen passes made, and the passes reduced to+ their disagreement, there. It needs no PyTorch here.++ Made, it asks the executor what it is (the probe) and holds the answer for its life: the identity of the torch backend+ it runs (`rife_torch.TorchRunner.identity`) and the protocol, the executor named besides (`identity`). Every request+ carries it (`expect`): an executor that is not it any more makes nothing and says what differs, and an answer from+ another is not handed on — so every frame handed on, and every frame a `Cache` keeps under its identity, is that+ executor's. An executor whose protocol or rife.py is another is refused as it is made. Nothing is asked again or run+ elsewhere: a send that fails, an answer that does not read and the executor's own failure are each a RIFE that fails+ (`RifeUnavailable`). Counts its calls of the send (`calls`)."""++ def __init__(self, send: Callable[[bytes], bytes], name: str):+ self.send, self.name = send, name+ self.calls = 0+ self.remote: dict = {}+ header, _ = self._sent({"probe": True}, {})+ if not isinstance(header.get("identity"), dict):+ raise RifeUnavailable(f"{name}: the executor's answer does not read: it names no identity")+ self.remote = header["identity"]++ def frame(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray]:+ made = self._made(colour, cover, t, read=False)+ return made["rgb"], made["alpha"]++ def read(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray, np.ndarray]:+ made = self._made(colour, cover, t, read=True)+ return made["rgb"], made["alpha"], made["apart"]++ def describe(self) -> dict[str, str]:+ return {"kind": "rife-torch", "executor": self.name, "device": str(self.remote.get("device")),+ **({"gpu": str(self.remote["gpu"])} if "gpu" in self.remote else {})}++ def identity(self) -> dict[str, object]:+ return {**self.remote, "executor": self.name}++ def _made(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float, *, read: bool) -> dict[str, np.ndarray]:+ """A frame, or a read, made by the executor in one call: the request names the identity held, the fraction, the+ augmentation this host asks for now (`tta()`) and whether the passes are read; the answer is that identity's, at+ the frame's size, or it is not handed on."""+ h, w = cover[0].shape+ header, arrays = self._sent({"expect": self.remote, "t": t, "modes": list(tta()), "read": read},+ {"colour_a": colour[0], "colour_b": colour[1], "cover_a": cover[0], "cover_b": cover[1]})+ if header.get("identity") != self.remote:+ raise RifeUnavailable(f"{self.name}: the executor answered as another than this RIFE runs as "+ f"({_differs(self.remote, header.get('identity'))}); its frame is not handed on")+ want = {"rgb": ((h, w, 3), np.uint8), "alpha": ((h, w), np.uint8), **({"apart": ((h, w), np.float32)} if read else {})}+ for name, (shape, dtype) in want.items():+ got = arrays.get(name)+ if got is None or got.shape != shape or got.dtype != dtype:+ found = "missing" if got is None else f"{got.shape} {got.dtype}"+ raise RifeUnavailable(f"{self.name}: the executor's answer does not read: {name} is {found}, not {shape} "+ f"{np.dtype(dtype)}")+ return arrays++ def _sent(self, header: dict, arrays: dict[str, np.ndarray]) -> tuple[dict, dict[str, np.ndarray]]:+ """One call of the send, this engine's protocol and code (CODE_SHA256) heading the request: the answer, read, or+ the refusal or failure it names (`RifeUnavailable`)."""+ request = pack({"protocol": PROTOCOL, "code": CODE_SHA256, **header}, arrays)+ self.calls += 1+ try:+ data = self.send(request)+ except Exception as exc: # what carries the bytes is the host's, failing in its own words+ raise RifeUnavailable(f"{self.name}: sending to the executor failed: {exc}") from exc+ try:+ answered, got = unpack(data)+ except ValueError as exc:+ raise RifeUnavailable(f"{self.name}: the executor's answer does not read: {exc}") from exc+ error = answered.get("error")+ if error == "protocol":+ raise RifeUnavailable(f"{self.name}: the executor speaks protocol {answered.get('protocol')}, this one {PROTOCOL}: "+ "run the same sprite-gen on both sides")+ if error == "code":+ raise RifeUnavailable(f"{self.name}: the executor's sprite-gen rife.py is {answered.get('code')}, this one's "+ f"{CODE_SHA256}: run the same sprite-gen on both sides")+ if error == "identity":+ raise RifeUnavailable(f"{self.name}: the executor is not the one this RIFE was made with "+ f"({_differs(self.remote, answered.get('identity'))}) and made nothing; a RIFE made now "+ "runs as it is")+ if error is not None:+ raise RifeUnavailable(f"{self.name}: the executor failed: {answered.get('message', error)}")+ return answered, got++def torch_runner(model: Path | None = None) -> object:"""The torch backend's runner (`rife_torch.TorchRunner`) on `device()`, from `model` (else `locate_model()`): the devicenamed is refused before anything is imported, PyTorch not installed is `RifeNotInstalled` with the extra that installs@@ -726,14 +954,16 @@ class Rife:times it ran RIFE (`calls`): the binary's calls, or the torch backend's runs.What runs it is its runner, which makes a frame whole from what `inputs` gives it (`frame`: two runs; `read`: two- runs and the sixteen passes) and names what its frames depend on (`identity`): `Binary` for the binary, or- `rife_torch.TorchRunner`. `Rife()` takes the one `backend()` names; `Rife(binary, model)` is the binary at those- paths; `Rife(runner=...)` any other."""+ runs and the sixteen passes) and names what its frames depend on (`identity`): `Binary` for the binary,+ `rife_torch.TorchRunner`, or `RemoteRunner` for the torch backend on a GPU elsewhere. `Rife()` takes the one+ `backend()` and `executor()` name; `Rife(binary, model)` is the binary at those paths; `Rife(runner=...)` any other."""def __init__(self, binary: Path | None = None, model: Path | None = None, *, runner: object | None = None):if runner is None:if binary is not None and model is not None:runner = Binary(binary, model)+ elif (named := executor()) is not None:+ runner = RemoteRunner(executor_send(named), named)elif backend() == "ncnn":runner = Binary(*locate())else:@@ -765,14 +995,18 @@ class Rife:def counts(self) -> dict[str, int]:"""The runs it made, as a report names them: the binary's calls (`binary_calls`), or the torch backend's runs- (`runs`)."""- return {"binary_calls" if isinstance(self.runner, Binary) else "runs": self.calls}+ (`runs`) and, on an executor, the calls of its send (`executor_calls`, the probe one of them)."""+ counted = {"binary_calls" if isinstance(self.runner, Binary) else "runs": self.calls}+ if isinstance(self.runner, RemoteRunner):+ counted["executor_calls"] = self.runner.calls+ return counteddef identity(self) -> str:"""What a frame this RIFE makes depends on besides its two frames and t, as one string (`Cache` keyson it): its runner's (`identity()` of the runner — for the binary its files, read again on every call, so abinary or model changed under this very RIFE is another identity; for the torch backend what it loaded and runs- on, the same for its life) and this host's, read as it is now on every call: the augmentation every call asks+ on, the same for its life; on an executor what the executor answered as this RIFE was made, held for its life and+ enforced on every frame it makes, and the executor's name) and this host's, read as it is now on every call: the augmentation every call asksfor now (`tta()`: SPRITE_GEN_RIFE_TTA is read on every call, and another is another frame); the engine code thisprocess loaded (`CODE_SHA256`: it turns a frame into RIFE's two runs and back) and NumPy's andPillow's versions; and the machine — the OS, and `uname`'s host name, OS release and build anddiff --git a/sprite_gen/video/rife_torch.py b/sprite_gen/video/rife_torch.pyindex bb0e85c..0c26a6f 100644--- a/sprite_gen/video/rife_torch.py+++ b/sprite_gen/video/rife_torch.py@@ -27,6 +27,9 @@ binary rounds it, floor(v * 255 + 0.5). From the upload to the uint8 frame everyOn CUDA every frame runs with cuDNN deterministic, its benchmark off and its convolutions in IEEE float32 (no TF32,which PyTorch turns on by default): the same frame twice is the same bytes on one GPU, driver and PyTorch, and whatthey are is in the runner's identity. On the CPU it is the same bytes whatever the thread count.++On a machine with the GPU for a host without one, `serve` is the executor's side of `rife.RemoteRunner`: a request's+bytes to its answer's, made by the torch backend of this process, loaded once (`rife.answer`)."""from __future__ import annotations@@ -273,24 +276,21 @@ class TorchRunner:return {"kind": "rife-torch", "device": self.device, "model": self.model.name}def frame(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray]:- self._settle()return rife.frame_runs(self.run, colour, cover, t)def read(self, colour: list[np.ndarray], cover: list[np.ndarray], t: float) -> tuple[np.ndarray, np.ndarray, np.ndarray]:- self._settle()return (*rife.frame_runs(self.run, colour, cover, t), rife.pass_runs(self.run, colour, cover, t))- def _settle(self) -> None:- """The settings a frame on CUDA is made under, set again for each (other code in the process may set them)."""- if self.device == "cuda":- _deterministic()-def run(self, x: np.ndarray, y: np.ndarray, t: float, modes: tuple[str, ...] | None) -> np.ndarray:"""One run, as one call of the binary makes it (`rife.Run`): `t` as the binary is handed it (`-s`, four- decimals), the augmentation `modes` (None: `rife.tta()`). A run that fails on the device (out of its memory) is- a RIFE that fails (`DEVICE_FAILURES`, `rife.RifeUnavailable`)."""+ decimals), the augmentation `modes` (None: `rife.tta()`); on CUDA under the settings that make it the same bytes,+ set again for each run (other code in the process may set them), whoever asks for it — a frame, a read, or an+ executor's request (`serve`). A run that fails on the device (out of its memory) is a RIFE that fails+ (`DEVICE_FAILURES`, `rife.RifeUnavailable`)."""modes = rife.tta() if modes is None else modestry:+ if self.device == "cuda":+ _deterministic()return self._run(x, y, float(f"{t:.4f}"), tuple(modes))except DEVICE_FAILURES as exc:raise rife.RifeUnavailable(f"rife-torch on {self.device} failed: {exc}") from exc@@ -368,3 +368,24 @@ class TorchRunner:return frames, Nonereturn frames, [(_swap if d else (lambda c: c))(_to_canon(torch.cat([gr["flow"][k:k + 1], gr["mask"][k:k + 1]], 1), g))for gr in groups for k, (g, d) in enumerate(gr["items"])]+++# The torch backend an executor serves (`serve`): loaded by the first request, from the model and on the device this+# process names (`rife.locate_model()`, `rife.device()`), and kept for every request after.+_served: TorchRunner | None = None+++def serve(request: bytes) -> bytes:+ """The executor's side of `rife.RemoteRunner`, on the machine with the GPU: a request's bytes to its answer's+ (`rife.answer`), the frame or the read made by the torch backend of this process. It is loaded once, by the first+ request — the probe a RIFE sends as it is made, which so takes the cold start — and a backend that cannot load is+ answered in its words, to be loaded again by the next. A host reaches it with the send it names+ (SPRITE_GEN_RIFE_EXECUTOR), the same sprite-gen on both sides."""+ return rife.answer(request, _loaded)+++def _loaded() -> TorchRunner:+ global _served+ if _served is None:+ _served = TorchRunner(rife.locate_model(), rife.device())+ return _served