Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

fix(video-loop): motion-auto searches a walk after the opening frames its anchor cannot track, as after a lead-in, where it refused the clip before the search

Soohong Kimd99ca7c94937
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 09b4af1..2d0fa9e 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -78,6 +78,10 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the
- the ledger counts each cell, and the warning lines and the stdout JSON say what is in it.
`tests/video/test_follow_inspect.py` now names `--region-mask`, `--region-rim`, `--region-shrink` and `--region-crease` among the arguments `video-sway` takes and `video-sway-inspect` does not.
+**Upgrading from 3.5.0: a walk or run whose first frames the automatic motion anchor cannot track is now searched after them, as after a lead-in, where it was refused; every clip the anchor tracks from its first frame keeps every cut and byte.**
+
+- **Fixed: `video-loop --anchor motion-auto` no longer refuses a walk or run for its opening frames alone.** Where `auto_motion.discover` finds no two regions through the whole clip (`automatic motion anchor needs two separated regions trackable throughout the clip`, or `found no stable textured region`), the analysis is read again on the clip after the fewest frames it opens on, fewer than the window's shortest cut, with which the anchor finds its two regions through the rest (`auto_motion.analyse_after_opening`, new exception `auto_motion.RegionsUnfound`). A clip filmed from a still opens on that still and on the standing pose the body sets off from, and the regions the middle of the clip shows may be out of the anchor's reach there; the refusal came before the search, so the gait fallback never ran. Those frames are left out of the search as a lead-in is (`lead_in.search_from`, every frame number the clip's own), recorded in the report's new `untracked_opening` (`frames`, `first`, `shorter_than`, `refused`) and, once a cut is found, a warning line `video-loop: warning: the automatic motion anchor found its two regions from frame <k> on, not from the clip's first frames: ...` that does not repeat the refusal's words. Where no such opening leaves the clip tracked, it is refused in the same words as before, followed by ` — nor through the clip after any of its first 1 to <n> frames left out (an opening shorter than the shortest cut, <n + 1> frames)`; where the cut the search took holds no two regions when its frames are read again for the correction, the refusal now ends ` — on the cut the search took (frames <a>-<b>)`. Tests: `tests/video/test_auto_motion_opening.py` (a synthetic walker whose first three frames carry its head and torso 90 px forward: refused as filmed before, now cut after them with every frame number the clip's own; the same walker without them unchanged; the lean again mid-clip refused with the new words).
+
## v3.5.0 - Widened square canvas framing for side walk and run stride and trailing parts, measured background matte cutout for flat backgrounds of any colour, and failure when a cutout removes nothing
**Upgrading from 3.4.0: `video-set` now frames a walk or run filmed from the side on a square widened by the room its still lacks, and a background the chroma engine did not recognise, which used to come out with nothing removed, is now cut out.** A still that leaves less than `GAIT_ROOM` clear in front of its silhouette (0.15 of the silhouette's height) or behind it (0.25) gains the columns it lacks on that side; the square gains the same number of rows above the still. A still that leaves the room keeps its canvas to the byte. Every other state, a front, back or diagonal view, `--shape` and `--fit tight` are framed as before. Each side walk or run's `canvas.report.json`, and its item in `set.report.json`, gains `room`. `video-canvas` gains `--view` and reads the room only with `--view side`. Backgrounds the chroma engine already recognised key byte for byte as before. A pure or recognised magenta/green background still goes to the chroma engine and a bright achromatic one to the white matte, with the same bytes.
diff --git a/docs/loop-review.md b/docs/loop-review.md
index 63ffa8c..2302921 100644
--- a/docs/loop-review.md
+++ b/docs/loop-review.md
@@ -170,7 +170,13 @@ The script searches a grid in the upper part of the foreground for two separated
textured regions. It ranks their normalized cross-correlation across time and
requires both to track throughout the sequence. These are stable image regions,
not semantic detections of a face, torso, species or limb. Untrackable patches are
-counted as rejected candidates; absence of a usable pair is an explicit failure.
+counted as rejected candidates; absence of a usable pair is an explicit failure. Where the pair is
+absent from the whole clip, the clip is read once more after the fewest frames it opens on, fewer than
+the window's shortest cut, with which a pair tracks through the rest (`auto_motion.analyse_after_opening`):
+a clip filmed from a still opens on that still and on the standing pose the body sets off from, where the
+regions of the walk may be out of reach. Those frames are left out of the search as a lead-in is
+([video-pipeline.md](video-pipeline.md) section 4), the report says so in `untracked_opening` and a
+warning line, and where no such opening leaves a pair tracked the clip is refused as before.
The correlation at every offset of a search window comes out of FFTs: per channel,
the masked patch sum, its squared sum and its product with the centered reference are
window correlations, so the masked patch is never copied out once per offset. The
diff --git a/sprite_gen/video/auto_motion.py b/sprite_gen/video/auto_motion.py
index 083184e..8e1c1ca 100644
--- a/sprite_gen/video/auto_motion.py
+++ b/sprite_gen/video/auto_motion.py
@@ -19,6 +19,12 @@ SAMPLE_COUNT = 9
MATCH_TIE = 1e-9
+class RegionsUnfound(ValueError):
+ """`discover` found no two regions to track: none stable and textured, or not two apart that track throughout
+ the frames. Every other refusal of the anchor (too few frames, a frame without the subject, a search that
+ reaches its edge on the cut) stays a plain ValueError."""
+
+
def correct_cycle(frames, regions, *, coarse_dx):
"""Do not turn an uncertain fine match into a potentially damaging shift.
@@ -204,7 +210,7 @@ def discover(frames: list[Image.Image], *, reference_index: int) -> tuple[list,
# rounding; read on MATCH_TIE's grain, such a tie is ordered by the box, the same on every machine.
candidates.sort(key=lambda row: (round(row['quality']/MATCH_TIE), row['box']))
if not candidates or candidates[0]['quality'] > .8:
- raise ValueError("automatic motion anchor found no stable textured region")
+ raise RegionsUnfound("automatic motion anchor found no stable textured region")
chosen = []
validated = []
for row in candidates:
@@ -226,7 +232,7 @@ def discover(frames: list[Image.Image], *, reference_index: int) -> tuple[list,
if len(chosen) == 2:
break
if len(chosen) != 2:
- raise ValueError("automatic motion anchor needs two separated regions trackable throughout the clip")
+ raise RegionsUnfound("automatic motion anchor needs two separated regions trackable throughout the clip")
ordered = sorted(zip(chosen, validated), key=lambda pair: pair[0]['box'][1])
chosen = [pair[0] for pair in ordered]
validated = [pair[1] for pair in ordered]
@@ -280,3 +286,22 @@ def analyse(frames: list[Image.Image], *, fps: float) -> tuple[np.ndarray, np.nd
report['analysis_only_translation_xy'] = (shifts*scale).tolist()
report['analysis_only'] = True
return distances, trajectory, report
+
+
+def analyse_after_opening(frames: list[Image.Image], *, fps: float, shorter_than: int,
+ unfound: RegionsUnfound) -> tuple[int, tuple[np.ndarray, np.ndarray, dict]]:
+ """`analyse` on `frames` after the fewest frames they open on, fewer than `shorter_than`, with which `discover`
+ finds its two regions through the rest: the opening (how many frames) and `analyse`'s three, on the frames after it.
+
+ A clip filmed from a still opens on that still, drawn as the still was and not as the video draws the walk, and on
+ the standing pose the body sets off from; the regions the middle of the clip shows may be nowhere in reach there
+ (`unfound`, the refusal on the whole clip). Left out as a lead-in is, those frames are no part of any cut, and one
+ fewer than `shorter_than` (the shortest cut the search takes) holds no cycle the search could have cut. Where every
+ such opening leaves the clip untracked, RegionsUnfound in `unfound`'s words and what was tried."""
+ for opening in range(1, min(shorter_than, len(frames) - 5)):
+ try:
+ return opening, analyse(frames[opening:], fps=fps)
+ except RegionsUnfound:
+ continue
+ raise RegionsUnfound(f"{unfound} — nor through the clip after any of its first 1 to {max(1, shorter_than - 1)} "
+ f"frames left out (an opening shorter than the shortest cut, {shorter_than} frames)") from unfound
diff --git a/sprite_gen/video/loop.py b/sprite_gen/video/loop.py
index 79ecf81..63e64d2 100644
--- a/sprite_gen/video/loop.py
+++ b/sprite_gen/video/loop.py
@@ -2083,7 +2083,23 @@ def _cut_loop(
files = gait_fallback.write_frames(held, [f.name for f in files], scratch.new())
else:
report_base["size_hold"] = {"applied": False, "why": "--size-hold off"}
- D, trajectory, analysis = auto_motion.analyse(source_frames, fps=fps)
+ try:
+ D, trajectory, analysis = auto_motion.analyse(source_frames, fps=fps)
+ except auto_motion.RegionsUnfound as unfound:
+ # No two regions track through the clip: the frames it opens on, fewer than the shortest cut, are left
+ # out as a lead-in is and the clip after them is searched (`auto_motion.analyse_after_opening`), said
+ # in the report's `untracked_opening` and, once a cut is found, a warning line; where no such opening
+ # leaves it tracked, the clip is refused as before, in the same words and what was tried.
+ opening, (D, trajectory, analysis) = auto_motion.analyse_after_opening(
+ source_frames, fps=fps, shorter_than=lo, unfound=unfound)
+ report_base["untracked_opening"] = {"frames": opening, "first": skip, "shorter_than": lo,
+ "refused": str(unfound)}
+ skip += opening
+ source_frames, files = source_frames[opening:], files[opening:]
+ window = CutWindow.of(prof, n - skip, fps, min_len, max_len)
+ lo, hi = window.lo, window.hi
+ report_base["window"] = [lo, hi]
+ report_base["lead_in"]["search_from"] = skip
report_base["automatic_motion_analysis"] = analysis
search = partial(_motion_auto, source_frames, files, D, trajectory, analysis, fps=fps, window=window, prof=prof,
steps=steps, seam_max=seam_max, seam_pop_max=seam_pop_max, scratch=scratch)
@@ -2140,6 +2156,13 @@ def _cut_loop(
else:
turning["verdict"] = "kept"
print(f"video-loop: warning: {body_turn_mod.kept(turning, cut=cut, held=holds)}", file=sys.stderr)
+ if opened := report_base.get("untracked_opening"):
+ # Not the refusal's words: a caller that reads the stderr for them reads a refusal, and this cut is found.
+ left = (f"frame {opened['first']}" if opened["frames"] == 1
+ else f"frames {opened['first']}-{opened['first'] + opened['frames'] - 1}")
+ print(f"video-loop: warning: the automatic motion anchor found its two regions from frame {skip} on, not from "
+ f"the clip's first frames: {left} left out of the search as a lead-in is (untracked_opening)",
+ file=sys.stderr)
elif cycle_mode == "once":
source_frames = [Image.open(f).convert("RGBA") for f in files]
cycle = nonloop.select_cut(D, masses, fps=fps, ending=partial(
@@ -2286,7 +2309,9 @@ def _cut_loop(
frames, anchor_regions, coarse_dx=body_wrap_offset(frames, search=motion_anchor.COARSE_SEARCH),
)
except ValueError as exc:
- error = f"video-loop: {exc}"
+ # The search found its regions on the clip; the cut's own frames, read again for the correction, may hold none.
+ on_cut = f" — on the cut the search took (frames {i}-{i + L - 1})" if isinstance(exc, auto_motion.RegionsUnfound) else ""
+ error = f"video-loop: {exc}{on_cut}"
refuse(target, out_dir, name, {**report_base, "status": "failed", "error": error, "cycle": cycle})
raise SystemExit(error) from exc
report_base["motion_anchor"] = motion
diff --git a/tests/video/test_auto_motion_opening.py b/tests/video/test_auto_motion_opening.py
new file mode 100644
index 0000000..faf4704
--- /dev/null
+++ b/tests/video/test_auto_motion_opening.py
@@ -0,0 +1,81 @@
+# SPDX-License-Identifier: Apache-2.0
+"""A walk whose first frames the automatic motion anchor cannot track is searched after them, as after a lead-in.
+
+A clip filmed from a still opens on that still and on the standing pose the body sets off from, and the two regions
+the middle of the clip shows may be out of the anchor's reach there: `--anchor motion-auto` refused the whole clip
+(`automatic motion anchor needs two separated regions trackable throughout the clip`) before its search, and the gait
+fallback after the search never ran. Here the walker's first frames carry its head and torso far forward over its legs,
+further than the anchor searches round the body's centre."""
+from __future__ import annotations
+
+import numpy as np
+import pytest
+from PIL import Image
+
+from sprite_gen.video import loop
+from tests.video import test_gait_fallback as walks
+
+LEAN = 90 # px the head and torso are carried forward: past the anchor's reach round the body's centre
+OPENING = 3
+
+
+def _shaded(k: int, *, lean: int = 0) -> Image.Image:
+ """The front walker with its head and torso shaded round (brightest in the middle, so a patch of them matches at
+ one place only), and carried `lean` px forward over its legs."""
+ image = walks.walker(k)
+ a = np.asarray(image).astype(np.float64)
+ hips = walks.FOOT[1] - 41
+ yy, xx = np.mgrid[0:a.shape[0], 0:a.shape[1]]
+ shade = 0.45 + 0.55 * np.exp(-((xx - walks.FOOT[0]) ** 2 + (yy - (walks.FOOT[1] - 85)) ** 2) / (2 * 22.0 ** 2))
+ upper = (yy < hips) & (a[:, :, 3] > 0)
+ a[upper, :3] *= shade[upper, None]
+ out = Image.fromarray(a.round().astype(np.uint8), "RGBA")
+ if not lean:
+ return out
+ top = out.crop((0, 0, out.width, hips))
+ moved = out.copy()
+ moved.paste((0, 0, 0, 0), (0, 0, out.width, hips))
+ moved.alpha_composite(top, (lean, 0))
+ return moved
+
+
+def _clip(leaning: range | tuple[int, ...]) -> list[Image.Image]:
+ return [_shaded(k, lean=LEAN if k in leaning else 0) for k in range(73)]
+
+
+def test_a_walk_the_anchor_tracks_only_after_its_opening_is_searched_after_it(tmp_path, capsys):
+ (tmp_path / "opening").mkdir()
+ code, report, _ = walks.run(tmp_path / "opening", _clip(range(OPENING)), "--size-hold", "off")
+ assert code == 0
+ opened = report["untracked_opening"]
+ assert opened["frames"] == OPENING and opened["first"] == 0
+ assert opened["refused"] == "automatic motion anchor needs two separated regions trackable throughout the clip"
+ assert opened["shorter_than"] == loop.profile_for("walk").window(73, 24.0)[0]
+ # Searched as after a lead-in: every frame number is the clip's own, none in the opening.
+ assert report["lead_in"]["frames"] == 0 and report["lead_in"]["search_from"] == OPENING
+ assert report["window"] == list(loop.profile_for("walk").window(73 - OPENING, 24.0))
+ cycle = report["cycle"]
+ assert abs(cycle["length"] - 24) <= 1 and cycle["start"] >= OPENING
+ assert all(row["start"] >= OPENING for row in cycle["candidates"])
+ err = capsys.readouterr().err
+ assert "the automatic motion anchor found its two regions from frame 3 on" in err
+ # The warning is not the refusal's line: a caller matching the stderr for it reads no refusal here.
+ assert "needs two separated regions" not in err
+
+ # The walker the anchor tracks from its first frame is searched from it, and says nothing of an opening.
+ (tmp_path / "plain").mkdir()
+ code, plain, _ = walks.run(tmp_path / "plain", _clip(()), "--size-hold", "off")
+ assert code == 0 and "untracked_opening" not in plain and plain["lead_in"]["search_from"] == 0
+
+
+def test_a_walk_the_anchor_cannot_track_after_any_short_opening_is_refused_as_before(tmp_path):
+ # The same lean again in the middle of the clip: no opening shorter than the shortest cut leaves it tracked.
+ code, report, _ = walks.run(tmp_path, _clip((0, 1, 2, 40, 41, 42)), "--size-hold", "off")
+ shortest = loop.profile_for("walk").window(73, 24.0)[0]
+ assert str(code) == (
+ "video-loop: automatic motion anchor needs two separated regions trackable throughout the clip — nor through the "
+ f"clip after any of its first 1 to {shortest - 1} frames left out (an opening shorter than the shortest cut, "
+ f"{shortest} frames)")
+ assert report["status"] == "failed" and "untracked_opening" not in report and report["lead_in"]["search_from"] == 0