Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix(video): a lead-in opens where height and size are both off the walk, not a pose
Soohong Kim6cc1443a5a13
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 722b974..e7a7245 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -7,9 +7,9 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the- **Output change: `video-canvas --shape wide` gives a walk or run no head-room.** A forced-wide canvas gave every state a jump's 35 % above the still, and above a square still that head-room set the width as well (2800 × 1575 for 1024 × 1024, 1216 columns in front of the still); a walker that small in its frame is one a video model may reframe. A walk or run forced wide now has a row of its own (`canvas.GAIT_WIDE`, read through `STATE_SHAPE_CANVAS`): 28 % in front, 20 % behind, nothing above — 1969 × 1108 for a 1024 × 1024 still. A jump, and every other state forced wide, keeps the forced-wide row (`WIDE_OVERRIDE`) and its canvas byte for byte; `--headroom` still sets the room above. [docs/video-pipeline.md](docs/video-pipeline.md#1-canvas--the-input-frame-decides-the-output-frame).- **`tests/video/test_edge_speck_lead_in.py`**: a walk or run forced wide is 1969 × 1108 with nothing above, a jump still 2800 × 1575.-- **Fix: `video-loop` leaves a walk's lead-in out of the search.** A video model may reframe a small subject in a clip's first frames — the body grows or shrinks by a third or more in under half a second, then walks at its new size. Searched with those frames, the motion analysis read the reframing as the clip's motion and lost the walk's repeat, and the gait fallback, scaling back by a line through the clip, read the step as steady growth and shrank the walk. A walk or run's sizes are now read one cycle on (`gait_fallback.cycle_drift`, the size hold's model) and the frames from the first on that are off it by 10 % or more (`LEAD_IN_MIN`) are its lead-in (`gait_fallback.lead_in`); `--cycle auto` and `periodic` — the size hold, the motion analysis, the cycle search and the gait fallback — read the clip after it as a clip of its own, window included. Every frame number the report gives is the clip's own: `cycle.start`, each candidate's `start` and `context_pair_range`, a refusal's windows too, so `--cycle fixed --start` taken from them cuts the frames they name. Every walk and run report gains `lead_in` (`frames`, `height_change`, `first_off`, `min`, `method`, `search_from`); a clip too short to read its size one cycle on has none (`why`). A cut the caller names (`fixed`, `pinned`) is the caller's frames. `cycle.coverage` counts the frames the search read. The gait fallback now scales back by the size read one cycle on, as the hold does, so a first pose settling into the walk no longer hides a body that grows from it (`gait_fallback.scale_drift` gains `method: one-cycle-on` in the report); a clip that reaches the fallback with 3 % or more of change is scaled by that model now, and its pixels and the drift its refusal quotes can differ in the last places. A walk or run with no lead-in that the first search cuts is cut as before, its strip, GIF, WebP, cycle frames and `strip.json` byte for byte. [docs/video-pipeline.md](docs/video-pipeline.md#4-loop--period-first-seam-second-then-the-gait-floor), [docs/loop-review.md](docs/loop-review.md#the-gait-fallback-a-slow-walk-or-a-walk-toward-the-camera).+- **Fix: `video-loop` leaves a walk's lead-in out of the search.** A video model may reframe a small subject in a clip's first frames — the body grows or shrinks by a third or more in under half a second, then walks at its new size. Searched with those frames, the motion analysis read the reframing as the clip's motion and lost the walk's repeat, and the gait fallback, scaling back by a line through the clip, read the step as steady growth and shrank the walk. A walk or run's sizes are now read one cycle on (`gait_fallback.cycle_drift`, the size hold's model) and a clip whose first frame's height and size (the square root of the frame's coverage) are both off it by 10 % or more, the same way (`LEAD_IN_MIN`), opens on a lead-in (`gait_fallback.lead_in`) that runs while the height stays that far off — a reframing scales the whole body, a pose such as an item held up from the second frame on changes the height alone; `--cycle auto` and `periodic` — the size hold, the motion analysis, the cycle search and the gait fallback — read the clip after it as a clip of its own, window included. Every frame number the report gives is the clip's own: `cycle.start`, each candidate's `start` and `context_pair_range`, a refusal's windows too, so `--cycle fixed --start` taken from them cuts the frames they name. Every walk and run report gains `lead_in` (`frames`, `height_change`, `first_off`, `first_off_size`, `min`, `method`, `search_from`); a clip too short to read its size one cycle on has none (`why`). A cut the caller names (`fixed`, `pinned`) is the caller's frames. `cycle.coverage` counts the frames the search read. The gait fallback now scales back by the size read one cycle on, as the hold does, so a first pose settling into the walk no longer hides a body that grows from it (`gait_fallback.scale_drift` gains `method: one-cycle-on` in the report); a clip that reaches the fallback with 3 % or more of change is scaled by that model now, and its pixels and the drift its refusal quotes can differ in the last places. A walk or run with no lead-in that the first search cuts is cut as before, its strip, GIF, WebP, cycle frames and `strip.json` byte for byte. [docs/video-pipeline.md](docs/video-pipeline.md#4-loop--period-first-seam-second-then-the-gait-floor), [docs/loop-review.md](docs/loop-review.md#the-gait-fallback-a-slow-walk-or-a-walk-toward-the-camera).- **Fix: after a lead-in, `--body-height` reads the standing height on the cut's first frame.** The clip's first frame is the still's pose as filmed, but a walk that opens on a lead-in films it at another scale: read there, the standing height scaled the walk up into the cell cap. Such a cut now reads it on its own first frame, and `strip.json` says so (`body_ref: cut-first-frame`, `body_ref_frame`, the clip frame); `video-cycle-align` rebuilds such a strip at the same height. A clip with no lead-in reads its first frame as before, and its `strip.json` is as before. [docs/video-pipeline.md](docs/video-pipeline.md#4-loop--period-first-seam-second-then-the-gait-floor).-- **`tests/video/test_edge_speck_lead_in.py`**: a walker filmed at 0.63 of its size and reframed over 9 frames: a lead-in of 5 frames, found no cycle before, now cuts its 24-frame walk after it, every frame number its own; too slow a walk after the same lead-in is refused with windows that all start after it; a fixed cut there reads its standing height on its own first frame, and `video-cycle-align` keeps it. A walker that settles from its first pose and grows 5 % reads 2.8 % on a line and was refused; read one cycle on it is scaled back and cut.+- **`tests/video/test_edge_speck_lead_in.py`**: a walker filmed at 0.63 of its size and reframed over 9 frames: a lead-in of 5 frames, found no cycle before, now cuts its 24-frame walk after it, every frame number its own; one filmed at 1.3 of its size has a lead-in too, and one that holds an item up from its second frame on has none; too slow a walk after the same lead-in is refused with windows that all start after it; a fixed cut there reads its standing height on its own first frame, and `video-cycle-align` keeps it. A walker that settles from its first pose and grows 5 % reads 2.8 % on a line and was refused; read one cycle on it is scaled back and cut.- **Fix: `video-frames` erases a speck far from the body before the edge check.** A fleck the model drew drifting across the background survives the matte as a few pale pixels; where it crossed an edge band in one frame, the edge check read it as the subject framed too tight and refused the clip, and `video-loop --anchor motion-auto`, which never cleans a frame, kept it in the loop and widened every cell to reach it. Now every keyed frame has its specks erased before anything reads it (`frames.drop_specks`): a piece of the frame — its opaque pixels, corners joining them — under 1 % of the body (`SPECK_MIN_FRACTION`, at least 8 px) and more than a tenth of the body's height from it (`SPECK_APART`). Small pieces near the body are the subject's own and stay: the outline's loose pixels, a shadow drawn under a shoe, a part of a shoe the matte cut loose. Each report row gains `specks`, how many its frame lost, and the report `specks` (`dropped`, `frames` and the rule's numbers); a frame with no speck is not written again, so its bytes are as before. `video-loop`'s own pass on a cut that is not motion-reviewed (`--anchor none|feet|body`) erases every island of solid pixels under the same 1 % as before, pixel for pixel, through the same function (a run-based labelling, about twenty times faster than the pixel-by-pixel search it replaces). [docs/video-pipeline.md](docs/video-pipeline.md#3-frames--extract-key-check-the-edges).- **`tests/video/test_edge_speck_lead_in.py`** keys a body on magenta with five pale pixels in its right edge band in one frame: before this the frames were refused as framed too tight, now the fleck is erased (`specks: 1`) and no frame touches the edge; a shadow under the feet, a loose pixel by the outline and a far block too large to be a speck are kept; `video-loop`'s own reading still erases the loose pieces. A walker on magenta with a fleck drifting far from it, keyed and cut `--anchor motion-auto`, is cut 49 px wide, the walker alone (147 px before).diff --git a/docs/video-pipeline.md b/docs/video-pipeline.mdindex d7b03ae..8810850 100644--- a/docs/video-pipeline.md+++ b/docs/video-pipeline.md@@ -841,16 +841,19 @@ or more in under half a second, then walks at its new size. Searched with thosemotion analysis reads the reframing as the clip's motion and the walk's repeat is lost; the gaitfallback, which then scaled by a line through the clip, read the step as a steady growth andshrank the walk. So the clip's sizes are read one cycle on (`gait_fallback.cycle_drift`, the-hold's model, a median over the whole clip), and the frames from the first on that are off that-model by 10 % or more (`LEAD_IN_MIN`; a walk's own frames stray from it by a head bob, a few-percent) are the lead-in (`gait_fallback.lead_in`). The search — the size hold, the motion+hold's model, a median over the whole clip). A clip opens on a lead-in when its first frame's+height and size (the square root of the frame's summed coverage) are both off that model by 10 % or+more, the same way (`LEAD_IN_MIN`): a reframing scales the whole body and moves both, a pose moves+one — an item held up from the second frame on makes the first frame shorter and no lighter, legs+spread in a side step add size and no height. The lead-in (`gait_fallback.lead_in`) runs on while+the height stays that far off, that way; a walk's own frames stray from the model by a few percent. The search — the size hold, the motionanalysis, the cycle search and the gait fallback — reads the clip after it as a clip of its own,its window included. What it finds, and what it refuses, is said in the clip's own frame numbers:every `start` and `context_pair_range` in the report's `cycle`, refused candidates too, so a cuttaken again from them (`--cycle fixed --start`) is the frames they name. The report records it onevery walk and run (`lead_in`: `frames`, 0 for none; `height_change`, the walk's height where the-lead-in ends over the first frame's, less 1; `first_off`; `search_from`, the first frame the search-read). A cut the caller names (`fixed`, `pinned`) is the caller's frames, lead-in or not. A clip+lead-in ends over the first frame's, less 1; `first_off` and `first_off_size`; `search_from`, the+first frame the search read). A cut the caller names (`fixed`, `pinned`) is the caller's frames, lead-in or not. A cliptoo short to read its size one cycle on has no lead-in (`why`). On a synthetic walker filmed at0.63 of its size and reframed over 9 frames, the search as filmed found no cycle, nor did thefallback; after its 5-frame lead-in it cuts the 24-frame walk (`tests/video/test_edge_speck_lead_in.py`).diff --git a/sprite_gen/video/gait_fallback.py b/sprite_gen/video/gait_fallback.pyindex 2fdaf95..9dda1eb 100644--- a/sprite_gen/video/gait_fallback.py+++ b/sprite_gen/video/gait_fallback.py@@ -29,10 +29,13 @@ the same reading, so both hold one size model.A clip can also open on a **lead-in**: the video model reframes a small subject in its firstframes, so the body grows (or shrinks) by a third or more in under half a second and then walks-at its new size. The one-cycle-on model, a median over the whole clip, holds the walk's size;-the frames off it by `LEAD_IN_MIN` or more from the first one on are the lead-in (`lead_in`). A-walk's own frames stray from it by a head bob and hair, a few percent; a reframing starts tens-of percent off it.+at its new size. The one-cycle-on model, a median over the whole clip, holds the walk's size. A+clip opens on a lead-in (`lead_in`) when its first frame's height and size are both off that model+by `LEAD_IN_MIN` or more, the same way: a reframing scales the whole body, so it moves both, and a+pose moves one — an item held up from the second frame on lengthens the body and adds next to+nothing to its mass, legs spread in a side step add mass and no height. The lead-in runs on while+the height stays that far off, that way; the height is the steadier of the two through a walk. A+walk's own frames stray from the model by a few percent; a reframing starts tens of percent off it."""from __future__ import annotations@@ -52,8 +55,9 @@ SCALE_DRIFT_MIN = 0.03# hair; above it a clip filmed from its first frame only grows or shrinks enough that the loop's# last frame is a different size from its first, and the loop pops at the wrap.SIZE_HOLD_MIN = 0.01-# A frame whose height is off the one-cycle-on model by this much is not yet walking at the clip's size:-# the first frames off it by this much are a lead-in (`lead_in`).+# How far off the one-cycle-on model a frame is when it is not yet walking at the clip's size: a first frame+# whose height and size are both this far off, the same way, opens a lead-in, and the frames whose height+# stays this far off, that way, are in it (`lead_in`).LEAD_IN_MIN = 0.1# The longest cycle the second search accepts, as a share of the clip and in seconds.LONG_CYCLE_FRACTION = 0.6@@ -176,26 +180,34 @@ def cycle_drift(frames: list[Image.Image], *, min_lag: int, max_lag: int) -> dicdef lead_in(frames: list[Image.Image], *, min_lag: int, max_lag: int) -> dict:- """The frames the clip opens on before its subject walks at the clip's size: from the first, every- frame whose height (`coverage_heights`) is off the one-cycle-on model (`cycle_drift`) by- LEAD_IN_MIN or more. `frames` is how many (0: none), `height_change` the model's height where the- lead-in ends over the first frame's, less 1 (+0.5: the body grew by half), `first_off` how far the- first frame is off the model. A frame with no subject ends it. A clip too short to show a cycle- twice has no model to read it on: no lead-in, and `why` says so."""+ """The frames the clip opens on before its subject walks at the clip's size. There are some when the+ first frame's height (`coverage_heights`) and size (the square root of its summed coverage) are+ both off the one-cycle-on model (`cycle_drift`; its growth, the size's own level) by LEAD_IN_MIN+ or more, the same way; from the first, every frame whose height stays that far off, that way, is+ one. `frames` is how many (0: none), `height_change` the model's height where the lead-in ends+ over the first frame's, less 1 (+0.5: the body grew by half), `first_off` and `first_off_size`+ how far the first frame's height and size are off the model. A frame with no subject ends it. A+ clip too short to show a cycle twice has no model to read it on: no lead-in, and `why` says so."""heights = coverage_heights(frames)measured = cycle_drift(frames, min_lag=min_lag, max_lag=max_lag)+ base = {"min": LEAD_IN_MIN, "method": "one-cycle-on"}if measured["lag"] is None:- return {"frames": 0, "height_change": 0.0, "first_off": None, "min": LEAD_IN_MIN, "method": "one-cycle-on",+ return {"frames": 0, "height_change": 0.0, "first_off": None, "first_off_size": None, **base,"why": "no lag in range: the clip is too short to read its size one cycle on"}model = measured["height"]+ growth = model / model[0]+ sizes = np.sqrt([float(np.asarray(frame.getchannel("A"), np.float64).sum()) / 255 for frame in frames])off = heights / model - 1+ off_size = sizes / (float(np.median(sizes / growth)) * growth) - 1count = 0- while count < len(frames) - 1 and np.isfinite(off[count]) and abs(off[count]) >= LEAD_IN_MIN:- count += 1+ way = np.sign(off[0]) if np.isfinite(off[0]) else 0.0+ if abs(off[0]) >= LEAD_IN_MIN and abs(off_size[0]) >= LEAD_IN_MIN and np.sign(off_size[0]) == way:+ while count < len(frames) - 1 and np.isfinite(off[count]) and way * off[count] >= LEAD_IN_MIN:+ count += 1change = float(model[count] / heights[0] - 1) if count and heights[0] > 0 else 0.0return {"frames": count, "height_change": round(change, 4),"first_off": round(float(off[0]), 4) if np.isfinite(off[0]) else None,- "min": LEAD_IN_MIN, "method": "one-cycle-on"}+ "first_off_size": round(float(off_size[0]), 4), **base}def undo_padding(frames: list[Image.Image], measured: dict) -> tuple[int, int, int, int]:diff --git a/tests/video/test_edge_speck_lead_in.py b/tests/video/test_edge_speck_lead_in.pyindex 9460e59..b82175f 100644--- a/tests/video/test_edge_speck_lead_in.py+++ b/tests/video/test_edge_speck_lead_in.py@@ -71,9 +71,9 @@ def test_small_pieces_near_the_body_and_a_large_one_far_from_it_are_the_subjectsimage.paste((60, 160, 60), (80, 30, 100, 60))files = _raw(tmp_path, 2, pieces)report = frames_mod.key_frames(files, tmp_path / "keyed", key="magenta")- assert [row["specks"] for row in report["rows"]] == [0, 0] and report["specks"]["dropped"] == 0keyed = Image.open(tmp_path / "keyed" / "frame-0000.png").convert("RGBA")assert keyed.getpixel((35, 92))[3] > 0 and keyed.getpixel((51, 40))[3] > 0 and keyed.getpixel((90, 45))[3] > 0+ assert [row["specks"] for row in report["rows"]] == [0, 0] and report["specks"]["dropped"] == 0def test_the_loop_cut_reads_its_loose_pieces_as_before():@@ -107,12 +107,12 @@ def test_a_motion_auto_loop_of_a_clip_with_a_fleck_is_cut_without_it(tmp_path):files.append(raw / f"frame-{k:04d}.png")image.save(files[-1])report = frames_mod.key_frames(files, tmp_path / "keyed", key="magenta")- assert all(row["specks"] == 1 for row in report["rows"])out = loop.run_loop(tmp_path / "keyed", tmp_path / "out", fps=24.0, state="walk", min_len=None, max_len=None,n_out=None, seam_max=loop.SEAM_RATIO_MAX, name="w", report_path=None, anchor="motion-auto",repair="off")walker_width = walks.walker(0).getchannel("A").getbbox()assert out["strip"]["w"] <= walker_width[2] - walker_width[0] + 2 * 8 + 8 # the walker, its margins, a little sway+ assert all(row["specks"] == 1 for row in report["rows"])# --- a walk's lead-in -----------------------------------------------------------------------------------------@@ -124,7 +124,7 @@ def zoomed(k: int, *, period: int = 24, frames: int = 9, start: float = 0.63) ->image = walks.walker(k, period=period)t = min(1.0, k / frames)s = start + (1 - start) * (1 - (1 - t) ** 2)- if s >= 1:+ if s == 1:return imageax, ay = walks.FOOTreturn image.convert("RGBa").transform(image.size, Image.Transform.AFFINE, (1 / s, 0, ax * (1 - 1 / s), 0, 1 / s, ay * (1 - 1 / s)),@@ -134,13 +134,21 @@ def zoomed(k: int, *, period: int = 24, frames: int = 9, start: float = 0.63) ->def test_a_lead_in_is_the_first_frames_off_the_walks_size():lead = gait_fallback.lead_in([zoomed(k) for k in range(73)], min_lag=12, max_lag=36)assert 4 <= lead["frames"] <= 7 and lead["first_off"] < -0.3 and 0.45 < lead["height_change"] < 0.65- shrinking = gait_fallback.lead_in([zoomed(k, start=1.5) for k in range(73)], min_lag=12, max_lag=36)- assert shrinking["frames"] >= 3 and shrinking["first_off"] > 0.3 and shrinking["height_change"] < -0.25+ assert lead["first_off_size"] < -0.3+ shrinking = gait_fallback.lead_in([zoomed(k, start=1.3) for k in range(73)], min_lag=12, max_lag=36)+ assert shrinking["frames"] >= 3 and shrinking["first_off"] > 0.25 and shrinking["height_change"] < -0.15# A walk, a walk toward the camera, a first pose that settles into the walk: none has a lead-in.for clip in ([walks.walker(k) for k in range(73)], [walks.walker(k, grow=0.2) for k in range(73)],[walks.settling(k) for k in range(73)]):found = gait_fallback.lead_in(clip, min_lag=12, max_lag=36)assert found["frames"] == 0 and abs(found["first_off"]) < gait_fallback.LEAD_IN_MIN+ # Nor a pose that changes the height alone: an item held up from the second frame on makes the first+ # frame 13 % shorter than the walk, and no lighter.+ held = [walks.walker(k) for k in range(73)]+ for image in held[1:]:+ image.paste((120, 120, 120, 255), (140, 30, 142, 48))+ found = gait_fallback.lead_in(held, min_lag=12, max_lag=36)+ assert found["frames"] == 0 and found["first_off"] < -gait_fallback.LEAD_IN_MIN and abs(found["first_off_size"]) < 0.02short = gait_fallback.lead_in([zoomed(k) for k in range(20)], min_lag=12, max_lag=36)assert short["frames"] == 0 and "too short" in short["why"]@@ -163,10 +171,10 @@ def test_a_refusal_after_a_lead_in_names_windows_in_the_clips_frames(tmp_path):# as the clip is, which `--cycle fixed --start` cuts on.code, report, _ = walks.run(tmp_path, [zoomed(k, period=200) for k in range(73)], "--size-hold", "off")assert str(code).startswith("video-loop: no periodic cycle found")- lead = report["lead_in"]["frames"]- assert lead >= 4starts = [row["start"] for row in report["cycle"]["candidates"]]- assert starts and min(starts) >= lead+ assert starts and min(starts) >= 4 # the reframing's first frames are 10 % and more off the walk's size+ lead = report["lead_in"]["frames"]+ assert lead >= 4 and min(starts) >= leaddef test_the_gait_fallback_scales_back_by_the_size_read_one_cycle_on(tmp_path):@@ -203,10 +211,10 @@ def _fixed(tmp_path: Path, clip: list[Image.Image], start: int) -> tuple[dict, ddef test_a_cut_after_a_lead_in_reads_the_standing_height_on_its_own_first_frame(tmp_path):# Read on the first frame, filmed small, the standing height scaled the walk up by half again.out, meta, keyed = _fixed(tmp_path, [zoomed(k) for k in range(73)], 12)- assert out["lead_in"]["frames"] >= 4 and out["lead_in"]["search_from"] == 0 # a named cut is the caller's frames- assert meta["body_ref"] == "cut-first-frame" and meta["body_ref_frame"] == 12assert meta["body_src_h"] == _height(keyed / "012.png") > _height(keyed / "000.png") + 30assert meta["scale"] == round(120 / meta["body_src_h"], 4)+ assert meta["body_ref"] == "cut-first-frame" and meta["body_ref_frame"] == 12+ assert out["lead_in"]["frames"] >= 4 and out["lead_in"]["search_from"] == 0 # a named cut is the caller's frames# `video-cycle-align` rebuilds the strip at the same standing height.for name in ("a", "b"):shutil.copytree(tmp_path / "out", tmp_path / name)@@ -218,9 +226,9 @@ def test_a_cut_after_a_lead_in_reads_the_standing_height_on_its_own_first_frame(def test_a_cut_of_a_clip_with_no_lead_in_reads_its_first_frame_as_before(tmp_path):out, meta, keyed = _fixed(tmp_path, [walks.walker(k) for k in range(73)], 12)- assert out["lead_in"]["frames"] == 0assert meta["body_ref"] == "first-frame" and "body_ref_frame" not in metaassert meta["body_src_h"] == _height(keyed / "000.png")+ assert out["lead_in"]["frames"] == 0# --- a walk forced wide ---------------------------------------------------------------------------------------@@ -231,11 +239,11 @@ def test_a_walk_or_run_forced_wide_gets_no_room_above():still.paste((180, 60, 30), (400, 100, 600, 1000))for state in ("walk", "run"):profile = canvas.profile_for(state, shape="wide")- assert profile is canvas.GAIT_WIDE and (profile.headroom, profile.lead, profile.trail) == (0.0, 0.28, 0.2)_, report = canvas.pad_canvas(still, profile)# 16:9 over the still's own height, the room in front and behind as forced wide gives any state.assert report["canvas"] == [1969, 1108] and report["offset"] == [394, 84]assert report["canvas"][0] == round(1024 / (1 - 0.28 - 0.2))+ assert profile is canvas.GAIT_WIDE and (profile.headroom, profile.lead, profile.trail) == (0.0, 0.28, 0.2)# A jump, or a state with no wide row of its own, keeps the jump's head-room.for state in ("jump", "idle", None):assert canvas.profile_for(state, shape="wide") is canvas.WIDE_OVERRIDE