Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix(video): the size hold and the gait fallback scale the same frames on every machine
Soohong Kim0e5c27b57e4e
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 8de206c..b0323e1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,8 +4,10 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the## Unreleased-- **Fix: `video-loop --anchor motion-auto` reads the same motion from the same pixels on every machine.** A patch of its region search that sits on a flat fill matches equally at several offsets; the FFTs that score them round differently between numpy builds (about 1e-14), and that rounding chose the offset. From the same frames a synthetic walk was tracked differently on Linux than on macOS, its distance matrix differed in most entries, and `tests/video/test_loop_prop_seam_held.py` failed on CI only (first choice popping 2.4 there, 11.7 on macOS). Now costs within 1e-9 of the lowest are one match (`auto_motion.MATCH_TIE`) and the offset nearest the centre the search was put on is taken; regions whose qualities tie on that grain are ranked by their box; and the drift lines that move the analysis frames are least-squares lines with exactly rounded sums (`auto_motion._line`), not `np.polyfit`, whose LAPACK solve rounded differently and tipped bilinear samples of those frames. The tracked offsets, the analysis frames and the distance matrix are now bit-identical on macOS arm64 and Linux x86-64; only the last digits of reported costs and scores differ. Footage whose patches never tie is cut as before: the same regions, offsets and cut, its analysis translations moved only in their last bits (about 1e-14), which can tip a bilinear sample and move a score in its sixth decimal. Drawn, flat-filled frames, where ties are the rule, are tracked — and may be cut — otherwise than v2.36.0 tracked them on any one machine.+- **Fix: `video-loop --anchor motion-auto` reads the same motion from the same pixels on every machine.** A patch of its region search that sits on a flat fill matches equally at several offsets; the FFTs that score them round differently between numpy builds (about 1e-14), and that rounding chose the offset. From the same frames a synthetic walk was tracked differently on Linux than on macOS, its distance matrix differed in most entries, and `tests/video/test_loop_prop_seam_held.py` failed on CI only (first choice popping 2.4 there, 11.7 on macOS). Now costs within 1e-9 of the lowest are one match (`auto_motion.MATCH_TIE`) and the offset nearest the centre the search was put on is taken; regions whose qualities tie on that grain are ranked by their box; and the drift lines that move the analysis frames are least-squares lines with exactly rounded sums (`auto_motion._line`, now `sprite_gen.util.lsq.line`), not `np.polyfit`, whose LAPACK solve rounded differently and tipped bilinear samples of those frames. The tracked offsets, the analysis frames and the distance matrix are now bit-identical on macOS arm64 and Linux x86-64; only the last digits of reported costs and scores differ. Footage whose patches never tie is cut as before: the same regions, offsets and cut, its analysis translations moved only in their last bits (about 1e-14), which can tip a bilinear sample and move a score in its sixth decimal. Drawn, flat-filled frames, where ties are the rule, are tracked — and may be cut — otherwise than v2.36.0 tracked them on any one machine.- **`tests/video/test_auto_motion.py`** perturbs every FFT correlation by up to 1e-12 of its largest value, seed by seed: the tied offset taken on the flat torso, the regions found and every tracked offset are the same under all of them (under v2.36.0 eight seeds took six offsets, 14 px apart, and four seeds found four different region pairs), and the drift line is `np.polyfit`'s to 1e-12 and bit-identical in any order of its points. `test_a_held_part_that_swings_with_the_step_is_cut_one_step_long` draws its pennant on a 53-frame beat (phase 0.75): its first choice pops 12.6, no other candidate's top over 8.5, the cut taken closes below the hand at 0.0; on v2.35.0 with this fix it fails where v2.35.0 reads nothing below the top.+- **Fix: the size hold and the gait fallback of `video-loop --anchor motion-auto` scale the same frames on every machine.** Both scale every frame back about a foot point read off a fitted line, and size the room the frames need from it. The line was `np.polyfit`'s, whose LAPACK solve rounds differently between builds (Accelerate on macOS, OpenBLAS on Linux; on the 10-05 synthetic grid their lines lay a few 1e-13 px apart). Feet that stand on the frame's bottom edge in every frame sit exactly where that rounding decides: on a synthetic walker growing 3 %, `np.polyfit` on Linux put them 4 units in the last place past the edge, and with its coefficients moved by up to 1e-12 of themselves, four seeds in eight gave the frames 2 px of room below and four none — another frame size, and so other frames for the cut search. The lines are now least-squares lines with exactly rounded sums (`sprite_gen.util.lsq`, the line `auto_motion` already drew its analysis drift with, moved there): feet on the edge read exactly the edge, and the foot point, the room and every held pixel are the same under any rounding. A clip the hold leaves as filmed and the fallback does not scale is cut as before.+- **`tests/video/test_gait_fallback.py`** holds a synthetic walker whose feet stand on the frame's bottom edge with `np.polyfit`'s coefficients moved by up to 1e-12 of themselves, seed by seed: the foot point, the room and every held pixel are the same under all of them (under v2.36.0 eight seeds gave eight foot points and two paddings); a walker that neither moves nor grows reads its box's foot point and height to the bit. `tests/test_lsq.py` checks the line against `np.polyfit` and in any order of its points (moved from `test_auto_motion.py`).## v2.36.0 - A slow walk cut one step long is named, and cut again on a count; a staff that closes only on two steps is cut as one cyclediff --git a/docs/loop-review.md b/docs/loop-review.mdindex 3b266b2..1c8cb0c 100644--- a/docs/loop-review.md+++ b/docs/loop-review.md@@ -92,9 +92,10 @@ box. Before this the rounding chose, and a synthetic walk was cut at another staLinux than on macOS from the same pixels.For selection only, the measured horizontal trajectory is locally fitted and the-vertical linear trend is removed (least-squares lines with exactly rounded sums, not-np.polyfit, whose LAPACK solve rounds differently between machines: the line moves the-analysis frames by a fraction of a pixel, and its last bit can tip a bilinear sample). Local lag minima can then identify a repeat in+vertical linear trend is removed (least-squares lines with exactly rounded sums,+`sprite_gen.util.lsq`, not np.polyfit, whose LAPACK solve rounds differently between machines:+the line moves the analysis frames by a fraction of a pixel, and its last bit can tip a bilinear+sample). Local lag minima can then identify a repeat ina clip with changing drift or cadence. Selection requires repeat depth, activemotion around both boundaries, and a supported doubled recurrence when a shortstep falls below the gait floor. Candidate ranking also penalizes drift that a@@ -132,7 +133,8 @@ back here if the first search found no cycle.)- **It walked toward the camera, or away from it.** Asked to walk in place, a front walksometimes comes closer, and the body grows through the clip, so the same pose never matches- itself in size. A straight-line fit of the subject's opaque height measures it; at 3 % or+ itself in size. A straight-line fit of the subject's opaque height measures it (the exact line,+ `sprite_gen.util.lsq`, as for the size hold); at 3 % ormore over the clip (`SCALE_DRIFT_MIN`) every frame is scaled back to the first frame's fittedheight about its fitted foot point, with its coverage and its colour mapped apart(`transform_cell`, [video-pipeline.md](video-pipeline.md) "Cells") so the soft edge keeps itsdiff --git a/docs/video-pipeline.md b/docs/video-pipeline.mdindex d6f1c48..d6b3089 100644--- a/docs/video-pipeline.md+++ b/docs/video-pipeline.md@@ -823,7 +823,11 @@ and when the feet also rose in the frame (a back walk going away toward the horilands above the frame. Every frame is first widened by the room the furthest one reaches pasteach edge (`gait_fallback.undo_padding`, the same for every frame; `padding_ltrb` in thereport), so no part is lost. A frame that needs no room keeps its size and bytes. The gait-fallback's own scale-back (below) widens the same way.+fallback's own scale-back (below) widens the same way. The foot point is a least-squares line+with exactly rounded sums (`sprite_gen.util.lsq`, as are the fallback's fitted heights), the same to+the last bit on every machine. With np.polyfit, whose last bit differs between LAPACK builds, feet+that stand on the frame's bottom edge read a hair past it or short of it, and whether the frames+got 2 px of room below — and so the frames the cut is searched on — turned on that bit.Gates, all fail-loud: no period (profile flat, below the recorded `periodicity_min`), loop seam ratioabove `--seam-max` (2.0), GIF/WebP re-opened and checked (frame count, `loop=0`,diff --git a/sprite_gen/util/lsq.py b/sprite_gen/util/lsq.pynew file mode 100644index 0000000..dc289d6--- /dev/null+++ b/sprite_gen/util/lsq.py@@ -0,0 +1,25 @@+# SPDX-License-Identifier: Apache-2.0+"""The least-squares line, the same to the last bit on every machine.++np.polyfit solves through LAPACK, and LAPACK builds round differently (Accelerate on macOS,+OpenBLAS on Linux): from the same points their lines lie a few 1e-13 px apart. Where a line+moves pixels or meets a threshold, that last bit can decide a resampled pixel's 8-bit level or+which side of zero a value falls on. Here the sums are exactly rounded (`math.fsum`) and the rest+is a fixed sequence of IEEE operations, so the line depends on the points alone: not on the+machine, nor on the order the points are given in. The lines that move the motion analysis's+frames (`auto_motion.analyse`) and the gait fallback's height and foot point+(`gait_fallback._trend`) are this line.+"""+from __future__ import annotations++import math+++def line(t, values) -> tuple[float, float]:+ """(slope, intercept) of the least-squares line through (t, values), t whole numbers."""+ t = [int(v) for v in t]+ values = [float(v) for v in values]+ n, st, stt = len(t), sum(t), sum(v*v for v in t)+ sv, stv = math.fsum(values), math.fsum(a*b for a, b in zip(t, values))+ det = n*stt-st*st+ return (n*stv-st*sv)/det, (stt*sv-st*stv)/detdiff --git a/sprite_gen/video/auto_motion.py b/sprite_gen/video/auto_motion.pyindex 8edef23..453eacb 100644--- a/sprite_gen/video/auto_motion.py+++ b/sprite_gen/video/auto_motion.py@@ -7,10 +7,9 @@ fine match. The caller must still gate the rendered, uncorrected output."""from __future__ import annotations-import math-from PIL import Imagefrom sprite_gen._deps import np+from sprite_gen.util import lsqfrom sprite_gen.video import motion_anchorANALYSIS_EDGE = 184@@ -206,20 +205,6 @@ def discover(frames: list[Image.Image], *, reference_index: int) -> tuple[list,return regions, report-def _line(t, values):- """(slope, intercept) of the least-squares line through (t, values), t whole numbers.-- Written out with exactly rounded sums rather than np.polyfit, whose LAPACK solve rounds- differently from machine to machine: the line moves the analysis frames by a fraction of a- pixel, and a last-bit change there can tip a bilinear sample to the next 8-bit level."""- t = [int(v) for v in t]- values = [float(v) for v in values]- n, st, stt = len(t), sum(t), sum(v*v for v in t)- sv, stv = math.fsum(values), math.fsum(a*b for a, b in zip(t, values))- det = n*stt-st*st- return (n*stv-st*sv)/det, (stt*sv-st*stv)/det--def analyse(frames: list[Image.Image], *, fps: float) -> tuple[np.ndarray, np.ndarray, dict]:_, report = discover(frames, reference_index=len(frames)//2)size = tuple(report['analysis_size'])@@ -232,10 +217,10 @@ def analyse(frames: list[Image.Image], *, fps: float) -> tuple[np.ndarray, np.ndtrajectory = []for k in range(n):js = np.arange(max(0, k-radius), min(n, k+radius+1))- trajectory.append(_line(js-k, positions[js, 0])[1])+ trajectory.append(lsq.line(js-k, positions[js, 0])[1])trajectory = np.asarray(trajectory)trajectory -= trajectory[n//2]- dy_slope = _line(range(n), positions[:, 1])[0]+ dy_slope = lsq.line(range(n), positions[:, 1])[0]shifts = np.column_stack((trajectory, dy_slope*(np.arange(n)-n//2)))pad = int(np.ceil(np.abs(shifts).max()))+4normalized = []diff --git a/sprite_gen/video/gait_fallback.py b/sprite_gen/video/gait_fallback.pyindex 156e78f..93e16cd 100644--- a/sprite_gen/video/gait_fallback.py+++ b/sprite_gen/video/gait_fallback.py@@ -33,6 +33,7 @@ from pathlib import Pathfrom PIL import Imagefrom sprite_gen._deps import np+from sprite_gen.util import lsqfrom sprite_gen.util.resample import transform_cell# Height change over the clip, from the fitted trend, at which the frames are scaled back.@@ -61,11 +62,17 @@ def subject_boxes(frames: list[Image.Image]) -> np.ndarray:def _trend(values: np.ndarray) -> np.ndarray:+ """The least-squares line through the known values, read at every frame. It is the exact line+ (`sprite_gen.util.lsq`), not np.polyfit's, whose last bit differs from machine to machine: the+ foot point and height it gives scale the frames back and size the room they need, so that bit+ could tip a resampled pixel, and feet standing on the frame's edge read a hair past it — on+ which side decided whether the frames got 2 px of room."""t = np.arange(len(values), dtype=np.float64)known = ~np.isnan(values)if known.sum() < 2:return np.full(len(values), np.nanmean(values) if known.any() else np.nan)- return np.polyval(np.polyfit(t[known], values[known], 1), t)+ slope, intercept = lsq.line(t[known], values[known])+ return slope*t+interceptdef scale_drift(frames: list[Image.Image]) -> dict:diff --git a/tests/test_lsq.py b/tests/test_lsq.pynew file mode 100644index 0000000..905e27e--- /dev/null+++ b/tests/test_lsq.py@@ -0,0 +1,26 @@+# SPDX-License-Identifier: Apache-2.0+"""The exact least-squares line (`sprite_gen.util.lsq`)."""+import numpy as np++from sprite_gen.util import lsq+++def test_line_is_the_least_squares_line_and_the_same_in_any_order():+ """np.polyfit's line, but written out with exactly rounded sums, so the order the points are+ summed in — which differs between machines' LAPACK — cannot reach its last bit."""+ rng = np.random.default_rng(3)+ t = np.arange(-12, 13)+ values = rng.normal(40, 9, len(t)).round(1)*0.7+ slope, intercept = lsq.line(t, values)+ np.testing.assert_allclose([slope, intercept], np.polyfit(t, values, 1), rtol=0, atol=1e-12)+ for seed in range(20):+ order = np.random.default_rng(seed).permutation(len(t))+ assert lsq.line(t[order], values[order]) == (slope, intercept)+++def test_a_line_through_equal_whole_or_half_pixels_is_that_value():+ """Box edges are whole pixels and their middles half pixels: a line through equal ones is flat+ at exactly that value."""+ t = np.arange(73)+ assert lsq.line(t, np.full(73, 640.0)) == (0.0, 640.0)+ assert lsq.line(t, np.full(73, 171.5)) == (0.0, 171.5)diff --git a/tests/video/test_auto_motion.py b/tests/video/test_auto_motion.pyindex ef338aa..a45fc6e 100644--- a/tests/video/test_auto_motion.py+++ b/tests/video/test_auto_motion.py@@ -141,19 +141,6 @@ def test_region_discovery_is_the_same_whatever_the_rounding(monkeypatch):assert len(seen) == 1-def test_line_is_the_least_squares_line_and_the_same_in_any_order():- """analyse's drift line: np.polyfit's, but written out with exactly rounded sums, so the order the- points are summed in — which differs between machines' LAPACK — cannot reach its last bit."""- rng = np.random.default_rng(3)- t = np.arange(-12, 13)- values = rng.normal(40, 9, len(t)).round(1)*0.7- slope, intercept = auto_motion._line(t, values)- np.testing.assert_allclose([slope, intercept], np.polyfit(t, values, 1), rtol=0, atol=1e-12)- for seed in range(20):- order = np.random.default_rng(seed).permutation(len(t))- assert auto_motion._line(t[order], values[order]) == (slope, intercept)--def test_region_discovery_refuses_blank_input():with pytest.raises(ValueError, match='visible foreground'):auto_motion.discover([Image.new('RGBA', (100, 100)) for _ in range(6)], reference_index=0)diff --git a/tests/video/test_gait_fallback.py b/tests/video/test_gait_fallback.pyindex 589b4d2..a8cfcba 100644--- a/tests/video/test_gait_fallback.py+++ b/tests/video/test_gait_fallback.py@@ -1,4 +1,5 @@"""The gait fallback: a slow walk and a walk toward the camera, on synthetic frames."""+import hashlibimport jsonimport math@@ -92,6 +93,52 @@ def test_drift_is_measured_on_the_fitted_height_and_undone_about_the_feet():assert abs(gait_fallback.scale_drift([walker(k) for k in range(73)])['drift']) < 0.02+def other_lapack(monkeypatch, seed):+ """np.polyfit as another machine's LAPACK might solve it: every coefficient off by up to 1e-12 of+ itself, seed by seed. Machines differ by less (on the 10-05 grid, lines a few 1e-13 px apart);+ a line read exactly is unmoved by any of it."""+ rng = np.random.default_rng(seed)+ polyfit = np.polyfit++ def solved_elsewhere(x, y, deg, *args, **kwargs):+ coef = polyfit(x, y, deg, *args, **kwargs)+ return coef*(1+rng.uniform(-1e-12, 1e-12, np.shape(coef)))+ monkeypatch.setattr(np, 'polyfit', solved_elsewhere)+++def test_a_body_that_keeps_its_place_and_size_reads_them_exactly():+ """A line through equal values is that value: the fitted foot point and height of a walker that+ neither moves nor grows are its box's, to the bit, so nothing about them is left to rounding."""+ frames = [walker(0)]*73+ x0, y0, x1, y1 = gait_fallback.subject_boxes(frames[:1])[0]+ measured = gait_fallback.scale_drift(frames)+ assert np.all(measured['foot_y'] == y1) and np.all(measured['foot_x'] == (x0+x1)/2)+ assert np.all(measured['height'] == y1-y0) and measured['drift'] == 0+++def test_the_size_hold_is_the_same_whatever_the_lapack_rounding(monkeypatch):+ """Feet that stand on the frame's bottom edge in every frame: the hold scales about them, so+ nothing reaches past that edge and no room is added below. The foot point is read off a fitted+ line; np.polyfit's last bit put it a hair past the edge or short of it, and the frames got 2 px+ of room below or none (v2.36.0). On the exact line the foot point, the room and every held pixel+ are the same under any rounding."""+ full = [walker(k, grow=0.03) for k in range(73)]+ floor = int(np.nanmax(gait_fallback.subject_boxes(full)[:, 3]))+ frames = [im.crop((0, 0, im.width, floor)) for im in full]+ assert np.all(gait_fallback.subject_boxes(frames)[:, 3] == floor)+ seen = set()+ for seed in range(8):+ other_lapack(monkeypatch, seed)+ measured = gait_fallback.cycle_drift(frames, min_lag=12, max_lag=36)+ pad = gait_fallback.undo_padding(frames, measured)+ held = gait_fallback.undo_scale(frames, measured, pad=pad)+ seen.add((pad, measured['foot_x'].tobytes(), measured['foot_y'].tobytes(),+ hashlib.sha256(b''.join(im.tobytes() for im in held)).hexdigest()))+ monkeypatch.undo()+ assert len(seen) == 1+ assert next(iter(seen))[0][3] == 0++def test_a_clip_the_first_search_cuts_is_cut_without_the_fallback(tmp_path):code, report, output = run(tmp_path, [walker(k) for k in range(73)])assert code == 0