Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix(decontam): video-frames keeps its edge band as the matte left it
Soohong Kimd92985bd37cf
diff --git a/docs/chroma-alpha.md b/docs/chroma-alpha.mdindex 6aa4501..37f8174 100644--- a/docs/chroma-alpha.md+++ b/docs/chroma-alpha.md@@ -97,11 +97,14 @@ B is the key background measured locally (mean of keyed pixels within 12 px). P| a blend within the unmix reach (`chroma.unmix_reach`, default 4) | alpha and colour || a blend deeper in the 6 px band | colour only, so a key reflection on a solid surface is recoloured rather than made see-through || a keyed pixel within 3 px of the subject that fits a blend above the background's noise floor and chains to the subject | coverage recovered: the faint sides of a strand the hard cut erased |+| `video-frames`: within its edge band (4 px of the top, left and right edges), a pixel the matte left transparent | stays transparent: the pass adds no edge contact to a clip (below) |Interior pixels are never touched, `off` returns the engine's output byte for byte, and everything is integer-exact or a fixed sequence of float64 operations (no random seed, no iteration-order dependence). The palette is a histogram k-means with a greedy weighted farthest-point seed and a fixed iteration count. There is no new dependency.Licensing: the pass is plain numpy inside this Apache-2.0 repository and installs nothing. Learned matting was considered as an optional install: BiRefNet and ViTMatte (MIT) and MODNet (Apache-2.0) are compatible, while RMBG-2.0 (non-commercial) and Robust Video Matting (GPL-3.0) cannot be dependencies. None was adopted: on the ground-truth set below, the two that were measured (BiRefNet-lite and ViTMatte-S alphas with foreground estimation or this repaint) left more key contamination than this pass, and each needs a model download.+**The frame edge.** A decoded frame's blur runs the subject on into the key as a smooth tail, and the video fit reads that tail as a faint blend at every distance, on the keyed ring just past the flank (background by the pass's own definition) as well as on the flank, so no floor tells a strand's side from it. In the interior that costs a few faint pixels in the subject's colour. Along the frame's top, left and right edges it decides a clip: `video-frames` refuses one where any visible pixel in its edge band has key colour in the source, as the key surviving the matte, so a head a few pixels under the frame edge would pass with the pass off and fail with it on. `video-frames` therefore hands the pass its edge band (`decontam_edge_band`, the gate's `EDGE_ROWS`), and there a pixel the matte left transparent stays transparent. Everywhere else the flank keeps what it recovers; on the ground-truth set below, the video fit's strand recall gain comes from it.+Two fits. `still` chooses P per pixel. `video` chooses P on a 3×3 mean of the observation, whose chroma is blurred anyway, and averages the projection alpha with a luma-only alpha where P and B differ in luma. `video-frames` uses the video fit and learns one palette per clip on the first frame, so the colours an edge may take cannot change from frame to frame.**Gold and yellow.** Gold sits between red and green, so a gold a little darker or yellower than the palette's own also fits as a warmer palette gold with some green mixed in. Up to v2.10.1 the still fit took that reading, and gold edges and small gold drops came out orange and partly see-through. Since v2.10.2 the still fit weighs the blend against the subject's own colour at the pixel's own light (the freedom the written colour already has), and a blend has to win by more than the subject's own spread: the 99th percentile of how far its confident interior sits from the palette, large for a textured gold and small for flat colour. A still's antialiased edge is at most two pixels deep, so deeper in the band a key share has to show as the key's hue: a pixel without it is the subject's own colour, and so is an edge pixel without it that has the colour of the material right behind it. Where the matte changed such a pixel, the source colour and coverage come back. The RGB matte (`cutout`'s default) needs that: it scores key tint on the channel average, which calls yellow green, so it unmixes gold as if it were a blend; the YCbCr matte (`gen`) does not. The video fit keeps its decisions, because a decoded frame's chroma is blurred and an edge pixel's colour is no evidence of what the subject is made of there.diff --git a/docs/video-pipeline.md b/docs/video-pipeline.mdindex 7c9c5fb..1ff5cc3 100644--- a/docs/video-pipeline.md+++ b/docs/video-pipeline.md@@ -345,7 +345,9 @@ anything one or two pixels wide, and despilling what is left turns thin red straorange. `video-frames --decontam palette` (also `video-set --decontam palette`) re-explainseach edge pixel as a blend of the local key background with one colour the subject owns,and writes that colour at the pixel's observed luma. It uses the video fit and one palette-per clip, learned on the first frame, so edge colours cannot flicker between palettes. The+per clip, learned on the first frame, so edge colours cannot flicker between palettes. Within+the edge band the edge gate reads, it gives no coverage to a pixel the matte left transparent,+so it adds no edge contact: a clip with none under `off` has none under `palette`. Thedefault `off` keeps frames byte-identical. Method, guards and measurements:[chroma-alpha.md](chroma-alpha.md#decontam--give-the-edge-the-subjects-own-colour-back).diff --git a/sprite_gen/frames/cutout.py b/sprite_gen/frames/cutout.pyindex 04c60bc..5057e2e 100644--- a/sprite_gen/frames/cutout.py+++ b/sprite_gen/frames/cutout.py@@ -274,7 +274,7 @@ def _matte_route(def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float | None = None,spill_min_tint: float | None = None,spill_require_hue: bool = False, decontam: str = "off", decontam_fit: str = "still",- decontam_palette: dict[str, Any] | None = None,+ decontam_palette: dict[str, Any] | None = None, decontam_edge_band: int = 0,background_key: tuple[int, int, int] | None = None) -> tuple[Image.Image, dict[str, Any]]:"""Magenta/green key background → reuse the verified `extract` chroma engine (no drift).@@ -307,7 +307,7 @@ def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float |decontam_stats: dict[str, Any] = {}if decontam != "off":extra.update(decontam=decontam, decontam_fit=decontam_fit, decontam_palette=decontam_palette,- decontam_stats=decontam_stats)+ decontam_stats=decontam_stats, decontam_edge_band=decontam_edge_band)# the colour detected above, not a second detection of the same bordersresult = remove_chroma_background(image, target, _EXTRACT_KEY_THRESHOLD, _EXTRACT_FRINGE_THRESHOLD, _EXTRACT_FRINGE_DELTA,@@ -339,6 +339,7 @@ def cutout(decontam: str = "off",decontam_fit: str = "still",decontam_palette: dict[str, Any] | None = None,+ decontam_edge_band: int = 0,) -> dict[str, Any]:"""Cut a uniform-background imported image to a clean transparent RGBA PNG.@@ -348,8 +349,9 @@ def cutout(engine default for either). `decontam` is the engine's edge decontaminationpass: "off" (the default) keeps the matte as it is; "auto" runs it on chroma routeswhere it applies and records why not elsewhere; "palette" demands it and fails where- it cannot run (the white matte has no key colour to remove). Returns a- stats dict. Raises SystemExit if the key is unknown, the background cannot be+ it cannot run (the white matte has no key colour to remove). `decontam_edge_band`+ keeps what the matte left transparent along the top, left and right edges+ transparent (`video-frames` passes its edge band). Returns a stats dict. Raises SystemExit if the key is unknown, the background cannot belocated, or any transparent pixel keeps non-zero RGB (No Silent Fallback)."""if key not in ("auto", "white", "magenta", "green"):@@ -365,7 +367,7 @@ def cutout(result, route_stats = extract_route(image, route, spill_max_fraction=spill_max_fraction,spill_min_tint=spill_min_tint, spill_require_hue=spill_require_hue,decontam=decontam, decontam_fit=decontam_fit,- decontam_palette=decontam_palette)+ decontam_palette=decontam_palette, decontam_edge_band=decontam_edge_band)else:if decontam == "palette":raise SystemExit(f"cutout: --decontam {decontam} needs a chroma key background (magenta/green); "diff --git a/sprite_gen/frames/decontam.py b/sprite_gen/frames/decontam.pyindex 065f0ac..53f2ea4 100644--- a/sprite_gen/frames/decontam.py+++ b/sprite_gen/frames/decontam.py@@ -28,6 +28,13 @@ Guards, in order of the decision they make:- *Background flank.* Keyed pixels within `FLANK_PX` of the subject may regaincoverage (the faint sides of a strand the hard cut erased), above a noisefloor measured on the background itself, and only when chained to the subject.+- *Frame edge.* Within `edge_band` of the frame's top, left and right edges (the+ band `video-frames` reads as the subject touching the frame or the key surviving+ the matte), a pixel the matte left transparent stays transparent. A decoded+ frame's blur runs the subject on into the key as a smooth tail that reads as a+ faint blend at every distance, the background past the flank as well, so a head+ a few pixels under the frame edge would get a faint flank there that no floor+ tells from the tail, and that flank alone would fail a clip the matte passes.- *Unexplained.* A pixel no palette colour explains (a small gem of a colour theinterior never shows) keeps the engine's bytes. In the still fit the blend mustalso reproduce the observed luma, so outline ink darker than any mix of a@@ -305,7 +312,7 @@ def palette_from_stats(stats: dict[str, Any]) -> dict[str, Any]:def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndarray,chroma_key: tuple[int, int, int], *, fit: str = "still", alpha_depth: int = 4,palette: dict[str, Any] | None = None, mode: str = "palette",- source_alpha: np.ndarray | None = None) -> tuple[np.ndarray, dict[str, Any]]:+ source_alpha: np.ndarray | None = None, edge_band: int = 0) -> tuple[np.ndarray, dict[str, Any]]:"""Re-explain the edge of an already keyed image. Returns (RGBA uint8, stats).`source_rgb` (H, W, 3) is the frame as generated and `source_alpha` (H, W) its own@@ -313,6 +320,8 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndarthe engine's hard-cut background (transparent input included). Interior pixels(deeper than `BAND_PX`) are returned byte-identical.`palette` is a `subject_palette` result to reuse (a clip's); None learns this frame's.+ `edge_band` (px) keeps what the matte left transparent along the top, left and right+ edges transparent (*Frame edge*); 0 leaves the frame edge to the pass like anywhere else.`mode="auto"` returns the input unchanged, with the reason in the stats, where thepass does not apply; `mode="palette"` raises SystemExit there instead."""@@ -340,6 +349,11 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar# the still fit reads an edge pixel's colour as evidence of the subject's own material; a decoded# video frame's chroma is blurred, so there the colour is no such evidence and the fit decides alonematerial_test = fit == "still"+ # *Frame edge*: what the matte left transparent along the top, left and right edges stays so+ held = np.zeros(keyed_mask.shape, dtype=bool)+ if edge_band > 0:+ held[:edge_band] = held[:, :edge_band] = held[:, -edge_band:] = True+ held &= out[..., 3] == 0far_background = keyed_mask & ~flankbackground = local_background(rgb, far_background, keyed_mask, BACKGROUND_RADIUS)@@ -425,13 +439,13 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndarsettled = hueless & (deep | ((behind_weight[start:start + len(r)] > 0) & (behind_distance <= margin)))else:settled = np.zeros(len(r), dtype=bool)- is_blend = explained & (alpha < 1.0) & (in_flank | (miss + margin < material)) & ~settled+ is_blend = explained & (alpha < 1.0) & (in_flank | (miss + margin < material)) & ~settled & ~held[r, c]# a tint: displaced from its colour toward the key clearly more than the fit misses by, but# not by enough to prove partial coverage, and by more than shading any palette colour# accounts for. Recoloured, coverage untouched.shift = (1.0 - alpha) * np.sqrt((line * line).sum(-1))is_tint = (explained & ~is_blend & ~in_flank & (shift >= TINT_SHIFT) & (shift >= TINT_RATIO * miss)- & ~own_colour & ~settled)+ & ~own_colour & ~settled & ~held[r, c])# coverage: refit within the unmix reach and on the flank; colour only deeper in the bandrefit = (in_flank | (depth[r, c] <= alpha_depth)) & ~is_tintfloor = np.maximum(MIN_RECOVERED_ALPHA, NOISE_SIGMAS * sigma_luma / np.maximum(np.abs((chosen - bg) @ LUMA), 1e-6))@@ -447,7 +461,8 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndarblend[start:start + len(r)] = is_blend | is_tinttinted[start:start + len(r)] = is_tintunexplained[start:start + len(r)] = ~explained- own[start:start + len(r)] = material_test & ~is_blend & ~is_tint & ~in_flank & ((material <= margin) | settled)+ own[start:start + len(r)] = (material_test & ~is_blend & ~is_tint & ~in_flank & ~held[r, c]+ & ((material <= margin) | settled))alpha8 = np.clip(np.round(alpha_new * 255), 0, 255).astype(np.uint8)colour8 = np.clip(np.round(colour_new), 0, 255).astype(np.uint8)@@ -487,6 +502,7 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar"band_px": BAND_PX,"alpha_depth": int(alpha_depth),"flank_px": FLANK_PX,+ "edge_band": int(edge_band),"palette_size": int(len(palette_rgb)),"palette_source": "frame" if palette is None else "given","palette": [[int(v) for v in colour] for colour in palette_rgb],diff --git a/sprite_gen/frames/extract.py b/sprite_gen/frames/extract.pyindex 4569856..ec126f6 100644--- a/sprite_gen/frames/extract.py+++ b/sprite_gen/frames/extract.py@@ -542,6 +542,7 @@ def remove_chroma_background(decontam_fit: str = "still",decontam_palette: dict[str, Any] | None = None,decontam_stats: dict[str, Any] | None = None,+ decontam_edge_band: int = 0,) -> Image.Image:"""Key `chroma_key` out of `image` (hard cut + soft-alpha fringe unmix + trapped-spill despill).@@ -553,7 +554,8 @@ def remove_chroma_background(`decontam="palette"` re-explains the edge afterwards with the subject's owncolours (`sprite_gen.frames.decontam`; "auto" does so where it applies and reportswhy not elsewhere); "off" (the default) returns exactly what the passes above produce. `decontam_stats`, when given, receives what- that pass did; `decontam_palette` reuses a palette learned elsewhere.+ that pass did; `decontam_palette` reuses a palette learned elsewhere, and+ `decontam_edge_band` keeps the frame edge as the matte left it (`decontaminate`'s `edge_band`)."""if decontam != "off":from sprite_gen.frames import decontam as decontam_module@@ -720,7 +722,7 @@ def remove_chroma_background(if decontam != "off":data, stats = decontam_module.decontaminate(source_rgb, data, keyed_mask, chroma_key, fit=decontam_fit,alpha_depth=unmix_reach, palette=decontam_palette, mode=decontam,- source_alpha=source_alpha)+ source_alpha=source_alpha, edge_band=decontam_edge_band)if decontam_stats is not None:decontam_stats.update(stats)# Back into the converted copy rather than a fresh Image.fromarray, so thediff --git a/sprite_gen/video/frames.py b/sprite_gen/video/frames.pyindex 238558f..c4a7683 100644--- a/sprite_gen/video/frames.py+++ b/sprite_gen/video/frames.py@@ -234,7 +234,7 @@ def key_frames(dst = keyed_dir / src.namestats = cutout(src, dst, key=key, spill_max_fraction=spill_max, spill_min_tint=spill_tint,spill_require_hue=spill == "full", decontam=decontam, decontam_fit=DECONTAM_FIT,- decontam_palette=clip_palette)+ decontam_palette=clip_palette, decontam_edge_band=EDGE_ROWS)image = Image.open(dst).convert("RGBA")hist = image.getchannel("A").histogram()w, h = image.size@@ -280,6 +280,7 @@ def key_frames("palette_keyfree": decontam_first["palette_keyfree"],"key_material_share": decontam_first["key_material_share"],"material_spread": decontam_first["material_spread"],+ "edge_band": decontam_first["edge_band"],"applied_frames": decontam_applied,"totals": decontam_totals,}diff --git a/tests/frames/test_chroma_decontam.py b/tests/frames/test_chroma_decontam.pyindex f9fd6ff..18b74ac 100644--- a/tests/frames/test_chroma_decontam.py+++ b/tests/frames/test_chroma_decontam.py@@ -415,6 +415,50 @@ def test_video_frames_learn_one_palette_per_clip(tmp_path: Path):assert all("decontam" not in row for row in off["rows"])+def _head_under_the_top_edge(t: int) -> np.ndarray:+ """A decoded frame of a dark head whose matte edge sits 6 px under the top of the frame.++ Above it the head's blur runs on into the key as a decoded clip shows it: coverage+ 0.3, 0.15, 0.08, 0.03 on rows 5 to 2, which the hard cut keys, and rows 0 to 1 are key.+ """+ F = np.zeros((96, 96, 3))+ A = np.zeros((96, 96))+ F[6:90, 30 + t:66 + t] = (60, 62, 66)+ F[40:70, 40 + t:56 + t] = (150, 148, 152) # a lighter patch, so the palette has two colours+ A[6:90, 30 + t:66 + t] = 1.0+ for row, coverage in ((5, 0.3), (4, 0.15), (3, 0.08), (2, 0.03)):+ F[row, 30 + t:66 + t] = (60, 62, 66)+ A[row, 30 + t:66 + t] = coverage+ return _subsample_420(_composite_on(F, A, GREEN_PAINTED))+++def test_video_frames_keep_the_frame_edge_as_the_matte_left_it(tmp_path: Path):+ """A clip that passes the edge gate with decontam off passes it with decontam on.++ Each blur row above the head fits a faint blend. Rows 3 to 5 lie within the flank,+ and row 3 inside the gate's edge band too: recovering it put the subject's grey on+ the frame edge, and the gate read it as the key surviving the matte.+ """+ raws = []+ for t in range(2):+ path = tmp_path / f"frame-{t + 1:04d}.png"+ Image.fromarray(_head_under_the_top_edge(t)).save(path)+ raws.append(path)+ off = frames_mod.key_frames(raws, tmp_path / "plain", key="green", spill="full")+ on = frames_mod.key_frames(raws, tmp_path / "keyed", key="green", spill="full", decontam="palette")+ assert off["edge_contacts"] == on["edge_contacts"] == []+ assert on["decontam"]["applied_frames"] == 2 and on["decontam"]["edge_band"] == frames_mod.EDGE_ROWS+ band = np.zeros((96, 96), dtype=bool)+ band[:frames_mod.EDGE_ROWS] = band[:, :frames_mod.EDGE_ROWS] = band[:, -frames_mod.EDGE_ROWS:] = True+ for raw in raws:+ plain = np.array(Image.open(tmp_path / "plain" / raw.name))[..., 3]+ keyed = np.array(Image.open(tmp_path / "keyed" / raw.name))[..., 3]+ assert not keyed[band & (plain == 0)].any()+ # below the edge band the flank still gives the blur rows their coverage back+ assert keyed[4:6][plain[4:6] == 0].any()+ assert on["decontam"]["totals"]["recovered_px"] > 0++def test_video_frames_rejects_an_unknown_mode(tmp_path: Path):with pytest.raises(SystemExit, match="unknown --decontam"):frames_mod.run_frames(tmp_path / "missing.mp4", tmp_path / "out", key="green", allow_edge_contact=True,