Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
feat(cutout): cut out a flat background of any colour from what the image painted, and fail a cutout that removes nothing
Soohong Kimd9d469075a89
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex cc039ed..8268c41 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -2,6 +2,18 @@All notable public changes to `sprite-gen` are recorded here. Versions track the `version:` field in `SKILL.md` and `pyproject.toml`.+## Unreleased - A flat background of any colour is cut out from what the image painted, and a cutout that removes nothing fails++**Upgrading: backgrounds the chroma engine already recognised key byte for byte as before; a background it did not recognise, which used to come out with nothing removed, is now cut out.** A pure or recognised magenta/green background still goes to the chroma engine and a bright achromatic one to the white matte, with the same bytes.++- **New: the measured background matte (`sprite_gen/frames/measured_matte.py`, rule in docs/chroma-alpha.md "The measured background").** It reads the background's colour and spread from a ring inside the image's edges, keys that colour and its hue in shade (a drop shadow, a gradient), thin dark strips along the edges, and enclosed pockets of a saturated background, then unmixes a 2 px edge between the local background and the nearest solid subject colour. No colour needs a route of its own: a muted yellow-green, an off-cyan, a grey or an off-white key the same way.+- **Fixed: `cutout --key green|magenta` and `extract_route` on a background the engine does not recognise as that key.** Such a background was keyed against the pure key and nothing came off, and `cutout` exited 0; a green behind dark edge strips was routed to the white matte. It is now measured (`route: measured`, `chroma_key_painted` the measured colour), so `video-canvas` repaints such a still to the exact key, and `cutout` keys the frames of such a clip one by one. `video-frames` keeps keying a clip from one exact key: it still refuses such a clip, with the same bytes and report, and its guidance to normalise the still with `video-canvas` now works on these backgrounds.+- **Fixed: `cutout` fails when it made nothing transparent**, like `gen/chroma.py`, instead of reporting success.+- CLI: `cutout --key measured` forces the measured matte; `auto` sends a corner that is neither a key nor bright and achromatic to it.+- Report fields (`route: measured`): `background`, `background_spread`, `background_ring_share`, `shade`, `tolerance`, `edge_strips`, `keyed_pixels`, `pocket_pixels`, `edge_pixels`, `band`, `key_residue_px`.+- Python callers: `extract_route(..., measure_unrecognised=True)` and `cutout(..., measure_unrecognised=True, refuse_empty=True)` (`video-frames` passes both False); new `cutout.key_recognised(image, kind)`, `measured_matte.measured_matte(image, band=2)`, `measured_matte.measure_background(rgb)`, `measured_matte.background_like(rgb, colour, tolerance, shade)`.+- **Tests**: new `tests/frames/test_measured_matte.py`, synthetic only: a subject with a dark outline, skin and hair colours on muted green, deep green, off-cyan and yellowish backgrounds with noise, a gradient, a darker shade round the subject, dark strips on two edges and a pocket of background inside; an off-white background measured on request; the routing of a green request on a muted green through `extract_route`, `cutout` and `video-canvas`; a cutout that keys nothing fails.+## v3.4.0 - Loop part pop retakes, cycle alignment walk beat repair and broken beats, ghost limb faults, and same foot, filmed jump, and seam jump retakes**Upgrading from 3.3.0: A walk or run loop in `video-loop` is cut to the same bytes, whether a part pops in it or not, and `video-cycle-align` changes only alignment outputs whose beat broke or where a made frame has a limb seen through under `--between auto`.** For `video-loop`, every walk and run report gains `part_pop` and `retake`; `--unfixed refuse` also refuses a loop recorded `retake` reason `part-pop`, whatever `--repair`; no flags are added, no frames are made and no other cut is taken. `video-cycle-align` now reads ghost bands in windows of the body to catch limbs seen through, keeps frames given way on the beat under `--between auto` by making them again at clean fractions or widening jolts across neighbouring frames, changing only alignment outputs whose beat broke (a jolt of 0.25 beat or more) or where a made frame has a limb seen through (ghost window over 1.4 %); loops keeping the beat, loops with no frame given way, and `video-loop` (and `video-set`) keep their bytes for strip PNG, GIF, WebP, and cycle frames, while gaining `beat`, `ghost_window`, (on back and back_diagonal views) `same_foot`, and `retake` reasons (`broken-beat`, `same-foot`) in alignment `strip.json` and reports; every aligned loop's `strip.json` and report gain `screen`, and loops whose cut jumps gain the reasons `filmed-jump` and `seam-jump`, frames unchanged; and `clip_feet` in `video-loop`'s `strip.json` and reports for every walk or run cut; all outputs from `video-loop-compare` and `video-loop-repair` are unchanged, bytes included, and `--same-foot` defaults to `warn` so nothing is refused by default.diff --git a/docs/chroma-alpha.md b/docs/chroma-alpha.mdindex 9f09683..36e6975 100644--- a/docs/chroma-alpha.md+++ b/docs/chroma-alpha.md@@ -174,6 +174,24 @@ Limitations:- Only on stills, outside a dark outline the flank can give a few faint pixels in the subject's colour.- Time and memory go up: keying on an Intel i5-7500 took 1.14 s a frame with `off` and 3.36 s with `auto` at 1080x1920, and peak memory rose from about 400 MB to 1155 MB (about 365 MB more per megapixel of frame, or about 2.3 GB more for `video-set` running three concurrent jobs).+## The measured background: any flat colour, one matte++Owns the rule; the code is `sprite_gen/frames/measured_matte.py`, routed from `cutout.py`.++A model asked for a key background rarely paints the key. It paints a muted yellow-green, a deep green that darkens toward pure green in shadow, an off-cyan, a grey; with noise, a gradient, a darker shade of the background round the subject, and sometimes thin dark strips along the edges. The chroma engine recognises a background as a key only through `is_border_key_candidate` or the hard cut of the pure key. Before this matte such a background was keyed against the pure key and nothing came off, and `cutout` still exited 0; a green background whose corners sat on dark strips was routed to the white matte.++No colour gets a route of its own. The measured matte reads the background off the image:++1. **Ring.** The colour is the mode of a ring just inside where an edge strip may reach (3 % of the short side, at least 2 px), refined to the mean of the ring pixels within 24 of it; its spread is their largest channel deviation. A background that holds under 20 % of the ring is refused.+2. **Background-like.** Within `max(26, 4 x spread)` on every channel of the colour, or, on a saturated background (chroma 40 and over), its hue in shade: within 30 degrees of its hue, at half its saturation or more, with chroma over `max(20, 0.15 x its chroma)` at any brightness. A model shades a green toward a deeper, purer green, not along the same RGB line, so shade is read in hue. Near-black lineart (chroma 10) is never shade.+3. **Edge strips.** A run of lines from an edge that are 60 % one colour and not background is a strip; its colour's pixels are keyed, a subject drawn over it keeps its own.+4. **Keyed.** Background-like pixels connected to the edges, read twice: the second time against the background's own colour nearby, so a gradient is followed. On a saturated background, enclosed pockets of it at least 5 px across (between an arm and the body, between strands of hair) are keyed too. On a pale background they are not: its pockets may be the subject's highlight or skin, the white matte's rule.+5. **Edge.** Within 2 px of the keyed set each pixel is unmixed between the local background and the nearest solid subject colour: alpha is its projection on that line, its colour the unmixed one (falling back to the subject colour as alpha drops). The subject's pixels farther in are byte-identical.++Routing (`cutout`): a pure or recognised magenta/green background still goes to the chroma engine, a bright achromatic corner to the white matte, and everything else to the measured matte. `--key measured` forces it. A magenta/green request whose background the engine does not recognise (`cutout.key_recognised`) is measured inside `extract_route`, so `video-canvas` repaints such a still to the exact key and `cutout` keys each frame of such a clip on its own; the stats say `route: measured` with `chroma_key_painted` the measured colour. `video-frames` does not measure (`measure_unrecognised=False`): a clip is keyed from one exact key, so it still refuses a clip whose background the engine does not recognise and sends its still to `video-canvas`, which now normalises it. `cutout` now fails when it made nothing transparent (`refuse_empty`; `video-frames` refuses such a frame itself).++Report fields (`route: measured`): `background`, `background_spread`, `background_ring_share`, `shade`, `tolerance`, `edge_strips` (depth per side), `keyed_pixels`, `pocket_pixels`, `edge_pixels`, `band`, and `key_residue_px`: opaque pixels in the edge band still within the tolerance of the local background colour, the visible key left on the cut. `--decontam palette` refuses this route (there is no key colour to remove); `auto` records that the matte unmixed its own edge.+## Related- [`../SKILL.md`](../SKILL.md) — 필수 게이트 (크로마 키 소재색 분기 + 변환 후 소재색 보존 검증)diff --git a/sprite_gen/frames/cutout.py b/sprite_gen/frames/cutout.pyindex 9de8e97..4c611f1 100644--- a/sprite_gen/frames/cutout.py+++ b/sprite_gen/frames/cutout.py@@ -20,6 +20,11 @@ It routes on the corner background colour:cut is safe there — the exact case a white matte's position guard is *not*needed. Empirically (2026-07-20) feeding an ivory key to the extract engineholes out bench stone / lamp glass; the two routes are deliberate.+- **any other flat colour** (a muted yellow-green, an off-cyan, a grey, a key+ behind dark edge strips) → the measured matte (`measured_matte.py`), which reads+ the colour, its spread and its shade off the image itself. A magenta or green+ request whose background the chroma engine does not recognise as that key goes+ there too (`extract_route`), instead of keying nothing.Sibling note: `unmix_key_blend`/`_matte_ycc` in `extract.py` and the mattedespill here are the same "despill + soft edge" idea in two colour spaces@@ -109,6 +114,17 @@ def _corner_average(image: Image.Image) -> tuple[int, int, int]:return tuple(sum(s[i] for s in samples) // n for i in range(3)) # type: ignore[return-value]+def _bright_corners(image: Image.Image) -> bool:+ """Whether a corner is bright and achromatic enough for the white matte's estimate+ (`estimate_background`): otherwise the white matte would key against its ivory guess."""+ width, height = image.size+ for x, y in ((0, 0), (width - 1, 0), (0, height - 1), (width - 1, height - 1)):+ r, g, b, a = image.crop((x, y, x + 1, y + 1)).convert("RGBA").getpixel((0, 0))+ if a > 0 and min(r, g, b) >= CORNER_MIN_BRIGHT:+ return True+ return False++def _detect_key_kind(color: tuple[int, int, int]) -> str:"""Classify a corner background colour into a route: magenta | green | white.@@ -271,12 +287,52 @@ def _matte_route(}+def key_recognised(image: Image.Image, kind: str) -> tuple[bool, list[int] | None]:+ """Whether the chroma engine would key ``image``'s background as the ``kind`` key: (recognised,+ the measured background colour or None).++ The background is measured on the ring inside any edge strip (`measured_matte.measure_background`).+ It is the key when that colour is a border key candidate for it (`extract.is_border_key_candidate`,+ the detector's own rule) or within the hard cut of the pure key; a background nothing on the ring+ holds is left to the engine, whose detector falls back to the declared key as before."""+ from sprite_gen._deps import np+ from sprite_gen.frames.extract import is_border_key_candidate+ from sprite_gen.frames.measured_matte import measure_background++ measured = measure_background(np.asarray(image.convert("RGB"), dtype=np.float32))+ if measured is None:+ return True, None+ colour = tuple(int(round(v)) for v in measured["colour"])+ target = KEY_TARGETS[kind]+ distance = sum((a - b) ** 2 for a, b in zip(colour, target)) ** 0.5+ return is_border_key_candidate(colour, target) or distance <= _EXTRACT_KEY_THRESHOLD, list(colour)+++def _measured_route(image: Image.Image, decontam: str, requested: str) -> tuple[Image.Image, dict[str, Any]]:+ """The measured matte, its stats carrying what was asked for and what the edge pass did."""+ from sprite_gen.frames.measured_matte import measured_matte++ if decontam == "palette":+ raise SystemExit(f"cutout: --decontam palette needs a chroma key background (magenta/green); "+ f"the background was keyed as measured (asked: {requested})")+ result, stats = measured_matte(image)+ stats["requested_key"] = requested+ if requested in KEY_TARGETS:+ stats["chroma_key"] = list(KEY_TARGETS[requested])+ stats["chroma_key_painted"] = list(stats["background"])+ if decontam == "auto":+ stats["decontam"] = {"mode": "auto", "applied": False,+ "reason": "the measured matte unmixes its own edge against the measured background"}+ return result, stats++def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float | None = None,spill_min_tint: float | None = None,spill_require_hue: bool = False, decontam: str = "off", decontam_fit: str = "still",decontam_palette: dict[str, Any] | None = None, decontam_edge_band: int = 0,background_key: tuple[int, int, int] | None = None,- decontam_history: dict[str, Any] | None = None) -> tuple[Image.Image, dict[str, Any]]:+ decontam_history: dict[str, Any] | None = None,+ measure_unrecognised: bool = True) -> tuple[Image.Image, dict[str, Any]]:"""Magenta/green key background → reuse the verified `extract` chroma engine (no drift).The engine keys from the background colour it detects on the borders@@ -291,10 +347,19 @@ def extract_route(image: Image.Image, kind: str, *, spill_max_fraction: float |`decontam` is the engine's edge decontamination pass (`extract.remove_chroma_background`);what it did comes back under `decontam` in the stats. `decontam_history` is a clip's record ofits last frame for the video fit, handed on to the pass.++ A background the engine does not recognise as the ``kind`` key (`key_recognised`: a muted or+ shifted colour, or a key behind dark edge strips) is keyed by the measured matte instead, its+ stats `route` "measured" and `chroma_key_painted` the measured colour; before it, such a+ background was keyed against the pure key and nothing came off (2026-10-26).+ `measure_unrecognised=False` keeps the engine on it anyway: `video-frames` refuses such a clip+ and sends its still to `video-canvas`, so every frame of a clip is keyed from one exact key."""from sprite_gen.frames.extract import detect_background_key_rgb, remove_chroma_backgroundtarget = KEY_TARGETS[kind]+ if background_key is None and measure_unrecognised and not key_recognised(image, kind)[0]:+ return _measured_route(image, decontam, kind)if background_key is None:painted = detect_background_key_rgb(image, target, _EXTRACT_KEY_THRESHOLD)else:@@ -344,11 +409,16 @@ def cutout(decontam_palette: dict[str, Any] | None = None,decontam_edge_band: int = 0,decontam_history: dict[str, Any] | None = None,+ measure_unrecognised: bool = True,+ refuse_empty: bool = True,) -> dict[str, Any]:"""Cut a uniform-background imported image to a clean transparent RGBA PNG.`key`: "auto" (detect from corners) | "white" (matte) | "magenta" | "green"- (reuse the extract chroma engine). `spill_max_fraction` overrides the chroma+ (reuse the extract chroma engine) | "measured" (the measured matte, any flat colour).+ Auto sends a corner that is neither a key nor bright and achromatic to the measured+ matte, and a magenta/green background the engine does not recognise goes there too+ (`extract_route`). `spill_max_fraction` overrides the chromaengine's trapped-spill cluster cap and `spill_min_tint` its tint bar (None = theengine default for either). `decontam` is the engine's edge decontaminationpass: "off" (the default) keeps the matte as it is; "auto" runs it on chroma routes@@ -356,11 +426,15 @@ def cutout(it cannot run (the white matte has no key colour to remove). `decontam_edge_band`keeps what the matte left transparent along the top, left and right edgestransparent (`video-frames` passes its edge band), and `decontam_history` carries a- clip's last frame to the next (`video-frames` keeps one per clip). Returns a stats dict. Raises SystemExit if the key is unknown, the background cannot be- located, or any transparent pixel keeps non-zero RGB (No Silent Fallback).+ clip's last frame to the next (`video-frames` keeps one per clip). `measure_unrecognised`+ is handed to `extract_route`, and `refuse_empty=False` leaves a cut with nothing transparent+ to the caller (`video-frames` passes both False: it refuses such frames itself, with its own+ report and guidance). Returns a stats dict. Raises SystemExit if the key is unknown, the background cannot be+ located, nothing was made transparent, or any transparent pixel keeps non-zero RGB+ (No Silent Fallback)."""- if key not in ("auto", "white", "magenta", "green"):- raise SystemExit(f"cutout: unknown key {key!r}; expected one of auto|white|magenta|green")+ if key not in ("auto", "white", "magenta", "green", "measured"):+ raise SystemExit(f"cutout: unknown key {key!r}; expected one of auto|white|magenta|green|measured")if decontam not in ("off", "auto", "palette"):raise SystemExit(f"cutout: unknown decontam {decontam!r}; expected off|auto|palette")@@ -368,12 +442,16 @@ def cutout(width, height = image.sizeroute = _detect_key_kind(_corner_average(image)) if key == "auto" else key- if route in ("magenta", "green"):+ if key == "auto" and route == "white" and not _bright_corners(image):+ route = "measured"+ if route == "measured":+ result, route_stats = _measured_route(image, decontam, key)+ elif route in ("magenta", "green"):result, route_stats = extract_route(image, route, spill_max_fraction=spill_max_fraction,spill_min_tint=spill_min_tint, spill_require_hue=spill_require_hue,decontam=decontam, decontam_fit=decontam_fit,decontam_palette=decontam_palette, decontam_edge_band=decontam_edge_band,- decontam_history=decontam_history)+ decontam_history=decontam_history, measure_unrecognised=measure_unrecognised)else:if decontam == "palette":raise SystemExit(f"cutout: --decontam {decontam} needs a chroma key background (magenta/green); "@@ -401,6 +479,11 @@ def cutout(raise SystemExit(f"cutout: transparent pixels still contain non-zero RGB ({stale_rgb} px) in {out_path}")+ if refuse_empty and not alpha_zero:+ raise SystemExit(+ f"cutout: nothing of {input_path} was made transparent (route {route_stats.get('route')}); "+ "refusing a successful cutout that keyed no background"+ )stats: dict[str, Any] = {"out": str(out_path),@@ -427,13 +510,14 @@ def cutout(def add_arguments(p: Any) -> None:- p.add_argument("input", type=Path, help="imported image with a uniform (white/ivory or magenta/green key) background")+ p.add_argument("input", type=Path, help="imported image with a flat background of any colour (white, a key, a muted green...)")p.add_argument("--out", type=Path, help="output PNG (default: <input>_cutout.png)")p.add_argument("--key",- choices=["auto", "white", "magenta", "green"],+ choices=["auto", "white", "magenta", "green", "measured"],default="auto",- help="background type; auto detects from corners (white→matte, magenta/green→extract engine)",+ help="background type; auto detects from corners (white→matte, magenta/green→extract engine, any other "+ "colour→measured matte); a magenta/green background the engine does not recognise is measured too",)p.add_argument("--strength",diff --git a/sprite_gen/frames/measured_matte.py b/sprite_gen/frames/measured_matte.pynew file mode 100644index 0000000..498cd7b--- /dev/null+++ b/sprite_gen/frames/measured_matte.py@@ -0,0 +1,285 @@+# SPDX-License-Identifier: Apache-2.0+"""The measured background matte: a flat background of any colour, keyed from what the image painted.++The chroma engine (`extract.remove_chroma_background`) keys magenta and green backgrounds it+recognises as a key (`extract.is_border_key_candidate`), and the white matte (`cutout._matte_route`)+keys bright achromatic ones. A model asked for a key background often paints neither: a muted+yellow-green, an off-cyan, a grey, with noise, a gradient, a darker same-colour shadow round the+subject and thin dark strips along the edges. This matte keys those from a measurement of the image+itself, so no colour needs a route of its own (docs/chroma-alpha.md "The measured background").++1. Edge strips: thin lines along an edge in one colour that is not the background (a letterbox bar).+2. The background colour and its spread, read on a ring inside the strips (`measure_background`).+3. Background-like pixels: within the spread of the colour, or, on a saturated background, its hue+ in shade (`background_like`), so a drop shadow and a gradient are background too.+4. What is keyed: the background-like pixels connected to the edges, the strips, and, on a+ saturated background, enclosed pockets of it thick enough not to be a speck of the subject.+5. The edge: within `band` px of the keyed set, each pixel is unmixed between the local background+ and the nearest solid subject colour, giving partial alpha and the subject's own colour back.++Numpy only, no per-pixel Python loop, so a clip's frames key at the speed of a still.+"""++from __future__ import annotations++from typing import Any++from PIL import Image, ImageFilter++from sprite_gen._deps import np++STRIP_MAX_FRACTION = 0.03 # an edge strip is at most this share of the side it runs across (min 2 px)+STRIP_LINE_SHARE = 0.6 # a strip line: this share of it is one colour+RING_WIDTH = 8 # px of the measuring ring, inside the deepest a strip may go+MODE_BIN = 8 # RGB histogram bin of the background mode+INLIER_TOLERANCE = 24 # max-channel distance from the mode that counts as the background when measuring it+MIN_BACKGROUND_SHARE = 0.2 # the background holds at least this share of the ring+TOLERANCE_FLOOR = 26 # max-channel distance from the colour that is background (the white matte's 26)+SPREAD_SCALE = 4.0 # ... or this many spreads of the measured background, if wider+SATURATED_CHROMA = 40 # max - min channel at which a background's colour is its own (shade is then read)+SHADE_HUE = 30.0 # degrees of hue from the background's that are its shade (a model shades by hue, not by scale)+SHADE_SATURATION = 0.5 # ... at this share of its saturation or more+SHADE_CHROMA = (20.0, 0.15) # ... and at least max(this, this share of its chroma): near-black lineart is not shade+HOLE_THICKNESS = 2 # an enclosed pocket survives this many erosions to be keyed (5 px across)+EDGE_BAND = 2 # px of edge unmixed outside the keyed set+ALPHA_FLOOR = 4 # alpha below this (of 255) is keyed outright+FULL_BACKGROUND_SHARE = 0.999 # keyed share at which nothing of a subject is left+++def _shift_or(mask: np.ndarray) -> np.ndarray:+ """The mask grown by one px in the four directions."""+ grown = mask.copy()+ grown[1:] |= mask[:-1]+ grown[:-1] |= mask[1:]+ grown[:, 1:] |= mask[:, :-1]+ grown[:, :-1] |= mask[:, 1:]+ return grown+++def _reconstruct(seed: np.ndarray, allowed: np.ndarray) -> np.ndarray:+ """The pixels of `allowed` connected (4-way) to `seed` inside it: geodesic reconstruction."""+ mask = seed & allowed+ while True:+ before = mask+ for _ in range(16):+ mask = _shift_or(mask) & allowed+ if np.array_equal(mask, before):+ return mask+++def _erode(mask: np.ndarray, times: int) -> np.ndarray:+ for _ in range(times):+ inner = mask.copy()+ inner[1:] &= mask[:-1]+ inner[:-1] &= mask[1:]+ inner[:, 1:] &= mask[:, :-1]+ inner[:, :-1] &= mask[:, 1:]+ mask = inner+ return mask+++def _filled_mean(rgb: np.ndarray, weight: np.ndarray, radii: tuple[int, ...]) -> tuple[np.ndarray, np.ndarray]:+ """Per pixel the mean colour of the weighted pixels nearby, at the smallest radius that has any.++ Returns (colour, found): `found` is False where no radius reached a weighted pixel."""+ height, width = weight.shape+ colour = np.zeros((height, width, 3), dtype=np.float32)+ found = np.zeros((height, width), dtype=bool)+ w8 = Image.fromarray((weight * 255).astype(np.uint8))+ channels = [Image.fromarray((rgb[..., c] * weight).astype(np.uint8)) for c in range(3)]+ for radius in radii:+ bw = np.asarray(w8.filter(ImageFilter.BoxBlur(radius))).astype(np.float32) / 255+ new = ~found & (bw > 0.02)+ if not new.any():+ continue+ for c in range(3):+ bc = np.asarray(channels[c].filter(ImageFilter.BoxBlur(radius))).astype(np.float32)+ colour[..., c][new] = (bc / np.maximum(bw, 1e-4))[new]+ found |= new+ return colour, found+++def _hue_chroma_saturation(rgb: np.ndarray) -> tuple[np.ndarray, np.ndarray, np.ndarray]:+ """HSV hue (degrees), chroma (max - min) and saturation (chroma / max) of RGB in 0..255."""+ high, low = rgb.max(axis=-1), rgb.min(axis=-1)+ chroma = high - low+ safe = np.maximum(chroma, 1e-4)+ r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2]+ hue = np.where(high == r, ((g - b) / safe) % 6, np.where(high == g, (b - r) / safe + 2, (r - g) / safe + 4)) * 60+ return hue, chroma, chroma / np.maximum(high, 1.0)+++def background_like(rgb: np.ndarray, colour: np.ndarray, tolerance: float, shade: bool) -> np.ndarray:+ """Pixels that are the background `colour` (one RGB, or one per pixel): within `tolerance` on+ every channel, or, when `shade`, its hue in shade: within `SHADE_HUE` of its hue, at+ `SHADE_SATURATION` of its saturation and with chroma over `SHADE_CHROMA`, at any brightness.+ A model darkens a green background toward a deeper, purer green, not toward black along the+ same RGB line, so the shade is read in hue."""+ like = np.abs(rgb - colour).max(axis=-1) <= tolerance+ if shade:+ hue, chroma, saturation = _hue_chroma_saturation(rgb)+ key_hue, key_chroma, key_saturation = _hue_chroma_saturation(np.broadcast_to(colour, rgb.shape))+ turn = np.abs(hue - key_hue) % 360+ like |= ((np.minimum(turn, 360 - turn) <= SHADE_HUE) & (saturation >= SHADE_SATURATION * key_saturation)+ & (chroma >= np.maximum(SHADE_CHROMA[0], SHADE_CHROMA[1] * key_chroma)))+ return like+++def _strip_depth(height: int, width: int) -> int:+ return max(2, round(min(height, width) * STRIP_MAX_FRACTION))+++def _ring(height: int, width: int) -> np.ndarray:+ inset = _strip_depth(height, width) + 2+ ring = np.zeros((height, width), dtype=bool)+ outer = min(inset + RING_WIDTH, height // 2, width // 2)+ if outer <= inset: # a tiny image: the whole border+ ring[:2] = ring[-2:] = True+ ring[:, :2] = ring[:, -2:] = True+ return ring+ ring[inset:outer, inset:width - inset] = True+ ring[height - outer:height - inset, inset:width - inset] = True+ ring[inset:height - inset, inset:outer] = True+ ring[inset:height - inset, width - outer:width - inset] = True+ return ring+++def measure_background(rgb: np.ndarray) -> dict[str, Any] | None:+ """The flat background's colour and spread, read on a ring just inside where an edge strip may+ reach: the mode of the ring's colours, then the mean and spread of the ring pixels within+ `INLIER_TOLERANCE` of it. None when no colour holds `MIN_BACKGROUND_SHARE` of the ring."""+ height, width = rgb.shape[:2]+ samples = rgb[_ring(height, width)]+ if not len(samples):+ return None+ bins = samples.astype(np.int32) // MODE_BIN+ slots = (bins[:, 0] << 10) | (bins[:, 1] << 5) | bins[:, 2]+ counts = np.bincount(slots)+ mode = int(np.argmax(counts))+ seed = samples[slots == mode].mean(axis=0)+ inliers = samples[np.abs(samples - seed).max(axis=-1) <= INLIER_TOLERANCE]+ share = len(inliers) / len(samples)+ if share < MIN_BACKGROUND_SHARE:+ return None+ colour = inliers.mean(axis=0)+ spread = float(inliers.std(axis=0).max())+ chroma = float(colour.max() - colour.min())+ return {"colour": colour, "spread": spread, "share": share, "chroma": chroma,+ "shade": chroma >= SATURATED_CHROMA,+ "tolerance": max(float(TOLERANCE_FLOOR), SPREAD_SCALE * spread)}+++def _edge_strips(rgb: np.ndarray, like: np.ndarray) -> tuple[np.ndarray, dict[str, int]]:+ """Lines along each edge in one colour that is not the background: (their pixels, depth per side).++ A line is a strip line when `STRIP_LINE_SHARE` of it is within `TOLERANCE_FLOOR` of its median+ and that colour is not mostly background; the strip is the run of such lines from the edge, at+ most `_strip_depth` deep. Only the strip colour's pixels (and the background's) are taken, so a+ subject drawn over the strip keeps its own pixels."""+ height, width = like.shape+ depth_max = _strip_depth(height, width)+ strip = np.zeros((height, width), dtype=bool)+ depths: dict[str, int] = {}+ views = {"top": (rgb, like, strip), "bottom": (rgb[::-1], like[::-1], strip[::-1]),+ "left": (rgb.transpose(1, 0, 2), like.T, strip.T),+ "right": (rgb.transpose(1, 0, 2)[::-1], like.T[::-1], strip.T[::-1])}+ for side, (lines, line_like, out) in views.items():+ depth = 0+ for k in range(min(depth_max, len(lines))):+ line = lines[k]+ if line_like[k].mean() >= STRIP_LINE_SHARE:+ break+ median = np.median(line, axis=0)+ same = np.abs(line - median).max(axis=-1) <= TOLERANCE_FLOOR+ if same.mean() < STRIP_LINE_SHARE:+ break+ out[k] |= same | line_like[k]+ depth = k + 1+ if depth:+ depths[side] = depth+ return strip, depths+++def measured_matte(image: Image.Image, *, band: int = EDGE_BAND) -> tuple[Image.Image, dict[str, Any]]:+ """Key ``image``'s flat background of whatever colour it was painted. Returns (RGBA, stats).++ Raises SystemExit when no flat background is found on the ring, or when keying it would leave+ nothing of a subject."""+ rgba = np.asarray(image.convert("RGBA"))+ rgb = rgba[..., :3].astype(np.float32)+ height, width = rgb.shape[:2]+ measured = measure_background(rgb)+ if measured is None:+ raise SystemExit("cutout: no flat background colour holds the image's border ring; "+ "is the background one colour?")+ colour, tolerance, shade = measured["colour"], measured["tolerance"], measured["shade"]++ like = background_like(rgb, colour, tolerance, shade) | (rgba[..., 3] == 0)+ strip, strips = _edge_strips(rgb, like)+ edge_seed = np.zeros_like(like)+ reach = _strip_depth(height, width) + 2 + RING_WIDTH+ edge_seed[:reach] = edge_seed[-reach:] = True+ edge_seed[:, :reach] = edge_seed[:, -reach:] = True+ connected = _reconstruct((edge_seed & like) | strip, like | strip)++ # a gradient: the background read again against its own colour nearby, the subject kept out+ local, found = _filled_mean(rgb, connected.astype(np.float32), (8, 32, 128))+ local = np.where(found[..., None], local, colour)+ like = background_like(rgb, local, tolerance, shade) | (rgba[..., 3] == 0)+ connected = _reconstruct((edge_seed & like) | strip, like | strip)++ # enclosed pockets of the background (between an arm and the body), thick enough not to be a speck;+ # only a saturated background's colour is its own: a pale one's pockets may be the subject's own+ # highlight or skin, so it is keyed by position alone (the white matte's rule)+ pockets = like & ~connected if shade else np.zeros_like(like)+ pockets = _reconstruct(_erode(pockets, HOLE_THICKNESS), pockets) if pockets.any() else pockets+ keyed = connected | pockets++ keyed_share = float(keyed.mean())+ if keyed_share >= FULL_BACKGROUND_SHARE:+ raise SystemExit(f"cutout: the measured background {tuple(int(v) for v in colour)} covers "+ f"{keyed_share:.1%} of the image; there is no subject to cut out")++ # the edge: unmixed between the local background and the nearest solid subject colour+ near = keyed.copy()+ for _ in range(band):+ near = _shift_or(near)+ edge = near & ~keyed+ solid = ~near+ background, found = _filled_mean(rgb, keyed.astype(np.float32), (2, 8, 32, 128))+ background = np.where(found[..., None], background, colour)+ subject, found = _filled_mean(rgb, solid.astype(np.float32), (1, 2, 4, 8, 16, 64))+ subject = np.where(found[..., None], subject, rgb)+ toward = subject - background+ offset = rgb - background+ span = (toward * toward).sum(axis=-1)+ alpha = np.where(span > 900.0, (offset * toward).sum(axis=-1) / np.maximum(span, 1.0),+ (np.abs(offset).max(axis=-1) - tolerance) / 60.0)+ alpha = np.clip(alpha, 0.0, 1.0)+ unmixed = np.clip((rgb - (1.0 - alpha[..., None]) * background) / np.maximum(alpha[..., None], 1e-3), 0, 255)+ edge_rgb = alpha[..., None] * unmixed + (1.0 - alpha[..., None]) * subject++ out = rgba.astype(np.float32).copy()+ out[edge, :3] = edge_rgb[edge]+ out[edge, 3] = np.minimum(out[edge, 3], np.round(alpha[edge] * 255))+ out[keyed] = 0+ faint = out[..., 3] < ALPHA_FLOOR+ out[faint] = 0+ result = Image.fromarray(np.clip(np.round(out), 0, 255).astype(np.uint8), "RGBA")++ residue = int((background_like(np.asarray(result)[..., :3].astype(np.float32), background, tolerance, False)+ & (np.asarray(result)[..., 3] >= 128) & near).sum())+ return result, {+ "route": "measured",+ "background": [int(round(v)) for v in colour],+ "background_spread": round(measured["spread"], 2),+ "background_ring_share": round(measured["share"], 4),+ "shade": shade,+ "tolerance": round(tolerance, 1),+ "edge_strips": strips,+ "keyed_pixels": int((out[..., 3] == 0).sum()),+ "pocket_pixels": int(pockets.sum()),+ "edge_pixels": int(edge.sum()),+ "band": band,+ "key_residue_px": residue,+ }diff --git a/sprite_gen/video/frames.py b/sprite_gen/video/frames.pyindex 01dbc73..27fe3d3 100644--- a/sprite_gen/video/frames.py+++ b/sprite_gen/video/frames.py@@ -548,7 +548,10 @@ def _key_chroma(raw_files: list[Path], keyed_dir: Path, *, key: str, check_edgesdst = keyed_dir / src.namestats = cutout(src, dst, key=key, spill_max_fraction=spill_max, spill_min_tint=spill_tint,spill_require_hue=spill == "full", decontam=decontam, decontam_fit=DECONTAM_FIT,- decontam_palette=clip_palette, decontam_edge_band=EDGE_ROWS, decontam_history=clip_history)+ decontam_palette=clip_palette, decontam_edge_band=EDGE_ROWS, decontam_history=clip_history,+ # one exact key for the whole clip: an unrecognised background is refused below and its+ # still normalized by `video-canvas` (docs/chroma-alpha.md "The measured background")+ measure_unrecognised=False, refuse_empty=False)image = Image.open(dst).convert("RGBA")# The specks go before anything reads the frame: the edge band, the coverage, the loop.image, specks = drop_specks(image)diff --git a/tests/frames/test_measured_matte.py b/tests/frames/test_measured_matte.pynew file mode 100644index 0000000..4009113--- /dev/null+++ b/tests/frames/test_measured_matte.py@@ -0,0 +1,157 @@+# SPDX-License-Identifier: Apache-2.0+"""The measured background matte keys a flat background of whatever colour was painted.++Synthetic stand-ins for what a model paints when asked for a key: a muted or shifted colour with+noise and a gradient, a darker shade of it round the subject, dark strips along two edges, and a+pocket of background enclosed by the subject. The subject keeps its dark outline and its colours.+"""++from __future__ import annotations++from pathlib import Path++import numpy as np+import pytest+from PIL import Image++from sprite_gen.frames.cutout import cutout, extract_route, key_recognised+from sprite_gen.frames.measured_matte import measure_background, measured_matte++W, H = 160, 200+SKIN = (242, 220, 203)+HAIR = (246, 164, 164)+LINE = (25, 35, 25)+BACKGROUNDS = {+ "muted-green": ((147, 189, 94), (60, 100, 30)), # the colour, and its shade as a model paints it+ "deep-green": ((69, 224, 122), (0, 104, 0)),+ "off-cyan": ((80, 200, 210), (20, 95, 110)),+ "yellowish": ((220, 210, 90), (120, 110, 30)),+}+++def _ellipse(cx: float, cy: float, rx: float, ry: float) -> np.ndarray:+ y, x = np.mgrid[0:H, 0:W]+ return ((x - cx) / rx) ** 2 + ((y - cy) / ry) ** 2 <= 1.0+++def _scene(colour: tuple[int, int, int], shade: tuple[int, int, int] | None, *, strips: bool = True,+ seed: int = 0) -> tuple[Image.Image, dict[str, np.ndarray]]:+ rng = np.random.default_rng(seed)+ gradient = np.linspace(-10, 10, H)[:, None, None]+ img = np.clip(np.array(colour, dtype=np.float32) + gradient + rng.normal(0, 3, (H, W, 3)), 0, 255)+ body = _ellipse(80, 105, 40, 62)+ halo = _ellipse(84, 110, 47, 70) & ~body+ if shade is not None:+ img[halo] = shade+ img[body] = LINE+ inner = _ellipse(80, 105, 37, 59)+ img[inner] = HAIR+ face = _ellipse(80, 80, 22, 22)+ img[face] = SKIN+ pocket = _ellipse(80, 135, 9, 12) # background seen through the subject+ img[pocket] = colour+ line_ring = _ellipse(80, 135, 11, 14) & ~pocket+ img[line_ring] = LINE+ if strips:+ img[:4] = (14, 15, 14)+ img[-3:] = (14, 15, 14)+ masks = {"body": inner & ~line_ring & ~pocket, "outline": body & ~inner, "face": face, "pocket": pocket,+ "halo": halo, "corner": np.zeros((H, W), bool)}+ masks["corner"][5:12, 5:12] = True+ return Image.fromarray(img.astype(np.uint8), "RGB").convert("RGBA"), masks++[email protected]("name", sorted(BACKGROUNDS))+def test_a_flat_background_of_any_colour_is_keyed_and_the_subject_kept(name: str) -> None:+ colour, shade = BACKGROUNDS[name]+ image, masks = _scene(colour, shade)+ out, stats = measured_matte(image)+ alpha = np.asarray(out)[..., 3]+ rgb = np.asarray(out)[..., :3]+ assert stats["route"] == "measured"+ assert np.abs(np.array(stats["background"]) - colour).max() <= 6+ assert (alpha[masks["corner"]] == 0).all()+ assert (alpha[masks["halo"]] == 0).mean() > 0.97, "the background's shade round the subject is background"+ assert (alpha[masks["pocket"]] == 0).mean() > 0.9, "background enclosed by the subject is keyed"+ assert (alpha[:4] == 0).all() and (alpha[-3:] == 0).all(), "the dark edge strips are keyed"+ assert set(stats["edge_strips"]) == {"top", "bottom"}+ assert (alpha[masks["body"]] == 255).all() and (alpha[masks["face"]] == 255).all()+ assert (rgb[masks["face"]] == SKIN).all(), "the subject's colour is byte-identical"+ assert (alpha[masks["outline"]] > 0).mean() > 0.97, "the dark outline is subject, not shade"+ assert stats["key_residue_px"] == 0+++def test_an_off_white_background_is_measured_when_asked() -> None:+ image, masks = _scene((232, 232, 222), None, strips=False)+ out, stats = measured_matte(image)+ alpha = np.asarray(out)[..., 3]+ assert stats["shade"] is False # an achromatic background is read within its spread only+ assert (alpha[masks["corner"]] == 0).all()+ assert (alpha[masks["face"]] == 255).all() and (alpha[masks["body"]] == 255).all()+++def test_the_measurement_skips_edge_strips() -> None:+ image, _ = _scene((147, 189, 94), (60, 100, 30))+ measured = measure_background(np.asarray(image.convert("RGB"), dtype=np.float32))+ assert measured is not None and np.abs(measured["colour"] - (147, 189, 94)).max() <= 6+++def test_a_green_request_on_a_muted_green_is_measured_instead_of_keying_nothing(tmp_path: Path) -> None:+ image, masks = _scene((147, 189, 94), (60, 100, 30))+ assert key_recognised(image, "green")[0] is False+ keyed, stats = extract_route(image, "green")+ assert stats["route"] == "measured"+ assert stats["chroma_key"] == [0, 255, 0]+ assert np.abs(np.array(stats["chroma_key_painted"]) - (147, 189, 94)).max() <= 6+ assert (np.asarray(keyed)[..., 3][masks["corner"]] == 0).all()++ src = tmp_path / "muted.png"+ image.save(src)+ for key in ("green", "auto", "measured"):+ result = cutout(src, tmp_path / f"out_{key}.png", key=key, decontam="auto")+ assert result["route"] == "measured"+ assert result["alpha_zero_pct"] > 20+ assert result["decontam"]["applied"] is False+++def test_a_pure_key_still_goes_to_the_chroma_engine() -> None:+ image, _ = _scene((0, 255, 0), None, strips=False)+ assert key_recognised(image, "green")[0] is True+ _, stats = extract_route(image, "green")+ assert stats["route"] == "extract:green"+++def test_a_cutout_that_keys_nothing_fails(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None:+ import sprite_gen.frames.cutout as cutout_mod++ src = tmp_path / "flat.png"+ image, _ = _scene((147, 189, 94), (60, 100, 30))+ image.save(src)+ monkeypatch.setattr(cutout_mod, "_measured_route", lambda image, *a, **k: (image, {"route": "measured"}))+ with pytest.raises(SystemExit, match="nothing of"):+ cutout(src, tmp_path / "o.png", key="measured")+++def test_a_clip_on_a_muted_green_keys_frame_by_frame(tmp_path: Path) -> None:+ """`video-canvas` normalises such a still to the exact key through `extract_route`; a clip's frames cut one+ by one with `cutout` are measured each; `video-frames` keeps the engine (`measure_unrecognised=False`) and+ refuses such a clip itself."""+ from sprite_gen.video.canvas import normalize_key++ still, masks = _scene((147, 189, 94), (60, 100, 30), seed=1)+ normalized, report = normalize_key(still.convert("RGB"), "green")+ data = np.asarray(normalized)+ assert (data[masks["corner"]] == (0, 255, 0)).all(), "the still's background is repainted to the exact key"+ assert (data[masks["face"]] == SKIN).all()+ for i in range(3):+ frame, masks = _scene((147, 189, 94), (60, 100, 30), seed=10 + i)+ src = tmp_path / f"f{i:04d}.png"+ frame.save(src)+ stats = cutout(src, tmp_path / f"k{i:04d}.png", key="green")+ alpha = np.asarray(Image.open(tmp_path / f"k{i:04d}.png"))[..., 3]+ assert stats["route"] == "measured"+ assert (alpha[masks["halo"]] == 0).mean() > 0.97 and (alpha[masks["face"]] == 255).all()+ keyed, stats = extract_route(frame, "green", measure_unrecognised=False)+ assert stats["route"] == "extract:green"+ quiet = cutout(src, tmp_path / "quiet.png", key="green", measure_unrecognised=False, refuse_empty=False)+ assert quiet["route"] == "extract:green"