Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

fix(decontam): keep gold and yellow as the subject's own colour on stills

Soohong Kim83e858f304de
diff --git a/docs/chroma-alpha.md b/docs/chroma-alpha.md
index 0fd49c2..6aa4501 100644
--- a/docs/chroma-alpha.md
+++ b/docs/chroma-alpha.md
@@ -91,8 +91,9 @@ B is the key background measured locally (mean of keyed pixels within 12 px). P
| The pixel | Outcome |
|---|---|
| no palette colour explains it (a colour the interior never shows), or, in the still fit, the explanation misses its luma (outline ink darker than any blend) | engine bytes |
-| closer to a palette colour as observed than the blend explains it (margin 10, or 3 noise sigmas of the background) | engine bytes; with a key-free palette, a leftover key hue (every keyed channel above every other by more than 8) loses the excess beyond 8 on its keyed channels, which never raises a channel and leaves colours without the key hue alone (a red or a skin tone under magenta is not magenta) |
-| pushed toward the key clearly more than the fit misses by, but not proven partial coverage | colour only |
+| closer to a palette colour than the blend explains it (margin 10, or 3 noise sigmas of the background): as observed, or, in the still fit, at its own luma within the subject's own spread (below), which then also raises the margin | engine bytes, except where the still fit finds that the matte changed it although it carries no key hue: then its own colour and coverage come back. With a key-free palette, a leftover key hue (every keyed channel above every other by more than 8) loses the excess beyond 8 on its keyed channels, which never raises a channel and leaves colours without the key hue alone (a red or a skin tone under magenta is not magenta) |
+| still fit: deeper than a still's antialiased edge (2 px, the engine's in-band unmix depth) without the key's hue, or on that edge without it and with the colour (at its own luma) of the material right behind it | the subject's own colour, never a blend or a tint; its own colour and coverage come back where the matte changed it |
+| pushed toward the key clearly more than the fit misses by, but not proven partial coverage (in the still fit, not when a shade of a palette colour explains it within the spread) | colour only |
| a blend within the unmix reach (`chroma.unmix_reach`, default 4) | alpha and colour |
| a blend deeper in the 6 px band | colour only, so a key reflection on a solid surface is recoloured rather than made see-through |
| a keyed pixel within 3 px of the subject that fits a blend above the background's noise floor and chains to the subject | coverage recovered: the faint sides of a strand the hard cut erased |
@@ -103,6 +104,8 @@ Licensing: the pass is plain numpy inside this Apache-2.0 repository and install
Two fits. `still` chooses P per pixel. `video` chooses P on a 3×3 mean of the observation, whose chroma is blurred anyway, and averages the projection alpha with a luma-only alpha where P and B differ in luma. `video-frames` uses the video fit and learns one palette per clip on the first frame, so the colours an edge may take cannot change from frame to frame.
+**Gold and yellow.** Gold sits between red and green, so a gold a little darker or yellower than the palette's own also fits as a warmer palette gold with some green mixed in. Up to v2.10.1 the still fit took that reading, and gold edges and small gold drops came out orange and partly see-through. Since v2.10.2 the still fit weighs the blend against the subject's own colour at the pixel's own light (the freedom the written colour already has), and a blend has to win by more than the subject's own spread: the 99th percentile of how far its confident interior sits from the palette, large for a textured gold and small for flat colour. A still's antialiased edge is at most two pixels deep, so deeper in the band a key share has to show as the key's hue: a pixel without it is the subject's own colour, and so is an edge pixel without it that has the colour of the material right behind it. Where the matte changed such a pixel, the source colour and coverage come back. The RGB matte (`cutout`'s default) needs that: it scores key tint on the channel average, which calls yellow green, so it unmixes gold as if it were a blend; the YCbCr matte (`gen`) does not. The video fit keeps its decisions, because a decoded frame's chroma is blurred and an edge pixel's colour is no evidence of what the subject is made of there.
+
Three modes: `palette` runs the pass and fails loud where it cannot (no key hue, or no subject pixel deeper than 6 px to learn from); `auto` runs it wherever it applies and records why not elsewhere; `off` leaves the matte as it is.
| Entry point | Default | Flag |
@@ -112,24 +115,26 @@ Three modes: `palette` runs the pass and fails loud where it cannot (no key hue,
| `sprite-gen video-frames`, `video-set` | `off` | `--decontam off\|auto\|palette` |
| row extractor (`extract`, `inspect`) | `off` | request `chroma.decontam`, or `--decontam` to override; written back to the request once in play |
-Reports record what the pass did under `decontam`: mode, whether it applied (and why not), fit, the palette, and changed / refit / tint / recovered / unexplained pixel counts.
+Reports record what the pass did under `decontam`: mode, whether it applied (and why not), fit, the palette and its material spread, and changed / refit / tint / recovered / unexplained / restored pixel counts.
Measured on a procedural ground-truth set (red strands 0.5–3 px wide, outline and interior ink, highlights, translucent wisps, white cloth and a gold accent on a painted green key; a still, and an H.264 4:2:0 round trip at CRF 16). Key contamination counts edge pixels covered in both output and truth whose hue turned 12° or more toward the key (green residue and the despill's warm drift both turn that way), or, on near-grey truth, whose a*/b* moved toward the key by 8. Halo is the mean CIEDE2000 against the true composite on #0A0A0D and on white:
| Source | Path | Key contamination | Halo dark / white | Strand recall |
|---|---|---|---|---|
| still | RGB engine (default) | 22.5 % | 6.65 / 7.16 | 84.3 % |
-| still | RGB engine + `decontam` (still fit) | 0.07 % | 1.14 / 1.14 | 99.0 % |
+| still | RGB engine + `decontam` (still fit) | 0.03 % | 1.08 / 1.09 | 99.0 % |
| still | YCbCr matte (`gen`) | 19.1 % | 9.77 / 9.36 | 77.3 % |
-| still | YCbCr matte + `decontam` | 0.16 % | 1.15 / 1.14 | 99.0 % |
+| still | YCbCr matte + `decontam` | 0.12 % | 1.09 / 1.08 | 99.0 % |
| H.264 frames | RGB engine + `--spill full` | 33.2 % | 12.79 / 12.43 | 82.1 % |
| H.264 frames | + `decontam` (video fit) | 10.8 % | 9.38 / 8.73 | 91.4 % |
+The two still-fit rows are v2.10.2's; v2.10.0 measured 0.07 % · 1.14 / 1.14 and 0.16 % · 1.15 / 1.14 on the same set, with the same recall. The video-fit row is unchanged.
+
**Defaults.** Every entry point defaults to `off`, which returns the engine's output byte for byte; pass `--decontam auto` (or `palette`) to use the pass. Stills defaulted to `auto` for a while during development. The full test suite then caught a regression on Grok's darker magenta: the key-hue cap greyed the warm colours of the palette, so a hard-edged brown square lost its opaque edge. The cap is fixed and tested, but the measurements above use a green key, so the default stays `off`. The row extractor stays `off` in any case: its frames are a derived cache that `heal` re-derives from raw, so a changed default would silently rewrite every existing run, and pixel-art rows binarize alpha downstream anyway. `video-frames` / `video-set` stay `off` (see the video numbers above).
Limitations:
-- An edge takes one of the subject's interior colours at its observed brightness. A thin feature whose colour appears nowhere inside the subject keeps the engine's pixels.
+- An edge takes one of the subject's interior colours at its observed brightness. A thin feature whose colour appears nowhere inside the subject keeps the engine's pixels. On the RGB matte that includes the matte's reading of a thin yellow line or dot, with no pixel deeper than two pixels, as a green blend.
- In video a thin strand's chroma is gone before keying, so its colour is chosen from blurred chroma; strands come back in the hair's hue, sometimes a shade darker or paler than drawn.
- A subject that owns key-coloured material keeps it in its palette, and its edges are repainted with whatever explains them best, key colour included. Keep the default when that material matters.
- The pixel-unfake path binarizes alpha downstream, so on pixel-art rows only edge colours change.
diff --git a/sprite_gen/frames/decontam.py b/sprite_gen/frames/decontam.py
index 8d5cec1..065f0ac 100644
--- a/sprite_gen/frames/decontam.py
+++ b/sprite_gen/frames/decontam.py
@@ -38,6 +38,22 @@ Guards, in order of the decision they make:
when the subject owns no key-coloured material is a residual key hue on such a
pixel capped (`KEY_HUE_TINT`), which lowers the keyed channels and never raises
one.
+- *Own colour (still fit).* Gold sits between red and green, so a gold a little
+ darker or yellower than the palette's own also reads as a warmer palette gold
+ with some green mixed in, and came out orange and see-through. The still fit
+ therefore also counts a palette colour at the pixel's own luma (the freedom the
+ written colour has too) when it comes within the subject's own spread: how far
+ its confident interior sits from the palette (`MATERIAL_SPREAD_QUANTILE`), which
+ also raises the margin a blend has to win by. And a still's antialiased edge is
+ at most `_IN_BAND_UNMIX_KEY_DEPTH` pixels deep, so deeper in the band a key share
+ shows only as the key's hue: a pixel without it is the subject's own colour, and
+ so is an edge pixel without it that has the colour of the plain material right
+ behind it. Such pixels are never blends or tints. Where the matte changed one of
+ them although it carries no key hue, the source colour and coverage come back:
+ the RGB matte scores tint on the channel average, which calls yellow green (and
+ red or blue magenta), so it unmixes gold as if it were a blend. The video fit
+ does none of this, since a decoded frame's chroma is blurred and an edge pixel's
+ colour is no evidence of what the subject is made of there.
- *Tint.* Between the two: a pixel pushed from its colour toward the key by at
least `TINT_SHIFT`, and `TINT_RATIO` times more than the fit misses by, is
recoloured with its coverage left alone (a faint cast, not proven coverage).
@@ -65,7 +81,8 @@ from __future__ import annotations
from typing import Any
from sprite_gen._deps import np
-from sprite_gen.frames.extract import _grow_chebyshev, _grow_into, _key_channel_split, _key_excess_field
+from sprite_gen.frames.extract import (_IN_BAND_UNMIX_KEY_DEPTH, _grow_chebyshev, _grow_into, _key_channel_split,
+ _key_excess_field)
# "palette" runs the pass and fails loud when it cannot; "auto" runs it wherever it applies (a
# chroma key, a subject interior to learn from) and otherwise reports why it did not.
@@ -81,6 +98,10 @@ PALETTE_ITERATIONS = 10
KEY_HUE_TINT = 8.0 # the key-hue bar of the full spill pass (extract._SPILL_FULL_MIN_TINT)
KEY_MATERIAL_MAX_SHARE = 0.005 # the key-material share `video-frames --spill auto` tolerates
OPAQUE_MARGIN = 10.0 # a blend must explain the pixel this much better than "opaque as observed"
+# ... and better than the subject's own interior sits from its palette: material this far from every
+# palette colour is still material (the quantile of the confident interior's distance to the palette)
+MATERIAL_SPREAD_QUANTILE = 99.0
+MATERIAL_SPREAD_SAMPLES = 1 << 16 # interior pixels the spread is measured on, at a fixed stride
NOISE_SIGMAS = 3.0
MIN_RECOVERED_ALPHA = 0.06
LUMA_TRUST_ALPHA = 0.35 # below this coverage the observed luma is noise; lean on the palette
@@ -125,6 +146,18 @@ def _box_sum(values: np.ndarray, radius: int) -> np.ndarray:
return table[y1][:, x1] - table[y0][:, x1] - table[y1][:, x0] + table[y0][:, x0]
+def _box_sum_at(values: np.ndarray, radius: int, rows: np.ndarray, cols: np.ndarray) -> np.ndarray:
+ """`_box_sum` of a 2-D array, at the given pixels only: one summed-area table, no full-size result."""
+ height, width = values.shape
+ table = np.zeros((height + 1, width + 1), dtype=np.float64)
+ table[1:, 1:] = values.cumsum(0).cumsum(1)
+ y0 = np.clip(rows - radius, 0, height)
+ y1 = np.clip(rows + radius + 1, 0, height)
+ x0 = np.clip(cols - radius, 0, width)
+ x1 = np.clip(cols + radius + 1, 0, width)
+ return table[y1, x1] - table[y0, x1] - table[y1, x0] + table[y0, x0]
+
+
def local_background(rgb: np.ndarray, preferred: np.ndarray, fallback: np.ndarray, radius: int) -> np.ndarray:
"""Mean keyed colour within `radius`, from `preferred` pixels where any, else from `fallback`."""
out = np.zeros(rgb.shape, dtype=np.float64)
@@ -194,6 +227,20 @@ def recolor_to_luma(colour: np.ndarray, luma: np.ndarray) -> np.ndarray:
return np.where(darker[:, None], colour * scale[:, None], colour + lift[:, None] * (255 - colour))
+def material_distance(obs: np.ndarray, palette: np.ndarray) -> np.ndarray:
+ """How far each observation (N, 3) is from being the subject's own material: the nearest
+ palette colour as it is, or at the observation's luma (the shading the colour written by
+ `decontaminate` is granted too)."""
+ obs = np.asarray(obs, dtype=np.float64)
+ light = obs @ LUMA
+ nearest = np.full(len(obs), np.inf)
+ for colour in np.asarray(palette, dtype=np.float64):
+ shaded = recolor_to_luma(np.broadcast_to(colour, obs.shape), light)
+ nearest = np.minimum(nearest, np.minimum(np.linalg.norm(obs - colour, axis=-1),
+ np.linalg.norm(obs - shaded, axis=-1)))
+ return nearest
+
+
def _box3(rgb: np.ndarray, rows: np.ndarray, cols: np.ndarray) -> np.ndarray:
"""3x3 mean (edge-clamped) of `rgb` at the given pixels."""
height, width = rgb.shape[:2]
@@ -220,9 +267,11 @@ def not_applicable(keyed: np.ndarray, keyed_mask: np.ndarray, chroma_key: tuple[
def subject_palette(keyed: np.ndarray, keyed_mask: np.ndarray, chroma_key: tuple[int, int, int]) -> dict[str, Any]:
"""The subject's colours, learned from its confident interior (opaque, deeper than `BAND_PX`).
- Returns {"palette": (k, 3) float64, "keyfree": bool, "key_material_share": float}. A clip
- can learn this once and hand it to every frame, so the colours an edge may take do not
- change from frame to frame. Raises SystemExit when `not_applicable` says why it cannot.
+ Returns {"palette": (k, 3) float64, "keyfree": bool, "key_material_share": float,
+ "material_spread": float}, the last being how far the interior itself sits from the
+ palette (`material_distance`, its `MATERIAL_SPREAD_QUANTILE`). A clip can learn this once
+ and hand it to every frame, so the colours an edge may take do not change from frame to
+ frame. Raises SystemExit when `not_applicable` says why it cannot.
"""
reason = not_applicable(keyed, keyed_mask, chroma_key)
if reason is not None:
@@ -241,23 +290,28 @@ def subject_palette(keyed: np.ndarray, keyed_mask: np.ndarray, chroma_key: tuple
palette = np.round(learn_palette(samples))
if keyfree:
palette = _cap_key_hue(palette, keyed_channels, unkeyed_channels, KEY_HUE_TINT)
- return {"palette": palette, "keyfree": keyfree, "key_material_share": key_material}
+ measured = samples[::max(1, len(samples) // MATERIAL_SPREAD_SAMPLES)]
+ spread = round(float(np.percentile(material_distance(measured, palette), MATERIAL_SPREAD_QUANTILE)), 3)
+ return {"palette": palette, "keyfree": keyfree, "key_material_share": key_material, "material_spread": spread}
def palette_from_stats(stats: dict[str, Any]) -> dict[str, Any]:
"""The `subject_palette` a `decontaminate` call used, rebuilt from its stats."""
return {"palette": np.asarray(stats["palette"], dtype=np.float64), "keyfree": bool(stats["palette_keyfree"]),
- "key_material_share": float(stats["key_material_share"])}
+ "key_material_share": float(stats["key_material_share"]),
+ "material_spread": float(stats["material_spread"])}
def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndarray,
chroma_key: tuple[int, int, int], *, fit: str = "still", alpha_depth: int = 4,
- palette: dict[str, Any] | None = None, mode: str = "palette") -> tuple[np.ndarray, dict[str, Any]]:
+ palette: dict[str, Any] | None = None, mode: str = "palette",
+ source_alpha: np.ndarray | None = None) -> tuple[np.ndarray, dict[str, Any]]:
"""Re-explain the edge of an already keyed image. Returns (RGBA uint8, stats).
- `source_rgb` (H, W, 3) is the frame as generated, `keyed` (H, W, 4) the engine's
- output for it and `keyed_mask` the engine's hard-cut background (transparent input
- included). Interior pixels (deeper than `BAND_PX`) are returned byte-identical.
+ `source_rgb` (H, W, 3) is the frame as generated and `source_alpha` (H, W) its own
+ coverage (None: opaque), `keyed` (H, W, 4) the engine's output for it and `keyed_mask`
+ the engine's hard-cut background (transparent input included). Interior pixels
+ (deeper than `BAND_PX`) are returned byte-identical.
`palette` is a `subject_palette` result to reuse (a clip's); None learns this frame's.
`mode="auto"` returns the input unchanged, with the reason in the stats, where the
pass does not apply; `mode="palette"` raises SystemExit there instead.
@@ -282,6 +336,10 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
palette_rgb = np.asarray(learned["palette"], dtype=np.float64)
keyfree = bool(learned["keyfree"])
key_material = float(learned["key_material_share"])
+ spread = float(learned["material_spread"])
+ # the still fit reads an edge pixel's colour as evidence of the subject's own material; a decoded
+ # video frame's chroma is blurred, so there the colour is no such evidence and the fit decides alone
+ material_test = fit == "still"
far_background = keyed_mask & ~flank
background = local_background(rgb, far_background, keyed_mask, BACKGROUND_RADIUS)
@@ -291,14 +349,23 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
sigma_rgb = 1.4826 * float(np.median(np.linalg.norm(residual, axis=-1)))
else:
sigma_luma = sigma_rgb = 0.0
- margin = max(OPAQUE_MARGIN, NOISE_SIGMAS * sigma_rgb)
+ margin = max(OPAQUE_MARGIN, NOISE_SIGMAS * sigma_rgb, spread if material_test else 0.0)
rows, cols = np.nonzero(band | flank)
+ edge = _IN_BAND_UNMIX_KEY_DEPTH # a still's antialiased edge: the engine's in-band unmix depth
+ if material_test:
+ # the subject's plain material right behind the edge: deeper than it and without the key's hue
+ source_int = np.asarray(source_rgb)[..., :3].astype(np.int32)
+ behind = subject & (depth > edge) & (_key_excess_field(source_int, keyed_channels, unkeyed_channels) <= 0)
+ behind_weight = _box_sum_at(behind.astype(np.float64), edge, rows, cols)
+ behind_mean = np.stack([_box_sum_at(rgb[..., channel] * behind, edge, rows, cols) for channel in range(3)],
+ axis=-1) / np.maximum(behind_weight, 1.0)[:, None]
alpha_new = np.zeros(len(rows))
colour_new = np.zeros((len(rows), 3))
blend = np.zeros(len(rows), dtype=bool)
tinted = np.zeros(len(rows), dtype=bool)
unexplained = np.zeros(len(rows), dtype=bool)
+ own = np.zeros(len(rows), dtype=bool)
for start in range(0, len(rows), _CHUNK):
r = rows[start:start + _CHUNK]
c = cols[start:start + _CHUNK]
@@ -312,7 +379,6 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
towards = choose_obs - choose_bg
best = np.full(len(r), np.inf)
pick = np.zeros(len(r), dtype=np.int64)
- opaque_distance = np.full(len(r), np.inf)
for index, colour in enumerate(palette_rgb):
line = colour - choose_bg
length = np.maximum((line * line).sum(-1), 1e-9)
@@ -321,7 +387,15 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
closer = miss < best
best[closer] = miss[closer]
pick[closer] = index
- opaque_distance = np.minimum(opaque_distance, np.linalg.norm(obs - colour, axis=-1))
+ near = np.min([np.linalg.norm(obs - colour, axis=-1) for colour in palette_rgb], axis=0)
+ if material_test:
+ # the subject's own colour at its own light, as closely as its interior sits to the palette
+ lit = material_distance(obs, palette_rgb)
+ own_colour = lit <= spread
+ material = np.where(own_colour, lit, near)
+ else:
+ own_colour = np.zeros(len(r), dtype=bool)
+ material = near
chosen = palette_rgb[pick]
line = chosen - bg
offset = obs - bg
@@ -339,11 +413,25 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
# pixel darker than the line allows (outline ink the interior never shows) is not a blend
explained &= np.abs(residual @ LUMA) <= LUMA_MISS_SLACK + NOISE_SIGMAS * sigma_luma
in_flank = flank[r, c]
- is_blend = explained & (alpha < 1.0) & (in_flank | (miss + margin < opaque_distance))
+ if material_test:
+ # Deeper than a still's antialiased edge, a key share shows only as the key's hue: a pixel
+ # without it is the subject's own colour. On the edge, so is a pixel without it that has
+ # the colour (at its own luma) of the plain material right behind it.
+ hueless = ~in_flank & (_key_excess_field(obs, keyed_channels, unkeyed_channels) <= 0)
+ deep = depth[r, c] > edge
+ mean = behind_mean[start:start + len(r)]
+ behind_distance = np.minimum(np.linalg.norm(obs - mean, axis=-1),
+ np.linalg.norm(obs - recolor_to_luma(mean, obs @ LUMA), axis=-1))
+ settled = hueless & (deep | ((behind_weight[start:start + len(r)] > 0) & (behind_distance <= margin)))
+ else:
+ settled = np.zeros(len(r), dtype=bool)
+ is_blend = explained & (alpha < 1.0) & (in_flank | (miss + margin < material)) & ~settled
# a tint: displaced from its colour toward the key clearly more than the fit misses by, but
- # not by enough to prove partial coverage. Recoloured, coverage untouched.
+ # not by enough to prove partial coverage, and by more than shading any palette colour
+ # accounts for. Recoloured, coverage untouched.
shift = (1.0 - alpha) * np.sqrt((line * line).sum(-1))
- is_tint = explained & ~is_blend & ~in_flank & (shift >= TINT_SHIFT) & (shift >= TINT_RATIO * miss)
+ is_tint = (explained & ~is_blend & ~in_flank & (shift >= TINT_SHIFT) & (shift >= TINT_RATIO * miss)
+ & ~own_colour & ~settled)
# coverage: refit within the unmix reach and on the flank; colour only deeper in the band
refit = (in_flank | (depth[r, c] <= alpha_depth)) & ~is_tint
floor = np.maximum(MIN_RECOVERED_ALPHA, NOISE_SIGMAS * sigma_luma / np.maximum(np.abs((chosen - bg) @ LUMA), 1e-6))
@@ -359,12 +447,22 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
blend[start:start + len(r)] = is_blend | is_tint
tinted[start:start + len(r)] = is_tint
unexplained[start:start + len(r)] = ~explained
+ own[start:start + len(r)] = material_test & ~is_blend & ~is_tint & ~in_flank & ((material <= margin) | settled)
alpha8 = np.clip(np.round(alpha_new * 255), 0, 255).astype(np.uint8)
colour8 = np.clip(np.round(colour_new), 0, 255).astype(np.uint8)
br, bc = rows[blend], cols[blend]
out[br, bc, :3] = colour8[blend]
out[br, bc, 3] = alpha8[blend]
+ # the subject's own material, with no key hue, that the matte changed anyway (its channel-average
+ # tint calls yellow green): the source pixel and its coverage come back
+ source8 = np.asarray(source_rgb)[rows, cols, :3].astype(np.int32)
+ source_a = np.full(len(rows), 255, dtype=np.int32) if source_alpha is None else np.asarray(source_alpha)[rows, cols].astype(np.int32)
+ back = (own & (_key_excess_field(source8, keyed_channels, unkeyed_channels) <= 0)
+ & ((out[rows, cols, :3] != source8).any(-1) | (out[rows, cols, 3] != source_a)))
+ out[rows[back], cols[back], :3] = source8[back].astype(np.uint8)
+ out[rows[back], cols[back], 3] = source_a[back].astype(np.uint8)
+ restored = int(np.count_nonzero(back))
clamped = 0
if keyfree:
kept = ~blend & band[rows, cols] & (out[rows, cols, 3] > 0)
@@ -395,12 +493,14 @@ def decontaminate(source_rgb: np.ndarray, keyed: np.ndarray, keyed_mask: np.ndar
"palette_keyfree": bool(keyfree),
"key_material_share": round(key_material, 5),
"noise_sigma_rgb": round(sigma_rgb, 3),
+ "material_spread": round(spread, 3),
"opaque_margin": round(margin, 3),
"changed_px": changed,
"refit_px": int(np.count_nonzero(blend & ~tinted)),
"tint_px": int(np.count_nonzero(tinted)),
"recovered_px": recovered,
"unexplained_px": int(np.count_nonzero(unexplained & band[rows, cols])),
+ "restored_px": restored,
"key_hue_capped_px": clamped,
}
return out, stats
diff --git a/sprite_gen/frames/extract.py b/sprite_gen/frames/extract.py
index 0811305..4569856 100644
--- a/sprite_gen/frames/extract.py
+++ b/sprite_gen/frames/extract.py
@@ -562,6 +562,7 @@ def remove_chroma_background(
width, height = rgba.size
data = np.array(rgba, dtype=np.uint8) # (H, W, 4); written back at the end
source_rgb = data[..., :3].astype(np.int32)
+ source_alpha = data[..., 3].copy() if decontam != "off" else None # coverage decontam may give back
keyed_channels, unkeyed_channels = _key_channel_split(chroma_key)
unseen = 255
@@ -718,7 +719,8 @@ def remove_chroma_background(
data[y, x] = (*despilled, alpha)
if decontam != "off":
data, stats = decontam_module.decontaminate(source_rgb, data, keyed_mask, chroma_key, fit=decontam_fit,
- alpha_depth=unmix_reach, palette=decontam_palette, mode=decontam)
+ alpha_depth=unmix_reach, palette=decontam_palette, mode=decontam,
+ source_alpha=source_alpha)
if decontam_stats is not None:
decontam_stats.update(stats)
# Back into the converted copy rather than a fresh Image.fromarray, so the
@@ -1105,7 +1107,8 @@ def remove_chroma_background_ycbcr(
keyed_mask, _ = hard_key_mask(source_rgb, source[..., 3], chroma_key, painted, DEFAULT_KEY_THRESHOLD)
data, stats = decontam_module.decontaminate(source_rgb, np.array(out, dtype=np.uint8), keyed_mask, chroma_key,
fit=decontam_fit, alpha_depth=DEFAULT_UNMIX_REACH,
- palette=decontam_palette, mode=decontam)
+ palette=decontam_palette, mode=decontam,
+ source_alpha=source[..., 3])
if decontam_stats is not None:
decontam_stats.update(stats)
out.frombytes(data.tobytes())
diff --git a/sprite_gen/video/frames.py b/sprite_gen/video/frames.py
index d3431f5..238558f 100644
--- a/sprite_gen/video/frames.py
+++ b/sprite_gen/video/frames.py
@@ -70,7 +70,7 @@ EDGE_MAX_PIXELS = 0 # any opaque pixel on the top/left/right edge band = contac
# learned on the first frame, so the colours an edge may take cannot change between frames.
DECONTAM_MODES = ("off", "auto", "palette")
DECONTAM_FIT = "video"
-DECONTAM_STAT_KEYS = ("changed_px", "refit_px", "tint_px", "recovered_px", "unexplained_px", "key_hue_capped_px")
+DECONTAM_STAT_KEYS = ("changed_px", "refit_px", "tint_px", "recovered_px", "unexplained_px", "restored_px", "key_hue_capped_px")
def _require(binary: str) -> str:
@@ -279,6 +279,7 @@ def key_frames(
"palette": decontam_first["palette"],
"palette_keyfree": decontam_first["palette_keyfree"],
"key_material_share": decontam_first["key_material_share"],
+ "material_spread": decontam_first["material_spread"],
"applied_frames": decontam_applied,
"totals": decontam_totals,
}
diff --git a/tests/frames/test_decontam_gold.py b/tests/frames/test_decontam_gold.py
new file mode 100644
index 0000000..465ba01
--- /dev/null
+++ b/tests/frames/test_decontam_gold.py
@@ -0,0 +1,150 @@
+"""`decontam` keeps gold and yellow as they are (synthetic fixture, known coverage and colour).
+
+Gold sits between red and green: a gold a little yellower or darker than the palette's own
+reads as a warmer palette gold with some green mixed in. Before, the pass took that reading,
+so gold edges and small gold drops came out orange and partly see-through. The RGB matte
+(`cutout`) makes it worse on its own: it scores key tint on the channel average, which calls
+yellow green, so it unmixes gold as if it were a blend. The scene: textured gold strokes
+with a crimson outline, drops of a darker shade of that gold, and drops of a yellower gold
+the strokes never show, drawn at 4x and box-reduced so every edge is a blend, on the pure
+and on a painted green key.
+"""
+from __future__ import annotations
+
+import numpy as np
+import pytest
+from PIL import Image
+
+from sprite_gen.frames.cutout import extract_route
+from sprite_gen.frames.extract import _grow_chebyshev, remove_chroma_background_ycbcr
+
+GREEN = (0, 255, 0)
+SIZE, SCALE = 160, 4
+EDGE_BAND = 6 # the pass's band: truth pixels this close to the background are the ones it may touch
+
+
+def _hsv(hue: np.ndarray, sat: float, val: np.ndarray) -> np.ndarray:
+ """Vectorised HSV (hue in degrees) to RGB 0-255."""
+ h = (hue % 360) / 60.0
+ c = val * sat
+ x = c * (1 - np.abs(h % 2 - 1))
+ zero = np.zeros_like(h)
+ sector = np.floor(h).astype(int) % 6
+ rgb = np.select([sector[..., None] == k for k in range(6)],
+ [np.stack(v, -1) for v in ((c, x, zero), (x, c, zero), (zero, c, x), (zero, x, c), (x, zero, c), (c, zero, x))])
+ return (rgb + (val - c)[..., None]) * 255
+
+
+def _scene(key: tuple[int, int, int]) -> tuple[Image.Image, np.ndarray, np.ndarray, np.ndarray]:
+ """(raw still, true coverage, true colour, gold mask) at SIZE x SIZE."""
+ n = SIZE * SCALE
+ yy, xx = np.mgrid[0:n, 0:n] / SCALE
+ cover = np.zeros((n, n), dtype=bool)
+ colour = np.zeros((n, n, 3))
+ gold = np.zeros((n, n), dtype=bool)
+ stroke = ((np.abs(yy - 45) < 16) & (xx > 12) & (xx < 148)) | ((np.abs(xx - 42) < 14) & (yy > 30) & (yy < 150))
+ outline = stroke.copy()
+ for _ in range(3 * SCALE):
+ outline = _grow_chebyshev(outline)
+ outline &= ~stroke
+ cover |= outline
+ colour[outline] = (140, 12, 14)
+ # a dry-brush gold: hue 41-48 degrees, value 0.80-1.00
+ hue = 44.5 + 3.5 * np.sin(xx / 4.7) * np.cos(yy / 6.3)
+ val = 0.9 + 0.1 * np.sin(xx / 2.9 + yy / 3.7)
+ cover |= stroke
+ colour[stroke] = _hsv(hue, 0.76, val)[stroke]
+ gold |= stroke
+ # drops the strokes' interior never shows: a darker shade of their gold, and a yellower gold
+ for cy, cx, radius, drop_hue, shade in ((100, 90, 4.5, 47.0, 0.80), (120, 118, 5.5, 46.0, 0.84), (140, 95, 4.0, 48.0, 0.82),
+ (98, 130, 5.0, 54.0, 0.97), (130, 142, 6.0, 55.0, 0.95), (148, 120, 4.5, 53.0, 0.98)):
+ drop = (yy - cy) ** 2 + (xx - cx) ** 2 <= radius ** 2
+ cover |= drop
+ colour[drop] = _hsv(np.full(int(drop.sum()), drop_hue), 0.74, np.full(int(drop.sum()), shade))
+ gold |= drop
+ big = np.where(cover[..., None], colour, np.asarray(key, dtype=np.float64))
+ raw = big.reshape(SIZE, SCALE, SIZE, SCALE, 3).mean((1, 3))
+ coverage = cover.reshape(SIZE, SCALE, SIZE, SCALE).mean((1, 3))
+ weight = np.maximum(cover.reshape(SIZE, SCALE, SIZE, SCALE).sum((1, 3)), 1)[..., None]
+ true_colour = (colour * cover[..., None]).reshape(SIZE, SCALE, SIZE, SCALE, 3).sum((1, 3)) / weight
+ gold_share = gold.reshape(SIZE, SCALE, SIZE, SCALE).mean((1, 3))
+ image = Image.fromarray(np.round(raw).astype(np.uint8)).convert("RGBA")
+ return image, coverage, true_colour, gold_share >= 0.999
+
+
+def _hue(rgb: np.ndarray) -> np.ndarray:
+ rgb = np.asarray(rgb, dtype=np.float64)
+ top, low = rgb.max(-1), rgb.min(-1)
+ span = np.maximum(top - low, 1e-9)
+ r, g, b = rgb[..., 0], rgb[..., 1], rgb[..., 2]
+ hue = np.where(top == r, (g - b) / span % 6, np.where(top == g, (b - r) / span + 2, (r - g) / span + 4))
+ return hue * 60.0
+
+
+def _solid_gold_at_the_edge(coverage: np.ndarray, is_gold: np.ndarray) -> np.ndarray:
+ """Fully covered gold pixels within the pass's band of the background."""
+ near = coverage < 0.02
+ for _ in range(EDGE_BAND):
+ near = _grow_chebyshev(near)
+ return is_gold & (coverage >= 0.999) & near
+
+
+def _scores(out: Image.Image, target: np.ndarray, true_colour: np.ndarray) -> tuple[float, np.ndarray]:
+ """(share of target pixels below alpha 250, signed hue change of every target pixel)."""
+ rgba = np.asarray(out.convert("RGBA")).astype(np.float64)
+ loss = float((rgba[target][:, 3] < 250).mean())
+ shift = (_hue(rgba[target][:, :3]) - _hue(true_colour[target]) + 180) % 360 - 180
+ return loss, shift
+
+
+MATTES = {
+ "ycbcr (gen)": lambda image, **kw: remove_chroma_background_ycbcr(image, GREEN, [], **kw),
+ "rgb (cutout)": lambda image, **kw: extract_route(image, "green", **kw)[0],
+}
+
+
[email protected]("key", [GREEN, (26, 217, 30)], ids=["pure key", "painted key"])
[email protected]("matte", list(MATTES))
+def test_decontam_leaves_gold_solid_and_gold(key: tuple[int, int, int], matte: str) -> None:
+ image, coverage, true_colour, is_gold = _scene(key)
+ target = _solid_gold_at_the_edge(coverage, is_gold)
+ assert target.sum() > 400
+ loss, shift = _scores(MATTES[matte](image, decontam="palette"), target, true_colour)
+ assert loss <= 0.01, f"{loss:.1%} of solid gold went see-through"
+ assert abs(float(shift.mean())) <= 1.0, f"gold hue moved {shift.mean():.2f} degrees on average"
+ assert float(np.percentile(shift, 5)) >= -2.0, f"gold turned orange: 5th percentile {np.percentile(shift, 5):.2f} degrees"
+
+
+def test_the_rgb_matte_alone_turns_gold_orange_and_the_pass_undoes_it() -> None:
+ image, coverage, true_colour, is_gold = _scene(GREEN)
+ target = _solid_gold_at_the_edge(coverage, is_gold)
+ matte_loss, matte_shift = _scores(MATTES["rgb (cutout)"](image), target, true_colour)
+ fixed_loss, fixed_shift = _scores(MATTES["rgb (cutout)"](image, decontam="palette"), target, true_colour)
+ # the channel-average tint reads yellow as green: the matte itself unmixes gold
+ assert matte_loss >= 0.2 and float(np.percentile(matte_shift, 5)) <= -5.0
+ assert fixed_loss <= 0.01 and float(np.percentile(fixed_shift, 5)) >= -2.0
+
+
[email protected]("matte", list(MATTES))
+def test_the_green_blend_around_gold_is_still_cleaned(matte: str) -> None:
+ """Keeping gold must not stop the pass: the green fringe the edge blends leave is still removed."""
+ image, coverage, true_colour, is_gold = _scene(GREEN)
+ fringe = (coverage > 0.1) & (coverage < 0.9)
+
+ def visible_green(out: Image.Image) -> int:
+ rgba = np.asarray(out.convert("RGBA")).astype(np.int32)
+ excess = rgba[..., 1] - np.maximum(rgba[..., 0], rgba[..., 2])
+ return int((fringe & (rgba[..., 3] > 25) & (excess > 8)).sum())
+
+ before = visible_green(MATTES[matte](image))
+ after = visible_green(MATTES[matte](image, decontam="palette"))
+ assert after <= before // 4, (before, after)
+
+
+def test_only_the_still_fit_takes_the_edge_colour_as_material() -> None:
+ """The video fit keeps its decisions: a decoded frame's chroma is blurred, so an edge colour proves nothing."""
+ image, *_ = _scene(GREEN)
+ _, still = extract_route(image, "green", decontam="palette")
+ _, video = extract_route(image, "green", decontam="palette", decontam_fit="video")
+ assert still["decontam"]["restored_px"] > 0
+ assert video["decontam"]["restored_px"] == 0