Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
docs(video-sway): the hold mask — docs/video-pipeline.md section 6 "Pixels a band leaves as they are" and the CHANGELOG's Unreleased
Soohong Kim32cff00dc813
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7f70604..a110b2e 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,11 +4,13 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the## Unreleased-**Upgrading from 3.5.0: `video-sway` can move each region as one mass with `--band`.** When `--band` is left out, strips, frames, and records remain byte for byte what they were. Passing `--band PX|auto` moves each region's ellipse and an outer margin at full weight, with the weight falling to 0 by cos² over a band of pixels around it, while carrying the region by its own pixels rather than the body's outline lay. The fifth region number `flat` and `--stretch-floor` are refused when `--band` is specified. In the strip's metadata, `follow` gains `band` (and `sway.json` gains `band` for `--frames`), while stdout JSON gains `band` with `band_px`, `margin_px`, and `lay_cells`. The legacy command name `video-follow` accepts the same flags.+**Upgrading from 3.5.0: `video-sway` can move each region as one mass with `--band`.** When `--band` is left out, strips, frames, and records remain byte for byte what they were. Passing `--band PX|auto` moves each region's ellipse and an outer margin at full weight, with the weight falling to 0 by cos² over a band of pixels around it, while carrying the region by its own pixels rather than the body's outline lay. The fifth region number `flat` and `--stretch-floor` are refused when `--band` is specified. In the strip's metadata, `follow` gains `band` (and `sway.json` gains `band` for `--frames`), while stdout JSON gains `band` with `band_px`, `margin_px`, and `lay_cells`. The legacy command name `video-follow` accepts the same flags. `video-sway --band` can now hold pixels with `--hold-mask`, leaving them as they are while regions move round them. It is off by default, and when not given, strips, frames, and records remain byte for byte as before.- **New: banded sway moving regions as one mass (`--band`, `--band-margin`, `--band-up`).** `--band PX|auto` keeps the weight at 1 over each region's ellipse expanded by the margin in every direction (the flat part), falling from 1 to 0 by cos² over a band of `PX` px past it, leaving pixels outside the band unchanged byte for byte. `--band-margin PX|auto` (default `auto`, `ceil(reach) + 1`) sets how far past the ellipse the weight stays 1. `--band auto` sets the band to `ceil(reach x pi / 2) + 1`, the narrowest band that does not fold. Reach is evaluated first at the requested `--gain`, after which band and margin are determined in two stages. The fold rule reads the narrowest band instead of radius, refusing moves where `reach x pi / (2 x band) >= 1` or lowering gain under `--on-fold lower`. `--band-up FLAT[,PX|auto]` shapes the area above the centre separately, stopping the flat part `FLAT` of the vertical radius above the centre and falling over `PX` px (defaulting to the band or `HANG_PAST` of 3 px above the top, whichever is further) so upper regions yield while the rest moves whole. Where the flat ellipse and the outer one differ in shape (above the centre under `--band-up`, where the flat part is short and the band long), the band measured across its ring is narrower than its px, so the weight falls steeper than the fold rule reads. After the fold rule, each region's steepest slope along the move is read over every pixel of every cell (`band_steepest`: per unit of gain, the most of the move at gain 1 dotted with the weight's gradient, the gradient read between each pixel's neighbours, and the cell where it is). A pixel moved by gain `g` keeps `1 - g x steepest` of its area, so a region folds where `g x steepest >= 1`. Then `--on-fold refuse` refuses, saying that `--band`'s weight falls steeper than its px where the band is narrower across its ring, in which cell and what area a pixel keeps at the gain asked, and to lower `--gain` or give a wider `--band` (or `--band-up`, when it was given); `--on-fold lower` lowers that region's gain to the largest whole number of `0.01` steps, no more than the gain it had, whose steepest slope stays under 1 (`steepest_gain`); a region that would have to go under gain 1 is held, and the run is refused only when every region is held, as the fold rule does. Each region is carried across cells in the loop's order from the cell the region was read on by its own solid pixels under the ellipse within `CARRY_SEARCH` (9) px of where the cell before had the region's pixels (around the cell's own body lay only where the cell before was carried by the lay, or is the cell the region was read on; or, with `--reference`, for the first cell) using least squared difference of premultiplied colour and subpixel parabolic fitting; uncertain matches revert to the outline lay and warn on stderr. See [video-pipeline](docs/video-pipeline.md) section 6.-- Python callers: `sway_loop`, `sway_frames`, `read_request`, and `read_ask` accept `band`, `band_margin`, and `band_up` (`band_up` as `(flat, px_or_"auto")` or command-line string). New in `sprite_gen.video.sway`: `band_weight`, `band_shapes`, `carry_match`, `carry_own`, `band_steepest`, `steepest_gain`, `band_ledger`, `band_notes`, and `jacobian` (moved from `sway_inspect`, which retains it under the same name); constants `BAND_POLICY`, `CARRY_BASIS_BAND`, `CARRY_SEARCH`, `HANG_PAST`, `PRESSED`, `STRETCHED`, `MOVED_PX`, and `AUTO`; and methods `Answer.band_field(k, xx, yy)` and `Answer.band_record()`.+- **New: holding pixels through banded sway (`--hold-mask`).** `--hold-mask MASK.png|DIR` leaves pixels as they are while a band moves its regions round them, requiring `--band` and off by default, byte for byte as before when not given. The mask is a PNG drawn on the cell the regions were read on (holding pixels where alpha is 128 or more; where the PNG has no alpha, grey 128 or more) or a folder of PNGs giving one mask per cell in file-name order. One mask is split into 8-connected pieces, each carried into every cell by its own solid pixels within `CARRY_SEARCH` (9) px by least squared difference of premultiplied colour and parabolic fitting (reverting to body lay with a warning on stderr when uncertain or covering no solid pixel), then grown by `HOLD_GROW` (3) px. Each cell keeps byte for byte the mask pixels, soft edges, and air within `ceil(reach) + HOLD_EDGE` (2) px. Other pixels are read short of kept pixels along the move direction by `f = min(s, h)` px, where `h` is `HOLD_SLOPE` (0.5) times free distance to kept or margin pixels (`HOLD_MARGIN` of 1 px), bounded to rise by at most `HOLD_SLOPE` per pixel (`read_share`). Reads touching forbidden pixels are renormalised or walked back by 1/`HOLD_STEPS` (1/16) of their move. Metadata gains `follow.hold` (and `sway.json` gains `hold` for `--frames`), while stdout JSON gains `hold` with `pieces`, `lay_cells`, and `read_folded_px_max`.+- Python callers: `sway_loop`, `sway_frames`, `read_request`, and `read_ask` accept `band`, `band_margin`, and `band_up` (`band_up` as `(flat, px_or_"auto")` or command-line string). New in `sprite_gen.video.sway`: `band_weight`, `band_shapes`, `carry_match`, `carry_own`, `band_steepest`, `steepest_gain`, `band_ledger`, `band_notes`, and `jacobian` (moved from `sway_inspect`, which retains it under the same name); constants `BAND_POLICY`, `CARRY_BASIS_BAND`, `CARRY_SEARCH`, `HANG_PAST`, `PRESSED`, `STRETCHED`, `MOVED_PX`, and `AUTO`; and methods `Answer.band_field(k, xx, yy)` and `Answer.band_record()`. `sway_loop`, `sway_frames`, `read_request`, and `read_ask` accept `hold_mask` (a Path to a PNG or folder; None: off); new in `sprite_gen.video.sway`: `Hold`, `read_hold`, `Held`, `hold_sets`, `free_back`, `read_share`, `draw_held`, `hold_ledger`, `hold_notes`, and `carry_mask`; constants `HOLD_POLICY`, `HOLD_GROW`, `HOLD_EDGE`, `HOLD_MARGIN`, `HOLD_SLOPE`, `HOLD_ROUND`, and `HOLD_STEPS`; `Ask.hold`, `Answer.hold`, `Answer.held_cell(k)`, and `Answer.hold_record()`; and `carry_own` now calls `carry_mask` (same behaviour).- Record fields: `follow.band` in strip metadata and `band` in `sway.json` record `policy` (`own-pixels/margin-band-cos2/1`), `asked` (`{band, margin, up}`), `reach_px`, `band_px`, and `margin_px`. They record per-region `flat_px` and `band_px` (across, above, below); per region in `follow.band.regions[i]`, `steepest`: `per_gain` (the slope per unit of gain), `cell`, and `ratio` (the region's gain times `per_gain`, under 1 on any written strip); `carry` with `search_px` (9) and per-cell, per-region `px`, `from` (`base`, `pixels`, or `lay`), `why` for lay, `least`, and `curvature`; `lay` entries for cells carried by lay; and `ledger` lists per cell (with `<key>_max` values) for `band_on_regions_px`, `pressed_px`, `stretched_px`, `pressed_box`, `stretched_box`, `area_least`, `area_most`, `outside_moved_px`, and `jacobian_min`. Only the four counts warn: `band_on_regions_px`, `pressed_px`, `stretched_px`, and `outside_moved_px`. For each that is not 0 in some cell, one warning line on stderr says in how many cells, up to how many px and in which cell (for `pressed_px` and `stretched_px` also the least or the most area and where), and ends with the record key: `follow.band.ledger.<key>` on `--loop-dir`, `sway.json band.ledger.<key>` on `--frames`. The lay warning names its record key the same way: `follow.band.lay` on `--loop-dir`, `sway.json band.lay` on `--frames`. `jacobian_min` (the least area of the source any pixel of a moved cell takes, per cell) gives no warning: it is recorded, and 0 or under in any cell refuses the run as the last check, named the engine's fault, nothing written. Command stdout JSON gains `band` containing `band_px`, `margin_px`, and `lay_cells`.+- Record fields: `follow.hold` in strip metadata and `hold` in `sway.json` record `policy` (`own-pixels-grown/kept-edge-air/read-short-sloped/1`), `asked` (`per_cell`, and `sha256`: the file's, or a list per file for a folder), `grow_px` (3; null for a folder), `edge_px`, `margin_px` (1), `slope` (0.5), `round_px` (1), and `steps` (16). They record `pieces` with each piece's `px` and `box` (`x0`, `y0`, `x1`, `y1`) on the cell it was drawn on (empty for a folder); `carry` with `search_px` (9) and per-cell, per-piece `px`, `from` (`base`, `pixels`, or `lay`), `why` for lay, `least`, and `curvature` (null for a folder); `lay` entries (`{cell, piece, why}`) for pieces carried by lay; and `ledger` lists per cell (with `<key>_max` values) for `held_px`, `soft_px`, `air_px`, `short_px`, `renormalised_px`, `walked_px`, `read_folded_px`, `read_folded_edge_px`, `read_least`, `read_turn_folded_px`, and `read_turn_least`. Warning lines on stderr report a piece carried by the lay (in how many cells and why), ending with the record key: `follow.hold.lay` on `--loop-dir`, `sway.json hold.lay` on `--frames`. For `renormalised_px`, `walked_px`, and `read_folded_px`, when not 0 in some cell, one warning line on stderr says in how many cells, up to how many px and in which cell, ending with the record key: `follow.hold.ledger.<key>` on `--loop-dir`, `sway.json hold.ledger.<key>` on `--frames`. Command stdout JSON gains `hold` containing `pieces`, `lay_cells`, and `read_folded_px_max`.- **Tests**: new file `tests/video/test_sway_band.py` covers these cases on synthetic figures. Each case first reads what a sway writes with or without the band (its cells, its fold, whether it refuses, what it says on stderr) and the band's own record after, so a release without the band fails on what comes out:- without `--band`, the strip and the record are as before (no `band`, no `carry_basis`);- `auto` reads the margin and the band off the move's reach at the gain asked, and the fold rule reads the band;@@ -25,6 +27,17 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the- the ledger's counts, boxes and warning lines;- `--frames` gives the strip's cells, and a turned loop under `--read-on` is the cut's banded sway cell for cell.`tests/video/test_follow_inspect.py` now names `--band`, `--band-margin` and `--band-up` among the arguments `video-sway` takes and `video-sway-inspect` does not.+- **Tests**: new file `tests/video/test_sway_hold.py` covers these cases on synthetic figures. Each case first reads what a sway writes with and without the mask (its cells, whether it refuses, what it says), so a release without the mask fails on what comes out; the hold's record and ledger are read after:+ - without `--hold-mask` the band is as before (no `hold` in the record);+ - an arm over the chest, a belt below it, a half-clear edge beside the arm and a clear notch at the waist keep their bytes in every cell; no arm-coloured pixel is drawn off the arm, and no pixel solid within 2 px of the arm is left clear; without the mask the band moves the arm and draws its colour on the chest; record: pieces, carry by own pixels where the body carries them, no lay, some reads short;+ - pixels no read round the mask comes near are the band's byte for byte;+ - a steep band beside the mask (the band's own least area under 0.2, the gain lowered by the fold rule): the arm keeps its bytes and no read takes the source backwards, checked by a direct computation and by the ledger (`read_folded_px` 0 in every cell, and where the read turns from the whole move to its share, `read_turn_folded_px` 0 and the least area over 0);+ - a folder of the carried masks gives the same strip and ledger as the one mask;+ - `--frames` gives the strip's cells and the same hold record;+ - under `--read-on`: a mask drawn on a cut's first cell (on its chest's rim); the held sway differs from the band alone in more than half of the cells, and a loop turned from that cut and swayed with `--read-on` on the cut's reading and the same mask gives the cut's held cells, cell for cell in the turned order, with the read-on relation `transported` and the same carried pixel count per piece in each cell, in the turned order (this case uses the read-on tests' runner figure);+ - each refusal leaves the loop as it was;+ - a piece on cloth with no pattern is carried by the lay, held all the same, and the record, the warning line, the stdout JSON and the ledger keys say so.+ `tests/video/test_follow_inspect.py` now names `--hold-mask` among the arguments `video-sway` takes and `video-sway-inspect` does not.## v3.5.0 - Widened square canvas framing for side walk and run stride and trailing parts, measured background matte cutout for flat backgrounds of any colour, and failure when a cutout removes nothingdiff --git a/docs/video-pipeline.md b/docs/video-pipeline.mdindex e49b8d3..dc59eaf 100644--- a/docs/video-pipeline.md+++ b/docs/video-pipeline.md@@ -601,10 +601,10 @@ A clip model draws a walk's body moving, but a soft part hanging off it (a chestsprite-gen video-sway --loop-dir set/front-walk/loop --region 136,164,60,50 [--region ...] \[--gain 2.5] [--on-fold refuse|lower] [--freq 2.4] [--zeta 0.6] [--board follow.png] \[--read-on <sha256>] [--stretch-floor D] [--gain-across G] \- [--band PX|auto] [--band-margin PX|auto] [--band-up FLAT[,PX|auto]]+ [--band PX|auto] [--band-margin PX|auto] [--band-up FLAT[,PX|auto]] [--hold-mask MASK.png|DIR]sprite-gen video-sway --frames set/front-idle/loop/cycle --out swayed --region cx,cy,rx,ry[,flat] [--fps F] \[--breathe breathed/breathe.json [--lag 0.15]] [--reference FRAME.png] [--gain G] [--gain-across G] [--on-fold ...] [--stretch-floor D] \- [--band PX|auto] [--band-margin PX|auto] [--band-up FLAT[,PX|auto]]+ [--band PX|auto] [--band-margin PX|auto] [--band-up FLAT[,PX|auto]] [--hold-mask MASK.png|DIR]```- **The region** is an ellipse over the part in the strip's first cell, in cell pixels@@ -710,6 +710,20 @@ With `--band`, each region is carried into each cell by its own pixels rather thCombining `--band` with a region's fifth number `flat` (other than 0, which counts as four numbers and is accepted) or with `--stretch-floor` is refused with a reason and a non-zero exit; nothing is written. Out-of-range values (a band of 0 or less, a negative margin, a `FLAT` of 0 or over 1, or text that is not a number or `auto`) are refused by the command line with exit 2. Only four ledger counts warn: `band_on_regions_px`, `pressed_px`, `stretched_px` and `outside_moved_px`. For each that is not 0 in some cell, one warning line on stderr says in how many cells, up to how many px and in which cell (for `pressed_px` and `stretched_px` also the least or the most area and where), ending with the record key: `follow.band.ledger.<key>` on `--loop-dir`, `sway.json band.ledger.<key>` on `--frames`. `jacobian_min` gives no warning. The option works on `--loop-dir` and on `--frames ... --out ...` alike, and under `--read-on` where a turned loop carried from its cut is moved cell for cell as the cut is. The alias `video-follow` takes the same flags. `video-sway-inspect` does not take `--band`.+#### Pixels a band leaves as they are: `--hold-mask`++`--hold-mask MASK.png|DIR` sets pixels that the band moves round and leaves as they are, whatever part of the figure they belong to, such as an arm that hangs over a moving chest, or a belt or a dress below it. The band's move field itself is not changed. It works only with `--band`; without `--band`, it is refused with the message "--hold-mask keeps pixels as they are while a band moves its regions round them; it needs --band" (a non-zero exit, writing nothing, the loop left as it was). It is off by default; when it is not given, nothing of it is read or written, and strips, frames, and records remain byte for byte as with `--band` alone (and as before without `--band`).++The mask is a PNG drawn on the cell the regions were read on (the first cell; the reference frame with `--reference`; or under `--read-on`, the cut's first cell where the loop holds it), pixel for pixel the cell's size. A pixel is held where the PNG's alpha is 128 (`ALPHA_SOLID`) or more; where the PNG has no alpha, where its grey value is 128 or more. When given as one mask, it is split into its 8-connected pieces. Each piece is carried into every cell by its own solid pixels (alpha 128 or more under it), using the same search a band uses to carry a region (`carry_mask`, built out of the band's `carry_own`): in the loop's order from the cell it was drawn on, within `CARRY_SEARCH` (9) px of where the cell before had it, by the least squared difference of premultiplied colour, with a parabola for the subpixel position. Where that is uncertain (such as unpatterned cloth) or the piece covers no solid pixel, that piece is carried by the body's lay in that cell, and a warning line names it. The carried pieces are then grown by `HOLD_GROW` (3) px in 4-neighbour steps, what a carry may be off by where the part held changes its shape. Alternatively, `--hold-mask` takes a folder of PNGs, one per cell, in file-name order; each is held as given in its cell (no carry, no growth).++In each cell, three sets of pixels are kept byte for byte: the mask's pixels (as carried and grown, or as given); the soft edge, which is every pixel that is not opaque (alpha over 0 and under 255) within `edge` 4-neighbour steps of the mask, where `edge = ceil(reach) + HOLD_EDGE` and `HOLD_EDGE` is 2 px (reach is the band's move reach); and the air, which is every clear pixel (alpha 0) within the same distance. This also keeps the half-clear rim of a gap beside the mask, such as a notch at the waist.++Every other pixel is read as the band reads it (premultiplied bilinear, the same sampling), except that a read headed into what is kept stops short. Within a cell, every move is `s` times one direction (the groups' offsets are one motion times their gains). For each pixel, the code finds how far back along that direction the read can go before its four sampled pixels reach a kept pixel or one within `HOLD_MARGIN` (1) px of them (`free_back`, followed an eighth of a pixel at a time). The read goes back `f = min(s, h)` px, where `h` is `HOLD_SLOPE` (0.5) times that free distance, then taken at its least within `HOLD_ROUND` (1) px each way and averaged over as far, then lowered until no pixel is more than `HOLD_SLOPE` over any of its 8 neighbours (`read_share`). The share of the move rises by at most `HOLD_SLOPE` per pixel in any direction, and the gap the part leaves is drawn from the part itself, stretched in order. The read `p - f x o` (`o` the move's unit direction) takes `1 - d_o f` of the source's area: where `f = h` a read takes at least `1 - HOLD_SLOPE` of the source's area, which read between neighbours across and down, as the ledger reads it, is at least `1 - HOLD_SLOPE x (|ox| + |oy|)` (at least `1 - HOLD_SLOPE x sqrt(2)`, about 0.29), while where `f = s` (the whole move) a read takes the band move's own area, over 0 wherever the band does not fold; where `f` turns from the one to the other, the two mix between neighbours and the bound does not hold, and the ledger reads those pixels separately (`read_turn_folded_px`, `read_turn_least`). A read whose four pixels still touch a pixel it may not (a solid pixel may not read any kept pixel; a clear pixel may not read a held one) is read from the others where they carry half its weight or more ("renormalised"), or else walked back by 1/`HOLD_STEPS` (1/16) of its move at a time until they do ("walked"). Where every read takes its whole move and touches no kept pixel, the cell is the band's, byte for byte.++In strip metadata under `follow.hold` (and `hold` in `sway.json` on `--frames`), the record stores `policy` (`own-pixels-grown/kept-edge-air/read-short-sloped/1`), `asked` (`per_cell`, and `sha256`: the file's, or a list per file for a folder), `grow_px` (3; null for a folder), `edge_px`, `margin_px` (1), `slope` (0.5), `round_px` (1), and `steps` (16). It records `pieces` (each piece's `px` and `box` on the cell it was drawn on, empty for a folder), `carry` (`search_px` of 9, and per cell and piece `px`, `from` (`base`, `pixels` or `lay`), `why` for lay, `least`, and `curvature`, null for a folder), and `lay` (`{cell, piece, why}` for each piece carried by the lay). Its `ledger` lists per cell, with `<key>_max` values beside each count: `held_px`, `soft_px`, `air_px`, `short_px`, `renormalised_px`, `walked_px`, `read_folded_px` (opaque pixels, not kept and not beside a kept one, whose read takes 0 or less of the source's area, read off the move each read took, as `jacobian` reads it), `read_folded_edge_px` (the same beside a kept pixel, where the part moves under the mask and the read is meant to jump), `read_least` (the least area a read of the first kind takes, and where: [area, x, y]; null where nothing moves), `read_turn_folded_px`, and `read_turn_least` (the same for pixels of the first kind where the read turns from the whole move to the share round the hold, where the bound does not hold). Warning lines on stderr, one line each, report a piece carried by the lay (in how many cells and why, ending with `follow.hold.lay` on `--loop-dir` or `sway.json hold.lay` on `--frames`), and for `renormalised_px`, `walked_px`, and `read_folded_px` when not 0 in some cell (in how many cells, up to how many px and in which cell, ending with `follow.hold.ledger.<key>` on `--loop-dir` or `sway.json hold.ledger.<key>`). Command stdout JSON gains `hold` with `pieces` (count), `lay_cells`, and `read_folded_px_max`.++A run is refused with a non-zero exit, writing nothing and leaving the loop as it was, when `--hold-mask` is used without `--band`; when a mask is of another size than the cells ("--hold-mask is WxH, the cells are WxH: a mask is drawn on a cell, pixel for pixel"); when a mask holds no pixel ("holds no pixel (alpha 128 or more, or that light where it has no alpha)"); when a file is not read ("...: not read (...)"); when a folder count of PNGs does not match the count of cells ("--hold-mask holds N masks and there are M cells: a folder gives one mask per cell, in name order"); or when a folder has no PNG in it ("a folder of masks with no PNG in it"). The option works on `--loop-dir` and on `--frames ... --out ...` alike, and under `--read-on`. The alias `video-follow` takes the same flag. `video-sway-inspect` does not take `--hold-mask`.+### Inspecting a sway — `video-sway-inspect`The ellipse is given in the first cell, where somebody looked. Into every other cell it is carried by the body's motion, and there nobody looked: an arm that swings across a chest is inside the chest's ellipse in those cells and moves with it, and so does a sleeve of the chest's own colour, which no count of colours finds. The Sway layer does not read what a pixel belongs to. `video-sway-inspect` (old name `video-follow-inspect`) writes what somebody needs in order to look, and moves nothing: