Skip to content

Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.

26.6 KB
Raw Blame

sprite-gen — Implemented Architecture

Owns: How the code is laid out: domains, stage ownership, the numeric SSoT, the cell model, extraction internals, runtime manifest · Index: docs/README.md

Status: reference (v2.2.0). Canonical behavior contract lives in ../SKILL.md; this doc explains how the code realizes that contract. If this doc and SKILL.md ever disagree, SKILL.md wins and this doc is the bug. The documentation index is README.md.

Contents

0. Domains — the taxonomy

sprite_gen/_modules.py is the one table that says which domain a module belongs to. sprite-gen --help, the scripts/ wrappers and the docs index enumerate from it; a module that is not in the table cannot be a verb (the CLI raises instead of inventing a group). Every domain package is a folder under sprite_gen/.

flowchart TB
    subgraph pkg["sprite_gen/  (_modules.py owns the mapping)"]
        direction LR
        spec["spec<br/>read-only asset adapters · sprite-request schema · layout resolver · runio · migrations"]
        gen["gen<br/>prepare · gen (codex / grok / openai) · gen-set · video (Grok Imagine)"]
        video["video<br/>canvas · frames · loop · batch (video-set)"]
        frames["frames<br/>extract · cutout · slice-sheet · unpack-atlas · segment"]
        curate["curate<br/>curation sidecar · direction anchors"]
        compose["compose<br/>atlas · cycle · gif · layers · export-pngs · export-aseprite"]
        background["background<br/>repeating RGBA tiles · join inspection"]
        effects["effects<br/>breathe · anatomy · recolor · interpolate · projected shadows"]
        qa["qa<br/>inspect-motion · inspect-holds · inspect · score · preview · correction-loop"]
        scene["scene<br/>placement · planes · camera · light · render · inspect"]
        serve["serve<br/>curation webview · composition canvas"]
    end
    cli["cli.py — one entrypoint, verbs grouped by domain"] --> pkg
    spec -.read by every stage.-> gen & frames & compose & qa

1. Pipelines, tools and scenes

A character image plus a numeric request becomes a transparent sprite atlas (pipeline A) or transparent motion loops (pipeline B). C/D/E are independent tool groups: imported image processing, finished-sheet refinement, and background/shadow/motion tools. Scene creation (S) is an optional workflow consuming existing assets. _modules.py owns separate PIPELINES, TOOL_GROUPS and WORKFLOWS catalogs. Domains express code ownership; workflows express the result; pipelines express an execution order.

flowchart TD
    subgraph A["A · atlas rows  (docs/run-contract.md)"]
        REQ["sprite-request.json<br/>(numeric SSoT)"] --> PREP["prepare"]
        PREP --> GUIDES["references/layout-guides/&lt;state&gt;.png<br/>prompts/&lt;state&gt;.txt"]
        GUIDES --> GEN["gen · gen-set<br/>(codex or grok; one strip per state — the only AI step)"]
        GEN --> RAW["raw/&lt;state&gt;.png"]
        RAW --> EXTRACT["extract<br/>chroma removal → connected components<br/>→ fit_to_cell or pixel-unfake"]
        EXTRACT --> FRAMES["frames/&lt;state&gt;/frame-N.png<br/>frames/frames-manifest.json"]
        FRAMES --> CURATE["curation (webview)<br/>curation.json sidecar"]
        FRAMES --> COMPOSE["compose-atlas"]
        CURATE -. "selected / transforms / breathe" .-> COMPOSE
        COMPOSE --> ATLAS["sprite-sheet-alpha.png + manifest.json<br/>(frame_layout = runtime SSoT)"]
        FRAMES --> QA["preview · inspect · score"]
    end
    subgraph B["B · video → loop  (docs/video-pipeline.md)"]
        STILL["one still on a flat key"] --> CANVAS["video-canvas<br/>state canvas (tall / wide / square)"]
        CANVAS --> VID["video<br/>Grok Imagine, in-place motion"]
        VID --> VFR["video-frames<br/>ffmpeg + cutout keying"]
        VFR --> LOOP["video-loop<br/>true period or one-shot → strip · GIF · WebP"]
        SET["video-set"] -. "directions × states" .-> CANVAS
    end
    subgraph C["C · utilities"]
        CUT["cutout<br/>white matte / chroma engine"] ~~~ SLICE["slice-sheet"] ~~~ UNPACK["unpack-atlas"]
    end
    subgraph D["D · post-processing"]
        RECOLOR["recolor · recolor-palette"] ~~~ EXPORT["export-pngs · export-aseprite"]
    end
    ATLAS --> D
    UNPACK --> CURATE
    subgraph E["E · independent asset tools"]
        TILE[background-tile] ~~~ SHADOW[shadow] ~~~ MOTION[inspect-motion] ~~~ HOLDS[inspect-holds]
    end
    subgraph S["S · optional scene workflow"]
        INPUT["existing assets + scene.json"] --> RENDER[scene-render]
        INPUT --> CHECK[scene-inspect]
    end

spec.assets adapts external descriptors, native loop strips and runtime atlases into one FrameSequence. Asset metadata owns frame order, native timing and anchor; scene.model owns references, placement, scale, playback rate, planes, camera and light. Scene never rewrites source assets. qa.motion measures contact evidence; scene applies only a verified measurement for the matching asset, in an explicitly chosen direction. effects.shadow.project_shadow is shared by standalone shadow export and scene render. Background generation remains a recipe for existing gen and cutout commands. The complete contracts live in asset-tools and scene.

2. Stage ownership (each script does one job)

The canonical stage → script → input → output table is owned by run-contract.md §1. This section explains the shared-import and locking behavior behind that table.

The scripts are explicit pipeline commands, not hidden imports. The shared imports are curation.py (schema + transform math + the frame_variant/ frame_filename resolver, kept single-source so the webview server and the compose scripts cannot drift) and runio.py (safe run-dir IO: a single-writer lock file .sprite-gen.lock taken by the extract/compose/export/unpack writers, plus atomic temp+os.replace writes for frames, atlas, manifests, and inspect/score/loop reports). The lock enforces the "one worker owns one character folder" rule at runtime — a second writer on the same run dir fails loudly with the holder's pid; a dead holder's lock is reclaimed automatically. curation.json stays outside the pipeline write lock: the webview writes it atomically and compose reads one consistent snapshot, so human edit sessions never block on a running compose. The curation write is, however, serialized against a --force re-import publish via the separate publish rwlock (read_guard/ publish_guard), and a curation POST whose echoed run generation (runRevision) no longer matches is rejected — a stale autosave cannot apply old selections to a re-imported run's frames, even when it keeps the same state names (run-contract.md §4).

The automatic correction loop is intentionally split into three owners: inspect.py measures deterministic signals (frame count, RGB histogram, dHash, motion presence, centroid jitter, existing extraction warnings), score.py turns only that report into scores and correction hints, and correction_loop.py preserves the best candidate while optionally invoking an explicit provider command. In dry-run mode it never generates new pixels; it only proves the defect → score → hint path.

3. The numeric SSoT: sprite-request.json

Every run starts here. It owns the recipe consumed by both prompts and scripts:

{
  "version": 1,
  "kind": "sprite-gen-request",
  "engine": "component-row",
  "character": { "id": "demo-hero", "description": "...", "base_image": "base-source.png" },
  "cell": { "shape": "square", "size": 256, "safe_margin": 24 },
  "chroma_key": { "name": "magenta", "hex": "#FF00FF", "rgb": [255, 0, 255] },
  "chroma": { "mode": "rgb" },
  "states": { "idle": { "frames": 4, "fps": 4, "loop": true, "action": "..." } },
  "fit": { "pixel_unfake": true, "logical_height": 64, "palette_size": 24, "align_x": "foot-centroid", "align_y": "bottom", "segmentation": "components" },
  "style": "..."
}
  • character.base_image is the canonical identity source for the pre-idle phase only (see §5).
  • chroma_key is chosen by sampling the base image (--chroma-key auto) unless forced. The key is picked away from the subject's dominant hue because extraction eats chroma-adjacent tint (behavior contract: chroma-alpha.md).
  • states is a free map; frames/fps/loop/action per state.
  • fit is optional (absent = legacy behavior): resample/align_x/align_y tuning, or the full deterministic pixel_unfake mode (§6.1). Behavior contract: pixel-unfake.md.
  • fit.segmentation is components by default; opt-in projection is the fused-pose recovery path (§6). The extract CLI can explicitly override it.
  • chroma.mode is rgb by default; opt-in ycbcr is owned by chroma-alpha.md. The extract and inspect CLIs can explicitly override it.

4. The cell model (read this — it is the most misunderstood part)

There is exactly one cell object, and it drives all three geometry stages identically:

  1. Generation — draw_guide() renders a guide of frames * cell_width wide × cell_height tall, and row_prompt() tells the model to treat the strip as "invisible cell_width×cell_height frame slots".
  2. Extraction — fit_to_cell() refits each extracted pose into a cell_width×cell_height transparent cell.
  3. Atlas — compose_sprite_atlas.py places every frame into a cell_width×cell_height slot; the sheet is max_frames*cell_width × num_states*cell_height.

4.1 Square vs rect — variabilized, but a single shape

size is a variable, not a hidden constant. The cell may be square or rect:

"cell": { "shape": "square", "size": 256, "safe_margin": 24 }
"cell": { "shape": "rect", "width": 192, "height": 208, "safe_margin_x": 18, "safe_margin_y": 16 }

normalize_cell() derives shape from width == height. Default is square 256. Picking rect makes the cell taller (hatch-pet-style) — but that one shape is then used end-to-end, generation and output alike.

4.2 Design note: one cell, NOT a two-stage tall→square transform

A common mental model is "generate in a tall/portrait cell to save image area, then cut/place into a square cell for the final atlas." The code does not do this. There is no separate gen_cell vs atlas_cell; the generated strip and the final atlas share the same cell. If you generate rect, the atlas is rect; if you generate square, the atlas is square.

What is true, and what makes a two-stage model only a small step away: extraction is content-based, not slot-based. connected_components() finds each pose blob, groups blobs by x-center, and fit_to_cell() crops the content bbox, rescales it preserving aspect ratio (scale = min(...) ≤ 1.0), and centers it in the target cell. So the refit already decouples a pose from the strip's pixel dimensions — a future change could feed a tall gen_cell to the guide/prompt and a square atlas_cell to fit_to_cell/compose without touching the extraction core. Today, that separation is not wired.

5. Idle-anchor architecture (identity ownership)

Stage 0 is a BLOCKING gate (the gate question and the five lock criteria live in ../SKILL.md). The ownership rule:

identity truth = accepted idle anchor
motion truth   = layout guide + paired/basis row when needed
base truth     = used only to create idle anchors, then dropped from row inputs

Base character images, original character sheets, and broad style references are allowed only before idle anchors are accepted. Once an idle anchor exists for the requested direction, later state rows must not attach the base character image as insurance. Re-attaching base makes the row model solve identity again and weakens the purpose of the idle-anchor workflow.

Reference ownership flow:

flowchart TD
    REFS["USER REFS / CHARACTER SHEET"] --> BASE["0. BASE IDLE 생성<br/>원본/캐릭터시트는 여기서만 사용<br/>비율/스타일/색/소품 고정"]
    BASE --> ANCHORS["1. 방향별 IDLE ANCHOR 생성<br/>idle-front-right · idle-front-left<br/>idle-back-right · idle-back-left"]
    ANCHORS --> DROP["BASE CHARACTER 폐기<br/>original refs 폐기 · character sheet 폐기<br/>이후 row 입력 금지"]
    DROP --> BASIS["2. BASIS ROW 생성<br/>input: target-direction idle<br/>+ target-state layout only<br/>output: basis row"]
    BASIS --> PAIRED["3. PAIRED ROW 생성<br/>input: paired-direction idle<br/>+ paired-state layout only + basis row<br/>output: paired row"]
    PAIRED --> ASSEMBLE["4. GIF / SHEET 조립<br/>row crop · gif preview · frame-index QA"]
  • The prompt text in row_prompt() enforces this ("Anchor lock" block): identity comes from the anchor, the row owns motion only.
  • Which image is "the accepted anchor" is code, not judgment (sprite_gen/curate/anchor.py, sprite-gen anchor): it resolves the human's pin (curation.json anchors.<direction>) or, absent one, the anchor row's curated sequence head; bakes that instance with the same primitives compose/export use (clone → pixel edits → transform → pixel-unfake re-quantization); and writes the derived cache references/anchors/<dir>-anchor-x8.png that row generation attaches. Reroll, the generation plan, and the curation view's anchor chip all go through it, so the picture the human approved on screen is the picture the next row inherits.
  • Default/simple states (idle/jump/attack/wave) skip the multi-anchor chain. Directional / 45° / locomotion use the chain documented in directional-anchor-workflow.md.

6. Extraction internals (extract_sprite_row_frames.py)

  1. remove_chroma_background() — drop near-key pixels (color-distance ball, --key-threshold default 96), clear fully-transparent RGB, then solve chroma-tinted boundary blends into despilled RGB + soft alpha. In-band blends are limited to key-distance <= 2; out-of-band blends use --fringe-unmix-reach (default 4). Small trapped spill clusters are despilled in place, while deeper key-tinted subject material survives. With request chroma.mode: "ycbcr" or CLI --chroma-mode ycbcr, the extractor uses remove_chroma_background_ycbcr() instead: border-mode key detection and chrominance-plane flood fill tolerate shaded keys and JPEG chroma noise. RGB remains the default.
  2. connected_components() — flood-fill alpha blobs, record bbox + area + x-center.
  3. separate_fused_poses() — only when fit.segmentation: "projection" or --segmentation projection is selected, projection profiles plus dynamic programming find the minimum-cost vertical cuts for poses joined by thin bridges. The default components path stays byte-identical and fails loudly when the declared pose count cannot be recovered.
  4. extract_component_frames() — pick frame_count seed blobs by area, sort by x-center, attach smaller noise blobs to the nearest seed; one group = one pose. If it cannot find frame_count poses, the row is blocked (no silent fallback). --allow-slot-fallback exists for debugging only and is reported as method: slots-explicit.
  5. fit_to_cell() — crop, aspect-preserving downscale (resample: lanczos | nearest | kcentroid), align_x/align_y placement, into the cell. lanczos is the smooth shrink: coverage by LANCZOS, colour mixed apart from it (sprite_gen.util.resample.resize_cell, the resample of the places that scaled with LANCZOS), so the fitted frame has no rim or key tint at its edge that the keyed frame did not have. On fit.pixel_unfake runs this legacy path still runs once per frame to produce the .plain.png twin (§6.1); the canonical frame goes through the pixel-unfake path instead.
  6. inspect_frames() — per-frame QA: empty/sparse, edge bleed, chroma-adjacent pixels, size outliers → frames-manifest.json.

This is intentionally closer to hatch-pet's cleanup than to a plain magick -transparent.

6.1 Pixel-unfake path (fit.pixel_unfake, deterministic)

The behavior contract is pixel-unfake.md; this is how the code realizes it. The path is unfake.js/pixeldetector-style and contains no model call and no non-integer resampling — same input, same output:

  1. Pitch detection — detect_pixel_pitch() scores each candidate pitch by edge-to-gridline alignment (the fraction of color boundaries landing within ±w of grid lines, minus the chance expectation (2w+1)/p) and takes the argmax. Detection runs per frame and each frame is snapped with its own detected pitch (maintainer 2026-07-20, plan sprite-gen/per-frame-pixel-grid: forcing a strip-wide pitch onto a frame accumulates the measurement gap across the width and cuts through block centers). The median across confident frames (collapse-filtered) is kept only as the fallback for frames whose own detection is inconclusive — applying it emits a per-frame warning, and a frame whose own pitch deviates from the consensus is warned, never overridden. If every frame fails, fit.pitch_hint (usually the base image's detected pitch) is used. estimate_pixel_grid_runlen() then derives an independent modal run-length estimate; crosscheck_pitch_runlen() records divisor/harmonic or axis-ratio disagreement as observable warnings. The run-length estimate is a crosscheck only and never silently replaces the canonical grid score.
  2. Grid-phase snap — the phase is measured, not inferred from the pitch detector: _best_phase() scores real cell uniformity per frame and picks the best offset, refine_edges_to_boundaries() then pulls each interior cut onto a real colour boundary (±pitch/3), and snap_by_edges() collapses each block to one output pixel via _dominant_block_color() voting. The histogram phase that _axis_refine() returns alongside the pitch is not used here — it can sit up to pitch/2 off the true phase, which is outside the ±pitch/3 recovery window and merged a character's eye from 4 rows into 3 (plan sprite-gen/frame-pitch-consensus-eats-a-row; see pixel-unfake.md "위상은 근사가 아니라 실측으로 고른다"). detail_bias (default true) prefers a near-black minority cluster (share ≥ 0.40, luma < 70/255) so eyes and 1px outlines survive dominant voting.
  3. Physical cap only — no conform squash — conform_row_logical() keeps the snapped native logical size and only enforces the physical cell cap (kCentroid, _kcentroid_downscale()), warning observably when a frame hits it. Squashing to a declared logical_height was removed (maintainer 2026-07-14/17: it merges cells and eats detail); fit.conform is now rejected loudly if present in a request.
  4. Alpha binarization — binarize_alpha(), applied on the outputs of steps 2-3 (snap, then cap) so every logical pixel is fully opaque or fully transparent.
  5. Inter-frame registration — register_row_frames() aligns frames on the stable upper body (alpha overlap, small slack), so locomotion rows don't jitter on per-frame content changes.
  6. Shared palette — build_shared_palette() + apply_palette(): one run-wide median-cut palette (palette_size) kills frame-to-frame color flicker.
  7. Outline — enforce_outline() draws a uniform 1px silhouette outline.
  8. Integer placement — fit_pixel_unfake() / row_placement() upscale by the integer factor cell_height // logical_height (NEAREST) and place with offsets honoring align_x/align_y. align_x: "alpha-centroid" computes the fringe-insensitive alpha-weighted centroid per frame (alpha ≤ 10 is ignored), cancelling residual horizontal registration jitter without flattening vertical motion arcs.

Alongside each canonical frame-N.png, the extractor writes two pre-pixel- perfect twins (same strip through the legacy fit path; if twin production fails it is a warning, only that twin is dropped): the cell-sized frame-N.plain.png and the hi-res orig/frame-N.png (legacy fit at S×cell, S = min(4, 1024//cell), display-only). The curation webview toggles the display to orig/ when present (crisp "off = original", else .plain.png), while curation.json.pixel_unfake: false makes compose/GIF/PNG export bake the cell-sized .plain.png variant — missing plain files are a hard error, not a silent fallback. The hi-res twin lives in a subdir so non-recursive frame-*.png globs (inspect, measure) never pick it up. The chosen variant is recorded as frame_variant in reports and the manifest. Display-contract SSoT: run-contract.md §3.

7. Curation sidecar (curation.json)

Optional, non-destructive. Written by the webview (serve_curation.py), consumed by compose_sprite_atlas.py and compose_selected_cycle.py. The original frames/<state>/frame-N.png files are never rewritten.

{ "version": 1, "kind": "sprite-gen-curation", "run_revision": "9f3c1a0b7e2d4c58",
  "pixel_unfake": true,
  "states": { "idle": { "selected": [0,2,3], "order": [0,2,3,1],
    "transforms": { "0": { "rotate": 15, "scale": 1.2, "dx": 10, "dy": -8, "flipX": 0 } } } } }
  • run_revision — required frame-generation stamp. load_curation applies the sidecar only when it matches the current run; a mismatched or missing stamp is stale and ignored (all-frames default, stderr-warned) — a re-import/re-extract or a legacy/hand-written sidecar never applies to frames it isn't proven to belong to.
  • pixel_unfake — top-level bake decision: false → compose/export bake the .plain.png twins (§6.1); absent/true → canonical frame-N.png.
  • selected — 0-based indices in play order; absent → all frames in order.
  • order — webview-owned full display order (sequence row then candidate-pool row) so reopening the curator restores both rows; compose ignores it and keys off selected.
  • transforms — per-index affine (rotate, scale, dx/dy, shx/shy shear, flipX), applied at compose time inside the cell so atlas geometry never changes.
  • A state missing from the sidecar uses the all-frames identity default — an explicit default, not a silent fallback.
  • curation.py owns the schema + transform math (single source for server and compose). Full field semantics: curation.md.

8. Runtime contract (manifest.json)

manifest.json.frame_layout is the runtime SSoT. Game code consumes absolute rectangles and must not recover frame rects from alpha at runtime.

Required fields:

  • game_input: "sprite-sheet-alpha.png"
  • degraded_static_fallback: false
  • animation.rows.<state> → { row, frames, fps, loop }
  • frame_layout.rows.<state>[i] → { x, y, w, h } absolute atlas rects
  • frame_layout also carries sheetWidth/sheetHeight/cellWidth/cellHeight
  • curation_applied records whether a sidecar was baked; frame_variant records which frame variant (canonical / plain) was baked

Static fallback is allowed only as explicit survival output when generation is blocked; it must not create sprite-sheet-alpha.png and is not a pass.

9. Output directory (one worker owns one character folder)

The canonical run-dir folder tree (every input/output file, plus which files drive the curation view) is owned by run-contract.md §2. The single-writer lock that enforces "one worker owns one character folder" at runtime is runio.py's .sprite-gen.lock (§2 above).

10. Inverse paths (editing finished assets)

  • unpack_atlas_run.py rebuilds a curator-ready run dir from a finished atlas (layout source priority: --grid > --manifest > auto-detect), or imports a folder of separate PNGs via --pngs-dir (carries a sibling meta.json for iso tile/anchor grid overlay).
  • export_curated_pngs.py bakes the curation transform back into named PNGs (the deliverable for imported still sets like furniture); the single atlas is the deliverable for animation frames / runtime perf.

11. Relationship to hatch-pet

sprite-gen is inspired by the Apache-2.0 hatch-pet skill but ships no pet assets. Shared DNA: one image-gen job per row/state, layout-guide grounding, chroma cleanup, deterministic assembly scripts, contact-sheet/video QA.

Key differences:

hatch-pet sprite-gen
Atlas fixed 8×9, fixed 192×208 cell, 9 fixed pet states variable state list, variable cell (square default 256)
Cell shape fixed rect 192×208 square or rect, one config end-to-end (§4)
Frame cut slot-based geometry content-based connected components + refit (+ pixel-unfake path)
Identity canonical base reference reused on every row idle-anchor ownership; base dropped after anchors
Curation n/a non-destructive curation.json + webview
Inverse n/a unpack atlas / import PNG pack / export stills
Packaging pet.json + spritesheet.webp into ~/.codex/pets/ runtime manifest.json frame_layout

Both generate row-by-row, one state per image-gen call — neither produces a whole atlas in a single generation.

Related