Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
docs: describe sound generation and game pack audio contracts
Soohong Kim576265be20fe
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f7fb187..3d0fac8 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,6 +4,8 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the## Unreleased+- Added `sprite-gen sound` CLI and game-pack audio integration for ElevenLabs SFX/BGM generation, featuring SQLite request deduplication, cosine loop crossfading, multi-pass FFmpeg loudnorm/alimiter normalization, dual Opus/AAC encoding, and Web Audio loop boundary metadata.+- Added `--cycle once` nonloop motion support with automatic selection for `death`, `fall`, `land`, `cast`, `hurt_hold`, and `transform`, rejection on cyclic states, no seam gate or wrap correction, and a terminal stability gate searching the whole clip for the first stable terminal window ($K = \max(4, \lceil\text{fps} \times 0.2\rceil)$) after action departure (ending at window end, discarding later recovery or shaking), with conservative candidate end pose geometric screening (alpha >= 8 body boxes; height <= 80% of initial for `death`/`fall`, changed geometry for other states) and retained cut edge contact validation (ignoring discarded tail).- Added nonloop-only flags `video-prompt --end-frame final.png` (refuses `--no-last-frame`, prints instructions without image validation) and `video-set --end-frame ITEM=PATH` (checks final canvas dimensions and corner key match initial canvas), with clip cache tracking end-image SHA256 where changed end pins reject until user explicitly `--force` regenerates.- Added endpoint-preserving strip downsampling with candidate events (`events:[{"frame": n, "name":"motion_peak"}]` using zero-based output cell indices mapped to keyed source frames via `source_cut.samples`) and `once` preview handling, with nonloop cycles insisting on existing `--anchor none` and `--fit state` while rejecting non-none anchor and tight fit.diff --git a/docs/README.md b/docs/README.mdindex 548753b..b5602d6 100644--- a/docs/README.md+++ b/docs/README.md@@ -78,6 +78,7 @@ grouping is derived from `sprite_gen/_modules.py`, the one taxonomy table.| Doc | Owns ||---|---|| [gen.md](gen.md) | `sprite-gen gen` / `gen-set`: providers, default resolution, transparency strategy per provider, when a refused provider request is sent again, row usage |+| [`sprite-gen sound`](sound.md) | Generate normalized SFX and BGM audio assets with Opus and AAC encoding, SQLite raw request caching, and Web Audio loop metadata. || [pixel-draw.md](pixel-draw.md) | `sprite-gen pixel-draw` (experimental): Claude Opus draws 8–48 px pixel art as a pixel grid — the inputs, the safe area, the run folder and its report, gates G1–G8, the row repair and the one redraw, the pick between candidates, Message Batches and `--resume`, re-rendering an edited `grid.txt` || [prompt-assembly.md](prompt-assembly.md) | What the engine adds to a still or clip prompt and when: the pieces, the 2.22.0 freeze without `--handed`, what counts as already said, and the notes on the caller's own words || [video.md](video.md) | `sprite-gen video` / `video-extend` / `video-edit` / `video-resume`: stills and clips to mp4 through Grok Imagine (image-to-video, last-frame pin, references, extension, editing, asking again for an unfinished generation) with the user's own credential |diff --git a/docs/game-pack.md b/docs/game-pack.mdindex 46fe7cc..9fd28fd 100644--- a/docs/game-pack.md+++ b/docs/game-pack.md@@ -13,7 +13,7 @@ sprite-gen game-pack --request game.json --out-dir game-assets* `--request <path>`: Path to the request JSON file. Paths within the request resolve relative to the request file directory.* `--out-dir <path>`: Target output directory. Must be a new directory path.* `--page-size <int>`: Maximum page dimension in pixels. Defaults to `4096`. Lower bound is `1`; cannot exceed `4096`.-* `--max-bytes <int>`: Total encoded byte budget for all output files (PNG, WebP, and JSON combined). Defaults to `50000000` (50 MB). Lower bound is `1`; configurable downward only.+* `--max-bytes <int>`: Total encoded byte budget for all output files (PNG, WebP, JSON, OGG, and M4A combined). Defaults to `50000000` (50 MB). Lower bound is `1`; configurable downward only.Invalid arguments, missing files, frame dimension overflows, or budget violations exit with a non-zero status.@@ -92,7 +92,7 @@ Source policies and row anchors are preserved by the asset adapter. Optional ent* Every frame receives 1-pixel edge extrusion on each side (+2 pixels to width and height). Maximum frame dimension plus 2 pixels must fit within `--page-size`.* Trimming and rotation are disabled (`trimmed: false`, `rotated: false`).* Each page has PNG and lossless WebP copies (preserving exact RGB and alpha).-* Total byte size across all PNG, WebP, and JSON outputs must not exceed `--max-bytes`.+* Total byte size across all PNG, WebP, JSON, OGG, and M4A outputs must not exceed `--max-bytes`.* Layout packing bins assets and frames using stable keys.## Output manifest@@ -160,3 +160,21 @@ function registerGamePack(scene, imagesInPageOrder, pack) {````addAtlasJSONHash` assigns `sourceIndex` offsets to frames across pages. The game runtime is responsible for reading `durations_ms` and applying animation playback policies (`repeat = anim.loop ? -1 : 0`, `hold_last`, `blend`, and `events`).++## Audio Assets and Playback++Game packs support optional bundled audio declared in pack recipes via `request.audio`:++```json+{+ "audio": [+ { "id": "hit", "source": "../hit/sound.json" }+ ]+}+```++### Integration Rules+- **Packaging and Manifest**: Pack assembly copies format files flat into the output directory as `audio-<id>.ogg` (Opus, 160 kbps) and `audio-<id>.m4a` (AAC, 192 kbps). The pack manifest preserves audio properties and format file metadata under `manifest.audio[id]`. Refer to [`sound.md`](sound.md) for generation options and full manifest metadata schemas.+- **Event Linkage**: String matching between audio `event` identifiers and animation `events[].name` provides runtime linkage for consumer engines to match strings. The pack does not enforce that every audio event has an associated animation (such as UI sounds or background music), and does not promise automatic runtime code.+- **Pack Limits**: Bundled audio file sizes count against the global 50,000,000-byte pack limit. Audio-free pack builds remain byte-for-byte unchanged.+- **Runtime Playback**: Web Audio uses `loopStart` (`0`) and `loopEnd` (`processed PCM duration`) metadata to exclude padding. HTML audio (such as `<audio loop>`) is not guaranteed gapless.diff --git a/docs/sound.md b/docs/sound.mdnew file mode 100644index 0000000..e70cf55--- /dev/null+++ b/docs/sound.md@@ -0,0 +1,89 @@+# Audio Asset Reference: `sprite-gen sound`++## Overview and CLI Usage++```bash+sprite-gen sound --kind sfx|bgm --prompt TEXT (or --prompt-file PATH) --duration SECONDS --event NAME --out-dir DIR [--cache-dir DIR] [--loop|--no-loop] [--target-lufs FLOAT] [--crossfade SECONDS]+```++### Options and System Requirements+- `--kind`: `sfx` (default `--no-loop`, -18 LUFS) or `bgm` (default `--loop`, -20 LUFS, instrumental).+- `--prompt` / `--prompt-file`: Prompt text or file. The provider may reject prompts containing artist names, franchises, or lyrics; the CLI omits a dialogue verb.+- `--duration`: Requested duration in seconds.+- `--event`: Safe identifier (1..128 characters); first character must be alphanumeric, remaining characters can be alphanumeric, dot, underscore, dash, or colon.+- `--out-dir`: Destination; must be new or match existing recipe and output file digests.+- `--cache-dir`: Defaults to `$XDG_CACHE_HOME/sprite-gen/audio` or `~/.cache/sprite-gen/audio` (never inside `--out-dir`; local by default but can be copied; no cross-cache-directory or cross-machine duplicate-request guarantee).+- `--target-lufs`: Target integrated loudness within `-36..-9` LUFS.+- `--crossfade`: Overlap duration in seconds.+- **Environment & PATH**: `ELEVENLABS_API_KEY` is required only for new paid requests (offline work needs none). `ffmpeg` with `libopus` and `aac` encoders is required on `PATH` and checked before paid calls (`ffprobe` is not required).++## Provider APIs and Cost Accounting++Requests use ElevenLabs (`output_format=mp3_44100_128`):+- **SFX**: `POST /v1/sound-generation` (`text`, `duration_seconds` 0.5..30 [API reference takes precedence over 0.1s overview bound], `loop`, `model_id="eleven_text_to_sound_v2"`, `prompt_influence=0.3`).+- **BGM**: `POST /v1/music` (`prompt` max 4100 characters, `music_length_ms` 3000..600000, `model_id="music_v1"`, `force_instrumental=true`).++### Billing and Usage Accounting+Checked 2026-10-13: [ElevenLabs Pricing](https://elevenlabs.io/pricing/api) quotes SFX at $0.12/min and music at $0.15/min (USD), FAQ notes generation metering, and legacy [Capabilities](https://elevenlabs.io/docs/overview/capabilities/sound-effects) states 40 credits/sec for fixed-duration SFX. Because plans vary, no currency conversion is hardcoded; the runtime records provider `character_cost` directly from responses when present (never guessed currency). Check quotas via [User Subscription](https://elevenlabs.io/docs/api-reference/user/subscription/get) (`character_count`/`character_limit`).++## Cache Idempotency and Retries++Raw paid audio is stored in SQLite keyed by provider parameter digest. Tuning `--event`, loudness, or exports reuses raw cache without new purchases:+- **Locking & Recovery**: Atomic claims precede requests; raw blobs and receipts commit on success. Duplicates, crashes, or uncertain results refuse resubmission. Do not delete unresolved cache entries blindly; inspect provider billing/history first. All errors remain unresolved.+- **Offline Processing**: Downstream processing runs offline. Identical runs require no network.+- **Retries**: Uses `sprite_gen.gen.retry` exclusively. ElevenLabs 429 retries only when `detail.status` is `system_busy` or `too_many_concurrent_requests` (max 2 retries via `Retry-After` or engine backoff; see [ElevenLabs 429 Errors](https://elevenlabs.io/docs/help-center/technical/api-error-code-429)). No other 429, 5xx, transport retries, or provider fallbacks occur. Emits only bounded status, code, and request IDs without error prose.++## Audio DSP, Looping, and Quality Gates++1. **Cosine Looping**: Blends tail and head with complementary cosine crossfade, subtracting overlap from provider length (default `min(0.25, requested_duration/8)`, under half source duration; 20s BGM yields ~19.75s). Cosine tail/head overlap is followed by a 5ms cosine taper to zero at both outer boundaries to suppress codec priming discontinuities; duration is not shortened further.+2. **Normalization**: Two-pass [FFmpeg Loudnorm](https://ffmpeg.org/ffmpeg-filters.html#loudnorm) targets -2 dBTP with repeating loop context. At most three per-format encode/measure passes correct codec reconstruction overshoot using measured gain and a tighter oversampled [FFmpeg Alimiter](https://ffmpeg.org/ffmpeg-filters.html#alimiter) (192kHz, auto-gain off, latency compensated); all passes are offline and never trigger provider retries.+3. **Quality Gates**: Output formats must decode to target +/-1 LU and true peak <=-1 dBTP. Silence or unmeasurable audio fails. Numeric checks do not judge musical beat alignment.+4. **Export Formats**: Outputs 48kHz stereo Opus in OGG (`sound.ogg`, `libopus`, 160 kbps) and AAC in M4A (`sound.m4a`, `aac`, 192 kbps).+5. **Playback**: Web Audio uses `loopStart` (`0`) and `loopEnd` (`processed PCM duration`) on decoded `AudioBuffer` to exclude padding; HTML audio (such as `<audio loop>`) is not guaranteed gapless.++## Manifest Schema and Pack Integration++Outputs `sound.json` (v1, `kind="sprite-gen-sound"`): `event`, `audio_kind`, `loop`, `duration_seconds`, `target_lufs`, `codec_taper_seconds` (0.005 for loops, 0 for nonloops), `loop_start_seconds` (0), `loop_end_seconds` (processed PCM duration), and `files` (`ogg` and `m4a` entries with `path`, decoded `duration_seconds`, `lufs`, `true_peak_dbfs` [true peak dBTP], `sha256`, `encode_passes`, `pcm_peak_limit_dbtp`).++Game pack recipes reference audio via `request.audio = [{"id": "hit", "source": "../hit/sound.json"}]`:+- Files copy flat as `audio-<id>.ogg` and `audio-<id>.m4a`; `manifest.audio[id]` preserves properties and metadata.+- Consumers match `event` strings against animation `events[].name` (packs require no animation for UI/music and generate no runtime code).+- Audio sizes count toward the 50,000,000-byte pack limit; audio-free packages remain unchanged.++## Examples++```bash+sprite-gen sound --kind sfx --prompt "laser" --duration 0.8 --event laser:shot --out-dir ./dist/laser+sprite-gen sound --kind bgm --prompt "synth cave" --duration 20.0 --event cave:bgm --out-dir ./dist/cave+```++### Manifest Audio Fragment (Partial)+```json+{+ "audio": {+ "hit": {+ "event": "hit",+ "audio_kind": "sfx",+ "loop": false,+ "duration_seconds": 1.25,+ "target_lufs": -18.0,+ "loop_start_seconds": 0.0,+ "loop_end_seconds": 1.25,+ "files": {+ "ogg": { "path": "audio-hit.ogg" },+ "m4a": { "path": "audio-hit.m4a" }+ }+ }+ }+}+```++## References (Checked 2026-10-13)+- [ElevenLabs SFX](https://elevenlabs.io/docs/api-reference/text-to-sound-effects/convert)+- [ElevenLabs Music](https://elevenlabs.io/docs/api-reference/music/compose)+- [ElevenLabs 429 Errors](https://elevenlabs.io/docs/help-center/technical/api-error-code-429)+- [ElevenLabs Capabilities](https://elevenlabs.io/docs/overview/capabilities/sound-effects)+- [ElevenLabs Pricing](https://elevenlabs.io/pricing/api)+- [ElevenLabs Subscription](https://elevenlabs.io/docs/api-reference/user/subscription/get)+- [FFmpeg Loudnorm](https://ffmpeg.org/ffmpeg-filters.html#loudnorm)+- [FFmpeg Alimiter](https://ffmpeg.org/ffmpeg-filters.html#alimiter)