Generate clean 2D game sprites & animation atlases — component-row pipeline: state rows, alpha cleanup, frame extraction, runtime atlases. Codex/Claude skill.
fix: keep processing version 3 sound bundles loadable by their own rule
Soohong Kim48cf31d809f2
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex accf717..f62fdc1 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -4,7 +4,7 @@ All notable public changes to `sprite-gen` are recorded here. Versions track the## Unreleased-- Bump audio processing recipe to version 4, measure BS.1770-4 loudness over normalized PCM frames and gated blocks, support a peak-limited loudness goal, normalize loudnorm-unanalysable short clips via limiter passes instead of refusing them, encode AAC using `-aac_coder fast`, record `normalization.pcm_lufs` and `pcm_true_peak_dbtp` required by bundle loaders, and reject recipe version 1-3 bundles while reusing cached provider audio.+- Bump audio processing recipe to version 4, measure BS.1770-4 loudness over normalized PCM frames and gated blocks, support a peak-limited loudness goal, normalize loudnorm-unanalysable short clips via limiter passes instead of refusing them, encode AAC using `-aac_coder fast`, record `normalization.pcm_lufs` and `pcm_true_peak_dbtp` required to load version 4 bundles, keep recipe version 3 bundles loadable in game packs by their own rule (target loudness, decoded files re-measured with loudnorm), reject recipe version 1-2 bundles, and reject recipe version 1-3 output directories in `sprite-gen sound` while reusing cached provider audio.- Bump audio processing recipe to version 3, require `ffprobe` on `PATH`, export M4A with `-movie_timescale 48000`, and validate AAC stream duration with `ffprobe`.- Add bounded one-frame decoded length tolerance, preserving loop duration, trimming one-shots, and rejecting shorter buffers and undersized overlaps at export.- Configure loop playback in `createAudioSource` using an internally cached L+1 guard frame buffer while preserving `prepareAudioBuffer` frame contracts.diff --git a/docs/game-pack.md b/docs/game-pack.mdindex a178b1f..5fa17f0 100644--- a/docs/game-pack.md+++ b/docs/game-pack.md@@ -181,7 +181,7 @@ Game packs support optional bundled audio declared in pack recipes via `request.- `duration_seconds`: Final prepared playback duration `(end_frame - start_frame - overlap_frames) / sample_rate`.- `target_lufs`: Target integrated loudness.- `playback`: Integer coordinate object (`version`, `sample_rate`, `start_frame`, `end_frame`, `overlap_frames`) preserved exactly from source `sound.json`.- - `files`: Format dictionary for `ogg` and `m4a` containing relative paths (`audio-<id>.ogg`, `audio-<id>.m4a`), `sha256`, decoded `duration_seconds`, `lufs`, and `true_peak_dbfs` (`encode_passes` and `pcm_peak_limit_dbtp` exist only in `sound.json` and are stripped on load). Source `sound.json` manifests must carry the `normalization` record of processing version 4 and satisfy the [loudness rule](sound.md#loudness-rule). For a peak-limited sound, `lufs` sits below `target_lufs`.+ - `files`: Format dictionary for `ogg` and `m4a` containing relative paths (`audio-<id>.ogg`, `audio-<id>.m4a`), `sha256`, decoded `duration_seconds`, `lufs`, and `true_peak_dbfs` (`encode_passes` and `pcm_peak_limit_dbtp` exist only in `sound.json` and are stripped on load). Source `sound.json` manifests must specify `recipe.processing_version` 3 or 4 (version 4 manifests require the `normalization` record, while version 3 manifests are decoded and re-measured under the version 3 [loudness rule](sound.md#loudness-rule)). For a peak-limited sound in version 4, `lufs` sits below `target_lufs` (version 3 has no peak-limited goal).- `player`: Runtime helper descriptor:```json{diff --git a/docs/sound.md b/docs/sound.mdindex 3ab0357..ed83c6e 100644--- a/docs/sound.md+++ b/docs/sound.md@@ -43,7 +43,7 @@ Raw paid audio is stored in SQLite keyed by provider parameter digest. Tuning `-### Loudness Rule-Loudness evaluation uses a unified rule across exporter passes and bundle loading:+Loudness evaluation uses a unified rule across exporter passes and version 4 bundle loading (version 3 bundles follow their own bullet below):- **Measurement** (`sprite_gen/audio/processing.py` `measure`):- Integrated loudness follows [ITU-R BS.1770-4](https://www.itu.int/rec/R-REC-BS.1770): double-precision FFmpeg `biquad` K-weighting at 48 kHz, 400 ms gating blocks every 100 ms, absolute gate at -70 LUFS, relative gate 10 LU under absolute-gated loudness. Audio shorter than 400 ms contains no gating block, is unmeasurable, and is refused.@@ -57,10 +57,14 @@ Loudness evaluation uses a unified rule across exporter passes and bundle loadin- **Refusal Criteria**: Refuses silent or unmeasurable audio, true peaks exceeding -1 dBTP, files more than 1 LU from the goal, quiet audio with headroom whose PCM peak never reached the ceiling, and files decoded quieter than their peak-limited PCM.- **Rejected Alternatives**: Measuring one-shots on repeated context counts a single hit as if it repeated and its repeated blocks still cross the gate together; a separate short-clip measure (maximum momentary loudness, or ungated loudness) would change what the target means for every short sound, and ungated loudness also counts trailing silence.- **AAC Encoder Configuration**: M4A encodes AAC 192 kbps with [`-aac_coder fast`](https://ffmpeg.org/ffmpeg-codecs.html#aac) ("Worse with low bitrates (less than 64kbps), but is better and much faster at higher bitrates."). In synthetic FFmpeg 5.1 tests, the default `twoloop` coder decoded up to 3.6 dB above PCM true peak (90th percentile 1.4 dB over 320 hits), prompting correction passes that cut transient loudness (a synthetic 0.6 s thud ended at -20.2 LUFS, -2.3 dBTP, and was refused). On a 27-hit sample, `fast` remained within 1.1 dB of PCM true peak (median -0.3 dB) compared to 1.9 dB for `twoloop`.+- **Version 3 Bundles**: Evaluated under their original export rules rather than version 4 manifest checks, re-measuring decoded audio:+ - **Measurement**: Requires `ffmpeg` and `ffprobe` on `PATH`. Evaluates files against `target_lufs` (version 3 has no peak-limited goal, even if its manifest carries `pcm_lufs` or `pcm_true_peak_dbtp`). Decodes each file and re-measures via `measure_version_3` using FFmpeg `loudnorm` first-pass analysis (`input_i` as integrated loudness, `input_tp` as true peak) over `[start_frame, end_frame)` (three repetitions for loops). Enforces container duration checks (`time_base=1/48000`, `duration_ts=end_frame-start_frame`) and decoded frame counts via `file_frames`. The game pack records re-measured `lufs` and `true_peak_dbfs`; values in `sound.json` are not trusted.+ - **Refusal Criteria**: Every file must measure finite, within 1 LU of `target_lufs`, and at a true peak at or under -1 dBTP. Refuses quiet files (such as a tone at -21.5 LUFS against a -18 LUFS target, even if the manifest claims -18.6 LUFS or carries a peak-limited normalization record), hot files (such as a -0.5 dBTP spike claiming -2 dBTP), and version 2 files with millisecond AAC timescales relabeled as version 3.+ - **Measurement Rationale**: Uses FFmpeg `loudnorm` rather than this engine's BS.1770-4 meter because version 3 bundles retain no PCM for gating. When measured with an own-audio gate, boundary blocks sitting at the relative gate diverge slightly between implementations. In synthetic tests with FFmpeg 5.1 on 200 synthetic sources (190 bundles produced, 10 refused by the version 3 exporter), `loudnorm` passed 190 of 190 and matched `sound.json` identically. The engine's own-gate meter passed only 189 of 190, failing a 1.5 s hit recorded at -17.98 LUFS as -19.16 LUFS because its third block sat 0.011 LU above the engine's gate while falling under `loudnorm`'s gate. Across the other 376 files, the two meters differed by a median of 0.03 LU (at most 0.09 LU) and at most 0.01 dB true peak.## Manifest Schema and Coordinates-Outputs `sound.json` (version 2, `kind="sprite-gen-sound"`, `recipe.processing_version=4`, independent of `playback` version 2). Existing output directories generated with recipe versions 1, 2, or 3 are rejected; specify a new `--out-dir`. Original provider request cache keys and blobs remain unchanged; reprocessing cached audio needs no key or new paid request. Provider APIs, caching, and retry behaviors remain unchanged.+Outputs `sound.json` (version 2, `kind="sprite-gen-sound"`, `recipe.processing_version=4`, independent of `playback` version 2). The bundle loader (`sprite-gen game-pack` and `sprite-gen sound` reuse) accepts `recipe.processing_version` 3 or 4 (an integer); version 1, 2, a missing recipe, or any other value is refused with a message to make the sound again into a new `--out-dir`. Reusing an existing `--out-dir` in `sprite-gen sound` still requires the bundle's recipe to equal the current one, so directories generated with recipe versions 1, 2, or 3 are rejected (specify a new `--out-dir`). Original provider request cache keys and blobs remain unchanged; reprocessing cached audio needs no key or new paid request. Provider APIs, caching, and retry behaviors remain unchanged.### Manifest Properties@@ -86,7 +90,7 @@ Outputs `sound.json` (version 2, `kind="sprite-gen-sound"`, `recipe.processing_v- Manifest `duration_seconds` is `(N - M) / sample_rate`.- The CLI `--crossfade` parameter (seconds) is converted to integer frame count `M`, defaulting to `min(0.25, requested_duration / 8)`.- `files`: Format dictionary for `ogg` and `m4a` containing `path`, decoded `duration_seconds` (full decoded file duration, which may include codec priming/tail padding), `lufs` (integrated loudness measured over normalized PCM frames and gated blocks; sits below `target_lufs` for peak-limited sounds), `true_peak_dbfs` (true peak level), `sha256`, `encode_passes`, and `pcm_peak_limit_dbtp`.-- `normalization`: Record containing `method` (`"loudnorm-two-pass"` or `"limiter"`), `peak_limit_dbtp` (`-2.0`), `gain_limiter_passes`, `pcm_lufs`, and `pcm_true_peak_dbtp`. The bundle loader requires `pcm_lufs` and `pcm_true_peak_dbtp` to validate against the loudness rule and rejects bundles without them.+- `normalization`: Record containing `method` (`"loudnorm-two-pass"` or `"limiter"`), `peak_limit_dbtp` (`-2.0`), `gain_limiter_passes`, `pcm_lufs`, and `pcm_true_peak_dbtp`. For version 4 bundles, the bundle loader requires `pcm_lufs` and `pcm_true_peak_dbtp` to validate against the loudness rule (checking recorded `lufs` and `true_peak_dbfs` against `loudness_goal` via `meets` without decoding files) and rejects bundles without them.- **Removed Fields**: `codec_taper_seconds`, `loop_start_seconds`, `loop_end_seconds`, and `crossfade_seconds` are removed from output manifest metadata (the recipe still accepts `--crossfade` in seconds).## Client Playback Helper: sprite-gen-audio-v2.mjsdiff --git a/sprite_gen/audio/manifest.py b/sprite_gen/audio/manifest.pyindex c61fc2d..0f82bb1 100644--- a/sprite_gen/audio/manifest.py+++ b/sprite_gen/audio/manifest.py@@ -40,12 +40,22 @@ def load(path, max_bytes):files = value.get('files')if not isinstance(files, dict) or set(files) != {'ogg', 'm4a'}:raise ValueError('audio source requires ogg and m4a files')- level = value.get('normalization')- if not isinstance(level, dict) or not {'pcm_lufs', 'pcm_true_peak_dbtp'} <= set(level):- raise ValueError('audio source predates sound processing version 4; make it again with sprite-gen sound '- 'into a new --out-dir (its cached provider audio is reused)')- goal = processing.loudness_goal(result['target_lufs'], _number(level['pcm_lufs'], 'PCM LUFS', -70, 0),- _number(level['pcm_true_peak_dbtp'], 'PCM true peak', -100, 0))+ recipe = value.get('recipe')+ version = recipe.get('processing_version') if isinstance(recipe, dict) else None+ if type(version) is not int or version not in (3, processing.PROCESSING_VERSION):+ raise ValueError('audio source must come from sound processing version 3 or 4; make it again with sprite-gen '+ 'sound into a new --out-dir (its cached provider audio is reused)')+ if version == 3:+ # Version 3 had no peak-limited goal. Its files are measured again here, the way it measured+ # them, and must meet their target.+ processing.require_programs()+ goal = result['target_lufs']+ else:+ level = value.get('normalization')+ if not isinstance(level, dict) or not {'pcm_lufs', 'pcm_true_peak_dbtp'} <= set(level):+ raise ValueError('a sound processing version 4 source requires normalization.pcm_lufs and pcm_true_peak_dbtp')+ goal = processing.loudness_goal(result['target_lufs'], _number(level['pcm_lufs'], 'PCM LUFS', -70, 0),+ _number(level['pcm_true_peak_dbtp'], 'PCM true peak', -100, 0))payloads, sources, size = {}, {path}, 0for fmt in ('ogg', 'm4a'):file = files[fmt]@@ -65,6 +75,9 @@ def load(path, max_bytes):'duration_seconds': _number(file.get('duration_seconds'), 'file duration', frames['end_frame']/frames['sample_rate'], 602),'lufs': _number(file.get('lufs'), 'LUFS', -70, 0),'true_peak_dbfs': _number(file.get('true_peak_dbfs'), 'true peak', -100, -1)}+ if version == 3:+ measured = processing.measure_version_3(processing.file_frames(source, frames, value['loop'])[1], goal)+ record.update(lufs=measured['lufs'], true_peak_dbfs=measured['true_peak'])if not processing.meets(record['lufs'], record['true_peak_dbfs'], goal):raise ValueError('audio loudness does not match its target or its peak-limited PCM')result['files'][fmt] = recorddiff --git a/sprite_gen/audio/processing.py b/sprite_gen/audio/processing.pyindex 6f7c080..d37359e 100644--- a/sprite_gen/audio/processing.py+++ b/sprite_gen/audio/processing.py@@ -29,10 +29,14 @@ K_WEIGHTING = ('aformat=sample_fmts=dblp,''aformat=sample_fmts=flt')-def require_tools():+def require_programs():for name in ('ffmpeg', 'ffprobe'):if not shutil.which(name):raise ValueError(f'{name} is required on PATH for sound processing')+++def require_tools():+ require_programs()encoders = _ffmpeg(['-encoders']).stdout.decode(errors='replace')for codec in ('libopus', 'aac'):if not re.search(r'(?m)^\s*A\S*\s+'+codec+r'\s', encoders):@@ -124,6 +128,30 @@ def measure(samples, gate=None):'true_peak': _true_peak(samples), 'gate': gate}+def file_frames(path, frames, loop):+ """The decoded length of an encoded file, and the frames its loudness is measured over: its+ playback frames, three times over for a loop."""+ if frames['sample_rate'] != RATE:+ raise ValueError(f'sound files are measured at {RATE} Hz frames')+ if path.suffix == '.m4a':+ _check_m4a_duration(path, frames['end_frame']-frames['start_frame'])+ decoded = decode(path)+ if len(decoded) < frames['end_frame']:+ raise ValueError(f'{path.suffix[1:]} truncated the audio')+ interval = decoded[frames['start_frame']:frames['end_frame']] # AAC may retain a final partial codec frame.+ return len(decoded), np.tile(interval, (3, 1)) if loop else interval+++def measure_version_3(samples, target):+ """Loudness and true peak as processing version 3 measured its files: loudnorm's first pass.++ A version 3 bundle keeps no PCM, so its files have no PCM gate to be measured over, and a+ block at the relative gate can move this engine's measure across the tolerance where+ loudnorm's accepted the file (docs/sound.md)."""+ values = _loudnorm(samples, target)+ return {'lufs': values['input_i'], 'true_peak': values['input_tp']}++def loudness_goal(target, pcm_lufs, pcm_peak):"""The integrated loudness every encoded file of a sound must reach, within TOLERANCE_LU.@@ -202,13 +230,8 @@ def export(source: Path, out_dir: Path, *, target_lufs, loop, crossfade):encoded_samples, ceiling = samples, PCM_PEAK_DBTPfor attempt in range(3):_ffmpeg([*_pcm_args(), '-map_metadata', '-1', *codec, str(path)], encoded_samples.astype('<f4').tobytes())- if fmt == 'm4a':- _check_m4a_duration(path, frames['end_frame']-frames['start_frame'])- decoded = decode(path)- if len(decoded) < len(samples):- raise ValueError(f'{fmt} truncated the audio')- interval = decoded[:len(samples)] # AAC may retain a final partial codec frame.- measured = measure(np.tile(interval, (3, 1)) if loop else interval, pcm['gate'])+ length, played = file_frames(path, frames, loop)+ measured = measure(played, pcm['gate'])lufs, peak = measured['lufs'], measured['true_peak']if meets(lufs, peak, goal):break@@ -219,7 +242,7 @@ def export(source: Path, out_dir: Path, *, target_lufs, loop, crossfade):# by the measured overshoot plus headroom, then encode and measure again.ceiling -= max(0, peak + 1.5) if peak > FILE_PEAK_DBTP else 0encoded_samples = limit_gain(encoded_samples, goal - lufs, ceiling, loop)- files[fmt] = {'path': path.name, 'duration_seconds': len(decoded)/RATE,+ files[fmt] = {'path': path.name, 'duration_seconds': length/RATE,'lufs': lufs, 'true_peak_dbfs': peak, 'encode_passes': attempt + 1,'pcm_peak_limit_dbtp': ceiling,'sha256': hashlib.sha256(path.read_bytes()).hexdigest()}diff --git a/tests/audio/test_sound.py b/tests/audio/test_sound.pyindex b84e5dd..d431678 100644--- a/tests/audio/test_sound.py+++ b/tests/audio/test_sound.py@@ -112,7 +112,12 @@ def test_a_bundle_is_refused_when_its_files_miss_the_rule(tmp_path, monkeypatch)hot['files']['m4a']['true_peak_dbfs'] = -.5old = json.loads(path.read_text())del old['normalization']- for manifest, refusal in ((headroom, 'does not match'), (hot, 'invalid audio true peak'), (old, 'version 4')):+ relabeled = json.loads(path.read_text())+ relabeled['recipe']['processing_version'] = 3 # version 3 had no peak-limited goal; its files measure against -18+ recipe2 = json.loads(path.read_text())+ recipe2['recipe']['processing_version'] = 2+ for manifest, refusal in ((headroom, 'does not match'), (hot, 'invalid audio true peak'), (old, 'version 4'),+ (relabeled, 'does not match'), (recipe2, 'version 3 or 4')):path.write_text(json.dumps(manifest))with pytest.raises(ValueError, match=refusal):load_sound(path, 50_000_000)diff --git a/tests/compose/test_game_pack_audio.py b/tests/compose/test_game_pack_audio.pyindex f520f1a..c5d61c9 100644--- a/tests/compose/test_game_pack_audio.py+++ b/tests/compose/test_game_pack_audio.py@@ -2,9 +2,11 @@import hashlibimport json+import numpy as npimport pytestfrom PIL import Image+from sprite_gen.audio import processingfrom sprite_gen.compose.game_pack import pack@@ -22,6 +24,7 @@ def inputs(tmp_path):'true_peak_dbfs':-2, 'sha256':hashlib.sha256(data).hexdigest()}manifest = {'kind':'sprite-gen-sound','version':2,'event':'combat.hit', 'audio_kind':'sfx','loop':False,'duration_seconds':1,'target_lufs':-18,'files':files,+ 'recipe': {'processing_version':4},'normalization': {'pcm_lufs':-18, 'pcm_true_peak_dbtp':-2},'playback': {'version':2, 'sample_rate':48000, 'start_frame':0,'end_frame':48000, 'overlap_frames':0}}@@ -59,7 +62,7 @@ def test_optional_audio_preserves_image_only_bytes_and_packs_events(tmp_path):@pytest.mark.parametrize('defect', ['duplicate', 'traversal', 'digest', 'nonfinite', 'badloop', 'old', 'fraction', 'overlap',- 'prerule', 'quiet'])+ 'prerule', 'quiet', 'norecipe', 'recipe2'])def test_broken_audio_cannot_publish_a_partial_pack(tmp_path, defect):path, request, manifest = inputs(tmp_path)request['audio'] = [{'id':'hit','source':'audio/sound.json'}]@@ -73,7 +76,63 @@ def test_broken_audio_cannot_publish_a_partial_pack(tmp_path, defect):if defect == 'overlap': manifest['playback']['overlap_frames'] = 24000if defect == 'prerule': del manifest['normalization']if defect == 'quiet': manifest['files']['m4a']['lufs'] = -19.5 # under the target, its PCM not peak-limited+ if defect == 'norecipe': del manifest['recipe']+ if defect == 'recipe2': manifest['recipe']['processing_version'] = 2path.write_text(json.dumps(request))(tmp_path/'audio'/'sound.json').write_text(json.dumps(manifest))with pytest.raises(ValueError): pack(path, tmp_path/'bad')assert not (tmp_path/'bad').exists()+++def version_3_bundle(tmp_path, lufs=-18., spike=None, timescale=48000, **changes):+ """A sound made by processing version 3: real encodes of a synthetic tone, the file numbers that+ version's loudnorm check recorded, and no PCM loudness record. 48003 frames are no whole+ millisecond, so a millisecond AAC timescale cannot store them."""+ t = np.arange(48003) / 48000+ pcm = np.stack([np.sin(2*np.pi*440*t), np.sin(2*np.pi*440*t + .3)], axis=1)+ pcm *= 10 ** ((lufs - processing.measure(pcm)['lufs']) / 20)+ if spike:+ pcm[24000:24048] = spike+ path, request, manifest = inputs(tmp_path)+ files = {}+ for fmt, codec in (('ogg', ['-c:a', 'libopus', '-b:a', '160k']),+ ('m4a', ['-c:a', 'aac', '-b:a', '192k', '-movflags', '+faststart',+ '-movie_timescale', str(timescale)])):+ file = tmp_path/'audio'/f'sound.{fmt}'+ processing._ffmpeg(['-f', 'f32le', '-ar', '48000', '-ac', '2', '-i', 'pipe:0', *codec, str(file)],+ pcm.astype('<f4').tobytes())+ files[fmt] = {'path':file.name, 'duration_seconds':1.02, 'lufs':-18.6, 'true_peak_dbfs':-2,+ 'encode_passes':1, 'pcm_peak_limit_dbtp':-2,+ 'sha256':hashlib.sha256(file.read_bytes()).hexdigest()}+ manifest['playback']['end_frame'], manifest['duration_seconds'] = 48003, 48003/48000+ manifest.update(files=files, recipe={'processing_version':3},+ normalization={'method':'loudnorm-two-pass', 'peak_limit_dbtp':-2, 'gain_limiter_passes':0})+ for key, value in changes.items():+ manifest[key] = {**manifest[key], **value} if isinstance(value, dict) else value+ request['audio'] = [{'id':'hit','source':'audio/sound.json'}]+ path.write_text(json.dumps(request))+ (tmp_path/'audio'/'sound.json').write_text(json.dumps(manifest))+ return path+++def test_a_version_3_sound_is_packed_by_the_target_it_was_made_for(tmp_path):+ result = pack(version_3_bundle(tmp_path), tmp_path/'pack')+ for file in result['audio']['hit']['files'].values():+ # its files are measured again over their frames, not taken from its loudnorm-era numbers+ assert abs(file['lufs'] + 18) <= .3 and file['lufs'] != -18.6+ assert -20 < file['true_peak_dbfs'] <= -1++[email protected]('defect', ['quiet', 'peak-limited', 'hot', 'timescale', 'recipe2', 'recipe1'])+def test_a_version_3_sound_that_misses_its_rule_is_refused(tmp_path, defect):+ path = version_3_bundle(+ tmp_path, lufs=-21.5 if defect in ('quiet', 'peak-limited') else -18.,+ spike=.94 if defect == 'hot' else None, # -0.5 dBTP while its manifest says -2+ timescale=1000 if defect == 'timescale' else 48000, # the millisecond AAC duration of version 2+ **({'normalization':{'pcm_lufs':-21.5, 'pcm_true_peak_dbtp':-2}} if defect == 'peak-limited' else {}),+ **({'recipe':{'processing_version':int(defect[-1])}} if defect.startswith('recipe') else {}))+ with pytest.raises(ValueError, match={'quiet':'does not match', 'peak-limited':'does not match',+ 'hot':'does not match', 'timescale':'container duration',+ 'recipe2':'version 3 or 4', 'recipe1':'version 3 or 4'}[defect]):+ pack(path, tmp_path/'bad')+ assert not (tmp_path/'bad').exists()