Skip to content

Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.

18.1 KB
Raw Blame

Launch profile + critique: evaluation

Generated by eval/launch/run_eval.py (numbers only; the films are local reference files of the author, never copied). Profile = earshot/data/launch_profile.json, built from the 7 references below.

Commands

EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --only ref_1,ref_2,ref_3 --fresh --measure-only
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --only ref_4,ref_5,ref_6,ref_7,clawd_real,clawd_synth --fresh --measure-only
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --build   # profile, LOO, Clawd, this file
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --neg base,quiet,hot,mono,muffled,late,deadair,fadeout,hole,squashed
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --neg sfx_only,amb_only,synth_bed
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py --neg sine,pink
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py           # rewrite this file from the caches

What was tuned on what

  • Detector thresholds (salient hit: >= 6 dB jump of the 100 ms level within 80 ms, landing within 8 LU of integrated, one hit per run, timed at the attack; visual event: frame-difference peak >= 4x the median of the surrounding second or >= 0.15, now earshot.video.diff_events; stop-down: >= 10 dB dip for 0.15-1.5 s) were chosen by looking at plots of all 9 films (7 refs + 2 Clawd), at the Clawd mix-report placements (hits) and at ffmpeg scene cuts (visual events). The Clawd real soundtrack is therefore partly tuning data for the hit detector.
  • Music threshold (AudioSet Music >= 0.3 per 5 s window) was set after looking at the per-window probabilities of all 9 films (music under voice or SFX scored 0.4-0.7, one sound-design stretch 0.1-0.15).
  • Changed after the first leave-one-out run (first build; not a blind test of these choices): true peak went from critical to major and is judged against 0 dBTP for decoded lossy files (the -1 dBTP master rule failed all 7 AAC-decoded references); a critical rule missed by <= 2 tolerances counts as major; the metric max_gap_s was added; the music_frac tolerance went from 0.1 to 0.05 after an SFX-only stem (music in 42 % of windows) came out as only a near miss.
  • Changed after the independent review, before the known-bad set below was first run: p_real uses the realness model's own thresholds (ok at its 'real or pro-produced' threshold, critical below 0.5, minor in between; no near-miss downgrade); a density range that would reach 0 keeps a floor at half the sparsest reference (salient hits >= 0.05 /s); a mono mix (identical channels) counts as -40 dB stereo width instead of n.a.; files under 10 s get no loudness-shape rows; brightness_hz replaces the power-weighted centroid (info only). The known-bad expectations were written into run_eval.py before its first run; nothing was changed after seeing its results.
  • Not adjusted after seeing results: the other tolerances, the weights of the other metrics and the verdict thresholds (90 / 75 / 50 %).
  • Ranges are data: min-max of the reference films widened by each metric's tolerance. Loudness and true peak use web-delivery practice (-16..-9 LUFS; <= -1 dBTP for a master, <= 0 dBTP for decoded AAC / MP3) instead, so references can fail them.

Per-film measurements

film dur s LUFS dBTP LRA music speech onsets/s hits/s vis events/s events w/ sound (chance) audio pass s (cold) picture pass s AudioSet pass s (cold, first build) warm s
ref_1 93.1 -12.1 0.1 1.7 100% 95% 5.87 1.56 0.3 89% (90%) 49.8 0.0 144.4 0.4
ref_2 28.9 -14.8 1.0 3.6 100% 83% 5.98 1.21 0.31 67% (89%) 9.7 0.0 37.0 0.7
ref_3 20.5 -10.4 0.6 6.4 100% 100% 3.81 1.66 0.83 88% (68%) 5.6 0.1 23.0 0.7
ref_4 32.3 -14.8 -0.8 7.0 100% 0% 8.22 1.18 1.3 100% (99%) 18.1 0.0 58.6 0.3
ref_5 29.5 -22.2 1.2 12.2 67% 0% 4.03 1.25 0.54 50% (48%) 4.0 1.9 41.6 0.2
ref_6 49.3 -17.4 0.1 5.5 100% 100% 5.66 1.3 0.63 81% (86%) 5.8 14.2 54.5 0.2
ref_7 41.8 -7.3 2.0 3.3 89% 0% 2.46 0.1 1.6 60% (44%) 5.8 5.1 98.4 0.3
clawd_real 60.0 -15.0 -1.4 10.4 100% 0% 2.48 0.27 0.43 62% (37%) 8.0 0.0 78.9 0.2
clawd_synth 60.0 -14.0 -2.1 7.2 100% 0% 2.1 0.38 0.43 54% (43%) 8.1 25.4 76.3 0.6

Audio pass = decode + features.measure(light) + librosa onsets / beats + loudness curves, timed cold in this round (this module's cache deleted; the core decode cache was cold too after its v2 bump). Picture pass = earshot.video.frame_diff (core cache, shared with timeline / sheet): 0.0 means another module had already decoded that film; ref_6 (4K) and clawd_synth were cold. AudioSet = 5 s windows through AST, measured cold in the first build (same code and cache key, so not re-run): ~4-9 s per 5 s window on this laptop under load from 3-4 other model jobs; earshot critique --fast skips it and realness. Warm = everything cached. EARSHOT_THREADS=2.

Leave-one-out: each reference vs a profile of the other six

film verdict weighted score scored metrics in range reference-range metrics in range out of range
ref_1 close 75% 17/22 16/20 true_peak high, lra low, first_sound_s high, end_silence_s high, side_to_mid_db low
ref_2 launch-grade 94% 21/22 20/20 true_peak high
ref_3 close 83% 17/22 16/20 true_peak high, end_lu low, band_sub low, band_lowmid high, band_air low
ref_4 close 89% 19/22 17/20 onset_rate high, intro_lu low, band_mid low
ref_5 needs work 56% 14/22 14/20 lufs_i low, true_peak high, music_frac low, cuts_with_sound low, lra high, max_gap_s high, band_air high, side_to_mid_db high
ref_6 close 86% 19/22 19/20 lufs_i low, true_peak high, intro_lu high
ref_7 close 77% 18/22 18/20 lufs_i high, true_peak high, onset_rate low, hit_rate low

Total: 125/154 scored metric checks in range (81 %); reference-range metrics only: 120/140 (86 %). For exchangeable data a held-out film falls outside the min-max of six others with probability 2/7 (29 %) per continuous metric before tolerance, so ~71 % is the no-overfit baseline; the tolerance adds a little. p_real is not part of this check (the realness model was trained on these films).

In-sample (each reference vs the full 7-film profile): every reference-range metric is in range by construction; the only failures are web-practice rules: ref_1: true_peak high; ref_2: true_peak high; ref_3: true_peak high; ref_5: lufs_i low, true_peak high; ref_6: lufs_i low, true_peak high; ref_7: lufs_i high, true_peak high.

Per metric (7 leave-one-out checks each)

metric weight ok low high n.a.
lufs_i critical 4 2 1 0
true_peak major 1 0 6 0
music_frac critical 6 1 0 0
cuts_with_sound major 6 1 0 0
lra major 5 1 1 0
crest_lu major 7 0 0 0
first_sound_s major 6 0 1 0
end_silence_s major 6 0 1 0
max_gap_s major 6 0 1 0
final_lu major 7 0 0 0
onset_rate major 5 1 1 0
hit_rate major 6 1 0 0
intro_lu minor 5 1 1 0
end_lu minor 6 1 0 0
fade_s minor 7 0 0 0
band_sub minor 6 1 0 0
band_low minor 7 0 0 0
band_lowmid minor 6 0 1 0
band_mid minor 6 1 0 0
band_presence minor 7 0 0 0
band_air minor 5 1 1 0
side_to_mid_db minor 5 1 1 0

Sensitivity: known-bad mixes

Each row is one defect made on purpose (ffmpeg, scratch files in out/launch/neg/), critiqued against the 7-film profile. 'Should flag' was written down before the first run; 'caught' = at least one of those metrics came out low / high. Rows without models skip AudioSet and realness (partial verdicts); the control is the untouched real soundtrack.

mix what should flag caught verdict weighted score all flagged rows p_real s (critique, cold)
base control: the real Clawd soundtrack untouched - - close (partial) 87% crest_lu high (major); first_sound_s high (major) - 27.5
quiet unmastered: 12 dB quieter lufs_i lufs_i needs work (partial) 77% lufs_i low (critical); crest_lu high (major); first_sound_s high (major) - 9.0
hot 12 dB hotter, clipped by the 16-bit file lufs_i, true_peak lufs_i, true_peak needs work (partial) 77% lufs_i high (critical); true_peak high (major); first_sound_s high (major) - 8.8
mono folded to mono side_to_mid_db side_to_mid_db close (partial) 77% lufs_i low (major); crest_lu high (major); first_sound_s high (major); side_to_mid_db low (minor) - 9.5
muffled low-passed at 3 kHz (12 dB/oct) band_presence, band_air band_presence, band_air close (partial) 81% crest_lu high (major); first_sound_s high (major); band_presence low (minor); band_air low (minor) - 10.0
late 2.5 s of silence before the start first_sound_s first_sound_s close (partial) 81% crest_lu high (major); first_sound_s high (major); onset_rate low (major) - 13.8
deadair 4 s of silence after the end end_silence_s end_silence_s close (partial) 81% crest_lu high (major); first_sound_s high (major); end_silence_s high (major) - 14.4
fadeout 12 s fade-out fade_s, final_lu, end_lu fade_s, final_lu close (partial) 77% crest_lu high (major); first_sound_s high (major); final_lu low (major); fade_s high (minor) - 12.6
hole 2.5 s muted mid-film max_gap_s max_gap_s needs work (partial) 74% crest_lu high (major); first_sound_s high (major); max_gap_s high (major); onset_rate low (major) - 13.2
squashed over-compressed: 20:1 from -36 dB + limiter lra, crest_lu no launch-grade (partial) 90% true_peak high (major); side_to_mid_db high (minor) - 14.6
sfx_only the SFX stem alone, with the picture music_frac music_frac needs work 77% lufs_i low (critical); music_frac low (critical); band_sub low (minor); band_low low (minor); band_mid high (minor) 0.787 194.8
amb_only the ambience stem alone music_frac, onset_rate, hit_rate music_frac needs work 68% lufs_i low (critical); music_frac low (critical); crest_lu high (major); band_sub low (minor); band_low low (minor); band_mid high (minor); band_presence high (minor) 0.882 90.6
synth_bed the retired code-synthesized music bed p_real p_real not launch-grade 49% lufs_i low (critical); p_real low (critical); music_frac low (critical); crest_lu high (major); first_sound_s high (major); onset_rate low (major); band_sub low (minor); band_low low (minor); band_mid high (minor); side_to_mid_db high (minor) 0.095 43.7
sine 30 s 440 Hz sine p_real, music_frac, onset_rate p_real, music_frac, onset_rate not launch-grade 35% lufs_i low (critical); p_real low (critical); music_frac low (critical); lra low (major); crest_lu low (major); onset_rate low (major); hit_rate low (major); band_sub low (minor); band_low low (minor); band_lowmid high (minor); band_mid low (minor); band_presence low (minor); band_air low (minor); side_to_mid_db low (minor) 0.03 47.5
pink 30 s pink noise p_real, music_frac, onset_rate p_real, music_frac, onset_rate not launch-grade 46% lufs_i low (critical); p_real low (critical); music_frac low (critical); lra low (major); crest_lu low (major); onset_rate low (major); hit_rate low (major); band_mid high (minor); band_presence high (minor); side_to_mid_db low (minor) 0.158 22.7

Caught: 13 of 14 known-bad mixes have the expected metric flagged (missed: squashed). Verdicts: launch-grade 1, close 5, needs work 5, not launch-grade 3 (control: close). A single structural defect on a mix that is already close usually stays close: the verdict is coarse, the flagged rows and FIX FIRST actions are the signal. Over-compression is not a defect by these references: two of them (LRA 1.7 / 3.3 LU, loudest 3 s only 1.1-1.2 LU above integrated) are as dense as the squashed mix.

The Clawd film: real soundtrack vs the retired synthesized soundtrack

Reading: the scorecard rates the real soundtrack close (90%) and the synthesized one needs work (80%). The structural metrics separate them only a little: both are loud enough, continuous and on the picture; the synth version fades away at the end, is sparser and has a long fade, while the real one starts late (a 1 s near-silent cold open) and has one loudness peak that sticks out. The difference a listener hears most - synthesized timbre - is not structural; it is what p_real measures: real 0.94 vs synth 0.33 (earshot.realness, verdict thresholds lo 0.4 / hi 0.77 at this run); below 0.5 the p_real row is critical, which is what moves the synth version's verdict. Caveat: the realness model (another module) was trained partly on these 7 launch films, on the Clawd project's real asset library and on the retired synthesized stems (eval/realness/), so p_real on the Clawd films is not an independent test.

clawd_real: Clawd short film with its real-recordings soundtrack (licensed music, foley, ambience)

Verdict close, weighted score 90%, 21/23 in range, 0 critical / 2 major issues; critique call 4.6 s with the measurement passes cached (realness runs inside it).

  • [major] nothing reaches body level until 0:01.01 (before that the sound sits around -42 LUFS, 27 LU under the body; refs 0.04 s .. 0.48 s): raise the opening or start the music at 0:00
  • [major] the loudest 3 s (around 0:41.50, -10.4 LUFS) stand 4.6 LU above the integrated level (refs 1.1 LU .. 3.5 LU): the rest of the film plays quiet next to it; lift the body (quietest stretch 0:06.20-0:09.20 at -24.5 LUFS short-term) or tame that peak
  • tip: 5 of 26 visual events get no sound at all (no onset within 100 ms, no whoosh / riser peak within 300 ms, no level peak within 150 ms; the matcher earshot timeline uses): 0:00.30, 0:00.80, 0:01.30, 0:30.00, 0:51.85; give the ones that matter a whoosh into the cut, a hit or a UI click, or keep one as a deliberate stop-down
  • tip: no stop-downs: 4 of 7 references drop the sound out for 0.2-1 s right before a big hit; try one before the biggest visual change at 0:20.00
  • realness: Clawd_1080p60.mp4: real or pro-produced (p_real 0.94, mean of 15 4 s windows; file thresholds lo 0.4 / hi 0.77, weak calibration): 0 of 15 windows lean synthesized (p_real < 0.5); mean CLAP head score +2.9. No window leans synthesized (lowest p_real 0.76); that does not rule out a code-made layer under a louder real mix (most are missed): judge the source files and stems.

clawd_synth: the same film with the retired code-synthesized soundtrack

Verdict needs work, weighted score 80%, 19/23 in range, 1 critical / 2 major issues; critique call 4.6 s with the measurement passes cached (realness runs inside it).

  • [critical] realness: likely code-synthesized (p_real 0.33, at or below the model's 0.40 threshold); 4 s windows at or below the window threshold 0.17 (code-synthesized): 0:56.00-1:00.00 (0.05): replace oscillator / noise layers with real recordings and produced music (skills/real-sound has the sourcing and licence workflow); judge each stem with earshot realness stem.wav (a finished mix is harder to judge than its sources)
  • [major] the final moment sits 7.5 LU below the body: end on a logo sting / hit at the last visual event (0:59.00) instead of fading away (refs -2.2 LU .. 5.6 LU)
  • [major] sparse: 2.1 onsets/s (refs 2.46 /s .. 8.22 /s): use music with more rhythmic drive, or add UI foley (clicks, typing, ticks) on the on-screen actions
  • [minor] 4.4 s fade-out (refs 0.08 s .. 2.54 s): end on a sting with a short ring-out, or cut hard on the last picture cut
  • tip: 8 of 26 visual events get no sound at all (no onset within 100 ms, no whoosh / riser peak within 300 ms, no level peak within 150 ms; the matcher earshot timeline uses): 0:00.80, 0:01.30, 0:28.33, 0:28.98, 0:30.00, 0:33.00, 0:40.88, 0:51.85; give the ones that matter a whoosh into the cut, a hit or a UI click, or keep one as a deliberate stop-down
  • realness: Clawd_1080p60_synth-audio.mp4: likely code-synthesized (p_real 0.33, mean of 15 4 s windows; file thresholds lo 0.4 / hi 0.77, weak calibration): 14 of 15 windows lean synthesized (p_real < 0.5); mean CLAP head score -0.5. Leaning synthesized: 0:00.00-0:48.00, 0:52.00-1:00.00; 1 at or below the 4 s-window threshold 0.17 (code-synthesized): 0:56.00-1:00.00.

Hit / onset timing vs ground truth (tuning data: the Clawd real soundtrack)

The mix report lists 264 sample placements (every key tick, pop, whoosh, hit, music motif note ...). earshot found 16 salient hits and 149 onsets in the mixed film audio; 11 of the 16 hits sit within 2 ms of a placement (median distance 1.0 ms).

tolerance hits on a placement (chance) onsets on a placement placements with an onset
+-20 ms 75% (10%) 61% 62%
+-50 ms 88% (25%) 75% 78%
+-100 ms 88% (39%) 79% 83%

Chance = the same hits shifted off the placements by fixed offsets. Hit and onset times are moved to the steepest rise of the 2 ms level (attack), which is how a mixer aligns a sample.

Visual-event detector (earshot.video frame difference) vs ffmpeg scene cuts

Scene cuts = video.cuts(path, method='scene') (ffmpeg scene score 0.3). 'Same as first build' = the core video.frame_diff signal is bit-identical to this module's first-build picture pass, so moving the detector into core changed no number.

film scene cuts visual events share of scene cuts also found (+-1 frame) same as first build
ref_1 0 28 n.a. (< 5 scene cuts) True
ref_2 2 9 n.a. (< 5 scene cuts) True
ref_3 15 17 1.0 True
ref_4 17 42 0.82 True
ref_5 2 16 n.a. (< 5 scene cuts) True
ref_6 11 31 1.0 True
ref_7 40 67 0.8 True
clawd_real 10 26 1.0 True
clawd_synth 10 26 1.0 True