Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.
Realness detector: held-out evaluation
Generated by eval/realness/run_eval.py --stage report; every number below comes from that run (results.json holds
all of them). The question: is a sound a real recording / professionally produced audio (positive), or amateur audio
synthesized in code: oscillators, filtered noise, simple envelopes, algorithmic reverb, beeps (negative)?
Professionally produced electronic music and pro sound design count as positive.
Data
1484 clean clips (691 real/pro, 793 synthesized) plus 1863 perturbed copies
(robustness tests and training-only augmentation). A clip is a whole sound up to 6 s, else a 4 s window: exactly the
clips earshot.realness scores at inference. Audio stays in out/realness/ (never committed).
| family | clean clips | what |
|---|---|---|
| cinetic | 318 | cinetic synth engine: generator types x seeds / parameters, short mixes, 6 full rendered scores |
| generic | 260 | independent numpy generators (14 types) + 6 generic code-made mixes |
| pro-film | 72 | audio of the 7 professional launch films (4 s windows) |
| pro-music | 119 | professionally produced music (assets/music, kept and rejected) |
| real-asset | 366 | Freesound / Mixkit recordings used in the Clawd film (7 categories) |
| real-rejected | 134 | real recordings rejected for quality (still real) |
| retired | 215 | the retired code-synthesized Clawd score: stems, beds, melody, events |
Tuning vs evaluation: model choices (regularisation C of the CLAP head, fusion weights) are fitted inside each training
fold by inner grouped CV; nothing is tuned on the held-out group. The verdict thresholds (lo 0.17 / hi 0.77)
are read from the scheme-M held-out predictions (>= 92.5% precision on each side), so the band numbers are in-sample
for that choice (two numbers fitted on 1484 clips); the file thresholds for long files likewise come from held-out
long files. Design choices made by looking at held-out results across the three build rounds: augmentation on
(robustness below), the precision target, music weighted as four sources (scheme M), the feature set, the file
threshold rule, and not training on layered clips (layer test below).
The final model (earshot/data/realness_model.json) is trained on all groups with the scheme-M grouping. Robustness copies are only ever scored, never trained
on; augmentation copies (synthetic noise floor, gating, short excerpts, codec round trips, with randomised parameters
different from the robustness set, never the real room tone) are only trained on.
Held-out results
Every clip is scored by a model that never saw its group (nor augmented copies of its clips).
| scheme | clips | ROC-AUC | [email protected] | Brier | log loss |
|---|---|---|---|---|---|
| A: leave one real category / synth family out (headline) | 1484 | 0.9219 | 0.8598 | 0.1088 | 0.36 |
| A without augmented training copies | 1484 | 0.9314 | 0.8693 | 0.1023 | 0.3387 |
| A, CLAP head only | 1484 | 0.9257 | 0.8612 | 0.1069 | 0.3514 |
| A, signal-measurement head only | 1484 | 0.721 | 0.7123 | 0.2448 | 0.7195 |
| B: leave one synth generator type out | 1484 | 0.9576 | 0.9016 | 0.0764 | 0.2599 |
| C: retired score never trained on | 1484 | 0.8854 | 0.8389 | 0.1298 | 0.4509 |
| D: retired score and cinetic engine never trained on | 1224 | 0.916 | 0.8554 | 0.1147 | 0.401 |
| M: as A, but music held out by provider (mk / sb / ch / other) | 1484 | 0.92 | 0.8801 | 0.0984 | 0.3549 |
D is scored on real vs the retired score and the cinetic engine only (its sole negatives were the generic generators).
The required check: real recordings rank above the retired synthesized score
The retired score (the synthesized soundtrack the Clawd project replaced) is held out of every training set (scheme C), or held out together with the whole cinetic engine that made it (scheme D):
| check | ROC-AUC | [email protected] | mean p_real of the synth side |
|---|---|---|---|
| C: real vs retired | 0.9475 | 0.862 | 0.166 |
| C: real recordings only (no music/films) vs retired | 0.9418 | - | - |
| D: real vs retired | 0.9085 | 0.8389 | 0.185 |
| D: real vs cinetic | 0.921 | 0.8484 | 0.145 |
| D: real recordings only (no music/films) vs retired | 0.9031 | - | - |
Verdict bands (scheme M held-out predictions)
real or pro-produced when p_real >= 0.77, likely code-synthesized when p_real <= 0.17, else uncertain.
Coverage 79% (the rest is uncertain); precision of the real verdict 0.9241, of the
synth verdict 0.9224; 6.7% of real clips are called synthesized and
5.5% of synthesized clips are called real. Other precision targets:
| precision target | lo | hi | coverage | real called synth | synth called real |
|---|---|---|---|---|---|
| 0.95 | 0.05 | 0.88 | 0.5195 | 0.0405 | 0.0277 |
| 0.925 | 0.17 | 0.77 | 0.7904 | 0.0666 | 0.0555 |
| 0.9 | 0.47 | 0.68 | 0.9407 | 0.11 | 0.0794 |
Calibration (scheme M, clean clips):
| p_real bin | clips | mean p_real | fraction real |
|---|---|---|---|
| 0.0-0.1 | 494 | 0.041 | 0.073 |
| 0.1-0.2 | 119 | 0.141 | 0.118 |
| 0.2-0.3 | 67 | 0.242 | 0.179 |
| 0.3-0.4 | 62 | 0.347 | 0.129 |
| 0.4-0.5 | 37 | 0.455 | 0.324 |
| 0.5-0.6 | 42 | 0.555 | 0.571 |
| 0.6-0.7 | 44 | 0.66 | 0.477 |
| 0.7-0.8 | 62 | 0.756 | 0.742 |
| 0.8-0.9 | 134 | 0.856 | 0.866 |
| 0.9-1.0 | 423 | 0.96 | 0.95 |
Pro music
Scheme A holds out all music at once, so the model scoring it has never heard real music while its negatives include synthesized scores: a pessimistic estimate (music mean p_real 0.595). Scheme M holds out one provider at a time (the model has heard other providers' music): 119 windows of 52 files, mean p_real 0.859, window [email protected] 0.9244, file [email protected] 0.9615; files (mean of windows) called real: 44, called code-synthesized: 0. The price of weighting music as four sources, on everything except music: scheme A AUC 0.9292 / Brier 0.1002 vs scheme M AUC 0.9123 / Brier 0.1022.
| provider fold | windows | mean p_real | [email protected] |
|---|---|---|---|
| music:ch | 30 | 0.83 | 0.9 |
| music:mk | 45 | 0.887 | 0.9333 |
| music:other | 8 | 0.77 | 0.75 |
| music:sb | 36 | 0.867 | 0.9722 |
Per group (A: the whole group held out; M: the same, music held out by provider)
| group | label | clips | A mean p_real | A [email protected] | A AUC vs the other class | M mean p_real | M [email protected] |
|---|---|---|---|---|---|---|---|
| ambience | real/pro | 77 | 0.78 | 0.974 | 0.9542 | 0.868 | 0.987 |
| cinetic | synth | 318 | 0.081 | 0.9686 | 0.9753 | 0.101 | 0.978 |
| film | real/pro | 72 | 0.75 | 0.9722 | 0.9449 | 0.886 | 0.9722 |
| foley | real/pro | 73 | 0.869 | 0.9726 | 0.9721 | 0.905 | 0.9726 |
| garden | real/pro | 78 | 0.835 | 0.9487 | 0.9572 | 0.857 | 0.9103 |
| generic | synth | 260 | 0.329 | 0.6731 | 0.8355 | 0.322 | 0.6846 |
| keyboard | real/pro | 77 | 0.969 | 1.0 | 0.9925 | 0.963 | 1.0 |
| magic | real/pro | 68 | 0.451 | 0.4853 | 0.753 | 0.405 | 0.4559 |
| motion | real/pro | 66 | 0.649 | 0.6667 | 0.8794 | 0.622 | 0.6667 |
| music | real/pro | 119 | 0.595 | 0.6891 | 0.8867 | 0.859 | 0.9244 |
| retired | synth | 215 | 0.166 | 0.9674 | 0.9474 | 0.157 | 0.9674 |
| toy | real/pro | 61 | 0.838 | 0.9672 | 0.9627 | 0.878 | 0.9672 |
Synthesized generator types (A: the whole family held out; B: only that type held out)
| type | clips | A mean p_real | A acc | B mean p_real | B acc |
|---|---|---|---|---|---|
| cinetic:bass | 10 | 0.025 | 1.0 | 0.021 | 1.0 |
| cinetic:blip | 10 | 0.036 | 1.0 | 0.035 | 1.0 |
| cinetic:clap | 10 | 0.016 | 1.0 | 0.013 | 1.0 |
| cinetic:crisp_tick | 10 | 0.045 | 1.0 | 0.015 | 1.0 |
| cinetic:drone | 10 | 0.108 | 1.0 | 0.144 | 1.0 |
| cinetic:fm_bell | 10 | 0.067 | 1.0 | 0.118 | 1.0 |
| cinetic:hat | 10 | 0.141 | 1.0 | 0.234 | 0.9 |
| cinetic:key_click | 10 | 0.069 | 1.0 | 0.058 | 1.0 |
| cinetic:keys | 10 | 0.131 | 1.0 | 0.203 | 0.8 |
| cinetic:kick | 10 | 0.026 | 1.0 | 0.066 | 1.0 |
| cinetic:marker | 10 | 0.016 | 1.0 | 0.01 | 1.0 |
| cinetic:mix | 16 | 0.094 | 1.0 | 0.298 | 0.875 |
| cinetic:motif | 10 | 0.012 | 1.0 | 0.016 | 1.0 |
| cinetic:pluck | 10 | 0.007 | 1.0 | 0.012 | 1.0 |
| cinetic:riser | 10 | 0.033 | 1.0 | 0.037 | 1.0 |
| cinetic:score | 42 | 0.038 | 1.0 | 0.148 | 1.0 |
| cinetic:snap | 10 | 0.132 | 0.9 | 0.246 | 0.8 |
| cinetic:sub_boom | 10 | 0.04 | 1.0 | 0.083 | 1.0 |
| cinetic:suck | 10 | 0.176 | 1.0 | 0.184 | 1.0 |
| cinetic:supersaw | 10 | 0.004 | 1.0 | 0.004 | 1.0 |
| cinetic:swell | 10 | 0.023 | 1.0 | 0.027 | 1.0 |
| cinetic:thock | 10 | 0.092 | 0.9 | 0.078 | 0.9 |
| cinetic:tick | 10 | 0.155 | 0.9 | 0.162 | 0.8 |
| cinetic:tock | 10 | 0.271 | 0.8 | 0.188 | 1.0 |
| cinetic:typing | 10 | 0.022 | 1.0 | 0.035 | 1.0 |
| cinetic:ui_click | 10 | 0.165 | 0.8 | 0.167 | 0.8 |
| cinetic:whoosh | 10 | 0.363 | 0.7 | 0.502 | 0.5 |
| cinetic:word_blip | 10 | 0.097 | 1.0 | 0.081 | 1.0 |
| generic:beep | 14 | 0.299 | 0.7143 | 0.126 | 0.9286 |
| generic:chiptune | 14 | 0.19 | 0.9286 | 0.019 | 1.0 |
| generic:chord | 14 | 0.335 | 0.7143 | 0.177 | 0.9286 |
| generic:click | 14 | 0.289 | 0.7143 | 0.2 | 1.0 |
| generic:drums | 14 | 0.009 | 1.0 | 0.027 | 1.0 |
| generic:dtmf | 14 | 0.713 | 0.0714 | 0.384 | 1.0 |
| generic:fm | 14 | 0.276 | 0.7857 | 0.155 | 0.8571 |
| generic:impact | 14 | 0.178 | 0.8571 | 0.19 | 0.8571 |
| generic:lfsr | 14 | 0.354 | 0.7143 | 0.079 | 1.0 |
| generic:mix | 36 | 0.609 | 0.3611 | 0.588 | 0.3889 |
| generic:pad | 14 | 0.3 | 0.7857 | 0.132 | 0.9286 |
| generic:pluck | 14 | 0.087 | 0.9286 | 0.077 | 0.9286 |
| generic:rain | 14 | 0.664 | 0.2143 | 0.61 | 0.2143 |
| generic:riser | 14 | 0.043 | 1.0 | 0.039 | 1.0 |
| generic:siren | 14 | 0.67 | 0.1429 | 0.177 | 0.8571 |
| generic:whoosh | 14 | 0.103 | 1.0 | 0.082 | 1.0 |
| generic:wind | 14 | 0.035 | 1.0 | 0.05 | 1.0 |
| retired:ambience | 15 | 0.224 | 0.9333 | 0.216 | 0.8667 |
| retired:bed | 15 | 0.285 | 0.9333 | 0.256 | 0.8667 |
| retired:melody | 12 | 0.151 | 1.0 | 0.1 | 1.0 |
| retired:melody-event | 25 | 0.174 | 0.96 | 0.147 | 0.92 |
| retired:stem_arp | 12 | 0.045 | 1.0 | 0.011 | 1.0 |
| retired:stem_bass | 13 | 0.059 | 1.0 | 0.027 | 1.0 |
| retired:stem_bed | 15 | 0.292 | 0.9333 | 0.265 | 0.8 |
| retired:stem_bells | 2 | 0.096 | 1.0 | 0.049 | 1.0 |
| retired:stem_bells-event | 1 | 0.028 | 1.0 | 0.004 | 1.0 |
| retired:stem_drums | 11 | 0.097 | 1.0 | 0.056 | 1.0 |
| retired:stem_drums-event | 25 | 0.089 | 1.0 | 0.084 | 1.0 |
| retired:stem_pad | 14 | 0.045 | 1.0 | 0.014 | 1.0 |
| retired:stem_reverb | 15 | 0.239 | 1.0 | 0.3 | 0.7333 |
| retired:stem_sfx | 15 | 0.241 | 0.9333 | 0.339 | 0.8 |
| retired:stem_sfx-event | 25 | 0.196 | 0.92 | 0.221 | 0.84 |
File level (long files: the mean of their window p_real)
Sources with >= 2 windows (films, music, ambience, retired stems, full synthesized mixes): 119 files (23 synthesized). Scheme A: AUC 0.9022, [email protected] 0.8235. Scheme M (the final model's grouping): AUC 0.9538, [email protected] 0.9244. File verdict thresholds (lo: 92.5% precision on held-out long files (scheme M); hi: max(that, the clip hi)): lo 0.4 / hi 0.77; with them 0.8655 of the files get a firm verdict, real files called synthesized 0.0, synthesized files called real 0.0435. Only 23 synthesized long files from three code bases: weak calibration. Scheme M synthesized files, lowest to highest: stems/bass.wav 0.053, stems/arp.wav 0.069, stems/pad.wav 0.072, cinetic_score_2/soundtrack.wav 0.081, stems/bells.wav 0.086, stems/drums.wav 0.089, cinetic_score_1/soundtrack.wav 0.105, cinetic_score_0/soundtrack.wav 0.106, cinetic_score_4/soundtrack.wav 0.129, cinetic_score_5/soundtrack.wav 0.13, audio/melody.wav 0.131, audio/ambience.wav 0.191, cinetic_score_3/soundtrack.wav 0.197, stems/sfx.wav 0.207, _retired_synth/bed.wav 0.254, stems/bed.wav 0.256, stems/reverb.wav 0.316, mixes/generic_mix_4.wav 0.513, mixes/generic_mix_0.wav 0.561, mixes/generic_mix_1.wav 0.614, mixes/generic_mix_2.wav 0.662, mixes/generic_mix_3.wav 0.769, mixes/generic_mix_5.wav 0.953. Scheme M real files under 0.6: music/ch-59068_sakuragirl-yay.mp3 0.401, music/mk-1142_gummies.mp3 0.435, ambience/fs-639459_summer-meadow-wind.ogg 0.499.
Robustness (perturbed copies, scored by the fold model that never saw their group)
noise70 = pink noise at -70 dBFS RMS added to synthesized clips; room70 = a real room tone (quietest 20 s of a rain recording) at -70 dBFS; gated = real clips gated to exact digital silence below -45 dB re peak; short = 0.15 s around the peak; mp3 = 128 kbps round trip. The final model is trained with augmentation (A, M); A_noaug shows the trade-off.
| perturbation | label | clips | acc perturbed (A) | acc unperturbed (A) | acc perturbed (A_noaug) | acc perturbed (M) |
|---|---|---|---|---|---|---|
| noise70 | synth | 115 | 0.8 | 0.9304 | 0.687 | 0.8348 |
| room70 | synth | 115 | 0.7391 | 0.8609 | 0.7217 | 0.7739 |
| gated | real/pro | 106 | 0.7453 | 0.8113 | 0.7264 | 0.7547 |
| short | synth | 115 | 0.8957 | 0.9043 | 0.9565 | 0.9043 |
| short | real/pro | 107 | 0.7383 | 0.8972 | 0.7196 | 0.729 |
| mp3 | synth | 115 | 0.8609 | 0.887 | 0.8696 | 0.8957 |
| mp3 | real/pro | 107 | 0.8505 | 0.8505 | 0.8411 | 0.8505 |
Level (final model, so in-sample clips; it measures sensitivity, not accuracy): the same clips at their own level, 20 dB quieter, normalised to -25 dBFS RMS, after the inference lift (sources under -30 dBFS RMS are raised to -30 dBFS RMS, peak <= -1 dBFS), and moved to -56 dBFS RMS without / with that lift (the first build only lifted sources under -60 dBFS RMS, so -56 dBFS input was scored as it was).
| clips | n | median RMS dBFS | lifted at inference | mean p_real own / -20dB / norm-25 / lifted / at-56 / at-56_lifted | mean abs change -20dB / norm-25 / lifted / at-56 / at-56_lifted | acc own / -20dB / norm-25 / lifted / at-56 / at-56_lifted |
|---|---|---|---|---|---|---|
| real | 34 | -30.9 | 18 | 0.887 / 0.867 / 0.86 / 0.876 / 0.864 / 0.863 | 0.046 / 0.033 / 0.012 / 0.063 / 0.032 | 0.941 / 0.941 / 0.912 / 0.912 / 0.941 / 0.912 |
| cinetic | 31 | -24.3 | 8 | 0.062 / 0.08 / 0.068 / 0.062 / 0.082 / 0.072 | 0.037 / 0.012 / 0.001 / 0.059 / 0.017 | 1.0 / 0.968 / 1.0 / 1.0 / 0.968 / 1.0 |
| generic | 28 | -24.6 | 7 | 0.042 / 0.05 / 0.043 / 0.042 / 0.064 / 0.042 | 0.015 / 0.006 / 0.0 / 0.03 / 0.007 | 1.0 / 1.0 / 1.0 / 1.0 / 1.0 / 1.0 |
Baselines (no training)
- CLAP zero-shot (log-sum-exp over 6 'real' prompts minus 6 'synth' prompts): AUC 0.8579, [email protected] 0.7487 on all clean clips.
- Single signal measurements (AUC, physical sign): silence_frac 0.5435, floor_db 0.4436, spec_floor_db 0.7546, log_flatness 0.7366, peakiness_db 0.7232, line_share 0.6516, bandwidth_khz 0.6971, quiet_flat 0.7859, tonal_frames 0.599, pitch_wobble 0.6744, env_resid_db 0.594, texture_kurt 0.5601, crest_db 0.7385.
- AST, on a stratified subset of 244 clean clips (AST costs ~5x CLAP here; the subset predates
the full synthesized mixes): max over the synth classes (Synthesizer, Sine wave, Beep, bleep, Electronic tuner, Chirp tone, Dial tone, Pink noise, White noise, Static, Buzz, Hum, Theremin, Video game music, Ringtone, Sound effect)
AUC 0.7555; single classes (high = real):
Synthesizer 0.8465, Sine wave 0.8253, Beep, bleep 0.7498, Electronic music 0.7551, Music 0.6642.
On the same subset: scheme A 0.951, CLAP zero-shot 0.8778,
clap+signal heads refit on the subset 0.9566, plus an AST head 0.9533,
AST head alone 0.7618. An AST head adds nothing, so AST is evidence only (
--ast).
What carries the decision
- Fusion weights (fitted on inner held-out head outputs): {'clap': 1.212409, 'signal': 0.0}. The CLAP head carries the decision; the signal head (0.721 AUC alone vs 0.9257 for CLAP alone) gets weight 0.0, so its measurements are reported as evidence (each with the share of training clips past that threshold that were of that side), not votes.
- The CLAP head's weight direction is closest to these vocabulary texts. Toward real: a high quality studio recording, a metal scrape, an old typewriter, a match being struck, a knife chopping on a board, a small object dropping on a table, pizzicato strings, a can opening. Toward synthesized: a square wave chiptune blip, a simple electronic synthesizer tone made by code, a warm analog synth pad, a cheap synthesized beep, a computer-generated electronic tone, filtered white noise, an 8-bit video game sound effect, an 8-bit sound effect.
- Signal-head coefficients (standardised, sign-constrained to their physics): {'silence_frac': -0.106636, 'floor_db': 0.317801, 'spec_floor_db': 0.0, 'log_flatness': 0.0, 'peakiness_db': 0.0, 'line_share': 0.0, 'bandwidth_khz': 0.0, 'quiet_flat': 0.898494, 'tonal_frames': 0.0, 'pitch_wobble': 0.199091, 'env_resid_db': 0.007039, 'texture_kurt': 0.0, 'crest_db': 0.755868}.
- Exact digital silence is weak evidence: over half of the clips of both classes contain some (edited and gated library recordings too), so it is never a verdict reason, only context quoted with its training base rates.
- One rule overrides the model at inference (earshot.realness.pure_tone): a pure, dead-steady tone (one spectral line in
every frame, crest <= 4.5 dB, no pitch drift) gets p_real <= 0.03, because bare test tones lie outside the training
data. Checked by
eval/realness/tone_guard.py(tone_guard.json): a 30 s 440 Hz sine gets 0.48 from the model alone and 0.03 with the rule; the rule fires on none of 4741 audible clips of the 345 real recordings.
Application files (whole soundtracks, 4 s windows)
| file | held-out model: p_real (mean) / verdict | windows < 0.5 / <= lo / all (held-out) | final model (in-sample): p_real / verdict |
|---|---|---|---|
| clawd_film_real_soundtrack | 0.91 / real or pro-produced | 0 / 0 / 15 | 0.942 / real or pro-produced |
| clawd_film_synth_soundtrack | 0.334 / likely code-synthesized | 13 / 1 / 15 | 0.327 / likely code-synthesized |
| clawd_soundtrack_wav | 0.901 / real or pro-produced | 0 / 0 / 15 | 0.933 / real or pro-produced |
| clawd_stem_pro_sfx | 0.646 / uncertain | 5 / 0 / 15 | 0.787 / real or pro-produced |
| clawd_stem_pro_amb | 0.84 / real or pro-produced | 0 / 0 / 15 | 0.882 / real or pro-produced |
| clawd_stem_pro_music | 0.811 / real or pro-produced | 0 / 0 / 15 | 0.915 / real or pro-produced |
The held-out model (1273 training clips) never saw accepted Clawd assets (incl. assets/music) and the retired score: the real soundtrack is built from 83 of those accepted recordings, and the synthesized soundtrack is the retired score, so this column is out of sample on both sides. The final model trained on both (in-sample, optimistic). Verdicts use the file thresholds for files over 6 s.
Professionally made synthetic sounds (Windows UI / alert sounds, never trained on)
70 top-level wav files of C:/Windows/Media (chimes, alarms, ringtones, notification tones; made by sound designers, mostly synthesized): 32 called 'likely code-synthesized', 32 'uncertain', 6 'real or pro-produced' (median p_real 0.238). Lowest: Windows Hardware Insert.wav 0.006, Windows Hardware Remove.wav 0.007, Alarm10.wav 0.009, Windows Notify System Generic.wav 0.009, Windows Background.wav 0.013. Highest: Ring08.wav 0.861, Speech Off.wav 0.885, ding.wav 0.893, Windows Recycle.wav 0.898, Windows Default.wav 0.946. The detector hears synthesis, not provenance: a clean, simple synthesized tone reads as code-made even when a professional made it and it is licensed. An agent using a licensed UI-sound pack must not treat that verdict as proof.
Layer test: a code-synthesized layer inside a real mix
369 clips: 184 real 4 s beds (ambience, films, music, foley ...; mean p_real alone
0.804, 0.114 of them under 0.5 alone) with one code-synthesized layer each (a
cinetic / generic / retired clip: one-shots 1-4 times, 4 s windows throughout), at a layer-to-bed RMS of -12 / -6 / 0 /
+6 dB (layer RMS over the span it plays). Every clip is scored by a model that never saw its bed's group nor its
layer's family (--stage layers). 'base' = the shipped recipe; 'train' = also trained on the layered clips of the
other groups.
| layer level | clips | base: mean p_real | base: < 0.5 | base: <= lo | base: >= hi (called real) | train: mean p_real | train: < 0.5 | train: <= lo |
|---|---|---|---|---|---|---|---|---|
| -12 dB | 93 | 0.673 | 0.269 | 0.172 | 0.538 | 0.591 | 0.344 | 0.204 |
| -6 dB | 91 | 0.621 | 0.33 | 0.121 | 0.429 | 0.536 | 0.418 | 0.132 |
| +0 dB | 94 | 0.633 | 0.298 | 0.149 | 0.479 | 0.524 | 0.457 | 0.17 |
| +6 dB | 91 | 0.622 | 0.286 | 0.143 | 0.396 | 0.512 | 0.484 | 0.154 |
Most layers are missed: the windows lean synthesized for only about a third of the clips, almost independent of the layer level, and many are still called real. Per bed group (base, -6..+6 dB, share < 0.5): {'ambience': 0.167, 'film': 0.167, 'foley': 0.214, 'garden': 0.133, 'keyboard': 0.133, 'magic': 0.833, 'motion': 0.682, 'music': 0.222, 'toy': 0.227} (magic and motion beds already lean synthesized on their own). Per layer family: {'cinetic': 0.375, 'generic': 0.221, 'retired': 0.295}. Training on layered clips (scheme M_layers: M plus the layered clips of the other groups) raised the held-out clip AUC to 0.9281 (acc 0.8935, Brier 0.0926; M: 0.92 / 0.8801 / 0.0984) and the layer detection above, but compressed the synthesized scores: at the same 92.5% precision target its thresholds are lo 0.05 / hi 0.68 with 0.5081 firm-verdict coverage (M: 0.7904). It was not adopted (decided after seeing these held-out numbers): judging source files firmly is the main use, and even trained, most layers stay unflagged. A dedicated layer head (CLAP logistic, bed vs bed
- layer, same pair folds; a scratch experiment, not in results.json) reached AUC 0.82-0.84 with 40-46% of the layers
at -6..+6 dB found at 5% false alarms on beds, but 11-17% false alarms on film and music beds: not shipped either.
So
earshot realnesssays plainly that a clean window does not rule out a code-made layer under a real mix.
Limits: what fools it
- Synthesized types it misses most (scheme A acc < 0.75, lowest first; acc / mean p_real): generic:dtmf (0.0714 / 0.713), generic:siren (0.1429 / 0.67), generic:rain (0.2143 / 0.664), generic:mix (0.3611 / 0.609), cinetic:whoosh (0.7 / 0.363), generic:beep (0.7143 / 0.299), generic:chord (0.7143 / 0.335), generic:click (0.7143 / 0.289), generic:lfsr (0.7143 / 0.354). Code-made noise textures that imitate nature (rain, wind beds) and telephone-like tones (DTMF, sirens) sound like recordings of real things to CLAP, and a generic code-made mix built on such beds can score real.
- Real groups it doubts (scheme A acc < 0.8; acc / mean p_real): magic (0.4853 / 0.451), motion (0.6667 / 0.649), music (0.6891 / 0.595). Library "magic" and "motion" sounds are themselves designed (often synthesized) by sound designers, and real electronic music sits near synthesized scores: read 'uncertain' there as "sounds designed", not "fake".
- Finished mixes are harder than sources, and a code-made layer under a louder real mix is usually missed (layer test). Judge the source files and stems, not only the final mix.
- Professionally made synthetic tones (UI sounds, chimes, alarms) are called code-synthesized: it hears synthesis, not provenance (Windows UI sounds above).
- A -70 dBFS noise floor added to synthesized audio lowers accuracy (robustness table): the detector mostly hears what a sound is, not microphone physics. Very short clips (< 0.2 s) of real sounds are harder.
- Training data: one sound library (Freesound / Mixkit picks for one film), 7 launch films, ~120 music windows; the negatives come from three code bases (cinetic, the retired score made with it, generic numpy generators). Synthesis from other tools (pro soft synths, granular, sample-based) was never seen: pro-produced synth music is meant to score real, and amateur code using techniques outside these three code bases may also score real.
Reproduce
.venv/Scripts/python.exe eval/realness/build_data.py # clips -> out/realness/
.venv/Scripts/python.exe eval/realness/build_data.py --only layers # layer-test clips
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/extract.py --ast 240 # repeat until "done"
OMP_NUM_THREADS=2 EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage cv # until "cv done"
OMP_NUM_THREADS=2 EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage layers # twice
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage apps
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/train.py
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage level # needs the final model
.venv/Scripts/python.exe eval/realness/run_eval.py --stage report # results.json + RESULTS.md
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/train.py # embeds the summary in the model