Skip to content

Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.

24.3 KB
Raw Blame

Realness detector: held-out evaluation

Generated by eval/realness/run_eval.py --stage report; every number below comes from that run (results.json holds all of them). The question: is a sound a real recording / professionally produced audio (positive), or amateur audio synthesized in code: oscillators, filtered noise, simple envelopes, algorithmic reverb, beeps (negative)? Professionally produced electronic music and pro sound design count as positive.

Data

1484 clean clips (691 real/pro, 793 synthesized) plus 1863 perturbed copies (robustness tests and training-only augmentation). A clip is a whole sound up to 6 s, else a 4 s window: exactly the clips earshot.realness scores at inference. Audio stays in out/realness/ (never committed).

family clean clips what
cinetic 318 cinetic synth engine: generator types x seeds / parameters, short mixes, 6 full rendered scores
generic 260 independent numpy generators (14 types) + 6 generic code-made mixes
pro-film 72 audio of the 7 professional launch films (4 s windows)
pro-music 119 professionally produced music (assets/music, kept and rejected)
real-asset 366 Freesound / Mixkit recordings used in the Clawd film (7 categories)
real-rejected 134 real recordings rejected for quality (still real)
retired 215 the retired code-synthesized Clawd score: stems, beds, melody, events

Tuning vs evaluation: model choices (regularisation C of the CLAP head, fusion weights) are fitted inside each training fold by inner grouped CV; nothing is tuned on the held-out group. The verdict thresholds (lo 0.17 / hi 0.77) are read from the scheme-M held-out predictions (>= 92.5% precision on each side), so the band numbers are in-sample for that choice (two numbers fitted on 1484 clips); the file thresholds for long files likewise come from held-out long files. Design choices made by looking at held-out results across the three build rounds: augmentation on (robustness below), the precision target, music weighted as four sources (scheme M), the feature set, the file threshold rule, and not training on layered clips (layer test below). The final model (earshot/data/realness_model.json) is trained on all groups with the scheme-M grouping. Robustness copies are only ever scored, never trained on; augmentation copies (synthetic noise floor, gating, short excerpts, codec round trips, with randomised parameters different from the robustness set, never the real room tone) are only trained on.

Held-out results

Every clip is scored by a model that never saw its group (nor augmented copies of its clips).

scheme clips ROC-AUC [email protected] Brier log loss
A: leave one real category / synth family out (headline) 1484 0.9219 0.8598 0.1088 0.36
A without augmented training copies 1484 0.9314 0.8693 0.1023 0.3387
A, CLAP head only 1484 0.9257 0.8612 0.1069 0.3514
A, signal-measurement head only 1484 0.721 0.7123 0.2448 0.7195
B: leave one synth generator type out 1484 0.9576 0.9016 0.0764 0.2599
C: retired score never trained on 1484 0.8854 0.8389 0.1298 0.4509
D: retired score and cinetic engine never trained on 1224 0.916 0.8554 0.1147 0.401
M: as A, but music held out by provider (mk / sb / ch / other) 1484 0.92 0.8801 0.0984 0.3549

D is scored on real vs the retired score and the cinetic engine only (its sole negatives were the generic generators).

The required check: real recordings rank above the retired synthesized score

The retired score (the synthesized soundtrack the Clawd project replaced) is held out of every training set (scheme C), or held out together with the whole cinetic engine that made it (scheme D):

check ROC-AUC [email protected] mean p_real of the synth side
C: real vs retired 0.9475 0.862 0.166
C: real recordings only (no music/films) vs retired 0.9418 - -
D: real vs retired 0.9085 0.8389 0.185
D: real vs cinetic 0.921 0.8484 0.145
D: real recordings only (no music/films) vs retired 0.9031 - -

Verdict bands (scheme M held-out predictions)

real or pro-produced when p_real >= 0.77, likely code-synthesized when p_real <= 0.17, else uncertain. Coverage 79% (the rest is uncertain); precision of the real verdict 0.9241, of the synth verdict 0.9224; 6.7% of real clips are called synthesized and 5.5% of synthesized clips are called real. Other precision targets:

precision target lo hi coverage real called synth synth called real
0.95 0.05 0.88 0.5195 0.0405 0.0277
0.925 0.17 0.77 0.7904 0.0666 0.0555
0.9 0.47 0.68 0.9407 0.11 0.0794

Calibration (scheme M, clean clips):

p_real bin clips mean p_real fraction real
0.0-0.1 494 0.041 0.073
0.1-0.2 119 0.141 0.118
0.2-0.3 67 0.242 0.179
0.3-0.4 62 0.347 0.129
0.4-0.5 37 0.455 0.324
0.5-0.6 42 0.555 0.571
0.6-0.7 44 0.66 0.477
0.7-0.8 62 0.756 0.742
0.8-0.9 134 0.856 0.866
0.9-1.0 423 0.96 0.95

Pro music

Scheme A holds out all music at once, so the model scoring it has never heard real music while its negatives include synthesized scores: a pessimistic estimate (music mean p_real 0.595). Scheme M holds out one provider at a time (the model has heard other providers' music): 119 windows of 52 files, mean p_real 0.859, window [email protected] 0.9244, file [email protected] 0.9615; files (mean of windows) called real: 44, called code-synthesized: 0. The price of weighting music as four sources, on everything except music: scheme A AUC 0.9292 / Brier 0.1002 vs scheme M AUC 0.9123 / Brier 0.1022.

provider fold windows mean p_real [email protected]
music:ch 30 0.83 0.9
music:mk 45 0.887 0.9333
music:other 8 0.77 0.75
music:sb 36 0.867 0.9722

Per group (A: the whole group held out; M: the same, music held out by provider)

group label clips A mean p_real A [email protected] A AUC vs the other class M mean p_real M [email protected]
ambience real/pro 77 0.78 0.974 0.9542 0.868 0.987
cinetic synth 318 0.081 0.9686 0.9753 0.101 0.978
film real/pro 72 0.75 0.9722 0.9449 0.886 0.9722
foley real/pro 73 0.869 0.9726 0.9721 0.905 0.9726
garden real/pro 78 0.835 0.9487 0.9572 0.857 0.9103
generic synth 260 0.329 0.6731 0.8355 0.322 0.6846
keyboard real/pro 77 0.969 1.0 0.9925 0.963 1.0
magic real/pro 68 0.451 0.4853 0.753 0.405 0.4559
motion real/pro 66 0.649 0.6667 0.8794 0.622 0.6667
music real/pro 119 0.595 0.6891 0.8867 0.859 0.9244
retired synth 215 0.166 0.9674 0.9474 0.157 0.9674
toy real/pro 61 0.838 0.9672 0.9627 0.878 0.9672

Synthesized generator types (A: the whole family held out; B: only that type held out)

type clips A mean p_real A acc B mean p_real B acc
cinetic:bass 10 0.025 1.0 0.021 1.0
cinetic:blip 10 0.036 1.0 0.035 1.0
cinetic:clap 10 0.016 1.0 0.013 1.0
cinetic:crisp_tick 10 0.045 1.0 0.015 1.0
cinetic:drone 10 0.108 1.0 0.144 1.0
cinetic:fm_bell 10 0.067 1.0 0.118 1.0
cinetic:hat 10 0.141 1.0 0.234 0.9
cinetic:key_click 10 0.069 1.0 0.058 1.0
cinetic:keys 10 0.131 1.0 0.203 0.8
cinetic:kick 10 0.026 1.0 0.066 1.0
cinetic:marker 10 0.016 1.0 0.01 1.0
cinetic:mix 16 0.094 1.0 0.298 0.875
cinetic:motif 10 0.012 1.0 0.016 1.0
cinetic:pluck 10 0.007 1.0 0.012 1.0
cinetic:riser 10 0.033 1.0 0.037 1.0
cinetic:score 42 0.038 1.0 0.148 1.0
cinetic:snap 10 0.132 0.9 0.246 0.8
cinetic:sub_boom 10 0.04 1.0 0.083 1.0
cinetic:suck 10 0.176 1.0 0.184 1.0
cinetic:supersaw 10 0.004 1.0 0.004 1.0
cinetic:swell 10 0.023 1.0 0.027 1.0
cinetic:thock 10 0.092 0.9 0.078 0.9
cinetic:tick 10 0.155 0.9 0.162 0.8
cinetic:tock 10 0.271 0.8 0.188 1.0
cinetic:typing 10 0.022 1.0 0.035 1.0
cinetic:ui_click 10 0.165 0.8 0.167 0.8
cinetic:whoosh 10 0.363 0.7 0.502 0.5
cinetic:word_blip 10 0.097 1.0 0.081 1.0
generic:beep 14 0.299 0.7143 0.126 0.9286
generic:chiptune 14 0.19 0.9286 0.019 1.0
generic:chord 14 0.335 0.7143 0.177 0.9286
generic:click 14 0.289 0.7143 0.2 1.0
generic:drums 14 0.009 1.0 0.027 1.0
generic:dtmf 14 0.713 0.0714 0.384 1.0
generic:fm 14 0.276 0.7857 0.155 0.8571
generic:impact 14 0.178 0.8571 0.19 0.8571
generic:lfsr 14 0.354 0.7143 0.079 1.0
generic:mix 36 0.609 0.3611 0.588 0.3889
generic:pad 14 0.3 0.7857 0.132 0.9286
generic:pluck 14 0.087 0.9286 0.077 0.9286
generic:rain 14 0.664 0.2143 0.61 0.2143
generic:riser 14 0.043 1.0 0.039 1.0
generic:siren 14 0.67 0.1429 0.177 0.8571
generic:whoosh 14 0.103 1.0 0.082 1.0
generic:wind 14 0.035 1.0 0.05 1.0
retired:ambience 15 0.224 0.9333 0.216 0.8667
retired:bed 15 0.285 0.9333 0.256 0.8667
retired:melody 12 0.151 1.0 0.1 1.0
retired:melody-event 25 0.174 0.96 0.147 0.92
retired:stem_arp 12 0.045 1.0 0.011 1.0
retired:stem_bass 13 0.059 1.0 0.027 1.0
retired:stem_bed 15 0.292 0.9333 0.265 0.8
retired:stem_bells 2 0.096 1.0 0.049 1.0
retired:stem_bells-event 1 0.028 1.0 0.004 1.0
retired:stem_drums 11 0.097 1.0 0.056 1.0
retired:stem_drums-event 25 0.089 1.0 0.084 1.0
retired:stem_pad 14 0.045 1.0 0.014 1.0
retired:stem_reverb 15 0.239 1.0 0.3 0.7333
retired:stem_sfx 15 0.241 0.9333 0.339 0.8
retired:stem_sfx-event 25 0.196 0.92 0.221 0.84

File level (long files: the mean of their window p_real)

Sources with >= 2 windows (films, music, ambience, retired stems, full synthesized mixes): 119 files (23 synthesized). Scheme A: AUC 0.9022, [email protected] 0.8235. Scheme M (the final model's grouping): AUC 0.9538, [email protected] 0.9244. File verdict thresholds (lo: 92.5% precision on held-out long files (scheme M); hi: max(that, the clip hi)): lo 0.4 / hi 0.77; with them 0.8655 of the files get a firm verdict, real files called synthesized 0.0, synthesized files called real 0.0435. Only 23 synthesized long files from three code bases: weak calibration. Scheme M synthesized files, lowest to highest: stems/bass.wav 0.053, stems/arp.wav 0.069, stems/pad.wav 0.072, cinetic_score_2/soundtrack.wav 0.081, stems/bells.wav 0.086, stems/drums.wav 0.089, cinetic_score_1/soundtrack.wav 0.105, cinetic_score_0/soundtrack.wav 0.106, cinetic_score_4/soundtrack.wav 0.129, cinetic_score_5/soundtrack.wav 0.13, audio/melody.wav 0.131, audio/ambience.wav 0.191, cinetic_score_3/soundtrack.wav 0.197, stems/sfx.wav 0.207, _retired_synth/bed.wav 0.254, stems/bed.wav 0.256, stems/reverb.wav 0.316, mixes/generic_mix_4.wav 0.513, mixes/generic_mix_0.wav 0.561, mixes/generic_mix_1.wav 0.614, mixes/generic_mix_2.wav 0.662, mixes/generic_mix_3.wav 0.769, mixes/generic_mix_5.wav 0.953. Scheme M real files under 0.6: music/ch-59068_sakuragirl-yay.mp3 0.401, music/mk-1142_gummies.mp3 0.435, ambience/fs-639459_summer-meadow-wind.ogg 0.499.

Robustness (perturbed copies, scored by the fold model that never saw their group)

noise70 = pink noise at -70 dBFS RMS added to synthesized clips; room70 = a real room tone (quietest 20 s of a rain recording) at -70 dBFS; gated = real clips gated to exact digital silence below -45 dB re peak; short = 0.15 s around the peak; mp3 = 128 kbps round trip. The final model is trained with augmentation (A, M); A_noaug shows the trade-off.

perturbation label clips acc perturbed (A) acc unperturbed (A) acc perturbed (A_noaug) acc perturbed (M)
noise70 synth 115 0.8 0.9304 0.687 0.8348
room70 synth 115 0.7391 0.8609 0.7217 0.7739
gated real/pro 106 0.7453 0.8113 0.7264 0.7547
short synth 115 0.8957 0.9043 0.9565 0.9043
short real/pro 107 0.7383 0.8972 0.7196 0.729
mp3 synth 115 0.8609 0.887 0.8696 0.8957
mp3 real/pro 107 0.8505 0.8505 0.8411 0.8505

Level (final model, so in-sample clips; it measures sensitivity, not accuracy): the same clips at their own level, 20 dB quieter, normalised to -25 dBFS RMS, after the inference lift (sources under -30 dBFS RMS are raised to -30 dBFS RMS, peak <= -1 dBFS), and moved to -56 dBFS RMS without / with that lift (the first build only lifted sources under -60 dBFS RMS, so -56 dBFS input was scored as it was).

clips n median RMS dBFS lifted at inference mean p_real own / -20dB / norm-25 / lifted / at-56 / at-56_lifted mean abs change -20dB / norm-25 / lifted / at-56 / at-56_lifted acc own / -20dB / norm-25 / lifted / at-56 / at-56_lifted
real 34 -30.9 18 0.887 / 0.867 / 0.86 / 0.876 / 0.864 / 0.863 0.046 / 0.033 / 0.012 / 0.063 / 0.032 0.941 / 0.941 / 0.912 / 0.912 / 0.941 / 0.912
cinetic 31 -24.3 8 0.062 / 0.08 / 0.068 / 0.062 / 0.082 / 0.072 0.037 / 0.012 / 0.001 / 0.059 / 0.017 1.0 / 0.968 / 1.0 / 1.0 / 0.968 / 1.0
generic 28 -24.6 7 0.042 / 0.05 / 0.043 / 0.042 / 0.064 / 0.042 0.015 / 0.006 / 0.0 / 0.03 / 0.007 1.0 / 1.0 / 1.0 / 1.0 / 1.0 / 1.0

Baselines (no training)

  • CLAP zero-shot (log-sum-exp over 6 'real' prompts minus 6 'synth' prompts): AUC 0.8579, [email protected] 0.7487 on all clean clips.
  • Single signal measurements (AUC, physical sign): silence_frac 0.5435, floor_db 0.4436, spec_floor_db 0.7546, log_flatness 0.7366, peakiness_db 0.7232, line_share 0.6516, bandwidth_khz 0.6971, quiet_flat 0.7859, tonal_frames 0.599, pitch_wobble 0.6744, env_resid_db 0.594, texture_kurt 0.5601, crest_db 0.7385.
  • AST, on a stratified subset of 244 clean clips (AST costs ~5x CLAP here; the subset predates the full synthesized mixes): max over the synth classes (Synthesizer, Sine wave, Beep, bleep, Electronic tuner, Chirp tone, Dial tone, Pink noise, White noise, Static, Buzz, Hum, Theremin, Video game music, Ringtone, Sound effect) AUC 0.7555; single classes (high = real): Synthesizer 0.8465, Sine wave 0.8253, Beep, bleep 0.7498, Electronic music 0.7551, Music 0.6642. On the same subset: scheme A 0.951, CLAP zero-shot 0.8778, clap+signal heads refit on the subset 0.9566, plus an AST head 0.9533, AST head alone 0.7618. An AST head adds nothing, so AST is evidence only (--ast).

What carries the decision

  • Fusion weights (fitted on inner held-out head outputs): {'clap': 1.212409, 'signal': 0.0}. The CLAP head carries the decision; the signal head (0.721 AUC alone vs 0.9257 for CLAP alone) gets weight 0.0, so its measurements are reported as evidence (each with the share of training clips past that threshold that were of that side), not votes.
  • The CLAP head's weight direction is closest to these vocabulary texts. Toward real: a high quality studio recording, a metal scrape, an old typewriter, a match being struck, a knife chopping on a board, a small object dropping on a table, pizzicato strings, a can opening. Toward synthesized: a square wave chiptune blip, a simple electronic synthesizer tone made by code, a warm analog synth pad, a cheap synthesized beep, a computer-generated electronic tone, filtered white noise, an 8-bit video game sound effect, an 8-bit sound effect.
  • Signal-head coefficients (standardised, sign-constrained to their physics): {'silence_frac': -0.106636, 'floor_db': 0.317801, 'spec_floor_db': 0.0, 'log_flatness': 0.0, 'peakiness_db': 0.0, 'line_share': 0.0, 'bandwidth_khz': 0.0, 'quiet_flat': 0.898494, 'tonal_frames': 0.0, 'pitch_wobble': 0.199091, 'env_resid_db': 0.007039, 'texture_kurt': 0.0, 'crest_db': 0.755868}.
  • Exact digital silence is weak evidence: over half of the clips of both classes contain some (edited and gated library recordings too), so it is never a verdict reason, only context quoted with its training base rates.
  • One rule overrides the model at inference (earshot.realness.pure_tone): a pure, dead-steady tone (one spectral line in every frame, crest <= 4.5 dB, no pitch drift) gets p_real <= 0.03, because bare test tones lie outside the training data. Checked by eval/realness/tone_guard.py (tone_guard.json): a 30 s 440 Hz sine gets 0.48 from the model alone and 0.03 with the rule; the rule fires on none of 4741 audible clips of the 345 real recordings.

Application files (whole soundtracks, 4 s windows)

file held-out model: p_real (mean) / verdict windows < 0.5 / <= lo / all (held-out) final model (in-sample): p_real / verdict
clawd_film_real_soundtrack 0.91 / real or pro-produced 0 / 0 / 15 0.942 / real or pro-produced
clawd_film_synth_soundtrack 0.334 / likely code-synthesized 13 / 1 / 15 0.327 / likely code-synthesized
clawd_soundtrack_wav 0.901 / real or pro-produced 0 / 0 / 15 0.933 / real or pro-produced
clawd_stem_pro_sfx 0.646 / uncertain 5 / 0 / 15 0.787 / real or pro-produced
clawd_stem_pro_amb 0.84 / real or pro-produced 0 / 0 / 15 0.882 / real or pro-produced
clawd_stem_pro_music 0.811 / real or pro-produced 0 / 0 / 15 0.915 / real or pro-produced

The held-out model (1273 training clips) never saw accepted Clawd assets (incl. assets/music) and the retired score: the real soundtrack is built from 83 of those accepted recordings, and the synthesized soundtrack is the retired score, so this column is out of sample on both sides. The final model trained on both (in-sample, optimistic). Verdicts use the file thresholds for files over 6 s.

Professionally made synthetic sounds (Windows UI / alert sounds, never trained on)

70 top-level wav files of C:/Windows/Media (chimes, alarms, ringtones, notification tones; made by sound designers, mostly synthesized): 32 called 'likely code-synthesized', 32 'uncertain', 6 'real or pro-produced' (median p_real 0.238). Lowest: Windows Hardware Insert.wav 0.006, Windows Hardware Remove.wav 0.007, Alarm10.wav 0.009, Windows Notify System Generic.wav 0.009, Windows Background.wav 0.013. Highest: Ring08.wav 0.861, Speech Off.wav 0.885, ding.wav 0.893, Windows Recycle.wav 0.898, Windows Default.wav 0.946. The detector hears synthesis, not provenance: a clean, simple synthesized tone reads as code-made even when a professional made it and it is licensed. An agent using a licensed UI-sound pack must not treat that verdict as proof.

Layer test: a code-synthesized layer inside a real mix

369 clips: 184 real 4 s beds (ambience, films, music, foley ...; mean p_real alone 0.804, 0.114 of them under 0.5 alone) with one code-synthesized layer each (a cinetic / generic / retired clip: one-shots 1-4 times, 4 s windows throughout), at a layer-to-bed RMS of -12 / -6 / 0 / +6 dB (layer RMS over the span it plays). Every clip is scored by a model that never saw its bed's group nor its layer's family (--stage layers). 'base' = the shipped recipe; 'train' = also trained on the layered clips of the other groups.

layer level clips base: mean p_real base: < 0.5 base: <= lo base: >= hi (called real) train: mean p_real train: < 0.5 train: <= lo
-12 dB 93 0.673 0.269 0.172 0.538 0.591 0.344 0.204
-6 dB 91 0.621 0.33 0.121 0.429 0.536 0.418 0.132
+0 dB 94 0.633 0.298 0.149 0.479 0.524 0.457 0.17
+6 dB 91 0.622 0.286 0.143 0.396 0.512 0.484 0.154

Most layers are missed: the windows lean synthesized for only about a third of the clips, almost independent of the layer level, and many are still called real. Per bed group (base, -6..+6 dB, share < 0.5): {'ambience': 0.167, 'film': 0.167, 'foley': 0.214, 'garden': 0.133, 'keyboard': 0.133, 'magic': 0.833, 'motion': 0.682, 'music': 0.222, 'toy': 0.227} (magic and motion beds already lean synthesized on their own). Per layer family: {'cinetic': 0.375, 'generic': 0.221, 'retired': 0.295}. Training on layered clips (scheme M_layers: M plus the layered clips of the other groups) raised the held-out clip AUC to 0.9281 (acc 0.8935, Brier 0.0926; M: 0.92 / 0.8801 / 0.0984) and the layer detection above, but compressed the synthesized scores: at the same 92.5% precision target its thresholds are lo 0.05 / hi 0.68 with 0.5081 firm-verdict coverage (M: 0.7904). It was not adopted (decided after seeing these held-out numbers): judging source files firmly is the main use, and even trained, most layers stay unflagged. A dedicated layer head (CLAP logistic, bed vs bed

  • layer, same pair folds; a scratch experiment, not in results.json) reached AUC 0.82-0.84 with 40-46% of the layers at -6..+6 dB found at 5% false alarms on beds, but 11-17% false alarms on film and music beds: not shipped either. So earshot realness says plainly that a clean window does not rule out a code-made layer under a real mix.

Limits: what fools it

  • Synthesized types it misses most (scheme A acc < 0.75, lowest first; acc / mean p_real): generic:dtmf (0.0714 / 0.713), generic:siren (0.1429 / 0.67), generic:rain (0.2143 / 0.664), generic:mix (0.3611 / 0.609), cinetic:whoosh (0.7 / 0.363), generic:beep (0.7143 / 0.299), generic:chord (0.7143 / 0.335), generic:click (0.7143 / 0.289), generic:lfsr (0.7143 / 0.354). Code-made noise textures that imitate nature (rain, wind beds) and telephone-like tones (DTMF, sirens) sound like recordings of real things to CLAP, and a generic code-made mix built on such beds can score real.
  • Real groups it doubts (scheme A acc < 0.8; acc / mean p_real): magic (0.4853 / 0.451), motion (0.6667 / 0.649), music (0.6891 / 0.595). Library "magic" and "motion" sounds are themselves designed (often synthesized) by sound designers, and real electronic music sits near synthesized scores: read 'uncertain' there as "sounds designed", not "fake".
  • Finished mixes are harder than sources, and a code-made layer under a louder real mix is usually missed (layer test). Judge the source files and stems, not only the final mix.
  • Professionally made synthetic tones (UI sounds, chimes, alarms) are called code-synthesized: it hears synthesis, not provenance (Windows UI sounds above).
  • A -70 dBFS noise floor added to synthesized audio lowers accuracy (robustness table): the detector mostly hears what a sound is, not microphone physics. Very short clips (< 0.2 s) of real sounds are harder.
  • Training data: one sound library (Freesound / Mixkit picks for one film), 7 launch films, ~120 music windows; the negatives come from three code bases (cinetic, the retired score made with it, generic numpy generators). Synthesis from other tools (pro soft synths, granular, sample-based) was never seen: pro-produced synth music is meant to score real, and amateur code using techniques outside these three code bases may also score real.

Reproduce

.venv/Scripts/python.exe eval/realness/build_data.py                                   # clips -> out/realness/
.venv/Scripts/python.exe eval/realness/build_data.py --only layers                     # layer-test clips
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/extract.py --ast 240           # repeat until "done"
OMP_NUM_THREADS=2 EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage cv   # until "cv done"
OMP_NUM_THREADS=2 EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage layers   # twice
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage apps
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/train.py
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage level     # needs the final model
.venv/Scripts/python.exe eval/realness/run_eval.py --stage report                       # results.json + RESULTS.md
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/train.py                       # embeds the summary in the model