Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.
Validation
Every number here comes from a command in this repo, run on a Ryzen 7 260 laptop CPU (bf16, EARSHOT_THREADS 2-3,
other jobs running). Full tables, data descriptions and "what was tuned on what" live in each area's results file;
this page is the summary. Data: the Clawd film's licensed recordings (219 used + 125 rejected Freesound/Mixkit files),
its finished 60 s soundtrack with every placement logged, the code-synthesized score that project retired, renders
of the Cinetic synth engine, independent numpy generators, and 7 professional launch films (measured locally, never
committed).
1. Matching descriptions, ranking, finding moments (eval/semantics/RESULTS.md)
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/semantics/run_eval.py analyze (tuning library) and ... heldout.
- Calibration (shipped: 5 prompt templates, term-bias removal + per-sound z): pooled AUC of (recording, description) pairs 0.966 on the tuning library (raw cosine 0.951). Tuning-library numbers are optimistic: 67 of the 322 vocabulary terms were added after seeing its classes.
- Held-out library (125 files labelled from file names before any scoring, nothing tuned on it), per verdict band:
| verdict | z | true : wrong (lenient) | precision at a 3% base rate | at even odds | strict (close variant = wrong) |
|---|---|---|---|---|---|
| strong | >= 3 | ~1300 : 1 (126 true vs 3 wrong pairs) | 0.98 | 1.00 | 51 : 1 (precision 0.61 at 3%) |
| plausible | 2.5-3 | 21 : 1 | 0.41 | 0.95 | 9.1 : 1 |
| weak | 2-2.5 | 12 : 1 | 0.28 | 0.92 | 6.6 : 1 |
| no match | < 2 | 0.25 : 1 | 0.008 | 0.20 | 0.25 : 1 |
Half of the true matches reach "strong" (recall 0.51), 76% reach at least "weak". The strong band's ratio rests on
only 3 wrong pairs: with that count's 95% Poisson range (0.6-8.8) it lies anywhere from 440:1 to ~6000:1, and the
tuning library gave 170:1. In a library search (3% of candidate pairs true) a "strong" hit is the right kind of
sound 98% of the time but the exact variant only 61% of the time (an Enter key for a spacebar counts as wrong).
- Concrete needs, tuning library only (24 queries written from its manifests, such as "a deep thocky keyboard press", "a glass ting", "rain on a window"; optimistic, never scored on the held-out library): a correct file in the top 3 for 23/24 (miss: "a cork pop", mouth pops rank first), rank 1 for 19/24.
find: on a recording of 17 single key presses, 16 usable trims on 16 distinct presses (the 17th is a false positive on a double hit the curator marked to skip), each starting 5-54 ms before the attack, never after; 9/10 on a 32-press board; in a 44 s scene of 8 real one-shots the right event is top-1 for 8/8 (7 tight trims, one 5 s window); in a 300 s scene every planted event is found as a tight trim (the first build never looked past ~95 s; fixed with gap-free windows and a two-stage search). Re-run after the final review's find changes (snapped windows for one-shot queries, no events across edit gaps,--kind event= onset-based trims only): the same numbers.- Speed (
eval/semantics/timing.json, models already loaded, nothing cached): model load 27 s once per process;describeof a 0.56 s key 17 s;match, new query on a cached file 2.0 s;findon an 18 s file 29.9 s for a new file, 1.2 s for a new query;rank2.8 s per new file. Every CLI call is a new process and pays the model load, so for more than a few calls use the MCP server or one Python script (from earshot import api).
2. Real vs code-synthesized (eval/realness/RESULTS.md)
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/realness/run_eval.py --stage report (data and training scripts in
eval/realness/; train.py reproduces the shipped weights bit for bit).
- 1484 clean clips (691 real / pro, 793 synthesized), grouped cross-validation that never tests on a trained group:
| scheme | ROC-AUC | [email protected] |
|---|---|---|
| leave one real category / synth family out (headline) | 0.922 | 0.860 |
| leave one synth generator type out | 0.958 | 0.902 |
| retired score never trained on: real vs retired | 0.948 | 0.862 |
| retired score + the whole engine that made it never trained on: real vs retired | 0.909 | 0.839 |
| zero-shot CLAP prompts (no training, baseline) | 0.858 | 0.749 |
- Verdict bands (held-out): "real or pro-produced" p_real >= 0.77 and "likely code-synthesized" <= 0.17 are each right ~92% of the time on a pool that is 47% real / 53% synthesized, with both thresholds chosen on these same held-out predictions (so the 92% is in-sample for that choice); 79% of clips get a firm verdict, the rest "uncertain". The base-rate-free error rates: 6.7% of real clips are called synthesized, 5.5% of synthesized clips are called real. Long files use file-level thresholds (lo 0.40) on the mean of their 4 s windows, read from only 23 synthesized long files (weak calibration; the tool says so).
- Pro-produced music held out by provider: mean p_real 0.86, 44 of 52 files called real, none called synthesized.
- A deterministic rule catches pure, dead-steady test tones: a 30 s 440 Hz sine gets 0.48 from the model alone and 0.03
with the rule; the rule fired on none of 4741 audible clips of the 345 real recordings
(
eval/realness/tone_guard.py->tone_guard.json).
3. Listening through a soundtrack (eval/timeline/RESULTS.md)
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/timeline/run_eval.py (ground truth: the Clawd mixer's placement log).
- Event detection vs the 147 placed sound effects: recall 93.2% (96.4% of those louder than the bed, 89.1% of those under it; 64.7% of the ones more than 6 dB under the music).
- Sync against the picture cue frames: for the 140 of the 153 cues with an attack-aligned placement, measured offsets within 5 ms of the true placement offset for 84%, within one frame for 96%. The "without a hit" list is the weak part: both cues it named on Clawd were sounded whooshes. Beat grid of the 120 BPM score found at 120.0 BPM.
- Event tags are hints: right kind for about 1 in 2 exposed events and 1 in 4 events buried in a bed; whooshes 56%, keyboard presses 58-82%, grass footsteps under music 0%.
- Time: 60 s soundtrack 81 s, 93 s film 93 s (detail normal, 2 threads, 2-4 other processes busy).
4. Launch scorecard (eval/launch/RESULTS.md)
EARSHOT_THREADS=2 .venv/Scripts/python.exe eval/launch/run_eval.py
- Leave-one-out: each of the 7 reference films vs a profile of the other six: 120/140 reference-range metric checks in
range (86%; the no-overfit baseline for min-max ranges of six is ~71%). Not a blind test: some rules were changed
after the first leave-one-out run (listed in the results file).
critiqueon one of these 7 films now detects it and scores it leave-one-out automatically. - The Clawd film: with its real-recordings soundtrack "close" (21/23 metrics, p_real 0.94; flags a quiet first second and one loud stretch), with the old code-synthesized soundtrack "needs work" (critical: p_real 0.33, the code-synthesized windows listed; major: fades away instead of ending on a hit, sparse).
- Known-bad variants written before the run (quiet, hot, mono, muffled, late start, dead air, long fade, hole, squashed, SFX-only, ambience-only, synth bed, sine, pink noise) and how each is flagged: in the results file.
5. Interfaces
.venv/Scripts/python.exe -m pytest tests/test_interfaces.py -q (+ --slow)
- All 13 MCP tools exercised over a real stdio connection on real files (silence, clicks, mono/stereo, ogg, mp3, the 60 s soundtrack, an mp4 film); bad input returns readable errors, never a crash; every JSON part valid.
- Models stay loaded between MCP calls (the stdio handshake takes a few seconds; not benchmarked).
- The Gemini critique has never been called live (no key on this machine, and a live call uploads audio): every path is tested against a fake HTTP layer, including the key never being printed and uploads deleted after use.
6. The real-sound skill reproduces the Clawd soundtrack exactly
.venv/Scripts/python.exe -I skills/real-sound/scripts/mix.py --root "<film project>" --design audio/sounddesign.json \
--cues out/cues_all.json --out out/repro/soundtrack.wav --stems out/repro/stems --report out/repro/mix_report.json \
--cues-out out/repro/cues_sounded.json --cache out/repro/cache
Bit-identical to the shipped out/audio/soundtrack.wav (sha1 b48c5e8872fe96765624f88db18a0b49a0a05d68), with an empty
decode cache, and mix_report.json byte-identical. (--cache must point to a writable folder.)
7. Tests
.venv/Scripts/python.exe -m pytest -q (fast; synthetic fixtures, no network) and --slow (loads the models / long
end-to-end runs). At release: 128 passed, 14 skipped (slow) in 65 s; the 14 slow tests also pass (3 min).