Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.
earshot: design notes
earshot gives coding agents "ears": an agent cannot hear, so earshot turns sound into things an agent can reason about: calibrated text-audio match scores, the best moment inside a long recording, similarity to real reference recordings, real-vs-synthesized judgement, a second-by-second listening transcript of a soundtrack, an annotated picture of it (agents can look at images), a launch-video scorecard against professional references, and (only with the user's own key) an audio-LLM critique. Everything runs on CPU; no GPU is ever used.
The repo also ships skills/real-sound/: the agent skill for building premium soundtracks for code-made films from
real licensed recordings (sourcing + licence audit + design JSON + mixer + mastering + credits), which uses earshot to
pick and verify sounds.
Principles
- Honest numbers. Every score says what it is (cosine, z vs background, probability) and how much to trust it.
Never present CLAP cosine as a probability. Validation results live in
docs/VALIDATION.mdwith the exact commands. - Agent-readable first. Every command prints a short plain-language
summaryfirst, then details;--jsonfor the full dict. Timestamps as seconds (3 decimals) and m:ss.ss in text. Levels in dBFS / LUFS / dBTP. - Deterministic and cached. No random crops; fixed windows; embeddings cached on disk (EARSHOT_HOME/emb).
- Gentle on the machine. CPU only, bf16 by default,
EARSHOT_THREADS(default min(4, cores/2)). Batch work. This laptop (Ryzen 7 260, 8 cores / 16 threads) is shared with other jobs: when developing setEARSHOT_THREADS=3, keep any single job under ~10 minutes, run jobs in the foreground, one heavy job at a time per agent. - No media in git. Never commit audio/video (reference films, downloaded recordings, renders). Derived numbers (profiles, trained weights, eval tables) are fine.
- No synthesized deliverables. earshot may synthesize test signals for tests/evaluation only.
Layout and ownership (one owner per file during the build)
| file | owner | purpose |
|---|---|---|
earshot/config.py, audio.py, features.py, video.py, report.py, cli.py, doctor.py, __init__.py, models/clap.py, models/ast.py |
core (lead) | shared base: read-only for builders. Need a change? Work around it in your module and report it. |
earshot/data/vocab.json, earshot/vocab.py, describe.py, match.py, search.py, compare.py |
semantics | describe / match / rank / find / compare / ab |
earshot/timeline.py, earshot/sheet.py |
timeline | listening transcript, sync analysis, annotated PNG |
earshot/realness.py, earshot/data/realness*.json, eval/realness/* |
realness | real vs synthesized detector + its evaluation |
earshot/profile.py, earshot/critique.py, earshot/data/launch_profile.json, docs/launch-video-sound.md |
launch | reference profile from pro launch films + scorecard |
earshot/llm.py, earshot/mcp_server.py, skills/earshot/SKILL.md |
interfaces | optional audio-LLM critique, MCP server, agent skill for using earshot |
skills/real-sound/**, docs/INTEGRATION.md |
real-sound | the soundtrack skill (moved from the Clawd project) + Cinetic integration notes |
tests/test_<module>.py |
that module's owner | fast tests (synthetic fixtures, no network). Model tests marked @pytest.mark.slow. |
README.md, docs/VALIDATION.md, docs/LIMITS.md |
lead (integration) | builders send their numbers in their final report |
Core API (already built; use it)
earshot.audio:load(path, sr=48000, mono=False, start=None, end=None)-> float32 (n,2)|(n,), any ffmpeg-readable file incl. the audio of mp4;probe(path);windows(x, win, hop);list_media(paths);resample;to_mono;file_id(path);SR = 48000.earshot.features:measure(path_or_array, music=False, light=False)-> dict (see module doc: lufs, true peak, noise floor, digital_silence_frac, attack_ms, decay_ms, centroid, bands_db, flatness, tonal_peak_ratio_db, harmonic_ratio, f0, onsets, tempo/beats/key);loudness_curve(x, win, hop);integrated_lufs;lra;onsets.earshot.models.clap.get()->Clap:embed_text(list)(n,512) unit;embed_audio(list_of_arrays, pad= 'repeatpad'|'zero', per_window=False);embed_file(path, start, end, pad)(cached);embed_files(paths);logit_scale(~38.7). ~0.3-1.2 s per 10 s window on 4 threads (bf16). Every input costs one 10 s window (short sounds are padded), so batch and avoid needless tiny windows.earshot.models.ast.get()->AST:probs(list_of_arrays, reduce='max'|'mean')(n,527) sigmoid;probs_file;top(p, k);labels;prob_of(p, label). AudioSet ontology labels. Hears up to 8 kHz (16 kHz input).earshot.video:cuts(path, threshold=0.3),frame(path, t, width),frames,info,has_video.earshot.report:emit(result, args),fmt_t,to_json.earshot.cli.common(parser)adds--json --out.
Public functions (the contract other modules and the MCP server rely on)
All return JSON-serialisable dicts with a summary string. Paths may be audio or video files.
describe(path, start=None, end=None, top=8)-> tags per vocab category with calibrated scores, AST top labels, key measurements, one-paragraphsummary.match(path, query, alts=None, start=None, end=None)->{'query', 'cosine', 'z', 'percentile', 'rank_in_vocab', 'p_vs_alts' (softmax over query+alts with logit_scale), 'verdict' ('strong'|'plausible'|'weak'|'no match'), ...}.rank(paths, query, top=None)-> files sorted by calibrated match.find(path, query, win=None, hop=None, top=5, start=None, end=None, kind='auto')-> best segments inside a long recording[{'start','end','score','z','kind', 'head_db','tail_db','cut'}]: sample trims, except one marked 'head cut' / 'tail cut'.compare(path, refs, k=5, start=None, end=None)-> nearest references + typicality vs the reference set;ab(a, b, query=None, refs=None, k=5, a_start=None, a_end=None, b_start=None, b_end=None)-> which is closer to the query / refs and how they differ (feature deltas in words); the trims judge only part of a file.realness(path)->{'p_real', 'verdict', 'evidence': [...]}(+ per-segment for long files).timeline(path, video=None, cues=None, ...)-> events (time, level, tags, AST labels), windows tags, loudness curve, cuts, sync stats (event-to-cut / event-to-cue offsets),text= the listening transcript.sheet(path, out_png, video=None, ...)-> writes an annotated PNG an agent can open; returns its path + summary.critique(path, profile=None, video=None)-> scorecard: each metric, the soundtrack's value, the reference range, status (ok/low/high), what to do.profileCLI builds a profile JSON from a folder of reference videos/soundtracks.llm.critique(path, prompt=None, provider='gemini')-> only when GEMINI_API_KEY / GOOGLE_API_KEY is set and the caller passes consent (--yes-upload); otherwise a clear message how to enable it. Never hardcode keys.
CLI: each module implements add_cli(sub) registering its subcommand(s) (names in earshot/cli.py COMMANDS; search
registers rank and find; compare registers compare and ab). Use from .cli import common for --json/--out.
Evaluation data (not redistributed)
Evaluation data: licensed recordings and third-party films, not redistributed. The evaluation scripts in eval/
read their locations from environment variables or a gitignored eval/local_paths.json (see eval/README.md).
What was used:
- A library of real licensed recordings (Freesound / Mixkit) in 8 categories (ambience, foley, garden, keyboard, magic, motion, music, toy): 219 files used for tuning, plus 126 real recordings rejected for quality, used as the held-out set. Labels come from per-file manifests (category, title, need, notes).
- Code-synthesized audio: a retired synthesized film score with its stems, and one-shots rendered by an open synth engine (kick, sub_boom, clap, hat, tick, tock, key_click, thock, ui_click, snap, blip, fm_bell, whoosh, riser, swell, pluck, bass, drone, keys ...).
- A finished 60 s real-recordings film soundtrack with ground truth: stems (sfx, amb, music), every placement with time and sample in a mix report, picture cue events at 60 fps, and the same film with the old synthesized soundtrack.
- 7 professional launch films as the quality bar (measured only, never copied), referred to as ref_1..ref_7 with
neutral style descriptions in
eval/launch/refs.json.
Environment
- Python 3.10+ with torch (CPU), transformers, librosa, scikit-learn, mcp. Run from the repo root (package importable
as
earshot), e.g..venv/bin/python -m earshot describe file.wav. - Models are cached in
.cache/hfinside the repo orEARSHOT_HOME(CLAP laion/larger_clap_general, AST MIT/ast-finetuned-audioset). - Scratch output:
out/<area>/(gitignored). - ffmpeg / ffprobe on PATH.