Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.
Integration: earshot inside a Cinetic film
Cinetic (~/.claude/skills/cinetic) directs and renders launch films from code. Its Step 5 builds the soundtrack
(scripts/audio/score.py) from a beat-locked timeline. Two routes give that soundtrack better sounds: an earshot
sound kit (the first section: forge sounds rendered from earshot's bank, the route cinetic 1.1 takes by default when
earshot is installed), and a real-recordings soundtrack from skills/real-sound (sections 1-4, which use Cinetic
exactly as it was before 1.1). Either way earshot gives Cinetic's review loop ears (section 3).
cinetic uses earshot
The effects of a cinetic film come from an earshot kit; cinetic keeps the conductor (timing, typing plan, levels, ducks, arc, master, report, stems) and earshot supplies the voices. Rendering a kit is pure DSP: no models, no torch import, seconds per film once cached.
src/sync.ts --export-cues--> out/cues.json ---------------------------------+
audio/score.json "sfx": {"engine": "auto", "kit": "audio/kit.json"} ------+--> score.py --> master.py --> soundtrack.wav
audio/kit.json --scripts/audio/kit.py--> earshot kit render --> audio/kit/*.wav + kit.lock.json --^
- The kit.
audio/kit.jsonstarts from a preset (earshot kit init --preset calm|crisp|minimal|launch; calm is the default for text-led films, launch for films cut to music) and is edited per film: the lockup hit tuned to the film's root (edits.pitch), another member for a role, a fitted sound. Roles are cinetic's cue kinds and their variants (below);nullroles are not sounded. - The bridge. cinetic's
scripts/audio/kit.pyfinds earshot (importable in cinetic's Python,sfx.pythoninscore.json,EARSHOT_PYTHON, orearshoton PATH), runsearshot kit render audio/kit.json --out audio/kitin one process (cached: an unchanged kit re-renders nothing) and keeps the takes inaudio/kit/, so the film rebuilds on a machine without earshot. - Placement. score.py plays one take per event (a new take per event by hash; typing cycles through the takes),
placed by the sync point the lock measured for the role's
align: the take's onset on the cue frame, a whoosh's loudest 5 ms on its apex, a riser's end on the hit (or its climax, for a riser or swell aligned on itspeakas the launch preset's are: their 0.4 members fall after the climax).level_dband the event weight set its level; a cue whose role isnullis skipped with a report line. A role with the 0.4 mix keys (the launch preset) asks for more: itsoffset_msmoves the sync point off the frame (kit.place), itsbed_dbsets the take against the music under the cue (kit.bed_level+kit.gain_db, instead oflevel_db), itsmin_gap_msspaces a run (kit.space_run), and the lock'smix.stop_downasks for a music dip before hits rather than a duck after them (docs/SFX.md). A renderer that ignores them still gets valid takes andlevel_db. - Fallback. earshot render -> the vendored
audio/kit/takes -> cinetic's own synth voices with a loud warning. The build never fails only because earshot is missing. - Listening.
scripts/audio/listen.pyrunsearshot timeline --cues,sheetandcritiquein one process and writesout/qa/listen.json. - The lockup is a stop-down and one clean element.
earshot kit checkwarns when the hit role has two attacks or a build before the hit, and when any role's pitch climbs (the rising synth whip); score.py's stack guard warns on a second designed effect within 3 frames of the lockup or payoff.
| cinetic cue (kind, variant) | kit role | align |
|---|---|---|
| tick / tick crisp | tick / tick.crisp |
onset |
| tock | tock |
onset |
| hit | hit |
onset |
| land / land light | land / land.light |
onset |
| pop | pop |
onset |
| whoosh (short / long move) | whoosh.short / whoosh.long |
peak |
| suck, riser, swell | suck, riser, swell (off in calm, crisp, minimal; launch sounds riser and swell, both aligned on the peak) |
end |
| drop | drop |
onset |
| key / key space / key return | key / key.space / key.return |
onset |
| click / click confirm | click / click.confirm |
onset |
| snap | snap |
onset |
| bell (motif notes, progress steps) | bell (one take per pitch) |
onset |
| a text fragment resolving | text |
onset |
What earshot guarantees the bridge: the role vocabulary, kit and lock schema 1 (docs/SFX.md#kits), the CLI flags above, deterministic takes, and a lock that names the bank version (a new bank version re-renders the kit, because it is part of every take's cache key). The bank, knobs, audit and presets are documented in docs/SFX.md.
1. What Cinetic produces that the soundtrack needs
| Cinetic output | content | used by real-sound / earshot as |
|---|---|---|
out/cues.json (npx tsx scripts/export-cues.ts) |
{fps, bpm, total, acts, cue, events: [{f, kind, weight, pan, pitch?, apexFrac?, dur?, variant?, id?}]}, sync frames computed from the picture code |
the cue file, unchanged: mix.py --cues out/cues.json (mix.py's cue format is this format). Rules match kind (regex), id (regex), variant, a time window; weight drives weight_db; pan drives pan: "cue" |
src/timeline.ts (FPS, BPM, TOTAL, ACT, CUE) |
the grid | TOTAL / FPS = the design's duration; BPM = arrange.py --film-bpm; ACT (exported as acts) = qa.py's per-act loudness |
audio/score.json |
key, bpm, one chord per bar, sections (intro / build / groove / drive / breakdown / resolve / hold), drops + silences, payoff, signature tick/tock/motif, typing mode, mix, master.lufs |
the musical intent: sections -> arc.json for arrange.py; payoff / drops -> drop_film_bar; key and chords -> which tuned recordings to source and whether a splice keeps the harmony; master -> the design's master |
TREATMENT.md / BRIEF.md |
the brand's three adjectives, the personality row (references/sound.md §5.5a), the beat sheet with a sound column |
the SOUND_BRIEF's concept and picture map |
public/audio/soundtrack.wav (written by npm run audio) |
the master Cinetic's render.sh --audio muxes |
replaced by mix.py's out/audio/soundtrack.wav, passed to render.sh --audio (render.sh takes any WAV; keeping it out of public/ keeps the premaster and stems out of the Remotion bundle) |
out/stems/ |
stems for av-audit.py --stems |
mix.py --stems out/stems writes sfx.wav, amb.wav, music.wav: av-audit picks sfx*.wav as the effects and music.wav as the music bus |
score.json sections -> an arrange.py arc
One number per film bar (dB relative to the track's loudest sections): intro -14 (rising to -10), build -10 -> -4
across its bars, groove -6, drive -2..0, breakdown -10, resolve -8 -> -14 over its bars, hold / tail -20.
payoff (usually the drop's bar) becomes drop_film_bar; a drop's silence becomes a deliberate hole in the music
edit (the segment before the drop ends silence early; qa.py lists it as a note, not a defect).
Synth voices -> real recordings
| Cinetic kind | synthesized today | real-sound replacement (category / source / rule) |
|---|---|---|
| music stems: drums, bass, pad, arp, bells, reverb | generated per chord and section | ONE licensed track edited at bar lines (arrange.py draft, music supervisor); its own drop lands on payoff |
tick / tock (signature clock) |
tuned wooden mallet | a real wood block / soft mechanical switch, two takes a fourth apart, resampled within +-2 st to the key |
key (typing) |
dark modal tock per key, thinned to ~12/s | slices of one real keyboard recording (slices + mode: cycle); variant: "space"/"return" -> a deeper spacebar / enter take; per-word style -> a rule on variant: "word" only |
click |
bright 2 kHz voice + release | a real mouse / trackpad press + release (two rules, offset ~0.085) |
pop |
pitched blip | a real cork / mouth pop / plastic tick, pitch_ramp up across a run |
snap |
click + body on the chord root | a real latch / Lego / magnet clack, layered with a low body |
land |
wooden knock + sub | real foley thumps by weight (weight_db), light taps for variant: "light" |
hit |
sub boom + clap + burst | a real drum / taiko / kick hit + a short top transient (two rules) |
drop |
long sub boom | the music's own drop (edit), optionally a real low impact under it |
whoosh |
noise band sweep | a real swish (bamboo, stick, cloth), align: "peak" |
riser / swell |
noise + saw glide / reverse chord | a real cymbal roll or orchestral swell, align: "end" (or the track's own build) |
suck |
reverse whoosh | a real air-suck recording, or nothing (mix.py does not reverse recordings) |
bell (variant: "motif") |
FM bell / the motif | a real celesta / bell / glass note in the key (one sample per note; the Clawd film built its motif this way) |
Cinetic's sound rules stay valid (the sound map, levels, the drop on the hero reveal, the resolved logo, restraint);
skills/real-sound/references/sound-design.md restates them for recordings.
2. The bridge, step by step (Cinetic untouched)
Inside a Cinetic film project (films/<name>/), with <rs> = skills/real-sound of this repo:
- Cinetic Steps 0-4 as usual. In
BRIEF.mdwrite the spec line withaudio: real recordings (real-sound). - Cues:
npx tsx scripts/export-cues.ts->out/cues.json. - Brief: copy
<rs>/templates/SOUND_BRIEF.mdtoaudio/SOUND_BRIEF.md; take the picture map from the treatment's beat sheet (its sound column) andout/cues.json; take the personality row and the arc fromaudio/score.json(write it even though score.py will not run: it is the musical intent). - Sourcing, licence audit, music edit, design fragments, critique and fix: the real-sound process
(
<rs>/SKILL.md, prompts in<rs>/templates/prompts/), with every path relative to the film project. - Render the soundtrack:
Set the design'spython -I <rs>/scripts/mix.py --root . --design audio/sounddesign.json --cues out/cues.json \ --out out/audio/soundtrack.wav --stems out/stems --report out/qa/mix_report.json --trace out/qa/adds.jsonmaster.lufstoaudio/score.json'smaster.lufs(-14 for launch films). Do not runnpm run audiofor this film (it writes the synthesized score). - Cinetic Step 6 with the real soundtrack:
bash scripts/render.sh Film out/preview.mp4 --preview --audio out/audio/soundtrack.wav(and--blurfor the master; never--build-audio, which re-runs score.py). Itscheck-sync.pygates lag and true peak on the decoded MP4 as before:python3 scripts/check-sync.py out/film.mp4 out/audio/soundtrack.wav. - Cinetic Step 7 (review loop) with ears (section 3).
- Credits:
audit.py --report out/qa/mix_report.jsonandcredits.py; putout/CREDITS_description.txtinto the delivery's README / video description.
real-sound's own mux.mjs is not needed in a Cinetic film: render.sh muxes. Use mux.mjs for films rendered
outside Cinetic.
3. Where earshot slots into Cinetic's self-critique loop
Cinetic's loop (§8): render -> measure (probe, forensics, av-audit) -> sheets -> critic lenses -> verify -> fix. earshot adds hearing to the measure and lens steps:
| Cinetic step | add | what it gives the critics |
|---|---|---|
| Step 5 gate (after the soundtrack is written) | python -I <rs>/scripts/qa.py --layers --stems out/stems |
clicks / cuts with the guilty sample, masked events, harsh 2-6 kHz peaks, chopped music splices, sync per event, partial coverage, loudness and LRA |
| Step 7 measure | python -m earshot timeline out/preview.mp4 --cues out/qa/cues_sounded.json |
a second-by-second listening transcript of the decoded film: events with tags, sections, loudness curve, beat grid, sound-to-cut and sound-to-cue offsets |
| Step 7 measure | python -m earshot sheet out/audio/soundtrack.wav -o out/qa/sound_sheet.png --video out/preview.mp4 --cues out/qa/cues_sounded.json |
one PNG (frames, sections, tagged events, spectrogram, loudness, cue sync) the director and sound lenses look at next to the contact sheets |
| Step 7 measure | python -m earshot critique out/audio/soundtrack.wav --video out/preview.mp4 |
the scorecard against the profile of professional launch films (metric, value, reference range, what to do) |
| av-audit | python3 scripts/av-audit.py out/preview.mp4 --cues out/qa/cues_sounded.json --stems out/stems |
Cinetic's own onset/sync audit on the real soundtrack; cues_sounded.json (written by mix.py) lists only events that received a sound, so deliberate silences are not counted as misses |
sound-sync lens (assets/critics/sound-sync.md) |
hand the lens qa.json, the earshot transcript, sheet and scorecard with av.json |
the lens can name the exact rule / sample to change; with real recordings its fixes are design operations (<rs>/templates/prompts/fixer.md), not score.py parameters |
verifier (assets/critics/verifier.md) |
refute a sound issue with earshot evidence (earshot match on the placed sample, the transcript at that time) |
fewer false P1s from reading code alone |
| a synthesized score already exists | python -m earshot realness public/audio/soundtrack.wav (score.py's output) |
shows (per 4 s window) where the synthesized score reads as code-made: useful when the brief asks for recorded sound |
4. A minimal change list for a future Cinetic
Each item is small and keeps the synthesized path as the default.
- One switch in
audio/score.json:"engine": "synth" | "real"and"design": "audio/sounddesign.json". Withreal,npm run audioruns real-sound'smix.pywith the same--cues/--stems,--outoutsidepublic/(render.sh's--audiodefault and--build-audiofollow it) and--jsonmapped to--report;score.pystays forsynth. export-cues.ts --thin-typing: applyplan_typing()'s thinning (<= 12 keys/s, per-word above 25 chars/s) at export, marking skipped keysvariant: "silent", so a recordings design sounds exactly the keys score.py would.av-audit.py: whenengineisreal, readout/qa/cues_sounded.jsonand skip the score.py typing replay.references/sound.md: a §1b "Real recordings" pointing at real-sound, and a "real palette" column in the personality table (calm: felt piano, soft wood; fast: tight drum hits, metal latches; bright: marimba, claps, cork pops; precise: sparse piano, glass; raw: switches, latches; warm: plucked strings, soft taps).assets/critics/sound-sync.md: withengine: real, replace "Never propose stock or sampled sounds" by "propose recordings from the manifests or a new sourcing need; never synthesized sounds", and readqa.jsonand the earshot transcript / scorecard.references/review-loop.mdMeasure step: addqa.py --layersandearshot timeline / sheet / critiqueto the scripts that writeout/qa/.scripts/new-film.sh --audio real: scaffoldaudio/SOUND_BRIEF.md,audio/sounddesign.json,audio/design/andaudio/assets/manifests/from real-sound's templates, and writeaudio: realin the BRIEF spec line.- One
master.py: real-sound vendors Cinetic'smaster.pybyte for byte (sha1 2af9044a923cdbe01a81bb1d7922b97d41641479 in both today); keep a single source when the two ship together.