Skip to content

Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.

docs: earshot 0.4.0

Leonxlnx426f4e726624
diff --git a/CHANGELOG.md b/CHANGELOG.md
index fef0625..2a83be5 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -1,5 +1,156 @@
# Changelog
+## 0.4.0 - 2026-10-22
+
+Sound for music-led launch films: 51 new recipe-made sounds in the bank (seven new families), a `launch` kit preset
+mixed against the music bed instead of at fixed levels, and the fit engine re-made and measured with a replication
+benchmark on a private set of licensed reference clips (not in the repo: no audio, no clip name and no per-clip
+model ships; aggregate measurements of them (medians, IQRs), CLAP cosines to them and parameters tuned toward them
+do). Tuned on a fixed 78-clip dev split; the 38 held-out clips were opened once, at the end of the first round, and a
+fresh 28-clip held-out set, frozen before any further tuning, at the end of round C2. No new dependencies.
+
+### Added
+
+- **Bank 0.4** (`earshot/forge/bank/forge04.json`, `forge04r1`; `bank.VERSION` is now
+ `forge_v2r6+forge04r1+fit1+pluck1`, 252 sounds): 51 recipe-made sounds, built from the recipe palettes by
+ `python -m earshot.forge.bank build --forge04`. New categories: `cut_hit` (sub, kick, punch, rumble, air),
+ `reverse_swell` (cymbal, ring, dark, short, breath), `ui_pop` (soft, step, boop, blip, fall, wide), `knock` (wood,
+ plastic, snap, desk, double), `mallet` (glock, pluck, celesta, marimba; chime runs flourish, up4), `drip` (drop,
+ bubble, plink, bloop, tiny), `shutter` (dslr, snap, mirror, film, quick). New members of old categories: whoosh_big
+ thump / thump_deep / thump_airy, riser dark / build / snatch, swish bright / flick, whoosh_fast whip, glitch burst /
+ mid / chirp, notify mid_tine / mid_wood. Every entry carries, besides its render block, meta and fingerprint: the
+ catalog profile (`profile`: length, attack, t20 / t40, centroid, low / high shares, tonal share, glide, L/R
+ correlation), its take audit in its kit role (`audit`: seeds n..n+5; 50 members 6 of 6 clean, glitch/mid 5 of 6)
+ and its originality (`original.max_ref_clap`: the largest CLAP cosine of its render to any of the reference clips,
+ three excerpt conventions, 0.53-0.84; the bar is 0.96). The build refuses a member whose own take is not clean,
+ with fewer than 5 of 6 clean takes, or without a measured originality under the bar; a recorded value holds only
+ for the render it was measured on (`bank.FORGE04_REF_ON`: the member's render block, and the shipped fingerprint),
+ so a member changed since needs a fresh one (`build --forge04 --ref-clap`).
+- **Recipes** (originals made from recipes, none copied from the reference clips): `ui_pop`, `knock`, `mallet` (with
+ chime runs), `drip`, `shutter` (shared parts in `recipes/_tonal.py`), notify's mid dings (`notify.FAMILY_MID`;
+ `ring` / `soft` / `bell_len` keys, off by default: earlier members render sample for sample); `cuthit`, `thump`,
+ `revcymbal`, `darkriser`, `brightswish`, `burstglitch`. Eight members came last, each found by a model-free search
+ from a hand-set corner toward a designed aim with every candidate's takes audited (drip bloop / tiny, shutter film /
+ quick, mallet celesta / marimba, reverse_swell short / breath).
+- **The launch preset** (`earshot kit init --preset launch`): music-led launch films. The 0.4 members where they fit
+ (ui_pop/soft for the pop, knock/wood, cut_hit/punch and kick, whoosh_big/thump, riser/dark, reverse_swell/dark,
+ notify/mid_tine, shutter/dslr, a wide gated mallet/glock bell and a soft mallet/pluck text), every role placed
+ against the music bed. riser and swell are sounded (at most one or two a film, at a section change), suck is off;
+ both are aligned on their peak (the 0.4 members fall after their climax: aligned on the end, the dark riser's
+ climax would come 0.27 s before the cut), and `kit check` notes an end-aligned role whose sound rings on more than
+ 30 ms past its peak. Its 72 takes are clean in `kit check`.
+- **Kit mix keys** (optional; a kit without them plans, keys and locks its takes as before): `bed_db` (the mix's
+ loudest 50 ms at the effect, K-weighted, over the music's median 50 ms level in the second before the cue; the takes
+ of such a role carry `l50_lufs`), `offset_ms` (the sync point against the picture: taps, knocks and clicks lead,
+ pop-in thumps and whoosh and swell peaks land after), `min_gap_ms` (run spacing), `width` (the L / R correlation
+ every take is set to by mid / side, measured into the lock as `corr`), and a kit-level `mix` block (the stop-down:
+ the music steps down before a hit and comes back on it; no ducking). Helpers for a mixer: `kit.bed_level(music, t)`
+ (None where the music is silent, under -60 LUFS: `gain_db` then falls back to `level_db`; a mono take or bed
+ measured as the dual mono it plays as), `kit.gain_db(take, role, bed_lufs)` (the take alone lands at bed + 10
+ log10(10^(bed_db/10) - 1), never over -1 dBFS at its peak), `kit.place(take, role, cue_s)`, `kit.space_run(times, min_gap_ms)`, `kit.stereo_width(y, corr)`,
+ `kit.loud_lufs(y)`. The launch preset's numbers are medians measured on the reference clips (effects 3-10 dB over
+ their bed; touches lead the cut by 11-60 ms, pop-in thumps and whoosh and swell peaks land 30-42 ms after it; runs
+ 66-159 ms apart).
+- **The audit gives a drip its rise**: `sfx.check` applies `recipes/drip.allows()` to the drip category, so a climb a
+ bubble makes (at most 14 st, starting within 60 ms, done within 250 ms) is a note, while any other rising tone is
+ still the rising-glide warning (drip/bloop needs it).
+- **The gesture model** (`earshot/forge/fit/gesture.py`, kind `gesture`): tones (the sines candidate's partials) +
+ air (a detail noise model of what the tones leave, split in the magnitude domain), with motion knobs (length,
+ sweep, doppler, proximity, pan, motion, width, roughness, tone_db / air_db, vary). Tried on every sound under 1 s
+ and second in the long gesture / mixed / glide / ringing plans; it won 38 of the 78 dev clips. `earshot fit --kind
+ gesture`; a bank fit entry's dur knob maps to its `length`.
+- **Stereo and space.** Sines and modal fits keep the recording's side over time per octave band (the part coherent
+ with the mid as a pan, the rest as the mid's quadrature): on 42 wide or decorrelated dev clips the side / mid error
+ fell from 13.4 to 0.15 dB, the tail correlation error from 0.39 to 0.04. A hybrid's sines layer keeps its side and
+ the residual the next layers are fitted on leaves it out (`fit_hybrid` takes the layer's stereo render out), so
+ the air does not model the tones' side a second time. `earshot/forge/fit/space.py`: a fitted
+ diffuse side layer for a stereo recording whose tail is decorrelated (`post['space']`). `quality` measures
+ `side_db`, `width_db`, `tail_corr`, `tail_corr_err`, and `score` weighs width and tail correlation.
+- **Beds.** A recording with a steady music bed in its pre-roll is stripped and the sound alone is fitted; the bed is
+ rendered back under it by default (`render(bed=False)`: the sound alone, which is what a bank entry's and a kit's
+ takes render: `bank add-fit` of such a fit keeps the bed out of its takes). `earshot fit --keep-bed` turns it off.
+ Analysis by synthesis decides whether it is a bed: the winning candidate's kind is also fitted on the recording as
+ it is, and the bed is dropped when that scores lower (`auto.BED_CHECK`; a whoosh's own steady approach had been
+ stripped as music).
+- **CLAP margin rule.** After the confirm stage a candidate whose CLAP is 0.02 higher on seed 0 and on the confirm
+ seeds' mean, within 1.5 of the score and 0.05 of the envelope r, is chosen instead (`auto.CLAP_MARGIN`; rows keep
+ `clap2`).
+- **Time shape.** A post-stage gate (cut-only, kept when it lowers the score and costs no CLAP) for gated pulses,
+ hard stops and merged runs (it follows the knobs that move time: a modal spacing moves each reopening with the
+ strike it opens for; a gesture length moves the gate and the attack clamp about the tones' first strike; a morph
+ warps each source's gate onto its timeline); modal noise bands may not rise before a strike (onset error 0.39 -> 0.25 octave on the
+ modal dev fits); the glide gate now fires at 2 semitones over 15 ms; the best three candidates are confirmed on two
+ more seeds.
+- **Spectral**: a detail fit (`spectral.DETAIL`, `quiet_db`), `trajectory()` and the motion knobs doppler,
+ proximity, pan and motion.
+- **Tonal knobs** (`knobs.apply(..., decay=, onset=, bright=, width=)`): t20, attack, centroid and L/R correlation
+ targets landed by measurement through one recipe parameter; pitch knobs for ui_pop, mallet, knock, drip and
+ cut_hit; a dur knob for reverse_swell.
+- **`eval/forge/replicate.py`**, the replication benchmark (targets read from a folder you pass, or a frozen split
+ file with `--split-file`; results go outside the repo or under `out/`; an explicit `PYTHONPATH` picks the engine).
+
+### Changed
+
+- Every render of a model fitted by 0.4 starts at zero (a fade-in of 2 ms, shorter when the onset is earlier) and
+ peaks at -1 dBFS at most. Models fitted before 0.4 (modal, sines and hybrid models without the `coherent` flag: the
+ 0.3 bank's fit entries; spectral models without the `finish` flag, which every 0.4 spectral fit carries) render as
+ 0.3.0 did, sample for sample (`auto.finished()`).
+- **calm, crisp and minimal sound as in 0.3.0**: rendered with the 0.3.0 code and with 0.4.0, all 191 of their takes
+ are byte-identical. The bank version is part of every take's cache key, so a kit re-renders once after the update.
+- Takes of fits made by 0.4 (models marked `coherent`): modal vary moves the modes by a smooth random function of
+ log frequency, the same in every strike, with phases kept on the first strike's timeline, and strike times no
+ longer jitter (independent per-mode detune undid the fit's cancellations: takes rang on); multi-strike modal fits
+ keep their strikes coherent under the size and spacing knobs too; sines vary moves neighbouring partials together.
+ A modal or hybrid fit's `vary_scale` 0.5 halves the bank's vary for these kinds.
+- The attack clamp of a noise model's post stage is high-passed and kept only where it helps.
+- Takes without clicks: the post gate's gain is smoothed over 1 ms, a later modal strike rises over 0.15 ms (coherent
+ models), and a coherent sines model jitters a strike's atoms together.
+- The gesture air's grain texture is levelled over 5 ms windows and kept only when it is stationary
+ (`spectral.fit(grain_norm_ms=, grain_halves=)`, `grains['norm_ms']`; older models render as before).
+- `kit check`'s restraint note says which presets keep riser, swell and suck off (launch sounds riser and swell).
+- Recipe comments no longer name a source film.
+
+### Measured (replication benchmark)
+
+Tuned on the dev split only; each held-out set opened once. Details and the regressions in
+[docs/FIT.md](docs/FIT.md#the-replication-benchmark-04). The fresh held-out set is harder and more bedded than the
+benchmark (its isolation bar is lower): compare engines on it, not its numbers with the other rows. The 0.4.0 rows were
+measured before the release review's hybrid side fix (a hybrid's residual now leaves its tone layer's stereo side
+out), which can change hybrid candidates on stereo tonal clips; they were not measured again.
+
+| split | engine | CLAP (median) | mel gap dB | high / good / fair / poor | takes pass |
+|---|---|---|---|---|---|
+| dev (78) | 0.3 | 0.884 | 1.43 | 19 / 20 / 31 / 8 | 29 |
+| dev (78) | 0.4, first round | 0.907 | 0.90 | 30 / 26 / 20 / 2 | 46 |
+| dev (78) | 0.4.0 | 0.921 | 0.85 | 40 / 20 / 17 / 1 | 42 |
+| held-out (38) | 0.3 | 0.878 | 1.73 | 7 / 8 / 19 / 4 | 21 |
+| held-out (38) | 0.4, first round | 0.909 | 0.98 | 14 / 9 / 12 / 3 | 25 |
+| held-out (38), opened | 0.4.0 | 0.903 | 1.04 | 14 / 11 / 12 / 1 | 25 |
+| fresh held-out (28) | 0.3 | 0.853 | 0.87 | 2 / 6 / 14 / 6 | 8 |
+| fresh held-out (28) | 0.4, first round | 0.830 | 0.75 | 4 / 3 / 14 / 7 | 12 |
+| fresh held-out (28) | 0.4.0 | 0.851 | 0.58 | 5 / 6 / 13 / 4 | 7 |
+
+On the fresh held-out set 0.4.0 is level with 0.3 on the median CLAP (0.851 against 0.853) and ahead on the mean
+(0.860 against 0.844), the mel gap, the grades (grade up on 7 clips, down on 2) and the take CLAP (0.863 against
+0.809); fewer takes pass the take check (7 against 8), and the onset error grew (1.00 octave against 0.53).
+
+### Not done (known gaps)
+
+- The four stereo sines fit entries (fit/click/floss_lid, fit/key.space/thock_space, fit/snap/switch_flip,
+ fit/whoosh.short/bamboo_swish) keep their 0.3 fits with the old, too quiet side: calm uses all four, crisp three
+ (not bamboo_swish), minimal switch_flip and launch floss_lid and thock_space, and a refit would change those
+ presets' takes (calm, crisp and minimal stay byte-identical to 0.3.0).
+- The bed check has no CLAP guard: on the fresh held-out set it dropped a knock's bed although the plain fit had 0.087
+ less CLAP, and a plain fit of a bedded clip keeps the bed in its sound (four of the six new take failures there).
+- Onsets: the onset error grew against 0.3 on the fresh held-out set (14 of 28 clips off by more than an octave).
+- Take clicks left: a loud grain inside an air-only key's attack and one modal-layer click in a hybrid take; a comb
+ ring in the ring-out tails of short modal clips (7-8 dev clips).
+- Multi-pulse counts on long gesture wins (7 of the 28 dev clips with 3 or more pulses miss by more than 2) and clips
+ of 0.5 s or longer (3 of 13 high on dev).
+- Takes passing fell this round (dev 46 -> 42, fresh held-out 12 -> 7) although the take CLAP rose.
+- Both held-out sets are now opened; a next round needs a new one.
+- No listening test of the 0.4 sounds: they were chosen and checked by measurement (profiles, audits, CLAP).
+
## 0.3.0 - 2026-10-19
The sound engine of a film: the forge's delivered sounds as a bank, one-sound renders aimed at the picture, a
diff --git a/README.md b/README.md
index 819a365..daf4cc8 100644
--- a/README.md
+++ b/README.md
@@ -20,7 +20,15 @@ professional launch films. It runs locally on CPU, as a CLI, a Python API and an
With ears, an agent can also **make its own sounds in code** and improve them by listening (`earshot forge`), and
**translate a recording into code** it can vary (`earshot fit`). Both below.
-**New in 0.3: sound for code-made films.** A bank of 201 sound effects stored as code that render by id in seconds,
+**New in 0.4: the sound of a music-led launch film.** 51 new sounds made in code for the soft, short, mid-register
+style those films use: rounded UI pops, wood and plastic knocks, short cut hits and sub hits with a kick, gated
+glockenspiel / celesta / marimba notes and chime runs, water drops, camera shutters, reverse cymbals, a thump-whoosh,
+short dark risers, bright swishes, burst glitches and mid dings (the bank now holds 252). A new `launch` kit preset
+mixes every effect against the music bed instead of at a fixed level, says where each one sits against the cut (taps
+lead it, whoosh peaks land after it) and how close a run may go. And `earshot fit` was re-made against professional
+sound effects and measured on held-out clips ([below](#earshot-fit-audio-to-code)).
+
+**New in 0.3: sound for code-made films.** A bank of sound effects stored as code that render by id in seconds,
with no models and no network: `earshot sfx` aims a whoosh, hit, click or note at a moment in the picture, and
`earshot kit` renders a whole film's effects with measured sync points ([below](#sound-for-films-bank-sfx-kits)).
@@ -56,8 +64,8 @@ $ earshot critique film.mp4
| `measure FILE` | objective measurements only (no models): LUFS, true peak, LRA, envelope, spectrum, tempo / key |
| `forge RECIPE --refset CAT` | make a sound in code and improve it by listening (see below) |
| `fit FILE -o recipe.py` | translate a recording into a forge recipe module that re-renders it, with knobs and morphs (see below) |
-| `sfx render ID -o x.wav` | a sound from the shipped bank of 201, aimed at the picture (`--dur`, `--apex`, `--pitch`) and audited; `sfx list`, `sfx check` (no models) |
-| `kit init / render / check` | a film's sound kit: one bank sound per role, rendered into cached takes with measured onset / peak / end (no models) |
+| `sfx render ID -o x.wav` | a sound from the shipped bank of 252, aimed at the picture (`--dur`, `--apex`, `--pitch`) and audited; `sfx list`, `sfx check` (no models) |
+| `kit init / render / check` | a film's sound kit: one bank sound per role, rendered into cached takes with measured onset / peak / end; the `launch` preset places each role against the music bed and the cut (no models) |
| `gallery DIR -o page.html` | one static HTML listening page of forge outputs: players, spectrograms, numbers |
| `llm FILE --yes-upload` | optional audio-LLM critique (Gemini), only with your own key and explicit consent |
| `mcp` | the same tools as an MCP server for Claude Code and other agents |
@@ -151,29 +159,70 @@ earshot fit --morph a.wav b.wav --amount 0.5 -o made/ab_morph.py
earshot fit --forge made/click_fit.py --budget 96 --dir made/click_variations # critic-searched variations
```
-Measured on 12 real recordings (2 each of click, key press, pop, impact, fast whoosh, notification): median CLAP
-cosine 0.945 between each recording and the render of its generated code (minimum 0.881), and the same top
-`earshot describe` tag in 10 of 12. In our 12-trial blind test (recording vs render, loudness-matched, random order)
-the listener could not tell which clip was the recording. One listener is a small test, not a proof. The renders are
-close, not identical: onsets, tone balance and noise texture still differ measurably; the selection weights were set
-on the same 12 recordings; risers, zips and long designed sounds are untested. A fit is a derivative of its recording,
-so keep the source's licence with the recipe. Limits: [docs/FIT.md](docs/FIT.md#limits-honest).
+Measured with the earlier engine (0.2-0.3) on 12 real recordings (2 each of click, key press, pop, impact, fast
+whoosh, notification): median CLAP cosine 0.945 between each recording and the render of its generated code (minimum
+0.881), and the same top `earshot describe` tag in 10 of 12. In our 12-trial blind test of that engine (recording vs
+render, loudness-matched, random order) the listener could not tell which clip was the recording. One listener is a
+small test, not a proof. Neither was re-run on 0.4.0, whose choice changed (the gesture model, the CLAP margin rule,
+the gate, the finish); its benchmark below is measurement only (CLAP and the mel gap are proxies; nobody listened).
+The renders are close, not identical: onsets, tone balance and noise texture still differ measurably; the selection
+weights were set on the same 12 recordings; zips, long glides and swipes are untested (the benchmark below covers
+risers, shimmers, swells and camera moves). A fit is a derivative of its recording, so keep the source's licence with
+the recipe. Limits: [docs/FIT.md](docs/FIT.md#limits-honest).
+
+**Re-making professional sound effects (0.4).** The replication benchmark (`eval/forge/replicate.py`) fits a private
+set of licensed reference clips (116 clean one-gesture effects in 24 categories; not in the repo: no audio, no clip
+name and no per-clip model ships; aggregate measurements of them (medians, IQRs), CLAP cosines to them and parameters
+tuned toward them do) and measures each fit and four takes against its clip. 0.4 adds a noise-gesture model (tones plus
+air), stereo width and decorrelated tails, music-bed stripping decided by analysis by synthesis, gates for pulses
+and hard stops, and takes that keep the fit's structure. It was tuned on 78 dev clips; two held-out sets were each
+opened once:
+
+| clips | earshot 0.3 | earshot 0.4.0 |
+|---|---|---|
+| dev (78): median CLAP, fits graded high | 0.884, 19 | 0.921, 40 |
+| held-out (38): median CLAP, high | 0.878, 7 | 0.909, 14 when opened (first-round engine); 0.903, 14 for 0.4.0 on the opened set |
+| fresh held-out (28, harder: more music under the effects): median / mean CLAP | 0.853 / 0.844 | 0.851 / 0.860 |
+| fresh held-out: mel gap to the clip, grades high / good / fair / poor | 0.87 dB, 2 / 6 / 14 / 6 | 0.58 dB, 5 / 6 / 13 / 4 |
+| fresh held-out: takes that pass the take check, take CLAP | 8, 0.809 | 7, 0.863 |
+
+On the harder fresh set 0.4.0 is level with 0.3 on the median and ahead on the mean, the spectrogram gap and the
+grades (up on 7 clips, down on 2), but fewer of its takes pass, the bed check can drop a bed the fit needed, and
+onsets came out blunter (1.00 octave off against 0.53). Every number and what is still weak:
+[docs/FIT.md](docs/FIT.md#the-replication-benchmark-04).
## Sound for films: bank, sfx, kits
-The forge's delivered sounds ship as a bank that renders by id, with no models and no network: 183 forge v2 finals
-(34 categories), each stored as its render block and re-rendering its delivered WAV to one 24-bit step, three tuned
-kalimba voices, and 15 sounds fitted from CC0 recordings (clicks, a tick, pops, touches, keys, a snap, a bell, a
-whoosh; credits in [docs/CREDITS_FIT.md](docs/CREDITS_FIT.md)). A film pipeline renders from it in seconds; searching
-and fitting new sounds stay offline steps.
+The forge's delivered sounds ship as a bank of 252 that renders by id, with no models and no network: 183 forge v2
+finals (34 categories), each stored as its render block and re-rendering its delivered WAV to one 24-bit step; 51
+sounds made in 0.4 from new recipes for music-led launch films; three tuned kalimba voices; and 15 sounds fitted from
+CC0 recordings (clicks, a tick, pops, touches, keys, a snap, a bell, a whoosh; credits in
+[docs/CREDITS_FIT.md](docs/CREDITS_FIT.md)). A film pipeline renders from it in seconds; searching and fitting new
+sounds stay offline steps.
+
+**The 0.4 families**, each a handful of different members, all made from recipes (none copied: the largest CLAP
+cosine of any of them to the reference clips is 0.84, the bar 0.96) and each with its measured profile and take
+audit in the bank:
+
+| family | members | for |
+|---|---|---|
+| `ui_pop` | soft, step, boop, blip, fall, wide | rounded mid pops (0.12-0.2 s, 550-1030 Hz, 13-44 ms onsets): chips, bubbles, rows arriving |
+| `knock` | wood, plastic, snap, desk, double | mid-band wood and plastic knocks (0.1-0.2 s) |
+| `cut_hit` | sub, kick, punch, rumble, air | short sub hits with a kick, a punchy cut hit (0.2-0.5 s; rumble 1.4 s) |
+| `mallet` | glock, celesta, marimba, pluck; flourish, up4 | gated mallet notes (~0.35 s, glock and celesta wide), a soft pluck, chime runs; tuned by `--pitch` |
+| `drip` | drop, bubble, plink, bloop, tiny | water drops whose pitch rises (the audit allows a bubble its climb) |
+| `shutter` | dslr, snap, mirror, film, quick | camera shutters: a two-stage mechanical click for snap zooms and whip-cuts |
+| `reverse_swell` | cymbal, ring, dark, short, breath | reverse cymbals and swells that peak just after a cut |
+| new members | whoosh_big thump x3, riser dark / build / snatch, swish bright / flick, whoosh_fast whip, glitch burst / mid / chirp, notify mid_tine / mid_wood | a thump-whoosh, short dark risers without a synth sweep, bright swishes and a falling whip, glitches that hit first, rounded mid dings |
```bash
earshot sfx list --kind whoosh # ids, lengths, peak times, cue kinds, knobs
earshot sfx render whoosh_fast/crisp -o w.wav --dur 0.8 --apex 0.32 # a new take aimed at a move (--seed N)
earshot sfx render pluck/kalimba -o e5.wav --pitch E5 # tuned through the recipe's own f0
earshot sfx check w.wav # clicks, cut tails, clipping, ring, rising glides
-earshot kit init --preset calm -o audio/kit.json # calm | crisp | minimal
+earshot kit init --preset calm -o audio/kit.json # calm | crisp | minimal | launch
earshot kit render audio/kit.json --out audio/kit # takes + kit.lock.json (measured onset / peak / end)
+earshot sfx render ui_pop/soft -o pop.wav --pitch A4 # a 0.4 sound, tuned
```
- **Knobs** edit each recipe's own parameters, never resample: `dur` lands exactly, `apex` puts a whoosh's loudest
@@ -181,12 +230,30 @@ earshot kit render audio/kit.json --out audio/kit # takes + k
sub drop G1 measure 0 cents off); noise sounds refuse a pitch and say why.
- **The audit** (`earshot sfx check`, run on every render) is measurement only: clicks, cut tails, clipping, DC, comb
ring, a re-rising tail, a tonal pitch that climbs more than 4 semitones (the rising "synth whip"), and for hits,
- stacked attacks or a build before the hit. 185 of the 201 sounds are clean; the presets use none of the others.
+ stacked attacks or a build before the hit (a water drop's rising bubble is allowed). 236 of the 252 sounds are clean
+ at their own seed (all 51 of 0.4's); the presets use none of the others.
- **Kits** name one sound per role (tick, hit, whoosh.short, key, bell, text ...). `kit render` writes deterministic,
cached takes and a lock with each take's measured sync points; a take that fails the audit is re-rolled on a new
- seed. The three presets sound every role but riser, swell and suck, use one clean hit for the lockup, and use the
- fitted recordings for clicks, ticks, keys, touches and the snap where they measure closer to the role than the
- forge member (earshot match and realness).
+ seed. calm, crisp and minimal sound every role but riser, swell and suck, use one clean hit for the lockup, and use
+ the fitted recordings for clicks, ticks, keys, touches and the snap where they measure closer to the role than the
+ forge member (earshot match and realness); they render byte for byte as in 0.3.0.
+- **The launch preset (0.4)** is for films cut to music. Each role names its level over the music bed (`bed_db`: the
+ mix's loudest 50 ms at the effect over the music's median level in the second before it, 3-10 dB), where it sits
+ against the cut (`offset_ms`: a tap -60 ms, a knock -40, a click -11, a pop-in thump +30, a whoosh peak +37, a swell
+ peak +42) and how close a run may go (`min_gap_ms`: ticks 66, keys 87, pops 114), and sets ticks and bells wide
+ (`width`, an L / R correlation). The numbers are medians measured on the reference clips. A mixer turns them into
+ placements: `kit.bed_level(music, t)`, `kit.gain_db(take, role, bed)`, `kit.place(take, role, cue)`,
+ `kit.space_run(times, gap)`; the music steps down before a hit instead of ducking under every effect (the lock's
+ `mix` block). All 72 takes pass the audit.
+
+```python
+from earshot.forge import kit
+lock = kit.render('audio/kit.json', 'audio/kit') # from `earshot kit init --preset launch`
+role = lock['roles']['pop']; take = role['takes'][0]
+start = kit.place(take, role, cue_s=4.20) # file start: its onset 4 ms after the cue
+gain = kit.gain_db(take, role, kit.bed_level('music.wav', 4.20)) # dB on the take: 8.5 dB over that bed
+times = kit.space_run([5.00, 5.05, 5.10, 5.16], role['min_gap_ms']) # a run of pops, at least 114 ms apart
+```
- **Python and MCP**: `from earshot.forge import bank; y, meta = bank.render('pro_sub_drop/ref', pitch='G1')`;
`earshot.api.sfx_render / kit_render`; MCP tools `sfx_list`, `sfx_render`, `kit_render`.
diff --git a/docs/FIT.md b/docs/FIT.md
index caece4b..c82b2fc 100644
--- a/docs/FIT.md
+++ b/docs/FIT.md
@@ -20,32 +20,69 @@ below. In the mini blind test (below) the listener could not tell the recordings
## How it works
-Three fitters, each analysis by synthesis, each producing a plain-JSON model:
+Four fitters, each analysis by synthesis, each producing a plain-JSON model:
| fitter | for | the model |
|---|---|---|
-| `fit/modal.py` | struck, ringing sounds (clicks, taps, keys, impacts) | per strike, damped sinusoids `[Hz, decay, amplitude, phase]` (ESPRIT on sub-bands + least squares); the contact as fast modes; a sub-20 Hz pulse; band noise envelopes |
-| `fit/sines.py` | tonal sounds (dings, chimes, pops with a pitch) | tracked partials with frequency and level breakpoints (McAulay-Quatieri / SMS; glides kept); strike atoms; 1/3-octave residual noise including the recording's own floor |
-| `fit/spectral.py` | noisy sounds (whooshes, swishes, cloth) | a time-varying 48-band noise gesture, stereo per band group, flutter / crackle texture statistics, tonal ridges |
-
-`fit/auto.py` is the engine on top of them.
-
-1. **Measure the sound.** `character()` measures attack, decay, crest, tonal share, partial persistence, glide and
+| `fit/modal.py` | struck, ringing sounds (clicks, taps, keys, impacts) | per strike, damped sinusoids `[Hz, decay, amplitude, phase]` (ESPRIT on sub-bands + least squares); the contact as fast modes; a sub-20 Hz pulse; band noise envelopes; for a stereo recording the side over time per octave band (`side_t`, 0.4) |
+| `fit/sines.py` | tonal sounds (dings, chimes, pops with a pitch) | tracked partials with frequency and level breakpoints (McAulay-Quatieri / SMS; glides kept); strike atoms; 1/3-octave residual noise including the recording's own floor; for a stereo recording the side over time per octave band (`side`, 0.4) |
+| `fit/spectral.py` | noisy sounds (whooshes, swishes, cloth) | a time-varying 48-band noise gesture, stereo per band group, flutter / crackle texture statistics, tonal ridges; motion knobs (doppler, proximity, pan, motion) read from its trajectory (0.4) |
+| `fit/gesture.py` (0.4) | designed gestures (whooshes, swells, risers, camera moves), and short sounds where tones plus a little air beat partials plus residual noise | tones (the deterministic part of the sines candidate) + air: a detail noise model of what the tones leave in the recording, split in the magnitude domain (an STFT mask), so a take is fresh air around the same tones, not a re-jittered bank of noise partials |
+
+`fit/space.py` (0.4) adds a fitted diffuse tail to the chosen model of a stereo recording whose tail is decorrelated
+(a seeded room in the side: the mono sum and every mono measure stay as they were). `fit/auto.py` is the engine on
+top of them.
+
+0. **Strip a bed.** A recording with a steady pre-roll 12-40 dB under the sound (a film's music under an effect) is
+ stripped (`bed_of`, `strip_bed`) and the sound alone is fitted from its onset. The bed's spectrum is kept as
+ `post['bed']`: `render()` puts a steady noise with it back under the sound, so the fit is scored against the
+ recording as it is; `render(bed=False)` gives the sound alone. `earshot fit --keep-bed` fits the recording as it
+ is. Analysis by synthesis decides whether it was a bed (0.4 round C2, `BED_CHECK`): after the choice (step 4) the
+ winning candidate's kind is fitted on the recording as it is too, and the bed is dropped when that scores lower
+ on the same seeds. A whoosh's own steady approach 13.6 dB under its peak had been stripped as music, and every
+ candidate fitted on the rest fell (CLAP 0.88 -> 0.64); on the 12 dev clips with a bed the plain fit won on 8.
+1. **Measure the sound.** `character()` measures attack, decay, crest, tonal share, partial persistence, glide (2
+ semitones over 15 ms count; before 0.4 the gate never fired on the reference clips' 2-6 semitone pops and dings) and
flatness, and names a class (glide, ringing, struck, gesture, mixed). The class orders the candidates.
-2. **Fit every candidate.** Under 1 s that is all six: modal, sines, spectral, and three hybrids. Longer sounds get
- the class's four.
+2. **Fit every candidate.** Under 1 s that is all seven: modal, sines, gesture, spectral, and three hybrids. Longer
+ sounds get the class's four (a long gesture, mixed, glide or ringing sound gets the gesture model second).
- In a **hybrid**, each layer but the last keeps only its deterministic part, and the next layer is fitted on
what is left. Example: sines partials, then modal modes on the rest, then spectral noise on what remains.
- So a pop becomes chirp + body + air, and a key press becomes body + air.
+ - A sines layer of a stereo recording keeps its fitted side (0.4), so what it leaves for the next layers is taken
+ out in stereo: the air models only the side the tones do not hold (a panned tone's side, counted twice before,
+ came out about 3 dB too wide).
3. **Measure each candidate's post stage** from two renders (analysis by synthesis again):
- a smooth third-octave EQ that moves the render's average balance onto the original's;
- the level;
- the original's attack envelope. Before the onset a render may not stand above it: band noise and window smear
leak energy before a strike, which softens the attack. Over the first 6-25 ms after a sharp onset the render
follows the original's contour within ±9 dB. This fixed a ding whose onset the sines fit had blunted from 60
- to 4 dB/ms, and noise up to 12 dB above the original in the 4 ms before a key's strike.
+ to 4 dB/ms, and noise up to 12 dB above the original in the 4 ms before a key's strike. For a noise model
+ (spectral, gesture) the clamp's correction is high-passed at 200 Hz and kept only when it lowers the score
+ (its gain steps follow the noise's own flicker and put energy into empty low bands);
+ - the gate (0.4): a cut-only gain curve where the renders stand over the original after the onset: the gaps of a
+ run of pulses, a hard stop, a tail the model lets ring on (modes and partials decay exponentially and cannot
+ stop; a gated pulse falls 30-60 dB in 5 ms). Kept only when it lowers the score and, with `--clap`, costs at
+ most 0.005 of CLAP. Knobs that move time move it: a sines stretch and a gesture length about the first strike
+ (the attack clamp too), a spectral length / speed about the start, and a modal spacing strike by strike (the
+ cut after a strike moves with it and holds across a longer gap or ends where a shorter one does; the reopening
+ in the last 2 ms before a strike moves with that strike). A morph warps each source's gate through its strike,
+ onset or peak anchors before the two mix, so the cuts sit in the morph's gaps.
4. **Score each candidate's render against the original** and keep the lowest (`quality.score`, metrics below).
- `--clap` adds CLAP to the score; loading CLAP takes about 50 s.
+ `--clap` adds CLAP to the score; loading CLAP takes about 50 s. With seed 0 alone, the best three are scored on
+ two more seeds and the choice is made on the mean (`CONFIRM`): on short noise-led sounds one seed's CLAP spread
+ 0.84-0.92, and the seed-0 choice took the lucky draws. Then the CLAP margin rule (0.4 round C2, `CLAP_MARGIN`): a
+ candidate whose CLAP is at least 0.02 higher on seed 0 and on the mean over the confirm seeds, whose score is at
+ most 1.5 higher (both) and whose envelope r is at most 0.05 lower, is chosen instead. The model-free terms (mel,
+ onset, envelope, excess) had decided near-ties against a clearly closer sound: on the first held-out check five of
+ six regressions were the gesture model chosen with 0.015-0.06 less CLAP than another candidate.
+5. **Finish** (0.4). A stereo recording with a decorrelated tail gives the chosen model `fit/space.py`'s side layer.
+ Every render ends with a fade-in (2 ms, shorter when the onset comes earlier) and a gain down to -1 dBFS when it
+ peaks over that (18 of 464 takes of the reference clips' fits had peaked over 0 dBFS). A model fitted before 0.4
+ (modal, sines or hybrid without the `coherent` flag: the 0.3 bank's fit entries; spectral without the `finish`
+ flag every 0.4 spectral fit carries) renders as it did, without either (`auto.finished()`): the presets' takes
+ stay byte-identical to 0.3.0, and a 0.3 spectral fit or its recipe module renders sample for sample.
`fit/codegen.py` then writes the model as a **forge recipe module**. It contains:
- `TARGET` / `ALTS`: from the refset category's recipe when the source is a refset file;
@@ -77,6 +114,8 @@ earshot fit click.wav -o made/click_fit.py # recipe + made/click_fit.
earshot fit click.wav -o made/click_fit.py --report # + render-vs-render floor + CLAP cosine -> click_fit.report.json
earshot fit pop.wav --kind hybrid --layers sines,modal,spectral --render pop_fit.wav
earshot fit whoosh.wav --clap # let CLAP help choose the fit (loads CLAP, ~50 s)
+earshot fit swell.wav --kind gesture # tones + air (0.4)
+earshot fit film_clip.wav --keep-bed # fit a clip with music under it as it is (0.4 strips the bed)
earshot fit --morph a.wav b.wav --amount 0.5 -o made/ab_morph.py
earshot fit --forge made/click_fit.py --budget 96 --dir made/click_variations # critic-searched variations
python made/click_fit.py 3 take3.wav # the module renders on its own (seed 3)
@@ -90,7 +129,7 @@ module name.
```python
from earshot.forge import fit
from earshot.forge.fit.quality import quality
-m = fit.fit('click.wav') # kind='auto' | 'modal' | 'sines' | 'spectral' | 'hybrid'
+m = fit.fit('click.wav') # kind='auto' | 'modal' | 'sines' | 'spectral' | 'gesture' | 'hybrid'
y = fit.render(m, seed=0, vary=0.3) # knobs: earshot.forge.fit.auto.knobs_of(m)
q = quality('click.wav', fit.render(m)) # the metrics below; clap=True / describe=True add the model checks
src = fit.to_recipe(m, name='click_fit', source='click.wav')
@@ -99,8 +138,8 @@ mm = fit.morph(m_a, m_b, 0.5)
## Quality metrics (`fit/quality.py`)
-Both sounds are mono-summed. Both are cut at the original's onset, so a timing error still counts. Each is levelled
-to its own loudest 50 ms; the level difference is reported separately.
+Both sounds are mono-summed (except for the stereo measures). Both are cut at the original's onset, so a timing error
+still counts. Each is levelled to its own loudest 50 ms; the level difference is reported separately.
| metric | what |
|---|---|
@@ -109,12 +148,17 @@ to its own loudest 50 ms; the level difference is reported separately.
| `onset_err` | \|log2\| of the onset-sharpness ratio. Sharpness: 17 dB over the time the first onset takes from 20 to 3 dB under its local peak (0.125 ms frames, interpolated) |
| `tone_db` | RMS third-octave balance difference (`earshot.forge.tone`) |
| `excess_db` / `missing_db` | energy the resynthesis has where the recording is quiet, or the reverse. Mean dB beyond 6 dB over third-octave band x 10 ms cells, looking 100 dB deep. It catches hiss bands, noise floors and rings that outlive the original, where mel and envelope stop looking. It tracks CLAP's verdicts closely |
+| `side_db`, `width_db` (0.4) | side / mid energy of each sound over the original's active span, and the per-octave-band width error weighted by the original's band level |
+| `tail_corr`, `tail_corr_err` (0.4) | L/R correlation over the original's tail (from its loudest 10 ms + 30 ms to the end of its tail) of each sound, and their difference: a reverberant space decorrelates |
| `clap_cos`, `describe` | the model checks (optional) |
**floor**: `quality(render seed 11, render seed 12)`. A model with noise in it can never beat its own floor on
`mel_db`.
-**score** = mel + 0.3 tone + 8 (1 - env r) + onset (capped at 2) + 1.5 excess + missing [+ 20 (1 - CLAP)].
+**score** = mel + 0.3 tone + 8 (1 - env r) + onset (capped at 2) + 1.5 excess + missing + 0.04 width (capped at
+20 dB) + tail correlation error [+ 20 (1 - CLAP)]. The stereo terms (0.4) cost a mono re-make of a wide recording
+about what 0.06 of CLAP costs: they decide between candidates that sound alike (before 0.4 a sines render of a wide
+ding at side / mid -16 dB against -5 scored as well as a wide one).
- With excess and missing, the model-free score's choices had a mean CLAP of 0.927 on the 12 refs; without them,
0.856 (the worst choice 0.52). Measured on the same refs, so in-sample.
- The CLAP-including choice agreed with the model-free one on 11 of 12.
@@ -245,6 +289,125 @@ Two more 0.3 changes:
noise layer keeps its 0.2 levelling): seed-0 and seed-averaged quality (seeds 0-7: mel, envelope, tone, onset,
excess, missing, CLAP) are identical, so the tables above stand.
+**0.4 takes.** Measured with the replication benchmark's take check (below), the bank's own check run on four takes
+(seeds 1-4, vary 0.3) of each fit:
+- **Modal vary keeps the fit's structure.** A modal fit's modes are tied to each other by the least squares (close
+ modes, and a later strike's modes, shape and cancel the earlier ones' ring). Vary used to detune every mode on its
+ own (0.6 % x vary) and jitter the strike times: even 0.1 % undid the cancellations, so a key's takes rang on 2.5x
+ the recording's tail. In a model fitted by 0.4 (marked `coherent`) vary moves the modes by a smooth random
+ function of log frequency (neighbouring modes and the same mode in every strike move together), strikes keep their
+ times, and when a knob moves a strike or a mode's frequency (vary, size, spacing) every mode keeps its phase on the
+ first strike's timeline. On the modal and hybrid dev fits the takes now score as vary-0 takes do (a key
+ 0.75 -> 0.89 CLAP to the recording, a pop 0.87 -> 0.94); 16 of 29 pass the take check, 11 before. A modal or
+ hybrid fit also carries `vary_scale` 0.5 (`auto.TAKE_VARY`): the bank's vary 0.3 jitters it as 0.15 did. Models
+ without the flag (the 0.3 bank's fit entries) render exactly as before.
+- **Sines vary moves neighbouring partials together** in a coherent model (`sines.vary_field`): independently
+ detuned, a harmonic series' upper partials went off the harmonic grid and takes read as aliasing or rang.
+- A coherent modal ring-out continues the noise un-faded, like the modes (it used to fall at the recording's
+ fade-out slope, leaving the modes alone in the added stretch).
+- Every render starts at zero and peaks at -1 dBFS at most (above).
+- All of it on the dev fits' saved models: takes passing 39 -> 44 of 78, median take CLAP 0.890 -> 0.896.
+- **Clicks** (round C2), traced to three sources and fixed where they start: the post gate's breakpoints sit 0.5 ms
+ apart, and a 30-80 dB step between two of them put a kink into a loud ring (its linear gain is now smoothed by a
+ 1 ms Hann window, `auto.GATE_SMOOTH_S`); a later modal strike's modes started at full amplitude inside the ring of
+ the strikes before it, a step in the fit itself (they now rise over 0.15 ms, `modal.STRIKE_RAMP_S`, in a coherent
+ model; the first strike keeps its step); a sines strike's atoms are fitted jointly and cancel each other's step at
+ the onset, and vary jittered each on its own (a coherent model now jitters a strike's atoms together).
+- **The gesture air's grains** (round C2): the air's grain texture is a Poisson click train with log-normal
+ amplitudes, and one draw could put its loudest clicks where the sound is quiet: takes of short noise-led fits
+ (keys, ticks, toggles fitted as air) spread 0.84-0.92 CLAP from seed to seed, and some clicked. The gesture air now
+ levels the train to unit power over 5 ms windows (at least 20 clicks; `grains['norm_ms']`, `spectral.fit(
+ grain_norm_ms=)`) and keeps grains only when their excess fast variance is in both halves of the active span
+ (`grain_halves`: stationary texture, as the AM is judged). Levelling every noise band instead (the first idea)
+ cost CLAP on 6 of 9 such fits: it removes fluctuation the recordings have. On 20 dev clips the takes' median gap
+ under seed 0 fell 0.046 -> 0.031 CLAP, take fails for dropping more than 0.06 under seed 0 7 -> 5.
+
+## The replication benchmark (0.4)
+
+`eval/forge/replicate.py` asks whether `earshot fit` can re-make professional sound effects. The 0.4 study used a
+private set of licensed reference clips: 116 clean one-gesture clips in 24 categories (they, the renders and the
+per-clip results stay local). The 0.4.0 rows below were measured before the release review's hybrid side fix (step 2:
+a sines layer's stereo side is left out of the residual), which can change hybrid candidates on stereo tonal clips;
+they were not measured again. Each clip is fitted with `--clap`; its seed-0 render, the model's floor and four
+takes (seeds 1-4, vary 0.3) are measured against it. Grades: **high** CLAP >= 0.92 and mel gap <= 2.5 dB (the bank's
+ship bar), **good** >= 0.88 / <= 3.5, **fair** >= 0.80 / <= 5.0, **poor** the rest. "Takes pass" is the bank's take
+check on all four takes. A fixed split by clip id holds 38 clips out; 0.4 was tuned on the other 78 only, and the
+held-out clips were fitted once, at the end.
+
+| split | engine | CLAP (median) | mel gap dB | high / good / fair / poor | takes pass |
+|---|---|---|---|---|---|
+| dev (78) | 0.3 | 0.884 | 1.43 | 19 / 20 / 31 / 8 | 29 |
+| dev (78) | 0.4, first round | 0.907 | 0.90 | 30 / 26 / 20 / 2 | 46 |
+| dev (78) | 0.4.0 (round C2) | 0.921 | 0.85 | 40 / 20 / 17 / 1 | 42 |
+| held-out (38) | 0.3 | 0.878 | 1.73 | 7 / 8 / 19 / 4 | 21 |
+| held-out (38) | 0.4, first round | 0.909 | 0.98 | 14 / 9 / 12 / 3 | 25 |
+| held-out (38), opened: a reference | 0.4.0 (round C2) | 0.903 | 1.04 | 14 / 11 / 12 / 1 | 25 |
+| fresh held-out (28) | 0.3 | 0.853 | 0.87 | 2 / 6 / 14 / 6 | 8 |
+| fresh held-out (28) | 0.4, first round | 0.830 | 0.75 | 4 / 3 / 14 / 7 | 12 |
+| fresh held-out (28) | 0.4.0 (round C2) | 0.851 | 0.58 | 5 / 6 / 13 / 4 | 7 |
+
+(The first-round dev row is the committed engine's own run; an earlier run of the same changes before the confirm
+stage went in read 0.907 / 0.86 / 32 / 24 / 18 / 4.) On the first held-out split the gain matched the dev gain (CLAP
++0.031 against +0.023; high 7 -> 14 against 19 -> 32): per held-out clip the grade went up on 14, stayed on 20 and
+went down on 4. The held-out width misses fell 15 -> 0 of 38,
+the tail correlation misses 9 -> 1 of 29. Of the six held-out regressions, five are the score choosing the gesture
+model on its mel, onset and envelope terms over a candidate with 0.015-0.06 more CLAP; one is a bed false positive (a
+whoosh's own steady approach 13.6 dB under its peak taken for music and stripped: CLAP 0.88 -> 0.64).
+
+**Round C2 and the fresh held-out set.** The first held-out split was opened by that check, so before any further
+tuning a second one was drawn from the same private set of licensed reference clips by a fixed rule (sha1 order
+inside per-category caps, nothing within CLAP 0.97 of a benchmark target, no fitting or listening) and frozen
+(`replicate.py --split-file`). The clean clips had all gone into the benchmark, so its isolation bar is lower (7 dB
+over the local bed against 12): a harder, more bedded set (the bed detector fires on half of it), compared engine
+against engine on the same 28 clips. Round C2 (bed check, CLAP margin rule, take clicks, gesture air grains: above)
+was tuned on dev only, run once on dev, then opened the fresh set once:
+- The first-round 0.4 engine had not beaten 0.3 on the fresh clips (median CLAP 0.830 against 0.853; grades up on 5,
+ down on 6): its bed stripping cost the stripped fits up to 0.16 of CLAP against a fit of the recording as it is.
+- Round C2 lifts it on the fresh clips by about what it gained on dev (median +0.021, mean 0.840 -> 0.860; grades up
+ on 8, down on 2; take CLAP 0.824 -> 0.863), level with 0.3 on the median and ahead on the mean (0.844), the mel gap,
+ the grades and the take CLAP.
+- Takes passing fell 12 -> 7 there (dev 46 -> 42): a plain fit of a bedded clip carries the bed as part of its sound,
+ so its takes vary it and end it at the clip's end (four of the six new failures). The onset error also grew (0.53 for
+ 0.3, 0.82 first round, 1.00 now).
+- On the opened first held-out split: CLAP mean 0.894 -> 0.903 (median 0.909 -> 0.903), poor 3 -> 1, takes 25 -> 25;
+ the whoosh whose approach was stripped as a bed is back at 0.88.
+
+What moved the dev numbers, each measured on the dev split by the change's own run:
+- **The gesture model** won 38 of the 78 dev clips. On the 9 dev clips of the gesture categories (camera air,
+ glitch, reverse swell, riser, swish, fast whoosh, shimmer) it lifted the median CLAP 0.858 -> 0.885 and the takes
+ passing 3 -> 7.
+- **Stereo**: on 29 wide-category clips the width misses (over 3 dB) fell 16 -> 0 and the tail correlation misses
+ (over 0.3) 13 -> 2.
+- **The gate** and the **modal pre-onset cap**: on the 67 gated, multi-pulse and short-click dev clips the gated
+ clips' gate error 21 -> 1.7 dB; modal onset error 0.39 -> 0.25 octave.
+- **Beds**: 12 of 78 dev clips carry a music bed; fitting the stripped sound and rendering the bed back under it
+ (rather than rendering it dry) kept clips like a camera move at CLAP 0.85 instead of 0.72.
+- **Takes**: coherent vary (above) 39 -> 44 of 78.
+- **Round C2** (from the first-round engine's own dev run, 0.907 / 30 / 26 / 20 / 2 / 46): the bed check on the 12 bed
+ clips CLAP 0.907 -> 0.928 (the plain fit won on 8); the CLAP margin rule 5 choices, all to a higher CLAP; the click
+ fixes cleared the traced clicks (four clips' takes); the grain levelling cut the takes' gap under seed 0 0.046 -> 0.031 on
+ 20 clips. Together: 0.921, 40 / 20 / 17 / 1, takes 42.
+
+Still weak on dev: takes (42 of 78; a take 0.06 under seed 0 on 12 clips, loudness spread over 2 LU on 9, a comb ring
+on 8); modal ring-outs of short clips read as a comb ring (modes whose decays the clip could not pin ring alone once
+the noise fades); multi-pulse counts on long gesture wins (7 of the 28 clips with 3 or more pulses miss by more than
+2); clips of 0.5 s or longer reach high on 3 of 13.
+
+**Known gaps after 0.4.0**, for the next round:
+- The bed check decides by the score alone: it needs a CLAP guard (a margin like the choice rule's) and should weigh
+ the takes. A plain fit of a bedded clip keeps the bed in its sound, which a kit should not have (`render(bed=False)`
+ separates only a stripped fit); one option is to use the plain fit's score only to decide whether the bed is real
+ and keep the stripped model's sound.
+- Onsets: 0.53 octave for 0.3 against 1.00 for 0.4.0 on the fresh held-out set (14 of 28 clips off by more than an
+ octave). Measuring the onset after the bed and a sharper first grain of the gesture air are the leads.
+- Take clicks left: a loud grain inside an air-only key's attack window (levelling over 5 ms cannot stop it) and one
+ modal-layer click in a hybrid take; the comb ring of short modal ring-outs (capping the ring-out trades against the
+ audit's 'truncated').
+- Takes passing fell in round C2 (dev 46 -> 42, fresh held-out 12 -> 7) although the take CLAP rose; the loudness
+ spread and ring borderlines need their own look.
+- Both held-out sets are opened. Under the fresh set's rule only 25 eligible clips remain, fewer in most categories:
+ a next held-out set needs more source material or a relaxed rule.
+
## Licence
A fit of a recording is a derivative of that recording: the parameters are measured from it. Keep the source's
@@ -262,7 +425,12 @@ attribution and licence with the generated recipe; its docstring names the sourc
floor. The worst: the pop mk-3000_1 at 6.1 dB against a 3.2 dB floor, and the key fs-561697_1 at 4.7 against 2.5.
**Onsets.** Even with the attack match, the measured onset sharpness differs by 0.6-1.5 octaves on 6 of 12. The
-metric itself is noisy on stochastic renders: different seeds of one model ranged 23-140 dB/ms on a key.
+metric itself is noisy on stochastic renders: different seeds of one model ranged 23-140 dB/ms on a key. On the 0.4
+dev fits, 8 seeds of one model spread 0.63 octave (standard deviation) on a whoosh and 0.91 on a riser: on 0.125 ms
+frames a noise's own flicker sets the crossings. Reading a slow original on 0.5 ms frames (and its render with it)
+cut that to 0.35 and 0.48 and a swish's error from 2.7 to 0.03 octave, but it also re-read 31 of the 78 dev
+originals, sharp two-part key onsets among them (58 -> 3 dB/ms), and worsened a glitch (0.9 -> 3.2 octaves); it
+needs its own benchmark run and is not in 0.4.
**Noise models.** A noise band is spectrally flat within its band, which fills valleys: the thock key's tone sits at
4.4 dB even after the EQ. The modal key fits have high floors (2.5-3.3 dB): their noise varies take to take more
@@ -271,7 +439,9 @@ than the recording suggests.
**Whooshes fitted as modes or partials.** The engine chose modal and sines for the two fast whooshes because they
re-render them closest. Such a model is a sinusoidal decomposition of noise, about 600 modes and a 274 KB recipe.
Its knob variations (size, pitch) may sound metallic. The spectral fitter is the generative choice for whooshes, but
-it lost on these two refs.
+it lost on these two refs. Since 0.4 the gesture model (tones + air) is a candidate for every short sound and second
+for long gestures; on the dev clips it won the whooshes, swells, risers and camera moves the sines and modal
+fits had taken.
**Morphs.**
- Strikes pair by index, so a key with 2 strikes and one with 4 pair crudely.
@@ -287,8 +457,13 @@ spectro-temporal ceiling was tried and left out: CLAP +0.004, mel -0.3 dB.
**Speed.** Default `earshot fit` takes 10-100 s for short sounds and minutes for long impacts. `--clap` adds about
50 s to load CLAP.
-**Choosing.** The weights of `quality.score` were set on the same 12 refs they were checked on. The selection has
-no held-out test yet.
+**Choosing.** The weights of `quality.score` were set on the same 12 refs they were checked on. The 0.4 engine was
+tuned on the replication benchmark's dev split and checked once on each held-out set (above). The CLAP margin rule
+settles clear near-ties for CLAP, but a candidate 0.02-0.03 higher on seed 0 and not on the confirm seeds' mean still
+loses. The bed check decides by the score alone: on the fresh held-out set it dropped a bed whose plain fit had 0.087
+less CLAP, and a plain fit of a bedded clip keeps the bed in its sound (`render(bed=False)` separates only a stripped
+fit), which costs its takes.
-**Not tested here:** zips and long glides (the sines fitter tracks chirps, but no zip ref exists), risers, swipes,
-shimmers, stereo-heavy designed sounds, and anything from the bench split.
+**Not tested here:** zips and long glides (the sines fitter tracks chirps, but no zip ref exists), swipes, and
+anything from the bench split. The replication benchmark (above) covers risers, shimmers, swells, camera moves and
+stereo designed sounds among its reference clips.
diff --git a/docs/INTEGRATION.md b/docs/INTEGRATION.md
index 570322c..9e89350 100644
--- a/docs/INTEGRATION.md
+++ b/docs/INTEGRATION.md
@@ -18,18 +18,23 @@ audio/score.json "sfx": {"engine": "auto", "kit": "audio/kit.json"} ------+-->
audio/kit.json --scripts/audio/kit.py--> earshot kit render --> audio/kit/*.wav + kit.lock.json --^
```
-1. **The kit.** `audio/kit.json` starts from a preset (`earshot kit init --preset calm|crisp|minimal`; calm is the
- default for text-led films) and is edited per film: the lockup hit tuned to the film's root (`edits.pitch`), another
- member for a role, a fitted sound. Roles are cinetic's cue kinds and their variants (below); `null` roles are not
- sounded.
+1. **The kit.** `audio/kit.json` starts from a preset (`earshot kit init --preset calm|crisp|minimal|launch`; calm
+ is the default for text-led films, launch for films cut to music) and is edited per film: the lockup hit tuned to
+ the film's root (`edits.pitch`), another member for a role, a fitted sound. Roles are cinetic's cue kinds and their
+ variants (below); `null` roles are not sounded.
2. **The bridge.** cinetic's `scripts/audio/kit.py` finds earshot (importable in cinetic's Python, `sfx.python` in
`score.json`, `EARSHOT_PYTHON`, or `earshot` on PATH), runs `earshot kit render audio/kit.json --out audio/kit` in
one process (cached: an unchanged kit re-renders nothing) and keeps the takes in `audio/kit/`, so the film rebuilds
on a machine without earshot.
3. **Placement.** score.py plays one take per event (a new take per event by hash; typing cycles through the takes),
placed by the sync point the lock measured for the role's `align`: the take's onset on the cue frame, a whoosh's
- loudest 5 ms on its apex, a riser's end on the hit. `level_db` and the event weight set its level; a cue whose role
- is `null` is skipped with a report line.
+ loudest 5 ms on its apex, a riser's end on the hit (or its climax, for a riser or swell aligned on its `peak` as
+ the launch preset's are: their 0.4 members fall after the climax). `level_db` and the event weight set its level;
+ a cue whose role is `null` is skipped with a report line. A role with the 0.4 mix keys (the launch preset) asks for more: its
+ `offset_ms` moves the sync point off the frame (`kit.place`), its `bed_db` sets the take against the music under
+ the cue (`kit.bed_level` + `kit.gain_db`, instead of `level_db`), its `min_gap_ms` spaces a run (`kit.space_run`),
+ and the lock's `mix.stop_down` asks for a music dip before hits rather than a duck after them
+ ([docs/SFX.md](SFX.md#mix-keys-04)). A renderer that ignores them still gets valid takes and `level_db`.
4. **Fallback.** earshot render -> the vendored `audio/kit/` takes -> cinetic's own synth voices with a loud warning.
The build never fails only because earshot is missing.
5. **Listening.** `scripts/audio/listen.py` runs `earshot timeline --cues`, `sheet` and `critique` in one process and
@@ -46,7 +51,7 @@ audio/kit.json --scripts/audio/kit.py--> earshot kit render --> audio/kit/*.wav
| land / land light | `land` / `land.light` | onset |
| pop | `pop` | onset |
| whoosh (short / long move) | `whoosh.short` / `whoosh.long` | peak |
-| suck, riser, swell | `suck`, `riser`, `swell` (off in every preset) | end |
+| suck, riser, swell | `suck`, `riser`, `swell` (off in calm, crisp, minimal; launch sounds riser and swell, both aligned on the peak) | end |
| drop | `drop` | onset |
| key / key space / key return | `key` / `key.space` / `key.return` | onset |
| click / click confirm | `click` / `click.confirm` | onset |
diff --git a/docs/SFX.md b/docs/SFX.md
index b95cb50..79116e0 100644
--- a/docs/SFX.md
+++ b/docs/SFX.md
@@ -1,8 +1,9 @@
# Sound for films: the bank, `earshot sfx`, `earshot kit`
-earshot 0.3.0 ships its forge sounds, three tuned voices and 15 sounds fitted from public-domain recordings as a bank
-that renders by id, aims a take at the picture, audits it without models, and renders a whole film's kit into takes
-with measured sync points. Everything here is pure DSP (numpy / scipy): no models, no network, no recorded audio (a
+earshot 0.4.0 ships its forge sounds, 51 sounds made in 0.4 for music-led launch films, three tuned voices and 15
+sounds fitted from public-domain recordings as a bank that renders by id, aims a take at the picture, audits it without
+models, and renders a whole film's kit into takes with measured sync points (placed against the music bed and the cut
+by the launch preset). Everything here is pure DSP (numpy / scipy): no models, no network, no recorded audio (a
fitted sound is stored as its modes, partials and band envelopes, not as samples). It is the sound engine of a
code-made film: a film pipeline renders from the bank in seconds and only ever searches (`earshot forge`) or fits
(`earshot fit`) offline.
@@ -12,30 +13,32 @@ earshot sfx list --kind whoosh # what is in
earshot sfx render whoosh_fast/crisp -o w.wav --dur 0.8 --apex 0.32 # one take aimed at a move
earshot sfx render pluck/kalimba -o e5.wav --pitch E5 # a tuned note
earshot sfx check w.wav # the model-free audit of any sound effect
-earshot kit init --preset calm -o audio/kit.json # a film's kit from a preset
+earshot kit init --preset calm -o audio/kit.json # a film's kit from a preset (or --preset launch)
earshot kit render audio/kit.json --out audio/kit # takes + audio/kit/kit.lock.json (cached)
earshot kit check audio/kit.json --out audio/kit # ids, knobs, the audit of every take
earshot kit audition audio/kit.json -o audition.wav # every take, role by role, at its level
```
-Contents: [the bank](#the-bank) · [knobs](#knobs-dur-apex-pitch-level) · [the audit](#the-audit-sfx-check) ·
-[kits](#kits) · [presets](#the-presets) · [Python, MCP](#python-and-mcp) · [fit entries](#fit-entries-from-public-domain-recordings) ·
-[cost](#cost) · [limits](#limits)
+Contents: [the bank](#the-bank) · [the 0.4 members](#the-04-members) · [knobs](#knobs-dur-apex-pitch-level) ·
+[the audit](#the-audit-sfx-check) · [kits](#kits) · [mix keys](#mix-keys-04) · [presets](#the-presets) ·
+[launch](#the-launch-preset-04) · [Python, MCP](#python-and-mcp) ·
+[fit entries](#fit-entries-from-public-domain-recordings) · [cost](#cost) · [limits](#limits)
## The bank
-`earshot/forge/bank/*.json` (package data; the forge file is 275 KB, 64 KB gzipped; the fit file 0.5 MB): 201 sounds
-today. `earshot doctor` reports the version and the counts.
+`earshot/forge/bank/*.json` (package data; the forge file is 275 KB, 64 KB gzipped; the 0.4 file 70 KB; the fit file
+0.5 MB): 252 sounds today. `earshot doctor` reports the version and the counts.
| file | version | entries | what |
|---|---|---|---|
| `forge_v2.json` | `forge_v2r6` | 183 | every delivered forge v2 final (34 categories, 4-6 named members each) |
+| `forge04.json` | `forge04r1` | 51 | the 0.4 recipe members: 7 new categories and new members of 6 old ones ([below](#the-04-members)) |
| `fit.json` | `fit1` | 15 | recipes fitted from CC0 recordings (`fit/<role>/<name>`, [below](#fit-entries-from-public-domain-recordings)) |
| `pluck.json` | `pluck1` | 3 | the tuned kalimba voices (`pluck/kalimba`, `pluck/kalimba_soft`, `pluck/kalimba_ring`) |
-`bank.VERSION` joins the file versions (`forge_v2r6+fit1+pluck1`); extra bank files load from `EARSHOT_BANK` (paths
-joined by the OS path separator, `os.pathsep`: `;` on Windows, `:` elsewhere; a missing or unreadable file is an error
-that names it). Ids are `<category>/<member>`: the category is the public name (a user never needs the
+`bank.VERSION` joins the file versions (`forge_v2r6+forge04r1+fit1+pluck1`: forge_v2 first, the other forge files,
+then by name); extra bank files load from `EARSHOT_BANK` (paths joined by the OS path separator, `os.pathsep`: `;` on
+Windows, `:` elsewhere; a missing or unreadable file is an error that names it). Ids are `<category>/<member>`: the category is the public name (a user never needs the
recipe module behind it: `typing_key` renders with `recipes/keypress.py`, `pro_pop` with `pop.render_v2`,
`pro_typing_run` with `typing.render_run`).
@@ -46,7 +49,8 @@ compares it with its delivered WAV; one pick off by more than one 24-bit step (1
```bash
python -m earshot.forge.bank build out/forge_v2/*/final -o earshot/forge/bank/forge_v2.json # 183 / 183 within 1.2e-7
python -m earshot.forge.bank build --pluck -o earshot/forge/bank/pluck.json
-python -m earshot.forge.bank check [IDS ...] # re-render against the stored fingerprints
+python -m earshot.forge.bank build --forge04 -o earshot/forge/bank/forge04.json # the 0.4 members, from the recipes
+python -m earshot.forge.bank check [IDS ...] # re-render against the stored fingerprints: 252 / 252
```
The forge's seed banks (`earshot/forge/seeds/`) are search starting points, not this: rounded parameters without
@@ -72,7 +76,7 @@ bit-exactly.
| shimmer, pro_shimmer | shimmer | chimes ringing out, granular sparkle |
| whoosh_fast, whoosh_big, pro_whoosh_big | whoosh.short, whoosh.long, suck, swell | fast passes; big air transitions; the designed low sub swell |
| swish, swipe | whoosh.short | thin flicks and text slides; cloth and paper swipes |
-| riser, pro_riser, downlifter | riser, drop | builds that stop on their loudest sample; energy falling away |
+| riser, pro_riser, downlifter | riser, drop | builds into a cut (forge_v2's stop on their loudest sample; 0.4's riser members peak and then fall, so a kit aligns them on `peak`); energy falling away |
| pro_zip | zip | tonal sweeps across a cut (up / up_hi / arc rise in pitch) |
| impact, pro_impact | hit, land | a cinematic hit (3-6 s tails); a designed gated sub impact (0.7-1.3 s) |
| sub_drop, pro_sub_drop | drop | weight under a hit; the designed drop that blooms after the cut |
@@ -81,6 +85,57 @@ bit-exactly.
| typing_key, pro_typing_key, pro_typing_run | key, key.space, key.return, key.run | mechanical keys; designed key ticks; a whole run |
| pluck | bell, text, hit | a kalimba tine: tuned notes (native C5) |
| fit | the role in the id, plus its `kinds` | recordings fitted as code: clicks, a tick, pops, light touches, keys, a space bar, a snap, a bell, a whoosh, a knock |
+| ui_pop (0.4) | pop, land.light | soft rounded mid pops: chips, bubbles, rows arriving |
+| knock (0.4) | tock, land.light | mid-band wood and plastic knocks |
+| cut_hit (0.4) | hit, drop, land | a short sub hit with a kick, or a short punchy hit, on a cut |
+| mallet (0.4) | bell, text | short gated mallet notes (glockenspiel, celesta, marimba), a soft pluck, chime runs |
+| drip (0.4) | pop | water drops: a bubble whose pitch rises |
+| shutter (0.4) | snap, click | camera shutters: a two-stage mechanical click for snap zooms and whip-cuts |
+| reverse_swell (0.4) | swell | reverse cymbals and swells that peak just after a cut (at most one a film) |
+
+## The 0.4 members
+
+`forge04.json` holds 51 sounds made from earshot 0.4's recipes for the soft, short, mid-register style of music-led
+launch films, tuned toward measurements of a private set of licensed reference clips (not in the repo: no audio, no
+clip name and no per-clip model ships; aggregate measurements of them (medians, IQRs), CLAP cosines to them and
+parameters tuned toward them do). Each entry is a palette member of its recipe module (`FAMILY` / `FAMILIES` /
+`FAMILY_MID` / `RUNS` by name) on its seed, so `python -m earshot.forge.bank build --forge04`
+re-makes the file from the code (`bank.FORGE04` lists the members, their seeds and kinds). Besides the render block,
+meta and fingerprint, every entry carries:
+
+| key | what |
+|---|---|
+| `profile` | the catalog's model-free measures of its render (`recipes/_tonal.measure`): dur_s, attack_ms, t20_ms / t40_ms, centroid_hz, lo150 / hi4k (energy shares under 150 Hz / over 4 kHz), tonal (share within a quarter octave of the strongest bin), glide_st, corr (L / R correlation) |
+| `audit` | its takes audited in its first kind's role, seeds n .. n+5: `clean` of 6 and the warning codes seen (50 members 6 / 6, glitch/mid 5 / 6: one take rings, which a kit re-rolls) |
+| `original` | `{made: 'recipe', max_ref_clap}`: the largest CLAP cosine of its render to any reference clip (116 clean clips and 28 more, three excerpt conventions: fixed 1.0 s and 2.5 s windows and the trimmed span, repeat-padded), 0.53-0.84 over the 51 |
+
+The build refuses a member whose own take is not clean, with fewer than 5 of 6 clean takes, or whose originality is
+unmeasured, stale or over 0.96 (`bank.FORGE04_REF_MAX`). The recorded cosines (`bank.FORGE04_REF_CLAP`) hold for the
+renders they were measured on: `bank.FORGE04_REF_ON` names each one's render block (recipe, fn, params, seed), and a
+member whose block changed, or whose render no longer matches the shipped fingerprint, needs a fresh measurement
+(`--ref-clap fresh.json`, `{id: cosine}`) instead of keeping the old number. Siblings differ: the largest CLAP cosine between two members of a
+new category is 0.93 (cut_hit sub and kick; the two mid dings 0.95), under the ~0.97 two renders of one sound reach.
+
+| category | members (profile: length, centroid, attack, L/R correlation) |
+|---|---|
+| `ui_pop` | soft (0.14 s, 550 Hz, 22 ms, 0.98), step (0.16 s, 710 Hz, 43 ms), boop (0.14 s, 770 Hz, a rounder falling boop), blip (0.04 s, 1030 Hz, 0.07: fully wide), fall (0.16 s, 890 Hz, an eased fall with a faint second pop), wide (0.14 s, 890 Hz, 0.51) |
+| `knock` | wood (0.11 s, 670 Hz: a 400 Hz block and its rattle), plastic (2.4 kHz, bounces 18 ms later), snap (3 kHz), desk (a knuckle on a desk, 81 % under 150 Hz), double (a knock-knock 55 ms apart) |
+| `cut_hit` | sub (0.26 s, 210 Hz: 225 -> 46 Hz with a kick), kick (0.22 s, 275 Hz), punch (0.28 s, 180 Hz, 83 % under 150 Hz), rumble (1.4 s, a wide granular tail), air (0.49 s, a wide air band, 0.18) |
+| `mallet` | glock (0.35 s, 1.3 kHz, doubled an octave under, 0.85), celesta (G5, 0.31 s, 0.48: wide), marimba (G4, 0.33 s), pluck (0.29 s, 460 Hz); runs flourish (3 notes in an arc), up4 (4 notes 110 ms apart, panned note by note) |
+| `drip` | drop (306 Hz rising 8 st), bubble (255 Hz rising 10 st, a daughter bubble), plink (690 Hz), bloop (420 Hz, ~5 st over 0.1-0.15 s: the drip allowance), tiny (1.15 kHz, 0.1 s) |
+| `shutter` | dslr (0.19 s, 1.07 kHz), snap (a dry 2 kHz snap-zoom), mirror (a slower mirror box), film (0.25 s, 625 Hz box after 95 ms of travel), quick (release and curtain 21 ms apart, 1.6 kHz) |
+| `reverse_swell` | cymbal (3.3 kHz, 0.16: wide, a plateau then a stop), ring (rings out), dark (420 Hz, near mono, a low body), short (0.33 s to the peak, 3.7 kHz), breath (1.3 kHz noise, half wide) |
+| whoosh_big | thump, thump_deep, thump_airy: a low whoosh that lands with a thump ~80 ms in and rings ~0.35 s |
+| riser | dark (0.8 s, 450 Hz), build (1.1 s, sub-heavy), snatch (0.63 s): noise, sub and a speeding stutter, no synth sweep |
+| swish / whoosh_fast | swish bright (2.4 kHz), swish flick (4.3 kHz, stops dead, wide); whoosh_fast whip (a wide ring that falls ~45 st) |
+| glitch | burst, mid, chirp: the first burst is the loudest (a glitch that hits first) |
+| notify | mid_tine, mid_wood: rounded mid dings (~690 Hz, 32 ms onsets, t20 90-130 ms) |
+
+Knobs: `pitch` on ui_pop (f0 250-1200 Hz), mallet (f0 150-3200 Hz, runs too), knock (f1 250-2600 Hz), drip (f0
+250-1600 Hz, not measured: the heard pitch is where it has risen to) and cut_hit (the sub's landing pitch, 36-130 Hz,
+not measured); `dur` on reverse_swell (its swell length) and the whoosh family members (shape, apex); risers and
+glitches through their length keys; the rest are trims. The tonal knobs (`knobs.apply(..., decay=, onset=, bright=,
+width=)`) reach ui_pop, knock, mallet, drip, shutter and notify.
## Knobs: dur, apex, pitch, level
@@ -129,6 +184,8 @@ Knobs edit the recipe's own parameters (`earshot/forge/knobs.py`); nothing is re
| impact, pro_impact | body_f1 (body_f0, sub_f) | 34-75 Hz |
| pro_whoosh_big | sub_f, only sub-led members | 25-120 Hz |
| shimmer, pro_shimmer | key: the pitch class only (the register stays) | - |
+ | ui_pop / mallet / knock (0.4) | f0 / f0 / f1 | 250-1200 / 150-3200 / 250-2600 Hz |
+ | drip / cut_hit (0.4) | f0 / sub_f1 (sub_f0), not measured | 250-1600 / 36-130 Hz |
Noise sounds are refused with the reason (whoosh_fast, whoosh_big, swish, swipe, glitch, pro_glitch, typing_key,
pro_typing_key, pro_typing_run), as are targets outside the range and members whose tonal layer is off. The render is
@@ -161,7 +218,7 @@ render runs it (`meta['checks']`). Warnings make `ok` false; notes are informati
| silent | nothing to hear: a sample peak under -60 dBFS, or less than 5 ms within 40 dB of the peak (the shortest bank sound holds 9 ms): digital silence, or a take faded to nothing | `sfx.live_ms` |
| silence | digital silence inside the sound (a note for glitches) | `polish.silence` |
| tail | the tail rises again by more than 6 dB (a note: a roll, a second event, a double member) | `polish.tail` |
-| rising_glide | a tonal sound whose pitch climbs more than 4 semitones over at least 50 ms: the rising "synth whip" | `sfx.rising_glide` |
+| rising_glide | a tonal sound whose pitch climbs more than 4 semitones over at least 50 ms: the rising "synth whip"; a note for the drip category when the climb is one a bubble makes (0.4) | `sfx.rising_glide`, `recipes/drip.allows` |
| stacked, pre_swell | for a hit (role hit / land): two separate attacks, or more than 100 ms of build before the hit | `sfx.layers` |
| pitch_off | a render with a pitch knob whose measured note is more than 50 cents off (bank renders and kit takes) | `knobs.pitch_warning` |
@@ -174,7 +231,13 @@ bells and plucks read 0. The layer heuristic reads a 25 ms RMS envelope (peaks w
dip between them, in the first 0.5 s): two noise hits 60 or 120 ms apart read 2, a gated grain texture reads 1
(pro_impact short / air / swell: 1; deep: 2; rumble: 3); impact swell builds 220 ms before its hit.
-Audit of the whole bank (each sound at its own seed): 185 of 201 clean. Flagged: the three rising zips, the two
+**The drip allowance** (0.4): a water drop's bubble rises as the cavity closes, by design. For the drip category a
+climb of at most 14 st that starts within the first 60 ms and is done within 250 ms (`drip.allows`) is a note; any
+other rising tone, and any climb in another category, stays the rising-glide warning. drip/bloop (about 5 st over
+0.1-0.15 s) is the member that needs it; the other drips rise inside the 50 ms window.
+
+Audit of the whole bank (each sound at its own seed): 236 of 252 clean, all 51 of the 0.4 members among them. Flagged
+(the same 16 as in 0.3): the three rising zips, the two
logo-sting leads, the two rising bubbles, typing_key light and laptop (comb ring), pro_hover_tick air and glass
(clicks), hover_tick detent and pro_typing_key soft (truncated), glitch dropout and logo_sting reverse_bloom, pro_shimmer
m3 (ring). The presets use none of them. All 15 fitted sounds are clean at their own seed; on some new seeds the
@@ -206,7 +269,7 @@ text fragment resolving). Other role names are allowed; a renderer must know the
| `seed` | default: the sound's own, so take 1 is the delivered final |
| `edits` | knobs for every take: `dur`, `apex`, `pitch`, `peak_db` (default -1, at most 0), `weight` |
| `pitches` | one set of takes per note (or Hz); each pitch once (`C#5`, `Db5`, `Cs5` and 554.37 are one pitch) |
-| `align` | the sync point a mixer puts on the cue: `onset`, `peak` (whoosh apexes), `end` (risers, swells); default per role |
+| `align` | the sync point a mixer puts on the cue: `onset`, `peak` (whoosh apexes; a riser or swell that falls after its climax, as the 0.4 members do), `end` (the last 5 ms within 50 dB of the loudest: builds that stop on their loudest sample); default per role. `kit check` notes an `end`-aligned role whose sound rings on more than 30 ms past its peak |
| `max_s` | a take longer than this is faded out to it (0.01 s or more) |
| `level_db` | the role's mix level relative to a take at its peak, a number (for the mixer; the audition applies it) |
| `note` | why this sound |
@@ -240,14 +303,58 @@ text fragment resolving). Other role names are allowed; a renderer must know the
take with its role as context (a `hit` warns on stacked attacks and pre-swells). It notes restraint (riser / swell /
suck sounded) and sounds the presets never use (rising zips, shepard risers and downlifters, logo-sting composites,
shimmers, glitches, tonal pops), so the choice is visible.
-- **Audition.** `earshot kit audition` writes every take role by role at its `level_db`, with an index `.txt`.
+- **Audition.** `earshot kit audition` writes every take role by role at its `level_db` (a role with `bed_db`: at
+ that level over a nominal -24 LUFS bed), with an index `.txt`.
+
+## Mix keys (0.4)
+
+Optional role keys for films cut to music. A kit without them plans, keys and locks its takes exactly as before
+(calm, crisp and minimal have none); a role that has them carries them into the lock, and its takes carry what they
+need.
+
+| key | meaning | in the lock |
+|---|---|---|
+| `bed_db` | the role's level over the music bed: the mix's loudest 50 ms at the effect (K-weighted, LUFS scale) this many dB over the bed's level, the median 50 ms K-weighted level of the music in the second before the cue; 0.5-30 | role `bed_db`; each take `l50_lufs` (its loudest 50 ms, K-weighted, at the file's level) |
+| `offset_ms` | where the take's sync point (its `align`) sits against the picture event: negative = the sound leads; -500..500 | role `offset_ms` |
+| `min_gap_ms` | the smallest spacing of two cues of the role in a run; 0-2000 | role `min_gap_ms` |
+| `width` | the L / R correlation every take is set to (1 = mono, 0 = decorrelated), by mid / side on the finished take: the side scaled down, or a decorrelated copy of the mid (three all-passes, above 250 Hz: the low end stays mono) added, solved to within 0.05; 0-1 | role `width`; each take `corr` (measured); `width` is part of the take's cache key |
+| `mix` (kit level) | information for the mixer: `{"level": "bed", "duck_db": 0, "stop_down": {"before": [roles], "depth_db": [6, 12], "length_s": [0.3, 0.5]}, ...}` | `mix` |
+
+A mixer turns them into placements with four helpers (`earshot.forge.kit`, numpy only):
+
+```python
+from earshot.forge import kit
+lock = kit.render('audio/kit.json', 'audio/kit')
+role = lock['roles']['pop']; take = role['takes'][k]
+bed = kit.bed_level(music, t_cue) # the music's median 50 ms K-weighted level over the second before t_cue
+gain = kit.gain_db(take, role, bed) # dB for this take on this cue (below); without bed_db: role level_db
+start = kit.place(take, role, t_cue) # file start: its align point lands offset_ms after the cue
+cues = kit.space_run(cue_times, role['min_gap_ms']) # a run's cues pushed apart to the gap, in time order
+y2, info = kit.stereo_width(y, 0.6) # any take to an L / R correlation (what `width` does at render)
+```
+
+`gain_db`: the reference over-bed levels were measured on the mix, the effect plus the bed under it, so the take
+alone goes to `bed + 10 log10(10 ** (bed_db / 10) - 1) - l50_lufs` (at 8.5 dB over the bed the take alone sits 7.8 dB
+over it; at 3 dB, level with it), never past -1 dBFS at the take's sample peak (a bed too loud for a short tick leaves
+the tick under its bed_db rather than clipping it). `kit.loud_lufs(y)` is the 50 ms measure on both sides,
+`kit.over_bed(bed_db)` the conversion. Both sides are measured as they play: a mono take or a mono (1-D) music array
+counts as the two identical channels a player makes of it, as a mono file does through `audio.load`. Where the
+music is silent or nearly so (under -60 LUFS, `kit.BED_FLOOR_LUFS`: before the music starts, a gap or a drop in the
+score, an end card after the track) `bed_level` returns `None` and `gain_db` falls back to the role's `level_db`
+(an effect set 8 dB over silence would be muted).
+
+The stop-down (`mix.stop_down`): in the reference clips 53 % of strong hits come after a dip of 6 dB or more in the
+0.5 s before them (9 % of random moments), and nothing ducks under the effects (the low band under bright effects
+does not dip). A mixer that follows it pulls the music down 6-12 dB for 0.3-0.5 s before a hit, a drop, a big whoosh
+or a swell, and brings it back on the hit.
## The presets
-`earshot kit init --preset calm|crisp|minimal` (`earshot/forge/kits/*.json`). All three sound every role except
-`riser`, `swell` and `suck` (calm also keeps `drop` off): a reveal resolves on one element, and one clean hit carries
-the lockup with nothing layered on it. Each role's `note` says what the sound is and what was measured. How the members
-were chosen:
+`earshot kit init --preset calm|crisp|minimal|launch` (`earshot/forge/kits/*.json`). calm, crisp and minimal sound
+every role except `riser`, `swell` and `suck` (calm also keeps `drop` off): a reveal resolves on one element, and one
+clean hit carries the lockup with nothing layered on it; rendered with 0.3.0 and with 0.4.0, their 191 takes are
+byte-identical. launch ([below](#the-launch-preset-04)) is the music-led film. Each role's `note` says what the sound
+is and what was measured. How the members of calm, crisp and minimal were chosen:
- **Every take passes the audit** (`earshot kit check`, all takes of all three presets); members the audit flags
(ringing keys, clicking airy ticks, rising zips) are out.
@@ -302,6 +409,43 @@ one pentatonic so a climbing progress motif finds a rung for each step; a tonal
degrees of one octave, so a chain of fragments can step through whichever of the chord's tones the takes hold. crisp
and minimal resolve text on an untuned soft tap (a new take per fragment), which never clashes with a note.
+## The launch preset (0.4)
+
+`earshot kit init --preset launch`: films cut to music, where effects ride 3-10 dB over the bed and read by timing
+and spectrum rather than level. Every number below is a median measured on the reference clips:
+`bed_db` from the effects' level over the music in the second before them, `offset_ms` from their signed offset to
+the nearest cut (sound minus picture), `min_gap_ms` from the spacing inside runs, `width` from their L / R
+correlation. `level_db` is the same balance for a mixer without a bed: the bed model on the takes at -1 dBFS (their
+measured `l50_lufs`), the loudest role at 0 dB.
+
+| role | sound | bed_db | offset_ms | min_gap_ms | width | level_db |
+|---|---|---|---|---|---|---|
+| tick / tick.crisp | pro_hover_tick/ping / pro_click/tink | 3 / 3 | 0 | 66 | 0.1 / 0.3 | 0 / -13 |
+| tock | knock/wood | 8 | -40 | 100 | - | -1.5 |
+| hit | cut_hit/punch | 7.5 | 0 | - | - | -9 |
+| land / land.light | pro_pop_in/round / tap/finger | 8 / 3 | +30 / -60 | 159 / 115 | - | -5.5 / -10 |
+| pop | ui_pop/soft | 8.5 | +4 | 114 | - | -8 |
+| whoosh.short / .long | whoosh_fast/crisp / whoosh_big/thump (peak) | 9 / 8.5 | +37 / +3 | - | - | -5.5 / -6.5 |
+| riser | riser/dark (peak) | 8 | -23 | - | - | -3.5 |
+| swell | reverse_swell/dark (peak) | 9.5 | +42 | - | - | -5.5 |
+| drop | cut_hit/kick | 7 | 0 | - | - | -8.5 |
+| key / .space / .return | typing_key/clicky / fit/key.space/thock_space / typing_key/clicky | 6.5 / 6.5 / 7 | +8 | 87 | - | -1 / -1.5 / -0.5 |
+| click / click.confirm | fit/click/floss_lid / notify/mid_tine | 6.5 / 8.5 | -11 / 0 | 85 | - | 0 / -8.5 |
+| snap | shutter/dslr | 4 | +9 | 66 | - | -7 |
+| bell | mallet/glock, C pentatonic G4-E6 (10 notes) | 7.5 | -20 | 110 | 0.6 | -8.5 |
+| text | mallet/pluck, C4 D4 E4 G4 A4 | 4 | 0 | 110 | - | -16 |
+| suck | off | | | | | |
+
+- **Restraint, the music way.** riser and swell are sounded (`kit check` notes it): about one a film, at a section
+ change, where the music would build anyway; suck stays off (a reverse whoosh into a cut stacks a lead-in on the
+ hit; the stop-down does that job). One gesture at a time; runs of UI sounds keep their gap with one accent at the
+ end (`mix.runs.accent_at_end`: click.confirm, land, drop).
+- **Ticks.** The reference ticks come from an effects-led mix only, so the ticks' `bed_db` is a choice: as low over
+ the bed as the light touches (3 dB), felt in the run rather than heard over the music.
+- **The stop-down** is in `mix`: depth 6-12 dB, 0.3-0.5 s before hit, drop, whoosh.long and swell, back on the hit;
+ `duck_db` 0.
+- **Checked.** All 72 takes pass the audit (three re-rolled on a new seed); the bell's ten notes land within 8 cents.
+
## Python and MCP
```python
@@ -314,7 +458,9 @@ from earshot import api # api.sfx_list / sfx_render / sfx_check / kit_init / k
```
MCP tools (`earshot mcp`): `sfx_list`, `sfx_render` (writes a WAV, returns the measured onset / peak / end and the
-audit) and `kit_render` (a kit.json into takes + lock). They load no models.
+audit) and `kit_render` (a kit.json into takes + lock). They load no models. The mix helpers (0.4):
+`kit.bed_level`, `kit.gain_db`, `kit.place`, `kit.space_run`, `kit.stereo_width`, `kit.loud_lufs`,
+`kit.over_bed` ([mix keys](#mix-keys-04)).
## Fit entries (from public-domain recordings)
@@ -369,7 +515,7 @@ seed (or kept with its warning when no re-roll is clean).
# 1. a CC0 / public-domain recording of ONE sound, trimmed close (a short fade-out at the cut is fine)
earshot fit door.wav -o made/door_fit.py --clap --takes --report # model made/door_fit.json + quality + floor
# 2. hear and audit takes the way the bank will render them
-python -c "from earshot.forge.fit import auto; from earshot.forge import sfx; import soundfile as sf; m = auto.load('made/door_fit.json'); [print(s, sfx.check(auto.render(m, seed=s, vary=0.3, take=True))['summary']) or sf.write(f'made/take{s}.wav', auto.render(m, seed=s, vary=0.3, take=True), 48000) for s in range(5)]"
+python -c "from earshot.forge.fit import auto; from earshot.forge import sfx; import soundfile as sf; m = auto.load('made/door_fit.json'); [print(s, sfx.check(auto.render(m, seed=s, vary=0.3, take=True, bed=False))['summary']) or sf.write(f'made/take{s}.wav', auto.render(m, seed=s, vary=0.3, take=True, bed=False), 48000) for s in range(5)]"
# 3. add it to a bank file of your own (or, in a checkout, the default earshot/forge/bank/fit.json)
python -m earshot.forge.bank add-fit made/door_fit.json --id fit/hit/door --source source.json --vary 0.3 --bank my_bank.json
# source.json: {"id": "freesound:123", "title": "...", "author": "...", "license": "CC0 1.0", "url": "https://..."}
@@ -378,7 +524,10 @@ EARSHOT_BANK=my_bank.json earshot sfx render fit/hit/door -o door_take.wav --see
- `--takes` scores the candidates on seeds 0-2 (a fit meant to make takes); `--clap` lets CLAP help choose (loads
CLAP, about 50 s). The report's `quality` against `floor` says how close the fit is (FIT.md, "Quality metrics").
-- Lower `--vary` (0.15, 0) when takes drift: listen to the takes, and run `earshot sfx check` on them.
+- Lower `--vary` (0.15, 0) when takes drift: listen to the takes, and run `earshot sfx check` on them. A fit with no
+ vary knob (an air-only gesture: `auto.knobs_of(model)` has none) is refused with `--vary` above 0.
+- Bank and kit takes are the sound alone: a fit of a recording whose bed was stripped (`post['bed']`, a steady
+ pre-roll under the sound) renders without that bed (`bed=False`), as step 2 does.
- The licence must be CC0 or public domain; `--allow-license` keeps an attribution licence (credit the source), and NC
or ND licences are always refused: a fit is a derivative of its recording and the bank ships in an MIT package.
- The entry keeps the model (its fit report dropped), seed, `vary`, the source, the measured note of a tonal fit
@@ -399,7 +548,7 @@ Measured on the development laptop (AMD Ryzen 7 260, 2 threads, while other jobs
| warm render / render + audit: pluck (3 s), impact (4.8 s) | 1.0-1.1 s / 1.4-2.2 s |
| a targeted whoosh (dur + apex: up to 3 renders + audit) | 1.9 s |
| render + audit, median over the 186 forge sounds and plucks (a cold process per batch) | 0.5 s |
-| `kit render`, whole preset, fresh process | calm 34 s (64 takes), crisp 22 s (64), minimal 9 s (63), with the fitted sounds (inside the process, numba kernels cached; the first run after an install adds the compile) |
+| `kit render`, whole preset, fresh process | calm 34 s (64 takes), crisp 22 s (64), minimal 9 s (63), with the fitted sounds (inside the process, numba kernels cached; the first run after an install adds the compile); launch 41 s (72 takes, 0.4.0, one thread) |
| `kit render`, everything cached | 3-5 s (the process start; 0.1-0.6 s inside it) |
Render a film's sounds in one process and keep the takes: the lock and its cache are made for that.
@@ -416,3 +565,11 @@ Render a film's sounds in one process and keep the takes: the lock and its cache
the edit. Ranges keep pitch inside each parameter's designed range.
- The whoosh engine picks its pan direction from the seed; a mixer mirrors a take for travel direction.
- No music and no ambience: the bank is effects and tuned notes.
+- The 0.4 members and the launch preset's numbers come from measurements on a private set of licensed reference clips
+ (profiles, offsets, levels, CLAP), with no listening test; the members were tuned toward the clips' measured style
+ and checked to stay originals (CLAP <= 0.96 to every clip, 0.84 at most), not copied from them.
+- The bed model is a median: the reference effects sit 3-10 dB over their bed with a wide spread per category
+ (pops 5.6-10.6 dB between quartiles), and the tick's level is a choice (no music-led reference ticks).
+- The four stereo sines fit entries (floss_lid, thock_space, switch_flip, bamboo_swish) keep their 0.3 fits and their
+ old, too quiet side: every preset that uses them would change its takes with a 0.4 refit, and those stay as they
+ were.
diff --git a/skills/earshot/SKILL.md b/skills/earshot/SKILL.md
index 837ac72..cd57115 100644
--- a/skills/earshot/SKILL.md
+++ b/skills/earshot/SKILL.md
@@ -1,6 +1,6 @@
---
name: earshot
-description: Listen with earshot whenever you must judge, choose, trim, verify or compare sound you cannot hear - picking a sound effect, music or ambience from candidates, finding the cleanest moment in a recording, checking whether audio reads as a real recording or as code-synthesized, reviewing a soundtrack or a video's audio (what happens when, sync to picture, loudness, mix), A/B-ing a mix change, or making a sound in code and improving it by listening (earshot forge). Use it before telling anyone how audio sounds.
+description: Listen with earshot whenever you must judge, choose, trim, verify or compare sound you cannot hear - picking a sound effect, music or ambience from candidates, finding the cleanest moment in a recording, checking whether audio reads as a real recording or as code-synthesized, reviewing a soundtrack or a video's audio (what happens when, sync to picture, loudness, mix), A/B-ing a mix change, making a sound in code and improving it by listening (earshot forge), or giving a code-made film its sound effects from the shipped bank (earshot sfx / kit). Use it before telling anyone how audio sounds.
---
# earshot: ears for an agent
@@ -144,26 +144,57 @@ one. The impact (darker, swell-led) and the click (short tails) were not re-made
Use this when a code-made film (a cinetic film, any cue sheet) needs its effects: render them from earshot's shipped
bank instead of writing synth voices. No models load; a whole kit renders in about a minute and re-renders in seconds.
1. Start from a preset: `earshot kit init --preset calm -o audio/kit.json` (calm for text-led films, crisp for
- energetic product films, minimal for designed effects-only films). Each role's `note` says why its sound was
- chosen; riser, swell and suck are off: a reveal resolves on one element, and the lockup is one clean hit.
+ energetic product films, minimal for designed effects-only films, launch for films cut to music). Each role's
+ `note` says why its sound was chosen. calm / crisp / minimal keep riser, swell and suck off: a reveal resolves on
+ one element, and the lockup is one clean hit. launch sounds one riser or reverse swell at a section change and
+ uses the 0.4 sounds made for music-led films (below).
2. Fit it to the film: `earshot sfx list --kind hit` (or `--category`) shows the candidates with lengths, peak times
and knobs. Change a role's `sound`, set `edits` (`pitch` to the film's root for the hit or drop, `dur` / `apex` for
a fixed move), `takes` (12 or more for typing, 2-4 elsewhere), `level_db`. Audition one candidate with
`earshot sfx render ID -o x.wav [--pitch E5] [--dur 0.8 --apex 0.32]` and read its measured onset / peak / end.
3. Render: `earshot kit render audio/kit.json --out audio/kit`. The takes and `kit.lock.json` (measured onset, peak,
end, loudness per take) are what the mixer places on cue frames: onset on the frame for hits and touches, the
- peak on a whoosh's apex, the end on a riser's hit.
+ peak on a whoosh's apex, the end on a riser's hit (its peak when the role aligns on `peak`, as launch's riser and
+ swell do: those members fall after their climax; `kit check` notes an end-aligned one).
4. Check: `earshot kit check audio/kit.json --out audio/kit`. It refuses what a render cannot use (a knob or pitch
the sound cannot take, a level over 0 dBFS, two takes that would share a file). Every warning names a take and a
reason (clicks, a cut tail, comb ring, silence, a pitch that climbs more than 4 semitones, a hit with two attacks
or a build before it, `pitch_off`: a note the knob did not land); swap the sound or the knob rather than keeping a
warned take. Read the notes too (a sounded riser, a composite sting).
5. Listen through the finished film with Loop 2 (`timeline --cues`, `sheet`) and check that each cue landed.
-Say plainly what the bank is: forge sounds made in code, and 15 sounds fitted as code from CC0 recordings
-(`fit/<role>/<name>`, each with its source; the presets use them for clicks, ticks, keys, touches and the snap), all
-chosen and checked by measurement. They are not recordings: when the brief asks for recorded sound, use
-`skills/real-sound` instead. To add a fit of your own (`earshot fit --takes`, then `bank add-fit`), see `docs/SFX.md`
-("Add your own fit").
+
+**The 0.4 families** (`earshot sfx list --category ui_pop`, ...), for the soft, short, mid-register sound of
+music-led launch films: `ui_pop` (rounded mid pops for chips, bubbles and rows arriving: the most used effect in
+those films), `knock` (mid-band wood and plastic), `cut_hit` (short sub hits with a kick, a punchy cut hit),
+`mallet` (gated glockenspiel / celesta / marimba notes, a soft pluck, chime runs; `--pitch` tunes them), `drip`
+(water drops whose pitch rises: the audit allows a bubble its climb), `shutter` (snap zooms, whip-cuts),
+`reverse_swell` (peaks just after a cut, one a film), plus whoosh_big thump*, riser dark / build / snatch, swish
+bright / flick, whoosh_fast whip, glitch burst / mid / chirp and notify mid_tine / mid_wood. Prefer them over the
+bright 5 ms blips and long cinematic tails of the older categories when the film is cut to music.
+
+**Levels against the music, not fixed** (launch, or any role with `bed_db`). In films cut to music effects ride 3-10
+dB over the music and read by timing and spectrum: a fixed `level_db` is either buried in loud passages or sticks
+out in quiet ones. For each cue, measure the bed and set the take from the lock:
+```python
+from earshot.forge import kit
+lock = kit.render('audio/kit.json', 'audio/kit') # takes carry l50_lufs; roles bed_db, offset_ms
+role = lock['roles']['pop']; take = role['takes'][i]
+bed = kit.bed_level(music, t_cue) # the music's median 50 ms level, second before
+gain = kit.gain_db(take, role, bed) # dB on the take file (never over -1 dBFS peak)
+start = kit.place(take, role, t_cue) # taps / knocks / clicks lead, whoosh peaks land after
+cues = kit.space_run(run_times, role['min_gap_ms']) # a run keeps its gap (pops 114 ms, keys 87)
+```
+Pull the music down 6-12 dB for 0.3-0.5 s before a hit, a drop, a big whoosh or a swell and bring it back on the hit
+(the lock's `mix.stop_down`); do not duck the music under every effect. Without a music bed (an effects-led film)
+`level_db` is the fallback; where the music is silent before a cue (a cold open, a gap in the score) `bed_level`
+returns `None` and `gain_db` uses `level_db`. Report the gains you set and the bed levels you measured.
+
+Say plainly what the bank is: forge sounds made in code (0.4's tuned toward the measured style of a private set of
+licensed reference clips, and checked to stay originals: CLAP 0.84 at most against any of them), and 15 sounds fitted
+as code from CC0 recordings (`fit/<role>/<name>`, each with its source; the presets use them for clicks, ticks, keys,
+touches and the snap), all chosen and checked by measurement, none by a listening test. They are not recordings: when
+the brief asks for recorded sound, use `skills/real-sound` instead. To add a fit of your own (`earshot fit --takes`,
+then `bank add-fit`), see `docs/SFX.md` ("Add your own fit").
## Optional: a second opinion in words (audio LLM)
`earshot llm mix.wav --yes-upload [--with-timeline] [--prompt "Is the riser at 0:12 cheap?"]` asks Google Gemini for