Skip to content

Ears for coding agents: let AI hear, review and make sound. CPU-only, open source.

forge v2: clean blind test kit + v1 -> v2 progress builder (eval/forge/blind_v2.py)

Leonxlnx40f03e150073
diff --git a/eval/forge/blind_v2.py b/eval/forge/blind_v2.py
new file mode 100644
index 0000000..fdc8091
--- /dev/null
+++ b/eval/forge/blind_v2.py
@@ -0,0 +1,759 @@
+"""Clean blind test (forge v2 finals vs held-out REAL recordings) + the v1 -> v2 progress listening file.
+
+ E:/earshot/.venv/Scripts/python.exe eval/forge/blind_v2.py --out out/showcase/blind_v2 --seed 2026
+ E:/earshot/.venv/Scripts/python.exe out/showcase/blind_v2/score_answers.py A B B A ... (after listening)
+
+One command rebuilds both from whatever is in out/forge_v2/<category>/final/ now (rerun it after a polish round).
+Nothing is synthesised here: every sound is a file that already exists (forge finals, v1 finals, licensed bench refs),
+only trimmed, faded, gain-changed and placed on silence. No markers, beeps or voice: trials are separated by silence.
+
+BLIND TEST (--out, default out/showcase/blind_v2)
+ Trials: N (default 20) over the licensed real categories that have bench refs (PLAN: two trials for the strongest
+ categories of the round-3 verdicts, one for the others, including the weak ones so the test is honest). Each trial
+ pairs one forge v2 final (<category>_NN.wav in out/forge_v2/<category>/final/) with one REAL recording of the same
+ category from the held-out BENCH split (earshot.forge.refsets: source recordings no forge search ever used).
+ Picks: the made final is drawn at random (seeded) among the category's length-compatible finals (audible extent
+ within 1.3x of an available bench ref; if too few, the closest in length), never by a score; the real one is the
+ bench ref whose audible extent is closest to it, with no real file and, where possible, no source recording used
+ twice in the test (whoosh_fast / swish and notify / logo_sting share bench files). Duration is the only property
+ either pick looks at, so the trims below cut little.
+ Trim: A and B get the SAME length. Usually both start 5 ms before their onset (first 1 ms frame within 40 dB of the
+ loudest; zero-padded if the file has no 5 ms lead) and run for the shorter of the two files, at most the longer
+ audible extent (last 5 ms frame within 50 dB of the loudest) + 30 ms; a clip cut before its file ends gets a
+ raised-cosine fade-out (15 % of the length, 10-300 ms), a clip that ends with its file keeps its own ending, both
+ get a 1 ms fade-in. Pairs that both build to their end (risers: loudest frame in the last 30 %) are end-aligned
+ instead: both end with their files and the longer one loses its quiet start under a fade-in of the same size.
+ Level: the made sound is matched to the real one. Short pairs (< 1 s): half the sample-peak difference plus half
+ the loudest-100-ms RMS difference, the RMS K-weighted (BS.1770 pre-filter: unweighted, a real whoosh's low rumble
+ counted as loud and its made partner played 5-6 LU louder). Long pairs: half the integrated-LUFS difference
+ (BS.1770, gated, of the equal-length clips) plus half the momentary-max-LUFS difference (integrated alone let a made
+ hit with a shorter tail play 3-4 LU louder at its head). What is left is half the two sounds' crest difference
+ (about +-2 LU momentary). Then one common gain for the pair: the louder momentary-max LUFS at -16 LUFS, never a peak
+ above -1 dBFS.
+ Order: trials shuffled (no category twice in a row), real-as-A on exactly half the trials, both from --seed.
+ Layout: 1 s lead, A, 1.2 s silence, B, 2.5 s silence, next trial ... 1 s tail; 48 kHz, 24-bit stereo.
+ Writes: blind_test.wav, answer_sheet.txt (trial, category, A start, B start, blank), KEY_open_after.json (the
+ answers, picks, trims, gains and levels), score_answers.py (per trial right / wrong, per category, per tier,
+ overall, one-sided binomial p), blind_test_trials/tNN_A.wav + tNN_B.wav (each trial on its own, same levels).
+ The key stores the md5 of every file used. If a final or bench ref changes while this runs (a polish round writing
+ finals), it stops before writing anything, so the folder always holds one complete, consistent build.
+
+PROGRESS (--progress, default <out>/../progress)
+ v1_vs_v2.wav: the five sounds v1 had (key press, UI click, pop, fast whoosh, impact): v1 final, v2 final, v1 final,
+ v2 final, each peak-matched to -1 dBFS (v1 = demo/forge-test/<sound>/final/<sound>_01 and _02; v2 = the v2 final
+ closest to that v1 final in 1/3-octave balance and audible length, so like is played against like). progress.txt
+ lists every start time. progress.md: the measured before / after numbers from eval/forge/results_v1.json against the
+ newest eval/forge/results_v2r<N>.json (both from the same frozen held-out benchmark, eval/forge/bench.py).
+"""
+import argparse
+import glob
+import json
+import os
+import re
+import sys
+import time
+
+import numpy as np
+import soundfile as sf
+
+HERE = os.path.dirname(os.path.abspath(__file__))
+REPO = os.path.dirname(os.path.dirname(HERE)).replace(chr(92), '/')
+for _p in (REPO, HERE):
+ if _p not in sys.path:
+ sys.path.insert(0, _p)
+import frozen_tone # noqa: E402 (1/3-octave profile, the benchmark's own frozen copy)
+from earshot import features # noqa: E402 (BS.1770 integrated LUFS)
+from earshot.forge import refsets # noqa: E402
+
+SR = 48000
+MADE = f'{REPO}/out/forge_v2'
+PEAK_MAX_DB = -1.0
+PAIR_LUFS_M = -16.0 # the louder sound of a pair at this momentary-max loudness (unless the peak cap is lower)
+PRE, FADE_IN = 0.005, 0.001
+LEAD, GAP, BETWEEN, TAIL = 1.0, 1.2, 2.5, 1.0
+SHORT_S = 1.0
+COMPAT = 1.3 # made finals whose audible extent is within this factor of an available bench ref
+
+# (category, answer-sheet name, trials at N = 20, tier from the round-3 verdicts in OVERNIGHT.md / results_v2r3.md)
+PLAN = [
+ ('whoosh_big', 'big whoosh', 2, 'strong'),
+ ('whoosh_fast', 'fast whoosh', 2, 'strong'),
+ ('click', 'UI click', 2, 'strong'),
+ ('typing_key', 'key press', 2, 'strong'),
+ ('shimmer', 'shimmer', 2, 'strong'),
+ ('pop', 'pop', 2, 'strong'),
+ ('swipe', 'cloth swipe', 1, 'strong'),
+ ('swish', 'swish', 1, 'strong'),
+ ('impact', 'impact', 1, 'strong'),
+ ('hover_tick', 'hover tick', 1, 'middle'),
+ ('tap', 'tap', 1, 'weak'),
+ ('toggle', 'toggle', 1, 'weak'),
+ ('notify', 'notification ding', 1, 'weak'),
+ ('riser', 'riser', 1, 'weak'),
+]
+EXCLUDED = {
+ 'logo_sting': 'its bench refs are a triangle hit and a struck metal ring (the very files of the notify bench): the '
+ 'library has no real logo sting, so a pair would test the form (a composed reveal vs one ring), not '
+ 'realism',
+ 'downlifter': 'no real or designed downlifter in the library (no bench refs)',
+ 'glitch': 'no glitch material in the library (no bench refs)',
+ 'sub_drop': 'one bench ref, a designed tom pitched down 24 semitones: not a recording of a real event',
+ 'pro_*': 'the pro refs are trims of launch films, most under music / voice beds: no clean A/B',
+}
+NOTES = {
+ 'impact': 'the only impact bench ref is a library cinematic hit (designed from recordings by a person)',
+ 'riser': 'the riser bench refs are real cymbal swells; the made risers are designed (synth / noise) risers',
+ 'swish': 'the swish bench refs are the same files as three whoosh_fast bench refs (used once each here)',
+}
+V1 = [ # (answer name, v2 category, v1 final folder, v1 file stem)
+ ('key press', 'typing_key', 'demo/forge-test/keypress/final', 'keypress'),
+ ('UI click', 'click', 'demo/forge-test/click/final', 'click'),
+ ('pop', 'pop', 'demo/forge-test/pop/final', 'pop'),
+ ('fast whoosh', 'whoosh_fast', 'demo/forge-test/whoosh/final', 'whoosh'),
+ ('impact', 'impact', 'demo/forge-test/impact/final', 'impact'),
+]
+
+
+# ----------------------------------------------------------------------------------------------------------- audio
+def rel(p):
+ p = os.path.abspath(p).replace(chr(92), '/')
+ return p[len(REPO) + 1:] if p.lower().startswith(REPO.lower() + '/') else p
+
+
+def load(p):
+ x, sr = sf.read(p, always_2d=True, dtype='float64')
+ if sr != SR:
+ from math import gcd
+ from scipy import signal
+ g = gcd(int(sr), SR)
+ x = signal.resample_poly(x, SR // g, int(sr) // g, axis=0)
+ if x.shape[1] == 1:
+ x = np.repeat(x, 2, axis=1)
+ return x[:, :2]
+
+
+def frames_db(x, ms):
+ p = (x ** 2).mean(axis=1)
+ n = max(1, int(SR * ms / 1000))
+ k = len(p) // n
+ if k < 1:
+ return np.array([10 * np.log10(p.mean() + 1e-20)]), n
+ return 10 * np.log10(p[:k * n].reshape(k, n).mean(axis=1) + 1e-20), n
+
+
+def onset(x, below=40.0):
+ e, n = frames_db(x, 1.0)
+ return int(np.nonzero(e >= e.max() - below)[0][0] * n)
+
+
+def end_of(x, below=50.0):
+ e, n = frames_db(x, 5.0)
+ return int((np.nonzero(e >= e.max() - below)[0][-1] + 1) * n)
+
+
+def extent_s(x):
+ return (end_of(x) - onset(x)) / SR
+
+
+def peak_db(x):
+ return float(20 * np.log10(np.abs(x).max() + 1e-12))
+
+
+def rms100_db(x, kw=False):
+ """RMS of the loudest 100 ms (the whole sound over 100 ms when shorter); kw=True: K-weighted first (BS.1770
+ pre-filter), so low rumble does not count as loud"""
+ if kw:
+ x = features._kw(np.concatenate([np.zeros((int(0.05 * SR), 2)), x]))[int(0.05 * SR):]
+ m = (x ** 2).mean(axis=1)
+ w = int(0.1 * SR)
+ c = np.convolve(m, np.ones(w) / w, mode='valid') if len(m) > w else np.array([m.sum() / w])
+ return float(10 * np.log10(c.max() + 1e-20))
+
+
+def lufs_int(x):
+ n = int(0.4 * SR)
+ if len(x) < n:
+ x = np.concatenate([x, np.zeros((n - len(x), 2))])
+ return float(features.integrated_lufs(x))
+
+
+def lufs_max(x):
+ """momentary-max loudness (K-weighted 400 ms windows, zero-padded when shorter), as a meter shows it"""
+ t, L = features.loudness_curve(x if len(x) >= int(0.4 * SR) else np.concatenate(
+ [x, np.zeros((int(0.4 * SR) - len(x), 2))]), 0.4, 0.01)
+ return float(np.max(L))
+
+
+def raised(n):
+ return 0.5 - 0.5 * np.cos(np.pi * np.arange(n) / max(1, n))
+
+
+def peak_late(x):
+ """riser-like: the loudest 5 ms frame lies in the last 30 % of the audible extent (it builds to its end)"""
+ e, n = frames_db(x, 5.0)
+ o, end = onset(x), end_of(x)
+ return int(np.argmax(e)) * n >= o + 0.7 * (end - o)
+
+
+def level_at(x, i):
+ """level (dB re the loudest 5 ms frame) of the 5 ms frame at sample i"""
+ e, n = frames_db(x, 5.0)
+ return round(float(e[max(0, min(len(e) - 1, i // n))] - e.max()), 1)
+
+
+def trim_pair(a, b):
+ """one common length for both clips (see the module doc); returns (a', b', length, align, cut levels).
+ 'start' alignment: both begin PRE before their onset; a clip that has to stop before its file ends gets a fade-out
+ (15 % of the length, 10-300 ms), a clip that ends with its file keeps its own ending. 'end' alignment (both build
+ to their end, risers): both end where their files end; the longer one loses its quiet start under a fade-in."""
+ pre = int(PRE * SR)
+ fo = lambda L: int(np.clip(0.15 * L, 0.010 * SR, 0.300 * SR))
+ if peak_late(a) and peak_late(b):
+ starts = [max(0, onset(x) - pre) for x in (a, b)]
+ L = int(min(len(a) - starts[0], len(b) - starts[1]))
+ out, cut = [], []
+ for x in (a, b):
+ s = len(x) - L
+ y = x[s:].copy()
+ n_in = fo(L) if s > max(0, onset(x) - pre) + int(0.001 * SR) else int(FADE_IN * SR)
+ cut.append(level_at(x, s) if n_in > int(FADE_IN * SR) else None)
+ y[:n_in] *= raised(n_in)[:, None]
+ out.append(y)
+ return out[0], out[1], L, 'end', cut
+ starts, avail, ext = [], [], []
+ for x in (a, b):
+ s = onset(x) - pre
+ starts.append(s)
+ avail.append(len(x) - s)
+ ext.append(end_of(x) - s)
+ L = int(min(min(avail), max(ext) + 0.03 * SR))
+ out, cut = [], []
+ for x, s, av in zip((a, b), starts, avail):
+ y = np.concatenate([np.zeros((-s, 2)), x[:s + L]]) if s < 0 else x[s:s + L].copy()
+ y = y[:L]
+ fi = int(FADE_IN * SR)
+ y[:fi] *= raised(fi)[:, None]
+ if L < av - int(0.001 * SR): # truncated: fade out (and say how loud it was there)
+ f = fo(L)
+ cut.append(level_at(x, max(0, s) + L - f))
+ y[L - f:] *= raised(f)[::-1][:, None]
+ else:
+ cut.append(None)
+ out.append(y)
+ return out[0], out[1], L, 'start', cut
+
+
+def match_pair(real, made):
+ """gain (dB) on the made clip so it plays as loud as the real one, the method name, and the two differences"""
+ if len(real) / SR < SHORT_S:
+ dp, dr = peak_db(real) - peak_db(made), rms100_db(real, kw=True) - rms100_db(made, kw=True)
+ return 0.5 * (dp + dr), 'short: mean of sample peak and K-weighted loudest-100-ms RMS', {
+ 'peak_diff_db': dp, 'rms100_kw_diff_db': dr}
+ di, dm = lufs_int(real) - lufs_int(made), lufs_max(real) - lufs_max(made)
+ return 0.5 * (di + dm), 'long: mean of integrated and momentary-max LUFS', {'lufs_i_diff_db': di,
+ 'lufs_m_max_diff_db': dm}
+
+
+def levels(x):
+ return {'peak_dbfs': round(peak_db(x), 2), 'rms100_dbfs': round(rms100_db(x), 2),
+ 'rms100_kw_db': round(rms100_db(x, kw=True), 2), 'lufs_i': round(lufs_int(x), 2),
+ 'lufs_m_max': round(lufs_max(x), 2)}
+
+
+def write(path, y):
+ os.makedirs(os.path.dirname(path), exist_ok=True)
+ assert np.isfinite(y).all(), path
+ sf.write(path, y, SR, subtype='PCM_24')
+
+
+def place(parts, total_s):
+ y = np.zeros((int(round(total_s * SR)) + 1, 2))
+ for t0, x in parts:
+ i = int(round(t0 * SR))
+ y[i:i + len(x)] += x
+ return y
+
+
+# ------------------------------------------------------------------------------------------------------ the picks
+def finals(cat):
+ return sorted(glob.glob(f'{MADE}/{cat}/final/{cat}_[0-9][0-9].wav'))
+
+
+def source_of(p):
+ """the source recording of a bench trim: 'fs-463763_2.wav' -> 'fs-463763' (sister trims share a source)"""
+ return re.sub(r'_\d+$', '', os.path.splitext(os.path.basename(p))[0])
+
+
+def allocate(n):
+ alloc = {c: 0 for c, *_ in PLAN}
+ left = n
+ for want in (1, 2):
+ for c, _, k, _ in PLAN:
+ if left and alloc[c] < min(want, k):
+ alloc[c] += 1
+ left -= 1
+ while left: # more than the plan: round-robin over the plan again
+ grew = False
+ for c, *_ in PLAN:
+ if left and alloc[c] < min(len(finals(c)), len(refsets.refset(c, 'bench'))):
+ alloc[c] += 1
+ left -= 1
+ grew = True
+ if not grew:
+ sys.exit(f'cannot build {n} trials: not enough finals / bench refs')
+ return {c: k for c, k in alloc.items() if k}
+
+
+def pick_trials(n, seed):
+ import hashlib
+ alloc = allocate(n)
+ rng_made = np.random.default_rng([seed, 1])
+ info = {c: (name, tier) for c, name, _, tier in PLAN}
+ bench = {c: refsets.refset(c, 'bench') for c in alloc}
+ digest = {}
+ for c in alloc:
+ for p in bench[c]:
+ digest[p] = hashlib.md5(load(p).astype(np.float32).tobytes()).hexdigest()
+ # most constrained category first: fewest distinct sources per trial
+ order = sorted(alloc, key=lambda c: (len({source_of(p) for p in bench[c]}) / alloc[c], c))
+ used_hash, used_src, trials = set(), set(), []
+ ext_cache = {}
+
+ def ext(p):
+ if p not in ext_cache:
+ ext_cache[p] = extent_s(load(p))
+ return ext_cache[p]
+
+ def available(c):
+ cands = [p for p in bench[c] if digest[p] not in used_hash and source_of(p) not in used_src]
+ if cands:
+ return cands, False
+ return [p for p in bench[c] if digest[p] not in used_hash], True
+
+ for c in order:
+ fin = finals(c)
+ if len(fin) < alloc[c]:
+ sys.exit(f'{c}: {len(fin)} finals, {alloc[c]} trials wanted')
+ # random among the length-compatible finals (audible extent within COMPAT of an available bench ref);
+ # if fewer than needed, the finals closest in length. Duration is the only property this pick looks at.
+ cands0, _ = available(c)
+ gap = {m: min(abs(np.log(ext(p) / ext(m))) for p in cands0) for m in fin}
+ pool = [m for m in fin if gap[m] <= np.log(COMPAT)]
+ if len(pool) < alloc[c]:
+ pool = sorted(fin, key=lambda m: (gap[m], m))[:alloc[c]]
+ made = [pool[i] for i in sorted(rng_made.choice(len(pool), alloc[c], replace=False))]
+ for m in made:
+ cands, relaxed = available(c)
+ if not cands:
+ sys.exit(f'{c}: no unused bench ref left')
+ em = ext(m)
+ real = min(cands, key=lambda p: (abs(np.log(ext(p) / em)), p))
+ used_hash.add(digest[real])
+ used_src.add(source_of(real))
+ trials.append({'category': c, 'sound': info[c][0], 'tier': info[c][1], 'made': m, 'real': real,
+ 'source_reused': relaxed, 'pool': [os.path.basename(q) for q in pool]})
+ # order: shuffled, no category twice in a row (deterministic from the seed)
+ rng_order = np.random.default_rng([seed, 2])
+ for _ in range(5000):
+ perm = rng_order.permutation(len(trials))
+ if all(trials[perm[i]]['category'] != trials[perm[i + 1]]['category'] for i in range(len(perm) - 1)):
+ break
+ trials = [trials[i] for i in perm]
+ rng_ab = np.random.default_rng([seed, 3])
+ ab = np.array(['A'] * (len(trials) // 2) + ['B'] * (len(trials) - len(trials) // 2))
+ if len(trials) % 2 and rng_ab.integers(0, 2):
+ ab[-1] = 'A'
+ ab = ab[rng_ab.permutation(len(ab))]
+ for t, r in zip(trials, ab):
+ t['real_is'] = str(r)
+ return trials
+
+
+# --------------------------------------------------------------------------------------------------- blind build
+SCORE_PY = r'''"""Score the blind test: python score_answers.py A B B A ... (one letter per trial, in trial order: which one
+was the REAL recording). Use ? for a trial you could not decide (it is left out of the count).
+
+Prints per trial right / wrong, per sound, per tier (the round-3 verdict of the category: strong / middle / weak) and
+overall, with the chance of doing at least that well by pure guessing (one-sided binomial p). Reads
+KEY_open_after.json next to this file.
+"""
+import json
+import os
+import sys
+from math import comb
+
+
+def p_at_least(k, n):
+ return sum(comb(n, i) for i in range(k, n + 1)) / 2 ** n if n else 1.0
+
+
+def p_at_most(k, n):
+ return sum(comb(n, i) for i in range(0, k + 1)) / 2 ** n if n else 1.0
+
+
+def main():
+ here = os.path.dirname(os.path.abspath(__file__))
+ key = json.load(open(os.path.join(here, 'KEY_open_after.json'), encoding='utf-8'))['trials']
+ ans = [a.strip().upper() for a in ' '.join(sys.argv[1:]).replace(',', ' ').split()]
+ if len(ans) != len(key) or any(a not in ('A', 'B', '?') for a in ans):
+ sys.exit(f'give {len(key)} answers, each A or B (which one was the REAL recording) or ? (undecided), in trial order')
+ per, tier = {}, {}
+ for k, a in zip(key, ans):
+ ok = None if a == '?' else a == k['real']
+ mark = 'skipped' if ok is None else ('right' if ok else 'wrong')
+ print(f"trial {k['trial']:2d} {k['sound']:<18} you: {a} real: {k['real']} {mark:<7} "
+ f"(real {k['real_file'].rsplit('/', 1)[-1]}, made {k['made_file'].rsplit('/', 1)[-1]})")
+ if ok is not None:
+ per.setdefault(k['sound'], []).append(ok)
+ tier.setdefault(k['tier'], []).append(ok)
+ print()
+ for s, v in per.items():
+ print(f'{s:<18} {sum(v)} of {len(v)} right')
+ print()
+ for s in ('strong', 'middle', 'weak'):
+ if s in tier:
+ v = tier[s]
+ print(f'{s + " categories":<18} {sum(v)} of {len(v)} right (p = {p_at_least(sum(v), len(v)):.3f})')
+ n = sum(len(v) for v in per.values())
+ right = sum(sum(v) for v in per.values())
+ print()
+ print(f'overall {right} of {n} right; chance of doing at least this well by guessing: p = {p_at_least(right, n):.3f}')
+ print('p < 0.05: you can tell made from real (at least for some sounds). Otherwise this test could not show it.')
+ if n and p_at_most(right, n) < 0.05:
+ print(f'note: this few right is itself unlikely by guessing (p = {p_at_most(right, n):.3f}): you heard the made '
+ 'sounds as the real ones more often than not.')
+
+
+if __name__ == '__main__':
+ main()
+'''
+
+
+def fingerprint(paths):
+ """{path: (size, mtime_ns)}: to notice a final being rewritten (a polish round) while this builds"""
+ out = {}
+ for p in paths:
+ try:
+ st = os.stat(p)
+ out[p] = (st.st_size, st.st_mtime_ns)
+ except OSError:
+ out[p] = None
+ return out
+
+
+def md5(p):
+ import hashlib
+ return hashlib.md5(open(p, 'rb').read()).hexdigest()
+
+
+def build_blind(out, n, seed, quiet=False):
+ inputs = sorted({q for c, *_ in PLAN for q in finals(c) + refsets.refset(c, 'bench')})
+ before = fingerprint(inputs)
+ trials = pick_trials(n, seed)
+ tdir = os.path.join(out, 'blind_test_trials')
+ parts, key, files, t = [], [], [], LEAD
+ for i, tr in enumerate(trials, 1):
+ real0, made0 = load(tr['real']), load(tr['made'])
+ real, made, L, align, cut = trim_pair(real0, made0)
+ g, how, diff = match_pair(real, made)
+ made = made * 10 ** (g / 20)
+ loud = max(lufs_max(real), lufs_max(made))
+ gc = min(PEAK_MAX_DB - max(peak_db(real), peak_db(made)), PAIR_LUFS_M - loud)
+ real, made = real * 10 ** (gc / 20), made * 10 ** (gc / 20)
+ A, B = (real, made) if tr['real_is'] == 'A' else (made, real)
+ slot = L / SR
+ ta = round(t, 2) # on the 10 ms grid: the sheet times are exact
+ tb = round(ta + slot + GAP, 2)
+ parts += [(ta, A), (tb, B)]
+ pad = np.zeros((int(0.25 * SR), 2))
+ files += [(os.path.join(tdir, f't{i:02d}_A.wav'), np.concatenate([pad, A, pad])),
+ (os.path.join(tdir, f't{i:02d}_B.wav'), np.concatenate([pad, B, pad]))]
+ lr, lm = levels(real), levels(made)
+ key.append({
+ 'trial': i, 'category': tr['category'], 'sound': tr['sound'], 'tier': tr['tier'],
+ 'A_start': round(ta, 2), 'B_start': round(tb, 2), 'length_s': round(slot, 3), 'real': tr['real_is'],
+ 'real_file': rel(tr['real']), 'made_file': rel(tr['made']),
+ 'real_source_reused_elsewhere': tr['source_reused'],
+ 'file_length_s': {'real': round(len(real0) / SR, 3), 'made': round(len(made0) / SR, 3)},
+ 'audible_extent_s': {'real': round(extent_s(real0), 3), 'made': round(extent_s(made0), 3)},
+ 'align': align,
+ 'cut_at_db': {'real': cut[0], 'made': cut[1]}, # level where a truncated clip's fade begins (None: kept)
+ 'made_drawn_from': tr['pool'],
+ 'match': how, 'made_gain_db': round(g, 2), 'pair_gain_db': round(gc, 2),
+ 'differences_before_match_db': {k: round(v, 2) for k, v in diff.items()},
+ 'levels_as_played': {'real': lr, 'made': lm},
+ 'note': NOTES.get(tr['category'], ''),
+ 'md5': {'real': md5(tr['real']), 'made': md5(tr['made'])},
+ })
+ t = round(tb + slot + BETWEEN, 2)
+ now = fingerprint(inputs)
+ if now != before: # nothing is written: the folder keeps the previous complete build
+ changed = [rel(q) for q in inputs if now.get(q) != before.get(q)]
+ sys.exit(f'input files changed while building ({", ".join(changed[:6])}): a polish round is writing finals; '
+ 'run this again when it is done (nothing written)')
+ total = t - BETWEEN + TAIL
+ y = place(parts, total)
+ assert np.abs(y).max() <= 10 ** (PEAK_MAX_DB / 20) + 1e-6
+ os.makedirs(tdir, exist_ok=True)
+ for old in glob.glob(os.path.join(tdir, 't*_[AB].wav')):
+ os.remove(old)
+ for f, x in files:
+ write(f, x)
+ write(os.path.join(out, 'blind_test.wav'), y)
+ json.dump({'seed': seed, 'built': time.strftime('%Y-%m-%d %H:%M'), 'trials': key,
+ 'excluded_categories': EXCLUDED,
+ 'how': 'built by eval/forge/blind_v2.py (see its docstring): forge v2 finals (random pick among the '
+ 'length-compatible ones) vs held-out bench refs (closest audible length, no file / source '
+ 'twice), equal-length trims (fade only where a clip is cut), made matched to real in level, '
+ 'random order and A/B (real = A on half the trials)'},
+ open(os.path.join(out, 'KEY_open_after.json'), 'w', encoding='utf-8'), indent=1)
+ open(os.path.join(out, 'score_answers.py'), 'w', encoding='utf-8').write(SCORE_PY)
+ from math import comb
+ nt = len(key)
+ pk = lambda k: sum(comb(nt, j) for j in range(k, nt + 1)) / 2 ** nt
+ k05 = next(k for k in range(nt + 1) if pk(k) < 0.05)
+ cats = sorted({k['category'] for k in key})
+ L = ['Blind test: blind_test.wav (or blind_test_trials/tNN_A.wav + tNN_B.wav, one trial at a time).',
+ 'In every trial you hear A, then B. One is a REAL recording, the other was made entirely in code by earshot',
+ 'forge. Write down which one you think is REAL (A or B; ? if you cannot decide). Do not open KEY_open_after.json',
+ 'first. The two sounds of a trial are matched in loudness and trimmed to the same length (short fades),',
+ 'and the real recordings come from a held-out set the forge never tuned against. Silence between trials, no',
+ 'markers: each trial starts at the times below.',
+ f'Then: .venv\\Scripts\\python.exe {rel(out).replace("/", chr(92))}\\score_answers.py A B ... '
+ f'({nt} letters, in trial order)',
+ f'By chance you would get about {nt // 2} of {nt} right; {k05} or more is unlikely by luck '
+ f'(p = {pk(k05):.3f}).', '',
+ f"{'trial':>5} {'sound':<18} {'A starts':>8} {'B starts':>8} real is (A/B)"]
+ for k in key:
+ L.append(f"{k['trial']:>5} {k['sound']:<18} {k['A_start']:8.2f} {k['B_start']:8.2f} ____")
+ L += ['', f'total length {len(y) / SR:.1f} s ({len(y) / SR / 60:.1f} min), 48 kHz, 24-bit stereo; '
+ f'{nt} trials over {len(cats)} sounds']
+ open(os.path.join(out, 'answer_sheet.txt'), 'w', encoding='utf-8').write(chr(10).join(L) + chr(10))
+ if not quiet:
+ print(chr(10).join(L))
+ return key
+
+
+# ------------------------------------------------------------------------------------------------ progress build
+def results_v2(path=None):
+ if path:
+ return path
+ found = []
+ for p in glob.glob(f'{HERE}/results_v2r*.json'):
+ m = re.match(r'results_v2r(\d+)\.json$', os.path.basename(p))
+ if m:
+ found.append((int(m.group(1)), p))
+ if not found:
+ sys.exit('no eval/forge/results_v2r<N>.json')
+ return max(found)[1]
+
+
+def nearest_v2(v1_file, cands):
+ a = load(v1_file)
+ pa, ea = frozen_tone.profile(a), extent_s(a)
+ best = None
+ for c in cands:
+ b = load(c)
+ d = float(np.sqrt(np.mean((frozen_tone.profile(b) - pa) ** 2))) + 6.0 * abs(np.log2(extent_s(b) / ea))
+ if best is None or d < best[0]:
+ best = (d, c)
+ return best
+
+
+def fmt(v, nd=2):
+ if v is None:
+ return '-'
+ if isinstance(v, float):
+ return f'{v:.{nd}f}'
+ return str(v)
+
+
+# (key in summary, label, rule, eps): what "closer to real" means per check. The real rows are the bench refs against
+# each other (real loo) and the forge refs against the bench (real x); a check only gets better / worse when it moves
+# relative to what those real recordings score ('same' within eps):
+# 'low' lower is better, but anything at or below the best real row counts as real (the excess over it is compared)
+# 'one' a ratio: log distance to the span of 1 and the real rows
+# 'span' distance to the span of the real rows' values (variety, texture)
+# 'model' a model check: higher is better per bench.py, but saturated; reported as up / down only
+METRICS = [
+ ('var_cos', "variety: CLAP cos between variants (the refs' own in the real columns)", 'span', 0.01),
+ ('tone_db', 'tone: 1/3-octave distance to the bench median, dB', 'low', 0.3),
+ ('env_oct', 'envelope: attack / t40 / length off, octaves', 'low', 0.03),
+ ('curve_db', 'envelope curve distance, dB', 'low', 0.5),
+ ('t40_ratio', 'tail t40 / bench median (1 = same)', 'one', 0.03),
+ ('artifacts', 'artifact flags (count)', 'low', 0.5),
+ ('stereo_corr', 'stereo correlation (1 = mono)', 'span', 0.02),
+ ('floor_db', 'floor, dB re the loudest 5 ms', 'span', 1.0),
+ ('darken', 'tail darkening (log2 centroid half-tail / peak)', 'span', 0.05),
+ ('typ_z', 'CLAP typicality z (model)', 'model', 0.05),
+ ('phrase_z', 'CLAP held-out phrasings z (model)', 'model', 0.05),
+ ('margin', 'phrasing margin over confusions (model)', 'model', 0.05),
+ ('as_names', 'AudioSet names the category, share (model)', 'model', 0.05),
+]
+
+
+def verdict(rule, a, b, reals, eps):
+ """'better' / 'worse' / 'same' (+ why) for a v1 -> v2 move of one check, given the real rows' values"""
+ if a is None or b is None:
+ return ''
+ reals = [r for r in reals if r is not None]
+ if rule == 'model':
+ v = 'up' if b > a + eps else ('down' if b < a - eps else 'same')
+ return v + (' (v1 sat above the real rows)' if reals and a > max(reals) else '')
+ if rule == 'low':
+ ref = min(reals) if reals else 0.0
+ dist = lambda v: max(0.0, v - ref)
+ elif rule == 'one':
+ lo, hi = min([1.0] + reals), max([1.0] + reals)
+ dist = lambda v: max(0.0, np.log(lo / max(v, 1e-3)), np.log(max(v, 1e-3) / hi))
+ else:
+ if not reals:
+ return ''
+ lo, hi = min(reals), max(reals)
+ dist = lambda v: max(0.0, lo - v, v - hi)
+ da, db = dist(a), dist(b)
+ if da == 0 and db == 0:
+ return 'same (both within the real rows)'
+ return 'better' if db < da - eps else ('worse' if db > da + eps else 'same')
+
+
+def build_progress(pdir, results_path=None, quiet=False):
+ os.makedirs(pdir, exist_ok=True)
+ r1 = json.load(open(f'{HERE}/results_v1.json', encoding='utf-8'))
+ rp = results_v2(results_path)
+ r2 = json.load(open(rp, encoding='utf-8'))
+ # listening file
+ parts, lines, t = [], [], LEAD
+ gain = 10 ** (PEAK_MAX_DB / 20)
+ picks = []
+ for name, cat, v1dir, stem in V1:
+ v1 = [f'{REPO}/{v1dir}/{stem}_01.wav', f'{REPO}/{v1dir}/{stem}_02.wav']
+ cands = finals(cat)
+ seq = []
+ for f in v1:
+ d, c = nearest_v2(f, cands)
+ cands = [q for q in cands if q != c]
+ seq += [('v1', f, None), ('v2', c, d)]
+ lines.append(f'{name} (v2 category {cat})')
+ for ver, f, d in seq:
+ x = load(f)
+ x = x * (gain / (np.abs(x).max() + 1e-12))
+ parts.append((t, x))
+ extra = f' (v2 final closest to the v1 final before it: distance {d:.1f})' if d is not None else ''
+ lines.append(f' {t:7.2f} s {ver} {rel(f)}{extra}')
+ picks.append({'sound': name, 'version': ver, 'start': round(t, 2), 'file': rel(f)})
+ t += len(x) / SR + 1.0
+ t += 2.0
+ total = t - 3.0 + TAIL
+ y = place(parts, total)
+ write(os.path.join(pdir, 'v1_vs_v2.wav'), y)
+ head = ['v1_vs_v2.wav: the five sounds forge v1 had, before (v1) and after (v2), alternating v1, v2, v1, v2.',
+ 'Every sound peak-matched to -1 dBFS (so the after is not just louder); 1 s between sounds, 3 s between',
+ 'sound groups. Each v2 final is the one closest to the v1 final before it in 1/3-octave balance + audible',
+ 'length (distance = RMS dB over the bands + 6 dB per octave of length), so each pair is like for like.',
+ f'total {len(y) / SR:.1f} s, 48 kHz, 24-bit stereo', '']
+ open(os.path.join(pdir, 'progress.txt'), 'w', encoding='utf-8').write(chr(10).join(head + lines) + chr(10))
+ # numbers
+ md = ['# forge v1 -> v2: measured before / after', '',
+ f'Same frozen, held-out benchmark (`eval/forge/bench.py`): v1 = `eval/forge/results_v1.json` '
+ f'({r1.get("created", "?")}), v2 = `{rel(rp)}` ({r2.get("created", "?")}). Made sounds are measured against '
+ 'bench recordings no forge search ever used. Columns: v1 and v2 made medians; "bench" = each bench ref against '
+ 'the other bench refs (real loo); "forge refs" = real recordings the bench never saw, scored the same way '
+ '(real x). Listening: `v1_vs_v2.wav` (`progress.txt` has the times).', '']
+ # scope
+ cats_v2 = sorted(d for d in os.listdir(MADE) if finals(d))
+ n_fin = sum(len(finals(d)) for d in cats_v2)
+ md += [f'Scope: v1 had 5 sounds, 3 finals each (15). v2 delivers {len(cats_v2)} categories, {n_fin} finals '
+ f'(`out/forge_v2/<category>/final/`).', '']
+ stale = []
+ for name, cat, *_ in V1:
+ listed = {os.path.normcase(os.path.abspath(os.path.join(REPO, m['file']))) for m in
+ r2['categories'].get(cat, {}).get('made', [])}
+ now = {os.path.normcase(os.path.abspath(p)) for p in finals(cat)}
+ newer = [p for p in finals(cat) if os.path.getmtime(p) > os.path.getmtime(rp)]
+ if listed != now or newer:
+ stale.append(cat)
+ if stale:
+ md += [f'**Stale:** the finals of {", ".join(stale)} changed after `{rel(rp)}` was written: these rows measure '
+ 'the earlier files. Re-run eval/forge/bench.py and rebuild.', '']
+ summary_rows = []
+ for name, cat, *_ in V1:
+ c1, c2 = r1['categories'][cat], r2['categories'][cat]
+ s1, s2 = c1['summary'], c2['summary']
+ loo, rx = s1.get('real_loo') or {}, s1.get('real_cross') or {}
+ m1, m2 = s1['made'], s2['made']
+ i1, i2 = c1.get('inside') or {}, c2.get('inside') or {}
+ marker = ''.join(f' ({os.path.splitext(os.path.basename(p))[0]})' for p in
+ glob.glob(f'{MADE}/{cat}/final/KEPT_*.txt'))
+ md += [f'## {name} (`{cat}`){marker}', '',
+ f'n: v1 {m1.get("n")}, v2 {m2.get("n")} finals. Bench refs: {c1.get("n_bench")}.', '',
+ '| check | v1 | v2 | bench (real loo) | forge refs (real x) | v1 -> v2 |', '|---|---|---|---|---|---|']
+ nb = nw = 0
+ for k, label, rule, eps in METRICS:
+ a, b = m1.get(k), m2.get(k)
+ if k == 'var_cos':
+ realv = (c1.get('diversity_bench') or {}).get('cos_mean')
+ rxv = (c1.get('diversity_forge') or {}).get('cos_mean')
+ else:
+ realv, rxv = loo.get(k), rx.get(k)
+ v = verdict(rule, a, b, [realv, rxv], eps)
+ if rule != 'model':
+ nb += v == 'better'
+ nw += v == 'worse'
+ md.append(f'| {label} | {fmt(a)} | {fmt(b)} | {fmt(realv)} | {fmt(rxv)} | {v} |')
+ ins = lambda d: (f'{np.median(d["per_sound"]):g} / {d["n_checks"]}' if d.get('per_sound') else '-')
+ real_ins = (f'{np.median(i1["real_per_sound"]):g} / {i1["n_checks"]}' if i1.get('real_per_sound') else '-')
+ md.append(f'| inside the real 80 % limits (median checks) | {ins(i1)} | {ins(i2)} | {real_ins} (real rows) | | '
+ f'{verdict("model", float(np.median(i1["per_sound"])), float(np.median(i2["per_sound"])), [], 0.4).replace("up", "more").replace("down", "fewer") if i1.get("per_sound") and i2.get("per_sound") else ""} |')
+ md.append('')
+ summary_rows.append((name, cat, nb, nw))
+ if c1.get('note'):
+ md += [f'Bench note: {c1["note"]}', '']
+ md += ['## In one line per sound (model-free checks: tone, envelope, curve, tail, artifacts, variety, texture)', '',
+ '| sound | better | worse |', '|---|---|---|']
+ md += [f'| {n} (`{c}`) | {b} | {w} |' for n, c, b, w in summary_rows]
+ md += ['', '## How to read this honestly', '',
+ '- "better" / "worse" for the model-free checks means closer to / further from what real recordings score '
+ 'on the same check (the bench refs against each other, and the forge refs against the bench): a tone, '
+ 'envelope or curve distance only counts above the best real row, a tail ratio or a variety / texture value '
+ 'only outside the span of the real rows ("same (both within the real rows)" otherwise). The counts weight '
+ 'every check the same.',
+ '- The model checks (CLAP typicality and phrasings, AudioSet) saturate: v1 sat above the real recordings on '
+ 'them after its searches tuned against CLAP, so a v2 drop toward the real rows is not by itself a loss. They '
+ 'are listed as up / down only. The "inside" row counts 7 checks of which 4 are model checks (CLAP '
+ 'typicality, phrasings, margin, AudioSet p) and 3 model-free (tone, envelope, curve): a fall there while '
+ 'the model-free rows improve may be the saturation unwinding rather than the sound '
+ 'getting worse; the numbers cannot settle which, the blind test can.',
+ '- v1 impact and fast-whoosh numbers are optimistic: v1 was tuned partly against kit refs that are bench '
+ 'refs today (kit impact_03, whoosh_med_01 / 02).',
+ '- Impact has a single bench ref (no spread, no variety baseline); typing_key has 12, click 7.',
+ '- Numbers are not ears: the clean blind test (`../blind_v2/`) is the deciding test.', '']
+ open(os.path.join(pdir, 'progress.md'), 'w', encoding='utf-8').write(chr(10).join(md) + chr(10))
+ json.dump({'results_v1': 'eval/forge/results_v1.json', 'results_v2': rel(rp), 'picks': picks,
+ 'built': time.strftime('%Y-%m-%d %H:%M')},
+ open(os.path.join(pdir, 'progress.json'), 'w', encoding='utf-8'), indent=1)
+ if not quiet:
+ print(chr(10).join(head + lines))
+ print(f'progress.md from {rel(rp)}' + (f'; STALE: {stale}' if stale else ''))
+
+
+def main():
+ ap = argparse.ArgumentParser(description=__doc__.split(chr(10))[0])
+ ap.add_argument('--out', default=f'{REPO}/out/showcase/blind_v2', help='blind test folder')
+ ap.add_argument('--progress', default=None, help='progress folder (default: <out>/../progress)')
+ ap.add_argument('--seed', type=int, default=2026)
+ ap.add_argument('--n', type=int, default=20, help='trials (default 20)')
+ ap.add_argument('--results', default=None, help='v2 benchmark JSON (default: the newest results_v2r<N>.json)')
+ ap.add_argument('--only', choices=('blind', 'progress'), default=None)
+ a = ap.parse_args()
+ out = os.path.abspath(a.out if os.path.isabs(a.out) else os.path.join(REPO, a.out))
+ pdir = a.progress or os.path.join(os.path.dirname(out), 'progress')
+ pdir = os.path.abspath(pdir if os.path.isabs(pdir) else os.path.join(REPO, pdir))
+ if a.only != 'progress':
+ build_blind(out, a.n, a.seed)
+ if a.only != 'blind':
+ build_progress(pdir, a.results)
+
+
+if __name__ == '__main__':
+ main()