Skip to content

Fastest and only macOS Dictation app with on-device STT and custom trained AI enhancement model. Windows, iOS testflight! Android, Linux waitlist open! A local Wispr Flow alternative. Follow https://x.com/fluidvoiceapp for more

test(dictionary): preserve reproducible pronunciation accuracy baselines

altic-deve089065bc436
diff --git a/docs/design/original-audio-learning-implementation.md b/docs/design/original-audio-learning-implementation.md
index 47de3ada..fb14a0be 100644
--- a/docs/design/original-audio-learning-implementation.md
+++ b/docs/design/original-audio-learning-implementation.md
@@ -60,3 +60,10 @@ The provider takes a bounded profile snapshot from the store actor on the first
The real-audio approval regression now starts with an empty profile snapshot, approves and learns the correction, resets for the next recording, and verifies actual replacement in short, word-timed and 60-second incremental dictation with Advanced Preview off. It also checks that disabling automatic learning and Advanced Preview restores baseline ASR. This repeated-recording proof is functional validation, not held-out recognition accuracy.
Validation: 76 dictionary/pronunciation and meeting-policy tests passed together, plus the separate production meeting-provider isolation test. Scoped provider lint is clean; full-source formatting and strict lint ran with existing repository violations preserved. The signed private FI build succeeded. Debug pronunciation timings are not treated as optimized latency measurements.
+
+
+## Reproducible evaluation
+
+The evaluation executables, comparison script, commands and machine-readable baseline now live in tools/DictionaryLearningEvaluation. A fresh Release run of both executables reproduced the saved results exactly. Model files, audio and large acoustic vectors remain external local inputs.
+
+The later real Huang occurrence was transcribed as Hong Kong and produced no acoustic hit above 0.70 from its earlier enrollment. This is an important unresolved example of the mean-pooled encoder's limitations; no threshold was lowered to make it pass. The multilingual combined result still equals text-only (19 true, 2 false substitutions) rather than exceeding it. Improvement on held-out personalized names remains an open acceptance gate, despite functional approval-to-reuse and composition regression tests passing.
diff --git a/tools/DictionaryLearningEvaluation/.gitignore b/tools/DictionaryLearningEvaluation/.gitignore
new file mode 100644
index 00000000..913901f7
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/.gitignore
@@ -0,0 +1,3 @@
+.build/
+Package.resolved
+__pycache__/
diff --git a/tools/DictionaryLearningEvaluation/Package.swift b/tools/DictionaryLearningEvaluation/Package.swift
new file mode 100644
index 00000000..b98a9da9
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/Package.swift
@@ -0,0 +1,20 @@
+// swift-tools-version: 6.2
+import Foundation
+import PackageDescription
+
+let package = Package(
+ name: "DictionaryLearningEvaluation",
+ platforms: [.macOS(.v14)],
+ dependencies: [
+ .package(
+ name: "FluidAudio",
+ path: ProcessInfo.processInfo.environment["FLUIDAUDIO_SOURCE"]
+ ?? "../../../FluidAudio_pronunciation_streaming")
+ ],
+ targets: [
+ .executableTarget(
+ name: "Evaluate", dependencies: [.product(name: "FluidAudio", package: "FluidAudio")]),
+ .executableTarget(
+ name: "PersonalProof", dependencies: [.product(name: "FluidAudio", package: "FluidAudio")]),
+ ]
+)
diff --git a/tools/DictionaryLearningEvaluation/README.md b/tools/DictionaryLearningEvaluation/README.md
new file mode 100644
index 00000000..6f7d0e06
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/README.md
@@ -0,0 +1,29 @@
+# Original-audio learning evaluation
+
+Offline diagnostics for the production FluidAudio encoder/search APIs. These tools never open a microphone, write the user's dictionary, or download a model. Supply real recordings and an already installed Parakeet v3 model directory. Outputs include transcribed text and acoustic vectors; store them locally.
+
+## Run
+
+Check `xcode-select -p`; use a full Xcode developer directory for Swift commands. From the FluidVoice root:
+
+```sh
+export DEVELOPER_DIR=/Applications/Xcode-beta6.app/Contents/Developer
+export FLUIDAUDIO_SOURCE=/absolute/path/to/FluidAudio_pronunciation_streaming
+swift run -c release --package-path tools/DictionaryLearningEvaluation PersonalProof /absolute/path/to/jensen-120s.wav /absolute/path/to/parakeet-tdt-0.6b-v3-coreml /tmp/personal-trials.json
+swift run -c release --package-path tools/DictionaryLearningEvaluation Evaluate /absolute/path/to/fleurs /absolute/path/to/parakeet-tdt-0.6b-v3-coreml /tmp/word-vectors.json
+python3 tools/DictionaryLearningEvaluation/analyze_search.py /tmp/word-vectors.json.trials.json /tmp/search-comparison.json
+```
+
+Baseline companion revision: 9725fb6. Default sibling checkout is FluidAudio_pronunciation_streaming; FLUIDAUDIO_SOURCE allows another checkout for comparison. No model or audio is committed here.
+
+Evaluate expects de_de, fr_fr, pl_pl, da_dk, el_gr, ru_ru, es_419 and en_us directories, each with WAV files and a matching LANGUAGE.trans.txt. Each reference line begins with the WAV stem, followed by its reference transcript. It excludes recordings longer than 14.88 seconds. Enrollment is from one clip; all evaluation clips differ from enrollment. Edit-distance reference alignment is approximate and must be inspected before interpreting individual errors.
+
+PersonalProof uses the first occurrence of NVIDIA, Jensen, Huang, design and AI in supplied interview speech. It excludes the entire enrollment window from subsequent search, preserves raw scores and reports release search timings. It ignores a final tail shorter than one second. Its positive counts use ASR text, not independent reference labels: “Hong Kong” will not be counted as a Huang occurrence. Do not use those counts as recall or WER. The 120-second source includes a speaker transition and is not a same-speaker-only benchmark.
+
+## Interpretation and limitations
+
+The analyzer compares known-text matches, acoustic hits at the library's 0.70 search threshold, and combined fallback with 0.85 required for unknown variants. It evaluates one intended label at a time; it does not simulate production competition between dictionary entries or every text-formatting rule. Tests in DictationE2ETests cover those composition behaviors. These thresholds must stay in sync with DictionaryPronunciationDecision; they are not probabilities.
+
+The current FLEURS subset produced text 19 true/2 false substitutions, acoustic 3 true/1 false, combined 19 true/2 false across 21 target and 547 non-target opportunities. This shows preserved text recall, not superiority. The independent later Huang/Hong Kong diagnostic produced no acoustic hit above 0.70. Do not lower thresholds to fit this single miss.
+
+Release search of one profile across a 14.88-second window: 28 calls, median 0.517 ms, range 0.494–0.669 ms on M5 Max. This excludes audio encoding, text reconciliation and Stop scheduling. See docs/design/original-audio-learning-implementation.md for full pipeline validation and resource bounds.
diff --git a/tools/DictionaryLearningEvaluation/Sources/Evaluate/main.swift b/tools/DictionaryLearningEvaluation/Sources/Evaluate/main.swift
new file mode 100644
index 00000000..8ca8c376
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/Sources/Evaluate/main.swift
@@ -0,0 +1,169 @@
+import FluidAudio
+import Foundation
+
+struct Word: Codable {
+ let clip: String
+ let language: String
+ let index: Int
+ let text: String
+ let reference: String?
+ let start: Double
+ let end: Double
+ let values: [Float]
+ let frames: Int
+}
+struct Hit: Codable {
+ let score: Float
+ let indices: [Int]
+}
+struct Trial: Codable {
+ let language: String
+ let target: String
+ let known: String
+ let clip: String
+ let words: [Word]
+ let hits: [Hit]
+}
+func normalized(_ text: String) -> String {
+ text.lowercased().filter { $0.isLetter || $0.isNumber }
+}
+func align(_ hypothesis: [String], _ reference: [String]) -> [Int: String] {
+ let n = hypothesis.count
+ let m = reference.count
+ var costs = Array(repeating: Array(repeating: 0, count: m + 1), count: n + 1)
+ for i in 0...n { costs[i][0] = i }
+ for j in 0...m { costs[0][j] = j }
+ if n > 0, m > 0 {
+ for i in 1...n {
+ for j in 1...m {
+ costs[i][j] = min(
+ costs[i - 1][j] + 1, costs[i][j - 1] + 1,
+ costs[i - 1][j - 1] + (hypothesis[i - 1] == reference[j - 1] ? 0 : 1))
+ }
+ }
+ }
+ var result: [Int: String] = [:]
+ var i = n
+ var j = m
+ while i > 0 || j > 0 {
+ if i > 0, j > 0,
+ costs[i][j] == costs[i - 1][j - 1] + (hypothesis[i - 1] == reference[j - 1] ? 0 : 1)
+ {
+ result[i - 1] = reference[j - 1]
+ i -= 1
+ j -= 1
+ } else if i > 0, costs[i][j] == costs[i - 1][j] + 1 {
+ i -= 1
+ } else {
+ j -= 1
+ }
+ }
+ return result
+}
+@main struct Evaluate {
+ static func main() async throws {
+ guard CommandLine.arguments.count == 4 else {
+ throw NSError(
+ domain: "DictionaryLearningEvaluation", code: 1,
+ userInfo: [
+ NSLocalizedDescriptionKey: "Usage: Evaluate FLEURS_DIRECTORY MODEL_DIRECTORY OUTPUT_JSON"
+ ])
+ }
+ let root = URL(fileURLWithPath: CommandLine.arguments[1])
+ let models = try await AsrModels.load(
+ from: URL(fileURLWithPath: CommandLine.arguments[2]), version: .v3)
+ let manager = AsrManager(
+ config: ASRConfig(
+ tdtConfig: TdtConfig(blankId: AsrModelVersion.v3.blankId), encoderHiddenSize: 1024))
+ try await manager.initialize(models: models)
+ var output: [Word] = []
+ var clips: [String: (samples: [Float], features: EncoderFeatureSequence)] = [:]
+ for language in ["en_us", "es_419", "fr_fr", "de_de", "ru_ru", "pl_pl", "el_gr", "da_dk"] {
+ let directory = root.appendingPathComponent(language)
+ let referenceLines = try String(
+ contentsOf: directory.appendingPathComponent("\(language).trans.txt"), encoding: .utf8
+ ).split(separator: "\n")
+ let references = Dictionary(
+ uniqueKeysWithValues: referenceLines.compactMap { line -> (String, [String])? in
+ let parts = line.split(separator: " ", maxSplits: 1)
+ guard parts.count == 2 else { return nil }
+ return (String(parts[0]), parts[1].split(separator: " ").map { normalized(String($0)) })
+ })
+ for file in try FileManager.default.contentsOfDirectory(
+ at: directory, includingPropertiesForKeys: nil
+ ).filter({ $0.pathExtension == "wav" }).sorted(by: { $0.path < $1.path }) {
+ let samples = try AudioConverter().resampleAudioFile(path: file.path)
+ guard samples.count <= 238080 else {
+ print("SKIP_LONG \(file.lastPathComponent)")
+ continue
+ }
+ await manager.setPronunciationCustomizationEnabled(true)
+ let result = try await manager.transcribe(samples, source: .microphone)
+ guard let features = await manager.consumePronunciationEncoderFeatures() else { continue }
+ let words = WordAudioChunkExtractor.words(from: result.tokenTimings ?? [])
+ let clip = file.deletingPathExtension().lastPathComponent
+ clips[clip] = (samples, features)
+ let labels = align(words.map { normalized($0.text) }, references[clip] ?? [])
+ for (index, word) in words.enumerated() {
+ let start = max(0, Int((word.startTime / 0.08).rounded(.down)))
+ let end = min(features.frameCount, Int((word.endTime / 0.08).rounded(.up)))
+ guard end > start,
+ let embedding = PronunciationEmbeddingMatcher.embedding(
+ from: features, frameRange: start..<end)
+ else { continue }
+ output.append(
+ Word(
+ clip: clip, language: language, index: index, text: normalized(word.text),
+ reference: labels[index], start: word.startTime, end: word.endTime,
+ values: embedding.values, frames: embedding.sourceFrameCount))
+ }
+ print("CLIP \(clip) \(words.count) words")
+ }
+ }
+ var trials: [Trial] = []
+ let grouped = Dictionary(grouping: output.filter { ($0.reference?.count ?? 0) >= 3 }) {
+ "\($0.language):\($0.reference!)"
+ }
+ for key in grouped.keys.sorted() {
+ let occurrences = grouped[key]!
+ guard Set(occurrences.map(\.clip)).count > 1,
+ let enrollment = occurrences.first(where: { $0.text != $0.reference }) ?? occurrences.first,
+ let source = clips[enrollment.clip]
+ else { continue }
+ let focalStart = Int((enrollment.start * 16000).rounded(.down))
+ let focalEnd = min(source.samples.count, Int((enrollment.end * 16000).rounded(.up)))
+ let contextStart = max(0, focalStart - 32000) / 1280 * 1280
+ let contextEnd = min(source.samples.count, contextStart + 238080)
+ guard focalEnd <= contextEnd, focalEnd > focalStart else { continue }
+ let prototype = try await manager.pronunciationEmbedding(
+ audioSamples: Array(source.samples[contextStart..<contextEnd]),
+ focalSampleRange: (focalStart - contextStart)..<(focalEnd - contextStart)
+ )
+ for clip in clips.keys.sorted()
+ where clip != enrollment.clip && clip.hasPrefix(enrollment.language) {
+ let words = output.filter { $0.clip == clip }
+ let matches = PronunciationEmbeddingMatcher.allMatches(
+ prototypes: [prototype], in: clips[clip]!.features)[0]
+ let hits = matches.map { match in
+ let start = Double(match.frameRange.lowerBound) * 0.08
+ let end = Double(match.frameRange.upperBound) * 0.08
+ let indices = words.indices.filter { index in
+ let word = words[index]
+ return max(0, min(word.end, end) - max(word.start, start))
+ / max(0.0001, word.end - word.start) >= 0.5
+ }
+ return Hit(score: match.score, indices: indices)
+ }
+ trials.append(
+ Trial(
+ language: enrollment.language, target: enrollment.reference!, known: enrollment.text,
+ clip: clip, words: words, hits: hits))
+ }
+ }
+ try JSONEncoder().encode(trials).write(
+ to: URL(fileURLWithPath: CommandLine.arguments[3] + ".trials.json"), options: .atomic)
+ try JSONEncoder().encode(output).write(
+ to: URL(fileURLWithPath: CommandLine.arguments[3]), options: .atomic)
+ await manager.cleanup()
+ }
+}
diff --git a/tools/DictionaryLearningEvaluation/Sources/PersonalProof/main.swift b/tools/DictionaryLearningEvaluation/Sources/PersonalProof/main.swift
new file mode 100644
index 00000000..6f207e23
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/Sources/PersonalProof/main.swift
@@ -0,0 +1,107 @@
+import FluidAudio
+import Foundation
+
+struct Token: Codable {
+ let text: String
+ let start: Double
+ let end: Double
+}
+struct Hit: Codable {
+ let score: Float
+ let text: String
+ let start: Double
+ let end: Double
+ let targetOnly: Bool
+}
+struct Trial: Codable {
+ let target: String
+ let enrollmentStart: Double
+ let positiveOccurrences: Int
+ let hits: [Hit]
+}
+func normalize(_ s: String) -> String { s.lowercased().filter { $0.isLetter || $0.isNumber } }
+@main struct PersonalProof {
+ static func main() async throws {
+ guard CommandLine.arguments.count == 4 else {
+ throw NSError(
+ domain: "DictionaryLearningEvaluation", code: 1,
+ userInfo: [
+ NSLocalizedDescriptionKey: "Usage: PersonalProof AUDIO MODEL_DIRECTORY OUTPUT_JSON"
+ ])
+ }
+ let source = try AudioConverter().resampleAudioFile(path: CommandLine.arguments[1])
+ let models = try await AsrModels.load(
+ from: URL(fileURLWithPath: CommandLine.arguments[2]), version: .v3)
+ let manager = AsrManager(
+ config: ASRConfig(
+ tdtConfig: TdtConfig(blankId: AsrModelVersion.v3.blankId), encoderHiddenSize: 1024))
+ try await manager.initialize(models: models)
+ var windows: [(offset: Int, features: EncoderFeatureSequence, words: [Token])] = []
+ var allWords: [Token] = []
+ for start in stride(from: 0, to: source.count, by: 238080) {
+ let samples = Array(source[start..<min(source.count, start + 238080)])
+ guard samples.count >= 16000 else {
+ print("SKIP_TAIL samples=\(samples.count)")
+ continue
+ }
+ await manager.setPronunciationCustomizationEnabled(true)
+ let result = try await manager.transcribe(samples, source: .microphone)
+ guard let features = await manager.consumePronunciationEncoderFeatures() else { continue }
+ let words = WordAudioChunkExtractor.words(from: result.tokenTimings ?? []).map {
+ Token(
+ text: normalize($0.text), start: $0.startTime + Double(start) / 16000,
+ end: $0.endTime + Double(start) / 16000)
+ }
+ windows.append((start, features, words))
+ allWords += words
+ print("WINDOW \(start) \(result.text)")
+ }
+ var trials: [Trial] = []
+ for target in ["nvidia", "jensen", "huang", "design", "ai"] {
+ let occurrences = allWords.filter { $0.text == target }
+ guard let enrollment = occurrences.first else { continue }
+ let lower = Int((enrollment.start * 16000).rounded(.down))
+ let upper = Int((enrollment.end * 16000).rounded(.up))
+ let contextStart = max(0, lower - 32000) / 1280 * 1280
+ let contextEnd = min(source.count, contextStart + 238080)
+ let prototype = try await manager.pronunciationEmbedding(
+ audioSamples: Array(source[contextStart..<contextEnd]),
+ focalSampleRange: (lower - contextStart)..<(upper - contextStart))
+ var hits: [Hit] = []
+ var positives = 0
+ for window in windows {
+ // Entire enrollment ASR window is excluded; never count replay as held-out speech.
+ if enrollment.start >= Double(window.offset) / 16000
+ && enrollment.start < Double(window.offset + 238080) / 16000
+ {
+ continue
+ }
+ positives += window.words.filter { $0.text == target }.count
+ let matchStart = ProcessInfo.processInfo.systemUptime
+ let matches = PronunciationEmbeddingMatcher.allMatches(
+ prototypes: [prototype], in: window.features)[0]
+ let matchMs = (ProcessInfo.processInfo.systemUptime - matchStart) * 1000
+ print("MATCH_MS target=\(target) frames=\(window.features.frameCount) elapsed=\(matchMs)")
+ for match in matches {
+ let start = Double(window.offset) / 16000 + Double(match.frameRange.lowerBound) * 0.08
+ let end = Double(window.offset) / 16000 + Double(match.frameRange.upperBound) * 0.08
+ let words = window.words.filter {
+ max(0, min($0.end, end) - max($0.start, start)) / max(0.0001, $0.end - $0.start) >= 0.5
+ }
+ guard !words.isEmpty else { continue }
+ hits.append(
+ Hit(
+ score: match.score, text: words.map(\.text).joined(separator: " "), start: start,
+ end: end, targetOnly: words.count == 1 && words[0].text == target))
+ }
+ }
+ trials.append(
+ Trial(
+ target: target, enrollmentStart: enrollment.start, positiveOccurrences: positives,
+ hits: hits))
+ }
+ try JSONEncoder().encode(trials).write(
+ to: URL(fileURLWithPath: CommandLine.arguments[3]), options: .atomic)
+ await manager.cleanup()
+ }
+}
diff --git a/tools/DictionaryLearningEvaluation/analyze_search.py b/tools/DictionaryLearningEvaluation/analyze_search.py
new file mode 100644
index 00000000..74100444
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/analyze_search.py
@@ -0,0 +1,28 @@
+import argparse, json, collections
+from pathlib import Path
+parser=argparse.ArgumentParser(description="Compare text and acoustic recovery on reference-aligned trials.")
+parser.add_argument("trials",type=Path)
+parser.add_argument("output",type=Path)
+args=parser.parse_args()
+trials=json.loads(args.trials.read_text())
+counts={name:collections.Counter() for name in ['text','audio70','combined_fallback']}
+errors=[]
+for t in trials:
+ text={i for i,w in enumerate(t['words']) if w['text']==t['known']}
+ audio=set(); combined=set()
+ for hit in t['hits']:
+ indices=hit['indices']
+ if not indices: continue
+ audio.update(indices)
+ known=' '.join(t['words'][i]['text'] for i in indices)==t['known']
+ if hit['score'] >= (.7 if known else .85): combined.update(indices)
+ combined.update(text)
+ for name,pred in [('text',text),('audio70',audio),('combined_fallback',combined)]:
+ for i,w in enumerate(t['words']):
+ if 'reference' not in w: continue
+ target=w['reference']==t['target']; accepted=i in pred
+ counts[name]['tp' if target and accepted else 'fn' if target else 'fp' if accepted else 'tn']+=1
+ if accepted and not target: errors.append(dict(mode=name,target=t['target'],language=t['language'],clip=t['clip'],text=w['text'],reference=w['reference']))
+result={'trials':len(trials),'targets':len(set((t['language'],t['target']) for t in trials)),'counts':counts,'errors':errors}
+args.output.write_text(json.dumps(result,indent=2))
+print(json.dumps(result,indent=2))
diff --git a/tools/DictionaryLearningEvaluation/baseline-summary.json b/tools/DictionaryLearningEvaluation/baseline-summary.json
new file mode 100644
index 00000000..e73e8c5f
--- /dev/null
+++ b/tools/DictionaryLearningEvaluation/baseline-summary.json
@@ -0,0 +1,52 @@
+{
+ "companion_revision": "9725fb6",
+ "model": "parakeet-tdt-0.6b-v3-coreml",
+ "hardware": "Apple M5 Max",
+ "multilingual": {
+ "clips": 27,
+ "languages": 8,
+ "timed_words": 536,
+ "targets": 11,
+ "trials": 26,
+ "counts": {
+ "text": {
+ "tn": 545,
+ "tp": 19,
+ "fp": 2,
+ "fn": 2
+ },
+ "audio70": {
+ "tn": 546,
+ "fn": 18,
+ "tp": 3,
+ "fp": 1
+ },
+ "combined_fallback": {
+ "tn": 545,
+ "tp": 19,
+ "fp": 2,
+ "fn": 2
+ }
+ },
+ "label_source": "reference-to-ASR edit alignment; approximate"
+ },
+ "held_out_name": {
+ "enrolled": "Huang",
+ "later_ASR": "Hong Kong",
+ "hits_at_or_above_0_70": 0,
+ "limitation": "diagnostic from one interview, not a representative name corpus"
+ },
+ "release_search": {
+ "calls": 28,
+ "window_seconds": 14.88,
+ "profiles_per_call": 1,
+ "median_ms": 0.516542,
+ "min_ms": 0.494125,
+ "max_ms": 0.668917,
+ "excludes": [
+ "encoding",
+ "text reconciliation",
+ "Stop scheduling"
+ ]
+ }
+}