Fastest and only macOS Dictation app with on-device STT and custom trained AI enhancement model. Windows, iOS testflight! Android, Linux waitlist open! A local Wispr Flow alternative. Follow https://x.com/fluidvoiceapp for more
feat(meeting): align words to speaker turns by timestamp
Shreeraman A K0492a4054bc9
diff --git a/MEETING_TRANSCRIPTION_PRD.md b/MEETING_TRANSCRIPTION_PRD.mdindex d0b54e75..9e052bb2 100644--- a/MEETING_TRANSCRIPTION_PRD.md+++ b/MEETING_TRANSCRIPTION_PRD.md@@ -181,20 +181,28 @@ Goal: make the working slice safe to ship.Written 2026-08-10 per DELIVERY-001, before M2 implementation. Numeric budgets referenced here are set in §21.7.-- [ ] **[MUST · M2] M2-TEST-001 — Relaunch recovery, deterministic.** Sessions left in `recording`, `stopping`, or `processing` are detected on launch; finalized chunks are salvaged; exactly one recovery offer appears per session; Resume produces a transcript from the salvaged chunks; Delete removes chunks, manifests, and checkpoints. Covered by tests that construct the on-disk states directly.-- [ ] **[MUST · M2] M2-TEST-002 — Real crash proof.** `kill -9` the app during a real recording at three points (first minute, mid-session, during processing). After relaunch, recovery succeeds and audio loss is bounded by the active chunk (≤ one chunk duration). Repeated for force-quit.-- [ ] **[MUST · M2] M2-TEST-003 — Transcript correction, deterministic.** On a fixture transcript: global rename updates every segment of that speaker without changing segment IDs; single-segment reassignment; merge of two speakers; undo restores the exact prior state after each operation; export reflects corrections.-- [ ] **[MUST · M2] M2-TEST-004 — History, export, deletion, retention.** Text and JSON exports exclude embeddings, model fingerprints, and internal confidence data (UX-RESULT-010). Delete Audio preserves the transcript; Delete Meeting removes chunks, manifests, checkpoints, and recovery remnants (PRIV-015). The chosen retention policy demonstrably deletes on schedule.-- [ ] **[MUST · M2] M2-TEST-005 — Permission lifecycle.** For microphone and Screen/System Audio independently: first grant, denial, return from System Settings, revocation mid-recording (typed interruption per REL-017 with all finalized audio preserved), regrant, relaunch.-- [ ] **[MUST · M2] M2-TEST-006 — Platform matrix at claimed tiers.** Real Zoom desktop, Google Meet in Chrome, and Teams desktop each: join, record ≥10 minutes, switch audio device mid-call (AirPods connect), stop, and produce a transcript with remote speakers separated. Run once with headphones and once with laptop speakers; the speaker run must show leaked speech labeled Unknown, never You.-- [ ] **[MUST · M2] M2-TEST-007 — One-hour capture through the app.** A 60-minute recording through the app's own writer meets the §21.7 budgets: no unexplained gap, drift within budget, all chunk files playable, capture CPU and memory within budget, and both track manifests consistent with the audio on disk.-- [ ] **[MUST · M2] M2-TEST-008 — Display and window failure family.** TEST-FAIL-007 (display sleep, screen lock, monitor switch) and TEST-FAIL-008 (capture-window loss) executed through the app runtime, verified by continued buffer delivery — Appendix A results re-confirmed on the release build.-- [ ] **[MUST · M2] M2-TEST-009 — Dictation coexistence.** Dictation hotkey during meeting recording shows the REL-009 message and does not open a competing session; meeting processing serializes model access (REL-010); with meeting idle, dictation first-PCM latency is unchanged from baseline (REL-011).-- [ ] **[MUST · M2] M2-TEST-010 — Intel honesty.** On real Intel hardware (TEST-REPO-006): recording and plain transcription work; no speaker labels are produced or promised; the capability difference is visible in the UI before recording starts (UX-SETUP-008).-- [ ] **[MUST · M2] M2-REVIEW-001** Independent architecture/reliability reviewer approves the recovery design, deletion coverage, permission-lapse handling, and the processing checkpoint/resume path.-- [ ] **[MUST · M2] M2-REVIEW-002** Independent UX/accessibility reviewer approves the recovery UX, correction flows (keyboard and VoiceOver complete), menu-bar lifecycle states, and capability copy.+- [x] **[MUST · M2] M2-TEST-001 — Relaunch recovery, deterministic.** Sessions left in `recording`, `stopping`, or `processing` are detected on launch; finalized chunks are salvaged; exactly one recovery offer appears per session; Resume produces a transcript from the salvaged chunks; Delete removes chunks, manifests, and checkpoints. Covered by tests that construct the on-disk states directly (automated: `Tests/FluidDictationIntegrationTests/MeetingRecoveryTests.swift`).+- [x] **[MUST · M2] M2-TEST-002 — Real crash proof.** `kill -9` the app during a real recording at three points (first minute, mid-session, during processing). After relaunch, recovery succeeds and audio loss is bounded by the active chunk (≤ one chunk duration). Repeated for force-quit. (Verified live 2026-08-11 on real hardware: `kill -9` during recording — recovered, ~15 s loss, under one chunk; `kill -9` during processing — recovered with the processing-specific message, retry completed; `kill -9` with a checkpoint present — resume confirmed by the log line "Meeting processing resuming from checkpoint after application track", checkpoint auto-deleted on success. Also observed: capture-source loss handled cleanly when the captured app quit mid-recording.)+- [x] **[MUST · M2] M2-TEST-003 — Transcript correction, deterministic.** On a fixture transcript: global rename updates every segment of that speaker without changing segment IDs; single-segment reassignment; merge of two speakers; undo restores the exact prior state after each operation; export reflects corrections. (Automated tests; merge keeps the source speaker as an alias so embeddings survive.)+- [x] **[MUST · M2] M2-TEST-004 — History, export, deletion, retention.** Text and JSON exports exclude embeddings, model fingerprints, and internal confidence data (UX-RESULT-010). Delete Audio preserves the transcript; Delete Meeting removes chunks, manifests, checkpoints, and recovery remnants (PRIV-015). The chosen retention policy demonstrably deletes on schedule. (Automated tests cover the eligibility matrix and the export key allowlist.)+- [ ] **[MUST · M2] M2-TEST-005 — Permission lifecycle.** For microphone and Screen/System Audio independently: first grant, denial, return from System Settings, revocation mid-recording (typed interruption per REL-017 with all finalized audio preserved), regrant, relaunch. Still open: needs manual TCC revocation mid-recording.+- [ ] **[MUST · M2] M2-TEST-006 — Platform matrix at claimed tiers.** Real Zoom desktop, Google Meet in Chrome, and Teams desktop each: join, record ≥10 minutes, switch audio device mid-call (AirPods connect), stop, and produce a transcript with remote speakers separated. Run once with headphones and once with laptop speakers; the speaker run must show leaked speech labeled Unknown, never You. Still open: only a 5-minute Zoom call has been run; needs ≥10 min on Zoom, Meet in Chrome, and Teams, plus one run on laptop speakers.+- [ ] **[MUST · M2] M2-TEST-007 — One-hour capture through the app.** A 60-minute recording through the app's own writer meets the §21.7 budgets: no unexplained gap, drift within budget, all chunk files playable, capture CPU and memory within budget, and both track manifests consistent with the audio on disk. Still open: in progress.+- [ ] **[MUST · M2] M2-TEST-008 — Display and window failure family.** TEST-FAIL-007 (display sleep, screen lock, monitor switch) and TEST-FAIL-008 (capture-window loss) executed through the app runtime, verified by continued buffer delivery — Appendix A results re-confirmed on the release build. Still open: display-sleep half automatable, screen-lock half manual.+- [x] **[MUST · M2] M2-TEST-009 — Dictation coexistence.** Dictation hotkey during meeting recording shows the REL-009 message and does not open a competing session; meeting processing serializes model access (REL-010); with meeting idle, dictation first-PCM latency is unchanged from baseline (REL-011). (Deterministic slice only: arbiter double-acquire and the processing serialization gate, including cancellation, are covered by automated tests. REL-009 hotkey messaging and REL-011 latency remain manual.)+- [ ] **[MUST · M2] M2-TEST-010 — Intel honesty.** On real Intel hardware (TEST-REPO-006): recording and plain transcription work; no speaker labels are produced or promised; the capability difference is visible in the UI before recording starts (UX-SETUP-008). Still open: needs Intel hardware.+- [ ] **[MUST · M2] M2-REVIEW-001** Independent architecture/reliability reviewer approves the recovery design, deletion coverage, permission-lapse handling, and the processing checkpoint/resume path. Still open: independent sign-off owed; multi-model adversarial reviews were run but are not a substitute unless the owner accepts them.+- [ ] **[MUST · M2] M2-REVIEW-002** Independent UX/accessibility reviewer approves the recovery UX, correction flows (keyboard and VoiceOver complete), menu-bar lifecycle states, and capability copy. Still open: independent sign-off owed; multi-model adversarial reviews were run but are not a substitute unless the owner accepts them.- [ ] **[MUST · M2] M2-REVIEW-003** Every blocking finding fixed or explicitly recorded against a later milestone before the M2 stacked PR merges.+#### Known limits++- Speaker-bleed echo detection covers the diarized path only; microphone whole-chunk-fallback segments are never classified.+- Echo suppression thresholds are calibrated against a single real call and are not yet validated per CAP-023; echo lines are hidden by default but revealable via a "Show probable echo" control.+- Word-timestamp alignment (one ASR pass per chunk, words assigned to turns by timing) engages only on providers that expose token timings and only when nothing can rewrite the text; every other case falls back to per-turn transcription. Up to 25% of words may be unplaceable before the chunk falls back — below that, stragglers are dropped and logged. Two consequences are accepted for now: vocabulary boosting disables alignment outright, and enrolled pronunciation corrections do not apply to the aligned path, because their matcher is gated to clips under 15s and meeting chunks are 60s.+- Retry restarts processing from preserved audio; it resumes only at the application-audio track boundary (checkpoint), not mid-track.+- Audio deletion is best-effort on APFS; no secure-erase claim (PRIV-015).+### Milestone 3 — Remembered speaker identities- [ ] **[MUST · M3] M3-EXIT-001** Explicit opt-in voice profiles store only confirmed clean samples.@@ -492,7 +500,7 @@ Decided 2026-08-10 from measured evidence on macOS 26.5.1; see Appendix A.## 12. Offline English transcription pipeline- [ ] **[MUST · V1] PIPE-001** Process the dual-track session manifest rather than pretending a captured meeting is one imported file.-- [ ] **[MUST · V1] PIPE-002** Remote/application track: diarize, preserve cluster IDs/embeddings/quality, and transcribe bounded turns.+- [ ] **[MUST · V1] PIPE-002** Remote/application track: diarize, preserve cluster IDs/embeddings/quality, and transcribe turn-bounded text — either per turn, or by assigning word timings from a single chunk pass back to the turns.- [ ] **[MUST · V1] PIPE-003** Microphone track: detect speech turns and correlate them against remote/system speech. Label clean personal-mic speech as **You**; mark probable leakage, shared-mic speech, or ambiguous overlap as **Microphone / Unknown** rather than misattributing it.- [ ] **[MUST · V1] PIPE-004** Merge tracks chronologically from their shared timestamp mapping while preserving overlaps.- [ ] **[MUST · V1] PIPE-005** Preserve raw meeting speech; do not automatically apply dictation filler removal, spoken-punctuation rewriting, or AI enhancement.diff --git a/Sources/Fluid/Services/ASRService.swift b/Sources/Fluid/Services/ASRService.swiftindex d06fc8e2..ee83e987 100644--- a/Sources/Fluid/Services/ASRService.swift+++ b/Sources/Fluid/Services/ASRService.swift@@ -3490,6 +3490,18 @@ final class ASRService: ObservableObject {}}+ func transcribeMeetingSamplesWithTimings(+ _ samples: [Float],+ provider: any TranscriptionProvider+ ) async throws -> (result: ASRTranscriptionResult, words: [ASRWordTiming]) {+ guard self.activeExclusiveActivity == .meeting else {+ throw ASRActivityError.activityInProgress(self.activeExclusiveActivity ?? .meeting)+ }+ return try await self.transcriptionExecutor.run { [provider] in+ try await provider.transcribeWithWordTimings(samples)+ }+ }+func transcribeMeetingFile(at fileURL: URL,provider: any TranscriptionProviderdiff --git a/Sources/Fluid/Services/FluidAudioProvider.swift b/Sources/Fluid/Services/FluidAudioProvider.swiftindex 64d3d988..ee3837a2 100644--- a/Sources/Fluid/Services/FluidAudioProvider.swift+++ b/Sources/Fluid/Services/FluidAudioProvider.swift@@ -30,6 +30,10 @@ final class FluidAudioProvider: TranscriptionProvider {true}+ /// Boosting rescores tokens without moving their timings. Checked up front so a chunk that+ /// cannot be aligned is never transcribed twice.+ var supportsWordTimings: Bool { !self.isWordBoostingActive }+private var streamingAsrManager: AsrManager?private var finalAsrManager: AsrManager?private var latestStreamingPreviewText: String = ""@@ -52,6 +56,12 @@ final class FluidAudioProvider: TranscriptionProvider {self.configureWordBoosting = configureWordBoosting}+ #if DEBUG+ func setWordBoostingActiveForTesting(_ active: Bool) {+ self.isWordBoostingActive = active+ }+ #endif+func prepare(progressHandler: ((ModelPreparationProgress) -> Void)? = nil) async throws {try Task.checkCancellation()guard self.isReady == false else { return }@@ -397,9 +407,9 @@ final class FluidAudioProvider: TranscriptionProvider {// manager so the user still gets a transcription (just without CTC rescoring).do {let startedAt = Date().timeIntervalSince1970- let result = try await self.transcribeFinalResult(samples, manager: manager)- self.logFinalBenchmark(samples: samples, text: result.text, startedAt: startedAt, usedFallback: false)- return result+ let outcome = try await self.transcribeFinalResult(samples, manager: manager)+ self.logFinalBenchmark(samples: samples, text: outcome.result.text, startedAt: startedAt, usedFallback: false)+ return outcome.result} catch {guard let fallback = self.streamingAsrManager, fallback !== manager else {throw error@@ -409,9 +419,9 @@ final class FluidAudioProvider: TranscriptionProvider {source: "FluidAudioProvider")let startedAt = Date().timeIntervalSince1970- let result = try await self.transcribeFinalResult(samples, manager: fallback)- self.logFinalBenchmark(samples: samples, text: result.text, startedAt: startedAt, usedFallback: true)- return result+ let outcome = try await self.transcribeFinalResult(samples, manager: fallback)+ self.logFinalBenchmark(samples: samples, text: outcome.result.text, startedAt: startedAt, usedFallback: true)+ return outcome.result}}@@ -488,7 +498,44 @@ final class FluidAudioProvider: TranscriptionProvider {self.incrementalAcceptedSampleCount = 0}- private func transcribeFinalResult(_ samples: [Float], manager: AsrManager) async throws -> ASRTranscriptionResult {+ func transcribeWithWordTimings(_ samples: [Float]) async throws -> (result: ASRTranscriptionResult, words: [ASRWordTiming]) {+ guard let manager = self.finalAsrManager ?? self.streamingAsrManager else {+ throw NSError(+ domain: "FluidAudioProvider",+ code: -1,+ userInfo: [NSLocalizedDescriptionKey: "ASR manager not initialized"]+ )+ }++ let outcome: (result: ASRTranscriptionResult, tokenTimings: [TokenTiming]?, textMayBeCorrected: Bool)+ do {+ let startedAt = Date().timeIntervalSince1970+ outcome = try await self.transcribeFinalResult(samples, manager: manager)+ self.logFinalBenchmark(samples: samples, text: outcome.result.text, startedAt: startedAt, usedFallback: false, source: "meeting")+ } catch {+ guard let fallback = self.streamingAsrManager, fallback !== manager else {+ throw error+ }+ let startedAt = Date().timeIntervalSince1970+ outcome = try await self.transcribeFinalResult(samples, manager: fallback)+ self.logFinalBenchmark(samples: samples, text: outcome.result.text, startedAt: startedAt, usedFallback: true, source: "meeting")+ }++ guard !outcome.textMayBeCorrected, let tokenTimings = outcome.tokenTimings else {+ return (outcome.result, [])+ }+ return (outcome.result, Self.makeWordTimings(from: tokenTimings))+ }++ static func makeWordTimings(from tokenTimings: [TokenTiming]) -> [ASRWordTiming] {+ WordAudioChunkExtractor.words(from: tokenTimings).map {+ ASRWordTiming(text: $0.text, start: $0.startTime, end: $0.endTime)+ }+ }++ private func transcribeFinalResult(+ _ samples: [Float], manager: AsrManager+ ) async throws -> (result: ASRTranscriptionResult, tokenTimings: [TokenTiming]?, textMayBeCorrected: Bool) {let matchingEnabled = SettingsStore.shared.pronunciationMatchingEnabled && samples.count <= 16_000 * 15let dictionaryLabels = Self.dictionaryLabels(from: SettingsStore.shared.customDictionaryEntries@@ -509,13 +556,15 @@ final class FluidAudioProvider: TranscriptionProvider {} else {profiles = []}+ // Rewritten text leaves timings on the old tokens; realigning would revert corrections.+ let textMayBeCorrected = self.isWordBoostingActive || !profiles.isEmptyawait manager.setPronunciationCustomizationEnabled(!profiles.isEmpty)do {let result = try await manager.transcribe(samples, source: AudioSource.microphone)let features = await manager.consumePronunciationEncoderFeatures()await manager.setPronunciationCustomizationEnabled(false)guard let features, !profiles.isEmpty else {- return ASRTranscriptionResult(text: result.text, confidence: result.confidence)+ return (ASRTranscriptionResult(text: result.text, confidence: result.confidence), result.tokenTimings, textMayBeCorrected)}let startedAt = Date().timeIntervalSince1970let corrected = self.applyPronunciationMatches(result: result, features: features, profiles: profiles)@@ -524,7 +573,7 @@ final class FluidAudioProvider: TranscriptionProvider {"PRONUNCIATION_MATCH profiles=\(profiles.count) elapsedMs=\(elapsedMs) changed=\(corrected != result.text)",source: "PronunciationMatching")- return ASRTranscriptionResult(text: corrected, confidence: result.confidence)+ return (ASRTranscriptionResult(text: corrected, confidence: result.confidence), result.tokenTimings, textMayBeCorrected)} catch {_ = await manager.consumePronunciationEncoderFeatures()await manager.setPronunciationCustomizationEnabled(false)diff --git a/Sources/Fluid/Services/Meeting/MeetingProcessingPipeline.swift b/Sources/Fluid/Services/Meeting/MeetingProcessingPipeline.swiftindex 73aedc33..70413349 100644--- a/Sources/Fluid/Services/Meeting/MeetingProcessingPipeline.swift+++ b/Sources/Fluid/Services/Meeting/MeetingProcessingPipeline.swift@@ -278,7 +278,7 @@ private final class MeetingProviderLanguagePin {@MainActorfinal class MeetingProcessingPipeline: MeetingProcessingControlling {- static let pipelineVersion = 2+ static let pipelineVersion = 3/// Conversational pauses run to a few seconds; 1.0s fragmented turns and starved the ASR.static let meetingTurnMergeGapSeconds = 3.0@@ -679,18 +679,7 @@ final class MeetingProcessingPipeline: MeetingProcessingControlling {let diarization = try await diarizer.diarizeWithProfiles(fileURL: url, maxGapSeconds: Self.meetingTurnMergeGapSeconds)let turns = diarization.turnsif !turns.isEmpty {- var allTurns: [(index: Int, turn: SpeakerDiarizationService.SpeakerTurn, text: String)] = []- for (index, turn) in turns.enumerated() {- let text = try await self.transcribeTurn(- turn,- fileURL: url,- provider: context.provider,- languageCode: context.languageCode,- asrService: context.asrService,- languagePin: context.languagePin- )- allTurns.append((index, turn, text))- }+ let allTurns = try await self.transcribeTurns(turns, chunkID: chunk.id, trackKind: .applicationAudio, fileURL: url, context: context)guard let transcribedTurns = MeetingTurnSelection.keepingNonEmpty(allTurns) else {throw LabeledPassError.emptyTurn}@@ -852,18 +841,7 @@ final class MeetingProcessingPipeline: MeetingProcessingControlling {let diarization = try await diarizer.diarizeWithProfiles(fileURL: url, maxGapSeconds: Self.meetingTurnMergeGapSeconds)let turns = diarization.turnsif !turns.isEmpty {- var allTurns: [(index: Int, turn: SpeakerDiarizationService.SpeakerTurn, text: String)] = []- for (index, turn) in turns.enumerated() {- let text = try await self.transcribeTurn(- turn,- fileURL: url,- provider: context.provider,- languageCode: context.languageCode,- asrService: context.asrService,- languagePin: context.languagePin- )- allTurns.append((index, turn, text))- }+ let allTurns = try await self.transcribeTurns(turns, chunkID: chunk.id, trackKind: .microphone, fileURL: url, context: context)guard let transcribedTurns = MeetingTurnSelection.keepingNonEmpty(allTurns) else {throw LabeledPassError.emptyTurn}@@ -1048,6 +1026,158 @@ final class MeetingProcessingPipeline: MeetingProcessingControlling {}}+ /// Each word lands in at most one turn: max overlap wins, ties to the lower index.+ nonisolated static func assignWords(+ _ words: [ASRWordTiming],+ toTurns turns: [(index: Int, start: TimeInterval, end: TimeInterval)],+ nearestTolerance: TimeInterval = 0.5+ ) -> (byTurn: [Int: String], unassigned: Int) {+ var accumulated: [Int: [String]] = [:]+ var unassignedCount = 0++ for word in words {+ // Half-open [start, end): a boundary instant scores 0 on both sides, never doubles.+ let overlaps = turns.map { turn in+ (index: turn.index, overlap: max(0, min(word.end, turn.end) - max(word.start, turn.start)))+ }+ if let maxOverlap = overlaps.map(\.overlap).max(), maxOverlap > 0,+ let winner = overlaps.filter({ $0.overlap == maxOverlap }).min(by: { $0.index < $1.index })+ {+ accumulated[winner.index, default: []].append(word.text)+ continue+ }++ let distances = turns.map { turn -> (index: Int, distance: TimeInterval) in+ let distance: TimeInterval+ if word.end <= turn.start {+ distance = turn.start - word.end+ } else if word.start >= turn.end {+ distance = word.start - turn.end+ } else {+ distance = 0+ }+ return (turn.index, distance)+ }+ if let nearest = distances.min(by: { $0.distance < $1.distance || ($0.distance == $1.distance && $0.index < $1.index) }),+ nearest.distance <= nearestTolerance+ {+ accumulated[nearest.index, default: []].append(word.text)+ } else {+ unassignedCount += 1+ }+ }++ return (accumulated.mapValues { $0.joined(separator: " ") }, unassignedCount)+ }++ /// Above this share of unplaceable words the timings are not trustworthy enough to keep.+ static let maxUnassignedWordFraction = 0.25++ /// Words outside every turn attach to the nearest turn within this window. Zero on the+ /// microphone, where bleed would otherwise be credited to the user as their own words.+ nonisolated static func wordAttachmentTolerance(for trackKind: MeetingAudioTrackKind) -> TimeInterval {+ trackKind == .microphone ? 0 : 0.5+ }++ /// The chunk-level catch classifies on these; swallowing one would mislabel the chunk.+ nonisolated static func alignmentErrorMustPropagate(_ error: Error) -> Bool {+ if error is CancellationError { return true }+ if case MeetingProcessingError.audioUnreadable = error { return true }+ return false+ }++ nonisolated static func tooManyWordsUnassigned(unassigned: Int, totalWords: Int) -> Bool {+ totalWords > 0 && Double(unassigned) > Double(totalWords) * Self.maxUnassignedWordFraction+ }++ private func transcribeTurnsWordAligned(+ _ turns: [SpeakerDiarizationService.SpeakerTurn],+ chunkID: MeetingAudioChunkID,+ trackKind: MeetingAudioTrackKind,+ fileURL: URL,+ context: ProcessingContext+ ) async throws -> [(index: Int, turn: SpeakerDiarizationService.SpeakerTurn, text: String)]? {+ let samples = try await Task.detached(priority: .userInitiated) {+ try Self.readSamples(fileURL: fileURL, startSeconds: 0, endSeconds: nil)+ }.value+ guard samples.count >= 16_000 else { return nil }++ context.languagePin.apply()+ let outcome = try await context.asrService.transcribeMeetingSamplesWithTimings(+ samples,+ provider: context.provider+ )+ guard !outcome.words.isEmpty else { return nil }++ let assignment = Self.assignWords(+ outcome.words,+ toTurns: turns.enumerated().map { (index: $0.offset, start: $0.element.startSeconds, end: $0.element.endSeconds) },+ nearestTolerance: Self.wordAttachmentTolerance(for: trackKind)+ )+ guard !Self.tooManyWordsUnassigned(unassigned: assignment.unassigned, totalWords: outcome.words.count) else {+ DebugLogger.shared.warning(+ "Meeting word alignment chunk \(chunkID): \(assignment.unassigned)/\(outcome.words.count) words unassigned, falling back to per-turn",+ source: "MeetingProcessingPipeline"+ )+ return nil+ }+ if assignment.unassigned > 0 {+ DebugLogger.shared.warning(+ "Meeting word alignment chunk \(chunkID): dropped \(assignment.unassigned) of \(outcome.words.count) unplaceable word(s)",+ source: "MeetingProcessingPipeline"+ )+ }+ return turns.enumerated().map { index, turn in+ (index, turn, assignment.byTurn[index] ?? "")+ }+ }++ private func transcribeTurns(+ _ turns: [SpeakerDiarizationService.SpeakerTurn],+ chunkID: MeetingAudioChunkID,+ trackKind: MeetingAudioTrackKind,+ fileURL: URL,+ context: ProcessingContext+ ) async throws -> [(index: Int, turn: SpeakerDiarizationService.SpeakerTurn, text: String)] {+ if !context.provider.supportsWordTimings {+ DebugLogger.shared.log(+ "Meeting chunk \(chunkID): provider \(context.provider.name) exposes no word timings; transcribing per turn",+ source: "MeetingProcessingPipeline"+ )+ } else {+ do {+ if let aligned = try await self.transcribeTurnsWordAligned(+ turns,+ chunkID: chunkID,+ trackKind: trackKind,+ fileURL: fileURL,+ context: context+ ) {+ return aligned+ }+ } catch {+ if Self.alignmentErrorMustPropagate(error) { throw error }+ DebugLogger.shared.warning(+ "Meeting word alignment chunk \(chunkID) failed; falling back to per-turn: \(error)",+ source: "MeetingProcessingPipeline"+ )+ }+ }+ var allTurns: [(index: Int, turn: SpeakerDiarizationService.SpeakerTurn, text: String)] = []+ for (index, turn) in turns.enumerated() {+ let text = try await self.transcribeTurn(+ turn,+ fileURL: fileURL,+ provider: context.provider,+ languageCode: context.languageCode,+ asrService: context.asrService,+ languagePin: context.languagePin+ )+ allTurns.append((index, turn, text))+ }+ return allTurns+ }+private func transcribeTurn(_ turn: SpeakerDiarizationService.SpeakerTurn,fileURL: URL,diff --git a/Sources/Fluid/Services/TranscriptionProvider.swift b/Sources/Fluid/Services/TranscriptionProvider.swiftindex 78903834..78fed611 100644--- a/Sources/Fluid/Services/TranscriptionProvider.swift+++ b/Sources/Fluid/Services/TranscriptionProvider.swift@@ -122,6 +122,12 @@ struct ASRTranscriptionResult {}}+nonisolated struct ASRWordTiming: Sendable {+ let text: String+ let start: TimeInterval+ let end: TimeInterval+}+// MARK: - Transcription Provider Protocol/// Protocol that abstracts speech-to-text transcription.@@ -156,6 +162,11 @@ protocol TranscriptionProvider {/// Providers can bypass final-output transforms that would distort the saved phrase.func transcribeDictionaryTraining(_ samples: [Float]) async throws -> ASRTranscriptionResult+ /// Meeting word alignment only; dictation never calls the method this gates.+ var supportsWordTimings: Bool { get }++ func transcribeWithWordTimings(_ samples: [Float]) async throws -> (result: ASRTranscriptionResult, words: [ASRWordTiming])+/// Whether this provider prefers to handle long-form file transcription itself./// This is useful when the backend already has model-native long-audio chunking/reassembly.var prefersNativeFileTranscription: Bool { get }@@ -180,6 +191,11 @@ extension TranscriptionProvider {func clearCache() async throws {}var shouldClearCacheAfterCancellation: Bool { true }var prefersNativeFileTranscription: Bool { false }+ var supportsWordTimings: Bool { false }++ func transcribeWithWordTimings(_ samples: [Float]) async throws -> (result: ASRTranscriptionResult, words: [ASRWordTiming]) {+ (try await self.transcribe(samples), [])+ }func transcribeStreaming(_ samples: [Float]) async throws -> ASRTranscriptionResult {try await self.transcribe(samples)}diff --git a/Tests/FluidDictationIntegrationTests/MeetingRecoveryTests.swift b/Tests/FluidDictationIntegrationTests/MeetingRecoveryTests.swiftindex 833573c9..603d13d9 100644--- a/Tests/FluidDictationIntegrationTests/MeetingRecoveryTests.swift+++ b/Tests/FluidDictationIntegrationTests/MeetingRecoveryTests.swift@@ -849,14 +849,15 @@ final class MeetingRecoveryTests: XCTestCase {private func makeCheckpoint(sessionID: MeetingSessionID = UUID(),completedTrackID: MeetingAudioTrackID = UUID(),- trackFingerprints: [MeetingAudioTrackID: [MeetingProcessingCheckpoint.ChunkFingerprint]] = [:]+ trackFingerprints: [MeetingAudioTrackID: [MeetingProcessingCheckpoint.ChunkFingerprint]] = [:],+ asrModel: String = "base") -> MeetingProcessingCheckpoint {MeetingProcessingCheckpoint(version: MeetingProcessingCheckpoint.currentVersion,sessionID: sessionID,pipelineVersion: MeetingProcessingPipeline.pipelineVersion,asrProvider: "apple",- asrModel: "base",+ asrModel: asrModel,languageCode: "en",completedTrackID: completedTrackID,trackFingerprints: trackFingerprints,@@ -950,6 +951,40 @@ final class MeetingRecoveryTests: XCTestCase {))}+ /// A checkpoint must validate a resume on the same model and be invalidated by a model+ /// switch, so a resume can never continue from a stale, wrong-model checkpoint.+ func testResumedVersusCleanCheckpointEquivalenceAcrossAModelSwitch() throws {+ let session = self.makeSession(state: .interrupted, endedAt: Date())+ let trackID = UUID()+ let fingerprint = MeetingProcessingCheckpoint.ChunkFingerprint(id: UUID(), byteCount: 128, sha256: "abc123")+ let fingerprints: [MeetingAudioTrackID: [MeetingProcessingCheckpoint.ChunkFingerprint]] = [trackID: [fingerprint]]++ let checkpoint = self.makeCheckpoint(+ sessionID: session.id, completedTrackID: trackID, trackFingerprints: fingerprints,+ asrModel: "parakeet-tdt-v2"+ )++ XCTAssertTrue(checkpoint.isValid(+ session: session,+ pipelineVersion: MeetingProcessingPipeline.pipelineVersion,+ provider: checkpoint.asrProvider,+ model: "parakeet-tdt-v2",+ language: "en",+ completedTrackID: trackID,+ expectedFingerprints: fingerprints+ ), "resuming with the same resolved model as the clean run must validate")++ XCTAssertFalse(checkpoint.isValid(+ session: session,+ pipelineVersion: MeetingProcessingPipeline.pipelineVersion,+ provider: checkpoint.asrProvider,+ model: "whisper-large-turbo",+ language: "en",+ completedTrackID: trackID,+ expectedFingerprints: fingerprints+ ), "a model switch between runs must invalidate the checkpoint rather than resume stale speaker/segment state")+ }+// MARK: - Slice 3, test 3: coordinator deletes checkpoint after successfunc testCoordinatorDeletesCheckpointAfterProcessingSuccess() async throws {@@ -1303,6 +1338,43 @@ final class MeetingRecoveryTests: XCTestCase {XCTAssertFalse(coordinator.canUndoCorrection(sessionID: session.id))}+ /// When the inverse's target is gone, undo drops the stale entry, throws+ /// `.noCorrectionToUndo`, and never saves — whatever other mutation produced is left as-is.+ func testUndoWithGoneInverseTargetDropsEntryAndSavesNothing() async throws {+ let dir = self.makeTempDirectory()+ defer { try? FileManager.default.removeItem(at: dir) }+ let store = MeetingSessionStore(rootDirectory: dir)+ let (session, _, speakerB, segmentID) = self.makeCorrectionSession(state: .completed)+ try await store.create(session)++ let coordinator = MeetingSessionCoordinator(+ store: store, capture: StubCaptureController(), processing: StubProcessingController(), audioArbiter: StubArbiter()+ )++ _ = try await coordinator.reassignSegment(sessionID: session.id, segmentID: segmentID, to: speakerB)+ XCTAssertTrue(coordinator.canUndoCorrection(sessionID: session.id))++ // Segment vanishes from under the undo stack: save without it, bypassing the coordinator.+ let loaded = try await store.load(id: session.id)+ var mutated = try XCTUnwrap(loaded)+ mutated.transcriptSegments.removeAll { $0.id == segmentID }+ try await store.save(mutated)++ do {+ _ = try await coordinator.undoTranscriptCorrection(sessionID: session.id)+ XCTFail("Expected undo to throw when the inverse target no longer exists")+ } catch let error as MeetingCoordinatorError {+ guard case .noCorrectionToUndo = error else {+ return XCTFail("Expected .noCorrectionToUndo, got \(error)")+ }+ }++ XCTAssertFalse(coordinator.canUndoCorrection(sessionID: session.id), "the stale entry must be dropped")+ let reloaded = try await store.load(id: session.id)+ XCTAssertEqual(reloaded?.transcriptSegments.count, mutated.transcriptSegments.count, "undo must never save on a dropped entry")+ XCTAssertFalse(reloaded?.transcriptSegments.contains { $0.id == segmentID } ?? true)+ }+func testCorrectionRefusedWhileProcessingSameSession() async throws {let dir = self.makeTempDirectory()defer { try? FileManager.default.removeItem(at: dir) }@@ -2161,6 +2233,92 @@ final class MeetingRecoveryTests: XCTestCase {}}+ /// A contended past-deadline session must re-arm within 60s instead of being skipped forever.+ /// Caught mid-`retryProcessing`: the sweep reads disk state (`.completed`, eligible) while+ /// `isSafeToSweep` reads in-memory state (`.processing`, contended) — the race the re-arm exists for.+ func testContendedSweepPastDeadlineSessionReArmsWithinSixtySeconds() async throws {+ let dir = self.makeTempDirectory()+ defer { try? FileManager.default.removeItem(at: dir) }+ let realStore = MeetingSessionStore(rootDirectory: dir)+ var (session, _, _, _) = self.makeCorrectionSession(state: .completed)+ session.startedAt = Date().addingTimeInterval(-30 * 24 * 60 * 60)+ session.endedAt = session.startedAt+ try await realStore.create(session)++ let recorder = EventRecorder()+ let gatedStore = GatedThrowingSaveStore(inner: realStore, recorder: recorder)+ await gatedStore.closeGate()+ let coordinator = MeetingSessionCoordinator(+ store: gatedStore, capture: StubCaptureController(), processing: StubProcessingController(), audioArbiter: StubArbiter()+ )+ await coordinator.ensureRestored()++ try await self.withRetentionPolicy(.days7) {+ let retryTask = Task { try await coordinator.retryProcessing(sessionID: session.id) }++ for _ in 0..<40 {+ if await recorder.events.contains("store.save.start") { break }+ try await Task.sleep(nanoseconds: 5_000_000)+ }+ let eventsSoFar = await recorder.events+ XCTAssertTrue(eventsSoFar.contains("store.save.start"))+ guard case .processing = coordinator.state else {+ await gatedStore.openGate()+ _ = try? await retryTask.value+ return XCTFail("retryProcessing did not reach .processing before its save landed")+ }++ await coordinator.sweepExpiredAudio()+ let reloaded = try await realStore.load(id: session.id)+ XCTAssertNil(reloaded?.retention.audioDeletedAt, "a contended session must not be swept")+ XCTAssertTrue(coordinator.hasScheduledRetentionSweep, "a contended past-deadline session must re-arm a retry")++ await gatedStore.openGate()+ _ = try await retryTask.value+ }+ }++ /// deleteSession racing shutdownForTermination must never resurrect the session as an+ /// interrupted offer: delete wins, coordinator settles at `.idle`, session gone from disk.+ func testDeleteVersusTerminationLeavesNoResurrectionAndSettlesAtIdle() async throws {+ let dir = self.makeTempDirectory()+ defer { try? FileManager.default.removeItem(at: dir) }+ let realStore = MeetingSessionStore(rootDirectory: dir)+ let session = self.makeSession(+ state: .interrupted,+ endedAt: Date(),+ audioTracks: [self.makeMicrophoneTrack(chunks: [self.makeFinalizedChunk()])]+ )+ try await realStore.create(session)++ let recorder = EventRecorder()+ let gatedStore = GatedDeleteStore(inner: realStore, recorder: recorder)+ let coordinator = MeetingSessionCoordinator(+ store: gatedStore, capture: StubCaptureController(), processing: StubProcessingController(), audioArbiter: StubArbiter()+ )+ await coordinator.ensureRestored()+ XCTAssertEqual(coordinator.activeSession?.id, session.id)++ let deleteTask = Task { try await coordinator.deleteSession(id: session.id) }+ for _ in 0..<40 {+ if await recorder.events.contains("store.delete.start") { break }+ try await Task.sleep(nanoseconds: 5_000_000)+ }+ let eventsSoFar = await recorder.events+ XCTAssertTrue(eventsSoFar.contains("store.delete.start"))++ // Returns promptly mid-delete: it waits on isMutatingSession, not isDeleting.+ await coordinator.shutdownForTermination()++ await gatedStore.openGate()+ try await deleteTask.value++ XCTAssertNil(coordinator.activeSession)+ XCTAssertEqual(coordinator.state, .idle)+ let reloaded = try? await realStore.load(id: session.id)+ XCTAssertNil(reloaded, "deleteSession must have the final say; termination must not resurrect it")+ }+/// O5: renaming a speaker to its current name must be a true no-op — no save, no undo entry.func testNoOpRenameDoesNotPushUndoOrTouchStore() async throws {let dir = self.makeTempDirectory()diff --git a/Tests/FluidDictationIntegrationTests/SpeakerTurnMergingTests.swift b/Tests/FluidDictationIntegrationTests/SpeakerTurnMergingTests.swiftindex 33aaadbb..a7ca593a 100644--- a/Tests/FluidDictationIntegrationTests/SpeakerTurnMergingTests.swift+++ b/Tests/FluidDictationIntegrationTests/SpeakerTurnMergingTests.swift@@ -3,6 +3,7 @@ import Foundationimport XCTest#if arch(arm64)+import FluidAudio// `mergeAdjacentTurns` joins consecutive turns from the same speaker across short pauses.// When clustering collapses to one label, that intentionally produces one longer segment;@@ -529,6 +530,38 @@ final class SpeakerLabeledTranscriptionPolicyTests: XCTestCase {]XCTAssertNil(SpeakerLabeledTranscriptionPolicy.assembleTurns(turns))++/// Word-timing mapping is meeting-only: `transcribeFinal` never reaches it.+final class FluidAudioProviderWordTimingTests: XCTestCase {+ func testMakeWordTimingsReconstructsWordsFromTokenTimings() {+ let tokenTimings: [TokenTiming] = [+ TokenTiming(token: "▁Hello", tokenId: 1, startTime: 0.0, endTime: 0.4, confidence: 0.9),+ TokenTiming(token: "▁world", tokenId: 2, startTime: 0.5, endTime: 0.9, confidence: 0.9),+ ]++ let words = FluidAudioProvider.makeWordTimings(from: tokenTimings)++ XCTAssertEqual(words.map(\.text), ["Hello", "world"])+ XCTAssertEqual(words.first?.start, 0.0)+ XCTAssertEqual(words.first?.end, 0.4)+ XCTAssertEqual(words.last?.start, 0.5)+ XCTAssertEqual(words.last?.end, 0.9)+ }++ func testMakeWordTimingsEmptyInputProducesNoWords() {+ XCTAssertTrue(FluidAudioProvider.makeWordTimings(from: []).isEmpty)+ }++ /// With boosting active, alignment must be off so no second full ASR pass ever runs.+ func testSupportsWordTimingsDisabledWhileBoostingActive() {+ let provider = FluidAudioProvider(configureWordBoosting: true)+ XCTAssertTrue(provider.supportsWordTimings, "no boosting configured yet: timings should be available")++ provider.setWordBoostingActiveForTesting(true)+ XCTAssertFalse(provider.supportsWordTimings, "boosting active: word alignment must be disabled")++ provider.setWordBoostingActiveForTesting(false)+ XCTAssertTrue(provider.supportsWordTimings)}}@@ -1306,6 +1339,137 @@ final class MeetingSessionModelTests: XCTestCase {XCTAssertEqual($0 as? MeetingModelValidationError, expected, file: file, line: line)}}++ // MARK: - assignWords++ private func word(_ text: String, _ start: Double, _ end: Double) -> ASRWordTiming {+ ASRWordTiming(text: text, start: start, end: end)+ }++ func testAssignWordsEmptyInput() {+ let result = MeetingProcessingPipeline.assignWords([], toTurns: [(index: 0, start: 0, end: 5)])+ XCTAssertTrue(result.byTurn.isEmpty)+ XCTAssertEqual(result.unassigned, 0)+ }++ func testAssignWordsWordFullyInsideATurn() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("hello", 1, 2)],+ toTurns: [(index: 0, start: 0, end: 5), (index: 1, start: 5, end: 10)]+ )+ XCTAssertEqual(result.byTurn[0], "hello")+ XCTAssertNil(result.byTurn[1])+ XCTAssertEqual(result.unassigned, 0)+ }++ func testAssignWordsStraddlingBoundaryGoesToMaxOverlapExactlyOnce() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("straddle", 4, 8)],+ toTurns: [(index: 0, start: 0, end: 5), (index: 1, start: 5, end: 8)]+ )+ XCTAssertNil(result.byTurn[0])+ XCTAssertEqual(result.byTurn[1], "straddle")+ XCTAssertEqual(result.unassigned, 0)+ }++ func testAssignWordsBoundaryInstantIsNotDoubleAssigned() {+ // Overlaps neither half-open turn, so nearest-turn attachment takes it (lower index wins).+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("tick", 5, 5)],+ toTurns: [(index: 0, start: 0, end: 5), (index: 1, start: 5, end: 10)]+ )+ XCTAssertEqual(result.byTurn[0], "tick")+ XCTAssertNil(result.byTurn[1])+ XCTAssertEqual(result.unassigned, 0)+ }++ func testAssignWordsInGapWithinToleranceAttachesToNearest() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("gap", 5.2, 5.3)],+ toTurns: [(index: 0, start: 0, end: 5), (index: 1, start: 5.5, end: 10)],+ nearestTolerance: 0.5+ )+ XCTAssertEqual(result.byTurn[0], "gap")+ XCTAssertNil(result.byTurn[1])+ XCTAssertEqual(result.unassigned, 0)+ }++ func testAssignWordsFarOutsideCountsAsUnassigned() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("far", 20, 21)],+ toTurns: [(index: 0, start: 0, end: 5)],+ nearestTolerance: 0.5+ )+ XCTAssertTrue(result.byTurn.isEmpty)+ XCTAssertEqual(result.unassigned, 1)+ }++ func testAssignWordsPreservesChronologicalOrderWithinATurn() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("one", 0, 1), self.word("two", 1, 2), self.word("three", 2, 3)],+ toTurns: [(index: 0, start: 0, end: 5)]+ )+ XCTAssertEqual(result.byTurn[0], "one two three")+ }++ // MARK: - Speaker bleed must never be attached to a local turn++ func testWordAttachmentToleranceIsZeroOnTheMicrophoneTrack() {+ XCTAssertEqual(MeetingProcessingPipeline.wordAttachmentTolerance(for: .microphone), 0)+ XCTAssertGreaterThan(MeetingProcessingPipeline.wordAttachmentTolerance(for: .applicationAudio), 0)+ }++ func testMicrophoneToleranceLeavesOutOfTurnWordsUnassignedInsteadOfCreditingTheUser() {+ // Bled 0.3s after the local turn ends: inside the application window, but not the user's.+ let bled = self.word("bleed", 5.3, 5.8)+ let turns = [(index: 0, start: 0.0, end: 5.0)]++ let onMicrophone = MeetingProcessingPipeline.assignWords(+ [self.word("mine", 1, 2), bled],+ toTurns: turns,+ nearestTolerance: MeetingProcessingPipeline.wordAttachmentTolerance(for: .microphone)+ )+ XCTAssertEqual(onMicrophone.byTurn[0], "mine")+ XCTAssertEqual(onMicrophone.unassigned, 1)++ let onApplication = MeetingProcessingPipeline.assignWords(+ [self.word("mine", 1, 2), bled],+ toTurns: turns,+ nearestTolerance: MeetingProcessingPipeline.wordAttachmentTolerance(for: .applicationAudio)+ )+ XCTAssertEqual(onApplication.byTurn[0], "mine bleed", "remote speakers still absorb near-boundary words")+ }++ func testZeroToleranceStillKeepsWordsOverlappingTheirTurn() {+ let result = MeetingProcessingPipeline.assignWords(+ [self.word("straddle", 4.8, 5.4)],+ toTurns: [(index: 0, start: 0, end: 5)],+ nearestTolerance: 0+ )+ XCTAssertEqual(result.byTurn[0], "straddle")+ XCTAssertEqual(result.unassigned, 0)+ }++ // MARK: - Alignment failures fall back, except what the chunk catch classifies on++ func testAlignmentErrorPropagationClassification() {+ XCTAssertTrue(MeetingProcessingPipeline.alignmentErrorMustPropagate(CancellationError()))+ XCTAssertTrue(MeetingProcessingPipeline.alignmentErrorMustPropagate(MeetingProcessingError.audioUnreadable))+ XCTAssertFalse(MeetingProcessingPipeline.alignmentErrorMustPropagate(MeetingProcessingError.modelUnavailable))+ XCTAssertFalse(+ MeetingProcessingPipeline.alignmentErrorMustPropagate(+ NSError(domain: "test", code: 1)+ )+ )+ }++ // MARK: - Aligned-path safety valve++ func testTooManyWordsUnassignedTripsAboveTwentyFivePercent() {+ XCTAssertTrue(MeetingProcessingPipeline.tooManyWordsUnassigned(unassigned: 26, totalWords: 100))+ XCTAssertFalse(MeetingProcessingPipeline.tooManyWordsUnassigned(unassigned: 25, totalWords: 100))+ XCTAssertFalse(MeetingProcessingPipeline.tooManyWordsUnassigned(unassigned: 0, totalWords: 0))+ }}private enum MeetingModelFixture {