Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.
Rust acpagent-client-protocolai-agentsai-assistantclaudeclaude-codecodexcoding-agentscursorgeminigemini-cligrok-bot-alternativeiosiphonelinuxmacosmcpopen-sourcerustself-hosted
feat(apps): voice calls reconnect and pick up again without hanging up
leepokai4db7aaef6fa3
diff --git a/apps/android/WhatsNew.md b/apps/android/WhatsNew.mdindex 19a21e8d..b2c6da97 100644--- a/apps/android/WhatsNew.md+++ b/apps/android/WhatsNew.md@@ -12,10 +12,12 @@ else a generic "Bug fixes and improvements" line. Delete old sections.### zh-Hant- Bot 工作時,Gemini 與 OpenAI 的語音會主動告訴你它在做什麼、有沒有等你核准或遇到問題,做完就念出結果。- 通話中短暫斷線再連上時,錯過的回覆會補念給你聽。+- Gemini 或 OpenAI 通話斷線時會自動重新連上並接著聊,不必重打。### en-US- While the bot works, the Gemini and OpenAI voice says what it's doing, when an approval waits or something fails, and reads the result.- Replies missed while a call reconnects are read once it's back.+- A dropped Gemini or OpenAI call reconnects without hanging up.## 2.25.0diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AnalyticsEvent.kt b/apps/android/app/src/main/kotlin/com/codync/android/AnalyticsEvent.ktindex 7af41b80..dae29f86 100644--- a/apps/android/app/src/main/kotlin/com/codync/android/AnalyticsEvent.kt+++ b/apps/android/app/src/main/kotlin/com/codync/android/AnalyticsEvent.kt@@ -68,6 +68,12 @@ internal sealed class AnalyticsEvent(val name: String) {data class VoiceCallStarted(val engine: VoiceEngineKind) : AnalyticsEvent("voice_call_started") {override val properties get() = mapOf("engine" to engine.json)}+ /** How one engine of a call fared ([com.codync.android.core.CallQuality]). */+ data class VoiceCallQuality(val engine: VoiceEngineKind, val report: com.codync.android.core.CallQuality.Report) : AnalyticsEvent("voice_call_quality") {+ override val properties get() = mapOf("engine" to engine.json, "reconnects" to JsonPrimitive(report.reconnects),+ "stuck" to JsonPrimitive(report.stuck.wire), "failure" to JsonPrimitive(report.failure?.wire ?: "none")) ++ listOfNotNull(report.readyMs?.let { "ready_ms" to JsonPrimitive(it) })+ }data class ScreenViewed(val screen: AnalyticsScreen) : AnalyticsEvent("screen_viewed") {override val properties get() = mapOf("screen" to screen.json)}diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.ktindex df5a7d80..fe32fadf 100644--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt@@ -31,6 +31,8 @@ internal class VoiceBinding(val id: String, val botId: String, val generation: Ivar reported = false/** The call on the computer. */var link: HostCallLink? = null+ /** Each engine's `voice_call_quality`. */+ val quality = CallQuality<VoiceEngineKind> { kind, report -> Analytics.capture(AnalyticsEvent.VoiceCallQuality(kind, report)) }}private val voiceHttp by lazy { OkHttpClient() }@@ -106,11 +108,14 @@ private fun AppStore.begin(call: VoiceBinding, provider: VoiceProvider?) {}}call.engine = engine+ call.quality.begin(kind)+ (engine as? GeminiLiveEngine)?.onStall = { contextScope.launch { if (call.engine === engine) call.quality.stalled() } }if (muted) engine.mute(true)call.cloudSince = if (provider != null) SystemClock.elapsedRealtime() else nullcontextScope.launch {val last = engine.state.first { status ->val current = voiceBinding === call && call.engine === engine+ if (current) call.quality.phase(status.phase, status.failureKind)val fallingBack = provider != null && status.phase == VoicePhase.Failedif (current && !fallingBack) {publish(call, status)@@ -198,6 +203,7 @@ internal fun AppStore.finishVoice(id: String, error: String? = null) {val call = voiceBinding?.takeIf { it.id == id }; voiceBinding = null; voiceGeneration = nullval engine = call?.engineengine?.end()+ call?.quality?.report()if (call != null) countCloudMinutes(call)mutableState.update { it.copy(voice = null, error = error ?: it.error) }PushDelivery.foreground = foreground@@ -219,6 +225,7 @@ internal suspend fun AppStore.retireVoice() {val callId = state.value.voice?.idmutableState.update { it.copy(voice = null) }if (callId != null) VoiceService.stop(getApplication(), callId)+ call?.quality?.report()call?.engine?.end(); call?.engine?.awaitEnd()}diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.ktindex 84b9703c..bb43c0b0 100644--- a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt@@ -83,7 +83,7 @@ internal fun openAppSettings(context: android.content.Context) =.semantics { contentDescription = if (speaking) "Interrupt" else bot?.name.orEmpty() }) {if (bot != null) BotAvatar(bot, state.bots, 34.dp)}- CallLevelDots(voice.level, speaking, bot?.status == "working", voice.phase == VoicePhase.Starting, color,+ CallLevelDots(voice.level, speaking, bot?.status == "working", voice.phase.connecting, color,Modifier.weight(1f).height(20.dp).semantics { contentDescription = statusText(voice, bot?.status == "working") })// Audio leaves the phone: say where.voice.provider?.let { provider ->@@ -121,6 +121,7 @@ internal fun openAppSettings(context: android.content.Context) =private fun statusText(voice: VoiceState, working: Boolean) = when (voice.phase) {VoicePhase.Starting -> "Connecting"+ VoicePhase.Reconnecting -> "Reconnecting"VoicePhase.Speaking -> "Speaking"VoicePhase.Failed -> "Can't listen"VoicePhase.Interrupted -> "Paused"diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.ktindex e3d855c3..a58dfea2 100644--- a/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt@@ -131,7 +131,7 @@ class VoiceService : Service() {return NotificationCompat.Builder(this, CHANNEL).setSmallIcon(R.drawable.ic_notification).setContentTitle("Voice chat · ${bot?.name ?: "Codync"}").setContentText(if (state.muted) "Microphone muted" else when (state.phase) {- VoicePhase.Starting -> "Connecting…"; VoicePhase.Speaking -> "Speaking"+ VoicePhase.Starting -> "Connecting…"; VoicePhase.Reconnecting -> "Reconnecting…"; VoicePhase.Speaking -> "Speaking"VoicePhase.Interrupted -> "Audio interrupted · return to resume"else -> "Listening · " + (state.provider?.title ?: "on this phone")}).setContentIntent(content).setOngoing(true).setCategory(NotificationCompat.CATEGORY_CALL)diff --git a/apps/android/app/src/test/kotlin/com/codync/android/AnalyticsTest.kt b/apps/android/app/src/test/kotlin/com/codync/android/AnalyticsTest.ktindex b7a69ac9..06ed8bda 100644--- a/apps/android/app/src/test/kotlin/com/codync/android/AnalyticsTest.kt+++ b/apps/android/app/src/test/kotlin/com/codync/android/AnalyticsTest.kt@@ -1,7 +1,10 @@package com.codync.android+import com.codync.android.core.CallQualityimport com.codync.android.core.HostErrorimport com.codync.android.core.PairingProblem+import com.codync.android.core.VoiceFailureKind+import com.codync.android.core.VoiceStuckimport java.io.IOExceptionimport kotlinx.serialization.json.JsonObjectimport org.junit.Assert.assertEquals@@ -22,6 +25,11 @@ class AnalyticsTest {assertEquals("""{"setting":"live_activities","on":false}""", json(AnalyticsEvent.SettingChanged(PhoneSetting.LiveActivities, false)))assertEquals("""{"via":"link","computer":"fbfcd4f4fe9f876c"}""", json(AnalyticsEvent.ComputerPaired(PairedVia.Link, "tZ1WMzsVltt-fj7mUDwZsg")))assertEquals("{}", json(AnalyticsEvent.SignedOut))+ assertEquals("""{"engine":"gemini","reconnects":1,"stuck":"none","failure":"connection","ready_ms":1200}""",+ json(AnalyticsEvent.VoiceCallQuality(VoiceEngineKind.Gemini, CallQuality.Report(1_200, 1, VoiceStuck.None, VoiceFailureKind.Connection))))+ // An engine that was never ready leaves `ready_ms` out.+ assertEquals("""{"engine":"on_device","reconnects":0,"stuck":"connecting","failure":"none"}""",+ json(AnalyticsEvent.VoiceCallQuality(VoiceEngineKind.OnDevice, CallQuality.Report(null, 0, VoiceStuck.Connecting, null))))}@Test fun `pairing failures map to the catalog's reasons`() {diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/CallQuality.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/CallQuality.ktnew file mode 100644index 00000000..c26c6cfa--- /dev/null+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/CallQuality.kt@@ -0,0 +1,69 @@+package com.codync.android.core++/** The first phase of a call's engine that lasted too long, for analytics (`voice_call_quality`). */+enum class VoiceStuck(val wire: String) { None("none"), Connecting("connecting"), Reconnecting("reconnecting"), Speaking("speaking") }++/**+ * How each engine of a call fares, for analytics (never content; iOS `CallQuality`): one report per engine (`E`, how+ * the app names it), made when it's replaced or the call ends — milliseconds until it was ready, reconnects, the first+ * phase stuck too long, why it failed.+ */+class CallQuality<E : Any>(private val clock: () -> Long = System::currentTimeMillis, private val capture: (E, Report) -> Unit) {+ /** `failure` is null when the engine didn't fail. */+ data class Report(val readyMs: Long?, val reconnects: Int, val stuck: VoiceStuck, val failure: VoiceFailureKind?)++ private var engine: E? = null+ private var startedAt = 0L+ private var readyMs: Long? = null+ private var reconnects = 0+ private var stuck = VoiceStuck.None+ private var failure: VoiceFailureKind? = null+ private var last = VoicePhase.Starting+ /** When the current connecting phase began. */+ private var since = 0L++ /** A new engine takes the call; the previous one's report goes out. */+ fun begin(engine: E) {+ report()+ this.engine = engine+ startedAt = clock(); since = startedAt+ readyMs = null; reconnects = 0; stuck = VoiceStuck.None; failure = null; last = VoicePhase.Starting+ }++ fun phase(phase: VoicePhase, kind: VoiceFailureKind?) {+ if (engine == null || phase == last) return+ if (last.connecting) checkStuck()+ when {+ phase == VoicePhase.Listening || phase == VoicePhase.Speaking -> {+ if (readyMs == null) readyMs = clock() - startedAt+ if (last == VoicePhase.Reconnecting) reconnects+++ }+ phase.connecting -> since = clock()+ phase == VoicePhase.Failed -> failure = kind ?: VoiceFailureKind.Connection+ else -> { }+ }+ last = phase+ }++ /** Playback stopped reporting progress after the voice's turn (the engine's watchdog took the call back). */+ fun stalled() { if (stuck == VoiceStuck.None) stuck = VoiceStuck.Speaking }++ /** The engine is done (replaced, or the call ended): its report. */+ fun report() {+ val engine = engine ?: return+ if (last.connecting) checkStuck()+ capture(engine, Report(readyMs, reconnects, stuck, failure))+ this.engine = null+ }++ private fun checkStuck() {+ if (stuck == VoiceStuck.None && clock() - since >= STUCK_MILLIS) {+ stuck = if (last == VoicePhase.Starting) VoiceStuck.Connecting else VoiceStuck.Reconnecting+ }+ }++ companion object {+ /** Connecting (or reconnecting) this long counts as stuck. */+ const val STUCK_MILLIS = 20_000L+ }+}diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.ktindex 87547e3c..27fa3d92 100644--- a/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt@@ -1,5 +1,6 @@package com.codync.android.core+import java.io.IOExceptionimport kotlinx.coroutines.*import kotlinx.coroutines.channels.Channelimport kotlinx.coroutines.flow.update@@ -27,8 +28,12 @@ class CloudSpeechEngine(parent: CoroutineScope, provider: VoiceProvider, settinginit {hop {- mic.start({ pcm, level -> hop { heard(pcm, level) } }) {- hop { fail("Couldn't use the microphone. Check its permission and other audio apps.") }+ try {+ mic.start({ pcm, level -> hop { heard(pcm, level) } }) {+ hop { fail("Couldn't use the microphone. Check its permission and other audio apps.", VoiceFailureKind.Microphone) }+ }+ } catch (error: IOException) {+ return@hop fail(error.message ?: "Couldn't use the microphone.", VoiceFailureKind.Microphone)}scope.launch {try {@@ -37,7 +42,7 @@ class CloudSpeechEngine(parent: CoroutineScope, provider: VoiceProvider, settingif (!ended && text.isNotEmpty()) speech.send(text)}} catch (cancelled: CancellationException) { throw cancelled }- catch (error: Exception) { hop { fail(error.message ?: "Couldn't transcribe what you said.") } }+ catch (error: Exception) { hop { fail(error.message ?: "Couldn't transcribe what you said.", VoiceFailureKind.Provider) } }}}}diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.ktindex dde35ff8..8a0342fa 100644--- a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt@@ -3,6 +3,8 @@ package com.codync.android.coreimport java.net.URLEncoderimport java.util.Base64import kotlinx.coroutines.CoroutineScope+import kotlinx.coroutines.Job+import kotlinx.coroutines.delayimport kotlinx.coroutines.launchimport kotlinx.coroutines.flow.updateimport kotlinx.serialization.json.*@@ -17,7 +19,7 @@ import okio.ByteString* Gemini Live over a WebSocket, phone ↔ Google directly (iOS `GeminiLiveEngine`): 16 kHz PCM up,* 24 kHz PCM down, echo-cancelled so you can talk over it. The computer runs the call ([CallLink]): it mints one-use* ephemeral tokens on the user's key and writes the voice's instructions and tools, again for each session; a session- * the server ends (`goAway`) is resumed with a fresh one.+ * the server ends (`goAway`) is resumed with a fresh one, and a dropped one is tried again.*/class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private val model: String, private val voice: String,private val link: CallLink,@@ -27,6 +29,10 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateprivate var socket: WebSocket? = null/** Setup is done on the current socket. */private var ready = false+ /** A session was set up once: a drop after that is tried again instead of ending the call. */+ private var connected = false+ /** The server said it ends this connection (`goAway`): carry on in a resumed one at once. */+ private var replacing = falseprivate var resumeHandle: String? = nullprivate var reconnects = 0private var audioRunning = false@@ -42,6 +48,10 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateprivate var session: CallSession? = null/** Tool calls still running on the computer: replies wait, so the model gets each answer first. */private var toolsRunning = 0+ private var lastPlayed = 0L+ private var watchdog: Job? = null+ /** Playback stopped reporting progress after the voice's turn, and the call went back to listening. */+ @Volatile var onStall: (() -> Unit)? = nullinit { hop { connect() } }@@ -70,7 +80,10 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private}private suspend fun connect() {- val session = link.start(VoiceProvider.Gemini, model, voice)+ val session = try { link.start(VoiceProvider.Gemini, model, voice) } catch (error: Exception) {+ // Starting fails over; mid-call, the computer or Google may be back in a moment.+ if (connected) return dropped(error.message.orEmpty()) else throw error+ }if (ended) returnval token = session.credential ?: throw IllegalStateException("Gemini sent no session credential.")this.session = session@@ -83,16 +96,34 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateoverride fun onOpen(webSocket: WebSocket, response: Response) = hop { if (socket === webSocket) send(setup(), webSocket) }override fun onMessage(webSocket: WebSocket, text: String) = hop { if (socket === webSocket) received(text) }override fun onMessage(webSocket: WebSocket, bytes: ByteString) = hop { if (socket === webSocket) received(bytes.utf8()) }- override fun onClosing(webSocket: WebSocket, code: Int, reason: String) { webSocket.close(1000, null); hop { closed(webSocket, reason) } }- override fun onClosed(webSocket: WebSocket, code: Int, reason: String) = hop { closed(webSocket, reason) }- override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) = hop { closed(webSocket, "") }+ override fun onClosing(webSocket: WebSocket, code: Int, reason: String) { webSocket.close(1000, null); hop { closed(webSocket, code, reason) } }+ override fun onClosed(webSocket: WebSocket, code: Int, reason: String) = hop { closed(webSocket, code, reason) }+ override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) = hop { closed(webSocket, null, "") }}- private suspend fun closed(webSocket: WebSocket, reason: String) {+ private suspend fun closed(webSocket: WebSocket, code: Int?, reason: String) {if (socket !== webSocket) returnsocket = null; ready = false- if (resumeHandle != null && reconnects < 3) { reconnects++; connect() }- else fail(reason.ifEmpty { "Lost the connection to Gemini." })+ when {+ !connected -> fail(reason.ifEmpty { "Couldn't reach Gemini." }, VoiceFailureKind.Provider)+ // Gemini refused the session (a key, a model, a request): trying again won't help.+ code == 1007 || code == 1008 -> fail(reason.ifEmpty { "Gemini ended the call." }, VoiceFailureKind.Provider)+ // The turn under way ends with this connection: the resumed one won't complete it.+ replacing -> { replacing = false; turnEnded(); connect() }+ else -> dropped(reason)+ }+ }++ /** The connection to Gemini dropped mid-call: tried again a few times (a resumed session when Gemini gave a handle,+ * else a fresh one the computer primes with the call so far) before the call falls back. */+ private fun dropped(reason: String) {+ val delays = Reconnect.delays+ if (reconnects >= delays.size) return fail(reason.ifEmpty { "Lost the connection to Gemini." })+ val wait = delays[reconnects]+ reconnects+++ stopPlayback(); turnDone = true; dropping = false+ if (phase != VoicePhase.Interrupted) phase(VoicePhase.Reconnecting)+ scope.launch { delay(wait); hop { connect() } }}private fun setup() = buildJsonObject {@@ -116,10 +147,14 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateprivate suspend fun received(text: String) {val message = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: returnif ("setupComplete" in message) {- ready = true; reconnects = 0+ ready = true; connected = true; reconnects = 0if (!audioRunning) {- mic.start({ pcm, level -> hop { captured(pcm, level) } }) {- hop { fail("Couldn't use the microphone. Check its permission and other audio apps.") }+ try {+ mic.start({ pcm, level -> hop { captured(pcm, level) } }) {+ hop { fail("Couldn't use the microphone. Check its permission and other audio apps.", VoiceFailureKind.Microphone) }+ }+ } catch (error: java.io.IOException) {+ return fail(error.message ?: "Couldn't use the microphone.", VoiceFailureKind.Microphone)}audioRunning = true}@@ -137,11 +172,7 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateturnDone = falsefor (part in parts) (part as? JsonObject)?.objectAt("inlineData")?.text("data")?.let { if (!dropping) play(it) }}- if (content.flag("turnComplete")) {- turnDone = true; dropping = false- if (queued == 0 && phase == VoicePhase.Speaking) phase(VoicePhase.Listening)- flush()- }+ if (content.flag("turnComplete")) turnEnded()}(message.objectAt("toolCall")?.get("functionCalls") as? JsonArray)?.let { calls ->val asked = calls.mapNotNull { call ->@@ -167,7 +198,13 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateif (update.flag("resumable")) update.text("newHandle")?.let { resumeHandle = it }}// The server ends this connection soon; carry on in a resumed one.- if ("goAway" in message) socket?.cancel()+ if ("goAway" in message) { replacing = true; socket?.cancel() }+ }++ /** The model's turn is over: listening once what's queued has played. */+ private fun turnEnded() {+ turnDone = true; dropping = false+ if (queued == 0) finishSpeaking() else watch()}private fun send(message: JsonObject, on: WebSocket? = null) {@@ -189,8 +226,8 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private}private fun captured(pcm: ByteArray, level: Float) {- // Connected once the microphone delivers after setup, not when the socket opens.- if (ready && phase == VoicePhase.Starting) phase(VoicePhase.Listening)+ // Ready once the microphone delivers after setup (again, after a reconnect), not when the socket opens.+ if (ready && phase.connecting) { phase(VoicePhase.Listening); flush() }if (muted || !active) returnlevel(if (phase == VoicePhase.Speaking) 0f else level)send(buildJsonObject {@@ -201,6 +238,8 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private}private fun play(base64: String) {+ // Another app has the audio: what the model says meanwhile isn't heard.+ if (phase == VoicePhase.Interrupted) returnval pcm = runCatching { Base64.getDecoder().decode(base64) }.getOrNull()?.takeIf { it.size >= 2 } ?: returnqueued++level(0f); phase(VoicePhase.Speaking)@@ -210,13 +249,39 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateprivate fun played(generation: Int) {if (generation != playGeneration) return+ lastPlayed = System.currentTimeMillis()queued = maxOf(0, queued - 1)if (queued != 0 || !turnDone) return+ finishSpeaking()+ }++ private fun finishSpeaking() {+ watchdog?.cancel(); watchdog = nullif (phase == VoicePhase.Speaking) phase(VoicePhase.Listening)flush()}- private fun stopPlayback() { playGeneration++; queued = 0; player.stop() }+ /** After the voice's turn, playback that stops reporting progress doesn't hold the call on "speaking" and its+ * replies unsaid (#214 on the iPhone): back to listening after [stallMillis] without progress. */+ private fun watch() {+ lastPlayed = System.currentTimeMillis()+ recheck()+ }++ private fun recheck() {+ watchdog?.cancel()+ watchdog = scope.launch { delay(stallMillis); hop { stalled() } }+ }++ private fun stalled() {+ if (queued == 0 || !turnDone) return+ if (System.currentTimeMillis() - lastPlayed < stallMillis) return recheck()+ playGeneration++; queued = 0+ onStall?.invoke()+ finishSpeaking()+ }++ private fun stopPlayback() { watchdog?.cancel(); watchdog = null; playGeneration++; queued = 0; player.stop() }override suspend fun release() {val open = socket; socket = null; ready = false@@ -230,6 +295,8 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, privateprivate fun JsonObject.flag(key: String): Boolean = (this[key] as? JsonPrimitive)?.booleanOrNull == truecompanion object {+ /** Without playback progress this long after the voice's turn, the call stops waiting for the speaker. */+ @Volatile var stallMillis = 4_000Lconst val ENDPOINT = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained"}}diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.ktindex 182bed56..a4370235 100644--- a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt@@ -41,20 +41,43 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, privprivate var session: CallSession? = null/** Tool calls still running on the computer: replies wait, so the model gets each result first. */private var toolsRunning = 0+ /** The session was set up once: a drop after that is tried again instead of ending the call. */+ private var connected = false+ private var reconnects = 0- init {- hop {+ init { hop { open() } }++ /** A session with OpenAI: the computer mints its client secret and writes the voice's instructions (with the call+ * so far, for a session that replaces a dropped one). */+ private suspend fun open() {+ try {val session = link.start(VoiceProvider.OpenAI, model, voice)- if (ended) return@hop+ if (ended) returnval secret = session.credential ?: throw IOException("OpenAI sent no session credential.")this.session = sessionval offer = peer.offer(this)- peer.mic(!muted)+ peer.mic(!muted && phase != VoicePhase.Interrupted)val answer = exchange(secret, offer)- if (ended) return@hop+ if (ended) returnpeer.answer(answer)- drop(15_000, "Couldn't reach OpenAI.") { phase == VoicePhase.Starting }+ } catch (cancelled: CancellationException) { throw cancelled }+ catch (error: Exception) {+ // Starting fails over; mid-call, the computer or OpenAI may be back in a moment.+ if (connected) return dropped(error.message.orEmpty()) else throw error}+ drop(15_000, "Couldn't reach OpenAI.") { phase.connecting }+ }++ /** The connection to OpenAI dropped mid-call: a new session, a few times, before the call falls back. */+ private suspend fun dropped(reason: String) {+ val delays = Reconnect.delays+ if (reconnects >= delays.size) return fail(reason.ifEmpty { "Lost the connection to OpenAI." })+ val wait = delays[reconnects]+ reconnects+++ dropTimer?.cancel(); responding = false+ peer.close()+ if (phase != VoicePhase.Interrupted) phase(VoicePhase.Reconnecting)+ scope.launch { delay(wait); hop { open() } }}override fun speak(markdown: String): Boolean {@@ -100,7 +123,10 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, privval message = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: return@hopwhen (message.text("type")) {// Connected once the session has the instructions and tools, not when the channel opens.- "session.updated" -> if (phase == VoicePhase.Starting) { dropTimer?.cancel(); phase(VoicePhase.Listening); flush() }+ "session.updated" -> if (phase.connecting) {+ dropTimer?.cancel(); connected = true; reconnects = 0+ phase(VoicePhase.Listening); flush()+ }"input_audio_buffer.speech_started" -> level(if (muted) 0f else .7f)"input_audio_buffer.speech_stopped" -> level(0f)"input_audio_buffer.committed" -> link.transcribing(true)@@ -135,19 +161,24 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, priv}override fun ice(state: RealtimePeer.Ice) = hop {- // While starting, only a failure counts: the start timer runs until the session is set up.- if (phase == VoicePhase.Starting && state != RealtimePeer.Ice.Failed) return@hop+ // While connecting, only a failure counts: the start timer runs until the session is set up.+ if (phase.connecting && state != RealtimePeer.Ice.Failed) return@hopwhen (state) {RealtimePeer.Ice.Connected -> dropTimer?.cancel()RealtimePeer.Ice.Disconnected -> drop(5_000, "Lost the connection to OpenAI.") { true }- RealtimePeer.Ice.Failed -> fail("Lost the connection to OpenAI.")+ RealtimePeer.Ice.Failed -> lost("Lost the connection to OpenAI.")}}- /** Fails the call after `millis` unless something cancels the timer first. */+ /** Gives the session up after `millis` unless something cancels the timer first. */private fun drop(millis: Long, message: String, still: () -> Boolean) {dropTimer?.cancel()- dropTimer = scope.launch { delay(millis); hop { if (still()) fail(message) } }+ dropTimer = scope.launch { delay(millis); hop { if (still()) lost(message) } }+ }++ /** The session is gone: mid-call it's tried again; starting, the call fails over. */+ private suspend fun lost(message: String) {+ if (connected) dropped(message) else fail(message)}private fun cancelResponse() {diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.ktindex a78cf0db..73501c83 100644--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt@@ -42,7 +42,17 @@ object SpokenText {}}-enum class VoicePhase { Starting, Listening, Speaking, Interrupted, Failed, Ended }+/**+ * A call's audio, the same on every engine (and on the iPhone and the desktop): connecting until the provider confirmed+ * the session and the microphone delivers, then listening or speaking; reconnecting after a drop; interrupted while+ * another app has the audio; failed when it can't go on; ended.+ */+enum class VoicePhase { Starting, Listening, Speaking, Reconnecting, Interrupted, Failed, Ended;+ /** The connecting line shows: not ready yet, or ready again soon. */+ val connecting: Boolean get() = this == Starting || this == Reconnecting+}+/** Why a call's engine gave up, for analytics (`voice_call_quality`). */+enum class VoiceFailureKind(val wire: String) { Microphone("microphone"), Connection("connection"), Provider("provider") }data class VoiceSettings(val pauseMillis: Long = 1_500, val rate: Float = 1f) {init { require(pauseMillis in listOf(1_000L, 1_500L, 2_500L)); require(rate in .5f..2f) }}@@ -50,7 +60,9 @@ data class VoiceState(val phase: VoicePhase = VoicePhase.Starting, val muted: Boval heard: String = "", val said: String = "", val level: Float = 0f, val engine: String = "",val settings: VoiceSettings = VoiceSettings(), val error: String? = null,/** Where the audio goes when it leaves the phone; null on device. */- val provider: VoiceProvider? = null)+ val provider: VoiceProvider? = null,+ /** Why it failed, with `error`. */+ val failureKind: VoiceFailureKind? = null)sealed interface VoiceInput {data class Text(val generation: Long, val value: String, val final: Boolean) : VoiceInputdata class Level(val generation: Long, val value: Float) : VoiceInputdiff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.ktindex 9d183ba2..72614c57 100644--- a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt@@ -36,6 +36,10 @@ interface VoiceEngine {suspend fun awaitEnd()}+/** How often a dropped provider connection is tried again before the call falls back to on-device voice: after these+ * waits (ms). Tests shorten them. */+object Reconnect { @Volatile var delays = listOf(1_000L, 2_000L, 4_000L) }+/** The microphone as 16 kHz mono PCM16 frames with their loudness (0…1), delivered on an audio thread. */interface MicSource {/** Throws when the microphone can't open; `failed` reports it stopping later. */@@ -76,7 +80,7 @@ abstract class ProviderEngine(parent: CoroutineScope, val provider: VoiceProvideif (ended) continuetry { task() }catch (cancelled: CancellationException) { throw cancelled }- catch (error: Exception) { fail(error.message ?: "Couldn't reach ${provider.title}.") }+ catch (error: Exception) { fail(error.message ?: "Couldn't reach ${provider.title}.", VoiceFailureKind.Provider) }}}}@@ -88,14 +92,14 @@ abstract class ProviderEngine(parent: CoroutineScope, val provider: VoiceProvide/** Stops the audio and the connection; may run twice. */protected abstract suspend fun release()- protected fun fail(message: String) { if (!ended) finish(VoicePhase.Failed, message) }+ protected fun fail(message: String, kind: VoiceFailureKind = VoiceFailureKind.Connection) { if (!ended) finish(VoicePhase.Failed, message, kind) }override fun end() { if (!ended) finish(VoicePhase.Ended, null) }override suspend fun awaitEnd() { teardown?.join() ?: job.join() }override fun settings(value: VoiceSettings) = mutable.update { it.copy(settings = value) }- private fun finish(final: VoicePhase, error: String?) {+ private fun finish(final: VoicePhase, error: String?, kind: VoiceFailureKind? = null) {ended = true- if (error != null) mutable.update { it.copy(phase = VoicePhase.Failed, error = error, level = 0f) }+ if (error != null) mutable.update { it.copy(phase = VoicePhase.Failed, error = error, failureKind = kind, level = 0f) }// Outside `job`, so cancelling the engine's own work can't cut the release short.teardown = scope.launch(NonCancellable) {try { release() }diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.ktindex 24e0b826..fa15e335 100644--- a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt@@ -91,9 +91,9 @@ class VoiceEnginesTest {/** The computer's side of a call, for the engines' tests: one session, tools answered here. */private class Link(val credential: String = "ek_secret") : CallLink {val tools = mutableListOf<Pair<String, JsonObject>>(); val lines = mutableListOf<Pair<String, String>>(); var turns = 0- var started: Triple<VoiceProvider, String, String>? = null+ var started: Triple<VoiceProvider, String, String>? = null; var starts = 0override suspend fun start(provider: VoiceProvider, model: String, voice: String): CallSession {- started = Triple(provider, model, voice)+ started = Triple(provider, model, voice); starts++return CallSession.decode(Json.parseToJsonElement("""{"credential":"$credential","instructions":"You are the voice relay for Ada","tools":[{"name":"send_to_bot","description":"Send","parameters":{"type":"object","properties":{"text":{"type":"string"}},"required":["text"]}},{"name":"bot_status","description":"Status","parameters":{"type":"object","properties":{}}}]}""").jsonObject)@@ -228,8 +228,50 @@ class VoiceEnginesTest {engine.mute(true); runCurrent(); assertEquals(false, peer.micOn)peer.events!!.ice(RealtimePeer.Ice.Disconnected); advanceTimeBy(4_000); runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)peer.events!!.ice(RealtimePeer.Ice.Connected); advanceTimeBy(2_000); runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)- peer.events!!.ice(RealtimePeer.Ice.Failed); runCurrent(); engine.awaitEnd()- assertEquals("Lost the connection to OpenAI.", engine.state.value.error); assertEquals(1, peer.closed)+ peer.events!!.ice(RealtimePeer.Ice.Failed); runCurrent()+ assertEquals(VoicePhase.Reconnecting, engine.state.value.phase); assertEquals(1, peer.closed) // Tried again, not hung up.+ engine.end(); engine.awaitEnd()+ }++ @Test fun openAIRealtimeReconnectsADroppedSessionPrimedByTheComputerAndGivesUpAfterThreeTries() = runTest {+ val peer = Peer(); val link = Link()+ var offers = 0+ val engine = OpenAIRealtimeEngine(backgroundScope, VoiceSettings(), "gpt-realtime", "marin", "gpt-transcribe", link, peer) { _, _ ->+ // OpenAI answers the first two sessions, then stops answering.+ if (++offers > 2) throw IOException("OpenAI answered 503."); "answer-sdp" }+ runCurrent(); peer.isOpen = true; peer.events!!.opened(); peer.events!!.received("""{"type":"session.updated"}"""); runCurrent()+ assertEquals(VoicePhase.Listening, engine.state.value.phase)+ peer.events!!.ice(RealtimePeer.Ice.Failed); runCurrent()+ assertEquals(VoicePhase.Reconnecting, engine.state.value.phase)+ engine.speak("Waits for the new session"); runCurrent()+ advanceTimeBy(Reconnect.delays[0] + 1); runCurrent()+ assertEquals(2, offers); assertEquals(2, link.starts) // A fresh secret and instructions from the computer.+ peer.events!!.opened(); peer.events!!.received("""{"type":"session.updated"}"""); runCurrent()+ assertEquals(VoicePhase.Listening, engine.state.value.phase)+ assertEquals("[Ada replied] Waits for the new session",+ peer.sent.last { it["type"]!!.jsonPrimitive.content == "conversation.item.create" }["item"]!!.jsonObject["content"]!!.jsonArray[0].jsonObject["text"]!!.jsonPrimitive.content)+ // A drop the new sessions never recover from: three tries, then the call fails over.+ peer.events!!.ice(RealtimePeer.Ice.Failed); runCurrent()+ repeat(Reconnect.delays.size) { advanceTimeBy(Reconnect.delays.max() + 15_001); runCurrent() }+ engine.awaitEnd()+ assertEquals(5, offers) // Each failed answer is a try of its own, not the end of the call.+ assertEquals(VoicePhase.Failed, engine.state.value.phase)+ assertEquals(VoiceFailureKind.Connection, engine.state.value.failureKind)+ }++ @Test fun callQualityReportsReadyTimeReconnectsStallsAndFailureOncePerEngine() {+ var now = 0L+ val reports = mutableListOf<Pair<String, CallQuality.Report>>()+ val quality = CallQuality<String>({ now }) { engine, report -> reports += engine to report }+ quality.begin("gemini")+ now = 1_200; quality.phase(VoicePhase.Listening, null)+ quality.phase(VoicePhase.Reconnecting, null); now = 2_000; quality.phase(VoicePhase.Listening, null)+ quality.stalled()+ quality.phase(VoicePhase.Failed, VoiceFailureKind.Connection)+ quality.begin("on_device")+ assertEquals("gemini" to CallQuality.Report(1_200, 1, VoiceStuck.Speaking, VoiceFailureKind.Connection), reports.single())+ now += CallQuality.STUCK_MILLIS; quality.report()+ assertEquals("on_device" to CallQuality.Report(null, 0, VoiceStuck.Connecting, null), reports.last())}@Test fun openAIRealtimeGivesUpWhenTheChannelNeverOpens() = runTest {@@ -319,4 +361,59 @@ class VoiceEnginesTest {scope.cancel(); dispatcher.close()}}++ @Test fun geminiLiveResumesADroppedSessionAndAStuckSpeakerGoesBackToListening() = runBlocking {+ val delays = Reconnect.delays; val stall = GeminiLiveEngine.stallMillis+ Reconnect.delays = listOf(20L, 20L, 20L); GeminiLiveEngine.stallMillis = 150L+ val inbound = LinkedBlockingQueue<Pair<Int, JsonObject>>(); val servers = LinkedBlockingQueue<WebSocket>()+ MockWebServer().use { mock ->+ repeat(2) { n ->+ mock.enqueue(MockResponse().withWebSocketUpgrade(object : WebSocketListener() {+ override fun onOpen(webSocket: WebSocket, response: Response) { servers += webSocket }+ override fun onMessage(webSocket: WebSocket, text: String) { inbound += n to Json.parseToJsonElement(text).jsonObject }+ override fun onClosing(webSocket: WebSocket, code: Int, reason: String) { webSocket.close(1000, null) }+ }))+ }+ val dispatcher = Executors.newSingleThreadExecutor().asCoroutineDispatcher()+ val scope = CoroutineScope(SupervisorJob() + dispatcher)+ val mic = Mic(); val player = Player(); val link = Link("token")+ val engine = GeminiLiveEngine(scope, VoiceSettings(), "gemini-live-test", "Puck", link, mic, player, OkHttpClient(),+ mock.url("/live").toString().replace("http://", "ws://"))+ var stalls = 0+ engine.onStall = { stalls++ }+ fun next(key: String): Pair<Int, JsonObject> {+ while (true) { val m = requireNotNull(inbound.poll(5, TimeUnit.SECONDS)) { "No $key arrived" }; if (key in m.second) return m }+ }+ suspend fun until(check: (VoiceState) -> Boolean) = withTimeout(5_000) { engine.state.first(check) }+ next("setup")+ val first = requireNotNull(servers.poll(5, TimeUnit.SECONDS))+ first.send("""{"setupComplete":{}}""")+ withTimeout(5_000) { while (mic.frames == null) delay(10) }+ mic.frames!!(byteArrayOf(1, 2), .5f)+ until { it.phase == VoicePhase.Listening }+ first.send("""{"sessionResumptionUpdate":{"newHandle":"h1","resumable":true}}""")+ first.close(1011, "Internal error")+ until { it.phase == VoicePhase.Reconnecting }+ engine.speak("Said after the drop")+ val (session, setup) = next("setup")+ assertEquals(1, session); assertEquals(2, link.starts) // A fresh credential from the computer.+ assertEquals("h1", setup["setup"]!!.jsonObject["sessionResumption"]!!.jsonObject["handle"]!!.jsonPrimitive.content)+ val second = requireNotNull(servers.poll(5, TimeUnit.SECONDS))+ second.send("""{"setupComplete":{}}""")+ assertEquals(1, next("clientContent").first) // The reply that waited goes to the new session.+ assertEquals(VoicePhase.Reconnecting, engine.state.value.phase) // Ready again once the microphone delivers.+ mic.frames!!(byteArrayOf(1, 2), .5f)+ until { it.phase == VoicePhase.Listening }+ // The speaker never reports the chunk heard: the watchdog takes the call back to listening.+ val audio = Base64.getEncoder().encodeToString(ByteArray(480))+ second.send("""{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"data":"$audio"}}]}}}""")+ until { it.phase == VoicePhase.Speaking }+ second.send("""{"serverContent":{"turnComplete":true}}""")+ until { it.phase == VoicePhase.Listening }+ assertEquals(1, stalls)+ engine.end(); withTimeout(5_000) { engine.awaitEnd() }+ scope.cancel(); dispatcher.close()+ }+ Reconnect.delays = delays; GeminiLiveEngine.stallMillis = stall+ }}diff --git a/apps/desktop/src/renderer/views/call/CallView.tsx b/apps/desktop/src/renderer/views/call/CallView.tsxindex 0d8646cf..3dc55f43 100644--- a/apps/desktop/src/renderer/views/call/CallView.tsx+++ b/apps/desktop/src/renderer/views/call/CallView.tsx@@ -15,13 +15,31 @@ import { VoiceChatSettingsView } from '../settings/VoiceChatSettingsView'import { voiceStatus } from '../settings/voice-settings'import { stopDictation } from '../thread/dictation'import { CallLink } from './call-link'+import { CallQuality } from './call-quality'import { CloudSpeechEngine } from './CloudSpeechEngine'import { GeminiLiveEngine } from './GeminiLiveEngine'import { OnDeviceEngine } from './OnDeviceEngine'import { OpenAIRealtimeEngine } from './OpenAIRealtimeEngine'-import { countVoiceUsage, fromBase64, onDeviceAvailable, onDeviceUnavailableReason, providerName, transcribeOnHost, VoiceEngine, VoiceSettings, voiceSettingsClosed, type VoiceProvider } from './voice'+import { countVoiceUsage, fromBase64, onDeviceAvailable, onDeviceUnavailableReason, providerName, transcribeOnHost, VoiceEngine, VoiceSettings, voiceSettingsClosed, type VoicePhase, type VoiceProvider } from './voice'import './call.css'+/** What the call bar says for `phase`. */+function statusText(phase: VoicePhase | null, muted: boolean, working: boolean) {+ switch (phase) {+ case null:+ case 'starting':+ return 'Connecting'+ case 'reconnecting':+ return 'Reconnecting'+ case 'speaking':+ return 'Speaking'+ case 'failed':+ return "Can't listen"+ case 'listening':+ return muted ? 'Muted' : working ? 'Working, listening' : 'Listening'+ }+}+/*** Opens the window's own Settings at Voice chat; without it (no Settings screen around) the* call bar shows the voice settings in their own sheet.@@ -63,6 +81,9 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {/** The call on the computer, which runs it. */const link = useRef<CallLink | null>(null)link.current ??= new CallLink(store, botId, crypto.randomUUID())+ /** Each engine's `voice_call_quality`. */+ const quality = useRef<CallQuality | null>(null)+ quality.current ??= new CallQuality((event) => analytics.track(event))const countCloudMinutes = () => {const provider = engine.current?.provider@@ -112,6 +133,8 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {void call.start('device').catch(() => {})engineName.current = 'on_device'}+ quality.current!.begin(engineName.current)+ if (next instanceof GeminiLiveEngine) next.onStall = () => quality.current!.stalled()// Picks up the computer's newest models for the next call; this one starts right away.if (provider) void rememberStatus()// The computer decides what the voice hears (host chat/calls/announce); the engine only says it.@@ -178,6 +201,7 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {onSpeaking(false)interrupt.current = nulllink.current?.end(Math.floor((Date.now() - startedAt) / 1000))+ quality.current?.report()}}, [])useEffect(() => voiceSettingsClosed.subscribe(switchEngine), [])@@ -185,6 +209,7 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {const phase = session?.phase ?? nulluseEffect(() => {if (!session) return+ if (phase) quality.current!.phase(phase, session.failureKind)onSpeaking(phase === 'speaking')interrupt.current = phase === 'speaking' ? () => session.interrupt() : null// Once per call, so an engine taking over doesn't count or open it again.@@ -204,18 +229,7 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {const speaking = phase === 'speaking'const muted = session?.muted === trueconst working = bot ? isWorking(bot) : false- const connecting = phase === null || phase === 'starting'- const statusText = connecting- ? 'Connecting'- : phase === 'speaking'- ? 'Speaking'- : phase === 'failed'- ? "Can't listen"- : muted- ? 'Muted'- : working- ? 'Working, listening'- : 'Listening'+ const connecting = session?.isConnecting ?? trueconst failure = blocked ?? (phase === 'failed' ? session?.failure : null)return (@@ -234,7 +248,7 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {sample={() => ({ level: session?.level ?? 0, speaking: session?.phase === 'speaking', working, connecting: connecting && !blocked })}color={color}width={120}- label={statusText}+ label={statusText(phase, muted, working)}className="call-dots"/>{/* Audio leaves the computer: say where. */}diff --git a/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts b/apps/desktop/src/renderer/views/call/CloudSpeechEngine.tsindex 54a93f96..31157011 100644--- a/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts+++ b/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts@@ -1,4 +1,4 @@-import { errorMessage, micPCM16k, SPEECH_CHUNK, spokenText, VoiceEngine, VoiceSettings, wav, type VoiceProvider } from './voice'+import { errorMessage, micPCM16k, onDevicesChanged, SPEECH_CHUNK, spokenText, VoiceEngine, VoiceSettings, wav, type VoiceProvider } from './voice'export interface CloudSpeech {transcribe: (wav: Uint8Array) => Promise<string>@@ -54,6 +54,7 @@ export function clips(text: string, limit: number): string[] {*/export class CloudSpeechEngine extends VoiceEngine {private stopMic: (() => void) | null = null+ private stopDevices: (() => void) | null = nullprivate chunk: number// Listening: PCM since speech began (plus a little before), and when it last went quiet.@@ -84,19 +85,36 @@ export class CloudSpeechEngine extends VoiceEngine {}async start() {+ await this.openMic()+ // A headset plugged in or out: the microphone starts again on what the system picks now.+ if (!this.ended && this.stopMic) this.stopDevices = onDevicesChanged(() => void this.restartMic())+ }++ private async openMic() {try {const stop = await micPCM16k((pcm, level) => this.heard(pcm, level))if (this.ended) return stop()this.stopMic = stop} catch (e) {- return this.fail(errorMessage(e))+ this.fail(errorMessage(e), 'microphone')}}+ private async restartMic() {+ if (this.ended || !this.stopMic) return+ this.stopMic()+ this.stopMic = null+ this.dropUtterance()+ if (this.phase === 'listening') this.setPhase('reconnecting')+ await this.openMic()+ }+end() {this.ended = truethis.onActivityChanged?.(false)this.stopSpeaking()+ this.stopDevices?.()+ this.stopDevices = nullthis.stopMic?.()this.stopMic = null}@@ -123,7 +141,7 @@ export class CloudSpeechEngine extends VoiceEngine {private heard(pcm: Int16Array, level: number) {// Listening once the microphone actually delivers.- if (!this.ended && this.phase === 'starting') this.setPhase('listening')+ if (!this.ended && this.isConnecting) this.setPhase('listening')if (this.ended || this.muted || this.phase !== 'listening' || Date.now() < this.deafUntil) returnthis.level = levelif (level > THRESHOLD) {@@ -158,7 +176,7 @@ export class CloudSpeechEngine extends VoiceEngine {if (!this.ended && text) this.speech.send(text)} catch (e) {console.error('transcribe:', errorMessage(e))- if (!this.ended) this.fail(errorMessage(e))+ if (!this.ended) this.fail(errorMessage(e), 'provider')}})}diff --git a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.tsindex a15e57a4..301c32cf 100644--- a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts+++ b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts@@ -1,7 +1,9 @@-import { PROGRESS_TAG, type CallLink, type CallSession } from './call-link'-import { errorMessage, fromBase64, micPCM16k, toBase64, VoiceEngine } from './voice'+import type { CallLink, CallSession } from './call-link'+import { browserPorts, type CallOutput, type GeminiPorts, type LiveSocket } from './gemini-ports'+import { errorMessage, fromBase64, onDevicesChanged, PROGRESS_TAG, reconnect, toBase64, VoiceEngine } from './voice'const URL_BASE = 'wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained'+const OPEN = 1/** Gemini's schema spells types in capitals (`OBJECT`, `STRING`). */function geminiSchema(schema: Record<string, unknown>): Record<string, unknown> {@@ -19,22 +21,31 @@ function geminiSchema(schema: Record<string, unknown>): Record<string, unknown>* Gemini Live over a WebSocket, this computer ↔ Google directly: 16 kHz PCM up, 24 kHz PCM down,* echo-cancelled so you can talk over it. The computer runs the call (`CallLink`): it mints one-use ephemeral tokens* on the user's key and writes the voice's instructions and tools, again for each session; a session the server ends- * (`goAway`) is resumed with a fresh one.+ * (`goAway`) is resumed with a fresh one, and a dropped one is tried again.*/export class GeminiLiveEngine extends VoiceEngine {+ /** Without playback progress this long (ms) after the voice's turn, the call stops waiting for the speaker. */+ static stallMs = 4000readonly provider = 'gemini' as const- private socket: WebSocket | null = null+ /** Playback stopped reporting progress after the voice's turn, and the call went back to listening. */+ onStall: (() => void) | null = null+ private socket: LiveSocket | null = null/** Setup is done on the current socket. */private ready = false+ /** A session was set up once: a drop after that is tried again instead of ending the call. */+ private connected = false+ /** The server said it ends this connection (`goAway`): carry on in a resumed one at once. */+ private replacing = falseprivate resumeHandle: string | null = nullprivate reconnects = 0private stopMic: (() => void) | null = null- private output: AudioContext | null = null- private sources = new Set<AudioBufferSourceNode>()- private nextStart = 0- /** Buffers scheduled and not played yet; bumping the generation drops late callbacks. */+ private stopDevices: (() => void) | null = null+ private output: CallOutput | null = null+ /** Pieces handed to the speaker and not played yet; bumping the generation drops late callbacks. */private queued = 0private playGeneration = 0+ private lastPlayed = 0+ private watchdog: ReturnType<typeof setInterval> | null = null/** The model finished its turn (nothing more is coming until we or the user speak). */private turnDone = true/** After a local interrupt, the rest of the turn's audio is dropped. */@@ -45,12 +56,17 @@ export class GeminiLiveEngine extends VoiceEngine {/** Tool calls still running on the computer: replies wait, so the model gets each answer first. */private toolsRunning = 0- constructor(- private model: string,- private voice: string,- private link: CallLink,- ) {+ private model: string+ private voice: string+ private link: CallLink+ private ports: GeminiPorts++ constructor(model: string, voice: string, link: CallLink, ports: GeminiPorts = browserPorts) {super()+ this.model = model+ this.voice = voice+ this.link = link+ this.ports = ports}async start() {@@ -63,9 +79,7 @@ export class GeminiLiveEngine extends VoiceEngine {if (this.ended) returnif (!session.credential) throw new Error('Gemini sent no session credential.')this.session = session- const token = session.credential- const socket = new WebSocket(`${URL_BASE}?access_token=${encodeURIComponent(token)}`)- socket.binaryType = 'arraybuffer'+ const socket = this.ports.socket(`${URL_BASE}?access_token=${encodeURIComponent(session.credential)}`)this.socket = socketthis.ready = falsesocket.onopen = () => this.sendSetup(socket)@@ -73,13 +87,16 @@ export class GeminiLiveEngine extends VoiceEngine {if (this.socket !== socket) returnthis.received(typeof e.data === 'string' ? e.data : new TextDecoder().decode(e.data as ArrayBuffer))}- socket.onclose = (e) => this.closed(socket, e.reason)+ socket.onclose = (e) => this.closed(socket, e.code, e.reason)} catch (e) {- if (!this.ended) this.fail(errorMessage(e))+ if (this.ended) return+ // Starting fails over; mid-call, the computer or Google may be back in a moment.+ if (this.connected) this.dropped(errorMessage(e))+ else this.fail(errorMessage(e), 'provider')}}- private sendSetup(socket: WebSocket) {+ private sendSetup(socket: LiveSocket) {const tools = (this.session?.tools ?? []).map((t) => ({name: t.name,description: t.description,@@ -110,9 +127,11 @@ export class GeminiLiveEngine extends VoiceEngine {this.socket = nullsocket?.close(1000)this.stopPlayback()+ this.stopDevices?.()+ this.stopDevices = nullthis.stopMic?.()this.stopMic = null- void this.output?.close()+ this.output?.close()this.output = null}@@ -150,18 +169,40 @@ export class GeminiLiveEngine extends VoiceEngine {// MARK: socket- private closed(socket: WebSocket, reason: string) {+ private closed(socket: LiveSocket, code: number, reason: string) {if (this.ended || this.socket !== socket) returnthis.socket = nullthis.ready = false- if (this.resumeHandle && this.reconnects < 3) {- this.reconnects+++ if (!this.connected) return this.fail(reason || "Couldn't reach Gemini.", 'provider')+ // Gemini refused the session (a key, a model, a request): trying again won't help.+ if (code === 1007 || code === 1008) return this.fail(reason || 'Gemini ended the call.', 'provider')+ if (this.replacing) {+ this.replacing = false+ // The turn under way ends with this connection: the resumed one won't complete it.+ this.turnEnded()void this.connect()} else {- this.fail(reason || 'Lost the connection to Gemini.')+ this.dropped(reason)}}+ /**+ * The connection to Gemini dropped mid-call: tried again a few times (a resumed session when Gemini gave a handle,+ * else a fresh one the computer primes with the call so far) before the call falls back.+ */+ private dropped(reason: string) {+ if (this.reconnects >= reconnect.delays.length) return this.fail(reason || 'Lost the connection to Gemini.', 'connection')+ const delay = reconnect.delays[this.reconnects]!+ this.reconnects+++ this.stopPlayback()+ this.turnDone = true+ this.dropping = false+ this.setPhase('reconnecting')+ setTimeout(() => {+ if (!this.ended) void this.connect()+ }, delay)+ }+private received(data: string) {if (this.ended) returnlet msg: Record<string, any>@@ -172,8 +213,9 @@ export class GeminiLiveEngine extends VoiceEngine {}if (msg.setupComplete !== undefined) {this.ready = true+ this.connected = truethis.reconnects = 0- if (!this.stopMic) void this.startAudio()+ if (!this.output) void this.startAudio()this.flush()}const content = msg.serverContent@@ -194,12 +236,7 @@ export class GeminiLiveEngine extends VoiceEngine {if (typeof audio === 'string' && !this.dropping) this.play(audio)}}- if (content.turnComplete === true) {- this.turnDone = true- this.dropping = false- if (this.queued === 0 && this.phase === 'speaking') this.setPhase('listening')- this.flush()- }+ if (content.turnComplete === true) this.turnEnded()}const calls = msg.toolCall?.functionCallsif (Array.isArray(calls)) {@@ -215,13 +252,24 @@ export class GeminiLiveEngine extends VoiceEngine {}const update = msg.sessionResumptionUpdateif (update?.resumable === true && typeof update.newHandle === 'string') this.resumeHandle = update.newHandle- // The server ends this connection soon; carry on in a resumed one.- if (msg.goAway !== undefined) this.socket?.close(1000)+ if (msg.goAway !== undefined) {+ // The server ends this connection soon; carry on in a resumed one.+ this.replacing = true+ this.socket?.close(1000)+ }+ }++ /** The model's turn is over: listening once what's queued has played. */+ private turnEnded() {+ this.turnDone = true+ this.dropping = false+ if (this.queued === 0) this.finishSpeaking()+ else this.watch()}- private send(message: unknown, socket?: WebSocket) {+ private send(message: unknown, socket?: LiveSocket) {const target = socket ?? (this.ready ? this.socket : null)- if (target?.readyState === WebSocket.OPEN) target.send(JSON.stringify(message))+ if (target?.readyState === OPEN) target.send(JSON.stringify(message))}/** Replies wait for the model to finish talking; `turnComplete` here would cut it off. */@@ -236,69 +284,91 @@ export class GeminiLiveEngine extends VoiceEngine {// MARK: audioprivate async startAudio() {+ this.output = this.ports.output()+ await this.openMic()+ // A headset plugged in or out: the microphone starts again on what the system picks now.+ if (!this.ended && this.stopMic) this.stopDevices ??= onDevicesChanged(() => void this.restartMic())+ }++ private async openMic() {try {- this.output = new AudioContext({ sampleRate: 24000 })- const stop = await micPCM16k((pcm, level) => this.captured(pcm, level))+ const stop = await this.ports.mic((pcm, level) => this.captured(pcm, level))if (this.ended) return stop()this.stopMic = stop} catch (e) {- if (!this.ended) this.fail(errorMessage(e))+ if (!this.ended) this.fail(errorMessage(e), 'microphone')}}+ private async restartMic() {+ if (this.ended || !this.stopMic) return+ this.stopMic()+ this.stopMic = null+ if (this.isActive) this.setPhase('reconnecting')+ await this.openMic()+ }+private captured(pcm: Int16Array, level: number) {- // Connected once the microphone delivers after setup, not when the socket opens.- if (!this.ended && this.ready && this.phase === 'starting') this.setPhase('listening')+ // Ready once the microphone delivers after setup (again, after a reconnect), not when the socket opens.+ if (!this.ended && this.ready && this.isConnecting) {+ this.setPhase('listening')+ this.flush()+ }if (this.ended || this.muted || !this.isActive) returnthis.level = this.phase === 'speaking' ? 0 : levelthis.send({ realtimeInput: { audio: { data: toBase64(new Uint8Array(pcm.buffer)), mimeType: 'audio/pcm;rate=16000' } } })}private play(base64: string) {- const ctx = this.output- if (!ctx) return+ const output = this.output+ if (!output) returnconst bytes = fromBase64(base64)- const frames = bytes.length >> 1- if (!frames) return- const view = new DataView(bytes.buffer)- const buffer = ctx.createBuffer(1, frames, 24000)- const out = buffer.getChannelData(0)- for (let i = 0; i < frames; i++) out[i] = view.getInt16(i * 2, true) / 32768- const source = ctx.createBufferSource()- source.buffer = buffer- source.connect(ctx.destination)- this.nextStart = Math.max(this.nextStart, ctx.currentTime)- source.start(this.nextStart)- this.nextStart += buffer.duration- this.sources.add(source)+ if (bytes.length < 2) returnthis.queued++this.level = 0if (this.phase !== 'speaking') this.setPhase('speaking')const generation = this.playGeneration- source.onended = () => {- this.sources.delete(source)- this.played(generation)- }+ output.play(bytes, () => this.played(generation))}private played(generation: number) {if (generation !== this.playGeneration || this.ended) return+ this.lastPlayed = Date.now()this.queued = Math.max(0, this.queued - 1)if (this.queued !== 0 || !this.turnDone) return+ this.finishSpeaking()+ }++ private finishSpeaking() {+ this.stopWatch()if (this.phase === 'speaking') this.setPhase('listening')this.flush()}+ /** After the voice's turn, playback that stops reporting progress doesn't hold the call (see `stallMs`). */+ private watch() {+ this.stopWatch()+ this.lastPlayed = Date.now()+ this.watchdog = setInterval(() => {+ if (this.ended || this.queued === 0 || !this.turnDone) return this.stopWatch()+ if (Date.now() - this.lastPlayed < GeminiLiveEngine.stallMs) return+ console.error('gemini playback stalled; back to listening')+ this.playGeneration+++ this.queued = 0+ this.onStall?.()+ this.finishSpeaking()+ }, GeminiLiveEngine.stallMs / 2)+ }++ private stopWatch() {+ if (this.watchdog) clearInterval(this.watchdog)+ this.watchdog = null+ }+private stopPlayback() {+ this.stopWatch()this.playGeneration++this.queued = 0- this.nextStart = 0- for (const s of this.sources) {- s.onended = null- try {- s.stop()- } catch {}- }- this.sources.clear()+ this.output?.stop()}}diff --git a/apps/desktop/src/renderer/views/call/OnDeviceEngine.ts b/apps/desktop/src/renderer/views/call/OnDeviceEngine.tsindex b7e5c34c..4fa15098 100644--- a/apps/desktop/src/renderer/views/call/OnDeviceEngine.ts+++ b/apps/desktop/src/renderer/views/call/OnDeviceEngine.ts@@ -1,5 +1,5 @@import type { SpeechEvent } from '@shared/ipc'-import { onDeviceUnavailableReason, spokenText, VoiceEngine, VoiceSettings } from './voice'+import { onDeviceUnavailableReason, spokenText, VoiceEngine, VoiceSettings, type VoiceFailureKind } from './voice'/*** The reply's language from its script, for picking a voice (replies may be English while the@@ -186,9 +186,10 @@ export class OnDeviceEngine extends VoiceEngine {this.listen()}- protected fail(message: string) {+ protected fail(message: string, kind: VoiceFailureKind = 'microphone') {this.stopListening()this.failure = message+ this.failureKind = kindthis.setPhase('failed')}}diff --git a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.tsindex e7c6e21c..303cd327 100644--- a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts+++ b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts@@ -1,5 +1,5 @@-import { PROGRESS_TAG, type CallLink, type CallSession } from './call-link'-import { errorMessage, microphone, VoiceEngine, VoiceSettings } from './voice'+import type { CallLink, CallSession } from './call-link'+import { errorMessage, microphone, onDevicesChanged, PROGRESS_TAG, reconnect, VoiceEngine, VoiceSettings } from './voice'/*** OpenAI Realtime over WebRTC, this computer ↔ OpenAI directly: the microphone is a WebRTC audio@@ -20,26 +20,78 @@ export class OpenAIRealtimeEngine extends VoiceEngine {private session: CallSession | null = null/** Tool calls still running on the computer: replies wait, so the model gets each result first. */private toolsRunning = 0+ /** The session was set up once: a drop after that is tried again instead of ending the call. */+ private connected = false+ private reconnects = 0+ private stopDevices: (() => void) | null = null- constructor(- private model: string,- private voice: string,- private link: CallLink,- ) {+ private model: string+ private voice: string+ private link: CallLink++ constructor(model: string, voice: string, link: CallLink) {super()+ this.model = model+ this.voice = voice+ this.link = link}async start() {try {this.stream = await microphone()- if (this.ended) return this.release()+ } catch (e) {+ return void (!this.ended && this.fail(errorMessage(e), 'microphone'))+ }+ if (this.ended) return this.release()+ // A headset plugged in or out: the call's microphone track moves to what the system picks now.+ this.stopDevices = onDevicesChanged(() => void this.replaceMic())+ await this.open()+ }++ /**+ * A session with OpenAI: the computer mints its client secret and writes the voice's instructions (with the call so+ * far, for a session that replaces a dropped one).+ */+ private async open() {+ try {const session = await this.link.start('realtime', 'openai', this.model, this.voice)if (this.ended) returnif (!session.credential) throw new Error('OpenAI sent no session credential.')this.session = sessionawait this.connect(session.credential)} catch (e) {- if (!this.ended) this.fail(errorMessage(e))+ if (this.ended) return+ if (this.connected) this.dropped(errorMessage(e))+ else this.fail(errorMessage(e), 'provider')+ }+ }++ /** The connection to OpenAI dropped mid-call: a new session, a few times, before the call falls back. */+ private dropped(reason: string) {+ if (this.ended) return+ if (this.reconnects >= reconnect.delays.length) return this.fail(reason, 'connection')+ const delay = reconnect.delays[this.reconnects]!+ this.reconnects+++ this.close()+ this.setPhase('reconnecting')+ setTimeout(() => {+ if (!this.ended) void this.open()+ }, delay)+ }++ private async replaceMic() {+ if (this.ended || !this.stream) return+ try {+ const next = await microphone()+ if (this.ended) return void next.getTracks().forEach((t) => t.stop())+ const track = next.getAudioTracks()[0]+ if (!track) return+ track.enabled = !this.muted+ await this.pc?.getSenders().find((s) => s.track?.kind === 'audio')?.replaceTrack(track)+ this.release()+ this.stream = next+ } catch (e) {+ console.error('realtime: microphone change', errorMessage(e))}}@@ -56,10 +108,11 @@ export class OpenAIRealtimeEngine extends VoiceEngine {pc.ontrack = (e) => {audio.srcObject = e.streams[0] ?? new MediaStream([e.track])}- pc.oniceconnectionstatechange = () => this.iceChanged(pc.iceConnectionState)+ pc.oniceconnectionstatechange = () => this.pc === pc && this.iceChanged(pc.iceConnectionState)const events = pc.createDataChannel('oai-events')events.onopen = () => this.opened()events.onmessage = (e) => typeof e.data === 'string' && this.received(e.data)+ events.onclose = () => this.events === events && this.connected && this.dropped('Lost the connection to OpenAI.')this.events = eventsconst offer = await pc.createOffer()@@ -81,7 +134,9 @@ export class OpenAIRealtimeEngine extends VoiceEngine {}await pc.setRemoteDescription({ type: 'answer', sdp: body })this.dropTimer = setTimeout(() => {- if (!this.ended && this.phase === 'starting') this.fail("Couldn't reach OpenAI.")+ if (this.ended || !this.isConnecting) return+ if (this.connected) this.dropped("Couldn't reach OpenAI.")+ else this.fail("Couldn't reach OpenAI.", 'connection')}, 15_000)}@@ -93,12 +148,21 @@ export class OpenAIRealtimeEngine extends VoiceEngine {end() {this.ended = truethis.onActivityChanged?.(false)+ this.stopDevices?.()+ this.stopDevices = null+ this.close()+ this.release()+ }++ /** Closes the session, keeping the microphone for the next one. */+ private close() {if (this.dropTimer) clearTimeout(this.dropTimer)- this.events?.close()- this.pc?.close()+ const events = this.eventsthis.events = null+ events?.close()+ this.pc?.close()this.pc = null- this.release()+ this.responding = falseif (this.audio) {this.audio.srcObject = nullthis.audio = null@@ -164,8 +228,10 @@ export class OpenAIRealtimeEngine extends VoiceEngine {switch (msg.type) {case 'session.updated':// Connected once the session has the instructions and tools, not when the channel opens.- if (this.phase !== 'starting') break+ if (!this.isConnecting) breakif (this.dropTimer) clearTimeout(this.dropTimer)+ this.connected = true+ this.reconnects = 0this.setPhase('listening')this.flush()break@@ -182,7 +248,7 @@ export class OpenAIRealtimeEngine extends VoiceEngine {this.link.transcribing(false)breakcase 'output_audio_buffer.started':- this.setPhase('speaking')+ if (this.isActive) this.setPhase('speaking')breakcase 'output_audio_buffer.stopped':case 'output_audio_buffer.cleared':@@ -229,8 +295,8 @@ export class OpenAIRealtimeEngine extends VoiceEngine {}private iceChanged(state: RTCIceConnectionState) {- // While starting, only a failure counts: the start timer runs until the session is set up.- if (this.ended || (this.phase === 'starting' && state !== 'failed')) return+ // While connecting, only a failure counts: the start timer runs until the session is set up.+ if (this.ended || (this.isConnecting && state !== 'failed')) returnswitch (state) {case 'connected':case 'completed':@@ -238,10 +304,11 @@ export class OpenAIRealtimeEngine extends VoiceEngine {breakcase 'disconnected':if (this.dropTimer) clearTimeout(this.dropTimer)- this.dropTimer = setTimeout(() => !this.ended && this.fail('Lost the connection to OpenAI.'), 5000)+ this.dropTimer = setTimeout(() => this.dropped('Lost the connection to OpenAI.'), 5000)breakcase 'failed':- this.fail('Lost the connection to OpenAI.')+ if (this.connected) this.dropped('Lost the connection to OpenAI.')+ else this.fail("Couldn't reach OpenAI.", 'connection')break}}diff --git a/apps/desktop/src/renderer/views/call/call-link.ts b/apps/desktop/src/renderer/views/call/call-link.tsindex 3919ebc1..37e9cb12 100644--- a/apps/desktop/src/renderer/views/call/call-link.ts+++ b/apps/desktop/src/renderer/views/call/call-link.ts@@ -1,10 +1,7 @@import type { CallLine } from '@shared/calls'import { sleep } from '../../store/bot-sync'import type { BotStore } from '../../store/bot-store'-import { spokenText, type VoiceProvider } from './voice'--/** How the voice's instructions name what the bot is doing. */-export const PROGRESS_TAG = '[Progress]'+import { PROGRESS_TAG, spokenText, type VoiceProvider } from './voice'/** How long a tool call waits for the user's words that are still being written down. */const HEARING_PATIENCE_MS = 2_000diff --git a/apps/desktop/src/renderer/views/call/call-quality.ts b/apps/desktop/src/renderer/views/call/call-quality.tsnew file mode 100644index 00000000..bf99ba09--- /dev/null+++ b/apps/desktop/src/renderer/views/call/call-quality.ts@@ -0,0 +1,83 @@+import type { AnalyticsEvent, VoiceCallEngine, VoiceFailure, VoiceStuck } from '@shared/analytics'+import type { VoiceFailureKind, VoicePhase } from './voice'++const isConnecting = (phase: VoicePhase) => phase === 'starting' || phase === 'reconnecting'++/**+ * How each engine of a call fares, for analytics (never content; kit `CallQuality`): one `voice_call_quality` per+ * engine, sent when it's replaced or the call ends — milliseconds until it was ready, reconnects, the first phase+ * stuck too long, why it failed.+ */+export class CallQuality {+ /** Connecting (or reconnecting) this long (ms) counts as stuck. */+ static readonly stuckMs = 20_000+ private engine: VoiceCallEngine | null = null+ private startedAt = 0+ private readyMs: number | undefined+ private reconnects = 0+ private stuck: VoiceStuck = 'none'+ private failure: VoiceFailure = 'none'+ private last: VoicePhase = 'starting'+ /** When the current connecting phase began. */+ private since = 0++ private capture: (event: AnalyticsEvent) => void+ private now: () => number++ constructor(capture: (event: AnalyticsEvent) => void, now: () => number = Date.now) {+ this.capture = capture+ this.now = now+ }++ /** A new engine takes the call; the previous one's report goes out. */+ begin(engine: VoiceCallEngine) {+ this.report()+ this.engine = engine+ this.startedAt = this.now()+ this.since = this.startedAt+ this.readyMs = undefined+ this.reconnects = 0+ this.stuck = 'none'+ this.failure = 'none'+ this.last = 'starting'+ }++ phase(phase: VoicePhase, failure: VoiceFailureKind | null) {+ if (!this.engine || phase === this.last) return+ if (isConnecting(this.last)) this.checkStuck()+ if (phase === 'listening' || phase === 'speaking') {+ this.readyMs ??= this.now() - this.startedAt+ if (this.last === 'reconnecting') this.reconnects+++ } else if (isConnecting(phase)) {+ this.since = this.now()+ } else if (phase === 'failed') {+ this.failure = failure ?? 'connection'+ }+ this.last = phase+ }++ /** Playback stopped reporting progress after the voice's turn (the engine's watchdog took the call back). */+ stalled() {+ if (this.stuck === 'none') this.stuck = 'speaking'+ }++ /** The engine is done (replaced, or the call ended): its report. */+ report() {+ if (!this.engine) return+ if (isConnecting(this.last)) this.checkStuck()+ this.capture({+ event: 'voice_call_quality',+ engine: this.engine,+ ...(this.readyMs === undefined ? {} : { ready_ms: this.readyMs }),+ reconnects: this.reconnects,+ stuck: this.stuck,+ failure: this.failure,+ })+ this.engine = null+ }++ private checkStuck() {+ if (this.stuck !== 'none' || this.now() - this.since < CallQuality.stuckMs) return+ this.stuck = this.last === 'starting' ? 'connecting' : 'reconnecting'+ }+}diff --git a/apps/desktop/src/renderer/views/call/call-recovery.test.ts b/apps/desktop/src/renderer/views/call/call-recovery.test.tsnew file mode 100644index 00000000..040fe9a6--- /dev/null+++ b/apps/desktop/src/renderer/views/call/call-recovery.test.ts@@ -0,0 +1,404 @@+import assert from 'node:assert/strict'+import { existsSync } from 'node:fs'+import { registerHooks } from 'node:module'+import { test } from 'node:test'++// Recorded Gemini Live and OpenAI Realtime event streams replayed through the engines with a scripted socket, microphone+// and speaker (kit CallRecoveryTests): phases, cues, resumption, the #214 shapes and failure.++// The renderer imports without extensions, as the bundler resolves them.+registerHooks({+ resolve(specifier, context, nextResolve) {+ if (specifier.startsWith('.') && context.parentURL && !/\.[a-z]+$/.test(specifier)) {+ const url = new URL(`${specifier}.ts`, context.parentURL)+ if (existsSync(url)) return nextResolve(url.href, context)+ }+ return nextResolve(specifier, context)+ },+})++const disk = new Map<string, string>()+Object.assign(globalThis, {+ localStorage: { getItem: (key: string) => disk.get(key) ?? null, setItem: (key: string, value: string) => disk.set(key, value) },+ window: { addEventListener() {} },+})+const { GeminiLiveEngine } = await import('./GeminiLiveEngine.ts')+const { OpenAIRealtimeEngine } = await import('./OpenAIRealtimeEngine.ts')+const { CallQuality } = await import('./call-quality.ts')+const { reconnect } = await import('./voice.ts')+reconnect.delays = [5, 5, 5]+GeminiLiveEngine.stallMs = 40++class FakeSocket {+ readyState = 0+ onopen: (() => void) | null = null+ onmessage: ((e: { data: unknown }) => void) | null = null+ onclose: ((e: { code: number; reason: string }) => void) | null = null+ sent: Record<string, any>[] = []+ open() {+ this.readyState = 1+ this.onopen?.()+ }+ play(message: unknown) {+ this.onmessage?.({ data: JSON.stringify(message) })+ }+ drop(reason = '', code = 1006) {+ if (this.readyState === 3) return+ this.readyState = 3+ this.onclose?.({ code, reason })+ }+ send(data: string) {+ this.sent.push(JSON.parse(data) as Record<string, any>)+ }+ close(code = 1000) {+ this.drop('', code)+ }+ of(key: string) {+ return this.sent.filter((m) => key in m)+ }+}++const until = async (condition: () => boolean, ms = 2000) => {+ for (let waited = 0; waited < ms; waited += 5) {+ if (condition()) return true+ await new Promise((resolve) => setTimeout(resolve, 5))+ }+ return condition()+}++function call() {+ const sockets: FakeSocket[] = []+ const plays: (() => void)[] = []+ let mic: ((pcm: Int16Array, level: number) => void) | null = null+ let starts = 0+ const link = {+ start: async () => {+ starts+++ return { credential: `token-${starts}`, instructions: 'Be the relay.', tools: [] }+ },+ add() {},+ endTurn() {},+ transcribing() {},+ tool: async () => ({}),+ reply: (text: string) => `[Bot replied] ${text}`,+ notice: (text: string) => `[Notice] ${text}`,+ progress: (text: string) => `[Progress] ${text}`,+ }+ const ports = {+ socket: () => {+ const socket = new FakeSocket()+ sockets.push(socket)+ return socket+ },+ mic: async (out: (pcm: Int16Array, level: number) => void) => {+ mic = out+ return () => {+ mic = null+ }+ },+ output: () => ({+ play: (_pcm: Uint8Array, done: () => void) => void plays.push(done),+ stop: () => void plays.splice(0),+ close() {},+ }),+ }+ const engine = new GeminiLiveEngine('gemini-live', 'Kore', link as never, ports)+ return {+ engine,+ sockets,+ plays,+ get starts() {+ return starts+ },+ get micOn() {+ return mic !== null+ },+ deliver: () => mic?.(new Int16Array(320), 0.5),+ last: () => sockets[sockets.length - 1]!,+ }+}++/** Started, set up, and the microphone delivering. */+async function ready(c: ReturnType<typeof call>) {+ void c.engine.start()+ assert.ok(await until(() => c.sockets.length === 1))+ c.last().open()+ c.last().play({ setupComplete: {} })+ assert.ok(await until(() => c.micOn))+ c.deliver()+ assert.equal(c.engine.phase, 'listening')+}++test('Gemini is ready only once the microphone delivers after setup', async () => {+ const c = call()+ void c.engine.start()+ assert.ok(await until(() => c.sockets.length === 1))+ c.last().open()+ assert.equal(c.last().of('setup').length, 1)+ c.last().play({ setupComplete: {} })+ assert.ok(await until(() => c.micOn))+ assert.equal(c.engine.phase, 'starting', 'setup alone is not ready')+ c.deliver()+ assert.equal(c.engine.phase, 'listening')+ assert.equal(c.last().of('realtimeInput').length, 1)+ c.engine.end()+})++test('a dropped Gemini connection resumes without hanging up', async () => {+ const c = call()+ await ready(c)+ c.last().play({ sessionResumptionUpdate: { newHandle: 'h1', resumable: true } })+ c.last().drop('network')+ assert.equal(c.engine.phase, 'reconnecting')+ assert.ok(c.engine.holdsCall, 'the call is not over')+ c.engine.speak('The tests pass.')+ assert.ok(await until(() => c.sockets.length === 2))+ assert.equal(c.starts, 2, 'a fresh credential from the computer')+ c.last().open()+ assert.equal(c.last().of('setup')[0]!.setup.sessionResumption.handle, 'h1')+ c.last().play({ setupComplete: {} })+ assert.equal(c.last().of('clientContent').length, 1, 'the reply that waited is said')+ assert.equal(c.engine.phase, 'reconnecting', 'ready again once the microphone delivers')+ c.deliver()+ assert.equal(c.engine.phase, 'listening')+ c.engine.end()+})++test('goAway carries on in a resumed session without reconnecting', async () => {+ const c = call()+ await ready(c)+ c.last().play({ sessionResumptionUpdate: { newHandle: 'h2', resumable: true } })+ c.last().play({ goAway: { timeLeft: '5s' } })+ assert.ok(await until(() => c.sockets.length === 2))+ assert.equal(c.engine.phase, 'listening')+ c.engine.end()+})++test('goAway mid-turn does not leave the call speaking', async () => {+ const c = call()+ await ready(c)+ c.last().play({ serverContent: { modelTurn: { parts: [{ inlineData: { data: 'AAAAAA==' } }] } } })+ assert.equal(c.engine.phase, 'speaking')+ c.last().play({ goAway: { timeLeft: '5s' } })+ assert.ok(await until(() => c.sockets.length === 2))+ c.plays.splice(0).forEach((done) => done())+ assert.equal(c.engine.phase, 'listening', 'the resumed session will not complete that turn')+ c.engine.speak('Deployed.')+ c.last().open()+ c.last().play({ setupComplete: {} })+ assert.equal(c.last().of('clientContent').length, 1, 'replies are not held behind it')+ c.engine.end()+})++test('a session Gemini refuses mid-call fails at once', async () => {+ const c = call()+ await ready(c)+ c.last().drop('API key not valid.', 1008)+ assert.equal(c.engine.phase, 'failed')+ assert.equal(c.engine.failureKind, 'provider', 'trying again will not help')+ assert.equal(c.engine.failure, 'API key not valid.')+ assert.equal(c.sockets.length, 1)+})++test('reconnects run out and the call fails', async () => {+ const c = call()+ await ready(c)+ for (let attempt = 1; attempt <= reconnect.delays.length; attempt++) {+ c.last().drop()+ assert.ok(await until(() => c.sockets.length === attempt + 1))+ }+ c.last().drop()+ assert.equal(c.engine.phase, 'failed')+ assert.equal(c.engine.failureKind, 'connection')+})++test('playback that never finishes goes back to listening (#214)', async () => {+ const c = call()+ let stalls = 0+ c.engine.onStall = () => stalls+++ await ready(c)+ c.last().play({ serverContent: { modelTurn: { parts: [{ inlineData: { data: 'AAAAAA==' } }] } } })+ assert.equal(c.engine.phase, 'speaking')+ c.last().play({ serverContent: { turnComplete: true } })+ c.engine.speak('Deployed.')+ assert.ok(await until(() => c.engine.phase === 'listening'))+ assert.equal(stalls, 1)+ assert.equal(c.last().of('clientContent').length, 1, 'the reply is not stuck behind it')+ c.engine.end()+})++test('playback that finishes returns to listening without a stall', async () => {+ const c = call()+ let stalls = 0+ c.engine.onStall = () => stalls+++ await ready(c)+ c.last().play({ serverContent: { modelTurn: { parts: [{ inlineData: { data: 'AAAAAA==' } }] } } })+ c.last().play({ serverContent: { turnComplete: true } })+ c.plays.splice(0).forEach((done) => done())+ assert.equal(c.engine.phase, 'listening')+ await new Promise((resolve) => setTimeout(resolve, 100))+ assert.equal(stalls, 0)+ c.engine.end()+})++test('a changed microphone reconnects until it delivers again', async () => {+ const c = call()+ await ready(c)+ await (c.engine as unknown as { restartMic(): Promise<void> }).restartMic()+ assert.equal(c.engine.phase, 'reconnecting')+ c.deliver()+ assert.equal(c.engine.phase, 'listening')+ c.engine.end()+})++test('OpenAI phases follow its events', () => {+ const engine = new OpenAIRealtimeEngine('gpt-realtime', 'marin', {} as never)+ const event = (type: string) => (engine as unknown as { received(data: string): void }).received(JSON.stringify({ type }))+ event('output_audio_buffer.started')+ assert.equal(engine.phase, 'starting', 'nothing plays before the session is set up')+ event('session.updated')+ assert.equal(engine.phase, 'listening')+ event('output_audio_buffer.started')+ assert.equal(engine.phase, 'speaking')+ event('output_audio_buffer.stopped')+ assert.equal(engine.phase, 'listening')+ engine.end()+})++/** OpenAI's side of a call: a WebRTC peer whose events channel and ICE state the test drives. */+class FakePeer {+ iceConnectionState = 'new'+ oniceconnectionstatechange: (() => void) | null = null+ ontrack: unknown = null+ answered = false+ channel = {+ readyState: 'connecting',+ onopen: null as (() => void) | null,+ onmessage: null as ((e: { data: unknown }) => void) | null,+ onclose: null as (() => void) | null,+ send() {},+ close() {+ this.readyState = 'closed'+ },+ }+ addTrack() {}+ createDataChannel() {+ return this.channel+ }+ async createOffer() {+ return { type: 'offer', sdp: 'offer-sdp' }+ }+ async setLocalDescription() {}+ async setRemoteDescription() {+ this.answered = true+ }+ getSenders() {+ return []+ }+ close() {}+ /** The session is set up. */+ ready() {+ this.channel.readyState = 'open'+ this.channel.onopen?.()+ this.channel.onmessage?.({ data: JSON.stringify({ type: 'session.updated' }) })+ }+ ice(state: string) {+ this.iceConnectionState = state+ this.oniceconnectionstatechange?.()+ }+}++function openAICall() {+ const peers: FakePeer[] = []+ const answers = { status: 200 }+ let starts = 0+ const track = { kind: 'audio', enabled: true, stop() {} }+ Object.defineProperty(globalThis, 'navigator', {+ configurable: true,+ value: {+ mediaDevices: { getUserMedia: async () => ({ getAudioTracks: () => [track], getTracks: () => [track] }), addEventListener() {}, removeEventListener() {} },+ },+ })+ Object.assign(globalThis, {+ RTCPeerConnection: class extends FakePeer {+ constructor() {+ super()+ peers.push(this)+ }+ },+ Audio: class {},+ fetch: async () => ({ ok: answers.status === 200, status: answers.status, text: async () => 'answer-sdp' }),+ })+ const link = {+ start: async () => {+ starts+++ return { credential: `ek-${starts}`, instructions: 'Be the relay.', tools: [] }+ },+ }+ const engine = new OpenAIRealtimeEngine('gpt-realtime', 'marin', link as never)+ return {+ engine,+ answers,+ get starts() {+ return starts+ },+ last: () => peers[peers.length - 1]!,+ peers,+ }+}++test('OpenAI reconnects a dropped session and gives up after three tries', async () => {+ const o = openAICall()+ void o.engine.start()+ assert.ok(await until(() => o.peers.length === 1 && o.last().answered))+ o.last().ready()+ assert.equal(o.engine.phase, 'listening')+ o.last().ice('failed')+ assert.equal(o.engine.phase, 'reconnecting', 'tried again, not hung up')+ assert.ok(await until(() => o.peers.length === 2 && o.last().answered))+ assert.equal(o.starts, 2, 'a fresh secret and instructions from the computer')+ o.last().ready()+ assert.equal(o.engine.phase, 'listening')+ // OpenAI stops answering: three more tries (the count started over once the session was back), then the call fails.+ o.answers.status = 503+ o.last().ice('failed')+ assert.ok(await until(() => o.engine.phase === 'failed'))+ assert.equal(o.engine.failureKind, 'connection')+ assert.equal(o.starts, 2 + reconnect.delays.length)+})++test('OpenAI that never connected fails over at once', async () => {+ const o = openAICall()+ o.answers.status = 401+ await o.engine.start()+ assert.equal(o.engine.phase, 'failed')+ assert.equal(o.engine.failureKind, 'provider')+ assert.equal(o.starts, 1)+})++test('call quality reports ready time, reconnects, stalls and failure once per engine', () => {+ const events: unknown[] = []+ const quality = new CallQuality((e) => events.push(e))+ quality.begin('gemini')+ quality.phase('listening', null)+ quality.phase('reconnecting', null)+ quality.phase('listening', null)+ quality.stalled()+ quality.phase('failed', 'connection')+ quality.begin('on_device')+ const first = events[0] as Record<string, unknown>+ assert.equal(first.event, 'voice_call_quality')+ assert.equal(first.engine, 'gemini')+ assert.equal(typeof first.ready_ms, 'number')+ assert.deepEqual([first.reconnects, first.stuck, first.failure], [1, 'speaking', 'connection'])+ quality.report()+ assert.equal(events.length, 2)++ let now = 0+ const stuck: unknown[] = []+ const slow = new CallQuality((e) => stuck.push(e), () => now)+ slow.begin('openai')+ now += CallQuality.stuckMs+ slow.report()+ assert.deepEqual(stuck, [{ event: 'voice_call_quality', engine: 'openai', reconnects: 0, stuck: 'connecting', failure: 'none' }])+})diff --git a/apps/desktop/src/renderer/views/call/gemini-ports.ts b/apps/desktop/src/renderer/views/call/gemini-ports.tsnew file mode 100644index 00000000..3f4759e6--- /dev/null+++ b/apps/desktop/src/renderer/views/call/gemini-ports.ts@@ -0,0 +1,81 @@+import { micPCM16k } from './voice'++// What Gemini Live needs from the computer (GeminiLiveEngine): its WebSocket, the microphone and the speaker.++/** The WebSocket the engine uses (a browser `WebSocket`; tests replay a recorded one). */+export interface LiveSocket {+ readonly readyState: number+ onopen: (() => void) | null+ onmessage: ((e: { data: unknown }) => void) | null+ onclose: ((e: { code: number; reason: string }) => void) | null+ send(data: string): void+ close(code?: number): void+}++/** The voice's audio out: 24 kHz PCM16 pieces, `done` once each has played. */+export interface CallOutput {+ play(pcm: Uint8Array, done: () => void): void+ /** Drops what's queued. */+ stop(): void+ close(): void+}++/** What the engine needs from the computer; tests script it. */+export interface GeminiPorts {+ socket(url: string): LiveSocket+ mic(out: (pcm: Int16Array, level: number) => void): Promise<() => void>+ output(): CallOutput+}++/** Web Audio playback, each piece scheduled after the last. */+class WebAudioOutput implements CallOutput {+ private ctx = new AudioContext({ sampleRate: 24000 })+ private sources = new Set<AudioBufferSourceNode>()+ private nextStart = 0++ play(bytes: Uint8Array, done: () => void) {+ const frames = bytes.length >> 1+ if (!frames) return+ const view = new DataView(bytes.buffer, bytes.byteOffset, bytes.byteLength)+ const buffer = this.ctx.createBuffer(1, frames, 24000)+ const out = buffer.getChannelData(0)+ for (let i = 0; i < frames; i++) out[i] = view.getInt16(i * 2, true) / 32768+ const source = this.ctx.createBufferSource()+ source.buffer = buffer+ source.connect(this.ctx.destination)+ this.nextStart = Math.max(this.nextStart, this.ctx.currentTime)+ source.start(this.nextStart)+ this.nextStart += buffer.duration+ this.sources.add(source)+ source.onended = () => {+ this.sources.delete(source)+ done()+ }+ }++ stop() {+ this.nextStart = 0+ for (const s of this.sources) {+ s.onended = null+ try {+ s.stop()+ } catch {}+ }+ this.sources.clear()+ }++ close() {+ this.stop()+ void this.ctx.close()+ }+}++export const browserPorts: GeminiPorts = {+ socket: (url) => {+ const socket = new WebSocket(url)+ socket.binaryType = 'arraybuffer'+ return socket as unknown as LiveSocket+ },+ mic: micPCM16k,+ output: () => new WebAudioOutput(),+}diff --git a/apps/desktop/src/renderer/views/call/voice.ts b/apps/desktop/src/renderer/views/call/voice.tsindex 01cdf2a5..f998d0a3 100644--- a/apps/desktop/src/renderer/views/call/voice.ts+++ b/apps/desktop/src/renderer/views/call/voice.ts@@ -1,4 +1,5 @@import type { HostClient } from '../../client/host-client'+import type { VoiceFailure } from '@shared/analytics'import type { CallCue } from '@shared/calls'import { Observable } from '../../lib/observable'import {@@ -76,7 +77,24 @@ export const countVoiceUsage = (seconds: number, p: VoiceProvider) => addVoiceUs// MARK: engines-export type VoicePhase = 'starting' | 'listening' | 'speaking' | 'failed'+/**+ * A call's audio, the same on every engine (and on the iPhone and Android): connecting until the provider confirmed+ * the session and the microphone delivers, then listening or speaking; reconnecting after a drop or a device change;+ * failed when it can't go on.+ */+export type VoicePhase = 'starting' | 'listening' | 'speaking' | 'reconnecting' | 'failed'++/** Why a call's engine gave up, for analytics (`voice_call_quality`). */+export type VoiceFailureKind = Exclude<VoiceFailure, 'none'>++/** How the voice's instructions name what the bot is doing. */+export const PROGRESS_TAG = '[Progress]'++/**+ * How often a dropped provider connection is tried again before the call falls back to on-device voice: after these+ * waits (ms). Tests shorten them.+ */+export const reconnect = { delays: [1000, 2000, 4000] }/*** What carries a call's audio: on-device speech or a realtime model on the user's own key.@@ -85,6 +103,7 @@ export type VoicePhase = 'starting' | 'listening' | 'speaking' | 'failed'export abstract class VoiceEngine extends Observable {phase: VoicePhase = 'starting'failure: string | null = null+ failureKind: VoiceFailureKind | null = null/** Microphone loudness, 0…1, while listening. Read every frame; not observed. */level = 0abstract readonly provider: VoiceProvider | null@@ -97,6 +116,16 @@ export abstract class VoiceEngine extends Observable {return this.phase === 'listening' || this.phase === 'speaking'}+ /** The connecting line shows: not ready yet, or ready again soon. */+ get isConnecting() {+ return this.phase === 'starting' || this.phase === 'reconnecting'+ }++ /** The call still holds its transport and hears the computer (a reconnect doesn't end it). */+ get holdsCall() {+ return this.isActive || this.phase === 'reconnecting'+ }+get muted() {return this.mutedValue}@@ -111,12 +140,13 @@ export abstract class VoiceEngine extends Observable {protected setPhase(phase: VoicePhase) {this.phase = phasethis.changed()- this.onActivityChanged?.(!this.ended && this.isActive)+ this.onActivityChanged?.(!this.ended && this.holdsCall)}- protected fail(message: string) {+ protected fail(message: string, kind: VoiceFailureKind = 'connection') {this.end()this.failure = message+ this.failureKind = kindthis.setPhase('failed')}@@ -144,6 +174,25 @@ export abstract class VoiceEngine extends Observable {export const errorMessage = (e: unknown) => (e instanceof Error ? e.message : String(e))export const MIC_DENIED = 'Allow the microphone for Codync in Settings to talk to your bots.'+/**+ * Runs `changed` when the computer's audio devices change (a headset plugged in or out), a moment after the system+ * settles; returns the function that stops listening.+ */+export function onDevicesChanged(changed: () => void): () => void {+ const devices = globalThis.navigator?.mediaDevices+ if (!devices) return () => {}+ let timer: ReturnType<typeof setTimeout> | null = null+ const listener = () => {+ if (timer) clearTimeout(timer)+ timer = setTimeout(changed, 500)+ }+ devices.addEventListener('devicechange', listener)+ return () => {+ if (timer) clearTimeout(timer)+ devices.removeEventListener('devicechange', listener)+ }+}+// MARK: audio helpers/** The microphone, echo-cancelled so the bot can be talked over. */diff --git a/apps/desktop/src/shared/analytics.ts b/apps/desktop/src/shared/analytics.tsindex 93dca598..5b70e148 100644--- a/apps/desktop/src/shared/analytics.ts+++ b/apps/desktop/src/shared/analytics.ts@@ -30,6 +30,10 @@ export type SignInFailure = 'cancelled' | 'timeout' | 'verification' | 'error'/** What carried a voice call: on-device speech, a realtime model, or cloud transcription and speech. */export type VoiceCallEngine = 'on_device' | 'openai' | 'gemini' | 'cloud'+/** The first phase of a call's engine that lasted too long. */+export type VoiceStuck = 'none' | 'connecting' | 'reconnecting' | 'speaking'+/** Why a call's engine gave up. */+export type VoiceFailure = 'none' | 'microphone' | 'connection' | 'provider'export type AnalyticsEvent =| { event: 'app_opened'; at_login: boolean }@@ -44,6 +48,7 @@ export type AnalyticsEvent =| { event: 'link_opened'; link: Link }| { event: 'screen_viewed'; screen: Screen }| { event: 'voice_call_started'; engine: VoiceCallEngine }+ | { event: 'voice_call_quality'; engine: VoiceCallEngine; ready_ms?: number; reconnects: number; stuck: VoiceStuck; failure: VoiceFailure }/** `track`'s body: the event with its properties, from the desktop app. */export const trackBody = (event: AnalyticsEvent) => ({ ...event, app: 'desktop' as const })diff --git a/apps/ios/Kit/Sources/CodyncKit/Analytics/AnalyticsEvent.swift b/apps/ios/Kit/Sources/CodyncKit/Analytics/AnalyticsEvent.swiftindex 4a079a98..bfbaeb56 100644--- a/apps/ios/Kit/Sources/CodyncKit/Analytics/AnalyticsEvent.swift+++ b/apps/ios/Kit/Sources/CodyncKit/Analytics/AnalyticsEvent.swift@@ -21,6 +21,9 @@ public enum AnalyticsEvent: Equatable, Sendable {case settingChanged(Setting, on: Bool)case linkOpened(Link)case voiceCallStarted(VoiceEngine)+ /// How one engine of a call fared, when it's replaced or the call ends: milliseconds until it was ready (nil:+ /// never), reconnects, the first phase stuck too long, why it failed.+ case voiceCallQuality(VoiceEngine, readyMs: Int?, reconnects: Int, stuck: VoiceStuck, failure: VoiceFailure)case screenViewed(Screen)public enum Notifications: String, Sendable { case allowed, denied, notAsked = "not_asked" }@@ -36,6 +39,8 @@ public enum AnalyticsEvent: Equatable, Sendable {case github, appStore = "app_store", demo, tailscale, privacy, installCommand = "install_command"}public enum VoiceEngine: String, Sendable { case onDevice = "on_device", openai, gemini, cloud }+ public enum VoiceStuck: String, Sendable { case none, connecting, reconnecting, speaking }+ public enum VoiceFailure: String, Sendable { case none, microphone, connection, provider }public enum Screen: String, Sendable {case bots, state, chat, trace, botDetails = "bot_details", memory, routines, marketplacecase settings, computers, usage, account, pairing, voiceSettings = "voice_settings", remoteScreen = "remote_screen"@@ -103,6 +108,7 @@ public enum AnalyticsEvent: Equatable, Sendable {case .settingChanged: "setting_changed"case .linkOpened: "link_opened"case .voiceCallStarted: "voice_call_started"+ case .voiceCallQuality: "voice_call_quality"case .screenViewed: "screen_viewed"}}@@ -128,6 +134,11 @@ public enum AnalyticsEvent: Equatable, Sendable {case let .settingChanged(setting, on): return ["setting": .string(setting.rawValue), "on": .bool(on)]case let .linkOpened(link): return ["link": .string(link.rawValue)]case let .voiceCallStarted(engine): return ["engine": .string(engine.rawValue)]+ case let .voiceCallQuality(engine, readyMs, reconnects, stuck, failure):+ var p: [String: Analytics.Value] = ["engine": .string(engine.rawValue), "reconnects": .int(reconnects),+ "stuck": .string(stuck.rawValue), "failure": .string(failure.rawValue)]+ p["ready_ms"] = readyMs.map(Analytics.Value.int)+ return pcase let .screenViewed(screen): return ["screen": .string(screen.rawValue)]}}diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CallAudio.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CallAudio.swiftnew file mode 100644index 00000000..f40820ea--- /dev/null+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CallAudio.swift@@ -0,0 +1,156 @@+import AVFoundation+import Foundation+import os++private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")++/// A realtime call's audio on the phone: the voice-processing microphone (16 kHz PCM16 up) and the speaker (24 kHz+/// PCM16 down). Tests use a scripted one.+@MainActor+protocol CallAudio: AnyObject {+ func permission() async -> Bool+ /// Starts the microphone, handing each buffer (base64 PCM16) and its loudness to `mic`, and the speaker.+ func start(mic: @escaping @MainActor (String, Float) -> Void) throws+ /// Plays a piece of the voice; `done` runs once it has been rendered.+ func play(_ pcm: Data, done: @escaping @MainActor () -> Void)+ /// Drops what's queued to play.+ func stopPlayback()+ /// Starts the microphone and speaker again after an interruption.+ func resume() throws+ func stop()+ /// Another app took the audio, or gave it back.+ var interruption: (@MainActor (AudioInterruption) -> Void)? { get set }+}++/// Another app's audio (a phone call, Siri, an alarm) pausing the call, and giving it back.+enum AudioInterruption {+ case began+ /// With whether the system says to pick up again on our own.+ case ended(shouldResume: Bool)+}++/// The audio session's events a call follows: interruptions, and `engine` reconfiguring (a route change such as+/// AirPods, voice processing settling), which stops it.+@MainActor+final class AudioSessionEvents {+ private var observers: [any NSObjectProtocol] = []++ init(engine: AVAudioEngine?, interrupted: @escaping @MainActor (AudioInterruption) -> Void,+ reconfigured: @escaping @MainActor () -> Void = {}) {+ let center = NotificationCenter.default+ observers.append(center.addObserver(forName: AVAudioSession.interruptionNotification, object: nil, queue: .main) { note in+ let type = (note.userInfo?[AVAudioSessionInterruptionTypeKey] as? UInt).flatMap(AVAudioSession.InterruptionType.init)+ let options = AVAudioSession.InterruptionOptions(rawValue: note.userInfo?[AVAudioSessionInterruptionOptionKey] as? UInt ?? 0)+ MainActor.assumeIsolated {+ switch type {+ case .began: interrupted(.began)+ case .ended: interrupted(.ended(shouldResume: options.contains(.shouldResume)))+ default: break+ }+ }+ })+ if let engine {+ observers.append(center.addObserver(forName: .AVAudioEngineConfigurationChange, object: engine, queue: .main) { _ in+ MainActor.assumeIsolated { reconfigured() }+ })+ }+ }++ func stop() {+ for observer in observers { NotificationCenter.default.removeObserver(observer) }+ observers = []+ }+}++/// The phone's audio for Gemini Live: an echo-cancelled (voice-processing) microphone tap and a player node.+final class VoiceProcessingAudio: CallAudio {+ var interruption: (@MainActor (AudioInterruption) -> Void)?+ private let engine = AVAudioEngine()+ private let player = AVAudioPlayerNode()+ private var started = false+ private var events: AudioSessionEvents?+ private var mic: (@MainActor (String, Float) -> Void)?++ private static let playFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 24000, channels: 1, interleaved: false)++ func permission() async -> Bool { await AVAudioApplication.requestRecordPermission() }++ func start(mic: @escaping @MainActor (String, Float) -> Void) throws {+ self.mic = mic+ let session = AVAudioSession.sharedInstance()+ try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP])+ try session.setActive(true)+ try engine.inputNode.setVoiceProcessingEnabled(true)+ engine.attach(player)+ // The engine stops itself when it reconfigures; without a restart the microphone goes quiet.+ events = AudioSessionEvents(engine: engine, interrupted: { [weak self] in self?.interruption?($0) },+ reconfigured: { [weak self] in self?.restart() })+ try run()+ started = true+ }++ /// Wires the speaker and taps the microphone for the current hardware, then starts the engine.+ private func run() throws {+ // Wired again on every start: after the engine reconfigures, the old connections stay in place but+ // play nothing, so the player never finishes its buffers and replies wait forever (#214).+ engine.connect(player, to: engine.mainMixerNode, format: Self.playFormat)+ engine.connect(engine.mainMixerNode, to: engine.outputNode, format: nil)+ let input = engine.inputNode+ input.removeTap(onBus: 0)+ let format = try MicTap.format(of: input)+ input.installTap(onBus: 0, bufferSize: 2048, format: format, block: MicTap.pcm16k(from: format) { [weak self] pcm, level in+ let audio = pcm.base64EncodedString()+ Task { @MainActor in self?.mic?(audio, level) }+ })+ engine.prepare()+ try engine.start()+ player.play()+ }++ private func restart() {+ guard started, !engine.isRunning else { return }+ log.info("call audio engine reconfigured; restarting")+ do { try run() } catch { log.error("call audio restart: \(error.localizedDescription, privacy: .public)") }+ }++ func resume() throws {+ try AVAudioSession.sharedInstance().setActive(true)+ if !engine.isRunning { try run() }+ }++ func play(_ pcm: Data, done: @escaping @MainActor () -> Void) {+ guard let format = Self.playFormat, pcm.count >= 2 else { return }+ let frames = pcm.count / 2+ guard let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(frames)),+ let out = buffer.floatChannelData?[0] else { return }+ buffer.frameLength = AVAudioFrameCount(frames)+ pcm.withUnsafeBytes { raw in+ for i in 0..<frames {+ let sample = Int16(littleEndian: raw.loadUnaligned(fromByteOffset: i * 2, as: Int16.self))+ out[i] = Float(sample) / 32768+ }+ }+ player.scheduleBuffer(buffer, completionCallbackType: .dataRendered) { _ in+ Task { @MainActor in done() }+ }+ }++ func stopPlayback() {+ guard started else { return }+ player.stop()+ // A stopped engine (mid-restart, or failing) would throw; restarting plays the player again.+ if engine.isRunning { player.play() }+ }++ func stop() {+ events?.stop()+ events = nil+ if started {+ player.stop()+ engine.inputNode.removeTap(onBus: 0)+ engine.stop()+ started = false+ }+ try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)+ }+}diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CallQuality.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CallQuality.swiftnew file mode 100644index 00000000..3109849c--- /dev/null+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CallQuality.swift@@ -0,0 +1,75 @@+import CodyncKit+import Foundation++/// How each engine of a call fares, for analytics (never content): one `voice_call_quality` per engine, sent when it's+/// replaced or the call ends — milliseconds until it was ready, reconnects, the first phase stuck too long, why it+/// failed.+@MainActor+final class CallQuality {+ /// Connecting (or reconnecting) this long counts as stuck.+ static let stuckAfter: Duration = .seconds(20)+ private let now: () -> ContinuousClock.Instant+ private let capture: @MainActor (AnalyticsEvent) -> Void+ private var engine: AnalyticsEvent.VoiceEngine?+ private var startedAt: ContinuousClock.Instant+ private var readyMs: Int?+ private var reconnects = 0+ private var stuck = AnalyticsEvent.VoiceStuck.none+ private var failure = AnalyticsEvent.VoiceFailure.none+ private var last: VoicePhase = .starting+ /// When the current connecting phase began.+ private var since: ContinuousClock.Instant++ init(capture: @escaping @MainActor (AnalyticsEvent) -> Void = { Analytics.shared.capture($0) },+ now: @escaping () -> ContinuousClock.Instant = { .now }) {+ self.capture = capture+ self.now = now+ startedAt = now()+ since = startedAt+ }++ /// A new engine takes the call; the previous one's report goes out.+ func begin(_ engine: AnalyticsEvent.VoiceEngine) {+ report()+ self.engine = engine+ startedAt = now()+ since = startedAt+ readyMs = nil+ reconnects = 0+ stuck = .none+ failure = .none+ last = .starting+ }++ func phase(_ phase: VoicePhase) {+ guard engine != nil, phase != last else { return }+ if last.isConnecting { checkStuck() }+ if phase.isActive {+ if readyMs == nil { readyMs = Int((now() - startedAt) / .milliseconds(1)) }+ if last == .reconnecting { reconnects += 1 }+ } else if phase.isConnecting {+ since = now()+ } else if case .failed(_, let kind) = phase {+ failure = AnalyticsEvent.VoiceFailure(rawValue: kind.rawValue) ?? .connection+ }+ last = phase+ }++ /// Playback stopped reporting progress after the voice's turn (the engine's watchdog took the call back).+ func stalled() {+ if stuck == .none { stuck = .speaking }+ }++ /// The engine is done (replaced, or the call ended): its report.+ func report() {+ guard let engine else { return }+ if last.isConnecting { checkStuck() }+ capture(.voiceCallQuality(engine, readyMs: readyMs, reconnects: reconnects, stuck: stuck, failure: failure))+ self.engine = nil+ }++ private func checkStuck() {+ guard stuck == .none, now() - since >= Self.stuckAfter else { return }+ stuck = last == .starting ? .connecting : .reconnecting+ }+}diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swiftindex dc6d5c25..73c61a04 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift@@ -26,6 +26,8 @@ struct CallView: View {@State private var settings = false/// `voice_call_started` went out for this call.@State private var reported = false+ /// Each engine's `voice_call_quality`.+ @State private var quality = CallQuality()private var bot: Bot? { model.bots[botId] }private var call: String { callID.uuidString }@@ -34,8 +36,13 @@ struct CallView: View {var body: some View {VStack(spacing: 6) {bar- if case .failed(let message) = session?.phase {+ if case .failed(let message, _) = session?.phase {line(message, color: Palette.danger)+ } else if session?.phase == .interrupted {+ Button { session?.resume() } label: {+ line("Audio paused by another app. Tap to resume.", color: Palette.secondary)+ }+ .buttonStyle(.plain)} else if let notice {line(notice, color: Palette.secondary)}@@ -55,8 +62,10 @@ struct CallView: View {interrupt = nilmodel.endVoiceCall(callID)link?.end(seconds: Int(Date.now.timeIntervalSince(startedAt)))+ quality.report()}.onChange(of: session?.phase) { _, phase in+ if let phase { quality.phase(phase) }isSpeaking = phase == .speakinginterrupt = phase == .speaking ? { session?.interrupt() } : nil// Counted once audio flows, so it names the engine the call ended up on.@@ -66,7 +75,7 @@ struct CallView: View {model.callStarted(botId, call: call)}// A cloud engine that can't start or drops mid-call hands over to on-device voice.- if case .failed(let message) = phase, let provider = session?.provider {+ if case .failed(let message, _) = phase, let provider = session?.provider {countCloudMinutes()notice = "\(provider.name) isn't available (\(message)). Using on-device voice."begin(cloud: nil)@@ -113,6 +122,8 @@ struct CallView: View {engine = OnDeviceEngine { [model, botId, call] in model.send($0, to: botId, call: call) }Task { _ = try? await link.start("device") }}+ quality.begin(engine.analyticsName)+ (engine as? GeminiLiveEngine)?.onStall = { [quality] in quality.stalled() }engine.onActivityChanged = { [weak engine, model, callID] active inif active {model.beginVoiceCall(callID, hear: { [weak engine] in engine?.hear($0) }, end: { [weak engine] in engine?.end() })@@ -157,7 +168,7 @@ struct CallView: View {HStack(spacing: 10) {avatarLevelDots(level: session?.level ?? 0, speaking: session?.phase == .speaking,- working: bot?.isWorking == true, connecting: (session?.phase ?? .starting) == .starting,+ working: bot?.isWorking == true, connecting: session?.phase.isConnecting ?? true,color: color, reduceMotion: reduceMotion).frame(minWidth: Self.dotsWidth, maxWidth: Self.dotsMaxWidth, minHeight: Self.dotsHeight, maxHeight: Self.dotsHeight).accessibilityLabel(statusText)@@ -229,6 +240,8 @@ struct CallView: View {private var statusText: String {switch session?.phase {case .starting, nil: "Connecting"+ case .reconnecting: "Reconnecting"+ case .interrupted: "Paused"case .speaking: "Speaking"case .failed: "Can't listen"case .listening where session?.muted == true: "Muted"diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swiftindex 73797f83..95de42c6 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift@@ -13,7 +13,7 @@ private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")@Observablefinal class CloudSpeechEngine: VoiceEngine {private(set) var phase: VoicePhase = .starting {- didSet { onActivityChanged?(!ended && phase.isActive) }+ didSet { onActivityChanged?(!ended && phase.holdsCall) }}private(set) var level: Float = 0var muted = false {@@ -36,6 +36,7 @@ final class CloudSpeechEngine: VoiceEngine {private let engine = AVAudioEngine()private var audioRunning = falseprivate var ended = false+ private var events: AudioSessionEvents?// Listening: PCM since speech began (plus a little before), and when it last went quiet.private var preroll = Data()@@ -77,29 +78,69 @@ final class CloudSpeechEngine: VoiceEngine {func start() async {guard await AVAudioApplication.requestRecordPermission() else {- return fail("Allow the microphone for Codync in Settings to talk to your bots.")+ return fail("Allow the microphone for Codync in Settings to talk to your bots.", .microphone)}guard !ended else { return }do {- let session = AVAudioSession.sharedInstance()- try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP])- try session.setActive(true)- let input = engine.inputNode- let format = try MicTap.format(of: input)- input.installTap(onBus: 0, bufferSize: 2048, format: format, block: MicTap.pcm16k(from: format) { [weak self] pcm, level in- Task { @MainActor in self?.heard(pcm, level: level) }- })- engine.prepare()- try engine.start()+ let session = AVAudioSession.sharedInstance()+ try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP])+ try session.setActive(true)+ try runMic()audioRunning = true} catch {- return fail("Couldn't use the microphone: \(error.localizedDescription)")+ return fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone)+ }+ // A route change (AirPods) stops the engine: the microphone picks up again on the new route.+ events = AudioSessionEvents(engine: engine, interrupted: { [weak self] in self?.interrupted($0) },+ reconfigured: { [weak self] in+ guard let self, !self.ended, self.audioRunning, self.phase != .interrupted, !self.engine.isRunning else { return }+ do { try self.runMic() } catch { self.fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone) }+ })+ }++ /// Taps the microphone in its current format and starts the engine.+ private func runMic() throws {+ let input = engine.inputNode+ input.removeTap(onBus: 0)+ let format = try MicTap.format(of: input)+ input.installTap(onBus: 0, bufferSize: 2048, format: format, block: MicTap.pcm16k(from: format) { [weak self] pcm, level in+ Task { @MainActor in self?.heard(pcm, level: level) }+ })+ engine.prepare()+ try engine.start()+ }++ /// Back from an interruption (it ended, or the user tapped the call's line): listening once the microphone+ /// delivers, then the replies that waited.+ func resume() {+ guard !ended, phase == .interrupted else { return }+ do {+ try AVAudioSession.sharedInstance().setActive(true)+ try runMic()+ } catch {+ return fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone)+ }+ phase = .reconnecting+ }++ private func interrupted(_ change: AudioInterruption) {+ guard !ended else { return }+ switch change {+ case .began:+ guard phase.isActive else { return }+ stopSpeaking()+ dropUtterance()+ phase = .interrupted+ case .ended(let shouldResume):+ if shouldResume { resume() }}}func end() {ended = trueonActivityChanged?(false)+ events?.stop()+ events = nilstopSpeaking()transcribing?.cancel()if audioRunning {@@ -124,7 +165,10 @@ final class CloudSpeechEngine: VoiceEngine {private func heard(_ pcm: Data, level: Float) {// Listening once the microphone actually delivers.- if !ended, phase == .starting { phase = .listening }+ if !ended, phase.isConnecting {+ phase = .listening+ if !queue.isEmpty, player == nil, next == nil { playNext() }+ }guard !ended, !muted, phase == .listening, Date.now >= deafUntil else { return }self.level = levelif level > Self.threshold {@@ -159,7 +203,7 @@ final class CloudSpeechEngine: VoiceEngine {speech.send(text)} catch {log.error("transcribe: \(error.localizedDescription, privacy: .public)")- self?.fail(error.localizedDescription)+ self?.fail(error.localizedDescription, .provider)}}}@@ -179,7 +223,8 @@ final class CloudSpeechEngine: VoiceEngine {private func say(_ text: String) {guard !ended, !phase.isFailed, phase != .starting, !text.isEmpty else { return }queue += Self.clips(text, limit: chunk)- if player == nil, next == nil { playNext() }+ // Interrupted or reconnecting, it waits for the audio to be back.+ if phase.isActive, player == nil, next == nil { playNext() }}private func playNext() {@@ -255,9 +300,9 @@ final class CloudSpeechEngine: VoiceEngine {player = nil}- private func fail(_ message: String) {+ private func fail(_ message: String, _ kind: VoiceFailureKind) {end()- phase = .failed(message)+ phase = .failed(message, kind)}/// Sentences grouped into clips of at most `limit` characters (longer sentences are cut).diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swiftindex edc86c49..72931b09 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift@@ -1,4 +1,3 @@-import AVFoundationimport CodyncKitimport Foundationimport Observation@@ -9,12 +8,12 @@ private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")/// Gemini Live over a WebSocket, phone ↔ Google directly: 16 kHz PCM up, 24 kHz PCM down,/// echo-cancelled by the voice-processing input so you can talk over it. The computer runs the call (`CallLink`): it/// mints one-use ephemeral tokens on the user's key and writes the voice's instructions and tools, again for each-/// session; a session the server ends (`goAway`) is resumed with a fresh one.+/// session; a session the server ends (`goAway`) is resumed with a fresh one, and a dropped one is tried again.@MainActor@Observablefinal class GeminiLiveEngine: VoiceEngine {private(set) var phase: VoicePhase = .starting {- didSet { onActivityChanged?(!ended && phase.isActive) }+ didSet { onActivityChanged?(!ended && phase.holdsCall) }}private(set) var level: Float = 0var muted = false {@@ -26,26 +25,33 @@ final class GeminiLiveEngine: VoiceEngine {}var provider: VoiceProvider? { .gemini }@ObservationIgnored var onActivityChanged: ((Bool) -> Void)?+ /// Playback stopped reporting progress after the voice's turn, and the call went back to listening.+ @ObservationIgnored var onStall: (() -> Void)?private let model: Stringprivate let voice: Stringprivate let link: CallLink+ private let audio: any CallAudio+ private let openSocket: @MainActor (URL) -> any LiveSocket/// The current provider session's instructions and tools, from the computer.private var session: CallSession?/// Tool calls still running on the computer: replies wait, so the model gets each answer first.private var toolsRunning = 0- private var socket: URLSessionWebSocketTask?+ private var socket: (any LiveSocket)?/// Setup is done on the current socket.private var ready = false+ /// A session was set up once: a drop after that is tried again instead of ending the call.+ private var connected = false+ /// The server said it ends this connection (`goAway`): carry on in a resumed one at once.+ private var replacing = falseprivate var resumeHandle: String?private var reconnects = 0- private let engine = AVAudioEngine()- private let player = AVAudioPlayerNode()private var audioStarted = false- private var configObserver: (any NSObjectProtocol)?- /// Buffers scheduled and not played yet; bumping the generation drops late callbacks.+ /// Pieces handed to the speaker and not rendered yet; bumping the generation drops late callbacks.private var queued = 0private var playGeneration = 0+ private var lastPlayed = ContinuousClock.now+ private var watchdog: Task<Void, Never>?/// The model finished its turn (nothing more is coming until we or the user speak).private var turnDone = true/// After a local interrupt, the rest of the turn's audio is dropped.@@ -53,18 +59,24 @@ final class GeminiLiveEngine: VoiceEngine {private var pending: [String] = []private var ended = false- private static let playFormat = AVAudioFormat(commonFormat: .pcmFormatFloat32, sampleRate: 24000, channels: 1, interleaved: false)+ /// Without playback progress this long after the voice's turn, the call stops waiting for the speaker (#214:+ /// buffers that never finish would keep it on "speaking" and the bot's replies unsaid).+ static var stallTimeout: Duration = .seconds(4)- init(model: String, voice: String, link: CallLink) {+ init(model: String, voice: String, link: CallLink, audio: any CallAudio = VoiceProcessingAudio(),+ socket: @escaping @MainActor (URL) -> any LiveSocket = { URLSessionLiveSocket($0) }) {self.model = modelself.voice = voiceself.link = link+ self.audio = audio+ openSocket = socket}func start() async {- guard await AVAudioApplication.requestRecordPermission() else {- return fail("Allow the microphone for Codync in Settings to talk to your bots.")+ guard await audio.permission() else {+ return fail("Allow the microphone for Codync in Settings to talk to your bots.", .microphone)}+ audio.interruption = { [weak self] in self?.interrupted($0) }await connect()}@@ -76,21 +88,20 @@ final class GeminiLiveEngine: VoiceEngine {self.session = sessionvar url = URLComponents(string: "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained")url?.queryItems = [URLQueryItem(name: "access_token", value: token)]- guard let url = url?.url else { throw VoiceError( "Couldn't reach Gemini.") }- let socket = URLSession.shared.webSocketTask(with: url)- socket.maximumMessageSize = 16 << 20+ guard let url = url?.url else { throw VoiceError("Couldn't reach Gemini.") }+ let socket = openSocket(url)self.socket = socketready = false- socket.resume()receive(socket)sendSetup(on: socket)} catch {guard !ended else { return }- fail(error.localizedDescription)+ // Starting fails over; mid-call, the computer or Google may be back in a moment.+ if connected { dropped(error.localizedDescription) } else { fail(error.localizedDescription, .provider) }}}- private func sendSetup(on socket: URLSessionWebSocketTask) {+ private func sendSetup(on socket: any LiveSocket) {let declared = session.flatMap { (try? JSONSerialization.jsonObject(with: $0.tools)) as? [[String: Any]] } ?? []let tools: [[String: Any]] = declared.map { tool invar declaration: [String: Any] = ["name": tool["name"] ?? "", "description": tool["description"] ?? ""]@@ -121,17 +132,14 @@ final class GeminiLiveEngine: VoiceEngine {func end() {ended = trueonActivityChanged?(false)- socket?.cancel(with: .normalClosure, reason: nil)+ socket?.close()socket = nilstopPlayback()- if let configObserver { NotificationCenter.default.removeObserver(configObserver) }- configObserver = nil+ audio.interruption = nilif audioStarted {- engine.inputNode.removeTap(onBus: 0)- engine.stop()+ audio.stop()audioStarted = false}- try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)}func interrupt() {@@ -161,56 +169,92 @@ final class GeminiLiveEngine: VoiceEngine {flush()}+ /// The audio is back (the user tapped the call's line, or the interruption ended): listening once the+ /// microphone delivers again.+ func resume() {+ guard !ended, phase == .interrupted else { return }+ do {+ try audio.resume()+ } catch {+ return fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone)+ }+ phase = .reconnecting+ }++ private func interrupted(_ change: AudioInterruption) {+ guard !ended else { return }+ switch change {+ case .began:+ guard phase.holdsCall || phase == .starting else { return }+ stopPlayback()+ dropping = !turnDone+ level = 0+ phase = .interrupted+ send(["realtimeInput": ["audioStreamEnd": true]])+ case .ended(let shouldResume):+ if shouldResume { resume() }+ }+ }+// MARK: socket- private func receive(_ socket: URLSessionWebSocketTask) {+ private func receive(_ socket: any LiveSocket) {Task { [weak self] inwhile true {- let message: URLSessionWebSocketTask.Message+ let text: Stringdo {- message = try await socket.receive()+ text = try await socket.receive()} catch {self?.closed(socket)return}guard let self, self.socket === socket else { return }- switch message {- case .data(let data): self.received(data)- case .string(let text): self.received(Data(text.utf8))- @unknown default: break- }+ self.received(Data(text.utf8))}}}- private func closed(_ socket: URLSessionWebSocketTask) {+ private func closed(_ socket: any LiveSocket) {guard !ended, self.socket === socket else { return }self.socket = nilready = false- let reason = socket.closeReason.map { String(decoding: $0, as: UTF8.self) } ?? ""- log.info("gemini closed: \(reason, privacy: .public)")- if resumeHandle != nil, reconnects < 3 {- reconnects += 1+ let (code, reason) = (socket.closeCode, socket.closeReason)+ log.info("gemini closed: \(code) \(reason, privacy: .public)")+ guard connected else { return fail(reason.isEmpty ? "Couldn't reach Gemini." : reason, .provider) }+ // Gemini refused the session (a key, a model, a request): trying again won't help.+ if code == 1007 || code == 1008 { return fail(reason.isEmpty ? "Gemini ended the call." : reason, .provider) }+ if replacing {+ replacing = false+ // The turn under way ends with this connection: the resumed one won't complete it.+ turnEnded()Task { await connect() }} else {- fail(reason.isEmpty ? "Lost the connection to Gemini." : reason)+ dropped(reason)+ }+ }++ /// The connection to Gemini dropped mid-call: tried again a few times (a resumed session when Gemini gave a handle,+ /// else a fresh one the computer primes with the call so far) before the call falls back.+ private func dropped(_ reason: String) {+ guard reconnects < Reconnect.delays.count else {+ return fail(reason.isEmpty ? "Lost the connection to Gemini." : reason, .connection)+ }+ let delay = Reconnect.delays[reconnects]+ reconnects += 1+ stopPlayback()+ turnDone = true+ dropping = false+ if phase != .interrupted { phase = .reconnecting }+ Task { [weak self] in+ try? await Task.sleep(for: .seconds(delay))+ guard let self, !self.ended else { return }+ await self.connect()}}private func received(_ data: Data) {guard !ended, let msg = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return }- if msg["setupComplete"] != nil {- ready = true- reconnects = 0- if !audioStarted {- do {- try startAudio()- } catch {- return fail("Couldn't use the microphone: \(error.localizedDescription)")- }- }- flush()- }+ if msg["setupComplete"] != nil { setupCompleted() }if let content = msg["serverContent"] as? [String: Any] {if let heard = (content["inputTranscription"] as? [String: Any])?["text"] as? String { link.add("user", heard) }if let said = (content["outputTranscription"] as? [String: Any])?["text"] as? String { link.add("voice", said) }@@ -226,12 +270,7 @@ final class GeminiLiveEngine: VoiceEngine {if let audio = (part["inlineData"] as? [String: Any])?["data"] as? String, !dropping { play(audio) }}}- if content["turnComplete"] as? Bool == true {- turnDone = true- dropping = false- if queued == 0, phase == .speaking { phase = .listening }- flush()- }+ if content["turnComplete"] as? Bool == true { turnEnded() }}if let calls = (msg["toolCall"] as? [String: Any])?["functionCalls"] as? [[String: Any]] {let asked: [(id: String, name: String, args: Data)] = calls.compactMap { call in@@ -259,21 +298,43 @@ final class GeminiLiveEngine: VoiceEngine {}if msg["goAway"] != nil {// The server ends this connection soon; carry on in a resumed one.- socket?.cancel(with: .normalClosure, reason: nil)+ replacing = true+ socket?.close()}}- private func send(_ message: [String: Any], on socket: URLSessionWebSocketTask? = nil) {+ private func setupCompleted() {+ ready = true+ connected = true+ reconnects = 0+ if !audioStarted {+ do {+ try audio.start { [weak self] audio, level in self?.captured(audio, level: level) }+ audioStarted = true+ } catch {+ return fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone)+ }+ }+ flush()+ }++ /// The model's turn is over: listening once what's queued has played.+ private func turnEnded() {+ turnDone = true+ dropping = false+ if queued == 0 { finishSpeaking() } else { watch() }+ }++ private func send(_ message: [String: Any], on socket: (any LiveSocket)? = nil) {guard let socket = socket ?? (ready ? self.socket : nil),let data = try? JSONSerialization.data(withJSONObject: message) else { return }- socket.send(.string(String(decoding: data, as: UTF8.self))) { error in- if let error { log.error("gemini send: \(error.localizedDescription, privacy: .public)") }- }+ socket.send(String(decoding: data, as: UTF8.self))}- /// Replies wait for the model to finish talking; `turnComplete` here would cut it off.+ /// Replies wait for the model to finish talking (`turnComplete` here would cut it off), and for the audio to be+ /// back after an interruption.private func flush() {- guard ready, turnDone, queued == 0, toolsRunning == 0, !pending.isEmpty else { return }+ guard ready, turnDone, queued == 0, toolsRunning == 0, phase != .interrupted, !pending.isEmpty else { return }let text = pending.joined(separator: "\n\n")pending.removeAll()turnDone = false@@ -282,100 +343,72 @@ final class GeminiLiveEngine: VoiceEngine {// MARK: audio- private func startAudio() throws {- let session = AVAudioSession.sharedInstance()- try session.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP])- try session.setActive(true)- try engine.inputNode.setVoiceProcessingEnabled(true)- engine.attach(player)- // The engine stops itself when it reconfigures (voice processing settling right after the start,- // a route change such as AirPods); without a restart the microphone goes quiet.- configObserver = NotificationCenter.default.addObserver(forName: .AVAudioEngineConfigurationChange, object: engine,- queue: .main) { [weak self] _ in- MainActor.assumeIsolated { self?.restartAudio() }- }- try runAudio()- audioStarted = true- }-- /// Wires the speaker and taps the microphone for the current hardware, then starts the engine.- private func runAudio() throws {- // Wired again on every start: after the engine reconfigures, the old connections stay in place but- // play nothing, so the player never finishes its buffers and replies wait forever (#214).- engine.connect(player, to: engine.mainMixerNode, format: Self.playFormat)- engine.connect(engine.mainMixerNode, to: engine.outputNode, format: nil)- let input = engine.inputNode- input.removeTap(onBus: 0)- let format = try MicTap.format(of: input)- input.installTap(onBus: 0, bufferSize: 2048, format: format, block: MicTap.pcm16k(from: format) { [weak self] pcm, level in- let audio = pcm.base64EncodedString()- Task { @MainActor in self?.captured(audio, level: level) }- })- engine.prepare()- try engine.start()- player.play()- }-- private func restartAudio() {- guard audioStarted, !ended, !engine.isRunning else { return }- log.info("gemini audio engine reconfigured; restarting")- do {- try runAudio()- } catch {- fail("Couldn't use the microphone: \(error.localizedDescription)")- }- }-private func captured(_ audio: String, level: Float) {- // Connected once the microphone delivers after setup, not when the socket opens.- if !ended, ready, phase == .starting { phase = .listening }+ // Ready once the microphone delivers after setup (again, after a reconnect), not when the socket opens.+ if !ended, ready, phase.isConnecting {+ phase = .listening+ flush()+ }guard !ended, !muted, phase.isActive else { return }self.level = phase == .speaking ? 0 : levelsend(["realtimeInput": ["audio": ["data": audio, "mimeType": "audio/pcm;rate=16000"]]])}private func play(_ base64: String) {- guard let format = Self.playFormat, let data = Data(base64Encoded: base64), data.count >= 2 else { return }- let frames = data.count / 2- guard let buffer = AVAudioPCMBuffer(pcmFormat: format, frameCapacity: AVAudioFrameCount(frames)),- let out = buffer.floatChannelData?[0] else { return }- buffer.frameLength = AVAudioFrameCount(frames)- data.withUnsafeBytes { raw in- for i in 0..<frames {- let sample = Int16(littleEndian: raw.loadUnaligned(fromByteOffset: i * 2, as: Int16.self))- out[i] = Float(sample) / 32768- }- }+ // Another app has the audio: what the model says meanwhile isn't heard.+ guard phase != .interrupted, let data = Data(base64Encoded: base64), data.count >= 2 else { return }queued += 1level = 0phase = .speakinglet generation = playGeneration- player.scheduleBuffer(buffer, completionCallbackType: .dataRendered) { [weak self] _ in- Task { @MainActor in self?.played(generation) }- }+ audio.play(data) { [weak self] in self?.played(generation) }}private func played(_ generation: Int) {guard generation == playGeneration, !ended else { return }+ lastPlayed = .nowqueued = max(0, queued - 1)guard queued == 0, turnDone else { return }+ finishSpeaking()+ }++ private func finishSpeaking() {+ watchdog?.cancel()+ watchdog = nilif phase == .speaking { phase = .listening }flush()}+ /// After the voice's turn, playback that stops reporting progress doesn't hold the call (see [`stallTimeout`]).+ private func watch() {+ watchdog?.cancel()+ lastPlayed = .now+ watchdog = Task { [weak self] in+ while !Task.isCancelled {+ try? await Task.sleep(for: Self.stallTimeout)+ guard let self, !Task.isCancelled, !self.ended, self.queued > 0, self.turnDone else { return }+ guard ContinuousClock.now - self.lastPlayed >= Self.stallTimeout else { continue }+ log.error("gemini playback stalled; back to listening")+ self.playGeneration += 1+ self.queued = 0+ self.onStall?()+ self.finishSpeaking()+ return+ }+ }+ }+private func stopPlayback() {+ watchdog?.cancel()+ watchdog = nilplayGeneration += 1queued = 0- if audioStarted {- player.stop()- // A stopped engine (mid-restart, or failing) would throw; restarting plays the player again.- if engine.isRunning { player.play() }- }+ if audioStarted { audio.stopPlayback() }}- private func fail(_ message: String) {+ private func fail(_ message: String, _ kind: VoiceFailureKind) {end()- phase = .failed(message)+ phase = .failed(message, kind)}// MARK: helpersdiff --git a/apps/ios/Kit/Sources/CodyncUI/Call/LiveSocket.swift b/apps/ios/Kit/Sources/CodyncUI/Call/LiveSocket.swiftnew file mode 100644index 00000000..aed5d8f3--- /dev/null+++ b/apps/ios/Kit/Sources/CodyncUI/Call/LiveSocket.swift@@ -0,0 +1,47 @@+import Foundation+import os++private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")++/// A provider's WebSocket, text frames each way (Gemini Live); tests replay a recorded one.+@MainActor+protocol LiveSocket: AnyObject {+ func send(_ text: String)+ /// The next frame; throws once the socket closed.+ func receive() async throws -> String+ func close()+ /// The code the server closed it with (0 when it didn't say).+ var closeCode: Int { get }+ /// Why the server closed it, if it said.+ var closeReason: String { get }+}++final class URLSessionLiveSocket: LiveSocket {+ private let task: URLSessionWebSocketTask++ init(_ url: URL) {+ task = URLSession.shared.webSocketTask(with: url)+ task.maximumMessageSize = 16 << 20+ task.resume()+ }++ func send(_ text: String) {+ task.send(.string(text)) { error in+ if let error { log.error("socket send: \(error.localizedDescription, privacy: .public)") }+ }+ }++ func receive() async throws -> String {+ switch try await task.receive() {+ case .data(let data): String(decoding: data, as: UTF8.self)+ case .string(let text): text+ @unknown default: ""+ }+ }++ func close() { task.cancel(with: .normalClosure, reason: nil) }++ var closeCode: Int { task.closeCode.rawValue }++ var closeReason: String { task.closeReason.map { String(decoding: $0, as: UTF8.self) } ?? "" }+}diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/OnDeviceEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/OnDeviceEngine.swiftindex 0cc1637e..305f94b1 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/OnDeviceEngine.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/OnDeviceEngine.swift@@ -13,7 +13,7 @@ private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")@Observablefinal class OnDeviceEngine: VoiceEngine {private(set) var phase: VoicePhase = .starting {- didSet { onActivityChanged?(!ended && phase.isActive) }+ didSet { onActivityChanged?(!ended && phase.holdsCall) }}var provider: VoiceProvider? { nil }/// Keep the transport attached while audio is active, even without a visible UI update.@@ -44,6 +44,9 @@ final class OnDeviceEngine: VoiceEngine {private let synthesizer = AVSpeechSynthesizer()private var speechDone: SpeechDone?private var ended = false+ private var events: AudioSessionEvents?+ /// Replies and notices that came while another app had the audio, said once it's back.+ private var unsaid: [String] = []/// Call settings (the gear), kept across calls.static let pauseKey = "callPause"@@ -72,13 +75,13 @@ final class OnDeviceEngine: VoiceEngine {func start() async {guard await AVAudioApplication.requestRecordPermission() else {- return fail("Allow the microphone for Codync in Settings to talk to your bots.")+ return fail("Allow the microphone for Codync in Settings to talk to your bots.", .microphone)}guard await Self.speechAuthorized() else {- return fail("Allow Speech Recognition for Codync in Settings to talk to your bots.")+ return fail("Allow Speech Recognition for Codync in Settings to talk to your bots.", .microphone)}guard let recognizer, recognizer.isAvailable else {- return fail("Speech recognition isn't available for this language right now.")+ return fail("Speech recognition isn't available for this language right now.", .provider)}guard !ended else { return }do {@@ -86,14 +89,47 @@ final class OnDeviceEngine: VoiceEngine {try audio.setCategory(.playAndRecord, mode: .voiceChat, options: [.defaultToSpeaker, .allowBluetoothHFP])try audio.setActive(true)} catch {- return fail("Couldn't start audio: \(error.localizedDescription)")+ return fail("Couldn't start audio: \(error.localizedDescription)", .microphone)}+ // A route change (AirPods) stops the engine mid-utterance: listen again on the new microphone.+ events = AudioSessionEvents(engine: engine, interrupted: { [weak self] in self?.interrupted($0) },+ reconfigured: { [weak self] in+ guard let self, !self.ended, self.phase == .listening, !self.engine.isRunning else { return }+ self.listen()+ })listen()}+ /// Back from an interruption (it ended, or the user tapped the call's line): what came meanwhile is said, then+ /// listening again.+ func resume() {+ guard !ended, phase == .interrupted else { return }+ try? AVAudioSession.sharedInstance().setActive(true)+ listen()+ guard !unsaid.isEmpty else { return }+ let waiting = unsaid.joined(separator: "\n\n")+ unsaid.removeAll()+ speak(reply: waiting)+ }++ private func interrupted(_ change: AudioInterruption) {+ guard !ended else { return }+ switch change {+ case .began:+ guard phase.isActive else { return }+ stopListening()+ synthesizer.stopSpeaking(at: .immediate)+ phase = .interrupted+ case .ended(let shouldResume):+ if shouldResume { resume() }+ }+ }+func end() {ended = trueonActivityChanged?(false)+ events?.stop()+ events = nilstopListening()synthesizer.stopSpeaking(at: .immediate)try? AVAudioSession.sharedInstance().setActive(false, options: .notifyOthersOnDeactivation)@@ -102,7 +138,9 @@ final class OnDeviceEngine: VoiceEngine {/// Reads a reply aloud (the bot's final message), pausing the microphone meanwhile.func speak(reply markdown: String) {let text = SpokenText.from(markdown)- guard !ended, !text.isEmpty, phase != .starting, !(phase.isFailed) else { return }+ guard !ended, !text.isEmpty else { return }+ if phase == .interrupted { return unsaid.append(markdown) }+ guard phase.isActive else { return }stopListening()said = textphase = .speaking@@ -145,7 +183,7 @@ final class OnDeviceEngine: VoiceEngine {try engine.start()} catch {log.error("audio engine: \(error.localizedDescription, privacy: .public)")- return fail("Couldn't use the microphone: \(error.localizedDescription)")+ return fail("Couldn't use the microphone: \(error.localizedDescription)", .microphone)}task = recognizer.recognitionTask(with: request, resultHandler: Self.results(self, generation: generation))}@@ -188,9 +226,9 @@ final class OnDeviceEngine: VoiceEngine {listen()}- private func fail(_ message: String) {+ private func fail(_ message: String, _ kind: VoiceFailureKind) {stopListening()- phase = .failed(message)+ phase = .failed(message, kind)}// MARK: audio-thread closures (built outside the main actor)diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swiftindex 6feb9f58..32e9149c 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift@@ -14,7 +14,7 @@ private let log = Logger(subsystem: "com.pokai.Codync", category: "Call")@Observablefinal class OpenAIRealtimeEngine: VoiceEngine {private(set) var phase: VoicePhase = .starting {- didSet { onActivityChanged?(!ended && phase.isActive) }+ didSet { onActivityChanged?(!ended && phase.holdsCall) }}private(set) var level: Float = 0var muted = false {@@ -42,6 +42,10 @@ final class OpenAIRealtimeEngine: VoiceEngine {private var responding = falseprivate var pending: [String] = []private var dropTimer: Task<Void, Never>?+ /// The session was set up once: a drop after that is tried again instead of ending the call.+ private var connected = false+ private var reconnects = 0+ private var audioEvents: AudioSessionEvents?init(model: String, voice: String, link: CallLink) {self.model = model@@ -51,8 +55,16 @@ final class OpenAIRealtimeEngine: VoiceEngine {func start() async {guard await AVAudioApplication.requestRecordPermission() else {- return fail("Allow the microphone for Codync in Settings to talk to your bots.")+ return fail("Allow the microphone for Codync in Settings to talk to your bots.", .microphone)}+ // WebRTC pauses and restarts its own audio; the call shows the pause.+ audioEvents = AudioSessionEvents(engine: nil, interrupted: { [weak self] in self?.interrupted($0) })+ await open()+ }++ /// A session with OpenAI: the computer mints its client secret and writes the voice's instructions (with the call+ /// so far, for a session that replaces a dropped one).+ private func open() async {do {let session = try await link.start("realtime", provider: .openAI, model: model, voice: voice)guard !ended else { return }@@ -61,7 +73,7 @@ final class OpenAIRealtimeEngine: VoiceEngine {try await connect(secret: secret)} catch {guard !ended else { return }- fail(error.localizedDescription)+ if connected { dropped(error.localizedDescription) } else { fail(error.localizedDescription, .provider) }}}@@ -107,14 +119,20 @@ final class OpenAIRealtimeEngine: VoiceEngine {try await pc.setRemoteDescription(RTCSessionDescription(type: .answer, sdp: String(decoding: data, as: UTF8.self)))dropTimer = Task { [weak self] intry? await Task.sleep(for: .seconds(15))- guard let self, !Task.isCancelled, self.phase == .starting else { return }- self.fail("Couldn't reach OpenAI.")+ guard let self, !Task.isCancelled, self.phase.isConnecting else { return }+ if self.connected { self.dropped("Couldn't reach OpenAI.") } else { self.fail("Couldn't reach OpenAI.", .connection) }}}func end() {ended = trueonActivityChanged?(false)+ audioEvents?.stop()+ audioEvents = nil+ close()+ }++ private func close() {dropTimer?.cancel()events?.close()pc?.close()@@ -122,6 +140,41 @@ final class OpenAIRealtimeEngine: VoiceEngine {pc = nilmic = nildelegate = nil+ responding = false+ }++ /// The connection to OpenAI dropped mid-call: a new session, primed by the computer with the call so far, a few+ /// times before the call falls back.+ private func dropped(_ reason: String) {+ guard !ended else { return }+ guard reconnects < Reconnect.delays.count else { return fail(reason, .connection) }+ let delay = Reconnect.delays[reconnects]+ reconnects += 1+ close()+ if phase != .interrupted { phase = .reconnecting }+ Task { [weak self] in+ try? await Task.sleep(for: .seconds(delay))+ guard let self, !self.ended else { return }+ await self.open()+ }+ }++ func resume() {+ guard !ended, phase == .interrupted else { return }+ phase = pc == nil ? .reconnecting : .listening+ flush()+ }++ private func interrupted(_ change: AudioInterruption) {+ guard !ended else { return }+ switch change {+ case .began:+ guard phase.isActive else { return }+ level = 0+ phase = .interrupted+ case .ended(let shouldResume):+ if shouldResume { resume() }+ }}func interrupt() {@@ -174,19 +227,21 @@ final class OpenAIRealtimeEngine: VoiceEngine {])}- fileprivate func received(_ data: Data) {+ func received(_ data: Data) {guard !ended, let msg = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return }switch msg["type"] as? String {// Connected once the session has the instructions and tools, not when the channel opens.- case "session.updated" where phase == .starting:+ case "session.updated" where phase.isConnecting:dropTimer?.cancel()+ connected = true+ reconnects = 0phase = .listeningflush()case "input_audio_buffer.speech_started": level = muted ? 0 : 0.7case "input_audio_buffer.speech_stopped": level = 0case "input_audio_buffer.committed": link.transcribing(true)case "conversation.item.input_audio_transcription.failed": link.transcribing(false)- case "output_audio_buffer.started": phase = .speaking+ case "output_audio_buffer.started" where phase.isActive: phase = .speakingcase "output_audio_buffer.stopped", "output_audio_buffer.cleared": if phase == .speaking { phase = .listening }case "response.created": responding = truecase "response.done":@@ -220,23 +275,29 @@ final class OpenAIRealtimeEngine: VoiceEngine {}}- fileprivate func iceChanged(_ state: RTCIceConnectionState) {- // While starting, only a failure counts: the start timer runs until the session is set up.- guard !ended, phase != .starting || state == .failed else { return }+ func iceChanged(_ state: RTCIceConnectionState) {+ // While connecting, only a failure counts: the start timer runs until the session is set up.+ guard !ended, !phase.isConnecting || state == .failed else { return }switch state {- case .connected, .completed: dropTimer?.cancel()+ case .connected, .completed: if phase.isActive || phase == .interrupted { dropTimer?.cancel() }case .disconnected:dropTimer?.cancel()dropTimer = Task { [weak self] intry? await Task.sleep(for: .seconds(5))guard let self, !Task.isCancelled else { return }- self.fail("Lost the connection to OpenAI.")+ self.dropped("Lost the connection to OpenAI.")}- case .failed: fail("Lost the connection to OpenAI.")+ case .failed: if connected { dropped("Lost the connection to OpenAI.") } else { fail("Couldn't reach OpenAI.", .connection) }default: break}}+ /// The events channel closed under a running session.+ fileprivate func channelClosed(_ channel: ObjectIdentifier) {+ guard !ended, connected, events.map(ObjectIdentifier.init) == channel else { return }+ dropped("Lost the connection to OpenAI.")+ }+/// Hands waiting replies (and tool results, as empty entries) to the model once it's free.private func flush() {guard !responding, toolsRunning == 0, phase.isActive, events?.readyState == .open, !pending.isEmpty else { return }@@ -256,9 +317,9 @@ final class OpenAIRealtimeEngine: VoiceEngine {events.sendData(RTCDataBuffer(data: data, isBinary: false))}- private func fail(_ message: String) {+ private func fail(_ message: String, _ kind: VoiceFailureKind) {end()- phase = .failed(message)+ phase = .failed(message, kind)}private static let factory: RTCPeerConnectionFactory = {@@ -290,9 +351,11 @@ private final class RealtimeEvents: NSObject, RTCPeerConnectionDelegate, RTCData}func dataChannelDidChangeState(_ dataChannel: RTCDataChannel) {- guard dataChannel.readyState == .open else { return }+ let (state, channel) = (dataChannel.readyState, ObjectIdentifier(dataChannel))let owner = owner- Task { @MainActor in owner.value?.opened() }+ Task { @MainActor in+ if state == .open { owner.value?.opened() } else if state == .closed { owner.value?.channelClosed(channel) }+ }}func dataChannel(_ dataChannel: RTCDataChannel, didReceiveMessageWith buffer: RTCDataBuffer) {diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swiftindex c72ffca6..15397e71 100644--- a/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift+++ b/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift@@ -2,14 +2,35 @@ import AVFoundationimport CodyncKitimport Foundation+/// A call's audio, the same on every engine (and on Android and the desktop): connecting until the provider confirmed+/// the session and the microphone delivers, then listening or speaking; reconnecting after a drop, interrupted while+/// another app (a phone call, Siri) has the audio, failed when it can't go on.enum VoicePhase: Equatable {case startingcase listeningcase speaking- case failed(String)+ case reconnecting+ case interrupted+ case failed(String, VoiceFailureKind)var isFailed: Bool { if case .failed = self { true } else { false } }var isActive: Bool { self == .listening || self == .speaking }+ /// The connecting line shows: not ready yet, or ready again soon.+ var isConnecting: Bool { self == .starting || self == .reconnecting }+ /// The call still holds its transport and hears the computer (a reconnect or an interruption doesn't end it).+ var holdsCall: Bool { isActive || self == .reconnecting || self == .interrupted }+}++/// Why a call's engine gave up, for analytics (`voice_call_quality`).+enum VoiceFailureKind: String, Sendable {+ case microphone, connection, provider+}++/// How often a dropped provider connection is tried again before the call falls back to on-device voice: after+/// these waits, in seconds.+@MainActor+enum Reconnect {+ static var delays: [Double] = [1, 2, 4]}/// What carries a call's audio: on-device speech (default) or a realtime model on the user's own key.@@ -34,6 +55,8 @@ protocol VoiceEngine: AnyObject, Observable {func announce(_ notice: String)/// What the bot is doing while it works, from the computer: for a realtime voice to mention.func progress(_ text: String)+ /// Picks the audio up again after an interruption that didn't end on its own (the user taps the call's line).+ func resume()}extension VoiceEngine {@@ -49,6 +72,8 @@ extension VoiceEngine {/// Engines that read replies aloud have nothing to say about progress.func progress(_ text: String) {}+ func resume() {}+/// The engine's name in analytics.var analyticsName: AnalyticsEvent.VoiceEngine {if self is CloudSpeechEngine { return .cloud }diff --git a/apps/ios/Kit/Tests/CodyncKitTests/AnalyticsTests.swift b/apps/ios/Kit/Tests/CodyncKitTests/AnalyticsTests.swiftindex a3df693b..5eec738a 100644--- a/apps/ios/Kit/Tests/CodyncKitTests/AnalyticsTests.swift+++ b/apps/ios/Kit/Tests/CodyncKitTests/AnalyticsTests.swift@@ -62,6 +62,11 @@ import Testing#expect(try json(.signInFailed(.google, reason: .cancelled)) == #"{"provider":"google","reason":"cancelled"}"#)#expect(try json(.screenViewed(.botDetails)) == #"{"screen":"bot_details"}"#)#expect(try json(.voiceCallStarted(.onDevice)) == #"{"engine":"on_device"}"#)+ #expect(try json(.voiceCallQuality(.gemini, readyMs: 1200, reconnects: 1, stuck: .none, failure: .connection))+ == #"{"engine":"gemini","failure":"connection","ready_ms":1200,"reconnects":1,"stuck":"none"}"#)+ // An engine that was never ready leaves `ready_ms` out.+ #expect(try json(.voiceCallQuality(.onDevice, readyMs: nil, reconnects: 0, stuck: .connecting, failure: .none))+ == #"{"engine":"on_device","failure":"none","reconnects":0,"stuck":"connecting"}"#)#expect(try json(.linkOpened(.installCommand)) == #"{"link":"install_command"}"#)#expect(AnalyticsEvent.notificationOpened(.needsInput).name == "notification_opened")#expect(AnalyticsEvent.NotificationKind(category: "needsInput") == .needsInput)diff --git a/apps/ios/Kit/Tests/CodyncUITests/CallRecoveryTests.swift b/apps/ios/Kit/Tests/CodyncUITests/CallRecoveryTests.swiftnew file mode 100644index 00000000..96fc6eef--- /dev/null+++ b/apps/ios/Kit/Tests/CodyncUITests/CallRecoveryTests.swift@@ -0,0 +1,365 @@+import Foundation+import Testing+@testable import CodyncKit+@testable import CodyncUI++// Recorded Gemini Live and OpenAI Realtime event streams replayed through the engines, with a scripted socket and+// scripted audio (no CoreAudio in tests): phases, cues, resumption, interruptions, the #214 shapes and failure.++/// A Gemini Live socket the test plays: it hands the engine the server's frames and keeps what the engine sent.+@MainActor+final class ScriptedSocket: LiveSocket {+ private(set) var sent: [[String: Any]] = []+ private var frames: [String] = []+ private var waiting: CheckedContinuation<String, Error>?+ private var closed = false+ var closeCode = 0+ var closeReason = ""++ func send(_ text: String) {+ sent.append((try? JSONSerialization.jsonObject(with: Data(text.utf8))) as? [String: Any] ?? [:])+ }++ func receive() async throws -> String {+ if !frames.isEmpty { return frames.removeFirst() }+ if closed { throw URLError(.networkConnectionLost) }+ return try await withCheckedThrowingContinuation { waiting = $0 }+ }++ func close() { drop() }++ /// The server sends `json`.+ func play(_ json: String) {+ if let waiting {+ self.waiting = nil+ waiting.resume(returning: json)+ } else {+ frames.append(json)+ }+ }++ /// The connection drops (with the server's close code and reason, when it gave them).+ func drop(code: Int = 0, reason: String = "") {+ closeCode = code+ closeReason = reason+ closed = true+ if let waiting {+ self.waiting = nil+ waiting.resume(throwing: URLError(.networkConnectionLost))+ }+ }++ func sent(_ key: String) -> [[String: Any]] { sent.filter { $0[key] != nil } }+}++@MainActor+final class ScriptedSockets {+ private(set) var all: [ScriptedSocket] = []+ var last: ScriptedSocket? { all.last }++ func open(_ url: URL) -> any LiveSocket {+ let socket = ScriptedSocket()+ all.append(socket)+ return socket+ }+}++/// The phone's audio, scripted: the test makes the microphone deliver and decides whether playback finishes.+@MainActor+final class ScriptedAudio: CallAudio {+ var interruption: (@MainActor (AudioInterruption) -> Void)?+ private var mic: (@MainActor (String, Float) -> Void)?+ private var playing: [@MainActor () -> Void] = []+ private(set) var resumed = 0+ var started: Bool { mic != nil }++ func permission() async -> Bool { true }+ func start(mic: @escaping @MainActor (String, Float) -> Void) throws { self.mic = mic }+ func play(_ pcm: Data, done: @escaping @MainActor () -> Void) { playing.append(done) }+ func stopPlayback() { playing.removeAll() }+ func resume() throws { resumed += 1 }+ func stop() { mic = nil }++ /// The microphone delivers a buffer.+ func deliver(level: Float = 0.5) { mic?(Data(count: 640).base64EncodedString(), level) }+ /// What was queued has played.+ func finish() {+ let done = playing+ playing.removeAll()+ done.forEach { $0() }+ }+}++/// A Gemini call on a scripted computer (`FakeRemote` answers `startCall` with a fresh credential each time).+@MainActor+final class GeminiCall {+ let fake = FakeRemote(.ready(.direct))+ let store: BotStore+ let audio = ScriptedAudio()+ let sockets = ScriptedSockets()+ let engine: GeminiLiveEngine+ private let suite: String++ init() async throws {+ let (storage, suite) = context()+ self.suite = suite+ let fake = self.fake+ store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }+ store.setActive(true)+ let link = CallLink(model: store, botId: "b1", call: UUID().uuidString)+ let sockets = self.sockets+ engine = GeminiLiveEngine(model: "gemini-live", voice: "Kore", link: link, audio: audio) { sockets.open($0) }+ #expect(await until { await fake.subscribed })+ await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Bot","rev":1,"lastAt":1}}"#)+ #expect(await until { self.store.connection == .online })+ }++ /// The engine started and its first session is set up, with the microphone delivering.+ func ready() async throws {+ Task { await engine.start() }+ #expect(await until { self.sockets.last?.sent("setup").isEmpty == false })+ sockets.last?.play(#"{"setupComplete":{}}"#)+ #expect(await until { self.audio.started })+ audio.deliver()+ #expect(await until { self.engine.phase == .listening })+ }++ func close() {+ engine.end()+ store.retire()+ UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite)+ }+}++@Suite(.serialized) @MainActor struct CallRecoveryTests {+ init() {+ Reconnect.delays = [0.01, 0.01, 0.01]+ GeminiLiveEngine.stallTimeout = .milliseconds(60)+ }++ @Test func geminiIsReadyOnlyOnceTheMicrophoneDelivers() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ Task { await call.engine.start() }+ #expect(await until { call.sockets.last?.sent("setup").isEmpty == false })+ call.sockets.last?.play(#"{"setupComplete":{}}"#)+ try await Task.sleep(for: .milliseconds(50))+ #expect(call.engine.phase == .starting, "setup alone isn't ready: the microphone hasn't delivered")+ call.audio.deliver()+ #expect(await until { call.engine.phase == .listening })+ #expect(call.sockets.last?.sent("realtimeInput").isEmpty == false, "the microphone's audio goes out")+ }++ @Test func aDroppedConnectionResumesWithoutHangingUp() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.sockets.last?.play(#"{"sessionResumptionUpdate":{"newHandle":"h1","resumable":true}}"#)+ try await Task.sleep(for: .milliseconds(20))+ call.sockets.last?.drop(reason: "network")+ #expect(await until { call.engine.phase == .reconnecting })+ // A reply from the computer while it reconnects waits for the new session.+ call.engine.speak(reply: "The tests pass.")+ #expect(await until { call.sockets.all.count == 2 })+ let setup = try #require(call.sockets.last?.sent("setup").first?["setup"] as? [String: Any])+ #expect((setup["sessionResumption"] as? [String: Any])?["handle"] as? String == "h1")+ #expect(await call.fake.calls.filter { $0 == "startCall" }.count == 2, "a fresh credential from the computer")+ call.sockets.last?.play(#"{"setupComplete":{}}"#)+ #expect(await until { call.sockets.last?.sent("clientContent").count == 1 }, "the waiting reply is said")+ #expect(call.engine.phase == .reconnecting, "ready again once the microphone delivers")+ call.audio.deliver()+ #expect(await until { call.engine.phase == .listening })+ }++ @Test func goAwayCarriesOnInAResumedSession() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.sockets.last?.play(#"{"sessionResumptionUpdate":{"newHandle":"h2","resumable":true}}"#)+ call.sockets.last?.play(#"{"goAway":{"timeLeft":"5s"}}"#)+ #expect(await until { call.sockets.all.count == 2 })+ #expect(call.engine.phase == .listening, "a planned handover isn't a reconnect")+ }++ @Test func reconnectsRunOutAndTheCallFails() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ for attempt in 1...Reconnect.delays.count {+ call.sockets.last?.drop()+ #expect(await until { call.sockets.all.count == attempt + 1 })+ }+ call.sockets.last?.drop()+ #expect(await until { call.engine.phase.isFailed })+ #expect(call.engine.phase == .failed("Lost the connection to Gemini.", .connection))+ }++ @Test func aRefusedSessionFailsAtOnce() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.sockets.last?.drop(code: 1008, reason: "API key not valid.")+ #expect(await until { call.engine.phase.isFailed })+ #expect(call.engine.phase == .failed("API key not valid.", .provider), "trying again won't help")+ #expect(call.sockets.all.count == 1)+ }++ @Test func goAwayMidTurnDoesNotLeaveTheCallSpeaking() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.sockets.last?.play(#"{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"data":"AAAAAA=="}}]}}}"#)+ #expect(await until { call.engine.phase == .speaking })+ call.sockets.last?.play(#"{"goAway":{"timeLeft":"5s"}}"#)+ #expect(await until { call.sockets.all.count == 2 })+ call.audio.finish()+ #expect(await until { call.engine.phase == .listening }, "the resumed session won't complete that turn")+ call.engine.speak(reply: "Deployed.")+ call.sockets.last?.play(#"{"setupComplete":{}}"#)+ #expect(await until { call.sockets.last?.sent("clientContent").count == 1 }, "replies aren't held behind it")+ }++ @Test func playbackThatNeverFinishesGoesBackToListening() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ var stalls = 0+ call.engine.onStall = { stalls += 1 }+ try await call.ready()+ call.sockets.last?.play(#"{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"mimeType":"audio/pcm;rate=24000","data":"AAAAAA=="}}]}}}"#)+ #expect(await until { call.engine.phase == .speaking })+ call.sockets.last?.play(#"{"serverContent":{"turnComplete":true}}"#)+ call.engine.speak(reply: "Deployed.")+ // The speaker never reports the buffer rendered (#214): the watchdog takes the call back.+ #expect(await until { call.engine.phase == .listening })+ #expect(stalls == 1)+ #expect(await until { call.sockets.last?.sent("clientContent").count == 1 }, "the reply isn't stuck behind it")+ }++ @Test func playbackThatFinishesReturnsToListeningAtOnce() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ var stalls = 0+ call.engine.onStall = { stalls += 1 }+ try await call.ready()+ call.sockets.last?.play(#"{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"data":"AAAAAA=="}}]}}}"#)+ #expect(await until { call.engine.phase == .speaking })+ call.sockets.last?.play(#"{"serverContent":{"turnComplete":true}}"#)+ try await Task.sleep(for: .milliseconds(20))+ call.audio.finish()+ #expect(await until { call.engine.phase == .listening })+ try await Task.sleep(for: .milliseconds(120))+ #expect(stalls == 0)+ }++ @Test func anInterruptionPausesTheCallAndItPicksUpAgain() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.audio.interruption?(.began)+ #expect(call.engine.phase == .interrupted)+ #expect(call.engine.phase.holdsCall, "the call isn't over")+ call.engine.speak(reply: "Done.")+ try await Task.sleep(for: .milliseconds(30))+ #expect(call.sockets.last?.sent("clientContent").isEmpty == true, "nothing is said while another app has the audio")+ call.audio.interruption?(.ended(shouldResume: true))+ #expect(call.audio.resumed == 1)+ #expect(call.engine.phase == .reconnecting)+ call.audio.deliver()+ #expect(await until { call.engine.phase == .listening })+ #expect(await until { call.sockets.last?.sent("clientContent").count == 1 })+ }++ @Test func theVoiceIsNotHeardWhileAnotherAppHasTheAudio() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.audio.interruption?(.began)+ // A tool's answer that was already on its way gets a spoken reply meanwhile.+ call.sockets.last?.play(#"{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"data":"AAAAAA=="}}]}}}"#)+ call.sockets.last?.play(#"{"serverContent":{"turnComplete":true}}"#)+ try await Task.sleep(for: .milliseconds(30))+ #expect(call.engine.phase == .interrupted, "still paused, so a tap or the interruption's end picks it up")+ call.audio.interruption?(.ended(shouldResume: true))+ #expect(call.audio.resumed == 1)+ }++ @Test func anInterruptionThatDoesNotEndWaitsForATap() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ try await call.ready()+ call.audio.interruption?(.began)+ call.audio.interruption?(.ended(shouldResume: false))+ #expect(call.engine.phase == .interrupted, "without shouldResume the call stays paused")+ call.engine.resume()+ call.audio.deliver()+ #expect(await until { call.engine.phase == .listening })+ }++ @Test func openAIPhasesFollowItsEvents() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ let engine = OpenAIRealtimeEngine(model: "gpt-realtime", voice: "marin", link: CallLink(model: call.store, botId: "b1", call: "c"))+ defer { engine.end() }+ let event = { (type: String) in engine.received(Data(#"{"type":"\#(type)"}"#.utf8)) }+ event("output_audio_buffer.started")+ #expect(engine.phase == .starting, "nothing plays before the session is set up")+ event("session.updated")+ #expect(engine.phase == .listening)+ event("output_audio_buffer.started")+ #expect(engine.phase == .speaking)+ event("output_audio_buffer.stopped")+ #expect(engine.phase == .listening)+ }++ @Test func openAIReconnectsADroppedSessionAndGivesUpAfterThreeTries() async throws {+ let call = try await GeminiCall()+ defer { call.close() }+ let link = CallLink(model: call.store, botId: "b1", call: "c")+ let starting = OpenAIRealtimeEngine(model: "gpt-realtime", voice: "marin", link: link)+ starting.iceChanged(.failed)+ #expect(starting.phase == .failed("Couldn't reach OpenAI.", .connection), "a call that never started falls back")++ let engine = OpenAIRealtimeEngine(model: "gpt-realtime", voice: "marin", link: link)+ defer { engine.end() }+ engine.received(Data(#"{"type":"session.updated"}"#.utf8))+ #expect(engine.phase == .listening)+ // The computer can't be reached for the new sessions either.+ await call.fake.fail("startCall", times: 10)+ engine.iceChanged(.failed)+ #expect(engine.phase == .reconnecting, "tried again, not hung up")+ #expect(engine.phase.holdsCall)+ #expect(await until { engine.phase.isFailed })+ guard case .failed(_, let kind) = engine.phase else { return }+ #expect(kind == .connection)+ #expect(await call.fake.calls.filter { $0 == "startCall" }.count == Reconnect.delays.count, "a new session each try")+ }++ @Test func qualityReportsReadyTimeReconnectsStallsAndFailure() async throws {+ var events: [AnalyticsEvent] = []+ let quality = CallQuality { events.append($0) }+ quality.begin(.gemini)+ quality.phase(.listening)+ quality.phase(.reconnecting)+ quality.phase(.listening)+ quality.stalled()+ quality.phase(.failed("Lost the connection to Gemini.", .connection))+ quality.begin(.onDevice)+ guard case let .voiceCallQuality(engine, readyMs, reconnects, stuck, failure) = try #require(events.first) else {+ Issue.record("not a quality event")+ return+ }+ #expect(engine == .gemini && readyMs != nil && reconnects == 1 && stuck == .speaking && failure == .connection)+ quality.report()+ #expect(events.count == 2, "one report per engine")+ }++ @Test func qualityCountsAPhaseStuckConnecting() async throws {+ var events: [AnalyticsEvent] = []+ var now = ContinuousClock.now+ let quality = CallQuality(capture: { events.append($0) }, now: { now })+ quality.begin(.openai)+ now += CallQuality.stuckAfter+ quality.report()+ #expect(events == [.voiceCallQuality(.openai, readyMs: nil, reconnects: 0, stuck: .connecting, failure: .none)])+ }+}diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swiftindex e0451f04..c6be9efb 100644--- a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift@@ -70,6 +70,10 @@ actor FakeRemote: RemoteTransport {replyTos.append(b["replyTo"])let entry = #"{"id":"e\#(sent.count)","seq":\#(sent.count),"botId":"\#(b["botId"] ?? "")","rev":\#(sent.count),"kind":"user","turn":1,"data":{"text":"\#(b["text"] ?? "")","clientNonce":"\#(b["clientNonce"] ?? "")"},"createdAt":1,"updatedAt":1}"#return Data(#"{"entry":\#(entry)}"#.utf8)+ case "startCall":+ // A fresh credential per provider session, so a test can tell a replaced session from the first.+ let n = calls.filter { $0 == "startCall" }.count+ return Data(#"{"instructions":"Be the relay.","tools":[],"credential":"token-\#(n)"}"#.utf8)default:return Data("{}".utf8)}@@ -122,7 +126,7 @@ actor FakeRemote: RemoteTransport {}}-private func randomComputer(_ name: String) -> Computer {+func randomComputer(_ name: String) -> Computer {let sk = Curve25519.Signing.PrivateKey().publicKey.rawRepresentationreturn Computer(id: RelayCrypto.computerId(signKey: sk), name: name, signKey: sk.base64URLForTests,boxKey: Curve25519.KeyAgreement.PrivateKey().publicKey.rawRepresentation.base64URLForTests,@@ -135,7 +139,7 @@ private extension Data {}}-private func context() -> (SharedStore.Context, String) {+func context() -> (SharedStore.Context, String) {let suite = "CodyncUITests.\(UUID().uuidString)"return (SharedStore.Context(accountID: "user_\(UUID().uuidString)", suite: suite), suite)}diff --git a/apps/ios/WhatsNew.md b/apps/ios/WhatsNew.mdindex 6e92493b..cabd89f8 100644--- a/apps/ios/WhatsNew.md+++ b/apps/ios/WhatsNew.md@@ -11,10 +11,12 @@ a version without one gets a generic "Bug fixes and improvements" line. Delete o### zh-Hant- Bot 工作時,Gemini 與 OpenAI 的語音會主動告訴你它在做什麼、有沒有等你核准或遇到問題,做完就念出結果。- 通話中短暫斷線再連上時,錯過的回覆會補念給你聽。+- Gemini 或 OpenAI 通話斷線時會自動重新連上並接著聊,不必重打;來電、Siri 或其他 App 佔用聲音時通話會暫停,結束後自動繼續,也可以點一下通話列下方繼續。### en-US- While the bot works, the Gemini and OpenAI voice keeps you posted on what it's doing, tells you when an approval waits or something fails, and reads the result when it's done.- A reply that arrives while a call briefly loses its connection is read once it's back.+- A Gemini or OpenAI call that loses its connection reconnects and carries on without hanging up; a phone call, Siri or another app's audio pauses the call, which picks up again when it ends, or with a tap under the call bar.## 2.25.0diff --git a/docs/features/voice-call.md b/docs/features/voice-call.mdindex ffaa7ba8..2fbb56fd 100644--- a/docs/features/voice-call.md+++ b/docs/features/voice-call.md@@ -253,15 +253,18 @@ carries audio and talks to these methods:the user what the bot says and asks for what's missing; it speaks of the bot in the third person and never claimsit did or saw something; anything about code, files, the computer or the project goes to the bot even when itthinks it knows. Then the background, as named sections that say where they came from, each capped (about 16k- characters in all on Gemini, six tenths of that on OpenAI), empty ones left out: now (the computer's date, time,+ characters in all on Gemini, plus up to 3k of the call so far for a session that replaces another; six tenths of+ that on OpenAI), empty ones left out: now (the computer's date, time,weekday and offset; the phone's language), about the user (the shared `MEMORY.md` and its notes' names anddescriptions, when the bot uses it), this bot (name, description, harness and model, folder and project,permission mode, connectors, skills, computer use, routines; a group's members), what the bot remembers (its`MEMORY.md` and notes index), the project now (git branch, changed files, the last three commits; outside arepository, or without the command line tools on macOS, nothing), current work (status, activity, a pendingapproval and its options, queued messages, the last failure), the conversation (the last 20 chat messages in- spoken form and the titles and gist of the last three calls), the other bots on this computer, and files and- artifacts shared in the chat. **All of it goes to the voice provider, on the user's own key.**+ spoken form and the titles and gist of the last three calls), the other bots on this computer, files and+ artifacts shared in the chat, and, when `startCall` comes again for the same call (a reconnect, or an engine+ switched mid-call), this call so far (its newest lines). **All of it goes to the voice provider, on the user's own+ key.**- **`callTool`** `{callId, name, arguments, lines}` → `{result}` runs the voice's tool calls on the host(`calls::tools`): `send_to_bot(text)`, `bot_status()` (a group's: its busy member's), `recent_messages(count)`,`answer_approval(option)` (only after the user said which; matched by option name or id), `stop_bot()`,@@ -337,10 +340,37 @@ carries audio and talks to these methods:player never finishes its buffers so the phase stays "speaking" and the bot's replies are never said.A reply counts as played once its buffers are rendered (`.dataRendered`).+### Phases and recovery++- Every engine on every client reports the same phases: connecting (`starting`) until the provider confirmed the+ session and the microphone delivers audio, then listening or speaking; reconnecting after a dropped connection or a+ changed microphone; interrupted while another app has the audio (iPhone: a phone call, Siri, an alarm; Android:+ audio focus, a removed headset); failed. The connecting line shows while connecting and reconnecting; a reconnect or+ an interruption keeps the call and its cues (`holdsCall`).+- A drop mid-call is tried again three times (after 1, 2 and 4 s) before the call falls back. Gemini resumes with its+ session handle and a fresh `startCall` credential; OpenAI gets a new session. Either way the repeated `startCall`+ adds "This call so far" to the voice's instructions (the call's newest lines, newest last, capped), so a session+ without a handle picks up the conversation. A `goAway` hands over at once, without showing a reconnect. A close+ Gemini gives with code 1007 or 1008 (an invalid key, model or request) fails at once: trying again won't help.+- Interruptions: the iPhone's realtime, speech and on-device engines pause on `AVAudioSession.interruptionNotification`+ and pick up again when it ends with `shouldResume`; otherwise the bar says "Audio paused by another app. Tap to+ resume." (as Android does). Route changes (AirPods) restart the iPhone's audio engine (`AVAudioEngineConfigurationChange`);+ the desktop restarts the microphone (and moves OpenAI's track) on `devicechange`. The desktop's on-device engine+ uses the speech helper's own microphone and isn't restarted.+- After the voice's turn, playback that reports no progress for 4 s returns the call to listening, so a stuck player+ (#214) can't hold the bot's replies back (`onStall`, counted as stuck `speaking`).+- Each engine of a call sends one `voice_call_quality` when it's replaced or the call ends: milliseconds until ready,+ reconnects, the first phase stuck too long (connecting or reconnecting over 20 s, or the stalled player), and why+ it failed (`microphone`, `connection`, `provider`).+- Tests replay recorded provider streams through the engines with a scripted socket, microphone and speaker: kit+ `CallRecoveryTests`, desktop `call-recovery.test.ts`, Android `VoiceEnginesTest` (Gemini over MockWebServer, OpenAI+ over a fake peer).+### Failure and cost-- No key on the computer, invalid key, quota or network failure at start, or a dropped connection mid-call: the- call switches to On device without hanging up, with a one-line notice under the bar.+- No key on the computer, invalid key, quota or network failure at start, a dropped connection that three tries+ don't bring back, or a refused session: the call switches to On device without hanging up, with a one-line notice+ under the bar.- Usage is billed by the provider to the user. The key form shows minutes used this month(`VoiceUsage`, counted on each device while a cloud engine is live and for each recordingdictated with the provider) as an estimate, not a bill.@@ -375,4 +405,6 @@ each engine through fake audio: utterance cutting and clip playback, OpenAI's evexchange (MockWebServer), and a Gemini Live session over a real WebSocket (setup, audio both ways,a tool call, a close with the provider's reason). Provider audio isn'tsimulated: on a physical iPhone, check both engines with a real key (talk over a reply, ask "what-are you doing?", approve by voice, and turn off Wi-Fi mid-call to see the fallback).+are you doing?", approve by voice, and turn Wi-Fi off for a moment mid-call: the bar shows Reconnecting, comes back+to Listening and still says the replies; with no network at all, after three tries, the call falls back to on-device+voice).diff --git a/docs/plans/voice-architecture.md b/docs/plans/voice-architecture.mdindex a51664bb..dcc470c5 100644--- a/docs/plans/voice-architecture.md+++ b/docs/plans/voice-architecture.md@@ -64,6 +64,10 @@ Each stage ships on its own (after-change, merged into `dev`), all clients toget### 3. One engine state machine+Shipped in 2.26.0 ([phases and recovery](../features/voice-call.md#phases-and-recovery)). Not covered: the desktop's+on-device engine on a device change (the speech helper owns its microphone), and OpenAI's WebRTC transport isn't+replayed end to end in the iPhone and desktop tests (its events are).+- Every client's engines share the phases: connecting → ready (the provider confirmed the session and the microphoneis delivering audio) → listening / speaking, with the connecting line until ready.- Drops resume (Gemini session resumption, a fresh OpenAI session with the transcript so far); route changes anddiff --git a/docs/reference/analytics-events.md b/docs/reference/analytics-events.mdindex 33a786bc..1b3cc62a 100644--- a/docs/reference/analytics-events.md+++ b/docs/reference/analytics-events.md@@ -81,6 +81,7 @@ host answers and sends it once the host is up.| `routine_run` | host | `runRoutine` | — || `routine_retried` | host | `retryRoutineRun` (the retry button on a failed run's notice) | — || `voice_call_started` | host `track` (desktop), iOS, Android | A voice call starts, with the engine it really uses after any fallback | `engine` (`on_device`; `openai`/`gemini` realtime; `cloud`: cloud speech to text and back) |+| `voice_call_quality` | host `track` (desktop), iOS, Android | One engine of a call is replaced or the call ends | `engine` (as `voice_call_started`), `ready_ms` (milliseconds until the provider confirmed the session and the microphone delivered; left out if it never was), `reconnects`, `stuck` (`none`/`connecting`/`reconnecting` over 20 s/`speaking`: playback stalled after the voice's turn), `failure` (`none`/`microphone`/`connection`/`provider`) || `voice_call_finished` | host | `endCall` (older apps `logCall`) | `seconds` || `remote_screen_opened` | host | `screenPrepare` (each connection, reconnects included) | — || `screen_taken_over` | host | `screenTakeover` (taking control, not handing it back) | — |diff --git a/host/src/analytics.rs b/host/src/analytics.rsindex 8e1d6fc4..2eb2035b 100644--- a/host/src/analytics.rs+++ b/host/src/analytics.rs@@ -78,6 +78,8 @@ pub enum Event {RoutineRetried,VoiceCallStarted,VoiceCallFinished,+ /// How one engine of a call fared (time to ready, reconnects, a phase stuck, failure).+ VoiceCallQuality,RemoteScreenOpened,ScreenTakenOver,ConnectorInstalled,@@ -137,6 +139,7 @@ impl Event {Self::RoutineRetried => "routine_retried",Self::VoiceCallStarted => "voice_call_started",Self::VoiceCallFinished => "voice_call_finished",+ Self::VoiceCallQuality => "voice_call_quality",Self::RemoteScreenOpened => "remote_screen_opened",Self::ScreenTakenOver => "screen_taken_over",Self::ConnectorInstalled => "connector_installed",diff --git a/host/src/analytics/reported.rs b/host/src/analytics/reported.rsindex dfd3a6cb..9293e41f 100644--- a/host/src/analytics/reported.rs+++ b/host/src/analytics/reported.rs@@ -73,6 +73,15 @@ pub enum Reported {VoiceCallStarted {engine: Engine,},+ VoiceCallQuality {+ engine: Engine,+ /// Milliseconds until the engine was ready; left out when it never was.+ #[serde(skip_serializing_if = "Option::is_none")]+ ready_ms: Option<u64>,+ reconnects: u32,+ stuck: VoiceStuck,+ failure: VoiceFailure,+ },}// Value lists. A value a newer client adds arrives as `other` instead of being refused.@@ -164,6 +173,30 @@ pub enum Engine {Other,}+/// The first phase of a call's engine that lasted too long.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum VoiceStuck {+ None,+ Connecting,+ Reconnecting,+ Speaking,+ #[serde(other)]+ Other,+}++/// Why a call's engine gave up.+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize, Deserialize)]+#[serde(rename_all = "snake_case")]+pub enum VoiceFailure {+ None,+ Microphone,+ Connection,+ Provider,+ #[serde(other)]+ Other,+}+impl Track {/// The body to send for `event`, as `app` (what the terminal UI posts).pub fn body(app: App, event: Reported) -> Value {@@ -254,6 +287,11 @@ mod tests {(json!({"event": "link_opened", "app": "desktop", "link": "install_command"}), Event::LinkOpened),(json!({"event": "screen_viewed", "app": "tui", "screen": "bot_details"}), Event::ScreenViewed),(json!({"event": "voice_call_started", "app": "desktop", "engine": "on_device"}), Event::VoiceCallStarted),+ (+ json!({"event": "voice_call_quality", "app": "desktop", "engine": "gemini", "ready_ms": 1200,+ "reconnects": 1, "stuck": "none", "failure": "none"}),+ Event::VoiceCallQuality,+ ),];for (body, event) in cases {let mut expected = body.clone();diff --git a/host/src/chat/calls/context.rs b/host/src/chat/calls/context.rsindex 7474aefe..329491fb 100644--- a/host/src/chat/calls/context.rs+++ b/host/src/chat/calls/context.rs@@ -1,7 +1,7 @@//! What the voice of a call hears before it speaks: its rules, then the background as named sections, each capped,//! empty ones left out, each saying where it came from. It goes to the voice provider, on the user's own key.-use super::speech::spoken;+use super::speech::{call_so_far, spoken};use super::tools::{runtime, waiting};use crate::hub::{BotStatus, Hub};use crate::store::{BotConfig, Entry, EntryKind, Lane};@@ -10,8 +10,9 @@ use codync_mem::files::{self as mf, MAIN};use std::path::Path;use std::sync::Arc;-/// Each section's cap (characters) and the whole background's: about 16k on Gemini. `OpenAI` Realtime keeps a-/// smaller context, so it gets [`OPENAI_TENTHS`] tenths of each.+/// Each section's cap (characters) and the whole background's: about 16k on Gemini, and up to 3k more of the call so+/// far for a session that replaces another. `OpenAI` Realtime keeps a smaller context, so it gets [`OPENAI_TENTHS`]+/// tenths of each.const CAPS: Caps = Caps {now: 300,user: 3_000,@@ -22,6 +23,7 @@ const CAPS: Caps = Caps {conversation: 3_500,others: 1_000,shared: 700,+ so_far: 3_000,};/// Of each cap, in tenths.const OPENAI_TENTHS: usize = 6;@@ -41,6 +43,7 @@ struct Caps {conversation: usize,others: usize,shared: usize,+ so_far: usize,}/// The voice's role first: a relay between the user and the bot (or group), never the bot itself.@@ -77,12 +80,14 @@ for speech (no code blocks, no long lists, no URLs).)}-/// The voice's instructions for a call with `cfg`'s bot.+/// The voice's instructions for a call with `cfg`'s bot; `so_far`: what was said on the call before this provider+/// session (one that replaces a dropped session picks up from it).pub async fn instructions(hub: &Arc<Hub>,cfg: &BotConfig,provider: Option<Provider>,language: Option<&str>,+ so_far: &[super::CallLine],) -> String {let tenths = if provider == Some(Provider::OpenAi) { OPENAI_TENTHS } else { 10 };let cap = |n: usize| n * tenths / 10;@@ -101,6 +106,7 @@ pub async fn instructions(("The conversation", "the bot's chat and earlier calls", sections.conversation, caps.conversation),("Other bots on this computer", "Codync", sections.others, caps.others),("Shared in this chat", "files and artifacts in the bot's chat", sections.shared, caps.shared),+ ("This call so far", "this call's transcript, newest last", call_so_far(so_far, cap(caps.so_far)), caps.so_far),];for (title, from, body, n) in named {let body = body.trim();@@ -421,7 +427,7 @@ pub(super) mod tests {.unwrap();}- let gemini = instructions(&hub, &cfg, Some(Provider::Gemini), Some("zh-TW")).await;+ let gemini = instructions(&hub, &cfg, Some(Provider::Gemini), Some("zh-TW"), &[]).await;assert!(gemini.starts_with("You are the voice relay for Mika"), "the role comes first: {}", &gemini[..80]);assert!(gemini.contains("You are not Mika") && gemini.contains("send_to_bot"));for heading in [@@ -450,7 +456,7 @@ pub(super) mod tests {assert!(conversation.chars().count() <= CAPS.conversation + 80, "the conversation keeps to its cap");let background = gemini.chars().count() - rules(&hub, &cfg).chars().count();assert!(background <= 16_000 + 1_000, "the background keeps to about 16k: {background}");- let openai = instructions(&hub, &cfg, Some(Provider::OpenAi), None).await;+ let openai = instructions(&hub, &cfg, Some(Provider::OpenAi), None, &[]).await;assert!(openai.chars().count() < gemini.chars().count(), "OpenAI's smaller context gets less");}diff --git a/host/src/chat/calls/mod.rs b/host/src/chat/calls/mod.rsindex d2a231c5..3f868c4d 100644--- a/host/src/chat/calls/mod.rs+++ b/host/src/chat/calls/mod.rs@@ -240,19 +240,23 @@ pub async fn start(hub: &Arc<Hub>, b: &Value, owner: &str) -> Result<Value> {let call = checked(b["callId"].as_str().unwrap_or_default())?;let mode = if b["mode"].is_null() { CallMode::Realtime } else { CallMode::deserialize(&b["mode"])? };let cfg = hub.store.bot(bot)?.filter(|r| !r.deleted).ok_or_else(|| anyhow!("unknown bot"))?.config;- {+ let so_far = {let _one = STATE.locked();match begun(&hub.store, bot, call)? {Some(s) if s.bot != bot => bail!("that call belongs to another bot"),// A session the provider replaced, maybe on another device or in another mode now.- Some(s) => save(&hub.store, call, &CallState { owner: owner.to_owned(), mode, ..s })?,+ Some(s) => {+ let s = CallState { owner: owner.to_owned(), mode, ..s };+ save(&hub.store, call, &s)?;+ s.lines+ }None => bail!("that call has ended"),}- }+ };announce::watch(hub, call, bot);let provider = b["provider"].as_str().map(crate::voice::Provider::parse).transpose()?;let mut out = json!({- "instructions": context::instructions(hub, &cfg, provider, b["language"].as_str()).await,+ "instructions": context::instructions(hub, &cfg, provider, b["language"].as_str(), &so_far).await,"tools": tools::declarations(),});if let (Some(p), CallMode::Realtime) = (provider, mode) {@@ -484,6 +488,10 @@ mod tests {assert!(started["tools"].as_array().unwrap().iter().any(|t| t["name"] == "search_memory"));assert!(started.get("credential").is_none(), "an on-device call has no provider");add_lines(&hub, "c1", &json!([{"from": "user", "text": "首頁", "at": 1}])).unwrap();+ // A provider session that replaces a dropped one picks up from what was said.+ let again = start(&hub, &json!({"botId": id, "callId": "c1", "mode": "device"}), "phone").await.unwrap();+ let instructions = again["instructions"].as_str().unwrap();+ assert!(instructions.contains("This call so far") && instructions.contains("User: 首頁"), "{instructions}");// The rest of what the user said comes with the tool call that hands it on.let unsent = json!([{"from": "user", "text": "改成藍色", "at": 2}, {"from": "voice", "text": "好"}]);diff --git a/host/src/chat/calls/speech.rs b/host/src/chat/calls/speech.rsindex eb4d0b4d..c7827720 100644--- a/host/src/chat/calls/speech.rs+++ b/host/src/chat/calls/speech.rs@@ -1,5 +1,7 @@//! Text as a voice says it.+use super::{CallLine, Speaker};+/// Text as speech can carry it: code blocks and links named, not read, markdown marks dropped, capped.pub(super) fn spoken(text: &str, max: usize) -> String {let mut out = String::new();@@ -33,6 +35,26 @@ pub(super) fn spoken(text: &str, max: usize) -> String {crate::agent::acp::truncate(out.split_whitespace().collect::<Vec<_>>().join(" ").as_str(), max)}+/// The call's lines, newest last, as many of the newest as fit `cap` characters.+pub(super) fn call_so_far(lines: &[CallLine], cap: usize) -> String {+ let mut kept: Vec<String> = Vec::new();+ let mut used = 0;+ for line in lines.iter().rev() {+ let who = match line.from {+ Speaker::User => "User",+ Speaker::Voice => "You",+ };+ let text = format!("{who}: {}", spoken(&line.text, 400));+ used += text.chars().count() + 1;+ if used > cap {+ break;+ }+ kept.push(text);+ }+ kept.reverse();+ kept.join("\n")+}+#[cfg(test)]mod tests {use super::*;@@ -49,4 +71,14 @@ mod tests {"Run cargo test --release. Done, keep snake_case");}++ #[test]+ fn a_replaced_session_hears_the_newest_of_the_call_so_far() {+ let line = |from, text: &str| CallLine { from, text: text.into(), at: None, id: None };+ let lines =+ [line(Speaker::User, "first thing"), line(Speaker::Voice, "noted"), line(Speaker::User, "second thing")];+ assert_eq!(call_so_far(&lines, 1_000), "User: first thing\nYou: noted\nUser: second thing");+ assert_eq!(call_so_far(&lines, 20), "User: second thing", "the newest lines win when they don't all fit");+ assert_eq!(call_so_far(&[], 1_000), "");+ }}diff --git a/web/content/docs/privacy.mdx b/web/content/docs/privacy.mdxindex 84f7ca72..92615103 100644--- a/web/content/docs/privacy.mdx+++ b/web/content/docs/privacy.mdx@@ -44,7 +44,7 @@ What leaves your device in a voice call or during dictation depends on the engin| Engine | Where your voice goes || --- | --- || On device (the default) | Recognized on your iPhone or Mac with Apple speech. Apple's servers may be used when on-device recognition isn't available for your language. Only the recognized text goes to your computer, as an ordinary message. |-| OpenAI or Gemini, Realtime mode | Straight from your device to the provider. Your computer gives the device a short-lived credential made from your key. Before the call, the provider also gets the background your computer writes for the voice: the date and time, what your bots remember about you (shared memory and the names of its notes), this bot's settings, routines and memory, the project's git branch, changed files and recent commits, what the bot is working on, the recent messages and calls in the chat, your other bots, and the names of files shared in the chat. During the call it gets the messages and results the voice reads. A written transcript of the call stays on your computer, in the chat. |+| OpenAI or Gemini, Realtime mode | Straight from your device to the provider. Your computer gives the device a short-lived credential made from your key. Before the call, the provider also gets the background your computer writes for the voice: the date and time, what your bots remember about you (shared memory and the names of its notes), this bot's settings, routines and memory, the project's git branch, changed files and recent commits, what the bot is working on, the recent messages and calls in the chat, your other bots, and the names of files shared in the chat. When a call reconnects or you switch engine mid-call, it also gets what was said on the call so far. During the call it gets the messages and results the voice reads. Your computer keeps the call's written transcript, in the chat. || OpenAI or Gemini, Speech mode, and dictation with a provider | Your recording goes to your computer over the encrypted connection, and your computer sends it to the provider with your key and the names of the bots on the computer as spelling hints. The provider's spoken replies come back the same way. |Your provider key stays on your computer, encrypted. It's never sent to the relay or Codync's cloud. Calls with your own key run under your provider account and its privacy policy.diff --git a/web/content/docs/voice.mdx b/web/content/docs/voice.mdxindex 52c64b25..a2e6cbf9 100644--- a/web/content/docs/voice.mdx+++ b/web/content/docs/voice.mdx@@ -64,11 +64,13 @@ Each provider has two modes, under **Mode**:In Realtime mode, the voice model is the bot's voice line, not the bot: it talks with you, and the bot does the work. It passes your requests on to the bot (along with your own words, so nothing gets lost in its wording), reads the bot's replies out in shortened form, keeps you posted while the bot works (and tells you what it's doing when you ask), lets you know when an approval is waiting, answers an approval for you once you've said which option, can stop the bot, and can look up what the bot remembers.-Before a Realtime call, your computer gives the voice model the background it needs: the date and time, what your bots remember about you, this bot's settings and memory, the project's git branch and recent changes, what the bot is working on, the recent messages and calls in this chat, your other bots, and the files shared in the chat. Like your audio, it goes to OpenAI or Google on your own key.+Before a Realtime call, your computer gives the voice model the background it needs: the date and time, what your bots remember about you, this bot's settings and memory, the project's git branch and recent changes, what the bot is working on, the recent messages and calls in this chat, your other bots, and the files shared in the chat. When the call reconnects or you switch engine mid-call, the voice model also gets what was said on the call so far. Like your audio, it goes to OpenAI or Google on your own key.Below the key you can pick the **Model** (or, in Speech mode, **Speech to text** and **Text to speech**) and the **Voice**. The default models follow each provider's newest ones. **This month** shows an estimate of the minutes used, counted on each device. The provider bills you; it isn't a bill.-If a provider can't start, or the connection drops mid-call, the call switches to on-device voice without hanging up and says so under the bar. (In the desktop app this happens on macOS, where on-device speech exists.)+If the connection drops mid-call, the bar shows the connecting line with "Reconnecting" and the call picks up where it left off, usually within a few seconds, without hanging up. A phone call, Siri or another app taking the audio pauses the call; it picks up again when that ends, or tap **Audio paused by another app. Tap to resume.** under the bar. On the iPhone and the computer, plugging in or removing a headset moves the call to the new microphone; on Android, removing the headset pauses the call until you tap to resume.++If a provider can't start, or the connection doesn't come back after a few tries, the call switches to on-device voice without hanging up and says so under the bar. (In the desktop app this happens on macOS, where on-device speech exists.)## Add your API key