Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

feat(apps): connecting line glides back and forth until the call is really connected

leepokaieec9e1532629
diff --git a/apps/android/WhatsNew.md b/apps/android/WhatsNew.md
index 40d0ac5f..34730dcb 100644
--- a/apps/android/WhatsNew.md
+++ b/apps/android/WhatsNew.md
@@ -19,7 +19,7 @@ else a generic "Bug fixes and improvements" line. Delete old sections.
- 在設定開啟「顯示思考細節」後,點一下工作中 Bot 的狀態膠囊,就能展開這一輪的思考與步驟。
- 點對話上方的 Bot 會打開它的詳情、資料庫與設定;「新會話」改到 Bot 列表的長按選單。
- 通話中在設定換了語音,關掉設定後通話就改用新的語音繼續。
-- 語音通話連線中,通話列會顯示一條跑動的進度線。
+- 語音通話連線中,通話列會顯示一條左右來回的線,真正連上後才換回點點。
### en-US
- Bots make artifacts, shown as chat cards; message the bot below one to edit it.
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
index 21a02938..84b9703c 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
@@ -27,7 +27,6 @@ import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.geometry.Size
import androidx.compose.ui.graphics.Color
import androidx.compose.ui.graphics.drawscope.DrawScope
-import androidx.compose.ui.graphics.drawscope.clipRect
import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.semantics.contentDescription
@@ -36,6 +35,8 @@ import androidx.compose.ui.text.style.TextAlign
import androidx.compose.ui.unit.dp
import androidx.compose.ui.text.style.TextOverflow
import com.kyant.shapes.Capsule
+import kotlin.math.PI
+import kotlin.math.cos
import kotlin.math.sin
import androidx.core.content.ContextCompat
import com.codync.android.core.*
@@ -127,14 +128,14 @@ private fun statusText(voice: VoiceState, working: Boolean) = when (voice.phase)
VoicePhase.Listening -> if (voice.muted) "Muted" else if (working) "Working, listening" else "Listening"
}
-/** Seconds for the connecting segment to cross the track. */
-private const val CONNECTING_PERIOD = 1.4f
+/** Seconds for the connecting segment's trip there and back. */
+private const val CONNECTING_PERIOD = 2.4f
/**
* The dotted line in the call bar: it follows your voice while listening, ripples while the bot
* talks, breathes while it works, and rests as faint dots otherwise. While the call connects it shows
- * an indeterminate progress line instead (a short segment sliding along a faint track, resting at the
- * left with reduced motion), and fades back to the dots.
+ * an indeterminate progress line instead (a short segment gliding back and forth along a faint track,
+ * resting at the left with reduced motion), and fades back to the dots.
*/
@Composable private fun CallLevelDots(level: Float, speaking: Boolean, working: Boolean, connecting: Boolean, color: Color, modifier: Modifier) {
val still = reduceMotion()
@@ -162,15 +163,14 @@ private const val CONNECTING_PERIOD = 1.4f
}
}
-/** The connecting line at `alpha`: a faint track and a segment a third its length sliding across it, at the left when `still`. */
+/** The connecting line at `alpha`: a faint track and a segment a third its length gliding back and forth, at the left when `still`. */
private fun DrawScope.connectingLine(accent: Color, alpha: Float, time: Float, still: Boolean) {
val mid = size.height / 2
drawRoundRect(accent.copy(alpha = 0.15f * alpha), Offset(0f, mid - 0.75.dp.toPx()), Size(size.width, 1.5.dp.toPx()),
CornerRadius(0.75.dp.toPx()))
val segment = size.width / 3
- val x = if (still) 0f else -segment + (time % CONNECTING_PERIOD) / CONNECTING_PERIOD * (size.width + segment)
- clipRect {
- drawRoundRect(accent.copy(alpha = alpha), Offset(x, mid - 1.75.dp.toPx()), Size(segment, 3.5.dp.toPx()),
- CornerRadius(1.75.dp.toPx()))
- }
+ // A cosine turns each end smoothly instead of jumping back to the start.
+ val progress = if (still) 0f else (1 - cos(2 * PI.toFloat() * time / CONNECTING_PERIOD)) / 2
+ drawRoundRect(accent.copy(alpha = alpha), Offset(progress * (size.width - segment), mid - 1.75.dp.toPx()),
+ Size(segment, 3.5.dp.toPx()), CornerRadius(1.75.dp.toPx()))
}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt
index d9d66f68..87547e3c 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt
@@ -30,7 +30,6 @@ class CloudSpeechEngine(parent: CoroutineScope, provider: VoiceProvider, setting
mic.start({ pcm, level -> hop { heard(pcm, level) } }) {
hop { fail("Couldn't use the microphone. Check its permission and other audio apps.") }
}
- phase(VoicePhase.Listening)
scope.launch {
try {
for (wav in utterances) {
@@ -60,6 +59,8 @@ class CloudSpeechEngine(parent: CoroutineScope, provider: VoiceProvider, setting
}
private fun heard(pcm: ByteArray, level: Float) {
+ // Listening once the microphone actually delivers.
+ if (!ended && phase == VoicePhase.Starting) phase(VoicePhase.Listening)
if (muted || phase != VoicePhase.Listening || clock() < deafUntil) return
level(level)
detector.feed(pcm, level, clock(), state.value.settings.pauseMillis)?.let { utterances.trySend(it) }
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
index d302e054..acbfda95 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
@@ -110,7 +110,6 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private
}
audioRunning = true
}
- if (phase == VoicePhase.Starting) phase(VoicePhase.Listening)
flush()
}
message.objectAt("serverContent")?.let { content ->
@@ -164,6 +163,8 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private
}
private fun captured(pcm: ByteArray, level: Float) {
+ // Connected once the microphone delivers after setup, not when the socket opens.
+ if (ready && phase == VoicePhase.Starting) phase(VoicePhase.Listening)
if (muted || !active) return
level(if (phase == VoicePhase.Speaking) 0f else level)
send(buildJsonObject {
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
index c091846e..8f62f717 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
@@ -73,7 +73,6 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, priv
}
override fun opened() = hop {
- dropTimer?.cancel()
send(buildJsonObject {
put("type", "session.update")
putJsonObject("session") {
@@ -83,13 +82,13 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, priv
putJsonObject("audio") { putJsonObject("input") { putJsonObject("turn_detection") { put("type", "semantic_vad") } } }
}
})
- phase(VoicePhase.Listening)
- flush()
}
override fun received(text: String) = hop {
val message = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: return@hop
when (message.text("type")) {
+ // Connected once the session has the instructions and tools, not when the channel opens.
+ "session.updated" -> if (phase == VoicePhase.Starting) { dropTimer?.cancel(); phase(VoicePhase.Listening); flush() }
"input_audio_buffer.speech_started" -> level(if (muted) 0f else .7f)
"input_audio_buffer.speech_stopped" -> level(0f)
"output_audio_buffer.started" -> if (phase == VoicePhase.Listening) phase(VoicePhase.Speaking)
@@ -112,6 +111,8 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, priv
}
override fun ice(state: RealtimePeer.Ice) = hop {
+ // While starting, only a failure counts: the start timer runs until the session is set up.
+ if (phase == VoicePhase.Starting && state != RealtimePeer.Ice.Failed) return@hop
when (state) {
RealtimePeer.Ice.Connected -> dropTimer?.cancel()
RealtimePeer.Ice.Disconnected -> drop(5_000, "Lost the connection to OpenAI.") { true }
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt
index fcb05f93..3525a923 100644
--- a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt
@@ -144,7 +144,7 @@ class VoiceEnginesTest {
val engine = CloudSpeechEngine(backgroundScope, VoiceProvider.Gemini, VoiceSettings(pauseMillis = 1_000), mic,
{ _, _ -> val clip = CompletableDeferred<Unit>().also { playing += it }
try { clip.await() } catch (cancelled: CancellationException) { clip.cancel(); throw cancelled } }, speech) { testScheduler.currentTime }
- runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ runCurrent(); assertEquals(VoicePhase.Starting, engine.state.value.phase) // Until the microphone delivers.
val frame = ByteArray(3_200)
repeat(6) { mic.frames!!(frame, .8f); advanceTimeBy(100); runCurrent() }
mic.frames!!(frame, 0f); advanceTimeBy(1_100); runCurrent(); mic.frames!!(frame, 0f); runCurrent()
@@ -196,6 +196,8 @@ class VoiceEnginesTest {
assertEquals("ek_secret" to "offer-sdp", exchanged); assertEquals("answer-sdp", peer.answer); assertEquals(true, peer.micOn)
engine.speak("Early reply"); runCurrent(); assertTrue(peer.sent.isEmpty()) // Waits for the channel.
peer.isOpen = true; peer.events!!.opened(); runCurrent()
+ assertEquals(listOf("session.update"), peer.types()); assertEquals(VoicePhase.Starting, engine.state.value.phase)
+ peer.events!!.received("""{"type":"session.updated"}"""); runCurrent()
assertEquals(listOf("session.update", "conversation.item.create", "response.create"), peer.types())
assertEquals("[Ada replied] Early reply", peer.sent[1]["item"]!!.jsonObject["content"]!!.jsonArray[0].jsonObject["text"]!!.jsonPrimitive.content)
assertEquals(VoicePhase.Listening, engine.state.value.phase)
@@ -224,6 +226,17 @@ class VoiceEnginesTest {
assertEquals(VoicePhase.Failed, engine.state.value.phase); assertEquals("Couldn't reach OpenAI.", engine.state.value.error)
}
+ @Test fun openAIRealtimeStartTimerOutlastsIceUntilTheSessionIsSetUp() = runTest {
+ val peer = Peer()
+ val engine = OpenAIRealtimeEngine(backgroundScope, VoiceSettings(), { "ek" }, CallOperator("ada", Desk(emptyList(), emptyList())), peer) { _, _ -> "sdp" }
+ runCurrent(); peer.isOpen = true; peer.events!!.opened()
+ peer.events!!.ice(RealtimePeer.Ice.Disconnected); advanceTimeBy(5_001); runCurrent()
+ assertEquals(VoicePhase.Starting, engine.state.value.phase) // No 5 s drop timer while starting.
+ peer.events!!.ice(RealtimePeer.Ice.Connected); advanceTimeBy(10_000); runCurrent()
+ assertEquals("Couldn't reach OpenAI.", engine.state.value.error) // ICE alone isn't connected.
+ engine.awaitEnd()
+ }
+
@Test fun openAICallsEndpointPostsTheOfferAndExplainsRefusals() = runBlocking {
MockWebServer().use { server ->
server.enqueue(MockResponse().setBody("v=0 answer")); server.enqueue(MockResponse().setResponseCode(401).setBody("""{"error":{"message":"Invalid client secret"}}"""))
@@ -267,9 +280,11 @@ class VoiceEnginesTest {
assertEquals(4, setup["tools"]!!.jsonArray[0].jsonObject["functionDeclarations"]!!.jsonArray.size)
engine.speak("Queued before setup")
server!!.send("""{"setupComplete":{}}""")
- until { it.phase == VoicePhase.Listening }
assertEquals("[Ada replied] Queued before setup", next()["clientContent"]!!.jsonObject["turns"]!!.jsonArray[0].jsonObject["parts"]!!.jsonArray[0].jsonObject["text"]!!.jsonPrimitive.content)
+ assertEquals(VoicePhase.Starting, engine.state.value.phase) // Connecting until the microphone delivers.
+ withTimeout(5_000) { while (mic.frames == null) delay(10) }
mic.frames!!(byteArrayOf(1, 2), .5f)
+ until { it.phase == VoicePhase.Listening }
assertEquals("audio/pcm;rate=16000", next()["realtimeInput"]!!.jsonObject["audio"]!!.jsonObject["mimeType"]!!.jsonPrimitive.content)
val audio = Base64.getEncoder().encodeToString(ByteArray(480))
server!!.send("""{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"mimeType":"audio/pcm;rate=24000","data":"$audio"}}]}}}""")
diff --git a/apps/desktop/src/renderer/components/LevelDots.tsx b/apps/desktop/src/renderer/components/LevelDots.tsx
index c94b52b7..769dda2a 100644
--- a/apps/desktop/src/renderer/components/LevelDots.tsx
+++ b/apps/desktop/src/renderer/components/LevelDots.tsx
@@ -11,14 +11,14 @@ export interface LevelSample {
/** The gap between dots, in points. */
const SPACING = 8
-/** Seconds for the connecting segment to cross the track. */
-const PERIOD = 1.4
+/** Seconds for the connecting segment's trip there and back. */
+const PERIOD = 2.4
/**
* A dotted line that follows your voice while listening, ripples while the other side talks,
* breathes while work goes on, and rests as faint dots otherwise. While a call connects it shows an
- * indeterminate progress line instead (a short segment sliding along a faint track, resting at the left
- * with reduced motion), and fades back to the dots. `sample` is read every frame.
+ * indeterminate progress line instead (a short segment gliding back and forth along a faint track, resting
+ * at the left with reduced motion), and fades back to the dots. `sample` is read every frame.
* `color` null is the text color.
*/
export function LevelDots({ sample, color, width, height = 16, label, className }: {
@@ -81,21 +81,17 @@ export function LevelDots({ sample, color, width, height = 16, label, className
return <canvas ref={canvas} role="img" aria-label={label} className={className} style={{ width, height }} />
}
-/** The connecting line: a faint track and a segment a third its length sliding across it (`t` in seconds), resting at the left with reduced motion. */
+/** The connecting line: a faint track and a segment a third its length gliding back and forth (`t` in seconds), resting at the left with reduced motion. */
function drawConnecting(ctx: CanvasRenderingContext2D, width: number, height: number, t: number, accent: string, reduceMotion: boolean) {
ctx.fillStyle = withAlpha(accent, 0.15)
ctx.beginPath()
ctx.roundRect(0, height / 2 - 0.75, width, 1.5, 0.75)
ctx.fill()
const segment = width / 3
- const x = reduceMotion ? 0 : -segment + ((t % PERIOD) / PERIOD) * (width + segment)
- ctx.save()
- ctx.beginPath()
- ctx.rect(0, 0, width, height)
- ctx.clip()
+ // A cosine turns each end smoothly instead of jumping back to the start.
+ const progress = reduceMotion ? 0 : (1 - Math.cos((2 * Math.PI * t) / PERIOD)) / 2
ctx.fillStyle = accent
ctx.beginPath()
- ctx.roundRect(x, height / 2 - 1.75, segment, 3.5, 1.75)
+ ctx.roundRect(progress * (width - segment), height / 2 - 1.75, segment, 3.5, 1.75)
ctx.fill()
- ctx.restore()
}
diff --git a/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts b/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts
index 0d52743e..54a93f96 100644
--- a/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts
+++ b/apps/desktop/src/renderer/views/call/CloudSpeechEngine.ts
@@ -91,7 +91,6 @@ export class CloudSpeechEngine extends VoiceEngine {
} catch (e) {
return this.fail(errorMessage(e))
}
- this.setPhase('listening')
}
end() {
@@ -123,6 +122,8 @@ export class CloudSpeechEngine extends VoiceEngine {
// MARK: listening
private heard(pcm: Int16Array, level: number) {
+ // Listening once the microphone actually delivers.
+ if (!this.ended && this.phase === 'starting') this.setPhase('listening')
if (this.ended || this.muted || this.phase !== 'listening' || Date.now() < this.deafUntil) return
this.level = level
if (level > THRESHOLD) {
diff --git a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
index eaa08604..b58637ca 100644
--- a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
+++ b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
@@ -156,7 +156,6 @@ export class GeminiLiveEngine extends VoiceEngine {
this.ready = true
this.reconnects = 0
if (!this.stopMic) void this.startAudio()
- if (this.phase === 'starting') this.setPhase('listening')
this.flush()
}
const content = msg.serverContent
@@ -225,6 +224,8 @@ export class GeminiLiveEngine extends VoiceEngine {
}
private captured(pcm: Int16Array, level: number) {
+ // Connected once the microphone delivers after setup, not when the socket opens.
+ if (!this.ended && this.ready && this.phase === 'starting') this.setPhase('listening')
if (this.ended || this.muted || !this.isActive) return
this.level = this.phase === 'speaking' ? 0 : level
this.send({ realtimeInput: { audio: { data: toBase64(new Uint8Array(pcm.buffer)), mimeType: 'audio/pcm;rate=16000' } } })
diff --git a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
index 69fda49b..6a9509ec 100644
--- a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
+++ b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
@@ -127,7 +127,6 @@ export class OpenAIRealtimeEngine extends VoiceEngine {
private opened() {
if (this.ended) return
- if (this.dropTimer) clearTimeout(this.dropTimer)
this.send({
type: 'session.update',
session: {
@@ -137,8 +136,6 @@ export class OpenAIRealtimeEngine extends VoiceEngine {
audio: { input: { turn_detection: { type: 'semantic_vad' } } },
},
})
- this.setPhase('listening')
- this.flush()
}
private received(data: string) {
@@ -150,6 +147,13 @@ export class OpenAIRealtimeEngine extends VoiceEngine {
return
}
switch (msg.type) {
+ case 'session.updated':
+ // Connected once the session has the instructions and tools, not when the channel opens.
+ if (this.phase !== 'starting') break
+ if (this.dropTimer) clearTimeout(this.dropTimer)
+ this.setPhase('listening')
+ this.flush()
+ break
case 'input_audio_buffer.speech_started':
this.level = this.muted ? 0 : 0.7
break
@@ -189,7 +193,8 @@ export class OpenAIRealtimeEngine extends VoiceEngine {
}
private iceChanged(state: RTCIceConnectionState) {
- if (this.ended) return
+ // While starting, only a failure counts: the start timer runs until the session is set up.
+ if (this.ended || (this.phase === 'starting' && state !== 'failed')) return
switch (state) {
case 'connected':
case 'completed':
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift
index 3e9f375f..73797f83 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CloudSpeechEngine.swift
@@ -95,7 +95,6 @@ final class CloudSpeechEngine: VoiceEngine {
} catch {
return fail("Couldn't use the microphone: \(error.localizedDescription)")
}
- phase = .listening
}
func end() {
@@ -124,6 +123,8 @@ final class CloudSpeechEngine: VoiceEngine {
// MARK: listening
private func heard(_ pcm: Data, level: Float) {
+ // Listening once the microphone actually delivers.
+ if !ended, phase == .starting { phase = .listening }
guard !ended, !muted, phase == .listening, Date.now >= deafUntil else { return }
self.level = level
if level > Self.threshold {
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
index cf4aa689..bce54cdf 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
@@ -192,7 +192,6 @@ final class GeminiLiveEngine: VoiceEngine {
return fail("Couldn't use the microphone: \(error.localizedDescription)")
}
}
- if phase == .starting { phase = .listening }
flush()
}
if let content = msg["serverContent"] as? [String: Any] {
@@ -297,6 +296,8 @@ final class GeminiLiveEngine: VoiceEngine {
}
private func captured(_ audio: String, level: Float) {
+ // Connected once the microphone delivers after setup, not when the socket opens.
+ if !ended, ready, phase == .starting { phase = .listening }
guard !ended, !muted, phase.isActive else { return }
self.level = phase == .speaking ? 0 : level
send(["realtimeInput": ["audio": ["data": audio, "mimeType": "audio/pcm;rate=16000"]]])
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
index 096e19dc..01364fcf 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
@@ -140,7 +140,6 @@ final class OpenAIRealtimeEngine: VoiceEngine {
fileprivate func opened() {
guard !ended else { return }
- dropTimer?.cancel()
let tools: [[String: Any]] = CallOperator.tools.map {
["type": "function", "name": $0.name, "description": $0.description, "parameters": $0.parameters]
}
@@ -153,13 +152,16 @@ final class OpenAIRealtimeEngine: VoiceEngine {
"audio": ["input": ["turn_detection": ["type": "semantic_vad"]]],
] as [String: Any],
])
- phase = .listening
- flush()
}
fileprivate func received(_ data: Data) {
guard !ended, let msg = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return }
switch msg["type"] as? String {
+ // Connected once the session has the instructions and tools, not when the channel opens.
+ case "session.updated" where phase == .starting:
+ dropTimer?.cancel()
+ phase = .listening
+ flush()
case "input_audio_buffer.speech_started": level = muted ? 0 : 0.7
case "input_audio_buffer.speech_stopped": level = 0
case "output_audio_buffer.started": phase = .speaking
@@ -183,7 +185,8 @@ final class OpenAIRealtimeEngine: VoiceEngine {
}
fileprivate func iceChanged(_ state: RTCIceConnectionState) {
- guard !ended else { return }
+ // While starting, only a failure counts: the start timer runs until the session is set up.
+ guard !ended, phase != .starting || state == .failed else { return }
switch state {
case .connected, .completed: dropTimer?.cancel()
case .disconnected:
diff --git a/apps/ios/Kit/Sources/CodyncUI/LevelDots.swift b/apps/ios/Kit/Sources/CodyncUI/LevelDots.swift
index 7bc063e6..ad9134af 100644
--- a/apps/ios/Kit/Sources/CodyncUI/LevelDots.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/LevelDots.swift
@@ -48,12 +48,12 @@ struct LevelDots: View {
}
}
-/// Indeterminate progress: a short capsule slides left to right along a faint track, on a loop;
-/// with reduced motion it rests at the left.
+/// Indeterminate progress: a short capsule glides back and forth along a faint track, easing in and out
+/// at both ends; with reduced motion it rests at the left. White in dark mode, black in light.
private struct ConnectingLine: View {
let reduceMotion: Bool
- /// Seconds for the segment to cross the track.
- private static let period = 1.4
+ /// Seconds for one trip there and back.
+ private static let period = 2.4
var body: some View {
TimelineView(.animation(paused: reduceMotion)) { context in
@@ -62,11 +62,10 @@ private struct ConnectingLine: View {
let track = CGRect(x: 0, y: mid - 0.75, width: size.width, height: 1.5)
canvas.fill(Path(roundedRect: track, cornerRadius: 0.75), with: .color(Palette.accent.opacity(0.15)))
let width = size.width / 3
+ // A cosine turns each end smoothly instead of jumping back to the start.
let progress = reduceMotion ? 0
- : context.date.timeIntervalSinceReferenceDate.truncatingRemainder(dividingBy: Self.period) / Self.period
- let segment = CGRect(x: reduceMotion ? 0 : -width + progress * (size.width + width), y: mid - 1.75,
- width: width, height: 3.5)
- canvas.clip(to: Path(CGRect(origin: .zero, size: size)))
+ : (1 - cos(2 * .pi * context.date.timeIntervalSinceReferenceDate / Self.period)) / 2
+ let segment = CGRect(x: progress * (size.width - width), y: mid - 1.75, width: width, height: 3.5)
canvas.fill(Path(roundedRect: segment, cornerRadius: 1.75), with: .color(Palette.accent))
}
}
diff --git a/apps/ios/WhatsNew.md b/apps/ios/WhatsNew.md
index 66b2abb2..7c3b0740 100644
--- a/apps/ios/WhatsNew.md
+++ b/apps/ios/WhatsNew.md
@@ -23,7 +23,7 @@ a version without one gets a generic "Bug fixes and improvements" line. Delete o
- 點對話上方的 Bot 會打開它的詳情、資料庫、記憶與設定;「新會話」改到 Bot 列表的長按選單。
- 用 Gemini 即時語音通話時,Gemini 聽得到你說話並會回答,也會念出 Bot 的回覆;先前通話可能一直沒有聲音、也沒有回應。
- 通話中在設定換了語音(例如從本機換成 Gemini),關掉設定後通話就改用新的語音繼續,不必掛斷重打。
-- 語音通話連線中,通話列會顯示一條跑動的進度線,連上後才換回音量點點。
+- 語音通話連線中,通話列會顯示一條左右來回的線,真正連上、聽得到你說話後才換回音量點點。
### en-US
- Bots can make artifacts: reports, trackers and dashboards you come back to show as a card in the chat. Open one full screen and message the bot below it to change it; it updates in place.
@@ -40,7 +40,7 @@ a version without one gets a generic "Bug fixes and improvements" line. Delete o
- Tapping the bot at the top of its chat opens its Details, Library, Memory and Settings; New session moved to the bot's touch-and-hold menu in the list.
- Gemini realtime voice calls hear you, answer and read the bot's replies aloud; before, a call could stay silent with no reply.
- Pick another voice in the call settings mid-call (on-device to Gemini, say) and the call carries on with it once you close them.
-- While a voice call connects, the call bar shows a moving progress line instead of flat dots.
+- While a voice call connects, the call bar shows a line gliding back and forth until the voice can really hear you.
## 2.24.0
diff --git a/docs/features/voice-call.md b/docs/features/voice-call.md
index c4b7f6fd..7df1fc4c 100644
--- a/docs/features/voice-call.md
+++ b/docs/features/voice-call.md
@@ -14,8 +14,12 @@ On iOS, [Open Live Mode in Shortcuts](ios-shortcuts.md) can select a bot and sta
chat stays readable and usable underneath. Left to right: the bot's avatar (pulses while
speaking; tap it to interrupt), a dotted level line (your voice while listening, a ripple while
the bot speaks, a slow breath while it works; while the engine connects, an indeterminate progress
- line instead: a short segment sliding along a faint track, still at the left under Reduce Motion,
- fading back to the dots once it listens), gear (call settings), mic (mute), red ✕ (end).
+ line instead: a short segment gliding back and forth along a faint track, easing at both ends
+ (white in dark mode, black in light), still at the left under Reduce Motion, fading back to the dots
+ once the call is really connected), gear (call settings), mic (mute), red ✕ (end). Connected means
+ Gemini has answered the setup and the microphone delivers audio; OpenAI has confirmed the
+ instructions and tools (`session.updated`, under the same 15-second start timeout); speech to text
+ hears the microphone. Opening the socket or data channel isn't enough.
- Desktop: the capsule sits 16 CSS pixels below the window top and explicitly excludes itself
from Electron's window-drag region so the gear, mute and end buttons receive mouse clicks.
The iPhone places its capsule below native navigation; the terminal has no call controls.
diff --git a/web/content/docs/voice.mdx b/web/content/docs/voice.mdx
index beac2ca7..a51dc266 100644
--- a/web/content/docs/voice.mdx
+++ b/web/content/docs/voice.mdx
@@ -14,7 +14,7 @@ When the message box is empty and the bot isn't working, the send button becomes
The call bar floats at the top of the chat, and the chat stays readable and usable underneath. From left to right:
- the bot's avatar: it pulses while the bot speaks, and tapping it then interrupts the reply
-- a line of dots that follows your voice while you speak, ripples while the bot speaks, and breathes slowly while it works; while the call connects, a moving progress line takes its place
+- a line of dots that follows your voice while you speak, ripples while the bot speaks, and breathes slowly while it works; while the call connects, a line gliding back and forth takes its place until the voice can really hear you
- the provider's name, whenever audio leaves your device (see "What goes where" below)
- **Call settings** (the gear), **Mute**, and **End call** (the red ✕)