Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

feat: the call's voice hears the bot's work from the computer

leepokai190232c08224
diff --git a/apps/android/WhatsNew.md b/apps/android/WhatsNew.md
index 64648fca..19a21e8d 100644
--- a/apps/android/WhatsNew.md
+++ b/apps/android/WhatsNew.md
@@ -7,6 +7,16 @@ released version's section; a version without one gets the PRs' "What's New: And
else a generic "Bug fixes and improvements" line. Delete old sections.
-->
+## 2.26.0
+
+### zh-Hant
+- Bot 工作時,Gemini 與 OpenAI 的語音會主動告訴你它在做什麼、有沒有等你核准或遇到問題,做完就念出結果。
+- 通話中短暫斷線再連上時,錯過的回覆會補念給你聽。
+
+### en-US
+- While the bot works, the Gemini and OpenAI voice says what it's doing, when an approval waits or something fails, and reads the result.
+- Replies missed while a call reconnects are read once it's back.
+
## 2.25.0
### zh-Hant
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
index 00ecd5d7..e313f86c 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
@@ -182,7 +182,7 @@ class AppStore(application: Application) : AndroidViewModel(application) {
mutableState.update { current -> current.copy(bots = snapshot.bots, entries = snapshot.entries,
pending = snapshot.pending, usage = snapshot.usage, screen = screen, mirrorRevision = snapshot.rev,
selectedBot = current.selectedBot?.takeIf { id -> snapshot.bots.any { it.id == id } }) }
- speakToCall(snapshot.entries, snapshot.bots)
+ if (voiceBinding != null && snapshot.bots.none { it.id == voiceBinding?.botId && !it.deleted }) endVoice("Voice chat ended because its bot was removed.")
if (PushPreferences.tasks(getApplication())) {
val feed = TaskFeed(local.directory)
val watched = withContext(Dispatchers.IO) { feed.all().filter { !it.finished }.map { it.botId }.toSet() }
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreConnection.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreConnection.kt
index 479d74fa..a0ddf056 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreConnection.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreConnection.kt
@@ -185,6 +185,7 @@ private suspend fun AppStore.consumeEvents(active: EncryptedChannel, cursor: Lon
withContext(Dispatchers.IO) { requireNotNull(database).apply(batch) }
refresh()
receiveArtifacts(batch)
+ hearCalls(batch)
acknowledgeVisible(readAfter(batch, state.value.readingChats))
if (resync) throw Resync()
}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
index 2dc2a86b..df5a7d80 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
@@ -14,9 +14,11 @@ import okhttp3.OkHttpClient
// messages, its replies are read aloud, and the call ends as one titled line. The engine is the one picked in the
// call settings; a cloud engine that can't start or drops mid-call hands over to on-device voice.
-/** The call in progress: its engine (replaced on a fallback), start times and which replies it has read. */
-internal class VoiceBinding(val id: String, val botId: String, val generation: Int, val startedAt: Long, val replies: VoiceReplyGate) {
+/** The call in progress: its engine (replaced on a fallback) and start times. */
+internal class VoiceBinding(val id: String, val botId: String, val generation: Int, val startedAt: Long) {
var engine: VoiceEngine? = null
+ /** The newest cue the call heard (`CallCue.seq`), kept across engines: a replayed one isn't said again. */
+ var heard = 0L
/** When the current cloud engine started, for this month's minutes. */
var cloudSince: Long? = null
/** The call settings the current engine was built from. */
@@ -52,9 +54,7 @@ fun AppStore.startVoice(botId: String, expectedGeneration: Int = state.value.gen
internal fun AppStore.attachVoice(id: String, botId: String): VoiceBinding? {
val call = state.value.voice ?: return null
if (call.id != id || call.botId != botId || voiceGeneration != state.value.generation || state.value.loading || voiceBinding != null) return null
- val entries = state.value.entries
- val head = maxOf(state.value.hello?.get("rev")?.jsonPrimitive?.longOrNull ?: 0, entries.maxOfOrNull(Entry::rev) ?: 0)
- val binding = VoiceBinding(id, botId, state.value.generation, SystemClock.elapsedRealtime(), VoiceReplyGate(botId, head, entries))
+ val binding = VoiceBinding(id, botId, state.value.generation, SystemClock.elapsedRealtime())
voiceBinding = binding
begin(binding, voiceChoices().provider)
connect()
@@ -177,12 +177,15 @@ private fun AppStore.sendFromCall(call: VoiceBinding, text: String) {
action("save:" + send.nonce) { savePending(send); deliver(send) }
}
-/** New final replies and approval notices for the call to say. */
-internal fun AppStore.speakToCall(entries: List<Entry>, bots: List<Bot>) {
+/** What the computer tells the call's voice (host chat/calls/announce), each cue once: the engine only says it. */
+internal fun AppStore.hearCalls(events: List<MirrorEvent>) {
val call = voiceBinding ?: return
- if (bots.none { it.id == call.botId && !it.deleted }) { endVoice("Voice chat ended because its bot was removed."); return }
- val engine = call.engine
- for (cue in call.replies.observe(entries, bots)) if (cue.notice) engine?.announce(cue.text) else engine?.speak(cue.text)
+ for (cue in events.filterIsInstance<MirrorEvent.Call>().map { it.cue }) {
+ val engine = call.engine
+ if (cue.call != call.id || cue.seq <= call.heard || engine == null) continue
+ call.heard = cue.seq
+ engine.hear(cue)
+ }
}
fun AppStore.endVoice(error: String? = null) {
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/MirrorDatabase.kt b/apps/android/app/src/main/kotlin/com/codync/android/MirrorDatabase.kt
index ccdaedf8..397c6aad 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/MirrorDatabase.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/MirrorDatabase.kt
@@ -85,7 +85,7 @@ class MirrorDatabase(context: Context, directory: File, computerId: String) :
is MirrorEvent.Usage -> setMeta(db, "usage", event.value.toString())
is MirrorEvent.Screen -> setMeta(db, "screen", event.value.toString())
// Artifacts live in memory (`AppStore.receiveArtifacts`).
- MirrorEvent.Resync, MirrorEvent.Ignored, is MirrorEvent.ArtifactChanged -> Unit
+ MirrorEvent.Resync, MirrorEvent.Ignored, is MirrorEvent.ArtifactChanged, is MirrorEvent.Call -> Unit
}
}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Calls.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Calls.kt
index 721911a6..a58acfd1 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Calls.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Calls.kt
@@ -38,6 +38,30 @@ interface CallLink {
/** How a final reply of the bot reaches the voice. */
fun reply(text: String): String
fun notice(text: String): String = "[Notice] $text"
+ /** What the bot is doing, as the voice's instructions name it. */
+ fun progress(text: String): String = "$PROGRESS_TAG $text"
+}
+
+const val PROGRESS_TAG = "[Progress]"
+
+/**
+ * Something the computer tells a call's voice while the call lasts: the bot's reply, a notice (an approval waiting, a
+ * failure) or what the bot is doing. Ready to say: worded for the call's voice, in a group with who said it. `seq`
+ * rises with each cue of the call: one heard already (replayed after a reconnect) is skipped.
+ */
+data class CallCue(val call: String, val kind: Kind, val text: String, val seq: Long) {
+ enum class Kind { Reply, Notice, Progress }
+
+ companion object {
+ /** A `call` event, or null for one this app can't use (a newer host's new kind is passed over). */
+ fun decode(value: JsonObject): CallCue? {
+ val kind = Kind.entries.firstOrNull { it.name.lowercase() == value["kind"]?.jsonPrimitive?.contentOrNull } ?: return null
+ val call = value["call"]?.jsonPrimitive?.contentOrNull ?: return null
+ val text = value["text"]?.jsonPrimitive?.contentOrNull ?: return null
+ val seq = value["seq"]?.jsonPrimitive?.longOrNull ?: return null
+ return CallCue(call, kind, text, seq)
+ }
+ }
}
/** One function the voice can call, in JSON Schema form; each provider spells it its own way. */
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Compatibility.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Compatibility.kt
index 9c8b1bb9..ee659c61 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Compatibility.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Compatibility.kt
@@ -12,7 +12,7 @@ sealed interface VersionMismatch {
companion object {
/** The oldest host this app works with; raise it only when a core flow needs something older hosts lack. */
- const val MIN_HOST = "2.25.0"
+ const val MIN_HOST = "2.26.0"
/** An unknown version is never a reason to lock someone out. */
fun check(app: String, hostVersion: String?, minApp: String?, minHost: String = MIN_HOST): VersionMismatch? = when {
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Conversation.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Conversation.kt
index b2aac51c..02ed57b7 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Conversation.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Conversation.kt
@@ -151,6 +151,8 @@ sealed interface MirrorEvent {
data class Screen(val value: JsonObject) : MirrorEvent
/** An artifact was published, updated or renamed (the catch-up replays it too), never a checkpoint. */
data class ArtifactChanged(val artifact: Artifact) : MirrorEvent
+ /** For the voice of a call on this device: live only, never a checkpoint. */
+ data class Call(val cue: CallCue) : MirrorEvent
data object Resync : MirrorEvent
data object Ignored : MirrorEvent
@@ -165,6 +167,7 @@ sealed interface MirrorEvent {
"screen" -> Screen(value.getValue("screen").jsonObject)
// One this app can't read is passed over rather than fetching everything again: a card or the Library reads it again.
"artifact" -> runCatching { ArtifactChanged(Artifact.decode(value.getValue("artifact"))) }.getOrDefault(Ignored)
+ "call" -> CallCue.decode(value)?.let(::Call) ?: Ignored
"resync" -> Resync
// Match the existing clients: unknown events do not advance the cursor.
else -> Ignored
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
index d956052c..dde35ff8 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
@@ -51,6 +51,8 @@ class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private
return true
}
override fun announce(notice: String) = hop { pending += link.notice(notice); flush() }
+ /** Only the newest progress waits to be said. */
+ override fun progress(text: String) = hop { pending.removeAll { it.startsWith(PROGRESS_TAG) }; pending += link.progress(text); flush() }
override fun mute(value: Boolean) = hop {
if (value == muted) return@hop
mutable.update { it.copy(muted = value, level = 0f) }
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
index 7268ee20..182bed56 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
@@ -63,6 +63,8 @@ class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, priv
return true
}
override fun announce(notice: String) = hop { pending += link.notice(notice); flush() }
+ /** Only the newest progress waits to be said. */
+ override fun progress(text: String) = hop { pending.removeAll { it.startsWith(PROGRESS_TAG) }; pending += link.progress(text); flush() }
override fun mute(value: Boolean) = hop {
mutable.update { it.copy(muted = value, level = if (value) 0f else it.level) }
if (phase != VoicePhase.Interrupted) peer.mic(!value)
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
index ced8ad5a..a78cf0db 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
@@ -3,8 +3,6 @@ package com.codync.android.core
import kotlinx.coroutines.*
import kotlinx.coroutines.channels.Channel
import kotlinx.coroutines.flow.*
-import kotlinx.serialization.json.booleanOrNull
-import kotlinx.serialization.json.jsonPrimitive
import java.io.IOException
class VoiceUnavailable(message: String) : IOException(message)
@@ -44,32 +42,6 @@ object SpokenText {
}
}
-/** Something a call says: a bot's reply, or a notice from the computer. */
-data class VoiceCue(val text: String, val notice: Boolean = false)
-
-/** Only newly finalized replies are spoken. The mirror remains the history owner. */
-class VoiceReplyGate(private val botId: String, private val startRev: Long, initial: List<Entry>) {
- private var previous = initial.associateBy(Entry::id)
- fun observe(entries: List<Entry>, bots: List<Bot>): List<VoiceCue> {
- val result = mutableListOf<VoiceCue>()
- val group = bots.firstOrNull { it.id == botId }?.kind == "group"
- for (entry in entries) {
- if (entry.botId != botId || entry.rev <= startRev) continue
- val before = previous[entry.id]
- val author = entry.data.text("author")?.let { id -> bots.firstOrNull { it.id == id }?.name }
- if (entry.kind == "agent" && entry.data["final"]?.jsonPrimitive?.booleanOrNull == true &&
- before?.data?.get("final")?.jsonPrimitive?.booleanOrNull != true && entry.text.isNotBlank())
- result.add(VoiceCue(if (group && author != null) "$author: ${entry.text}" else entry.text))
- if (entry.kind == "permission" && entry.status == "pending" && before?.status != "pending") {
- val name = author ?: bots.firstOrNull { it.id == botId && !group }?.name ?: "A bot"
- result.add(VoiceCue("$name needs your approval in the chat.", notice = true))
- }
- }
- previous = entries.associateBy(Entry::id)
- return result
- }
-}
-
enum class VoicePhase { Starting, Listening, Speaking, Interrupted, Failed, Ended }
data class VoiceSettings(val pauseMillis: Long = 1_500, val rate: Float = 1f) {
init { require(pauseMillis in listOf(1_000L, 1_500L, 2_500L)); require(rate in .5f..2f) }
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt
index 7eab2111..9d183ba2 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt
@@ -15,6 +15,17 @@ interface VoiceEngine {
fun speak(markdown: String): Boolean
/** A notice from the computer (an approval is waiting) to say. */
fun announce(notice: String)
+ /** What the bot is doing while it works, from the computer: for a realtime voice to mention (engines that read
+ * replies aloud have nothing to say about it). */
+ fun progress(text: String) {}
+ /** What the computer tells the call's voice, handed to the method that says it. */
+ fun hear(cue: CallCue) {
+ when (cue.kind) {
+ CallCue.Kind.Reply -> speak(cue.text)
+ CallCue.Kind.Notice -> announce(cue.text)
+ CallCue.Kind.Progress -> progress(cue.text)
+ }
+ }
fun mute(value: Boolean)
fun settings(value: VoiceSettings)
/** Android audio focus: false pauses the call's audio, true resumes it. */
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/CompatibilityTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/CompatibilityTest.kt
index 923850c3..3e49d35a 100644
--- a/apps/android/core/src/test/kotlin/com/codync/android/core/CompatibilityTest.kt
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/CompatibilityTest.kt
@@ -7,8 +7,8 @@ class CompatibilityTest {
@Test fun `each side names the oldest version of the other`() {
assertNull(VersionMismatch.check("2.17.5-dev", null, "9.0.0"))
assertEquals(VersionMismatch.UpdateApp("2.18.0"), VersionMismatch.check("2.17.5-dev", "2.18.0", "2.18.0"))
- assertEquals(VersionMismatch.UpdateHost("2.24.0", "2.25.0"), VersionMismatch.check("2.25.0", "2.24.0", "2.3.0"))
- assertNull(VersionMismatch.check("2.25.0", "2.25.0", "2.3.0"))
+ assertEquals(VersionMismatch.UpdateHost("2.25.0", "2.26.0"), VersionMismatch.check("2.26.0", "2.25.0", "2.3.0"))
+ assertNull(VersionMismatch.check("2.26.0", "2.26.0", "2.3.0"))
assertTrue(AppVersion.isBelow("2.9.3", "2.14.1"))
assertFalse(AppVersion.isBelow("garbled", "2.14.1"))
}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
index 1ff46230..e143360f 100644
--- a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
@@ -114,25 +114,10 @@ class VoiceTest {
audio.text("Late recognition", true); runCurrent()
assertFalse(session.speak("A late reply")); assertEquals(1, audio.closed)
}
- @Test fun replyGateIgnoresHistoryOtherBotsAndAlreadyFinalizedReplies() {
- fun entry(id: String, rev: Long, bot: String = "active", final: Boolean = true) = Entry(
- id, rev, bot, rev, "agent", data = buildJsonObject { put("text", id); put("final", final) })
- val old = entry("old", 5); val streaming = entry("streaming", 9, final = false)
- val gate = VoiceReplyGate("active", 10, listOf(old, streaming))
- val values = listOf(entry("old", 20), entry("streaming", 21), entry("reply", 22),
- entry("other", 24, bot = "other"), entry("history", 6))
- val bots = listOf(Bot("active", "Ada"))
- assertEquals(listOf("streaming", "reply"), gate.observe(values, bots).map(VoiceCue::text))
- assertTrue(gate.observe(values, bots).isEmpty())
- val permission = Entry("approval", 25, "active", rev = 25, kind = "permission", data = buildJsonObject { put("status", "pending") })
- assertEquals(listOf(VoiceCue("Ada needs your approval in the chat.", notice = true)), gate.observe(values + permission, bots))
- assertTrue(gate.observe(values + permission, bots).isEmpty())
- }
- @Test fun groupRepliesAndApprovalsNameTheMemberWhoSentThem() {
- val bots = listOf(Bot("team", "Team", kind = "group", members = listOf("ada")), Bot("ada", "Ada"))
- val gate = VoiceReplyGate("team", 0, emptyList())
- val reply = Entry("r", 1, "team", rev = 1, kind = "agent", data = buildJsonObject { put("text", "Done"); put("final", true); put("author", "ada") })
- val card = Entry("p", 2, "team", rev = 2, kind = "permission", data = buildJsonObject { put("status", "pending"); put("author", "ghost") })
- assertEquals(listOf(VoiceCue("Ada: Done"), VoiceCue("A bot needs your approval in the chat.", true)), gate.observe(listOf(reply, card), bots))
+ @Test fun theComputersCuesForACallDecodeByKind() {
+ fun event(kind: String) = buildJsonObject { put("type", "call"); put("call", "c1"); put("kind", kind); put("text", "Ada: Running tests"); put("owner", "phone"); put("seq", 7) }
+ assertEquals(MirrorEvent.Call(CallCue("c1", CallCue.Kind.Progress, "Ada: Running tests", 7)), MirrorEvent.decode(event("progress")))
+ // A newer host's new kind is passed over, never a reason to fetch everything again.
+ assertEquals(MirrorEvent.Ignored, MirrorEvent.decode(event("whisper")))
}
}
diff --git a/apps/desktop/src/renderer/client/host-client.ts b/apps/desktop/src/renderer/client/host-client.ts
index fb1b471b..72c90ba3 100644
--- a/apps/desktop/src/renderer/client/host-client.ts
+++ b/apps/desktop/src/renderer/client/host-client.ts
@@ -11,6 +11,7 @@ import {
type Usage,
} from '@shared/models'
import type { Artifact, ArtifactChunk, LibraryItem } from '@shared/artifacts'
+import { parseCallCue, type CallCue } from '@shared/calls'
/** How a client reaches a host API. Only loopback today: the computer's own host or an SSH tunnel. */
export type HostRoute = 'loopback' | 'direct' | 'relay'
@@ -72,6 +73,8 @@ export type HostEvent =
| { type: 'screen'; screen: ScreenState }
| { type: 'accessRequests'; requests: AccessRequest[] }
| { type: 'cloud'; cloud: CloudStatus }
+ /** For the voice of a call in this app (only the computer's own apps receive this computer's calls' cues). */
+ | { type: 'call'; cue: CallCue }
| { type: 'resync' }
/** An event this app version couldn't read (host newer/older than the app). */
| { type: 'undecodable'; kind: string }
@@ -118,6 +121,10 @@ export function parseEvent(line: string): HostEvent | null {
return { type, requests: (raw.requests as AccessRequest[]) ?? [] }
case 'cloud':
return raw.cloud ? { type, cloud: raw.cloud as CloudStatus } : null
+ case 'call': {
+ const cue = parseCallCue(raw)
+ return cue ? { type, cue } : null
+ }
case 'resync':
return { type }
default:
diff --git a/apps/desktop/src/renderer/store/bot-sync.ts b/apps/desktop/src/renderer/store/bot-sync.ts
index f3640bd8..df5554c9 100644
--- a/apps/desktop/src/renderer/store/bot-sync.ts
+++ b/apps/desktop/src/renderer/store/bot-sync.ts
@@ -3,6 +3,7 @@ import { isOutsideLoadedWindow, snapshotFloors } from '@shared/mirror-window'
import { HostClient, HostError, type HostEvent } from '../client/host-client'
import { BotMirror, type Connection } from './bot-mirror'
import { Artifacts } from './artifacts'
+import type { CallCue } from '@shared/calls'
const DROP_GRACE = 5000
const INITIAL_GRACE = 1000
@@ -28,6 +29,10 @@ function keptEntries(entries: Map<string, Entry[]>, keep: (e: Entry) => boolean)
*/
export class BotSync extends BotMirror {
readonly artifacts = new Artifacts(() => this.changed())
+ /** The calls in this app, by id: each hands what the computer tells its voice to its engine. */
+ readonly callCues = new Map<string, (cue: CallCue) => void>()
+ /** The newest cue each call heard (`CallCue.seq`), kept across engine switches: a replayed one isn't said again. */
+ private readonly heardCues = new Map<string, number>()
private heldDrop: Connection | null = null
private dropTimer: ReturnType<typeof setTimeout> | null = null
private closeEvents: (() => void) | null = null
@@ -233,6 +238,14 @@ export class BotSync extends BotMirror {
case 'cloud':
this.cloud = event.cloud
break
+ case 'call': {
+ const { cue } = event
+ const hear = this.callCues.get(cue.call)
+ if (!hear || cue.seq <= (this.heardCues.get(cue.call) ?? 0)) break
+ this.heardCues.set(cue.call, cue.seq)
+ hear(cue)
+ break
+ }
case 'resync':
this.restartEvents()
break
diff --git a/apps/desktop/src/renderer/views/call/CallView.tsx b/apps/desktop/src/renderer/views/call/CallView.tsx
index bdd9bb6d..0d8646cf 100644
--- a/apps/desktop/src/renderer/views/call/CallView.tsx
+++ b/apps/desktop/src/renderer/views/call/CallView.tsx
@@ -1,4 +1,5 @@
import type { VoiceCallEngine } from '@shared/analytics'
+import type { CallCue } from '@shared/calls'
import { isWorking } from '@shared/models'
import { createContext, useContext, useEffect, useRef, useState } from 'react'
import { BotAvatar } from '../../components/Avatar'
@@ -19,7 +20,6 @@ import { GeminiLiveEngine } from './GeminiLiveEngine'
import { OnDeviceEngine } from './OnDeviceEngine'
import { OpenAIRealtimeEngine } from './OpenAIRealtimeEngine'
import { countVoiceUsage, fromBase64, onDeviceAvailable, onDeviceUnavailableReason, providerName, transcribeOnHost, VoiceEngine, VoiceSettings, voiceSettingsClosed, type VoiceProvider } from './voice'
-import { watchVoiceCall } from './voice-call'
import './call.css'
/**
@@ -114,14 +114,11 @@ export function CallView({ botId, onSpeaking, interrupt, onEnd }: {
}
// Picks up the computer's newest models for the next call; this one starts right away.
if (provider) void rememberStatus()
- let stopWatching: (() => void) | null = null
+ // The computer decides what the voice hears (host chat/calls/announce); the engine only says it.
+ const hear = (cue: CallCue) => next.hear(cue)
next.onActivityChanged = (active) => {
- if (active && !stopWatching) {
- stopWatching = watchVoiceCall(store, botId, { speak: (t) => next.speak(t), announce: (t) => next.announce(t) })
- } else if (!active && stopWatching) {
- stopWatching()
- stopWatching = null
- }
+ if (active) store.callCues.set(call.call, hear)
+ else if (store.callCues.get(call.call) === hear) store.callCues.delete(call.call)
}
// A muted call stays muted on the engine that takes over.
next.muted = engine.current?.muted === true
diff --git a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
index ddd14fc1..a15e57a4 100644
--- a/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
+++ b/apps/desktop/src/renderer/views/call/GeminiLiveEngine.ts
@@ -1,4 +1,4 @@
-import type { CallLink, CallSession } from './call-link'
+import { PROGRESS_TAG, type CallLink, type CallSession } from './call-link'
import { errorMessage, fromBase64, micPCM16k, toBase64, VoiceEngine } from './voice'
const URL_BASE = 'wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained'
@@ -140,6 +140,14 @@ export class GeminiLiveEngine extends VoiceEngine {
this.flush()
}
+ /** Only the newest progress waits to be said. */
+ progress(text: string) {
+ if (this.ended || this.phase === 'failed') return
+ this.pending = this.pending.filter((p) => !p.startsWith(PROGRESS_TAG))
+ this.pending.push(this.link.progress(text))
+ this.flush()
+ }
+
// MARK: socket
private closed(socket: WebSocket, reason: string) {
diff --git a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
index 3289b347..e7c6e21c 100644
--- a/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
+++ b/apps/desktop/src/renderer/views/call/OpenAIRealtimeEngine.ts
@@ -1,4 +1,4 @@
-import type { CallLink, CallSession } from './call-link'
+import { PROGRESS_TAG, type CallLink, type CallSession } from './call-link'
import { errorMessage, microphone, VoiceEngine, VoiceSettings } from './voice'
/**
@@ -129,6 +129,14 @@ export class OpenAIRealtimeEngine extends VoiceEngine {
this.flush()
}
+ /** Only the newest progress waits to be said. */
+ progress(text: string) {
+ if (this.ended || this.phase === 'failed') return
+ this.pending = this.pending.filter((p) => !p.startsWith(PROGRESS_TAG))
+ this.pending.push(this.link.progress(text))
+ this.flush()
+ }
+
// MARK: events
private opened() {
diff --git a/apps/desktop/src/renderer/views/call/call-link.ts b/apps/desktop/src/renderer/views/call/call-link.ts
index c5a60ca5..3919ebc1 100644
--- a/apps/desktop/src/renderer/views/call/call-link.ts
+++ b/apps/desktop/src/renderer/views/call/call-link.ts
@@ -3,6 +3,9 @@ import { sleep } from '../../store/bot-sync'
import type { BotStore } from '../../store/bot-store'
import { spokenText, type VoiceProvider } from './voice'
+/** How the voice's instructions name what the bot is doing. */
+export const PROGRESS_TAG = '[Progress]'
+
/** How long a tool call waits for the user's words that are still being written down. */
const HEARING_PATIENCE_MS = 2_000
@@ -103,6 +106,11 @@ export class CallLink {
return `[Notice] ${text}`
}
+ /** What the bot is doing, as the voice's instructions name it. */
+ progress(text: string) {
+ return `${PROGRESS_TAG} ${text}`
+ }
+
/** The call is over: its line in the chat, with what's left of the transcript. */
end(seconds: number) {
void this.serial(() => this.store.endCall(this.botId, this.call, seconds, this.drain(true)))
diff --git a/apps/desktop/src/renderer/views/call/voice-call.ts b/apps/desktop/src/renderer/views/call/voice-call.ts
deleted file mode 100644
index ba765872..00000000
--- a/apps/desktop/src/renderer/views/call/voice-call.ts
+++ /dev/null
@@ -1,44 +0,0 @@
-import { isGroup, needsInput } from '@shared/models'
-import type { BotStore } from '../../store/bot-store'
-
-/**
- * While a call's audio is active, speaks each new final reply in the bot's main chat and
- * announces when the bot starts waiting for an approval (kit BotStore `beginVoiceCall`). Only
- * entries newer (by rev) than the call's start count. Returns the function that stops it.
- */
-export function watchVoiceCall(store: BotStore, botId: string, handlers: { speak: (text: string) => void; announce: (text: string) => void }) {
- let startRev = 0
- for (const b of store.bots.values()) startRev = Math.max(startRev, b.rev)
- for (const list of store.entries.values()) for (const e of list) startRev = Math.max(startRev, e.rev)
- // Replies already final when the call began are never spoken (a reaction bumps their rev).
- const finals = new Set(store.allEntries(botId).filter((e) => e.data.final).map((e) => e.id))
- let waiting = (() => {
- const bot = store.bots.get(botId)
- return bot ? needsInput(bot) : false
- })()
-
- // A group has no status of its own: its members' approvals arrive as cards.
- const cards = new Set(store.allEntries(botId).filter((e) => e.kind === 'permission').map((e) => e.id))
- const nameOf = (author?: string) => (author ? store.bots.get(author)?.name : undefined)
-
- return store.subscribe(() => {
- const group = store.bots.get(botId)
- const inGroup = !!group && isGroup(group)
- for (const e of store.allEntries(botId)) {
- if (inGroup && e.kind === 'permission' && !cards.has(e.id)) {
- cards.add(e.id)
- if (e.data.status === 'pending' && e.rev > startRev) handlers.announce(`${nameOf(e.data.author) ?? 'A bot'} needs your approval in the chat.`)
- }
- if (!e.data.final || finals.has(e.id)) continue
- finals.add(e.id)
- if (e.kind !== 'agent' || e.rev <= startRev || !e.data.text) continue
- // In a group, a reply is said with its speaker's name.
- const speaker = inGroup ? nameOf(e.data.author) : undefined
- handlers.speak(speaker ? `${speaker}: ${e.data.text}` : e.data.text)
- }
- const bot = store.bots.get(botId)
- const now = bot ? needsInput(bot) : false
- if (bot && now && !waiting) handlers.announce(`${bot.name} needs your approval in the chat.`)
- waiting = now
- })
-}
diff --git a/apps/desktop/src/renderer/views/call/voice.ts b/apps/desktop/src/renderer/views/call/voice.ts
index f2cae576..01cdf2a5 100644
--- a/apps/desktop/src/renderer/views/call/voice.ts
+++ b/apps/desktop/src/renderer/views/call/voice.ts
@@ -1,4 +1,5 @@
import type { HostClient } from '../../client/host-client'
+import type { CallCue } from '@shared/calls'
import { Observable } from '../../lib/observable'
import {
addVoiceUsage,
@@ -128,6 +129,16 @@ export abstract class VoiceEngine extends Observable {
abstract speak(reply: string): void
/** A notice from the computer (an approval is waiting) to say. */
abstract announce(notice: string): void
+ /** What the bot is doing while it works, from the computer: for a realtime voice to mention (engines that read
+ * replies aloud have nothing to say about it). */
+ progress(_text: string) {}
+
+ /** What the computer tells the call's voice, handed to the method that says it. */
+ hear(cue: CallCue) {
+ if (cue.kind === 'reply') this.speak(cue.text)
+ else if (cue.kind === 'notice') this.announce(cue.text)
+ else this.progress(cue.text)
+ }
}
export const errorMessage = (e: unknown) => (e instanceof Error ? e.message : String(e))
diff --git a/apps/desktop/src/shared/calls.test.ts b/apps/desktop/src/shared/calls.test.ts
index 1158e02f..6390a9ff 100644
--- a/apps/desktop/src/shared/calls.test.ts
+++ b/apps/desktop/src/shared/calls.test.ts
@@ -1,6 +1,6 @@
import assert from 'node:assert/strict'
import { test } from 'node:test'
-import { callConversation, hidingCalls, withCallEntries } from './calls.ts'
+import { callConversation, hidingCalls, parseCallCue, withCallEntries } from './calls.ts'
import type { Entry } from './models.ts'
const entry = (id: string, kind: string, at: number, data: Entry['data']): Entry => ({
@@ -46,3 +46,11 @@ test("a call's own entries fill in what the loaded chat doesn't hold", () => {
assert.deepEqual(callConversation(merged, line).map((m) => `${m.speaker}: ${m.text}`), ['user: from before the loaded chat', 'bot: its reply'])
assert.equal(withCallEntries(merged, fetched).length, merged.length, 'each entry once')
})
+
+test("the computer's cues for a call's voice parse by kind; a new kind is passed over", () => {
+ assert.deepEqual(parseCallCue({ type: 'call', call: 'c', kind: 'progress', text: 'Mika: Running tests', owner: 'local', seq: 7 }), {
+ call: 'c', kind: 'progress', text: 'Mika: Running tests', seq: 7,
+ })
+ assert.equal(parseCallCue({ type: 'call', call: 'c', kind: 'whisper', text: '?', seq: 8 }), null)
+ assert.equal(parseCallCue({ type: 'call', kind: 'reply', text: 'no call', seq: 9 }), null)
+})
diff --git a/apps/desktop/src/shared/calls.ts b/apps/desktop/src/shared/calls.ts
index 820fd4b0..9172d720 100644
--- a/apps/desktop/src/shared/calls.ts
+++ b/apps/desktop/src/shared/calls.ts
@@ -12,6 +12,26 @@ export interface CallLine {
id?: string
}
+/** Something the computer tells a call's voice while the call lasts: the bot's reply, a notice (an approval waiting,
+ * a failure) or what the bot is doing. Ready to say: worded for the call's voice, in a group with who said it. */
+export interface CallCue {
+ call: string
+ kind: 'reply' | 'notice' | 'progress'
+ text: string
+ /** Rises with each cue of the call: one heard already (replayed after a reconnect) is skipped. */
+ seq: number
+}
+
+const CUE_KINDS: ReadonlySet<string> = new Set(['reply', 'notice', 'progress'])
+
+/** A `call` event, or null for one this app can't use (a newer host's new kind is passed over). */
+export function parseCallCue(raw: Record<string, unknown>): CallCue | null {
+ const { call, kind, text, seq } = raw
+ return typeof call === 'string' && typeof kind === 'string' && CUE_KINDS.has(kind) && typeof text === 'string' && typeof seq === 'number'
+ ? { call, kind: kind as CallCue['kind'], text, seq }
+ : null
+}
+
/** One message of a call's conversation. */
export interface CallMessage {
speaker: 'user' | 'voice' | 'voiceToBot' | 'bot'
diff --git a/apps/desktop/src/shared/compat.test.ts b/apps/desktop/src/shared/compat.test.ts
index e5b89b45..02850eb8 100644
--- a/apps/desktop/src/shared/compat.test.ts
+++ b/apps/desktop/src/shared/compat.test.ts
@@ -13,6 +13,6 @@ test('versions parse like the host and the iPhone app', () => {
test('the side that must update', () => {
assert.deepEqual(checkVersions('2.6.0', '2.7.0', '2.7.0'), { kind: 'updateApp', minimum: '2.7.0' })
- assert.deepEqual(checkVersions('2.25.0', '2.24.9', null), { kind: 'updateHost', version: '2.24.9', minimum: '2.25.0' })
- assert.equal(checkVersions('2.25.0', '2.25.0', '2.3.0'), null)
+ assert.deepEqual(checkVersions('2.26.0', '2.25.9', null), { kind: 'updateHost', version: '2.25.9', minimum: '2.26.0' })
+ assert.equal(checkVersions('2.26.0', '2.26.0', '2.3.0'), null)
})
diff --git a/apps/desktop/src/shared/compat.ts b/apps/desktop/src/shared/compat.ts
index f06580f3..8b02109e 100644
--- a/apps/desktop/src/shared/compat.ts
+++ b/apps/desktop/src/shared/compat.ts
@@ -6,7 +6,7 @@ export type VersionMismatch =
| { kind: 'updateHost'; version: string; minimum: string }
/** The oldest host this app works with. */
-export const MIN_HOST = '2.25.0'
+export const MIN_HOST = '2.26.0'
/** `major.minor.patch`, ignoring a leading `v` and any pre-release or build suffix. */
export function parseVersion(version: string): number[] | null {
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
index 038234cc..d37c6e48 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
@@ -9,6 +9,8 @@ public enum HostEvent: Sendable {
case screen(ScreenState)
/// An artifact published or renamed (the catch-up replays it too).
case artifact(Artifact)
+ /// For the voice of a call on this device (only the device holding the call receives it).
+ case call(CallCue)
case resync
/// An event this app version couldn't decode (host newer/older than the app).
case undecodable(type: String)
@@ -120,6 +122,8 @@ public struct HostClient: Sendable {
case "artifact":
// Not worth a rewind: a card or the Library reads the artifact again.
return (try? d.decode(ArtifactWrap.self, from: data)).map { .artifact($0.artifact) }
+ case "call":
+ return (try? d.decode(CallCue.self, from: data)).map { .call($0) }
case "resync":
return .resync
default:
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/Calls.swift b/apps/ios/Kit/Sources/CodyncKit/Models/Calls.swift
index 6bcd1c5b..cb96ccb9 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Models/Calls.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/Calls.swift
@@ -23,6 +23,19 @@ public struct CallSession: Sendable {
}
}
+/// Something the computer tells a call's voice while the call lasts: the bot's reply, a notice (an approval
+/// waiting, a failure) or what the bot is doing.
+public struct CallCue: Decodable, Sendable, Equatable {
+ public enum Kind: String, Decodable, Sendable { case reply, notice, progress }
+ /// The call's id.
+ public var call: String
+ public var kind: Kind
+ /// Ready to say: worded for the call's voice, and in a group with who said it.
+ public var text: String
+ /// Rises with each cue of the call: one heard already (replayed after a reconnect) is skipped.
+ public var seq: Int64
+}
+
/// One message of a call's conversation.
public struct CallMessage: Hashable, Sendable {
public enum Speaker: Sendable { case user, voice, voiceToBot, bot }
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/Compatibility.swift b/apps/ios/Kit/Sources/CodyncKit/Models/Compatibility.swift
index 6d5d6a10..c319bf93 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Models/Compatibility.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/Compatibility.swift
@@ -11,7 +11,7 @@ public enum VersionMismatch: Equatable, Sendable {
/// The oldest host this app works with. Raise it only when the app starts depending on
/// something older hosts don't have (an optional feature hides its control instead).
- public static let minHost = "2.25.0"
+ public static let minHost = "2.26.0"
/// Checks this app (`app`) against a host's version and `minApp` (nil from hosts that predate it).
public static func check(app: String, minHost: String = minHost, hostVersion: String, minApp: String?) -> VersionMismatch? {
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CallLink.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CallLink.swift
index bf7102ca..92a9ddd8 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/CallLink.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CallLink.swift
@@ -95,6 +95,11 @@ final class CallLink {
func notice(_ text: String) -> String { "[Notice] \(text)" }
+ static let progressTag = "[Progress]"
+
+ /// What the bot is doing, as the voice's instructions name it.
+ func progress(_ text: String) -> String { "\(Self.progressTag) \(text)" }
+
/// The call is over: its line in the chat, with what's left of the transcript.
func end(seconds: Int) {
serial { [self] in await model.endCall(botId, seconds: seconds, call: call, lines: drain(withOpen: true)) }
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift b/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift
index 9d13a725..dc6d5c25 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/CallView.swift
@@ -113,12 +113,9 @@ struct CallView: View {
engine = OnDeviceEngine { [model, botId, call] in model.send($0, to: botId, call: call) }
Task { _ = try? await link.start("device") }
}
- engine.onActivityChanged = { [weak engine, model, botId, callID] active in
+ engine.onActivityChanged = { [weak engine, model, callID] active in
if active {
- model.beginVoiceCall(callID, botId: botId,
- speak: { [weak engine] in engine?.speak(reply: $0) },
- announce: { [weak engine] in engine?.announce($0) },
- end: { [weak engine] in engine?.end() })
+ model.beginVoiceCall(callID, hear: { [weak engine] in engine?.hear($0) }, end: { [weak engine] in engine?.end() })
} else {
model.endVoiceCall(callID)
}
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
index 944419bc..edc86c49 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/GeminiLiveEngine.swift
@@ -153,6 +153,14 @@ final class GeminiLiveEngine: VoiceEngine {
flush()
}
+ /// Only the newest progress waits to be said.
+ func progress(_ text: String) {
+ guard !ended, !phase.isFailed else { return }
+ pending.removeAll { $0.hasPrefix(CallLink.progressTag) }
+ pending.append(link.progress(text))
+ flush()
+ }
+
// MARK: socket
private func receive(_ socket: URLSessionWebSocketTask) {
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
index adfe60fb..6feb9f58 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/OpenAIRealtimeEngine.swift
@@ -143,6 +143,14 @@ final class OpenAIRealtimeEngine: VoiceEngine {
flush()
}
+ /// Only the newest progress waits to be said.
+ func progress(_ text: String) {
+ guard !ended, !phase.isFailed else { return }
+ pending.removeAll { $0.hasPrefix(CallLink.progressTag) }
+ pending.append(link.progress(text))
+ flush()
+ }
+
// MARK: events
fileprivate func opened() {
diff --git a/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift b/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift
index 11d3096c..c72ffca6 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Call/VoiceEngine.swift
@@ -32,9 +32,23 @@ protocol VoiceEngine: AnyObject, Observable {
func speak(reply: String)
/// A notice from the computer (an approval is waiting) to say.
func announce(_ notice: String)
+ /// What the bot is doing while it works, from the computer: for a realtime voice to mention.
+ func progress(_ text: String)
}
extension VoiceEngine {
+ /// What the computer tells the call's voice, handed to the method that says it.
+ func hear(_ cue: CallCue) {
+ switch cue.kind {
+ case .reply: speak(reply: cue.text)
+ case .notice: announce(cue.text)
+ case .progress: progress(cue.text)
+ }
+ }
+
+ /// Engines that read replies aloud have nothing to say about progress.
+ func progress(_ text: String) {}
+
/// The engine's name in analytics.
var analyticsName: AnalyticsEvent.VoiceEngine {
if self is CloudSpeechEngine { return .cloud }
diff --git a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Connection.swift b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Connection.swift
index abc8c41a..e1110563 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Connection.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Connection.swift
@@ -46,10 +46,9 @@ extension BotStore {
}
/// Audio owns this subscription, so background calls don't depend on SwiftUI updates.
- func beginVoiceCall(_ id: UUID, botId: String, speak: @escaping @MainActor (String) -> Void,
- announce: (@MainActor (String) -> Void)? = nil, end: @escaping @MainActor () -> Void) {
+ func beginVoiceCall(_ id: UUID, hear: @escaping @MainActor (CallCue) -> Void, end: @escaping @MainActor () -> Void) {
guard !retired, voiceCalls[id] == nil else { return }
- voiceCalls[id] = VoiceCall(botId: botId, startRev: rev, speak: speak, announce: announce ?? speak, end: end)
+ voiceCalls[id] = VoiceCall(hear: hear, end: end)
if streamTask == nil { restartStream() }
}
diff --git a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Sync.swift b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Sync.swift
index a4773e66..ff4cb403 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Sync.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore+Sync.swift
@@ -92,17 +92,11 @@ extension BotStore {
catchUpRev = hostRev
bump(rev)
case let .bot(bot):
- let neededInput = bots[bot.id]?.needsInput == true
bots[bot.id] = bot
bump(bot.rev)
onBotUpdated?(bot)
onRosterChanged?()
if bot.unread > 0 { acknowledgeVisibleConversations(bot.id) }
- if bot.needsInput && !neededInput {
- for call in Array(voiceCalls.values) where call.botId == bot.id {
- call.announce("\(bot.name) needs your approval in the chat.")
- }
- }
case let .botDeleted(id, r):
removeComposerDrafts(for: id)
bots[id] = nil
@@ -116,20 +110,15 @@ extension BotStore {
bump(e.rev)
break
}
- let known = entries[e.botId]?.first(where: { $0.id == e.id })
upsert(e)
if let card = e.data.artifact { artifacts.announce(card) }
bump(e.rev)
acknowledgeVisibleConversations(e.botId, entry: e)
- let calls = Array(voiceCalls.values).filter { $0.botId == e.botId && e.rev > $0.startRev }
- if e.kind == "agent", e.data.final == true, known?.data.final != true, let text = e.data.text {
- calls.forEach { $0.speak(spokenReply(e, text)) }
- }
- // A group has no status of its own: its members' approvals arrive as cards.
- if e.kind == "permission", known == nil, e.data.status == "pending", bots[e.botId]?.isGroup == true {
- let name = e.data.author.flatMap { bots[$0]?.name } ?? "A bot"
- calls.forEach { $0.announce("\(name) needs your approval in the chat.") }
- }
+ case let .call(cue):
+ // The computer decides what the voice hears; this device hands each cue over once.
+ guard let id = UUID(uuidString: cue.call), let call = voiceCalls[id], cue.seq > heardCues[id, default: 0] else { break }
+ heardCues[id] = cue.seq
+ call.hear(cue)
case let .usage(u):
setUsage(u)
case let .screen(s):
@@ -154,12 +143,6 @@ extension BotStore {
scheduleSave()
}
- /// In a group, a reply is said with its speaker's name.
- private func spokenReply(_ e: Entry, _ text: String) -> String {
- guard bots[e.botId]?.isGroup == true, let name = e.data.author.flatMap({ bots[$0]?.name }) else { return text }
- return "\(name): \(text)"
- }
-
private func resetMirror() {
composerDrafts = [:]
persistComposerDrafts()
diff --git a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore.swift b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore.swift
index f357626b..01faaa70 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Store/BotStore.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Store/BotStore.swift
@@ -109,15 +109,14 @@ public final class BotStore {
var streamTask: Task<Void, Never>?
var eventsTask: Task<Void, Never>?
var isActive = true
+ /// A call's audio: what the computer tells its voice (`CallCue`), and its end.
struct VoiceCall {
- let botId: String
- let startRev: Int64
- let speak: @MainActor (String) -> Void
- /// A notice from the computer (approval needed), as opposed to the bot's reply.
- let announce: @MainActor (String) -> Void
+ let hear: @MainActor (CallCue) -> Void
let end: @MainActor () -> Void
}
@ObservationIgnored var voiceCalls: [UUID: VoiceCall] = [:]
+ /// The newest cue each call heard (`CallCue.seq`), kept across engine switches: a replayed one isn't said again.
+ @ObservationIgnored var heardCues: [UUID: Int64] = [:]
@ObservationIgnored var shortcutSends = 0
var saveTask: Task<Void, Never>?
var rewound = false
diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
index d9db3d79..e0451f04 100644
--- a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
@@ -167,7 +167,7 @@ private func botEvent(_ id: String, name: String, rev: Int) -> String {
let call = UUID()
var spoken: [String] = []
- store.beginVoiceCall(call, botId: "b1", speak: { spoken.append($0) }, end: {})
+ store.beginVoiceCall(call, hear: { spoken.append($0.text) }, end: {})
store.setActive(false)
// Allow shutdown to run if backgrounding incorrectly scheduled one.
try await Task.sleep(for: .milliseconds(30))
@@ -176,13 +176,10 @@ private func botEvent(_ id: String, name: String, rev: Int) -> String {
#expect(await fake.sent == ["spoken while backgrounded"])
#expect(await fake.shutdownCount == 0)
- let reply = #"{"type":"entry","entry":{"id":"reply","seq":2,"botId":"b1","rev":2,"kind":"agent","turn":1,"data":{"text":"The answer","final":true},"createdAt":1,"updatedAt":1}}"#
- await fake.emit(reply)
+ // The computer tells the call what to say; another call's cue isn't this one's.
+ await fake.emit(#"{"type":"call","call":"\#(UUID().uuidString)","kind":"reply","text":"Another call","seq":1}"#)
+ await fake.emit(#"{"type":"call","call":"\#(call.uuidString)","kind":"reply","text":"The answer","seq":1}"#)
#expect(await until { spoken == ["The answer"] })
- await fake.emit(reply) // Event replays must not read the same reply twice.
- await fake.emit(#"{"type":"entry","entry":{"id":"other","seq":4,"botId":"b2","rev":4,"kind":"agent","turn":1,"data":{"text":"Other bot","final":true},"createdAt":1,"updatedAt":1}}"#)
- #expect(await until { store.chat("b2").last?.id == "other" })
- #expect(spoken == ["The answer"])
store.setActive(true)
#expect(await fake.shutdownCount == 0)
@@ -203,8 +200,8 @@ private func botEvent(_ id: String, name: String, rev: Int) -> String {
await fake.emit(botEvent("b1", name: "Bot", rev: 1))
#expect(await until { store.connection == .online })
let first = UUID(), second = UUID()
- store.beginVoiceCall(first, botId: "b1", speak: { _ in }, end: {})
- store.beginVoiceCall(second, botId: "b1", speak: { _ in }, end: {})
+ store.beginVoiceCall(first, hear: { _ in }, end: {})
+ store.beginVoiceCall(second, hear: { _ in }, end: {})
store.setActive(false)
store.endVoiceCall(first)
try await Task.sleep(for: .milliseconds(30))
@@ -224,13 +221,13 @@ private func botEvent(_ id: String, name: String, rev: Int) -> String {
#expect(await until { store.connection == .online })
let call = UUID()
var ended = false
- store.beginVoiceCall(call, botId: "b1", speak: { _ in Issue.record("Retired call received speech") }, end: { ended = true })
+ store.beginVoiceCall(call, hear: { _ in Issue.record("Retired call received speech") }, end: { ended = true })
store.setActive(false)
store.retire()
#expect(ended)
#expect(await until { await fake.shutdownCount == 1 })
store.send("must not send", to: "b1")
- store.beginVoiceCall(UUID(), botId: "b1", speak: { _ in }, end: {})
+ store.beginVoiceCall(UUID(), hear: { _ in }, end: {})
#expect(store.chat("b1").isEmpty)
#expect(await fake.sent.isEmpty)
}
@@ -563,8 +560,9 @@ extension FakeRemote {
#expect(await fake.calls.filter { $0 == "respondPermission" }.count == 1)
}
-/// Approvals reach a call as notices, replies as replies (a realtime operator words them differently).
-@MainActor @Test func voiceCallGetsApprovalsAsNotices() async throws {
+/// The computer's cues reach a call once each: one replayed after a reconnect isn't said again, even after an engine
+/// switch.
+@MainActor @Test func voiceCallHearsEachOfTheComputersCuesOnce() async throws {
let (storage, suite) = context()
defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
let fake = FakeRemote(.ready(.direct))
@@ -574,12 +572,27 @@ extension FakeRemote {
#expect(await until { await fake.subscribed })
await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Ada","rev":1,"lastAt":1}}"#)
#expect(await until { store.connection == .online })
- var replies: [String] = []
- var notices: [String] = []
- store.beginVoiceCall(UUID(), botId: "b1", speak: { replies.append($0) }, announce: { notices.append($0) }, end: {})
- await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Ada","rev":2,"lastAt":2,"status":"needsInput"}}"#)
- #expect(await until { notices == ["Ada needs your approval in the chat."] })
- #expect(replies.isEmpty)
+ var heard: [CallCue] = []
+ let call = UUID()
+ store.beginVoiceCall(call, hear: { heard.append($0) }, end: {})
+ let cues = [("notice", "Ada needs the user's approval: Edit a.txt", 1), ("progress", "Ada: Running tests", 2), ("reply", "Done.", 3)]
+ func emit(_ kind: String, _ text: String, _ seq: Int) async {
+ await fake.emit(#"{"type":"call","call":"\#(call.uuidString)","kind":"\#(kind)","text":"\#(text)","seq":\#(seq)}"#)
+ }
+ for (kind, text, seq) in cues { await emit(kind, text, seq) }
+ #expect(await until { heard.count == 3 })
+ #expect(heard.map(\.kind) == [.notice, .progress, .reply])
+ // The engine switches; the reconnect's catch-up replays what the computer kept.
+ store.endVoiceCall(call)
+ store.beginVoiceCall(call, hear: { heard.append($0) }, end: {})
+ await emit("reply", "Done.", 3)
+ await emit("reply", "Missed while away.", 4)
+ #expect(await until { heard.last?.text == "Missed while away." })
+ #expect(heard.map(\.text) == cues.map(\.1) + ["Missed while away."])
+ // A newer host's new cue kind is passed over, not a reason to rewind.
+ await fake.emit(#"{"type":"call","call":"\#(call.uuidString)","kind":"whisper","text":"?","seq":5}"#)
+ try await Task.sleep(for: .milliseconds(30))
+ #expect(store.unreadable == false)
}
// MARK: - Version compatibility (docs/reference/compatibility.md)
diff --git a/apps/ios/WhatsNew.md b/apps/ios/WhatsNew.md
index 3fb12942..6e92493b 100644
--- a/apps/ios/WhatsNew.md
+++ b/apps/ios/WhatsNew.md
@@ -6,6 +6,16 @@ any left). tools/asc-submit.py reads the released version's section;
a version without one gets a generic "Bug fixes and improvements" line. Delete old sections.
-->
+## 2.26.0
+
+### zh-Hant
+- Bot 工作時,Gemini 與 OpenAI 的語音會主動告訴你它在做什麼、有沒有等你核准或遇到問題,做完就念出結果。
+- 通話中短暫斷線再連上時,錯過的回覆會補念給你聽。
+
+### en-US
+- While the bot works, the Gemini and OpenAI voice keeps you posted on what it's doing, tells you when an approval waits or something fails, and reads the result when it's done.
+- A reply that arrives while a call briefly loses its connection is read once it's back.
+
## 2.25.0
### zh-Hant
diff --git a/docs/architecture/file-structure.md b/docs/architecture/file-structure.md
index c1adbb55..59bdb751 100644
--- a/docs/architecture/file-structure.md
+++ b/docs/architecture/file-structure.md
@@ -91,7 +91,7 @@ Codync/
| Agent process, ACP, queue and session lifecycle | `host/src/agent/bot/` (`launch.rs`, `queue.rs`, `session.rs`, `turn.rs`, `updates.rs`), `acp.rs` |
| Compaction timing and rollover | `host/src/agent/bot/compaction.rs`, the handoff card in `host/src/chat/handoff.rs`; the rules in `codync-mem`'s `compaction.rs` |
| Group room turns / bot-to-bot requests | `host/src/chat/group.rs` / `team.rs` |
-| Voice calls (the call, the voice's instructions and tools, transcripts, call lines) | `host/src/chat/calls/` (`mod.rs`, `context.rs`, `project.rs` the git state, `tools.rs`); the user's messages it relays go through `host/src/chat/messages.rs` like typed ones; clients' `CallLink` (iPhone `CodyncUI/Call/CallLink.swift`, desktop `views/call/call-link.ts`, Android `core/Calls.kt` + `HostCallLink.kt`) and call rows (`CallRows`) |
+| Voice calls (the call, the voice's instructions and tools, transcripts, call lines) | `host/src/chat/calls/` (`mod.rs`, `context.rs`, `project.rs` the git state, `tools.rs`, `announce.rs` what the voice hears while the call lasts, `speech.rs` text as a voice says it); the user's messages it relays go through `host/src/chat/messages.rs` like typed ones; clients' `CallLink` (iPhone `CodyncUI/Call/CallLink.swift`, desktop `views/call/call-link.ts`, Android `core/Calls.kt` + `HostCallLink.kt`) and call rows (`CallRows`) |
| Team management (bots creating and changing bots) | `host/src/chat/manage.rs`, its approval card (`Cmd::Approve`) in `agent/bot/updates.rs` |
| Prompt snapshots and memory | `host/src/chat/context.rs`, `memory/` (`api.rs` the apps' read-only view, `tools.rs` and `file_tools.rs` the MCP tools, `files.rs` the folders and scan, `turns.rs` which bot's turn a found change is credited to, `file_api.rs`, `dream.rs`, `import.rs`); the rules in `host/crates/memory` (`codync-mem`: files and facts in `files/`, dreaming in `dream/`, tombstones in `store.rs`, the old entries in `legacy.rs`); harnesses' own memory turned off in `host/src/agent/own_memory.rs` |
| Identity, encryption and direct channel | `host/src/remote/identity.rs`, `crypto.rs`, `channel.rs` |
diff --git a/docs/features/voice-call.md b/docs/features/voice-call.md
index ca03f5b2..ffaa7ba8 100644
--- a/docs/features/voice-call.md
+++ b/docs/features/voice-call.md
@@ -46,9 +46,8 @@ way open with a sentence or two, read aloud like any reply and part of the call
foreground/background transitions. Ending or failing the last call releases that
connection when backgrounded, restoring normal push delivery. Removing the computer
or switching accounts ends its calls and closes the transport.
-- Final replies and approval notices are delivered from the store's incoming events
- directly to the audio session, rather than waiting for a SwiftUI view update.
- Replayed final replies and other bots' replies are not read into this call.
+- What to say arrives from the computer as the call's `call` events (see "How the voice talks"), delivered from
+ the store's incoming events directly to the audio session, rather than waiting for a SwiftUI view update.
## Dictation
@@ -305,9 +304,27 @@ carries audio and talks to these methods:
### How the voice talks
-- Each new final reply is added to the realtime conversation as `[<bot> replied] …` and an approval
- as `[Notice] …` (`BotStore` hands them over as `speak` / `announce`); the model says them,
- shortened for speech. They wait while the model is talking, so they don't cut it off.
+- The host decides what the voice hears while the call lasts (`chat::calls::announce`, one watcher per call): each
+ new final reply in the bot's chat (one already final when the call began isn't; one that began before it and
+ finishes during it is), an approval card waiting, a failure notice, and the bot's activity line while it works:
+ at most one progress line every 15 s, only when it changed, and "<bot> is still working" after 45 s with nothing
+ said. It sends each as a `call` event `{call, owner, kind: reply | notice | progress, text, seq}`, worded for the
+ call's mode as `startCall` last gave it: a realtime voice gets replies shortened for speech, approvals with their
+ options ("Mika needs the user's approval: … (options: …)") and progress; an on-device or speech-to-text call gets
+ replies as written (its engine shortens them), approvals for the listener ("Mika needs your approval in the chat:
+ …") and no progress. In a group, replies and notices name the member. `api::events` delivers each cue only to the
+ device that holds the call (`owner`, its device key, set from the caller on `startCall`); older apps ignore the
+ event type.
+- Each cue's `seq` rises through the call (from the clock, so across host restarts too). The host keeps the call's
+ last 10 replies and notices and adds them to that device's catch-up when its event stream reconnects; the app
+ skips any `seq` it already heard (kept per call, across engine switches). A watcher that missed hub events
+ rereads the chat's latest entries.
+- The watcher starts with `startCall` and runs until `endCall`. As the host starts it watches again every call
+ started in the last day (`rewatch`). A call none of its app's requests (`startCall`, `callLines`, `callTool`, a
+ message said on it) touched for 2 hours stops being watched; the next one starts it again.
+- The app hands each cue to its engine (`hear`): realtime voices get `[<bot> replied] …`, `[Notice] …` and
+ `[Progress] …` (only the newest progress waits); on-device and speech-to-text calls read replies and notices
+ aloud. They wait while the model is talking, so they don't cut it off.
- Barge-in is the provider's voice activity detection (OpenAI semantic VAD; Gemini's automatic
VAD, with the voice-processing input cancelling the device's own playback). Tapping the avatar
cancels the response locally.
@@ -346,8 +363,14 @@ A route change mid-call (AirPods) is not checked yet.
Host tests (`chat::calls`) cover a call's flow (start, lines, `send_to_bot` with the user's words, the line at the
end), the voice's instructions (the relay role and the bot's name first, each section present and within its cap,
-OpenAI's smaller share), transcript checks and dreaming over a call. `StoreTests` cover approvals reaching a call
-as notices; each client tests hiding a call's messages and merging its conversation. On Android, `VoiceEnginesTest` (core, JVM) drives
+OpenAI's smaller share), transcript checks and dreaming over a call. `chat::calls::announce` tests cover what the
+voice hears: new replies, approvals and failures said once, a reply unfinished when the call began, a group's
+member named, the wording per call mode, progress rate-limited with a "still working" after a silence, and the
+replies and notices kept for the device holding the call; `api::events` tests check that only that device gets
+them, and `calls_e2e` that an on-device call hears its approval and reply. `StoreTests` (iPhone) hand each cue to
+the call once, also when a reconnect replays it after an engine switch; the desktop tests parse cues
+(`parseCallCue`) and Android's decode them (`CallCue`). Each client tests hiding a call's messages and merging its
+conversation. On Android, `VoiceEnginesTest` (core, JVM) drives
each engine through fake audio: utterance cutting and clip playback, OpenAI's event sequence and SDP
exchange (MockWebServer), and a Gemini Live session over a real WebSocket (setup, audio both ways,
a tool call, a close with the provider's reason). Provider audio isn't
diff --git a/docs/plans/voice-architecture.md b/docs/plans/voice-architecture.md
index 53aab2f6..a51664bb 100644
--- a/docs/plans/voice-architecture.md
+++ b/docs/plans/voice-architecture.md
@@ -1,6 +1,6 @@
# Voice calls owned by the host
-Status: stage 1 shipped (2.25.0); stages 2–4 planned. Supersedes [voice call transcripts](voice-call-transcripts.md),
+Status: stage 1 shipped (2.25.0), stage 2 (2.26.0); stages 3–4 planned. Supersedes [voice call transcripts](voice-call-transcripts.md),
which became stage 1 of this plan. Current behavior: [voice call](../features/voice-call.md#the-call-on-the-host).
A realtime voice model (Gemini Live, OpenAI Realtime) stays in front of the bot: coding agents are text-only and
@@ -53,7 +53,7 @@ Each stage ships on its own (after-change, merged into `dev`), all clients toget
- **Memory:** dreaming reads the transcript; the user's spoken lines are the user's own words, the voice's lines and
the voice-written `send_to_bot` text are context only. A call's user lines count toward the dream trigger.
-### 2. The bot's work reaches the call
+### 2. The bot's work reaches the call (shipped)
- The host pushes what the voice should say to the client holding the call (a `call` event, which older clients
ignore): the bot's final replies, approvals waiting, failures, and short progress while it works ("running the
diff --git a/docs/reference/compatibility.md b/docs/reference/compatibility.md
index aa8486c8..01e5a52b 100644
--- a/docs/reference/compatibility.md
+++ b/docs/reference/compatibility.md
@@ -141,6 +141,7 @@ Update this table in the release that changes a floor, with the reason.
| Release | Host `minApp` | Client `minHost` | Why |
|---|---|---|---|
+| 2.26.0 | 2.21.0 | 2.26.0 (iPhone, Android, desktop) | What a call's voice hears comes from the host as `call` events (replies, approvals, failures, progress); the apps no longer derive it from the chat, so on an older host a call would never hear the bot. |
| 2.25.0 | 2.21.0 | 2.25.0 (iPhone, Android, desktop) | Voice calls run on the host: the apps start a call, run the voice's tools and end it through `startCall`, `callTool`, `callLines` and `endCall`, which older hosts lack (a Linux desktop or an SSH computer can run one). `logCall` stays for older apps. |
| 2.21.0 | 2.21.0 | 2.20.1 | Memory is files the bot keeps, and the apps only read it: the edit methods (`saveMemory`, `pinMemory`, `archiveMemory`, `forgetMemory`, `clearMemory`, `undoDream`, `answerMemory`, `undoMemory`, `saveMemoryFile`, `deleteMemoryFile`, `restoreMemoryFile`) are gone, so older apps' memory buttons would fail. `memory` keeps its shape, so new apps still read an older host's memory. |
| 2.20.1 | 2.20.1 | 2.20.1 | Reply threads gave way to quote replies: the `thread` method, `send.threadId`, `markRead.threadId`, entry `threadId`, `data.thread` and bot `workingThread` are gone, and old thread replies sit in the main chat. Older apps would open empty threads. The clients no longer leave thread replies out of the chat or read threads, so an older host would show them inline, routine results twice and thread unread counts that never clear. |
diff --git a/host/src/api/bots.rs b/host/src/api/bots.rs
index 2a311ff2..97cdc3bc 100644
--- a/host/src/api/bots.rs
+++ b/host/src/api/bots.rs
@@ -1,5 +1,6 @@
//! Bot and chat methods: roster, messages, uploads, memory, artifacts, routines and the bots' own MCP calls.
+use super::devices::Caller;
use super::str_arg;
use crate::agent::bot::Cmd;
use crate::hub::Hub;
@@ -9,7 +10,7 @@ use serde_json::{Value, json};
use std::ops::ControlFlow;
use std::sync::Arc;
-pub(super) async fn call(hub: &Arc<Hub>, method: &str, b: Value) -> Result<ControlFlow<Value, Value>> {
+pub(super) async fn call(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value) -> Result<ControlFlow<Value, Value>> {
Ok(ControlFlow::Break(match method {
"routineSchedule" => crate::routines::schedule_preview(&b, crate::store::now_ms())?,
"routines" => hub.routines.list(str_arg(&b, "botId")?),
@@ -168,7 +169,7 @@ pub(super) async fn call(hub: &Arc<Hub>, method: &str, b: Value) -> Result<Contr
// Said on a voice call (on-device or speech to text): the apps show it in the call's conversation.
if let Some(call) = b["call"].as_str() {
data["call"] = crate::chat::calls::checked(call)?.into();
- crate::chat::calls::heard(&hub.store, bot, call)?;
+ crate::chat::calls::heard(hub, bot, call, caller.device_key())?;
}
json!({"entry": crate::chat::messages::post(hub, &row.config, data)?})
}
@@ -187,8 +188,9 @@ pub(super) async fn call(hub: &Arc<Hub>, method: &str, b: Value) -> Result<Contr
let e = hub.add_entry(&Lane::main(bot), EntryKind::Notice, hub.store.max_turn(bot), &data);
json!({"entry": e})
}
- // A voice call, owned by the host (`chat::calls`).
- "startCall" => crate::chat::calls::start(hub, &b).await?,
+ // A voice call, owned by the host (`chat::calls`); only the device that starts it hears what its voice should
+ // say.
+ "startCall" => crate::chat::calls::start(hub, &b, caller.device_key()).await?,
// `lines`: what was said on the call that the app hadn't sent yet, which the tool may hand on.
"callTool" => {
let call = crate::chat::calls::checked(str_arg(&b, "callId")?)?;
@@ -196,11 +198,7 @@ pub(super) async fn call(hub: &Arc<Hub>, method: &str, b: Value) -> Result<Contr
json!({"result": result.await})
}
"callLines" => {
- crate::chat::calls::add_lines(
- &hub.store,
- crate::chat::calls::checked(str_arg(&b, "callId")?)?,
- &b["lines"],
- )?;
+ crate::chat::calls::add_lines(hub, crate::chat::calls::checked(str_arg(&b, "callId")?)?, &b["lines"])?;
json!({})
}
// What a call carried, for its conversation: the chat an app holds may not reach back that far.
diff --git a/host/src/api/events.rs b/host/src/api/events.rs
index fbff24fc..b3f2e932 100644
--- a/host/src/api/events.rs
+++ b/host/src/api/events.rs
@@ -77,14 +77,20 @@ pub fn events_stream(
"screen": hub.screen.state(),
});
catch_up.insert(0, hello);
+ // What the voice of a call this device holds should have heard while it was away.
+ catch_up.extend(crate::chat::calls::replay(caller.device_key()));
let local = matches!(caller, Caller::Local);
+ let key = caller.device_key().to_owned();
let guard = Arc::new(matches!(client, Some("ios" | "android")).then(|| PhoneClientGuard::new(hub.clone())));
let tail = live.filter_map(move |msg| {
let _keep = guard.clone();
+ let key = key.clone();
async move {
match msg {
Ok(v) if !local && v["type"].as_str().is_some_and(|t| LOCAL_EVENTS.contains(&t)) => None,
+ // What a call's voice should say goes to the device holding that call only.
+ Ok(v) if v["type"] == "call" && v["owner"] != key.as_str() => None,
Ok(v) => Some(v),
// Lagged: tell the client to resync from its last rev.
Err(_) => Some(json!({"type": "resync"})),
diff --git a/host/src/api/events/tests.rs b/host/src/api/events/tests.rs
index 107bb2d0..301eee8d 100644
--- a/host/src/api/events/tests.rs
+++ b/host/src/api/events/tests.rs
@@ -157,3 +157,22 @@ async fn renaming_an_artifact_keeps_the_reactions_its_card_gets_meanwhile() {
assert_eq!(reactions, all, "no rename wrote over a reaction");
assert_eq!(card.data["artifact"]["title"], "Title 39");
}
+
+#[tokio::test]
+async fn what_a_calls_voice_should_say_reaches_only_the_device_holding_it() {
+ let f = Fixture::new();
+ let phone = Caller::Device { key: "phone".into(), scopes: vec![] };
+ let other = Caller::Device { key: "tablet".into(), scopes: vec![] };
+ let streams = [&phone, &other, &Caller::Local].map(|c| Box::pin(events_stream(&f.hub, 0, None, c).unwrap()));
+ let _ =
+ f.hub.events.send(json!({"type": "call", "call": "c1", "owner": "phone", "kind": "reply", "text": "Done."}));
+ let mut heard = Vec::new();
+ for mut stream in streams {
+ let mut got = false;
+ while let Ok(Some(event)) = tokio::time::timeout(std::time::Duration::from_millis(50), stream.next()).await {
+ got |= event["type"] == "call";
+ }
+ heard.push(got);
+ }
+ assert_eq!(heard, [true, false, false]);
+}
diff --git a/host/src/api/mod.rs b/host/src/api/mod.rs
index c65e5cb0..1544c213 100644
--- a/host/src/api/mod.rs
+++ b/host/src/api/mod.rs
@@ -91,7 +91,7 @@ async fn route(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value) -> Resul
ControlFlow::Break(v) => return Ok(v),
ControlFlow::Continue(b) => b,
};
- let b = match bots::call(hub, method, b).await? {
+ let b = match bots::call(hub, caller, method, b).await? {
ControlFlow::Break(v) => return Ok(v),
ControlFlow::Continue(b) => b,
};
diff --git a/host/src/chat/calls/announce.rs b/host/src/chat/calls/announce.rs
new file mode 100644
index 00000000..db0aff8e
--- /dev/null
+++ b/host/src/chat/calls/announce.rs
@@ -0,0 +1,497 @@
+//! What the call's voice hears from the computer while the call lasts (`docs/plans/voice-architecture.md`, stage
+//! 2): the bot's final replies, approvals waiting, failures, and short progress while it works. Each goes out as a
+//! `call` event that only the app holding the call receives (`api::events`), worded for the call's voice (a
+//! realtime model relays it; on-device and speech-to-text calls read it aloud); the app hands it to its voice.
+
+use super::speech::spoken;
+use super::tools::{approval, runtime};
+use super::{CallMode, state};
+use crate::LockExt;
+use crate::hub::{BotStatus, Hub, Runtime};
+use crate::store::{BotConfig, Entry, EntryKind, Store, now_ms};
+use serde::Serialize;
+use serde_json::{Value, json};
+use std::collections::{HashMap, HashSet, VecDeque};
+use std::sync::atomic::{AtomicU64, Ordering};
+use std::sync::{Arc, LazyLock, Mutex};
+use tokio::sync::broadcast::error::RecvError;
+use tokio::time::{Duration, Instant};
+
+/// The fewest seconds between two progress lines.
+const PROGRESS_EVERY: Duration = Duration::from_secs(15);
+/// A working bot the voice hasn't heard about for this long gets a "still working".
+const SILENCE: Duration = Duration::from_secs(45);
+const TICK: Duration = Duration::from_secs(5);
+/// A call its app hasn't touched for this long (the app is gone, or the call silent) stops being watched; the app's
+/// next request for it brings the watcher back.
+// ponytail: an on-device call silent for this long misses cues until the user speaks again; track the owner's
+// connection if that matters.
+const IDLE: Duration = Duration::from_secs(2 * 60 * 60);
+/// The chat's latest entries read as a watcher starts (replies still unfinished) and after missed events.
+const RECENT: i64 = 50;
+/// Replies and notices kept for an app that reconnects mid-call.
+const KEPT: usize = 10;
+const REPLY_CHARS: usize = 3_000;
+const LINE_CHARS: usize = 300;
+
+/// A call this host process watches.
+struct Watch {
+ /// Which watcher task: a stopped one never removes the one after it.
+ task: u64,
+ /// When the app last touched the call.
+ touched: Instant,
+ /// The last cue's `seq` (ms or above, so it keeps rising across host restarts): an app skips cues it heard.
+ seq: i64,
+ /// The latest replies and notices, for an app that missed them ([`replay`]).
+ kept: VecDeque<Value>,
+}
+
+static WATCHED: LazyLock<Mutex<HashMap<String, Watch>>> = LazyLock::new(Mutex::default);
+static TASKS: AtomicU64 = AtomicU64::new(0);
+
+/// What a cue is on the wire: the bot's reply, a notice from the computer, or what the bot is doing.
+#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
+#[serde(rename_all = "lowercase")]
+enum Kind {
+ Reply,
+ Notice,
+ Progress,
+}
+
+/// Something for the voice, worded for the call's voice as it goes out ([`Cue::words`]).
+#[derive(Debug, PartialEq, Eq)]
+enum Cue {
+ /// A final reply as written; in a group, with the member who wrote it.
+ Reply { from: Option<String>, text: String },
+ /// An approval card waiting: who asks, for what, and its options' names.
+ Approval { from: String, request: String, options: Vec<String> },
+ /// An error notice in the chat.
+ Failure { from: String, text: String },
+ /// What the bot is doing.
+ Progress(String),
+}
+
+impl Cue {
+ /// The cue as the call's voice takes it: a realtime model gets lines to relay, shortened for speech; a voice that
+ /// reads aloud gets them worded for the listener (it shortens replies itself, and skips progress).
+ fn words(&self, mode: CallMode) -> Option<(Kind, String)> {
+ let relayed = mode == CallMode::Realtime;
+ match self {
+ Self::Reply { from, text } => {
+ let text = if relayed { spoken(text, REPLY_CHARS) } else { text.clone() };
+ let text = match from {
+ Some(name) => format!("{name}: {text}"),
+ None => text,
+ };
+ Some((Kind::Reply, text))
+ }
+ Self::Approval { from, request, options } => {
+ let request = spoken(request, LINE_CHARS);
+ let text = if relayed {
+ format!("{from} needs the user's approval: {request} (options: {})", options.join(", "))
+ } else {
+ format!("{from} needs your approval in the chat: {request}")
+ };
+ Some((Kind::Notice, text))
+ }
+ Self::Failure { from, text } => {
+ Some((Kind::Notice, format!("{from} hit a problem: {}", spoken(text, LINE_CHARS))))
+ }
+ Self::Progress(line) => relayed.then(|| (Kind::Progress, line.clone())),
+ }
+ }
+}
+
+/// What a call has told its voice so far, and what it tells it next.
+struct Announcer {
+ /// The call's bot or group: its chat is the one followed.
+ cfg: BotConfig,
+ /// Entries already said.
+ said: HashSet<String>,
+ /// Replies unfinished when the call began: said when they finish, though they began before it.
+ open: HashSet<String>,
+ /// When the call began (ms): what the chat held before it isn't said.
+ since: i64,
+ last_cue: Instant,
+ /// The last progress line's activity and when it went out.
+ progress: Option<(String, Instant)>,
+}
+
+impl Announcer {
+ /// `recent`: the chat's latest entries as the call begins.
+ fn new(cfg: BotConfig, recent: &[Entry], since: i64, now: Instant) -> Self {
+ let open = recent
+ .iter()
+ .filter(|e| e.kind == EntryKind::Agent.as_str() && e.data["final"] != true)
+ .map(|e| e.id.clone())
+ .collect();
+ Self { cfg, said: HashSet::new(), open, since, last_cue: now, progress: None }
+ }
+
+ /// An event of the hub's stream.
+ fn event(&mut self, hub: &Hub, v: &Value, now: Instant) -> Option<Cue> {
+ match v["type"].as_str()? {
+ "entry" => self.entry(hub, &v["entry"], now),
+ "bot" if v["bot"]["id"] == self.cfg.id.as_str() => self.progress(&runtime(hub, &self.cfg), now),
+ _ => None,
+ }
+ }
+
+ /// After missed events: the chat's latest entries stand in for them (each is still said once).
+ fn rescan(&mut self, hub: &Hub, now: Instant) -> Vec<Cue> {
+ let recent = hub.store.history(&self.cfg.id, i64::MAX, RECENT).unwrap_or_default();
+ recent.iter().filter_map(|e| self.entry(hub, &serde_json::to_value(e).ok()?, now)).collect()
+ }
+
+ /// Every few seconds: progress that waited out the rate limit, else a "still working" after a long silence.
+ fn tick(&mut self, rt: &Runtime, now: Instant) -> Option<Cue> {
+ self.progress(rt, now)
+ .or_else(|| if now.duration_since(self.last_cue) >= SILENCE { self.still_working(rt, now) } else { None })
+ }
+
+ fn entry(&mut self, hub: &Hub, e: &Value, now: Instant) -> Option<Cue> {
+ let id = e["id"].as_str()?;
+ if e["botId"] != self.cfg.id.as_str() || self.said.contains(id) {
+ return None;
+ }
+ let new = e["createdAt"].as_i64().unwrap_or(0) >= self.since;
+ let data = &e["data"];
+ let cue = match e["kind"].as_str()? {
+ k if k == EntryKind::Agent.as_str() => self.reply(hub, id, data, new)?,
+ k if k == EntryKind::Permission.as_str() && new && data["status"] == "pending" => {
+ let (request, options) = approval(data);
+ let options = options.into_iter().map(str::to_owned).collect();
+ Cue::Approval { from: self.author(hub, data), request: request.to_owned(), options }
+ }
+ k if k == EntryKind::Notice.as_str() && new && data["style"] == "error" => Cue::Failure {
+ from: self.author(hub, data),
+ text: data["text"].as_str().unwrap_or_default().to_owned(),
+ },
+ _ => return None,
+ };
+ self.said.insert(id.to_owned());
+ self.last_cue = now;
+ // Progress right after a cue is stale (`send_message` leaves "Working…" just before the turn ends): the next
+ // line waits out the limit.
+ let last = self.progress.take().map(|(last, _)| last).unwrap_or_default();
+ self.progress = Some((last, now));
+ Some(cue)
+ }
+
+ /// A final reply begun during the call, or one unfinished when it began.
+ fn reply(&self, hub: &Hub, id: &str, data: &Value, new: bool) -> Option<Cue> {
+ let text = data["text"].as_str().unwrap_or_default().trim();
+ if data["final"] != true || text.is_empty() || !(new || self.open.contains(id)) {
+ return None;
+ }
+ Some(Cue::Reply { from: self.cfg.is_group().then(|| self.author(hub, data)), text: text.to_owned() })
+ }
+
+ /// Who wrote an entry: in a group, the member.
+ fn author(&self, hub: &Hub, data: &Value) -> String {
+ data["author"].as_str().map_or_else(|| self.cfg.name.clone(), |id| hub.bot_name(id))
+ }
+
+ /// What the bot is doing, when that changed and the last line is old enough.
+ fn progress(&mut self, rt: &Runtime, now: Instant) -> Option<Cue> {
+ let doing = doing(rt)?;
+ let changed =
+ self.progress.as_ref().is_none_or(|(last, at)| *last != doing && now.duration_since(*at) >= PROGRESS_EVERY);
+ if doing.is_empty() || !changed {
+ return None;
+ }
+ let line = format!("{}: {doing}", self.cfg.name.trim());
+ Some(self.progressed(doing, line, now))
+ }
+
+ /// Whatever the bot is doing, after a long silence.
+ fn still_working(&mut self, rt: &Runtime, now: Instant) -> Option<Cue> {
+ let doing = doing(rt)?;
+ let name = self.cfg.name.trim();
+ let line = if doing.is_empty() {
+ format!("{name} is still working.")
+ } else {
+ format!("{name} is still working: {doing}")
+ };
+ Some(self.progressed(doing, line, now))
+ }
+
+ fn progressed(&mut self, doing: String, line: String, now: Instant) -> Cue {
+ self.progress = Some((doing, now));
+ self.last_cue = now;
+ Cue::Progress(line)
+ }
+}
+
+/// The bot's activity line while it works, shortened for speech.
+fn doing(rt: &Runtime) -> Option<String> {
+ (rt.status == BotStatus::Working).then(|| spoken(rt.activity.trim(), LINE_CHARS))
+}
+
+/// Tells the call's app what its voice should say until the call ends or the app leaves it idle; for a call watched
+/// already, notes that the app touched it.
+pub(super) fn watch(hub: &Arc<Hub>, call: &str, bot: &str) {
+ let task = TASKS.fetch_add(1, Ordering::Relaxed);
+ {
+ let mut watched = WATCHED.locked();
+ if let Some(w) = watched.get_mut(call) {
+ w.touched = Instant::now();
+ return;
+ }
+ watched.insert(call.to_owned(), Watch { task, touched: Instant::now(), seq: 0, kept: VecDeque::new() });
+ }
+ let Ok(Some(row)) = hub.store.bot(bot) else {
+ unwatch(call, task);
+ return;
+ };
+ let mut events = hub.events.subscribe();
+ let recent = hub.store.history(bot, i64::MAX, RECENT).unwrap_or_default();
+ let mut announcer = Announcer::new(row.config, &recent, now_ms(), Instant::now());
+ let (hub, call) = (hub.clone(), call.to_owned());
+ tokio::spawn(async move {
+ let mut tick = tokio::time::interval(TICK);
+ loop {
+ let cues: Vec<Cue> = tokio::select! {
+ event = events.recv() => match event {
+ Ok(v) => announcer.event(&hub, &v, Instant::now()).into_iter().collect(),
+ Err(RecvError::Lagged(_)) => announcer.rescan(&hub, Instant::now()),
+ Err(RecvError::Closed) => break,
+ },
+ _ = tick.tick() => {
+ if !watching(&hub.store, &call, task) {
+ break;
+ }
+ announcer.tick(&runtime(&hub, &announcer.cfg), Instant::now()).into_iter().collect()
+ }
+ };
+ for cue in &cues {
+ tell(&hub, &call, cue);
+ }
+ }
+ unwatch(&call, task);
+ });
+}
+
+/// The call is still on, this task watches it and its app touched it lately; else the watcher stops here.
+fn watching(store: &Store, call: &str, task: u64) -> bool {
+ let live = super::is_live(store, call);
+ let mut watched = WATCHED.locked();
+ let Some(w) = watched.get(call).filter(|w| w.task == task) else { return false };
+ let keep = live && w.touched.elapsed() < IDLE;
+ if !keep {
+ watched.remove(call);
+ }
+ keep
+}
+
+fn unwatch(call: &str, task: u64) {
+ let mut watched = WATCHED.locked();
+ if watched.get(call).is_some_and(|w| w.task == task) {
+ watched.remove(call);
+ }
+}
+
+/// Sends the app holding the call a cue, worded for its voice now (a replaced session may be on another device, or
+/// carry another mode).
+fn tell(hub: &Hub, call: &str, cue: &Cue) {
+ let Some(s) = state(&hub.store, call) else { return };
+ let Some((kind, text)) = cue.words(s.mode) else { return };
+ let event = {
+ let mut watched = WATCHED.locked();
+ let Some(w) = watched.get_mut(call) else { return };
+ w.seq = now_ms().max(w.seq + 1);
+ let event = json!({"type": "call", "call": call, "owner": s.owner, "kind": kind, "text": text, "seq": w.seq});
+ if kind != Kind::Progress {
+ if w.kept.len() == KEPT {
+ w.kept.pop_front();
+ }
+ w.kept.push_back(event.clone());
+ }
+ event
+ };
+ let _ = hub.events.send(event);
+}
+
+/// The latest replies and notices for the calls `owner` holds, for its catch-up: an app that missed them while it
+/// reconnected hears them now (it skips the ones it heard by `seq`).
+pub fn replay(owner: &str) -> Vec<Value> {
+ WATCHED.locked().values().flat_map(|w| &w.kept).filter(|v| v["owner"] == owner).cloned().collect()
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+ use crate::store::{BotKind, Lane};
+
+ fn entry(hub: &Hub, bot: &str, kind: EntryKind, data: &Value) -> Value {
+ let e = hub.store.insert_entry(&Lane::main(bot), kind, 1, data).unwrap();
+ json!({"type": "entry", "entry": e})
+ }
+
+ fn setup(id: &str) -> (Arc<Hub>, BotConfig) {
+ let hub = super::super::context::tests::hub();
+ let cfg = super::super::context::tests::bot(id, "Mika", "/tmp");
+ hub.store.save_bot(&cfg).unwrap();
+ (hub, cfg)
+ }
+
+ fn working(activity: &str) -> Runtime {
+ Runtime { status: BotStatus::Working, activity: activity.into(), ..Runtime::default() }
+ }
+
+ #[tokio::test]
+ async fn the_voice_hears_new_replies_approvals_and_failures_once() {
+ let (hub, cfg) = setup("b-cue");
+ let now = Instant::now();
+ let old = entry(&hub, &cfg.id, EntryKind::Agent, &json!({"text": "Before the call.", "final": true}));
+ std::thread::sleep(Duration::from_millis(2));
+ let mut a = Announcer::new(cfg.clone(), &[], now_ms(), now);
+ assert_eq!(a.event(&hub, &old, now), None, "a reply from before the call isn't said");
+
+ let reply =
+ entry(&hub, &cfg.id, EntryKind::Agent, &json!({"text": "**Tests pass.** See `cargo test`", "final": true}));
+ let said = a.event(&hub, &reply, now).unwrap();
+ assert_eq!(said.words(CallMode::Realtime), Some((Kind::Reply, "Tests pass. See cargo test".into())));
+ assert_eq!(a.event(&hub, &reply, now), None, "said once, though its entry changes again");
+
+ let streaming = entry(&hub, &cfg.id, EntryKind::Agent, &json!({"text": "Work"}));
+ assert_eq!(a.event(&hub, &streaming, now), None, "not until it's final");
+
+ let card =
+ json!({"title": "Run rm -rf build", "status": "pending", "options": [{"name": "Allow"}, {"name": "Deny"}]});
+ let card = entry(&hub, &cfg.id, EntryKind::Permission, &card);
+ let cue = a.event(&hub, &card, now).unwrap();
+ assert_eq!(
+ cue.words(CallMode::Realtime),
+ Some((Kind::Notice, "Mika needs the user's approval: Run rm -rf build (options: Allow, Deny)".into()))
+ );
+
+ let failed = entry(&hub, &cfg.id, EntryKind::Notice, &json!({"text": "Rate limited.", "style": "error"}));
+ let cue = a.event(&hub, &failed, now).unwrap();
+ assert_eq!(cue.words(CallMode::Device), Some((Kind::Notice, "Mika hit a problem: Rate limited.".into())));
+
+ let elsewhere = entry(&hub, "another-bot", EntryKind::Agent, &json!({"text": "Not this chat.", "final": true}));
+ assert_eq!(a.event(&hub, &elsewhere, now), None);
+ }
+
+ #[tokio::test]
+ async fn a_reply_unfinished_when_the_call_began_is_said_when_it_finishes() {
+ let (hub, cfg) = setup("b-open");
+ let begun = entry(&hub, &cfg.id, EntryKind::Agent, &json!({"text": "Let me run the full test suite."}));
+ let id = begun["entry"]["id"].as_str().unwrap().to_owned();
+ std::thread::sleep(Duration::from_millis(2));
+ let recent = hub.store.history(&cfg.id, i64::MAX, RECENT).unwrap();
+ let now = Instant::now();
+ let mut a = Announcer::new(cfg.clone(), &recent, now_ms(), now);
+ let finished = hub.set_entry(&id, &json!({"text": "All 412 tests pass.", "final": true})).unwrap();
+ let cue = a.event(&hub, &json!({"type": "entry", "entry": finished}), now);
+ assert_eq!(cue, Some(Cue::Reply { from: None, text: "All 412 tests pass.".into() }));
+ }
+
+ #[tokio::test]
+ async fn in_a_group_the_voice_hears_which_member_replied_or_asks() {
+ let (hub, ada) = setup("b-ada");
+ let ada = BotConfig { name: "Ada".into(), ..ada };
+ hub.store.save_bot(&ada).unwrap();
+ let group = BotConfig {
+ kind: BotKind::Group,
+ members: vec![ada.id.clone()],
+ ..super::super::context::tests::bot("g-team", "Team", "")
+ };
+ hub.store.save_bot(&group).unwrap();
+ let now = Instant::now();
+ let mut a = Announcer::new(group.clone(), &[], now_ms(), now);
+ let reply = json!({"text": "Done.", "final": true, "author": ada.id});
+ let cue = a.event(&hub, &entry(&hub, &group.id, EntryKind::Agent, &reply), now).unwrap();
+ assert_eq!(cue.words(CallMode::Realtime), Some((Kind::Reply, "Ada: Done.".into())));
+ let card =
+ json!({"title": "Edit a.txt", "status": "pending", "options": [{"name": "Allow"}], "author": ada.id});
+ let cue = a.event(&hub, &entry(&hub, &group.id, EntryKind::Permission, &card), now).unwrap();
+ assert_eq!(
+ cue.words(CallMode::Device),
+ Some((Kind::Notice, "Ada needs your approval in the chat: Edit a.txt".into()))
+ );
+ }
+
+ #[test]
+ fn a_voice_that_reads_aloud_gets_cues_worded_for_the_listener() {
+ let reply = Cue::Reply { from: None, text: "See `main.rs`:\n```\nfn main() {}\n```".into() };
+ assert_eq!(
+ reply.words(CallMode::Speech),
+ Some((Kind::Reply, "See `main.rs`:\n```\nfn main() {}\n```".into())),
+ "it shortens replies itself"
+ );
+ assert_eq!(reply.words(CallMode::Realtime), Some((Kind::Reply, "See main.rs: (code)".into())));
+ let card = Cue::Approval { from: "Mika".into(), request: "Edit a.txt".into(), options: vec!["Allow".into()] };
+ assert_eq!(
+ card.words(CallMode::Device),
+ Some((Kind::Notice, "Mika needs your approval in the chat: Edit a.txt".into()))
+ );
+ assert_eq!(Cue::Progress("Mika: Running tests".into()).words(CallMode::Device), None);
+ }
+
+ #[tokio::test]
+ async fn progress_is_rate_limited_and_silence_gets_a_still_working() {
+ let (hub, cfg) = setup("b-progress");
+ let t0 = Instant::now();
+ let mut a = Announcer::new(cfg.clone(), &[], now_ms(), t0);
+ let bot = json!({"type": "bot", "bot": {"id": cfg.id}});
+ assert_eq!(a.event(&hub, &bot, t0), None, "an idle bot has no progress");
+
+ hub.set_runtime(&cfg.id, |r| *r = working("Running `cargo test`"));
+ assert_eq!(a.event(&hub, &bot, t0), Some(Cue::Progress("Mika: Running cargo test".into())));
+ let editing = working("Editing main.rs");
+ let soon = t0 + Duration::from_secs(5);
+ assert_eq!(a.tick(&editing, soon), None, "too soon after the last line");
+ let later = t0 + PROGRESS_EVERY;
+ assert_eq!(a.tick(&editing, later), Some(Cue::Progress("Mika: Editing main.rs".into())), "the tick catches up");
+ assert_eq!(a.tick(&editing, later + Duration::from_secs(20)), None, "nothing changed, not silent long enough");
+ let quiet = later + SILENCE;
+ assert_eq!(a.tick(&editing, quiet), Some(Cue::Progress("Mika is still working: Editing main.rs".into())));
+ assert_eq!(a.tick(&Runtime::default(), quiet + SILENCE), None, "an idle bot isn't working");
+ }
+
+ #[tokio::test]
+ async fn no_progress_right_after_a_reply() {
+ let (hub, cfg) = setup("b-after-reply");
+ let t0 = Instant::now();
+ let mut a = Announcer::new(cfg.clone(), &[], now_ms(), t0);
+ let bot = json!({"type": "bot", "bot": {"id": cfg.id}});
+ hub.set_runtime(&cfg.id, |r| *r = working("Running tests"));
+ assert!(a.event(&hub, &bot, t0).is_some());
+ let later = t0 + PROGRESS_EVERY;
+ let reply = entry(&hub, &cfg.id, EntryKind::Agent, &json!({"text": "Done.", "final": true}));
+ assert!(a.event(&hub, &reply, later).is_some());
+ hub.set_runtime(&cfg.id, |r| *r = working("Working…"));
+ assert_eq!(a.event(&hub, &bot, later), None, "the turn is ending, not starting new work");
+ let still = a.tick(&working("Working…"), later + PROGRESS_EVERY);
+ assert_eq!(still, Some(Cue::Progress("Mika: Working…".into())), "still working once the limit passed");
+ }
+
+ #[tokio::test]
+ async fn replies_and_notices_wait_for_the_app_holding_the_call() {
+ let (hub, cfg) = setup("b-kept");
+ let call = uuid::Uuid::new_v4().to_string();
+ let s = super::super::CallState {
+ owner: "phone".into(),
+ mode: CallMode::Realtime,
+ ..super::super::CallState::new(&cfg.id)
+ };
+ super::super::save(&hub.store, &call, &s).unwrap();
+ WATCHED
+ .locked()
+ .insert(call.clone(), Watch { task: u64::MAX, touched: Instant::now(), seq: 0, kept: VecDeque::new() });
+ let mut events = hub.events.subscribe();
+ tell(&hub, &call, &Cue::Progress("Mika: Running tests".into()));
+ tell(&hub, &call, &Cue::Reply { from: None, text: "Done.".into() });
+ let (first, second) = (events.try_recv().unwrap(), events.try_recv().unwrap());
+ assert!(second["seq"].as_i64() > first["seq"].as_i64(), "{first} {second}");
+
+ let kept = replay("phone");
+ let kept: Vec<_> = kept.iter().filter(|v| v["call"] == call.as_str()).collect();
+ assert_eq!(kept.len(), 1, "progress isn't kept: {kept:?}");
+ assert_eq!((kept[0]["kind"].as_str(), &kept[0]["seq"]), (Some("reply"), &second["seq"]));
+ assert!(replay("tablet").iter().all(|v| v["call"] != call.as_str()), "only for the device holding it");
+ unwatch(&call, u64::MAX);
+ }
+}
diff --git a/host/src/chat/calls/context.rs b/host/src/chat/calls/context.rs
index 1caa9cd2..7474aefe 100644
--- a/host/src/chat/calls/context.rs
+++ b/host/src/chat/calls/context.rs
@@ -1,6 +1,7 @@
//! What the voice of a call hears before it speaks: its rules, then the background as named sections, each capped,
//! empty ones left out, each saying where it came from. It goes to the voice provider, on the user's own key.
+use super::speech::spoken;
use super::tools::{runtime, waiting};
use crate::hub::{BotStatus, Hub};
use crate::store::{BotConfig, Entry, EntryKind, Lane};
@@ -69,6 +70,8 @@ word for word. Never do the work or invent results.
- Call answer_approval only after the user has clearly said which option to pick.
- Messages starting with [{name} replied] or [Notice] come from the computer: tell the user what they say, shortened \
for speech (no code blocks, no long lists, no URLs).
+- Messages starting with [Progress] come from the computer while {name} works: the latest one answers \"what is \
+{name} doing?\"; mention it briefly if the user is waiting, never make up progress or results.
- Speak the user's language. Keep spoken answers short and natural.
- The background below is for context: use it, don't read it out."
)
@@ -267,11 +270,7 @@ fn conversation(hub: &Hub, cfg: &BotConfig, recent: &[Entry]) -> String {
let gist = e.data["transcript"].as_array().into_iter().flatten().find(|l| l["from"] == "user");
let gist =
gist.and_then(|l| l["text"].as_str()).map(|t| format!(": \"{}\"", spoken(t, 120))).unwrap_or_default();
- format!(
- "- {} ({}){gist}",
- e.data["text"].as_str().unwrap_or(super::UNTITLED),
- crate::chat::memory::ymd(e.created_at)
- )
+ format!("- {} ({}){gist}", e.data["text"].as_str().unwrap_or(super::UNTITLED), local_time(e.created_at))
})
.collect();
if !calls.is_empty() {
@@ -280,6 +279,12 @@ fn conversation(hub: &Hub, cfg: &BotConfig, recent: &[Entry]) -> String {
lines.join("\n")
}
+/// When something happened, on this computer's clock (as "Now" reads).
+fn local_time(ms: i64) -> String {
+ chrono::DateTime::from_timestamp_millis(ms)
+ .map_or_else(|| "unknown date".into(), |t| t.with_timezone(&chrono::Local).format("%Y-%m-%d %H:%M").to_string())
+}
+
fn others(hub: &Hub, cfg: &BotConfig) -> String {
let rows = hub.store.bots().unwrap_or_default();
rows.iter()
@@ -315,38 +320,6 @@ fn shared(recent: &[Entry]) -> String {
titles.join("\n")
}
-/// Text as speech can carry it: code blocks and links named, not read, markdown marks dropped, capped.
-pub fn spoken(text: &str, max: usize) -> String {
- let mut out = String::new();
- let mut in_code = false;
- for line in text.lines() {
- if line.trim_start().starts_with("```") {
- if !in_code {
- out.push_str(" (code) ");
- }
- in_code = !in_code;
- continue;
- }
- if in_code {
- continue;
- }
- let words: Vec<String> = line
- .split_whitespace()
- .map(|w| {
- if w.starts_with("http://") || w.starts_with("https://") {
- "(link)".to_owned()
- } else {
- w.trim_matches(['*', '`', '#', '>', '_']).to_owned()
- }
- })
- .filter(|w| !w.is_empty())
- .collect();
- out.push_str(&words.join(" "));
- out.push(' ');
- }
- crate::agent::acp::truncate(out.split_whitespace().collect::<Vec<_>>().join(" ").as_str(), max)
-}
-
#[cfg(test)]
pub(super) mod tests {
use super::*;
@@ -482,11 +455,19 @@ pub(super) mod tests {
}
#[test]
- fn speech_names_code_and_links() {
- assert_eq!(
- spoken("**Done.** See https://x.dev\n```\nlet a = 1;\n```\n- `cargo test` passed", 200),
- "Done. See (link) (code) - cargo test passed"
+ fn earlier_calls_are_dated_on_this_computers_clock() {
+ let hub = hub();
+ let cfg = bot("b-clock", "Mika", "/tmp");
+ hub.store.save_bot(&cfg).unwrap();
+ let data =
+ json!({"text": "Release plan", "callSeconds": 30, "transcript": [{"from": "user", "text": "ship it"}]});
+ let call = hub.store.insert_entry(&Lane::main(&cfg.id), EntryKind::Notice, 1, &data).unwrap();
+ let recent = hub.store.history(&cfg.id, i64::MAX, RECENT).unwrap();
+ let local = chrono::DateTime::from_timestamp_millis(call.created_at).unwrap().with_timezone(&chrono::Local);
+ let section = conversation(&hub, &cfg, &recent);
+ assert!(
+ section.contains(&format!("- Release plan ({}): \"ship it\"", local.format("%Y-%m-%d %H:%M"))),
+ "{section}"
);
- assert!(spoken(&"word ".repeat(500), 50).chars().count() <= 51);
}
}
diff --git a/host/src/chat/calls/mod.rs b/host/src/chat/calls/mod.rs
index e2f46f16..d2a231c5 100644
--- a/host/src/chat/calls/mod.rs
+++ b/host/src/chat/calls/mod.rs
@@ -3,13 +3,17 @@
//! voice's tool calls (`callTool`, [`tools`]), keeps the transcript as it is spoken (`callLines`, and whatever lines
//! the app still holds ride along with `callTool` and `endCall`) and writes the call's line when it ends (`endCall`).
//! What a call sends the bot carries its id (`call`), and so do the bot's replies while it lasts: the apps show those
-//! only in the call's conversation, which its line opens.
+//! only in the call's conversation, which its line opens. While it lasts, the bot's replies, approvals, failures
+//! and progress reach the voice from here ([`announce`]).
+mod announce;
pub mod context;
mod project;
+mod speech;
mod sweep;
pub mod tools;
+pub use announce::replay;
pub use sweep::sweep_loop;
use crate::LockExt;
@@ -66,19 +70,34 @@ struct CallState {
/// When the call last heard from its app.
#[serde(default)]
last: i64,
+ /// The device that holds the call (`Caller::device_key`): only it hears what the voice should say.
+ #[serde(default)]
+ owner: String,
+ /// What carries its voice now: how what it hears is worded ([`announce`]).
+ #[serde(default)]
+ mode: CallMode,
}
impl CallState {
fn new(bot: &str) -> Self {
let now = now_ms();
- Self { bot: bot.to_owned(), lines: Vec::new(), handoff: 0, started: now, last: now }
+ Self {
+ bot: bot.to_owned(),
+ lines: Vec::new(),
+ handoff: 0,
+ started: now,
+ last: now,
+ owner: String::new(),
+ mode: CallMode::default(),
+ }
}
}
/// What carries a call's voice: a realtime provider, speech to text and back, or the device's own speech.
-#[derive(Clone, Copy, PartialEq, Eq, Deserialize)]
+#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "lowercase")]
enum CallMode {
+ #[default]
Realtime,
Speech,
Device,
@@ -120,13 +139,36 @@ fn begun(store: &Store, bot: &str, call: &str) -> Result<Option<CallState>> {
Ok(Some(CallState::new(bot)))
}
-/// A message said on call `call` reached `bot`'s chat (`send`): the call is alive, and the host knows of it even
-/// if its `startCall` never came.
-pub fn heard(store: &Store, bot: &str, call: &str) -> Result<()> {
- let _one = STATE.locked();
- match begun(store, bot, call)? {
- Some(s) if s.bot == bot => save(store, call, &s),
- _ => Ok(()),
+/// A message said on call `call` reached `bot`'s chat (`send`) from device `owner`: the call is alive, and the host
+/// knows of it even if its `startCall` never came (then it's an on-device or speech call: only those send
+/// messages), and its voice keeps hearing the bot's work.
+pub fn heard(hub: &Arc<Hub>, bot: &str, call: &str, owner: &str) -> Result<()> {
+ {
+ let _one = STATE.locked();
+ let known = state(&hub.store, call).is_some();
+ match begun(&hub.store, bot, call)? {
+ Some(mut s) if s.bot == bot => {
+ if !known {
+ s.mode = CallMode::Device;
+ }
+ if s.owner.is_empty() {
+ owner.clone_into(&mut s.owner);
+ }
+ save(&hub.store, call, &s)?;
+ }
+ _ => return Ok(()),
+ }
+ }
+ announce::watch(hub, call, bot);
+ Ok(())
+}
+
+/// As the host starts: the calls still on hear the bot's work again (the sweep ends abandoned ones).
+pub fn rewatch(hub: &Arc<Hub>) {
+ for call in sweep::open_calls(&hub.store) {
+ if let Some(s) = state(&hub.store, &call) {
+ announce::watch(hub, &call, &s.bot);
+ }
}
}
@@ -191,9 +233,9 @@ fn append(s: &mut CallState, lines: Vec<CallLine>) {
s.handoff = s.handoff.saturating_sub(over);
}
-/// `startCall`: the call is on (again, for a session the provider replaced: its transcript stays). Hands back what
-/// the client needs: the voice's instructions and tools, and a realtime provider's credential.
-pub async fn start(hub: &Arc<Hub>, b: &Value) -> Result<Value> {
+/// `startCall` from device `owner`: the call is on (again, for a session the provider replaced: its transcript
+/// stays). Hands back what the client needs: the voice's instructions and tools, and a realtime provider's credential.
+pub async fn start(hub: &Arc<Hub>, b: &Value, owner: &str) -> Result<Value> {
let bot = b["botId"].as_str().ok_or_else(|| anyhow!("`botId` is required"))?;
let call = checked(b["callId"].as_str().unwrap_or_default())?;
let mode = if b["mode"].is_null() { CallMode::Realtime } else { CallMode::deserialize(&b["mode"])? };
@@ -202,10 +244,12 @@ pub async fn start(hub: &Arc<Hub>, b: &Value) -> Result<Value> {
let _one = STATE.locked();
match begun(&hub.store, bot, call)? {
Some(s) if s.bot != bot => bail!("that call belongs to another bot"),
- Some(s) => save(&hub.store, call, &s)?,
+ // A session the provider replaced, maybe on another device or in another mode now.
+ Some(s) => save(&hub.store, call, &CallState { owner: owner.to_owned(), mode, ..s })?,
None => bail!("that call has ended"),
}
}
+ announce::watch(hub, call, bot);
let provider = b["provider"].as_str().map(crate::voice::Provider::parse).transpose()?;
let mut out = json!({
"instructions": context::instructions(hub, &cfg, provider, b["language"].as_str()).await,
@@ -220,17 +264,21 @@ pub async fn start(hub: &Arc<Hub>, b: &Value) -> Result<Value> {
Ok(out)
}
-/// `callLines`: what was just said on the call (`raw`, see [`transcript`]).
-pub fn add_lines(store: &Store, call: &str, raw: &Value) -> Result<()> {
+/// `callLines`: what was just said on the call (`raw`, see [`transcript`]). Its voice keeps hearing the bot's work
+/// (again, after a host restart).
+pub fn add_lines(hub: &Arc<Hub>, call: &str, raw: &Value) -> Result<()> {
let lines = transcript(raw);
+ let _one = STATE.locked();
+ let Some(mut s) = state(&hub.store, call) else {
+ return if lines.is_empty() { Ok(()) } else { Err(anyhow!("no such call")) };
+ };
+ announce::watch(hub, call, &s.bot);
if lines.is_empty() {
return Ok(());
}
- let _one = STATE.locked();
- let mut s = state(store, call).ok_or_else(|| anyhow!("no such call"))?;
s.last = now_ms();
append(&mut s, lines);
- save(store, call, &s)
+ save(&hub.store, call, &s)
}
/// What the user said since the bot last heard from the call, word for word; the bot hears it now.
@@ -431,11 +479,11 @@ mod tests {
// No agent to run: the title writer gives up at once.
let cfg = BotConfig { backend: "none".into(), ..context::tests::bot(&id, "Mika", "/tmp") };
hub.store.save_bot(&cfg).unwrap();
- let started = start(&hub, &json!({"botId": id, "callId": "c1", "mode": "device"})).await.unwrap();
+ let started = start(&hub, &json!({"botId": id, "callId": "c1", "mode": "device"}), "phone").await.unwrap();
assert!(started["instructions"].as_str().unwrap().contains("voice relay for Mika"));
assert!(started["tools"].as_array().unwrap().iter().any(|t| t["name"] == "search_memory"));
assert!(started.get("credential").is_none(), "an on-device call has no provider");
- add_lines(&hub.store, "c1", &json!([{"from": "user", "text": "首頁", "at": 1}])).unwrap();
+ add_lines(&hub, "c1", &json!([{"from": "user", "text": "首頁", "at": 1}])).unwrap();
// The rest of what the user said comes with the tool call that hands it on.
let unsent = json!([{"from": "user", "text": "改成藍色", "at": 2}, {"from": "voice", "text": "好"}]);
@@ -455,21 +503,27 @@ mod tests {
assert_eq!(line.data["transcript"].as_array().map(Vec::len), Some(4), "with the lines the app still held");
assert!(end(&hub, &id, "c1", 31, &last).unwrap().is_none(), "an app trying again ends it once");
assert!(bot_of(&hub.store, "c1").is_none(), "an ended call takes no more lines");
- assert!(start(&hub, &json!({"botId": id, "callId": "c1", "mode": "device"})).await.is_err());
+ assert!(start(&hub, &json!({"botId": id, "callId": "c1", "mode": "device"}), "phone").await.is_err());
assert!(of_turn(&hub.store, &[relayed[0].id.clone()]).is_none(), "its replies show again");
assert!(tools::call(&hub, "c1", "bot_status", &json!({}), &Value::Null).await["error"].is_string());
}
- #[test]
- fn a_line_sent_again_is_kept_once() {
- let store = Store::open(std::path::Path::new(":memory:")).unwrap();
- heard(&store, "b", "c1").unwrap();
+ #[tokio::test]
+ async fn a_line_sent_again_is_kept_once() {
+ let hub = context::tests::hub();
+ heard(&hub, "b", "c1", "phone").unwrap();
let batch = json!([{"from": "user", "text": "hi", "at": 1, "id": "l1"}, {"from": "voice", "text": "Hello.", "id": "l2"}]);
- add_lines(&store, "c1", &batch).unwrap();
+ add_lines(&hub, "c1", &batch).unwrap();
// The answer to the first send was lost, so the app sends the batch again, and one more line.
- add_lines(&store, "c1", &batch).unwrap();
- add_lines(&store, "c1", &json!([{"from": "user", "text": "hi", "id": "l3"}])).unwrap();
- let lines: Vec<String> = state(&store, "c1").unwrap().lines.iter().map(|l| l.text.clone()).collect();
+ add_lines(&hub, "c1", &batch).unwrap();
+ add_lines(&hub, "c1", &json!([{"from": "user", "text": "hi", "id": "l3"}])).unwrap();
+ let s = state(&hub.store, "c1").unwrap();
+ assert_eq!(
+ (s.owner.as_str(), s.mode),
+ ("phone", CallMode::Device),
+ "a call first heard by a message is on-device"
+ );
+ let lines: Vec<String> = s.lines.iter().map(|l| l.text.clone()).collect();
assert_eq!(lines, ["hi", "Hello.", "hi"], "repeats skipped, a new line with the same words kept");
}
diff --git a/host/src/chat/calls/speech.rs b/host/src/chat/calls/speech.rs
new file mode 100644
index 00000000..eb4d0b4d
--- /dev/null
+++ b/host/src/chat/calls/speech.rs
@@ -0,0 +1,52 @@
+//! Text as a voice says it.
+
+/// Text as speech can carry it: code blocks and links named, not read, markdown marks dropped, capped.
+pub(super) fn spoken(text: &str, max: usize) -> String {
+ let mut out = String::new();
+ let mut in_code = false;
+ for line in text.lines() {
+ if line.trim_start().starts_with("```") {
+ if !in_code {
+ out.push_str(" (code) ");
+ }
+ in_code = !in_code;
+ continue;
+ }
+ if in_code {
+ continue;
+ }
+ let words: Vec<String> = line
+ .split_whitespace()
+ .map(|w| {
+ if w.starts_with("http://") || w.starts_with("https://") {
+ "(link)".to_owned()
+ } else {
+ // Inline code and emphasis can span words ("`cargo test`."): their marks go wherever they are.
+ w.replace(['`', '*'], "").trim_matches(['#', '>', '_']).to_owned()
+ }
+ })
+ .filter(|w| !w.is_empty())
+ .collect();
+ out.push_str(&words.join(" "));
+ out.push(' ');
+ }
+ crate::agent::acp::truncate(out.split_whitespace().collect::<Vec<_>>().join(" ").as_str(), max)
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn speech_names_code_and_links() {
+ assert_eq!(
+ spoken("**Done.** See https://x.dev\n```\nlet a = 1;\n```\n- `cargo test` passed", 200),
+ "Done. See (link) (code) - cargo test passed"
+ );
+ assert!(spoken(&"word ".repeat(500), 50).chars().count() <= 51);
+ assert_eq!(
+ spoken("Run `cargo test --release`. **Done**, keep snake_case", 200),
+ "Run cargo test --release. Done, keep snake_case"
+ );
+ }
+}
diff --git a/host/src/chat/calls/sweep.rs b/host/src/chat/calls/sweep.rs
index 26b5b806..92fa3563 100644
--- a/host/src/chat/calls/sweep.rs
+++ b/host/src/chat/calls/sweep.rs
@@ -102,7 +102,7 @@ mod tests {
async fn an_abandoned_call_gets_its_line() {
let (hub, id) = hub_with_bot();
// A message said on a call whose app then went away (its startCall never reached the host).
- heard(&hub.store, &id, "gone").unwrap();
+ heard(&hub, &id, "gone", "phone").unwrap();
assert!(is_live(&hub.store, "gone"));
sweep(&hub, now_ms());
assert!(is_live(&hub.store, "gone"), "a call heard from just now stays open");
@@ -116,7 +116,7 @@ mod tests {
#[tokio::test]
async fn a_call_resumed_before_the_sweep_ends_it_stays_open() {
let (hub, id) = hub_with_bot();
- heard(&hub.store, &id, "back").unwrap();
+ heard(&hub, &id, "back", "phone").unwrap();
let later = now_ms() + ABANDONED_MS + 1;
// What the sweep read first: quiet for long enough.
let read = state(&hub.store, "back").unwrap();
@@ -132,7 +132,7 @@ mod tests {
#[tokio::test]
async fn a_call_its_bot_works_on_stays_open() {
let (hub, id) = hub_with_bot();
- heard(&hub.store, &id, "long").unwrap();
+ heard(&hub, &id, "long", "phone").unwrap();
// The app last spoke long ago: the user asked for a long job and stayed quiet.
let mut s = state(&hub.store, "long").unwrap();
(s.started, s.last) = (0, 0);
diff --git a/host/src/chat/calls/tools.rs b/host/src/chat/calls/tools.rs
index 1eede41e..ecb57bee 100644
--- a/host/src/chat/calls/tools.rs
+++ b/host/src/chat/calls/tools.rs
@@ -31,7 +31,7 @@ pub fn declarations() -> Value {
pub async fn call(hub: &Arc<Hub>, call: &str, name: &str, args: &Value, lines: &Value) -> Value {
let run = async {
let bot = super::bot_of(&hub.store, call).ok_or_else(|| anyhow!("The call has ended."))?;
- super::add_lines(&hub.store, call, lines)?;
+ super::add_lines(hub, call, lines)?;
let cfg =
hub.store.bot(&bot)?.filter(|r| !r.deleted).ok_or_else(|| anyhow!("The bot isn't available."))?.config;
match name {
@@ -98,11 +98,14 @@ pub(super) struct Waiting<'a> {
pub(super) fn waiting(recent: &[Entry]) -> Option<Waiting<'_>> {
let card =
recent.iter().rev().find(|e| e.kind == EntryKind::Permission.as_str() && e.data["status"] == "pending")?;
- Some(Waiting {
- card,
- request: card.data["title"].as_str().or_else(|| card.data["command"].as_str()).unwrap_or_default(),
- options: card.data["options"].as_array().into_iter().flatten().filter_map(|o| o["name"].as_str()).collect(),
- })
+ let (request, options) = approval(&card.data);
+ Some(Waiting { card, request, options })
+}
+
+/// What an approval card asks and its options' names.
+pub(super) fn approval(data: &Value) -> (&str, Vec<&str>) {
+ let request = data["title"].as_str().or_else(|| data["command"].as_str()).unwrap_or_default();
+ (request, data["options"].as_array().into_iter().flatten().filter_map(|o| o["name"].as_str()).collect())
}
fn answer(hub: &Hub, cfg: &BotConfig, option: &str) -> Result<Value> {
diff --git a/host/src/chat/lead/tests.rs b/host/src/chat/lead/tests.rs
index 9afff0a2..15cb4681 100644
--- a/host/src/chat/lead/tests.rs
+++ b/host/src/chat/lead/tests.rs
@@ -131,7 +131,7 @@ async fn the_lead_speaks_first_on_a_call() {
let lead = make(&f.hub, f.cfg(NAME)).unwrap();
let other = f.hub.create_bot(f.cfg("Other")).unwrap()["id"].as_str().unwrap().to_owned();
call_started(&f.hub, &other, None).unwrap();
- crate::chat::calls::heard(&f.hub.store, &lead, "c1").unwrap();
+ crate::chat::calls::heard(&f.hub, &lead, "c1", "local").unwrap();
call_started(&f.hub, &lead, Some("c1")).unwrap();
until(|| f.hub.runtime(&lead).status == BotStatus::Idle && !f.agent_texts(&lead).is_empty(), "the lead opens")
.await;
diff --git a/host/src/hub.rs b/host/src/hub.rs
index 017706fe..68415247 100644
--- a/host/src/hub.rs
+++ b/host/src/hub.rs
@@ -153,6 +153,7 @@ impl Hub {
self.spawn_bot(row.config);
}
}
+ crate::chat::calls::rewatch(self);
crate::routines::start(self);
crate::analytics::start(self);
Ok(())
diff --git a/host/tests/calls_e2e.rs b/host/tests/calls_e2e.rs
index 860540bc..427f1389 100644
--- a/host/tests/calls_e2e.rs
+++ b/host/tests/calls_e2e.rs
@@ -31,6 +31,18 @@ fn reply(e: &Value) -> bool {
e["kind"] == "agent" && e["data"]["final"] == true
}
+/// The next thing the computer tells call `call`'s voice, of `kind`.
+async fn cue(events: &mut tokio::sync::mpsc::UnboundedReceiver<Value>, call: &str, kind: &str) -> String {
+ let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
+ loop {
+ let event =
+ tokio::time::timeout_at(deadline, events.recv()).await.expect("no cue in time").expect("stream ended");
+ if event["type"] == "call" && event["call"] == call && event["kind"] == kind {
+ return event["text"].as_str().unwrap().to_owned();
+ }
+ }
+}
+
async fn tool(host: &Host, call: &str, name: &str, arguments: Value) -> Value {
host.call("callTool", json!({"callId": call, "name": name, "arguments": arguments})).await["result"].clone()
}
@@ -43,6 +55,7 @@ async fn replies_are_the_calls_while_it_lasts() {
let bot = json!({"name": "Mika", "backend": "custom", "command": common::python_agent(&agent), "permission": "ask",
"connectors": [], "notify": false});
let bot = host.call("createBot", bot).await["bot"]["id"].as_str().unwrap().to_owned();
+ let mut events = host.events().await;
host.call("startCall", json!({"botId": bot, "callId": "c1", "mode": "device"})).await;
// The user's words the app hadn't sent yet ride along with the hand-off.
@@ -54,6 +67,9 @@ async fn replies_are_the_calls_while_it_lasts() {
assert!(relayed["data"]["text"].as_str().unwrap().contains("«改一下 a.txt»"), "{relayed}");
wait_for(&host, &bot, 0, pending).await;
+ // The voice hears of the approval from the computer, worded for an on-device call's listener.
+ let approval = cue(&mut events, "c1", "notice").await;
+ assert_eq!(approval, "Mika needs your approval in the chat: Edit a.txt");
// While it waits: a message said on the call and a typed one, answered together.
host.call("send", json!({"botId": bot, "text": "and b.txt", "clientNonce": "n1", "call": "c1"})).await;
host.call("send", json!({"botId": bot, "text": "typed", "clientNonce": "n2"})).await;
@@ -62,6 +78,8 @@ async fn replies_are_the_calls_while_it_lasts() {
assert_eq!(tool(&host, "c1", "answer_approval", json!({"option": "allow"})).await["answered"], "Allow");
let first = wait_for(&host, &bot, 0, reply).await;
assert_eq!(first["data"]["call"], "c1", "a reply to the call is the call's: {first}");
+ let said = cue(&mut events, "c1", "reply").await;
+ assert_eq!(said, first["data"]["text"].as_str().unwrap().trim(), "the voice hears the reply");
wait_for(&host, &bot, 0, pending).await;
assert_eq!(tool(&host, "c1", "answer_approval", json!({"option": "ALLOW"})).await["answered"], "Allow");
diff --git a/web/content/docs/voice.mdx b/web/content/docs/voice.mdx
index 833e1cc1..52c64b25 100644
--- a/web/content/docs/voice.mdx
+++ b/web/content/docs/voice.mdx
@@ -50,7 +50,8 @@ Under **Engine**, pick one. The choice is kept on this device. Changed during a
- You speak in the language set under **Language** (by default, your device's first language).
- A pause ends what you said and sends it to the bot as part of the call. If the bot is busy, it's queued like any other message. Set the pause under **Send after a pause of** (1 s, 1.5 s or 2.5 s).
- Each reply is read aloud in the language it's written in, skipping code blocks and formatting. Set **Reading speed** to Slower, Normal or Faster. The microphone pauses while a reply is read.
-- When the bot asks for approval, the call tells you; answer it in the chat.
+- When the bot asks for approval, the call tells you what it asks; answer it in the chat. If it runs into a problem, the call tells you that too.
+- A reply that arrives while the call briefly loses its connection is read once it's back.
In the desktop app, on-device speech is available on macOS only. On Windows and Linux, calls use OpenAI or Gemini; without a key, the call asks you to add one in call settings.
@@ -61,7 +62,7 @@ Each provider has two modes, under **Mode**:
- **Realtime:** a live conversation with the provider's realtime model. You can talk over replies.
- **Speech:** the same loop as On device, using the provider's speech-to-text and text-to-speech models instead. What you say is sent when you pause, and replies are read aloud.
-In Realtime mode, the voice model is the bot's voice line, not the bot: it talks with you, and the bot does the work. It passes your requests on to the bot (along with your own words, so nothing gets lost in its wording), reads the bot's replies out in shortened form, tells you what the bot is doing ("what are you working on?"), answers an approval for you once you've said which option, can stop the bot, and can look up what the bot remembers.
+In Realtime mode, the voice model is the bot's voice line, not the bot: it talks with you, and the bot does the work. It passes your requests on to the bot (along with your own words, so nothing gets lost in its wording), reads the bot's replies out in shortened form, keeps you posted while the bot works (and tells you what it's doing when you ask), lets you know when an approval is waiting, answers an approval for you once you've said which option, can stop the bot, and can look up what the bot remembers.
Before a Realtime call, your computer gives the voice model the background it needs: the date and time, what your bots remember about you, this bot's settings and memory, the project's git branch and recent changes, what the bot is working on, the recent messages and calls in this chat, your other bots, and the files shared in the chat. Like your audio, it goes to OpenAI or Google on your own key.