Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

feat(android): cloud voice calls and the remote screen (#152)

leepokai277caba0f116
diff --git a/apps/android/README.md b/apps/android/README.md
index 44559c82..afddf46b 100644
--- a/apps/android/README.md
+++ b/apps/android/README.md
@@ -146,7 +146,7 @@ That speech prototype does not establish full voice parity.
AndroidX, Compose, CameraX, Glance, WorkManager, Firebase and desugaring artifacts
come from Google Maven. Other dependencies use the configured repositories.
AndroidX native libraries, Vosk and JNA are upstream artifacts; no custom native
-crypto or camera library is introduced. WebRTC artifact selection remains open.
+crypto or camera library is introduced. WebRTC comes from the webrtc-sdk build of libwebrtc.
| Dependency | Version | Source/license and role |
| --- | --- | --- |
@@ -171,6 +171,7 @@ crypto or camera library is introduced. WebRTC artifact selection remains open.
| xterm / fit / web-links | 6.0.0 / 0.11.0 / 0.12.0 | [xterm.js](https://github.com/xtermjs/xterm.js/tree/6.0.0), MIT; bundled terminal rendering and explicit HTTP links, with each license included in the assets |
| Vosk Android / English model | 0.3.75 / small-en-us-0.15 | [Vosk](https://github.com/alphacep/vosk-api), Apache-2.0; bundled offline recognition and checksum-pinned model |
| JNA | 5.18.1 | [Java Native Access](https://github.com/java-native-access/jna/tree/5.18.1), Apache-2.0 option with notice; native speech binding |
+| WebRTC (webrtc-sdk) | 125.6422.07 | [webrtc-sdk/android](https://github.com/webrtc-sdk/webrtc), BSD-3-Clause; remote screen video and input channels, OpenAI Realtime voice |
| Kyant backdrop / shapes | 2.0.1 / 1.2.1 | [AndroidLiquidGlass](https://github.com/Kyant0/AndroidLiquidGlass), Apache-2.0; glass blur, lens and highlight (RenderEffect API 31+, RuntimeShader API 33+), continuous-corner shapes |
| Inter | 4.x variable | [Inter](https://rsms.me/inter/), OFL-1.1; the app's type face in `design/src/main/res/font` |
| Material Symbols Rounded | upstream SVGs | [Material Design Icons](https://github.com/google/material-design-icons), Apache-2.0; vector stand-ins for SF Symbols |
diff --git a/apps/android/app/build.gradle.kts b/apps/android/app/build.gradle.kts
index ec71b412..e795917f 100644
--- a/apps/android/app/build.gradle.kts
+++ b/apps/android/app/build.gradle.kts
@@ -160,6 +160,8 @@ dependencies {
implementation(libs.firebase.messaging)
implementation(libs.work)
implementation(libs.glance)
+ // Remote screen video and OpenAI Realtime voice, as the iPhone app's WebRTC.
+ implementation(libs.webrtc)
implementation(libs.camera.core)
implementation(libs.camera.camera2)
implementation(libs.camera.lifecycle)
diff --git a/apps/android/app/src/androidTest/kotlin/com/codync/android/NativeVoiceTest.kt b/apps/android/app/src/androidTest/kotlin/com/codync/android/NativeVoiceTest.kt
index ca825298..3b2004a9 100644
--- a/apps/android/app/src/androidTest/kotlin/com/codync/android/NativeVoiceTest.kt
+++ b/apps/android/app/src/androidTest/kotlin/com/codync/android/NativeVoiceTest.kt
@@ -40,7 +40,7 @@ class NativeVoiceTest {
val samples = requireNotNull(pcm)
val text = withContext(Dispatchers.IO) {
LibVosk.vosk_set_log_level(-1)
- val directory = VoiceModels.english(instrumentation.targetContext)
+ val directory = VoskModels.english(instrumentation.targetContext)
Model(directory.absolutePath).use { model -> Recognizer(model, 16_000f).use { recognizer ->
val results = mutableListOf<String>(); var position = 0
while (position < samples.size) {
diff --git a/apps/android/app/src/main/AndroidManifest.xml b/apps/android/app/src/main/AndroidManifest.xml
index 85f02e8a..70aa32ea 100644
--- a/apps/android/app/src/main/AndroidManifest.xml
+++ b/apps/android/app/src/main/AndroidManifest.xml
@@ -55,7 +55,8 @@
<!-- Direct ws:// channels already encrypt every inner request with Codync E2E.
Core accepts HTTP only for validated direct candidates; cloud remains HTTPS. -->
<activity android:name=".MainActivity" android:exported="true" android:launchMode="singleTop"
- android:windowSoftInputMode="adjustResize">
+ android:windowSoftInputMode="adjustResize"
+ android:configChanges="orientation|screenSize|screenLayout|smallestScreenSize|keyboard|keyboardHidden|navigation">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
@@ -84,6 +85,12 @@
<category android:name="android.intent.category.BROWSABLE" />
<data android:scheme="${urlScheme}" android:host="usage" />
</intent-filter>
+ <intent-filter>
+ <action android:name="android.intent.action.VIEW" />
+ <category android:name="android.intent.category.DEFAULT" />
+ <category android:name="android.intent.category.BROWSABLE" />
+ <data android:scheme="${urlScheme}" android:host="screen" />
+ </intent-filter>
</activity>
</application>
</manifest>
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppState.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppState.kt
index 5e61fec7..cf5236f0 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppState.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppState.kt
@@ -39,6 +39,10 @@ data class AppState(
val routines: Map<String, RoutineListing> = emptyMap(),
val plugins: PluginListing? = null,
val voice: VoiceCallViewState? = null,
+ /** The computer's remote screen (`null`: the host predates it). */
+ val screen: ScreenState? = null,
+ /** The remote screen viewer is open (`codync://screen`, the thread header, Computers). */
+ val screenRequest: ScreenRequest? = null,
/** The host sent data this app can't read even after fetching everything again. */
val unreadable: Boolean = false,
val error: String? = null,
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
index 25b3914c..6da7eb24 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStore.kt
@@ -24,6 +24,7 @@ class AppStore(application: Application) : AndroidViewModel(application) {
private var contextJob = SupervisorJob(viewModelScope.coroutineContext[Job])
internal var contextScope = CoroutineScope(contextJob + Dispatchers.Main.immediate)
internal val transitions = Mutex()
+ internal val screenTakeovers = Mutex()
internal var cloud: CloudClient? = null
private var sessionJob: Job? = null
internal var linkJob: Job? = null
@@ -110,11 +111,7 @@ class AppStore(application: Application) : AndroidViewModel(application) {
widgetOwner = null
WidgetRefreshWorker.stop(getApplication())
try { CodyncWidgets.update(getApplication()) } catch (_: Exception) { }
- val call = voiceBinding; voiceBinding = null; voiceGeneration = null
- val callId = state.value.voice?.id
- mutableState.update { it.copy(voice = null) }
- if (callId != null) VoiceService.stop(getApplication(), callId)
- call?.session?.end(); call?.session?.awaitEnd()
+ retireVoice()
val writers = actions.values.toList()
writers.forEach { it.cancel() }; writers.forEach { it.join() }
PushDelivery.retire(getApplication())
@@ -179,13 +176,12 @@ class AppStore(application: Application) : AndroidViewModel(application) {
val retainedDraftFiles = state.value.drafts.values.flatMap { it.files }.map { it.path }.toSet()
val retired = state.value.pending.filter { old -> snapshot.pending.none { it.nonce == old.nonce } }.flatMap { it.files }.filter { it.path !in retainedDraftFiles }
withContext(Dispatchers.IO) { retired.forEach(attachments::discard) }
+ // Like iOS, a screen state this app can't read is a host without remote screen.
+ val screen = snapshot.screen?.let { runCatching { ScreenState.decode(it) }.getOrNull() }
mutableState.update { current -> current.copy(bots = snapshot.bots, entries = snapshot.entries,
- pending = snapshot.pending, usage = snapshot.usage, mirrorRevision = snapshot.rev,
+ pending = snapshot.pending, usage = snapshot.usage, screen = screen, mirrorRevision = snapshot.rev,
selectedBot = current.selectedBot?.takeIf { id -> snapshot.bots.any { it.id == id } }) }
- voiceBinding?.let { call ->
- if (snapshot.bots.none { it.id == call.botId && !it.deleted }) endVoice("Voice chat ended because its bot was removed.")
- else for (reply in call.replies.observe(snapshot.entries)) call.session.speak(reply)
- }
+ speakToCall(snapshot.entries, snapshot.bots)
if (PushPreferences.tasks(getApplication())) {
val feed = TaskFeed(local.directory)
val watched = withContext(Dispatchers.IO) { feed.all().filter { !it.finished }.map { it.botId }.toSet() }
@@ -245,7 +241,7 @@ class AppStore(application: Application) : AndroidViewModel(application) {
override fun onCleared() {
state.value.voice?.id?.let { VoiceService.stop(getApplication(), it) }
- voiceBinding?.session?.end()
+ voiceBinding?.engine?.end()
channel?.close()
val old = database
val retiring = contextJob
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreNotifications.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreNotifications.kt
index 3618b53f..aa3da0dd 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreNotifications.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreNotifications.kt
@@ -28,13 +28,16 @@ fun AppStore.taskNotifications(enabled: Boolean) = action("taskSettings") {
if (!enabled) NotificationDisplay.cancel(getApplication(), 2)
syncPush(); PushDelivery.schedule(getApplication())
}
+/** The state once the signed-in (or local) account has loaded: where a link can be opened. */
+internal suspend fun AppStore.activated(): AppState = state.first { current -> !current.loading &&
+ (current.account.ready || !current.account.configured || current.account.error != null) &&
+ current.contextId == LocalStore.referenceId(current.account.userId ?: "local") }
+
fun AppStore.openLink(uri: Uri) {
linkJob?.cancel()
linkJob = viewModelScope.launch {
val completed = withTimeoutOrNull(30_000) {
- val activated = state.first { current -> !current.loading &&
- (current.account.ready || !current.account.configured || current.account.error != null) &&
- current.contextId == LocalStore.referenceId(current.account.userId ?: "local") }
+ val activated = activated()
val usage = uri.host == "usage"
val destination = try {
if (usage) { val target = UsageDestination.parse(uri.toString(), BuildConfig.URL_SCHEME); BotDestination(target.scope, target.computerId, "") }
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreScreen.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreScreen.kt
new file mode 100644
index 00000000..de5596a9
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreScreen.kt
@@ -0,0 +1,46 @@
+package com.codync.android
+
+import androidx.lifecycle.viewModelScope
+import com.codync.android.core.*
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.flow.update
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.sync.withLock
+import kotlinx.coroutines.withContext
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.withTimeoutOrNull
+import kotlinx.serialization.json.buildJsonObject
+import kotlinx.serialization.json.jsonObject
+import kotlinx.serialization.json.put
+
+// The remote screen viewer: opening it, and taking control from bots.
+
+/** Opens the screen viewer, optionally watching a bot that's using the computer. */
+data class ScreenRequest(val watching: String? = null)
+
+fun AppStore.openScreen(watching: String? = null) = mutableState.update { it.copy(screenRequest = ScreenRequest(watching)) }
+
+fun AppStore.closeScreen() = mutableState.update { it.copy(screenRequest = null) }
+
+/** `codync://screen`: the paired computer's screen, once the account has loaded. */
+fun AppStore.openScreenLink() {
+ linkJob?.cancel()
+ linkJob = viewModelScope.launch {
+ val ready = withTimeoutOrNull(30_000) { activated() }
+ if (ready?.computers?.isNotEmpty() == true) openScreen()
+ }
+}
+
+/** Takes control from bots (they can still look) or hands it back; requests go out in the order they're made. */
+fun AppStore.screenTakeover(on: Boolean) {
+ contextScope.launch {
+ screenTakeovers.withLock {
+ try {
+ val state = ready().call("screenTakeover", buildJsonObject { put("on", on) }).jsonObject
+ withContext(Dispatchers.IO) { database?.apply(MirrorEvent.Screen(state)) }
+ refresh()
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { showError(error) }
+ }
+ }
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
index 6d8d653b..de2de196 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/AppStoreVoice.kt
@@ -1,61 +1,160 @@
package com.codync.android
+import android.os.SystemClock
+import java.io.IOException
+import java.util.Base64
import java.util.UUID
import com.codync.android.core.*
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.*
import kotlinx.serialization.json.*
+import okhttp3.OkHttpClient
-// A voice call with one bot: what you say is sent, its replies are read aloud.
+// A voice call with one bot: what you say is sent, its replies are read aloud. The engine is the one
+// picked in the call settings; a cloud engine that can't start or drops mid-call hands over to on-device voice.
+
+/** The call in progress: its engine (replaced on a fallback), start times and which replies it has read. */
+internal class VoiceBinding(val id: String, val botId: String, val generation: Int, val startedAt: Long, val replies: VoiceReplyGate) {
+ var engine: VoiceEngine? = null
+ /** When the current cloud engine started, for this month's minutes. */
+ var cloudSince: Long? = null
+ /** The microphone's audio session, so Android silencing it pauses the call. */
+ var recording: () -> Int? = { null }
+ /** What the voice service sees: the current engine, never the failure it fell back from. */
+ val state = MutableStateFlow(VoiceState())
+}
+
+private val voiceHttp by lazy { OkHttpClient() }
+
+internal fun AppStore.voiceChoices(): VoiceChoices = VoicePreferences.choices(getApplication())
-/** The call in progress and what it needs: its session, start time and which replies it has read. */
-internal data class VoiceBinding(val id: String, val botId: String, val generation: Int,
- val startedAt: Long, val session: VoiceSession, val replies: VoiceReplyGate)
fun AppStore.startVoice(botId: String, expectedGeneration: Int = state.value.generation) {
if (state.value.generation != expectedGeneration || state.value.loading || state.value.voice != null) return
- if (!foreground || state.value.actualConnection !is LinkState.Ready || state.value.bots.none { it.id == botId && !it.deleted && it.kind != "group" }) {
+ if (!foreground || state.value.actualConnection !is LinkState.Ready || state.value.bots.none { it.id == botId && !it.deleted }) {
mutableState.update { it.copy(error = "Open the connected bot's conversation to start voice chat.") }; return
}
val id = UUID.randomUUID().toString()
voiceGeneration = expectedGeneration
- mutableState.update { it.copy(voice = VoiceCallViewState(id, botId, VoiceState(settings = VoicePreferences.load(getApplication())))) }
+ val starting = VoiceState(settings = VoicePreferences.load(getApplication()), provider = voiceChoices().provider)
+ mutableState.update { it.copy(voice = VoiceCallViewState(id, botId, starting)) }
try { VoiceService.start(getApplication(), this, id, botId); PushDelivery.foreground = true }
catch (_: Exception) { finishVoice(id, "Couldn't start voice chat. Allow microphone access and wait for any previous call to finish.") }
}
-internal fun AppStore.attachVoice(id: String, botId: String, audio: VoiceAudio): VoiceSession? {
+
+/** Starts the call's engine for the voice service; null when the call is gone. */
+internal fun AppStore.attachVoice(id: String, botId: String): VoiceBinding? {
val call = state.value.voice ?: return null
if (call.id != id || call.botId != botId || voiceGeneration != state.value.generation || state.value.loading || voiceBinding != null) return null
- val generation = state.value.generation
val entries = state.value.entries
val head = maxOf(state.value.hello?.get("rev")?.jsonPrimitive?.longOrNull ?: 0, entries.maxOfOrNull(Entry::rev) ?: 0)
- val session = VoiceSession(contextScope, audio, call.state.settings) { text ->
- if (state.value.generation == generation && state.value.voice?.id == id && !state.value.loading) {
- val send = PendingSend(UUID.randomUUID().toString(), botId, null, text, System.currentTimeMillis())
- action("save:" + send.nonce) { savePending(send); deliver(send) }
+ val binding = VoiceBinding(id, botId, state.value.generation, SystemClock.elapsedRealtime(), VoiceReplyGate(botId, head, entries))
+ voiceBinding = binding
+ begin(binding, voiceChoices().provider)
+ connect()
+ return binding
+}
+
+/** Starts an engine: a provider on the key kept by the computer, or on-device speech. */
+private fun AppStore.begin(call: VoiceBinding, provider: VoiceProvider?) {
+ val choices = voiceChoices()
+ // Picks up the computer's newest models for the next call; this one starts right away.
+ if (provider != null) contextScope.launch { try { choices.remember(voiceStatus()) } catch (_: Exception) { } }
+ val settings = VoicePreferences.load(getApplication())
+ val operator = CallOperator(call.botId, object : CallTarget {
+ override val bots get() = state.value.bots
+ override val entries get() = state.value.entries
+ override fun send(text: String) = sendFromCall(call, text)
+ override fun respond(entry: Entry, optionId: String) = [email protected](entry, optionId)
+ })
+ call.recording = { null }
+ val engine: VoiceEngine = if (provider == null) {
+ val audio = LocalVoiceAudio(getApplication())
+ call.recording = { audio.recordingSessionId }
+ VoiceSession(contextScope, audio, settings) { sendFromCall(call, it) }
+ } else {
+ val voice = choices.voice(provider)
+ val model = choices.model(provider, VoiceChoices.Model.Realtime)
+ val credential: suspend () -> String = { voiceSession(provider, model, voice) }
+ fun mic() = AndroidMic(getApplication()).also { mic -> call.recording = { mic.recordingSessionId } }
+ when {
+ choices.speechMode(provider) -> {
+ val transcribe = choices.model(provider, VoiceChoices.Model.Transcribe)
+ val speech = choices.model(provider, VoiceChoices.Model.Speech)
+ CloudSpeechEngine(contextScope, provider, settings, mic(), AndroidClipPlayer, CloudSpeechEngine.Speech(
+ transcribe = { inReadersScript(voiceTranscribe(provider, transcribe, it)) },
+ synthesize = { voiceSpeak(provider, speech, voice, it) },
+ send = { sendFromCall(call, it) }))
+ }
+ // WebRTC records the microphone itself.
+ provider == VoiceProvider.OpenAI -> OpenAIRealtimeEngine(contextScope, settings, credential, operator, WebRtcPeer(getApplication())) { secret, offer ->
+ OpenAIRealtimeEngine.answer(voiceHttp, secret, offer)
+ }
+ else -> GeminiLiveEngine(contextScope, settings, model, voice, credential, operator, mic(), AndroidPcmPlayer(), voiceHttp)
}
}
- voiceBinding = VoiceBinding(id, botId, generation, android.os.SystemClock.elapsedRealtime(), session, VoiceReplyGate(botId, head, entries))
- contextScope.launch { session.state.takeWhile { it.phase != VoicePhase.Ended }.collect { status ->
- mutableState.update { current -> if (current.voice?.id == id) current.copy(voice = current.voice.copy(state = status)) else current }
- } }
- connect()
- return session
+ call.engine = engine
+ call.cloudSince = if (provider != null) SystemClock.elapsedRealtime() else null
+ contextScope.launch {
+ val last = engine.state.first { status ->
+ val current = voiceBinding === call && call.engine === engine
+ val fallingBack = provider != null && status.phase == VoicePhase.Failed
+ if (current && !fallingBack) publish(call, status)
+ !current || fallingBack || status.phase == VoicePhase.Ended
+ }
+ if (provider == null || last.phase != VoicePhase.Failed || voiceBinding !== call) return@launch
+ countCloudMinutes(call)
+ val notice = "${provider.title} isn't available (${last.error ?: "no answer"}). Using on-device voice."
+ mutableState.update { it.copy(voice = it.voice?.takeIf { voice -> voice.id == call.id }?.copy(notice = notice) ?: it.voice) }
+ engine.end(); engine.awaitEnd()
+ if (voiceBinding === call) begin(call, null)
+ }
+}
+
+private fun AppStore.publish(call: VoiceBinding, status: VoiceState) {
+ call.state.value = status
+ mutableState.update { current -> if (current.voice?.id == call.id) current.copy(voice = current.voice.copy(state = status)) else current }
}
+
+private fun AppStore.countCloudMinutes(call: VoiceBinding) {
+ val provider = call.engine?.state?.value?.provider ?: return
+ val since = call.cloudSince ?: return
+ call.cloudSince = null
+ voiceChoices().addUsage(provider, (SystemClock.elapsedRealtime() - since) / 1_000)
+}
+
+/** What you said goes out as an ordinary message, unless the call has moved on. */
+private fun AppStore.sendFromCall(call: VoiceBinding, text: String) {
+ if (state.value.generation != call.generation || state.value.voice?.id != call.id || state.value.loading) return
+ val send = PendingSend(UUID.randomUUID().toString(), call.botId, null, text, System.currentTimeMillis())
+ action("save:" + send.nonce) { savePending(send); deliver(send) }
+}
+
+/** New final replies and approval notices for the call to say. */
+internal fun AppStore.speakToCall(entries: List<Entry>, bots: List<Bot>) {
+ val call = voiceBinding ?: return
+ if (bots.none { it.id == call.botId && !it.deleted }) { endVoice("Voice chat ended because its bot was removed."); return }
+ val engine = call.engine
+ for (cue in call.replies.observe(entries, bots)) if (cue.notice) engine?.announce(cue.text) else engine?.speak(cue.text)
+}
+
fun AppStore.endVoice(error: String? = null) {
val id = state.value.voice?.id ?: return
finishVoice(id, error); VoiceService.stop(getApplication(), id)
}
+
internal fun AppStore.finishVoice(id: String, error: String? = null) {
if (state.value.voice?.id != id) return
val call = voiceBinding?.takeIf { it.id == id }; voiceBinding = null; voiceGeneration = null
- call?.session?.end()
+ val engine = call?.engine
+ engine?.end()
+ if (call != null) countCloudMinutes(call)
mutableState.update { it.copy(voice = null, error = error ?: it.error) }
PushDelivery.foreground = foreground
contextScope.launch {
try {
- call?.session?.awaitEnd()
+ engine?.awaitEnd()
if (call != null && state.value.generation == call.generation && channel?.state?.value is LinkState.Ready) {
- val seconds = ((android.os.SystemClock.elapsedRealtime() - call.startedAt) / 1_000).coerceAtLeast(0)
+ val seconds = ((SystemClock.elapsedRealtime() - call.startedAt) / 1_000).coerceAtLeast(0)
withTimeoutOrNull(3_000) { hostQuery("logCall", buildJsonObject { put("botId", call.botId); put("seconds", seconds) }, 3_000, call.generation) }
}
} catch (cancelled: CancellationException) { throw cancelled }
@@ -63,7 +162,43 @@ internal fun AppStore.finishVoice(id: String, error: String? = null) {
finally { if (!transportRequired) { connectionJob?.cancel(); graceJob?.cancel(); channel?.close() } }
}
}
-fun AppStore.muteVoice() { voiceBinding?.session?.let { it.mute(!it.state.value.muted) } }
-fun AppStore.interruptVoice() { voiceBinding?.session?.interrupt() }
-fun AppStore.resumeVoice() { voiceBinding?.session?.focus(true) }
-fun AppStore.voiceSettings(settings: VoiceSettings) { VoicePreferences.save(getApplication(), settings); voiceBinding?.session?.settings(settings) }
+
+/** Ends the engine of a retiring account and waits for its audio to close. */
+internal suspend fun AppStore.retireVoice() {
+ val call = voiceBinding; voiceBinding = null; voiceGeneration = null
+ val callId = state.value.voice?.id
+ mutableState.update { it.copy(voice = null) }
+ if (callId != null) VoiceService.stop(getApplication(), callId)
+ call?.engine?.end(); call?.engine?.awaitEnd()
+}
+
+fun AppStore.muteVoice() { voiceBinding?.engine?.let { it.mute(!it.state.value.muted) } }
+fun AppStore.interruptVoice() { voiceBinding?.engine?.interrupt() }
+fun AppStore.resumeVoice() { voiceBinding?.engine?.focus(true) }
+internal fun AppStore.focusVoice(active: Boolean) { voiceBinding?.engine?.focus(active) }
+internal fun AppStore.voiceRecordingSession(): Int? = voiceBinding?.recording?.invoke()
+fun AppStore.voiceSettings(settings: VoiceSettings) { VoicePreferences.save(getApplication(), settings); voiceBinding?.engine?.settings(settings) }
+
+// The computer's voice methods: it holds the provider keys and does the provider calls that need them.
+
+internal suspend fun AppStore.voiceStatus(): VoiceStatus = WireJson.decodeFromJsonElement(hostCall("voiceStatus", JsonObject(emptyMap()), 30_000))
+internal suspend fun AppStore.voiceModels(provider: VoiceProvider): VoiceModels =
+ WireJson.decodeFromJsonElement(hostCall("voiceModels", body("provider" to provider.id), 30_000))
+
+/** A short-lived credential for one call's audio session. */
+internal suspend fun AppStore.voiceSession(provider: VoiceProvider, model: String, voice: String): String =
+ hostCall("voiceSession", body("provider" to provider.id, "model" to model, "voice" to voice), 30_000).jsonObject["credential"]
+ ?.jsonPrimitive?.contentOrNull ?: throw IOException("${provider.title} didn't start the call.")
+
+/** Speech to text for one utterance (a WAV file), with the computer's key. */
+internal suspend fun AppStore.voiceTranscribe(provider: VoiceProvider, model: String, wav: ByteArray): String =
+ hostCall("voiceTranscribe", body("provider" to provider.id, "model" to model, "audio" to Base64.getEncoder().encodeToString(wav)), 60_000)
+ .jsonObject["text"]?.jsonPrimitive?.contentOrNull.orEmpty()
+
+/** Text to speech for a stretch of a reply: the audio and its format (`aac` or `wav`). */
+internal suspend fun AppStore.voiceSpeak(provider: VoiceProvider, model: String, voice: String, text: String): Pair<ByteArray, String> {
+ val clip = hostCall("voiceSpeak", body("provider" to provider.id, "model" to model, "voice" to voice, "text" to text), 90_000).jsonObject
+ val audio = clip["audio"]?.jsonPrimitive?.contentOrNull?.let { runCatching { Base64.getDecoder().decode(it) }.getOrNull() }
+ ?: throw IOException("Unreadable audio.")
+ return audio to (clip["format"]?.jsonPrimitive?.contentOrNull ?: "aac")
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/BotEditor.kt b/apps/android/app/src/main/kotlin/com/codync/android/BotEditor.kt
index ec5243ca..e74a1e9f 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/BotEditor.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/BotEditor.kt
@@ -27,7 +27,6 @@ import com.codync.android.core.LinkState
import com.codync.android.design.*
import com.kyant.shapes.RoundedRectangle
import kotlinx.serialization.json.JsonArray
-import kotlinx.serialization.json.JsonNull
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.booleanOrNull
import kotlinx.serialization.json.contentOrNull
@@ -79,7 +78,7 @@ private val JsonObject.curated get() = this["curated"]?.jsonPrimitive?.booleanOr
/** Bot settings: a big avatar, then sections in the Settings style (`CardSection`). */
@Composable private fun BotSettingsForm(state: AppState, store: AppStore, draft: BotDraft, change: (BotDraft) -> Unit, chooseFolder: () -> Unit) {
val online = state.actualConnection is LinkState.Ready
- val screen = state.hello?.get("screen")?.takeUnless { it == JsonNull } as? JsonObject
+ val screen = state.screen
Column(Modifier.fillMaxSize().verticalScroll(rememberScrollState()).padding(20.dp), verticalArrangement = Arrangement.spacedBy(30.dp)) {
Column(Modifier.fillMaxWidth(), horizontalAlignment = Alignment.CenterHorizontally, verticalArrangement = Arrangement.spacedBy(16.dp)) {
Box(Modifier.clip(RoundedRectangle(24.dp)).background(Palette.bubbleAgent).padding(18.dp)) {
@@ -154,9 +153,12 @@ private val JsonObject.curated get() = this["curated"]?.jsonPrimitive?.booleanOr
CardSection("Activity") {
row { SwitchRow("Notifications", "Get notified when this bot finishes or needs you", draft.notify != false, { change(draft.copy(notify = it)) }) }
if (screen != null) row {
- SwitchRow("Use the computer", if (screen["enabled"]?.jsonPrimitive?.booleanOrNull == true)
- "Let this bot see the screen and use the mouse and keyboard. You can watch and take over from your phone."
- else "Let this bot see the screen and use the mouse and keyboard. On that computer, open Codync → Settings → Computer access to set up permissions first.",
+ SwitchRow("Use the computer", when {
+ !screen.controlReady -> "Let this bot use apps. On that computer, open Codync → Settings → Computer access to set up permissions first."
+ screen.enabled -> "Let this bot see the screen and use the mouse and keyboard. You can watch and take over from your phone."
+ screen.computerUse -> "Let this bot see the screen and use the mouse and keyboard."
+ else -> "Let this bot see the screen and use the mouse and keyboard. On that computer, open Codync → Settings → Computer access to set up permissions first."
+ },
draft.computer, { change(draft.copy(computer = it)) })
}
}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ChatScreen.kt b/apps/android/app/src/main/kotlin/com/codync/android/ChatScreen.kt
index 1979a156..3b132caa 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/ChatScreen.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ChatScreen.kt
@@ -1,11 +1,17 @@
package com.codync.android
import androidx.compose.animation.AnimatedVisibility
+import androidx.compose.animation.core.RepeatMode
+import androidx.compose.animation.core.animateFloat
+import androidx.compose.animation.core.infiniteRepeatable
+import androidx.compose.animation.core.rememberInfiniteTransition
+import androidx.compose.animation.core.tween
import androidx.compose.animation.fadeIn
import androidx.compose.animation.fadeOut
import androidx.compose.animation.slideInVertically
import androidx.compose.animation.slideOutVertically
import androidx.compose.foundation.background
+import androidx.compose.foundation.clickable
import androidx.compose.foundation.layout.Arrangement
import androidx.compose.foundation.layout.Box
import androidx.compose.foundation.layout.Column
@@ -47,12 +53,14 @@ import androidx.compose.ui.Modifier
import androidx.compose.ui.geometry.Offset
import androidx.compose.ui.graphics.Brush
import androidx.compose.ui.graphics.Color
+import androidx.compose.ui.graphics.graphicsLayer
import androidx.compose.ui.input.nestedscroll.NestedScrollConnection
import androidx.compose.ui.input.nestedscroll.NestedScrollSource
import androidx.compose.ui.input.nestedscroll.nestedScroll
import androidx.compose.ui.layout.onSizeChanged
import androidx.compose.ui.platform.LocalContext
import androidx.compose.ui.platform.LocalDensity
+import androidx.compose.ui.semantics.Role
import androidx.compose.ui.semantics.contentDescription
import androidx.compose.ui.semantics.semantics
import androidx.compose.ui.text.style.TextOverflow
@@ -173,6 +181,23 @@ import kotlinx.coroutines.launch
ConnectionSubtitle(state)
}
}
+ // The corner is the computer: it pulses while this bot is operating it (watch, then take over).
+ val screen = state.screen
+ if (screen != null) {
+ val operating = screen.agentBot == bot.id
+ ComputerButton(operating, Modifier.align(Alignment.CenterEnd)) { store.openScreen(bot.id.takeIf { operating }) }
+ }
+ }
+}
+
+@Composable private fun ComputerButton(operating: Boolean, modifier: Modifier, action: () -> Unit) {
+ val (source, pressed) = rememberPress()
+ val pulse by rememberInfiniteTransition(label = "Operating").animateFloat(1f, 0.35f,
+ infiniteRepeatable(tween(900), RepeatMode.Reverse), label = "Pulse")
+ Box(modifier.size(44.dp).glass(Capsule(), pressed = pressed)
+ .clickable(source, null, role = Role.Button, onClick = action).semantics { contentDescription = "Computer" },
+ contentAlignment = Alignment.Center) {
+ Symbol("desktopcomputer", Modifier.graphicsLayer { alpha = if (operating) pulse else 1f }, size = 22.dp, tint = Palette.text)
}
}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/CodyncApp.kt b/apps/android/app/src/main/kotlin/com/codync/android/CodyncApp.kt
index 8ed4c325..e91db962 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/CodyncApp.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/CodyncApp.kt
@@ -2,6 +2,8 @@ package com.codync.android
import androidx.activity.compose.BackHandler
import androidx.compose.animation.AnimatedContent
+import androidx.compose.animation.AnimatedVisibility
+import androidx.compose.animation.core.MutableTransitionState
import androidx.compose.animation.fadeIn
import androidx.compose.animation.fadeOut
import androidx.compose.animation.slideInHorizontally
@@ -111,11 +113,26 @@ private enum class Root { Loading, Welcome, Pairing, Tabs }
CodyncSheet(editor.takeIf { !state.loading }, { editor = null }) { draft ->
key(state.contextId, state.generation, draft.id) { BotEditor(state, draft, store) }
}
+ // The remote screen opens over the app once its sheets have slid away.
+ LaunchedEffect(state.screenRequest) { if (state.screenRequest != null) { closeAll(); showUsage = false } }
+ ScreenLayer(state, store)
// One place for errors the screen showing them doesn't handle itself (pairing, access, the editors do).
val inline = root == Root.Pairing || showPairing || access != null || editor != null || groupEditor != null
CodyncDialog("Something went wrong", state.error != null && !inline, store::clearError, state.error, cancel = "OK") { emptyList() }
}
+/** The remote screen as a full-window layer that fades in and out (iOS `.codyncOverlay`). */
+@Composable private fun ScreenLayer(state: AppState, store: AppStore) {
+ val visible = remember { MutableTransitionState(false) }
+ visible.targetState = state.screenRequest != null
+ if (!visible.currentState && !visible.targetState) return
+ Overlay {
+ AnimatedVisibility(visible, enter = fadeIn(Motion.layout()), exit = fadeOut(Motion.fade())) {
+ ScreenView(state, store, state.screenRequest?.watching, store::closeScreen)
+ }
+ }
+}
+
/** The app-level analytics events (iOS reports the same): setup finished, signed in, signed out. */
@Composable private fun AnalyticsMilestones(state: AppState) {
var setup by remember { mutableStateOf<Boolean?>(null) }
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/LocalVoiceAudio.kt b/apps/android/app/src/main/kotlin/com/codync/android/LocalVoiceAudio.kt
index 6aecfcc3..7530a47e 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/LocalVoiceAudio.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/LocalVoiceAudio.kt
@@ -10,6 +10,7 @@ import android.os.Looper
import android.speech.tts.TextToSpeech
import android.speech.tts.UtteranceProgressListener
import androidx.core.content.ContextCompat
+import androidx.core.content.edit
import com.codync.android.core.*
import java.io.File
import java.io.IOException
@@ -21,26 +22,25 @@ import org.json.JSONObject
import org.vosk.LibVosk
import org.vosk.Model
import org.vosk.Recognizer
-import kotlin.math.log10
/** Offline English recognition and an installed local TTS voice. No PCM is written to disk. */
internal class LocalVoiceAudio(private val context: Context) : VoiceAudio {
- private val worker = CoroutineScope(SupervisorJob() + Dispatchers.IO)
private val main = Handler(Looper.getMainLooper())
@Volatile private var model: Model? = null
- @Volatile private var recorder: AudioRecord? = null
- private var recording: Job? = null
+ private val mic = AndroidMic(context)
+ /** Vosk, fed from the microphone's thread; closed only after that thread has stopped. */
+ private var decoder: Recognizer? = null
private var speech: TextToSpeech? = null
private var completion: ((Long, String?) -> Unit)? = null
private var speakingId: Long? = null; private var lastChunk: String? = null
- val recordingSessionId: Int? get() = recorder?.audioSessionId
+ val recordingSessionId: Int? get() = mic.recordingSessionId
override suspend fun initialize(): String {
require(ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED)
withContext(Dispatchers.IO) {
try {
LibVosk.vosk_set_log_level(-1)
- val directory = VoiceModels.english(context)
+ val directory = VoskModels.english(context)
currentCoroutineContext().ensureActive()
model = Model(directory.absolutePath)
} catch (error: LinkageError) { throw IOException("The local speech engine couldn't load.", error) }
@@ -75,55 +75,27 @@ internal class LocalVoiceAudio(private val context: Context) : VoiceAudio {
return "Offline English · Vosk"
}
override suspend fun listen(generation: Long, events: (VoiceInput) -> Unit) {
- check(recording == null)
- require(ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED)
- val installed = requireNotNull(model)
- recording = worker.launch {
- var microphone: AudioRecord? = null; var decoder: Recognizer? = null
- try {
- val format = AudioFormat.Builder().setSampleRate(16_000).setChannelMask(AudioFormat.CHANNEL_IN_MONO)
- .setEncoding(AudioFormat.ENCODING_PCM_16BIT).build()
- val minimum = AudioRecord.getMinBufferSize(16_000, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT)
- check(minimum > 0)
- microphone = AudioRecord.Builder().setAudioSource(MediaRecorder.AudioSource.VOICE_COMMUNICATION)
- .setAudioFormat(format).setBufferSizeInBytes(maxOf(minimum, 12_800)).build()
- recorder = microphone
- check(microphone.state == AudioRecord.STATE_INITIALIZED)
- decoder = Recognizer(installed, 16_000f)
- ensureActive(); microphone.startRecording()
- check(microphone.recordingState == AudioRecord.RECORDSTATE_RECORDING)
- val buffer = ShortArray(3_200); var previous = ""
- while (isActive) {
- val count = microphone.read(buffer, 0, buffer.size, AudioRecord.READ_BLOCKING)
- if (!isActive) break
- if (count < 0) throw IOException("Microphone read failed")
- if (count == 0) { delay(10); continue }
- var power = 0.0
- for (i in 0 until count) { val sample = buffer[i].toDouble() / 32_768; power += sample * sample }
- val db = 10 * log10(maxOf(power / count, 1e-10))
- events(VoiceInput.Level(generation, ((db + 50) / 50).toFloat().coerceIn(0f, 1f)))
- if (decoder.acceptWaveForm(buffer, count)) {
- val text = JSONObject(decoder.result).optString("text")
- previous = ""; if (text.isNotBlank()) events(VoiceInput.Text(generation, text, true))
- } else {
- val text = JSONObject(decoder.partialResult).optString("partial")
- if (text != previous) { previous = text; events(VoiceInput.Text(generation, text, false)) }
- }
+ check(decoder == null)
+ val recognizer = Recognizer(requireNotNull(model), Pcm.RATE.toFloat()).also { decoder = it }
+ var previous = ""
+ try {
+ mic.start({ pcm, level ->
+ events(VoiceInput.Level(generation, level))
+ if (recognizer.acceptWaveForm(pcm, pcm.size)) {
+ val text = JSONObject(recognizer.result).optString("text")
+ previous = ""; if (text.isNotBlank()) events(VoiceInput.Text(generation, text, true))
+ } else {
+ val text = JSONObject(recognizer.partialResult).optString("partial")
+ if (text != previous) { previous = text; events(VoiceInput.Text(generation, text, false)) }
}
- } catch (cancelled: CancellationException) { throw cancelled }
- catch (_: Exception) { if (isActive) events(VoiceInput.Failed(generation, "Couldn't use the microphone. Check its permission and other audio apps.")) }
- finally {
- runCatching { microphone?.stop() }; runCatching { microphone?.release() }
- if (recorder === microphone) recorder = null
- decoder?.close()
- }
+ }) { events(VoiceInput.Failed(generation, "Couldn't use the microphone. Check its permission and other audio apps.")) }
+ } catch (error: Exception) {
+ decoder = null; recognizer.close()
+ events(VoiceInput.Failed(generation, error.message ?: "Couldn't use the microphone."))
}
}
override suspend fun stopListening() {
- withContext(NonCancellable) {
- val old = recording; recording = null
- old?.cancel(); runCatching { recorder?.stop() }; old?.join()
- }
+ withContext(NonCancellable) { mic.stop(); decoder?.close(); decoder = null }
}
override suspend fun speak(id: Long, text: String, rate: Float, completed: (Long, String?) -> Unit) {
val engine = requireNotNull(speech)
@@ -138,7 +110,6 @@ internal class LocalVoiceAudio(private val context: Context) : VoiceAudio {
override suspend fun close() {
try { stopListening() }
finally {
- worker.cancel()
try { stopSpeaking() }
finally {
val engine = speech; speech = null
@@ -150,7 +121,7 @@ internal class LocalVoiceAudio(private val context: Context) : VoiceAudio {
}
/** Public linguistic weights are shared; account credentials and captured audio never enter this directory. */
-internal object VoiceModels {
+internal object VoskModels {
private val lock = Mutex()
private const val source = "voice/vosk-model-small-en-us-0.15"
suspend fun english(context: Context): File = lock.withLock {
@@ -177,13 +148,20 @@ internal object VoiceModels {
}
}
+/** Call settings on this phone: the pause and reading speed, and the cloud engine choices (`VoiceChoices`). */
internal object VoicePreferences {
+ private fun stored(context: Context) = context.getSharedPreferences("codync-voice", Context.MODE_PRIVATE)
fun load(context: Context): VoiceSettings {
- val stored = context.getSharedPreferences("codync-voice", Context.MODE_PRIVATE)
+ val stored = stored(context)
return runCatching { VoiceSettings(stored.getLong("pause", 1_500), stored.getFloat("rate", 1f)) }.getOrDefault(VoiceSettings())
}
fun save(context: Context, settings: VoiceSettings) {
- context.getSharedPreferences("codync-voice", Context.MODE_PRIVATE).edit()
- .putLong("pause", settings.pauseMillis).putFloat("rate", settings.rate).apply()
+ stored(context).edit { putLong("pause", settings.pauseMillis); putFloat("rate", settings.rate) }
+ }
+ fun choices(context: Context): VoiceChoices {
+ val stored = stored(context)
+ return VoiceChoices({ runCatching { stored.getString(it, null) }.getOrNull() }) { key, value ->
+ stored.edit { if (value == null) remove(key) else putString(key, value) }
+ }
}
}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/MainActivity.kt b/apps/android/app/src/main/kotlin/com/codync/android/MainActivity.kt
index 386bb7a8..be978547 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/MainActivity.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/MainActivity.kt
@@ -76,6 +76,7 @@ class MainActivity : ComponentActivity() {
// Retain the scoped, non-secret destination for Activity/process restoration.
if (uri.scheme == BuildConfig.URL_SCHEME && uri.host in setOf("bot", "usage")) store.openLink(uri)
if (uri.scheme == BuildConfig.URL_SCHEME && uri.host == "oauth") { intent.data = null; store.finishConnectorSignIn(uri) }
+ if (uri.scheme == BuildConfig.URL_SCHEME && uri.host == "screen") { intent.data = null; store.openScreenLink() }
}
}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/MessageComposer.kt b/apps/android/app/src/main/kotlin/com/codync/android/MessageComposer.kt
index 37970143..a9250663 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/MessageComposer.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/MessageComposer.kt
@@ -111,7 +111,7 @@ private enum class Trailing { Stop, Call, Send }
val working = bot.status in listOf("working", "needsInput") && (thread == null || bot.workingThread == thread)
val trailing = when {
working && empty -> Trailing.Stop
- onCall != null && empty && thread == null && bot.kind != "group" -> Trailing.Call
+ onCall != null && empty && thread == null -> Trailing.Call
else -> Trailing.Send
}
val placeholder = when {
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ScreenCanvas.kt b/apps/android/app/src/main/kotlin/com/codync/android/ScreenCanvas.kt
new file mode 100644
index 00000000..4303e961
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ScreenCanvas.kt
@@ -0,0 +1,257 @@
+package com.codync.android
+
+import android.annotation.SuppressLint
+import android.content.Context
+import android.graphics.Matrix
+import android.graphics.RectF
+import android.graphics.SurfaceTexture
+import android.view.TextureView
+import androidx.compose.foundation.background
+import androidx.compose.foundation.gestures.awaitEachGesture
+import androidx.compose.foundation.gestures.awaitFirstDown
+import androidx.compose.foundation.layout.Box
+import androidx.compose.foundation.layout.fillMaxSize
+import androidx.compose.foundation.layout.heightIn
+import androidx.compose.foundation.layout.padding
+import androidx.compose.foundation.layout.size
+import androidx.compose.foundation.layout.widthIn
+import androidx.compose.runtime.Composable
+import androidx.compose.runtime.LaunchedEffect
+import androidx.compose.runtime.SideEffect
+import androidx.compose.runtime.getValue
+import androidx.compose.runtime.mutableIntStateOf
+import androidx.compose.runtime.mutableStateOf
+import androidx.compose.runtime.remember
+import androidx.compose.runtime.rememberCoroutineScope
+import androidx.compose.runtime.rememberUpdatedState
+import androidx.compose.runtime.setValue
+import androidx.compose.ui.Alignment
+import androidx.compose.ui.Modifier
+import androidx.compose.ui.draw.clipToBounds
+import androidx.compose.ui.geometry.Offset
+import androidx.compose.ui.graphics.Color
+import androidx.compose.ui.hapticfeedback.HapticFeedbackType
+import androidx.compose.ui.input.pointer.PointerInputChange
+import androidx.compose.ui.input.pointer.PointerInputScope
+import androidx.compose.ui.input.pointer.pointerInput
+import androidx.compose.ui.input.pointer.util.VelocityTracker
+import androidx.compose.ui.layout.onSizeChanged
+import androidx.compose.ui.platform.LocalDensity
+import androidx.compose.ui.platform.LocalHapticFeedback
+import androidx.compose.ui.unit.dp
+import androidx.compose.ui.viewinterop.AndroidView
+import com.codync.android.core.*
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.delay
+import kotlinx.coroutines.launch
+import org.webrtc.EglBase
+import org.webrtc.EglRenderer
+import org.webrtc.GlRectDrawer
+import org.webrtc.VideoFrame
+import org.webrtc.VideoSink
+import kotlin.math.abs
+import kotlin.math.hypot
+
+/**
+ * The video plus every touch and key gesture (CodyncUI `ScreenCanvas`). `ScreenControl` turns
+ * them into input; coordinates leave here as view points (dp), the control maps them to the display.
+ */
+@Composable internal fun ScreenCanvas(session: ScreenSession, display: ScreenDisplay, mode: TouchMode, interactive: Boolean,
+ keyboard: Boolean, armed: List<String>, onArmed: (List<String>) -> Unit, onZoomed: (Boolean) -> Unit, resetToken: Int,
+ modifier: Modifier = Modifier) {
+ val density = LocalDensity.current.density
+ val haptics = LocalHapticFeedback.current
+ val scope = rememberCoroutineScope()
+ val control = remember(session) { ScreenControl(session::send) }
+ var frame by remember(session) { mutableStateOf(control.videoFrame) }
+ var composing by remember { mutableStateOf(false) }
+ val zoomed by rememberUpdatedState(onZoomed)
+ val viewRequest = remember { object { var job: Job? = null } }
+ SideEffect {
+ control.mode = mode
+ control.interactive = interactive
+ control.armed = armed
+ control.onArmedConsumed = { onArmed(emptyList()) }
+ control.onChanged = { request ->
+ frame = control.videoFrame
+ zoomed(control.zoomed)
+ // Streams just the visible region at the view's pixel size, once the gesture settles.
+ if (request) {
+ viewRequest.job?.cancel()
+ viewRequest.job = scope.launch {
+ delay(180)
+ if (control.pinching) return@launch
+ val view = control.viewport.view
+ session.view(control.viewport.crop, ScreenSize(view.width * density, view.height * density))
+ }
+ }
+ }
+ }
+ LaunchedEffect(control, display) { control.setDisplay(display) }
+ LaunchedEffect(control, session.streamCrop) { control.setStreamRegion(session.streamCrop) }
+ var handledReset by remember { mutableIntStateOf(resetToken) }
+ LaunchedEffect(resetToken) { if (resetToken != handledReset) { handledReset = resetToken; control.resetZoom() } }
+
+ Box(modifier.fillMaxSize().background(Color.Black).clipToBounds()
+ .onSizeChanged { control.setViewSize(ScreenSize(it.width / density.toDouble(), it.height / density.toDouble())) }) {
+ AndroidView({ context -> ScreenVideoView(context).also { session.sink = it } }, Modifier.fillMaxSize(),
+ onRelease = { session.sink = null; it.release() }) { view ->
+ view.place(frame.x * density, frame.y * density, frame.width * density, frame.height * density)
+ }
+ Box(Modifier.fillMaxSize().pointerInput(control) {
+ screenGestures(control) { haptics.performHapticFeedback(HapticFeedbackType.LongPress) }
+ })
+ // Hidden while idle; while an IME composes, the field shows above the bottom edge so its marked text is visible.
+ AndroidView({ context ->
+ KeyProxyField(context).apply {
+ onText = control::typed
+ onKey = control::pressKey
+ onComposingChanged = { composing = it }
+ post { requestFocus() }
+ }
+ }, if (composing) Modifier.align(Alignment.BottomCenter).padding(16.dp).widthIn(min = 80.dp).heightIn(min = 40.dp)
+ else Modifier.align(Alignment.TopStart).size(1.dp)) { field ->
+ field.showsSoftKeyboard = keyboard
+ }
+ }
+}
+
+private enum class Gesture { Undecided, OnePan, Press, TwoFingers, Panning, Done }
+
+/** UIKit's recognizers in one detector: tap, two-finger tap, pan, long-press drag, two-finger pan and pinch together. */
+private suspend fun PointerInputScope.screenGestures(control: ScreenControl, haptic: () -> Unit) {
+ val slop = viewConfiguration.touchSlop
+ fun point(o: Offset) = ScreenPoint(o.x / density.toDouble(), o.y / density.toDouble())
+ awaitEachGesture {
+ val first = awaitFirstDown(requireUnconsumed = false)
+ val velocity = VelocityTracker().apply { addPosition(first.uptimeMillis, first.position) }
+ var gesture = Gesture.Undecided
+ var last = first.position
+ var pair: List<PointerInputChange> = emptyList()
+ var startCenter = Offset.Zero
+ var startSpread = 0f
+ var center = Offset.Zero
+ var spread = 0f
+ var pinching = false
+ try {
+ while (true) {
+ val wait = PRESS_MILLIS - (currentEvent.changes.first().uptimeMillis - first.uptimeMillis)
+ val event = if (gesture == Gesture.Undecided) withTimeoutOrNull(maxOf(wait, 0)) { awaitPointerEvent() } else awaitPointerEvent()
+ if (event == null) {
+ gesture = Gesture.Press
+ if (control.pressBegan(point(last))) haptic()
+ continue
+ }
+ event.changes.forEach { it.consume() }
+ val down = event.changes.filter { it.pressed }
+ val finger = event.changes.firstOrNull { it.id == first.id }
+ when (gesture) {
+ Gesture.Undecided -> when {
+ down.size >= 2 -> {
+ gesture = Gesture.TwoFingers
+ pair = down.take(2)
+ startCenter = (pair[0].position + pair[1].position) / 2f
+ startSpread = (pair[0].position - pair[1].position).getDistance()
+ center = startCenter
+ spread = startSpread
+ }
+ finger == null || !finger.pressed -> {
+ control.tap(point(finger?.position ?: last), finger?.uptimeMillis ?: first.uptimeMillis)
+ return@awaitEachGesture
+ }
+ (finger.position - first.position).getDistance() > slop -> {
+ gesture = Gesture.OnePan
+ last = finger.position
+ velocity.addPosition(finger.uptimeMillis, finger.position)
+ }
+ }
+ Gesture.OnePan -> {
+ if (finger == null || !finger.pressed) return@awaitEachGesture
+ velocity.addPosition(finger.uptimeMillis, finger.position)
+ val v = velocity.calculateVelocity()
+ control.onePan(point(finger.position - last), hypot(v.x, v.y) / density.toDouble())
+ last = finger.position
+ }
+ Gesture.Press -> {
+ if (finger == null || !finger.pressed) return@awaitEachGesture
+ control.pressMoved(point(finger.position))
+ last = finger.position
+ }
+ Gesture.TwoFingers, Gesture.Panning -> {
+ val now = pair.map { p -> event.changes.firstOrNull { it.id == p.id } }
+ if (now.any { it == null || !it.pressed }) {
+ val quick = event.changes.first().uptimeMillis - first.uptimeMillis < TWO_TAP_MILLIS
+ if (gesture == Gesture.TwoFingers && quick) control.twoFingerTap(point(startCenter))
+ if (pinching) { pinching = false; control.pinchEnded() }
+ gesture = Gesture.Done
+ } else {
+ val a = requireNotNull(now[0]).position
+ val b = requireNotNull(now[1]).position
+ val nextCenter = (a + b) / 2f
+ val nextSpread = (a - b).getDistance()
+ if (gesture == Gesture.TwoFingers && ((nextCenter - startCenter).getDistance() > slop || abs(nextSpread - startSpread) > slop))
+ gesture = Gesture.Panning
+ if (gesture == Gesture.Panning) {
+ if (!pinching && abs(nextSpread - startSpread) > slop) { pinching = true; control.pinchBegan() }
+ if (pinching && spread > 0f) control.pinch((nextSpread / spread).toDouble(), point(nextCenter))
+ control.twoPan(point(nextCenter - center), point(nextCenter))
+ center = nextCenter
+ spread = nextSpread
+ }
+ }
+ }
+ Gesture.Done -> if (down.isEmpty()) return@awaitEachGesture
+ }
+ }
+ } finally {
+ if (gesture == Gesture.Press) control.pressEnded()
+ if (pinching) control.pinchEnded()
+ }
+ }
+}
+
+private const val PRESS_MILLIS = 350L
+private const val TWO_TAP_MILLIS = 500L
+
+/** WebRTC video in a TextureView: a normal view, so it moves, clips and sits under Compose chrome. */
+@SuppressLint("ViewConstructor")
+internal class ScreenVideoView(context: Context) : TextureView(context), TextureView.SurfaceTextureListener, VideoSink {
+ private val renderer = EglRenderer("screen")
+
+ init {
+ renderer.init(ScreenRtc.egl.eglBaseContext, EglBase.CONFIG_PLAIN, GlRectDrawer())
+ surfaceTextureListener = this
+ }
+
+ override fun onFrame(frame: VideoFrame) = renderer.onFrame(frame)
+
+ private val placement = RectF()
+
+ /** Shows the whole stream at this rectangle (pixels in the view); what falls outside is clipped. */
+ fun place(left: Double, top: Double, width: Double, height: Double) {
+ placement.set(left.toFloat(), top.toFloat(), (left + width).toFloat(), (top + height).toFloat())
+ applyPlacement()
+ }
+
+ override fun onSizeChanged(w: Int, h: Int, oldw: Int, oldh: Int) {
+ super.onSizeChanged(w, h, oldw, oldh)
+ applyPlacement()
+ }
+
+ private fun applyPlacement() {
+ if (width == 0 || height == 0 || placement.isEmpty) return
+ setTransform(Matrix().apply { setRectToRect(RectF(0f, 0f, width.toFloat(), height.toFloat()), placement, Matrix.ScaleToFit.FILL) })
+ invalidate()
+ }
+
+ fun release() = renderer.release()
+
+ override fun onSurfaceTextureAvailable(surface: SurfaceTexture, width: Int, height: Int) = renderer.createEglSurface(surface)
+ override fun onSurfaceTextureSizeChanged(surface: SurfaceTexture, width: Int, height: Int) = Unit
+ // The EGL surface goes first; the texture is released once the render thread lets go of it.
+ override fun onSurfaceTextureDestroyed(surface: SurfaceTexture): Boolean {
+ renderer.releaseEglSurface { surface.release() }
+ return false
+ }
+ override fun onSurfaceTextureUpdated(surface: SurfaceTexture) = Unit
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ScreenKeys.kt b/apps/android/app/src/main/kotlin/com/codync/android/ScreenKeys.kt
new file mode 100644
index 00000000..0b6abe0d
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ScreenKeys.kt
@@ -0,0 +1,126 @@
+package com.codync.android
+
+import android.annotation.SuppressLint
+import android.content.Context
+import android.graphics.Color
+import android.graphics.drawable.GradientDrawable
+import android.text.Editable
+import android.text.InputType
+import android.text.TextWatcher
+import android.util.TypedValue
+import android.view.KeyEvent
+import android.view.inputmethod.BaseInputConnection
+import android.view.inputmethod.EditorInfo
+import android.view.inputmethod.InputConnection
+import android.view.inputmethod.InputConnectionWrapper
+import android.view.inputmethod.InputMethodManager
+import android.widget.EditText
+
+/**
+ * Invisible text field that turns the soft keyboard (including IME input such as Zhuyin or
+ * Pinyin) and hardware keys into events for the computer (CodyncUI `KeyProxyField`). It keeps
+ * focus with the soft keyboard hidden, so a hardware keyboard just works.
+ */
+@SuppressLint("AppCompatCustomView", "ViewConstructor")
+internal class KeyProxyField(context: Context) : EditText(context) {
+ var onText: (String) -> Unit = {}
+ var onKey: (String, List<String>) -> Unit = { _, _ -> }
+ /** The composition started or ended: the canvas shows the field while an IME composes. */
+ var onComposingChanged: (Boolean) -> Unit = {}
+ private var composing = false
+
+ var showsSoftKeyboard = false
+ set(value) {
+ if (field == value) return
+ field = value
+ requestFocus()
+ val keyboard = context.getSystemService(InputMethodManager::class.java)
+ if (value) keyboard.showSoftInput(this, 0) else keyboard.hideSoftInputFromWindow(windowToken, 0)
+ }
+
+ init {
+ inputType = InputType.TYPE_CLASS_TEXT or InputType.TYPE_TEXT_FLAG_NO_SUGGESTIONS or InputType.TYPE_TEXT_FLAG_MULTI_LINE
+ imeOptions = EditorInfo.IME_FLAG_NO_EXTRACT_UI or EditorInfo.IME_FLAG_NO_FULLSCREEN or EditorInfo.IME_FLAG_NO_PERSONALIZED_LEARNING
+ showSoftInputOnFocus = false
+ isFocusableInTouchMode = true
+ alpha = IDLE_ALPHA
+ setTextColor(Color.WHITE)
+ setTextSize(TypedValue.COMPLEX_UNIT_SP, 17f)
+ val dp = resources.displayMetrics.density
+ background = GradientDrawable().apply { setColor(Color.rgb(31, 31, 31)); cornerRadius = 12 * dp }
+ setPadding((14 * dp).toInt(), (8 * dp).toInt(), (14 * dp).toInt(), (8 * dp).toInt())
+ addTextChangedListener(object : TextWatcher {
+ override fun beforeTextChanged(s: CharSequence, start: Int, count: Int, after: Int) = Unit
+ override fun onTextChanged(s: CharSequence, start: Int, before: Int, count: Int) = Unit
+ override fun afterTextChanged(s: Editable) = changed(s)
+ })
+ setOnKeyListener { _, _, event -> key(event) }
+ }
+
+ /** Sends committed text; text still being composed by an IME waits. */
+ private fun changed(text: Editable) {
+ val now = BaseInputConnection.getComposingSpanStart(text) != -1
+ if (now != composing) {
+ composing = now
+ alpha = if (now) 1f else IDLE_ALPHA
+ onComposingChanged(now)
+ }
+ if (now || text.isEmpty()) return
+ val committed = text.toString()
+ text.clear()
+ // Return is its own key; the multi-line field would type it as a newline.
+ committed.split('\n').forEachIndexed { i, part ->
+ if (i > 0) onKey("return", emptyList())
+ if (part.isNotEmpty()) onText(part)
+ }
+ }
+
+ /** While an IME is composing, Return, Delete, arrows and Escape pick and edit candidates. */
+ private fun key(event: KeyEvent): Boolean {
+ if (composing) return false
+ val (name, mods) = ScreenKeys.map(event.keyCode, event.metaState, event.getUnicodeChar(0)) ?: return false
+ if (event.action == KeyEvent.ACTION_DOWN) onKey(name, mods)
+ return true
+ }
+
+ /** Backspace on an empty field still deletes on the computer, however the IME sends it. */
+ override fun onCreateInputConnection(outAttrs: EditorInfo): InputConnection? {
+ val base = super.onCreateInputConnection(outAttrs) ?: return null
+ return object : InputConnectionWrapper(base, true) {
+ override fun deleteSurroundingText(beforeLength: Int, afterLength: Int): Boolean {
+ if (text.isNullOrEmpty() && beforeLength > 0) { onKey("delete", emptyList()); return true }
+ return super.deleteSurroundingText(beforeLength, afterLength)
+ }
+ }
+ }
+
+ private companion object { const val IDLE_ALPHA = 0.02f }
+}
+
+/** Hardware keys as the computer names them. Shortcuts and special keys go straight through; plain typing takes the text path (IME-aware). */
+internal object ScreenKeys {
+ private val special = mapOf(
+ KeyEvent.KEYCODE_DPAD_UP to "up", KeyEvent.KEYCODE_DPAD_DOWN to "down", KeyEvent.KEYCODE_DPAD_LEFT to "left", KeyEvent.KEYCODE_DPAD_RIGHT to "right",
+ KeyEvent.KEYCODE_ESCAPE to "escape", KeyEvent.KEYCODE_TAB to "tab", KeyEvent.KEYCODE_ENTER to "return", KeyEvent.KEYCODE_NUMPAD_ENTER to "return",
+ KeyEvent.KEYCODE_DEL to "delete", KeyEvent.KEYCODE_FORWARD_DEL to "forwardDelete",
+ KeyEvent.KEYCODE_MOVE_HOME to "home", KeyEvent.KEYCODE_MOVE_END to "end", KeyEvent.KEYCODE_PAGE_UP to "pageUp", KeyEvent.KEYCODE_PAGE_DOWN to "pageDown",
+ ) + (1..12).associate { KeyEvent.KEYCODE_F1 + it - 1 to "f$it" }
+
+ fun modifiers(meta: Int): List<String> = buildList {
+ if (meta and KeyEvent.META_META_ON != 0) add("cmd")
+ if (meta and KeyEvent.META_ALT_ON != 0) add("option")
+ if (meta and KeyEvent.META_CTRL_ON != 0) add("ctrl")
+ if (meta and KeyEvent.META_SHIFT_ON != 0) add("shift")
+ }
+
+ /** `char` is the key's character without modifiers (0: none). `null`: let the field type it. */
+ fun map(keyCode: Int, meta: Int, char: Int): Pair<String, List<String>>? {
+ val mods = modifiers(meta)
+ special[keyCode]?.let { return it to mods }
+ // Ctrl-Space switches the keyboard's input language.
+ if (keyCode == KeyEvent.KEYCODE_SPACE && mods == listOf("ctrl")) return null
+ val chord = "cmd" in mods || "ctrl" in mods || "option" in mods
+ if (!chord || char <= 0) return null
+ return String(Character.toChars(char)).lowercase() to mods
+ }
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt b/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt
new file mode 100644
index 00000000..7d53861f
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt
@@ -0,0 +1,352 @@
+package com.codync.android
+
+import android.content.Context
+import android.util.Log
+import androidx.compose.runtime.getValue
+import androidx.compose.runtime.mutableStateOf
+import androidx.compose.runtime.setValue
+import com.codync.android.core.*
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.SupervisorJob
+import kotlinx.coroutines.delay
+import kotlinx.coroutines.isActive
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.suspendCancellableCoroutine
+import kotlinx.coroutines.withTimeoutOrNull
+import kotlinx.serialization.json.JsonObject
+import kotlinx.serialization.json.buildJsonObject
+import kotlinx.serialization.json.decodeFromJsonElement
+import kotlinx.serialization.json.jsonPrimitive
+import kotlinx.serialization.json.put
+import org.webrtc.DataChannel
+import org.webrtc.DefaultVideoDecoderFactory
+import org.webrtc.DefaultVideoEncoderFactory
+import org.webrtc.EglBase
+import org.webrtc.IceCandidate
+import org.webrtc.MediaConstraints
+import org.webrtc.MediaStream
+import org.webrtc.MediaStreamTrack
+import org.webrtc.PeerConnection
+import org.webrtc.PeerConnectionFactory
+import org.webrtc.RtpTransceiver
+import org.webrtc.SdpObserver
+import org.webrtc.SessionDescription
+import org.webrtc.VideoSink
+import org.webrtc.VideoTrack
+import java.io.IOException
+import java.nio.ByteBuffer
+import kotlin.coroutines.resume
+import kotlin.coroutines.resumeWithException
+
+/**
+ * One live view of the computer's screen (CodyncUI `ScreenSession`): a WebRTC peer that receives
+ * the display as H.264 and sends input over two data channels (`input` reliable, `input-fast`
+ * unordered for pointer moves). Signaling goes through the host API, non-trickle (offer and
+ * answer carry every candidate). Lives on the main thread; WebRTC callbacks hop onto it.
+ */
+class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
+ sealed interface Phase {
+ data object Connecting : Phase
+ data object Live : Phase
+ data object Reconnecting : Phase
+ data class Failed(val message: String) : Phase
+ data object Closed : Phase
+ }
+
+ var phase by mutableStateOf<Phase>(Phase.Connecting)
+ private set
+ /** Text the computer copied, waiting for the user to pull it onto the phone. */
+ var remoteClipboard by mutableStateOf<String?>(null)
+ private set
+ var lastError by mutableStateOf<String?>(null)
+ private set
+ var display by mutableStateOf(display)
+ private set
+ /** The display region the video currently shows (points; `null` = all of it), as confirmed by the computer. */
+ var streamCrop by mutableStateOf<ScreenRect?>(null)
+ private set
+
+ private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate)
+ private val context: Context = store.getApplication<android.app.Application>()
+ private var session: String? = null
+ private var pc: PeerConnection? = null
+ private var track: VideoTrack? = null
+ private var reliable: DataChannel? = null
+ private var fast: DataChannel? = null
+ private var gathered: CompletableDeferred<Unit>? = null
+ private var recoverJob: Job? = null
+ private var renewJob: Job? = null
+ private var connectionTimer: Job? = null
+ private var generation = 0
+ /** Where the video draws; the session moves it between tracks and detaches it before a track is freed. */
+ var sink: VideoSink? = null
+ set(value) {
+ field?.let { track?.removeSink(it) }
+ field = value
+ value?.let { track?.addSink(it) }
+ }
+
+ /** A newer connection or a close replaced the one being set up. */
+ private class Superseded : Exception()
+
+ // MARK: connection
+
+ fun start() {
+ phase = Phase.Connecting
+ scope.launch {
+ try { connect() }
+ catch (cancelled: CancellationException) { throw cancelled }
+ catch (_: Superseded) { /* Its replacement reports for itself. */ }
+ catch (error: Exception) {
+ if (phase == Phase.Closed) return@launch
+ close()
+ phase = Phase.Failed(error.message ?: "Couldn't connect to the remote screen.")
+ }
+ }
+ }
+
+ private suspend fun connect() {
+ val preparing = generation
+ val active = store.ready()
+ val relay = (active.state.value as? LinkState.Ready)?.route == HostRoute.Relay
+ val connection = WireJson.decodeFromJsonElement<ScreenConnection>(active.call("screenPrepare", buildJsonObject { put("relay", relay) }))
+ if (generation != preparing || phase == Phase.Closed) {
+ closeRemote(connection.session)
+ throw Superseded()
+ }
+ // Reserve the replacement before closing the old viewer so takeover survives renewal.
+ val old = session
+ teardown()
+ val current = generation
+ session = connection.session
+ if (old != null) closeRemote(old).join()
+ if (generation != current || phase == Phase.Closed) throw Superseded()
+ val config = PeerConnection.RTCConfiguration(connection.iceServers.map { server ->
+ PeerConnection.IceServer.builder(server.urls).apply {
+ server.username?.let(::setUsername)
+ server.credential?.let(::setPassword)
+ }.createIceServer()
+ }).apply {
+ sdpSemantics = PeerConnection.SdpSemantics.UNIFIED_PLAN
+ tcpCandidatePolicy = PeerConnection.TcpCandidatePolicy.ENABLED
+ continualGatheringPolicy = PeerConnection.ContinualGatheringPolicy.GATHER_ONCE
+ bundlePolicy = PeerConnection.BundlePolicy.MAXBUNDLE
+ }
+ val factory = ScreenRtc.factory(context)
+ val pc = factory.createPeerConnection(config, Events(current)) ?: throw IOException("Couldn't start the video connection.")
+ this.pc = pc
+ val video = pc.addTransceiver(MediaStreamTrack.MediaType.MEDIA_TYPE_VIDEO,
+ RtpTransceiver.RtpTransceiverInit(RtpTransceiver.RtpTransceiverDirection.RECV_ONLY))
+ val codecs = factory.getRtpReceiverCapabilities(MediaStreamTrack.MediaType.MEDIA_TYPE_VIDEO).codecs
+ video.setCodecPreferences(preferredScreenCodecs(codecs, { it.name }, { it.parameters["profile-level-id"] }))
+ track = (video.receiver.track() as? VideoTrack)?.also { t -> sink?.let(t::addSink) }
+ reliable = pc.createDataChannel("input", DataChannel.Init())?.also { it.registerObserver(Messages(current)) }
+ fast = pc.createDataChannel("input-fast", DataChannel.Init().apply { ordered = false; maxRetransmits = 0 })
+
+ negotiate(pc)
+ if (generation != current || phase == Phase.Closed) throw Superseded()
+ // Refresh both endpoints before TURN credentials expire, including an idle screen.
+ renewJob = scope.launch {
+ delay(connection.renewIn(System.currentTimeMillis()))
+ if (generation != current || phase == Phase.Closed) return@launch
+ phase = Phase.Reconnecting
+ scheduleRecovery(0)
+ }
+ connectionTimer = scope.launch {
+ delay(30_000)
+ if (generation != current || phase == Phase.Live) return@launch
+ close()
+ phase = Phase.Failed("Couldn't connect to the remote screen. Check the computer's connection and try again.")
+ }
+ }
+
+ /** Offer → host → answer with the reserved session's ICE configuration. */
+ private suspend fun negotiate(pc: PeerConnection) {
+ val offer = suspendCancellableCoroutine { done ->
+ pc.createOffer(Sdp({ done.resume(it) }, { done.resumeWithException(IOException(it)) }), MediaConstraints())
+ }
+ pc.await { pc.setLocalDescription(it, offer) }
+ waitForCandidates(pc)
+ if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
+ val sdp = pc.localDescription?.description ?: offer.description
+ val response = store.hostCall("screenOffer", buildJsonObject {
+ put("sdp", sdp)
+ session?.let { put("session", it) }
+ display?.let { put("display", it.id) }
+ }, 30_000)
+ val answer = WireJson.decodeFromJsonElement<ScreenAnswer>(response)
+ if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
+ session = answer.session
+ pc.await { pc.setRemoteDescription(it, SessionDescription(SessionDescription.Type.ANSWER, answer.sdp)) }
+ }
+
+ private suspend fun PeerConnection.await(set: (SdpObserver) -> Unit) = suspendCancellableCoroutine { done ->
+ set(Sdp(onSet = { done.resume(Unit) }, failed = { done.resumeWithException(IOException(it)) }))
+ }
+
+ private suspend fun waitForCandidates(pc: PeerConnection) {
+ if (pc.iceGatheringState() == PeerConnection.IceGatheringState.COMPLETE) return
+ val done = CompletableDeferred<Unit>()
+ gathered = done
+ withTimeoutOrNull(10_000) { done.await() }
+ if (gathered === done) gathered = null
+ }
+
+ private fun iceChanged(state: PeerConnection.IceConnectionState, generation: Int) {
+ if (generation != this.generation || phase == Phase.Closed) return
+ when (state) {
+ PeerConnection.IceConnectionState.CONNECTED, PeerConnection.IceConnectionState.COMPLETED -> {
+ connectionTimer?.cancel()
+ phase = Phase.Live
+ lastError = null
+ }
+ // Often recovers by itself (Wi-Fi ↔ cellular); reconnect if it does not.
+ PeerConnection.IceConnectionState.DISCONNECTED -> { phase = Phase.Reconnecting; scheduleRecovery(2_000) }
+ PeerConnection.IceConnectionState.FAILED -> { phase = Phase.Reconnecting; scheduleRecovery(0) }
+ else -> Unit
+ }
+ }
+
+ private fun scheduleRecovery(delayMillis: Long) {
+ recoverJob?.cancel()
+ recoverJob = scope.launch {
+ var attempt = 0
+ delay(delayMillis)
+ while (isActive && phase == Phase.Reconnecting) {
+ try {
+ // Re-evaluate the signaling route after Wi-Fi/cellular changes and fetch fresh ICE credentials.
+ connect()
+ return@launch
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (_: Superseded) { return@launch }
+ catch (error: Exception) {
+ if (phase == Phase.Closed) return@launch
+ Log.i("CodyncScreen", "screen reconnect failed: ${error.message}")
+ attempt += 1
+ if (attempt >= 3) {
+ close()
+ phase = Phase.Failed(error.message ?: "Couldn't reconnect to the remote screen.")
+ return@launch
+ }
+ delay(minOf(attempt * 2_000L, 10_000L))
+ }
+ }
+ }
+ }
+
+ fun close() {
+ recoverJob?.cancel()
+ phase = Phase.Closed
+ session?.let(::closeRemote)
+ session = null
+ teardown()
+ }
+
+ /** Best effort: a viewer whose close never arrives ends with its WebRTC connection. */
+ private fun closeRemote(id: String): Job = scope.launch {
+ try { store.hostCall("screenClose", body("session" to id), 15_000) }
+ catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { Log.i("CodyncScreen", "screen close failed: ${error.message}") }
+ }
+
+ private fun teardown() {
+ generation += 1
+ renewJob?.cancel()
+ connectionTimer?.cancel()
+ gathered?.complete(Unit)
+ gathered = null
+ sink?.let { track?.removeSink(it) }
+ track = null
+ reliable?.unregisterObserver()
+ for (channel in listOfNotNull(reliable, fast)) { channel.close(); channel.dispose() }
+ reliable = null
+ fast = null
+ pc?.close()
+ pc?.dispose()
+ pc = null
+ }
+
+ // MARK: input
+
+ /** Sends one input event (display points). Pointer moves go unordered: a late move is useless. */
+ fun send(event: JsonObject) {
+ val move = event["type"]?.jsonPrimitive?.content == "move"
+ val channel = (if (move && fast?.state() == DataChannel.State.OPEN) fast else reliable) ?: return
+ if (channel.state() != DataChannel.State.OPEN) return
+ channel.send(DataChannel.Buffer(ByteBuffer.wrap(event.toString().toByteArray()), false))
+ }
+
+ /** What to stream: a region of the display (points; `null` = all of it) at the viewer's pixel size. */
+ fun view(crop: ScreenRect?, pixels: ScreenSize) = send(ScreenRequests.view(display?.id, crop, pixels))
+
+ fun switchDisplay(d: ScreenDisplay) {
+ display = d
+ send(ScreenRequests.switchDisplay(d.id))
+ }
+
+ fun pushClipboard(text: String) = send(ScreenRequests.clipboard(text))
+
+ fun takeRemoteClipboard(): String? = remoteClipboard.also { remoteClipboard = null }
+
+ private fun received(reply: ScreenReply) {
+ when (reply) {
+ is ScreenReply.Clipboard -> remoteClipboard = reply.text
+ is ScreenReply.View -> streamCrop = reply.crop
+ is ScreenReply.Error -> lastError = reply.message
+ }
+ }
+
+ // MARK: WebRTC callbacks (its signaling thread) → main
+
+ private inner class Events(private val generation: Int) : PeerConnection.Observer {
+ override fun onIceConnectionChange(state: PeerConnection.IceConnectionState) { scope.launch { iceChanged(state, generation) } }
+ override fun onIceGatheringChange(state: PeerConnection.IceGatheringState) {
+ if (state != PeerConnection.IceGatheringState.COMPLETE) return
+ scope.launch { if (generation == [email protected]) gathered?.complete(Unit) }
+ }
+ override fun onSignalingChange(state: PeerConnection.SignalingState) = Unit
+ override fun onIceConnectionReceivingChange(receiving: Boolean) = Unit
+ override fun onIceCandidate(candidate: IceCandidate) = Unit
+ override fun onIceCandidatesRemoved(candidates: Array<out IceCandidate>) = Unit
+ override fun onAddStream(stream: MediaStream) = Unit
+ override fun onRemoveStream(stream: MediaStream) = Unit
+ override fun onDataChannel(channel: DataChannel) = Unit
+ override fun onRenegotiationNeeded() = Unit
+ }
+
+ private inner class Messages(private val generation: Int) : DataChannel.Observer {
+ override fun onMessage(buffer: DataChannel.Buffer) {
+ // The buffer is only valid during this call.
+ val bytes = ByteArray(buffer.data.remaining()).also { buffer.data.get(it) }
+ val reply = ScreenReply.decode(bytes.decodeToString()) ?: return
+ scope.launch { if (generation == [email protected]) received(reply) }
+ }
+ override fun onBufferedAmountChange(previous: Long) = Unit
+ override fun onStateChange() = Unit
+ }
+
+ private class Sdp(val created: (SessionDescription) -> Unit = {}, val failed: (String?) -> Unit, val onSet: () -> Unit = {}) : SdpObserver {
+ override fun onCreateSuccess(description: SessionDescription) = created(description)
+ override fun onCreateFailure(error: String?) = failed(error)
+ override fun onSetSuccess() = onSet()
+ override fun onSetFailure(error: String?) = failed(error)
+ }
+}
+
+/** The process's WebRTC: one EGL context shared by the hardware decoder and the video views. */
+internal object ScreenRtc {
+ val egl: EglBase by lazy { EglBase.create() }
+ private var factory: PeerConnectionFactory? = null
+
+ @Synchronized fun factory(context: Context): PeerConnectionFactory = factory ?: run {
+ WebRtc.load(context)
+ PeerConnectionFactory.builder()
+ .setVideoDecoderFactory(DefaultVideoDecoderFactory(egl.eglBaseContext))
+ .setVideoEncoderFactory(DefaultVideoEncoderFactory(egl.eglBaseContext, true, true))
+ .createPeerConnectionFactory().also { factory = it }
+ }
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ScreenView.kt b/apps/android/app/src/main/kotlin/com/codync/android/ScreenView.kt
new file mode 100644
index 00000000..51a4c34c
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ScreenView.kt
@@ -0,0 +1,307 @@
+package com.codync.android
+
+import android.content.ClipData
+import android.content.ClipDescription
+import android.content.ClipboardManager
+import android.content.pm.ActivityInfo
+import android.content.res.Configuration
+import androidx.activity.compose.BackHandler
+import androidx.activity.compose.LocalActivity
+import androidx.compose.animation.AnimatedVisibility
+import androidx.compose.animation.animateColorAsState
+import androidx.compose.animation.fadeIn
+import androidx.compose.animation.fadeOut
+import androidx.compose.animation.slideInVertically
+import androidx.compose.animation.slideOutVertically
+import androidx.compose.foundation.background
+import androidx.compose.foundation.horizontalScroll
+import androidx.compose.foundation.layout.*
+import androidx.compose.foundation.rememberScrollState
+import androidx.compose.foundation.shape.CircleShape
+import androidx.compose.foundation.shape.RoundedCornerShape
+import androidx.compose.runtime.*
+import androidx.compose.ui.Alignment
+import androidx.compose.ui.Modifier
+import androidx.compose.ui.draw.clip
+import androidx.compose.ui.graphics.Color
+import androidx.compose.ui.graphics.graphicsLayer
+import androidx.compose.ui.layout.onSizeChanged
+import androidx.compose.ui.platform.LocalConfiguration
+import androidx.compose.ui.platform.LocalContext
+import androidx.compose.ui.platform.LocalDensity
+import androidx.compose.ui.platform.LocalView
+import androidx.compose.ui.semantics.Role
+import androidx.compose.ui.semantics.contentDescription
+import androidx.compose.ui.semantics.selected
+import androidx.compose.ui.semantics.semantics
+import androidx.compose.ui.text.style.TextAlign
+import androidx.compose.ui.text.style.TextOverflow
+import androidx.compose.ui.unit.dp
+import androidx.compose.foundation.clickable
+import androidx.compose.material3.LocalContentColor
+import androidx.compose.material3.Text
+import androidx.core.view.WindowCompat
+import androidx.core.view.WindowInsetsCompat
+import androidx.core.view.WindowInsetsControllerCompat
+import androidx.lifecycle.Lifecycle
+import androidx.lifecycle.compose.LifecycleEventEffect
+import com.codync.android.core.*
+import com.codync.android.design.*
+
+/**
+ * The computer's screen, full screen on the phone (CodyncUI `ScreenView`). Opened from Computers
+ * (to use the computer) or from a bot's chat (to watch it work, then take over).
+ */
+@Composable internal fun ScreenView(state: AppState, store: AppStore, watching: String?, close: () -> Unit) {
+ val screen = state.screen
+ val agent = screen?.agentBot?.let { id -> state.bots.firstOrNull { it.id == id } }
+ var session by remember { mutableStateOf<ScreenSession?>(null) }
+ var display by remember { mutableStateOf<ScreenDisplay?>(null) }
+ var interactive by remember { mutableStateOf(false) }
+ var mode by remember { mutableStateOf(TouchMode.Trackpad) }
+ var keyboard by remember { mutableStateOf(false) }
+ var armed by remember { mutableStateOf(emptyList<String>()) }
+ var zoomed by remember { mutableStateOf(false) }
+ var resetToken by remember { mutableIntStateOf(0) }
+ var tookOver by remember { mutableStateOf(false) }
+ val portrait = LocalConfiguration.current.orientation != Configuration.ORIENTATION_LANDSCAPE
+
+ fun setInteractive(on: Boolean) {
+ val allowed = on && state.screen?.input == true
+ interactive = allowed
+ if (allowed != tookOver) {
+ tookOver = allowed
+ store.screenTakeover(allowed)
+ }
+ if (!allowed) keyboard = false
+ }
+ /** Also runs when the screen becomes available after opening. */
+ fun start() {
+ val current = state.screen
+ if (session != null || current == null || !current.available) return
+ val d = display ?: current.mainDisplay ?: return
+ display = d
+ session = ScreenSession(store, d).also { it.start() }
+ // Watching a bot: look first. Otherwise take the screen (bots may still look).
+ setInteractive(watching == null)
+ }
+ fun stop() {
+ session?.close()
+ session = null
+ if (tookOver) store.screenTakeover(false)
+ tookOver = false
+ }
+ // In the background the stream stops and the bots get the computer back, as if the viewer closed.
+ LifecycleEventEffect(Lifecycle.Event.ON_START) { start() }
+ LifecycleEventEffect(Lifecycle.Event.ON_STOP) { stop() }
+ DisposableEffect(Unit) { onDispose { stop() } }
+ LaunchedEffect(screen?.available) { if (screen?.available == true && session == null) start() }
+ LaunchedEffect(screen?.input) { if (screen?.input != true) setInteractive(false) }
+ BackHandler(onBack = close)
+ FullScreenWindow()
+ KeyboardDismissal(keyboard) { keyboard = false }
+
+ CompositionLocalProvider(LocalDark provides true, LocalContentColor provides Color.White) {
+ val bar = remember { mutableIntStateOf(0) }
+ val barHeight = with(LocalDensity.current) { bar.intValue.toDp() }
+ val showBar = keyboard && interactive
+ GlassStage(Modifier.fillMaxSize().background(Color.Black), chrome = {
+ TopBar(state, session, display, agent, interactive, mode, keyboard, zoomed, close,
+ toggleKeyboard = { keyboard = !keyboard },
+ toggleMode = { mode = if (mode == TouchMode.Trackpad) TouchMode.Direct else TouchMode.Trackpad },
+ fit = { resetToken += 1 },
+ pick = { d -> display = d; session?.switchDisplay(d); resetToken += 1 },
+ takeOver = { setInteractive(!interactive) })
+ AnimatedVisibility(showBar, Modifier.align(Alignment.BottomCenter).imePadding().onSizeChanged { bar.intValue = it.height },
+ enter = slideInVertically(Motion.layoutOffset) { it } + fadeIn(Motion.fade()),
+ exit = slideOutVertically(Motion.layoutOffset) { it } + fadeOut(Motion.fade())) {
+ ModifierBar(armed, { armed = it }) { key -> session?.send(ScreenRequests.key(key, armed)); armed = emptyList() }
+ }
+ }) {
+ val s = session
+ val d = display
+ if (s != null && d != null && screen?.available == true) {
+ // Portrait: the screen moves above the keyboard. Landscape: the keyboard overlays it,
+ // since shrinking there would leave the screen too small.
+ ScreenCanvas(s, d, mode, interactive, keyboard, armed, { armed = it }, { zoomed = it }, resetToken,
+ if (portrait) Modifier.imePadding().padding(bottom = if (showBar) barHeight else 0.dp) else Modifier)
+ Status(s, screen)
+ } else Unavailable(state)
+ }
+ }
+}
+
+/** The status and navigation bars step aside while the screen is open; a swipe shows them for a moment. */
+@Composable private fun FullScreenWindow() {
+ val activity = LocalActivity.current ?: return
+ val view = LocalView.current
+ DisposableEffect(activity) {
+ // The Rotate button's choice lasts only while the screen is open.
+ val orientation = activity.requestedOrientation
+ val controller = WindowCompat.getInsetsController(activity.window, view)
+ controller.systemBarsBehavior = WindowInsetsControllerCompat.BEHAVIOR_SHOW_TRANSIENT_BARS_BY_SWIPE
+ controller.hide(WindowInsetsCompat.Type.systemBars())
+ val light = controller.isAppearanceLightStatusBars
+ controller.isAppearanceLightStatusBars = false
+ onDispose {
+ activity.requestedOrientation = orientation
+ controller.show(WindowInsetsCompat.Type.systemBars())
+ controller.isAppearanceLightStatusBars = light
+ }
+ }
+}
+
+/** Hiding the soft keyboard with Back also turns the keyboard button off. */
+@OptIn(ExperimentalLayoutApi::class)
+@Composable private fun KeyboardDismissal(keyboard: Boolean, dismissed: () -> Unit) {
+ val visible = WindowInsets.isImeVisible
+ var seen by remember { mutableStateOf(false) }
+ LaunchedEffect(keyboard, visible) {
+ if (!keyboard) seen = false
+ else if (visible) seen = true
+ else if (seen) { seen = false; dismissed() }
+ }
+}
+
+@Composable private fun TopBar(state: AppState, session: ScreenSession?, display: ScreenDisplay?, agent: Bot?,
+ interactive: Boolean, mode: TouchMode, keyboard: Boolean, zoomed: Boolean, close: () -> Unit, toggleKeyboard: () -> Unit,
+ toggleMode: () -> Unit, fit: () -> Unit, pick: (ScreenDisplay) -> Unit, takeOver: () -> Unit) {
+ val activity = LocalActivity.current
+ Row(Modifier.fillMaxWidth().windowInsetsPadding(WindowInsets.displayCutout.only(WindowInsetsSides.Top + WindowInsetsSides.Horizontal))
+ .padding(horizontal = 12.dp).padding(top = 6.dp), verticalAlignment = Alignment.CenterVertically,
+ horizontalArrangement = Arrangement.spacedBy(8.dp)) {
+ OverlayButton("Close", "xmark", action = close)
+ AnimatedVisibility(agent != null && !interactive, Modifier.weight(1f, fill = false), enter = fadeIn(Motion.fade()), exit = fadeOut(Motion.fade())) {
+ Row(Modifier.clip(CircleShape).frost(20.dp, MATERIAL).padding(horizontal = 10.dp, vertical = 6.dp),
+ verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(6.dp)) {
+ Symbol("cursorarrow.motionlines", size = 14.dp, tint = Color.White)
+ Text("${agent?.name.orEmpty()} is using the computer", style = Type.caption.medium, color = Color.White,
+ maxLines = 1, overflow = TextOverflow.Ellipsis)
+ }
+ }
+ Spacer(Modifier.weight(1f))
+ // An explicit rotation works even with the phone's auto-rotate off.
+ OverlayButton("Rotate", "rotate.right") { activity?.let(::rotate) }
+ if (session != null) {
+ if (interactive) {
+ OverlayButton(if (keyboard) "Hide keyboard" else "Keyboard", if (keyboard) "keyboard.chevron.compact.down" else "keyboard", action = toggleKeyboard)
+ OverlayButton(if (mode == TouchMode.Trackpad) "Touch mode: trackpad" else "Touch mode: direct",
+ if (mode == TouchMode.Trackpad) "rectangle.and.hand.point.up.left" else "hand.point.up.left", action = toggleMode)
+ ClipboardMenu(session)
+ }
+ AnimatedVisibility(zoomed, enter = fadeIn(Motion.fade()), exit = fadeOut(Motion.fade())) {
+ OverlayButton("Fit screen", "arrow.down.right.and.arrow.up.left", action = fit)
+ }
+ val displays = state.screen?.displays.orEmpty()
+ if (displays.size > 1) {
+ DropdownMenu({ displays.map { d -> MenuItem(d.name, icon = "display", selected = d.id == display?.id) { pick(d) } } },
+ label = "Displays") { IconLabel("display.2") }
+ }
+ OverlayButton(if (interactive) "Hand back to bots" else "Take over", if (interactive) "hand.raised.slash" else "hand.raised",
+ enabled = state.screen?.input == true, action = takeOver)
+ }
+ }
+}
+
+/** Flip between portrait and landscape; leaving the viewer gives rotation back to the phone. */
+private fun rotate(activity: android.app.Activity) {
+ val landscape = activity.resources.configuration.orientation == Configuration.ORIENTATION_LANDSCAPE
+ activity.requestedOrientation = if (landscape) ActivityInfo.SCREEN_ORIENTATION_PORTRAIT else ActivityInfo.SCREEN_ORIENTATION_LANDSCAPE
+}
+
+/** Menu items have no disabled state on iOS: unavailable actions are left out instead. */
+@Composable private fun ClipboardMenu(session: ScreenSession) {
+ val clipboard = LocalContext.current.getSystemService(ClipboardManager::class.java)
+ DropdownMenu({
+ buildList {
+ // The description says whether there's text without reading it (which Android announces).
+ if (clipboard.primaryClipDescription?.hasMimeType(ClipDescription.MIMETYPE_TEXT_PLAIN) == true) {
+ add(MenuItem("Send my clipboard", icon = "arrow.up.doc.on.clipboard") {
+ clipboard.primaryClip?.getItemAt(0)?.text?.toString()?.let(session::pushClipboard)
+ })
+ }
+ if (session.remoteClipboard != null) {
+ add(MenuItem("Copy from computer", icon = "arrow.down.doc.on.clipboard") {
+ session.takeRemoteClipboard()?.let { clipboard.setPrimaryClip(ClipData.newPlainText("Codync", it)) }
+ })
+ }
+ }
+ }, label = "Clipboard") { IconLabel(if (session.remoteClipboard == null) "doc.on.clipboard" else "doc.on.clipboard.fill") }
+}
+
+@Composable private fun BoxScope.Status(session: ScreenSession, screen: ScreenState) {
+ when (val phase = session.phase) {
+ ScreenSession.Phase.Connecting -> Notice(null, "Connecting to the screen…")
+ ScreenSession.Phase.Reconnecting -> Notice(null, "Reconnecting…")
+ is ScreenSession.Phase.Failed -> Notice("exclamationmark.triangle", phase.message) { session.start() }
+ ScreenSession.Phase.Live, ScreenSession.Phase.Closed -> {
+ val text = if (!screen.input) "View only. To allow control, open Codync → Settings → Computer access on the computer." else session.lastError
+ if (text != null) Text(text, Modifier.align(Alignment.BottomCenter).navigationBarsPadding().padding(bottom = 24.dp)
+ .padding(horizontal = 16.dp).clip(RoundedCornerShape(if (screen.input) 8.dp else 12.dp)).frost(20.dp, MATERIAL)
+ .padding(if (screen.input) 8.dp else 12.dp), style = Type.caption, color = Color.White)
+ }
+ }
+}
+
+@Composable private fun BoxScope.Unavailable(state: AppState) {
+ val screen = state.screen
+ when {
+ state.connection !is LinkState.Ready -> Notice("wifi.slash", "This computer is offline.")
+ screen == null -> Notice("arrow.down.circle", "Update Codync on your computer to use its screen from here.")
+ else -> Notice("lock.display", screen.accessGuidance)
+ }
+}
+
+/** A centered icon (or a spinner), a line and an optional retry. */
+@Composable private fun BoxScope.Notice(icon: String?, text: String, retry: (() -> Unit)? = null) {
+ Column(Modifier.align(Alignment.Center).widthIn(max = 420.dp).padding(32.dp), horizontalAlignment = Alignment.CenterHorizontally,
+ verticalArrangement = Arrangement.spacedBy(12.dp)) {
+ if (icon != null) Symbol(icon, size = 40.dp, tint = Palette.secondary) else Spinner(22.dp)
+ Text(text, style = Type.callout, color = Palette.secondary, textAlign = TextAlign.Center)
+ if (retry != null) SecondaryButton(retry) { Symbol("arrow.clockwise", size = 20.dp, contentDescription = "Try again") }
+ }
+}
+
+@Composable private fun OverlayButton(label: String, icon: String, enabled: Boolean = true, action: () -> Unit) {
+ val (source, pressed) = rememberPress()
+ Box(Modifier.pressScale(pressed).graphicsLayer { alpha = if (enabled) 1f else 0.4f }
+ .clickable(source, null, enabled = enabled, role = Role.Button, onClick = action)
+ .semantics { contentDescription = label }) { IconLabel(icon) }
+}
+
+/** `.ultraThinMaterial` in a 36 pt circle, a white glyph on top. */
+@Composable private fun IconLabel(icon: String) {
+ Box(Modifier.size(36.dp).clip(CircleShape).frost(20.dp, MATERIAL), contentAlignment = Alignment.Center) {
+ Symbol(icon, size = 18.dp, tint = Color.White)
+ }
+}
+
+/** Keys the phone keyboard lacks. Modifiers arm for the next key or click. */
+@Composable private fun ModifierBar(armed: List<String>, arm: (List<String>) -> Unit, press: (String) -> Unit) {
+ Row(Modifier.fillMaxWidth().background(Color.Black.copy(alpha = 0.85f)).horizontalScroll(rememberScrollState())
+ .padding(horizontal = 12.dp, vertical = 6.dp), verticalAlignment = Alignment.CenterVertically,
+ horizontalArrangement = Arrangement.spacedBy(6.dp)) {
+ for ((name, icon) in MODIFIERS) {
+ val on = name in armed
+ BarKey(name, icon, on) { arm(if (on) armed - name else armed + name) }
+ }
+ Box(Modifier.width(0.5.dp).height(24.dp).background(Color.White.copy(alpha = 0.3f)))
+ for ((name, icon) in KEYS) BarKey(name, icon, false) { press(name) }
+ }
+}
+
+@Composable private fun BarKey(name: String, icon: String, on: Boolean, action: () -> Unit) {
+ val (source, pressed) = rememberPress()
+ val fill by animateColorAsState(if (on) Color.White else Color.White.copy(alpha = 0.12f), Motion.hover(), label = "Key")
+ Box(Modifier.pressScale(pressed).size(40.dp, 34.dp).clip(RoundedCornerShape(8.dp)).background(fill)
+ .clickable(source, null, role = Role.Button, onClick = action)
+ .semantics { contentDescription = name; selected = on }, contentAlignment = Alignment.Center) {
+ Symbol(icon, size = 18.dp, tint = if (on) Color.Black else Color.White)
+ }
+}
+
+/** `.ultraThinMaterial` over the dark screen. */
+private val MATERIAL = Color.Black.copy(alpha = 0.32f)
+private val MODIFIERS = listOf("cmd" to "command", "option" to "option", "ctrl" to "control", "shift" to "shift")
+private val KEYS = listOf("escape" to "escape", "tab" to "arrow.right.to.line", "left" to "arrow.left", "up" to "arrow.up",
+ "down" to "arrow.down", "right" to "arrow.right")
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/SettingsView.kt b/apps/android/app/src/main/kotlin/com/codync/android/SettingsView.kt
index f7d472ed..c41174a8 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/SettingsView.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/SettingsView.kt
@@ -58,7 +58,7 @@ import kotlinx.serialization.json.put
CardSection("Computers", if (signedIn) "Computers in your account need your OK on the computer before this phone can use them."
else "Each bot runs on its own computer. Sign in to see the computers in your account.") {
if (paired != null) row {
- ComputerRow(state, store, state.cloudComputers.any { it.computerId == paired.id }, extensions) { confirmForget = true }
+ ComputerRow(state, store, state.cloudComputers.any { it.computerId == paired.id }, extensions, store::openScreen) { confirmForget = true }
}
for (computer in state.cloudComputers.filter { it.computerId != paired?.id }) row {
AccountComputerRow(computer, state.access?.takeIf { it.computerId == computer.computerId }?.code, canAsk = paired == null) { ask(computer) }
@@ -71,12 +71,12 @@ import kotlinx.serialization.json.put
}
}
}
- if (state.actualConnection is LinkState.Ready) CardSection("Voice chat", "Android speech on this phone; only the text reaches the computer.") {
+ if (state.actualConnection is LinkState.Ready) CardSection("Voice chat", "Android speech on this phone, or OpenAI and Gemini on your own API key, kept on the computer.") {
row {
Row(Modifier.fillMaxWidth().plainButton { voice = true }, verticalAlignment = Alignment.CenterVertically,
horizontalArrangement = Arrangement.spacedBy(10.dp)) {
Symbol("waveform", size = 20.dp, tint = Palette.text)
- Text("Voice chat", Modifier.weight(1f), style = Type.body, color = Palette.text)
+ Text("Voice and API keys", Modifier.weight(1f), style = Type.body, color = Palette.text)
Symbol("chevron.right", size = 13.dp, tint = Palette.tertiary)
}
}
@@ -112,7 +112,7 @@ import kotlinx.serialization.json.put
}
}
}
- CodyncSheet(voice, { voice = false }) { VoiceChatSettings(store) }
+ CodyncSheet(voice, { voice = false }) { VoiceChatSettings(state, store) }
CodyncSheet(aboutAnalytics, { aboutAnalytics = false }) { AnalyticsSheet() }
CodyncDialog("Start over?", confirmStartOver, { confirmStartOver = false },
"Signs out of every account and forgets every computer on this phone, then shows the welcome again. Your computers keep their bots and chats; pair again to use them.") {
@@ -143,10 +143,12 @@ import kotlinx.serialization.json.put
}
/** The paired computer: its connection, how it's reached, and whether it's in the account. */
-@Composable private fun ComputerRow(state: AppState, store: AppStore, inAccount: Boolean, extensions: () -> Unit, remove: () -> Unit) {
+@Composable private fun ComputerRow(state: AppState, store: AppStore, inAccount: Boolean, extensions: () -> Unit, screen: () -> Unit,
+ remove: () -> Unit) {
val computer = state.computers.first()
val link = state.connection
val online = link is LinkState.Ready
+ val viewable = online && state.screen != null
val detail = if (link is LinkState.Ready) {
val count = state.bots.count { !it.deleted && it.kind != "group" }
"$count bot${if (count == 1) "" else "s"} · ${if (link.route == HostRoute.Direct) "Wi-Fi/Tailscale" else "Cloudflare"}"
@@ -154,6 +156,7 @@ import kotlinx.serialization.json.put
Row(Modifier.fillMaxWidth().contextActions(items = {
listOfNotNull(
if (online) MenuItem("Extensions", icon = "square.grid.2x2", action = extensions) else null,
+ if (viewable) MenuItem("Screen", icon = "display", action = screen) else null,
MenuItem("Remove", icon = "trash", destructive = true, divider = online, action = remove),
)
}), verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(12.dp)) {
@@ -168,6 +171,7 @@ import kotlinx.serialization.json.put
}
}
if (online) IconButton("Extensions", "square.grid.2x2", action = extensions)
+ if (viewable) IconButton("Screen", "display", action = screen)
// Which route goes first, and whether Cloudflare is a fallback at all.
DropdownMenu({ ConnectionRoute.entries.map { route -> MenuItem(route.title, icon = route.icon, selected = route == computer.route) { store.setRoute(route) } } },
Modifier.size(36.dp), label = "Connection") {
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceAudioPorts.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceAudioPorts.kt
new file mode 100644
index 00000000..0213bd66
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceAudioPorts.kt
@@ -0,0 +1,165 @@
+package com.codync.android
+
+import android.Manifest
+import android.content.Context
+import android.content.pm.PackageManager
+import android.icu.text.Transliterator
+import android.icu.util.ULocale
+import android.media.*
+import android.media.audiofx.AcousticEchoCanceler
+import android.os.LocaleList
+import androidx.core.content.ContextCompat
+import com.codync.android.core.*
+import java.io.IOException
+import java.util.Locale
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.TimeUnit
+import kotlin.concurrent.thread
+import kotlin.coroutines.resume
+import kotlin.coroutines.resumeWithException
+import kotlinx.coroutines.*
+
+// The phone's audio for calls: the microphone and the players the voice engines in core drive.
+
+internal val callAudio: AudioAttributes = AudioAttributes.Builder().setUsage(AudioAttributes.USAGE_VOICE_COMMUNICATION)
+ .setContentType(AudioAttributes.CONTENT_TYPE_SPEECH).build()
+
+/** The microphone: 16 kHz mono PCM16 frames with their level, echo-cancelled so the bot's voice isn't heard back. */
+internal class AndroidMic(private val context: Context) : MicSource {
+ private val worker = CoroutineScope(SupervisorJob() + Dispatchers.IO)
+ private var reading: Job? = null
+ @Volatile private var recorder: AudioRecord? = null
+ val recordingSessionId: Int? get() = recorder?.audioSessionId
+
+ override suspend fun start(frames: (ByteArray, Float) -> Unit, failed: () -> Unit) {
+ check(reading == null)
+ if (ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED)
+ throw IOException("Allow microphone access before starting voice chat.")
+ val microphone = withContext(Dispatchers.IO) {
+ val minimum = AudioRecord.getMinBufferSize(Pcm.RATE, AudioFormat.CHANNEL_IN_MONO, AudioFormat.ENCODING_PCM_16BIT)
+ val format = AudioFormat.Builder().setSampleRate(Pcm.RATE).setChannelMask(AudioFormat.CHANNEL_IN_MONO)
+ .setEncoding(AudioFormat.ENCODING_PCM_16BIT).build()
+ val record = try {
+ check(minimum > 0)
+ AudioRecord.Builder().setAudioSource(MediaRecorder.AudioSource.VOICE_COMMUNICATION).setAudioFormat(format)
+ .setBufferSizeInBytes(maxOf(minimum, 12_800)).build()
+ } catch (_: Exception) { throw IOException(UNAVAILABLE) }
+ if (record.state != AudioRecord.STATE_INITIALIZED) { record.release(); throw IOException(UNAVAILABLE) }
+ record
+ }
+ recorder = microphone
+ reading = worker.launch {
+ val echo = if (AcousticEchoCanceler.isAvailable()) AcousticEchoCanceler.create(microphone.audioSessionId)?.apply { enabled = true } else null
+ try {
+ microphone.startRecording()
+ check(microphone.recordingState == AudioRecord.RECORDSTATE_RECORDING)
+ val buffer = ShortArray(1_024)
+ while (isActive) {
+ val count = microphone.read(buffer, 0, buffer.size, AudioRecord.READ_BLOCKING)
+ if (!isActive) break
+ if (count < 0) throw IOException("Microphone read failed")
+ if (count == 0) { delay(10); continue }
+ frames(Pcm.bytes(buffer, count), Pcm.level(buffer, count))
+ }
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (_: Exception) { if (isActive) failed() }
+ finally {
+ runCatching { microphone.stop() }; echo?.release(); microphone.release()
+ if (recorder === microphone) recorder = null
+ }
+ }
+ }
+
+ override suspend fun stop() = withContext(NonCancellable) {
+ val old = reading; reading = null
+ old?.cancel(); runCatching { recorder?.stop() }; old?.join()
+ Unit
+ }
+
+ private companion object { const val UNAVAILABLE = "Couldn't use the microphone. Check its permission and other audio apps." }
+}
+
+/**
+ * Streams the model's 24 kHz voice. One writer thread owns the track: it writes chunks in order and
+ * reports each once the playback head has passed it, so the call knows when the model stops talking.
+ */
+internal class AndroidPcmPlayer : PcmPlayer {
+ private class Chunk(val pcm: ByteArray, val played: () -> Unit)
+ private val chunks = LinkedBlockingQueue<Chunk>()
+ @Volatile private var flushes = 0
+ @Volatile private var closed = false
+ private val track = AudioTrack.Builder().setAudioAttributes(callAudio)
+ .setAudioFormat(AudioFormat.Builder().setSampleRate(RATE).setChannelMask(AudioFormat.CHANNEL_OUT_MONO)
+ .setEncoding(AudioFormat.ENCODING_PCM_16BIT).build())
+ .setTransferMode(AudioTrack.MODE_STREAM)
+ .setBufferSizeInBytes(maxOf(AudioTrack.getMinBufferSize(RATE, AudioFormat.CHANNEL_OUT_MONO, AudioFormat.ENCODING_PCM_16BIT), RATE * 2 / 5))
+ .build().apply { play() }
+ private val writer = thread(name = "codync-call-playback") { write() }
+
+ override fun play(pcm: ByteArray, played: () -> Unit) { chunks += Chunk(pcm, played) }
+ override fun stop() { chunks.clear(); flushes++ }
+ override suspend fun close() = withContext(Dispatchers.IO) { closed = true; writer.join(); track.release() }
+
+ private fun write() {
+ val waiting = ArrayDeque<Pair<Long, () -> Unit>>()
+ var written = 0L; var flushed = flushes; var padded = false; var stalled = 0; var last = -1L
+ while (!closed) {
+ if (flushed != flushes) {
+ flushed = flushes; waiting.clear(); written = 0; padded = false; last = -1
+ track.pause(); track.flush(); track.play()
+ }
+ val next = chunks.poll(20, TimeUnit.MILLISECONDS)
+ if (next != null && flushed == flushes) {
+ track.write(next.pcm, 0, next.pcm.size and 1.inv(), AudioTrack.WRITE_BLOCKING)
+ written += next.pcm.size / 2; waiting.addLast(written to next.played); padded = false
+ }
+ val head = track.playbackHeadPosition.toLong() and 0xFFFF_FFFFL
+ while (waiting.isNotEmpty() && head >= waiting.first().first) waiting.removeFirst().second()
+ if (head != last) { last = head; stalled = 0 } else if (next == null) stalled++
+ // A short last chunk can sit under the track's start threshold: push it out with silence.
+ if (waiting.isNotEmpty() && !padded && stalled > 10) {
+ track.write(ByteArray(RATE / 5 * 2), 0, RATE / 5 * 2, AudioTrack.WRITE_BLOCKING); padded = true
+ }
+ }
+ track.pause(); track.flush()
+ }
+
+ private companion object { const val RATE = 24_000 }
+}
+
+/** Plays one text-to-speech clip from memory, through the call's audio route. */
+internal object AndroidClipPlayer : ClipPlayer {
+ override suspend fun play(audio: ByteArray, format: String) = suspendCancellableCoroutine { done ->
+ val player = MediaPlayer()
+ var released = false
+ fun release() { if (!released) { released = true; runCatching { player.stop() }; player.release() } }
+ player.setAudioAttributes(callAudio)
+ player.setDataSource(object : MediaDataSource() {
+ override fun readAt(position: Long, buffer: ByteArray, offset: Int, size: Int): Int {
+ if (position >= audio.size) return -1
+ val count = minOf(size.toLong(), audio.size - position).toInt()
+ System.arraycopy(audio, position.toInt(), buffer, offset, count)
+ return count
+ }
+ override fun getSize(): Long = audio.size.toLong()
+ override fun close() { }
+ })
+ player.setOnPreparedListener { it.start() }
+ player.setOnCompletionListener { release(); if (done.isActive) done.resume(Unit) }
+ player.setOnErrorListener { _, _, _ -> release(); if (done.isActive) done.resumeWithException(IOException("Couldn't play the $format reply.")); true }
+ done.invokeOnCancellation { player.setOnCompletionListener(null); release() }
+ player.prepareAsync()
+ }
+}
+
+/**
+ * Speech to text writes Chinese in Simplified characters whatever it's told: turn it into Traditional
+ * for someone who reads Traditional (their first Chinese language, or a Traditional region when none is listed).
+ */
+internal fun inReadersScript(text: String): String {
+ val languages = LocaleList.getDefault().let { list -> (0 until list.size()).map(list::get) }
+ val chinese = languages.firstOrNull { it.language == "zh" }
+ val traditional = if (chinese != null) ULocale.addLikelySubtags(ULocale.forLocale(chinese)).script == "Hant"
+ else Locale.getDefault().country in listOf("TW", "HK", "MO")
+ return if (traditional) Transliterator.getInstance("Hans-Hant").transliterate(text) else text
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceChatSettings.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceChatSettings.kt
new file mode 100644
index 00000000..c497f1d9
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceChatSettings.kt
@@ -0,0 +1,162 @@
+package com.codync.android
+
+import androidx.compose.animation.animateContentSize
+import androidx.compose.foundation.background
+import androidx.compose.foundation.layout.*
+import androidx.compose.material3.Text
+import androidx.compose.runtime.*
+import androidx.compose.ui.Alignment
+import androidx.compose.ui.Modifier
+import androidx.compose.ui.graphics.graphicsLayer
+import androidx.compose.ui.platform.LocalContext
+import androidx.compose.ui.unit.dp
+import com.codync.android.core.*
+import com.codync.android.design.*
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.launch
+
+/**
+ * Voice chat settings (iOS `VoiceChatSettingsView`): which voice carries calls (Android speech on this
+ * phone, or OpenAI / Gemini on the user's own key kept by the computer), the mode, models and voice.
+ * Opened from a call's gear and from Settings.
+ */
+@Composable internal fun VoiceChatSettings(state: AppState, store: AppStore) {
+ val context = LocalContext.current
+ val choices = remember { store.voiceChoices() }
+ var engine by remember { mutableStateOf(choices.provider) }
+ var settings by remember { mutableStateOf(VoicePreferences.load(context)) }
+ fun change(next: VoiceSettings) { settings = next; store.voiceSettings(next) }
+ Column(Modifier.fillMaxSize().background(Palette.background)) {
+ ModalHeader("Voice chat")
+ CardForm {
+ val provider = engine
+ var speech by remember(provider) { mutableStateOf(provider?.let(choices::speechMode) == true) }
+ CardSection("Voice", footer = provider?.let { "Your own ${it.title} key, billed by ${it.title}. Applies from the next call." }
+ ?: "Android speech, on this device. Applies from the next call.", Modifier.animateContentSize(Motion.layoutSize)) {
+ row {
+ ValueRow("Engine") {
+ ChoicePicker(provider, listOf<Pair<VoiceProvider?, String>>(null to "On device") + VoiceProvider.entries.map { it to it.title },
+ { engine = it; choices.provider = it }, label = "Engine")
+ }
+ }
+ if (provider != null) {
+ row {
+ ValueRow("Mode", detail = if (speech) "Sent when you pause, replies read aloud." else "A live conversation. Talk over replies.") {
+ ChoicePicker(speech, listOf(false to "Realtime", true to "Speech"), { speech = it; choices.speechMode(provider, it) }, label = "Mode")
+ }
+ }
+ if (speech) row { PauseRow(settings, ::change) }
+ } else {
+ row { ValueRow("Language", "English", detail = "The language you speak in calls.") }
+ row {
+ ValueRow("Reading speed") {
+ ChoicePicker(settings.rate, listOf(.84f to "Slower", 1f to "Normal", 1.12f to "Faster"),
+ { change(settings.copy(rate = it)) }, label = "Reading speed")
+ }
+ }
+ row { PauseRow(settings, ::change) }
+ }
+ }
+ if (provider != null) key(provider) { ProviderVoiceSettings(provider, speech, state, store, choices) }
+ }
+ }
+}
+
+/** How long a silence ends what you say, before it's sent. */
+@Composable private fun PauseRow(settings: VoiceSettings, change: (VoiceSettings) -> Unit) {
+ ValueRow("Send after a pause of", detail = "Longer leaves time to think mid-sentence.") {
+ ChoicePicker(settings.pauseMillis, listOf(1_000L to "1 s", 1_500L to "1.5 s", 2_500L to "2.5 s"),
+ { change(settings.copy(pauseMillis = it)) }, label = "Send after a pause of")
+ }
+}
+
+/**
+ * One provider: the key the computer keeps, models, voice and this month's minutes. Keys are entered
+ * on the computer itself (the host takes `setVoiceKey` from loopback only).
+ */
+@Composable private fun ProviderVoiceSettings(provider: VoiceProvider, speech: Boolean, state: AppState, store: AppStore, choices: VoiceChoices) {
+ val scope = rememberCoroutineScope()
+ val host = state.hostName
+ /** null while the computer hasn't answered yet. */
+ var saved by remember { mutableStateOf<Boolean?>(null) }
+ var busy by remember { mutableStateOf(false) }
+ var result by remember { mutableStateOf<Pair<Boolean, String>?>(null) }
+ var voice by remember { mutableStateOf(choices.voice(provider)) }
+ var model by remember { mutableStateOf(choices.model(provider, VoiceChoices.Model.Realtime)) }
+ var transcribe by remember { mutableStateOf(choices.model(provider, VoiceChoices.Model.Transcribe)) }
+ var speechModel by remember { mutableStateOf(choices.model(provider, VoiceChoices.Model.Speech)) }
+ var models by remember { mutableStateOf(choices.cachedModels(provider) ?: VoiceModels()) }
+ val hasKey = saved == true
+
+ suspend fun loadModels() {
+ val list = try { store.voiceModels(provider) } catch (cancelled: CancellationException) { throw cancelled } catch (_: Exception) { return }
+ models = list
+ // A newer default moves every choice the user hasn't made themselves.
+ choices.remember(list.defaults, provider)
+ model = choices.model(provider, VoiceChoices.Model.Realtime)
+ transcribe = choices.model(provider, VoiceChoices.Model.Transcribe)
+ speechModel = choices.model(provider, VoiceChoices.Model.Speech)
+ choices.cache(provider, list)
+ }
+ LaunchedEffect(provider) {
+ if (state.actualConnection !is LinkState.Ready) { saved = false; result = false to "Not connected to $host."; return@LaunchedEffect }
+ try {
+ val status = store.voiceStatus()
+ choices.remember(status); saved = status.configured(provider)
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { saved = false; result = false to (error.message ?: "Couldn't reach $host."); return@LaunchedEffect }
+ if (saved == true) loadModels()
+ }
+ // With a key saved, Test starts a call credential (or reads "OK" aloud) with the chosen model and voice.
+ fun test() {
+ if (busy || !hasKey) return
+ busy = true; result = null
+ scope.launch {
+ try {
+ if (speech) store.voiceSpeak(provider, speechModel, voice, "OK") else store.voiceSession(provider, model, voice)
+ result = true to "Key works"
+ loadModels()
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { result = false to (error.message ?: "The key didn't work.") }
+ finally { busy = false }
+ }
+ }
+ @Composable fun picker(label: String, selection: String, list: List<String>, select: (String) -> Unit) {
+ // The current choice stays listed before the provider's list arrives.
+ val options = if (selection in list) list else listOf(selection) + list
+ ValueRow(label, Modifier.graphicsLayer { alpha = if (hasKey) 1f else .55f }) {
+ ChoicePicker(selection, options.map { it to it }, select, label = label, enabled = hasKey)
+ }
+ }
+
+ CardSection(provider.title, footer = if (hasKey) null else "Add a key to choose from ${provider.title}'s models.",
+ modifier = Modifier.animateContentSize(Motion.layoutSize)) {
+ if (saved != null || result != null) row {
+ Column(verticalArrangement = Arrangement.spacedBy(8.dp)) {
+ if (hasKey) Row(verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(10.dp)) {
+ Symbol("key.fill", size = 18.dp, tint = Palette.secondary)
+ Column(Modifier.weight(1f), verticalArrangement = Arrangement.spacedBy(1.dp)) {
+ Text("Key saved", style = Type.body, color = Palette.text, maxLines = 1)
+ Text("Stays on $host", style = Type.caption, color = Palette.tertiary, maxLines = 1)
+ }
+ if (busy) Spinner() else IconButton("Test key", "checkmark.seal", action = ::test)
+ } else if (saved == false) Row(horizontalArrangement = Arrangement.spacedBy(10.dp)) {
+ Symbol("key", Modifier.padding(top = 2.dp), size = 18.dp, tint = Palette.tertiary)
+ Column(verticalArrangement = Arrangement.spacedBy(2.dp)) {
+ Text("No ${provider.title} key on $host", style = Type.body, color = Palette.text)
+ Text("Add it in Codync on that computer: Computers & devices, then Voice chat.", style = Type.caption, color = Palette.secondary)
+ }
+ }
+ result?.let { (ok, text) ->
+ IconLabel(text, if (ok) "checkmark.circle.fill" else "exclamationmark.circle.fill", if (ok) Palette.secondary else Palette.danger)
+ }
+ }
+ }
+ if (speech) {
+ row { picker("Speech to text", transcribe, models.transcribe) { transcribe = it; choices.choose(provider, VoiceChoices.Model.Transcribe, it) } }
+ row { picker("Text to speech", speechModel, models.speech) { speechModel = it; choices.choose(provider, VoiceChoices.Model.Speech, it) } }
+ } else row { picker("Model", model, models.models) { model = it; choices.choose(provider, VoiceChoices.Model.Realtime, it) } }
+ row { picker("Voice", voice, provider.voices) { voice = it; choices.voice(provider, it) } }
+ row { ValueRow("This month", "≈ ${choices.minutes(provider)} min", Modifier.graphicsLayer { alpha = if (hasKey) 1f else .55f }) }
+ }
+}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
index 9ba34606..7cc47e93 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceControls.kt
@@ -80,6 +80,11 @@ internal fun openAppSettings(context: android.content.Context) =
}
CallLevelDots(voice.level, speaking, bot?.status == "working", color,
Modifier.weight(1f).height(20.dp).semantics { contentDescription = statusText(voice, bot?.status == "working") })
+ // Audio leaves the phone: say where.
+ voice.provider?.let { provider ->
+ Text(provider.title, Modifier.semantics { contentDescription = "Voice by ${provider.title}" },
+ style = Type.caption2.semibold, color = Palette.secondary, maxLines = 1)
+ }
RoundButton("Call settings", "gearshape") { settings = true }
RoundButton(if (voice.muted) "Unmute" else "Mute", if (voice.muted) "mic.slash" else "mic", on = voice.muted, action = store::muteVoice)
Box(Modifier.size(46.dp).clip(Capsule()).background(Palette.danger).plainButton { store.endVoice() }
@@ -91,9 +96,10 @@ internal fun openAppSettings(context: android.content.Context) =
when {
failure != null -> CallLine(failure, Palette.danger)
voice.phase == VoicePhase.Interrupted -> CallLine("Audio paused by another app. Tap to resume.", Palette.secondary, store::resumeVoice)
+ call.notice != null -> CallLine(call.notice, Palette.secondary)
}
}
- CodyncSheet(settings, { settings = false }) { VoiceChatSettings(store) }
+ CodyncSheet(settings, { settings = false }) { VoiceChatSettings(state, store) }
}
@Composable private fun CallLine(text: String, color: Color, action: (() -> Unit)? = null) {
@@ -144,30 +150,3 @@ private fun statusText(voice: VoiceState, working: Boolean) = when (voice.phase)
}
}
}
-
-/** Voice chat settings (iOS `VoiceChatSettingsView`): Android speech on this phone, its pause and reading speed. */
-@Composable internal fun VoiceChatSettings(store: AppStore) {
- val context = LocalContext.current
- var settings by remember { mutableStateOf(VoicePreferences.load(context)) }
- fun change(next: VoiceSettings) { settings = next; store.voiceSettings(next) }
- Column(Modifier.fillMaxSize().background(Palette.background)) {
- ModalHeader("Voice chat")
- CardForm {
- CardSection("Voice", footer = "Android speech, on this phone. Only the text reaches the computer.") {
- row { ValueRow("Engine", "On device") }
- row {
- ValueRow("Reading speed") {
- ChoicePicker(settings.rate, listOf(.84f to "Slower", 1f to "Normal", 1.12f to "Faster"), { change(settings.copy(rate = it)) },
- label = "Reading speed")
- }
- }
- row {
- ValueRow("Send after a pause of", detail = "Longer leaves time to think mid-sentence.") {
- ChoicePicker(settings.pauseMillis, listOf(1_000L to "1 s", 1_500L to "1.5 s", 2_500L to "2.5 s"),
- { change(settings.copy(pauseMillis = it)) }, label = "Send after a pause of")
- }
- }
- }
- }
- }
-}
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt b/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt
index 210931d8..e3d855c3 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/VoiceService.kt
@@ -14,14 +14,14 @@ import com.codync.android.core.*
import kotlinx.coroutines.*
import kotlinx.coroutines.flow.collectLatest
-data class VoiceCallViewState(val id: String, val botId: String, val state: VoiceState = VoiceState())
+/** The call the UI shows; `notice` is the line under the bar, e.g. why it fell back to on-device voice. */
+data class VoiceCallViewState(val id: String, val botId: String, val state: VoiceState = VoiceState(), val notice: String? = null)
/** Started only by the visible call UI. Audio is retained in background, never restarted by Android. */
class VoiceService : Service() {
private val scope = CoroutineScope(SupervisorJob() + Dispatchers.Main.immediate)
private var store: AppStore? = null
- private var session: VoiceSession? = null
- private var audio: LocalVoiceAudio? = null
+ private var session: VoiceBinding? = null
private var callId: String? = null
private var focus: AudioFocusRequest? = null
private lateinit var manager: AudioManager
@@ -33,19 +33,19 @@ class VoiceService : Service() {
private var receivers = false
private var ending = false
private val noisy = object : BroadcastReceiver() {
- override fun onReceive(context: Context?, intent: Intent?) { session?.focus(false) }
+ override fun onReceive(context: Context?, intent: Intent?) { store?.focusVoice(false) }
}
private val devices = object : AudioDeviceCallback() {
override fun onAudioDevicesAdded(addedDevices: Array<out AudioDeviceInfo>) { chooseRoute() }
override fun onAudioDevicesRemoved(removedDevices: Array<out AudioDeviceInfo>) {
- if (removedDevices.any { it.id == selectedDevice?.id && it.type != AudioDeviceInfo.TYPE_BUILTIN_SPEAKER }) session?.focus(false)
+ if (removedDevices.any { it.id == selectedDevice?.id && it.type != AudioDeviceInfo.TYPE_BUILTIN_SPEAKER }) store?.focusVoice(false)
chooseRoute()
}
}
private val recording = object : AudioManager.AudioRecordingCallback() {
override fun onRecordingConfigChanged(configs: MutableList<AudioRecordingConfiguration>?) {
- val id = audio?.recordingSessionId ?: return
- if (configs.orEmpty().any { it.clientAudioSessionId == id && it.isClientSilenced }) session?.focus(false)
+ val id = store?.voiceRecordingSession() ?: return
+ if (configs.orEmpty().any { it.clientAudioSessionId == id && it.isClientSilenced }) store?.focusVoice(false)
}
}
override fun onCreate() {
@@ -72,9 +72,9 @@ class VoiceService : Service() {
val ownedFocus = AudioFocusRequest.Builder(AudioManager.AUDIOFOCUS_GAIN).setAudioAttributes(attributes)
.setOnAudioFocusChangeListener({ change ->
when (change) {
- AudioManager.AUDIOFOCUS_GAIN -> session?.focus(true)
+ AudioManager.AUDIOFOCUS_GAIN -> store?.focusVoice(true)
AudioManager.AUDIOFOCUS_LOSS -> end("Voice chat ended because another app took audio focus.")
- else -> session?.focus(false)
+ else -> store?.focusVoice(false)
}
}, Handler(Looper.getMainLooper())).build()
focus = ownedFocus
@@ -88,8 +88,7 @@ class VoiceService : Service() {
manager.registerAudioRecordingCallback(recording, Handler(Looper.getMainLooper()))
ContextCompat.registerReceiver(this, noisy, IntentFilter(AudioManager.ACTION_AUDIO_BECOMING_NOISY), ContextCompat.RECEIVER_NOT_EXPORTED)
receivers = true
- val engine = LocalVoiceAudio(applicationContext); audio = engine
- val call = request.store.attachVoice(request.id, request.botId, engine) ?: throw IllegalStateException("The active account changed")
+ val call = request.store.attachVoice(request.id, request.botId) ?: throw IllegalStateException("The active account changed")
session = call
scope.launch { call.state.collectLatest { state ->
if (state.phase in listOf(VoicePhase.Failed, VoicePhase.Ended)) end(state.error)
@@ -97,7 +96,7 @@ class VoiceService : Service() {
} }
} catch (_: Exception) { end("Couldn't start voice chat. Check microphone access and Android's audio settings.") }
} else if (id == callId) {
- when (intent?.action) { MUTE -> session?.let { it.mute(!it.state.value.muted) }; END -> end(null) }
+ when (intent?.action) { MUTE -> store?.muteVoice(); END -> end(null) }
} else if (session == null) stopSelf(startId)
return START_NOT_STICKY
}
@@ -132,8 +131,9 @@ class VoiceService : Service() {
return NotificationCompat.Builder(this, CHANNEL).setSmallIcon(R.drawable.ic_notification)
.setContentTitle("Voice chat · ${bot?.name ?: "Codync"}")
.setContentText(if (state.muted) "Microphone muted" else when (state.phase) {
- VoicePhase.Starting -> "Starting offline speech…"; VoicePhase.Speaking -> "Reading a reply"
- VoicePhase.Interrupted -> "Audio interrupted · return to resume"; else -> "Listening · offline English"
+ VoicePhase.Starting -> "Connecting…"; VoicePhase.Speaking -> "Speaking"
+ VoicePhase.Interrupted -> "Audio interrupted · return to resume"
+ else -> "Listening · " + (state.provider?.title ?: "on this phone")
}).setContentIntent(content).setOngoing(true).setCategory(NotificationCompat.CATEGORY_CALL)
.setVisibility(NotificationCompat.VISIBILITY_PRIVATE).setOnlyAlertOnce(true)
.addAction(0, if (state.muted) "Unmute" else "Mute", action(MUTE)).addAction(0, "End call", action(END)).build()
@@ -143,23 +143,21 @@ class VoiceService : Service() {
ending = true
val id = callId
if (id != null) store?.finishVoice(id, error)
- val call = session
- call?.end()
- scope.launch { try { call?.awaitEnd() } finally { stopSelf() } }
+ val engine = session?.engine
+ scope.launch { try { engine?.awaitEnd() } finally { stopSelf() } }
}
override fun onTaskRemoved(rootIntent: Intent?) { end(null); super.onTaskRemoved(rootIntent) }
override fun onDestroy() {
val id = callId
if (id != null) store?.finishVoice(id)
- val call = session
- call?.end(); session = null; audio = null
+ val engine = session?.engine; session = null
if (receivers) unregisterReceiver(noisy)
runCatching { manager.unregisterAudioDeviceCallback(devices) }
runCatching { manager.unregisterAudioRecordingCallback(recording) }
// A UI end or account retirement can destroy this service first. Keep
// the route and audio focus until its recorder worker has joined.
scope.launch {
- try { call?.awaitEnd() }
+ try { engine?.awaitEnd() }
finally {
runCatching { releaseRoute() }
if (currentId == id) currentId = null
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/WebRtcPeer.kt b/apps/android/app/src/main/kotlin/com/codync/android/WebRtcPeer.kt
new file mode 100644
index 00000000..e749bcfc
--- /dev/null
+++ b/apps/android/app/src/main/kotlin/com/codync/android/WebRtcPeer.kt
@@ -0,0 +1,114 @@
+package com.codync.android
+
+import android.content.Context
+import com.codync.android.core.RealtimePeer
+import java.io.IOException
+import java.nio.ByteBuffer
+import kotlin.coroutines.resume
+import kotlin.coroutines.resumeWithException
+import kotlinx.coroutines.Dispatchers
+import kotlinx.coroutines.suspendCancellableCoroutine
+import kotlinx.coroutines.withContext
+import org.webrtc.*
+import org.webrtc.audio.JavaAudioDeviceModule
+
+/**
+ * The WebRTC peer of an OpenAI Realtime call: the microphone as an audio track (hardware echo
+ * cancelling on), the model's voice played from the remote track, and the `oai-events` data channel.
+ */
+internal class WebRtcPeer(private val context: Context) : RealtimePeer {
+ private var audio: JavaAudioDeviceModule? = null
+ private var factory: PeerConnectionFactory? = null
+ private var connection: PeerConnection? = null
+ private var source: AudioSource? = null
+ private var track: AudioTrack? = null
+ private var channel: DataChannel? = null
+
+ override val isOpen: Boolean get() = channel?.state() == DataChannel.State.OPEN
+
+ override suspend fun offer(events: RealtimePeer.Events): String {
+ WebRtc.load(context)
+ val audio = JavaAudioDeviceModule.builder(context).setAudioAttributes(callAudio)
+ .setUseHardwareAcousticEchoCanceler(true).setUseHardwareNoiseSuppressor(true).createAudioDeviceModule()
+ this.audio = audio
+ val factory = PeerConnectionFactory.builder().setAudioDeviceModule(audio).createPeerConnectionFactory().also { factory = it }
+ val config = PeerConnection.RTCConfiguration(emptyList()).apply {
+ sdpSemantics = PeerConnection.SdpSemantics.UNIFIED_PLAN; bundlePolicy = PeerConnection.BundlePolicy.MAXBUNDLE
+ }
+ val connection = factory.createPeerConnection(config, Observer(events)) ?: throw IOException("Couldn't start the call audio.")
+ this.connection = connection
+ val source = factory.createAudioSource(MediaConstraints()).also { source = it }
+ track = factory.createAudioTrack("mic", source).also { connection.addTrack(it, listOf("call")) }
+ channel = connection.createDataChannel("oai-events", DataChannel.Init()).apply {
+ registerObserver(object : DataChannel.Observer {
+ override fun onBufferedAmountChange(previousAmount: Long) { }
+ override fun onStateChange() { if (state() == DataChannel.State.OPEN) events.opened() }
+ override fun onMessage(buffer: DataChannel.Buffer) {
+ val bytes = ByteArray(buffer.data.remaining()).also { buffer.data.get(it) }
+ events.received(String(bytes, Charsets.UTF_8))
+ }
+ })
+ }
+ val offer = sdp<SessionDescription> { connection.createOffer(it, MediaConstraints()) }
+ sdp<Unit> { connection.setLocalDescription(it, offer) }
+ return offer.description
+ }
+
+ override suspend fun answer(sdp: String) {
+ val connection = connection ?: throw IOException("The call ended.")
+ sdp<Unit> { connection.setRemoteDescription(it, SessionDescription(SessionDescription.Type.ANSWER, sdp)) }
+ }
+
+ override fun send(text: String) { channel?.send(DataChannel.Buffer(ByteBuffer.wrap(text.toByteArray()), false)) }
+ override fun mic(enabled: Boolean) { track?.setEnabled(enabled) }
+
+ override suspend fun close() = withContext(Dispatchers.IO) {
+ channel?.unregisterObserver(); channel?.close(); channel?.dispose(); channel = null
+ connection?.dispose(); connection = null; track = null
+ source?.dispose(); source = null
+ factory?.dispose(); factory = null
+ audio?.release(); audio = null
+ }
+
+ private class Observer(private val events: RealtimePeer.Events) : PeerConnection.Observer {
+ override fun onIceConnectionChange(state: PeerConnection.IceConnectionState) {
+ when (state) {
+ PeerConnection.IceConnectionState.CONNECTED, PeerConnection.IceConnectionState.COMPLETED -> events.ice(RealtimePeer.Ice.Connected)
+ PeerConnection.IceConnectionState.DISCONNECTED -> events.ice(RealtimePeer.Ice.Disconnected)
+ PeerConnection.IceConnectionState.FAILED -> events.ice(RealtimePeer.Ice.Failed)
+ else -> { }
+ }
+ }
+ override fun onSignalingChange(state: PeerConnection.SignalingState) { }
+ override fun onIceConnectionReceivingChange(receiving: Boolean) { }
+ override fun onIceGatheringChange(state: PeerConnection.IceGatheringState) { }
+ override fun onIceCandidate(candidate: IceCandidate) { }
+ override fun onIceCandidatesRemoved(candidates: Array<out IceCandidate>) { }
+ override fun onAddStream(stream: MediaStream) { }
+ override fun onRemoveStream(stream: MediaStream) { }
+ override fun onDataChannel(channel: DataChannel) { }
+ override fun onRenegotiationNeeded() { }
+ }
+
+ /** One SDP step as a suspending call: `create` results come back as the description, `set` as Unit. */
+ @Suppress("UNCHECKED_CAST")
+ private suspend fun <T> sdp(step: (SdpObserver) -> Unit): T = suspendCancellableCoroutine { done ->
+ step(object : SdpObserver {
+ override fun onCreateSuccess(description: SessionDescription) { if (done.isActive) done.resume(description as T) }
+ override fun onSetSuccess() { if (done.isActive) done.resume(Unit as T) }
+ override fun onCreateFailure(error: String?) { if (done.isActive) done.resumeWithException(IOException("Couldn't start the call audio.")) }
+ override fun onSetFailure(error: String?) { if (done.isActive) done.resumeWithException(IOException("OpenAI's answer didn't fit this call.")) }
+ })
+ }
+
+}
+
+/** Loads WebRTC once per process, for voice calls and the remote screen. */
+internal object WebRtc {
+ private var loaded = false
+ @Synchronized fun load(context: Context) {
+ if (loaded) return
+ PeerConnectionFactory.initialize(PeerConnectionFactory.InitializationOptions.builder(context.applicationContext).createInitializationOptions())
+ loaded = true
+ }
+}
diff --git a/apps/android/app/src/test/kotlin/com/codync/android/ScreenKeysTest.kt b/apps/android/app/src/test/kotlin/com/codync/android/ScreenKeysTest.kt
new file mode 100644
index 00000000..e9c529c8
--- /dev/null
+++ b/apps/android/app/src/test/kotlin/com/codync/android/ScreenKeysTest.kt
@@ -0,0 +1,19 @@
+package com.codync.android
+
+import android.view.KeyEvent
+import org.junit.Assert.*
+import org.junit.Test
+
+class ScreenKeysTest {
+ @Test fun `special keys and shortcuts go to the computer, plain typing stays with the field`() {
+ assertEquals("left" to emptyList<String>(), ScreenKeys.map(KeyEvent.KEYCODE_DPAD_LEFT, 0, 0))
+ assertEquals("tab" to listOf("shift"), ScreenKeys.map(KeyEvent.KEYCODE_TAB, KeyEvent.META_SHIFT_ON, 0))
+ assertEquals("f12" to emptyList<String>(), ScreenKeys.map(KeyEvent.KEYCODE_F12, 0, 0))
+ assertEquals("forwardDelete" to emptyList<String>(), ScreenKeys.map(KeyEvent.KEYCODE_FORWARD_DEL, 0, 0))
+ assertEquals("c" to listOf("cmd", "shift"), ScreenKeys.map(KeyEvent.KEYCODE_C, KeyEvent.META_META_ON or KeyEvent.META_SHIFT_ON, 'C'.code))
+ assertEquals("a" to listOf("ctrl"), ScreenKeys.map(KeyEvent.KEYCODE_A, KeyEvent.META_CTRL_ON, 'a'.code))
+ assertNull("typing goes through the IME", ScreenKeys.map(KeyEvent.KEYCODE_A, KeyEvent.META_SHIFT_ON, 'a'.code))
+ assertNull("Ctrl-Space switches the input language", ScreenKeys.map(KeyEvent.KEYCODE_SPACE, KeyEvent.META_CTRL_ON, ' '.code))
+ assertNull(ScreenKeys.map(KeyEvent.KEYCODE_VOLUME_UP, KeyEvent.META_CTRL_ON, 0))
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/CallOperator.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/CallOperator.kt
new file mode 100644
index 00000000..9ef925fe
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/CallOperator.kt
@@ -0,0 +1,142 @@
+package com.codync.android.core
+
+import kotlinx.serialization.json.*
+
+/** What a call's operator works against: the app's mirror and its send/approval actions. */
+interface CallTarget {
+ val bots: List<Bot>
+ val entries: List<Entry>
+ fun send(text: String)
+ fun respond(entry: Entry, optionId: String)
+}
+
+/** One function the realtime model can call, in JSON Schema form (both providers take the same shape). */
+data class CallTool(val name: String, val description: String, val parameters: JsonObject) {
+ fun openAI(): JsonObject = buildJsonObject {
+ put("type", "function"); put("name", name); put("description", description); put("parameters", parameters)
+ }
+ fun gemini(): JsonObject = buildJsonObject {
+ put("name", name); put("description", description)
+ if (parameters["properties"]?.jsonObject?.isNotEmpty() == true) put("parameters", geminiSchema(parameters))
+ }
+ /** Gemini's schema spells types in capitals (`OBJECT`, `STRING`). */
+ private fun geminiSchema(schema: JsonObject): JsonObject = JsonObject(schema.mapValues { (key, value) ->
+ when {
+ key == "type" && value is JsonPrimitive -> JsonPrimitive(value.content.uppercase())
+ key == "properties" && value is JsonObject -> JsonObject(value.mapValues { (_, it) -> (it as? JsonObject)?.let(::geminiSchema) ?: JsonObject(emptyMap()) })
+ else -> value
+ }
+ })
+}
+
+/**
+ * A realtime model is an operator in front of the bot, not a replacement for it (iOS `CallOperator`):
+ * the bot does the work, the operator talks. Its tools run here against the app; only `send_to_bot`
+ * messages and the bot's replies become chat entries.
+ */
+class CallOperator(private val botId: String, private val target: CallTarget) {
+ private val bot: Bot? get() = target.bots.firstOrNull { it.id == botId }
+
+ val instructions: String get() {
+ val bot = bot
+ val name = bot?.name ?: "the bot"
+ val about = if (bot == null || bot.description.isEmpty()) "" else " (${bot.description})"
+ val who = if (bot?.kind == "group") {
+ val members = target.bots.filter { it.id in bot.members }.joinToString(", ") { it.name }
+ "the group chat $name$about on the user's computer, where the bots $members answer in turn " +
+ "(a message that @-mentions one of them asks just that bot). The bots do all the actual work; you relay"
+ } else "$name$about, a coding agent running on the user's computer. $name does all the actual work; you relay"
+ return """
+ You are the voice on a phone call between the user and $who.
+ - When the user asks for something to be done, changed, checked or answered about their code or computer, call send_to_bot with their request in their own words (keep details, names and code terms exactly). Don't do the work or invent results yourself.
+ - Use bot_status and recent_messages for "what are you doing?" or "what did it say?".
+ - Call answer_approval only after the user has clearly said which option to pick.
+ - Messages starting with [$name replied] or [Notice] come from the computer: tell the user what they say, shortened for speech (no code blocks, no long lists, no URLs).
+ - Reply in the user's language. Keep spoken answers short and natural.
+ """.trimIndent()
+ }
+
+ /** How a reply is handed to the operator. */
+ fun reply(text: String): String = "[${bot?.name ?: "Bot"} replied] ${SpokenText.from(text)}"
+ fun notice(text: String): String = "[Notice] $text"
+
+ /** Runs one tool call; the result goes back to the model as JSON. */
+ fun call(name: String, arguments: JsonObject): JsonObject = when (name) {
+ "send_to_bot" -> send(arguments.text("text").orEmpty())
+ "bot_status" -> status()
+ "recent_messages" -> recent(arguments["count"]?.jsonPrimitive?.let { it.intOrNull ?: it.doubleOrNull?.toInt() } ?: 6)
+ "answer_approval" -> answer(arguments.text("option").orEmpty())
+ else -> error("Unknown tool $name.")
+ }
+
+ private fun send(text: String): JsonObject {
+ val message = text.trim()
+ if (message.isEmpty()) return error("Nothing to send.")
+ target.send(message)
+ return buildJsonObject { put("sent", true); put("note", "The bot's reply will arrive as a separate message.") }
+ }
+
+ private fun status(): JsonObject {
+ val bot = bot ?: return error("The bot isn't available.")
+ return buildJsonObject {
+ put("status", bot.status); put("activity", bot.activity)
+ pendingApproval?.let { card ->
+ putJsonObject("approval") {
+ put("request", card.data.text("title") ?: card.data.text("command") ?: "")
+ put("options", JsonArray(options(card).map { JsonPrimitive(it.second) }))
+ }
+ }
+ }
+ }
+
+ private fun recent(count: Int): JsonObject {
+ val name = bot?.name ?: "bot"
+ val messages = chat.filter(Entry::isChat).takeLast(count.coerceIn(1, 20)).map { entry ->
+ val from = when (entry.kind) {
+ "user" -> "user"
+ "agent" -> entry.data.text("author")?.let { author -> target.bots.firstOrNull { it.id == author }?.name } ?: name
+ "permission" -> "approval request"
+ else -> "notice"
+ }
+ buildJsonObject { put("from", from); put("text", SpokenText.from(entry.data.text("text") ?: entry.data.text("title") ?: "")) }
+ }
+ return buildJsonObject { put("messages", JsonArray(messages)) }
+ }
+
+ private fun answer(option: String): JsonObject {
+ val card = pendingApproval ?: return error("Nothing is waiting for approval.")
+ val options = options(card)
+ val wanted = option.lowercase()
+ val pick = options.firstOrNull { it.second.lowercase() == wanted || it.first == option }
+ ?: options.firstOrNull { it.second.lowercase().contains(wanted) || wanted.contains(it.second.lowercase()) }
+ ?: return buildJsonObject { put("error", "No such option."); put("options", JsonArray(options.map { JsonPrimitive(it.second) })) }
+ target.respond(card, pick.first)
+ return buildJsonObject { put("answered", pick.second) }
+ }
+
+ private val chat: List<Entry> get() = target.entries.filter { it.botId == botId && it.threadId == null }.sortedBy(Entry::seq)
+ private val pendingApproval: Entry? get() = chat.lastOrNull { it.kind == "permission" && it.status == "pending" }
+
+ /** A permission card's options: id → name. */
+ private fun options(card: Entry): List<Pair<String, String>> = (card.data["options"] as? JsonArray).orEmpty().mapNotNull { option ->
+ val fields = option as? JsonObject ?: return@mapNotNull null
+ val id = fields.text("optionId") ?: return@mapNotNull null
+ id to (fields.text("name") ?: id)
+ }
+
+ private fun error(message: String) = buildJsonObject { put("error", message) }
+
+ companion object {
+ val tools = listOf(
+ CallTool("send_to_bot", "Send a message to the bot on the computer, exactly as if the user typed it. Its reply arrives later.",
+ schema { putJsonObject("text") { put("type", "string"); put("description", "The message") } }.withRequired("text")),
+ CallTool("bot_status", "Whether the bot is idle or working, what it is doing, and any approval it is waiting for.", schema { }),
+ CallTool("recent_messages", "The latest messages in the chat with the bot, oldest first.",
+ schema { putJsonObject("count") { put("type", "integer"); put("description", "How many, 1–20") } }),
+ CallTool("answer_approval", "Answer the bot's pending approval request with one of its options.",
+ schema { putJsonObject("option") { put("type", "string"); put("description", "The option's name") } }.withRequired("option")),
+ )
+ private fun schema(properties: JsonObjectBuilder.() -> Unit) = buildJsonObject { put("type", "object"); putJsonObject("properties", properties) }
+ private fun JsonObject.withRequired(name: String) = JsonObject(this + ("required" to JsonArray(listOf(JsonPrimitive(name)))))
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt
new file mode 100644
index 00000000..d9d66f68
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/CloudSpeechEngine.kt
@@ -0,0 +1,108 @@
+package com.codync.android.core
+
+import kotlinx.coroutines.*
+import kotlinx.coroutines.channels.Channel
+import kotlinx.coroutines.flow.update
+
+/**
+ * The provider's speech models without a live conversation (iOS `CloudSpeechEngine`): each utterance
+ * is cut at a pause, transcribed with the provider's speech to text and sent as an ordinary message;
+ * the bot's replies are read aloud with its text to speech. Both go through the computer, which holds the key.
+ */
+class CloudSpeechEngine(parent: CoroutineScope, provider: VoiceProvider, settings: VoiceSettings,
+ private val mic: MicSource, private val player: ClipPlayer, private val speech: Speech,
+ private val clock: () -> Long = { System.nanoTime() / 1_000_000 },
+) : ProviderEngine(parent, provider, settings, "${provider.title} speech") {
+ class Speech(val transcribe: suspend (ByteArray) -> String, val synthesize: suspend (String) -> Pair<ByteArray, String>,
+ val send: (String) -> Unit)
+
+ private val detector = UtteranceDetector()
+ /** Transcriptions finish in the order they were spoken. */
+ private val utterances = Channel<ByteArray>(Channel.UNLIMITED)
+ /** The reply's clips still to fetch and play. */
+ private val queue = ArrayDeque<String>()
+ private var speaking: Job? = null
+ /** Playback just ended; the room's echo of it isn't speech. */
+ private var deafUntil = 0L
+
+ init {
+ hop {
+ mic.start({ pcm, level -> hop { heard(pcm, level) } }) {
+ hop { fail("Couldn't use the microphone. Check its permission and other audio apps.") }
+ }
+ phase(VoicePhase.Listening)
+ scope.launch {
+ try {
+ for (wav in utterances) {
+ val text = speech.transcribe(wav).trim()
+ if (!ended && text.isNotEmpty()) speech.send(text)
+ }
+ } catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { hop { fail(error.message ?: "Couldn't transcribe what you said.") } }
+ }
+ }
+ }
+
+ override fun speak(markdown: String): Boolean {
+ if (ended) return false
+ hop { say(runCatching { SpokenText.from(markdown) }.getOrDefault("")) }
+ return true
+ }
+ override fun announce(notice: String) = hop { say(notice) }
+ override fun mute(value: Boolean) = hop {
+ mutable.update { it.copy(muted = value) }
+ if (value) { detector.reset(); level(0f) }
+ }
+ override fun interrupt() = hop { if (phase == VoicePhase.Speaking) { stopSpeaking(); phase(VoicePhase.Listening) } }
+ override fun focus(active: Boolean) = hop {
+ if (!active && this.active) { stopSpeaking(); detector.reset(); level(0f); phase(VoicePhase.Interrupted) }
+ else if (active && phase == VoicePhase.Interrupted) { phase(VoicePhase.Listening); if (queue.isNotEmpty()) playQueue() }
+ }
+
+ private fun heard(pcm: ByteArray, level: Float) {
+ if (muted || phase != VoicePhase.Listening || clock() < deafUntil) return
+ level(level)
+ detector.feed(pcm, level, clock(), state.value.settings.pauseMillis)?.let { utterances.trySend(it) }
+ }
+
+ /** Queues text in clips the provider can turn into one channel message each. */
+ private fun say(text: String) {
+ if (ended || phase == VoicePhase.Starting || text.isEmpty()) return
+ queue.addAll(speechClips(text, provider.speechChunk))
+ if (phase != VoicePhase.Interrupted) playQueue()
+ }
+
+ /** Plays the queue, fetching each clip while the one before it plays. A clip that can't be made ends the reply. */
+ private fun playQueue() {
+ if (speaking != null) return
+ speaking = scope.launch {
+ coroutineScope {
+ fun fetch(text: String) = async {
+ try { speech.synthesize(text) } catch (cancelled: CancellationException) { throw cancelled } catch (_: Exception) { null }
+ }
+ var next: Deferred<Pair<ByteArray, String>?>? = null
+ while (true) {
+ val pending = next ?: queue.removeFirstOrNull()?.let(::fetch) ?: break
+ next = null
+ val clip = pending.await()
+ if (clip == null) { queue.clear(); break }
+ detector.reset(); level(0f); phase(VoicePhase.Speaking)
+ queue.removeFirstOrNull()?.let { next = fetch(it) }
+ try { player.play(clip.first, clip.second) }
+ catch (cancelled: CancellationException) { throw cancelled }
+ catch (_: Exception) { /* An unplayable clip is skipped. */ }
+ }
+ }
+ speaking = null
+ deafUntil = clock() + 400
+ if (phase == VoicePhase.Speaking) phase(VoicePhase.Listening)
+ }
+ }
+
+ private fun stopSpeaking() { speaking?.cancel(); speaking = null; queue.clear() }
+
+ override suspend fun release() {
+ stopSpeaking(); utterances.close()
+ mic.stop()
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
new file mode 100644
index 00000000..d302e054
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/GeminiLiveEngine.kt
@@ -0,0 +1,208 @@
+package com.codync.android.core
+
+import java.net.URLEncoder
+import java.util.Base64
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.flow.update
+import kotlinx.serialization.json.*
+import okhttp3.OkHttpClient
+import okhttp3.Request
+import okhttp3.Response
+import okhttp3.WebSocket
+import okhttp3.WebSocketListener
+import okio.ByteString
+
+/**
+ * Gemini Live over a WebSocket, phone ↔ Google directly (iOS `GeminiLiveEngine`): 16 kHz PCM up,
+ * 24 kHz PCM down, echo-cancelled so you can talk over it. The user's key stays on the computer,
+ * which mints one-use ephemeral tokens; a session the server ends (`goAway`) is resumed with a fresh one.
+ */
+class GeminiLiveEngine(parent: CoroutineScope, settings: VoiceSettings, private val model: String, private val voice: String,
+ private val credential: suspend () -> String, private val operator: CallOperator,
+ private val mic: MicSource, private val player: PcmPlayer, private val http: OkHttpClient,
+ private val endpoint: String = ENDPOINT,
+) : ProviderEngine(parent, VoiceProvider.Gemini, settings, "Gemini Live") {
+ private var socket: WebSocket? = null
+ /** Setup is done on the current socket. */
+ private var ready = false
+ private var resumeHandle: String? = null
+ private var reconnects = 0
+ private var audioRunning = false
+ /** Chunks handed to the player and not heard yet; bumping the generation drops late callbacks. */
+ private var queued = 0
+ private var playGeneration = 0
+ /** The model finished its turn (nothing more is coming until we or the user speak). */
+ private var turnDone = true
+ /** After a local interrupt, the rest of the turn's audio is dropped. */
+ private var dropping = false
+ private val pending = mutableListOf<String>()
+
+ init { hop { connect() } }
+
+ override fun speak(markdown: String): Boolean {
+ if (ended) return false
+ hop { pending += operator.reply(markdown); flush() }
+ return true
+ }
+ override fun announce(notice: String) = hop { pending += operator.notice(notice); flush() }
+ override fun mute(value: Boolean) = hop {
+ if (value == muted) return@hop
+ mutable.update { it.copy(muted = value, level = 0f) }
+ if (value) send(buildJsonObject { putJsonObject("realtimeInput") { put("audioStreamEnd", true) } })
+ }
+ override fun interrupt() = hop {
+ if (phase != VoicePhase.Speaking) return@hop
+ stopPlayback(); dropping = !turnDone; phase(VoicePhase.Listening)
+ }
+ override fun focus(active: Boolean) = hop {
+ if (!active && this.active) {
+ stopPlayback(); dropping = !turnDone; level(0f); phase(VoicePhase.Interrupted)
+ send(buildJsonObject { putJsonObject("realtimeInput") { put("audioStreamEnd", true) } })
+ } else if (active && phase == VoicePhase.Interrupted) { phase(VoicePhase.Listening); flush() }
+ }
+
+ private suspend fun connect() {
+ val token = credential()
+ if (ended) return
+ ready = false
+ val url = "$endpoint?access_token=" + URLEncoder.encode(token, "UTF-8")
+ socket = http.newWebSocket(Request.Builder().url(url).build(), Listener())
+ }
+
+ private inner class Listener : WebSocketListener() {
+ override fun onOpen(webSocket: WebSocket, response: Response) = hop { if (socket === webSocket) send(setup(), webSocket) }
+ override fun onMessage(webSocket: WebSocket, text: String) = hop { if (socket === webSocket) received(text) }
+ override fun onMessage(webSocket: WebSocket, bytes: ByteString) = hop { if (socket === webSocket) received(bytes.utf8()) }
+ override fun onClosing(webSocket: WebSocket, code: Int, reason: String) { webSocket.close(1000, null); hop { closed(webSocket, reason) } }
+ override fun onClosed(webSocket: WebSocket, code: Int, reason: String) = hop { closed(webSocket, reason) }
+ override fun onFailure(webSocket: WebSocket, t: Throwable, response: Response?) = hop { closed(webSocket, "") }
+ }
+
+ private suspend fun closed(webSocket: WebSocket, reason: String) {
+ if (socket !== webSocket) return
+ socket = null; ready = false
+ if (resumeHandle != null && reconnects < 3) { reconnects++; connect() }
+ else fail(reason.ifEmpty { "Lost the connection to Gemini." })
+ }
+
+ private fun setup() = buildJsonObject {
+ putJsonObject("setup") {
+ put("model", "models/$model")
+ putJsonObject("generationConfig") {
+ putJsonArray("responseModalities") { add("AUDIO") }
+ putJsonObject("speechConfig") { putJsonObject("voiceConfig") { putJsonObject("prebuiltVoiceConfig") { put("voiceName", voice) } } }
+ }
+ putJsonObject("systemInstruction") { putJsonArray("parts") { addJsonObject { put("text", operator.instructions) } } }
+ putJsonArray("tools") { addJsonObject { put("functionDeclarations", JsonArray(CallOperator.tools.map(CallTool::gemini))) } }
+ // Audio sessions are otherwise capped at 15 minutes.
+ putJsonObject("contextWindowCompression") { putJsonObject("slidingWindow") { } }
+ putJsonObject("sessionResumption") { resumeHandle?.let { put("handle", it) } }
+ }
+ }
+
+ private suspend fun received(text: String) {
+ val message = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: return
+ if ("setupComplete" in message) {
+ ready = true; reconnects = 0
+ if (!audioRunning) {
+ mic.start({ pcm, level -> hop { captured(pcm, level) } }) {
+ hop { fail("Couldn't use the microphone. Check its permission and other audio apps.") }
+ }
+ audioRunning = true
+ }
+ if (phase == VoicePhase.Starting) phase(VoicePhase.Listening)
+ flush()
+ }
+ message.objectAt("serverContent")?.let { content ->
+ if (content.flag("interrupted")) {
+ stopPlayback(); dropping = false
+ if (phase == VoicePhase.Speaking) phase(VoicePhase.Listening)
+ }
+ (content.objectAt("modelTurn")?.get("parts") as? JsonArray)?.let { parts ->
+ turnDone = false
+ for (part in parts) (part as? JsonObject)?.objectAt("inlineData")?.text("data")?.let { if (!dropping) play(it) }
+ }
+ if (content.flag("turnComplete")) {
+ turnDone = true; dropping = false
+ if (queued == 0 && phase == VoicePhase.Speaking) phase(VoicePhase.Listening)
+ flush()
+ }
+ }
+ (message.objectAt("toolCall")?.get("functionCalls") as? JsonArray)?.let { calls ->
+ val responses = calls.mapNotNull { call ->
+ val fields = call as? JsonObject ?: return@mapNotNull null
+ val id = fields.text("id") ?: return@mapNotNull null
+ val name = fields.text("name") ?: return@mapNotNull null
+ val result = operator.call(name, fields.objectAt("args") ?: JsonObject(emptyMap()))
+ buildJsonObject { put("id", id); put("name", name); putJsonObject("response") { put("result", result) } }
+ }
+ send(buildJsonObject { putJsonObject("toolResponse") { put("functionResponses", JsonArray(responses)) } })
+ }
+ message.objectAt("sessionResumptionUpdate")?.let { update ->
+ if (update.flag("resumable")) update.text("newHandle")?.let { resumeHandle = it }
+ }
+ // The server ends this connection soon; carry on in a resumed one.
+ if ("goAway" in message) socket?.cancel()
+ }
+
+ private fun send(message: JsonObject, on: WebSocket? = null) {
+ val target = on ?: socket?.takeIf { ready } ?: return
+ target.send(message.toString())
+ }
+
+ /** Replies wait for the model to finish talking; `turnComplete` here would cut it off. */
+ private fun flush() {
+ if (!ready || !turnDone || queued != 0 || pending.isEmpty() || phase == VoicePhase.Interrupted) return
+ val text = pending.joinToString("\n\n"); pending.clear()
+ turnDone = false
+ send(buildJsonObject {
+ putJsonObject("clientContent") {
+ putJsonArray("turns") { addJsonObject { put("role", "user"); putJsonArray("parts") { addJsonObject { put("text", text) } } } }
+ put("turnComplete", true)
+ }
+ })
+ }
+
+ private fun captured(pcm: ByteArray, level: Float) {
+ if (muted || !active) return
+ level(if (phase == VoicePhase.Speaking) 0f else level)
+ send(buildJsonObject {
+ putJsonObject("realtimeInput") {
+ putJsonObject("audio") { put("data", Base64.getEncoder().encodeToString(pcm)); put("mimeType", "audio/pcm;rate=16000") }
+ }
+ })
+ }
+
+ private fun play(base64: String) {
+ val pcm = runCatching { Base64.getDecoder().decode(base64) }.getOrNull()?.takeIf { it.size >= 2 } ?: return
+ queued++
+ level(0f); phase(VoicePhase.Speaking)
+ val generation = playGeneration
+ player.play(pcm) { hop { played(generation) } }
+ }
+
+ private fun played(generation: Int) {
+ if (generation != playGeneration) return
+ queued = maxOf(0, queued - 1)
+ if (queued != 0 || !turnDone) return
+ if (phase == VoicePhase.Speaking) phase(VoicePhase.Listening)
+ flush()
+ }
+
+ private fun stopPlayback() { playGeneration++; queued = 0; player.stop() }
+
+ override suspend fun release() {
+ val open = socket; socket = null; ready = false
+ open?.close(1000, null)
+ stopPlayback()
+ if (audioRunning) { audioRunning = false; mic.stop() }
+ player.close()
+ }
+
+ private fun JsonObject.objectAt(key: String): JsonObject? = this[key] as? JsonObject
+ private fun JsonObject.flag(key: String): Boolean = (this[key] as? JsonPrimitive)?.booleanOrNull == true
+
+ companion object {
+ const val ENDPOINT = "wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1alpha.GenerativeService.BidiGenerateContentConstrained"
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
new file mode 100644
index 00000000..c091846e
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/OpenAIRealtimeEngine.kt
@@ -0,0 +1,168 @@
+package com.codync.android.core
+
+import java.io.IOException
+import java.util.concurrent.TimeUnit
+import kotlinx.coroutines.*
+import kotlinx.coroutines.flow.update
+import kotlinx.serialization.json.*
+import okhttp3.MediaType.Companion.toMediaType
+import okhttp3.OkHttpClient
+import okhttp3.Request
+import okhttp3.RequestBody.Companion.toRequestBody
+
+/** The WebRTC side of an OpenAI Realtime call: the microphone track, the model's voice and the `oai-events` channel. */
+interface RealtimePeer {
+ enum class Ice { Connected, Disconnected, Failed }
+ interface Events { fun opened(); fun received(text: String); fun ice(state: Ice) }
+ /** Creates the connection and returns its SDP offer. */
+ suspend fun offer(events: Events): String
+ suspend fun answer(sdp: String)
+ val isOpen: Boolean
+ fun send(text: String)
+ fun mic(enabled: Boolean)
+ suspend fun close()
+}
+
+/**
+ * OpenAI Realtime over WebRTC, phone ↔ OpenAI directly (iOS `OpenAIRealtimeEngine`): the microphone is
+ * a WebRTC audio track, the model's voice plays from the remote track, and events (tools, replies) go
+ * over `oai-events`. The user's key stays on the computer, which mints the client secret.
+ */
+class OpenAIRealtimeEngine(parent: CoroutineScope, settings: VoiceSettings, private val credential: suspend () -> String,
+ private val operator: CallOperator, private val peer: RealtimePeer,
+ private val exchange: suspend (secret: String, offer: String) -> String,
+) : ProviderEngine(parent, VoiceProvider.OpenAI, settings, "OpenAI Realtime"), RealtimePeer.Events {
+ /** A response is being generated; new replies wait for it to finish. */
+ private var responding = false
+ /** Replies to hand over, and tool results as empty entries (a response is asked for once free). */
+ private val pending = mutableListOf<String>()
+ private var dropTimer: Job? = null
+
+ init {
+ hop {
+ val secret = credential()
+ if (ended) return@hop
+ val offer = peer.offer(this)
+ peer.mic(!muted)
+ val answer = exchange(secret, offer)
+ if (ended) return@hop
+ peer.answer(answer)
+ drop(15_000, "Couldn't reach OpenAI.") { phase == VoicePhase.Starting }
+ }
+ }
+
+ override fun speak(markdown: String): Boolean {
+ if (ended) return false
+ hop { pending += operator.reply(markdown); flush() }
+ return true
+ }
+ override fun announce(notice: String) = hop { pending += operator.notice(notice); flush() }
+ override fun mute(value: Boolean) = hop {
+ mutable.update { it.copy(muted = value, level = if (value) 0f else it.level) }
+ if (phase != VoicePhase.Interrupted) peer.mic(!value)
+ }
+ override fun interrupt() = hop { if (phase == VoicePhase.Speaking) { cancelResponse(); phase(VoicePhase.Listening) } }
+ override fun focus(active: Boolean) = hop {
+ if (!active && this.active) {
+ peer.mic(false)
+ if (phase == VoicePhase.Speaking) cancelResponse()
+ level(0f); phase(VoicePhase.Interrupted)
+ } else if (active && phase == VoicePhase.Interrupted) {
+ peer.mic(!muted); phase(VoicePhase.Listening); flush()
+ }
+ }
+
+ override fun opened() = hop {
+ dropTimer?.cancel()
+ send(buildJsonObject {
+ put("type", "session.update")
+ putJsonObject("session") {
+ put("type", "realtime")
+ put("instructions", operator.instructions)
+ put("tools", JsonArray(CallOperator.tools.map(CallTool::openAI)))
+ putJsonObject("audio") { putJsonObject("input") { putJsonObject("turn_detection") { put("type", "semantic_vad") } } }
+ }
+ })
+ phase(VoicePhase.Listening)
+ flush()
+ }
+
+ override fun received(text: String) = hop {
+ val message = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: return@hop
+ when (message.text("type")) {
+ "input_audio_buffer.speech_started" -> level(if (muted) 0f else .7f)
+ "input_audio_buffer.speech_stopped" -> level(0f)
+ "output_audio_buffer.started" -> if (phase == VoicePhase.Listening) phase(VoicePhase.Speaking)
+ "output_audio_buffer.stopped", "output_audio_buffer.cleared" -> if (phase == VoicePhase.Speaking) phase(VoicePhase.Listening)
+ "response.created" -> responding = true
+ "response.done" -> { responding = false; flush() }
+ "response.function_call_arguments.done" -> {
+ val id = message.text("call_id") ?: return@hop
+ val name = message.text("name") ?: return@hop
+ val arguments = message.text("arguments")?.let { runCatching { WireJson.parseToJsonElement(it).jsonObject }.getOrNull() }
+ val output = operator.call(name, arguments ?: JsonObject(emptyMap()))
+ send(buildJsonObject {
+ put("type", "conversation.item.create")
+ putJsonObject("item") { put("type", "function_call_output"); put("call_id", id); put("output", output.toString()) }
+ })
+ // An empty entry asks for a response once this one is done, so the model speaks the result.
+ pending += ""
+ }
+ }
+ }
+
+ override fun ice(state: RealtimePeer.Ice) = hop {
+ when (state) {
+ RealtimePeer.Ice.Connected -> dropTimer?.cancel()
+ RealtimePeer.Ice.Disconnected -> drop(5_000, "Lost the connection to OpenAI.") { true }
+ RealtimePeer.Ice.Failed -> fail("Lost the connection to OpenAI.")
+ }
+ }
+
+ /** Fails the call after `millis` unless something cancels the timer first. */
+ private fun drop(millis: Long, message: String, still: () -> Boolean) {
+ dropTimer?.cancel()
+ dropTimer = scope.launch { delay(millis); hop { if (still()) fail(message) } }
+ }
+
+ private fun cancelResponse() {
+ send(buildJsonObject { put("type", "response.cancel") })
+ send(buildJsonObject { put("type", "output_audio_buffer.clear") })
+ }
+
+ /** Hands waiting replies (and tool results, as empty entries) to the model once it's free. */
+ private fun flush() {
+ if (responding || !active || !peer.isOpen || pending.isEmpty()) return
+ for (text in pending) if (text.isNotEmpty()) send(buildJsonObject {
+ put("type", "conversation.item.create")
+ putJsonObject("item") {
+ put("type", "message"); put("role", "user")
+ putJsonArray("content") { addJsonObject { put("type", "input_text"); put("text", text) } }
+ }
+ })
+ pending.clear()
+ responding = true
+ send(buildJsonObject { put("type", "response.create") })
+ }
+
+ private fun send(event: JsonObject) { if (peer.isOpen) peer.send(event.toString()) }
+
+ override suspend fun release() { dropTimer?.cancel(); peer.close() }
+
+
+ companion object {
+ const val CALLS = "https://api.openai.com/v1/realtime/calls"
+
+ /** Posts the offer with the call's client secret and returns OpenAI's SDP answer. */
+ suspend fun answer(http: OkHttpClient, secret: String, offer: String, url: String = CALLS): String = withContext(Dispatchers.IO) {
+ val request = Request.Builder().url(url).header("Authorization", "Bearer $secret")
+ .post(offer.toByteArray().toRequestBody("application/sdp".toMediaType())).build()
+ http.newBuilder().callTimeout(15, TimeUnit.SECONDS).build().newCall(request).execute().use { response ->
+ val body = response.body?.string().orEmpty()
+ if (response.isSuccessful) return@use body
+ val message = runCatching { WireJson.parseToJsonElement(body).jsonObject["error"]?.jsonObject?.text("message") }.getOrNull()
+ throw IOException(message ?: "OpenAI answered ${response.code}.")
+ }
+ }
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt
new file mode 100644
index 00000000..c206704f
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt
@@ -0,0 +1,133 @@
+package com.codync.android.core
+
+import kotlinx.serialization.Serializable
+import kotlinx.serialization.json.JsonArray
+import kotlinx.serialization.json.JsonObject
+import kotlinx.serialization.json.JsonPrimitive
+import kotlinx.serialization.json.add
+import kotlinx.serialization.json.buildJsonArray
+import kotlinx.serialization.json.buildJsonObject
+import kotlinx.serialization.json.decodeFromJsonElement
+import kotlinx.serialization.json.doubleOrNull
+import kotlinx.serialization.json.jsonObject
+import kotlinx.serialization.json.put
+
+/** The computer's remote screen: whether phones can view/control it, and who's in control (CodyncKit `ScreenState`). */
+@Serializable
+data class ScreenState(
+ val enabled: Boolean = false,
+ /** The screen helper is running. */
+ val connected: Boolean = false,
+ val platform: String = "",
+ /** App name shown in the computer's system permission settings. */
+ val permissionApp: String? = null,
+ /** Screen recording is permitted. */
+ val capture: Boolean = false,
+ /** Input injection is permitted. */
+ val input: Boolean = false,
+ val displays: List<ScreenDisplay> = emptyList(),
+ /** A phone took over: bots may only look. */
+ val userControl: Boolean = false,
+ /** The bot using the computer right now. */
+ val agentBot: String? = null,
+ /** Bots with computer use can act here (Windows: always; elsewhere Remote screen is on). */
+ val computerUse: Boolean = false,
+ val viewers: Int = 0,
+) {
+ val available: Boolean get() = enabled && connected && capture
+ val controlReady: Boolean get() = if (platform == "windows") computerUse else available && input
+ val mainDisplay: ScreenDisplay? get() = displays.firstOrNull { it.main } ?: displays.firstOrNull()
+
+ /** Remote clients explain setup; only the computer itself can request OS permissions. */
+ val accessGuidance: String get() {
+ if (platform == "windows") return "Remote screen viewing is not available on Windows yet. Bots can still use apps on that computer."
+ val setup = "On the computer, open Codync → Settings → Computer access. Permissions are set separately on each computer."
+ val app = permissionApp ?: "Codync Screen"
+ return when {
+ !enabled -> "Remote screen is off. $setup"
+ !connected -> "Codync Screen is not running. $setup"
+ platform == "linux" && (!capture || !input) -> "Choose a screen and allow control in the system sharing dialog. $setup"
+ !capture -> "Allow screen recording for $app. $setup"
+ !input -> "Allow computer control for $app. macOS may call this Device Control and Data Access or Accessibility. $setup"
+ else -> "Computer access is ready."
+ }
+ }
+
+ companion object {
+ fun decode(value: JsonObject): ScreenState = WireJson.decodeFromJsonElement(value)
+ }
+}
+
+@Serializable
+data class ScreenDisplay(
+ val id: Long,
+ val name: String,
+ /** Size in points: the coordinate space of screen input. */
+ val width: Double,
+ val height: Double,
+ val main: Boolean = false,
+)
+
+@Serializable data class ScreenIceServer(val urls: List<String>, val username: String? = null, val credential: String? = null)
+
+/** A reserved viewer session and the ICE servers to reach it (`screenPrepare`). */
+@Serializable
+data class ScreenConnection(val session: String, val iceServers: List<ScreenIceServer>, /** Unix milliseconds. */ val expiresAt: Long) {
+ /** Renew five minutes before the TURN credentials expire, never sooner than a second from now. */
+ fun renewIn(nowMillis: Long): Long = maxOf(1_000, expiresAt - nowMillis - 300_000)
+}
+
+@Serializable data class ScreenAnswer(val session: String, val sdp: String)
+
+/** What the computer said on the `input` channel. */
+sealed interface ScreenReply {
+ /** Text the computer copied. */
+ data class Clipboard(val text: String?) : ScreenReply
+ /** The region the video now shows (`null`: the whole display). */
+ data class View(val crop: ScreenRect?) : ScreenReply
+ data class Error(val message: String?) : ScreenReply
+
+ companion object {
+ fun decode(text: String): ScreenReply? {
+ val msg = runCatching { WireJson.parseToJsonElement(text).jsonObject }.getOrNull() ?: return null
+ return when (msg.text("type")) {
+ "clipboard" -> Clipboard(msg.text("text"))
+ "view" -> View((msg["crop"] as? JsonArray)?.mapNotNull { (it as? JsonPrimitive)?.doubleOrNull }
+ ?.takeIf { it.size == 4 }?.let { ScreenRect(it[0], it[1], it[2], it[3]) })
+ "error" -> Error(msg.text("message"))
+ else -> null
+ }
+ }
+ }
+}
+
+/** The messages a viewer sends besides pointer and keys (`ScreenSession.view`, `switchDisplay`, `pushClipboard`). */
+object ScreenRequests {
+ /** Stream a region of the display (points; `null` = all of it) at the viewer's pixel size. */
+ fun view(display: Long?, crop: ScreenRect?, pixels: ScreenSize): JsonObject = buildJsonObject {
+ put("type", "view")
+ put("long", maxOf(pixels.width, pixels.height))
+ put("short", minOf(pixels.width, pixels.height))
+ if (display != null) put("display", display)
+ if (crop != null) put("crop", buildJsonArray { add(crop.x); add(crop.y); add(crop.width); add(crop.height) })
+ }
+
+ fun switchDisplay(display: Long): JsonObject = buildJsonObject { put("type", "view"); put("display", display) }
+
+ fun clipboard(text: String): JsonObject = buildJsonObject { put("type", "clipboard"); put("text", text) }
+
+ /** A key by name (`return`, `left`, `c`) with the modifiers held for it. */
+ fun key(name: String, modifiers: List<String>): JsonObject = buildJsonObject {
+ put("type", "key"); put("key", name)
+ put("modifiers", buildJsonArray { modifiers.forEach { add(it) } })
+ }
+}
+
+/**
+ * Hardware H.264 first (High, then Constrained Baseline), every other codec only as a fallback.
+ * `profile` reads a codec's `profile-level-id`.
+ */
+fun <T> preferredScreenCodecs(codecs: List<T>, name: (T) -> String, profile: (T) -> String?): List<T> {
+ val h264 = codecs.filter { name(it) == "H264" }.sortedByDescending { profile(it).orEmpty().startsWith("640c") }
+ return h264 + codecs.filter { name(it) != "H264" }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenControl.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenControl.kt
new file mode 100644
index 00000000..60335236
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenControl.kt
@@ -0,0 +1,221 @@
+package com.codync.android.core
+
+import kotlinx.serialization.json.JsonObject
+import kotlinx.serialization.json.add
+import kotlinx.serialization.json.buildJsonObject
+import kotlinx.serialization.json.put
+import kotlinx.serialization.json.putJsonArray
+import kotlin.math.hypot
+import kotlin.math.max
+import kotlin.math.min
+
+enum class TouchMode {
+ /** The phone is a trackpad: drag moves the pointer, tap clicks where it is. */
+ Trackpad,
+ /** Tap clicks where you touch. */
+ Direct,
+}
+
+/**
+ * Every touch and key gesture on the remote screen turned into input for the computer, in
+ * display points (CodyncUI `ScreenCanvas` without its view). Gestures arrive in view points.
+ */
+class ScreenControl(private val send: (JsonObject) -> Unit) {
+ var mode = TouchMode.Trackpad
+ var interactive = true
+ /** One-shot modifiers from the on-screen bar, applied to the next key or click. */
+ var armed: List<String> = emptyList()
+ var onArmedConsumed: () -> Unit = {}
+ /** The video moved; `request` also asks for the visible region once the gesture settles. */
+ var onChanged: (request: Boolean) -> Unit = {}
+
+ var viewport = ScreenViewport(ScreenSize(1.0, 1.0), ScreenSize(1.0, 1.0))
+ private set
+ /** Mid-pinch the region request waits for the fingers to lift. */
+ var pinching = false
+ private set
+ private var streamRegion: ScreenRect? = null
+ private var cursor = ScreenPoint(0.0, 0.0)
+ private var lastTap: Tap? = null
+ private var dragging = false
+ private var lastPress = ScreenPoint(0.0, 0.0)
+
+ private class Tap(val at: Long, val point: ScreenPoint, val count: Int)
+
+ val zoomed: Boolean get() = viewport.crop != null
+ /** Where the streamed region sits in the view. */
+ val videoFrame: ScreenRect get() = viewport.frame(streamRegion ?: viewport.displayBounds)
+
+ // MARK: state from the view
+
+ fun setDisplay(d: ScreenDisplay) {
+ val size = ScreenSize(d.width, d.height)
+ if (size == viewport.display) return
+ viewport = ScreenViewport(size, viewport.view)
+ cursor = ScreenPoint(d.width / 2, d.height / 2)
+ streamRegion = null
+ onChanged(true)
+ }
+
+ fun setViewSize(size: ScreenSize) {
+ if (size == viewport.view) return
+ viewport = viewport.resized(size)
+ onChanged(true)
+ }
+
+ /** The region the computer confirmed it streams (`null`: the whole display). */
+ fun setStreamRegion(crop: ScreenRect?) {
+ if (crop == streamRegion) return
+ streamRegion = crop
+ onChanged(false)
+ }
+
+ fun resetZoom() {
+ viewport = viewport.reset()
+ onChanged(true)
+ }
+
+ // MARK: gestures
+
+ /** One finger moved by `delta` at `speed` view points a second. */
+ fun onePan(delta: ScreenPoint, speed: Double) {
+ if (!interactive) return pan(delta)
+ when (mode) {
+ TouchMode.Trackpad -> {
+ // Pointer speed grows with finger speed, like a real trackpad.
+ val accel = min(2.5, max(1.0, speed / 900))
+ moveCursor(ScreenPoint(delta.x * accel / viewport.scale, delta.y * accel / viewport.scale))
+ }
+ TouchMode.Direct -> if (zoomed) pan(delta)
+ else send(scroll(cursor, -delta.x, -delta.y))
+ }
+ }
+
+ /** Successive quick taps become double/triple clicks without waiting to tell them apart. */
+ fun tap(at: ScreenPoint, nowMillis: Long) {
+ if (!interactive) return
+ val p = point(at)
+ cursor = p
+ val last = lastTap
+ val count = if (last != null && nowMillis - last.at < 350 && hypot(last.point.x - p.x, last.point.y - p.y) < 8 / viewport.scale)
+ min(last.count + 1, 3) else 1
+ lastTap = Tap(nowMillis, p, count)
+ send(click(p, "left", count))
+ }
+
+ fun twoFingerTap(at: ScreenPoint) {
+ if (!interactive) return
+ send(click(point(at), "right", 1))
+ }
+
+ /** Long-press then move: drag (select text, move windows). Returns whether a drag began (for the haptic). */
+ fun pressBegan(at: ScreenPoint): Boolean {
+ if (!interactive) return false
+ val p = point(at)
+ cursor = p
+ dragging = true
+ send(button("down", p))
+ lastPress = at
+ return true
+ }
+
+ fun pressMoved(at: ScreenPoint) {
+ if (!dragging) return
+ if (mode == TouchMode.Direct) {
+ cursor = clampToDisplay(viewport.toDisplay(at))
+ send(move(cursor))
+ } else {
+ moveCursor(ScreenPoint((at.x - lastPress.x) / viewport.scale, (at.y - lastPress.y) / viewport.scale))
+ }
+ lastPress = at
+ }
+
+ /** A drag that began while interactive still lifts the button, or it stays held on the computer. */
+ fun pressEnded() {
+ if (!dragging) return
+ send(button("up", cursor))
+ dragging = false
+ }
+
+ /** Two fingers moved: scroll the computer, or move the view while zooming or only looking. */
+ fun twoPan(delta: ScreenPoint, at: ScreenPoint) {
+ if (pinching || !interactive) return pan(delta)
+ // Natural scrolling: content follows the fingers.
+ val p = if (mode == TouchMode.Direct) clampToDisplay(viewport.toDisplay(at)) else cursor
+ send(scroll(p, -delta.x * 1.5, -delta.y * 1.5))
+ }
+
+ fun pinchBegan() { pinching = true }
+
+ fun pinch(factor: Double, at: ScreenPoint) {
+ viewport = viewport.zoomed(factor, at)
+ onChanged(false)
+ }
+
+ fun pinchEnded() {
+ pinching = false
+ onChanged(true)
+ }
+
+ // MARK: keys
+
+ /** Committed text: one character with armed modifiers is a shortcut. */
+ fun typed(text: String) {
+ if (!interactive) return
+ val mods = consumeArmed()
+ send(if (mods.isNotEmpty() && text.codePointCount(0, text.length) == 1) ScreenRequests.key(text.lowercase(), mods)
+ else buildJsonObject { put("type", "text"); put("text", text) })
+ }
+
+ fun pressKey(name: String, modifiers: List<String> = emptyList()) {
+ if (!interactive) return
+ send(ScreenRequests.key(name, (modifiers + consumeArmed()).distinct()))
+ }
+
+ private fun consumeArmed(): List<String> {
+ val mods = armed
+ if (mods.isNotEmpty()) {
+ armed = emptyList()
+ onArmedConsumed()
+ }
+ return mods
+ }
+
+ // MARK: helpers
+
+ private fun pan(delta: ScreenPoint) {
+ viewport = viewport.panned(delta)
+ onChanged(true)
+ }
+
+ private fun moveCursor(d: ScreenPoint) {
+ val bounds = viewport.displayBounds
+ cursor = ScreenPoint(min(max(cursor.x + d.x, 0.0), bounds.width - 1), min(max(cursor.y + d.y, 0.0), bounds.height - 1))
+ send(move(cursor))
+ val followed = viewport.following(cursor)
+ if (followed != viewport) {
+ viewport = followed
+ onChanged(true)
+ }
+ }
+
+ private fun point(at: ScreenPoint) = if (mode == TouchMode.Direct) clampToDisplay(viewport.toDisplay(at)) else cursor
+
+ private fun clampToDisplay(p: ScreenPoint) =
+ ScreenPoint(min(max(p.x, 0.0), viewport.display.width - 1), min(max(p.y, 0.0), viewport.display.height - 1))
+
+ private fun move(p: ScreenPoint) = buildJsonObject { put("type", "move"); put("x", p.x); put("y", p.y) }
+
+ private fun button(type: String, p: ScreenPoint) = buildJsonObject {
+ put("type", type); put("x", p.x); put("y", p.y); put("button", "left")
+ }
+
+ private fun click(p: ScreenPoint, button: String, count: Int) = buildJsonObject {
+ put("type", "click"); put("x", p.x); put("y", p.y); put("button", button); put("count", count)
+ putJsonArray("modifiers") { consumeArmed().forEach { add(it) } }
+ }
+
+ private fun scroll(p: ScreenPoint, dx: Double, dy: Double) = buildJsonObject {
+ put("type", "scroll"); put("x", p.x); put("y", p.y); put("dx", dx); put("dy", dy); put("units", "pixel")
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenViewport.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenViewport.kt
new file mode 100644
index 00000000..1c575094
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenViewport.kt
@@ -0,0 +1,118 @@
+package com.codync.android.core
+
+import kotlin.math.max
+import kotlin.math.min
+
+data class ScreenPoint(val x: Double, val y: Double)
+data class ScreenSize(val width: Double, val height: Double)
+data class ScreenRect(val x: Double, val y: Double, val width: Double, val height: Double) {
+ val maxX: Double get() = x + width
+ val maxY: Double get() = y + height
+
+ fun intersection(other: ScreenRect): ScreenRect {
+ val left = max(x, other.x)
+ val top = max(y, other.y)
+ return ScreenRect(left, top, max(0.0, min(maxX, other.maxX) - left), max(0.0, min(maxY, other.maxY) - top))
+ }
+}
+
+/**
+ * Which part of a remote display the viewer shows, and the mapping between view points and
+ * display points (CodyncKit `ScreenViewport`). Zoom 1 fits the whole display; beyond that the
+ * visible region takes the view's aspect and is kept on the display.
+ */
+data class ScreenViewport(
+ /** Display size in points. */
+ val display: ScreenSize,
+ /** Viewer size in points. */
+ val view: ScreenSize,
+ val zoom: Double = 1.0,
+ /** The display point at the middle of the view. */
+ val center: ScreenPoint = ScreenPoint(display.width / 2, display.height / 2),
+) {
+ private val fitScale: Double get() {
+ // Before layout the view is 0×0: any positive scale keeps the math finite.
+ if (display.width <= 0 || display.height <= 0 || view.width <= 0 || view.height <= 0) return 1.0
+ return min(view.width / display.width, view.height / display.height)
+ }
+
+ val maxZoom: Double get() = max(1.0, MAX_POINT_SCALE / max(fitScale, 0.0001))
+
+ /** View points per display point. */
+ val scale: Double get() = fitScale * zoom
+
+ /** The display region the view covers (may extend past the display when letterboxed). */
+ val visible: ScreenRect get() {
+ val w = view.width / scale
+ val h = view.height / scale
+ return ScreenRect(center.x - w / 2, center.y - h / 2, w, h)
+ }
+
+ val displayBounds: ScreenRect get() = ScreenRect(0.0, 0.0, display.width, display.height)
+
+ /** The region to stream: `null` while the whole display is in view. */
+ val crop: ScreenRect? get() = if (zoom <= 1.001) null else visible.intersection(displayBounds)
+
+ fun toDisplay(p: ScreenPoint): ScreenPoint {
+ val v = visible
+ return ScreenPoint(v.x + p.x / scale, v.y + p.y / scale)
+ }
+
+ fun toView(p: ScreenPoint): ScreenPoint {
+ val v = visible
+ return ScreenPoint((p.x - v.x) * scale, (p.y - v.y) * scale)
+ }
+
+ /** Where a streamed region (display points) sits in the view. */
+ fun frame(region: ScreenRect): ScreenRect {
+ val o = toView(ScreenPoint(region.x, region.y))
+ return ScreenRect(o.x, o.y, region.width * scale, region.height * scale)
+ }
+
+ /** Pinch: zoom by `factor` keeping the display point under `anchor` (view point) in place. */
+ fun zoomed(factor: Double, anchor: ScreenPoint): ScreenViewport {
+ val fixed = toDisplay(anchor)
+ val next = copy(zoom = min(max(zoom * factor, 1.0), maxZoom))
+ val s = next.scale
+ return next.copy(center = ScreenPoint(fixed.x - (anchor.x - view.width / 2) / s, fixed.y - (anchor.y - view.height / 2) / s)).clamped()
+ }
+
+ /** Moves the view by a drag of `delta` view points. */
+ fun panned(delta: ScreenPoint): ScreenViewport =
+ copy(center = ScreenPoint(center.x - delta.x / scale, center.y - delta.y / scale)).clamped()
+
+ /** Scrolls just enough to keep `p` (display point) inside the inner 80% of the view. */
+ fun following(p: ScreenPoint): ScreenViewport {
+ val v = visible
+ val mx = v.width * 0.1
+ val my = v.height * 0.1
+ val cx = when {
+ p.x < v.x + mx -> center.x - (v.x + mx - p.x)
+ p.x > v.maxX - mx -> center.x + (p.x - (v.maxX - mx))
+ else -> center.x
+ }
+ val cy = when {
+ p.y < v.y + my -> center.y - (v.y + my - p.y)
+ p.y > v.maxY - my -> center.y + (p.y - (v.maxY - my))
+ else -> center.y
+ }
+ return copy(center = ScreenPoint(cx, cy)).clamped()
+ }
+
+ fun reset(): ScreenViewport = copy(zoom = 1.0, center = ScreenPoint(display.width / 2, display.height / 2))
+
+ /** The same region on a resized view (rotation, the keyboard), kept on the display. */
+ fun resized(view: ScreenSize): ScreenViewport = copy(view = view).clamped()
+
+ private fun clamped(): ScreenViewport {
+ val v = visible
+ val x = if (v.width >= display.width) display.width / 2 else min(max(center.x, v.width / 2), display.width - v.width / 2)
+ val y = if (v.height >= display.height) display.height / 2 else min(max(center.y, v.height / 2), display.height - v.height / 2)
+ return copy(center = ScreenPoint(x, y))
+ }
+
+ private companion object {
+ /** Closest zoom: one display point spans this many view points. */
+ const val MAX_POINT_SCALE = 4.0
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
index ca0db43d..ede85338 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Voice.kt
@@ -44,18 +44,26 @@ object SpokenText {
}
}
+/** Something a call says: a bot's reply, or a notice from the computer. */
+data class VoiceCue(val text: String, val notice: Boolean = false)
+
/** Only newly finalized main-chat replies are spoken. The mirror remains the history owner. */
class VoiceReplyGate(private val botId: String, private val startRev: Long, initial: List<Entry>) {
private var previous = initial.associateBy(Entry::id)
- fun observe(entries: List<Entry>): List<String> {
- val result = mutableListOf<String>()
+ fun observe(entries: List<Entry>, bots: List<Bot>): List<VoiceCue> {
+ val result = mutableListOf<VoiceCue>()
+ val group = bots.firstOrNull { it.id == botId }?.kind == "group"
for (entry in entries) {
if (entry.botId != botId || entry.threadId != null || entry.rev <= startRev) continue
val before = previous[entry.id]
+ val author = entry.data.text("author")?.let { id -> bots.firstOrNull { it.id == id }?.name }
if (entry.kind == "agent" && entry.data["final"]?.jsonPrimitive?.booleanOrNull == true &&
- before?.data?.get("final")?.jsonPrimitive?.booleanOrNull != true && entry.text.isNotBlank()) result.add(entry.text)
- if (entry.kind == "permission" && entry.status == "pending" && before?.status != "pending")
- result.add("Your bot needs approval. Open the conversation to allow or deny the request.")
+ before?.data?.get("final")?.jsonPrimitive?.booleanOrNull != true && entry.text.isNotBlank())
+ result.add(VoiceCue(if (group && author != null) "$author: ${entry.text}" else entry.text))
+ if (entry.kind == "permission" && entry.status == "pending" && before?.status != "pending") {
+ val name = author ?: bots.firstOrNull { it.id == botId && !group }?.name ?: "A bot"
+ result.add(VoiceCue("$name needs your approval in the chat.", notice = true))
+ }
}
previous = entries.associateBy(Entry::id)
return result
@@ -68,7 +76,9 @@ data class VoiceSettings(val pauseMillis: Long = 1_500, val rate: Float = 1f) {
}
data class VoiceState(val phase: VoicePhase = VoicePhase.Starting, val muted: Boolean = false,
val heard: String = "", val said: String = "", val level: Float = 0f, val engine: String = "",
- val settings: VoiceSettings = VoiceSettings(), val error: String? = null)
+ val settings: VoiceSettings = VoiceSettings(), val error: String? = null,
+ /** Where the audio goes when it leaves the phone; null on device. */
+ val provider: VoiceProvider? = null)
sealed interface VoiceInput {
data class Text(val generation: Long, val value: String, val final: Boolean) : VoiceInput
data class Level(val generation: Long, val value: Float) : VoiceInput
@@ -85,7 +95,7 @@ interface VoiceAudio {
/** Serial audio ownership: no late recognizer/TTS callback can submit or restart a retired call. */
class VoiceSession(parent: CoroutineScope, private val audio: VoiceAudio, settings: VoiceSettings,
- private val send: (String) -> Unit) {
+ private val send: (String) -> Unit) : VoiceEngine {
private sealed interface Event {
data class Input(val value: VoiceInput) : Event
data class Submit(val generation: Long, val revision: Long) : Event
@@ -100,7 +110,7 @@ class VoiceSession(parent: CoroutineScope, private val audio: VoiceAudio, settin
private val scope = CoroutineScope(parent.coroutineContext + job)
private val commands = Channel<Event>(64)
private val mutableState = MutableStateFlow(VoiceState(settings = settings))
- val state = mutableState.asStateFlow()
+ override val state = mutableState.asStateFlow()
private var generation = 0L; private var revision = 0L; private var utterance = 0L
private var silence: Job? = null
private val replies = ArrayDeque<String>()
@@ -128,13 +138,14 @@ class VoiceSession(parent: CoroutineScope, private val audio: VoiceAudio, settin
}
} }
private fun post(event: Event): Boolean = job.isActive && commands.trySend(event).isSuccess
- fun speak(markdown: String): Boolean = post(Event.Reply(markdown))
- fun mute(value: Boolean) { post(Event.Mute(value)) }
- fun settings(value: VoiceSettings) { post(Event.Settings(value)) }
- fun focus(active: Boolean) { post(Event.Focus(active)) }
- fun interrupt() { post(Event.Interrupt) }
- fun end() { job.cancel() }
- suspend fun awaitEnd() { job.join() }
+ override fun speak(markdown: String): Boolean = post(Event.Reply(markdown))
+ override fun announce(notice: String) { post(Event.Reply(notice)) }
+ override fun mute(value: Boolean) { post(Event.Mute(value)) }
+ override fun settings(value: VoiceSettings) { post(Event.Settings(value)) }
+ override fun focus(active: Boolean) { post(Event.Focus(active)) }
+ override fun interrupt() { post(Event.Interrupt) }
+ override fun end() { job.cancel() }
+ override suspend fun awaitEnd() { job.join() }
private suspend fun stopListening() {
generation++; revision++; silence?.cancel(); silence = null
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt
new file mode 100644
index 00000000..7eab2111
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceEngine.kt
@@ -0,0 +1,98 @@
+package com.codync.android.core
+
+import kotlinx.coroutines.*
+import kotlinx.coroutines.channels.Channel
+import kotlinx.coroutines.flow.*
+
+/**
+ * What carries a call's audio (iOS `VoiceEngine`): on-device speech (`VoiceSession`), or a provider
+ * on the user's own key (`CloudSpeechEngine`, `GeminiLiveEngine`, `OpenAIRealtimeEngine`).
+ * The call bar and the voice service only talk to this.
+ */
+interface VoiceEngine {
+ val state: StateFlow<VoiceState>
+ /** A new final reply from the bot to say; false once the call has ended. */
+ fun speak(markdown: String): Boolean
+ /** A notice from the computer (an approval is waiting) to say. */
+ fun announce(notice: String)
+ fun mute(value: Boolean)
+ fun settings(value: VoiceSettings)
+ /** Android audio focus: false pauses the call's audio, true resumes it. */
+ fun focus(active: Boolean)
+ /** Cuts what's being said short. */
+ fun interrupt()
+ fun end()
+ suspend fun awaitEnd()
+}
+
+/** The microphone as 16 kHz mono PCM16 frames with their loudness (0…1), delivered on an audio thread. */
+interface MicSource {
+ /** Throws when the microphone can't open; `failed` reports it stopping later. */
+ suspend fun start(frames: (ByteArray, Float) -> Unit, failed: () -> Unit)
+ suspend fun stop()
+}
+
+/** Streams 24 kHz mono PCM16. `played` runs (on any thread) once a chunk has been heard; `stop` drops what's queued. */
+interface PcmPlayer {
+ fun play(pcm: ByteArray, played: () -> Unit)
+ fun stop()
+ suspend fun close()
+}
+
+/** Plays one clip (`aac` or `wav`) to its end; cancelling stops it. */
+fun interface ClipPlayer { suspend fun play(audio: ByteArray, format: String) }
+
+/**
+ * The lifecycle shared by the provider engines: one state, and one inbox that runs every callback
+ * from the audio and network threads in order on the caller's dispatcher, so engine fields are only
+ * touched there. Ports are released once, on `end()` or a failure.
+ */
+abstract class ProviderEngine(parent: CoroutineScope, val provider: VoiceProvider, settings: VoiceSettings, label: String) : VoiceEngine {
+ private val job = SupervisorJob(parent.coroutineContext[Job])
+ protected val scope = CoroutineScope(parent.coroutineContext + job)
+ protected val mutable = MutableStateFlow(VoiceState(engine = label, settings = settings, provider = provider))
+ final override val state = mutable.asStateFlow()
+ private val inbox = Channel<suspend () -> Unit>(Channel.UNLIMITED)
+ private var teardown: Job? = null
+ protected var ended = false; private set
+ protected val phase: VoicePhase get() = state.value.phase
+ protected val muted: Boolean get() = state.value.muted
+ protected val active: Boolean get() = phase == VoicePhase.Listening || phase == VoicePhase.Speaking
+
+ init {
+ scope.launch {
+ for (task in inbox) {
+ if (ended) continue
+ try { task() }
+ catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) { fail(error.message ?: "Couldn't reach ${provider.title}.") }
+ }
+ }
+ }
+
+ /** Runs `block` in the inbox: from a port's thread, or from the app. */
+ protected fun hop(block: suspend () -> Unit) { if (!ended) inbox.trySend(block) }
+ protected fun phase(value: VoicePhase) = mutable.update { it.copy(phase = value) }
+ protected fun level(value: Float) = mutable.update { it.copy(level = value) }
+ /** Stops the audio and the connection; may run twice. */
+ protected abstract suspend fun release()
+
+ protected fun fail(message: String) { if (!ended) finish(VoicePhase.Failed, message) }
+ override fun end() { if (!ended) finish(VoicePhase.Ended, null) }
+ override suspend fun awaitEnd() { teardown?.join() ?: job.join() }
+ override fun settings(value: VoiceSettings) = mutable.update { it.copy(settings = value) }
+
+ private fun finish(final: VoicePhase, error: String?) {
+ ended = true
+ if (error != null) mutable.update { it.copy(phase = VoicePhase.Failed, error = error, level = 0f) }
+ // Outside `job`, so cancelling the engine's own work can't cut the release short.
+ teardown = scope.launch(NonCancellable) {
+ try { release() }
+ catch (_: Exception) { mutable.update { it.copy(error = it.error ?: "Voice audio couldn't finish closing. Start a new call when Android releases the microphone.") } }
+ finally {
+ if (final == VoicePhase.Ended) mutable.update { it.copy(phase = VoicePhase.Ended, level = 0f) }
+ inbox.close(); job.cancel()
+ }
+ }
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceProvider.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceProvider.kt
new file mode 100644
index 00000000..b94f91e0
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/VoiceProvider.kt
@@ -0,0 +1,170 @@
+package com.codync.android.core
+
+import java.io.ByteArrayOutputStream
+import java.nio.ByteBuffer
+import java.nio.ByteOrder
+import java.time.YearMonth
+import kotlin.math.log10
+import kotlinx.serialization.Serializable
+
+// Realtime voice on the user's own provider keys (host/src/voice.rs). The keys stay on the
+// computer; a call asks the host for a short-lived credential and streams audio to the provider.
+
+enum class VoiceProvider(val id: String, val title: String, val keyPage: String, val keyPrompt: String,
+ /** Used until the user picks one, and until the computer has said what's newest (`VoiceDefaults`). */
+ val model: String, val transcribeModel: String, val speechModel: String,
+ /** Longest stretch of a reply turned into audio at once: each clip has to fit one channel message (1 MiB). */
+ val speechChunk: Int, val voices: List<String>) {
+ OpenAI("openai", "OpenAI", "https://platform.openai.com/api-keys", "sk-…", "gpt-realtime-2.1", "gpt-transcribe",
+ "gpt-4o-mini-tts", 600, listOf("marin", "cedar", "alloy", "ash", "ballad", "coral", "echo", "sage", "shimmer", "verse")),
+ Gemini("gemini", "Gemini", "https://aistudio.google.com/apikey", "AIza…", "gemini-3.8-live", "gemini-3.8-flash",
+ "gemini-3.8-flash-tts", 150, listOf("Kore", "Puck", "Charon", "Fenrir", "Aoede", "Leda", "Orus", "Zephyr"));
+
+ val defaults: VoiceDefaults get() = VoiceDefaults(model, transcribeModel, speechModel)
+
+ companion object { fun of(id: String?): VoiceProvider? = entries.firstOrNull { it.id == id } }
+}
+
+/** The provider's newest model for each voice mode, as the computer last found them. */
+@Serializable data class VoiceDefaults(val realtime: String, val transcribe: String, val speech: String)
+
+@Serializable data class VoiceStatus(val providers: List<ProviderStatus> = emptyList()) {
+ /** `provider` stays a string so a provider added by a newer computer doesn't break the list. */
+ @Serializable data class ProviderStatus(val provider: String, val configured: Boolean = false, val defaults: VoiceDefaults? = null)
+ fun configured(provider: VoiceProvider): Boolean = providers.any { it.provider == provider.id && it.configured }
+}
+
+/** The models the computer's key can use, per voice mode. */
+@Serializable data class VoiceModels(val models: List<String> = emptyList(), val transcribe: List<String> = emptyList(),
+ val speech: List<String> = emptyList(), val defaults: VoiceDefaults? = null)
+
+/**
+ * The call settings kept on this phone (iOS `VoiceSettings`): the engine, each provider's mode,
+ * models and voice, and this month's minutes. A default model isn't stored, so a newer one reaches the user.
+ */
+class VoiceChoices(private val read: (String) -> String?, private val write: (String, String?) -> Unit) {
+ enum class Model(val prefix: String) { Realtime("callModel"), Transcribe("callSpeechToText"), Speech("callSpeechModel") }
+
+ var provider: VoiceProvider?
+ get() = VoiceProvider.of(read("callEngine"))
+ set(value) = write("callEngine", value?.id)
+
+ fun defaults(provider: VoiceProvider): VoiceDefaults = read("callDefaults.${provider.id}")
+ ?.let { runCatching { WireJson.decodeFromString(VoiceDefaults.serializer(), it) }.getOrNull() } ?: provider.defaults
+ fun remember(defaults: VoiceDefaults?, provider: VoiceProvider) {
+ if (defaults != null) write("callDefaults.${provider.id}", WireJson.encodeToString(VoiceDefaults.serializer(), defaults))
+ }
+ fun remember(status: VoiceStatus) { for (entry in status.providers) VoiceProvider.of(entry.provider)?.let { remember(entry.defaults, it) } }
+
+ /** Speech: speech to text, then the reply read aloud. Otherwise a realtime conversation. */
+ fun speechMode(provider: VoiceProvider): Boolean = read("callMode.${provider.id}") == "speech"
+ fun speechMode(provider: VoiceProvider, on: Boolean) = write("callMode.${provider.id}", if (on) "speech" else "realtime")
+
+ fun model(provider: VoiceProvider, kind: Model): String = read("${kind.prefix}.${provider.id}") ?: fallback(provider, kind)
+ fun choose(provider: VoiceProvider, kind: Model, model: String) =
+ write("${kind.prefix}.${provider.id}", model.takeUnless { it.isEmpty() || it == fallback(provider, kind) })
+ private fun fallback(provider: VoiceProvider, kind: Model) = defaults(provider).let {
+ when (kind) { Model.Realtime -> it.realtime; Model.Transcribe -> it.transcribe; Model.Speech -> it.speech }
+ }
+
+ fun voice(provider: VoiceProvider): String = read("callVoice.${provider.id}")?.takeIf { it in provider.voices } ?: provider.voices[0]
+ fun voice(provider: VoiceProvider, voice: String) = write("callVoice.${provider.id}", voice)
+
+ /** The provider's model list as last fetched with the user's key. */
+ fun cachedModels(provider: VoiceProvider): VoiceModels? = read("callModels.${provider.id}")
+ ?.let { runCatching { WireJson.decodeFromString(VoiceModels.serializer(), it) }.getOrNull() }
+ fun cache(provider: VoiceProvider, models: VoiceModels) = write("callModels.${provider.id}", WireJson.encodeToString(VoiceModels.serializer(), models))
+
+ /** Cloud voice used this month, counted on this device: an estimate, not a bill. */
+ fun addUsage(provider: VoiceProvider, seconds: Long, month: YearMonth = YearMonth.now()) {
+ if (seconds <= 0) return
+ write(usageKey(provider, month), (usedSeconds(provider, month) + seconds).toString())
+ }
+ fun minutes(provider: VoiceProvider, month: YearMonth = YearMonth.now()): Long = (usedSeconds(provider, month) + 59) / 60
+ private fun usedSeconds(provider: VoiceProvider, month: YearMonth) = read(usageKey(provider, month))?.toLongOrNull() ?: 0
+ private fun usageKey(provider: VoiceProvider, month: YearMonth) = "callUsage.${provider.id}.$month"
+}
+
+/** Microphone audio: 16 kHz mono PCM16, little-endian. */
+object Pcm {
+ const val RATE = 16_000
+
+ /** Loudness, 0…1: -50 dB (a quiet room) … 0 dB. */
+ fun level(samples: ShortArray, count: Int): Float {
+ if (count <= 0) return 0f
+ var power = 0.0
+ for (i in 0 until count) { val sample = samples[i] / 32_768.0; power += sample * sample }
+ val db = 10 * log10(maxOf(power / count, 1e-10))
+ return ((db + 50) / 50).toFloat().coerceIn(0f, 1f)
+ }
+
+ fun bytes(samples: ShortArray, count: Int): ByteArray = ByteBuffer.allocate(count * 2).order(ByteOrder.LITTLE_ENDIAN)
+ .apply { asShortBuffer().put(samples, 0, count) }.array()
+
+ /** PCM wrapped in a WAV header. */
+ fun wav(pcm: ByteArray, rate: Int = RATE): ByteArray = ByteBuffer.allocate(44 + pcm.size).order(ByteOrder.LITTLE_ENDIAN).apply {
+ put("RIFF".toByteArray()); putInt(36 + pcm.size); put("WAVEfmt ".toByteArray()); putInt(16)
+ putShort(1); putShort(1); putInt(rate); putInt(rate * 2); putShort(2); putShort(16)
+ put("data".toByteArray()); putInt(pcm.size); put(pcm)
+ }.array()
+}
+
+/** Sentences grouped into clips of at most `limit` characters (longer sentences are cut). */
+fun speechClips(text: String, limit: Int): List<String> {
+ val out = mutableListOf<String>(); val current = StringBuilder(); val sentence = StringBuilder()
+ fun flush() { val clip = current.trim(); if (clip.isNotEmpty()) out += clip.toString(); current.clear() }
+ fun add(piece: String) {
+ if (current.length + piece.length > limit) flush()
+ var rest = piece
+ while (rest.length > limit) { val cut = SpokenText.chunks(rest, limit).first(); out += cut; rest = rest.substring(cut.length) }
+ current.append(rest)
+ }
+ for (character in text) {
+ sentence.append(character)
+ if (character in ".!?。!?\n") { add(sentence.toString()); sentence.clear() }
+ }
+ add(sentence.toString()); flush()
+ return out
+}
+
+/**
+ * Cuts the microphone into utterances at a pause, for speech to text: the PCM since speech began
+ * (plus a little before), sent once it has gone quiet for `pauseMillis` or reached the size one
+ * channel message holds. Under 0.4 s of voice is a cough or noise (and speech-to-text models
+ * invent words for noise), so it is dropped.
+ */
+class UtteranceDetector {
+ private var preroll = ByteArray(0)
+ private val utterance = ByteArrayOutputStream()
+ private var speaking = false
+ private var quietSince: Long? = null
+ /** How much of the utterance was actually loud, in bytes of PCM. */
+ private var voiced = 0
+
+ /** Takes one frame; returns the finished utterance as WAV, or null. */
+ fun feed(pcm: ByteArray, level: Float, now: Long, pauseMillis: Long): ByteArray? {
+ if (level > THRESHOLD) { speaking = true; quietSince = null; voiced += pcm.size }
+ else if (speaking && quietSince == null) quietSince = now
+ if (!speaking) {
+ preroll = (preroll + pcm).let { if (it.size > PREROLL) it.copyOfRange(it.size - PREROLL, it.size) else it }
+ return null
+ }
+ utterance.write(pcm)
+ val paused = quietSince?.let { now - it >= pauseMillis } ?: false
+ if (!paused && utterance.size() < MAX_BYTES) return null
+ val wav = Pcm.wav(preroll + utterance.toByteArray())
+ val enough = voiced >= MIN_VOICED
+ reset()
+ return wav.takeIf { enough }
+ }
+
+ fun reset() { preroll = ByteArray(0); utterance.reset(); speaking = false; quietSince = null; voiced = 0 }
+
+ companion object {
+ const val THRESHOLD = .3f
+ private const val PREROLL = Pcm.RATE * 2 * 3 / 10
+ /** An utterance has to fit one channel message once base64-encoded. */
+ const val MAX_BYTES = Pcm.RATE * 2 * 20
+ private const val MIN_VOICED = Pcm.RATE * 2 * 4 / 10
+ }
+}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenTest.kt
new file mode 100644
index 00000000..788dfc2c
--- /dev/null
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenTest.kt
@@ -0,0 +1,146 @@
+package com.codync.android.core
+
+import kotlinx.serialization.json.*
+import org.junit.Assert.*
+import org.junit.Test
+
+class ScreenTest {
+ private val display = ScreenDisplay(7, "Studio Display", 1600.0, 1000.0, main = true)
+
+ @Test fun `screen state decodes what older and newer hosts send and explains setup`() {
+ val state = ScreenState.decode(WireJson.parseToJsonElement("""{"enabled":true,"connected":true,"platform":"macos","capture":true,
+ "displays":[{"id":1,"name":"Side","width":800,"height":600,"main":false},{"id":7,"name":"Studio Display","width":1600,"height":1000,"main":true}],
+ "agentBot":"b1","future":1}""").jsonObject)
+ assertTrue(state.available)
+ assertFalse(state.controlReady)
+ assertEquals(7L, state.mainDisplay?.id)
+ assertEquals("b1", state.agentBot)
+ assertTrue(state.accessGuidance.startsWith("Allow computer control for Codync Screen."))
+ assertEquals(ScreenState(), ScreenState.decode(buildJsonObject {}))
+ assertTrue(ScreenState(enabled = true, connected = true, platform = "linux", capture = true).accessGuidance.startsWith("Choose a screen"))
+ assertTrue(ScreenState(platform = "windows", computerUse = true).controlReady)
+ assertTrue(ScreenState().accessGuidance.startsWith("Remote screen is off."))
+ }
+
+ @Test fun `replies and requests match the data channel protocol`() {
+ assertEquals(ScreenReply.Clipboard("hi"), ScreenReply.decode("""{"type":"clipboard","text":"hi"}"""))
+ assertEquals(ScreenReply.View(ScreenRect(1.0, 2.0, 3.0, 4.0)), ScreenReply.decode("""{"type":"view","crop":[1,2,3,4]}"""))
+ assertEquals(ScreenReply.View(null), ScreenReply.decode("""{"type":"view"}"""))
+ assertEquals(ScreenReply.Error("nope"), ScreenReply.decode("""{"type":"error","message":"nope"}"""))
+ assertNull(ScreenReply.decode("not json"))
+ assertNull(ScreenReply.decode("""{"type":"later"}"""))
+ assertEquals("""{"type":"view","long":2400.0,"short":1080.0,"display":7,"crop":[0.0,0.0,800.0,500.0]}""",
+ ScreenRequests.view(7, ScreenRect(0.0, 0.0, 800.0, 500.0), ScreenSize(1080.0, 2400.0)).toString())
+ assertEquals("""{"type":"view","long":10.0,"short":5.0}""", ScreenRequests.view(null, null, ScreenSize(10.0, 5.0)).toString())
+ assertEquals("""{"type":"view","display":2}""", ScreenRequests.switchDisplay(2).toString())
+ }
+
+ @Test fun `hardware H264 High leads and renewal leaves five minutes`() {
+ val codecs = listOf("VP8" to null, "H264" to "42e01f", "AV1" to null, "H264" to "640c1f")
+ assertEquals(listOf("H264" to "640c1f", "H264" to "42e01f", "VP8" to null, "AV1" to null),
+ preferredScreenCodecs(codecs, { it.first }, { it.second }))
+ val connection = ScreenConnection("s", emptyList(), expiresAt = 3_600_000)
+ assertEquals(3_300_000, connection.renewIn(0))
+ assertEquals(1_000, connection.renewIn(3_500_000))
+ }
+
+ @Test fun `viewport zooms around the fingers, stays on the display and streams the visible part`() {
+ var v = ScreenViewport(ScreenSize(1600.0, 1000.0), ScreenSize(400.0, 800.0))
+ assertEquals(0.25, v.scale, 1e-9)
+ assertNull(v.crop)
+ assertEquals(ScreenRect(0.0, 275.0, 400.0, 250.0), v.frame(v.displayBounds))
+ val anchor = ScreenPoint(100.0, 400.0)
+ val before = v.toDisplay(anchor)
+ v = v.zoomed(4.0, anchor)
+ assertEquals(before.x, v.toDisplay(anchor).x, 1e-9)
+ assertEquals(ScreenRect(300.0, 100.0, 400.0, 800.0), v.crop)
+ // Pans stop at the display's edge.
+ assertEquals(ScreenRect(1200.0, 100.0, 400.0, 800.0), v.panned(ScreenPoint(-10_000.0, 0.0)).crop)
+ assertEquals(16.0, v.maxZoom, 1e-9)
+ assertEquals(16.0, v.zoomed(100.0, anchor).zoom, 1e-9)
+ assertNull(v.reset().crop)
+ // A pointer near the edge pulls the view along.
+ assertTrue(v.following(ScreenPoint(1599.0, 500.0)).center.x > v.center.x)
+ assertEquals(v, v.following(v.center))
+ }
+
+ @Test fun `trackpad moves the pointer faster with the finger and taps click there`() {
+ val sent = mutableListOf<JsonObject>()
+ val control = ScreenControl { sent += it }
+ control.setViewSize(ScreenSize(400.0, 250.0))
+ control.setDisplay(display)
+ control.onePan(ScreenPoint(10.0, 0.0), speed = 0.0)
+ assertEquals(840.0, sent.last().double("x"), 1e-9)
+ control.onePan(ScreenPoint(10.0, 0.0), speed = 1800.0)
+ assertEquals(920.0, sent.last().double("x"), 1e-9)
+ control.tap(ScreenPoint(0.0, 0.0), 1_000)
+ control.tap(ScreenPoint(0.0, 0.0), 1_200)
+ control.tap(ScreenPoint(0.0, 0.0), 1_400)
+ control.tap(ScreenPoint(0.0, 0.0), 1_600)
+ assertEquals(listOf(1, 2, 3, 3), sent.takeLast(4).map { it.getValue("count").jsonPrimitive.int })
+ assertEquals(920.0, sent.last().double("x"), 1e-9)
+ control.tap(ScreenPoint(0.0, 0.0), 3_000)
+ assertEquals(1, sent.last().getValue("count").jsonPrimitive.int)
+ }
+
+ @Test fun `direct taps land under the finger, two fingers scroll naturally or pan while pinching`() {
+ val sent = mutableListOf<JsonObject>()
+ val control = ScreenControl { sent += it }
+ control.mode = TouchMode.Direct
+ control.setViewSize(ScreenSize(400.0, 250.0))
+ control.setDisplay(display)
+ control.tap(ScreenPoint(100.0, 50.0), 0)
+ assertEquals(400.0 to 200.0, sent.last().double("x") to sent.last().double("y"))
+ control.twoFingerTap(ScreenPoint(400.0, 250.0))
+ assertEquals("right", sent.last().text("button"))
+ assertEquals(1599.0, sent.last().double("x"), 1e-9)
+ control.twoPan(ScreenPoint(0.0, 10.0), ScreenPoint(0.0, 0.0))
+ assertEquals("scroll", sent.last().text("type"))
+ assertEquals(-15.0, sent.last().double("dy"), 1e-9)
+ val count = sent.size
+ control.pinchBegan()
+ control.pinch(2.0, ScreenPoint(200.0, 125.0))
+ control.twoPan(ScreenPoint(10.0, 0.0), ScreenPoint(200.0, 125.0))
+ control.pinchEnded()
+ assertEquals("pinching only moves the view", count, sent.size)
+ assertTrue(control.zoomed)
+ }
+
+ @Test fun `a drag holds the button until the finger lifts, even after handing control back`() {
+ val sent = mutableListOf<JsonObject>()
+ val control = ScreenControl { sent += it }
+ control.setViewSize(ScreenSize(400.0, 250.0))
+ control.setDisplay(display)
+ assertTrue(control.pressBegan(ScreenPoint(10.0, 10.0)))
+ control.pressMoved(ScreenPoint(20.0, 10.0))
+ control.interactive = false
+ control.pressEnded()
+ assertEquals(listOf("down", "move", "up"), sent.map { it.text("type") })
+ assertFalse(control.pressBegan(ScreenPoint(0.0, 0.0)))
+ control.tap(ScreenPoint(0.0, 0.0), 0)
+ control.typed("x")
+ assertEquals(3, sent.size)
+ }
+
+ @Test fun `armed modifiers apply once, to the next click or key`() {
+ val sent = mutableListOf<JsonObject>()
+ var consumed = 0
+ val control = ScreenControl { sent += it }
+ control.onArmedConsumed = { consumed++ }
+ control.armed = listOf("cmd")
+ control.typed("C")
+ assertEquals("""{"type":"key","key":"c","modifiers":["cmd"]}""", sent.last().toString())
+ control.typed("hi")
+ assertEquals("""{"type":"text","text":"hi"}""", sent.last().toString())
+ control.armed = listOf("shift")
+ control.pressKey("tab", listOf("shift", "ctrl"))
+ assertEquals(listOf("shift", "ctrl"), sent.last().getValue("modifiers").jsonArray.map { it.jsonPrimitive.content })
+ control.armed = listOf("option")
+ control.tap(ScreenPoint(0.0, 0.0), 0)
+ assertEquals(listOf("option"), sent.last().getValue("modifiers").jsonArray.map { it.jsonPrimitive.content })
+ assertEquals(3, consumed)
+ assertTrue(control.armed.isEmpty())
+ }
+
+ private fun JsonObject.double(key: String) = getValue(key).jsonPrimitive.double
+}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt
new file mode 100644
index 00000000..f7957d9f
--- /dev/null
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceEnginesTest.kt
@@ -0,0 +1,273 @@
+package com.codync.android.core
+
+import java.io.IOException
+import java.time.YearMonth
+import java.util.Base64
+import java.util.concurrent.Executors
+import java.util.concurrent.LinkedBlockingQueue
+import java.util.concurrent.TimeUnit
+import kotlinx.coroutines.*
+import kotlinx.coroutines.flow.first
+import kotlinx.coroutines.test.*
+import kotlinx.serialization.json.*
+import okhttp3.OkHttpClient
+import okhttp3.Response
+import okhttp3.WebSocket
+import okhttp3.WebSocketListener
+import okhttp3.mockwebserver.MockResponse
+import okhttp3.mockwebserver.MockWebServer
+import org.junit.Assert.*
+import org.junit.Test
+
+@OptIn(ExperimentalCoroutinesApi::class)
+class VoiceEnginesTest {
+ private class Prefs { val values = mutableMapOf<String, String>(); val choices = VoiceChoices({ values[it] }) { k, v -> if (v == null) values.remove(k) else values[k] = v } }
+
+ @Test fun choicesKeepOnlyWhatTheUserPickedAndFollowNewerDefaults() {
+ val prefs = Prefs(); val choices = prefs.choices
+ assertNull(choices.provider)
+ choices.provider = VoiceProvider.Gemini; assertEquals(VoiceProvider.Gemini, choices.provider)
+ choices.choose(VoiceProvider.Gemini, VoiceChoices.Model.Realtime, VoiceProvider.Gemini.model)
+ assertTrue(prefs.values.keys.none { it.startsWith("callModel.") })
+ choices.remember(VoiceStatus(listOf(VoiceStatus.ProviderStatus("gemini", true, VoiceDefaults("live-9", "flash-9", "tts-9")),
+ VoiceStatus.ProviderStatus("future", true, VoiceDefaults("x", "y", "z")))))
+ assertEquals("live-9", choices.model(VoiceProvider.Gemini, VoiceChoices.Model.Realtime))
+ assertEquals("tts-9", choices.model(VoiceProvider.Gemini, VoiceChoices.Model.Speech))
+ choices.choose(VoiceProvider.Gemini, VoiceChoices.Model.Transcribe, "flash-8")
+ assertEquals("flash-8", choices.model(VoiceProvider.Gemini, VoiceChoices.Model.Transcribe))
+ prefs.values["callVoice.openai"] = "retired"; assertEquals("marin", choices.voice(VoiceProvider.OpenAI))
+ prefs.values["callDefaults.openai"] = "{broken"; assertEquals(VoiceProvider.OpenAI.defaults, choices.defaults(VoiceProvider.OpenAI))
+ assertFalse(choices.speechMode(VoiceProvider.OpenAI)); choices.speechMode(VoiceProvider.OpenAI, true); assertTrue(choices.speechMode(VoiceProvider.OpenAI))
+ val october = YearMonth.of(2026, 10)
+ choices.addUsage(VoiceProvider.OpenAI, 61, october); choices.addUsage(VoiceProvider.OpenAI, -5, october)
+ assertEquals(2, choices.minutes(VoiceProvider.OpenAI, october)); assertEquals(0, choices.minutes(VoiceProvider.OpenAI, october.plusMonths(1)))
+ assertEquals("61", prefs.values["callUsage.openai.2026-10"])
+ }
+
+ @Test fun utterancesStartAtSpeechEndAtThePauseAndDropCoughs() {
+ val detector = UtteranceDetector(); val frame = ByteArray(3_200)
+ assertNull(detector.feed(frame, 0f, 0, 1_000))
+ assertNull(detector.feed(frame, .8f, 100, 1_000))
+ assertNull(detector.feed(frame, 0f, 200, 1_000)) // A cough: one loud frame, then quiet.
+ assertNull(detector.feed(frame, 0f, 1_300, 1_000))
+ for (t in 0 until 6) assertNull(detector.feed(frame, .8f, 2_000L + t * 100, 1_000))
+ assertNull(detector.feed(frame, 0f, 2_700, 1_000))
+ val wav = requireNotNull(detector.feed(frame, 0f, 3_700, 1_000))
+ assertEquals("RIFF", String(wav, 0, 4)); assertEquals("WAVE", String(wav, 8, 4))
+ assertEquals(44 + 8 * 3_200, wav.size) // The quiet frame before speech is kept as preroll.
+ assertNull(detector.feed(frame, 0f, 5_000, 1_000))
+ val long = ByteArray(UtteranceDetector.MAX_BYTES)
+ assertNotNull(detector.feed(long, .9f, 6_000, 1_000)) // A monologue is cut at one message's size.
+ }
+
+ @Test fun clipsGroupSentencesAndCutLongOnes() {
+ assertEquals(listOf("One. Two.", "Three!"), speechClips("One. Two. Three!", 10))
+ assertEquals(listOf("abcde", "fghij", "k."), speechClips("abcdefghijk.", 5))
+ assertTrue(speechClips(" ", 5).isEmpty())
+ assertEquals(listOf(1, 0, 1, 0, 128, 62, 0, 0, 0, 125, 0, 0), Pcm.wav(ByteArray(0)).copyOfRange(20, 32).map { it.toInt() and 0xFF })
+ assertEquals(1f, Pcm.level(ShortArray(4) { Short.MAX_VALUE }, 4), .01f); assertEquals(0f, Pcm.level(ShortArray(4), 4))
+ }
+
+ private class Desk(override val bots: List<Bot>, override var entries: List<Entry>) : CallTarget {
+ val sent = mutableListOf<String>(); val answered = mutableListOf<Pair<String, String>>()
+ override fun send(text: String) { sent += text }
+ override fun respond(entry: Entry, optionId: String) { answered += entry.id to optionId }
+ }
+ private fun entry(id: String, seq: Long, kind: String, data: JsonObjectBuilder.() -> Unit, bot: String = "ada", thread: String? = null) =
+ Entry(id, seq, bot, thread, seq, kind, data = buildJsonObject(data))
+ private val card = entry("card", 4, "permission", {
+ put("status", "pending"); put("title", "Run npm test")
+ putJsonArray("options") {
+ addJsonObject { put("optionId", "allow_once"); put("name", "Allow once"); put("kind", "allow_once") }
+ addJsonObject { put("optionId", "reject"); put("name", "Deny"); put("kind", "reject_once") }
+ }
+ })
+
+ @Test fun theOperatorRelaysToTheBotAndAnswersApprovalsByName() {
+ val desk = Desk(listOf(Bot("ada", "Ada", description = "web app", status = "working", activity = "Running tests")), listOf(
+ entry("u", 1, "user", { put("text", "Run the tests") }),
+ entry("a", 2, "agent", { put("text", "**Started** `npm test`"); put("final", true) }),
+ entry("t", 3, "agent", { put("text", "In a thread"); put("final", true) }, thread = "root"), card))
+ val operator = CallOperator("ada", desk)
+ assertTrue(operator.instructions.startsWith("You are the voice on a phone call between the user and Ada (web app), a coding agent"))
+ assertTrue("[Ada replied]" in operator.instructions)
+ assertEquals("Nothing to send.", operator.call("send_to_bot", buildJsonObject { put("text", " ") })["error"]?.jsonPrimitive?.content)
+ operator.call("send_to_bot", buildJsonObject { put("text", " Fix the build ") }); assertEquals(listOf("Fix the build"), desk.sent)
+ val status = operator.call("bot_status", JsonObject(emptyMap()))
+ assertEquals("Run npm test", status["approval"]!!.jsonObject["request"]!!.jsonPrimitive.content)
+ val messages = operator.call("recent_messages", buildJsonObject { put("count", 2.0) })["messages"]!!.jsonArray
+ assertEquals(listOf("ada" to "Started npm test", "approval request" to "Run npm test"),
+ messages.map { it.jsonObject["from"]!!.jsonPrimitive.content.lowercase() to it.jsonObject["text"]!!.jsonPrimitive.content })
+ assertEquals("Allow once", operator.call("answer_approval", buildJsonObject { put("option", "allow") })["answered"]?.jsonPrimitive?.content)
+ assertEquals(listOf("card" to "allow_once"), desk.answered)
+ assertEquals(2, operator.call("answer_approval", buildJsonObject { put("option", "maybe") })["options"]!!.jsonArray.size)
+ assertEquals("Unknown tool shell.", operator.call("shell", JsonObject(emptyMap()))["error"]?.jsonPrimitive?.content)
+ assertEquals("[Ada replied] Done", operator.reply("**Done**")); assertEquals("[Notice] Hi", operator.notice("Hi"))
+ val group = CallOperator("team", Desk(listOf(Bot("team", "Team", kind = "group", members = listOf("ada", "bo")), Bot("ada", "Ada"), Bot("bo", "Bo")), emptyList()))
+ assertTrue("the group chat Team on the user's computer, where the bots Ada, Bo answer in turn" in group.instructions)
+ val declarations = CallOperator.tools.map(CallTool::gemini)
+ assertNull(declarations.first { it["name"]!!.jsonPrimitive.content == "bot_status" }["parameters"])
+ assertEquals("OBJECT", declarations[0]["parameters"]!!.jsonObject["type"]!!.jsonPrimitive.content)
+ assertEquals("STRING", declarations[0]["parameters"]!!.jsonObject["properties"]!!.jsonObject["text"]!!.jsonObject["type"]!!.jsonPrimitive.content)
+ assertEquals("function", CallOperator.tools[0].openAI()["type"]!!.jsonPrimitive.content)
+ }
+
+ private class Mic : MicSource {
+ var frames: ((ByteArray, Float) -> Unit)? = null; var stops = 0; var refuse = false
+ override suspend fun start(frames: (ByteArray, Float) -> Unit, failed: () -> Unit) { if (refuse) throw IOException("No microphone"); this.frames = frames }
+ override suspend fun stop() { stops++ }
+ }
+
+ @Test fun cloudSpeechSendsEachUtteranceAndReadsRepliesClipByClip() = runTest {
+ val mic = Mic(); val sent = mutableListOf<String>(); val heard = mutableListOf<Int>(); val made = mutableListOf<String>()
+ val playing = mutableListOf<CompletableDeferred<Unit>>()
+ val speech = CloudSpeechEngine.Speech({ heard += it.size; " hello there " }, { made += it; it.toByteArray() to "aac" }, sent::add)
+ val engine = CloudSpeechEngine(backgroundScope, VoiceProvider.Gemini, VoiceSettings(pauseMillis = 1_000), mic,
+ { _, _ -> val clip = CompletableDeferred<Unit>().also { playing += it }
+ try { clip.await() } catch (cancelled: CancellationException) { clip.cancel(); throw cancelled } }, speech) { testScheduler.currentTime }
+ runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ val frame = ByteArray(3_200)
+ repeat(6) { mic.frames!!(frame, .8f); advanceTimeBy(100); runCurrent() }
+ mic.frames!!(frame, 0f); advanceTimeBy(1_100); runCurrent(); mic.frames!!(frame, 0f); runCurrent()
+ assertEquals(listOf("hello there"), sent)
+ val reply = "First sentence here. " + "x".repeat(200) + "."
+ engine.speak("**$reply**"); runCurrent()
+ assertEquals(VoicePhase.Speaking, engine.state.value.phase); assertEquals(2, made.size) // The next clip is fetched while one plays.
+ mic.frames!!(frame, .9f); runCurrent(); assertEquals(0f, engine.state.value.level) // The mic doesn't count while it talks.
+ playing[0].complete(Unit); runCurrent(); assertEquals(VoicePhase.Speaking, engine.state.value.phase)
+ playing[1].complete(Unit); runCurrent(); assertEquals(VoicePhase.Speaking, engine.state.value.phase)
+ playing[2].complete(Unit); runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ engine.speak("Long reply."); runCurrent(); engine.interrupt(); runCurrent()
+ assertEquals(VoicePhase.Listening, engine.state.value.phase); assertTrue(playing.last().isCancelled)
+ engine.end(); runCurrent(); engine.awaitEnd()
+ assertEquals(VoicePhase.Ended, engine.state.value.phase); assertEquals(1, mic.stops); assertFalse(engine.speak("Late"))
+ }
+
+ @Test fun aRejectedTranscriptionFailsTheCallWithTheProvidersReason() = runTest {
+ val mic = Mic()
+ val engine = CloudSpeechEngine(backgroundScope, VoiceProvider.OpenAI, VoiceSettings(pauseMillis = 1_000), mic, { _, _ -> },
+ CloudSpeechEngine.Speech({ throw IOException("Incorrect API key") }, { error("unused") }) { fail("Nothing to send") }) { testScheduler.currentTime }
+ runCurrent()
+ repeat(6) { mic.frames!!(ByteArray(3_200), .8f); runCurrent() }
+ mic.frames!!(ByteArray(3_200), 0f); advanceTimeBy(1_100); runCurrent(); mic.frames!!(ByteArray(3_200), 0f); runCurrent()
+ engine.awaitEnd()
+ assertEquals(VoicePhase.Failed, engine.state.value.phase); assertEquals("Incorrect API key", engine.state.value.error); assertEquals(1, mic.stops)
+ val refused = CloudSpeechEngine(backgroundScope, VoiceProvider.OpenAI, VoiceSettings(), Mic().apply { refuse = true }, { _, _ -> },
+ CloudSpeechEngine.Speech({ "" }, { error("unused") }) { })
+ runCurrent(); assertEquals("No microphone", refused.state.value.error)
+ }
+
+ private class Peer : RealtimePeer {
+ var events: RealtimePeer.Events? = null; var answer: String? = null; var micOn: Boolean? = null; var closed = 0
+ override var isOpen = false
+ val sent = mutableListOf<JsonObject>()
+ override suspend fun offer(events: RealtimePeer.Events): String { this.events = events; return "offer-sdp" }
+ override suspend fun answer(sdp: String) { answer = sdp }
+ override fun send(text: String) { sent += Json.parseToJsonElement(text).jsonObject }
+ override fun mic(enabled: Boolean) { micOn = enabled }
+ override suspend fun close() { closed++ }
+ fun types() = sent.map { it["type"]!!.jsonPrimitive.content }
+ }
+
+ @Test fun openAIRealtimeRelaysRepliesOneResponseAtATimeAndRunsTools() = runTest {
+ val peer = Peer(); val desk = Desk(listOf(Bot("ada", "Ada")), emptyList()); var exchanged: Pair<String, String>? = null
+ val engine = OpenAIRealtimeEngine(backgroundScope, VoiceSettings(), { "ek_secret" }, CallOperator("ada", desk), peer) { secret, offer ->
+ exchanged = secret to offer; "answer-sdp" }
+ runCurrent()
+ assertEquals("ek_secret" to "offer-sdp", exchanged); assertEquals("answer-sdp", peer.answer); assertEquals(true, peer.micOn)
+ engine.speak("Early reply"); runCurrent(); assertTrue(peer.sent.isEmpty()) // Waits for the channel.
+ peer.isOpen = true; peer.events!!.opened(); runCurrent()
+ assertEquals(listOf("session.update", "conversation.item.create", "response.create"), peer.types())
+ assertEquals("[Ada replied] Early reply", peer.sent[1]["item"]!!.jsonObject["content"]!!.jsonArray[0].jsonObject["text"]!!.jsonPrimitive.content)
+ assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ peer.events!!.received("""{"type":"response.created"}"""); engine.announce("Approval waiting"); runCurrent()
+ assertEquals(3, peer.sent.size)
+ peer.events!!.received("""{"type":"response.function_call_arguments.done","call_id":"c1","name":"send_to_bot","arguments":"{\"text\":\"Ship it\"}"}""")
+ peer.events!!.received("""{"type":"response.done"}"""); runCurrent()
+ assertEquals(listOf("Ship it"), desk.sent)
+ assertEquals(listOf("function_call_output", "message"), peer.sent.drop(3).dropLast(1).map { it["item"]!!.jsonObject["type"]!!.jsonPrimitive.content })
+ assertEquals("response.create", peer.types().last())
+ peer.events!!.received("""{"type":"output_audio_buffer.started"}"""); runCurrent()
+ assertEquals(VoicePhase.Speaking, engine.state.value.phase)
+ engine.interrupt(); runCurrent()
+ assertEquals(listOf("response.cancel", "output_audio_buffer.clear"), peer.types().takeLast(2))
+ engine.mute(true); runCurrent(); assertEquals(false, peer.micOn)
+ peer.events!!.ice(RealtimePeer.Ice.Disconnected); advanceTimeBy(4_000); runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ peer.events!!.ice(RealtimePeer.Ice.Connected); advanceTimeBy(2_000); runCurrent(); assertEquals(VoicePhase.Listening, engine.state.value.phase)
+ peer.events!!.ice(RealtimePeer.Ice.Failed); runCurrent(); engine.awaitEnd()
+ assertEquals("Lost the connection to OpenAI.", engine.state.value.error); assertEquals(1, peer.closed)
+ }
+
+ @Test fun openAIRealtimeGivesUpWhenTheChannelNeverOpens() = runTest {
+ val peer = Peer()
+ val engine = OpenAIRealtimeEngine(backgroundScope, VoiceSettings(), { "ek" }, CallOperator("ada", Desk(emptyList(), emptyList())), peer) { _, _ -> "sdp" }
+ advanceTimeBy(15_001); runCurrent(); engine.awaitEnd()
+ assertEquals(VoicePhase.Failed, engine.state.value.phase); assertEquals("Couldn't reach OpenAI.", engine.state.value.error)
+ }
+
+ @Test fun openAICallsEndpointPostsTheOfferAndExplainsRefusals() = runBlocking {
+ MockWebServer().use { server ->
+ server.enqueue(MockResponse().setBody("v=0 answer")); server.enqueue(MockResponse().setResponseCode(401).setBody("""{"error":{"message":"Invalid client secret"}}"""))
+ val url = server.url("/v1/realtime/calls").toString()
+ assertEquals("v=0 answer", OpenAIRealtimeEngine.answer(OkHttpClient(), "ek_1", "v=0 offer", url))
+ val request = server.takeRequest()
+ assertEquals("Bearer ek_1", request.getHeader("Authorization")); assertTrue(request.getHeader("Content-Type")!!.startsWith("application/sdp"))
+ assertEquals("v=0 offer", request.body.readUtf8())
+ assertEquals("Invalid client secret", runCatching { OpenAIRealtimeEngine.answer(OkHttpClient(), "ek_2", "offer", url) }.exceptionOrNull()?.message)
+ }
+ }
+
+ private class Player : PcmPlayer {
+ val chunks = LinkedBlockingQueue<() -> Unit>(); var stops = 0; var closed = 0
+ override fun play(pcm: ByteArray, played: () -> Unit) { chunks += played }
+ override fun stop() { stops++ }
+ override suspend fun close() { closed++ }
+ }
+
+ @Test fun geminiLiveSetsUpTalksAndRunsToolsOverTheSocket() = runBlocking {
+ val inbound = LinkedBlockingQueue<JsonObject>(); var server: WebSocket? = null
+ val opened = CompletableDeferred<Unit>()
+ MockWebServer().use { mock ->
+ mock.enqueue(MockResponse().withWebSocketUpgrade(object : WebSocketListener() {
+ override fun onOpen(webSocket: WebSocket, response: Response) { server = webSocket; opened.complete(Unit) }
+ override fun onMessage(webSocket: WebSocket, text: String) { inbound += Json.parseToJsonElement(text).jsonObject }
+ }))
+ val dispatcher = Executors.newSingleThreadExecutor().asCoroutineDispatcher()
+ val scope = CoroutineScope(SupervisorJob() + dispatcher)
+ val mic = Mic(); val player = Player(); val desk = Desk(listOf(Bot("ada", "Ada", status = "idle")), emptyList())
+ val endpoint = mock.url("/live").toString().replace("http://", "ws://")
+ val engine = GeminiLiveEngine(scope, VoiceSettings(), "gemini-live-test", "Puck", { "token/1" }, CallOperator("ada", desk), mic, player,
+ OkHttpClient(), endpoint)
+ fun next() = requireNotNull(inbound.poll(5, TimeUnit.SECONDS)) { "Nothing arrived" }
+ suspend fun until(check: (VoiceState) -> Boolean) = withTimeout(5_000) { engine.state.first(check) }
+ assertEquals("/live?access_token=token%2F1", mock.takeRequest().path)
+ withTimeout(5_000) { opened.await() }
+ val setup = next()["setup"]!!.jsonObject
+ assertEquals("models/gemini-live-test", setup["model"]!!.jsonPrimitive.content)
+ assertEquals("Puck", setup["generationConfig"]!!.jsonObject["speechConfig"]!!.jsonObject["voiceConfig"]!!.jsonObject["prebuiltVoiceConfig"]!!.jsonObject["voiceName"]!!.jsonPrimitive.content)
+ assertEquals(4, setup["tools"]!!.jsonArray[0].jsonObject["functionDeclarations"]!!.jsonArray.size)
+ engine.speak("Queued before setup")
+ server!!.send("""{"setupComplete":{}}""")
+ until { it.phase == VoicePhase.Listening }
+ assertEquals("[Ada replied] Queued before setup", next()["clientContent"]!!.jsonObject["turns"]!!.jsonArray[0].jsonObject["parts"]!!.jsonArray[0].jsonObject["text"]!!.jsonPrimitive.content)
+ mic.frames!!(byteArrayOf(1, 2), .5f)
+ assertEquals("audio/pcm;rate=16000", next()["realtimeInput"]!!.jsonObject["audio"]!!.jsonObject["mimeType"]!!.jsonPrimitive.content)
+ val audio = Base64.getEncoder().encodeToString(ByteArray(480))
+ server!!.send("""{"serverContent":{"modelTurn":{"parts":[{"inlineData":{"mimeType":"audio/pcm;rate=24000","data":"$audio"}}]}}}""")
+ until { it.phase == VoicePhase.Speaking }
+ server!!.send("""{"serverContent":{"turnComplete":true}}""")
+ requireNotNull(player.chunks.poll(5, TimeUnit.SECONDS))()
+ until { it.phase == VoicePhase.Listening }
+ server!!.send("""{"toolCall":{"functionCalls":[{"id":"f1","name":"bot_status","args":{}}]}}""")
+ val response = next()["toolResponse"]!!.jsonObject["functionResponses"]!!.jsonArray[0].jsonObject
+ assertEquals("f1", response["id"]!!.jsonPrimitive.content)
+ assertEquals("idle", response["response"]!!.jsonObject["result"]!!.jsonObject["status"]!!.jsonPrimitive.content)
+ server!!.close(1008, "API key not valid")
+ until { it.phase == VoicePhase.Failed }
+ assertEquals("API key not valid", engine.state.value.error)
+ withTimeout(5_000) { engine.awaitEnd() }
+ assertEquals(1, mic.stops); assertEquals(1, player.closed)
+ scope.cancel(); dispatcher.close()
+ }
+ }
+}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
index 193a88af..81f584fb 100644
--- a/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/VoiceTest.kt
@@ -104,10 +104,18 @@ class VoiceTest {
val gate = VoiceReplyGate("active", 10, listOf(old, streaming))
val values = listOf(entry("old", 20), entry("streaming", 21), entry("reply", 22),
entry("thread", 23, thread = "root"), entry("other", 24, bot = "other"), entry("history", 6))
- assertEquals(listOf("streaming", "reply"), gate.observe(values))
- assertTrue(gate.observe(values).isEmpty())
+ val bots = listOf(Bot("active", "Ada"))
+ assertEquals(listOf("streaming", "reply"), gate.observe(values, bots).map(VoiceCue::text))
+ assertTrue(gate.observe(values, bots).isEmpty())
val permission = Entry("approval", 25, "active", rev = 25, kind = "permission", data = buildJsonObject { put("status", "pending") })
- assertEquals(1, gate.observe(values + permission).size)
- assertTrue(gate.observe(values + permission).isEmpty())
+ assertEquals(listOf(VoiceCue("Ada needs your approval in the chat.", notice = true)), gate.observe(values + permission, bots))
+ assertTrue(gate.observe(values + permission, bots).isEmpty())
+ }
+ @Test fun groupRepliesAndApprovalsNameTheMemberWhoSentThem() {
+ val bots = listOf(Bot("team", "Team", kind = "group", members = listOf("ada")), Bot("ada", "Ada"))
+ val gate = VoiceReplyGate("team", 0, emptyList())
+ val reply = Entry("r", 1, "team", rev = 1, kind = "agent", data = buildJsonObject { put("text", "Done"); put("final", true); put("author", "ada") })
+ val card = Entry("p", 2, "team", rev = 2, kind = "permission", data = buildJsonObject { put("status", "pending"); put("author", "ghost") })
+ assertEquals(listOf(VoiceCue("Ada: Done"), VoiceCue("A bot needs your approval in the chat.", true)), gate.observe(listOf(reply, card), bots))
}
}
diff --git a/apps/android/design/src/main/kotlin/com/codync/android/design/Controls.kt b/apps/android/design/src/main/kotlin/com/codync/android/design/Controls.kt
index c0faaa56..74a93cdb 100644
--- a/apps/android/design/src/main/kotlin/com/codync/android/design/Controls.kt
+++ b/apps/android/design/src/main/kotlin/com/codync/android/design/Controls.kt
@@ -193,10 +193,10 @@ import com.kyant.shapes.RoundedRectangle
/** Shows the current choice in a pill with a chevron and opens the menu: `ChoicePicker`. */
@Composable fun <T> ChoicePicker(selection: T, options: List<Pair<T, String>>, select: (T) -> Unit, modifier: Modifier = Modifier,
- fill: Color = Color.Transparent, label: String? = null) {
+ fill: Color = Color.Transparent, label: String? = null, enabled: Boolean = true) {
val current = options.firstOrNull { it.first == selection }?.second ?: "Choose"
DropdownMenu({ options.map { (id, text) -> MenuItem(text, selected = id == selection) { select(id) } } }, modifier,
- label = label?.let { "$it: $current" }) {
+ enabled = enabled, label = label?.let { "$it: $current" }) {
Row(Modifier.then(if (fill == Color.Transparent) Modifier.outline() else Modifier).pill(fill),
verticalAlignment = Alignment.CenterVertically, horizontalArrangement = Arrangement.spacedBy(6.dp)) {
Text(current, style = Type.body, color = Palette.text, maxLines = 1, overflow = TextOverflow.Ellipsis)
diff --git a/apps/android/design/src/main/kotlin/com/codync/android/design/Symbols.kt b/apps/android/design/src/main/kotlin/com/codync/android/design/Symbols.kt
index 174d1151..ce11fa8b 100644
--- a/apps/android/design/src/main/kotlin/com/codync/android/design/Symbols.kt
+++ b/apps/android/design/src/main/kotlin/com/codync/android/design/Symbols.kt
@@ -146,6 +146,7 @@ import androidx.annotation.DrawableRes
"return" -> R.drawable.sf_return
"rotate.right" -> R.drawable.sf_rotate_right
"safari" -> R.drawable.sf_safari
+ "shift" -> R.drawable.sf_shift
"shippingbox" -> R.drawable.sf_shippingbox
"square.and.arrow.down" -> R.drawable.sf_square_and_arrow_down
"square.and.arrow.up" -> R.drawable.sf_square_and_arrow_up
diff --git a/apps/android/design/src/main/res/drawable/sf_shift.xml b/apps/android/design/src/main/res/drawable/sf_shift.xml
new file mode 100644
index 00000000..7ffea76c
--- /dev/null
+++ b/apps/android/design/src/main/res/drawable/sf_shift.xml
@@ -0,0 +1,8 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- Material Symbols Rounded `shift`, Generated by tools/symbols.py. -->
+<vector xmlns:android="http://schemas.android.com/apk/res/android"
+ android:width="24dp" android:height="24dp" android:viewportWidth="960" android:viewportHeight="960">
+ <group android:translateY="960">
+ <path android:fillColor="#FF000000" android:pathData="M315.22-160.72v-274.5H205.67q-29.34 0-41.52-25.73-12.17-25.72 6.22-48.35L444.7-844.63q13.67-16.91 35.3-16.91 21.63 0 35.3 16.91L789.63-509.3q18.39 22.63 6.22 48.35-12.18 25.73-41.52 25.73H644.78v274.5q0 19.15-13.17 32.33-13.18 13.17-32.33 13.17H360.72q-19.15 0-32.33-13.17-13.17-13.18-13.17-32.33Zm91-45.5h147.56v-320h104.07L480-744.2 302.15-526.22h104.07v320Zm73.78-320Z"/>
+ </group>
+</vector>
diff --git a/apps/android/gradle/libs.versions.toml b/apps/android/gradle/libs.versions.toml
index c28f5e01..8c290a16 100644
--- a/apps/android/gradle/libs.versions.toml
+++ b/apps/android/gradle/libs.versions.toml
@@ -25,6 +25,7 @@ jna = "5.18.1"
glance = "1.2.0"
backdrop = "2.0.1"
kyant-shapes = "1.2.1"
+webrtc = "125.6422.07"
[libraries]
compose-bom = { module = "androidx.compose:compose-bom", version.ref = "compose" }
@@ -58,6 +59,7 @@ firebase-bom = { module = "com.google.firebase:firebase-bom", version.ref = "fir
firebase-messaging = { module = "com.google.firebase:firebase-messaging" }
work = { module = "androidx.work:work-runtime", version.ref = "work" }
camera-core = { module = "androidx.camera:camera-core", version.ref = "camera" }
+webrtc = { module = "io.github.webrtc-sdk:android", version.ref = "webrtc" }
camera-camera2 = { module = "androidx.camera:camera-camera2", version.ref = "camera" }
camera-lifecycle = { module = "androidx.camera:camera-lifecycle", version.ref = "camera" }
camera-view = { module = "androidx.camera:camera-view", version.ref = "camera" }
diff --git a/apps/android/tools/symbols.py b/apps/android/tools/symbols.py
index 9e3bffe0..9d47785e 100644
--- a/apps/android/tools/symbols.py
+++ b/apps/android/tools/symbols.py
@@ -157,6 +157,7 @@ MAP = {
"return": "keyboard_return",
"rotate.right": "rotate_right",
"safari": "explore",
+ "shift": "shift",
"shippingbox": "package_2",
"square.and.arrow.down": "download",
"square.and.arrow.up": "ios_share",
diff --git a/apps/desktop/src/renderer/views/tray-summary.ts b/apps/desktop/src/renderer/views/tray-summary.ts
index 03a133aa..872c9f2a 100644
--- a/apps/desktop/src/renderer/views/tray-summary.ts
+++ b/apps/desktop/src/renderer/views/tray-summary.ts
@@ -138,7 +138,7 @@ function screenSubtitle(app: AppModel) {
if (app.screenNeedsApproval) return 'Needs approval in System Settings'
if (!screen.connected) return 'Starting Codync Screen…'
if (!screen.capture || !screen.input) return 'Needs permission'
- if (screen.viewers > 0) return screen.viewers === 1 ? 'Your iPhone is viewing' : `${screen.viewers} viewers`
+ if (screen.viewers > 0) return screen.viewers === 1 ? 'Your phone is viewing' : `${screen.viewers} viewers`
const bot = screen.agentBot ? local.bots.get(screen.agentBot) : null
if (bot) return `${bot.name} is using it`
const display = screen.displays.find((d) => d.main)?.name ?? screen.displays[0]?.name
diff --git a/apps/screen-macos/ViewingBadge.swift b/apps/screen-macos/ViewingBadge.swift
index ceb70e8e..1017fe5c 100644
--- a/apps/screen-macos/ViewingBadge.swift
+++ b/apps/screen-macos/ViewingBadge.swift
@@ -64,7 +64,7 @@ private struct BadgeView: View {
.opacity(pulse ? 0.35 : 1)
.animation(.easeInOut(duration: 0.9).repeatForever(autoreverses: true), value: pulse)
.onAppear { pulse = true }
- Text(model.viewers == 1 ? "iPhone is viewing" : "\(model.viewers) viewers")
+ Text(model.viewers == 1 ? "Phone is viewing" : "\(model.viewers) viewers")
.font(.system(size: 12, weight: .medium))
.foregroundStyle(.white)
Button(action: disconnect) {
diff --git a/docs/features/remote-screen.md b/docs/features/remote-screen.md
index d77aa49c..c2064a50 100644
--- a/docs/features/remote-screen.md
+++ b/docs/features/remote-screen.md
@@ -27,12 +27,19 @@ The iPad already supports all orientations. The desktop app and the TUI do not
use the iPhone viewer or its orientation policy; the capture helpers stream the
computer display without choosing the phone's orientation.
+The Android viewer follows auto-rotate like the rest of the Android app. Rotate locks
+the other orientation until the viewer closes. The activity handles orientation and
+keyboard changes itself, so turning the phone or attaching a hardware keyboard keeps
+the stream. In the background the Android viewer closes its stream and hands control
+back to the bots, then reconnects when the app returns.
+
## Boundaries
- `host/src/screen/` coordinates access to the local helper and owns viewer sessions. Another device cannot renegotiate or close a session it does not own.
- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions, including those of the computer-use driver it starts.
- `apps/screen-linux/` implements the Linux helper using desktop portals and GStreamer, including ICE URL conversion and TURN transport configuration. The host starts it from beside its own executable (then `PATH`). Linux host releases ship `codync-screen` in the same archive (built on Ubuntu 24.04: glibc 2.39+ and GStreamer 1.22+ with the base, good and bad plugins, PipeWire and xdg-desktop-portal at run time); `install.sh`, Homebrew and `codync-host update` put it next to the host.
- Helpers communicate locally through `~/.codync/screen.sock`. SDP is non-trickle; input uses the `input` and `input-fast` data channels.
+- Phone viewers: iOS in `apps/ios/Kit/Sources/CodyncUI/Screen/`. Android keeps the same structure in `apps/android/app/.../Screen*.kt`, with the gesture-to-input and viewport logic in `core` (`ScreenControl`, `ScreenViewport`). Both offer hardware H.264 first, High then Constrained Baseline.
- The Linux helper answers with the packetized H.264 payload the phone offers, feeds the encoder 8-bit 4:2:0 video and timestamps capture on the system clock. It answers before capture starts, so the profile is negotiated before the encoder fixes its output.
- Screen access is on by default on macOS and on Linux while a graphical session (Wayland or X11) is running; headless servers stay off. Turning it on or off is remembered. Enabling through `setScreenEnabled` requires a loopback caller. Interactive OS permission prompts must be completed on the computer.
- Bots with their computer capability enabled receive the built-in `computer` MCP tools, run by the computer-use driver that Codync Screen starts on macOS ([computer use](computer-use.md)). Their permission policy still applies. An interactive phone can take over; bots may still look.
@@ -57,11 +64,12 @@ References: [Cloudflare credential issuance](https://developers.cloudflare.com/r
## Verification
-Run cloud tests/type checking, host formatting/Clippy/tests, iOS Kit package tests, and builds of the iOS app and the macOS `Screen` helper target. Run Linux helper tests in a Linux environment with GStreamer and PipeWire development packages.
+Run cloud tests/type checking, host formatting/Clippy/tests, iOS Kit package tests, Android `:core:test` and `:app:testDebugUnitTest`, and builds of the iOS app, the Android app and the macOS `Screen` helper target. Run Linux helper tests in a Linux environment with GStreamer and PipeWire development packages.
Live acceptance requires the configured Worker and fresh host/helper/phone builds:
- Check screen-recording and input permissions, capture, click/type/scroll, clipboard, display changes and closing the viewer.
+- Run the checks below on an Android phone too, including a hardware keyboard and IME composition (Zhuyin, Pinyin).
- Open the iPhone viewer while upright and while already held sideways, then rotate
in both directions. Check video fitting and direct taps after each turn. Repeat
with rotation lock on and with the Rotate button; closing returns to portrait.
diff --git a/docs/features/voice-call.md b/docs/features/voice-call.md
index 8bc05e9a..4c577556 100644
--- a/docs/features/voice-call.md
+++ b/docs/features/voice-call.md
@@ -1,6 +1,6 @@
# Voice call
-A hands-free call with one bot from the iPhone or desktop chat (the waveform button in an empty composer).
+A hands-free call with one bot or group from the iPhone, Android or desktop chat (the waveform button in an empty composer).
On Linux the desktop app offers only the cloud engines (on-device speech is macOS only). The terminal
UI has no calls (no audio stack); it shows the "Voice chat · 00:16" notice like every client.
The bot is the same agent on the computer; the call only changes how you talk to it.
@@ -8,6 +8,7 @@ The bot is the same agent on the computer; the call only changes how you talk to
## On-device voice
- `apps/ios/Kit/Sources/CodyncUI/Call/` (iPhone) and `apps/desktop/src/renderer/views/call/` (desktop): `CallView` (the call bar) over a `VoiceEngine`; `OnDeviceEngine` is the default audio loop. On the Mac the desktop app runs speech recognition in `codync-speech` (`apps/desktop/native/speech-macos/`, driven by `apps/desktop/src/main/speech.ts`).
+- Android: `CallBar` (`apps/android/app/.../VoiceControls.kt`) over the same `VoiceEngine` set, ported into `apps/android/core` (`VoiceEngine.kt`, `CloudSpeechEngine.kt`, `GeminiLiveEngine.kt`, `OpenAIRealtimeEngine.kt`, `CallOperator.kt`) with the phone's audio in the app (`VoiceAudioPorts.kt`, `WebRtcPeer.kt`). On device is `VoiceSession` with offline English Vosk recognition and an installed Android voice (`LocalVoiceAudio.kt`); `VoiceService` (a microphone foreground service) holds audio focus and the Bluetooth/wired route for the call. Settings keys match the iPhone's (`callEngine`, `callMode.<provider>`, …).
- UI follows Grok Bot: a floating capsule over the top of the chat (`ThreadView` overlay), and the
chat stays readable and usable underneath. Left to right: the bot's avatar (pulses while
speaking; tap it to interrupt), a dotted level line (your voice while listening, a ripple while
@@ -118,7 +119,7 @@ built into the host and the app (today `gpt-realtime-2.1`, `gpt-transcribe`, `gp
list arrives.
Voice settings (`VoiceChatSettingsView`) open from the call bar's gear and from settings: on the
-iPhone, **Computers & settings → Voice chat**; on the desktop, **Settings → Voice chat** (this
+iPhone, **Computers & settings → Voice chat**; on Android, **Settings → Voice and API keys**; on the desktop, **Settings → Voice chat** (this
computer) or the waveform button on a computer's card in **Computers & devices**. Keys are entered
on the computer only: the host takes `setVoiceKey` from loopback, and the iPhone shows whether a key
is there and can test it. Picking a cloud engine shows its key form instead of the on-device settings: a secure field, a link
@@ -184,6 +185,9 @@ a spoken reply, and `send_to_bot` called for a Chinese request. The iOS Simulato
audio unit can abort with an RPC timeout when WebRTC starts its audio; use a physical iPhone for audio.
`StoreTests` cover the operator's tools against a scripted host (send, status with an approval,
-option matching, recent messages) and approvals reaching a call as notices. Provider audio isn't
+option matching, recent messages) and approvals reaching a call as notices. On Android, `VoiceEnginesTest` (core, JVM) drives
+each engine through fake audio: utterance cutting and clip playback, OpenAI's event sequence and SDP
+exchange (MockWebServer), and a Gemini Live session over a real WebSocket (setup, audio both ways,
+a tool call, a close with the provider's reason). Provider audio isn't
simulated: on a physical iPhone, check both engines with a real key (talk over a reply, ask "what
are you doing?", approve by voice, and turn off Wi-Fi mid-call to see the fallback).
diff --git a/docs/plans/android-app.md b/docs/plans/android-app.md
index a6cb2e0f..cf053b48 100644
--- a/docs/plans/android-app.md
+++ b/docs/plans/android-app.md
@@ -22,12 +22,12 @@ emulator against a real host; device acceptance is listed separately.
| Memory | Ported: search, filters, add/edit, pin, review, history and timeline, export/import, forget all | — |
| Routines | Ported: list with switches, editor (schedule, webhook panel, run settings), test run, delete | Live provider/routine acceptance |
| Extensions | Ported: agents (install, sign-in, setup terminal), registry connectors, Composio apps, skills, installed list, custom connector, own skill, credentials, chat connection request card | Real provider OAuth, 1Password |
-| Voice | Ported call bar and Voice chat settings; on-device Android speech | OpenAI Realtime / Gemini Live engines |
+| Voice | Ported: call bar (provider name, fallback notice), Voice chat settings (engine, mode, models, voice, minutes this month), on device (Vosk English + Android TTS), OpenAI Realtime (WebRTC), Gemini Live, Speech mode, fallback to on device, group calls | Both providers with a real key on a device; on-device recognition in languages other than English |
| State/widgets | Ported: Widget (with the animated Add a widget guide), Live update and Status bar pages, Usage sheet, Choose the look; Provider usage, Bots and Team widgets with launcher Edit widget | Physical launcher refresh |
| Live updates | Android 16 Live updates (`ProgressStyle`), the status bar chip on Android 16 QPR1 (API 36.1) | Background updates need FCM in the deployment |
| Version check | `VersionMismatch.MIN_HOST`, the host's `minApp`, the update card in the roster and in place of the composer | Set `MIN_HOST` to the first release that ships Android support when it's cut |
-| Remote screen | Not ported | WebRTC view and control |
-| Analytics | Not ported | PostHog with the Settings switch |
+| Remote screen | Ported: viewer from Computers, the chat header (watch a bot, then take over) and `codync-dev://screen`; WebRTC H.264, trackpad and direct touch, long-press drag, two-finger scroll, pinch zoom streaming the visible region, keyboard with IME and modifier keys, clipboard, displays, take over/hand back, Rotate | Pinch, IME composition and a hardware keyboard on a device; TURN relay path and renewal |
+| Analytics | Ported: PostHog feature events with the Settings switch (and its sheet), crash reports sent on the next launch | — |
| Packaging | Debug, unsigned R8 release, CI, optional Firebase/private configuration | Play signing, distribution, `versionCode` policy |
## Gates before merge into dev