Skip to content

Open source Grok Bot alternative, Bring your own agent and subscription: message Claude Code, Codex, Gemini CLI, Cursor, Copilot and 40+ AI agents as bots on your own computer, from iPhone, Mac, Windows, Linux or terminal. Agent-agnostic and free. Also a Muse and Dots alternative. Built in Rust.

fix(screen): connect remote screens with trickle ICE (#40, #235)

Sean Ferguson6c28b3b7ad43
diff --git a/.github/workflows/screen-macos.yml b/.github/workflows/screen-macos.yml
index 7c85e12a..1c50661f 100644
--- a/.github/workflows/screen-macos.yml
+++ b/.github/workflows/screen-macos.yml
@@ -8,6 +8,7 @@ on:
pull_request:
paths:
- "apps/screen-macos/**"
+ - "apps/screen-macos-tests/**"
- "apps/project.yml"
- "apps/ios/Kit/Package.swift"
- "apps/ios/Kit/Package.resolved"
@@ -40,3 +41,6 @@ jobs:
-destination 'generic/platform=macOS'
-derivedDataPath build/screen-ci
CODE_SIGNING_ALLOWED=NO
+
+ - name: Test helper socket lifecycle and candidates
+ run: swift test --package-path apps/screen-macos-tests --scratch-path build/dd/screen-macos-tests
diff --git a/CLAUDE.md b/CLAUDE.md
index 364d9b89..6bb2506c 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -31,7 +31,7 @@ Bot-based remote for coding agents: persistent named bots on your computer, mess
- **Sync**: every mutation stamps a global `rev`. Clients call `GET /events?since=<rev>` (catch-up in rev order, then live). Emission happens under `Hub::emit_lock` so events leave in rev order. Clients upsert by id; never skip undecodable events (the iOS app rewinds to rev 0).
- API: `POST /api/<method>` + SSE with the bearer token (`~/.codync/token`) is **loopback only** (desktop app, SSH tunnel, local helpers). Phones and other remote clients use the E2E channel (`/channel` direct, or the Cloudflare relay). Default port **19222**.
- Remote access (Cloudflare relay primary, direct LAN/Tailscale alternative, accounts, SSH, public routine webhooks queued in the relay §7.8, the cloud assistant MCP connector at mcp.codync.dev §7.9): [docs/reference/remote-relay.md](docs/reference/remote-relay.md).
-- Remote screen (`host/src/screen/` is the reference): phones view/control the computer over WebRTC (hardware H.264, non-trickle SDP relayed by `screenOffer`, input on data channels `input` / `input-fast`); bots get the built-in `computer` MCP server (`codync-host mcp computer`, `host/src/mcp.rs`) when their `computer` flag is on; its tools run on the bundled cua-driver, which Codync Screen starts on macOS so its grants apply ([computer use](docs/features/computer-use.md)). Capture/input live in a helper on `~/.codync/screen.sock`: `apps/screen-macos` (macOS, launchd agent the desktop app registers via `SMAppService`, owns the TCC grants) or `apps/screen-linux` (`codync-screen`: portals + GStreamer, started by the host). On by default (Linux: only with a graphical session); `setScreenEnabled` is accepted only from loopback. An interactive phone takes over (bots may only look).
+- Remote screen (`host/src/screen/` is the reference): phones view/control the computer over WebRTC (hardware H.264, negotiated trickle ICE with complete-SDP fallback, signaled through `screenPrepare` / `screenOffer` and `screenCandidate` / `screenCandidates`, input on data channels `input` / `input-fast`); bots get the built-in `computer` MCP server (`codync-host mcp computer`, `host/src/mcp.rs`) when their `computer` flag is on; its tools run on the bundled cua-driver, which Codync Screen starts on macOS so its grants apply ([computer use](docs/features/computer-use.md)). Capture/input live in a helper on `~/.codync/screen.sock`: `apps/screen-macos` (macOS, launchd agent the desktop app registers via `SMAppService`, owns the TCC grants) or `apps/screen-linux` (`codync-screen`: portals + GStreamer, started by the host). On by default (Linux: only with a graphical session); `setScreenEnabled` is accepted only from loopback. An interactive phone takes over (bots may only look).
- Multiple computers per account are future work, not a current priority: the product targets one computer. The multi-computer structure (`AccountStore`, `BotReference`) stays because relay/accounts/SSH build on it; don't extend or polish multi-computer features unless asked. Bots on the user's other computers reach each other through `codync-network` (`host/crates/network`, peer to peer over iroh, signed roster with an owner; `codync-host network`): [codync-network](docs/features/codync-network.md).
- Environments: `iOS Dev` / desktop `dist:*:dev` use the isolated Codync Dev app; `iOS` / `dist:*:main` retain production identities. Stop only the variant being rebuilt. Resource map, signing and diagnostics: [Dev app isolation](docs/guides/dev-app.md). Local secrets: git-ignored `.env.development` / `.env.production` at the root, templates `.env.<env>.example` ([local secrets](docs/guides/dev-app.md#local-secrets)).
- Push: iOS registers its APNs token with `relay/` → gets an AES-GCM ticket → gives it (plus its X25519 push key) to the host; the host seals title/body to that key and a Notification Service Extension opens it, so `relay/` sees only generic text. Alert kinds: *needs you*, *done*, and *failed*, suppressed while the iOS app is connected. Delivery and lifecycle: [notification design](docs/design/push-and-live-activity.md).
diff --git a/apps/Codync.xcodeproj/project.pbxproj b/apps/Codync.xcodeproj/project.pbxproj
index ffa06fbb..90d5a76d 100644
--- a/apps/Codync.xcodeproj/project.pbxproj
+++ b/apps/Codync.xcodeproj/project.pbxproj
@@ -16,6 +16,7 @@
1302DAC1A6D9AE9E33654B84 /* RowLabel.swift in Sources */ = {isa = PBXBuildFile; fileRef = B708DF60FB00E1724260BCD2 /* RowLabel.swift */; };
16C24A622E33414C24B0C148 /* AppearanceSettings.swift in Sources */ = {isa = PBXBuildFile; fileRef = 0814BD0C30F01A163FBDF896 /* AppearanceSettings.swift */; };
17DD96A91B1EC2AE699F4A2F /* ViewingBadge.swift in Sources */ = {isa = PBXBuildFile; fileRef = 2C0E04490F5C56FD070539E6 /* ViewingBadge.swift */; };
+ 17EFF498D834DBBE41BD0FDB /* SessionCandidate.swift in Sources */ = {isa = PBXBuildFile; fileRef = D55F2906A970053C26B95A41 /* SessionCandidate.swift */; };
1855DC364C2F8E5B62878BD8 /* CodyncBot-teardrop-Ink.ttf in Resources */ = {isa = PBXBuildFile; fileRef = 203A43D8DF8D3832F95FA837 /* CodyncBot-teardrop-Ink.ttf */; };
198096547EAA6C5FBB68558F /* CodyncBot-hex-Ink.ttf in Resources */ = {isa = PBXBuildFile; fileRef = 89A0D530BEF30AF36F9DBEA8 /* CodyncBot-hex-Ink.ttf */; };
2002C16E427B4E4E7279E1C4 /* ComputerStatus.swift in Sources */ = {isa = PBXBuildFile; fileRef = 6BC00039A6C2C0D3F39A5F67 /* ComputerStatus.swift */; };
@@ -83,6 +84,7 @@
F842AE1C5ED96AF9A7F5D16C /* Widgets.appex in Embed Foundation Extensions */ = {isa = PBXBuildFile; fileRef = 6D734BEE7CD3F84677B0218B /* Widgets.appex */; settings = {ATTRIBUTES = (RemoveHeadersOnCopy, ); }; };
F97C489DDB806224CEFCEB57 /* ScreenHelper.swift in Sources */ = {isa = PBXBuildFile; fileRef = 28B2CB561A43ED9F2F1A4391 /* ScreenHelper.swift */; };
FCE6E509AEFC88C3D05D8E3D /* CodyncBot-tablet-Tint.ttf in Resources */ = {isa = PBXBuildFile; fileRef = F7CAD5C4F330637102B40288 /* CodyncBot-tablet-Tint.ttf */; };
+ FFCAB62428CD3D4FB479B4AB /* SessionCandidates.swift in Sources */ = {isa = PBXBuildFile; fileRef = 46538A6E6E6B7C5E212F5CDF /* SessionCandidates.swift */; };
/* End PBXBuildFile section */
/* Begin PBXContainerItemProxy section */
@@ -139,6 +141,7 @@
36C62636FD789C49762696A3 /* RosterSectionHeader.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = RosterSectionHeader.swift; sourceTree = "<group>"; };
3F4C4D3B64C608AEF2F85F91 /* Kit */ = {isa = PBXFileReference; lastKnownFileType = folder; name = Kit; path = ios/Kit; sourceTree = SOURCE_ROOT; };
445DB0A03CFBB5CC7A830139 /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist; path = Info.plist; sourceTree = "<group>"; };
+ 46538A6E6E6B7C5E212F5CDF /* SessionCandidates.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SessionCandidates.swift; sourceTree = "<group>"; };
4815867922559696EDCA47FD /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = "<group>"; };
4EB282B31FBC94EE70D07638 /* AccountSwitcherView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AccountSwitcherView.swift; sourceTree = "<group>"; };
4FA78A055791BBBA79DF8C8B /* Permissions.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = Permissions.swift; sourceTree = "<group>"; };
@@ -180,6 +183,7 @@
D0DE72EEAEC947C1D3CA237F /* PrivacyInfo.xcprivacy */ = {isa = PBXFileReference; path = PrivacyInfo.xcprivacy; sourceTree = "<group>"; };
D0E552136D473567E0C25ED4 /* CodyncBot-blob-Tint.ttf */ = {isa = PBXFileReference; lastKnownFileType = file; path = "CodyncBot-blob-Tint.ttf"; sourceTree = "<group>"; };
D4D13418ABB5FD7CFC087677 /* AXTree.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AXTree.swift; sourceTree = "<group>"; };
+ D55F2906A970053C26B95A41 /* SessionCandidate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = SessionCandidate.swift; sourceTree = "<group>"; };
D57A7C09154869DD84BACE19 /* WelcomeView.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WelcomeView.swift; sourceTree = "<group>"; };
D5F23C3F9FCE63B4AFB4771A /* CodyncBot-squircle-Tint.ttf */ = {isa = PBXFileReference; lastKnownFileType = file; path = "CodyncBot-squircle-Tint.ttf"; sourceTree = "<group>"; };
D827E43025A592B41CB68EFD /* CodyncBot-hex-Tint.ttf */ = {isa = PBXFileReference; lastKnownFileType = file; path = "CodyncBot-hex-Tint.ttf"; sourceTree = "<group>"; };
@@ -378,6 +382,8 @@
4FA78A055791BBBA79DF8C8B /* Permissions.swift */,
28B2CB561A43ED9F2F1A4391 /* ScreenHelper.swift */,
F99C11684BFBEB49006600A9 /* Session.swift */,
+ D55F2906A970053C26B95A41 /* SessionCandidate.swift */,
+ 46538A6E6E6B7C5E212F5CDF /* SessionCandidates.swift */,
2C0E04490F5C56FD070539E6 /* ViewingBadge.swift */,
);
path = "screen-macos";
@@ -690,6 +696,8 @@
A1B7C7774E7FAF6D45D1BEF8 /* Permissions.swift in Sources */,
F97C489DDB806224CEFCEB57 /* ScreenHelper.swift in Sources */,
921ED46A1643AB90BB3D4CF7 /* Session.swift in Sources */,
+ 17EFF498D834DBBE41BD0FDB /* SessionCandidate.swift in Sources */,
+ FFCAB62428CD3D4FB479B4AB /* SessionCandidates.swift in Sources */,
17DD96A91B1EC2AE699F4A2F /* ViewingBadge.swift in Sources */,
);
runOnlyForDeploymentPostprocessing = 0;
diff --git a/apps/android/README.md b/apps/android/README.md
index e9e764cc..9ac5ef4f 100644
--- a/apps/android/README.md
+++ b/apps/android/README.md
@@ -76,6 +76,12 @@ current pairing in Computer management before pairing another. Multiple-computer
support is tracked in the checklist. Accounts have separate device keys, mirrors,
files, image caches and composer drafts.
+Remote screen negotiates incremental ICE with hosts and helpers that advertise
+support. It exchanges offers and answers while candidates are still gathering,
+buffers candidates until the answer is installed, and keeps established media
+running through a brief signaling disconnect. Older hosts and helpers continue
+using complete SDP.
+
The roster shows activity, previews, unread counts, groups, pin/hide actions and sections
(Move to section in a bot's menu; tap a section's header to fold it on this phone, touch and hold it to rename or remove it; bots in none sit under a "Bots" label);
touch and hold a bot, then move it, to reorder within its section (the order syncs through the host). Cached bots remain readable offline and are labeled as saved. An
diff --git a/apps/android/WhatsNew.md b/apps/android/WhatsNew.md
index 9bf7ba54..bd70e405 100644
--- a/apps/android/WhatsNew.md
+++ b/apps/android/WhatsNew.md
@@ -11,6 +11,8 @@ else a generic "Bug fixes and improvements" line. Delete old sections.
### zh-Hant
- Bot 清單可以分成自己的分組,例如「工作」和「個人」:長按 Bot 選「移到分組」,點分組標題即可收合,收合時會顯示裡面的未讀數。
+- 遠端畫面連線更快,網路短暫中斷時也不會斷掉。
### en-US
- Sort the bot list into your own sections, like Work and Personal: touch and hold a bot and choose Move to section, then tap a section's heading to fold it, with its unread count showing while it's folded.
+- The remote screen connects faster and stays up through a brief connection drop.
diff --git a/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt b/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt
index 7d53861f..e3368e2a 100644
--- a/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt
+++ b/apps/android/app/src/main/kotlin/com/codync/android/ScreenSession.kt
@@ -20,8 +20,10 @@ import kotlinx.coroutines.withTimeoutOrNull
import kotlinx.serialization.json.JsonObject
import kotlinx.serialization.json.buildJsonObject
import kotlinx.serialization.json.decodeFromJsonElement
+import kotlinx.serialization.json.encodeToJsonElement
import kotlinx.serialization.json.jsonPrimitive
import kotlinx.serialization.json.put
+import org.webrtc.AddIceObserver
import org.webrtc.DataChannel
import org.webrtc.DefaultVideoDecoderFactory
import org.webrtc.DefaultVideoEncoderFactory
@@ -45,8 +47,8 @@ import kotlin.coroutines.resumeWithException
/**
* One live view of the computer's screen (CodyncUI `ScreenSession`): a WebRTC peer that receives
* the display as H.264 and sends input over two data channels (`input` reliable, `input-fast`
- * unordered for pointer moves). Signaling goes through the host API, non-trickle (offer and
- * answer carry every candidate). Lives on the main thread; WebRTC callbacks hop onto it.
+ * unordered for pointer moves). Signaling negotiates trickle ICE, with complete-SDP fallback
+ * for older hosts/helpers. Lives on the main thread; WebRTC callbacks hop onto it.
*/
class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
sealed interface Phase {
@@ -78,6 +80,8 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
private var reliable: DataChannel? = null
private var fast: DataChannel? = null
private var gathered: CompletableDeferred<Unit>? = null
+ private var signaling: ScreenSignaling? = null
+ private var recoveryAttempts = 0
private var recoverJob: Job? = null
private var renewJob: Job? = null
private var connectionTimer: Job? = null
@@ -96,13 +100,14 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
// MARK: connection
fun start() {
+ recoveryAttempts = 0
phase = Phase.Connecting
scope.launch {
try { connect() }
catch (cancelled: CancellationException) { throw cancelled }
catch (_: Superseded) { /* Its replacement reports for itself. */ }
catch (error: Exception) {
- if (phase == Phase.Closed) return@launch
+ if (phase != Phase.Connecting) return@launch
close()
phase = Phase.Failed(error.message ?: "Couldn't connect to the remote screen.")
}
@@ -113,7 +118,8 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
val preparing = generation
val active = store.ready()
val relay = (active.state.value as? LinkState.Ready)?.route == HostRoute.Relay
- val connection = WireJson.decodeFromJsonElement<ScreenConnection>(active.call("screenPrepare", buildJsonObject { put("relay", relay) }))
+ val connection = WireJson.decodeFromJsonElement<ScreenConnection>(active.call("screenPrepare", buildJsonObject { put("relay", relay); put("trickle", true) }))
+ if (BuildConfig.DEBUG) Log.d("CodyncScreen", "screen prepared: trickle=${connection.trickle}")
if (generation != preparing || phase == Phase.Closed) {
closeRemote(connection.session)
throw Superseded()
@@ -137,8 +143,33 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
bundlePolicy = PeerConnection.BundlePolicy.MAXBUNDLE
}
val factory = ScreenRtc.factory(context)
- val pc = factory.createPeerConnection(config, Events(current)) ?: throw IOException("Couldn't start the video connection.")
+ val events = Events(current)
+ val pc = factory.createPeerConnection(config, events) ?: throw IOException("Couldn't start the video connection.")
this.pc = pc
+ if (connection.trickle) {
+ val next = ScreenSignaling(scope, screenCandidates(active, connection.session), send = { candidate ->
+ active.call("screenCandidate", buildJsonObject {
+ put("session", connection.session)
+ put("candidate", WireJson.encodeToJsonElement(candidate))
+ })
+ }, apply = { candidate ->
+ addCandidate(pc, candidate)
+ }, isMediaConnected = {
+ this.pc === pc && pc.iceConnectionState() in listOf(PeerConnection.IceConnectionState.CONNECTED, PeerConnection.IceConnectionState.COMPLETED)
+ }, onFailure = { error ->
+ if (generation == current && phase != Phase.Closed) {
+ lastError = error.message
+ teardown()
+ phase = Phase.Reconnecting
+ if (recoverJob?.isActive != true) scheduleRecovery(0)
+ }
+ })
+ signaling = next
+ events.signaling = next
+ next.subscribe()
+ if (this.pc !== pc || generation != current || phase == Phase.Closed) throw Superseded()
+ next.check()
+ }
val video = pc.addTransceiver(MediaStreamTrack.MediaType.MEDIA_TYPE_VIDEO,
RtpTransceiver.RtpTransceiverInit(RtpTransceiver.RtpTransceiverDirection.RECV_ONLY))
val codecs = factory.getRtpReceiverCapabilities(MediaStreamTrack.MediaType.MEDIA_TYPE_VIDEO).codecs
@@ -147,8 +178,9 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
reliable = pc.createDataChannel("input", DataChannel.Init())?.also { it.registerObserver(Messages(current)) }
fast = pc.createDataChannel("input-fast", DataChannel.Init().apply { ordered = false; maxRetransmits = 0 })
- negotiate(pc)
+ negotiate(pc, active)
if (generation != current || phase == Phase.Closed) throw Superseded()
+ signaling?.check()
// Refresh both endpoints before TURN credentials expire, including an idle screen.
renewJob = scope.launch {
delay(connection.renewIn(System.currentTimeMillis()))
@@ -165,15 +197,18 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
}
/** Offer → host → answer with the reserved session's ICE configuration. */
- private suspend fun negotiate(pc: PeerConnection) {
+ private suspend fun negotiate(pc: PeerConnection, active: EncryptedChannel) {
+ val signaling = signaling
val offer = suspendCancellableCoroutine { done ->
pc.createOffer(Sdp({ done.resume(it) }, { done.resumeWithException(IOException(it)) }), MediaConstraints())
}
+ if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
pc.await { pc.setLocalDescription(it, offer) }
- waitForCandidates(pc)
if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
- val sdp = pc.localDescription?.description ?: offer.description
- val response = store.hostCall("screenOffer", buildJsonObject {
+ if (signaling == null) waitForCandidates(pc)
+ if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
+ val sdp = if (signaling == null) pc.localDescription?.description ?: offer.description else offer.description
+ val response = active.call("screenOffer", buildJsonObject {
put("sdp", sdp)
session?.let { put("session", it) }
display?.let { put("display", it.id) }
@@ -182,6 +217,19 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
session = answer.session
pc.await { pc.setRemoteDescription(it, SessionDescription(SessionDescription.Type.ANSWER, answer.sdp)) }
+ if (this.pc !== pc || phase == Phase.Closed) throw Superseded()
+ signaling?.installedAnswer()
+ }
+
+ private suspend fun addCandidate(pc: PeerConnection, event: ScreenCandidate) = suspendCancellableCoroutine<Unit> { done ->
+ // The Linux helper sends no sdpMid; native WebRTC can't take null, and an empty mid falls back to the index.
+ val candidate = IceCandidate(event.sdpMid ?: "", requireNotNull(event.sdpMLineIndex), requireNotNull(event.candidate))
+ pc.addIceCandidate(candidate, object : AddIceObserver {
+ override fun onAddSuccess() { if (done.isActive) done.resume(Unit) }
+ override fun onAddFailure(error: String?) {
+ if (done.isActive) done.resumeWithException(ScreenProtocolException("Couldn't add a screen ICE candidate."))
+ }
+ })
}
private suspend fun PeerConnection.await(set: (SdpObserver) -> Unit) = suspendCancellableCoroutine { done ->
@@ -200,6 +248,8 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
if (generation != this.generation || phase == Phase.Closed) return
when (state) {
PeerConnection.IceConnectionState.CONNECTED, PeerConnection.IceConnectionState.COMPLETED -> {
+ signaling?.mediaConnected()
+ recoveryAttempts = 0
connectionTimer?.cancel()
phase = Phase.Live
lastError = null
@@ -214,32 +264,51 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
private fun scheduleRecovery(delayMillis: Long) {
recoverJob?.cancel()
recoverJob = scope.launch {
- var attempt = 0
delay(delayMillis)
while (isActive && phase == Phase.Reconnecting) {
+ if (recoveryAttempts >= 3) {
+ val message = lastError ?: "Couldn't reconnect to the remote screen."
+ close()
+ phase = Phase.Failed(message)
+ return@launch
+ }
+ // Wait for a usable signaling route before spending a negotiation attempt.
+ try { waitForScreenRoute { store.channel?.state?.value } }
+ catch (cancelled: CancellationException) { throw cancelled }
+ catch (error: Exception) {
+ if (phase != Phase.Reconnecting) return@launch
+ close()
+ phase = Phase.Failed(error.message ?: "Couldn't reach the computer.")
+ return@launch
+ }
try {
- // Re-evaluate the signaling route after Wi-Fi/cellular changes and fetch fresh ICE credentials.
+ if (phase != Phase.Reconnecting) return@launch
+ recoveryAttempts += 1
connect()
return@launch
} catch (cancelled: CancellationException) { throw cancelled }
- catch (_: Superseded) { return@launch }
+ catch (_: Superseded) { /* A failed negotiation may have retired this peer. */ }
catch (error: Exception) {
if (phase == Phase.Closed) return@launch
+ lastError = error.message
Log.i("CodyncScreen", "screen reconnect failed: ${error.message}")
- attempt += 1
- if (attempt >= 3) {
+ // The host answers most errors with 400 (a restarting helper, a replaced session): only a denial or a
+ // required update ends recovery early, and the last attempt fails at once, as on the iPhone.
+ if ((error is HostError && error.status in screenTerminalStatuses) || recoveryAttempts >= 3) {
close()
- phase = Phase.Failed(error.message ?: "Couldn't reconnect to the remote screen.")
+ phase = Phase.Failed(lastError ?: "Couldn't reconnect to the remote screen.")
return@launch
}
- delay(minOf(attempt * 2_000L, 10_000L))
}
+ if (phase != Phase.Reconnecting) return@launch
+ delay(minOf(maxOf(1, recoveryAttempts) * 2_000L, 10_000L))
}
}
}
fun close() {
recoverJob?.cancel()
+ recoverJob = null
phase = Phase.Closed
session?.let(::closeRemote)
session = null
@@ -255,6 +324,8 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
private fun teardown() {
generation += 1
+ signaling?.stop()
+ signaling = null
renewJob?.cancel()
connectionTimer?.cancel()
gathered?.complete(Unit)
@@ -303,14 +374,18 @@ class ScreenSession(private val store: AppStore, display: ScreenDisplay?) {
// MARK: WebRTC callbacks (its signaling thread) → main
private inner class Events(private val generation: Int) : PeerConnection.Observer {
+ @Volatile var signaling: ScreenSignaling? = null
override fun onIceConnectionChange(state: PeerConnection.IceConnectionState) { scope.launch { iceChanged(state, generation) } }
override fun onIceGatheringChange(state: PeerConnection.IceGatheringState) {
if (state != PeerConnection.IceGatheringState.COMPLETE) return
+ signaling?.offerLocal(ScreenCandidate.complete)
scope.launch { if (generation == [email protected]) gathered?.complete(Unit) }
}
override fun onSignalingChange(state: PeerConnection.SignalingState) = Unit
override fun onIceConnectionReceivingChange(receiving: Boolean) = Unit
- override fun onIceCandidate(candidate: IceCandidate) = Unit
+ override fun onIceCandidate(candidate: IceCandidate) {
+ signaling?.offerLocal(ScreenCandidate(ScreenCandidate.Kind.Candidate, candidate.sdp, candidate.sdpMLineIndex, candidate.sdpMid))
+ }
override fun onIceCandidatesRemoved(candidates: Array<out IceCandidate>) = Unit
override fun onAddStream(stream: MediaStream) = Unit
override fun onRemoveStream(stream: MediaStream) = Unit
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/EncryptedChannel.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/EncryptedChannel.kt
index f7587432..bcbb85cb 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/EncryptedChannel.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/EncryptedChannel.kt
@@ -65,7 +65,8 @@ class EncryptedChannel internal constructor(
private var opener: FrameOpener? = null
private val ids = AtomicLong()
private val calls = ConcurrentHashMap<Long, CompletableDeferred<JsonElement>>()
- private val streams = ConcurrentHashMap<Long, Channel<JsonObject>>()
+ private data class Subscription(val kind: String, val events: Channel<JsonObject>)
+ private val streams = ConcurrentHashMap<Long, Subscription>()
private val mailboxCalls = ConcurrentHashMap<String, CompletableDeferred<JsonObject>>()
private val listLock = Mutex()
private val mailboxUpdates = MutableSharedFlow<JsonObject>(extraBufferCapacity = 256)
@@ -189,20 +190,29 @@ class EncryptedChannel internal constructor(
require(id in 1..0xffff_ffffL) { "Invalid RPC ID" }
when {
"ev" in message -> {
- val events = streams[id] ?: return
+ val subscription = streams[id] ?: return
+ val events = subscription.events
+ if (subscription.kind == "screenCandidates") {
+ val result = events.trySend(message.getValue("ev").jsonObject)
+ if (result.isFailure && !result.isClosed && streams.remove(id, subscription)) {
+ events.close(ScreenProtocolException("Screen ICE candidate queue overflowed."))
+ cancelRemote(id)
+ }
+ return
+ }
// Bounded suspension reaches the socket reader instead of dropping catch-up.
// A closed subscription is expected when the user leaves that screen.
try { events.send(message.getValue("ev").jsonObject) }
catch (_: kotlinx.coroutines.channels.ClosedSendChannelException) { }
catch (error: CancellationException) { val task = kotlinx.coroutines.currentCoroutineContext(); task.ensureActive() }
}
- message["end"]?.jsonPrimitive?.booleanOrNull == true -> streams.remove(id)?.close()
+ message["end"]?.jsonPrimitive?.booleanOrNull == true -> streams.remove(id)?.events?.close()
"err" in message -> {
val detail = message.getValue("err").jsonObject
val failure = HostError(detail["status"]?.jsonPrimitive?.longOrNull?.toInt() ?: 500,
detail.text("message") ?: "The computer could not complete the request.")
calls.remove(id)?.completeExceptionally(failure)
- streams.remove(id)?.close(failure)
+ streams.remove(id)?.events?.close(failure)
}
"ok" in message -> calls.remove(id)?.complete(message.getValue("ok"))
else -> throw IOException("Invalid RPC response")
@@ -241,17 +251,17 @@ class EncryptedChannel internal constructor(
awaitReady()
require(!pairing)
val id = nextId()
- val events = Channel<JsonObject>(256)
- streams[id] = events
+ val events = Channel<JsonObject>(if (kind == "screenCandidates") ScreenCandidate.maxCount + 2 else 256)
+ streams[id] = Subscription(kind, events)
try {
sendInner(buildJsonObject {
put("id", id); put("sub", kind); put("b", body)
})
for (event in events) emit(event)
} finally {
- streams.remove(id)
+ val subscribed = streams.remove(id) != null
events.cancel()
- cancelRemote(id)
+ if (subscribed) cancelRemote(id)
}
}
@@ -319,7 +329,7 @@ class EncryptedChannel internal constructor(
private fun dropPending(error: Throwable) {
calls.keys.toList().forEach { calls.remove(it)?.completeExceptionally(error) }
- streams.keys.toList().forEach { streams.remove(it)?.close(error) }
+ streams.keys.toList().forEach { streams.remove(it)?.events?.close(error) }
}
private fun fail(error: Throwable) {
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt
index c206704f..0c46df74 100644
--- a/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/Screen.kt
@@ -72,7 +72,8 @@ data class ScreenDisplay(
/** A reserved viewer session and the ICE servers to reach it (`screenPrepare`). */
@Serializable
-data class ScreenConnection(val session: String, val iceServers: List<ScreenIceServer>, /** Unix milliseconds. */ val expiresAt: Long) {
+data class ScreenConnection(val session: String, val iceServers: List<ScreenIceServer>, /** Unix milliseconds. */ val expiresAt: Long,
+ /** False when an older host or helper requires complete SDP. */ val trickle: Boolean = false) {
/** Renew five minutes before the TURN credentials expire, never sooner than a second from now. */
fun renewIn(nowMillis: Long): Long = maxOf(1_000, expiresAt - nowMillis - 300_000)
}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenCandidate.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenCandidate.kt
new file mode 100644
index 00000000..18efb88a
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenCandidate.kt
@@ -0,0 +1,55 @@
+package com.codync.android.core
+
+import java.io.IOException
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.flow.map
+import kotlinx.serialization.SerialName
+import kotlinx.serialization.Serializable
+import kotlinx.serialization.json.decodeFromJsonElement
+import kotlinx.serialization.json.buildJsonObject
+import kotlinx.serialization.json.put
+
+/** Private, bounded ICE signaling. Candidate contents must never be logged. */
+@Serializable
+data class ScreenCandidate(
+ val type: Kind,
+ val candidate: String? = null,
+ val sdpMLineIndex: Int? = null,
+ val sdpMid: String? = null,
+ val message: String? = null,
+) {
+ @Serializable enum class Kind {
+ @SerialName("ready") Ready,
+ @SerialName("candidate") Candidate,
+ @SerialName("complete") Complete,
+ @SerialName("error") Error,
+ }
+
+ fun validate() {
+ if (type != Kind.Candidate) return
+ val sdp = candidate
+ if (sdp == null || !sdp.startsWith("candidate:") || sdp.toByteArray(Charsets.UTF_8).size > 4096 ||
+ '\r' in sdp || '\n' in sdp || sdpMLineIndex == null || sdpMLineIndex !in 0..16 ||
+ (sdpMid?.toByteArray(Charsets.UTF_8)?.size ?: 0) > 256) {
+ throw ScreenProtocolException("Invalid screen ICE candidate.")
+ }
+ }
+
+ companion object {
+ const val maxCount = 128
+ val complete = ScreenCandidate(Kind.Complete)
+ }
+}
+
+class ScreenProtocolException(message: String, cause: Throwable? = null) : IOException(message, cause)
+
+fun screenCandidates(channel: EncryptedChannel, session: String): Flow<ScreenCandidate> =
+ channel.subscribe("screenCandidates", buildJsonObject { put("session", session) }).map { value ->
+ try {
+ val event = WireJson.decodeFromJsonElement<ScreenCandidate>(value)
+ event.validate()
+ event
+ } catch (error: IllegalArgumentException) {
+ throw ScreenProtocolException("Invalid screen signaling event.", error)
+ }
+ }
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenRoute.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenRoute.kt
new file mode 100644
index 00000000..07e068f4
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenRoute.kt
@@ -0,0 +1,23 @@
+package com.codync.android.core
+
+import kotlinx.coroutines.delay
+
+/** Statuses that end screen recovery: access was taken away, or the app must update. */
+val screenTerminalStatuses = listOf(403, 426)
+
+/**
+ * Waits out a route outage (presence can briefly report offline while signaling reconnects) so it doesn't spend an
+ * ICE retry; like the iPhone, only a denial or a required update ends it. Closing the screen cancels the wait.
+ */
+suspend fun waitForScreenRoute(current: () -> LinkState?) {
+ while (true) {
+ when (val link = current()) {
+ is LinkState.Ready -> return
+ is LinkState.Failed -> if (link.status in screenTerminalStatuses) {
+ throw HostError(requireNotNull(link.status), link.message)
+ }
+ else -> Unit
+ }
+ delay(100)
+ }
+}
diff --git a/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenSignaling.kt b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenSignaling.kt
new file mode 100644
index 00000000..d0d824f8
--- /dev/null
+++ b/apps/android/core/src/main/kotlin/com/codync/android/core/ScreenSignaling.kt
@@ -0,0 +1,165 @@
+package com.codync.android.core
+
+import java.io.IOException
+import kotlinx.coroutines.CancellationException
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.Job
+import kotlinx.coroutines.TimeoutCancellationException
+import kotlinx.coroutines.channels.Channel
+import kotlinx.coroutines.flow.Flow
+import kotlinx.coroutines.launch
+import kotlinx.coroutines.withTimeout
+
+/** One consumer per direction; both wait for the answer and preserve native candidate order. */
+class ScreenSignaling(
+ private val scope: CoroutineScope,
+ private val events: Flow<ScreenCandidate>,
+ private val send: suspend (ScreenCandidate) -> Unit,
+ private val apply: suspend (ScreenCandidate) -> Unit,
+ private val isMediaConnected: () -> Boolean,
+ private val onFailure: (Throwable) -> Unit,
+) {
+ private enum class State { Active, Detached, Stopped }
+ private var state = State.Active
+ private val ready = CompletableDeferred<Unit>()
+ private val answer = CompletableDeferred<Unit>()
+ private val local = Channel<ScreenCandidate>(ScreenCandidate.maxCount + 1)
+ private var remoteJob: Job? = null
+ private var localJob: Job? = null
+ private var failure: Throwable? = null
+ private var mediaEstablished = false
+ private var localComplete = false
+ private var remoteComplete = false
+ private var remoteCount = 0
+
+ val isDetached: Boolean get() = state == State.Detached
+ val candidatesComplete: Boolean get() = localComplete && remoteComplete
+
+ suspend fun subscribe() {
+ remoteJob = scope.launch {
+ try {
+ events.collect { receive(it) }
+ transportEnded(IOException("Screen signaling ended before connecting."))
+ } catch (error: CancellationException) {
+ if (state == State.Active) fail(error)
+ } catch (error: Exception) {
+ handle(error)
+ }
+ }
+ try {
+ withTimeout(10_000) { ready.await() }
+ check()
+ } catch (error: TimeoutCancellationException) {
+ val timeout = IOException("The screen signaling subscription didn't become ready.", error)
+ fail(timeout)
+ throw timeout
+ } catch (error: CancellationException) {
+ stop()
+ throw error
+ }
+ }
+
+ /** WebRTC invokes this on its signaling thread; the bounded channel is the only shared state. */
+ fun offerLocal(event: ScreenCandidate) {
+ val result = local.trySend(event)
+ if (result.isFailure && !result.isClosed) {
+ val error = ScreenProtocolException("Too many queued screen ICE candidates.")
+ if (local.close(error)) scope.launch { fail(error) }
+ }
+ }
+
+ fun installedAnswer() {
+ check()
+ if (state != State.Active) return
+ answer.complete(Unit)
+ if (localJob != null) throw ScreenProtocolException("The screen answer was already installed.")
+ localJob = scope.launch {
+ var count = 0
+ try {
+ for (event in local) {
+ event.validate()
+ if (localComplete || event.type !in listOf(ScreenCandidate.Kind.Candidate, ScreenCandidate.Kind.Complete)) {
+ throw ScreenProtocolException("Screen candidates arrived out of order.")
+ }
+ if (event.type == ScreenCandidate.Kind.Candidate && ++count > ScreenCandidate.maxCount) {
+ throw ScreenProtocolException("Too many screen ICE candidates.")
+ }
+ send(event)
+ if (state != State.Active) return@launch
+ if (event.type == ScreenCandidate.Kind.Complete) localComplete = true
+ }
+ } catch (error: CancellationException) {
+ if (state == State.Active) fail(error)
+ } catch (error: Exception) {
+ handle(error)
+ }
+ }
+ }
+
+ fun mediaConnected() { if (state == State.Active) mediaEstablished = true }
+
+ fun check() {
+ failure?.let { throw it }
+ if (state == State.Stopped) throw CancellationException("Screen signaling stopped.")
+ }
+
+ private suspend fun receive(event: ScreenCandidate) {
+ if (state != State.Active) return
+ event.validate()
+ when (event.type) {
+ ScreenCandidate.Kind.Ready -> {
+ if (ready.isCompleted) throw ScreenProtocolException("Duplicate screen subscription readiness.")
+ ready.complete(Unit)
+ }
+ ScreenCandidate.Kind.Error -> throw ScreenProtocolException(event.message ?: "Screen signaling failed.")
+ ScreenCandidate.Kind.Candidate, ScreenCandidate.Kind.Complete -> {
+ if (!ready.isCompleted || remoteComplete) throw ScreenProtocolException("Screen candidates arrived out of order.")
+ if (event.type == ScreenCandidate.Kind.Candidate && ++remoteCount > ScreenCandidate.maxCount) {
+ throw ScreenProtocolException("Too many screen ICE candidates.")
+ }
+ answer.await()
+ if (state != State.Active) return
+ if (event.type == ScreenCandidate.Kind.Candidate) {
+ try { apply(event) }
+ catch (error: CancellationException) { throw error }
+ catch (error: Exception) { throw ScreenProtocolException("Couldn't apply a screen ICE candidate.", error) }
+ } else remoteComplete = true
+ }
+ }
+ }
+
+ private fun handle(error: Exception) {
+ if (error is IOException && error !is ScreenProtocolException && error !is HostError) transportEnded(error)
+ else fail(error)
+ }
+
+ private fun transportEnded(error: Throwable) {
+ if (state != State.Active) return
+ if (mediaEstablished || isMediaConnected() || candidatesComplete) {
+ state = State.Detached
+ finish()
+ } else fail(error)
+ }
+
+ private fun fail(error: Throwable) {
+ if (state != State.Active) return
+ failure = error
+ stop()
+ onFailure(error)
+ }
+
+ fun stop() {
+ if (state == State.Stopped) return
+ state = State.Stopped
+ finish()
+ }
+
+ private fun finish() {
+ ready.completeExceptionally(failure ?: CancellationException("Screen signaling stopped."))
+ answer.cancel()
+ local.cancel()
+ localJob?.cancel()
+ remoteJob?.cancel()
+ }
+}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/ChannelIntegrationTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/ChannelIntegrationTest.kt
index 9c549261..f1f2745f 100644
--- a/apps/android/core/src/test/kotlin/com/codync/android/core/ChannelIntegrationTest.kt
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/ChannelIntegrationTest.kt
@@ -3,6 +3,9 @@ package com.codync.android.core
import java.util.concurrent.CopyOnWriteArrayList
import java.util.concurrent.atomic.AtomicReference
import java.io.IOException
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.async
+import kotlinx.coroutines.cancelAndJoin
import kotlinx.coroutines.currentCoroutineContext
import kotlinx.coroutines.isActive
import kotlinx.coroutines.flow.take
@@ -42,7 +45,7 @@ class ChannelIntegrationTest {
private fun identity(): DeviceIdentity = DeviceIdentity(ByteArray(32) { 3 }, ByteArray(32) { 12 })
private fun enqueueHost(forgeSignature: Boolean = false, holdCalls: Boolean = false, burst: Int = 0, welcomeDelay: Long = 0,
- snapshotHead: Int = 5, reject: String? = null) {
+ snapshotHead: Int = 5, reject: String? = null, screenBurst: Int = 0) {
server.enqueue(MockResponse().withWebSocketUpgrade(object : WebSocketListener() {
private lateinit var opener: FrameOpener
private lateinit var sealer: FrameSealer
@@ -81,7 +84,14 @@ class ChannelIntegrationTest {
return
}
if (holdCalls) return
- if (request.text("sub") == "events") {
+ if (request.text("sub") == "screenCandidates") {
+ send(webSocket, buildJsonObject { put("id", id); put("ev", buildJsonObject { put("type", "ready") }) })
+ repeat(screenBurst) { index -> send(webSocket, buildJsonObject {
+ put("id", id); put("ev", buildJsonObject {
+ put("type", "candidate"); put("candidate", "candidate:$index 1 udp 1 192.0.2.1 1234 typ host"); put("sdpMLineIndex", 0)
+ })
+ }) }
+ } else if (request.text("sub") == "events") {
assertEquals("android", request.getValue("b").jsonObject.text("client"))
if (burst > 0) {
repeat(burst) { index -> send(webSocket, buildJsonObject {
@@ -109,6 +119,33 @@ class ChannelIntegrationTest {
}))
}
+ @Test fun `screen overflow cannot block unrelated RPC responses`() = runBlocking {
+ enqueueHost(screenBurst = 300)
+ val channel = HostConnector.connect(computer(), identity())
+ val waiting = CompletableDeferred<Unit>()
+ val release = CompletableDeferred<Unit>()
+ val stream = async {
+ runCatching {
+ channel.subscribe("screenCandidates", buildJsonObject { put("session", "screen") }).collect { event ->
+ if (event.text("type") == "ready") { waiting.complete(Unit); release.await() }
+ }
+ }
+ }
+ try {
+ withTimeout(5_000) { waiting.await() }
+ val response = withTimeout(5_000) { channel.call("probe") }
+ assertEquals(cid, response.jsonObject.text("computerId"))
+ release.complete(Unit)
+ val result = withTimeout(5_000) { stream.await() }
+ assertTrue(result.exceptionOrNull() is ScreenProtocolException)
+ withTimeout(5_000) { while (cancelled.get() == null) delay(10) }
+ } finally {
+ release.complete(Unit)
+ stream.cancelAndJoin()
+ channel.close()
+ }
+ }
+
@After fun shutdown() {
sockets.forEach { it.close(1000, null) }
server.shutdown()
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenRouteTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenRouteTest.kt
new file mode 100644
index 00000000..dc2fbf40
--- /dev/null
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenRouteTest.kt
@@ -0,0 +1,48 @@
+package com.codync.android.core
+
+import kotlinx.coroutines.ExperimentalCoroutinesApi
+import kotlinx.coroutines.async
+import kotlinx.coroutines.test.advanceTimeBy
+import kotlinx.coroutines.test.runCurrent
+import kotlinx.coroutines.test.runTest
+import org.junit.Assert.*
+import org.junit.Test
+
+@OptIn(ExperimentalCoroutinesApi::class)
+class ScreenRouteTest {
+ @Test fun `transient presence loss waits until a usable route returns`() = runTest {
+ var link: LinkState = LinkState.ComputerOffline(null)
+ val waiting = async { waitForScreenRoute { link } }
+ runCurrent(); advanceTimeBy(1_000); runCurrent()
+ assertFalse(waiting.isCompleted)
+ link = LinkState.Connecting
+ advanceTimeBy(1_000); runCurrent()
+ assertFalse(waiting.isCompleted)
+ link = LinkState.Ready(HostRoute.Relay)
+ advanceTimeBy(100); runCurrent()
+ waiting.await()
+ }
+
+ @Test fun `access denial and required update fail without retrying`() = runTest {
+ for (status in listOf(403, 426)) {
+ val result = runCatching { waitForScreenRoute { LinkState.Failed("Unavailable", status) } }
+ assertEquals(status, (result.exceptionOrNull() as HostError).status)
+ }
+ }
+
+ @Test fun `a long outage keeps waiting, as on the iPhone`() = runTest {
+ var link: LinkState = LinkState.ComputerOffline(null)
+ val waiting = async { waitForScreenRoute { link } }
+ runCurrent(); advanceTimeBy(120_000); runCurrent()
+ assertFalse(waiting.isCompleted)
+ link = LinkState.Ready(HostRoute.Relay)
+ advanceTimeBy(100); runCurrent()
+ waiting.await()
+ }
+
+ @Test fun `closing the screen cancels its route wait`() = runTest {
+ val waiting = async { waitForScreenRoute { null } }
+ runCurrent(); waiting.cancel(); runCurrent()
+ assertTrue(waiting.isCancelled)
+ }
+}
diff --git a/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenSignalingTest.kt b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenSignalingTest.kt
new file mode 100644
index 00000000..808493e6
--- /dev/null
+++ b/apps/android/core/src/test/kotlin/com/codync/android/core/ScreenSignalingTest.kt
@@ -0,0 +1,194 @@
+package com.codync.android.core
+
+import java.io.IOException
+import kotlinx.coroutines.CompletableDeferred
+import kotlinx.coroutines.CoroutineScope
+import kotlinx.coroutines.ExperimentalCoroutinesApi
+import kotlinx.coroutines.async
+import kotlinx.coroutines.channels.Channel
+import kotlinx.coroutines.flow.receiveAsFlow
+import kotlinx.coroutines.test.advanceTimeBy
+import kotlinx.coroutines.test.runCurrent
+import kotlinx.coroutines.test.runTest
+import kotlinx.serialization.json.decodeFromJsonElement
+import org.junit.Assert.*
+import org.junit.Test
+
+@OptIn(ExperimentalCoroutinesApi::class)
+class ScreenSignalingTest {
+ private fun candidate(index: Int) = ScreenCandidate(ScreenCandidate.Kind.Candidate,
+ "candidate:$index 1 udp 1 192.0.2.1 1234 typ host", 0, "video")
+
+ private class Fixture(scope: CoroutineScope) {
+ val incoming = Channel<ScreenCandidate>(Channel.UNLIMITED)
+ val uploads = mutableListOf<ScreenCandidate>()
+ val applied = mutableListOf<ScreenCandidate>()
+ val failures = mutableListOf<Throwable>()
+ var connected = false
+ var uploadGate: CompletableDeferred<Unit>? = null
+ var applyGate: CompletableDeferred<Unit>? = null
+ val signal = ScreenSignaling(scope, incoming.receiveAsFlow(), send = {
+ uploadGate?.await()
+ uploads += it
+ }, apply = {
+ applyGate?.await()
+ applied += it
+ }, isMediaConnected = { connected }, onFailure = { failures += it })
+ fun ready() { incoming.trySend(ScreenCandidate(ScreenCandidate.Kind.Ready)).getOrThrow() }
+ }
+
+ @Test fun `older hosts keep complete SDP unless they explicitly negotiate trickle`() {
+ val old = WireJson.parseToJsonElement("""{"session":"s","iceServers":[],"expiresAt":1}""")
+ assertFalse(WireJson.decodeFromJsonElement<ScreenConnection>(old).trickle)
+ val newer = WireJson.parseToJsonElement("""{"session":"s","iceServers":[],"expiresAt":1,"trickle":true}""")
+ assertTrue(WireJson.decodeFromJsonElement<ScreenConnection>(newer).trickle)
+ }
+
+ @Test fun `candidates are bounded by UTF8 bytes and reject malformed fields`() {
+ candidate(1).validate()
+ ScreenCandidate.complete.validate()
+ val invalid = listOf(candidate(1).copy(candidate = "candidate:x\n"), candidate(1).copy(candidate = "no prefix"),
+ candidate(1).copy(candidate = "candidate:" + "é".repeat(2048)),
+ candidate(1).copy(sdpMid = "é".repeat(129)), candidate(1).copy(sdpMLineIndex = -1),
+ candidate(1).copy(sdpMLineIndex = 17), candidate(1).copy(sdpMLineIndex = null))
+ for (event in invalid) assertThrows(ScreenProtocolException::class.java) { event.validate() }
+ }
+
+ @Test fun `both directions wait for answer then preserve ordering and completion acknowledgments`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.signal.offerLocal(candidate(1)); f.signal.offerLocal(candidate(2)); f.signal.offerLocal(ScreenCandidate.complete)
+ f.incoming.send(candidate(3)); f.incoming.send(candidate(4)); f.incoming.send(ScreenCandidate.complete)
+ runCurrent()
+ assertTrue(f.uploads.isEmpty()); assertTrue(f.applied.isEmpty())
+ val upload = CompletableDeferred<Unit>(); f.uploadGate = upload
+ val apply = CompletableDeferred<Unit>(); f.applyGate = apply
+ f.signal.installedAnswer(); runCurrent()
+ assertTrue(f.uploads.isEmpty()); assertTrue(f.applied.isEmpty()); assertFalse(f.signal.candidatesComplete)
+ apply.complete(Unit); runCurrent()
+ assertEquals(listOf(candidate(3), candidate(4)), f.applied)
+ assertFalse(f.signal.candidatesComplete)
+ upload.complete(Unit); runCurrent()
+ assertEquals(listOf(candidate(1), candidate(2), ScreenCandidate.complete), f.uploads)
+ assertTrue(f.signal.candidatesComplete); assertTrue(f.failures.isEmpty())
+ f.signal.stop()
+ }
+
+ @Test fun `a signaling failure before ready throws its reason`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { runCatching { f.signal.subscribe() } }
+ runCurrent()
+ f.incoming.trySend(ScreenCandidate(ScreenCandidate.Kind.Error, message = "unknown screen session")).getOrThrow()
+ runCurrent()
+ val error = subscription.await().exceptionOrNull()
+ assertTrue(error is ScreenProtocolException)
+ assertEquals("unknown screen session", error?.message)
+ }
+
+ @Test fun `losing signaling before media and completion fails negotiation`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.incoming.close(IOException("socket closed")); runCurrent()
+ assertEquals(1, f.failures.size); assertFalse(f.signal.isDetached)
+ assertThrows(IOException::class.java) { f.signal.check() }
+ }
+
+ @Test fun `native connected state preserves media even before delegate callback arrives`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.connected = true
+ f.incoming.close(IOException("socket closed")); runCurrent()
+ assertTrue(f.signal.isDetached); assertTrue(f.failures.isEmpty()); f.signal.check()
+ f.signal.offerLocal(candidate(1)); runCurrent()
+ assertTrue(f.uploads.isEmpty())
+ }
+
+ @Test fun `completed directions preserve peer when signaling ends before connected callback`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.signal.installedAnswer()
+ f.signal.offerLocal(ScreenCandidate.complete); f.incoming.send(ScreenCandidate.complete); runCurrent()
+ assertTrue(f.signal.candidatesComplete)
+ f.incoming.close(); runCurrent()
+ assertTrue(f.signal.isDetached); assertTrue(f.failures.isEmpty())
+ }
+
+ @Test fun `unacknowledged completion does not detach an unconnected negotiation`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.uploadGate = CompletableDeferred()
+ f.signal.installedAnswer()
+ f.signal.offerLocal(ScreenCandidate.complete); f.incoming.send(ScreenCandidate.complete); runCurrent()
+ assertFalse(f.signal.candidatesComplete)
+ f.incoming.close(IOException("socket closed")); runCurrent()
+ assertEquals(1, f.failures.size); assertFalse(f.signal.isDetached)
+ }
+
+ @Test fun `protocol errors stay fatal after media connects`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ f.connected = true; f.signal.installedAnswer()
+ f.incoming.send(ScreenCandidate.complete); f.incoming.send(candidate(1)); runCurrent()
+ assertTrue(f.failures.single() is ScreenProtocolException); assertFalse(f.signal.isDetached)
+ }
+
+ @Test fun `native candidate rejection stays fatal on a connected peer`() = runTest {
+ val incoming = Channel<ScreenCandidate>(4)
+ val failures = mutableListOf<Throwable>()
+ val signal = ScreenSignaling(backgroundScope, incoming.receiveAsFlow(), {},
+ { throw IOException("native failure") }, { true }, { failures += it })
+ val subscription = async { signal.subscribe() }
+ runCurrent(); incoming.send(ScreenCandidate(ScreenCandidate.Kind.Ready)); runCurrent(); subscription.await()
+ signal.installedAnswer(); incoming.send(candidate(1)); runCurrent()
+ assertTrue(failures.single() is ScreenProtocolException); assertFalse(signal.isDetached)
+ }
+
+ @Test fun `host rejection stays fatal on a connected peer`() = runTest {
+ val incoming = Channel<ScreenCandidate>(4)
+ val failures = mutableListOf<Throwable>()
+ val signal = ScreenSignaling(backgroundScope, incoming.receiveAsFlow(),
+ { throw HostError(500, "Helper rejected the candidate.") }, {}, { true }, { failures += it })
+ val subscription = async { signal.subscribe() }
+ runCurrent(); incoming.send(ScreenCandidate(ScreenCandidate.Kind.Ready)); runCurrent(); subscription.await()
+ signal.installedAnswer(); signal.offerLocal(candidate(1)); runCurrent()
+ assertTrue(failures.single() is HostError); assertFalse(signal.isDetached)
+ }
+
+ @Test fun `local overflow fails before offer can silently lose candidates`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await()
+ repeat(10_000) { f.signal.offerLocal(candidate(it)) }; runCurrent()
+ assertTrue(f.failures.single() is ScreenProtocolException); assertTrue(f.uploads.isEmpty())
+ }
+
+ @Test fun `remote count limit rejects the candidate beyond the bound`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.ready(); runCurrent(); subscription.await(); f.signal.installedAnswer()
+ repeat(ScreenCandidate.maxCount + 1) { f.incoming.send(candidate(it)) }; runCurrent()
+ assertEquals(ScreenCandidate.maxCount, f.applied.size)
+ assertTrue(f.failures.single() is ScreenProtocolException)
+ }
+
+ @Test fun `stopping an unopened subscription unblocks readiness without failure`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { f.signal.subscribe() }
+ runCurrent(); f.signal.stop(); runCurrent()
+ assertTrue(subscription.isCancelled); assertTrue(f.failures.isEmpty())
+ }
+
+ @Test fun `subscription readiness has a finite deadline`() = runTest {
+ val f = Fixture(backgroundScope)
+ val subscription = async { runCatching { f.signal.subscribe() } }
+ runCurrent(); advanceTimeBy(10_000); runCurrent()
+ val result = subscription.await()
+ assertTrue(result.isFailure); assertEquals(1, f.failures.size)
+ }
+}
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Calls.swift b/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Calls.swift
index 89fafd81..35a29ca1 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Calls.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Calls.swift
@@ -22,13 +22,36 @@ extension ChannelTransport {
case .ready, .connecting:
guard ContinuousClock.now < deadline else { throw HostError.unreachable }
await stateChange(until: deadline)
- case let .hostOffline(lastSeen): throw HostError.computerOffline(lastSeen: lastSeen)
- case let .unauthorized(message): throw HostError.unauthorized(message)
- case .failed: throw needsUpgrade ? HostError.upgradeRequired : HostError.unreachable
+ case .hostOffline, .unauthorized, .failed: throw linkError(state)
}
}
}
+ /// Waits out outages and an offline computer; throws only once the channel can't come back.
+ public func waitForRoute() async throws {
+ for await state in states() {
+ try Task.checkCancellation()
+ switch state {
+ case .ready: return
+ case .connecting, .hostOffline: continue
+ case .failed where !isStopped: continue
+ case .unauthorized, .failed: throw linkError(state)
+ }
+ }
+ try Task.checkCancellation()
+ throw HostError.unreachable
+ }
+
+ /// What a link state the caller can't use means to it.
+ private func linkError(_ state: LinkState) -> HostError {
+ switch state {
+ case let .hostOffline(lastSeen): .computerOffline(lastSeen: lastSeen)
+ case let .unauthorized(message): .unauthorized(message)
+ case .failed where needsUpgrade: .upgradeRequired
+ case .connecting, .ready, .failed: .unreachable
+ }
+ }
+
func perform(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
try await ready(within: timeout)
let b: Any = body.isEmpty ? NSNull() : try JSONSerialization.jsonObject(with: body, options: .fragmentsAllowed)
@@ -57,7 +80,7 @@ extension ChannelTransport {
private func abandon(_ id: UInt32, error: Error) {
guard let c = calls.removeValue(forKey: id) else { return }
c.resume(throwing: error)
- try? sendInner(JSONSerialization.data(withJSONObject: ["id": id, "cancel": true]))
+ try? sendCancellation(id)
}
private func sendInner(_ message: Data) throws {
@@ -75,7 +98,7 @@ extension ChannelTransport {
}
public nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> {
- AsyncThrowingStream { continuation in
+ AsyncThrowingStream(bufferingPolicy: request.bufferLimit.map { .bufferingOldest($0) } ?? .unbounded) { continuation in
let key = UUID()
let task = Task { await self.openStream(request, key: key, continuation) }
continuation.onTermination = { _ in
@@ -93,6 +116,7 @@ extension ChannelTransport {
let id = nextId()
let sub: [String: Any] = switch request {
case let .events(since, client): ["id": id, "sub": "events", "b": ["since": since, "client": client]]
+ case let .screenCandidates(session): ["id": id, "sub": "screenCandidates", "b": ["session": session]]
case let .term(term): ["id": id, "sub": "term", "b": ["term": term]]
}
streams[id] = continuation
@@ -103,9 +127,20 @@ extension ChannelTransport {
}
}
+ /// Tells the host to stop a call or stream.
+ func sendCancellation(_ id: UInt32) throws {
+ try sendInner(JSONSerialization.data(withJSONObject: ["id": id, "cancel": true]))
+ }
+
/// The consumer went away: stop the host's stream.
private func closeStream(_ key: UUID) {
guard let id = streamIds.removeValue(forKey: key), streams.removeValue(forKey: id) != nil else { return }
- try? sendInner(JSONSerialization.data(withJSONObject: ["id": id, "cancel": true]))
+ try? sendCancellation(id)
+ }
+
+ /// The host ended a stream (or it overflowed): finish it and forget its id.
+ func finishStream(_ id: UInt32, throwing error: Error? = nil) {
+ streamIds = streamIds.filter { $0.value != id }
+ streams.removeValue(forKey: id)?.finish(throwing: error)
}
}
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Incoming.swift b/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Incoming.swift
index daf38968..261d0dd1 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Incoming.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/ChannelTransport+Incoming.swift
@@ -153,18 +153,21 @@ extension ChannelTransport {
let rawId = obj["id"] as? NSNumber else { return }
let id = rawId.uint32Value
if let ev = obj["ev"] {
- if let data = try? JSONSerialization.data(withJSONObject: ev, options: .fragmentsAllowed) { streams[id]?.yield(data) }
+ // Only a bounded stream (`HostStreamRequest.bufferLimit`) drops: screen candidates.
+ if let data = try? JSONSerialization.data(withJSONObject: ev, options: .fragmentsAllowed),
+ case .dropped = streams[id]?.yield(data) {
+ finishStream(id, throwing: ScreenError.candidateOverflow)
+ do { try sendCancellation(id) } catch { link?.socket.close(code: 1000) }
+ }
} else if obj["end"] as? Bool == true {
- streams.removeValue(forKey: id)?.finish()
- streamIds = streamIds.filter { $0.value != id }
+ finishStream(id)
} else if let err = obj["err"] as? [String: Any] {
let status = (err["status"] as? NSNumber)?.intValue ?? 500
let error = HostError.http(status, err["message"] as? String ?? "Host error \(status)")
if let call = calls.removeValue(forKey: id) {
call.resume(throwing: error)
- } else if let stream = streams.removeValue(forKey: id) {
- stream.finish(throwing: error)
- streamIds = streamIds.filter { $0.value != id }
+ } else {
+ finishStream(id, throwing: error)
}
} else if obj.keys.contains("ok") {
let data = (try? JSONSerialization.data(withJSONObject: obj["ok"] ?? NSNull(), options: .fragmentsAllowed)) ?? Data("null".utf8)
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/HostClient+ScreenCandidates.swift b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient+ScreenCandidates.swift
new file mode 100644
index 00000000..0f51e7f0
--- /dev/null
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient+ScreenCandidates.swift
@@ -0,0 +1,28 @@
+import Foundation
+
+public extension HostClient {
+ func screenCandidates(session: String) -> AsyncThrowingStream<ScreenCandidate, Error> {
+ let source = transport.stream(.screenCandidates(session))
+ return AsyncThrowingStream(bufferingPolicy: .bufferingOldest(ScreenCandidate.maxCount + 2)) { c in
+ let task = Task {
+ do {
+ for try await data in source {
+ let event = try JSONDecoder().decode(ScreenCandidate.self, from: data)
+ try event.validate()
+ if case .dropped = c.yield(event) {
+ throw ScreenError.candidateOverflow
+ }
+ }
+ c.finish()
+ } catch { c.finish(throwing: error) }
+ }
+ c.onTermination = { _ in task.cancel() }
+ }
+ }
+
+ func screenCandidate(session: String, candidate: ScreenCandidate) async throws {
+ try candidate.validate()
+ struct Body: Encodable { var session: String; var candidate: ScreenCandidate }
+ let _: Empty = try await call("screenCandidate", Body(session: session, candidate: candidate))
+ }
+}
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
index f05c0ce4..949f3970 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/HostClient.swift
@@ -367,11 +367,11 @@ public extension HostClient {
func screenPrepare() async throws -> ScreenConnection {
let relay = if let channel = transport as? ChannelTransport { try await channel.screenRelayRequired() } else { false }
- return try await call("screenPrepare", ["relay": relay])
+ return try await call("screenPrepare", ["relay": relay, "trickle": true])
}
- /// Sends a WebRTC offer (with all its ICE candidates); returns the session id and the answer.
- /// Passing `session` restarts ICE on an existing session.
+ /// Sends the offer using the prepared session’s negotiated ICE mode.
+ /// Trickle negotiations use a fresh session for every connection attempt.
func screenOffer(sdp: String, session: String?, display: UInt32?) async throws -> ScreenAnswer {
struct Body: Encodable { var sdp: String; var session: String?; var display: UInt32? }
return try await call("screenOffer", Body(sdp: sdp, session: session, display: display), timeout: 30)
diff --git a/apps/ios/Kit/Sources/CodyncKit/Client/HostTransport.swift b/apps/ios/Kit/Sources/CodyncKit/Client/HostTransport.swift
index 19d88f4c..4a852880 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Client/HostTransport.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Client/HostTransport.swift
@@ -8,6 +8,12 @@ public enum HostRoute: Sendable, Equatable {
public enum HostStreamRequest: Sendable {
case events(since: Int64, client: String)
case term(String)
+ case screenCandidates(String)
+
+ /// Events a slow consumer may leave unread before the stream ends; `nil` keeps every one.
+ var bufferLimit: Int? {
+ if case .screenCandidates = self { ScreenCandidate.maxCount + 2 } else { nil }
+ }
}
public enum LinkState: Sendable, Equatable {
@@ -28,6 +34,23 @@ public protocol HostTransport: Sendable {
func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error>
/// A new stream per call: the current state first, then every change.
func states() -> AsyncStream<LinkState>
+ /// Waits however long an outage lasts until the host can be reached again; throws once it can't be.
+ func waitForRoute() async throws
+}
+
+public extension HostTransport {
+ func waitForRoute() async throws {
+ for await state in states() {
+ try Task.checkCancellation()
+ switch state {
+ case .ready: return
+ case let .unauthorized(message): throw HostError.unauthorized(message)
+ case .connecting, .hostOffline, .failed: continue
+ }
+ }
+ try Task.checkCancellation()
+ throw HostError.unreachable
+ }
}
/// A transport to a remote computer: learns its addresses and keys, and holds messages
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/CandidateProgress.swift b/apps/ios/Kit/Sources/CodyncKit/Models/CandidateProgress.swift
new file mode 100644
index 00000000..17576df5
--- /dev/null
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/CandidateProgress.swift
@@ -0,0 +1,20 @@
+import Foundation
+
+/// One direction of a screen candidate exchange, under the host's rules:
+/// at most `ScreenCandidate.maxCount` candidates, then one completion, then nothing.
+public struct CandidateProgress: Sendable {
+ public private(set) var count = 0
+ public private(set) var isComplete = false
+
+ public init() {}
+
+ public mutating func accept(_ event: ScreenCandidate) throws {
+ guard !isComplete else { throw ScreenError.candidatesAlreadyComplete }
+ if event.type == .complete {
+ isComplete = true
+ } else {
+ guard count < ScreenCandidate.maxCount else { throw ScreenError.tooManyCandidates }
+ count += 1
+ }
+ }
+}
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/ScreenCandidate.swift b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenCandidate.swift
new file mode 100644
index 00000000..fd160e60
--- /dev/null
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenCandidate.swift
@@ -0,0 +1,43 @@
+import Foundation
+
+/// Private screen signaling events; credentials and candidates stay in memory.
+public struct ScreenCandidate: Codable, Sendable, Equatable {
+ public static let maxCount = 128
+ private static let maxBytes = 4096
+ private static let maxMediaIndex: Int32 = 16
+ private static let maxMidBytes = 256
+ public enum Kind: String, Codable, Sendable { case ready, candidate, complete, error }
+ public let type: Kind
+ public let candidate: String?
+ public let sdpMLineIndex: Int32?
+ public let sdpMid: String?
+ public let message: String?
+
+ public init(candidate: String, sdpMLineIndex: Int32, sdpMid: String?) {
+ type = .candidate
+ self.candidate = candidate
+ self.sdpMLineIndex = sdpMLineIndex
+ self.sdpMid = sdpMid
+ message = nil
+ }
+
+ private init(_ type: Kind) {
+ self.type = type
+ candidate = nil
+ sdpMLineIndex = nil
+ sdpMid = nil
+ message = nil
+ }
+
+ public static let complete = ScreenCandidate(.complete)
+
+ public func validate() throws {
+ if type == .candidate {
+ guard let candidate, candidate.hasPrefix("candidate:"), candidate.utf8.count <= Self.maxBytes,
+ !candidate.contains("\r"), !candidate.contains("\n"),
+ let sdpMLineIndex, (0...Self.maxMediaIndex).contains(sdpMLineIndex),
+ (sdpMid?.utf8.count ?? 0) <= Self.maxMidBytes
+ else { throw ScreenError.invalidCandidate }
+ }
+ }
+}
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/ScreenConnection.swift b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenConnection.swift
index 48b288a3..63fa6d13 100644
--- a/apps/ios/Kit/Sources/CodyncKit/Models/ScreenConnection.swift
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenConnection.swift
@@ -6,6 +6,17 @@ public struct ScreenConnection: Codable, Sendable {
public var iceServers: [ScreenIceServer]
/// Unix milliseconds, matching the host/cloud API.
public var expiresAt: Int64
+ public var trickle: Bool
+
+ private enum CodingKeys: String, CodingKey { case session, iceServers, expiresAt, trickle }
+
+ public init(from decoder: Decoder) throws {
+ let c = try decoder.container(keyedBy: CodingKeys.self)
+ session = try c.decode(String.self, forKey: .session)
+ iceServers = try c.decode([ScreenIceServer].self, forKey: .iceServers)
+ expiresAt = try c.decode(Int64.self, forKey: .expiresAt)
+ trickle = try c.decodeIfPresent(Bool.self, forKey: .trickle) ?? false
+ }
}
public struct ScreenIceServer: Codable, Sendable {
diff --git a/apps/ios/Kit/Sources/CodyncKit/Models/ScreenError.swift b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenError.swift
new file mode 100644
index 00000000..cd2f8a2f
--- /dev/null
+++ b/apps/ios/Kit/Sources/CodyncKit/Models/ScreenError.swift
@@ -0,0 +1,26 @@
+import Foundation
+
+/// Local negotiation and candidate validation failures, separate from host HTTP responses.
+public enum ScreenError: LocalizedError, Sendable, Equatable {
+ case invalidCandidate
+ case candidateOverflow
+ case candidatesAlreadyComplete
+ case tooManyCandidates
+ case duplicateSubscription
+ case candidatesOutOfOrder
+ case signalingFailed(String)
+ case connectionFailed
+
+ public var errorDescription: String? {
+ switch self {
+ case .invalidCandidate: "Invalid screen candidate."
+ case .candidateOverflow: "Screen candidate queue overflow."
+ case .candidatesAlreadyComplete: "Screen candidates already complete."
+ case .tooManyCandidates: "Too many screen candidates."
+ case .duplicateSubscription: "Duplicate screen signaling subscription."
+ case .candidatesOutOfOrder: "Screen candidates out of order."
+ case let .signalingFailed(message): message
+ case .connectionFailed: "Couldn't start the video connection."
+ }
+ }
+}
diff --git a/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSession.swift b/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSession.swift
index 80096b8a..5601ee1f 100644
--- a/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSession.swift
+++ b/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSession.swift
@@ -9,7 +9,7 @@ private let log = Logger(subsystem: "com.pokai.Codync", category: "Screen")
/// One live view of the computer's screen: a WebRTC peer that receives the
/// display as H.264 and sends input over two data channels (`input` reliable,
/// `input-fast` unordered for pointer moves). Signaling goes through the host
-/// API, non-trickle (offer and answer carry every candidate).
+/// API, with full trickle ICE when the host and helper support it.
@MainActor
@Observable
public final class ScreenSession {
@@ -36,12 +36,15 @@ public final class ScreenSession {
private var reliable: RTCDataChannel?
private var fast: RTCDataChannel?
private var events: PeerEvents?
+ private var signaling: ScreenSignaling?
private var gathered: CheckedContinuation<Void, Never>?
private var recoverTask: Task<Void, Never>?
private var gatherTask: Task<Void, Never>?
private var renewTask: Task<Void, Never>?
private var connectionTimer: Task<Void, Never>?
private var generation = 0
+ private var recoveryAttempts = 0
+ private static let maxRecoveryAttempts = 3
static let factory: RTCPeerConnectionFactory = {
RTCInitializeSSL()
@@ -56,13 +59,13 @@ public final class ScreenSession {
// MARK: connection
public func start() async {
+ recoveryAttempts = 0
phase = .connecting
do {
try await connect()
} catch {
- guard phase != .closed else { return }
- close()
- phase = .failed(error.localizedDescription)
+ guard phase == .connecting else { return }
+ fail(error.localizedDescription)
}
}
@@ -89,17 +92,37 @@ public final class ScreenSession {
config.tcpCandidatePolicy = .enabled
config.continualGatheringPolicy = .gatherOnce
config.bundlePolicy = .maxBundle
- let events = PeerEvents(owner: self, generation: current)
+ let candidatePair = connection.trickle ? AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(ScreenCandidate.maxCount + 1)) : nil
+ let events = PeerEvents(owner: self, generation: current, candidates: candidatePair?.continuation)
guard let pc = Self.factory.peerConnection(with: config, constraints: Self.noConstraints, delegate: events) else {
- throw HostError.http(0, "Couldn't start the video connection.")
+ throw ScreenError.connectionFailed
}
self.events = events
self.pc = pc
+ if let candidatePair {
+ let signaling = ScreenSignaling(client: client, session: connection.session, pc: pc, localPair: candidatePair) { [weak self] error in
+ guard let self, self.generation == current, self.phase != .closed else { return }
+ log.info("screen signaling failed: \(error.localizedDescription)")
+ self.lastError = error.localizedDescription
+ // Retire this peer now, before fetching a replacement. Its queued callbacks
+ // must not restore .live or keep sending input after signaling has failed.
+ self.teardown()
+ self.phase = .reconnecting
+ // An ongoing retry observes the failure when negotiation resumes.
+ // Keep its attempt counter instead of replacing it with a new retry loop.
+ if self.recoverTask == nil { self.scheduleRecovery(after: .zero) }
+ }
+ self.signaling = signaling
+ try await signaling.subscribe()
+ // Subscription failure can retire this peer before the ready waiter resumes.
+ try ensureCurrent(pc)
+ try signaling.check()
+ }
let initVideo = RTCRtpTransceiverInit()
initVideo.direction = .recvOnly
guard let video = pc.addTransceiver(of: .video, init: initVideo) else {
- throw HostError.http(0, "Couldn't start the video connection.")
+ throw ScreenError.connectionFailed
}
// Hardware H.264 first (High, then Constrained Baseline); the rest only as a fallback.
let codecs = Self.factory.rtpReceiverCapabilities(forKind: kRTCMediaStreamTrackKindVideo).codecs
@@ -117,6 +140,7 @@ public final class ScreenSession {
try await negotiate(pc)
guard generation == current, phase != .closed else { throw CancellationError() }
+ try signaling?.check()
// Refresh both endpoints before TURN credentials expire, including an idle screen.
let seconds = max(1, Double(connection.expiresAt) / 1000 - Date().timeIntervalSince1970 - 300)
renewTask = Task { [weak self] in
@@ -128,24 +152,33 @@ public final class ScreenSession {
connectionTimer = Task { [weak self] in
do { try await Task.sleep(for: .seconds(30)) } catch { return }
guard let self, self.generation == current, self.phase != .live else { return }
- self.close()
- self.phase = .failed("Couldn't connect to the remote screen. Check the computer's connection and try again.")
+ self.fail("Couldn't connect to the remote screen. Check the computer's connection and try again.")
}
}
/// Offer → host → answer with the reserved session’s ICE configuration.
private func negotiate(_ pc: RTCPeerConnection) async throws {
+ let signaling = signaling
let offer = try await pc.offer(for: Self.noConstraints)
+ try ensureCurrent(pc)
try await pc.setLocalDescription(offer)
- await waitForCandidates(pc)
+ try ensureCurrent(pc)
+ if signaling == nil { await waitForCandidates(pc) }
try Task.checkCancellation()
- guard self.pc === pc, phase != .closed else { throw CancellationError() }
- let sdp = pc.localDescription?.sdp ?? offer.sdp
+ try ensureCurrent(pc)
+ let sdp = signaling == nil ? (pc.localDescription?.sdp ?? offer.sdp) : offer.sdp
let answer = try await client.screenOffer(sdp: sdp, session: session, display: display?.id)
try Task.checkCancellation()
- guard self.pc === pc, phase != .closed else { throw CancellationError() }
+ try ensureCurrent(pc)
session = answer.session
try await pc.setRemoteDescription(RTCSessionDescription(type: .answer, sdp: answer.sdp))
+ try ensureCurrent(pc)
+ try await signaling?.installedAnswer()
+ }
+
+ /// A newer peer or a close retired `pc` while we waited.
+ private func ensureCurrent(_ pc: RTCPeerConnection) throws {
+ guard self.pc === pc, phase != .closed else { throw CancellationError() }
}
private func waitForCandidates(_ pc: RTCPeerConnection) async {
@@ -172,6 +205,8 @@ public final class ScreenSession {
guard generation == self.generation, phase != .closed else { return }
switch state {
case .connected, .completed:
+ signaling?.mediaConnected()
+ recoveryAttempts = 0
connectionTimer?.cancel()
phase = .live
lastError = nil
@@ -190,30 +225,50 @@ public final class ScreenSession {
private func scheduleRecovery(after delay: Duration) {
recoverTask?.cancel()
recoverTask = Task { [weak self] in
- var attempt = 0
try? await Task.sleep(for: delay)
while let self, !Task.isCancelled, self.phase == .reconnecting {
+ guard self.recoveryAttempts < Self.maxRecoveryAttempts else {
+ self.fail(self.lastError ?? HostError.unreachable.localizedDescription)
+ return
+ }
+ // A route outage is not a failed WebRTC attempt: wait for the host channel to recover.
+ do { try await self.client.transport.waitForRoute() }
+ catch {
+ guard !Task.isCancelled, self.phase != .closed else { return }
+ self.fail(error.localizedDescription)
+ return
+ }
+ guard !Task.isCancelled, self.phase == .reconnecting else { break }
+ self.recoveryAttempts += 1
do {
// Re-evaluate the signaling route after Wi-Fi/cellular changes and fetch fresh ICE credentials.
try await self.connect()
+ self.recoverTask = nil
return
} catch {
guard !Task.isCancelled, self.phase != .closed else { return }
log.info("screen reconnect failed: \(error.localizedDescription)")
- attempt += 1
- if attempt >= 3 {
- self.close()
- self.phase = .failed(error.localizedDescription)
+ // A failed signaling attempt cancels the rest of it; keep the reason its failure recorded.
+ if !(error is CancellationError) { self.lastError = error.localizedDescription }
+ if self.recoveryAttempts >= Self.maxRecoveryAttempts {
+ self.fail(self.lastError ?? error.localizedDescription)
return
}
- try? await Task.sleep(for: .seconds(min(Double(attempt) * 2, 10)))
+ try? await Task.sleep(for: .seconds(min(Double(self.recoveryAttempts) * 2, 10)))
}
}
+ if !Task.isCancelled { self?.recoverTask = nil }
}
}
+ private func fail(_ message: String) {
+ close()
+ phase = .failed(message)
+ }
+
public func close() {
recoverTask?.cancel()
+ recoverTask = nil
phase = .closed
if let session {
let client = client
@@ -225,6 +280,8 @@ public final class ScreenSession {
private func teardown() {
generation += 1
+ signaling?.stop()
+ signaling = nil
renewTask?.cancel()
connectionTimer?.cancel()
finishGathering()
@@ -270,7 +327,8 @@ public final class ScreenSession {
return remoteClipboard
}
- fileprivate func received(_ data: Data) {
+ fileprivate func received(_ data: Data, generation: Int) {
+ guard generation == self.generation, phase != .closed else { return }
guard let msg = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return }
switch msg["type"] as? String {
case "clipboard": remoteClipboard = msg["text"] as? String
@@ -292,17 +350,28 @@ public final class ScreenSession {
private final class PeerEvents: NSObject, RTCPeerConnectionDelegate, RTCDataChannelDelegate, Sendable {
private let owner: WeakBox
private let generation: Int
+ let candidates: AsyncThrowingStream<ScreenCandidate, Error>.Continuation?
- @MainActor init(owner: ScreenSession, generation: Int) {
+ @MainActor init(owner: ScreenSession, generation: Int, candidates: AsyncThrowingStream<ScreenCandidate, Error>.Continuation? = nil) {
self.owner = WeakBox(owner)
self.generation = generation
+ self.candidates = candidates
}
func peerConnection(_ peerConnection: RTCPeerConnection, didChange stateChanged: RTCSignalingState) {}
func peerConnection(_ peerConnection: RTCPeerConnection, didAdd stream: RTCMediaStream) {}
func peerConnection(_ peerConnection: RTCPeerConnection, didRemove stream: RTCMediaStream) {}
func peerConnectionShouldNegotiate(_ peerConnection: RTCPeerConnection) {}
- func peerConnection(_ peerConnection: RTCPeerConnection, didGenerate candidate: RTCIceCandidate) {}
+ func peerConnection(_ peerConnection: RTCPeerConnection, didGenerate candidate: RTCIceCandidate) {
+ emit(ScreenCandidate(candidate: candidate.sdp, sdpMLineIndex: candidate.sdpMLineIndex, sdpMid: candidate.sdpMid))
+ }
+
+ private func emit(_ event: ScreenCandidate) {
+ guard let candidates else { return }
+ if case .dropped = candidates.yield(event) {
+ candidates.finish(throwing: ScreenError.candidateOverflow)
+ }
+ }
func peerConnection(_ peerConnection: RTCPeerConnection, didRemove candidates: [RTCIceCandidate]) {}
func peerConnection(_ peerConnection: RTCPeerConnection, didOpen dataChannel: RTCDataChannel) {}
func dataChannelDidChangeState(_ dataChannel: RTCDataChannel) {}
@@ -315,6 +384,7 @@ private final class PeerEvents: NSObject, RTCPeerConnectionDelegate, RTCDataChan
func peerConnection(_ peerConnection: RTCPeerConnection, didChange newState: RTCIceGatheringState) {
guard newState == .complete else { return }
+ emit(.complete)
let owner = owner
let generation = generation
Task { @MainActor in owner.value?.finishGathering(generation: generation) }
@@ -323,7 +393,8 @@ private final class PeerEvents: NSObject, RTCPeerConnectionDelegate, RTCDataChan
func dataChannel(_ dataChannel: RTCDataChannel, didReceiveMessageWith buffer: RTCDataBuffer) {
let data = buffer.data
let owner = owner
- Task { @MainActor in owner.value?.received(data) }
+ let generation = generation
+ Task { @MainActor in owner.value?.received(data, generation: generation) }
}
}
diff --git a/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSignaling.swift b/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSignaling.swift
new file mode 100644
index 00000000..c2f93423
--- /dev/null
+++ b/apps/ios/Kit/Sources/CodyncUI/Screen/ScreenSignaling.swift
@@ -0,0 +1,162 @@
+import CodyncKit
+import Foundation
+@preconcurrency import WebRTC
+
+/// Two bounded FIFOs. Each direction has one consumer; neither applies candidates before the answer.
+@MainActor
+final class ScreenSignaling {
+ private enum State { case active, detached, stopped }
+ private let client: HostClient
+ private let session: String
+ private let applyCandidate: @MainActor (ScreenCandidate) async throws -> Void
+ private let isMediaConnected: @MainActor () -> Bool
+ private let onFailure: @MainActor (Error) -> Void
+ nonisolated let local: AsyncThrowingStream<ScreenCandidate, Error>.Continuation
+ private var localTask: Task<Void, Never>?
+ private var remoteTask: Task<Void, Never>?
+ private var deadline: Task<Void, Never>?
+ private var ready: CheckedContinuation<Void, Error>?
+ private var isReady = false
+ private var answered = false
+ private var state = State.active
+ private var failure: Error?
+ private var mediaEstablished = false
+ private var localComplete = false
+ private var remote = CandidateProgress()
+ private var answerWaiter: CheckedContinuation<Void, Never>?
+ private let localStream: AsyncThrowingStream<ScreenCandidate, Error>
+
+ var isDetached: Bool { state == .detached }
+ var candidatesComplete: Bool { localComplete && remote.isComplete }
+
+ typealias LocalPair = (stream: AsyncThrowingStream<ScreenCandidate, Error>, continuation: AsyncThrowingStream<ScreenCandidate, Error>.Continuation)
+
+ convenience init(client: HostClient, session: String, pc: RTCPeerConnection, localPair: LocalPair, onFailure: @escaping @MainActor (Error) -> Void) {
+ self.init(client: client, session: session, localPair: localPair, applyCandidate: { event in
+ guard let sdp = event.candidate, let index = event.sdpMLineIndex else { throw ScreenError.invalidCandidate }
+ try await pc.add(RTCIceCandidate(sdp: sdp, sdpMLineIndex: index, sdpMid: event.sdpMid))
+ }, isMediaConnected: { pc.iceConnectionState == .connected || pc.iceConnectionState == .completed }, onFailure: onFailure)
+ }
+
+ init(client: HostClient, session: String, localPair: LocalPair, applyCandidate: @escaping @MainActor (ScreenCandidate) async throws -> Void, isMediaConnected: @escaping @MainActor () -> Bool = { false }, onFailure: @escaping @MainActor (Error) -> Void) {
+ self.client = client
+ self.session = session
+ self.applyCandidate = applyCandidate
+ self.isMediaConnected = isMediaConnected
+ self.onFailure = onFailure
+ local = localPair.continuation
+ localStream = localPair.stream
+ }
+
+ func subscribe() async throws {
+ remoteTask = Task { [weak self, client, session] in
+ do {
+ for try await event in client.screenCandidates(session: session) {
+ guard let self, self.state == .active else { return }
+ try await self.receive(event)
+ }
+ self?.transportEnded(HostError.unreachable)
+ } catch { self?.handle(error) }
+ }
+ deadline = Task { [weak self] in
+ do { try await Task.sleep(for: .seconds(10)) } catch { return }
+ self?.fail(HostError.unreachable)
+ }
+ try await withTaskCancellationHandler {
+ try await withCheckedThrowingContinuation { c in
+ if isReady { c.resume() }
+ else if state == .stopped { c.resume(throwing: failure ?? CancellationError()) }
+ else { ready = c }
+ }
+ } onCancel: { Task { @MainActor in self.stop() } }
+ deadline?.cancel()
+ }
+
+ func check() throws {
+ if let failure { throw failure }
+ guard state != .stopped else { throw CancellationError() }
+ }
+
+ func mediaConnected() {
+ guard state == .active else { return }
+ mediaEstablished = true
+ }
+
+ func installedAnswer() async throws {
+ try check()
+ guard state == .active else { return }
+ answered = true
+ answerWaiter?.resume()
+ answerWaiter = nil
+ localTask = Task { [weak self, localStream, client, session] in
+ do {
+ var local = CandidateProgress()
+ for try await event in localStream {
+ try Task.checkCancellation()
+ try local.accept(event)
+ try await client.screenCandidate(session: session, candidate: event)
+ try Task.checkCancellation()
+ guard let self, self.state == .active else { return }
+ if event.type == .complete { self.localComplete = true }
+ }
+ } catch { self?.handle(error) }
+ }
+ }
+
+ private func receive(_ event: ScreenCandidate) async throws {
+ switch event.type {
+ case .ready:
+ guard !isReady else { throw ScreenError.duplicateSubscription }
+ isReady = true
+ ready?.resume()
+ ready = nil
+ case .error: throw ScreenError.signalingFailed(event.message ?? "Screen signaling failed.")
+ case .candidate, .complete:
+ guard isReady, !remote.isComplete else { throw ScreenError.candidatesOutOfOrder }
+ try remote.accept(event)
+ // Suspend this consumer: its bounded transport stream buffers further events.
+ if !answered { await withCheckedContinuation { answerWaiter = $0 } }
+ try Task.checkCancellation()
+ guard state == .active else { throw CancellationError() }
+ if event.type == .candidate { try await applyCandidate(event) }
+ // Native Apple WebRTC exposes no usable empty/null end-of-candidates operation.
+ }
+ }
+
+ private func fail(_ error: Error) {
+ guard state == .active else { return }
+ failure = error
+ stop()
+ onFailure(error)
+ }
+
+ private func handle(_ error: Error) {
+ if let error = error as? HostError, error.isTransient { transportEnded(error) }
+ else { fail(error) }
+ }
+
+ private func transportEnded(_ error: Error) {
+ guard state == .active else { return }
+ // The native state may lead its queued main-actor delegate callback during a route change.
+ guard mediaEstablished || isMediaConnected() || candidatesComplete else { fail(error); return }
+ state = .detached
+ finishTasks()
+ }
+
+ func stop() {
+ guard state != .stopped else { return }
+ state = .stopped
+ finishTasks()
+ }
+
+ private func finishTasks() {
+ deadline?.cancel()
+ ready?.resume(throwing: failure ?? CancellationError())
+ ready = nil
+ answerWaiter?.resume()
+ answerWaiter = nil
+ local.finish()
+ localTask?.cancel()
+ remoteTask?.cancel()
+ }
+}
diff --git a/apps/ios/Kit/Tests/CodyncKitTests/ChannelTransportTests.swift b/apps/ios/Kit/Tests/CodyncKitTests/ChannelTransportTests.swift
index 59176426..db2d1e80 100644
--- a/apps/ios/Kit/Tests/CodyncKitTests/ChannelTransportTests.swift
+++ b/apps/ios/Kit/Tests/CodyncKitTests/ChannelTransportTests.swift
@@ -218,6 +218,36 @@ private func waitFor(_ states: AsyncStream<LinkState>, _ match: (LinkState) -> B
await t.shutdown()
}
+@Test(.timeLimit(.minutes(1))) func screenCandidateStreamIsBoundedAndCancelledOnOverflow() async throws {
+ let (dial, sockets) = fakeDialer()
+ var computer = v.computer
+ computer.urls = ["http://127.0.0.1:1"]
+ let t = ChannelTransport(computer: computer, identity: v.identity, pairingCode: nil, dial: dial, watchesNetwork: false)
+ await t.start()
+ var it = sockets.makeAsyncIterator()
+ var host = HostSide(try #require(await it.next()))
+ try await host.accept()
+ let hello = try await host.receive()
+ try await host.send(["id": hello["id"] ?? 0, "ok": helloBody()])
+
+ let candidates = t.stream(.screenCandidates("s"))
+ let sub = try await host.receive()
+ #expect(sub["sub"] as? String == "screenCandidates")
+ #expect((sub["b"] as? [String: Any])?["session"] as? String == "s")
+ // Nobody reads: one event past the buffer ends the stream and the host's subscription.
+ for _ in 0...ScreenCandidate.maxCount + 2 {
+ try await host.send(["id": sub["id"] ?? 0, "ev": ["type": "candidate"]])
+ }
+ let cancel = try await host.receive()
+ #expect(cancel["id"] as? Int == sub["id"] as? Int && cancel["cancel"] as? Bool == true)
+ var read = 0
+ await #expect(throws: ScreenError.candidateOverflow) {
+ for try await _ in candidates { read += 1 }
+ }
+ #expect(read == ScreenCandidate.maxCount + 2)
+ await t.shutdown()
+}
+
@Test(.timeLimit(.minutes(1))) func changedHostIdentityIsNotRetried() async throws {
let (dial, sockets) = fakeDialer()
var computer = v.computer
diff --git a/apps/ios/Kit/Tests/CodyncKitTests/ScreenCandidateTests.swift b/apps/ios/Kit/Tests/CodyncKitTests/ScreenCandidateTests.swift
new file mode 100644
index 00000000..8990ce80
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncKitTests/ScreenCandidateTests.swift
@@ -0,0 +1,34 @@
+import Foundation
+import Testing
+@testable import CodyncKit
+
+@Test func oldScreenPreparationDefaultsToCompleteSDP() throws {
+ let data = Data(#"{"session":"s","iceServers":[],"expiresAt":123}"#.utf8)
+ let connection = try JSONDecoder().decode(ScreenConnection.self, from: data)
+ #expect(!connection.trickle)
+ let updated = Data(#"{"session":"s","iceServers":[],"expiresAt":123,"trickle":true}"#.utf8)
+ let trickle = try JSONDecoder().decode(ScreenConnection.self, from: updated)
+ #expect(trickle.trickle)
+}
+
+@Test func candidateWireFormatPreservesMediaAndCompletion() throws {
+ let candidate = ScreenCandidate(candidate: "candidate:1 1 UDP 1 192.0.2.1 5000 typ relay", sdpMLineIndex: 1, sdpMid: "video")
+ let data = try JSONEncoder().encode(candidate)
+ let roundtrip = try JSONDecoder().decode(ScreenCandidate.self, from: data)
+ #expect(roundtrip == candidate)
+ try roundtrip.validate()
+ let completed = try JSONDecoder().decode(ScreenCandidate.self, from: Data(#"{"type":"complete"}"#.utf8))
+ #expect(completed == .complete)
+}
+
+@Test func malformedScreenCandidatesAreRejected() throws {
+ for json in [
+ #"{"type":"candidate","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":"candidate:1\nline","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":"candidate:1","sdpMLineIndex":-1}"#,
+ #"{"type":"candidate","candidate":"candidate:1","sdpMLineIndex":17}"#
+ ] {
+ let event = try JSONDecoder().decode(ScreenCandidate.self, from: Data(json.utf8))
+ #expect(throws: ScreenError.invalidCandidate) { try event.validate() }
+ }
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingLifetimeTests.swift b/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingLifetimeTests.swift
new file mode 100644
index 00000000..bdbbfe7a
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingLifetimeTests.swift
@@ -0,0 +1,127 @@
+import CryptoKit
+import Foundation
+import Testing
+@testable import CodyncKit
+@testable import CodyncUI
+
+@Test @MainActor func connectedMediaKeepsRunningAfterSignalingTransportLoss() async throws {
+ for nativeCallbackPending in [false, true] {
+ let transport = ScreenTransport()
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ let stopped = AsyncStream<Bool>.makeStream()
+ local.continuation.onTermination = { _ in stopped.continuation.yield(true); stopped.continuation.finish() }
+ var failures: [Error] = []
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { _ in }, isMediaConnected: { nativeCallbackPending }, onFailure: { failures.append($0) })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"ready"}"#)
+ try await subscription.value
+ try await signaling.installedAnswer()
+ if !nativeCallbackPending { signaling.mediaConnected() }
+ transport.sink.finish(throwing: HostError.unreachable)
+ let detached = await waitForTestEvent(stopped.stream, matching: { $0 })
+ try #require(detached)
+ try signaling.check()
+ #expect(failures.isEmpty)
+ if case .terminated = local.continuation.yield(.complete) {} else { Issue.record("local candidate production continued after detaching") }
+ let uploads = await transport.uploads
+ #expect(uploads.isEmpty, "the detached signaling must never replay uploads after a route change")
+ signaling.stop()
+ }
+}
+
+@Test @MainActor func completedCandidatesSurviveTransportLossOnlyAfterUploadAcknowledgment() async throws {
+ for acknowledged in [false, true] {
+ let transport = ScreenTransport(gated: true)
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ let stopped = AsyncStream<Bool>.makeStream()
+ local.continuation.onTermination = { _ in stopped.continuation.yield(true); stopped.continuation.finish() }
+ var failures: [Error] = []
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { _ in }, onFailure: { failures.append($0) })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"ready"}"#)
+ try await subscription.value
+ try await signaling.installedAnswer()
+ local.continuation.yield(.complete)
+ transport.emit(#"{"type":"complete"}"#)
+ let uploading = await waitForTestEvent(transport.uploaded, matching: { $0 == 1 })
+ try #require(uploading)
+ if acknowledged {
+ await transport.acknowledge()
+ try await waitForScreenCondition { signaling.candidatesComplete }
+ }
+ transport.sink.finish()
+ let ended = await waitForTestEvent(stopped.stream, matching: { $0 })
+ try #require(ended)
+ #expect(signaling.isDetached == acknowledged)
+ #expect(failures.count == (acknowledged ? 0 : 1))
+ if acknowledged { try signaling.check() }
+ else {
+ await transport.acknowledge()
+ #expect(throws: HostError.unreachable) { try signaling.check() }
+ }
+ signaling.stop()
+ }
+}
+
+@Test(.timeLimit(.minutes(1))) @MainActor func screenRecoveryWaitsForAReadyRouteWithoutSpendingAttempts() async throws {
+ let transport = LateSignalingFailureTransport(gateRecovery: true)
+ let session = ScreenSession(client: HostClient(transport: transport), display: nil)
+ defer { session.close() }
+ await session.start()
+ let installations = await transport.answerInstalled("late-1")
+ let installed = await waitForTestEvent(installations, matching: { $0 })
+ try #require(installed)
+ await transport.setRoute(.connecting)
+ await transport.failSignaling("late-1")
+ let waiting = await waitForTestEvent(transport.routeSubscribed, matching: { $0 })
+ try #require(waiting)
+ for state in [LinkState.failed("No network"), .connecting, .hostOffline(lastSeen: nil), .connecting] {
+ await transport.setRoute(state)
+ }
+ let before = await transport.preparations
+ #expect(before == 1)
+ #expect(session.phase == .reconnecting)
+ await transport.setRoute(.ready(.relay))
+ let recovering = await waitForTestEvent(transport.recoveryRequested, matching: { $0 })
+ try #require(recovering)
+ let after = await transport.preparations
+ #expect(after == 2)
+ session.close()
+ await transport.resumePreparation()
+}
+
+@Test(arguments: [false, true]) @MainActor func stoppedSignalingRoutesEndRecoveryWithTheirFailure(needsUpgrade: Bool) async throws {
+ let identity = DeviceIdentity(signing: Curve25519.Signing.PrivateKey(), push: Curve25519.KeyAgreement.PrivateKey())
+ let channel = ChannelTransport(computer: randomComputer("Mac"), identity: identity, pairingCode: nil,
+ dial: { _ in throw HostError.unreachable }, watchesNetwork: false)
+ defer { Task { await channel.shutdown() } }
+ let failed = AsyncStream<Bool>.makeStream()
+ let waiting = Task {
+ do {
+ try await channel.waitForRoute()
+ Issue.record("a stopped route must fail recovery")
+ } catch {
+ #expect(error as? HostError == (needsUpgrade ? .upgradeRequired : .unreachable))
+ failed.continuation.yield(true)
+ }
+ failed.continuation.finish()
+ }
+ defer { waiting.cancel() }
+ await channel.stopForScreenRecoveryTest(needsUpgrade: needsUpgrade)
+ let ended = await waitForTestEvent(failed.stream, matching: { $0 })
+ try #require(ended, "a terminal failed route must not wait indefinitely")
+}
+
+private extension ChannelTransport {
+ func stopForScreenRecoveryTest(needsUpgrade: Bool) {
+ isStopped = true
+ if needsUpgrade {
+ let outcome = upgrade()
+ if case let .stop(state) = outcome { setState(state) }
+ } else {
+ setState(.failed("Channel stopped."))
+ }
+ }
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingTests.swift b/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingTests.swift
new file mode 100644
index 00000000..8814d77e
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncUITests/ScreenSignalingTests.swift
@@ -0,0 +1,368 @@
+import Foundation
+import Testing
+@testable import CodyncKit
+@testable import CodyncUI
+@preconcurrency import WebRTC
+
+actor ScreenTransport: HostTransport {
+ nonisolated let events: AsyncThrowingStream<Data, Error>
+ nonisolated let sink: AsyncThrowingStream<Data, Error>.Continuation
+ nonisolated let uploaded: AsyncStream<Int>
+ private let uploadSink: AsyncStream<Int>.Continuation
+ private(set) var uploads: [ScreenCandidate] = []
+ private var acknowledgment: CheckedContinuation<Void, Never>?
+ let gated: Bool
+
+ init(gated: Bool = false) {
+ self.gated = gated
+ let pair = AsyncThrowingStream<Data, Error>.makeStream()
+ events = pair.stream
+ sink = pair.continuation
+ let uploads = AsyncStream<Int>.makeStream()
+ uploaded = uploads.stream
+ uploadSink = uploads.continuation
+ }
+
+ nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> { events }
+ nonisolated func states() -> AsyncStream<LinkState> { AsyncStream { $0.finish() } }
+
+ func call(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
+ struct Body: Decodable { var session: String; var candidate: ScreenCandidate }
+ let value = try JSONDecoder().decode(Body.self, from: body)
+ #expect(method == "screenCandidate" && value.session == "session")
+ uploads.append(value.candidate)
+ uploadSink.yield(uploads.count)
+ if gated { await withCheckedContinuation { acknowledgment = $0 } }
+ return Data("{}".utf8)
+ }
+
+ func acknowledge() { acknowledgment?.resume(); acknowledgment = nil }
+
+ nonisolated func emit(_ json: String) { sink.yield(Data(json.utf8)) }
+}
+
+@MainActor
+func waitForScreenCondition(_ condition: @MainActor () async -> Bool) async throws {
+ let deadline = ContinuousClock.now + .seconds(10)
+ while ContinuousClock.now < deadline {
+ let satisfied = await condition()
+ if satisfied { return }
+ try await Task.sleep(for: .milliseconds(5))
+ }
+ throw HostError.unreachable
+}
+
+private let firstCandidate = ScreenCandidate(candidate: "candidate:1 1 UDP 1 192.0.2.1 5000 typ relay", sdpMLineIndex: 0, sdpMid: "0")
+private let secondCandidate = ScreenCandidate(candidate: "candidate:2 1 UDP 1 192.0.2.2 5001 typ relay", sdpMLineIndex: 0, sdpMid: "0")
+
+@Test @MainActor func screenCandidatesWaitForAnswerThenUploadWithSerialAcknowledgments() async throws {
+ let transport = ScreenTransport(gated: true)
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ var applied: [ScreenCandidate] = []
+ var failures: [String] = []
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { applied.append($0) }, onFailure: { failures.append($0.localizedDescription) })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"ready"}"#)
+ transport.emit(#"{"type":"candidate","candidate":"candidate:1 1 UDP 1 192.0.2.1 5000 typ relay","sdpMLineIndex":0,"sdpMid":"0"}"#)
+ try await subscription.value
+ local.continuation.yield(firstCandidate)
+ local.continuation.yield(secondCandidate)
+ local.continuation.yield(.complete)
+ try await Task.sleep(for: .milliseconds(30))
+ let before = await transport.uploads
+ #expect(before.isEmpty && applied.isEmpty)
+ try await signaling.installedAnswer()
+ try await waitForScreenCondition { let uploads = await transport.uploads; return uploads.count == 1 }
+ #expect(applied == [firstCandidate])
+ let one = await transport.uploads
+ #expect(one == [firstCandidate])
+ await transport.acknowledge()
+ try await waitForScreenCondition { let uploads = await transport.uploads; return uploads.count == 2 }
+ await transport.acknowledge()
+ try await waitForScreenCondition { let uploads = await transport.uploads; return uploads.count == 3 }
+ let all = await transport.uploads
+ #expect(all == [firstCandidate, secondCandidate, .complete])
+ await transport.acknowledge()
+ #expect(failures.isEmpty)
+ signaling.stop()
+}
+
+@Test @MainActor func remoteCandidatesAwaitNativeCompletionBeforeApplyingTheNext() async throws {
+ let transport = ScreenTransport()
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ var applied: [ScreenCandidate] = []
+ var acknowledgment: CheckedContinuation<Void, Never>?
+ var failure: Error?
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { event in applied.append(event); await withCheckedContinuation { acknowledgment = $0 } }, onFailure: { failure = $0 })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"ready"}"#)
+ try await subscription.value
+ try await signaling.installedAnswer()
+ for candidate in [firstCandidate, secondCandidate, .complete] {
+ let data = try JSONEncoder().encode(candidate)
+ transport.sink.yield(data)
+ }
+ try await waitForScreenCondition { applied.count == 1 }
+ #expect(applied == [firstCandidate])
+ acknowledgment?.resume()
+ try await waitForScreenCondition { applied.count == 2 }
+ #expect(applied == [firstCandidate, secondCandidate])
+ acknowledgment?.resume()
+ // Completion is recorded at app level, never added as a null native candidate.
+ try await Task.sleep(for: .milliseconds(20))
+ #expect(failure == nil)
+ signaling.stop()
+}
+
+@Test @MainActor func signalingLossAndCandidateAfterCompletionFailTheAttempt() async throws {
+ for disconnected in [false, true] {
+ let transport = ScreenTransport()
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ var failures: [Error] = []
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { _ in Issue.record("unexpected native candidate") }, onFailure: { failures.append($0) })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"ready"}"#)
+ try await subscription.value
+ try await signaling.installedAnswer()
+ if disconnected { transport.sink.finish() }
+ else {
+ transport.emit(#"{"type":"complete"}"#)
+ transport.sink.yield(try JSONEncoder().encode(firstCandidate))
+ }
+ try await waitForScreenCondition { !failures.isEmpty }
+ #expect(failures.count == 1)
+ if disconnected { #expect(throws: HostError.self) { try signaling.check() } }
+ else { #expect(throws: ScreenError.candidatesOutOfOrder) { try signaling.check() } }
+ signaling.stop()
+ }
+}
+
+@Test @MainActor func aSignalingFailureBeforeReadyThrowsItsReason() async throws {
+ let transport = ScreenTransport()
+ let local = AsyncThrowingStream<ScreenCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(129))
+ let signaling = ScreenSignaling(client: HostClient(transport: transport), session: "session", localPair: local,
+ applyCandidate: { _ in }, onFailure: { _ in })
+ let subscription = Task { try await signaling.subscribe() }
+ transport.emit(#"{"type":"error","message":"unknown screen session"}"#)
+ await #expect(throws: ScreenError.signalingFailed("unknown screen session")) { try await subscription.value }
+}
+
+private actor FailingSignalingTransport: HostTransport {
+ private(set) var preparations = 0
+
+ func call(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
+ switch method {
+ case "screenPrepare":
+ preparations += 1
+ return Data(#"{"session":"attempt-\#(preparations)","iceServers":[],"expiresAt":4102444800000,"trickle":true}"#.utf8)
+ case "screenOffer": throw HostError.unreachable
+ default: return Data("{}".utf8)
+ }
+ }
+
+ nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> {
+ AsyncThrowingStream { c in
+ c.yield(Data(#"{"type":"ready"}"#.utf8))
+ c.yield(Data(#"{"type":"error","message":"Signaling disconnected."}"#.utf8))
+ c.finish()
+ }
+ }
+
+ nonisolated func states() -> AsyncStream<LinkState> { AsyncStream { $0.yield(.ready(.direct)); $0.finish() } }
+}
+
+@Test(.timeLimit(.minutes(1))) @MainActor func repeatedSignalingLossUsesFreshSessionsAndExhaustsTheExistingRetryBudget() async throws {
+ let transport = FailingSignalingTransport()
+ let session = ScreenSession(client: HostClient(transport: transport), display: nil)
+ await session.start()
+ try await waitForScreenCondition {
+ if case .failed = session.phase { return true }
+ return false
+ }
+ let preparations = await transport.preparations
+ #expect(preparations == 4, "one initial attempt and the existing three recovery attempts")
+ session.close()
+}
+
+actor LateSignalingFailureTransport: HostTransport {
+ private(set) var preparations = 0
+ private var streams: [String: AsyncThrowingStream<Data, Error>.Continuation] = [:]
+ private var installedAnswers: Set<String> = []
+ private var answerWaiters: [String: AsyncStream<Bool>.Continuation] = [:]
+ private let gateRecovery: Bool
+ private let preparationGate: AsyncStream<Void>
+ private let preparationRelease: AsyncStream<Void>.Continuation
+ nonisolated let recoveryRequested: AsyncStream<Bool>
+ private let recoverySink: AsyncStream<Bool>.Continuation
+ private var route = LinkState.ready(.direct)
+ private var routeObservers: [UUID: AsyncStream<LinkState>.Continuation] = [:]
+ nonisolated let routeSubscribed: AsyncStream<Bool>
+ private let routeSubscribedSink: AsyncStream<Bool>.Continuation
+
+ init(gateRecovery: Bool = false) {
+ self.gateRecovery = gateRecovery
+ let gate = AsyncStream<Void>.makeStream()
+ preparationGate = gate.stream
+ preparationRelease = gate.continuation
+ let requested = AsyncStream<Bool>.makeStream()
+ recoveryRequested = requested.stream
+ recoverySink = requested.continuation
+ let routes = AsyncStream<Bool>.makeStream()
+ routeSubscribed = routes.stream
+ routeSubscribedSink = routes.continuation
+ }
+
+ func resumePreparation() { preparationRelease.yield(()) }
+
+ func answerInstalled(_ session: String) -> AsyncStream<Bool> {
+ let pair = AsyncStream<Bool>.makeStream()
+ if installedAnswers.contains(session) {
+ pair.continuation.yield(true)
+ pair.continuation.finish()
+ } else {
+ answerWaiters[session] = pair.continuation
+ }
+ return pair.stream
+ }
+
+ func failSignaling(_ session: String) {
+ streams.removeValue(forKey: session)?.finish(throwing: HostError.unreachable)
+ }
+
+ private func installed(_ session: String) {
+ guard installedAnswers.insert(session).inserted else { return }
+ if let waiter = answerWaiters.removeValue(forKey: session) {
+ waiter.yield(true)
+ waiter.finish()
+ }
+ }
+
+ func call(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
+ switch method {
+ case "screenPrepare":
+ preparations += 1
+ if gateRecovery && preparations == 2 {
+ recoverySink.yield(true)
+ for await _ in preparationGate { break }
+ try Task.checkCancellation()
+ }
+ return Data(#"{"session":"late-\#(preparations)","iceServers":[],"expiresAt":4102444800000,"trickle":true}"#.utf8)
+ case "screenOffer":
+ struct Offer: Decodable { let session: String; let sdp: String }
+ let offer = try JSONDecoder().decode(Offer.self, from: body)
+ let answer = try await Self.answer(offer.sdp)
+ struct Answer: Encodable { let session: String; let sdp: String }
+ return try JSONEncoder().encode(Answer(session: offer.session, sdp: answer))
+ case "screenCandidate":
+ struct Upload: Decodable { let session: String }
+ let upload = try JSONDecoder().decode(Upload.self, from: body)
+ // Candidate uploads start only after the phone's native answer completion.
+ installed(upload.session)
+ return Data("{}".utf8)
+ case "screenClose":
+ struct Close: Decodable { let session: String }
+ let close = try JSONDecoder().decode(Close.self, from: body)
+ streams.removeValue(forKey: close.session)?.finish()
+ answerWaiters.removeValue(forKey: close.session)?.finish()
+ return Data("{}".utf8)
+ default: return Data("{}".utf8)
+ }
+ }
+
+ @MainActor private static func answer(_ offer: String) async throws -> String {
+ let config = RTCConfiguration()
+ config.sdpSemantics = .unifiedPlan
+ config.bundlePolicy = .maxBundle
+ let constraints = RTCMediaConstraints(mandatoryConstraints: nil, optionalConstraints: nil)
+ guard let pc = ScreenSession.factory.peerConnection(with: config, constraints: constraints, delegate: nil) else {
+ throw HostError.unreachable
+ }
+ defer { pc.close() }
+ try await pc.setRemoteDescription(RTCSessionDescription(type: .offer, sdp: offer))
+ let answer = try await pc.answer(for: constraints)
+ try await pc.setLocalDescription(answer)
+ return answer.sdp
+ }
+
+ nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> {
+ AsyncThrowingStream { c in
+ guard case let .screenCandidates(session) = request else { c.finish(); return }
+ Task { await self.open(session, sink: c) }
+ }
+ }
+
+ private func open(_ session: String, sink: AsyncThrowingStream<Data, Error>.Continuation) {
+ streams[session] = sink
+ sink.yield(Data(#"{"type":"ready"}"#.utf8))
+ }
+
+ func setRoute(_ state: LinkState) {
+ route = state
+ for sink in routeObservers.values { sink.yield(state) }
+ }
+
+ nonisolated func states() -> AsyncStream<LinkState> {
+ AsyncStream { sink in
+ let id = UUID()
+ Task { await self.observeRoute(id, sink: sink) }
+ sink.onTermination = { _ in Task { await self.removeRouteObserver(id) } }
+ }
+ }
+
+ private func observeRoute(_ id: UUID, sink: AsyncStream<LinkState>.Continuation) {
+ routeObservers[id] = sink
+ sink.yield(route)
+ routeSubscribedSink.yield(true)
+ }
+
+ private func removeRouteObserver(_ id: UUID) { routeObservers[id] = nil }
+}
+
+@Test(.timeLimit(.minutes(1))) @MainActor func signalingLossAfterAnswerDoesNotResetRecoveryUntilMediaIsLive() async throws {
+ let transport = LateSignalingFailureTransport()
+ let session = ScreenSession(client: HostClient(transport: transport), display: nil)
+ defer { session.close() }
+ await session.start()
+ #expect(session.phase == .connecting, "the initial answer was installed successfully")
+ for attempt in 1...4 {
+ let id = "late-\(attempt)"
+ let installations = await transport.answerInstalled(id)
+ let installed = await waitForTestEvent(installations, matching: { $0 })
+ try #require(installed)
+ #expect(session.track != nil, "the negotiated peer exists before its signaling is lost")
+ #expect(session.phase == (attempt == 1 ? .connecting : .reconnecting))
+ let preparations = await transport.preparations
+ #expect(preparations == attempt)
+ await transport.failSignaling(id)
+ }
+ let exhausted = await waitForObservedCondition {
+ if case .failed = session.phase { return true }
+ return false
+ }
+ try #require(exhausted)
+ let preparations = await transport.preparations
+ #expect(preparations == 4, "one initial attempt and three retries, even when answers succeed")
+}
+
+@Test(.timeLimit(.minutes(1))) @MainActor func fatalSignalingLossRetiresTheOldPeerBeforePreparingItsReplacement() async throws {
+ let transport = LateSignalingFailureTransport(gateRecovery: true)
+ let session = ScreenSession(client: HostClient(transport: transport), display: nil)
+ defer { session.close() }
+ await session.start()
+ #expect(session.track != nil)
+ let installations = await transport.answerInstalled("late-1")
+ let installed = await waitForTestEvent(installations, matching: { $0 })
+ try #require(installed)
+ await transport.failSignaling("late-1")
+ let preparing = await waitForTestEvent(transport.recoveryRequested, matching: { $0 })
+ try #require(preparing)
+ let preparations = await transport.preparations
+ #expect(preparations == 2)
+ #expect(session.phase == .reconnecting)
+ #expect(session.track == nil, "the failed peer must close even while replacement preparation waits")
+ session.close()
+ await transport.resumePreparation()
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreConnectionTests.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreConnectionTests.swift
new file mode 100644
index 00000000..03455b30
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreConnectionTests.swift
@@ -0,0 +1,239 @@
+import CryptoKit
+import Foundation
+import Testing
+@testable import CodyncKit
+@testable import CodyncUI
+
+@MainActor @Test func visibleConversationAcknowledgesEntriesBeforeUnreadAndFinalUpdates() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.ready(.direct))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ store.setActive(true)
+ defer { store.setActive(false) }
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.bots["b1"] != nil })
+ let token = UUID()
+ store.setReading(token, botId: "b1", active: true)
+ #expect(await until { await fake.readReceipts.count == 1 })
+ // Streaming and final content have the same ID. The final update must be read
+ // even though the roster has not yet reported any unread messages.
+ await fake.emit(#"{"type":"entry","entry":{"id":"answer","seq":1,"botId":"b1","rev":2,"kind":"agent","turn":1,"data":{"text":"draft","final":false},"createdAt":1,"updatedAt":1}}"#)
+ #expect(await until { store.chat("b1").last?.rev == 2 })
+ await fake.emit(#"{"type":"entry","entry":{"id":"answer","seq":1,"botId":"b1","rev":3,"kind":"agent","turn":1,"data":{"text":"done","final":true},"createdAt":1,"updatedAt":2}}"#)
+ #expect(await until { await fake.readReceipts.count == 2 })
+ await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Bot","rev":4,"unread":1}}"#)
+ #expect(await until { await fake.readReceipts.count == 3 })
+ store.setReading(token, botId: "b1", active: false)
+ await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Bot","rev":5,"unread":2}}"#)
+ #expect(await until { store.bots["b1"]?.rev == 5 })
+ try await Task.sleep(for: .milliseconds(50))
+ #expect(await fake.readReceipts.count == 3)
+ // Returning to the foreground acknowledges without needing navigation.
+ store.setReading(token, botId: "b1", active: true)
+ #expect(await until { await fake.readReceipts.count == 4 })
+}
+
+
+@MainActor @Test func initialOfflineReportWaitsForReconnect() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.hostOffline(lastSeen: nil))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ try await Task.sleep(for: .milliseconds(200))
+ #expect(store.connection == .connecting)
+ #expect(store.lastError == nil)
+ await fake.set(.ready(.relay))
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ try await Task.sleep(for: .seconds(1))
+ #expect(store.connection == .online)
+}
+
+@MainActor @Test func initialUnreachableEventuallyShowsOffline() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.failed("Can't reach"))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ try await Task.sleep(for: .milliseconds(200))
+ #expect(store.connection == .connecting)
+ #expect(await until { store.connection == .offline("Can't reach") })
+ #expect(store.lastError == nil)
+}
+
+@MainActor @Test func failedAutomaticReadDoesNotInterruptChatAndRecoversWithTheLink() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.ready(.direct))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ await fake.setReadFailure(true)
+ store.setReading(UUID(), botId: "b1", active: true)
+ #expect(await until { await fake.readAttempts == 1 })
+ try await Task.sleep(for: .milliseconds(50))
+ #expect(store.lastError == nil)
+ await fake.set(.hostOffline(lastSeen: nil))
+ #expect(await until { store.hostRoute == nil })
+ #expect(store.connection == .online)
+ #expect(store.canQueue)
+ await fake.setReadFailure(false)
+ await fake.set(.ready(.relay))
+ #expect(await until { await fake.eventSubscriptionCount == 2 })
+ await fake.emit(botEvent("b2", name: "Other bot", rev: 2))
+ #expect(await until { await fake.readReceipts.count == 1 })
+ #expect(store.connection == .online)
+ #expect(store.lastError == nil)
+}
+
+@MainActor @Test func actionDuringReconnectWaitsInsteadOfFailing() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.ready(.direct))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ // A drop still inside its grace: the header says connected, the link is gone.
+ await fake.set(.connecting)
+ #expect(await until { store.hostRoute == nil })
+ store.stop("b1")
+ #expect(await until { store.shownConnection == .connecting })
+ #expect(await !fake.calls.contains("stop"))
+ await fake.set(.ready(.relay))
+ #expect(await until { await fake.eventSubscriptionCount == 2 })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 2))
+ #expect(await until { await fake.calls.contains("stop") })
+ #expect(store.shownConnection == .online)
+ #expect(store.lastError == nil)
+}
+
+@MainActor @Test func droppedCallIsRetriedOnlyWhenSafeToRepeat() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.ready(.direct))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ await fake.fail("stop", times: 1)
+ store.stop("b1")
+ #expect(await until { await fake.calls.filter { $0 == "stop" }.count == 2 })
+ #expect(store.lastError == nil)
+ // A new session can't be told apart from a second one, so it isn't sent twice.
+ await fake.fail("newSession", times: 1)
+ store.newSession("b1")
+ #expect(await until { store.lastError != nil })
+ #expect(await fake.calls.filter { $0 == "newSession" }.count == 1)
+}
+
+@MainActor @Test(.timeLimit(.minutes(1))) func offlineTapReconnectsBeforeGivingUp() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ // The first link can't reach the computer; a fresh attempt would.
+ let down = FakeRemote(.failed("Can't reach"))
+ let up = FakeRemote(.ready(.relay))
+ var made = 0
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in
+ made += 1
+ return made == 1 ? down : up
+ }
+ defer { store.retire() }
+ store.setActive(true)
+ let offline = await waitForObservedCondition { store.connection == .offline("Can't reach") }
+ try #require(offline)
+ store.stop("b1")
+ let subscriptions = await up.checkpoints()
+ let subscribed = await waitForTestEvent(subscriptions, matching: { $0 == .eventsSubscribed })
+ try #require(subscribed)
+ await up.emit(botEvent("b1", name: "Bot", rev: 1))
+ let calls = await up.checkpoints()
+ let stopped = await waitForTestEvent(calls, matching: { $0 == .called("stop") })
+ try #require(stopped)
+ let downCalls = await down.calls
+ #expect(made == 2)
+ #expect(downCalls.isEmpty)
+ #expect(store.lastError == nil)
+}
+
+@MainActor @Test(.timeLimit(.minutes(1))) func offlineActionWaitsForFreshTransportBeforeSending() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let down = FakeRemote(.failed("Can't reach"))
+ let up = FakeRemote(.ready(.relay))
+ let requested = AsyncStream<Bool>.makeStream()
+ let release = AsyncStream<Void>.makeStream()
+ defer { release.continuation.finish(); requested.continuation.finish() }
+ var made = 0
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in
+ made += 1
+ if made == 1 { return down }
+ requested.continuation.yield(true)
+ for await _ in release.stream { break }
+ try Task.checkCancellation()
+ return up
+ }
+ defer { store.retire() }
+ store.setActive(true)
+ let offline = await waitForObservedCondition { store.connection == .offline("Can't reach") }
+ try #require(offline)
+ store.stop("b1")
+ let preparing = await waitForTestEvent(requested.stream, matching: { $0 })
+ try #require(preparing)
+ let waiting = await waitForObservedCondition { store.waiting == 1 }
+ try #require(waiting)
+ let downCalls = await down.calls
+ let before = await up.calls
+ #expect(store.shownConnection == .connecting)
+ #expect(downCalls.isEmpty)
+ #expect(before.isEmpty)
+ #expect(store.lastError == nil)
+
+ release.continuation.yield(())
+ let subscriptions = await up.checkpoints()
+ let subscribed = await waitForTestEvent(subscriptions, matching: { $0 == .eventsSubscribed })
+ try #require(subscribed)
+ await up.emit(botEvent("b1", name: "Bot", rev: 1))
+ let calls = await up.checkpoints()
+ let stopped = await waitForTestEvent(calls, matching: { $0 == .called("stop") })
+ try #require(stopped)
+ let after = await up.calls
+ #expect(after.filter { $0 == "stop" }.count == 1)
+ #expect(made == 2)
+ #expect(store.lastError == nil)
+}
+
+@MainActor @Test func permissionAnswerShowsOnTheCardAndIsSentOnce() async throws {
+ let (storage, suite) = context()
+ defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
+ let fake = FakeRemote(.ready(.direct))
+ let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
+ defer { store.retire() }
+ store.setActive(true)
+ #expect(await until { await fake.subscribed })
+ await fake.emit(botEvent("b1", name: "Bot", rev: 1))
+ #expect(await until { store.connection == .online })
+ let card = Entry(id: "p1", seq: 1, botId: "b1", rev: 1, kind: "permission", turn: 1,
+ data: EntryData(status: "pending"), createdAt: 1, updatedAt: 1)
+ store.respond(card, option: "allow")
+ store.respond(card, option: "deny")
+ #expect(store.answering["p1"] == "allow")
+ #expect(await until { await fake.calls.contains("respondPermission") })
+ try await Task.sleep(for: .milliseconds(100))
+ #expect(await fake.calls.filter { $0 == "respondPermission" }.count == 1)
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreTestFixtures.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreTestFixtures.swift
new file mode 100644
index 00000000..2995047c
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreTestFixtures.swift
@@ -0,0 +1,194 @@
+import CryptoKit
+import Foundation
+import Testing
+@testable import CodyncKit
+@testable import CodyncUI
+
+/// A scripted transport: tests set its link state and push events; it records what the store asked.
+actor FakeRemote: RemoteTransport {
+ enum Checkpoint: Equatable, Sendable {
+ case eventsSubscribed
+ case called(String)
+ }
+
+ private var state: LinkState
+ private var stateSinks: [AsyncStream<LinkState>.Continuation] = []
+ private var eventSinks: [AsyncThrowingStream<Data, Error>.Continuation] = []
+ private var mailboxSinks: [AsyncStream<MailboxEvent>.Continuation] = []
+ private(set) var sent: [String] = []
+ /// Each `send`'s quoted message, in order.
+ private(set) var replyTos: [String?] = []
+ private(set) var readReceipts: [Data] = []
+ private var failReadReceipts = false
+ private(set) var readAttempts = 0
+ private(set) var shutdownCount = 0
+ private var isShutdown = false
+
+ private var checkpointSinks: [UUID: AsyncStream<Checkpoint>.Continuation] = [:]
+
+ func checkpoints() -> AsyncStream<Checkpoint> {
+ let id = UUID()
+ let pair = AsyncStream<Checkpoint>.makeStream()
+ checkpointSinks[id] = pair.continuation
+ if subscribed { pair.continuation.yield(.eventsSubscribed) }
+ for call in calls { pair.continuation.yield(.called(call)) }
+ pair.continuation.onTermination = { [weak self] _ in
+ Task { await self?.removeCheckpointSink(id) }
+ }
+ return pair.stream
+ }
+
+ private func removeCheckpointSink(_ id: UUID) { checkpointSinks[id] = nil }
+
+ private func record(_ checkpoint: Checkpoint) {
+ for sink in checkpointSinks.values { sink.yield(checkpoint) }
+ }
+
+ func setReadFailure(_ fail: Bool) { failReadReceipts = fail }
+ private(set) var calls: [String] = []
+ private var failures: [String: Int] = [:]
+ /// The next `count` calls of `method` fail as if the link dropped under them.
+ func fail(_ method: String, times count: Int) { failures[method] = count }
+ private(set) var enqueued: [String] = []
+ var cancelResult = MailboxCancel.cancelled
+
+ private var hello = #"{"hostId":"h1","name":"Mac","version":"3.0.0","os":"macos","backends":[],"rev":0}"#
+ /// What `hello` answers from now on (a host updated in place keeps its link).
+ func setHello(_ json: String) { hello = json }
+
+ init(_ state: LinkState) { self.state = state }
+
+ var subscribed: Bool { !eventSinks.isEmpty }
+ var eventSubscriptionCount: Int { eventSinks.count }
+
+ func set(_ new: LinkState) {
+ state = new
+ for s in stateSinks { s.yield(new) }
+ }
+
+ func emit(_ json: String) {
+ for s in eventSinks { s.yield(Data(json.utf8)) }
+ }
+
+ func emit(_ event: MailboxEvent) {
+ for s in mailboxSinks { s.yield(event) }
+ }
+
+ private func respond(_ method: String, _ body: Data) throws -> Data {
+ guard !isShutdown else { throw HostError.unreachable }
+ calls.append(method)
+ record(.called(method))
+ if let left = failures[method], left > 0 {
+ failures[method] = left - 1
+ throw HostError.unreachable
+ }
+ switch method {
+ case "hello":
+ return Data(hello.utf8)
+ case "markRead":
+ readAttempts += 1
+ if failReadReceipts { throw HostError.unreachable }
+ readReceipts.append(body)
+ return Data("{}".utf8)
+ case "send":
+ let b = (try JSONSerialization.jsonObject(with: body) as? [String: String]) ?? [:]
+ sent.append(b["text"] ?? "")
+ replyTos.append(b["replyTo"])
+ let entry = #"{"id":"e\#(sent.count)","seq":\#(sent.count),"botId":"\#(b["botId"] ?? "")","rev":\#(sent.count),"kind":"user","turn":1,"data":{"text":"\#(b["text"] ?? "")","clientNonce":"\#(b["clientNonce"] ?? "")"},"createdAt":1,"updatedAt":1}"#
+ return Data(#"{"entry":\#(entry)}"#.utf8)
+ case "startCall":
+ // A fresh credential per provider session, so a test can tell a replaced session from the first.
+ let n = calls.filter { $0 == "startCall" }.count
+ return Data(#"{"instructions":"Be the relay.","tools":[],"credential":"token-\#(n)"}"#.utf8)
+ default:
+ return Data("{}".utf8)
+ }
+ }
+
+ nonisolated func call(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
+ try await respond(method, body)
+ }
+
+ private func addEvents(_ c: AsyncThrowingStream<Data, Error>.Continuation) {
+ eventSinks.append(c)
+ record(.eventsSubscribed)
+ }
+ private func addState(_ c: AsyncStream<LinkState>.Continuation) {
+ isShutdown = false
+ c.yield(state)
+ stateSinks.append(c)
+ }
+ private func addMailbox(_ c: AsyncStream<MailboxEvent>.Continuation) { mailboxSinks.append(c) }
+
+ nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> {
+ AsyncThrowingStream { c in Task { await self.addEvents(c) } }
+ }
+
+ nonisolated func states() -> AsyncStream<LinkState> {
+ AsyncStream { c in Task { await self.addState(c) } }
+ }
+
+ nonisolated func computerUpdates() -> AsyncStream<Computer> { AsyncStream { _ in } }
+
+ private var enqueueError: (any Error)?
+ /// Every later `enqueue` throws `error`.
+ func failEnqueue(_ error: any Error) { enqueueError = error }
+
+ func enqueue(botId: String, text: String, clientNonce: String, replyTo: String?) async throws {
+ if let enqueueError { throw enqueueError }
+ enqueued.append(clientNonce)
+ }
+ func cancelQueued(clientNonce: String) async -> MailboxCancel { cancelResult }
+ func listQueued() async throws -> [QueuedItem] { enqueued.map { QueuedItem(nonce: $0, exp: nil, state: "queued") } }
+
+ nonisolated func mailboxEvents() -> AsyncStream<MailboxEvent> {
+ AsyncStream { c in Task { await self.addMailbox(c) } }
+ }
+
+ func shutdown() async {
+ shutdownCount += 1
+ isShutdown = true
+ for s in eventSinks { s.finish() }
+ for s in stateSinks { s.finish() }
+ eventSinks = []
+ stateSinks = []
+ }
+}
+
+func randomComputer(_ name: String) -> Computer {
+ let sk = Curve25519.Signing.PrivateKey().publicKey.rawRepresentation
+ return Computer(id: RelayCrypto.computerId(signKey: sk), name: name, signKey: sk.base64URLForTests,
+ boxKey: Curve25519.KeyAgreement.PrivateKey().publicKey.rawRepresentation.base64URLForTests,
+ cloud: URL(string: "https://cloud.example.dev"))
+}
+
+private extension Data {
+ var base64URLForTests: String {
+ base64EncodedString().replacingOccurrences(of: "+", with: "-").replacingOccurrences(of: "/", with: "_").replacingOccurrences(of: "=", with: "")
+ }
+}
+
+func context() -> (SharedStore.Context, String) {
+ let suite = "CodyncUITests.\(UUID().uuidString)"
+ return (SharedStore.Context(accountID: "user_\(UUID().uuidString)", suite: suite), suite)
+}
+
+/// Polls until `condition` holds (2 s at most).
+@MainActor
+func until(_ condition: @MainActor () async -> Bool) async -> Bool {
+ for _ in 0..<200 {
+ let satisfied = await condition()
+ if satisfied { return true }
+ try? await Task.sleep(for: .milliseconds(10))
+ }
+ return false
+}
+
+func botEvent(_ id: String, name: String, rev: Int) -> String {
+ #"{"type":"bot","bot":{"id":"\#(id)","name":"\#(name)","rev":\#(rev),"lastAt":\#(rev)}}"#
+}
+
+
+extension FakeRemote {
+ func setCancelResult(_ result: MailboxCancel) { cancelResult = result }
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreTestReadiness.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreTestReadiness.swift
new file mode 100644
index 00000000..384c96bc
--- /dev/null
+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreTestReadiness.swift
@@ -0,0 +1,64 @@
+import Foundation
+import Observation
+@testable import CodyncUI
+
+/// Wait for a recorded fixture event, allowing the store's real action deadline.
+/// Cancelling the losing stream iterator also removes its fixture subscription.
+@MainActor
+func waitForTestEvent<Event: Sendable>(_ events: AsyncStream<Event>, matching: @escaping @Sendable (Event) -> Bool) async -> Bool {
+ let timeout = BotStore.actionPatience
+ return await withTaskGroup(of: Bool.self) { group in
+ group.addTask {
+ for await event in events {
+ if matching(event) { return true }
+ }
+ return false
+ }
+ group.addTask {
+ do { try await Task.sleep(for: timeout) } catch { return false }
+ return false
+ }
+ let result = await group.next()
+ group.cancelAll()
+ return result ?? false
+ }
+}
+
+/// Observe store state instead of racing a two-second polling window on MainActor.
+@MainActor
+func waitForObservedCondition(_ condition: @escaping @MainActor () -> Bool) async -> Bool {
+ let observer = StoreConditionObserver(condition)
+ defer { observer.stop() }
+ return await waitForTestEvent(observer.events, matching: { $0 })
+}
+
+@MainActor
+private final class StoreConditionObserver {
+ let events: AsyncStream<Bool>
+ private let sink: AsyncStream<Bool>.Continuation
+ private let condition: @MainActor () -> Bool
+ private var stopped = false
+
+ init(_ condition: @escaping @MainActor () -> Bool) {
+ self.condition = condition
+ let pair = AsyncStream<Bool>.makeStream()
+ events = pair.stream
+ sink = pair.continuation
+ observe()
+ }
+
+ func stop() {
+ stopped = true
+ sink.finish()
+ }
+
+ private func observe() {
+ guard !stopped else { return }
+ let value = withObservationTracking {
+ condition()
+ } onChange: { [weak self] in
+ Task { @MainActor in self?.observe() }
+ }
+ sink.yield(value)
+ }
+}
diff --git a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
index 747053d4..1315e1ec 100644
--- a/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
+++ b/apps/ios/Kit/Tests/CodyncUITests/StoreTests.swift
@@ -4,160 +4,6 @@ import Testing
@testable import CodyncKit
@testable import CodyncUI
-/// A scripted transport: tests set its link state and push events; it records what the store asked.
-actor FakeRemote: RemoteTransport {
- private var state: LinkState
- private var stateSinks: [AsyncStream<LinkState>.Continuation] = []
- private var eventSinks: [AsyncThrowingStream<Data, Error>.Continuation] = []
- private var mailboxSinks: [AsyncStream<MailboxEvent>.Continuation] = []
- private(set) var sent: [String] = []
- /// Each `send`'s quoted message, in order.
- private(set) var replyTos: [String?] = []
- private(set) var readReceipts: [Data] = []
- private var failReadReceipts = false
- private(set) var readAttempts = 0
- private(set) var shutdownCount = 0
- private var isShutdown = false
-
- func setReadFailure(_ fail: Bool) { failReadReceipts = fail }
- private(set) var calls: [String] = []
- private var failures: [String: Int] = [:]
- /// The next `count` calls of `method` fail as if the link dropped under them.
- func fail(_ method: String, times count: Int) { failures[method] = count }
- private(set) var enqueued: [String] = []
- var cancelResult = MailboxCancel.cancelled
-
- private var hello = #"{"hostId":"h1","name":"Mac","version":"3.0.0","os":"macos","backends":[],"rev":0}"#
- /// What `hello` answers from now on (a host updated in place keeps its link).
- func setHello(_ json: String) { hello = json }
-
- init(_ state: LinkState) { self.state = state }
-
- var subscribed: Bool { !eventSinks.isEmpty }
- var eventSubscriptionCount: Int { eventSinks.count }
-
- func set(_ new: LinkState) {
- state = new
- for s in stateSinks { s.yield(new) }
- }
-
- func emit(_ json: String) {
- for s in eventSinks { s.yield(Data(json.utf8)) }
- }
-
- func emit(_ event: MailboxEvent) {
- for s in mailboxSinks { s.yield(event) }
- }
-
- private func respond(_ method: String, _ body: Data) throws -> Data {
- guard !isShutdown else { throw HostError.unreachable }
- calls.append(method)
- if let left = failures[method], left > 0 {
- failures[method] = left - 1
- throw HostError.unreachable
- }
- switch method {
- case "hello":
- return Data(hello.utf8)
- case "markRead":
- readAttempts += 1
- if failReadReceipts { throw HostError.unreachable }
- readReceipts.append(body)
- return Data("{}".utf8)
- case "send":
- let b = (try JSONSerialization.jsonObject(with: body) as? [String: String]) ?? [:]
- sent.append(b["text"] ?? "")
- replyTos.append(b["replyTo"])
- let entry = #"{"id":"e\#(sent.count)","seq":\#(sent.count),"botId":"\#(b["botId"] ?? "")","rev":\#(sent.count),"kind":"user","turn":1,"data":{"text":"\#(b["text"] ?? "")","clientNonce":"\#(b["clientNonce"] ?? "")"},"createdAt":1,"updatedAt":1}"#
- return Data(#"{"entry":\#(entry)}"#.utf8)
- case "startCall":
- // A fresh credential per provider session, so a test can tell a replaced session from the first.
- let n = calls.filter { $0 == "startCall" }.count
- return Data(#"{"instructions":"Be the relay.","tools":[],"credential":"token-\#(n)"}"#.utf8)
- default:
- return Data("{}".utf8)
- }
- }
-
- nonisolated func call(_ method: String, body: Data, timeout: TimeInterval) async throws -> Data {
- try await respond(method, body)
- }
-
- private func addEvents(_ c: AsyncThrowingStream<Data, Error>.Continuation) { eventSinks.append(c) }
- private func addState(_ c: AsyncStream<LinkState>.Continuation) {
- isShutdown = false
- c.yield(state)
- stateSinks.append(c)
- }
- private func addMailbox(_ c: AsyncStream<MailboxEvent>.Continuation) { mailboxSinks.append(c) }
-
- nonisolated func stream(_ request: HostStreamRequest) -> AsyncThrowingStream<Data, Error> {
- AsyncThrowingStream { c in Task { await self.addEvents(c) } }
- }
-
- nonisolated func states() -> AsyncStream<LinkState> {
- AsyncStream { c in Task { await self.addState(c) } }
- }
-
- nonisolated func computerUpdates() -> AsyncStream<Computer> { AsyncStream { _ in } }
-
- private var enqueueError: (any Error)?
- /// Every later `enqueue` throws `error`.
- func failEnqueue(_ error: any Error) { enqueueError = error }
-
- func enqueue(botId: String, text: String, clientNonce: String, replyTo: String?) async throws {
- if let enqueueError { throw enqueueError }
- enqueued.append(clientNonce)
- }
- func cancelQueued(clientNonce: String) async -> MailboxCancel { cancelResult }
- func listQueued() async throws -> [QueuedItem] { enqueued.map { QueuedItem(nonce: $0, exp: nil, state: "queued") } }
-
- nonisolated func mailboxEvents() -> AsyncStream<MailboxEvent> {
- AsyncStream { c in Task { await self.addMailbox(c) } }
- }
-
- func shutdown() async {
- shutdownCount += 1
- isShutdown = true
- for s in eventSinks { s.finish() }
- for s in stateSinks { s.finish() }
- eventSinks = []
- stateSinks = []
- }
-}
-
-func randomComputer(_ name: String) -> Computer {
- let sk = Curve25519.Signing.PrivateKey().publicKey.rawRepresentation
- return Computer(id: RelayCrypto.computerId(signKey: sk), name: name, signKey: sk.base64URLForTests,
- boxKey: Curve25519.KeyAgreement.PrivateKey().publicKey.rawRepresentation.base64URLForTests,
- cloud: URL(string: "https://cloud.example.dev"))
-}
-
-private extension Data {
- var base64URLForTests: String {
- base64EncodedString().replacingOccurrences(of: "+", with: "-").replacingOccurrences(of: "/", with: "_").replacingOccurrences(of: "=", with: "")
- }
-}
-
-func context() -> (SharedStore.Context, String) {
- let suite = "CodyncUITests.\(UUID().uuidString)"
- return (SharedStore.Context(accountID: "user_\(UUID().uuidString)", suite: suite), suite)
-}
-
-/// Polls until `condition` holds (2 s at most).
-@MainActor
-func until(_ condition: @MainActor () async -> Bool) async -> Bool {
- for _ in 0..<200 {
- if await condition() { return true }
- try? await Task.sleep(for: .milliseconds(10))
- }
- return false
-}
-
-private func botEvent(_ id: String, name: String, rev: Int) -> String {
- #"{"type":"bot","bot":{"id":"\#(id)","name":"\#(name)","rev":\#(rev),"lastAt":\#(rev)}}"#
-}
-
@MainActor @Test func voiceCallSendsAndReceivesInBackgroundWithoutViewUpdates() async throws {
let (storage, suite) = context()
defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
@@ -392,43 +238,8 @@ private func botEvent(_ id: String, name: String, rev: Int) -> String {
#expect(account.store(for: saved.id)?.computer.notifications == nil)
}
-extension FakeRemote {
- func setCancelResult(_ result: MailboxCancel) { cancelResult = result }
-}
-@MainActor @Test func visibleConversationAcknowledgesEntriesBeforeUnreadAndFinalUpdates() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.ready(.direct))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- store.setActive(true)
- defer { store.setActive(false) }
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.bots["b1"] != nil })
- let token = UUID()
- store.setReading(token, botId: "b1", active: true)
- #expect(await until { await fake.readReceipts.count == 1 })
- // Streaming and final content have the same ID. The final update must be read
- // even though the roster has not yet reported any unread messages.
- await fake.emit(#"{"type":"entry","entry":{"id":"answer","seq":1,"botId":"b1","rev":2,"kind":"agent","turn":1,"data":{"text":"draft","final":false},"createdAt":1,"updatedAt":1}}"#)
- #expect(await until { store.chat("b1").last?.rev == 2 })
- await fake.emit(#"{"type":"entry","entry":{"id":"answer","seq":1,"botId":"b1","rev":3,"kind":"agent","turn":1,"data":{"text":"done","final":true},"createdAt":1,"updatedAt":2}}"#)
- #expect(await until { await fake.readReceipts.count == 2 })
- await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Bot","rev":4,"unread":1}}"#)
- #expect(await until { await fake.readReceipts.count == 3 })
- store.setReading(token, botId: "b1", active: false)
- await fake.emit(#"{"type":"bot","bot":{"id":"b1","name":"Bot","rev":5,"unread":2}}"#)
- #expect(await until { store.bots["b1"]?.rev == 5 })
- try await Task.sleep(for: .milliseconds(50))
- #expect(await fake.readReceipts.count == 3)
- // Returning to the foreground acknowledges without needing navigation.
- store.setReading(token, botId: "b1", active: true)
- #expect(await until { await fake.readReceipts.count == 4 })
-}
@MainActor @Test func readingOneChatDoesNotAcknowledgeAnother() async throws {
let (storage, suite) = context()
@@ -456,150 +267,12 @@ extension FakeRemote {
}
}
-@MainActor @Test func initialOfflineReportWaitsForReconnect() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.hostOffline(lastSeen: nil))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- try await Task.sleep(for: .milliseconds(200))
- #expect(store.connection == .connecting)
- #expect(store.lastError == nil)
- await fake.set(.ready(.relay))
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- try await Task.sleep(for: .seconds(1))
- #expect(store.connection == .online)
-}
-@MainActor @Test func initialUnreachableEventuallyShowsOffline() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.failed("Can't reach"))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- try await Task.sleep(for: .milliseconds(200))
- #expect(store.connection == .connecting)
- #expect(await until { store.connection == .offline("Can't reach") })
- #expect(store.lastError == nil)
-}
-@MainActor @Test func failedAutomaticReadDoesNotInterruptChatAndRecoversWithTheLink() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.ready(.direct))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- await fake.setReadFailure(true)
- store.setReading(UUID(), botId: "b1", active: true)
- #expect(await until { await fake.readAttempts == 1 })
- try await Task.sleep(for: .milliseconds(50))
- #expect(store.lastError == nil)
- await fake.set(.hostOffline(lastSeen: nil))
- #expect(await until { store.hostRoute == nil })
- #expect(store.connection == .online)
- #expect(store.canQueue)
- await fake.setReadFailure(false)
- await fake.set(.ready(.relay))
- #expect(await until { await fake.eventSubscriptionCount == 2 })
- await fake.emit(botEvent("b2", name: "Other bot", rev: 2))
- #expect(await until { await fake.readReceipts.count == 1 })
- #expect(store.connection == .online)
- #expect(store.lastError == nil)
-}
-@MainActor @Test func actionDuringReconnectWaitsInsteadOfFailing() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.ready(.direct))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- // A drop still inside its grace: the header says connected, the link is gone.
- await fake.set(.connecting)
- #expect(await until { store.hostRoute == nil })
- store.stop("b1")
- #expect(await until { store.shownConnection == .connecting })
- #expect(await !fake.calls.contains("stop"))
- await fake.set(.ready(.relay))
- #expect(await until { await fake.eventSubscriptionCount == 2 })
- await fake.emit(botEvent("b1", name: "Bot", rev: 2))
- #expect(await until { await fake.calls.contains("stop") })
- #expect(store.shownConnection == .online)
- #expect(store.lastError == nil)
-}
-@MainActor @Test func droppedCallIsRetriedOnlyWhenSafeToRepeat() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.ready(.direct))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- await fake.fail("stop", times: 1)
- store.stop("b1")
- #expect(await until { await fake.calls.filter { $0 == "stop" }.count == 2 })
- #expect(store.lastError == nil)
- // A new session can't be told apart from a second one, so it isn't sent twice.
- await fake.fail("newSession", times: 1)
- store.newSession("b1")
- #expect(await until { store.lastError != nil })
- #expect(await fake.calls.filter { $0 == "newSession" }.count == 1)
-}
-@MainActor @Test func offlineTapReconnectsBeforeGivingUp() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- // The first link can't reach the computer; a fresh attempt would.
- let down = FakeRemote(.failed("Can't reach"))
- let up = FakeRemote(.ready(.relay))
- var made = 0
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in
- made += 1
- return made == 1 ? down : up
- }
- defer { store.retire() }
- store.setActive(true)
- #expect(await until { store.connection == .offline("Can't reach") })
- store.stop("b1")
- #expect(await until { await up.subscribed })
- await up.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { await up.calls.contains("stop") })
- #expect(store.lastError == nil)
-}
-@MainActor @Test func permissionAnswerShowsOnTheCardAndIsSentOnce() async throws {
- let (storage, suite) = context()
- defer { UserDefaults(suiteName: suite)?.removePersistentDomain(forName: suite) }
- let fake = FakeRemote(.ready(.direct))
- let store = BotStore(computer: randomComputer("Mac"), storage: storage) { _ in fake }
- defer { store.retire() }
- store.setActive(true)
- #expect(await until { await fake.subscribed })
- await fake.emit(botEvent("b1", name: "Bot", rev: 1))
- #expect(await until { store.connection == .online })
- let card = Entry(id: "p1", seq: 1, botId: "b1", rev: 1, kind: "permission", turn: 1,
- data: EntryData(status: "pending"), createdAt: 1, updatedAt: 1)
- store.respond(card, option: "allow")
- store.respond(card, option: "deny")
- #expect(store.answering["p1"] == "allow")
- #expect(await until { await fake.calls.contains("respondPermission") })
- try await Task.sleep(for: .milliseconds(100))
- #expect(await fake.calls.filter { $0 == "respondPermission" }.count == 1)
-}
/// The computer's cues reach a call once each: one replayed after a reconnect isn't said again, even after an engine
/// switch.
diff --git a/apps/ios/WhatsNew.md b/apps/ios/WhatsNew.md
index cc836d60..22e4a7d8 100644
--- a/apps/ios/WhatsNew.md
+++ b/apps/ios/WhatsNew.md
@@ -10,6 +10,8 @@ a version without one gets a generic "Bug fixes and improvements" line. Delete o
### zh-Hant
- Bot 清單可以分成自己的分組,例如「工作」和「個人」:長按 Bot 選「移到分組」,點分組標題即可收合,收合時會顯示裡面的未讀數。
+- 遠端畫面連線更快,網路短暫中斷時也不會斷掉。
### en-US
- Sort the bot list into your own sections, like Work and Personal: touch and hold a bot and choose Move to section, then tap a section's heading to fold it, with its unread count showing while it's folded.
+- The remote screen connects faster and stays up through a brief connection drop.
diff --git a/apps/screen-linux/Cargo.lock b/apps/screen-linux/Cargo.lock
index aff12e2a..59b417fa 100644
--- a/apps/screen-linux/Cargo.lock
+++ b/apps/screen-linux/Cargo.lock
@@ -299,6 +299,7 @@ dependencies = [
"gstreamer-app",
"gstreamer-sdp",
"gstreamer-webrtc",
+ "serde",
"serde_json",
"tokio",
"tracing",
diff --git a/apps/screen-linux/Cargo.toml b/apps/screen-linux/Cargo.toml
index d166d43c..d948fcea 100644
--- a/apps/screen-linux/Cargo.toml
+++ b/apps/screen-linux/Cargo.toml
@@ -16,6 +16,7 @@ gst = { package = "gstreamer", version = "0.25" }
gst-app = { package = "gstreamer-app", version = "0.25" }
gst-sdp = { package = "gstreamer-sdp", version = "0.25" }
gst-webrtc = { package = "gstreamer-webrtc", version = "0.25", features = ["v1_22"] }
+serde = { version = "1", features = ["derive"] }
serde_json = "1"
tokio = { version = "1", features = ["rt-multi-thread", "macros", "net", "io-util", "sync", "time", "process"] }
tracing = "0.1"
diff --git a/apps/screen-linux/src/candidates.rs b/apps/screen-linux/src/candidates.rs
new file mode 100644
index 00000000..f71c2933
--- /dev/null
+++ b/apps/screen-linux/src/candidates.rs
@@ -0,0 +1,261 @@
+//! Ordered ICE candidates: out from GStreamer's signaling callbacks, in from the host.
+
+use crate::Helper;
+use anyhow::{Result, anyhow, bail};
+use gst::prelude::*;
+use serde::{Deserialize, Serialize};
+use serde_json::json;
+use std::sync::{Arc, Mutex};
+
+const MAX_CANDIDATES: usize = 128;
+const MAX_CANDIDATE_BYTES: usize = 4096;
+const MAX_MEDIA_INDEX: u32 = 16;
+
+/// One candidate event on the wire, as the host's `Candidate` (this helper needs no `sdpMid`).
+#[derive(Debug, Deserialize, Serialize)]
+#[serde(tag = "type", rename_all = "camelCase")]
+pub(super) enum Candidate {
+ Candidate {
+ candidate: String,
+ #[serde(rename = "sdpMLineIndex")]
+ sdp_m_line_index: u32,
+ },
+ Complete,
+}
+
+/// One direction of the exchange: at most `MAX_CANDIDATES` valid candidates, then one completion.
+#[derive(Default)]
+pub(super) struct Candidates {
+ count: usize,
+ complete: bool,
+}
+
+impl Candidates {
+ pub(super) fn accept(&mut self, candidate: &Candidate) -> Result<()> {
+ if self.complete {
+ bail!("screen candidates already complete");
+ }
+ match candidate {
+ Candidate::Complete => self.complete = true,
+ Candidate::Candidate {
+ candidate,
+ sdp_m_line_index,
+ } => {
+ if !candidate.starts_with("candidate:")
+ || candidate.len() > MAX_CANDIDATE_BYTES
+ || candidate.contains(['\r', '\n'])
+ || *sdp_m_line_index > MAX_MEDIA_INDEX
+ {
+ bail!("invalid screen candidate");
+ }
+ if self.count >= MAX_CANDIDATES {
+ bail!("too many screen candidates");
+ }
+ self.count += 1;
+ }
+ }
+ Ok(())
+ }
+}
+
+pub(super) fn observe(webrtc: &gst::Element, id: &str, helper: &Arc<Helper>) {
+ let state = Arc::new(Mutex::new(Candidates::default()));
+ let runtime = tokio::runtime::Handle::current();
+ let emit = Arc::new({
+ let id = id.to_owned();
+ let helper = Arc::downgrade(helper);
+ move |candidate: Result<Candidate>| {
+ let Some(helper) = helper.upgrade() else {
+ return;
+ };
+ // Both callback kinds enter one FIFO synchronously; no spawned task can reorder them.
+ let mut state = state
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner);
+ let sent = candidate.and_then(|candidate| {
+ state.accept(&candidate)?;
+ helper.notify("candidate", json!({"session": id, "candidate": candidate}))
+ });
+ if let Err(error) = sent {
+ tracing::warn!(%error, "screen candidate exchange failed");
+ let _guard = runtime.enter();
+ helper.ended(&id);
+ }
+ }
+ });
+ webrtc.connect("on-ice-candidate", false, {
+ let emit = emit.clone();
+ move |values| {
+ match (values[1].get::<u32>(), values[2].get::<String>()) {
+ // webrtcbin emits an empty end-of-candidates marker for each
+ // transport. Aggregate gathering-state Complete below ends the
+ // wire exchange only after every transport has finished.
+ (Ok(_), Ok(candidate)) if candidate.is_empty() => {}
+ (Ok(sdp_m_line_index), Ok(candidate)) => emit(Ok(Candidate::Candidate {
+ candidate,
+ sdp_m_line_index,
+ })),
+ _ => emit(Err(anyhow!("malformed native ICE candidate"))),
+ }
+ None
+ }
+ });
+ webrtc.connect_notify(Some("ice-gathering-state"), move |peer, _| {
+ if peer.property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state")
+ == gst_webrtc::WebRTCICEGatheringState::Complete
+ {
+ emit(Ok(Candidate::Complete));
+ }
+ });
+}
+
+#[cfg(test)]
+mod tests {
+ #![allow(clippy::unwrap_used)]
+ use super::*;
+ use crate::stream::Session;
+ use serde_json::Value;
+ use tokio::sync::{Mutex as AsyncMutex, mpsc};
+
+ fn wire(sdp: &str) -> Candidate {
+ Candidate::Candidate {
+ candidate: sdp.to_owned(),
+ sdp_m_line_index: 0,
+ }
+ }
+
+ fn fixture() -> (Arc<Helper>, mpsc::UnboundedReceiver<Value>, Session) {
+ gst::init().unwrap();
+ let (out, events) = mpsc::unbounded_channel();
+ let helper = Arc::new(Helper {
+ portal: Arc::new(AsyncMutex::new(None)),
+ last_status: AsyncMutex::new(None),
+ sessions: crate::sessions::Sessions::default(),
+ out,
+ rt: tokio::runtime::Handle::current(),
+ });
+ let session = Session {
+ pipeline: gst::Pipeline::new(),
+ webrtc: gst::ElementFactory::make("webrtcbin").build().unwrap(),
+ incoming: Some(Mutex::default()),
+ };
+ (helper, events, session)
+ }
+
+ #[tokio::test]
+ async fn native_transport_end_markers_do_not_close_the_viewer_or_finish_other_candidates() {
+ let (helper, mut events, session) = fixture();
+ let peer = session.webrtc.clone();
+ observe(&peer, "viewer", &helper);
+ let registered = helper
+ .sessions
+ .answer("viewer".into(), "offer", async {
+ Ok((session, "answer".into()))
+ })
+ .await;
+ registered.unwrap();
+ let sdp = "candidate:1 1 UDP 1 192.0.2.1 5000 typ host";
+ peer.emit_by_name::<()>("on-ice-candidate", &[&0u32, &sdp]);
+ assert_eq!(
+ events.try_recv().unwrap()["params"]["candidate"]["candidate"],
+ sdp
+ );
+
+ // One media transport can finish while another still emits candidates.
+ peer.emit_by_name::<()>("on-ice-candidate", &[&0u32, &""]);
+ peer.emit_by_name::<()>("on-ice-candidate", &[&1u32, &""]);
+ tokio::task::yield_now().await;
+ assert!(
+ events.try_recv().is_err(),
+ "native end markers are not wire candidates or global completion"
+ );
+ let live = helper.sessions.is_live("viewer", &peer).await;
+ assert!(live, "a native gathering end marker must not stop media");
+ peer.emit_by_name::<()>("on-ice-candidate", &[&1u32, &sdp]);
+ assert_eq!(
+ events.try_recv().unwrap()["params"]["candidate"]["sdpMLineIndex"],
+ 1
+ );
+ helper.sessions.close_all().await;
+ }
+
+ #[tokio::test]
+ async fn native_gathering_exports_one_global_completion_after_all_candidates() {
+ let (helper, mut events, session) = fixture();
+ session.pipeline.add(&session.webrtc).unwrap();
+ session.pipeline.set_state(gst::State::Ready).unwrap();
+ let peer = session.webrtc.clone();
+ observe(&peer, "viewer", &helper);
+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\
+ a=group:BUNDLE 0\r\nm=application 9 UDP/DTLS/SCTP webrtc-datachannel\r\n\
+ c=IN IP4 0.0.0.0\r\na=mid:0\r\na=setup:actpass\r\na=sctp-port:5000\r\n\
+ a=ice-ufrag:test\r\na=ice-pwd:testpasswordforthedataoffer\r\n\
+ a=fingerprint:sha-256 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00\r\n";
+ let answered = session.answer(offer).await;
+ let answer = answered.unwrap();
+ let registered = helper
+ .sessions
+ .answer("viewer".into(), offer, async { Ok((session, answer)) })
+ .await;
+ registered.unwrap();
+ let gathering = async {
+ let mut count = 0;
+ loop {
+ let received = events.recv().await;
+ let event = received.unwrap();
+ assert_eq!(
+ event["method"], "candidate",
+ "gathering must not close the media session"
+ );
+ let candidate = &event["params"]["candidate"];
+ if candidate["type"] == "complete" {
+ break;
+ }
+ assert_eq!(candidate["type"], "candidate");
+ assert!(
+ candidate["candidate"]
+ .as_str()
+ .unwrap()
+ .starts_with("candidate:")
+ );
+ count += 1;
+ }
+ assert!(count > 0, "the native peer must actually gather candidates");
+ };
+ let gathered = tokio::time::timeout(std::time::Duration::from_secs(10), gathering).await;
+ gathered.unwrap();
+ assert_eq!(
+ peer.property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state"),
+ gst_webrtc::WebRTCICEGatheringState::Complete
+ );
+ tokio::task::yield_now().await;
+ assert!(
+ events.try_recv().is_err(),
+ "completion is exported once, after every native candidate"
+ );
+ let live = helper.sessions.is_live("viewer", &peer).await;
+ assert!(live, "finishing native gathering must not close media");
+ helper.sessions.close_all().await;
+ }
+
+ #[test]
+ fn native_end_markers_do_not_relax_wire_candidate_validation() {
+ for sdp in ["", " ", "candidate:1\r\n"] {
+ let mut stream = Candidates::default();
+ assert!(stream.accept(&wire(sdp)).is_err());
+ }
+ }
+
+ #[test]
+ fn completion_follows_all_candidates_and_bounds_are_enforced() {
+ let mut stream = Candidates::default();
+ let candidate = wire("candidate:1 1 UDP 1 192.0.2.1 5000 typ relay");
+ for _ in 0..128 {
+ stream.accept(&candidate).unwrap();
+ }
+ assert!(stream.accept(&candidate).is_err());
+ stream.accept(&Candidate::Complete).unwrap();
+ assert!(stream.accept(&candidate).is_err());
+ assert!(stream.accept(&Candidate::Complete).is_err());
+ }
+}
diff --git a/apps/screen-linux/src/connection.rs b/apps/screen-linux/src/connection.rs
new file mode 100644
index 00000000..a99a3900
--- /dev/null
+++ b/apps/screen-linux/src/connection.rs
@@ -0,0 +1,159 @@
+//! Native ICE state notifications run on GStreamer's threads, outside Tokio.
+
+use crate::Helper;
+use gst::prelude::*;
+use std::sync::Arc;
+
+pub(super) fn observe(peer: &gst::Element, id: String, helper: &Arc<Helper>) {
+ let runtime = tokio::runtime::Handle::current();
+ let helper = Arc::downgrade(helper);
+ peer.connect_notify(Some("ice-connection-state"), move |peer, _| {
+ let state = peer.property::<gst_webrtc::WebRTCICEConnectionState>("ice-connection-state");
+ let Some(helper) = helper.upgrade() else {
+ return;
+ };
+ let id = id.clone();
+ let peer = peer.clone();
+ runtime.spawn(async move { changed(&helper, &id, &peer, state).await });
+ });
+}
+
+async fn changed(
+ helper: &Arc<Helper>,
+ id: &str,
+ peer: &gst::Element,
+ state: gst_webrtc::WebRTCICEConnectionState,
+) {
+ use gst_webrtc::WebRTCICEConnectionState as Ice;
+ match state {
+ Ice::Connected | Ice::Completed => helper.connected(id, peer).await,
+ Ice::Failed | Ice::Closed => helper.ended_peer(id, peer).await,
+ // A disconnected connection may recover without renegotiating.
+ _ => {}
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ #![allow(clippy::unwrap_used)]
+ use super::*;
+ use crate::stream::Session;
+ use tokio::sync::{Mutex, mpsc};
+
+ fn session() -> Session {
+ gst::init().unwrap();
+ Session {
+ pipeline: gst::Pipeline::new(),
+ webrtc: gst::ElementFactory::make("webrtcbin").build().unwrap(),
+ incoming: Some(std::sync::Mutex::default()),
+ }
+ }
+
+ #[tokio::test]
+ async fn connection_notifications_ignore_closed_and_replaced_native_peers() {
+ let (out, mut events) = mpsc::unbounded_channel();
+ let helper = Arc::new(Helper {
+ portal: Arc::new(Mutex::new(None)),
+ last_status: Mutex::new(None),
+ sessions: crate::sessions::Sessions::default(),
+ out,
+ rt: tokio::runtime::Handle::current(),
+ });
+ let original = session();
+ let old_peer = original.webrtc.clone();
+ let answer = helper
+ .sessions
+ .answer("viewer".into(), "offer", async {
+ Ok((original, "answer".into()))
+ })
+ .await;
+ answer.unwrap();
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Connected,
+ )
+ .await;
+ let connected = events.try_recv().unwrap();
+ assert_eq!(connected["method"], "session");
+ assert_eq!(connected["params"]["state"], "connected");
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Disconnected,
+ )
+ .await;
+ assert!(
+ events.try_recv().is_err(),
+ "temporary ICE disconnects keep media alive"
+ );
+ helper.sessions.close("viewer").await;
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Completed,
+ )
+ .await;
+ assert!(events.try_recv().is_err());
+ let replacement = session();
+ let peer = replacement.webrtc.clone();
+ let answer = helper
+ .sessions
+ .answer("viewer".into(), "offer", async {
+ Ok((replacement, "answer".into()))
+ })
+ .await;
+ answer.unwrap();
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Connected,
+ )
+ .await;
+ assert!(
+ events.try_recv().is_err(),
+ "late callbacks belong only to their native peer"
+ );
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Failed,
+ )
+ .await;
+ let still_live = helper.sessions.is_live("viewer", &peer).await;
+ assert!(still_live, "an old peer cannot close its replacement");
+ changed(
+ &helper,
+ "viewer",
+ &old_peer,
+ gst_webrtc::WebRTCICEConnectionState::Closed,
+ )
+ .await;
+ let still_live = helper.sessions.is_live("viewer", &peer).await;
+ assert!(still_live);
+ assert!(events.try_recv().is_err());
+ changed(
+ &helper,
+ "viewer",
+ &peer,
+ gst_webrtc::WebRTCICEConnectionState::Completed,
+ )
+ .await;
+ assert_eq!(events.try_recv().unwrap()["params"]["state"], "connected");
+ changed(
+ &helper,
+ "viewer",
+ &peer,
+ gst_webrtc::WebRTCICEConnectionState::Closed,
+ )
+ .await;
+ assert_eq!(events.try_recv().unwrap()["params"]["state"], "closed");
+ let still_live = helper.sessions.is_live("viewer", &peer).await;
+ assert!(!still_live);
+ }
+}
diff --git a/apps/screen-linux/src/link.rs b/apps/screen-linux/src/link.rs
new file mode 100644
index 00000000..7df8ef12
--- /dev/null
+++ b/apps/screen-linux/src/link.rs
@@ -0,0 +1,147 @@
+//! Requests belong to one host socket; none may outlive that connection.
+
+use crate::Outbox;
+use anyhow::{Result, anyhow};
+use serde_json::{Value, json};
+use std::future::Future;
+use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
+use tokio::net::UnixStream;
+use tokio::sync::mpsc;
+use tokio::task::JoinSet;
+
+pub(super) async fn serve<F, R>(
+ sock: UnixStream,
+ out: Outbox,
+ mut outbox: mpsc::UnboundedReceiver<Value>,
+ handle: F,
+) -> Result<()>
+where
+ F: Fn(String, Value) -> R,
+ R: Future<Output = Result<Value>> + Send + 'static,
+{
+ let (rd, mut wr) = sock.into_split();
+ let mut writer = tokio::spawn(async move {
+ while let Some(msg) = outbox.recv().await {
+ wr.write_all(format!("{msg}\n").as_bytes()).await?;
+ }
+ Ok::<_, std::io::Error>(())
+ });
+ let mut requests = JoinSet::new();
+ let mut lines = BufReader::new(rd).lines();
+ let result = loop {
+ tokio::select! {
+ line = lines.next_line() => {
+ let line = match line {
+ Ok(Some(line)) => line,
+ Ok(None) => break Ok(()),
+ Err(error) => break Err(error.into()),
+ };
+ dispatch(&line, &out, &handle, &mut requests);
+ }
+ Some(completed) = requests.join_next(), if !requests.is_empty() => {
+ if let Err(error) = completed { tracing::warn!(%error, "screen request task failed"); }
+ }
+ completed = &mut writer => {
+ break match completed {
+ Ok(Err(error)) => Err(error.into()),
+ Err(error) => Err(error.into()),
+ Ok(Ok(())) => Err(anyhow!("host writer disconnected")),
+ };
+ }
+ }
+ };
+ requests.abort_all();
+ loop {
+ let completed = requests.join_next().await;
+ if completed.is_none() {
+ break;
+ }
+ }
+ writer.abort();
+ result
+}
+
+fn dispatch<F, R>(line: &str, out: &Outbox, handle: &F, requests: &mut JoinSet<()>)
+where
+ F: Fn(String, Value) -> R,
+ R: Future<Output = Result<Value>> + Send + 'static,
+{
+ let Ok(msg) = serde_json::from_str::<Value>(line) else {
+ return;
+ };
+ let (Some(method), Some(id)) = (msg["method"].as_str(), msg.get("id").cloned()) else {
+ return;
+ };
+ let response = handle(method.to_owned(), msg["params"].clone());
+ let out = out.clone();
+ requests.spawn(async move {
+ let result = response.await;
+ let reply = match result {
+ Ok(result) => json!({"jsonrpc": "2.0", "id": id, "result": result}),
+ Err(error) => json!({"jsonrpc": "2.0", "id": id, "error": {"code": -32000, "message": format!("{error:#}")}}),
+ };
+ // A closed writer is handled by the enclosing connection loop.
+ let _ = out.send(reply);
+ });
+}
+
+#[cfg(test)]
+mod tests {
+ #![allow(clippy::unwrap_used)]
+ use super::*;
+ use std::sync::{
+ Arc,
+ atomic::{AtomicBool, Ordering},
+ };
+ use std::time::Duration;
+ use tokio::sync::Notify;
+
+ struct Dropped(Arc<AtomicBool>);
+ impl Drop for Dropped {
+ fn drop(&mut self) {
+ self.0.store(true, Ordering::SeqCst);
+ }
+ }
+
+ #[tokio::test]
+ async fn disconnect_cancels_pending_answers_before_session_cleanup() {
+ for invalid_line in [false, true] {
+ let (sock, mut host) = UnixStream::pair().unwrap();
+ let (out, outbox) = mpsc::unbounded_channel();
+ let started = Arc::new(Notify::new());
+ let dropped = Arc::new(AtomicBool::new(false));
+ let serving = tokio::spawn(serve(sock, out, outbox, {
+ let started = started.clone();
+ let dropped = dropped.clone();
+ move |_, _| {
+ let started = started.clone();
+ let dropped = dropped.clone();
+ async move {
+ let _guard = Dropped(dropped);
+ started.notify_one();
+ std::future::pending::<Result<Value>>().await
+ }
+ }
+ }));
+ let sent = host
+ .write_all(b"{\"id\":1,\"method\":\"answer\",\"params\":{}}\n")
+ .await;
+ sent.unwrap();
+ let starting = tokio::time::timeout(Duration::from_secs(1), started.notified()).await;
+ starting.unwrap();
+ if invalid_line {
+ let sent = host.write_all(b"\xff\n").await;
+ sent.unwrap();
+ } else {
+ drop(host);
+ }
+ let ended = tokio::time::timeout(Duration::from_secs(1), serving).await;
+ let result = ended.unwrap().unwrap();
+ assert_eq!(result.is_err(), invalid_line);
+ assert!(
+ dropped.load(Ordering::SeqCst),
+ "negotiation cannot continue after cleanup"
+ );
+ }
+ }
+}
diff --git a/apps/screen-linux/src/main.rs b/apps/screen-linux/src/main.rs
index deaf54d4..45e4ee27 100644
--- a/apps/screen-linux/src/main.rs
+++ b/apps/screen-linux/src/main.rs
@@ -6,18 +6,21 @@
//! GStreamer `webrtcbin`, and the accessibility tree over AT-SPI.
mod a11y;
+mod candidates;
+mod connection;
mod input;
+mod link;
+mod negotiation;
mod portal;
+mod sessions;
mod stream;
mod video;
use anyhow::{Context, Result, anyhow};
use serde_json::{Value, json};
-use std::collections::HashMap;
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
-use tokio::io::{AsyncBufReadExt, AsyncWriteExt, BufReader};
use tokio::net::UnixStream;
use tokio::sync::{Mutex, mpsc};
@@ -34,7 +37,7 @@ pub type Outbox = mpsc::UnboundedSender<Value>;
pub struct Helper {
portal: Arc<Mutex<Option<portal::Portal>>>,
last_status: Mutex<Option<Value>>,
- sessions: Mutex<HashMap<String, stream::Session>>,
+ sessions: sessions::Sessions,
out: Outbox,
/// GStreamer calls `ended` from its own threads, outside the Tokio runtime.
rt: tokio::runtime::Handle,
@@ -66,21 +69,11 @@ async fn main() -> Result<()> {
}
async fn serve(sock: UnixStream, portal: &Arc<Mutex<Option<portal::Portal>>>) -> Result<()> {
- let (rd, mut wr) = sock.into_split();
- let (out, mut outbox) = mpsc::unbounded_channel::<Value>();
- let writer = tokio::spawn(async move {
- while let Some(msg) = outbox.recv().await {
- let mut line = msg.to_string();
- line.push('\n');
- if wr.write_all(line.as_bytes()).await.is_err() {
- break;
- }
- }
- });
+ let (out, outbox) = mpsc::unbounded_channel::<Value>();
let helper = Arc::new(Helper {
portal: portal.clone(),
last_status: Mutex::new(None),
- sessions: Mutex::default(),
+ sessions: sessions::Sessions::default(),
out: out.clone(),
rt: tokio::runtime::Handle::current(),
});
@@ -92,38 +85,14 @@ async fn serve(sock: UnixStream, portal: &Arc<Mutex<Option<portal::Portal>>>) ->
watched.send_status().await;
}
});
- let mut lines = BufReader::new(rd).lines();
- let result = loop {
- let line = match lines.next_line().await {
- Ok(Some(line)) => line,
- Ok(None) => break Ok(()),
- Err(error) => break Err(error.into()),
- };
- let Ok(msg) = serde_json::from_str::<Value>(&line) else {
- continue;
- };
- let (Some(method), Some(id)) = (
- msg["method"].as_str().map(str::to_owned),
- msg.get("id").cloned(),
- ) else {
- continue;
- };
- let helper = helper.clone();
- tokio::spawn(async move {
- let reply = match handle(&helper, &method, &msg["params"]).await {
- Ok(result) => json!({"jsonrpc": "2.0", "id": id, "result": result}),
- Err(error) => {
- json!({"jsonrpc": "2.0", "id": id, "error": {"code": -32000, "message": format!("{error:#}")}})
- }
- };
- let _ = helper.out.send(reply);
- });
- };
- writer.abort();
+ let serving = helper.clone();
+ let result = link::serve(sock, out, outbox, move |method, params| {
+ let helper = serving.clone();
+ async move { handle(&helper, &method, &params).await }
+ })
+ .await;
status_task.abort();
- for (_, s) in helper.sessions.lock().await.drain() {
- s.close();
- }
+ helper.sessions.close_all().await;
result
}
@@ -140,9 +109,7 @@ async fn handle(h: &Arc<Helper>, method: &str, p: &Value) -> Result<Value> {
if previous.is_active() {
previous.session().close().await?;
}
- for (_, session) in h.sessions.lock().await.drain() {
- session.close();
- }
+ h.sessions.close_all().await;
}
*portal = Some(next);
}
@@ -152,16 +119,12 @@ async fn handle(h: &Arc<Helper>, method: &str, p: &Value) -> Result<Value> {
}
// Close requests remain valid before setup; all other methods require a granted session.
if method == "closeAll" {
- for (_, s) in h.sessions.lock().await.drain() {
- s.close();
- }
+ h.sessions.close_all().await;
return Ok(json!({}));
}
if method == "close" {
- if let Some(id) = p["session"].as_str()
- && let Some(s) = h.sessions.lock().await.remove(id)
- {
- s.close();
+ if let Some(id) = p["session"].as_str() {
+ h.sessions.close(id).await;
}
return Ok(json!({}));
}
@@ -176,16 +139,25 @@ async fn handle(h: &Arc<Helper>, method: &str, p: &Value) -> Result<Value> {
let sdp = p["sdp"]
.as_str()
.ok_or_else(|| anyhow!("sdp is required"))?;
- if let Some(s) = h.sessions.lock().await.get(&id) {
- return Ok(json!({"sdp": s.answer(sdp).await?}));
- }
- let d = display()?;
- let fd = portal.pipewire_fd().await?;
- let session = stream::Session::new(id.clone(), &d, fd, h.clone(), p)?;
- let answer = session.answer(sdp).await?;
- h.sessions.lock().await.insert(id, session);
+ let creating = id.clone();
+ let answer = h
+ .sessions
+ .answer(id, sdp, async {
+ let d = display()?;
+ let fd = portal.pipewire_fd().await?;
+ let session = stream::Session::new(creating, &d, fd, h, p)?;
+ let answer = session.answer(sdp).await?;
+ Ok((session, answer))
+ })
+ .await?;
json!({"sdp": answer})
}
+ "candidate" => {
+ let id = p["session"].as_str().context("missing session")?;
+ let candidate = serde_json::from_value(p["candidate"].clone())?;
+ h.sessions.candidate(id, &candidate).await?;
+ json!({})
+ }
"screenshot" => {
let (w, h2) = (
p["width"].as_u64().unwrap_or(1280),
@@ -218,27 +190,44 @@ async fn handle(h: &Arc<Helper>, method: &str, p: &Value) -> Result<Value> {
impl Helper {
async fn portal(&self) -> Result<portal::Portal> {
- self.portal
- .lock()
- .await
+ let portal = self.portal.lock().await;
+ portal
.clone()
.filter(portal::Portal::is_active)
.ok_or_else(|| anyhow!("Set up Computer access in Codync on this computer first."))
}
async fn send_status(&self) {
- let status = self.portal.lock().await.as_ref().map_or_else(
- || json!({"platform": "linux", "permissionApp": "Codync Screen", "capture": false, "input": false, "displays": []}),
+ let portal = self.portal.lock().await;
+ let status = portal.as_ref().map_or_else(
+ || json!({"platform": "linux", "permissionApp": "Codync Screen", "version": env!("CARGO_PKG_VERSION"), "trickle": true, "capture": false, "input": false, "displays": []}),
portal::Portal::status,
);
+ drop(portal);
let mut last = self.last_status.lock().await;
if last.as_ref() == Some(&status) {
return;
}
*last = Some(status.clone());
- let _ = self
- .out
- .send(json!({"jsonrpc": "2.0", "method": "status", "params": status}));
+ // Fails only once the host link is gone; the next link starts a new Helper that sends its status first.
+ let _ = self.notify("status", status);
+ }
+
+ /// One notification to the host; fails once the host link is gone.
+ fn notify(&self, method: &str, params: Value) -> Result<()> {
+ let mut message = json!({"jsonrpc": "2.0", "method": method});
+ message["params"] = params;
+ self.out
+ .send(message)
+ .map_err(|_| anyhow!("the host link closed"))
+ }
+
+ /// Tells the host a session's state. It fails only once the host link is gone, and the host drops every session
+ /// of a helper whose link ended, so a lost notice changes nothing there.
+ fn notify_session(&self, id: &str, state: &str) {
+ if let Err(error) = self.notify("session", json!({"session": id, "state": state})) {
+ tracing::debug!(%error, session = id, state, "session notice not sent");
+ }
}
/// A session's WebRTC connection ended on its own.
@@ -246,10 +235,23 @@ impl Helper {
let this = self.clone();
let id = id.to_owned();
self.rt.spawn(async move {
- if let Some(s) = this.sessions.lock().await.remove(&id) {
- s.close();
- }
- let _ = this.out.send(json!({"jsonrpc": "2.0", "method": "session", "params": {"session": id, "state": "closed"}}));
+ this.sessions.close(&id).await;
+ this.notify_session(&id, "closed");
});
}
+
+ async fn connected(&self, id: &str, peer: &gst::Element) {
+ let live = self.sessions.is_live(id, peer).await;
+ if live {
+ self.notify_session(id, "connected");
+ }
+ }
+
+ async fn ended_peer(&self, id: &str, peer: &gst::Element) {
+ self.sessions
+ .close_peer(id, peer, || {
+ self.notify_session(id, "closed");
+ })
+ .await;
+ }
}
diff --git a/apps/screen-linux/src/negotiation.rs b/apps/screen-linux/src/negotiation.rs
new file mode 100644
index 00000000..34388dce
--- /dev/null
+++ b/apps/screen-linux/src/negotiation.rs
@@ -0,0 +1,152 @@
+//! WebRTC SDP and ICE negotiation.
+
+use crate::candidates::Candidate;
+use crate::stream::Session;
+use anyhow::{Context, Result, anyhow, bail};
+use gst::prelude::*;
+use std::time::Duration;
+use tokio::sync::oneshot;
+
+impl Session {
+ /// Trickle sends the original answer immediately; complete-SDP peers wait for gathering.
+ pub async fn answer(&self, offer: &str) -> Result<String> {
+ let sdp = gst_sdp::SDPMessage::parse_buffer(offer.as_bytes())
+ .map_err(|_| anyhow!("bad offer"))?;
+ let offer =
+ gst_webrtc::WebRTCSessionDescription::new(gst_webrtc::WebRTCSDPType::Offer, sdp);
+ let (p, done) = promise();
+ self.webrtc
+ .emit_by_name::<()>("set-remote-description", &[&offer, &p]);
+ done.await?;
+ let (p, done) = promise();
+ self.webrtc
+ .emit_by_name::<()>("create-answer", &[&None::<gst::Structure>, &p]);
+ let reply = done.await?;
+ let reply = reply.ok_or_else(|| anyhow!("no answer"))?;
+ let answer = reply
+ .get::<gst_webrtc::WebRTCSessionDescription>("answer")
+ .map_err(|_| anyhow!("no answer"))?;
+ let (p, done) = promise();
+ self.webrtc
+ .emit_by_name::<()>("set-local-description", &[&answer, &p]);
+ done.await?;
+ // Negotiate before the encoder chooses fixed caps. Starting first can
+ // produce High-profile caps that reject a phone's Baseline offer.
+ self.pipeline
+ .set_state(gst::State::Playing)
+ .context("starting the video pipeline")?;
+ if self.incoming.is_some() {
+ return answer
+ .sdp()
+ .as_text()
+ .map_err(|_| anyhow!("couldn't write the answer"));
+ }
+ // Complete-SDP clients need all transports, including late TURN TLS/443 candidates.
+ for _ in 0..200 {
+ if self
+ .webrtc
+ .property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state")
+ == gst_webrtc::WebRTCICEGatheringState::Complete
+ {
+ break;
+ }
+ tokio::time::sleep(Duration::from_millis(50)).await;
+ }
+ let local = self
+ .webrtc
+ .property::<Option<gst_webrtc::WebRTCSessionDescription>>("local-description");
+ let local = local.unwrap_or(answer);
+ local
+ .sdp()
+ .as_text()
+ .map_err(|_| anyhow!("couldn't write the answer"))
+ }
+
+ pub fn candidate(&self, candidate: &Candidate) -> Result<()> {
+ let mut incoming = self
+ .incoming
+ .as_ref()
+ .context("screen session does not support trickle ICE")?
+ .lock()
+ .unwrap_or_else(std::sync::PoisonError::into_inner);
+ incoming.accept(candidate)?;
+ match candidate {
+ // 1.22 supports end-of-candidates without add-ice-candidate-full.
+ Candidate::Complete => self
+ .webrtc
+ .emit_by_name::<()>("add-ice-candidate", &[&0u32, &None::<String>]),
+ Candidate::Candidate {
+ candidate,
+ sdp_m_line_index,
+ } => self
+ .webrtc
+ .emit_by_name::<()>("add-ice-candidate", &[sdp_m_line_index, candidate]),
+ }
+ Ok(())
+ }
+}
+
+/// Convert WebRTC's ICE URL syntax to GStreamer's URI syntax without exposing credentials.
+pub(super) fn ice_uri(
+ url: &str,
+ username: Option<&str>,
+ credential: Option<&str>,
+) -> Result<String> {
+ let (scheme, address) = url.split_once(':').context("invalid ICE URL")?;
+ if scheme == "stun" {
+ return Ok(format!("stun://{address}"));
+ }
+ if !matches!(scheme, "turn" | "turns") {
+ bail!("unsupported ICE scheme");
+ }
+ let username = username.context("missing TURN username")?;
+ let credential = credential.context("missing TURN credential")?;
+ let escape = |s: &str| -> String {
+ use std::fmt::Write as _;
+ s.bytes().fold(String::new(), |mut out, byte| {
+ if byte.is_ascii_alphanumeric() || b"-._~".contains(&byte) {
+ out.push(char::from(byte));
+ } else {
+ let _ = write!(out, "%{byte:02X}");
+ }
+ out
+ })
+ };
+ Ok(format!(
+ "{scheme}://{}:{}@{address}",
+ escape(username),
+ escape(credential)
+ ))
+}
+
+fn promise() -> (gst::Promise, oneshot::Receiver<Option<gst::Structure>>) {
+ let (tx, rx) = oneshot::channel();
+ let p = gst::Promise::with_change_func(move |reply| {
+ let _ = tx.send(reply.ok().flatten().map(ToOwned::to_owned));
+ });
+ (p, rx)
+}
+
+#[cfg(test)]
+mod tests {
+ #![allow(clippy::unwrap_used)]
+ use super::*;
+
+ #[test]
+ fn turn_uri_preserves_transport_and_escapes_credentials() {
+ assert_eq!(
+ ice_uri(
+ "turns:turn.cloudflare.com:443?transport=tcp",
+ Some("a:b"),
+ Some("c+/@")
+ )
+ .unwrap(),
+ "turns://a%3Ab:c%2B%2F%[email protected]:443?transport=tcp"
+ );
+ assert_eq!(
+ ice_uri("stun:stun.cloudflare.com:3478", None, None).unwrap(),
+ "stun://stun.cloudflare.com:3478"
+ );
+ assert!(ice_uri("turn:turn.cloudflare.com:3478", None, None).is_err());
+ }
+}
diff --git a/apps/screen-linux/src/portal.rs b/apps/screen-linux/src/portal.rs
index 11d2aff4..19ad025a 100644
--- a/apps/screen-linux/src/portal.rs
+++ b/apps/screen-linux/src/portal.rs
@@ -136,7 +136,9 @@ impl Portal {
json!({
"platform": "linux",
"permissionApp": "Codync Screen",
+ "version": env!("CARGO_PKG_VERSION"),
"capture": self.is_active(),
+ "trickle": true,
"input": self.has_input(),
"displays": self.0.displays.iter().map(|d| json!({
"id": d.id,
diff --git a/apps/screen-linux/src/sessions.rs b/apps/screen-linux/src/sessions.rs
new file mode 100644
index 00000000..19ebc678
--- /dev/null
+++ b/apps/screen-linux/src/sessions.rs
@@ -0,0 +1,243 @@
+//! Serialize registration with close so a late answer cannot resurrect capture.
+
+use crate::candidates::Candidate;
+use crate::stream::Session;
+use anyhow::{Context, Result, bail};
+use std::collections::HashMap;
+use std::future::Future;
+use std::sync::Arc;
+use tokio::sync::Mutex;
+
+enum State {
+ Preparing,
+ Live(Session),
+ Closed,
+}
+
+#[derive(Default)]
+pub(super) struct Sessions(Mutex<HashMap<String, Arc<Mutex<State>>>>);
+
+impl Sessions {
+ pub(super) async fn answer(
+ &self,
+ id: String,
+ offer: &str,
+ create: impl Future<Output = Result<(Session, String)>>,
+ ) -> Result<String> {
+ let slot = {
+ let mut sessions = self.0.lock().await;
+ sessions
+ .entry(id)
+ .or_insert_with(|| Arc::new(Mutex::new(State::Preparing)))
+ .clone()
+ };
+ // Only this viewer waits for creation; other viewers can negotiate and close concurrently.
+ let mut state = slot.lock().await;
+ match &*state {
+ State::Live(session) => return session.answer(offer).await,
+ State::Closed => bail!("screen session closed"),
+ State::Preparing => {}
+ }
+ let created = create.await;
+ match created {
+ Ok((session, answer)) => {
+ *state = State::Live(session);
+ Ok(answer)
+ }
+ Err(error) => {
+ *state = State::Closed;
+ Err(error)
+ }
+ }
+ }
+
+ pub(super) async fn candidate(&self, id: &str, candidate: &Candidate) -> Result<()> {
+ let slot = {
+ let sessions = self.0.lock().await;
+ sessions.get(id).context("unknown screen session")?.clone()
+ };
+ let state = slot.lock().await;
+ let State::Live(session) = &*state else {
+ bail!("screen session is not ready");
+ };
+ session.candidate(candidate)
+ }
+
+ pub(super) async fn close(&self, id: &str) {
+ let slot = {
+ let mut sessions = self.0.lock().await;
+ sessions.remove(id)
+ };
+ if let Some(slot) = slot {
+ Self::close_slot(&slot).await;
+ }
+ }
+
+ pub(super) async fn is_live(&self, id: &str, peer: &gst::Element) -> bool {
+ let slot = {
+ let sessions = self.0.lock().await;
+ let Some(slot) = sessions.get(id) else {
+ return false;
+ };
+ slot.clone()
+ };
+ let state = slot.lock().await;
+ matches!(&*state, State::Live(session) if session.webrtc == *peer)
+ }
+
+ pub(super) async fn close_all(&self) {
+ let slots = {
+ let mut sessions = self.0.lock().await;
+ sessions.drain().map(|(_, slot)| slot).collect::<Vec<_>>()
+ };
+ for slot in slots {
+ Self::close_slot(&slot).await;
+ }
+ }
+
+ pub(super) async fn close_peer(&self, id: &str, peer: &gst::Element, closed: impl FnOnce()) {
+ let slot = {
+ let sessions = self.0.lock().await;
+ let Some(slot) = sessions.get(id) else { return };
+ slot.clone()
+ };
+ let mut state = slot.lock().await;
+ if !matches!(&*state, State::Live(session) if session.webrtc == *peer) {
+ return;
+ }
+ if let State::Live(session) = std::mem::replace(&mut *state, State::Closed) {
+ session.close();
+ }
+ let mut sessions = self.0.lock().await;
+ if sessions
+ .get(id)
+ .is_some_and(|current| Arc::ptr_eq(current, &slot))
+ {
+ sessions.remove(id);
+ // Emit before the ID can be registered again, preserving close/replacement ordering.
+ closed();
+ }
+ }
+
+ async fn close_slot(slot: &Mutex<State>) {
+ let mut state = slot.lock().await;
+ if let State::Live(session) = std::mem::replace(&mut *state, State::Closed) {
+ session.close();
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ #![allow(clippy::unwrap_used)]
+ use super::*;
+ use gst::prelude::*;
+ use std::sync::Arc;
+ use std::time::Duration;
+ use tokio::sync::oneshot;
+
+ fn session() -> (Session, gst::Pipeline) {
+ gst::init().unwrap();
+ let pipeline = gst::parse::launch("videotestsrc is-live=true ! fakesink")
+ .unwrap()
+ .downcast::<gst::Pipeline>()
+ .unwrap();
+ pipeline.set_state(gst::State::Playing).unwrap();
+ let session = Session {
+ pipeline: pipeline.clone(),
+ webrtc: gst::ElementFactory::make("webrtcbin").build().unwrap(),
+ incoming: Some(std::sync::Mutex::default()),
+ };
+ (session, pipeline)
+ }
+
+ #[tokio::test]
+ async fn close_during_answer_waits_and_stops_the_late_pipeline() {
+ for all in [false, true] {
+ let (session, pipeline) = session();
+ let sessions = Arc::new(Sessions::default());
+ let (started, starting) = oneshot::channel();
+ let (finish, finishing) = oneshot::channel();
+ let answering = tokio::spawn({
+ let sessions = sessions.clone();
+ async move {
+ sessions
+ .answer("viewer".into(), "offer", async {
+ started.send(()).unwrap();
+ let finished = finishing.await;
+ finished.unwrap();
+ Ok((session, "answer".into()))
+ })
+ .await
+ }
+ });
+ let started = starting.await;
+ started.unwrap();
+ let mut closing = tokio::spawn({
+ let sessions = sessions.clone();
+ async move {
+ if all {
+ sessions.close_all().await;
+ } else {
+ sessions.close("viewer").await;
+ }
+ }
+ });
+ let early = tokio::time::timeout(Duration::from_millis(25), &mut closing).await;
+ assert!(
+ early.is_err(),
+ "close cannot acknowledge while an answer can still register"
+ );
+ finish.send(()).unwrap();
+ let answer = answering.await;
+ assert_eq!(answer.unwrap().unwrap(), "answer");
+ let closed = closing.await;
+ closed.unwrap();
+ assert_eq!(pipeline.current_state(), gst::State::Null);
+ let candidate = sessions.candidate("viewer", &Candidate::Complete).await;
+ assert!(
+ candidate.is_err(),
+ "the closed session was not registered later"
+ );
+ }
+ }
+
+ #[tokio::test]
+ async fn a_pending_answer_does_not_block_another_viewer_from_connecting_or_closing() {
+ let (slow, _) = session();
+ let sessions = Arc::new(Sessions::default());
+ let (started, starting) = oneshot::channel();
+ let (finish, finishing) = oneshot::channel();
+ let pending = tokio::spawn({
+ let sessions = sessions.clone();
+ async move {
+ sessions
+ .answer("slow".into(), "offer", async {
+ started.send(()).unwrap();
+ let finished = finishing.await;
+ finished.unwrap();
+ Ok((slow, "slow answer".into()))
+ })
+ .await
+ }
+ });
+ let started = starting.await;
+ started.unwrap();
+ let (fast, pipeline) = session();
+ let answered = tokio::time::timeout(
+ Duration::from_secs(1),
+ sessions.answer("fast".into(), "offer", async {
+ Ok((fast, "fast answer".into()))
+ }),
+ )
+ .await;
+ assert_eq!(answered.unwrap().unwrap(), "fast answer");
+ let closed = tokio::time::timeout(Duration::from_secs(1), sessions.close("fast")).await;
+ closed.unwrap();
+ assert_eq!(pipeline.current_state(), gst::State::Null);
+ finish.send(()).unwrap();
+ let finished = pending.await;
+ finished.unwrap().unwrap();
+ sessions.close_all().await;
+ }
+}
diff --git a/apps/screen-linux/src/stream.rs b/apps/screen-linux/src/stream.rs
index 21180562..2374e204 100644
--- a/apps/screen-linux/src/stream.rs
+++ b/apps/screen-linux/src/stream.rs
@@ -1,6 +1,7 @@
//! Video: a PipeWire monitor stream encoded as H.264 and sent over GStreamer
-//! `webrtcbin` (non-trickle, like the macOS helper), plus one-off JPEG stills.
+//! `webrtcbin` with negotiated trickle ICE, plus one-off JPEG stills.
+use crate::negotiation::ice_uri;
use crate::portal::Display;
use crate::video::{h264_format, h264_payload};
use crate::{Helper, input};
@@ -10,8 +11,7 @@ use gst::prelude::*;
use serde_json::{Value, json};
use std::os::fd::{IntoRawFd, OwnedFd};
use std::sync::{Arc, Mutex};
-use std::time::Duration;
-use tokio::sync::{mpsc, oneshot};
+use tokio::sync::mpsc;
/// The PipeWire source for one portal stream. Only properties every supported PipeWire has
/// (Ubuntu 24.04's 1.0 onward) go in the builder: an unknown one panics there.
@@ -41,8 +41,10 @@ const ENCODERS: &[&str] = &[
const BITRATE_KBPS: u32 = 8000;
pub struct Session {
- pipeline: gst::Pipeline,
- webrtc: gst::Element,
+ pub(super) pipeline: gst::Pipeline,
+ pub(super) webrtc: gst::Element,
+ /// The phone's candidates so far; `None` for a complete-SDP session.
+ pub(super) incoming: Option<Mutex<crate::candidates::Candidates>>,
}
impl Session {
@@ -50,7 +52,7 @@ impl Session {
id: String,
display: &Display,
fd: OwnedFd,
- helper: Arc<Helper>,
+ helper: &Arc<Helper>,
config: &Value,
) -> Result<Self> {
let payload = h264_payload(config["sdp"].as_str().context("missing offer")?)?;
@@ -178,66 +180,19 @@ impl Session {
None
}
});
- webrtc.connect_notify(Some("ice-connection-state"), move |w, _| {
- let state = w.property::<gst_webrtc::WebRTCICEConnectionState>("ice-connection-state");
- if matches!(
- state,
- gst_webrtc::WebRTCICEConnectionState::Failed
- | gst_webrtc::WebRTCICEConnectionState::Closed
- ) {
- helper.ended(&id);
- }
- });
+ let trickle = config["trickle"] == true;
+ if trickle {
+ crate::candidates::observe(&webrtc, &id, helper);
+ }
+ crate::connection::observe(&webrtc, id, helper);
pipeline
.set_state(gst::State::Ready)
.context("preparing the video pipeline")?;
- Ok(Self { pipeline, webrtc })
- }
-
- /// Answers an offer (first connect or ICE restart) once every local candidate is in.
- pub async fn answer(&self, offer: &str) -> Result<String> {
- let sdp = gst_sdp::SDPMessage::parse_buffer(offer.as_bytes())
- .map_err(|_| anyhow!("bad offer"))?;
- let offer =
- gst_webrtc::WebRTCSessionDescription::new(gst_webrtc::WebRTCSDPType::Offer, sdp);
- let (p, done) = promise();
- self.webrtc
- .emit_by_name::<()>("set-remote-description", &[&offer, &p]);
- done.await?;
- let (p, done) = promise();
- self.webrtc
- .emit_by_name::<()>("create-answer", &[&None::<gst::Structure>, &p]);
- let reply = done.await?;
- let reply = reply.ok_or_else(|| anyhow!("no answer"))?;
- let answer = reply
- .get::<gst_webrtc::WebRTCSessionDescription>("answer")
- .map_err(|_| anyhow!("no answer"))?;
- let (p, done) = promise();
- self.webrtc
- .emit_by_name::<()>("set-local-description", &[&answer, &p]);
- done.await?;
- // Negotiate the profile before the encoder fixes its output caps.
- self.pipeline
- .set_state(gst::State::Playing)
- .context("starting the video pipeline")?;
- for _ in 0..200 {
- if self
- .webrtc
- .property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state")
- == gst_webrtc::WebRTCICEGatheringState::Complete
- {
- break;
- }
- tokio::time::sleep(Duration::from_millis(50)).await;
- }
- let local = self
- .webrtc
- .property::<Option<gst_webrtc::WebRTCSessionDescription>>("local-description");
- let local = local.unwrap_or(answer);
- local
- .sdp()
- .as_text()
- .map_err(|_| anyhow!("couldn't write the answer"))
+ Ok(Self {
+ pipeline,
+ webrtc,
+ incoming: trickle.then(Mutex::default),
+ })
}
pub fn close(self) {
@@ -359,35 +314,6 @@ fn make(name: &str) -> Result<gst::Element> {
.with_context(|| format!("GStreamer element {name} is missing"))
}
-/// Convert WebRTC's ICE URL syntax to GStreamer's URI syntax without exposing credentials.
-fn ice_uri(url: &str, username: Option<&str>, credential: Option<&str>) -> Result<String> {
- let (scheme, address) = url.split_once(':').context("invalid ICE URL")?;
- if scheme == "stun" {
- return Ok(format!("stun://{address}"));
- }
- if !matches!(scheme, "turn" | "turns") {
- bail!("unsupported ICE scheme");
- }
- let username = username.context("missing TURN username")?;
- let credential = credential.context("missing TURN credential")?;
- let escape = |s: &str| -> String {
- use std::fmt::Write as _;
- s.bytes().fold(String::new(), |mut out, byte| {
- if byte.is_ascii_alphanumeric() || b"-._~".contains(&byte) {
- out.push(char::from(byte));
- } else {
- let _ = write!(out, "%{byte:02X}");
- }
- out
- })
- };
- Ok(format!(
- "{scheme}://{}:{}@{address}",
- escape(username),
- escape(credential)
- ))
-}
-
fn encoder(bitrate_kbps: u32) -> Result<gst::Element> {
let name = ENCODERS
.iter()
@@ -414,14 +340,6 @@ fn encoder(bitrate_kbps: u32) -> Result<gst::Element> {
Ok(e)
}
-fn promise() -> (gst::Promise, oneshot::Receiver<Option<gst::Structure>>) {
- let (tx, rx) = oneshot::channel();
- let p = gst::Promise::with_change_func(move |reply| {
- let _ = tx.send(reply.ok().flatten().map(ToOwned::to_owned));
- });
- (p, rx)
-}
-
/// One still for agents: `width`×`height` JPEG, base64.
pub async fn screenshot(fd: OwnedFd, d: &Display, width: u32, height: u32) -> Result<String> {
let desc = format!(
@@ -452,26 +370,70 @@ pub async fn screenshot(fd: OwnedFd, d: &Display, width: u32, height: u32) -> Re
}
#[cfg(test)]
-mod ice_tests {
+mod tests {
#![allow(clippy::unwrap_used)]
use super::*;
+ use crate::candidates::Candidate;
- #[test]
- fn turn_uri_preserves_transport_and_escapes_credentials() {
- assert_eq!(
- ice_uri(
- "turns:turn.cloudflare.com:443?transport=tcp",
- Some("a:b"),
- Some("c+/@")
- )
- .unwrap(),
- "turns://a%3Ab:c%2B%2F%[email protected]:443?transport=tcp"
+ #[tokio::test]
+ async fn trickle_answer_does_not_wait_for_candidate_gathering() {
+ gst::init().unwrap();
+ let pipeline = gst::parse::launch(
+ "videotestsrc is-live=true ! videoconvert ! video/x-raw,format=I420 \
+ ! x264enc tune=zerolatency ! h264parse ! rtph264pay pt=96 \
+ ! application/x-rtp,media=video,encoding-name=H264,payload=96,clock-rate=90000 \
+ ! webrtcbin name=peer",
+ )
+ .unwrap()
+ .downcast::<gst::Pipeline>()
+ .unwrap();
+ let webrtc = pipeline.by_name("peer").unwrap();
+ pipeline.set_state(gst::State::Ready).unwrap();
+ let session = Session {
+ pipeline,
+ webrtc,
+ incoming: Some(Mutex::default()),
+ };
+ let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\
+ m=video 9 UDP/TLS/RTP/SAVPF 96\r\nc=IN IP4 0.0.0.0\r\n\
+ a=mid:0\r\na=recvonly\r\na=rtcp-mux\r\na=setup:actpass\r\n\
+ a=ice-ufrag:test\r\na=ice-pwd:testpasswordforthevideooffer\r\n\
+ a=fingerprint:sha-256 00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00:00\r\n\
+ a=rtpmap:96 H264/90000\r\n\
+ a=fmtp:96 packetization-mode=1;profile-level-id=42e01f\r\n";
+ // A silent local STUN endpoint keeps gathering active deterministically.
+ let silent_stun = std::net::UdpSocket::bind("127.0.0.1:0").unwrap();
+ let port = silent_stun.local_addr().unwrap().port();
+ session
+ .webrtc
+ .set_property("stun-server", format!("stun://127.0.0.1:{port}"));
+ let answer_result =
+ tokio::time::timeout(std::time::Duration::from_secs(2), session.answer(offer)).await;
+ let answer_result = answer_result.expect("the answer must not wait for STUN gathering");
+ let answer = answer_result.unwrap();
+ let sdp = gst_sdp::SDPMessage::parse_buffer(answer.as_bytes()).unwrap();
+ let video = sdp.media(0).unwrap();
+ assert_ne!(video.port(), 0, "the H.264 video offer must be accepted");
+ assert_eq!(video.formats().collect::<Vec<_>>(), ["96"]);
+ assert!(video.attributes().any(|a| a.key() == "sendonly"));
+ assert_eq!(session.pipeline.current_state(), gst::State::Playing);
+ assert_ne!(
+ session
+ .webrtc
+ .property::<gst_webrtc::WebRTCICEGatheringState>("ice-gathering-state"),
+ gst_webrtc::WebRTCICEGatheringState::Complete
);
- assert_eq!(
- ice_uri("stun:stun.cloudflare.com:3478", None, None).unwrap(),
- "stun://stun.cloudflare.com:3478"
+ assert!(
+ !answer.contains("a=candidate:"),
+ "trickle returns the original SDP, not a gathered snapshot"
);
- assert!(ice_uri("turn:turn.cloudflare.com:3478", None, None).is_err());
+ let host = Candidate::Candidate {
+ candidate: "candidate:1 1 UDP 1 192.0.2.1 5000 typ host".into(),
+ sdp_m_line_index: 0,
+ };
+ session.candidate(&host).unwrap();
+ session.candidate(&Candidate::Complete).unwrap();
+ assert!(session.candidate(&Candidate::Complete).is_err());
}
}
diff --git a/apps/screen-linux/src/video_tests.rs b/apps/screen-linux/src/video_tests.rs
index eb3e3c9b..46ce2697 100644
--- a/apps/screen-linux/src/video_tests.rs
+++ b/apps/screen-linux/src/video_tests.rs
@@ -30,7 +30,11 @@ async fn baseline_offer_starts_video_after_the_answer_is_negotiated() {
.unwrap();
let webrtc = pipeline.by_name("peer").unwrap();
pipeline.set_state(gst::State::Ready).unwrap();
- let session = Session { pipeline, webrtc };
+ let session = Session {
+ pipeline,
+ webrtc,
+ incoming: None,
+ };
let offer = "v=0\r\no=- 0 0 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\n\
m=video 9 UDP/TLS/RTP/SAVPF 102\r\nc=IN IP4 0.0.0.0\r\n\
a=mid:0\r\na=recvonly\r\na=rtcp-mux\r\na=setup:actpass\r\n\
diff --git a/apps/screen-macos-tests/Package.swift b/apps/screen-macos-tests/Package.swift
new file mode 100644
index 00000000..81085284
--- /dev/null
+++ b/apps/screen-macos-tests/Package.swift
@@ -0,0 +1,11 @@
+// swift-tools-version: 6.0
+import PackageDescription
+
+let package = Package(
+ name: "ScreenLink",
+ platforms: [.macOS(.v14)],
+ targets: [
+ .target(name: "ScreenLink", path: "Sources"),
+ .testTarget(name: "ScreenLinkTests", dependencies: ["ScreenLink"], path: "Tests"),
+ ]
+)
diff --git a/apps/screen-macos-tests/Sources/HostLink.swift b/apps/screen-macos-tests/Sources/HostLink.swift
new file mode 120000
index 00000000..64069966
--- /dev/null
+++ b/apps/screen-macos-tests/Sources/HostLink.swift
@@ -0,0 +1 @@
+../../screen-macos/HostLink.swift
\ No newline at end of file
diff --git a/apps/screen-macos-tests/Sources/SessionCandidate.swift b/apps/screen-macos-tests/Sources/SessionCandidate.swift
new file mode 120000
index 00000000..cc1d8761
--- /dev/null
+++ b/apps/screen-macos-tests/Sources/SessionCandidate.swift
@@ -0,0 +1 @@
+../../screen-macos/SessionCandidate.swift
\ No newline at end of file
diff --git a/apps/screen-macos-tests/Tests/HostLinkTests.swift b/apps/screen-macos-tests/Tests/HostLinkTests.swift
new file mode 100644
index 00000000..0951909c
--- /dev/null
+++ b/apps/screen-macos-tests/Tests/HostLinkTests.swift
@@ -0,0 +1,104 @@
+import Foundation
+import Network
+import Testing
+@testable import ScreenLink
+
+@MainActor private func eventually(_ message: String, _ condition: () -> Bool) async throws {
+ for _ in 0..<1000 {
+ if condition() { return }
+ try await Task.sleep(for: .milliseconds(5))
+ }
+ throw HelperError("socket check timed out: \(message)")
+}
+
+@MainActor private final class Host {
+ let listener: NWListener
+ private(set) var isReady = false
+ private(set) var connections: [NWConnection] = []
+ private(set) var replies: [ObjectIdentifier: [[String: Any]]] = [:]
+ private var buffers: [ObjectIdentifier: Data] = [:]
+
+ init() throws {
+ listener = try NWListener(using: .tcp, on: .any)
+ listener.stateUpdateHandler = { [weak self] state in
+ MainActor.assumeIsolated {
+ if case .ready = state { self?.isReady = true }
+ }
+ }
+ listener.newConnectionHandler = { [weak self] c in
+ MainActor.assumeIsolated {
+ guard let self else { return }
+ self.connections.append(c)
+ c.start(queue: .main)
+ self.receive(on: c)
+ }
+ }
+ listener.start(queue: .main)
+ }
+
+ func send(_ method: String, on c: NWConnection) {
+ let data = Data((#"{"id":1,"method":"\#(method)","params":{}}"# + "\n").utf8)
+ c.send(content: data, completion: .contentProcessed { _ in })
+ }
+
+ private func receive(on c: NWConnection) {
+ c.receive(minimumIncompleteLength: 1, maximumLength: 1 << 20) { [weak self] data, _, done, error in
+ MainActor.assumeIsolated {
+ guard let self else { return }
+ let key = ObjectIdentifier(c)
+ if let data {
+ self.buffers[key, default: Data()].append(data)
+ while let nl = self.buffers[key]?.firstIndex(of: 0x0a), let buffer = self.buffers[key] {
+ let line = buffer[buffer.startIndex..<nl]
+ self.buffers[key]?.removeSubrange(buffer.startIndex...nl)
+ if let reply = try? JSONSerialization.jsonObject(with: line) as? [String: Any] {
+ self.replies[key, default: []].append(reply)
+ }
+ }
+ }
+ if !done && error == nil { self.receive(on: c) }
+ }
+ }
+ }
+
+ func stop() {
+ listener.cancel()
+ for c in connections { c.cancel() }
+ }
+}
+
+@Test(arguments: [false, true]) @MainActor func oldSocketRepliesCannotAnswerANewerConnectionsRequest(throwsError: Bool) async throws {
+ let host = try Host()
+ defer { host.stop() }
+ try await eventually("listener ready") { host.isReady }
+ let port = try #require(host.listener.port)
+ let link = HostLink(endpoint: .hostPort(host: "127.0.0.1", port: port))
+ defer { link.stop() }
+ var pending: CheckedContinuation<Void, Never>?
+ var resumed = false
+ link.handler = { method, _ in
+ if method == "old" {
+ await withCheckedContinuation { pending = $0 }
+ resumed = true
+ if throwsError { throw HelperError("old socket failure") }
+ }
+ return ["method": method]
+ }
+ link.start()
+ try await eventually("first connection ready") { host.connections.count == 1 && link.isConnected }
+ host.send("old", on: host.connections[0])
+ try await eventually("first handler suspended") { pending != nil }
+ host.connections[0].cancel()
+ try await eventually("replacement connection ready") { host.connections.count == 2 && link.isConnected }
+ let replacement = host.connections[1]
+ let key = ObjectIdentifier(replacement)
+ host.send("new", on: replacement)
+ try await eventually("replacement reply received") { host.replies[key]?.count == 1 }
+ pending?.resume()
+ pending = nil
+ try await eventually("first handler resumed") { resumed }
+ try await Task.sleep(for: .milliseconds(100))
+ let replies = host.replies[key, default: []]
+ #expect(replies.count == 1, "the old handler must not reuse its request ID on the new socket")
+ #expect((replies.first?["result"] as? [String: String])?["method"] == "new")
+}
diff --git a/apps/screen-macos-tests/Tests/SessionCandidateTests.swift b/apps/screen-macos-tests/Tests/SessionCandidateTests.swift
new file mode 100644
index 00000000..4612f14f
--- /dev/null
+++ b/apps/screen-macos-tests/Tests/SessionCandidateTests.swift
@@ -0,0 +1,45 @@
+import Foundation
+import Testing
+@testable import ScreenLink
+
+/// Parsed the way `HostLink` parses the host's requests.
+private func wire(_ json: String) throws -> [String: Any] {
+ try #require(JSONSerialization.jsonObject(with: Data(json.utf8)) as? [String: Any])
+}
+
+private let relay = #"{"type":"candidate","candidate":"candidate:1 1 UDP 1 192.0.2.1 5000 typ relay","sdpMLineIndex":0}"#
+
+@Test func completionFollowsAllCandidatesAndBoundsAreEnforced() throws {
+ var progress = CandidateProgress()
+ let candidate = try SessionCandidate(json: wire(relay))
+ for _ in 0..<CandidateProgress.maxCount { try progress.accept(candidate) }
+ #expect(throws: HelperError.self) { try progress.accept(candidate) }
+ try progress.accept(.complete)
+ #expect(throws: HelperError.self) { try progress.accept(candidate) }
+ #expect(throws: HelperError.self) { try progress.accept(.complete) }
+}
+
+@Test func wireCandidatesAreValidatedBeforeTheyReachThePeer() throws {
+ let long = "candidate:" + String(repeating: "a", count: 4096)
+ let rejected = [
+ #"{"type":"candidate","candidate":"","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":" ","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":"candidate:1\r\n","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":"\#(long)","sdpMLineIndex":0}"#,
+ #"{"type":"candidate","candidate":"candidate:1","sdpMLineIndex":17}"#,
+ #"{"type":"candidate","candidate":"candidate:1"}"#,
+ #"{"type":"ready"}"#,
+ "{}",
+ ]
+ for json in rejected {
+ let value = try wire(json)
+ #expect(throws: HelperError.self) { try SessionCandidate(json: value) }
+ }
+ var withMid = try wire(relay)
+ withMid["sdpMid"] = "0"
+ let parsed = try SessionCandidate(json: withMid)
+ #expect(parsed == .candidate(sdp: "candidate:1 1 UDP 1 192.0.2.1 5000 typ relay", index: 0, mid: "0"))
+ #expect(parsed.json["sdpMid"] as? String == "0")
+ #expect(try SessionCandidate(json: wire(#"{"type":"complete"}"#)) == .complete)
+ #expect(SessionCandidate.complete.json["type"] as? String == "complete")
+}
diff --git a/apps/screen-macos/HostLink.swift b/apps/screen-macos/HostLink.swift
index 28a27b15..45b0d2ca 100644
--- a/apps/screen-macos/HostLink.swift
+++ b/apps/screen-macos/HostLink.swift
@@ -15,18 +15,28 @@ final class HostLink {
/// Answers the host's requests.
var handler: Handler?
private var connection: NWConnection?
+ private let endpoint: NWEndpoint
+ private var reconnectTask: Task<Void, Never>?
private var buffer = Data()
private(set) var isConnected = false
/// Runs on every (re)connect: the host forgets us when we drop.
var onConnect: (@MainActor () -> Void)?
+ var onDisconnect: (@MainActor () -> Void)?
static var socketPath: String {
let base = ProcessInfo.processInfo.environment["CODYNC_HOME"] ?? NSHomeDirectory() + "/" + (Bundle.main.object(forInfoDictionaryKey: "CodyncDataFolder") as? String ?? ".codync")
return base + "/screen.sock"
}
+ init(endpoint: NWEndpoint? = nil) {
+ self.endpoint = endpoint ?? .unix(path: Self.socketPath)
+ }
+
func start() {
- let c = NWConnection(to: .unix(path: Self.socketPath), using: .tcp)
+ reconnectTask?.cancel()
+ reconnectTask = nil
+ guard connection == nil else { return }
+ let c = NWConnection(to: endpoint, using: .tcp)
connection = c
c.stateUpdateHandler = { [weak self] state in
MainActor.assumeIsolated { self?.changed(state, of: c) }
@@ -34,6 +44,18 @@ final class HostLink {
c.start(queue: .main)
}
+ func stop() {
+ reconnectTask?.cancel()
+ reconnectTask = nil
+ if let c = connection {
+ connection = nil
+ c.stateUpdateHandler = nil
+ c.cancel()
+ }
+ isConnected = false
+ onDisconnect?()
+ }
+
private func changed(_ state: NWConnection.State, of c: NWConnection) {
guard c === connection else { return }
switch state {
@@ -53,11 +75,12 @@ final class HostLink {
private func drop(_ c: NWConnection) {
guard c === connection else { return }
isConnected = false
+ onDisconnect?()
connection = nil
c.stateUpdateHandler = nil
c.cancel()
- Task { [weak self] in
- try? await Task.sleep(for: .seconds(2))
+ reconnectTask = Task { [weak self] in
+ do { try await Task.sleep(for: .seconds(2)) } catch { return }
self?.start()
}
}
@@ -68,7 +91,7 @@ final class HostLink {
guard let self, c === self.connection else { return }
if let data {
self.buffer.append(data)
- self.drain()
+ self.drain(on: c)
}
if done || error != nil {
self.drop(c)
@@ -79,7 +102,7 @@ final class HostLink {
}
}
- private func drain() {
+ private func drain(on c: NWConnection) {
while let nl = buffer.firstIndex(of: 0x0A) {
let line = buffer[buffer.startIndex..<nl]
buffer.removeSubrange(buffer.startIndex...nl)
@@ -88,16 +111,19 @@ final class HostLink {
else { continue }
let id = msg["id"].map { "\($0)" }
let params = msg["params"] as? [String: Any] ?? [:]
- Task { await self.run(method, params, id: id) }
+ Task { await self.run(method, params, id: id, on: c) }
}
}
- private func run(_ method: String, _ params: [String: Any], id: String?) async {
+ private func run(_ method: String, _ params: [String: Any], id: String?, on c: NWConnection) async {
+ guard c === connection, isConnected else { return }
do {
guard let handler else { throw HelperError("not ready") }
let result = try await handler(method, params)
+ guard c === connection, isConnected else { return }
if let id { send(["jsonrpc": "2.0", "id": Int(id) ?? 0, "result": result]) }
} catch {
+ guard c === connection, isConnected else { return }
if let id {
send(["jsonrpc": "2.0", "id": Int(id) ?? 0, "error": ["code": -32000, "message": error.localizedDescription]])
}
@@ -111,6 +137,10 @@ final class HostLink {
private func send(_ msg: [String: Any]) {
guard let connection, isConnected, var data = try? JSONSerialization.data(withJSONObject: msg) else { return }
data.append(0x0A)
- connection.send(content: data, completion: .contentProcessed { _ in })
+ connection.send(content: data, completion: .contentProcessed { [weak self] error in
+ if error != nil {
+ MainActor.assumeIsolated { self?.drop(connection) }
+ }
+ })
}
}
diff --git a/apps/screen-macos/ScreenHelper.swift b/apps/screen-macos/ScreenHelper.swift
index 2f32269f..d1acf730 100644
--- a/apps/screen-macos/ScreenHelper.swift
+++ b/apps/screen-macos/ScreenHelper.swift
@@ -21,6 +21,7 @@ final class ScreenHelper {
lastStatus = [:]
sendStatus()
}
+ link.onDisconnect = { [unowned self] in closeAll() }
badge.onDisconnect = { [unowned self] in closeAll() }
}
@@ -62,7 +63,7 @@ final class ScreenHelper {
}
let s = Session(id: id, display: display(p["display"]), helper: self, iceServers: iceServers,
maxBitrate: min(16_000_000, max(100_000, p["maxBitrateBps"] as? Int ?? 16_000_000)),
- maxFramerate: min(60, max(1, p["maxFramerate"] as? Int ?? 60)))
+ maxFramerate: min(60, max(1, p["maxFramerate"] as? Int ?? 60)), trickle: p["trickle"] as? Bool ?? false)
sessions[id] = s
do {
let answer = try await s.answer(offer: sdp)
@@ -73,6 +74,12 @@ final class ScreenHelper {
s.close()
throw error
}
+ case "candidate":
+ guard let id = p["session"] as? String, let session = sessions[id],
+ let candidate = p["candidate"] as? [String: Any]
+ else { throw HelperError("unknown screen session") }
+ try await session.candidate(candidate)
+ return [String: Any]()
case "close":
if let id = p["session"] as? String, let s = sessions.removeValue(forKey: id) {
s.close()
@@ -94,6 +101,16 @@ final class ScreenHelper {
}
}
+ func candidate(session: String, candidate: [String: Any]) {
+ guard sessions[session] != nil else { return }
+ link.notify("candidate", ["session": session, "candidate": candidate])
+ }
+
+ func sessionConnected(_ session: Session) {
+ guard sessions[session.id] === session else { return }
+ link.notify("session", ["session": session.id, "state": "connected"])
+ }
+
func sessionEnded(_ id: String) {
guard let s = sessions.removeValue(forKey: id) else { return }
s.close()
@@ -143,6 +160,7 @@ final class ScreenHelper {
"permissionApp": Bundle.main.object(forInfoDictionaryKey: "CFBundleDisplayName") as? String ?? "Codync Screen",
"displays": displays,
"capture": CGPreflightScreenCaptureAccess(),
+ "trickle": true,
"input": AXIsProcessTrusted(),
]
guard status != lastStatus, link.isConnected else { return }
diff --git a/apps/screen-macos/Session.swift b/apps/screen-macos/Session.swift
index d1d71ab6..ad764359 100644
--- a/apps/screen-macos/Session.swift
+++ b/apps/screen-macos/Session.swift
@@ -25,10 +25,15 @@ final class Session: NSObject {
private var gatherTask: Task<Void, Never>?
private let maxBitrate: Int
private let maxFramerate: Int
+ private let candidates: SessionCandidates?
+ nonisolated private let candidateSink: AsyncThrowingStream<SessionCandidate, Error>.Continuation?
var display: CGDirectDisplayID { capture.display }
- init(id: String, display: CGDirectDisplayID, helper: ScreenHelper, iceServers: [RTCIceServer], maxBitrate: Int, maxFramerate: Int) {
+ init(id: String, display: CGDirectDisplayID, helper: ScreenHelper, iceServers: [RTCIceServer], maxBitrate: Int, maxFramerate: Int, trickle: Bool) {
+ let candidates = trickle ? SessionCandidates(helper: helper, session: id) : nil
+ self.candidates = candidates
+ candidateSink = candidates?.sink
self.maxBitrate = maxBitrate
self.maxFramerate = maxFramerate
self.id = id
@@ -47,11 +52,11 @@ final class Session: NSObject {
pc = Self.factory.peerConnection(with: config, constraints: RTCMediaConstraints(mandatoryConstraints: nil, optionalConstraints: nil), delegate: self)
}
- /// Answers the phone's offer (first connect or an ICE restart). Non-trickle: waits
- /// for local candidates so the answer carries them all.
+ /// Trickle returns the original answer; complete-SDP peers wait for local candidates.
func answer(offer: String) async throws -> String {
guard let pc else { throw HelperError("session closed") }
try await pc.setRemoteDescription(RTCSessionDescription(type: .offer, sdp: offer))
+ guard self.pc === pc else { throw HelperError("session closed") }
if !capturing, let video = pc.transceivers.first(where: { $0.mediaType == .video }) {
video.sender.track = track
video.sender.streamIds = ["screen"]
@@ -60,8 +65,10 @@ final class Session: NSObject {
if let error { throw error }
}
let answer = try await pc.answer(for: RTCMediaConstraints(mandatoryConstraints: nil, optionalConstraints: nil))
+ guard self.pc === pc else { throw HelperError("session closed") }
try await pc.setLocalDescription(answer)
- await waitForCandidates()
+ guard self.pc === pc else { throw HelperError("session closed") }
+ if candidates == nil { await waitForCandidates() }
guard self.pc === pc else { throw HelperError("session closed") }
if !capturing {
tune()
@@ -74,7 +81,7 @@ final class Session: NSObject {
keepAwake(true)
watchClipboard()
}
- return pc.localDescription?.sdp ?? answer.sdp
+ return candidates == nil ? (pc.localDescription?.sdp ?? answer.sdp) : answer.sdp
}
/// Screen content: keep text sharp (drop frames before resolution), allow a high bitrate on fast links.
@@ -107,7 +114,14 @@ final class Session: NSObject {
gathered = nil
}
+ func candidate(_ value: [String: Any]) async throws {
+ guard let candidates, let pc else { throw HelperError("screen session is not ready for candidates") }
+ try await candidates.add(value, to: pc)
+ guard self.pc === pc else { throw HelperError("session closed") }
+ }
+
func close() {
+ candidates?.stop()
finishGathering()
clipboardTask?.cancel()
keepAwake(false)
@@ -164,6 +178,7 @@ final class Session: NSObject {
// MARK: messages from the phone
private func received(_ data: Data) {
+ guard pc != nil else { return }
guard let msg = (try? JSONSerialization.jsonObject(with: data)) as? [String: Any] else { return }
switch msg["type"] as? String {
case "clipboard":
@@ -179,6 +194,7 @@ final class Session: NSObject {
}
let capture = capture
Task {
+ guard self.pc != nil else { return }
do {
let applied = try await capture.view(display: display, crop: crop, long: InputInjector.number(msg["long"]), short: InputInjector.number(msg["short"]))
// Frames from here on show this region: the phone re-aligns its view on this.
@@ -192,6 +208,7 @@ final class Session: NSObject {
guard let helper else { return }
let display = self.display
Task {
+ guard self.pc != nil else { return }
do {
try await helper.input.perform(msg, display: display)
} catch {
@@ -202,7 +219,10 @@ final class Session: NSObject {
}
private func connectionChanged(_ state: RTCIceConnectionState) {
+ guard pc != nil else { return }
switch state {
+ case .connected, .completed:
+ helper?.sessionConnected(self)
case .failed, .closed:
// `disconnected` can recover (or the phone restarts ICE); `failed` can't.
helper?.sessionEnded(id)
@@ -212,6 +232,7 @@ final class Session: NSObject {
}
private func opened(_ channel: RTCDataChannel) {
+ guard pc != nil else { channel.close(); return }
channel.delegate = self
channels.append(channel)
}
@@ -222,7 +243,16 @@ extension Session: RTCPeerConnectionDelegate, RTCDataChannelDelegate {
nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didAdd stream: RTCMediaStream) {}
nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didRemove stream: RTCMediaStream) {}
nonisolated func peerConnectionShouldNegotiate(_ peerConnection: RTCPeerConnection) {}
- nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didGenerate candidate: RTCIceCandidate) {}
+ nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didGenerate candidate: RTCIceCandidate) {
+ emitCandidate(.candidate(sdp: candidate.sdp, index: candidate.sdpMLineIndex, mid: candidate.sdpMid))
+ }
+
+ nonisolated private func emitCandidate(_ candidate: SessionCandidate) {
+ guard let candidateSink else { return }
+ if case .dropped = candidateSink.yield(candidate) {
+ candidateSink.finish(throwing: HelperError("screen candidate queue overflow"))
+ }
+ }
nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didRemove candidates: [RTCIceCandidate]) {}
nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didChange newState: RTCIceConnectionState) {
@@ -231,6 +261,7 @@ extension Session: RTCPeerConnectionDelegate, RTCDataChannelDelegate {
nonisolated func peerConnection(_ peerConnection: RTCPeerConnection, didChange newState: RTCIceGatheringState) {
guard newState == .complete else { return }
+ emitCandidate(.complete)
Task { @MainActor in self.finishGathering() }
}
diff --git a/apps/screen-macos/SessionCandidate.swift b/apps/screen-macos/SessionCandidate.swift
new file mode 100644
index 00000000..3c744bdb
--- /dev/null
+++ b/apps/screen-macos/SessionCandidate.swift
@@ -0,0 +1,50 @@
+import Foundation
+
+/// One screen candidate event on the wire, matching the host's `Candidate`.
+/// Plain values, so they can leave WebRTC's signaling thread.
+enum SessionCandidate: Sendable, Equatable {
+ case candidate(sdp: String, index: Int32, mid: String?)
+ case complete
+
+ private static let maxBytes = 4096
+ private static let maxMediaIndex: Int32 = 16
+
+ /// Parses and validates one candidate from the host.
+ init(json: [String: Any]) throws {
+ switch json["type"] as? String {
+ case "complete": self = .complete
+ case "candidate":
+ guard let sdp = json["candidate"] as? String, sdp.hasPrefix("candidate:"), sdp.utf8.count <= Self.maxBytes,
+ !sdp.contains("\r"), !sdp.contains("\n"),
+ let index = json["sdpMLineIndex"] as? Int32, (0...Self.maxMediaIndex).contains(index)
+ else { throw HelperError("invalid screen candidate") }
+ self = .candidate(sdp: sdp, index: index, mid: json["sdpMid"] as? String)
+ default: throw HelperError("invalid screen candidate")
+ }
+ }
+
+ var json: [String: Any] {
+ guard case let .candidate(sdp, index, mid) = self else { return ["type": "complete"] }
+ var value: [String: Any] = ["type": "candidate", "candidate": sdp, "sdpMLineIndex": index]
+ if let mid { value["sdpMid"] = mid }
+ return value
+ }
+}
+
+/// One direction of the exchange, under the host's rules: at most `maxCount`
+/// candidates, then one completion, then nothing.
+struct CandidateProgress {
+ static let maxCount = 128
+ private(set) var count = 0
+ private(set) var isComplete = false
+
+ mutating func accept(_ candidate: SessionCandidate) throws {
+ guard !isComplete else { throw HelperError("screen candidates already complete") }
+ if candidate == .complete {
+ isComplete = true
+ } else {
+ guard count < Self.maxCount else { throw HelperError("too many screen candidates") }
+ count += 1
+ }
+ }
+}
diff --git a/apps/screen-macos/SessionCandidates.swift b/apps/screen-macos/SessionCandidates.swift
new file mode 100644
index 00000000..91e700a4
--- /dev/null
+++ b/apps/screen-macos/SessionCandidates.swift
@@ -0,0 +1,43 @@
+import Foundation
+@preconcurrency import WebRTC
+
+/// A trickle session's candidates: ours go to the host in order, the phone's go to the peer.
+@MainActor
+final class SessionCandidates {
+ nonisolated let sink: AsyncThrowingStream<SessionCandidate, Error>.Continuation
+ private var task: Task<Void, Never>?
+ private var incoming = CandidateProgress()
+
+ init(helper: ScreenHelper, session: String) {
+ let pair = AsyncThrowingStream<SessionCandidate, Error>.makeStream(bufferingPolicy: .bufferingOldest(CandidateProgress.maxCount + 1))
+ sink = pair.continuation
+ task = Task { [weak helper] in
+ do {
+ var outgoing = CandidateProgress()
+ for try await candidate in pair.stream {
+ try Task.checkCancellation()
+ try outgoing.accept(candidate)
+ guard let helper else { return }
+ helper.candidate(session: session, candidate: candidate.json)
+ }
+ } catch {
+ guard !Task.isCancelled else { return }
+ helper?.sessionEnded(session)
+ }
+ }
+ }
+
+ func add(_ value: [String: Any], to pc: RTCPeerConnection) async throws {
+ let candidate = try SessionCandidate(json: value)
+ try incoming.accept(candidate)
+ // Native Apple WebRTC has no usable empty/null end-of-candidates operation.
+ guard case let .candidate(sdp, index, mid) = candidate else { return }
+ try await pc.add(RTCIceCandidate(sdp: sdp, sdpMLineIndex: index, sdpMid: mid))
+ }
+
+ func stop() {
+ sink.finish()
+ task?.cancel()
+ task = nil
+ }
+}
diff --git a/docs/architecture/file-structure.md b/docs/architecture/file-structure.md
index 7ec81d9b..d14093c7 100644
--- a/docs/architecture/file-structure.md
+++ b/docs/architecture/file-structure.md
@@ -55,6 +55,7 @@ Codync/
│ │ ├── app/ # Screens, the store (AppStore), notifications, Live updates, Glance widgets
│ │ └── tools/ # symbols.py (SF Symbol stand-ins), launcher_icon.py, agent_icons.py
│ ├── screen-macos/ # macOS screen helper (Xcode Screen target)
+│ ├── screen-macos-tests/ # Swift package testing the helper's socket lifecycle and candidate rules
│ └── screen-linux/ # Linux portal/GStreamer screen helper
├── cloud/ # Account API + encrypted relay Worker and Durable Object
│ ├── src/
@@ -78,6 +79,7 @@ Codync/
- `apps/desktop/` — desktop app for macOS, Linux and Windows (Electron; `npm run dev`, packaged by `.github/workflows/release-desktop.yml`)
- `iOS` (`apps/ios/`) — iOS app
- `Screen` (`apps/screen-macos/`) — Codync Screen: capture, input and WebRTC for Remote screen, embedded in the macOS desktop app (`Contents/Library/LoginItems`); starts the computer-use driver it carries in `Contents/Helpers`
+- `apps/screen-macos-tests/` — native helper tests: socket lifecycle and candidate rules (`swift test --package-path apps/screen-macos-tests --scratch-path build/dd/screen-macos-tests`)
- `apps/screen-linux/` — `codync-screen` (Rust, GStreamer + xdg portals), the Linux Remote screen helper
- `Widgets` (`apps/ios/Widgets/`) — usage widget + bot Live Activity (bundle id `com.pokai.Codync.ios.LiveActivity`)
- `apps/android/` — Android app (`com.pokai.Codync.android`, Dev `.dev`; Gradle modules `app`, `core`, `design`; CI `.github/workflows/android.yml`)
diff --git a/docs/features/remote-screen.md b/docs/features/remote-screen.md
index c2064a50..9974ff0e 100644
--- a/docs/features/remote-screen.md
+++ b/docs/features/remote-screen.md
@@ -4,11 +4,11 @@ Remote screen lets an authorized phone view and control a computer. Signaling tr
## Connection path
-1. The phone calls `screenPrepare {relay}`. `relay` is true when its active host channel uses Cloudflare; direct LAN/Tailscale connections do not contact a STUN/TURN provider.
+1. The phone calls `screenPrepare {relay, trickle: true}`. `relay` is true when its active host channel uses Cloudflare; direct LAN/Tailscale connections do not contact a STUN/TURN provider.
2. The host checks the `screen` scope and that screen access is enabled. For relay connections it signs `POST /v1/host/screen-ice {deviceKey}`. The Worker requires a registered, active computer belonging to an active account. The host attests the device's local authorization, so account grants and local QR pairing both work on a claimed computer.
3. The Worker issues Cloudflare Realtime credentials with a one-hour TTL, tags usage with `computerId:deviceFingerprint` (the fingerprint is the base64url encoding of the first 16 bytes of SHA-256 of the device public key; the resulting 45-character tag fits Cloudflare’s 64-character limit), and returns `Cache-Control: no-store`. Permanent TURN keys stay in Worker secrets.
-4. The host reserves a random session ID bound to the calling device and keeps the exact ICE configuration in memory. The phone and helper use that configuration. Clients cannot supply arbitrary ICE servers to the host.
-5. `screenOffer {session, sdp, display?}` carries the completed offer to the helper. Both peers gather for up to ten seconds. ICE prefers direct candidates and uses TURN when necessary, with UDP, TCP and TLS on port 443 available.
+4. The host reserves a random session ID bound to the calling device and helper connection, and keeps the exact ICE configuration in memory. It returns `trickle: true` only when the phone requests it and the helper advertises support. The negotiated mode stays with that session. Clients cannot supply arbitrary ICE servers to the host.
+5. For trickle sessions, the phone opens a private `screenCandidates {session}` subscription on its encrypted channel and waits for `ready` before sending `screenOffer {session, sdp, display?}`. Both peers send their original candidate-free SDP after installing their local description, while gathering continues concurrently. The helper's candidates arrive on that subscription; the phone uploads its candidates through `screenCandidate {session, candidate}` after installing the answer, one acknowledged request at a time. Late TURN TCP/TLS candidates remain available. Older phones, hosts or helpers use complete SDP and gather for up to ten seconds. ICE prefers direct candidates and uses TURN when necessary, with UDP, TCP and TLS on port 443 available.
6. The phone reconnects with fresh credentials five minutes before expiry. The host checks active sessions every five seconds and closes expired sessions, revoked devices and expired account leases. A prepared session must start within one minute.
Connections prepared through the cloud use a **4 Mbps / 30 fps** encoding limit even if ICE subsequently finds a direct path. Direct sessions retain their platform defaults (Mac: 16 Mbps / 60 fps). These are encoding limits, not exact bandwidth or billing caps. TURN does not carry plaintext desktop content; WebRTC DTLS-SRTP and data-channel encryption remain between the phone and helper.
@@ -38,14 +38,39 @@ back to the bots, then reconnects when the app returns.
- `host/src/screen/` coordinates access to the local helper and owns viewer sessions. Another device cannot renegotiate or close a session it does not own.
- `apps/screen-macos/` is the macOS capture/input helper (Xcode `Screen` target, `CodyncScreen.app`). The desktop app bundles it and registers it through `SMAppService` (`apps/desktop/src/main/screen.ts`); it is responsible for the OS permissions, including those of the computer-use driver it starts.
- `apps/screen-linux/` implements the Linux helper using desktop portals and GStreamer, including ICE URL conversion and TURN transport configuration. The host starts it from beside its own executable (then `PATH`). Linux host releases ship `codync-screen` in the same archive (built on Ubuntu 24.04: glibc 2.39+ and GStreamer 1.22+ with the base, good and bad plugins, PipeWire and xdg-desktop-portal at run time); `install.sh`, Homebrew and `codync-host update` put it next to the host.
-- Helpers communicate locally through `~/.codync/screen.sock`. SDP is non-trickle; input uses the `input` and `input-fast` data channels.
-- Phone viewers: iOS in `apps/ios/Kit/Sources/CodyncUI/Screen/`. Android keeps the same structure in `apps/android/app/.../Screen*.kt`, with the gesture-to-input and viewport logic in `core` (`ScreenControl`, `ScreenViewport`). Both offer hardware H.264 first, High then Constrained Baseline.
+- Helpers communicate locally through `~/.codync/screen.sock`. Updated helpers advertise `trickle` in `status`, emit ordered `candidate {session, candidate}` notifications, and accept the host's `candidate` request. Input uses the `input` and `input-fast` data channels.
+- Linux waits for an in-progress answer before acknowledging that viewer's close, and cancels pending helper requests when the socket ends. A late answer cannot restart closed capture; other viewers can still negotiate and close independently.
+- macOS replies stay bound to the socket that received the request. Closed peers ignore queued input and channel callbacks. The phone retires a failed peer before preparing its replacement and checks peer identity after each native negotiation await.
+- A trickle session accepts one offer and one candidate subscription. Each direction permits 128 candidates of at most 4096 bytes followed by one `complete` event. Candidates include `candidate`, `sdpMLineIndex` and optional `sdpMid`; `type` distinguishes `candidate` and `complete`. The subscription starts with `ready` and may report `error` before ending. It never uses the shared chat broadcast.
+- Both directions buffer candidates until the phone installs its answer. Native candidate additions and upload acknowledgments are awaited in order. Completion follows every candidate; Apple WebRTC records completion at app level because its native API has no usable null-candidate operation. The finite connection deadline remains in place.
+- Overflow and candidate protocol errors end the attempt. Signaling loss ends an unfinished negotiation; once ICE connects or both directions deliver completion (including the phone upload acknowledgment), it detaches the candidate subscription and leaves media and input running. Credentials, Screen authorization and the on/off switch continue to govern the viewer independently. Recovery reserves a fresh session rather than resubscribing, reoffering or replaying an uncertain candidate addition. The phone waits for a ready host channel before spending one of its three recovery attempts; a successful answer alone does not reset that budget. A permanently stopped channel ends recovery with its authorization or update error. Closing, disabling screen access, changing helpers or removing Screen authorization closes media.
+- Android negotiates the same way (`core` `ScreenSignaling`, `ScreenCandidate`, `ScreenRoute`); desktop, terminal UI and Apple Watch have no remote-screen viewer. Their screen-helper management and permission controls are unaffected. Cloud and relay forwarding already carry the encrypted subscription and RPC frames; they need no signaling changes.
+- Phone viewers: iOS in `apps/ios/Kit/Sources/CodyncUI/Screen/`. Android keeps the same structure in `apps/android/app/.../Screen*.kt`, with gesture-to-input and viewport logic in `core` (`ScreenControl`, `ScreenViewport`). Both offer hardware H.264 first, High then Constrained Baseline.
- The Linux helper answers with the packetized H.264 payload the phone offers, feeds the encoder 8-bit 4:2:0 video and timestamps capture on the system clock. It answers before capture starts, so the profile is negotiated before the encoder fixes its output.
- Screen access is on by default on macOS and on Linux while a graphical session (Wayland or X11) is running; headless servers stay off. Turning it on or off is remembered. Enabling through `setScreenEnabled` requires a loopback caller. Interactive OS permission prompts must be completed on the computer.
- Bots with their computer capability enabled receive the built-in `computer` MCP tools, run by the computer-use driver that Codync Screen starts on macOS ([computer use](computer-use.md)). Their permission policy still applies. An interactive phone can take over; bots may still look.
- Each device may hold four sessions; the host allows 32 total, including pending sessions. Credential issuance is limited to 12 requests per minute per account by `TURN_LIMITER` (Cloudflare's per-location rate limiter).
- Closing a session stops its media; issued TURN credentials remain usable until their one-hour TTL. Issuance limits and encoder limits do not constitute a hard monthly spending cap. Track Realtime egress by the credential's custom identifier.
+## Android incremental ICE
+
+Android advertises `trickle: true` in `screenPrepare`, then uses incremental ICE
+only when the host returns `trickle: true`. An omitted or false capability keeps
+complete-SDP negotiation for older hosts and helpers. The host/helper protocol
+is introduced in [PR #40](https://github.com/leepokai/Codync/pull/40).
+
+The Android viewer subscribes to `screenCandidates` before offering, waits for
+`ready`, and installs the answer before applying or uploading candidates. Each
+direction permits 128 candidates of at most 4096 UTF-8 bytes, followed by one
+completion. A full screen-signaling queue fails that session without blocking
+unrelated RPCs. Protocol errors remain fatal; a transport disconnect after media
+connects or both directions acknowledge completion leaves WebRTC running.
+
+Android core owns `ScreenCandidate` and `ScreenSignaling`; the app's
+`ScreenSession` connects them to native WebRTC. The iOS implementation and
+host/helper contract are covered by PR #40. Desktop, terminal UI and Watch have
+no viewer and receive no client signaling changes here.
+
## Cloud setup
Create a separate Cloudflare Realtime TURN key for each environment. Save its key ID as `TURN_KEY_ID` and its credential-issuing API token as `TURN_KEY_API_TOKEN`, both Worker secrets. Every environment needs the `TURN_LIMITER` binding in `cloud/wrangler.toml`.
@@ -70,13 +95,16 @@ Live acceptance requires the configured Worker and fresh host/helper/phone build
- Check screen-recording and input permissions, capture, click/type/scroll, clipboard, display changes and closing the viewer.
- Run the checks below on an Android phone too, including a hardware keyboard and IME composition (Zhuyin, Pinyin).
+- Measure first-frame connection time and input-to-visible-frame delay, including after several seconds of idle. Verify video continues through pinch/pan/reset and frame-size changes.
- Open the iPhone viewer while upright and while already held sideways, then rotate
in both directions. Check video fitting and direct taps after each turn. Repeat
with rotation lock on and with the Rotate button; closing returns to portrait.
- Repeat with the phone on cellular and the computer on a different network, without Tailscale. Verify a selected ICE candidate pair with `candidateType=relay` in WebRTC diagnostics and observe Realtime egress; signaling via Cloudflare alone is insufficient.
-- Verify UDP-blocked connectivity using TURN over TCP/TLS 443.
+- Verify UDP-blocked connectivity using TURN over TCP/TLS 443 on both Linux and macOS. Check the selected relay's actual server URL and relay transport; a candidate's `tcp` token alone does not establish that TURN used TLS.
+- Confirm that offer/answer exchange finishes before gathering completes, that late candidates reach both peers, and that candidates generated before the answer are applied afterward in order. Completion must come last.
+- Check mixed old/new phone, host and helper combinations, duplicate offers/subscriptions, another device's session ID, queue overflow, signaling loss, fresh-session recovery and stale-session rejection.
- Verify Wi-Fi/cellular changes, reconnect after sleep and renewal before the one-hour expiry.
-- Revoke the phone or disable screen access while streaming; input and media must stop. Verify a control-only device cannot call `screenPrepare` or `screenOffer`.
+- Revoke the phone or disable screen access while streaming; input and media must stop. Verify a control-only device cannot call `screenPrepare`, `screenOffer`, `screenCandidate` or subscribe to `screenCandidates`.
- Verify LAN/direct-only mode without internet access, and the explicit error when relay secrets are absent.
See [Cloudflare testing](../guides/cloudflare-testing.md) for transport checks and [architecture](../architecture/overview.md) for data flow.
diff --git a/docs/guides/development.md b/docs/guides/development.md
index d4b28f2c..dfec60be 100644
--- a/docs/guides/development.md
+++ b/docs/guides/development.md
@@ -143,6 +143,12 @@ resources, dependencies, `apps/shared/Config`, `apps/screen-macos`, `packaging/`
Left to CI: the iOS Kit tests (an iPhone simulator) and the platform builds (Linux screen
helper: GStreamer development packages; macOS: the `Screen` Xcode target against WebRTC).
+The macOS helper's socket lifecycle and candidate rule checks run without capture permissions:
+
+```sh
+swift test --package-path apps/screen-macos-tests --scratch-path build/dd/screen-macos-tests
+```
+
Host development: `cargo run --manifest-path host/Cargo.toml -- serve`. Avoid competing with an installed host on port 19222; isolated tests should use a temporary `CODYNC_HOME` and another port. Stop test hosts when finished.
## Visual checks
diff --git a/docs/plans/android-app.md b/docs/plans/android-app.md
index 2ddeb629..f903ea62 100644
--- a/docs/plans/android-app.md
+++ b/docs/plans/android-app.md
@@ -28,7 +28,7 @@ emulator against a real host; device acceptance is listed separately.
| Notifications | Ported: alerts as conversations with the bot's face (a group's face and its speaking member), conversation shortcuts, what happened ("Task complete") as sub text | Android's conversation layout doesn't draw sub text; background delivery needs FCM in the deployment |
| Live updates | Android 16 Live updates (`ProgressStyle`), the status bar chip on Android 16 QPR1 (API 36.1) | Background updates need FCM in the deployment |
| Version check | `VersionMismatch.MIN_HOST`, the host's `minApp`, the update card in the roster and in place of the composer | Set `MIN_HOST` to the first release that ships Android support when it's cut |
-| Remote screen | Ported: viewer from Computers, the chat header (watch a bot, then take over) and `codync-dev://screen`; WebRTC H.264, trackpad and direct touch, long-press drag, two-finger scroll, pinch zoom streaming the visible region, keyboard with IME and modifier keys, clipboard, displays, take over/hand back, Rotate | Pinch, IME composition and a hardware keyboard on a device; TURN relay path and renewal |
+| Remote screen | Ported: viewer from Computers, the chat header (watch a bot, then take over) and `codync-dev://screen`; WebRTC H.264, trackpad and direct touch, long-press drag, two-finger scroll, pinch zoom streaming the visible region, keyboard with IME and modifier keys, clipboard, displays, take over/hand back, Rotate, trickle ICE that keeps the video through a channel reconnect | Pinch, IME composition and a hardware keyboard on a device; TURN relay path and renewal |
| Analytics | Ported: PostHog feature events with the Settings switch (and its sheet), crash reports sent on the next launch | — |
| Packaging | Debug, R8 release signed with the Play upload key, `versionCode` from the version, Firebase (`codync-auth`) and private configuration, `Submit Android` to Google Play (internal testing since 2.24.0) | Production access (closed test), see [Before a Play release](#before-a-play-release) |
diff --git a/docs/reference/remote-relay.md b/docs/reference/remote-relay.md
index 5110b18b..09526104 100644
--- a/docs/reference/remote-relay.md
+++ b/docs/reference/remote-relay.md
@@ -304,6 +304,7 @@ blob = epk(32) ‖ sig(64) ‖ ct
- `id`:device 端遞增 u32,連線內唯一。
- 每條 channel 最多 32 個進行中的 request、4 個訂閱;超過回 `err 429`。
+- `screenCandidates` subscribes with `{"id":N,"sub":"screenCandidates","b":{"session":"<screen-session-id>"}}`. Its `ev` values are `ready`, ordered `candidate`, `complete`, or `error`; `end` closes the subscription. It requires screen scope and ownership of that screen session.
- `events` 訂閱的語意與 SSE 完全相同(先 subscribe、再 catch-up、`hello` 在最前且 `catchUp` 為其後 catch-up 事件數、落後時送 `{"type":"resync"}`);`client == "ios"` 時計入 `hub.ios_clients`(推播抑制)。
- Bot 刪除標記會保留在 host 資料庫;增量同步與 `since = 0` 的全量同步都傳送它們。手機及桌面在版本更新或解碼重試時會保留快取並重設同步游標,必須收到刪除標記才能移除舊 Bot、對話與草稿;終端使用相同的事件串流。
- 串流沒有額外的 keep-alive;存活由 WebSocket ping(§7.6)負責。
@@ -314,9 +315,11 @@ blob = epk(32) ‖ sig(64) ‖ ct
|---|---|---|
| loopback 專用 | `setScreenEnabled`, `pairing`, `computerCall`, `computerTools`, `chatCall`, `teamCall`, `manageCall`, `memoryCall`, `routineCall`, `composioCall`, `claimSign`, `unclaim`, `devices`, `revokeDevice`, `accessRequests`, `decideAccessRequest`, `cloudStatus`, `setCloud`, `setCloudUrl` | `Caller::Local`(loopback + token) |
| 配對中 | `pair` | 只在 pairing 狀態的 channel;配對完成後不可再呼叫 |
-| screen scope | `screenPrepare`, `screenOffer`, `screenClose`, `screenTakeover` | Local,或 scopes 含 `screen` 的裝置 |
+| screen scope | `screenPrepare`, `screenOffer`, `screenCandidate`, `screenClose`, `screenTakeover` | Local,或 scopes 含 `screen` 的裝置 |
| control scope | 其餘所有既有方法(`hello`, `history`, `createBot`, `updateBot`, `renameSection`, `deleteBot`, `markRead`, `send`, `stop`, `newSession`, `respondPermission`, `registerDevice`, `unregisterDevice`, `registerActivity`, `refreshBackends`, `usage`, `listDirs`, market/skills/connectors、`agentSetup`, `agentAuth`, `agentAuthenticate`, `setAgentEnv`, `termInput`, `termResize`, `termClose`, `screenStatus`) | Local,或 scopes 含 `control` 的裝置 |
+The `screenCandidates` subscription also requires `screen` scope and belongs to its device’s screen session. It carries ordered candidates and completion separately from chat events; see [remote screen](../features/remote-screen.md).
+
v1 發出的 grant 一律 `["control","screen"]`。
### 6.7 推播封裝(host → APNs → 手機)
diff --git a/host/src/api/devices.rs b/host/src/api/devices.rs
index e770a794..abb372b8 100644
--- a/host/src/api/devices.rs
+++ b/host/src/api/devices.rs
@@ -79,7 +79,8 @@ const LOCAL_ONLY: &[&str] = &[
// Who may write to this computer's bots is decided at the computer, like who is in.
"setNetworkRules",
];
-const SCREEN: &[&str] = &["screenPrepare", "screenOffer", "screenClose", "screenTakeover"];
+const SCREEN: &[&str] =
+ &["screenPrepare", "screenOffer", "screenClose", "screenTakeover", "screenCandidate", "screenCandidates"];
/// A method the caller isn't allowed to call (403).
#[derive(Debug)]
@@ -155,6 +156,12 @@ pub fn authorize(hub: &Hub, key: &str) -> Result<Device, RejectCode> {
Ok(d)
}
+/// Whether `key` may still view this computer's screen: `local` always, a device while
+/// it is authorized and holds the screen scope.
+pub fn may_view_screen(hub: &Hub, key: &str) -> bool {
+ key == "local" || authorize(hub, key).is_ok_and(|device| device.scopes.contains(&Scope::Screen))
+}
+
// MARK: pairing codes
const CODE_TTL_MS: i64 = 10 * 60 * 1000;
@@ -254,6 +261,10 @@ mod tests {
assert!(permit(&both, "screenOffer").is_ok());
assert!(permit(&both, "screenPrepare").is_ok());
assert!(permit(&control, "screenPrepare").is_err());
+ assert!(permit(&control, "screenCandidate").is_err());
+ assert!(permit(&control, "screenCandidates").is_err());
+ assert!(permit(&both, "screenCandidate").is_ok());
+ assert!(permit(&both, "screenCandidates").is_ok());
assert!(permit(&control, "screenOffer").is_err(), "screen needs the screen scope");
assert!(permit(&control, "screenStatus").is_ok(), "status is control");
assert!(permit(&control, "installHostUpdate").is_ok(), "a phone can update a computer that's behind");
diff --git a/host/src/api/host.rs b/host/src/api/host.rs
index 2ada0277..522ea8fb 100644
--- a/host/src/api/host.rs
+++ b/host/src/api/host.rs
@@ -147,7 +147,7 @@ pub(super) async fn call(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value
} else {
crate::screen::IceConfig::default()
};
- let result = hub.screen.prepare(caller.device_key(), ice)?;
+ let result = hub.screen.prepare(caller.device_key(), ice, b["trickle"].as_bool().unwrap_or(false))?;
let session = result["session"].as_str().context("missing screen session")?.to_owned();
tokio::spawn(crate::screen::watch_viewer(hub.clone(), session, caller.device_key().to_owned()));
result
@@ -156,6 +156,11 @@ pub(super) async fn call(hub: &Arc<Hub>, caller: &Caller, method: &str, b: Value
let display = b["display"].as_u64().and_then(|d| u32::try_from(d).ok());
hub.screen.offer(caller.device_key(), str_arg(&b, "sdp")?, str_arg(&b, "session")?, display).await?
}
+ "screenCandidate" => {
+ let candidate = serde_json::from_value(b["candidate"].clone())?;
+ hub.screen.candidate(str_arg(&b, "session")?, caller.device_key(), candidate).await?;
+ json!({})
+ }
"screenClose" => {
let session = str_arg(&b, "session")?;
if hub.screen.owns_session(session, caller.device_key()) {
diff --git a/host/src/remote/channel.rs b/host/src/remote/channel.rs
index 9b913b20..65447e9d 100644
--- a/host/src/remote/channel.rs
+++ b/host/src/remote/channel.rs
@@ -8,6 +8,8 @@
//! The device's authorization is re-checked on every request, whenever the device
//! table changes and when its lease runs out; a revoked device is closed at once.
+mod subscriptions;
+
use crate::LockExt;
use crate::api;
use crate::api::devices::{self, Caller, RejectCode};
@@ -494,53 +496,6 @@ impl Channel {
});
}
- async fn subscribe(&self, id: u64, kind: &str, b: &Value) {
- if self.count(true) >= MAX_SUBS {
- self.send(err(id, 429, "too many subscriptions")).await;
- return;
- }
- let stream = match kind {
- "events" => match api::events_stream(
- &self.hub,
- b["since"].as_i64().unwrap_or(0),
- b["client"].as_str(),
- &self.caller,
- ) {
- Ok(s) => s.boxed(),
- Err(e) => {
- self.send(err(id, 500, &format!("{e:#}"))).await;
- return;
- }
- },
- "term" => {
- let Some(s) = api::term_events(&self.hub, b["term"].as_str().unwrap_or_default()) else {
- self.send(err(id, 404, "that terminal is gone")).await;
- return;
- };
- s.boxed()
- }
- _ => {
- self.send(err(id, 400, "unknown subscription")).await;
- return;
- }
- };
- let (jobs, tx) = (self.jobs.clone(), self.tx.clone());
- // Held while spawning so the task can't finish (and remove itself) before it's listed.
- let mut list = self.jobs.locked();
- let task = tokio::spawn(async move {
- let mut stream = stream;
- while let Some(ev) = stream.next().await {
- if tx.send(ToDevice::Inner(json!({"id": id, "ev": ev}))).await.is_err() {
- return;
- }
- }
- if jobs.locked().remove(&id).is_some() {
- let _ = tx.send(ToDevice::Inner(json!({"id": id, "end": true}))).await;
- }
- });
- list.insert(id, Job::Sub(task.abort_handle()));
- }
-
async fn cancel(&self, id: u64) {
let job = self.jobs.locked().remove(&id);
if let Some(Job::Sub(h)) = job {
@@ -614,6 +569,7 @@ impl Channel {
#[cfg(test)]
mod tests {
#![allow(clippy::unwrap_used, clippy::expect_used)]
+ mod screen_signaling;
use super::*;
use crate::remote::identity::Identity;
use crate::store::Store;
diff --git a/host/src/remote/channel/subscriptions.rs b/host/src/remote/channel/subscriptions.rs
new file mode 100644
index 00000000..5187a246
--- /dev/null
+++ b/host/src/remote/channel/subscriptions.rs
@@ -0,0 +1,114 @@
+//! Private encrypted channel subscriptions.
+
+use super::*;
+use tokio::sync::watch;
+
+/// Who may keep receiving one kind of subscription.
+type Gate = fn(&Hub, &str) -> bool;
+
+/// Keep checking permissions even while a slow device fills the outbound queue.
+pub(super) async fn send_allowed(
+ hub: &Hub,
+ owner: &str,
+ allowed: Gate,
+ auth: &mut watch::Receiver<u64>,
+ tx: &mpsc::Sender<ToDevice>,
+ packet: Value,
+) -> bool {
+ if !allowed(hub, owner) {
+ return false;
+ }
+ let send = tx.send(ToDevice::Inner(packet));
+ tokio::pin!(send);
+ loop {
+ tokio::select! {
+ biased;
+ changed = auth.changed() => {
+ if changed.is_err() || !allowed(hub, owner) { return false; }
+ }
+ result = &mut send => return result.is_ok(),
+ }
+ }
+}
+
+impl Channel {
+ pub(super) async fn subscribe(&self, id: u64, kind: &str, b: &Value) {
+ if self.count(true) >= MAX_SUBS {
+ self.send(err(id, 429, "too many subscriptions")).await;
+ return;
+ }
+ // Subscribe before reserving the receiver so permission changes cannot fall
+ // between the initial authorization and the spawned subscription task.
+ let mut auth = self.hub.auth.subscribe();
+ let stream = match kind {
+ "events" => match api::events_stream(
+ &self.hub,
+ b["since"].as_i64().unwrap_or(0),
+ b["client"].as_str(),
+ &self.caller,
+ ) {
+ Ok(s) => s.boxed(),
+ Err(e) => {
+ self.send(err(id, 500, &format!("{e:#}"))).await;
+ return;
+ }
+ },
+ "screenCandidates" => {
+ if let Err(error) = devices::permit(&self.caller, "screenCandidates") {
+ self.send(err(id, 403, &error.to_string())).await;
+ return;
+ }
+ match self.hub.screen.candidates(b["session"].as_str().unwrap_or_default(), self.caller.device_key()) {
+ Ok(stream) => stream.boxed(),
+ Err(error) => {
+ self.send(err(id, 400, &error.to_string())).await;
+ return;
+ }
+ }
+ }
+ "term" => {
+ let Some(s) = api::term_events(&self.hub, b["term"].as_str().unwrap_or_default()) else {
+ self.send(err(id, 404, "that terminal is gone")).await;
+ return;
+ };
+ s.boxed()
+ }
+ _ => {
+ self.send(err(id, 400, "unknown subscription")).await;
+ return;
+ }
+ };
+ let allowed: Gate = if kind == "screenCandidates" { devices::may_view_screen } else { |_, _| true };
+ let hub = self.hub.clone();
+ let owner = self.caller.device_key().to_owned();
+ let (jobs, tx) = (self.jobs.clone(), self.tx.clone());
+ // Held while spawning so the task can't finish (and remove itself) before it's listed.
+ let mut list = self.jobs.locked();
+ let task = tokio::spawn(async move {
+ let mut stream = stream;
+ loop {
+ if !allowed(&hub, &owner) {
+ break;
+ }
+ let ev = tokio::select! {
+ biased;
+ changed = auth.changed() => {
+ if changed.is_err() || !allowed(&hub, &owner) { break; }
+ continue;
+ }
+ event = stream.next() => { let Some(event) = event else { break }; event }
+ };
+ if !send_allowed(&hub, &owner, allowed, &mut auth, &tx, json!({"id": id, "ev": ev})).await {
+ break;
+ }
+ }
+ // Dropping a screen receiver closes its viewer, even if the final end
+ // notification cannot yet fit in this device's outgoing queue.
+ drop(stream);
+ if jobs.locked().remove(&id).is_some() {
+ let _ = tx.send(ToDevice::Inner(json!({"id": id, "end": true}))).await;
+ }
+ });
+ list.insert(id, Job::Sub(task.abort_handle()));
+ }
+}
diff --git a/host/src/remote/channel/tests/screen_signaling.rs b/host/src/remote/channel/tests/screen_signaling.rs
new file mode 100644
index 00000000..bba5fc4c
--- /dev/null
+++ b/host/src/remote/channel/tests/screen_signaling.rs
@@ -0,0 +1,77 @@
+use super::*;
+use crate::remote::channel::subscriptions::send_allowed;
+
+#[tokio::test]
+async fn screen_candidate_rpc_and_private_subscription_require_screen_scope() {
+ let hub = temp_hub();
+ let key = paired_phone(&hub).await;
+ let dk = crypto::b64(key.verifying_key().as_bytes());
+ let mut device = hub.store.device(&dk).unwrap();
+ device.scopes = vec![Scope::Control];
+ hub.store.put_device(&device).unwrap();
+ let mut phone = Phone::direct(&hub, key.clone());
+ let welcome = phone.hello(&hub, false).await;
+ assert!(matches!(welcome, Out::Msg(v) if v["t"] == "welcome"));
+ let reply = phone
+ .call(1, "screenCandidate", json!({"session": "someone-elses-session", "candidate": {"type": "complete"}}))
+ .await;
+ assert_eq!(reply["err"]["status"], 403);
+ phone.send(json!({"id": 2, "sub": "screenCandidates", "b": {"session": "someone-elses-session"}})).await;
+ let reply = phone.recv().await;
+ assert_eq!(reply.unwrap()["err"]["status"], 403);
+ let reply = phone.call(3, "hello", json!({})).await;
+ assert!(reply["ok"].is_object(), "denied screen access doesn't end authorized chat access");
+}
+
+#[tokio::test]
+async fn queued_screen_candidates_stop_when_screen_permission_is_removed() {
+ let hub = temp_hub();
+ let key = paired_phone(&hub).await;
+ let owner = crypto::b64(key.verifying_key().as_bytes());
+ let mut auth = hub.auth.subscribe();
+ let (tx, mut rx) = mpsc::channel(1);
+ let filled = tx.send(ToDevice::Inner(json!({"filler": true}))).await;
+ filled.unwrap();
+ let candidate = json!({"id": 1, "ev": {"type": "candidate"}});
+ let sending = send_allowed(&hub, &owner, devices::may_view_screen, &mut auth, &tx, candidate.clone());
+ tokio::pin!(sending);
+ assert!(futures::poll!(&mut sending).is_pending());
+
+ let mut device = hub.store.device(&owner).unwrap();
+ device.scopes = vec![Scope::Control];
+ hub.store.put_device(&device).unwrap();
+ hub.auth_changed();
+ let sent = timeout(Duration::from_secs(1), sending).await;
+ assert!(!sent.unwrap(), "revocation cancels a send without waiting for queue space");
+ let queued = rx.recv().await;
+ assert!(matches!(queued, Some(ToDevice::Inner(v)) if v["filler"] == true));
+ assert!(rx.try_recv().is_err());
+
+ // Also reject a stale caller whose scope changed before its task began.
+ let mut auth = hub.auth.subscribe();
+ let sent = send_allowed(&hub, &owner, devices::may_view_screen, &mut auth, &tx, candidate).await;
+ assert!(!sent);
+ assert!(rx.try_recv().is_err());
+}
+
+#[tokio::test]
+async fn unrelated_permission_updates_preserve_a_backpressured_screen_candidate() {
+ let hub = temp_hub();
+ let key = paired_phone(&hub).await;
+ let owner = crypto::b64(key.verifying_key().as_bytes());
+ let mut auth = hub.auth.subscribe();
+ let (tx, mut rx) = mpsc::channel(1);
+ let filled = tx.send(ToDevice::Inner(json!({"filler": true}))).await;
+ filled.unwrap();
+ let candidate = json!({"id": 1, "ev": {"type": "candidate"}});
+ let sending = send_allowed(&hub, &owner, devices::may_view_screen, &mut auth, &tx, candidate.clone());
+ tokio::pin!(sending);
+ assert!(futures::poll!(&mut sending).is_pending());
+ hub.auth_changed();
+ assert!(futures::poll!(&mut sending).is_pending());
+ let _ = rx.recv().await;
+ let sent = sending.await;
+ assert!(sent);
+ let queued = rx.recv().await;
+ assert!(matches!(queued, Some(ToDevice::Inner(v)) if v == candidate));
+}
diff --git a/host/src/screen/helper.rs b/host/src/screen/helper.rs
index 080b9397..ff222859 100644
--- a/host/src/screen/helper.rs
+++ b/host/src/screen/helper.rs
@@ -21,7 +21,6 @@ pub(super) type Pending = Mutex<HashMap<i64, oneshot::Sender<Result<Value, Strin
/// One connected screen helper.
pub(super) struct Link {
- #[cfg_attr(windows, expect(dead_code, reason = "Windows has no screen helper yet"))]
pub(super) id: u64,
pub(super) tx: mpsc::UnboundedSender<String>,
pub(super) pending: Pending,
@@ -140,6 +139,7 @@ pub async fn supervise_linux_helper(screen: Arc<Screen>) {
#[cfg(unix)]
async fn run_link(screen: Arc<Screen>, stream: UnixStream) {
+ let mut generation = screen.helper_generation.subscribe();
let (rd, mut wr) = stream.into_split();
let (tx, mut rx) = mpsc::unbounded_channel::<String>();
let link = Arc::new(Link {
@@ -148,7 +148,12 @@ async fn run_link(screen: Arc<Screen>, stream: UnixStream) {
pending: Mutex::default(),
next_id: AtomicI64::new(1),
});
- *screen.link.locked() = Some(link.clone());
+ {
+ let mut slot = screen.link.locked();
+ *slot = Some(link.clone());
+ screen.reset_helper_state();
+ screen.helper_generation.send_replace(link.id);
+ }
tracing::info!("screen helper connected");
let writer = tokio::spawn(async move {
while let Some(mut line) = rx.recv().await {
@@ -159,37 +164,27 @@ async fn run_link(screen: Arc<Screen>, stream: UnixStream) {
}
});
let mut lines = BufReader::new(rd).lines();
- while let Ok(Some(line)) = lines.next_line().await {
- let Ok(msg) = serde_json::from_str::<Value>(&line) else {
- tracing::debug!(line, "ignoring non-JSON from the screen helper");
+ loop {
+ let packet = tokio::select! {
+ packet = lines.next_line() => { let Ok(Some(packet)) = packet else { break }; packet }
+ changed = generation.changed() => {
+ if changed.is_err() || *generation.borrow() != link.id { break; }
+ continue;
+ }
+ };
+ let Ok(msg) = serde_json::from_str::<Value>(&packet) else {
+ tracing::debug!(packet, "ignoring non-JSON from the screen helper");
continue;
};
- match (msg["method"].as_str(), msg["id"].as_i64()) {
- (None, Some(id)) => {
- if let Some(waiter) = link.pending.locked().remove(&id) {
- let res = match msg.get("error") {
- Some(e) => Err(e["message"].as_str().map_or_else(|| e.to_string(), str::to_owned)),
- None => Ok(msg["result"].clone()),
- };
- let _ = waiter.send(res);
- }
- }
- (Some("status"), _) => match serde_json::from_value::<HelperStatus>(msg["params"].clone()) {
- Ok(st) => {
- *screen.status.locked() = st;
- screen.emit();
- }
- Err(error) => tracing::warn!(%error, "bad status from the screen helper"),
- },
- (Some("session"), _) => {
- if msg["params"]["state"] == "closed"
- && let Some(s) = msg["params"]["session"].as_str()
- {
- screen.session_closed(s);
- }
- }
- (Some(method), _) => tracing::debug!(method, "unknown screen helper notification"),
- _ => {}
+ // Keep replacement atomic with every notification's state changes.
+ let current = screen.link.locked();
+ if current.as_ref().is_none_or(|current| current.id != link.id) {
+ break;
+ }
+ let announce = handle_message(&screen, &link, &msg);
+ drop(current);
+ if announce {
+ screen.emit();
}
}
writer.abort();
@@ -199,79 +194,74 @@ async fn run_link(screen: Arc<Screen>, stream: UnixStream) {
let current = slot.as_ref().is_some_and(|l| l.id == link.id);
if current {
*slot = None;
+ screen.reset_helper_state();
}
current
};
if current {
- *screen.status.locked() = HelperStatus::default();
- screen.sessions.locked().clear();
- screen.control.locked().user = false;
screen.emit();
}
tracing::info!("screen helper disconnected");
}
-#[cfg(all(test, unix))]
-mod tests {
- use super::*;
- use crate::screen::tests::retina;
- use crate::screen::viewer::{IceConfig, IceServer};
- use tokio::sync::broadcast;
-
- #[tokio::test]
- async fn helper_roundtrip_over_the_socket() {
- let (tx, _) = broadcast::channel(8);
- let screen = Arc::new(Screen::new(Some(true), tx));
- let (a, b) = UnixStream::pair().unwrap();
- let (observed_tx, mut observed_rx) = mpsc::unbounded_channel();
- // Fake helper reports a display and records the host's ICE configuration.
- tokio::spawn(async move {
- let (rd, mut wr) = b.into_split();
- let status = json!({"jsonrpc": "2.0", "method": "status", "params": {"platform": "test", "capture": true, "displays": [retina()]}});
- wr.write_all(format!("{status}\n").as_bytes()).await.unwrap();
- let mut lines = BufReader::new(rd).lines();
- while let Ok(Some(l)) = lines.next_line().await {
- let m: Value = serde_json::from_str(&l).unwrap();
- observed_tx.send(m.clone()).unwrap();
- let res = json!({"jsonrpc": "2.0", "id": m["id"], "result": {"sdp": format!("answer to {}", m["params"]["sdp"].as_str().unwrap_or_default())}});
- wr.write_all(format!("{res}\n").as_bytes()).await.unwrap();
+/// Applies one message from the helper; true when clients should hear about the change.
+#[cfg(unix)]
+fn handle_message(screen: &Screen, link: &Link, msg: &Value) -> bool {
+ match (msg["method"].as_str(), msg["id"].as_i64()) {
+ (None, Some(id)) => {
+ if let Some(waiter) = link.pending.locked().remove(&id) {
+ let res = match msg.get("error") {
+ Some(e) => Err(e["message"].as_str().map_or_else(|| e.to_string(), str::to_owned)),
+ None => Ok(msg["result"].clone()),
+ };
+ let _ = waiter.send(res);
}
- });
- tokio::spawn(run_link(screen.clone(), a));
- for _ in 0..50 {
- if screen.status.locked().capture {
- break;
+ false
+ }
+ (Some("status"), _) => match serde_json::from_value::<HelperStatus>(msg["params"].clone()) {
+ Ok(st) => {
+ *screen.status.locked() = st;
+ true
+ }
+ Err(error) => {
+ tracing::warn!(%error, "bad status from the screen helper");
+ false
+ }
+ },
+ (Some("candidate"), _) => {
+ if let Some(session) = msg["params"]["session"].as_str() {
+ screen.helper_candidate(session, msg["params"]["candidate"].clone());
}
- tokio::time::sleep(Duration::from_millis(10)).await;
+ false
}
- let local = screen.prepare("local", IceConfig::default()).unwrap();
- let res = screen.offer("local", "offer", local["session"].as_str().unwrap(), None).await.unwrap();
- assert_eq!(res["sdp"], "answer to offer");
- assert_eq!(screen.state()["viewers"], 1);
- assert_eq!(screen.state()["displays"][0]["name"], "Built-in");
- let direct = observed_rx.recv().await.unwrap();
- assert_eq!(direct["params"]["iceServers"], json!([]));
- assert_eq!(direct["params"]["maxBitrateBps"], 16_000_000);
- let config = IceConfig {
- ice_servers: vec![IceServer {
- urls: vec!["turn:turn.cloudflare.com:3478?transport=udp".into()],
- username: Some("u".into()),
- credential: Some("p".into()),
- }],
- ..IceConfig::default()
- };
- screen.takeover(true);
- let prepared = screen.prepare("phone", config).unwrap();
- screen.session_closed(res["session"].as_str().unwrap());
- assert_eq!(screen.state()["userControl"], true, "renewal keeps takeover while the replacement is pending");
- let id = prepared["session"].as_str().unwrap();
- assert!(screen.offer("other", "offer", id, None).await.is_err());
- screen.offer("phone", "offer", id, None).await.unwrap();
- let remote = observed_rx.recv().await.unwrap();
- assert_eq!(remote["params"]["iceServers"], prepared["iceServers"]);
- assert_eq!(remote["params"]["maxBitrateBps"], 4_000_000);
- assert_eq!(remote["params"]["maxFramerate"], 30);
- screen.sessions.locked().get_mut(id).unwrap().ice.expires_at = 0;
- assert!(screen.offer("phone", "offer", id, None).await.is_err());
+ (Some("session"), _) => {
+ let Some(session) = msg["params"]["session"].as_str() else { return false };
+ match msg["params"]["state"].as_str() {
+ Some("closed") => screen.remove_session(session),
+ Some("connected") => {
+ screen.session_connected(session);
+ false
+ }
+ _ => false,
+ }
+ }
+ (Some(method), _) => {
+ tracing::debug!(method, "unknown screen helper notification");
+ false
+ }
+ _ => false,
}
}
+
+#[cfg(unix)]
+impl Screen {
+ /// Nothing from a previous helper carries over to the next one.
+ fn reset_helper_state(&self) {
+ self.sessions.locked().clear();
+ *self.status.locked() = HelperStatus::default();
+ self.control.locked().user = false;
+ }
+}
+
+#[cfg(all(test, unix))]
+mod tests;
diff --git a/host/src/screen/helper/tests.rs b/host/src/screen/helper/tests.rs
new file mode 100644
index 00000000..f664145b
--- /dev/null
+++ b/host/src/screen/helper/tests.rs
@@ -0,0 +1,211 @@
+#![allow(clippy::unwrap_used)]
+
+use super::*;
+use crate::screen::tests::retina;
+use crate::screen::viewer::{IceConfig, IceServer};
+use std::sync::mpsc as sync_mpsc;
+use tokio::sync::broadcast;
+
+fn screen() -> Arc<Screen> {
+ let (events, _) = broadcast::channel(8);
+ Arc::new(Screen::new(Some(true), events))
+}
+
+fn status_line(params: &Value) -> String {
+ format!("{}\n", json!({"jsonrpc": "2.0", "method": "status", "params": params}))
+}
+
+/// Connects a fake helper that reports `status`, once the host has applied it.
+async fn connect_helper(screen: &Arc<Screen>, status: &Value) -> UnixStream {
+ let (link, mut peer) = UnixStream::pair().unwrap();
+ tokio::spawn(run_link(screen.clone(), link));
+ let sent = peer.write_all(status_line(status).as_bytes()).await;
+ sent.unwrap();
+ for _ in 0..50 {
+ if screen.status.locked().capture {
+ break;
+ }
+ tokio::time::sleep(Duration::from_millis(10)).await;
+ }
+ peer
+}
+
+/// Another thread holding the status lock, as a status update in progress does.
+struct StatusLock {
+ checked: sync_mpsc::Receiver<bool>,
+ release: sync_mpsc::Sender<()>,
+ thread: std::thread::JoinHandle<()>,
+}
+
+/// Returns once the lock is held; the thread then watches whether anyone holds the link lock.
+fn hold_status_lock(screen: &Arc<Screen>) -> StatusLock {
+ let (held, holding) = sync_mpsc::channel();
+ let (checked, checking) = sync_mpsc::channel();
+ let (release, releasing) = sync_mpsc::channel();
+ let thread = std::thread::spawn({
+ let screen = screen.clone();
+ move || {
+ let _status = screen.status.locked();
+ held.send(()).unwrap();
+ let protected = (0..100).any(|_| {
+ let taken = screen.link.try_lock().is_err();
+ if !taken {
+ std::thread::sleep(Duration::from_millis(5));
+ }
+ taken
+ });
+ checked.send(protected).unwrap();
+ // A timeout also releases the lock if an assertion fails.
+ let _ = releasing.recv_timeout(Duration::from_secs(3));
+ }
+ });
+ holding.recv_timeout(Duration::from_secs(1)).unwrap();
+ StatusLock { checked: checking, release, thread }
+}
+
+impl StatusLock {
+ /// Whether the link lock was held while the status lock was.
+ fn link_was_held(&self) -> bool {
+ self.checked.recv_timeout(Duration::from_secs(1)).unwrap()
+ }
+
+ fn release(self) {
+ self.release.send(()).unwrap();
+ self.thread.join().unwrap();
+ }
+}
+
+async fn wait_for_replacement(screen: &Screen, old_id: u64) {
+ for _ in 0..50 {
+ if screen.link().unwrap().id != old_id && screen.status.locked().capture {
+ break;
+ }
+ tokio::time::sleep(Duration::from_millis(10)).await;
+ }
+}
+
+#[tokio::test]
+async fn helper_roundtrip_over_the_socket() {
+ let screen = screen();
+ let peer = connect_helper(&screen, &json!({"platform": "test", "capture": true, "displays": [retina()]})).await;
+ let (observed_tx, mut observed_rx) = mpsc::unbounded_channel();
+ // Fake helper records the host's ICE configuration.
+ tokio::spawn(async move {
+ let (rd, mut wr) = peer.into_split();
+ let mut lines = BufReader::new(rd).lines();
+ while let Ok(Some(l)) = lines.next_line().await {
+ let m: Value = serde_json::from_str(&l).unwrap();
+ observed_tx.send(m.clone()).unwrap();
+ if m["method"] == "answer" {
+ let connected = json!({"jsonrpc": "2.0", "method": "session", "params": {"session": m["params"]["session"], "state": "connected"}});
+ let written = wr.write_all(format!("{connected}\n").as_bytes()).await;
+ written.unwrap();
+ }
+ let res = json!({"jsonrpc": "2.0", "id": m["id"], "result": {"sdp": format!("answer to {}", m["params"]["sdp"].as_str().unwrap_or_default())}});
+ wr.write_all(format!("{res}\n").as_bytes()).await.unwrap();
+ }
+ });
+ let local = screen.prepare("local", IceConfig::default(), false).unwrap();
+ let res = screen.offer("local", "offer", local["session"].as_str().unwrap(), None).await.unwrap();
+ assert_eq!(res["sdp"], "answer to offer");
+ assert_eq!(screen.state()["viewers"], 1);
+ assert!(matches!(
+ screen.sessions.locked()[local["session"].as_str().unwrap()].state,
+ super::super::ViewerState::Connected
+ ));
+ assert_eq!(screen.state()["displays"][0]["name"], "Built-in");
+ let direct = observed_rx.recv().await.unwrap();
+ assert_eq!(direct["params"]["iceServers"], json!([]));
+ assert_eq!(direct["params"]["maxBitrateBps"], 16_000_000);
+ let config = IceConfig {
+ ice_servers: vec![IceServer {
+ urls: vec!["turn:turn.cloudflare.com:3478?transport=udp".into()],
+ username: Some("u".into()),
+ credential: Some("p".into()),
+ }],
+ ..IceConfig::default()
+ };
+ screen.takeover(true);
+ let prepared = screen.prepare("phone", config, false).unwrap();
+ screen.session_closed(res["session"].as_str().unwrap());
+ assert_eq!(screen.state()["userControl"], true, "renewal keeps takeover while the replacement is pending");
+ let id = prepared["session"].as_str().unwrap();
+ assert!(screen.offer("other", "offer", id, None).await.is_err());
+ screen.offer("phone", "offer", id, None).await.unwrap();
+ let remote = observed_rx.recv().await.unwrap();
+ assert_eq!(remote["params"]["iceServers"], prepared["iceServers"]);
+ assert_eq!(remote["params"]["maxBitrateBps"], 4_000_000);
+ assert_eq!(remote["params"]["maxFramerate"], 30);
+ screen.sessions.locked().get_mut(id).unwrap().ice.expires_at = 0;
+ assert!(screen.offer("phone", "offer", id, None).await.is_err());
+}
+
+#[tokio::test]
+async fn replacing_a_helper_ends_its_socket_and_private_signaling() {
+ use futures::StreamExt as _;
+ use tokio::io::AsyncReadExt as _;
+ let screen = screen();
+ let status = json!({"capture": true, "trickle": true});
+ let mut first_peer = connect_helper(&screen, &status).await;
+ let prepared = screen.prepare("phone", IceConfig::default(), true).unwrap();
+ let session = prepared["session"].as_str().unwrap();
+ let mut candidates = screen.candidates(session, "phone").unwrap().boxed();
+ let ready = candidates.next().await;
+ assert_eq!(ready.unwrap()["type"], "ready");
+ let old_id = screen.link().unwrap().id;
+ let _second_peer = connect_helper(&screen, &status).await;
+ let mut buffer = [0; 1];
+ let closed = tokio::time::timeout(Duration::from_secs(1), first_peer.read(&mut buffer)).await;
+ assert_eq!(closed.unwrap().unwrap(), 0, "the old helper must stop media when its socket ends");
+ let ended = tokio::time::timeout(Duration::from_secs(1), candidates.next()).await;
+ assert!(ended.unwrap().is_none());
+ assert!(!screen.owns_session(session, "phone"));
+ assert_ne!(screen.link().unwrap().id, old_id);
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn helper_replacement_cannot_overtake_an_in_progress_status_update() {
+ use std::io::Write as _;
+ let screen = screen();
+ let first_peer = connect_helper(&screen, &json!({"capture": true, "trickle": true})).await;
+ let old_id = screen.link().unwrap().id;
+ let (second, mut second_peer) = UnixStream::pair().unwrap();
+ let sent = second_peer.write_all(status_line(&json!({"capture": true, "trickle": true})).as_bytes()).await;
+ sent.unwrap();
+ let mut first_peer = first_peer.into_std().unwrap();
+ first_peer.set_nonblocking(false).unwrap();
+ let status_lock = hold_status_lock(&screen);
+ first_peer.write_all(status_line(&json!({"capture": false, "trickle": false})).as_bytes()).unwrap();
+ let protected = status_lock.link_was_held();
+ tokio::spawn(run_link(screen.clone(), second));
+ status_lock.release();
+ assert!(protected, "the previous helper must hold its generation until the update completes");
+ wait_for_replacement(&screen, old_id).await;
+ assert_ne!(screen.link().unwrap().id, old_id);
+ assert!(screen.status.locked().trickle, "the replacement's capability must win");
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
+async fn preparing_a_viewer_uses_one_helper_generation_and_its_capabilities() {
+ let screen = screen();
+ let _first_peer = connect_helper(&screen, &json!({"capture": true, "trickle": true})).await;
+ let old_id = screen.link().unwrap().id;
+ let (second, mut second_peer) = UnixStream::pair().unwrap();
+ let sent = second_peer.write_all(status_line(&json!({"capture": true, "trickle": false})).as_bytes()).await;
+ sent.unwrap();
+ let status_lock = hold_status_lock(&screen);
+ let preparing = std::thread::spawn({
+ let screen = screen.clone();
+ move || screen.prepare("phone", IceConfig::default(), true)
+ });
+ let protected = status_lock.link_was_held();
+ tokio::spawn(run_link(screen.clone(), second));
+ status_lock.release();
+ let prepared = preparing.join().unwrap().unwrap();
+ assert!(protected, "replacement must wait until preparation stores the helper generation");
+ assert_eq!(prepared["trickle"], true);
+ wait_for_replacement(&screen, old_id).await;
+ assert!(!screen.owns_session(prepared["session"].as_str().unwrap(), "phone"));
+ let fresh = screen.prepare("phone", IceConfig::default(), true).unwrap();
+ assert_eq!(fresh["trickle"], false);
+}
diff --git a/host/src/screen/mod.rs b/host/src/screen/mod.rs
index dbb61d0e..1a8f7814 100644
--- a/host/src/screen/mod.rs
+++ b/host/src/screen/mod.rs
@@ -14,9 +14,11 @@
//! - helper → host notifications
//! - `status` [`HelperStatus`]
//! - `session {session, state: "connected" | "closed"}`
+//! - `candidate {session, candidate}`: ordered ICE candidates and completion for trickle sessions
//! - host → helper requests
-//! - `answer {session, sdp, display?, iceServers, maxBitrateBps, maxFramerate}` → `{sdp}`:
-//! non-trickle ICE, with short-lived STUN/TURN credentials for remote connections.
+//! - `answer {session, sdp, trickle?, display?, iceServers, maxBitrateBps, maxFramerate}` → `{sdp}`:
+//! incremental ICE when negotiated by `screenPrepare`, complete SDP otherwise.
+//! - `candidate {session, candidate}` → `{}`: apply one candidate after answering.
//! - `close {session}`, `closeAll {}`
//! - `focusedField {pid?}` → `{app, pid, url?, secure}`: the focused control of that app (the
//! frontmost one by default), for `type_login`
@@ -37,6 +39,7 @@ mod helper;
mod helper_tools;
mod permissions;
mod protocol;
+mod signaling;
mod viewer;
pub use computer::{computer, computer_tools};
@@ -55,7 +58,7 @@ use std::sync::Arc;
use std::sync::Mutex;
use std::sync::atomic::AtomicU64;
use std::time::{Duration, Instant};
-use tokio::sync::broadcast;
+use tokio::sync::{broadcast, watch};
/// How long after its last computer call a bot still counts as "using the computer".
const AGENT_HOLD: Duration = Duration::from_secs(20);
@@ -79,11 +82,39 @@ impl Control {
}
}
+#[derive(PartialEq, Eq)]
+enum ViewerState {
+ Prepared,
+ Started,
+ #[cfg(any(unix, test))]
+ Connected,
+ Failed,
+}
+
struct Viewer {
owner: String,
ice: IceConfig,
- started: bool,
+ state: ViewerState,
pending_until: i64,
+ link_id: u64,
+ signaling: Option<signaling::Signaling>,
+}
+
+impl Viewer {
+ fn started(&self) -> bool {
+ !matches!(self.state, ViewerState::Prepared | ViewerState::Failed)
+ }
+
+ #[cfg(any(unix, test))]
+ fn media_connected(&self) -> bool {
+ self.state == ViewerState::Connected
+ }
+
+ fn expired(&self, now: i64) -> bool {
+ self.state == ViewerState::Failed
+ || self.ice.expires_at <= now
+ || (!self.started() && self.pending_until <= now)
+ }
}
pub struct Screen {
@@ -97,10 +128,13 @@ pub struct Screen {
driver: cua::Driver,
#[cfg_attr(windows, expect(dead_code, reason = "Windows has no screen helper yet"))]
next_link: AtomicU64,
+ #[cfg_attr(windows, expect(dead_code, reason = "Windows has no screen helper yet"))]
+ helper_generation: watch::Sender<u64>,
}
impl Screen {
pub fn new(enabled: Option<bool>, events: broadcast::Sender<Value>) -> Self {
+ let (helper_generation, _) = watch::channel(0);
Self {
enabled: Mutex::new(enabled),
events,
@@ -110,6 +144,7 @@ impl Screen {
sessions: Mutex::default(),
driver: cua::Driver::default(),
next_link: AtomicU64::new(1),
+ helper_generation,
}
}
@@ -132,7 +167,7 @@ impl Screen {
pub fn state(&self) -> Value {
let st = self.status.locked().clone();
let connected = self.link.locked().is_some();
- let viewers = self.sessions.locked().values().filter(|v| v.started).count();
+ let viewers = self.sessions.locked().values().filter(|v| v.started()).count();
let c = self.control.locked();
json!({
"enabled": self.enabled(),
@@ -154,10 +189,16 @@ impl Screen {
}
fn link(&self) -> Result<Arc<Link>> {
+ self.with_link(|link| Ok(link.clone()))
+ }
+
+ fn with_link<T>(&self, action: impl FnOnce(&Arc<Link>) -> Result<T>) -> Result<T> {
if !self.enabled() {
bail!("Remote screen is turned off on this computer. Turn it on in Codync's menu there.");
}
- self.link.locked().clone().ok_or_else(|| anyhow!("The screen helper isn't running on this computer."))
+ let slot = self.link.locked();
+ let link = slot.as_ref().ok_or_else(|| anyhow!("The screen helper isn't running on this computer."))?;
+ action(link)
}
#[cfg(target_os = "linux")]
diff --git a/host/src/screen/protocol.rs b/host/src/screen/protocol.rs
index 365015dc..3d537342 100644
--- a/host/src/screen/protocol.rs
+++ b/host/src/screen/protocol.rs
@@ -32,4 +32,7 @@ pub struct HelperStatus {
/// Input injection (and the accessibility tree) is permitted.
#[serde(default)]
pub input: bool,
+ /// Both directions support incremental ICE candidates.
+ #[serde(default)]
+ pub trickle: bool,
}
diff --git a/host/src/screen/signaling.rs b/host/src/screen/signaling.rs
new file mode 100644
index 00000000..dc94d6fc
--- /dev/null
+++ b/host/src/screen/signaling.rs
@@ -0,0 +1,305 @@
+//! One ordered, private ICE exchange per prepared viewer.
+
+use super::{Screen, Viewer, ViewerState};
+use crate::LockExt;
+use anyhow::{Context, Result, bail};
+use futures::{Stream, StreamExt as _, stream};
+use serde::{Deserialize, Serialize};
+use serde_json::{Value, json};
+use std::sync::Arc;
+use tokio::sync::{mpsc, watch};
+
+const MAX_CANDIDATES: usize = 128;
+const MAX_CANDIDATE_BYTES: usize = 4096;
+const MAX_MEDIA_INDEX: u32 = 16;
+const MAX_MID_BYTES: usize = 256;
+
+#[derive(Clone, Debug, Deserialize, Serialize)]
+#[serde(tag = "type", rename_all = "camelCase")]
+pub enum Candidate {
+ Candidate {
+ candidate: String,
+ #[serde(rename = "sdpMLineIndex")]
+ sdp_m_line_index: u32,
+ #[serde(rename = "sdpMid", default)]
+ sdp_mid: Option<String>,
+ },
+ Complete,
+}
+
+impl Candidate {
+ fn validate(&self) -> Result<()> {
+ if let Self::Candidate { candidate, sdp_m_line_index, sdp_mid } = self
+ && (!candidate.starts_with("candidate:")
+ || candidate.len() > MAX_CANDIDATE_BYTES
+ || *sdp_m_line_index > MAX_MEDIA_INDEX
+ || sdp_mid.as_ref().is_some_and(|mid| mid.len() > MAX_MID_BYTES)
+ || candidate.contains(['\r', '\n']))
+ {
+ bail!("invalid screen candidate");
+ }
+ Ok(())
+ }
+}
+
+#[derive(Default)]
+struct Progress {
+ count: usize,
+ complete: bool,
+}
+
+impl Progress {
+ fn accept(&mut self, candidate: &Candidate) -> Result<()> {
+ candidate.validate()?;
+ if self.complete {
+ bail!("screen candidates already complete");
+ }
+ match candidate {
+ Candidate::Complete => self.complete = true,
+ Candidate::Candidate { .. } => {
+ if self.count >= MAX_CANDIDATES {
+ bail!("too many screen candidates");
+ }
+ self.count += 1;
+ }
+ }
+ Ok(())
+ }
+}
+
+#[derive(Default, PartialEq, Eq)]
+enum Offer {
+ #[default]
+ Prepared,
+ Answering,
+ Answered,
+}
+
+pub(super) struct Signaling {
+ #[cfg(any(unix, test))]
+ tx: mpsc::Sender<Value>,
+ rx: Option<mpsc::Receiver<Value>>,
+ failure: watch::Sender<Option<String>>,
+ offer: Offer,
+ incoming: Progress,
+ #[cfg(any(unix, test))]
+ outgoing: Progress,
+ uploading: bool,
+ detached: bool,
+ outgoing_complete_delivered: bool,
+}
+
+impl Signaling {
+ pub(super) fn new() -> Self {
+ let (tx, rx) = mpsc::channel(MAX_CANDIDATES + 1);
+ // Windows has no helper socket to produce candidates yet.
+ #[cfg(not(any(unix, test)))]
+ drop(tx);
+ let (failure, _) = watch::channel(None);
+ Self {
+ #[cfg(any(unix, test))]
+ tx,
+ rx: Some(rx),
+ failure,
+ offer: Offer::Prepared,
+ incoming: Progress::default(),
+ #[cfg(any(unix, test))]
+ outgoing: Progress::default(),
+ uploading: false,
+ detached: false,
+ outgoing_complete_delivered: false,
+ }
+ }
+
+ pub(super) fn fail(&self, message: &str) {
+ self.failure.send_if_modified(|reason| {
+ if reason.is_some() {
+ return false;
+ }
+ *reason = Some(message.to_owned());
+ true
+ });
+ }
+
+ fn check(&self) -> Result<()> {
+ if let Some(reason) = self.failure.borrow().as_ref() {
+ bail!("{reason}");
+ }
+ Ok(())
+ }
+
+ pub(super) fn begin_offer(&mut self) -> Result<()> {
+ self.check()?;
+ if self.rx.is_some() {
+ bail!("open screenCandidates before offering");
+ }
+ if self.offer != Offer::Prepared {
+ bail!("screen session already offered");
+ }
+ self.offer = Offer::Answering;
+ Ok(())
+ }
+
+ pub(super) fn answered(&mut self) -> Result<()> {
+ self.check()?;
+ self.offer = Offer::Answered;
+ Ok(())
+ }
+
+ fn complete(&self) -> bool {
+ self.incoming.complete && self.outgoing_complete_delivered && !self.uploading
+ }
+}
+
+impl Viewer {
+ /// The viewer is done: watchers close it and its candidate stream reports `reason`.
+ fn fail(&mut self, reason: &str) {
+ self.state = ViewerState::Failed;
+ if let Some(signaling) = &self.signaling {
+ signaling.fail(reason);
+ }
+ }
+
+ fn signaling(&mut self) -> Result<&mut Signaling> {
+ if self.expired(crate::store::now_ms()) {
+ bail!("screen session expired");
+ }
+ let signaling = self.signaling.as_mut().context("screen session does not support trickle ICE")?;
+ if signaling.detached {
+ bail!("screen signaling disconnected");
+ }
+ Ok(signaling)
+ }
+}
+
+impl Screen {
+ pub fn candidates(
+ self: &Arc<Self>,
+ session: &str,
+ owner: &str,
+ ) -> Result<impl Stream<Item = Value> + Send + 'static> {
+ self.link()?;
+ let (rx, failure) = {
+ let mut sessions = self.sessions.locked();
+ let viewer = sessions.get_mut(session).filter(|v| v.owner == owner).context("unknown screen session")?;
+ let signaling = viewer.signaling()?;
+ signaling.check()?;
+ let rx = signaling.rx.take().context("screen candidates already subscribed")?;
+ (rx, signaling.failure.subscribe())
+ };
+ let guard = Subscription { screen: self.clone(), session: session.to_owned() };
+ // Ends right after reporting a failure, without draining possibly stale candidates.
+ let events = stream::unfold((Some((rx, failure)), guard), |(live, guard)| async move {
+ let (mut rx, mut failure) = live?;
+ let event = tokio::select! {
+ biased;
+ reason = failure.wait_for(Option::is_some) => {
+ let message = reason.ok()?.clone().unwrap_or_default();
+ return Some((json!({"type": "error", "message": message}), (None, guard)));
+ }
+ event = rx.recv() => event?,
+ };
+ if event["type"] == "complete" {
+ let mut sessions = guard.screen.sessions.locked();
+ if let Some(signaling) = sessions.get_mut(&guard.session).and_then(|viewer| viewer.signaling.as_mut()) {
+ signaling.outgoing_complete_delivered = true;
+ }
+ }
+ Some((event, (Some((rx, failure)), guard)))
+ });
+ Ok(stream::once(futures::future::ready(json!({"type": "ready"}))).chain(events))
+ }
+
+ pub async fn candidate(&self, session: &str, owner: &str, candidate: Candidate) -> Result<()> {
+ let link = self.link()?;
+ {
+ let mut sessions = self.sessions.locked();
+ let viewer = sessions.get_mut(session).filter(|v| v.owner == owner).context("unknown screen session")?;
+ let signaling = viewer.signaling()?;
+ signaling.check()?;
+ if signaling.offer != Offer::Answered || signaling.uploading {
+ bail!("screen candidate is not ready");
+ }
+ if let Err(error) = signaling.incoming.accept(&candidate) {
+ viewer.fail(&error.to_string());
+ return Err(error);
+ }
+ signaling.uploading = true;
+ }
+ let result = link.request("candidate", json!({"session": session, "candidate": candidate})).await;
+ let mut sessions = self.sessions.locked();
+ let viewer = sessions.get_mut(session).context("screen session closed")?;
+ viewer.signaling()?.uploading = false;
+ if let Err(error) = &result {
+ viewer.fail(&error.to_string());
+ }
+ result?;
+ viewer.signaling()?.check()
+ }
+
+ #[cfg(any(unix, test))]
+ pub(super) fn helper_candidate(&self, session: &str, candidate: Value) {
+ let mut sessions = self.sessions.locked();
+ let Some(viewer) = sessions.get_mut(session) else { return };
+ let Some(signaling) = viewer.signaling.as_mut() else { return };
+ if signaling.detached {
+ return;
+ }
+ let result = serde_json::from_value::<Candidate>(candidate.clone())
+ .map_err(anyhow::Error::from)
+ .and_then(|candidate| signaling.outgoing.accept(&candidate));
+ if let Err(error) = result {
+ viewer.fail(&error.to_string());
+ return;
+ }
+ if signaling.offer == Offer::Prepared || signaling.tx.try_send(candidate).is_err() {
+ viewer.fail("screen candidate stream overflow or closed");
+ }
+ }
+
+ pub(super) fn fail_signaling(&self, session: &str, message: &str) {
+ if let Some(viewer) = self.sessions.locked().get_mut(session) {
+ viewer.fail(message);
+ }
+ }
+
+ /// Candidate delivery ends with the channel; established media has its own lifetime.
+ fn signaling_disconnected(&self, session: &str) -> bool {
+ let mut sessions = self.sessions.locked();
+ let Some(viewer) = sessions.get_mut(session) else { return false };
+ #[cfg(any(unix, test))]
+ let connected = viewer.media_connected();
+ #[cfg(not(any(unix, test)))]
+ let connected = false;
+ let Some(signaling) = viewer.signaling.as_mut() else { return false };
+ if viewer.state != ViewerState::Failed && (connected || signaling.complete()) {
+ signaling.detached = true;
+ return false;
+ }
+ viewer.fail("screen signaling disconnected");
+ true
+ }
+}
+
+struct Subscription {
+ screen: Arc<Screen>,
+ session: String,
+}
+
+impl Drop for Subscription {
+ fn drop(&mut self) {
+ if !self.screen.signaling_disconnected(&self.session) {
+ return;
+ }
+ let screen = self.screen.clone();
+ let session = self.session.clone();
+ tokio::spawn(async move {
+ if let Err(error) = screen.close(&session).await {
+ tracing::warn!(error = format!("{error:#}"), "couldn't close screen signaling");
+ }
+ });
+ }
+}
+
+#[cfg(test)]
+mod tests;
diff --git a/host/src/screen/signaling/tests.rs b/host/src/screen/signaling/tests.rs
new file mode 100644
index 00000000..11e5c318
--- /dev/null
+++ b/host/src/screen/signaling/tests.rs
@@ -0,0 +1,302 @@
+use super::*;
+use crate::screen::helper::Link;
+use crate::screen::{IceConfig, protocol::HelperStatus};
+use std::sync::{Mutex, atomic::AtomicI64};
+use tokio::sync::broadcast;
+
+fn fixture(trickle: bool) -> (Arc<Screen>, mpsc::UnboundedReceiver<String>) {
+ let (events, _) = broadcast::channel(8);
+ let screen = Arc::new(Screen::new(Some(true), events));
+ let (tx, rx) = mpsc::unbounded_channel();
+ *screen.link.locked() = Some(Arc::new(Link { id: 1, tx, pending: Mutex::default(), next_id: AtomicI64::new(1) }));
+ *screen.status.locked() = HelperStatus { capture: true, trickle, ..HelperStatus::default() };
+ (screen, rx)
+}
+
+fn prepared(screen: &Screen, requested: bool) -> String {
+ let result = screen.prepare("phone", IceConfig::default(), requested).unwrap();
+ result["session"].as_str().unwrap().to_owned()
+}
+
+fn candidate(n: usize) -> Value {
+ json!({"type": "candidate", "candidate": format!("candidate:{n} 1 UDP 1 192.0.2.1 5000 typ relay"), "sdpMLineIndex": 0})
+}
+
+async fn answer(screen: &Arc<Screen>, requests: &mut mpsc::UnboundedReceiver<String>, session: &str) {
+ let task = tokio::spawn({
+ let screen = screen.clone();
+ let session = session.to_owned();
+ async move { screen.offer("phone", "candidate-free offer", &session, None).await }
+ });
+ let request = requests.recv().await;
+ let request = request.unwrap();
+ let request: Value = serde_json::from_str(&request).unwrap();
+ assert_eq!(request["params"]["trickle"], true);
+ screen.helper_candidate(session, candidate(1));
+ let link = screen.link().unwrap();
+ link.pending
+ .locked()
+ .remove(&request["id"].as_i64().unwrap())
+ .unwrap()
+ .send(Ok(json!({"sdp": "candidate-free answer"})))
+ .unwrap();
+ let result = task.await;
+ let result = result.unwrap().unwrap();
+ assert_eq!(result["sdp"], "candidate-free answer");
+}
+
+#[tokio::test]
+async fn preparation_negotiates_support_once_and_falls_back_for_old_peers() {
+ let (screen, _requests) = fixture(false);
+ let session = prepared(&screen, true);
+ assert!(screen.sessions.locked()[&session].signaling.is_none());
+ assert!(screen.candidates(&session, "phone").is_err());
+ screen.status.locked().trickle = true;
+ let session = prepared(&screen, false);
+ assert!(screen.sessions.locked()[&session].signaling.is_none());
+ let result = screen.prepare("phone", IceConfig::default(), true).unwrap();
+ assert_eq!(result["trickle"], true);
+}
+
+#[tokio::test]
+async fn private_stream_is_ready_before_offer_and_keeps_late_candidates_in_order() {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ assert!(screen.candidates(&session, "other").is_err());
+ let result = screen.offer("phone", "offer", &session, None).await;
+ assert!(result.is_err());
+ let mut stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let ready = stream.next().await;
+ assert_eq!(ready.unwrap()["type"], "ready");
+ assert!(screen.candidates(&session, "phone").is_err());
+ answer(&screen, &mut requests, &session).await;
+ let first = stream.next().await;
+ assert_eq!(first.unwrap(), candidate(1));
+ screen.helper_candidate(&session, candidate(2));
+ screen.helper_candidate(&session, json!({"type": "complete"}));
+ let second = stream.next().await;
+ assert_eq!(second.unwrap(), candidate(2));
+ let completed = stream.next().await;
+ assert_eq!(completed.unwrap()["type"], "complete");
+ let result = screen.offer("phone", "again", &session, None).await;
+ assert!(result.is_err());
+ let result = screen.candidate(&session, "other", Candidate::Complete).await;
+ assert!(result.is_err());
+ screen.sessions.locked().get_mut(&session).unwrap().ice.expires_at = 0;
+ let result = screen.candidate(&session, "phone", Candidate::Complete).await;
+ assert!(result.is_err());
+}
+
+#[tokio::test]
+async fn concurrent_offers_and_pre_answer_uploads_are_rejected() {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let _stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let offering = tokio::spawn({
+ let screen = screen.clone();
+ let session = session.clone();
+ async move { screen.offer("phone", "offer", &session, None).await }
+ });
+ let request = requests.recv().await;
+ let request = request.unwrap();
+ let request: Value = serde_json::from_str(&request).unwrap();
+ let result = screen.offer("phone", "offer", &session, None).await;
+ assert!(result.is_err());
+ let result = screen.candidate(&session, "phone", Candidate::Complete).await;
+ assert!(result.is_err());
+ let link = screen.link().unwrap();
+ link.pending.locked().remove(&request["id"].as_i64().unwrap()).unwrap().send(Ok(json!({"sdp": "answer"}))).unwrap();
+ let result = offering.await;
+ result.unwrap().unwrap();
+}
+
+#[tokio::test]
+async fn overflow_and_post_completion_candidates_fail_without_delivering_stale_data() {
+ for post_completion in [false, true] {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let mut stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let ready = stream.next().await;
+ ready.unwrap();
+ answer(&screen, &mut requests, &session).await;
+ if post_completion {
+ screen.helper_candidate(&session, json!({"type": "complete"}));
+ } else {
+ for n in 2..=128 {
+ screen.helper_candidate(&session, candidate(n));
+ }
+ }
+ screen.helper_candidate(&session, candidate(129));
+ let error = stream.next().await;
+ assert_eq!(error.unwrap()["type"], "error");
+ let ended = stream.next().await;
+ assert!(ended.is_none());
+ }
+}
+
+#[tokio::test]
+async fn signalling_loss_prevents_resubscription_and_a_fresh_session_can_connect() {
+ let (screen, _requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let stream = screen.candidates(&session, "phone").unwrap().boxed();
+ drop(stream);
+ assert!(screen.candidates(&session, "phone").is_err());
+ let result = screen.offer("phone", "offer", &session, None).await;
+ assert!(result.is_err());
+ let fresh = prepared(&screen, true);
+ assert_ne!(fresh, session);
+ assert!(screen.candidates(&fresh, "phone").is_ok());
+}
+
+#[tokio::test]
+async fn connected_media_survives_candidate_subscription_loss() {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let mut stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let ready = stream.next().await;
+ ready.unwrap();
+ answer(&screen, &mut requests, &session).await;
+ screen.session_connected(&session);
+ screen.takeover(true);
+ drop(stream);
+ // Candidates already in flight after the channel disconnects must not fail healthy media.
+ screen.helper_candidate(&session, candidate(2));
+ let stale_upload = screen.candidate(&session, "phone", Candidate::Complete).await;
+ assert!(stale_upload.is_err(), "detached candidates never reuse the old signaling exchange");
+ assert!(screen.owns_session(&session, "phone"));
+ assert_eq!(screen.state()["viewers"], 1);
+ assert_eq!(screen.state()["userControl"], true);
+ assert!(!screen.sessions.locked()[&session].expired(crate::store::now_ms()));
+ tokio::task::yield_now().await;
+ assert!(requests.try_recv().is_err(), "signaling loss never sends a media close");
+}
+
+#[tokio::test]
+async fn completed_exchange_survives_subscription_loss_only_after_delivery_and_acknowledgment() {
+ for (delivered, acknowledged) in [(false, true), (true, false), (true, true)] {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let mut stream = Some(screen.candidates(&session, "phone").unwrap().boxed());
+ let ready = stream.as_mut().unwrap().next().await;
+ ready.unwrap();
+ answer(&screen, &mut requests, &session).await;
+ screen.helper_candidate(&session, json!({"type": "complete"}));
+ if delivered {
+ let candidate = stream.as_mut().unwrap().next().await;
+ assert_eq!(candidate.unwrap()["type"], "candidate");
+ let complete = stream.as_mut().unwrap().next().await;
+ assert_eq!(complete.unwrap()["type"], "complete");
+ }
+ let uploading = tokio::spawn({
+ let screen = screen.clone();
+ let session = session.clone();
+ async move { screen.candidate(&session, "phone", Candidate::Complete).await }
+ });
+ let request = requests.recv().await;
+ let request: Value = serde_json::from_str(&request.unwrap()).unwrap();
+ let link = screen.link().unwrap();
+ let ack = link.pending.locked().remove(&request["id"].as_i64().unwrap()).unwrap();
+ let result = if acknowledged {
+ ack.send(Ok(json!({}))).unwrap();
+ let result = uploading.await;
+ Some(result.unwrap())
+ } else {
+ drop(stream.take());
+ ack.send(Ok(json!({}))).unwrap();
+ let result = uploading.await;
+ Some(result.unwrap())
+ };
+ if acknowledged {
+ result.unwrap().unwrap();
+ drop(stream.take());
+ } else {
+ assert!(result.unwrap().is_err(), "a late completion cannot revive a failed exchange");
+ }
+ let sessions = screen.sessions.locked();
+ let viewer = &sessions[&session];
+ assert_eq!(viewer.expired(crate::store::now_ms()), !(delivered && acknowledged));
+ assert!(viewer.ice.expires_at > crate::store::now_ms(), "failure is separate from credential expiry");
+ }
+}
+
+#[tokio::test]
+async fn late_connected_notifications_cannot_revive_failed_or_closed_sessions() {
+ let (screen, _requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let stream = screen.candidates(&session, "phone").unwrap().boxed();
+ drop(stream);
+ screen.session_connected(&session);
+ assert!(screen.sessions.locked()[&session].expired(crate::store::now_ms()));
+ screen.session_closed(&session);
+ screen.session_connected(&session);
+ assert!(!screen.owns_session(&session, "phone"));
+}
+
+#[test]
+fn candidate_bounds_and_completion_apply_to_both_directions() {
+ let mut progress = Progress::default();
+ for n in 0..128 {
+ progress.accept(&serde_json::from_value(candidate(n)).unwrap()).unwrap();
+ }
+ assert!(progress.accept(&serde_json::from_value(candidate(128)).unwrap()).is_err());
+ progress.accept(&Candidate::Complete).unwrap();
+ assert!(progress.accept(&Candidate::Complete).is_err());
+ let invalid = Candidate::Candidate { candidate: "candidate:bad\nline".into(), sdp_m_line_index: 0, sdp_mid: None };
+ assert!(invalid.validate().is_err());
+}
+
+#[tokio::test]
+async fn candidate_upload_waits_for_acknowledgment_and_never_retries_an_error() {
+ let (screen, mut requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let mut stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let ready = stream.next().await;
+ ready.unwrap();
+ answer(&screen, &mut requests, &session).await;
+ let uploading = tokio::spawn({
+ let screen = screen.clone();
+ let session = session.clone();
+ async move { screen.candidate(&session, "phone", Candidate::Complete).await }
+ });
+ let request = requests.recv().await;
+ let request: Value = serde_json::from_str(&request.unwrap()).unwrap();
+ assert!(!uploading.is_finished());
+ assert_eq!(request["method"], "candidate");
+ assert_eq!(request["params"]["candidate"]["type"], "complete");
+ let result = screen.candidate(&session, "phone", Candidate::Complete).await;
+ assert!(result.is_err(), "concurrent additions are rejected");
+ let link = screen.link().unwrap();
+ link.pending
+ .locked()
+ .remove(&request["id"].as_i64().unwrap())
+ .unwrap()
+ .send(Err("candidate rejected".into()))
+ .unwrap();
+ let result = uploading.await;
+ assert!(result.unwrap().is_err());
+ let first = stream.next().await;
+ assert_eq!(first.unwrap()["type"], "error", "the failure overrides buffered helper candidates");
+ let result = screen.candidate(&session, "phone", Candidate::Complete).await;
+ assert!(result.is_err());
+ assert!(requests.try_recv().is_err(), "the uncertain addition was never replayed");
+}
+
+#[tokio::test]
+async fn disabling_screen_ends_private_signaling_and_rejects_stale_sessions() {
+ let (screen, _requests) = fixture(true);
+ let session = prepared(&screen, true);
+ let mut stream = screen.candidates(&session, "phone").unwrap().boxed();
+ let ready = stream.next().await;
+ ready.unwrap();
+ let directory = std::env::temp_dir().join(format!("codync-screen-{}", uuid::Uuid::new_v4()));
+ std::fs::create_dir_all(&directory).unwrap();
+ let store = crate::store::Store::open(&directory.join("test.db")).unwrap();
+ // No helper is needed once the screen is off; the disconnect path has already stopped media.
+ *screen.link.locked() = None;
+ let result = screen.set_enabled(&store, false).await;
+ result.unwrap();
+ assert!(!screen.enabled());
+ let ended = stream.next().await;
+ assert!(ended.is_none());
+ assert!(screen.candidates(&session, "phone").is_err());
+}
diff --git a/host/src/screen/viewer.rs b/host/src/screen/viewer.rs
index 946e2714..da0254e5 100644
--- a/host/src/screen/viewer.rs
+++ b/host/src/screen/viewer.rs
@@ -1,6 +1,6 @@
//! Phones viewing the screen: prepared WebRTC sessions, offers, takeover and the on/off switch.
-use super::{Control, KV_ENABLED, Screen, Viewer};
+use super::{Control, KV_ENABLED, Screen, Viewer, ViewerState};
use crate::LockExt;
use crate::hub::Hub;
use anyhow::{Context, Result, anyhow, bail};
@@ -34,18 +34,31 @@ impl Default for IceConfig {
impl Screen {
/// Reserve one session before the phone gathers ICE; the host keeps the exact configuration.
- pub fn prepare(&self, owner: &str, ice: IceConfig) -> Result<Value> {
- self.link()?;
- let mut sessions = self.sessions.locked();
- let now = crate::store::now_ms();
- sessions.retain(|_, v| v.started || v.pending_until > now);
- if sessions.len() >= 32 || sessions.values().filter(|v| v.owner == owner).count() >= 4 {
- bail!("too many screen sessions; close another screen and try again");
- }
- let session = uuid::Uuid::new_v4().to_string();
- let result = json!({"session": session, "iceServers": ice.ice_servers, "expiresAt": ice.expires_at});
- sessions.insert(session, Viewer { owner: owner.to_owned(), ice, started: false, pending_until: now + 60_000 });
- Ok(result)
+ pub fn prepare(&self, owner: &str, ice: IceConfig, trickle: bool) -> Result<Value> {
+ self.with_link(|link| {
+ let trickle = trickle && self.status.locked().trickle;
+ let mut sessions = self.sessions.locked();
+ let now = crate::store::now_ms();
+ // Started/failed viewers stay registered until their watcher closes the helper.
+ sessions.retain(|_, v| v.state != ViewerState::Prepared || v.pending_until > now);
+ if sessions.len() >= 32 || sessions.values().filter(|v| v.owner == owner).count() >= 4 {
+ bail!("too many screen sessions; close another screen and try again");
+ }
+ let session = uuid::Uuid::new_v4().to_string();
+ let result = json!({"session": session, "iceServers": ice.ice_servers, "expiresAt": ice.expires_at, "trickle": trickle});
+ sessions.insert(
+ session,
+ Viewer {
+ owner: owner.to_owned(),
+ ice,
+ state: ViewerState::Prepared,
+ pending_until: now + 60_000,
+ link_id: link.id,
+ signaling: trickle.then(super::signaling::Signaling::new),
+ },
+ );
+ Ok(result)
+ })
}
pub fn owns_session(&self, session: &str, owner: &str) -> bool {
@@ -58,21 +71,28 @@ impl Screen {
if !self.status.locked().capture {
bail!("Codync isn't allowed to record this computer's screen yet. Allow it in System Settings there.");
}
- let ice = {
- let sessions = self.sessions.locked();
- let viewer = sessions.get(session).filter(|v| v.owner == owner).context("unknown screen session")?;
+ let (ice, trickle) = {
+ let mut sessions = self.sessions.locked();
+ let viewer = sessions.get_mut(session).filter(|v| v.owner == owner).context("unknown screen session")?;
let now = crate::store::now_ms();
- if viewer.ice.expires_at <= now || (!viewer.started && viewer.pending_until <= now) {
+ if viewer.expired(now) {
bail!("screen connection expired; reconnect to continue");
}
- viewer.ice.clone()
+ if viewer.link_id != link.id {
+ bail!("screen helper changed; reconnect");
+ }
+ let trickle = viewer.signaling.is_some();
+ if let Some(signaling) = &mut viewer.signaling {
+ signaling.begin_offer()?;
+ }
+ (viewer.ice.clone(), trickle)
};
let remote = !ice.ice_servers.is_empty();
let res = link
.request(
"answer",
json!({
- "session": session, "sdp": sdp, "display": display, "iceServers": ice.ice_servers,
+ "session": session, "sdp": sdp, "trickle": trickle, "display": display, "iceServers": ice.ice_servers,
"maxBitrateBps": if remote { 4_000_000 } else { 16_000_000 },
"maxFramerate": if remote { 30 } else { 60 },
}),
@@ -83,7 +103,7 @@ impl Screen {
Err(error) => {
// A helper can finish after the RPC times out. Keep retrying close until it acknowledges.
if let Some(viewer) = self.sessions.locked().get_mut(session) {
- viewer.ice.expires_at = 0;
+ viewer.state = ViewerState::Failed;
}
if let Err(close_error) = self.close(session).await {
tracing::warn!(error = format!("{close_error:#}"), "couldn't close failed screen session");
@@ -95,7 +115,12 @@ impl Screen {
let accepted = {
let mut sessions = self.sessions.locked();
if let Some(viewer) = sessions.get_mut(session) {
- viewer.started = true;
+ if let Some(signaling) = &mut viewer.signaling {
+ signaling.answered()?;
+ }
+ if viewer.state == ViewerState::Prepared {
+ viewer.state = ViewerState::Started;
+ }
true
} else {
false
@@ -110,6 +135,7 @@ impl Screen {
}
pub async fn close(&self, session: &str) -> Result<()> {
+ self.fail_signaling(session, "screen session closed");
let link = self.link.locked().clone();
if let Some(link) = link {
link.request("close", json!({"session": session})).await?;
@@ -119,20 +145,30 @@ impl Screen {
}
pub(super) fn session_closed(&self, session: &str) {
- let removed = {
- let mut sessions = self.sessions.locked();
- let removed = sessions.remove(session).is_some();
- if sessions.is_empty() {
- // Nobody is watching: hand control back to the bots.
- self.control.locked().user = false;
- }
- removed
- };
- if removed {
+ if self.remove_session(session) {
self.emit();
}
}
+ #[cfg(any(unix, test))]
+ pub(super) fn session_connected(&self, session: &str) {
+ if let Some(viewer) = self.sessions.locked().get_mut(session)
+ && viewer.state != ViewerState::Failed
+ {
+ viewer.state = ViewerState::Connected;
+ }
+ }
+
+ pub(super) fn remove_session(&self, session: &str) -> bool {
+ let mut sessions = self.sessions.locked();
+ let removed = sessions.remove(session).is_some();
+ if sessions.is_empty() {
+ // Nobody is watching: hand control back to the bots.
+ self.control.locked().user = false;
+ }
+ removed
+ }
+
pub fn takeover(&self, on: bool) {
self.control.locked().user = on;
self.emit();
@@ -168,12 +204,9 @@ pub async fn watch_viewer(hub: Arc<Hub>, session: String, owner: String) {
let sessions = hub.screen.sessions.locked();
let Some(viewer) = sessions.get(&session) else { return };
let now = crate::store::now_ms();
- viewer.ice.expires_at <= now || (!viewer.started && viewer.pending_until <= now)
+ viewer.expired(now)
};
- let authorized = owner == "local"
- || crate::api::devices::authorize(&hub, &owner)
- .is_ok_and(|d| d.scopes.contains(&crate::store::Scope::Screen));
- if expired || !authorized || !hub.screen.enabled() {
+ if expired || !crate::api::devices::may_view_screen(&hub, &owner) || !hub.screen.enabled() {
if let Err(error) = hub.screen.close(&session).await {
tracing::warn!(error = format!("{error:#}"), "couldn't close expired screen session");
continue;
@@ -206,7 +239,7 @@ mod tests {
}],
..IceConfig::default()
};
- let prepared = screen.prepare("phone", config).unwrap();
+ let prepared = screen.prepare("phone", config, false).unwrap();
let id = prepared["session"].as_str().unwrap();
assert!(screen.owns_session(id, "phone"));
assert!(!screen.owns_session(id, "other"));
@@ -214,14 +247,27 @@ mod tests {
assert_eq!(screen.state()["viewers"], 0);
assert!(!screen.state().to_string().contains("secret"));
for _ in 0..3 {
- screen.prepare("phone", IceConfig::default()).unwrap();
+ screen.prepare("phone", IceConfig::default(), false).unwrap();
}
- assert!(screen.prepare("phone", IceConfig::default()).is_err());
+ assert!(screen.prepare("phone", IceConfig::default(), false).is_err());
for viewer in screen.sessions.locked().values_mut() {
viewer.pending_until = 0;
}
- assert!(screen.prepare("phone", IceConfig::default()).is_ok());
+ assert!(screen.prepare("phone", IceConfig::default(), false).is_ok());
+ let live = screen.prepare("watching", IceConfig::default(), false).unwrap();
+ let live_id = live["session"].as_str().unwrap();
+ {
+ let mut sessions = screen.sessions.locked();
+ let viewer = sessions.get_mut(live_id).unwrap();
+ viewer.state = ViewerState::Started;
+ viewer.ice.expires_at = 0;
+ }
+ screen.prepare("another", IceConfig::default(), false).unwrap();
+ assert!(
+ screen.owns_session(live_id, "watching"),
+ "credential expiry must still close the helper through the viewer watcher"
+ );
*screen.enabled.locked() = Some(false);
- assert!(screen.prepare("phone", IceConfig::default()).is_err());
+ assert!(screen.prepare("phone", IceConfig::default(), false).is_err());
}
}
diff --git a/web/content/docs/computer-use.mdx b/web/content/docs/computer-use.mdx
index 6b8e0633..99fba20c 100644
--- a/web/content/docs/computer-use.mdx
+++ b/web/content/docs/computer-use.mdx
@@ -50,6 +50,7 @@ If the computer allows viewing but not control, the screen says "View only" and
- Video and input go over WebRTC and stay encrypted between your phone and Codync Screen. The setup messages travel over the same end-to-end encrypted connection as your chats.
- On a direct connection (same Wi-Fi, or Tailscale), Codync uses no STUN or TURN server. On a Mac, the picture is encoded at up to 16 Mbps and 60 fps.
- Through the Codync relay, the stream may be forwarded by Cloudflare's TURN service, which passes along encrypted packets and never sees your screen's content. This needs the computer linked to a signed-in account; otherwise the phone asks you to sign in on the computer. Relayed sessions are encoded at up to 4 Mbps and 30 fps, even if a direct path turns up later.
+- With updated iPhone and computer builds, an established screen connection can continue through a temporary chat reconnection. Setup failures reconnect with a fresh screen session.
- Revoking the phone's access or turning off remote screen stops the picture and input.
## Let a bot use the computer