Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.
The phone's sound on the PC while mirroring
Bee8d6d80554c17
diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xmlindex 623c6a6..591b3c7 100644--- a/android/app/src/main/AndroidManifest.xml+++ b/android/app/src/main/AndroidManifest.xml@@ -16,6 +16,9 @@<uses-permission android:name="android.permission.FOREGROUND_SERVICE_CONNECTED_DEVICE" /><!-- Sharing the screen with a PC (each time, after Android's prompt). --><uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />+ <!-- The phone's sound while mirrored (playback capture, with the screen+ share's consent). Never the microphone. -->+ <uses-permission android:name="android.permission.RECORD_AUDIO" /><!-- Elevated: turning wireless debugging back on after a reboot or a newWi-Fi network. Not grantable to apps; Nectarlink grants it to itselfthrough the shell once the user has paired it with wireless debugging. -->diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.ktindex ddc5fe6..ec7ab6d 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt@@ -3,6 +3,7 @@ package app.nectarlink.android.coreimport android.content.Contextimport android.net.wifi.WifiManager+import android.os.Buildimport android.service.notification.NotificationListenerServiceimport android.util.Logimport android.net.Uri@@ -277,11 +278,6 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener}}- /**- * Sends picked or shared files to a PC; problems arrive as messages.- * The files are opened right away: Android's permission to read a- * shared item ends with the activity that received it.- *//** Opens this phone's screen stream to a PC (after the user agreed). */suspend fun mirrorOpen(pcId: String): MirrorStream {startJob?.join()@@ -289,6 +285,18 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListenerreturn node.mirrorOpen(pcId)}+ /** Opens this phone's sound stream to a PC that asked for it with the screen. */+ suspend fun mirrorOpenAudio(pcId: String): MirrorStream {+ startJob?.join()+ val node = checkNotNull(node) { "the core isn't running" }+ return node.mirrorOpenAudio(pcId)+ }++ /**+ * Sends picked or shared files to a PC; problems arrive as messages.+ * The files are opened right away: Android's permission to read a+ * shared item ends with the activity that received it.+ */fun sendFiles(pcId: String, uris: List<Uri>) = send(pcId, OutgoingFiles.open(context, uris))/** Sends a folder the user picked, with everything in it. */@@ -387,7 +395,9 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener(if (PhoneCalls.canControl(context) && CallCompanion.allowed(context)) listOf("call.incall") else emptyList()) +(if (PhoneSms.canRead(context)) listOf("sms.read") else emptyList()) +(if (PhoneSms.canSend(context)) listOf("sms.send") else emptyList()) +- (if (InputService.running || Elevated.running) listOf("mirror.input") else emptyList())+ (if (InputService.running || Elevated.running) listOf("mirror.input") else emptyList()) ++ // Asked for when a PC wants the sound.+ (if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q) listOf("mirror.audio.playback") else emptyList())/*** Elevated while the wireless debugging helper runs, Assist when thediff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.ktindex 1c5658d..fe1b7f2 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt@@ -65,8 +65,8 @@ internal class PhonePlatform(override fun callCommand(id: String, command: CallCommand): Boolean = onCall(id, command)- override fun mirrorRequested(pcId: String, maxSize: UInt, fps: UInt, bitrate: UInt): Boolean =- onMirror(pcId, MirrorRequest(pcId, maxSize.toInt(), fps.toInt(), bitrate.toInt()))+ override fun mirrorRequested(pcId: String, maxSize: UInt, fps: UInt, bitrate: UInt, audio: Boolean): Boolean =+ onMirror(pcId, MirrorRequest(pcId, maxSize.toInt(), fps.toInt(), bitrate.toInt(), audio))override fun mirrorStopRequested(pcId: String) {MirrorRequests.dismiss(context, pcId)diff --git a/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorRequest.kt b/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorRequest.ktindex af5356b..dbb31af 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorRequest.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorRequest.kt@@ -10,6 +10,7 @@ import android.content.Contextimport android.content.Intentimport android.content.pm.PackageManagerimport android.media.projection.MediaProjectionManager+import android.os.Buildimport android.os.Bundleimport androidx.activity.ComponentActivityimport androidx.activity.result.contract.ActivityResultContracts@@ -18,17 +19,24 @@ import androidx.core.app.NotificationManagerCompatimport androidx.core.content.ContextCompatimport app.nectarlink.android.R-/** A PC asking for the screen, with what it wants. */-data class MirrorRequest(val pcId: String, val maxSize: Int, val fps: Int, val bitrate: Int) {+/** A PC asking for the screen (and its sound), with what it wants. */+data class MirrorRequest(val pcId: String, val maxSize: Int, val fps: Int, val bitrate: Int, val audio: Boolean) {fun toIntent(intent: Intent): Intent = intent.putExtra("pc", pcId).putExtra("maxSize", maxSize).putExtra("fps", fps).putExtra("bitrate", bitrate)+ .putExtra("audio", audio)companion object {fun of(intent: Intent): MirrorRequest? = intent.getStringExtra("pc")?.let {- MirrorRequest(it, intent.getIntExtra("maxSize", 1920), intent.getIntExtra("fps", 60), intent.getIntExtra("bitrate", 8_000_000))+ MirrorRequest(+ it,+ intent.getIntExtra("maxSize", 1920),+ intent.getIntExtra("fps", 60),+ intent.getIntExtra("bitrate", 8_000_000),+ intent.getBooleanExtra("audio", false),+ )}}}@@ -81,10 +89,16 @@ object MirrorRequests {private const val REQUEST_TIMEOUT_MS = 2 * 60 * 1000L}-/** Shows Android's screen capture prompt for a PC's request, then starts sharing. */+/**+ * Shows Android's screen capture prompt for a PC's request, then starts+ * sharing. A PC that wants the sound too first gets the permission asked+ * for (once; without it, the screen is shared alone).+ */class MirrorConsentActivity : ComponentActivity() {private var request: MirrorRequest? = null+ private val sound = registerForActivityResult(ActivityResultContracts.RequestPermission()) { askForScreen() }+private val prompt = registerForActivityResult(ActivityResultContracts.StartActivityForResult()) { result ->val data = result.dataval asked = request@@ -99,8 +113,13 @@ class MirrorConsentActivity : ComponentActivity() {request = MirrorRequest.of(intent)val asked = request ?: return finish()MirrorRequests.dismiss(this, asked.pcId)- if (savedInstanceState == null) {- prompt.launch(getSystemService(MediaProjectionManager::class.java).createScreenCaptureIntent())- }+ if (savedInstanceState != null) return+ val wantsSound = asked.audio && Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q &&+ ContextCompat.checkSelfPermission(this, Manifest.permission.RECORD_AUDIO) != PackageManager.PERMISSION_GRANTED+ if (wantsSound) sound.launch(Manifest.permission.RECORD_AUDIO) else askForScreen()+ }++ private fun askForScreen() {+ prompt.launch(getSystemService(MediaProjectionManager::class.java).createScreenCaptureIntent())}}diff --git a/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorService.kt b/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorService.ktindex d9188ea..7c772c9 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorService.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/mirror/MirrorService.kt@@ -38,6 +38,7 @@ class MirrorService : Service() {private val main = Handler(Looper.getMainLooper())private var projection: MediaProjection? = nullprivate var encoder: ScreenEncoder? = null+ private var sound: SoundCapture? = nullprivate var pcId: String? = nulloverride fun onBind(intent: Intent?): IBinder? = null@@ -78,6 +79,7 @@ class MirrorService : Service() {val maxSize = intent.getIntExtra(EXTRA_MAX_SIZE, 1920)val fps = intent.getIntExtra(EXTRA_FPS, 60)val bitrate = intent.getIntExtra(EXTRA_BITRATE, 8_000_000)+ val audio = intent.getBooleanExtra(EXTRA_AUDIO, false)scope.launch {val stream = runCatching { core.mirrorOpen(pc) }.getOrElse {Log.w(TAG, "can't stream to the PC", it)@@ -94,10 +96,28 @@ class MirrorService : Service() {main.post { stopSharing() }}.also { it.start() }}+ if (audio && SoundCapture.canCapture(this@MirrorService)) startSound(core, pc)}return START_NOT_STICKY}+ /** The sound, on a stream of its own; the screen goes on without it if it can't. */+ private suspend fun startSound(core: app.nectarlink.android.core.Core, pc: String) {+ if (Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) return+ val stream = runCatching { core.mirrorOpenAudio(pc) }.getOrElse {+ Log.w(TAG, "can't send the sound", it)+ return+ }+ main.post {+ val running = projection+ if (running == null || pcId != pc || Build.VERSION.SDK_INT < Build.VERSION_CODES.Q) {+ stream.end()+ return@post+ }+ sound = SoundCapture(running, stream).also { it.start() }+ }+ }+private fun notification(pcName: String): Notification {val stop = PendingIntent.getService(this, 0, Intent(this, MirrorService::class.java).setAction(ACTION_STOP),@@ -116,6 +136,8 @@ class MirrorService : Service() {private fun stopSharing() {encoder?.stop()encoder = null+ sound?.stop()+ sound = nullprojection?.let { runCatching { it.stop() } }projection = nullpcId = null@@ -141,6 +163,7 @@ class MirrorService : Service() {private const val EXTRA_MAX_SIZE = "maxSize"private const val EXTRA_FPS = "fps"private const val EXTRA_BITRATE = "bitrate"+ private const val EXTRA_AUDIO = "audio"/** The service sharing the screen now, if any. */@Volatile private var current: MirrorService? = null@@ -160,6 +183,7 @@ class MirrorService : Service() {.putExtra(EXTRA_MAX_SIZE, request.maxSize).putExtra(EXTRA_FPS, request.fps).putExtra(EXTRA_BITRATE, request.bitrate)+ .putExtra(EXTRA_AUDIO, request.audio)ContextCompat.startForegroundService(context, intent)}diff --git a/android/app/src/main/kotlin/app/nectarlink/android/mirror/ScreenEncoder.kt b/android/app/src/main/kotlin/app/nectarlink/android/mirror/ScreenEncoder.ktindex 825d873..51425c8 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/mirror/ScreenEncoder.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/mirror/ScreenEncoder.kt@@ -143,7 +143,8 @@ internal class ScreenEncoder(if (key) parameters + bytes else bytes,)when (result) {- MirrorSendResult.NEED_KEYFRAME -> keyframeWanted = true+ // A video stream never drops without asking for a keyframe.+ MirrorSendResult.NEED_KEYFRAME, MirrorSendResult.DROPPED -> keyframeWanted = trueMirrorSendResult.CLOSED -> return trueMirrorSendResult.QUEUED -> {}}diff --git a/android/app/src/main/kotlin/app/nectarlink/android/mirror/SoundCapture.kt b/android/app/src/main/kotlin/app/nectarlink/android/mirror/SoundCapture.ktnew file mode 100644index 0000000..e1ebef2--- /dev/null+++ b/android/app/src/main/kotlin/app/nectarlink/android/mirror/SoundCapture.kt@@ -0,0 +1,118 @@+// SPDX-License-Identifier: GPL-3.0-or-later+package app.nectarlink.android.mirror++import android.Manifest+import android.annotation.SuppressLint+import android.content.Context+import android.content.pm.PackageManager+import android.media.AudioAttributes+import android.media.AudioFormat+import android.media.AudioPlaybackCaptureConfiguration+import android.media.AudioRecord+import android.media.projection.MediaProjection+import android.os.Build+import android.os.SystemClock+import android.util.Log+import androidx.annotation.RequiresApi+import androidx.core.content.ContextCompat+import app.nectarlink.core.MirrorSendResult+import app.nectarlink.core.MirrorStream+import app.nectarlink.core.VideoPacketKind+import app.nectarlink.core.mirrorAudioConfig++/**+ * The phone's sound while its screen is shared (Android 10+): what apps+ * play (media, games, and apps that don't say), captured with the screen+ * share's consent, as 48 kHz stereo PCM in 10 ms packets. Apps that opt out+ * of playback capture, and calls, aren't heard.+ */+class SoundCapture(+ private val projection: MediaProjection,+ private val stream: MirrorStream,+) {+ @Volatile private var running = false+ private var thread: Thread? = null++ @SuppressLint("MissingPermission") // Checked by canCapture, before this is made.+ @RequiresApi(Build.VERSION_CODES.Q)+ fun start() {+ val config = AudioPlaybackCaptureConfiguration.Builder(projection)+ .addMatchingUsage(AudioAttributes.USAGE_MEDIA)+ .addMatchingUsage(AudioAttributes.USAGE_GAME)+ .addMatchingUsage(AudioAttributes.USAGE_UNKNOWN)+ .build()+ val format = AudioFormat.Builder()+ .setEncoding(AudioFormat.ENCODING_PCM_16BIT)+ .setSampleRate(RATE)+ .setChannelMask(AudioFormat.CHANNEL_IN_STEREO)+ .build()+ val minimum = AudioRecord.getMinBufferSize(RATE, AudioFormat.CHANNEL_IN_STEREO, AudioFormat.ENCODING_PCM_16BIT)+ val record = runCatching {+ AudioRecord.Builder()+ .setAudioFormat(format)+ .setBufferSizeInBytes(maxOf(minimum, PACKET_BYTES * 8))+ .setAudioPlaybackCaptureConfig(config)+ .build()+ }.getOrElse {+ Log.w(TAG, "no sound capture", it)+ stream.end()+ return+ }+ if (record.state != AudioRecord.STATE_INITIALIZED) {+ Log.w(TAG, "sound capture didn't start")+ record.release()+ stream.end()+ return+ }+ running = true+ thread = Thread({ run(record) }, "sound-capture").apply {+ priority = Thread.MAX_PRIORITY+ start()+ }+ }++ private fun run(record: AudioRecord) {+ try {+ record.startRecording()+ if (stream.send(VideoPacketKind.CONFIG, 0u, mirrorAudioConfig(RATE.toUInt(), CHANNELS.toUByte())) == MirrorSendResult.CLOSED) return+ val started = SystemClock.elapsedRealtimeNanos()+ val packet = ByteArray(PACKET_BYTES)+ while (running) {+ var filled = 0+ while (filled < packet.size && running) {+ val read = record.read(packet, filled, packet.size - filled)+ if (read < 0) return+ filled += read+ }+ if (!running) return+ val time = ((SystemClock.elapsedRealtimeNanos() - started) / 1000).toULong()+ if (stream.send(VideoPacketKind.FRAME, time, packet.copyOf()) == MirrorSendResult.CLOSED) return+ }+ } catch (e: Exception) {+ Log.w(TAG, "sound capture stopped", e)+ } finally {+ running = false+ runCatching { record.stop() }+ record.release()+ stream.end()+ }+ }++ fun stop() {+ running = false+ thread = null+ }++ companion object {+ private const val TAG = "SoundCapture"+ private const val RATE = 48_000+ private const val CHANNELS = 2+ /** 10 ms of 16-bit stereo. */+ private const val PACKET_BYTES = RATE / 100 * CHANNELS * 2++ /** Whether this phone can share its sound (Android 10+, with the permission). */+ fun canCapture(context: Context): Boolean =+ Build.VERSION.SDK_INT >= Build.VERSION_CODES.Q &&+ ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) == PackageManager.PERMISSION_GRANTED+ }+}diff --git a/core/nectarlink-cli/src/main.rs b/core/nectarlink-cli/src/main.rsindex 135ba03..30170a0 100644--- a/core/nectarlink-cli/src/main.rs+++ b/core/nectarlink-cli/src/main.rs@@ -175,6 +175,9 @@ enum Command {/// with a forwarded port).#[arg(long)]at: Vec<std::net::SocketAddr>,+ /// Also ask for the phone's sound, and save it here (WAV).+ #[arg(long)]+ sound: Option<PathBuf>,},/// Use a paired phone's screen: tap at x,y (fractions of the screen),/// swipe, scroll, press a key (back, home, recents, enter...) or type.@@ -194,6 +197,10 @@ enum Command {height: u32,#[arg(long, default_value_t = 60)]fps: u32,+ /// Sound to share too, in a loop, with PCs that ask for it (a WAV+ /// file of 16-bit PCM).+ #[arg(long)]+ sound: Option<PathBuf>,},/// Act as a phone with a few sample conversations (use with --as-phone),/// so a PC can read them and text through this client; stays online.@@ -359,6 +366,8 @@ static RECORDING: std::sync::OnceLock<std::sync::Arc<Recording>> = std::sync::Onstruct Recording {file: std::sync::Mutex<Option<std::fs::File>>,stats: std::sync::Mutex<RecordingStats>,+ /// The sound's format and samples, when asked for.+ sound: std::sync::Mutex<(Option<nectarlink_core::MirrorAudioConfig>, Vec<u8>)>,}#[derive(Debug, Default, Clone, Copy)]@@ -387,10 +396,91 @@ impl nectarlink_core::MirrorSink for Recording {fn ended(&self) {println!("The stream ended.");}+ fn audio_config(&self, config: nectarlink_core::MirrorAudioConfig) {+ println!("The phone streams its sound: {} Hz, {} channels.", config.rate, config.channels);+ self.sound.lock().unwrap().0 = Some(config);+ }+ fn audio(&self, _time_us: u64, data: Vec<u8>) {+ self.sound.lock().unwrap().1.extend_from_slice(&data);+ }+ fn audio_ended(&self) {+ println!("The sound ended.");+ }}-/// PCs that asked for the screen, for `screen`.-static SCREEN_ASKS: std::sync::OnceLock<tokio::sync::mpsc::UnboundedSender<DeviceId>> =+/// A WAV file of 16-bit PCM: its format and samples.+fn read_wav(bytes: &[u8]) -> Option<(u32, u8, &[u8])> {+ if bytes.len() < 12 || &bytes[..4] != b"RIFF" || &bytes[8..12] != b"WAVE" {+ return None;+ }+ let (mut at, mut format) = (12, None);+ while at + 8 <= bytes.len() {+ let id = &bytes[at..at + 4];+ let len = u32::from_le_bytes(bytes[at + 4..at + 8].try_into().ok()?) as usize;+ let body = bytes.get(at + 8..(at + 8 + len).min(bytes.len()))?;+ match id {+ b"fmt " if body.len() >= 16 => {+ let tag = u16::from_le_bytes([body[0], body[1]]);+ let channels = u16::from_le_bytes([body[2], body[3]]);+ let rate = u32::from_le_bytes(body[4..8].try_into().ok()?);+ let bits = u16::from_le_bytes([body[14], body[15]]);+ if tag != 1 || bits != 16 || !(1..=2).contains(&channels) {+ return None;+ }+ format = Some((rate, channels as u8));+ }+ b"data" => return format.map(|(rate, channels)| (rate, channels, body)),+ _ => {}+ }+ at += 8 + len + (len & 1);+ }+ None+}++fn write_wav(path: &std::path::Path, rate: u32, channels: u8, samples: &[u8]) -> std::io::Result<()> {+ let block = 2 * u32::from(channels);+ let mut out = Vec::with_capacity(44 + samples.len());+ out.extend_from_slice(b"RIFF");+ out.extend_from_slice(&(36 + samples.len() as u32).to_le_bytes());+ out.extend_from_slice(b"WAVEfmt ");+ out.extend_from_slice(&16u32.to_le_bytes());+ out.extend_from_slice(&1u16.to_le_bytes());+ out.extend_from_slice(&u16::from(channels).to_le_bytes());+ out.extend_from_slice(&rate.to_le_bytes());+ out.extend_from_slice(&(rate * block).to_le_bytes());+ out.extend_from_slice(&(block as u16).to_le_bytes());+ out.extend_from_slice(&16u16.to_le_bytes());+ out.extend_from_slice(b"data");+ out.extend_from_slice(&(samples.len() as u32).to_le_bytes());+ out.extend_from_slice(samples);+ std::fs::write(path, out)+}++/// Streams PCM in 10 ms packets, in real time, in a loop, until the PC+/// stops listening.+fn stream_sound(sound: &nectarlink_core::MirrorStream, rate: u32, channels: u8, samples: &[u8]) {+ use nectarlink_core::{MirrorAudioConfig, MirrorSend, PacketKind};+ let config = MirrorAudioConfig { codec: nectarlink_core::MIRROR_PCM.into(), rate, channels };+ if sound.send(PacketKind::Config, 0, config.to_cbor()) == MirrorSend::Closed {+ return;+ }+ let packet = (rate / 100) as usize * config.frame_bytes();+ let started = std::time::Instant::now();+ for (sent, chunk) in (0u64..).zip(samples.chunks_exact(packet).cycle()) {+ let due = std::time::Duration::from_millis(sent * 10);+ if let Some(wait) = due.checked_sub(started.elapsed()) {+ std::thread::sleep(wait);+ }+ if sound.send(PacketKind::Frame, sent * 10_000, chunk.to_vec()) == MirrorSend::Closed {+ break;+ }+ }+ println!("Stopped sharing the sound.");+}++/// PCs that asked for the screen (and whether they want the sound), for+/// `screen`.+static SCREEN_ASKS: std::sync::OnceLock<tokio::sync::mpsc::UnboundedSender<(DeviceId, bool)>> =std::sync::OnceLock::new();/// A sample text for `texts`: (thread, number, name, incoming, body, Unix ms).@@ -451,8 +541,11 @@ impl Platform for TerminalPlatform {options: &nectarlink_core::MirrorStart,) -> std::result::Result<(), String> {let asks = SCREEN_ASKS.get().ok_or("not sharing a screen (run `screen`)")?;+ if options.audio {+ println!("It wants the sound too.");+ }println!("A PC asked for the screen ({}px, {} fps); sharing it.", options.max_size, options.fps);- asks.send(*peer).map_err(|e| e.to_string())+ asks.send((*peer, options.audio)).map_err(|e| e.to_string())}fn mirror_stop_requested(&self, _peer: &DeviceId) {println!("The PC stopped watching.");@@ -863,20 +956,28 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {tokio::time::sleep(Duration::from_millis(500)).await;println!("Sent.");}- Command::Mirror { device, out, seconds, at } => {+ Command::Mirror { device, out, seconds, at, sound } => {let recording = std::sync::Arc::new(Recording::default());*recording.file.lock().unwrap() =Some(std::fs::File::create(out).context("can't create the file")?);let _ = RECORDING.set(recording.clone());let mut offers = cli.offers.clone();offers.push(nectarlink_core::MIRROR_VIEW.into());+ if sound.is_some() {+ offers.push(nectarlink_core::MIRROR_LISTEN.into());+ }node.update_power(node_power(cli), offers).await;let id = resolve(node, device)?;if !at.is_empty() {node.add_known_addrs(id, at);}wait_until_online(node, id).await?;- let options = nectarlink_core::MirrorStart { max_size: 1920, fps: 60, bitrate: 8_000_000 };+ let options = nectarlink_core::MirrorStart {+ max_size: 1920,+ fps: 60,+ bitrate: 8_000_000,+ audio: sound.is_some(),+ };node.mirror_start(id, options).await.context("the phone didn't ask its user")?;println!("Asked the phone; accept on the phone. Recording for {seconds} s after the first frame."@@ -898,8 +999,28 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {bytes as f64 * 8.0 / secs / 1e6,size.map(|(w, h)| format!(", {w}x{h}")).unwrap_or_default());+ if let Some(path) = sound {+ let (config, samples) = std::mem::take(&mut *recording.sound.lock().unwrap());+ match config {+ Some(config) => {+ let peak = samples+ .as_chunks::<2>()+ .0+ .iter()+ .map(|s| i16::from_le_bytes(*s).unsigned_abs())+ .max()+ .unwrap_or(0);+ let ms = samples.len() as u64 * 1000+ / (u64::from(config.rate) * config.frame_bytes() as u64);+ write_wav(path, config.rate, config.channels, &samples)+ .context("can't save the sound")?;+ println!("Sound: {ms} ms, peak {peak} of 32767.");+ }+ None => println!("No sound came."),+ }+ }}- Command::Screen { video, width, height, fps } => {+ Command::Screen { video, width, height, fps, sound } => {if !cli.as_phone {bail!("screens are shared by phones: add --as-phone");}@@ -908,16 +1029,27 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {if units.is_empty() {bail!("no access unit delimiters in the video");}+ let sound = sound.as_deref().map(std::fs::read).transpose().context("can't read the sound")?;+ let sound = match &sound {+ Some(bytes) => {+ let (rate, channels, samples) = read_wav(bytes).context("not a 16-bit PCM WAV file")?;+ Some((rate, channels, std::sync::Arc::new(samples.to_vec())))+ }+ None => None,+ };let (asks, mut asked) = tokio::sync::mpsc::unbounded_channel();let _ = SCREEN_ASKS.set(asks);let mut offers = cli.offers.clone();offers.push(nectarlink_core::MIRROR_CAPTURE.into());+ if sound.is_some() {+ offers.push(nectarlink_core::MIRROR_AUDIO_PLAYBACK.into());+ }node.update_power(node_power(cli), offers).await;println!("Sharing a {width}x{height} screen ({} frames) with PCs that ask.", units.len());let streamer = node.clone();let (width, height, fps) = (*width, *height, *fps);tokio::spawn(async move {- while let Some(pc) = asked.recv().await {+ while let Some((pc, wants_sound)) = asked.recv().await {let mirror = match streamer.mirror_open(pc).await {Ok(mirror) => std::sync::Arc::new(mirror),Err(e) => {@@ -927,6 +1059,14 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {};let units = units.clone();std::thread::spawn(move || stream_screen(&mirror, &units, width, height, fps));+ if let (true, Some((rate, channels, samples))) = (wants_sound, sound.clone()) {+ match streamer.mirror_open_audio(pc).await {+ Ok(stream) => {+ std::thread::spawn(move || stream_sound(&stream, rate, channels, &samples));+ }+ Err(e) => println!("Couldn't open the sound stream: {e}"),+ }+ }}});watch(node, false).await?;@@ -1666,7 +1806,7 @@ fn stream_screen(let time = started.elapsed().as_micros() as u64;match mirror.send(kind, time, unit.clone()) {MirrorSend::Queued => {}- MirrorSend::NeedKeyframe => dropped += 1,+ MirrorSend::NeedKeyframe | MirrorSend::Dropped => dropped += 1,MirrorSend::Closed => break,}}diff --git a/core/nectarlink-core/src/features.rs b/core/nectarlink-core/src/features.rsindex b0d097d..c6b4bf5 100644--- a/core/nectarlink-core/src/features.rs+++ b/core/nectarlink-core/src/features.rs@@ -679,9 +679,17 @@ pub const FEATURES: &[FeatureDef] = &[FeatureDef {id: "mirroring.audio",group: FeatureGroup::Mirroring,- requires: &[phone("mirror.audio", ELEVATED)],+ requires: &[+ phone("mirror.audio", ELEVATED),+ desktop("mirror.listen", UPDATE),+ DeviceToggle("mirroring"),+ ],partial: Some(PartialDef {- requires: &[phone("mirror.audio.playback", UPDATE)],+ requires: &[+ phone("mirror.audio.playback", UPDATE),+ desktop("mirror.listen", UPDATE),+ DeviceToggle("mirroring"),+ ],limit: "mirroring.limit.audio_some_apps",}),},@@ -781,6 +789,7 @@ mod tests {"input.inject","deck.actions","mirror.view",+ "mirror.listen",];fn pc() -> DeviceFacts {diff --git a/core/nectarlink-core/src/lib.rs b/core/nectarlink-core/src/lib.rsindex c8d8e12..857011d 100644--- a/core/nectarlink-core/src/lib.rs+++ b/core/nectarlink-core/src/lib.rs@@ -42,15 +42,16 @@ pub use nectarlink_protocol::messages::clip::{pub use nectarlink_protocol::{DeviceId, PacketKind,messages::{- Battery, CLIP_MAX_BYTES, CallControls, CallState, DeviceInfo, DeviceKind, MediaPlayer, MirrorConfig,- MirrorInput, MirrorStart, Notification, NotificationAction, PhotoNew as Photo, PowerLevel,- SmsMessage, SmsPart, SmsThread, TouchAction,+ Battery, CLIP_MAX_BYTES, CallControls, CallState, DeviceInfo, DeviceKind, MediaPlayer,+ MirrorAudioConfig, MirrorConfig, MirrorInput, MirrorStart, Notification, NotificationAction,+ PhotoNew as Photo, PowerLevel, SmsMessage, SmsPart, SmsThread, TouchAction,calls::{CONTROL as CALLS_CONTROL, IN_CALL as CALLS_IN_CALL, SHOW as CALLS_SHOW, STATE as CALLS_STATE,},mirror::{- CAPTURE as MIRROR_CAPTURE, INPUT as MIRROR_INPUT, MAX_TEXT_BYTES as MIRROR_MAX_TEXT_BYTES,- VIEW as MIRROR_VIEW,+ AUDIO as MIRROR_AUDIO, AUDIO_PLAYBACK as MIRROR_AUDIO_PLAYBACK, CAPTURE as MIRROR_CAPTURE,+ INPUT as MIRROR_INPUT, LISTEN as MIRROR_LISTEN, MAX_TEXT_BYTES as MIRROR_MAX_TEXT_BYTES,+ PCM as MIRROR_PCM, VIEW as MIRROR_VIEW,},mirror_keys,photos::{MAX_THUMB_BYTES as PHOTO_MAX_THUMB_BYTES, READ as PHOTOS_READ, SHOW as PHOTOS_SHOW},diff --git a/core/nectarlink-core/src/mirror.rs b/core/nectarlink-core/src/mirror.rsindex dd374c6..7f27776 100644--- a/core/nectarlink-core/src/mirror.rs+++ b/core/nectarlink-core/src/mirror.rs@@ -6,6 +6,9 @@//! The phone side keeps latency low over throughput: when the network//! can't keep up, frames are dropped (never queued for long) and the//! encoder is asked for a keyframe to start clean from.+//!+//! When the PC asks for it and the phone can, the phone's sound comes on a+//! second stream (PCM), dropped rather than delayed in the same way.use std::sync::{Arc, Mutex,@@ -15,7 +18,7 @@ use std::sync::{use iroh::endpoint::{RecvStream, SendStream, VarInt};use nectarlink_protocol::{DeviceId, Envelope, ErrorCode, PacketKind,- messages::{MirrorConfig, MirrorInput, MirrorStart, StreamHeader, mirror, types},+ messages::{MirrorAudioConfig, MirrorConfig, MirrorInput, MirrorStart, StreamHeader, mirror, types},read_video_packet, video_packet_header, write_frame,};use tokio::sync::mpsc;@@ -27,6 +30,8 @@ pub(crate) const TOGGLE: &str = "mirroring";/// Packets waiting to go out, at most. Beyond this the network is behind:/// sending more would only add delay.const QUEUE: usize = 4;+/// Sound packets waiting to go out, at most (each is a few milliseconds).+const AUDIO_QUEUE: usize = 24;/// What the PC's app gets of a mirroring stream. Called from the core's/// threads; must return quickly (hand the work to a thread of its own).@@ -37,6 +42,12 @@ pub trait MirrorSink: Send + Sync {fn packet(&self, keyframe: bool, time_us: u64, data: Vec<u8>);/// The stream ended (the phone stopped, or the connection dropped).fn ended(&self);+ /// The sound's format: first on the sound stream, and when it changes.+ fn audio_config(&self, _config: MirrorAudioConfig) {}+ /// Sound, in that format.+ fn audio(&self, _time_us: u64, _data: Vec<u8>) {}+ /// The sound stream ended.+ fn audio_ended(&self) {}}impl std::fmt::Debug for dyn MirrorSink {@@ -90,7 +101,7 @@ pub(crate) async fn receive(shared: Arc<Shared>, peer: DeviceId, mut send: SendSreturn;};shared.emit(NodeEvent::Mirroring { device: peer, on: true });- let stopped = shared.new_mirror_stop(&peer);+ let stopped = shared.new_mirror_stop(&peer, mirror::OP_VIDEO);loop {let packet = tokio::select! {_ = stopped.notified() => break,@@ -118,6 +129,47 @@ pub(crate) async fn receive(shared: Arc<Shared>, peer: DeviceId, mut send: SendSshared.emit(NodeEvent::Mirroring { device: peer, on: false });}+/// A phone opened a sound stream (its header already read).+pub(crate) async fn receive_audio(+ shared: Arc<Shared>,+ peer: DeviceId,+ mut send: SendStream,+ mut recv: RecvStream,+) {+ let listens = shared.local_capabilities().iter().any(|c| c == mirror::LISTEN);+ let sink = listens.then(|| shared.platform.mirror_sink(&peer)).flatten();+ let Some(sink) = sink.filter(|_| shared.toggle_on(&peer, TOGGLE)) else {+ let _ = send.reset(VarInt::from_u32(mirror::STOPPED));+ let _ = recv.stop(VarInt::from_u32(mirror::STOPPED));+ return;+ };+ let stopped = shared.new_mirror_stop(&peer, mirror::OP_AUDIO);+ loop {+ let packet = tokio::select! {+ _ = stopped.notified() => break,+ packet = read_video_packet(&mut recv) => packet,+ };+ match packet {+ Ok(Some(p)) if p.kind == PacketKind::Config => match MirrorAudioConfig::from_cbor(&p.data) {+ Ok(config) if config.is_valid() => sink.audio_config(config),+ _ => {+ tracing::warn!("a phone sent a sound format this PC can't play");+ break;+ }+ },+ Ok(Some(p)) => sink.audio(p.time_us, p.data),+ Ok(None) => break,+ Err(e) => {+ tracing::debug!(error = %e, "mirrored sound ended");+ break;+ }+ }+ }+ let _ = recv.stop(VarInt::from_u32(mirror::STOPPED));+ let _ = send.finish();+ sink.audio_ended();+}+// ---- The phone ----/// Handles `mirror.*` on a session. Returns false for other types.@@ -171,6 +223,8 @@ pub(crate) async fn handle(shared: &Arc<Shared>, session: &Arc<Session>, env: &E#[derive(Debug, Clone, Copy, PartialEq, Eq)]pub enum MirrorSend {Queued,+ /// Sound dropped to keep the delay low (the next packet may go).+ Dropped,/// Dropped to keep the delay low: send a keyframe next (frames are/// dropped until one comes).NeedKeyframe,@@ -178,13 +232,15 @@ pub enum MirrorSend {Closed,}-/// A phone's video stream to one PC.+/// A phone's video (or sound) stream to one PC.#[derive(Debug)]pub struct MirrorStream {queue: mpsc::Sender<(PacketKind, u64, Vec<u8>)>,closed: Arc<AtomicBool>,/// Frames are being dropped until a keyframe.resyncing: Mutex<bool>,+ /// Sound: each packet stands alone, so a dropped one needs no keyframe.+ audio: bool,}impl MirrorStream {@@ -195,7 +251,7 @@ impl MirrorStream {return MirrorSend::Closed;}let mut resyncing = self.resyncing.lock().unwrap_or_else(|e| e.into_inner());- if *resyncing && kind == PacketKind::Frame {+ if *resyncing && kind == PacketKind::Frame && !self.audio {return MirrorSend::NeedKeyframe;}let sent = if kind == PacketKind::Config {@@ -214,6 +270,7 @@ impl MirrorStream {MirrorSend::Queued}Err(()) if self.closed.load(Ordering::Acquire) => MirrorSend::Closed,+ Err(()) if self.audio => MirrorSend::Dropped,Err(()) => {*resyncing = true;MirrorSend::NeedKeyframe@@ -231,18 +288,21 @@ impl MirrorStream {}}-/// Opens a video stream to a PC that asked for the screen.-pub(crate) async fn open(shared: &Arc<Shared>, session: &Session) -> Result<MirrorStream> {+/// Opens a video stream to a PC that asked for the screen, or (`audio`) a+/// sound stream to one that asked for the sound too.+pub(crate) async fn open(shared: &Arc<Shared>, session: &Session, audio: bool) -> Result<MirrorStream> {if !shared.toggle_on(&session.peer, TOGGLE) {return Err(Error::Denied);}let (mut send, mut recv) = session.conn.open_bi().await.map_err(crate::error::net)?;- let header =- StreamHeader { svc: mirror::SERVICE.into(), op: mirror::OP_VIDEO.into(), v: mirror::VERSION };+ let op = if audio { mirror::OP_AUDIO } else { mirror::OP_VIDEO };+ let header = StreamHeader { svc: mirror::SERVICE.into(), op: op.into(), v: mirror::VERSION };write_frame(&mut send, &Envelope::new(types::STREAM, &header)?.to_cbor()).await?;- // Video goes out as soon as it's written.- let _ = send.set_priority(1);- let (queue, mut packets) = mpsc::channel::<(PacketKind, u64, Vec<u8>)>(QUEUE);+ // Sound and video go out as soon as they're written; sound first, as a+ // gap in it is the more noticeable.+ let _ = send.set_priority(if audio { 2 } else { 1 });+ let (queue, mut packets) =+ mpsc::channel::<(PacketKind, u64, Vec<u8>)>(if audio { AUDIO_QUEUE } else { QUEUE });let closed = Arc::new(AtomicBool::new(false));let done = closed.clone();tokio::spawn(async move {@@ -271,21 +331,26 @@ pub(crate) async fn open(shared: &Arc<Shared>, session: &Session) -> Result<Mirrdone.store(true, Ordering::Release);let _ = send.finish();});- Ok(MirrorStream { queue, closed, resyncing: Mutex::new(false) })+ Ok(MirrorStream { queue, closed, resyncing: Mutex::new(false), audio })}impl Shared {- /// A new stop signal for the PC's reader of `peer`'s stream.- fn new_mirror_stop(&self, peer: &DeviceId) -> Arc<tokio::sync::Notify> {+ /// A new stop signal for the PC's reader of `peer`'s video or sound+ /// stream (`op`).+ fn new_mirror_stop(&self, peer: &DeviceId, op: &'static str) -> Arc<tokio::sync::Notify> {let signal = Arc::new(tokio::sync::Notify::new());- self.mirror_stops.lock().unwrap_or_else(|e| e.into_inner()).insert(*peer, signal.clone());+ self.mirror_stops.lock().unwrap_or_else(|e| e.into_inner()).insert((*peer, op), signal.clone());signal}- /// Stops showing `peer`'s screen here (kept until the reader sees it).+ /// Stops showing (and playing) `peer`'s screen here (kept until the+ /// readers see it).pub(crate) fn stop_showing(&self, peer: &DeviceId) {- if let Some(signal) = self.mirror_stops.lock().unwrap_or_else(|e| e.into_inner()).remove(peer) {- signal.notify_one();+ let mut stops = self.mirror_stops.lock().unwrap_or_else(|e| e.into_inner());+ for op in [mirror::OP_VIDEO, mirror::OP_AUDIO] {+ if let Some(signal) = stops.remove(&(*peer, op)) {+ signal.notify_one();+ }}}}@@ -297,7 +362,8 @@ mod tests {#[tokio::test(flavor = "multi_thread")]async fn a_full_queue_drops_frames_until_a_keyframe() {let (queue, mut packets) = mpsc::channel(2);- let stream = MirrorStream { queue, closed: Arc::default(), resyncing: Mutex::new(false) };+ let stream =+ MirrorStream { queue, closed: Arc::default(), resyncing: Mutex::new(false), audio: false };let send = |kind| tokio::task::block_in_place(|| stream.send(kind, 0, vec![1]));assert_eq!(send(PacketKind::Keyframe), MirrorSend::Queued);assert_eq!(send(PacketKind::Frame), MirrorSend::Queued);@@ -311,4 +377,17 @@ mod tests {drop(packets);assert_eq!(send(PacketKind::Keyframe), MirrorSend::Closed);}++ #[tokio::test(flavor = "multi_thread")]+ async fn sound_drops_only_what_doesnt_fit() {+ let (queue, mut packets) = mpsc::channel(1);+ let stream =+ MirrorStream { queue, closed: Arc::default(), resyncing: Mutex::new(false), audio: true };+ let send = |kind| tokio::task::block_in_place(|| stream.send(kind, 0, vec![1]));+ assert_eq!(send(PacketKind::Frame), MirrorSend::Queued);+ assert_eq!(send(PacketKind::Frame), MirrorSend::Dropped, "full");+ packets.recv().await;+ // No keyframe to wait for: the next one goes.+ assert_eq!(send(PacketKind::Frame), MirrorSend::Queued);+ }}diff --git a/core/nectarlink-core/src/node.rs b/core/nectarlink-core/src/node.rsindex 43f672f..b3288a9 100644--- a/core/nectarlink-core/src/node.rs+++ b/core/nectarlink-core/src/node.rs@@ -120,7 +120,7 @@ pub(crate) struct Shared {/// This phone's call in progress (docs/protocol/calls.md).pub(crate) calls: crate::calls::Current,/// Stop signals for phone screens shown here, by phone.- pub(crate) mirror_stops: Mutex<HashMap<DeviceId, Arc<tokio::sync::Notify>>>,+ pub(crate) mirror_stops: Mutex<HashMap<(DeviceId, &'static str), Arc<tokio::sync::Notify>>>,pub data_dir: std::path::PathBuf,/// Where received files go.pub downloads_dir: std::path::PathBuf,@@ -957,7 +957,14 @@ impl Node {/// Opens this phone's video stream to a PC that asked for the screen.pub async fn mirror_open(&self, peer: DeviceId) -> Result<crate::MirrorStream> {let session = self.connected(&peer)?;- crate::mirror::open(&self.shared, &session).await+ crate::mirror::open(&self.shared, &session, false).await+ }++ /// Opens this phone's sound stream to a PC that asked for the screen+ /// with its sound.+ pub async fn mirror_open_audio(&self, peer: DeviceId) -> Result<crate::MirrorStream> {+ let session = self.connected(&peer)?;+ crate::mirror::open(&self.shared, &session, true).await}// ---- Messages (docs/protocol/sms.md) ----diff --git a/core/nectarlink-core/src/transfer.rs b/core/nectarlink-core/src/transfer.rsindex 718f29b..e6e4802 100644--- a/core/nectarlink-core/src/transfer.rs+++ b/core/nectarlink-core/src/transfer.rs@@ -560,6 +560,9 @@ pub(crate) async fn accept_stream(Some(h) if h.svc == mirror::SERVICE && h.op == mirror::OP_VIDEO && h.v == mirror::VERSION => {crate::mirror::receive(shared, session.peer, send, recv).await;}+ Some(h) if h.svc == mirror::SERVICE && h.op == mirror::OP_AUDIO && h.v == mirror::VERSION => {+ crate::mirror::receive_audio(shared, session.peer, send, recv).await;+ }_ => {let reply = Envelope::error(ErrorCode::Unsupported, "unknown stream");let _ = write_frame(&mut send, &reply.to_cbor()).await;diff --git a/core/nectarlink-core/tests/two_devices.rs b/core/nectarlink-core/tests/two_devices.rsindex d12c6ac..a07f33f 100644--- a/core/nectarlink-core/tests/two_devices.rs+++ b/core/nectarlink-core/tests/two_devices.rs@@ -86,7 +86,8 @@ impl Platform for RecordingPlatform {_peer: &nectarlink_core::DeviceId,options: &nectarlink_core::MirrorStart,) -> Result<(), String> {- self.mirror_asks.lock().unwrap().push(format!("start {}", options.max_size));+ let sound = if options.audio { " with sound" } else { "" };+ self.mirror_asks.lock().unwrap().push(format!("start {}{sound}", options.max_size));Ok(())}fn mirror_stop_requested(&self, _peer: &nectarlink_core::DeviceId) {@@ -178,6 +179,15 @@ impl nectarlink_core::MirrorSink for ScreenSink {fn ended(&self) {self.got.lock().unwrap().push("ended".into());}+ fn audio_config(&self, config: nectarlink_core::MirrorAudioConfig) {+ self.got.lock().unwrap().push(format!("sound {} Hz x{}", config.rate, config.channels));+ }+ fn audio(&self, time_us: u64, data: Vec<u8>) {+ self.got.lock().unwrap().push(format!("sound {time_us} {}", data.len()));+ }+ fn audio_ended(&self) {+ self.got.lock().unwrap().push("sound ended".into());+ }}struct TestDevice {@@ -1034,19 +1044,28 @@ async fn folders_arrive_with_their_layout() {#[tokio::test(flavor = "multi_thread")]async fn the_phone_screen_streams_to_the_pc() {use nectarlink_core::{MirrorSend, PacketKind};- let mut pc = device_with("Desktop", DeviceKind::Desktop, &[nectarlink_core::MIRROR_VIEW]).await;+ let mut pc = device_with(+ "Desktop",+ DeviceKind::Desktop,+ &[nectarlink_core::MIRROR_VIEW, nectarlink_core::MIRROR_LISTEN],+ )+ .await;let mut phone = device_with("Pixel",DeviceKind::Phone,- &[nectarlink_core::MIRROR_CAPTURE, nectarlink_core::MIRROR_INPUT],+ &[+ nectarlink_core::MIRROR_CAPTURE,+ nectarlink_core::MIRROR_INPUT,+ nectarlink_core::MIRROR_AUDIO_PLAYBACK,+ ],).await;pair_qr(&mut pc, &mut phone).await;let (pc_id, phone_id) = (pc.node.device_id(), phone.node.device_id());- let options = nectarlink_core::MirrorStart { max_size: 1920, fps: 60, bitrate: 8_000_000 };+ let options = nectarlink_core::MirrorStart { max_size: 1920, fps: 60, bitrate: 8_000_000, audio: true };with_timeout("start", pc.node.mirror_start(phone_id, options)).await.unwrap();- assert_eq!(*phone.platform.mirror_asks.lock().unwrap(), ["start 1920"]);+ assert_eq!(*phone.platform.mirror_asks.lock().unwrap(), ["start 1920 with sound"]);// The user agreed: the phone streams.let stream = Arc::new(with_timeout("open", phone.node.mirror_open(pc_id)).await.unwrap());@@ -1070,6 +1089,29 @@ async fn the_phone_screen_streams_to_the_pc() {}assert_eq!(*screen.got.lock().unwrap(), ["config 1080x2400", "key 1 90000", "frame 16667 4000"]);+ // And its sound, on a stream of its own.+ let sound = Arc::new(with_timeout("open sound", phone.node.mirror_open_audio(pc_id)).await.unwrap());+ let format = nectarlink_core::MirrorAudioConfig {+ codec: nectarlink_core::MIRROR_PCM.into(),+ rate: 48_000,+ channels: 2,+ };+ let sender = sound.clone();+ let sent = tokio::task::spawn_blocking(move || {+ [+ sender.send(PacketKind::Config, 0, format.to_cbor()),+ sender.send(PacketKind::Frame, 10_000, vec![0; 1_920]),+ ]+ })+ .await+ .unwrap();+ assert!(sent.iter().all(|s| *s == MirrorSend::Queued), "{sent:?}");+ let deadline = tokio::time::Instant::now() + WAIT;+ while screen.got.lock().unwrap().len() < 5 && tokio::time::Instant::now() < deadline {+ tokio::time::sleep(std::time::Duration::from_millis(20)).await;+ }+ assert_eq!(screen.got.lock().unwrap()[3..], ["sound 48000 Hz x2", "sound 10000 1920"]);+pc.node.mirror_keyframe(phone_id).await;// The PC's mouse on the phone's screen; nonsense never leaves the PC.use nectarlink_core::{MirrorInput, TouchAction};@@ -1082,7 +1124,14 @@ async fn the_phone_screen_streams_to_the_pc() {// The PC stops watching: the phone hears it, and its stream closes.pc.node.mirror_stop(phone_id).await;wait_for(&mut pc, "stopped", |e| matches!(e, NodeEvent::Mirroring { on: false, .. }).then_some(())).await;- assert_eq!(screen.got.lock().unwrap().last().map(String::as_str), Some("ended"));+ let deadline = tokio::time::Instant::now() + WAIT;+ while !screen.got.lock().unwrap().iter().any(|g| g == "sound ended")+ && tokio::time::Instant::now() < deadline+ {+ tokio::time::sleep(std::time::Duration::from_millis(20)).await;+ }+ let got = screen.got.lock().unwrap().clone();+ assert!(got.iter().any(|g| g == "ended") && got.iter().any(|g| g == "sound ended"), "{got:?}");let deadline = tokio::time::Instant::now() + WAIT;while !stream.is_closed() && tokio::time::Instant::now() < deadline {let s = stream.clone();@@ -1090,6 +1139,13 @@ async fn the_phone_screen_streams_to_the_pc() {tokio::time::sleep(std::time::Duration::from_millis(20)).await;}assert!(stream.is_closed(), "the phone's stream closes");+ let deadline = tokio::time::Instant::now() + WAIT;+ while !sound.is_closed() && tokio::time::Instant::now() < deadline {+ let s = sound.clone();+ tokio::task::spawn_blocking(move || s.send(PacketKind::Frame, 0, vec![0])).await.unwrap();+ tokio::time::sleep(std::time::Duration::from_millis(20)).await;+ }+ assert!(sound.is_closed(), "and its sound stream");let asks = phone.platform.mirror_asks.lock().unwrap().clone();for wanted in ["keyframe", "Touch(Down)", "Key(back)", "stop"] {assert!(asks.iter().any(|a| a == wanted), "{wanted}: {asks:?}");diff --git a/core/nectarlink-ffi/src/lib.rs b/core/nectarlink-ffi/src/lib.rsindex 713d47a..3a65595 100644--- a/core/nectarlink-ffi/src/lib.rs+++ b/core/nectarlink-ffi/src/lib.rs@@ -519,10 +519,12 @@ impl From<core::MirrorInput> for MirrorInputEvent {}}-/// What became of a video packet.+/// What became of a video (or sound) packet.#[derive(Debug, Clone, Copy, PartialEq, Eq, uniffi::Enum)]pub enum MirrorSendResult {Queued,+ /// Sound dropped: the network is behind (the next packet may go).+ Dropped,/// The network is behind: request a keyframe from the encoder (frames/// are dropped until one comes).NeedKeyframe,@@ -530,7 +532,7 @@ pub enum MirrorSendResult {Closed,}-/// This phone's screen going to one PC.+/// This phone's screen (or its sound) going to one PC.#[derive(Debug, uniffi::Object)]pub struct MirrorStream(core::MirrorStream);@@ -545,6 +547,7 @@ impl MirrorStream {};match self.0.send(kind, time_us, data) {core::MirrorSend::Queued => MirrorSendResult::Queued,+ core::MirrorSend::Dropped => MirrorSendResult::Dropped,core::MirrorSend::NeedKeyframe => MirrorSendResult::NeedKeyframe,core::MirrorSend::Closed => MirrorSendResult::Closed,}@@ -562,6 +565,12 @@ pub fn mirror_config(width: u32, height: u32) -> Vec<u8> {core::MirrorConfig { codec: "h264".into(), width, height }.to_cbor()}+/// The bytes of a config packet for 16-bit PCM sound.+#[uniffi::export]+pub fn mirror_audio_config(rate: u32, channels: u8) -> Vec<u8> {+ core::MirrorAudioConfig { codec: core::MIRROR_PCM.into(), rate, channels }.to_cbor()+}+/// A conversation (docs/protocol/sms.md).#[derive(Clone, PartialEq, Eq, uniffi::Record)]pub struct SmsThread {@@ -1198,9 +1207,10 @@ pub trait Platform: Send + Sync {/// A PC asked to answer, decline or silence call `id` (the call in/// progress). False if the phone couldn't.fn call_command(&self, id: String, command: CallCommand) -> bool;- /// A PC asked for this phone's screen: ask the user (then call- /// `mirror_open`). False if the user couldn't be asked.- fn mirror_requested(&self, pc_id: String, max_size: u32, fps: u32, bitrate: u32) -> bool;+ /// A PC asked for this phone's screen (and, with `audio`, its sound):+ /// ask the user (then call `mirror_open`, and `mirror_open_audio` for+ /// the sound). False if the user couldn't be asked.+ fn mirror_requested(&self, pc_id: String, max_size: u32, fps: u32, bitrate: u32, audio: bool) -> bool;/// The PC stopped watching: stop sharing.fn mirror_stop_requested(&self, pc_id: String);/// The PC needs a keyframe.@@ -1281,7 +1291,13 @@ impl core::Platform for PlatformAdapter {}}fn mirror_requested(&self, peer: &DeviceId, options: &core::MirrorStart) -> Result<(), String> {- if self.0.mirror_requested(peer.to_string(), options.max_size, options.fps, options.bitrate) {+ if self.0.mirror_requested(+ peer.to_string(),+ options.max_size,+ options.fps,+ options.bitrate,+ options.audio,+ ) {Ok(())} else {Err("the user couldn't be asked".into())@@ -1665,6 +1681,14 @@ impl NectarlinkNode {self.run(async move { Ok(Arc::new(MirrorStream(node.mirror_open(id).await?))) }).await}+ /// Opens this phone's sound stream to a PC that asked for it with the+ /// screen.+ pub async fn mirror_open_audio(&self, pc_id: String) -> Result<Arc<MirrorStream>> {+ let id = parse_id(&pc_id)?;+ let node = self.node.clone();+ self.run(async move { Ok(Arc::new(MirrorStream(node.mirror_open_audio(id).await?))) }).await+ }+/// This phone's messages changed (in `thread`, or anywhere when null):/// PCs that show them catch up.pub async fn sms_changed(&self, thread: Option<String>) {diff --git a/core/nectarlink-protocol/src/messages.rs b/core/nectarlink-protocol/src/messages.rsindex a9ae6eb..476a35a 100644--- a/core/nectarlink-protocol/src/messages.rs+++ b/core/nectarlink-protocol/src/messages.rs@@ -484,6 +484,15 @@ pub mod mirror {/// Typed text in one message: at most this many bytes.pub const MAX_TEXT_BYTES: usize = 4096;pub const H264: &str = "h264";+ /// Offered by phones that stream all their sound while mirrored.+ pub const AUDIO: &str = "mirror.audio";+ /// Offered by phones that stream the sound of apps that allow it.+ pub const AUDIO_PLAYBACK: &str = "mirror.audio.playback";+ /// Offered by PCs that play a mirrored phone's sound.+ pub const LISTEN: &str = "mirror.listen";+ pub const OP_AUDIO: &str = "audio";+ /// 16-bit little-endian PCM, channels interleaved.+ pub const PCM: &str = "pcm_s16le";}/// Body of `mirror.start`: show me your screen.@@ -495,6 +504,9 @@ pub struct MirrorStart {pub fps: u32,/// Bits per second to aim for.pub bitrate: u32,+ /// Also stream the phone's sound, when it can.+ #[serde(default)]+ pub audio: bool,}/// Body of `mirror.input`: the PC's mouse and keyboard, on the mirrored@@ -597,6 +609,39 @@ impl MirrorConfig {}}+/// The format of a mirrored phone's sound (an audio stream's config+/// packet, in CBOR).+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub struct MirrorAudioConfig {+ /// `pcm_s16le`.+ pub codec: String,+ /// Samples per second.+ pub rate: u32,+ pub channels: u8,+}++impl MirrorAudioConfig {+ pub fn to_cbor(&self) -> Vec<u8> {+ let mut out = Vec::new();+ ciborium::into_writer(self, &mut out).expect("writing to a Vec cannot fail");+ out+ }++ pub fn from_cbor(bytes: &[u8]) -> Result<MirrorAudioConfig, crate::ProtocolError> {+ ciborium::from_reader(bytes).map_err(|e| crate::ProtocolError::BadMessage(e.to_string()))+ }++ /// A format this version can play: PCM, 8–96 kHz, mono or stereo.+ pub fn is_valid(&self) -> bool {+ self.codec == mirror::PCM && (8_000..=96_000).contains(&self.rate) && (1..=2).contains(&self.channels)+ }++ /// Bytes in one sample frame (a sample for each channel).+ pub fn frame_bytes(&self) -> usize {+ 2 * usize::from(self.channels)+ }+}+// ---- Messages (docs/protocol/sms.md) ----pub mod sms {diff --git a/desktop/app/qml/Icons.qml b/desktop/app/qml/Icons.qmlindex 8fc2729..c77f2a5 100644--- a/desktop/app/qml/Icons.qml+++ b/desktop/app/qml/Icons.qml@@ -29,6 +29,7 @@ QtObject {readonly property string mic: "M12 2a3 3 0 0 1 3 3v6a3 3 0 0 1-6 0V5a3 3 0 0 1 3-3zM5 11a7 7 0 0 0 14 0M12 18v4"readonly property string micOff: "M15 10V5a3 3 0 0 0-5.8-1.1M9 9v2a3 3 0 0 0 4.9 2.3M5 11a7 7 0 0 0 11.3 5.5M19 11a7 7 0 0 1-.6 2.8M12 18v4M3 3l18 18"readonly property string speaker: "M4 9h3.5L12 5v14l-4.5-4H4zM15.5 9a4 4 0 0 1 0 6M18.5 6.5a8 8 0 0 1 0 11"+ readonly property string soundOff: "M4 9h3.5L12 5v14l-4.5-4H4zM16 9.5l5 5M21 9.5l-5 5"readonly property string volumeUp: "M4 9h3.5L12 5v14l-4.5-4H4zM16 12h5M18.5 9.5v5"readonly property string volumeDown: "M4 9h3.5L12 5v14l-4.5-4H4zM16 12h5"readonly property string dialpad: "M6 4.5h.01M12 4.5h.01M18 4.5h.01M6 10h.01M12 10h.01M18 10h.01M6 15.5h.01M12 15.5h.01M18 15.5h.01M12 21h.01"diff --git a/desktop/app/qml/MirrorWindow.qml b/desktop/app/qml/MirrorWindow.qmlindex 955f99d..e9b4af6 100644--- a/desktop/app/qml/MirrorWindow.qml+++ b/desktop/app/qml/MirrorWindow.qml@@ -123,6 +123,28 @@ Window {}}+ // The phone's sound on this PC: on or off, shown while the mouse is+ // over the window (and while it's off).+ HoverHandler { id: hovering }+ Rectangle {+ anchors.top: parent.top+ anchors.right: parent.right+ anchors.margins: 12+ width: 40; height: 40+ radius: 20+ color: Qt.rgba(0, 0, 0, 0.6)+ visible: Mirror.sound && Mirror.phase === "showing"+ opacity: hovering.hovered || Mirror.muted ? 1 : 0+ Behavior on opacity { NumberAnimation { duration: Theme.fadeFast } }+ IconButton {+ anchors.centerIn: parent+ iconPath: Mirror.muted ? Icons.soundOff : Icons.speaker+ iconColor: "white"+ label: Mirror.muted ? qsTr("Play the phone's sound here") : qsTr("Mute the phone's sound here")+ onClicked: Mirror.toggleSound()+ }+ }+// How to control it, until the phone allows it.Rectangle {anchors.left: parent.leftdiff --git a/desktop/app/src/bridge/mirror.rs b/desktop/app/src/bridge/mirror.rsindex f23b535..fbe671b 100644--- a/desktop/app/src/bridge/mirror.rs+++ b/desktop/app/src/bridge/mirror.rs@@ -33,6 +33,10 @@ pub mod qobject {#[qproperty(QString, reason)]/// The PC's mouse and keyboard work on the phone.#[qproperty(bool, can_control, cxx_name = "canControl")]+ /// The phone's sound is coming in.+ #[qproperty(bool, sound)]+ /// The user turned the sound off on this PC.+ #[qproperty(bool, muted)]type Mirror = super::MirrorRust;}@@ -62,6 +66,10 @@ pub mod qobject {/// Types the PC's clipboard text on the phone.#[qinvokable]fn paste(self: &Mirror);+ /// Turns the phone's sound off on this PC, or back on.+ #[qinvokable]+ #[cxx_name = "toggleSound"]+ fn toggle_sound(self: &Mirror);}}@@ -72,6 +80,8 @@ pub struct MirrorRust {phase: QString,reason: QString,can_control: bool,+ sound: bool,+ muted: bool,shown: Option<DeviceId>,}@@ -92,6 +102,8 @@ impl qobject::Mirror {== Some(nectarlink_core::FeatureState::Available)});self.as_mut().set_can_control(control);+ self.as_mut().set_sound(mirror::has_sound(&device));+ self.as_mut().set_muted(mirror::muted());let (phase, reason) = match mirror::phase(&device) {None => ("", String::new()),Some(Phase::Asking) => ("asking", String::new()),@@ -168,6 +180,10 @@ impl qobject::Mirror {}}+ pub fn toggle_sound(&self) {+ mirror::set_muted(!mirror::muted());+ }+pub fn stop(mut self: Pin<&mut Self>) {if let Some(id) = self.as_mut().rust_mut().shown.take() {mirror::stop(id);diff --git a/desktop/app/src/core_host.rs b/desktop/app/src/core_host.rsindex 65d71c7..5a712be 100644--- a/desktop/app/src/core_host.rs+++ b/desktop/app/src/core_host.rs@@ -102,6 +102,7 @@ async fn run(data_dir: PathBuf, platform: Arc<dyn Platform>) {nectarlink_core::CALLS_SHOW.into(),nectarlink_core::SMS_SHOW.into(),nectarlink_core::MIRROR_VIEW.into(),+ nectarlink_core::MIRROR_LISTEN.into(),];let node = match Node::start(config, platform).await {Ok(node) => node,diff --git a/desktop/app/src/mirror.rs b/desktop/app/src/mirror.rsindex d913a1a..d9bb09c 100644--- a/desktop/app/src/mirror.rs+++ b/desktop/app/src/mirror.rs@@ -3,29 +3,33 @@//! decodes its video on a thread of its own and hands each picture to the//! mirror window's `VideoView`. Only the newest picture is shown; when the//! decoder falls behind or loses its place, the phone is asked for a-//! keyframe.+//! keyframe. The phone's sound, when it sends it, plays on the PC's+//! speakers.use std::{collections::HashMap,sync::{- Arc, Mutex,+ Arc, Mutex, OnceLock,atomic::{AtomicUsize, Ordering},mpsc::{Receiver, SyncSender, TrySendError, sync_channel},},time::{Duration, Instant},};-use nectarlink_core::{DeviceId, Error, MirrorConfig, MirrorSink, MirrorStart, NodeEvent};+use nectarlink_core::{DeviceId, Error, MirrorAudioConfig, MirrorConfig, MirrorSink, MirrorStart, NodeEvent};use crate::{bridge::{app::describe, native::ffi},core_host,state::Changes,- win::h264::{Decoder, to_bgrx},+ win::{+ audio_out::Player,+ h264::{Decoder, to_bgrx},+ },};/// What the PC asks for: the phone scales its screen to fit.-const OPTIONS: MirrorStart = MirrorStart { max_size: 1920, fps: 60, bitrate: 8_000_000 };+const OPTIONS: MirrorStart = MirrorStart { max_size: 1920, fps: 60, bitrate: 8_000_000, audio: true };/// Packets waiting for the decoder, at most (about two seconds).const BACKLOG: usize = 120;/// How often decode times are logged.@@ -44,6 +48,30 @@ pub enum Phase {#[derive(Debug, Default)]struct State {phases: HashMap<DeviceId, Phase>,+ /// Phones whose sound is coming in.+ sound: std::collections::HashSet<DeviceId>,+}++/// The user turned the phones' sound off on this PC (until turned on).+static MUTED: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);++pub fn muted() -> bool {+ MUTED.load(Ordering::Relaxed)+}++pub fn set_muted(muted: bool) {+ MUTED.store(muted, Ordering::Relaxed);+ core_host::host().hub.changed(Changes::MIRROR);+}++/// Whether `device`'s sound is coming in.+pub fn has_sound(device: &DeviceId) -> bool {+ state(|s| s.sound.contains(device))+}++fn set_sound(device: DeviceId, on: bool) {+ state(|s| if on { s.sound.insert(device) } else { s.sound.remove(&device) });+ core_host::host().hub.changed(Changes::MIRROR);}static STATE: Mutex<Option<State>> = Mutex::new(None);@@ -122,10 +150,11 @@ pub fn input(device: DeviceId, input: nectarlink_core::MirrorInput) {let _ = queue.send((device, input));}-/// Where the core puts a phone's video: only for phones this PC asked.+/// Where the core puts a phone's video (or sound, from a stream of its+/// own): only for phones this PC asked.pub fn sink(device: &DeviceId) -> Option<Arc<dyn MirrorSink>> {matches!(phase(device), Some(Phase::Asking | Phase::Showing))- .then(|| Arc::new(Sink::start(*device)) as Arc<dyn MirrorSink>)+ .then(|| Arc::new(Sink::new(*device)) as Arc<dyn MirrorSink>)}enum Item {@@ -134,16 +163,29 @@ enum Item {Ended,}-/// Hands the core's packets to the decoder thread.+/// Hands the core's packets to the decoder thread, or its sound to the+/// player; each starts with its stream's first packet (the format).struct Sink {device: DeviceId,+ video: OnceLock<Video>,+ sound: Mutex<Option<Player>>,+}++/// The decoder thread's end of a sink.+struct Video {queue: SyncSender<Item>,/// Packets waiting for the decoder.waiting: Arc<AtomicUsize>,}impl Sink {- fn start(device: DeviceId) -> Sink {+ fn new(device: DeviceId) -> Sink {+ Sink { device, video: OnceLock::new(), sound: Mutex::new(None) }+ }+}++impl Video {+ fn start(device: DeviceId) -> Video {let (queue, items) = sync_channel(BACKLOG);let waiting = Arc::new(AtomicUsize::new(0));let counter = waiting.clone();@@ -153,9 +195,11 @@ impl Sink {if let Err(e) = started {tracing::error!(error = %e, "can't start the video decoder");}- Sink { device, queue, waiting }+ Video { queue, waiting }}+}+impl Sink {fn keyframe_please(&self) {let device = self.device;if let Some(node) = core_host::node() {@@ -166,14 +210,16 @@ impl Sink {impl MirrorSink for Sink {fn config(&self, config: MirrorConfig) {+ let video = self.video.get_or_init(|| Video::start(self.device));// The format matters: wait for room rather than drop it.- let _ = self.queue.send(Item::Config(config));+ let _ = video.queue.send(Item::Config(config));}fn packet(&self, keyframe: bool, time_us: u64, data: Vec<u8>) {- match self.queue.try_send(Item::Packet { keyframe, time_us, data }) {+ let Some(video) = self.video.get() else { return };+ match video.queue.try_send(Item::Packet { keyframe, time_us, data }) {Ok(()) => {- self.waiting.fetch_add(1, Ordering::AcqRel);+ video.waiting.fetch_add(1, Ordering::AcqRel);}Err(TrySendError::Disconnected(_)) => {}// The decoder is behind; it starts over at the next keyframe.@@ -182,7 +228,36 @@ impl MirrorSink for Sink {}fn ended(&self) {- let _ = self.queue.send(Item::Ended);+ if let Some(video) = self.video.get() {+ let _ = video.queue.send(Item::Ended);+ }+ }++ fn audio_config(&self, config: MirrorAudioConfig) {+ let mut sound = self.sound.lock().unwrap_or_else(|e| e.into_inner());+ // A new format: a new player (the old one ends when dropped).+ *sound = match Player::start(config.rate, u16::from(config.channels)) {+ Ok(player) => Some(player),+ Err(e) => {+ tracing::warn!(error = %e, "can't play the phone's sound");+ None+ }+ };+ set_sound(self.device, sound.is_some());+ }++ fn audio(&self, _time_us: u64, data: Vec<u8>) {+ if muted() {+ return;+ }+ if let Some(player) = self.sound.lock().unwrap_or_else(|e| e.into_inner()).as_ref() {+ player.push(data);+ }+ }++ fn audio_ended(&self) {+ self.sound.lock().unwrap_or_else(|e| e.into_inner()).take();+ set_sound(self.device, false);}}diff --git a/desktop/app/src/win/audio_out.rs b/desktop/app/src/win/audio_out.rsnew file mode 100644index 0000000..46da01b--- /dev/null+++ b/desktop/app/src/win/audio_out.rs@@ -0,0 +1,293 @@+// SPDX-License-Identifier: GPL-3.0-or-later+//! Playing a mirrored phone's sound on the PC's speakers: 16-bit PCM from+//! the phone goes through a small jitter buffer to the default output+//! device (WASAPI, shared mode), on a thread of its own.+//!+//! The buffer favors low delay: playback starts once a few packets are in,+//! falls back to silence (and waits again) when the network stalls, and+//! drops the oldest sound when it piles up, so the sound stays in step+//! with the picture.++use std::{+ collections::VecDeque,+ sync::mpsc::{Receiver, RecvTimeoutError, SyncSender, TrySendError, sync_channel},+ time::Duration,+};++use windows::Win32::{+ Foundation::{CloseHandle, HANDLE},+ Media::Audio::{+ AUDCLNT_SHAREMODE_SHARED, AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM, AUDCLNT_STREAMFLAGS_EVENTCALLBACK,+ AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY, IAudioClient, IAudioRenderClient, IMMDeviceEnumerator,+ MMDeviceEnumerator, WAVE_FORMAT_PCM, WAVEFORMATEX, eConsole, eRender,+ },+ System::{+ Com::{CLSCTX_ALL, COINIT_MULTITHREADED, CoCreateInstance, CoInitializeEx, CoUninitialize},+ Threading::{CreateEventW, WaitForSingleObject},+ },+};++/// Sound packets waiting for the player thread, at most.+const QUEUE: usize = 64;+/// How much sound is gathered before playing (and again after a stall).+const START_MS: u32 = 40;+/// More than this waiting, and the oldest is dropped down to [`TRIM_TO_MS`].+const MOST_MS: u32 = 200;+const TRIM_TO_MS: u32 = 60;+/// The device's buffer.+const DEVICE_BUFFER: Duration = Duration::from_millis(40);++/// Plays sound until dropped.+#[derive(Debug)]+pub struct Player {+ queue: SyncSender<Vec<u8>>,+}++impl Player {+ /// Starts playing `channels` × 16-bit PCM at `rate` on the default+ /// output device.+ pub fn start(rate: u32, channels: u16) -> std::io::Result<Player> {+ let (queue, packets) = sync_channel(QUEUE);+ std::thread::Builder::new().name("mirror-sound".into()).spawn(move || {+ let mut attempts = 0;+ // The device can go away (headphones unplugged): start over on+ // the new default, a few times.+ while attempts < 5 {+ match play(rate, channels, &packets) {+ Ok(()) => return,+ Err(e) => {+ tracing::warn!(error = %e, "the sound stopped; trying the default device again");+ attempts += 1;+ std::thread::sleep(Duration::from_millis(300));+ }+ }+ }+ })?;+ Ok(Player { queue })+ }++ /// Hands over a packet without waiting (dropped if the player is far+ /// behind: it would be too late to hear anyway).+ pub fn push(&self, data: Vec<u8>) {+ if let Err(TrySendError::Full(_)) = self.queue.try_send(data) {+ tracing::debug!("a sound packet was dropped");+ }+ }+}++/// The jitter buffer, in bytes of interleaved 16-bit samples.+#[derive(Debug)]+struct Jitter {+ bytes: VecDeque<u8>,+ /// Bytes in one sample frame.+ frame: usize,+ /// Bytes in one millisecond.+ per_ms: usize,+ /// Gathering sound before playing.+ waiting: bool,+ /// Times it ran dry, and bytes dropped for piling up (for the log).+ stalls: u32,+ trimmed: usize,+}++impl Jitter {+ fn new(rate: u32, channels: u16) -> Jitter {+ let frame = 2 * usize::from(channels);+ Jitter {+ bytes: VecDeque::new(),+ frame,+ per_ms: frame * rate as usize / 1000,+ waiting: true,+ stalls: 0,+ trimmed: 0,+ }+ }++ fn push(&mut self, data: &[u8]) {+ self.bytes.extend(data);+ if self.bytes.len() > self.per_ms * MOST_MS as usize {+ let keep = self.per_ms * TRIM_TO_MS as usize;+ // Whole frames only, or the channels swap.+ let drop = (self.bytes.len() - keep) / self.frame * self.frame;+ self.bytes.drain(..drop);+ self.trimmed += drop;+ }+ }++ /// Fills `out` (whole frames): sound when there is some, silence when+ /// gathering or stalled.+ fn fill(&mut self, out: &mut [u8]) {+ if self.waiting && self.bytes.len() >= self.per_ms * START_MS as usize {+ self.waiting = false;+ }+ let mut n = 0;+ if !self.waiting {+ n = out.len().min(self.bytes.len()) / self.frame * self.frame;+ for (o, b) in out[..n].iter_mut().zip(self.bytes.drain(..n)) {+ *o = b;+ }+ if n < out.len() {+ // Ran dry: gather again before playing on.+ self.waiting = true;+ self.stalls += 1;+ }+ }+ out[n..].fill(0);+ }+}++/// Plays until the sender is gone (`Ok`) or the device fails (`Err`).+fn play(rate: u32, channels: u16, packets: &Receiver<Vec<u8>>) -> windows::core::Result<()> {+ // SAFETY: balanced below; this thread is ours and uses only the MTA.+ unsafe { CoInitializeEx(None, COINIT_MULTITHREADED) }.ok()?;+ let event = Event::new()?;+ let result = render(rate, channels, packets, event.0);+ drop(event);+ // SAFETY: pairs with CoInitializeEx above; every COM object is gone.+ unsafe { CoUninitialize() };+ result+}++fn render(rate: u32, channels: u16, packets: &Receiver<Vec<u8>>, event: HANDLE) -> windows::core::Result<()> {+ let block = 2 * channels;+ let format = WAVEFORMATEX {+ wFormatTag: WAVE_FORMAT_PCM as u16,+ nChannels: channels,+ nSamplesPerSec: rate,+ nAvgBytesPerSec: rate * u32::from(block),+ nBlockAlign: block,+ wBitsPerSample: 16,+ cbSize: 0,+ };+ // SAFETY: COM calls on live objects, in this thread's apartment; the+ // render buffer is written only within its frame count, between+ // GetBuffer and ReleaseBuffer.+ unsafe {+ let devices: IMMDeviceEnumerator = CoCreateInstance(&MMDeviceEnumerator, None, CLSCTX_ALL)?;+ let device = devices.GetDefaultAudioEndpoint(eRender, eConsole)?;+ let client: IAudioClient = device.Activate(CLSCTX_ALL, None)?;+ // Windows converts the rate and channels to the device's own.+ client.Initialize(+ AUDCLNT_SHAREMODE_SHARED,+ AUDCLNT_STREAMFLAGS_EVENTCALLBACK+ | AUDCLNT_STREAMFLAGS_AUTOCONVERTPCM+ | AUDCLNT_STREAMFLAGS_SRC_DEFAULT_QUALITY,+ (DEVICE_BUFFER.as_nanos() / 100) as i64,+ 0,+ &format,+ None,+ )?;+ client.SetEventHandle(event)?;+ let size = client.GetBufferSize()?;+ let output: IAudioRenderClient = client.GetService()?;+ let mut jitter = Jitter::new(rate, channels);+ client.Start()?;+ tracing::debug!(rate, channels, frames = size, "playing a phone's sound");+ let result = loop {+ // Woken when the device wants sound (or after a while regardless,+ // to notice the end).+ let _ = WaitForSingleObject(event, 200);+ // Everything that came in.+ let mut gone = false;+ loop {+ match packets.recv_timeout(Duration::ZERO) {+ Ok(data) => jitter.push(&data),+ Err(RecvTimeoutError::Timeout) => break,+ Err(RecvTimeoutError::Disconnected) => {+ gone = true;+ break;+ }+ }+ }+ if gone {+ break Ok(());+ }+ let free = match client.GetCurrentPadding() {+ Ok(padding) => size - padding,+ Err(e) => break Err(e),+ };+ if free == 0 {+ continue;+ }+ let buffer = match output.GetBuffer(free) {+ Ok(buffer) => buffer,+ Err(e) => break Err(e),+ };+ let out = std::slice::from_raw_parts_mut(buffer, free as usize * usize::from(block));+ jitter.fill(out);+ if let Err(e) = output.ReleaseBuffer(free, 0) {+ break Err(e);+ }+ };+ let _ = client.Stop();+ tracing::debug!(+ stalls = jitter.stalls,+ dropped_ms = jitter.trimmed / jitter.per_ms.max(1),+ "the phone's sound stopped"+ );+ result+ }+}++/// An auto-reset event, closed when dropped.+struct Event(HANDLE);++impl Event {+ fn new() -> windows::core::Result<Event> {+ // SAFETY: creates an unnamed event with default security.+ unsafe { CreateEventW(None, false, false, None) }.map(Event)+ }+}++impl Drop for Event {+ fn drop(&mut self) {+ // SAFETY: the handle came from CreateEventW and is closed once.+ unsafe {+ let _ = CloseHandle(self.0);+ }+ }+}++#[cfg(test)]+mod tests {+ use super::*;++ /// 48 kHz stereo: 192 bytes a millisecond.+ const MS: usize = 192;++ #[test]+ fn plays_once_enough_has_come() {+ let mut jitter = Jitter::new(48_000, 2);+ let mut out = vec![9u8; 10 * MS];+ jitter.push(&vec![1; 30 * MS]);+ jitter.fill(&mut out);+ assert!(out.iter().all(|&b| b == 0), "silence while gathering");+ jitter.push(&vec![1; 10 * MS]);+ jitter.fill(&mut out);+ assert!(out.iter().all(|&b| b == 1), "then the sound");+ }++ #[test]+ fn a_stall_is_silence_then_gathers_again() {+ let mut jitter = Jitter::new(48_000, 2);+ jitter.push(&vec![1; 45 * MS]);+ let mut out = vec![9u8; 50 * MS];+ jitter.fill(&mut out);+ assert!(out[..45 * MS].iter().all(|&b| b == 1));+ assert!(out[45 * MS..].iter().all(|&b| b == 0), "silence when dry");+ jitter.push(&vec![2; 10 * MS]);+ let mut out = vec![9u8; 5 * MS];+ jitter.fill(&mut out);+ assert!(out.iter().all(|&b| b == 0), "gathers before playing on");+ }++ #[test]+ fn a_pile_up_drops_the_oldest_whole_frames() {+ let mut jitter = Jitter::new(48_000, 2);+ jitter.push(&vec![1; 150 * MS]);+ jitter.push(&vec![2; 60 * MS]);+ assert_eq!(jitter.bytes.len(), TRIM_TO_MS as usize * MS);+ assert_eq!(jitter.bytes.len() % 4, 0, "whole frames");+ assert!(jitter.bytes.iter().all(|&b| b == 2), "the newest stays");+ }+}diff --git a/desktop/app/src/win/mod.rs b/desktop/app/src/win/mod.rsindex f68e1dc..f3d0e6e 100644--- a/desktop/app/src/win/mod.rs+++ b/desktop/app/src/win/mod.rs@@ -5,6 +5,7 @@#![allow(unsafe_code)]+pub mod audio_out;pub mod autostart;pub mod clipboard;pub mod doctor;diff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.mdindex 1867871..ac955ac 100644--- a/docs/protocol/capabilities.md+++ b/docs/protocol/capabilities.md@@ -59,6 +59,7 @@ capability is missing.| `call.show` | Shows a phone's calls ([calls](calls.md)) | app update || `sms.show` | Shows a phone's text messages ([messages](sms.md)) | app update || `mirror.view` | Shows a phone's screen ([mirroring](mirror.md)) | app update |+| `mirror.listen` | Plays a mirrored phone's sound ([mirroring](mirror.md)) | app update || `files.browse` | Lets a paired phone browse and fetch its files | app update || `input.inject` | Accepts pointer and keyboard input (touchpad, air mouse, voice typing) | app update || `deck.actions` | Runs Deck actions | app update |diff --git a/docs/protocol/mirror.md b/docs/protocol/mirror.mdindex 6dd6c81..61c2fd7 100644--- a/docs/protocol/mirror.md+++ b/docs/protocol/mirror.md@@ -4,8 +4,9 @@> allowed until v1. License: CC BY 4.0.A PC shows a phone's screen. The phone's user agrees each time (Android-asks); the phone then streams its screen as H.264 on a stream of its own.-The PC can also use the phone with its mouse and keyboard (§4).+asks); the phone then streams its screen as H.264 on a stream of its own,+and its sound, when the PC asks, on another (§5). The PC can also use the+phone with its mouse and keyboard (§4).## 1. Capabilities@@ -14,6 +15,9 @@ The PC can also use the phone with its mouse and keyboard (§4).| `mirror.capture` | phone | Shares its screen when its user agrees || `mirror.view` | PC | Shows a phone's screen || `mirror.input` | phone | Takes the PC's mouse and keyboard while mirrored (Assist: an accessibility service the user turns on; Elevated: injected input) |+| `mirror.audio.playback` | phone | Shares the sound of apps that allow it while mirrored (Android 10+) |+| `mirror.audio` | phone | Shares all its sound while mirrored (planned, Elevated) |+| `mirror.listen` | PC | Plays a mirrored phone's sound |Both devices must allow it: the `mirroring` device toggle, on by default,on each side.@@ -23,7 +27,7 @@ on each side.On the control stream.```-t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint }+t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool }t = "mirror.stop" id = nt = "mirror.keyframe"t = "ok" re = n@@ -31,9 +35,11 @@ t = "ok" re = n- `mirror.start` (PC → phone): show me your screen, its longer side at most`max_size` pixels, at most `fps` frames a second, aiming for `bitrate`- bits a second. The phone answers `ok` once it has asked its user; video- follows only if the user agrees. Errors: `DENIED` (mirroring is off for- the PC), `UNSUPPORTED` (no `mirror.capture`).+ bits a second, and with `audio` (default false), its sound too. The+ phone answers `ok` once it has asked its user; video follows only if the+ user agrees, and sound only if the phone can (it may share the screen+ without it). Errors: `DENIED` (mirroring is off for the PC),+ `UNSUPPORTED` (no `mirror.capture`).- `mirror.stop` (either way): stop. The PC stops reading the video stream;the phone stops sharing and finishes it.- `mirror.keyframe` (PC → phone): the decoder lost its place; send a@@ -93,7 +99,29 @@ bottom), so they hold at any size and rotation.The Windows app maps the left button to a finger, right-click to Back,middle-click to Home and Ctrl+V to typing the PC's clipboard.-## 5. Latency over completeness+## 5. Sound++When the PC asked for it, the phone opens a second stream with the header+`{ svc: "mirror", op: "audio", v: 1 }`, with packets as in §3:++- **config**: the sound's format, CBOR `{ codec: "pcm_s16le", rate: uint,+ channels: uint }`: 16-bit little-endian PCM, channels interleaved, 8 to+ 96 kHz, mono or stereo. First, and again if it changes.+- **frame**: a few milliseconds of sound (Android sends 10 ms of 48 kHz+ stereo, 1,920 bytes), whole sample frames only.++A PC that doesn't offer `mirror.listen` resets the stream. The stream ends+with the video: when either side stops mirroring, or the connection drops.++On Android, the sound is what apps play (media, games, and apps that don't+say otherwise), captured with the screen share's consent and the+permission to record audio, which the phone asks for the first time a PC+wants the sound. Apps that don't allow playback capture, and calls, aren't+heard. Sharing all sound (`mirror.audio`) is planned for Elevated.++The phone's own speaker keeps playing; the PC can mute its copy.++## 6. Latency over completenessThe stream is reliable, so a slow network would otherwise build up delay.Instead, the phone keeps only a few packets waiting to go out: when that's@@ -102,8 +130,14 @@ its encoder for one. The PC likewise decodes everything it gets but showsonly the newest picture when it's behind, and asks for a keyframe when apicture doesn't decode.-## 6. Privacy+Sound works the same way without keyframes: a packet that doesn't fit is+dropped on its own. The PC plays from a short buffer (about 40 ms),+plays silence and refills it when the network stalls, and drops the oldest+sound when more than 200 ms piles up, so sound stays in step with the+picture.++## 7. Privacy-The screen is never recorded or logged by either side. Android shows that+The screen and sound are never recorded or logged by either side. Android shows thatthe screen is being shared (in the status bar), and the user can stop itthere, from the phone's notification, or from the PC.