Skip to content

Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.

Control media across PC and phone

Beef4ccdf20c85e
diff --git a/README.md b/README.md
index e177a52..bf00770 100644
--- a/README.md
+++ b/README.md
@@ -84,6 +84,10 @@ place and tested; the user-facing features of Phase 1 come next.
phone, share them from any app or pick them in Nectarlink. They land in `Downloads\Nectarlink` on the PC and
`Download/Nectarlink` on the phone, with progress, cancelling, and transfers
that pick up where they left off after a dropped connection.
+- **Media** both ways: what plays on the phone shows on the PC, in the app and
+ in Windows' own media flyout (so the keyboard's media keys work too), and
+ what plays on the PC shows in the phone's media controls, with artwork and
+ a seek bar.
- **Capability matrix**: both apps compute which features work for a pair of
devices and what would enable the rest, so the interface never offers
something that can't work.
@@ -95,8 +99,7 @@ place and tested; the user-facing features of Phase 1 come next.
connection service, and an identity key protected by the Android Keystore.
- **Command-line client** for development and testing (`nectarlink`).
-**Next (Phase 1)**: media controls, notification filters and history, and a
-Windows installer. Mirroring, messages, calls and the other
+**Next (Phase 1)**: notification filters and history, and a Windows installer. Mirroring, messages, calls and the other
features in the plan follow in later phases.
## Screenshots
diff --git a/android/README.md b/android/README.md
index 5096a3e..834eeb0 100644
--- a/android/README.md
+++ b/android/README.md
@@ -45,6 +45,9 @@ sources; nothing generated is checked in except the theme tokens
- `files/`: opening files to send, publishing received files to
Download/Nectarlink (MediaStore on Android 10+), and the transfer
notifications with progress and Cancel.
+- `media/`: this phone's media sessions for PCs (`PhoneMedia`, with
+ notification access), and a PC's player in Android's media controls
+ (`PcMedia`, a media session and its notification).
- `service/ConnectionService`: a "connected device" foreground service that
keeps the phone reachable and holds the Wi-Fi multicast lock local
discovery needs.
diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xml
index d49bd3a..3c40e3e 100644
--- a/android/app/src/main/AndroidManifest.xml
+++ b/android/app/src/main/AndroidManifest.xml
@@ -86,6 +86,11 @@
android:name=".files.TransferNotifications$CancelReceiver"
android:exported="false" />
+ <!-- The PC media notification's buttons (Android 12 and earlier). -->
+ <receiver
+ android:name=".media.PcMedia$CommandReceiver"
+ android:exported="false" />
+
<!-- Opening received files on Android 8–9 (later versions use MediaStore),
and images from a PC on the clipboard. -->
<provider
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
index abe4650..91a3f02 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
@@ -8,6 +8,8 @@ import android.util.Log
import android.net.Uri
import app.nectarlink.android.BuildConfig
import app.nectarlink.android.clipboard.PhoneClipboard
+import app.nectarlink.android.media.PcMedia
+import app.nectarlink.android.media.PhoneMedia
import app.nectarlink.android.files.OutgoingFiles
import app.nectarlink.android.files.ReceivedFiles
import app.nectarlink.android.files.TransferNotifications
@@ -15,6 +17,7 @@ import app.nectarlink.android.R
import app.nectarlink.android.notifications.NotificationListener
import app.nectarlink.core.Event
import app.nectarlink.core.EventListener
+import app.nectarlink.core.Link
import app.nectarlink.core.NectarlinkException
import app.nectarlink.core.NectarlinkNode
import app.nectarlink.core.NodeOptions
@@ -48,7 +51,16 @@ import kotlinx.coroutines.withContext
class Core(context: Context, private val scope: CoroutineScope) : EventListener {
private val context = context.applicationContext
private val ringer = Ringer(this.context)
- private val platform = PhonePlatform(this.context, ringer)
+ /** What plays on this phone, for PCs (needs notification access). */
+ private val phoneMedia = PhoneMedia(this.context) { players ->
+ notificationOps.trySend { it.mediaChanged(players) }
+ }
+ private val platform = PhonePlatform(this.context, ringer, phoneMedia)
+
+ /** What plays on the PCs, in Android's media controls. */
+ val pcMedia = PcMedia(this.context) { pc, player, action, position ->
+ command { it.mediaCommand(pc, player, action, position) }
+ }
/**
* Notification changes, in order: a removal must never overtake the post
* it removes. Runs once the node is up.
@@ -128,6 +140,7 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
*/
fun notificationAccessChanged(granted: Boolean, showing: List<Notification>) {
_state.update { it.copy(notificationAccess = granted) }
+ scope.launch(Dispatchers.Main) { if (granted) phoneMedia.start() else phoneMedia.stop() }
notificationOps.trySend { node ->
node.updatePower(PowerLevel.BASIC, capabilities(granted))
node.notificationsReset(showing)
@@ -158,7 +171,15 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
override fun onEvent(event: Event) {
_state.update { it.reduce(event) }
- if (event is Event.Transfer) onTransfer(event.transfer)
+ when (event) {
+ is Event.Transfer -> onTransfer(event.transfer)
+ is Event.MediaChanged -> pcMedia.update(event.id, _state.value.nameOf(event.id).orEmpty(), event.players)
+ // A PC that's gone isn't playing for this phone anymore; it
+ // sends what plays when it's back.
+ is Event.LinkChanged -> if (event.link is Link.Offline) pcMedia.update(event.id, "", emptyList())
+ is Event.DeviceRemoved -> pcMedia.update(event.id, "", emptyList())
+ else -> {}
+ }
}
// ---- Files ----
@@ -253,7 +274,7 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
/** What this phone offers PCs (docs/protocol/capabilities.md). */
private fun capabilities(notificationAccess: Boolean): List<String> =
- CLIPBOARD_CAPABILITIES + if (notificationAccess) NOTIFICATION_CAPABILITIES else emptyList()
+ CLIPBOARD_CAPABILITIES + if (notificationAccess) NOTIFICATION_CAPABILITIES + MEDIA_CAPABILITIES else emptyList()
// ---- Pairing ----
@@ -352,6 +373,8 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
const val TAG = "Nectarlink"
/** Offered while notification access is granted (docs/protocol/capabilities.md). */
val NOTIFICATION_CAPABILITIES = listOf("notify.mirror", "notify.reply")
+ /** Sharing this phone's players, which also needs notification access. */
+ val MEDIA_CAPABILITIES = listOf("media.control")
/** Accepting the PC's clipboard, and sending this one when asked. */
val CLIPBOARD_CAPABILITIES = listOf("clip.write", "clip.share")
}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/CoreState.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/CoreState.kt
index df98a14..37f844b 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/CoreState.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/CoreState.kt
@@ -102,6 +102,8 @@ data class CoreState(
// Android shows its own "copied" confirmation.
is Event.ClipboardReceived -> this
is Event.Transfer -> copy(transfers = withTransfer(event.transfer))
+ // Shown in Android's media controls (see media/PcMedia).
+ is Event.MediaChanged -> this
}
private fun withTransfer(transfer: Transfer): List<Transfer> {
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
index db5805e..ca7afa1 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
@@ -10,6 +10,8 @@ import android.os.Build
import android.os.Bundle
import app.nectarlink.android.clipboard.PhoneClipboard
import app.nectarlink.android.notifications.NotificationListener
+import app.nectarlink.android.media.PhoneMedia
+import app.nectarlink.core.MediaAction
import app.nectarlink.core.NotificationFailure
import app.nectarlink.core.Platform
@@ -17,7 +19,11 @@ import app.nectarlink.core.Platform
* What the core asks of the phone: ring it, and act on its notifications
* for a PC (dismiss, reply, run an action).
*/
-internal class PhonePlatform(context: Context, private val ringer: Ringer) : Platform {
+internal class PhonePlatform(
+ context: Context,
+ private val ringer: Ringer,
+ private val media: PhoneMedia,
+) : Platform {
private val context = context.applicationContext
override fun startRinging() = ringer.startRinging()
@@ -29,6 +35,9 @@ internal class PhonePlatform(context: Context, private val ringer: Ringer) : Pla
override fun setClipboardImage(mime: String, bytes: ByteArray): Boolean =
PhoneClipboard.writeImage(context, mime, bytes)
+ override fun mediaCommand(player: String, action: MediaAction, position: ULong?) =
+ media.command(player, action, position)
+
override fun dismissNotification(key: String) {
val listener = NotificationListener.instance ?: throw NotificationFailure.Unsupported()
listener.dismiss(key)
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/media/PcMedia.kt b/android/app/src/main/kotlin/app/nectarlink/android/media/PcMedia.kt
new file mode 100644
index 0000000..c5336b9
--- /dev/null
+++ b/android/app/src/main/kotlin/app/nectarlink/android/media/PcMedia.kt
@@ -0,0 +1,211 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+package app.nectarlink.android.media
+
+import android.Manifest
+import android.app.Notification
+import android.app.NotificationChannel
+import android.app.NotificationManager
+import android.app.PendingIntent
+import android.content.BroadcastReceiver
+import android.content.Context
+import android.content.Intent
+import android.content.pm.PackageManager
+import android.graphics.Bitmap
+import android.graphics.BitmapFactory
+import android.graphics.drawable.Icon
+import android.media.MediaMetadata
+import android.media.session.MediaSession
+import android.media.session.PlaybackState
+import android.os.Handler
+import android.os.Looper
+import android.os.SystemClock
+import androidx.core.content.ContextCompat
+import app.nectarlink.android.NectarlinkApplication
+import app.nectarlink.android.R
+import app.nectarlink.android.ui.MainActivity
+import app.nectarlink.core.MediaAction
+import app.nectarlink.core.MediaPlayer
+
+/**
+ * What plays on a paired PC, in Android's own media controls (the
+ * notification shade and Quick Settings player): a media session that
+ * stands for the PC's player, and its notification. What the user presses
+ * goes to the PC through [send]; the PC's new state comes back as an update.
+ *
+ * Everything runs on the main thread.
+ */
+class PcMedia(
+ context: Context,
+ private val send: (pc: String, player: String, action: MediaAction, position: ULong?) -> Unit,
+) {
+ private val context = context.applicationContext
+ private val main = Handler(Looper.getMainLooper())
+
+ /** Players per PC, in the PC's order, and the PCs' names. */
+ private val players = linkedMapOf<String, List<MediaPlayer>>()
+ private val names = mutableMapOf<String, String>()
+ /** Decoded artwork by PC and key (artwork arrives once per key). */
+ private val art = object : LinkedHashMap<String, Bitmap>(8, 0.75f, true) {
+ override fun removeEldestEntry(eldest: MutableMap.MutableEntry<String, Bitmap>) = size > 8
+ }
+
+ private var session: MediaSession? = null
+ private var shown: Pair<String, MediaPlayer>? = null
+
+ private val callback = object : MediaSession.Callback() {
+ override fun onPlay() = command(MediaAction.PLAY)
+ override fun onPause() = command(MediaAction.PAUSE)
+ override fun onSkipToNext() = command(MediaAction.NEXT)
+ override fun onSkipToPrevious() = command(MediaAction.PREVIOUS)
+ override fun onSeekTo(pos: Long) = command(MediaAction.SEEK, pos.coerceAtLeast(0).toULong())
+ }
+
+ /** A PC's players changed (empty: nothing plays there, or it's offline). */
+ fun update(pc: String, pcName: String, list: List<MediaPlayer>) = main.post {
+ names[pc] = pcName
+ for (player in list) {
+ val key = player.artKey ?: continue
+ val bytes = player.art ?: continue
+ BitmapFactory.decodeByteArray(bytes, 0, bytes.size)?.let { art["$pc/$key"] = it }
+ }
+ if (list.isEmpty()) players.remove(pc) else players[pc] = list
+ render()
+ }
+
+ private fun command(action: MediaAction, position: ULong? = null) {
+ val (pc, player) = shown ?: return
+ send(pc, player.id, action, position)
+ }
+
+ private fun render() {
+ val first = players.entries.firstNotNullOfOrNull { (pc, list) -> list.firstOrNull()?.let { pc to it } }
+ if (first == null) return hide()
+ val (pc, player) = first
+ shown = first
+ val session = session ?: MediaSession(context, "Nectarlink PC media").also {
+ it.setCallback(callback, main)
+ it.setSessionActivity(openApp())
+ session = it
+ }
+ val picture = player.artKey?.let { art["$pc/$it"] }
+ val where = context.getString(R.string.media_on, player.app, names[pc].orEmpty())
+ session.setMetadata(
+ MediaMetadata.Builder()
+ .putString(MediaMetadata.METADATA_KEY_TITLE, player.title ?: player.app)
+ .putString(MediaMetadata.METADATA_KEY_ARTIST, player.artist ?: where)
+ .putString(MediaMetadata.METADATA_KEY_ALBUM, player.album)
+ .putString(MediaMetadata.METADATA_KEY_DISPLAY_SUBTITLE, where)
+ .apply { player.duration?.let { putLong(MediaMetadata.METADATA_KEY_DURATION, it.toLong()) } }
+ .apply { picture?.let { putBitmap(MediaMetadata.METADATA_KEY_ALBUM_ART, it) } }
+ .build(),
+ )
+ var actions = 0L
+ if (MediaAction.PLAY in player.actions) actions = actions or PlaybackState.ACTION_PLAY
+ if (MediaAction.PAUSE in player.actions) actions = actions or PlaybackState.ACTION_PAUSE
+ if (MediaAction.PLAY in player.actions && MediaAction.PAUSE in player.actions) {
+ actions = actions or PlaybackState.ACTION_PLAY_PAUSE
+ }
+ if (MediaAction.NEXT in player.actions) actions = actions or PlaybackState.ACTION_SKIP_TO_NEXT
+ if (MediaAction.PREVIOUS in player.actions) actions = actions or PlaybackState.ACTION_SKIP_TO_PREVIOUS
+ if (MediaAction.SEEK in player.actions) actions = actions or PlaybackState.ACTION_SEEK_TO
+ session.setPlaybackState(
+ PlaybackState.Builder()
+ .setActions(actions)
+ .setState(
+ if (player.playing) PlaybackState.STATE_PLAYING else PlaybackState.STATE_PAUSED,
+ player.position?.toLong() ?: PlaybackState.PLAYBACK_POSITION_UNKNOWN,
+ if (player.playing) 1f else 0f,
+ SystemClock.elapsedRealtime(),
+ )
+ .build(),
+ )
+ session.isActive = true
+ post(notification(session, player, where, picture))
+ }
+
+ private fun hide() {
+ shown = null
+ session?.run {
+ isActive = false
+ release()
+ }
+ session = null
+ context.getSystemService(NotificationManager::class.java).cancel(NOTIFICATION_TAG, NOTIFICATION_ID)
+ }
+
+ private fun notification(session: MediaSession, player: MediaPlayer, where: String, picture: Bitmap?): Notification {
+ val buttons = buildList {
+ if (MediaAction.PREVIOUS in player.actions) add(button(R.drawable.ic_skip_previous, R.string.media_previous, MediaAction.PREVIOUS))
+ if (player.playing && MediaAction.PAUSE in player.actions) add(button(R.drawable.ic_pause, R.string.media_pause, MediaAction.PAUSE))
+ if (!player.playing && MediaAction.PLAY in player.actions) add(button(R.drawable.ic_play, R.string.media_play, MediaAction.PLAY))
+ if (MediaAction.NEXT in player.actions) add(button(R.drawable.ic_skip_next, R.string.media_next, MediaAction.NEXT))
+ }
+ return Notification.Builder(context, CHANNEL)
+ .setSmallIcon(R.drawable.ic_notification)
+ .setContentTitle(player.title ?: player.app)
+ .setContentText(listOfNotNull(player.artist, where).joinToString(" · "))
+ .setLargeIcon(picture)
+ .setContentIntent(openApp())
+ .setOnlyAlertOnce(true)
+ .setShowWhen(false)
+ .setVisibility(Notification.VISIBILITY_PUBLIC)
+ .setCategory(Notification.CATEGORY_TRANSPORT)
+ .apply { buttons.forEach(::addAction) }
+ .setStyle(
+ Notification.MediaStyle()
+ .setMediaSession(session.sessionToken)
+ .setShowActionsInCompactView(*buttons.indices.take(3).toIntArray()),
+ )
+ .build()
+ }
+
+ private fun button(icon: Int, label: Int, action: MediaAction): Notification.Action {
+ val intent = Intent(context, CommandReceiver::class.java).putExtra(EXTRA_ACTION, action.name)
+ val pending = PendingIntent.getBroadcast(
+ context,
+ action.ordinal,
+ intent,
+ PendingIntent.FLAG_IMMUTABLE or PendingIntent.FLAG_UPDATE_CURRENT,
+ )
+ return Notification.Action.Builder(Icon.createWithResource(context, icon), context.getString(label), pending).build()
+ }
+
+ private fun openApp(): PendingIntent = PendingIntent.getActivity(
+ context,
+ 0,
+ Intent(context, MainActivity::class.java).addFlags(Intent.FLAG_ACTIVITY_NEW_TASK),
+ PendingIntent.FLAG_IMMUTABLE,
+ )
+
+ private fun post(notification: Notification) {
+ if (ContextCompat.checkSelfPermission(context, Manifest.permission.POST_NOTIFICATIONS) != PackageManager.PERMISSION_GRANTED) {
+ return
+ }
+ context.getSystemService(NotificationManager::class.java).notify(NOTIFICATION_TAG, NOTIFICATION_ID, notification)
+ }
+
+ /** The notification's buttons on Android 12 and earlier (later ones use the session). */
+ class CommandReceiver : BroadcastReceiver() {
+ override fun onReceive(context: Context, intent: Intent) {
+ val action = intent.getStringExtra(EXTRA_ACTION)?.let { name -> MediaAction.entries.find { it.name == name } } ?: return
+ (context.applicationContext as NectarlinkApplication).core.pcMedia.pressed(action)
+ }
+ }
+
+ /** A button in the notification. */
+ internal fun pressed(action: MediaAction) = main.post { command(action) }
+
+ companion object {
+ private const val CHANNEL = "pc_media"
+ private const val NOTIFICATION_TAG = "pc_media"
+ private const val NOTIFICATION_ID = 1
+ private const val EXTRA_ACTION = "action"
+
+ fun createChannel(context: Context) {
+ context.getSystemService(NotificationManager::class.java).createNotificationChannel(
+ NotificationChannel(CHANNEL, context.getString(R.string.channel_pc_media), NotificationManager.IMPORTANCE_LOW)
+ .apply { setShowBadge(false) },
+ )
+ }
+ }
+}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/media/PhoneMedia.kt b/android/app/src/main/kotlin/app/nectarlink/android/media/PhoneMedia.kt
new file mode 100644
index 0000000..87d6af1
--- /dev/null
+++ b/android/app/src/main/kotlin/app/nectarlink/android/media/PhoneMedia.kt
@@ -0,0 +1,214 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+package app.nectarlink.android.media
+
+import android.content.Context
+import android.graphics.Bitmap
+import android.media.MediaMetadata
+import android.media.session.MediaController
+import android.media.session.MediaSessionManager
+import android.media.session.PlaybackState
+import android.os.Handler
+import android.os.Looper
+import android.os.SystemClock
+import android.util.Log
+import androidx.core.graphics.scale
+import app.nectarlink.android.notifications.NotificationListener
+import app.nectarlink.core.MediaAction
+import app.nectarlink.core.MediaFailure
+import app.nectarlink.core.MediaPlayer
+import java.io.ByteArrayOutputStream
+
+/**
+ * What plays on this phone, for paired PCs (docs/protocol/media.md): the
+ * apps' media sessions, read with notification access, and the PCs'
+ * commands for them.
+ *
+ * Runs on the main thread. Nectarlink's own session (a PC's media, see
+ * [PcMedia]) is left out, so a PC never gets its own music back.
+ */
+class PhoneMedia(context: Context, private val onChange: (List<MediaPlayer>) -> Unit) {
+ private val context = context.applicationContext
+ private val manager = context.getSystemService(MediaSessionManager::class.java)
+ private val main = Handler(Looper.getMainLooper())
+
+ /** Watched sessions, with their callbacks. */
+ private val watched = mutableMapOf<MediaController, MediaController.Callback>()
+ /** The players last sent, by ID, for commands. */
+ @Volatile
+ private var controllers: Map<String, MediaController> = emptyMap()
+ private val art = ArtCache()
+ private var running = false
+
+ private val sessionsChanged = MediaSessionManager.OnActiveSessionsChangedListener { list ->
+ watch(list.orEmpty())
+ }
+ private val publish = Runnable { publishNow() }
+
+ /** Starts watching; needs notification access. */
+ fun start() {
+ if (running) return
+ val component = NotificationListener.component(context)
+ try {
+ manager.addOnActiveSessionsChangedListener(sessionsChanged, component, main)
+ watch(manager.getActiveSessions(component))
+ running = true
+ } catch (e: SecurityException) {
+ Log.w(TAG, "no access to media sessions", e)
+ }
+ }
+
+ fun stop() {
+ if (!running) return
+ running = false
+ manager.removeOnActiveSessionsChangedListener(sessionsChanged)
+ watched.forEach { (controller, callback) -> controller.unregisterCallback(callback) }
+ watched.clear()
+ controllers = emptyMap()
+ main.removeCallbacks(publish)
+ onChange(emptyList())
+ }
+
+ private fun watch(sessions: List<MediaController>) {
+ val current = sessions.filter { it.packageName != context.packageName }
+ watched.keys.filter { old -> current.none { it.sessionToken == old.sessionToken } }.forEach { gone ->
+ watched.remove(gone)?.let(gone::unregisterCallback)
+ }
+ for (controller in current) {
+ if (watched.keys.any { it.sessionToken == controller.sessionToken }) continue
+ val callback = object : MediaController.Callback() {
+ override fun onMetadataChanged(metadata: MediaMetadata?) = schedule()
+ override fun onPlaybackStateChanged(state: PlaybackState?) = schedule()
+ override fun onSessionDestroyed() = schedule()
+ }
+ controller.registerCallback(callback, main)
+ watched[controller] = callback
+ }
+ schedule()
+ }
+
+ /** Changes come in bursts (metadata, then state, then artwork). */
+ private fun schedule() {
+ main.removeCallbacks(publish)
+ main.postDelayed(publish, SETTLE_MS)
+ }
+
+ private fun publishNow() {
+ val ids = mutableMapOf<String, MediaController>()
+ val players = watched.keys
+ .mapNotNull { controller -> read(controller)?.let { controller to it } }
+ .sortedWith(
+ compareByDescending<Pair<MediaController, Reading>> { it.second.playing }
+ .thenByDescending { it.second.updated },
+ )
+ .take(MAX_PLAYERS)
+ .map { (controller, reading) ->
+ // One app may have more than one session.
+ var id = controller.packageName
+ var n = 2
+ while (id in ids) id = "${controller.packageName}#${n++}"
+ ids[id] = controller
+ reading.player.copy(id = id)
+ }
+ controllers = ids
+ onChange(players)
+ }
+
+ private class Reading(val player: MediaPlayer, val playing: Boolean, val updated: Long)
+
+ private fun read(controller: MediaController): Reading? {
+ val metadata = controller.metadata ?: return null
+ val title = metadata.text(MediaMetadata.METADATA_KEY_TITLE)
+ ?: metadata.text(MediaMetadata.METADATA_KEY_DISPLAY_TITLE)
+ ?: return null
+ val state = controller.playbackState
+ val playing = state?.state == PlaybackState.STATE_PLAYING ||
+ state?.state == PlaybackState.STATE_BUFFERING
+ val duration = metadata.getLong(MediaMetadata.METADATA_KEY_DURATION).takeIf { it > 0 }
+ val position = state?.let { s ->
+ val moved = if (playing && s.lastPositionUpdateTime > 0) {
+ ((SystemClock.elapsedRealtime() - s.lastPositionUpdateTime) * s.playbackSpeed).toLong()
+ } else {
+ 0L
+ }
+ (s.position + moved).coerceAtLeast(0).let { if (duration != null) it.coerceAtMost(duration) else it }
+ }
+ val bits = state?.actions ?: 0L
+ val actions = buildList {
+ if (bits and (PlaybackState.ACTION_PLAY or PlaybackState.ACTION_PLAY_PAUSE) != 0L) add(MediaAction.PLAY)
+ if (bits and (PlaybackState.ACTION_PAUSE or PlaybackState.ACTION_PLAY_PAUSE) != 0L) add(MediaAction.PAUSE)
+ if (bits and PlaybackState.ACTION_SKIP_TO_NEXT != 0L) add(MediaAction.NEXT)
+ if (bits and PlaybackState.ACTION_SKIP_TO_PREVIOUS != 0L) add(MediaAction.PREVIOUS)
+ if (bits and PlaybackState.ACTION_SEEK_TO != 0L && duration != null) add(MediaAction.SEEK)
+ }
+ val picture = metadata.getBitmap(MediaMetadata.METADATA_KEY_ART)
+ ?: metadata.getBitmap(MediaMetadata.METADATA_KEY_ALBUM_ART)
+ ?: metadata.getBitmap(MediaMetadata.METADATA_KEY_DISPLAY_ICON)
+ val (artKey, artBytes) = picture?.let(art::get) ?: (null to null)
+ val player = MediaPlayer(
+ id = controller.packageName,
+ app = appName(controller.packageName),
+ title = title,
+ artist = metadata.text(MediaMetadata.METADATA_KEY_ARTIST)
+ ?: metadata.text(MediaMetadata.METADATA_KEY_ALBUM_ARTIST)
+ ?: metadata.text(MediaMetadata.METADATA_KEY_DISPLAY_SUBTITLE),
+ album = metadata.text(MediaMetadata.METADATA_KEY_ALBUM),
+ playing = playing,
+ duration = duration?.toULong(),
+ position = position?.toULong(),
+ actions = actions,
+ artKey = artKey,
+ art = artBytes,
+ )
+ return Reading(player, playing, state?.lastPositionUpdateTime ?: 0)
+ }
+
+ private fun MediaMetadata.text(key: String): String? = getString(key)?.trim()?.takeIf { it.isNotEmpty() }
+
+ private fun appName(packageName: String): String = runCatching {
+ val pm = context.packageManager
+ pm.getApplicationLabel(pm.getApplicationInfo(packageName, 0)).toString()
+ }.getOrDefault(packageName)
+
+ /** A PC's command for one of the players last sent. */
+ fun command(player: String, action: MediaAction, position: ULong?) {
+ val controller = controllers[player] ?: throw MediaFailure.NotFound()
+ val controls = controller.transportControls
+ when (action) {
+ MediaAction.PLAY -> controls.play()
+ MediaAction.PAUSE -> controls.pause()
+ MediaAction.NEXT -> controls.skipToNext()
+ MediaAction.PREVIOUS -> controls.skipToPrevious()
+ MediaAction.SEEK -> controls.seekTo(position?.toLong() ?: throw MediaFailure.Unsupported())
+ }
+ }
+
+ /** Artwork as a small JPEG, made once per picture. */
+ private class ArtCache {
+ private val made = object : LinkedHashMap<String, Pair<String, ByteArray>>(8, 0.75f, true) {
+ override fun removeEldestEntry(eldest: MutableMap.MutableEntry<String, Pair<String, ByteArray>>) = size > 8
+ }
+
+ fun get(bitmap: Bitmap): Pair<String, ByteArray> {
+ val identity = "${System.identityHashCode(bitmap)}:${bitmap.generationId}"
+ made[identity]?.let { return it }
+ val scale = minOf(1f, ART_SIZE.toFloat() / maxOf(bitmap.width, bitmap.height))
+ val small = if (scale < 1f) {
+ bitmap.scale((bitmap.width * scale).toInt().coerceAtLeast(1), (bitmap.height * scale).toInt().coerceAtLeast(1))
+ } else {
+ bitmap
+ }
+ val jpeg = ByteArrayOutputStream().also { small.compress(Bitmap.CompressFormat.JPEG, 88, it) }.toByteArray()
+ if (small !== bitmap) small.recycle()
+ val key = jpeg.contentHashCode().toUInt().toString(16) + "-" + jpeg.size.toString(16)
+ return (key to jpeg).also { made[identity] = it }
+ }
+ }
+
+ private companion object {
+ const val TAG = "PhoneMedia"
+ const val SETTLE_MS = 200L
+ const val MAX_PLAYERS = 4
+ /** Artwork's longest side, in pixels. */
+ const val ART_SIZE = 360
+ }
+}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/service/ConnectionService.kt b/android/app/src/main/kotlin/app/nectarlink/android/service/ConnectionService.kt
index d20c52a..0d3d4a5 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/service/ConnectionService.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/service/ConnectionService.kt
@@ -37,6 +37,7 @@ class ConnectionService : LifecycleService() {
super.onCreate()
createChannels(this)
app.nectarlink.android.files.TransferNotifications.createChannels(this)
+ app.nectarlink.android.media.PcMedia.createChannel(this)
val core = (application as NectarlinkApplication).core
startInForeground(core.state.value)
multicastLock = getSystemService(WifiManager::class.java)
diff --git a/android/app/src/main/res/drawable/ic_pause.xml b/android/app/src/main/res/drawable/ic_pause.xml
new file mode 100644
index 0000000..4e1117b
--- /dev/null
+++ b/android/app/src/main/res/drawable/ic_pause.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
+<vector xmlns:android="http://schemas.android.com/apk/res/android"
+ android:width="24dp" android:height="24dp" android:viewportWidth="24" android:viewportHeight="24">
+ <path android:fillColor="#FFFFFFFF" android:pathData="M7,5h3.5v14H7zM13.5,5H17v14h-3.5z" />
+</vector>
diff --git a/android/app/src/main/res/drawable/ic_play.xml b/android/app/src/main/res/drawable/ic_play.xml
new file mode 100644
index 0000000..58cc10b
--- /dev/null
+++ b/android/app/src/main/res/drawable/ic_play.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
+<vector xmlns:android="http://schemas.android.com/apk/res/android"
+ android:width="24dp" android:height="24dp" android:viewportWidth="24" android:viewportHeight="24">
+ <path android:fillColor="#FFFFFFFF" android:pathData="M8,5.14v13.72a1,1 0,0 0,1.5 0.86l11.2,-6.86a1,1 0,0 0,0 -1.72L9.5,4.28A1,1 0,0 0,8 5.14z" />
+</vector>
diff --git a/android/app/src/main/res/drawable/ic_skip_next.xml b/android/app/src/main/res/drawable/ic_skip_next.xml
new file mode 100644
index 0000000..67550b0
--- /dev/null
+++ b/android/app/src/main/res/drawable/ic_skip_next.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
+<vector xmlns:android="http://schemas.android.com/apk/res/android"
+ android:width="24dp" android:height="24dp" android:viewportWidth="24" android:viewportHeight="24">
+ <path android:fillColor="#FFFFFFFF" android:pathData="M6,6.5v11a1,1 0,0 0,1.55 0.83l8.2,-5.5a1,1 0,0 0,0 -1.66l-8.2,-5.5A1,1 0,0 0,6 6.5zM16.5,6h2v12h-2z" />
+</vector>
diff --git a/android/app/src/main/res/drawable/ic_skip_previous.xml b/android/app/src/main/res/drawable/ic_skip_previous.xml
new file mode 100644
index 0000000..28e3645
--- /dev/null
+++ b/android/app/src/main/res/drawable/ic_skip_previous.xml
@@ -0,0 +1,6 @@
+<?xml version="1.0" encoding="utf-8"?>
+<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
+<vector xmlns:android="http://schemas.android.com/apk/res/android"
+ android:width="24dp" android:height="24dp" android:viewportWidth="24" android:viewportHeight="24">
+ <path android:fillColor="#FFFFFFFF" android:pathData="M18,6.5v11a1,1 0,0 1,-1.55 0.83l-8.2,-5.5a1,1 0,0 1,0 -1.66l8.2,-5.5A1,1 0,0 1,18 6.5zM5.5,6h2v12h-2z" />
+</vector>
diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml
index 535600b..0c8e4a2 100644
--- a/android/app/src/main/res/values/strings.xml
+++ b/android/app/src/main/res/values/strings.xml
@@ -59,6 +59,12 @@
<string name="clip_share_label">Send to PC</string>
<string name="channel_transfers">File transfers</string>
<string name="channel_received">Received files</string>
+ <string name="channel_pc_media">Media on your PC</string>
+ <string name="media_on">%1$s on %2$s</string>
+ <string name="media_play">Play</string>
+ <string name="media_pause">Pause</string>
+ <string name="media_next">Next</string>
+ <string name="media_previous">Previous</string>
<string name="action_send_files">Send files</string>
<string name="transfers_title">Transfers</string>
<string name="transfer_sending">Sending to %1$s</string>
diff --git a/core/nectarlink-cli/src/main.rs b/core/nectarlink-cli/src/main.rs
index f6df376..ae404a8 100644
--- a/core/nectarlink-cli/src/main.rs
+++ b/core/nectarlink-cli/src/main.rs
@@ -10,8 +10,9 @@ use anyhow::{Context, Result, bail};
use clap::{Parser, Subcommand, ValueEnum};
use nectarlink_core::{
Battery, ConnectionPath, DeviceId, DeviceInfo, DeviceKind, Direction, FeatureState, FileSource,
- LinkState, Node, NodeConfig, NodeEvent, Notification, NotificationAction, NotificationError,
- OutgoingFile, PairedDevice, PairingEvent, Platform, PowerLevel, TransferState,
+ LinkState, MediaAction, MediaError, MediaPlayer, Node, NodeConfig, NodeEvent, Notification,
+ NotificationAction, NotificationError, OutgoingFile, PairedDevice, PairingEvent, Platform, PowerLevel,
+ TransferState,
features::{Effort, FEATURES, Role, UnsupportedReason, Upgrade, UpgradeAction},
};
use tokio::sync::broadcast::error::RecvError;
@@ -140,6 +141,35 @@ enum Command {
},
/// Show a notification on paired PCs as if this were a phone (use with
/// --as-phone), then stay online to show what the PC does with it.
+ /// Control what plays on a paired device: play, pause, next, previous,
+ /// or seek with --position.
+ Media {
+ device: String,
+ #[arg(value_enum)]
+ action: MediaCommandArg,
+ /// The player (`run` prints them); default: the device's first.
+ #[arg(long)]
+ player: Option<String>,
+ /// Where to seek to, in seconds.
+ #[arg(long, required_if_eq("action", "seek"))]
+ position: Option<u64>,
+ },
+ /// Pretend to play a song and share it with paired devices, which can
+ /// control it; stays online until Ctrl+C.
+ Play {
+ title: String,
+ #[arg(long, default_value = "Test Artist")]
+ artist: String,
+ /// App name shown with it.
+ #[arg(long, default_value = "Music")]
+ app: String,
+ /// Length, in seconds.
+ #[arg(long, default_value_t = 215)]
+ length: u64,
+ /// Artwork (JPEG or PNG).
+ #[arg(long)]
+ art: Option<PathBuf>,
+ },
Notify {
title: String,
text: String,
@@ -162,6 +192,31 @@ enum Power {
Elevated,
}
+#[derive(Debug, Clone, Copy, PartialEq, Eq, ValueEnum)]
+enum MediaCommandArg {
+ Play,
+ Pause,
+ Next,
+ Previous,
+ Seek,
+}
+
+impl From<MediaCommandArg> for MediaAction {
+ fn from(a: MediaCommandArg) -> Self {
+ match a {
+ MediaCommandArg::Play => MediaAction::Play,
+ MediaCommandArg::Pause => MediaAction::Pause,
+ MediaCommandArg::Next => MediaAction::Next,
+ MediaCommandArg::Previous => MediaAction::Previous,
+ MediaCommandArg::Seek => MediaAction::Seek,
+ }
+ }
+}
+
+/// Media commands from paired devices, for `play`.
+static MEDIA_COMMANDS: std::sync::OnceLock<tokio::sync::mpsc::UnboundedSender<(MediaAction, Option<u64>)>> =
+ std::sync::OnceLock::new();
+
#[derive(Debug, Clone, Copy, ValueEnum)]
enum OnOff {
On,
@@ -190,6 +245,23 @@ impl Platform for TerminalPlatform {
println!("Clipboard image from a paired device ({} bytes): {}", bytes.len(), path.display());
Ok(())
}
+ fn media_command(
+ &self,
+ player: &str,
+ action: MediaAction,
+ position: Option<u64>,
+ ) -> Result<(), MediaError> {
+ println!(
+ "A paired device asked {player} to {}{}",
+ action.as_str(),
+ match position {
+ Some(ms) => format!(" to {}", clock(ms)),
+ None => String::new(),
+ }
+ );
+ let sender = MEDIA_COMMANDS.get().ok_or(MediaError::NotFound)?;
+ sender.send((action, position)).map_err(|_| MediaError::NotFound)
+ }
fn dismiss_notification(&self, key: &str) -> Result<(), NotificationError> {
println!("The PC dismissed {key}");
Ok(())
@@ -387,6 +459,24 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {
println!("{name} is now {}", if on { "on" } else { "off" });
}
Command::Run => watch(node).await?,
+ Command::Media { device, action, player, position } => {
+ let id = resolve(node, device)?;
+ wait_until_online(node, id).await?;
+ let player = match player {
+ Some(player) => player.clone(),
+ None => first_player(node, id).await?,
+ };
+ node.media_command(id, player.clone(), (*action).into(), position.map(|s| s * 1000))
+ .await
+ .context("media command failed")?;
+ println!("Done ({player}).");
+ }
+ Command::Play { title, artist, app, length, art } => {
+ let mut offers = cli.offers.clone();
+ offers.push("media.control".into());
+ node.update_power(node_power(cli), offers).await;
+ play(node, title, artist, app, *length, art.as_deref()).await?;
+ }
Command::Notify { title, text, app, reply, silent } => {
if !cli.as_phone {
bail!("notifications come from phones: add --as-phone");
@@ -514,6 +604,87 @@ async fn send_files(node: &Node, device: DeviceId, paths: &[PathBuf]) -> Result<
}
}
+/// "3:07".
+fn clock(ms: u64) -> String {
+ let s = ms / 1000;
+ format!("{}:{:02}", s / 60, s % 60)
+}
+
+/// The first player a device reports (it reports them when connecting).
+async fn first_player(node: &Node, device: DeviceId) -> Result<String> {
+ let mut events = node.events();
+ // Ask again in case the report came before we listened.
+ node.set_device_toggle(device, "media", true)?;
+ let wait = async {
+ loop {
+ if let Ok(NodeEvent::MediaChanged { device: d, players }) = events.recv().await
+ && d == device
+ && let Some(p) = players.first()
+ {
+ return p.id.clone();
+ }
+ }
+ };
+ tokio::time::timeout(Duration::from_secs(10), wait).await.context("nothing is playing on that device")
+}
+
+/// A pretend player that follows the commands it gets.
+async fn play(
+ node: &Node,
+ title: &str,
+ artist: &str,
+ app: &str,
+ length: u64,
+ art: Option<&std::path::Path>,
+) -> Result<()> {
+ let (sender, mut commands) = tokio::sync::mpsc::unbounded_channel();
+ let _ = MEDIA_COMMANDS.set(sender);
+ let art = art.map(std::fs::read).transpose().context("can't read the artwork")?;
+ let art_key = art.as_ref().map(|a| format!("{:x}", a.len()));
+ let length = length * 1000;
+ let mut track = 1u32;
+ let mut position = 0u64;
+ let mut playing = true;
+ let mut since = std::time::Instant::now();
+ let player = |track: u32, position: u64, playing: bool| MediaPlayer {
+ id: "cli.player".into(),
+ app: app.into(),
+ title: Some(if track == 1 { title.to_owned() } else { format!("{title} ({track})") }),
+ artist: Some(artist.into()),
+ album: Some("Nectarlink Test Album".into()),
+ playing,
+ duration: Some(length),
+ position: Some(position),
+ actions: ["play", "pause", "next", "previous", "seek"].map(String::from).to_vec(),
+ art_key: art_key.clone(),
+ art: art.clone(),
+ };
+ node.media_changed(vec![player(track, position, playing)]).await;
+ println!("Playing \"{title}\". Paired devices can control it; Ctrl+C to stop.");
+ loop {
+ tokio::select! {
+ _ = tokio::signal::ctrl_c() => {
+ node.media_changed(Vec::new()).await;
+ return Ok(());
+ }
+ Some((action, to)) = commands.recv() => {
+ if playing {
+ position = (position + since.elapsed().as_millis() as u64).min(length);
+ }
+ since = std::time::Instant::now();
+ match action {
+ MediaAction::Play => playing = true,
+ MediaAction::Pause => playing = false,
+ MediaAction::Next => { track += 1; position = 0; }
+ MediaAction::Previous => { track = track.saturating_sub(1).max(1); position = 0; }
+ MediaAction::Seek => position = to.unwrap_or(position).min(length),
+ }
+ node.media_changed(vec![player(track, position, playing)]).await;
+ }
+ }
+ }
+}
+
async fn watch(node: &Node) -> Result<()> {
println!("Online as {}. Press Ctrl+C to stop.", node.device_id());
for d in node.paired_devices()? {
@@ -581,6 +752,25 @@ fn print_event(node: &Node, event: &NodeEvent) {
println!("{}: a notification went away", name(device))
}
NodeEvent::ClipboardReceived { device } => println!("{}: sent its clipboard", name(device)),
+ NodeEvent::MediaChanged { device, players } if players.is_empty() => {
+ println!("{}: nothing playing", name(device));
+ }
+ NodeEvent::MediaChanged { device, players } => {
+ for p in players {
+ println!(
+ "{}: {} {} – {} ({}, {} of {}{}) [player {}]",
+ name(device),
+ if p.playing { "▶" } else { "⏸" },
+ p.title.as_deref().unwrap_or("?"),
+ p.artist.as_deref().unwrap_or("?"),
+ p.app,
+ clock(p.position.unwrap_or(0)),
+ clock(p.duration.unwrap_or(0)),
+ if p.art.is_some() { ", with artwork" } else { "" },
+ p.id,
+ );
+ }
+ }
NodeEvent::Transfer(t) if t.direction == Direction::Incoming => match &t.state {
TransferState::Done { saved } => {
for path in saved {
diff --git a/core/nectarlink-core/src/config.rs b/core/nectarlink-core/src/config.rs
index ecae6d6..e033f04 100644
--- a/core/nectarlink-core/src/config.rs
+++ b/core/nectarlink-core/src/config.rs
@@ -3,6 +3,7 @@ use std::{fmt, path::PathBuf};
use nectarlink_protocol::messages::{DeviceInfo, PowerLevel};
+use crate::media::{MediaAction, MediaError};
use crate::{identity::KeyProtector, notifications::NotificationError};
/// Configuration for starting a [`Node`](crate::Node).
@@ -30,6 +31,9 @@ pub struct NodeConfig {
/// Where received files go once complete. `None`: a `received` folder
/// in `data_dir`.
pub downloads_dir: Option<PathBuf>,
+ /// Capabilities this device offers beyond the ones every build does
+ /// (docs/protocol/capabilities.md), e.g. `media.control`.
+ pub capabilities: Vec<String>,
}
impl NodeConfig {
@@ -44,6 +48,7 @@ impl NodeConfig {
port: 0,
key_protector: None,
downloads_dir: None,
+ capabilities: Vec::new(),
}
}
}
@@ -99,6 +104,17 @@ pub trait Platform: Send + Sync + 'static {
fn set_clipboard_image(&self, _mime: &str, _bytes: &[u8]) -> Result<(), String> {
Err("this device has no clipboard".into())
}
+
+ /// Run a command on one of this device's media players (a paired device
+ /// asked); `position` is set for [`MediaAction::Seek`].
+ fn media_command(
+ &self,
+ _player: &str,
+ _action: MediaAction,
+ _position: Option<u64>,
+ ) -> Result<(), MediaError> {
+ Err(MediaError::Unsupported)
+ }
}
/// A platform that does nothing; useful for tests and headless tools.
diff --git a/core/nectarlink-core/src/events.rs b/core/nectarlink-core/src/events.rs
index 97392da..fb4f20f 100644
--- a/core/nectarlink-core/src/events.rs
+++ b/core/nectarlink-core/src/events.rs
@@ -3,7 +3,7 @@
use nectarlink_protocol::{
DeviceId,
- messages::{Battery, DeviceInfo, Notification, PowerLevel},
+ messages::{Battery, DeviceInfo, MediaPlayer, Notification, PowerLevel},
};
use crate::{features::CapabilityMatrix, transfer::Transfer};
@@ -121,4 +121,11 @@ pub enum NodeEvent {
},
/// A file transfer started, progressed or finished.
Transfer(Transfer),
+ /// A device's media players changed (all of them; empty when nothing
+ /// plays there, or the user turned media off for it). Artwork is
+ /// attached when it's new for this session; keep it by `art_key`.
+ MediaChanged {
+ device: DeviceId,
+ players: Vec<MediaPlayer>,
+ },
}
diff --git a/core/nectarlink-core/src/features.rs b/core/nectarlink-core/src/features.rs
index c46f6f3..ebba517 100644
--- a/core/nectarlink-core/src/features.rs
+++ b/core/nectarlink-core/src/features.rs
@@ -330,6 +330,7 @@ pub const DEVICE_TOGGLES: &[(&str, bool)] = &[
("calls", true),
("clipboard", true),
("files", true),
+ ("media", true),
("photos", true),
("pc_actions", true),
// Browsing the PC's files from anywhere is opt-in.
@@ -656,13 +657,17 @@ pub const FEATURES: &[FeatureDef] = &[
FeatureDef {
id: "media.phone_control",
group: FeatureGroup::Media,
- requires: &[phone("media.control", Unlock::Permission(Permission::NotificationAccess))],
+ requires: &[
+ phone("media.control", Unlock::Permission(Permission::NotificationAccess)),
+ desktop("media.remote", UPDATE),
+ DeviceToggle("media"),
+ ],
partial: None,
},
FeatureDef {
id: "media.pc_control",
group: FeatureGroup::Media,
- requires: &[desktop("media.control", UPDATE)],
+ requires: &[desktop("media.control", UPDATE), phone("media.remote", UPDATE), DeviceToggle("media")],
partial: None,
},
// Camera
@@ -732,10 +737,18 @@ mod tests {
"mirror.capture",
"mirror.audio.playback",
"media.control",
+ "media.remote",
"camera.stream",
];
- const PC: &[&str] =
- &["pc.power", "files.transfer", "files.browse", "media.control", "input.inject", "deck.actions"];
+ const PC: &[&str] = &[
+ "pc.power",
+ "files.transfer",
+ "files.browse",
+ "media.control",
+ "media.remote",
+ "input.inject",
+ "deck.actions",
+ ];
fn pc() -> DeviceFacts {
facts(DeviceKind::Laptop, "windows", "10.0.26200", PowerLevel::NotApplicable, PC)
diff --git a/core/nectarlink-core/src/lib.rs b/core/nectarlink-core/src/lib.rs
index 21f2998..c7b647c 100644
--- a/core/nectarlink-core/src/lib.rs
+++ b/core/nectarlink-core/src/lib.rs
@@ -12,6 +12,7 @@ mod error;
mod events;
pub mod features;
mod identity;
+mod media;
mod node;
mod notifications;
mod pairing;
@@ -26,13 +27,15 @@ pub use events::{
};
pub use features::{CapabilityMatrix, FeatureState};
pub use identity::{KeyProtector, PlainKeyProtector, default_protector};
+pub use media::{MediaAction, MediaError};
pub use nectarlink_protocol::messages::clip::{
IMAGE_TYPES as CLIP_IMAGE_TYPES, MAX_IMAGE_BYTES as CLIP_MAX_IMAGE_BYTES,
};
pub use nectarlink_protocol::{
DeviceId,
messages::{
- Battery, CLIP_MAX_BYTES, DeviceInfo, DeviceKind, Notification, NotificationAction, PowerLevel,
+ Battery, CLIP_MAX_BYTES, DeviceInfo, DeviceKind, MediaPlayer, Notification, NotificationAction,
+ PowerLevel,
},
pairing::PairingUri,
};
diff --git a/core/nectarlink-core/src/media.rs b/core/nectarlink-core/src/media.rs
new file mode 100644
index 0000000..66e12fb
--- /dev/null
+++ b/core/nectarlink-core/src/media.rs
@@ -0,0 +1,257 @@
+// SPDX-License-Identifier: MPL-2.0
+//! Media control between devices (docs/protocol/media.md).
+//!
+//! A device that shares its players (`media.control`) keeps every allowed
+//! device that shows them (`media.remote`) up to date: the full list when a
+//! session starts or the user changes what that device may see, then after
+//! each change. Artwork goes once per key and session. The other side turns
+//! that into [`NodeEvent::MediaChanged`] and sends the user's commands back.
+
+use std::{
+ collections::HashMap,
+ sync::{Arc, Mutex},
+};
+
+use nectarlink_protocol::{
+ DeviceId, Envelope, ErrorCode,
+ messages::{MediaCommand, MediaPlayer, MediaState, media_actions, media_limits::PLAYERS, types},
+};
+
+use crate::{Result, events::NodeEvent, node::Shared, session::Session};
+
+/// The device toggle that allows media for a device.
+pub(crate) const TOGGLE: &str = "media";
+/// Offered by devices that show and control other devices' players.
+const REMOTE: &str = "media.remote";
+/// Offered by devices that share their own players.
+const SHARES: &str = "media.control";
+
+/// A command for a player.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum MediaAction {
+ Play,
+ Pause,
+ Next,
+ Previous,
+ /// To the position given with it.
+ Seek,
+}
+
+impl MediaAction {
+ pub fn as_str(self) -> &'static str {
+ match self {
+ MediaAction::Play => media_actions::PLAY,
+ MediaAction::Pause => media_actions::PAUSE,
+ MediaAction::Next => media_actions::NEXT,
+ MediaAction::Previous => media_actions::PREVIOUS,
+ MediaAction::Seek => media_actions::SEEK,
+ }
+ }
+
+ pub fn parse(action: &str) -> Option<MediaAction> {
+ Some(match action {
+ media_actions::PLAY => MediaAction::Play,
+ media_actions::PAUSE => MediaAction::Pause,
+ media_actions::NEXT => MediaAction::Next,
+ media_actions::PREVIOUS => MediaAction::Previous,
+ media_actions::SEEK => MediaAction::Seek,
+ _ => return None,
+ })
+ }
+}
+
+/// Why a media command didn't run.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub enum MediaError {
+ /// The player is gone.
+ NotFound,
+ /// The player (or this device) can't do that.
+ Unsupported,
+ /// Anything else; the string is for logs.
+ Failed(String),
+}
+
+/// This device's players, as the app last reported them, with their
+/// artwork kept apart for sending once per session.
+#[derive(Default)]
+pub(crate) struct Players {
+ inner: Mutex<PlayersState>,
+}
+
+#[derive(Default)]
+struct PlayersState {
+ /// Set once the app reports players: only then does this device send.
+ active: bool,
+ players: Vec<MediaPlayer>,
+ art: HashMap<String, Vec<u8>>,
+}
+
+impl Players {
+ fn lock(&self) -> std::sync::MutexGuard<'_, PlayersState> {
+ self.inner.lock().unwrap_or_else(|e| e.into_inner())
+ }
+
+ fn set(&self, players: Vec<MediaPlayer>) {
+ let mut state = self.lock();
+ state.active = true;
+ state.players.clear();
+ let mut art = HashMap::new();
+ for mut player in players.into_iter().filter_map(MediaPlayer::sanitized).take(PLAYERS) {
+ if let (Some(key), Some(bytes)) = (&player.art_key, player.art.take()) {
+ art.insert(key.clone(), bytes);
+ } else if let Some(key) = &player.art_key
+ && let Some(bytes) = state.art.remove(key)
+ {
+ // Artwork is sent with every report; keep what's still in use.
+ art.insert(key.clone(), bytes);
+ }
+ state.players.push(player);
+ }
+ state.art = art;
+ }
+
+ /// The players for one session, artwork attached where it hasn't had it.
+ fn for_session(&self, session: &Session) -> MediaState {
+ let state = self.lock();
+ let mut sent = session.sent_art();
+ let players = state
+ .players
+ .iter()
+ .map(|player| {
+ let mut player = player.clone();
+ if let Some(key) = &player.art_key
+ && let Some(art) = state.art.get(key)
+ && sent.insert(key.clone())
+ {
+ player.art = Some(art.clone());
+ }
+ player
+ })
+ .collect();
+ MediaState { players }
+ }
+}
+
+impl Shared {
+ fn shows_media(&self, peer: &DeviceId) -> bool {
+ self.store.get_peer(peer).ok().flatten().is_some_and(|p| p.caps.contains(REMOTE))
+ }
+
+ /// Sends this device's players to one device: all of them when the
+ /// user allows it, none (which clears that device) when not.
+ pub(crate) async fn send_media_state(&self, session: &Arc<Session>) {
+ if !self.players.lock().active || !self.shows_media(&session.peer) {
+ return;
+ }
+ let state = if self.toggle_on(&session.peer, TOGGLE) {
+ self.players.for_session(session)
+ } else {
+ MediaState { players: Vec::new() }
+ };
+ if let Ok(env) = Envelope::new(types::MEDIA_STATE, &state) {
+ let _ = session.send(env).await;
+ }
+ }
+
+ pub(crate) async fn media_changed(&self, players: Vec<MediaPlayer>) {
+ self.players.set(players);
+ for session in self.live_sessions() {
+ if self.toggle_on(&session.peer, TOGGLE) {
+ self.send_media_state(&session).await;
+ }
+ }
+ }
+
+ /// Brings both sides in line after the user allowed or stopped media
+ /// for a device.
+ pub(crate) fn media_toggled(self: &Arc<Self>, peer: DeviceId, enabled: bool) {
+ let Some(session) = self.session(&peer) else {
+ if !enabled {
+ self.emit(NodeEvent::MediaChanged { device: peer, players: Vec::new() });
+ }
+ return;
+ };
+ let shared = self.clone();
+ self.runtime.spawn(async move {
+ // What this device shares: everything, or nothing.
+ shared.send_media_state(&session).await;
+ if enabled {
+ // What the other one shares: ask for it.
+ let _ = session.send(Envelope::empty(types::MEDIA_SYNC)).await;
+ } else {
+ shared.emit(NodeEvent::MediaChanged { device: peer, players: Vec::new() });
+ }
+ });
+ }
+}
+
+/// Handles a `media.*` message on a session. Returns false for other types.
+pub(crate) async fn handle(shared: &Arc<Shared>, session: &Arc<Session>, env: &Envelope) -> Result<bool> {
+ let peer = session.peer;
+ match env.t.as_str() {
+ types::MEDIA_STATE => {
+ if shared.toggle_on(&peer, TOGGLE) {
+ let state: MediaState = env.body()?;
+ let players =
+ state.players.into_iter().filter_map(MediaPlayer::sanitized).take(PLAYERS).collect();
+ shared.emit(NodeEvent::MediaChanged { device: peer, players });
+ }
+ }
+ types::MEDIA_SYNC => shared.send_media_state(session).await,
+ types::MEDIA_COMMAND => {
+ let command: MediaCommand = env.body()?;
+ let reply = run_command(shared, &peer, command).await;
+ session.send(reply.reply_to(env.id)).await?;
+ }
+ _ => return Ok(false),
+ }
+ Ok(true)
+}
+
+async fn run_command(shared: &Arc<Shared>, peer: &DeviceId, command: MediaCommand) -> Envelope {
+ if !shared.local_capabilities().iter().any(|c| c == SHARES) {
+ return Envelope::error(ErrorCode::Unsupported, "media control is not available");
+ }
+ if !shared.toggle_on(peer, TOGGLE) {
+ return Envelope::error(ErrorCode::Denied, "media is off for this device");
+ }
+ let Some(action) = MediaAction::parse(&command.action) else {
+ return Envelope::error(ErrorCode::Unsupported, "unknown media action");
+ };
+ if action == MediaAction::Seek && command.position.is_none() {
+ return Envelope::error(ErrorCode::BadMessage, "seek needs a position");
+ }
+ let platform = shared.platform.clone();
+ let MediaCommand { player, position, .. } = command;
+ let ran = tokio::task::spawn_blocking(move || platform.media_command(&player, action, position))
+ .await
+ .unwrap_or_else(|e| Err(MediaError::Failed(e.to_string())));
+ match ran {
+ Ok(()) => Envelope::empty(types::OK),
+ Err(MediaError::NotFound) => Envelope::error(ErrorCode::NotFound, "no such player"),
+ Err(MediaError::Unsupported) => Envelope::error(ErrorCode::Unsupported, "the player can't do that"),
+ Err(MediaError::Failed(reason)) => {
+ tracing::warn!(reason, "media command failed");
+ Envelope::error(ErrorCode::Internal, "failed")
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn actions_round_trip() {
+ for action in [
+ MediaAction::Play,
+ MediaAction::Pause,
+ MediaAction::Next,
+ MediaAction::Previous,
+ MediaAction::Seek,
+ ] {
+ assert_eq!(MediaAction::parse(action.as_str()), Some(action));
+ }
+ assert_eq!(MediaAction::parse("rewind"), None);
+ }
+}
diff --git a/core/nectarlink-core/src/node.rs b/core/nectarlink-core/src/node.rs
index 0018be1..3d439a3 100644
--- a/core/nectarlink-core/src/node.rs
+++ b/core/nectarlink-core/src/node.rs
@@ -20,8 +20,8 @@ use iroh_mdns_address_lookup::{DiscoveryEvent, MdnsAddressLookup};
use nectarlink_protocol::{
ALPN_PAIR, ALPN_SESSION, DeviceId, Envelope, ErrorCode,
messages::{
- Battery, ClipSet, DeviceInfo, HelloUpdate, Notification, NotifyAction, NotifyKey, PowerLevel, Ring,
- types,
+ Battery, ClipSet, DeviceInfo, HelloUpdate, MediaCommand, MediaPlayer, Notification, NotifyAction,
+ NotifyKey, PowerLevel, Ring, types,
},
pairing::PairingUri,
};
@@ -54,7 +54,7 @@ const IDLE_TIMEOUT: Duration = Duration::from_secs(8);
/// Capabilities every build offers.
const BASE_CAPABILITIES: &[&str] =
- &["core.ping", "device.battery", "device.ring", "files.transfer", "clip.image"];
+ &["core.ping", "device.battery", "device.ring", "files.transfer", "clip.image", "media.remote"];
/// This device's mutable description, sent to peers.
#[derive(Debug, Clone)]
@@ -103,13 +103,15 @@ pub(crate) struct Shared {
memory: MemoryLookup,
/// This device's notifications, when it mirrors them (phones).
pub notifications: Feed,
+ /// This device's media players (docs/protocol/media.md).
+ pub players: crate::media::Players,
pub data_dir: std::path::PathBuf,
/// Where received files go.
pub downloads_dir: std::path::PathBuf,
/// Running transfers, to cancel them.
transfers: Mutex<HashMap<String, CancellationToken>>,
/// The node's runtime, for work started from non-async callers.
- runtime: tokio::runtime::Handle,
+ pub(crate) runtime: tokio::runtime::Handle,
}
impl std::fmt::Debug for Shared {
@@ -337,6 +339,7 @@ impl Shared {
let _ = s.send(env).await;
}
shared.send_notification_snapshot(&s).await;
+ shared.send_media_state(&s).await;
});
Some(session)
}
@@ -575,7 +578,7 @@ impl Node {
app_version: config.app_version.clone(),
device: config.device.clone(),
power: config.power,
- extra_capabilities: Vec::new(),
+ extra_capabilities: config.capabilities.clone(),
battery: None,
}),
pairing: PairingState::default(),
@@ -589,6 +592,7 @@ impl Node {
away_mode: config.away_mode,
memory,
notifications: Feed::default(),
+ players: Default::default(),
data_dir: config.data_dir.clone(),
downloads_dir: config.downloads_dir.clone().unwrap_or_else(|| config.data_dir.join("received")),
transfers: Mutex::new(HashMap::new()),
@@ -797,6 +801,9 @@ impl Node {
if toggle == notifications::TOGGLE {
self.notifications_toggled(peer, enabled);
}
+ if toggle == crate::media::TOGGLE {
+ self.shared.media_toggled(peer, enabled);
+ }
Ok(())
}
@@ -869,6 +876,32 @@ impl Node {
crate::clipboard::send_image(&self.shared, &session, mime, bytes).await
}
+ // ---- Media (docs/protocol/media.md) ----
+
+ /// This device's media players changed (most relevant first); sent to
+ /// every connected device the user allows. Include each player's
+ /// artwork every time: the core sends it once per device and session.
+ pub async fn media_changed(&self, players: Vec<MediaPlayer>) {
+ self.shared.media_changed(players).await;
+ }
+
+ /// Runs a command on a paired device's media player.
+ pub async fn media_command(
+ &self,
+ peer: DeviceId,
+ player: String,
+ action: crate::MediaAction,
+ position: Option<u64>,
+ ) -> Result<()> {
+ if !self.shared.toggle_on(&peer, crate::media::TOGGLE) {
+ return Err(Error::Denied);
+ }
+ let command = MediaCommand { player, action: action.as_str().into(), position };
+ let env = Envelope::new(types::MEDIA_COMMAND, &command)?;
+ self.request(peer, env).await?.expect(types::OK)?;
+ Ok(())
+ }
+
// ---- Notifications (docs/protocol/notifications.md) ----
/// A notification appeared or changed on this phone; sent to every
diff --git a/core/nectarlink-core/src/session.rs b/core/nectarlink-core/src/session.rs
index d2107e1..26af288 100644
--- a/core/nectarlink-core/src/session.rs
+++ b/core/nectarlink-core/src/session.rs
@@ -114,6 +114,8 @@ pub(crate) struct Session {
rtt_ms: AtomicU32,
/// Apps whose notification icon this peer already got.
sent_icons: crate::notifications::SentIcons,
+ /// Artwork keys sent to this device (docs/protocol/media.md).
+ sent_art: crate::notifications::SentIcons,
pub(crate) cancel: CancellationToken,
}
@@ -154,6 +156,7 @@ impl Session {
next_id: AtomicU64::new(1),
rtt_ms: AtomicU32::new(initial_rtt),
sent_icons: Default::default(),
+ sent_art: Default::default(),
cancel: shared.cancel.child_token(),
});
@@ -204,6 +207,10 @@ impl Session {
self.sent_icons.lock().unwrap_or_else(|e| e.into_inner())
}
+ pub(crate) fn sent_art(&self) -> std::sync::MutexGuard<'_, std::collections::HashSet<String>> {
+ self.sent_art.lock().unwrap_or_else(|e| e.into_inner())
+ }
+
pub fn is_alive(&self) -> bool {
!self.cancel.is_cancelled() && self.conn.close_reason().is_none()
}
@@ -339,6 +346,7 @@ async fn handle(shared: &Arc<Shared>, session: &Arc<Session>, env: Envelope) ->
shared.forget_peer(&peer).await?;
}
t if t.starts_with("notify.") && crate::notifications::handle(shared, session, &env).await? => {}
+ t if t.starts_with("media.") && crate::media::handle(shared, session, &env).await? => {}
t if t.starts_with("clip.") && crate::clipboard::handle(shared, session, &env).await? => {}
other => {
if env.id.is_some() {
diff --git a/core/nectarlink-core/tests/two_devices.rs b/core/nectarlink-core/tests/two_devices.rs
index 6dc8935..b02e0b3 100644
--- a/core/nectarlink-core/tests/two_devices.rs
+++ b/core/nectarlink-core/tests/two_devices.rs
@@ -9,10 +9,10 @@ use std::{
};
use nectarlink_core::{
- Battery, DeviceInfo, DeviceKind, Direction, Error, FeatureState, FileSource, LinkState, Node, NodeConfig,
- NodeEvent, Notification, NotificationAction, NotificationError, OutgoingFile, PairingEvent,
- PairingFailure, PairingUri, PlainKeyProtector, Platform, PowerLevel, Transfer, TransferFailure,
- TransferState,
+ Battery, DeviceInfo, DeviceKind, Direction, Error, FeatureState, FileSource, LinkState, MediaAction,
+ MediaError, MediaPlayer, Node, NodeConfig, NodeEvent, Notification, NotificationAction,
+ NotificationError, OutgoingFile, PairingEvent, PairingFailure, PairingUri, PlainKeyProtector, Platform,
+ PowerLevel, Transfer, TransferFailure, TransferState,
features::{Effort, Upgrade, UpgradeAction},
};
use tempfile::TempDir;
@@ -27,6 +27,7 @@ struct RecordingPlatform {
actions: Mutex<Vec<(String, String, Option<String>)>>,
clipboard: Mutex<Vec<String>>,
images: Mutex<Vec<(String, Vec<u8>)>>,
+ media: Mutex<Vec<(String, MediaAction, Option<u64>)>>,
}
impl Platform for RecordingPlatform {
@@ -60,6 +61,18 @@ impl Platform for RecordingPlatform {
self.images.lock().unwrap().push((mime.to_owned(), bytes.to_vec()));
Ok(())
}
+ fn media_command(
+ &self,
+ player: &str,
+ action: MediaAction,
+ position: Option<u64>,
+ ) -> Result<(), MediaError> {
+ if player == "gone" {
+ return Err(MediaError::NotFound);
+ }
+ self.media.lock().unwrap().push((player.to_owned(), action, position));
+ Ok(())
+ }
}
struct TestDevice {
@@ -95,6 +108,7 @@ async fn start_in(dir: TempDir, name: &'static str, kind: DeviceKind) -> TestDev
config.lan_discovery = false;
config.away_mode = false;
config.key_protector = Some(Arc::new(PlainKeyProtector));
+ config.capabilities = vec!["media.control".into()];
let platform = Arc::new(RecordingPlatform::default());
let node = Node::start(config, platform.clone()).await.expect("node starts");
let events = node.events();
@@ -640,6 +654,114 @@ async fn clipboard_images_go_both_ways_with_consent() {
.expect_err("still off");
}
+fn player(id: &str, title: &str, art: Option<(&str, Vec<u8>)>) -> MediaPlayer {
+ MediaPlayer {
+ id: id.into(),
+ app: "Music".into(),
+ title: Some(title.into()),
+ artist: Some("Artist".into()),
+ album: None,
+ playing: true,
+ duration: Some(200_000),
+ position: Some(12_000),
+ actions: vec!["play".into(), "pause".into(), "next".into(), "seek".into(), "bogus".into()],
+ art_key: art.as_ref().map(|(key, _)| (*key).into()),
+ art: art.map(|(_, bytes)| bytes),
+ }
+}
+
+fn media_of(device: nectarlink_core::DeviceId) -> impl FnMut(&NodeEvent) -> Option<Vec<MediaPlayer>> {
+ move |e| match e {
+ NodeEvent::MediaChanged { device: d, players } if *d == device => Some(players.clone()),
+ _ => None,
+ }
+}
+
+#[tokio::test(flavor = "multi_thread")]
+async fn media_is_shared_and_controlled_with_consent() {
+ let mut pc = device("Desktop", DeviceKind::Desktop).await;
+ let mut phone = device("Pixel", DeviceKind::Phone).await;
+ pair_qr(&mut pc, &mut phone).await;
+ let (pc_id, phone_id) = (pc.node.device_id(), phone.node.device_id());
+
+ // The phone plays something; the PC gets it with its artwork.
+ let art = data(30_000, 7);
+ phone.node.media_changed(vec![player("com.music", "Song", Some(("a1", art.clone())))]).await;
+ let players = wait_for(&mut pc, "media", media_of(phone_id)).await;
+ assert_eq!(players.len(), 1);
+ assert_eq!(players[0].title.as_deref(), Some("Song"));
+ assert_eq!(players[0].art.as_deref(), Some(&art[..]));
+ assert_eq!(players[0].actions, ["play", "pause", "next", "seek"], "unknown actions are dropped");
+
+ // Artwork goes once per session; a new picture goes again.
+ phone.node.media_changed(vec![player("com.music", "Song", Some(("a1", art.clone())))]).await;
+ let players = wait_for(&mut pc, "same art", media_of(phone_id)).await;
+ assert_eq!((players[0].art_key.as_deref(), players[0].art.as_ref()), (Some("a1"), None));
+ phone.node.media_changed(vec![player("com.music", "Next song", Some(("a2", vec![9; 10])))]).await;
+ let players = wait_for(&mut pc, "new art", media_of(phone_id)).await;
+ assert_eq!(players[0].art.as_deref(), Some(&[9; 10][..]));
+
+ // The PC controls it.
+ with_timeout("pause", pc.node.media_command(phone_id, "com.music".into(), MediaAction::Pause, None))
+ .await
+ .expect("paused");
+ with_timeout(
+ "seek",
+ pc.node.media_command(phone_id, "com.music".into(), MediaAction::Seek, Some(60_000)),
+ )
+ .await
+ .expect("sought");
+ assert_eq!(
+ *phone.platform.media.lock().unwrap(),
+ [
+ ("com.music".to_owned(), MediaAction::Pause, None),
+ ("com.music".to_owned(), MediaAction::Seek, Some(60_000))
+ ]
+ );
+ let gone = pc.node.media_command(phone_id, "gone".into(), MediaAction::Play, None).await;
+ assert!(matches!(gone, Err(Error::NotFound)), "{gone:?}");
+ let no_position = pc.node.media_command(phone_id, "com.music".into(), MediaAction::Seek, None).await;
+ assert!(no_position.is_err(), "{no_position:?}");
+
+ // The PC turns media off for the phone: it's cleared there at once,
+ // and the phone stops taking its commands only if the phone turns it off.
+ pc.node.set_device_toggle(phone_id, "media", false).unwrap();
+ assert!(wait_for(&mut pc, "cleared", media_of(phone_id)).await.is_empty());
+ let off = pc.node.media_command(phone_id, "com.music".into(), MediaAction::Play, None).await;
+ assert!(matches!(off, Err(Error::Denied)), "{off:?}");
+ // On again: the PC asks for what plays now.
+ pc.node.set_device_toggle(phone_id, "media", true).unwrap();
+ let players = wait_for(&mut pc, "synced", media_of(phone_id)).await;
+ assert_eq!(players[0].title.as_deref(), Some("Next song"));
+
+ // The phone turns it off for the PC: the PC is told nothing plays, and
+ // commands are refused.
+ phone.node.set_device_toggle(pc_id, "media", false).unwrap();
+ assert!(wait_for(&mut pc, "cleared by the phone", media_of(phone_id)).await.is_empty());
+ let denied = pc.node.media_command(phone_id, "com.music".into(), MediaAction::Play, None).await;
+ assert!(matches!(denied, Err(Error::Denied)), "{denied:?}");
+
+ // And the other way: the PC's own players reach the phone once the
+ // phone allows media again.
+ phone.node.set_device_toggle(pc_id, "media", true).unwrap();
+ pc.node.media_changed(vec![player("Spotify.exe", "On the PC", None)]).await;
+ let players = wait_for(&mut phone, "pc media", |e| match e {
+ NodeEvent::MediaChanged { device, players } if *device == pc_id && !players.is_empty() => {
+ Some(players.clone())
+ }
+ _ => None,
+ })
+ .await;
+ assert_eq!(players[0].title.as_deref(), Some("On the PC"));
+ with_timeout(
+ "phone controls the pc",
+ phone.node.media_command(pc_id, "Spotify.exe".into(), MediaAction::Next, None),
+ )
+ .await
+ .expect("next");
+ assert_eq!(pc.platform.media.lock().unwrap()[0], ("Spotify.exe".to_owned(), MediaAction::Next, None));
+}
+
/// Deterministic, incompressible-looking test data.
fn data(len: usize, seed: u64) -> Vec<u8> {
let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
diff --git a/core/nectarlink-ffi/src/lib.rs b/core/nectarlink-ffi/src/lib.rs
index 9ec5c5a..3facb82 100644
--- a/core/nectarlink-ffi/src/lib.rs
+++ b/core/nectarlink-ffi/src/lib.rs
@@ -165,6 +165,105 @@ pub struct Notification {
pub icon: Option<Vec<u8>>,
}
+/// Something playing (or paused) on a device (docs/protocol/media.md).
+#[derive(Clone, PartialEq, Eq, uniffi::Record)]
+pub struct MediaPlayer {
+ /// Stable while the player exists (Android: the app's package name).
+ pub id: String,
+ /// User-visible app name.
+ pub app: String,
+ pub title: Option<String>,
+ pub artist: Option<String>,
+ pub album: Option<String>,
+ pub playing: bool,
+ /// Milliseconds.
+ pub duration: Option<u64>,
+ /// Milliseconds, now; it moves on in real time while `playing`.
+ pub position: Option<u64>,
+ pub actions: Vec<MediaAction>,
+ /// Identifies the artwork: the same key, the same picture.
+ pub art_key: Option<String>,
+ /// JPEG or PNG. Pass it every time; the core sends it once per device
+ /// and connection. Arrives the same way: keep it by `art_key`.
+ pub art: Option<Vec<u8>>,
+}
+
+/// Never prints what's playing (protocol v0 §11).
+impl std::fmt::Debug for MediaPlayer {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("MediaPlayer").field("id", &self.id).finish_non_exhaustive()
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq, uniffi::Enum)]
+pub enum MediaAction {
+ Play,
+ Pause,
+ Next,
+ Previous,
+ Seek,
+}
+
+impl From<MediaAction> for core::MediaAction {
+ fn from(a: MediaAction) -> Self {
+ match a {
+ MediaAction::Play => core::MediaAction::Play,
+ MediaAction::Pause => core::MediaAction::Pause,
+ MediaAction::Next => core::MediaAction::Next,
+ MediaAction::Previous => core::MediaAction::Previous,
+ MediaAction::Seek => core::MediaAction::Seek,
+ }
+ }
+}
+
+impl From<core::MediaAction> for MediaAction {
+ fn from(a: core::MediaAction) -> Self {
+ match a {
+ core::MediaAction::Play => MediaAction::Play,
+ core::MediaAction::Pause => MediaAction::Pause,
+ core::MediaAction::Next => MediaAction::Next,
+ core::MediaAction::Previous => MediaAction::Previous,
+ core::MediaAction::Seek => MediaAction::Seek,
+ }
+ }
+}
+
+impl From<MediaPlayer> for core::MediaPlayer {
+ fn from(p: MediaPlayer) -> Self {
+ core::MediaPlayer {
+ id: p.id,
+ app: p.app,
+ title: p.title,
+ artist: p.artist,
+ album: p.album,
+ playing: p.playing,
+ duration: p.duration,
+ position: p.position,
+ actions: p.actions.into_iter().map(|a| core::MediaAction::from(a).as_str().to_owned()).collect(),
+ art_key: p.art_key,
+ art: p.art,
+ }
+ }
+}
+
+impl From<core::MediaPlayer> for MediaPlayer {
+ fn from(p: core::MediaPlayer) -> Self {
+ MediaPlayer {
+ id: p.id,
+ app: p.app,
+ title: p.title,
+ artist: p.artist,
+ album: p.album,
+ playing: p.playing,
+ duration: p.duration,
+ position: p.position,
+ actions: p.actions.iter().filter_map(|a| core::MediaAction::parse(a)).map(Into::into).collect(),
+ art_key: p.art_key,
+ art: p.art,
+ }
+ }
+}
+
/// Never prints content (protocol v0 §11).
impl std::fmt::Debug for Notification {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
@@ -308,6 +407,12 @@ pub enum Event {
Transfer {
transfer: Transfer,
},
+ /// A device's media players (all of them; empty when nothing plays
+ /// there, or media is off for it).
+ MediaChanged {
+ id: String,
+ players: Vec<MediaPlayer>,
+ },
}
#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, uniffi::Error)]
@@ -667,6 +772,10 @@ impl From<NodeEvent> for Event {
}
NodeEvent::ClipboardReceived { device } => Event::ClipboardReceived { id: device.to_string() },
NodeEvent::Transfer(t) => Event::Transfer { transfer: t.into() },
+ NodeEvent::MediaChanged { device, players } => Event::MediaChanged {
+ id: device.to_string(),
+ players: players.into_iter().map(Into::into).collect(),
+ },
}
}
}
@@ -707,6 +816,36 @@ impl From<NotificationFailure> for core::NotificationError {
}
}
+/// Why a media command didn't run.
+#[derive(Debug, Clone, PartialEq, Eq, thiserror::Error, uniffi::Error)]
+pub enum MediaFailure {
+ /// The player is gone.
+ #[error("no such player")]
+ NotFound,
+ /// The player can't do that (or media access was revoked).
+ #[error("not available")]
+ Unsupported,
+ /// `reason` is for logs.
+ #[error("failed: {reason}")]
+ Failed { reason: String },
+}
+
+impl From<uniffi::UnexpectedUniFFICallbackError> for MediaFailure {
+ fn from(e: uniffi::UnexpectedUniFFICallbackError) -> Self {
+ MediaFailure::Failed { reason: e.reason }
+ }
+}
+
+impl From<MediaFailure> for core::MediaError {
+ fn from(f: MediaFailure) -> Self {
+ match f {
+ MediaFailure::NotFound => core::MediaError::NotFound,
+ MediaFailure::Unsupported => core::MediaError::Unsupported,
+ MediaFailure::Failed { reason } => core::MediaError::Failed(reason),
+ }
+ }
+}
+
/// What the app does for the core: ring the phone, act on its
/// notifications.
#[uniffi::export(with_foreign)]
@@ -729,6 +868,14 @@ pub trait Platform: Send + Sync {
/// A PC sent this image (`image/png` or `image/jpeg`): put it on the
/// clipboard. False if that failed.
fn set_clipboard_image(&self, mime: String, bytes: Vec<u8>) -> bool;
+ /// A paired device asked one of this phone's media players to do
+ /// something; `position` (milliseconds) is set for seeking.
+ fn media_command(
+ &self,
+ player: String,
+ action: MediaAction,
+ position: Option<u64>,
+ ) -> Result<(), MediaFailure>;
}
/// Encrypts the device key at rest (Android: a Keystore key).
@@ -773,6 +920,14 @@ impl core::Platform for PlatformAdapter {
fn set_clipboard(&self, text: &str) -> Result<(), String> {
if self.0.set_clipboard(text.to_owned()) { Ok(()) } else { Err("the clipboard rejected it".into()) }
}
+ fn media_command(
+ &self,
+ player: &str,
+ action: core::MediaAction,
+ position: Option<u64>,
+ ) -> Result<(), core::MediaError> {
+ Ok(self.0.media_command(player.to_owned(), action.into(), position)?)
+ }
fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
if self.0.set_clipboard_image(mime.to_owned(), bytes.to_vec()) {
Ok(())
@@ -997,6 +1152,29 @@ impl NectarlinkNode {
self.run(async move { Ok(node.send_clipboard_image(id, mime, bytes).await?) }).await
}
+ // ---- Media ----
+
+ /// This phone's media players changed (most relevant first).
+ pub async fn media_changed(&self, players: Vec<MediaPlayer>) {
+ let node = self.node.clone();
+ self.run(async move { node.media_changed(players.into_iter().map(Into::into).collect()).await })
+ .await;
+ }
+
+ /// Runs a command on a paired device's media player; `position`
+ /// (milliseconds) for seeking.
+ pub async fn media_command(
+ &self,
+ id: String,
+ player: String,
+ action: MediaAction,
+ position: Option<u64>,
+ ) -> Result<()> {
+ let id = parse_id(&id)?;
+ let node = self.node.clone();
+ self.run(async move { Ok(node.media_command(id, player, action.into(), position).await?) }).await
+ }
+
// ---- Notifications ----
/// A notification appeared or changed on this phone.
diff --git a/core/nectarlink-protocol/src/messages.rs b/core/nectarlink-protocol/src/messages.rs
index 748ca30..41ad778 100644
--- a/core/nectarlink-protocol/src/messages.rs
+++ b/core/nectarlink-protocol/src/messages.rs
@@ -35,6 +35,9 @@ pub mod types {
pub const CLIP_SET: &str = "clip.set";
pub const NOTIFY_DISMISS: &str = "notify.dismiss";
pub const NOTIFY_ACTION: &str = "notify.action";
+ pub const MEDIA_STATE: &str = "media.state";
+ pub const MEDIA_SYNC: &str = "media.sync";
+ pub const MEDIA_COMMAND: &str = "media.command";
}
/// What kind of device this is.
@@ -317,6 +320,119 @@ impl std::fmt::Debug for NotifyAction {
}
}
+// ---- Media (docs/protocol/media.md) ----
+
+/// Limits from docs/protocol/media.md.
+pub mod media_limits {
+ pub const PLAYERS: usize = 8;
+ pub const ID_BYTES: usize = 256;
+ pub const TEXT_CHARS: usize = 256;
+ pub const ART_KEY_BYTES: usize = 64;
+ pub const ART_BYTES: usize = 256 * 1024;
+}
+
+/// What a player can be asked to do (`MediaPlayer::actions`,
+/// `MediaCommand::action`).
+pub mod media_actions {
+ pub const PLAY: &str = "play";
+ pub const PAUSE: &str = "pause";
+ pub const NEXT: &str = "next";
+ pub const PREVIOUS: &str = "previous";
+ pub const SEEK: &str = "seek";
+ pub const ALL: &[&str] = &[PLAY, PAUSE, NEXT, PREVIOUS, SEEK];
+}
+
+/// Something playing (or paused) on a device.
+#[derive(Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct MediaPlayer {
+ /// The device's ID for the player, stable while it exists (e.g. the
+ /// app's package name).
+ pub id: String,
+ /// User-visible app name.
+ pub app: String,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub title: Option<String>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub artist: Option<String>,
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub album: Option<String>,
+ #[serde(default)]
+ pub playing: bool,
+ /// Length, milliseconds.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub duration: Option<u64>,
+ /// Position when this was sent, milliseconds; it moves on in real time
+ /// while `playing`.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub position: Option<u64>,
+ /// Commands it takes, from [`media_actions`].
+ #[serde(default)]
+ pub actions: Vec<String>,
+ /// Identifies the artwork; the same key means the same picture.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub art_key: Option<String>,
+ /// The artwork (JPEG or PNG), sent once per key and session.
+ #[serde(default, skip_serializing_if = "Option::is_none", with = "serde_bytes")]
+ pub art: Option<Vec<u8>>,
+}
+
+/// Never prints what's playing (protocol v0 §11).
+impl std::fmt::Debug for MediaPlayer {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ f.debug_struct("MediaPlayer")
+ .field("id", &self.id)
+ .field("playing", &self.playing)
+ .field("actions", &self.actions)
+ .finish_non_exhaustive()
+ }
+}
+
+impl MediaPlayer {
+ /// Applies the spec's limits. `None` if it can't be shown (no ID, or
+ /// nothing to show it by).
+ pub fn sanitized(mut self) -> Option<MediaPlayer> {
+ use media_limits::*;
+ if self.id.is_empty() || self.id.len() > ID_BYTES {
+ return None;
+ }
+ truncate_chars(&mut self.app, TEXT_CHARS);
+ self.title = clean_text(self.title, TEXT_CHARS);
+ self.artist = clean_text(self.artist, TEXT_CHARS);
+ self.album = clean_text(self.album, TEXT_CHARS);
+ if self.title.is_none() && self.app.trim().is_empty() {
+ return None;
+ }
+ self.actions.retain(|a| media_actions::ALL.contains(&a.as_str()));
+ self.actions.dedup();
+ if let (Some(position), Some(duration)) = (self.position, self.duration) {
+ self.position = Some(position.min(duration));
+ }
+ if self.art_key.as_ref().is_some_and(|k| k.is_empty() || k.len() > ART_KEY_BYTES) {
+ self.art_key = None;
+ }
+ if self.art_key.is_none() || self.art.as_ref().is_some_and(|a| a.is_empty() || a.len() > ART_BYTES) {
+ self.art = None;
+ }
+ Some(self)
+ }
+}
+
+/// Body of `media.state`: every player, most relevant first.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct MediaState {
+ pub players: Vec<MediaPlayer>,
+}
+
+/// Body of `media.command`.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct MediaCommand {
+ pub player: String,
+ pub action: String,
+ /// For `seek`: where to, milliseconds.
+ #[serde(default, skip_serializing_if = "Option::is_none")]
+ pub position: Option<u64>,
+}
+
// ---- Files (docs/protocol/files.md) ----
/// The files service and its one operation, for stream headers.
diff --git a/desktop/app/Cargo.toml b/desktop/app/Cargo.toml
index 8f24735..a83fe5d 100644
--- a/desktop/app/Cargo.toml
+++ b/desktop/app/Cargo.toml
@@ -35,6 +35,10 @@ windows = { workspace = true, features = [
"Data_Xml_Dom",
"Foundation",
"Foundation_Collections",
+ "Media",
+ "Media_Control",
+ "Storage_Streams",
+ "ApplicationModel",
"UI_Notifications",
"Win32_Foundation",
"Win32_Graphics_Dwm",
@@ -49,6 +53,7 @@ windows = { workspace = true, features = [
"Win32_System_Power",
"Win32_System_Registry",
"Win32_System_Threading",
+ "Win32_System_WinRT",
"Win32_UI_Shell",
"Win32_UI_WindowsAndMessaging",
"Win32_NetworkManagement_IpHelper",
diff --git a/desktop/app/build.rs b/desktop/app/build.rs
index 8bed864..0f0f261 100644
--- a/desktop/app/build.rs
+++ b/desktop/app/build.rs
@@ -31,6 +31,7 @@ const QML: &[&str] = &[
"qml/components/LockChip.qml",
"qml/components/NavItem.qml",
"qml/components/NotificationItem.qml",
+ "qml/components/NowPlaying.qml",
"qml/components/QrCode.qml",
"qml/components/Segmented.qml",
"qml/components/Sheet.qml",
@@ -186,6 +187,7 @@ fn main() {
"src/bridge/native.rs",
"src/bridge/app.rs",
"src/bridge/devices.rs",
+ "src/bridge/media.rs",
"src/bridge/notifications.rs",
"src/bridge/pairing.rs",
"src/bridge/prefs.rs",
diff --git a/desktop/app/qml/Icons.qml b/desktop/app/qml/Icons.qml
index d062619..2da8666 100644
--- a/desktop/app/qml/Icons.qml
+++ b/desktop/app/qml/Icons.qml
@@ -15,6 +15,11 @@ QtObject {
readonly property string clipboard: "M9 3h6v4H9zM16 5h2a2 2 0 0 1 2 2v12a2 2 0 0 1-2 2H6a2 2 0 0 1-2-2V7a2 2 0 0 1 2-2h2"
readonly property string mirror: "M4 4h10a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2H4a2 2 0 0 1-2-2V6a2 2 0 0 1 2-2zM18.5 8h2a1.5 1.5 0 0 1 1.5 1.5v7a1.5 1.5 0 0 1-1.5 1.5h-2A1.5 1.5 0 0 1 17 16.5v-7A1.5 1.5 0 0 1 18.5 8zM6 18h6"
readonly property string send: "M22 2L11 13M22 2l-7 20-4-9-9-4z"
+ readonly property string play: "M8 5.5v13a1 1 0 0 0 1.5.86l10.5-6.5a1 1 0 0 0 0-1.72L9.5 4.64A1 1 0 0 0 8 5.5z"
+ readonly property string pause: "M9 5v14M15 5v14"
+ readonly property string skipNext: "M5 6.2v11.6a1 1 0 0 0 1.6.8l7.4-5.8a1 1 0 0 0 0-1.6L6.6 5.4a1 1 0 0 0-1.6.8zM19 5v14"
+ readonly property string skipPrevious: "M19 6.2v11.6a1 1 0 0 1-1.6.8L10 12.8a1 1 0 0 1 0-1.6l7.4-5.8a1 1 0 0 1 1.6.8zM5 5v14"
+ readonly property string music: "M9 18V5l12-2v13M9 18a3 3 0 1 1-6 0 3 3 0 0 1 6 0zM21 16a3 3 0 1 1-6 0 3 3 0 0 1 6 0z"
readonly property string ring: "M6 8a6 6 0 1 1 12 0c0 7 3 8 3 8H3s3-1 3-8M10 20a2 2 0 0 0 4 0M2 8c0-2 1-4 2-5M22 8c0-2-1-4-2-5"
readonly property string mic: "M12 2a3 3 0 0 1 3 3v6a3 3 0 0 1-6 0V5a3 3 0 0 1 3-3zM5 11a7 7 0 0 0 14 0M12 18v4"
readonly property string lock: "M7 11h10a2 2 0 0 1 2 2v6a2 2 0 0 1-2 2H7a2 2 0 0 1-2-2v-6a2 2 0 0 1 2-2zM8 11V8a4 4 0 0 1 8 0v3"
diff --git a/desktop/app/qml/components/NowPlaying.qml b/desktop/app/qml/components/NowPlaying.qml
new file mode 100644
index 0000000..9f0e27d
--- /dev/null
+++ b/desktop/app/qml/components/NowPlaying.qml
@@ -0,0 +1,201 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+import QtQuick
+import app.nectarlink
+
+// What plays on a phone: artwork, title and artist, a seek bar that moves
+// with the music, and the player's controls. Commands go to the phone;
+// what it plays next comes back as new state.
+Card {
+ id: card
+ required property string deviceId
+ required property string playerId
+ required property string app
+ required property string title
+ required property string artist
+ required property bool playing
+ required property real duration
+ required property real position
+ required property real positionAt
+ required property string art
+ required property bool canPlay
+ required property bool canPause
+ required property bool canNext
+ required property bool canPrevious
+ required property bool canSeek
+
+ // Time, ticking while it plays so the bar and clock move.
+ property real now: Date.now()
+ Timer {
+ interval: 250
+ repeat: true
+ running: card.visible && card.playing && card.duration > 0
+ onTriggered: card.now = Date.now()
+ onRunningChanged: card.now = Date.now()
+ }
+ onPositionAtChanged: now = Date.now()
+
+ readonly property real elapsed: position < 0 ? 0
+ : Math.min(duration > 0 ? duration : Infinity, position + (playing ? Math.max(0, now - positionAt) : 0))
+ // While dragging, the bar shows where the user is pointing.
+ property real dragTo: -1
+ readonly property real shown: dragTo >= 0 ? dragTo : elapsed
+
+ function send(action, to) { MediaList.command(deviceId, playerId, action, to === undefined ? 0 : to) }
+ function clock(ms) {
+ const s = Math.floor(ms / 1000)
+ const h = Math.floor(s / 3600)
+ const m = Math.floor((s % 3600) / 60)
+ const ss = String(s % 60).padStart(2, "0")
+ return h > 0 ? h + ":" + String(m).padStart(2, "0") + ":" + ss : m + ":" + ss
+ }
+
+ Column {
+ width: parent.width
+ spacing: 14
+
+ Txt { text: qsTr("Now playing"); role: "label"; muted: true }
+
+ Row {
+ width: parent.width
+ spacing: 14
+ Rectangle {
+ id: cover
+ width: 72; height: 72
+ radius: Theme.radiusMd
+ color: Theme.secondaryContainer
+ clip: true
+ Icon {
+ anchors.centerIn: parent
+ width: 28; height: 28
+ visible: artwork.status !== Image.Ready
+ path: Icons.music
+ color: Theme.secondaryContainerContent
+ }
+ Image {
+ id: artwork
+ anchors.fill: parent
+ source: card.art
+ fillMode: Image.PreserveAspectCrop
+ asynchronous: true
+ sourceSize: Qt.size(144, 144)
+ opacity: status === Image.Ready ? 1 : 0
+ Behavior on opacity { NumberAnimation { duration: Theme.fadeNormal } }
+ }
+ }
+ Column {
+ width: parent.width - cover.width - parent.spacing
+ anchors.verticalCenter: parent.verticalCenter
+ spacing: 3
+ Txt {
+ width: parent.width
+ role: "title"
+ text: card.title.length > 0 ? card.title : card.app
+ elide: Text.ElideRight
+ maximumLineCount: 2
+ wrapMode: Text.WordWrap
+ }
+ Txt {
+ width: parent.width
+ visible: card.artist.length > 0
+ role: "bodySmall"
+ muted: true
+ text: card.artist
+ elide: Text.ElideRight
+ }
+ Txt {
+ width: parent.width
+ role: "caption"
+ muted: true
+ text: card.app
+ elide: Text.ElideRight
+ }
+ }
+ }
+
+ // ---- Seek bar ----
+ Column {
+ width: parent.width
+ spacing: 4
+ visible: card.duration > 0
+ Item {
+ id: bar
+ width: parent.width
+ height: 16
+ readonly property real fraction: card.duration > 0 ? Math.max(0, Math.min(1, card.shown / card.duration)) : 0
+ Rectangle {
+ id: track
+ anchors.verticalCenter: parent.verticalCenter
+ width: parent.width
+ height: 4
+ radius: 2
+ color: Theme.surfaceContainerHighest
+ Rectangle {
+ width: parent.width * bar.fraction
+ height: parent.height
+ radius: parent.radius
+ color: Theme.primary
+ }
+ }
+ Rectangle {
+ visible: card.canSeek && (seek.containsMouse || seek.pressed)
+ x: bar.width * bar.fraction - width / 2
+ anchors.verticalCenter: parent.verticalCenter
+ width: 12; height: 12; radius: 6
+ color: Theme.primary
+ }
+ MouseArea {
+ id: seek
+ anchors.fill: parent
+ anchors.margins: -4
+ enabled: card.canSeek
+ hoverEnabled: true
+ cursorShape: card.canSeek ? Qt.PointingHandCursor : Qt.ArrowCursor
+ function at(x) { return Math.max(0, Math.min(1, (x - 4) / bar.width)) * card.duration }
+ onPressed: (mouse) => card.dragTo = at(mouse.x)
+ onPositionChanged: (mouse) => { if (pressed) card.dragTo = at(mouse.x) }
+ onReleased: (mouse) => {
+ card.send("seek", Math.round(at(mouse.x)))
+ card.dragTo = -1
+ }
+ onCanceled: card.dragTo = -1
+ }
+ }
+ Item {
+ width: parent.width
+ height: elapsedText.height
+ Txt { id: elapsedText; role: "caption"; muted: true; text: card.clock(card.shown) }
+ Txt { anchors.right: parent.right; role: "caption"; muted: true; text: card.clock(card.duration) }
+ }
+ }
+
+ // ---- Controls ----
+ Row {
+ anchors.horizontalCenter: parent.horizontalCenter
+ spacing: 18
+ IconButton {
+ anchors.verticalCenter: parent.verticalCenter
+ iconPath: Icons.skipPrevious
+ label: qsTr("Previous")
+ enabled: card.canPrevious
+ onClicked: card.send("previous")
+ }
+ IconButton {
+ id: playPause
+ anchors.verticalCenter: parent.verticalCenter
+ width: 48; height: 48
+ tonal: true
+ iconPath: card.playing ? Icons.pause : Icons.play
+ label: card.playing ? qsTr("Pause") : qsTr("Play")
+ enabled: card.playing ? card.canPause : card.canPlay
+ onClicked: card.send(card.playing ? "pause" : "play")
+ }
+ IconButton {
+ anchors.verticalCenter: parent.verticalCenter
+ iconPath: Icons.skipNext
+ label: qsTr("Next")
+ enabled: card.canNext
+ onClicked: card.send("next")
+ }
+ }
+ }
+}
diff --git a/desktop/app/qml/pages/HomePage.qml b/desktop/app/qml/pages/HomePage.qml
index 662ba05..cc3ca72 100644
--- a/desktop/app/qml/pages/HomePage.qml
+++ b/desktop/app/qml/pages/HomePage.qml
@@ -480,6 +480,17 @@ Item {
width: home.sideWidth
spacing: Theme.gutter
+ // What plays on the phone (its first player).
+ Repeater {
+ model: MediaList
+ delegate: NowPlaying {
+ required property int index
+ width: home.sideWidth
+ visible: deviceId === home.deviceId
+ && MediaList.revision >= 0 && index === MediaList.firstFor(home.deviceId)
+ }
+ }
+
Card {
width: parent.width
Column {
diff --git a/desktop/app/qml/pages/SettingsPage.qml b/desktop/app/qml/pages/SettingsPage.qml
index 30fa963..579f337 100644
--- a/desktop/app/qml/pages/SettingsPage.qml
+++ b/desktop/app/qml/pages/SettingsPage.qml
@@ -22,6 +22,8 @@ Item {
messages: qsTr("Messages"),
calls: qsTr("Calls"),
clipboard: qsTr("Clipboard"),
+ files: qsTr("Files"),
+ media: qsTr("Media playing on either device"),
photos: qsTr("Photos"),
pc_actions: qsTr("Lock and sleep this PC"),
remote_files: qsTr("Browse this PC's files while away")
diff --git a/desktop/app/src/bridge/media.rs b/desktop/app/src/bridge/media.rs
new file mode 100644
index 0000000..f2d260e
--- /dev/null
+++ b/desktop/app/src/bridge/media.rs
@@ -0,0 +1,285 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//! `MediaList`: what plays on paired devices as a Qt list model, plus the
+//! commands QML sends to those players.
+
+use std::{pin::Pin, time::UNIX_EPOCH};
+
+use cxx_qt::{CxxQtType, Threading};
+use cxx_qt_lib::{QByteArray, QHash, QHashPair_i32_QByteArray, QList, QModelIndex, QString, QUrl, QVariant};
+use nectarlink_core::MediaAction;
+
+use super::{Edit, diff};
+use crate::{
+ core_host, media,
+ state::{Changes, PlayerView},
+};
+
+#[cxx_qt::bridge]
+pub mod qobject {
+ unsafe extern "C++" {
+ include!(<QtCore/QAbstractListModel>);
+ type QAbstractListModel;
+
+ include!("cxx-qt-lib/qmodelindex.h");
+ type QModelIndex = cxx_qt_lib::QModelIndex;
+ include!("cxx-qt-lib/qvariant.h");
+ type QVariant = cxx_qt_lib::QVariant;
+ include!("cxx-qt-lib/qhash.h");
+ type QHash_i32_QByteArray = cxx_qt_lib::QHash<cxx_qt_lib::QHashPair_i32_QByteArray>;
+ include!("cxx-qt-lib/qlist.h");
+ type QList_i32 = cxx_qt_lib::QList<i32>;
+ include!("cxx-qt-lib/qstring.h");
+ type QString = cxx_qt_lib::QString;
+ }
+
+ unsafe extern "RustQt" {
+ #[qobject]
+ #[base = QAbstractListModel]
+ #[qml_element]
+ #[qml_singleton]
+ #[qproperty(i32, count)]
+ /// Bumped when players come or go, for bindings on [`first_for`].
+ #[qproperty(i32, revision)]
+ type MediaList = super::MediaListRust;
+ }
+
+ // Inherited from QAbstractListModel.
+ unsafe extern "RustQt" {
+ #[inherit]
+ #[cxx_name = "beginInsertRows"]
+ unsafe fn begin_insert_rows(self: Pin<&mut MediaList>, parent: &QModelIndex, first: i32, last: i32);
+ #[inherit]
+ #[cxx_name = "endInsertRows"]
+ unsafe fn end_insert_rows(self: Pin<&mut MediaList>);
+ #[inherit]
+ #[cxx_name = "beginRemoveRows"]
+ unsafe fn begin_remove_rows(self: Pin<&mut MediaList>, parent: &QModelIndex, first: i32, last: i32);
+ #[inherit]
+ #[cxx_name = "endRemoveRows"]
+ unsafe fn end_remove_rows(self: Pin<&mut MediaList>);
+ #[inherit]
+ fn index(self: &MediaList, row: i32, column: i32, parent: &QModelIndex) -> QModelIndex;
+
+ #[inherit]
+ #[qsignal]
+ #[cxx_name = "dataChanged"]
+ fn data_changed(
+ self: Pin<&mut MediaList>,
+ top_left: &QModelIndex,
+ bottom_right: &QModelIndex,
+ roles: &QList_i32,
+ );
+ }
+
+ unsafe extern "RustQt" {
+ #[qinvokable]
+ #[cxx_override]
+ fn data(self: &MediaList, index: &QModelIndex, role: i32) -> QVariant;
+ #[qinvokable]
+ #[cxx_override]
+ #[cxx_name = "roleNames"]
+ fn role_names(self: &MediaList) -> QHash_i32_QByteArray;
+ #[qinvokable]
+ #[cxx_override]
+ #[cxx_name = "rowCount"]
+ fn row_count(self: &MediaList, parent: &QModelIndex) -> i32;
+
+ /// Runs "play", "pause", "next", "previous" or "seek" (to
+ /// `position_ms`) on a device's player.
+ #[qinvokable]
+ fn command(self: &MediaList, device: &QString, player: &QString, action: &QString, position_ms: f64);
+ /// The row of a device's first player (the one to show), or -1.
+ #[qinvokable]
+ #[cxx_name = "firstFor"]
+ fn first_for(self: &MediaList, device: &QString) -> i32;
+ }
+
+ impl cxx_qt::Threading for MediaList {}
+ impl cxx_qt::Initialize for MediaList {}
+}
+
+#[derive(Default)]
+pub struct MediaListRust {
+ count: i32,
+ revision: i32,
+ rows: Vec<PlayerView>,
+}
+
+/// Model roles (Qt::UserRole + n).
+const ROLES: &[&str] = &[
+ "deviceId",
+ "playerId",
+ "app",
+ "title",
+ "artist",
+ "album",
+ "playing",
+ "duration",
+ "position",
+ "positionAt",
+ "art",
+ "canPlay",
+ "canPause",
+ "canNext",
+ "canPrevious",
+ "canSeek",
+];
+const USER_ROLE: i32 = 0x0100;
+
+fn role_value(row: &PlayerView, role: &str) -> QVariant {
+ let text = |s: &str| QVariant::from(&QString::from(s));
+ let p = &row.player;
+ let can = |action: &str| QVariant::from(&p.actions.iter().any(|a| a == action));
+ match role {
+ "deviceId" => text(&row.device.to_string()),
+ "playerId" => text(&p.id),
+ "app" => text(&p.app),
+ "title" => text(p.title.as_deref().unwrap_or_default()),
+ "artist" => text(p.artist.as_deref().unwrap_or_default()),
+ "album" => text(p.album.as_deref().unwrap_or_default()),
+ "playing" => QVariant::from(&p.playing),
+ // Milliseconds; -1 when the player doesn't say.
+ "duration" => QVariant::from(&p.duration.map_or(-1.0, |d| d as f64)),
+ "position" => QVariant::from(&p.position.map_or(-1.0, |d| d as f64)),
+ // When `position` was current, as JavaScript time.
+ "positionAt" => {
+ QVariant::from(&row.at.duration_since(UNIX_EPOCH).map_or(0.0, |d| d.as_millis() as f64))
+ }
+ "art" => text(
+ &row.art
+ .as_deref()
+ .map(|path| {
+ String::from(
+ &QUrl::from_local_file(&QString::from(&*path.to_string_lossy())).to_qstring(),
+ )
+ })
+ .unwrap_or_default(),
+ ),
+ "canPlay" => can("play"),
+ "canPause" => can("pause"),
+ "canNext" => can("next"),
+ "canPrevious" => can("previous"),
+ "canSeek" => can("seek"),
+ _ => QVariant::default(),
+ }
+}
+
+impl cxx_qt::Initialize for qobject::MediaList {
+ fn initialize(self: Pin<&mut Self>) {
+ super::subscribe(self.qt_thread(), Changes::MEDIA, Self::refresh);
+ }
+}
+
+impl qobject::MediaList {
+ fn refresh(mut self: Pin<&mut Self>) {
+ let new: Vec<PlayerView> = core_host::host().hub.read(|s| s.media.clone());
+ let root = QModelIndex::default();
+ for edit in diff(&self.rows, &new, |r| (r.device, r.player.id.clone())) {
+ match edit {
+ Edit::Remove(row) => {
+ let r = row as i32;
+ // SAFETY: row exists; begin/end are balanced.
+ unsafe {
+ self.as_mut().begin_remove_rows(&root, r, r);
+ self.as_mut().rust_mut().rows.remove(row);
+ self.as_mut().end_remove_rows();
+ }
+ }
+ Edit::Insert(row) => {
+ let r = row as i32;
+ // SAFETY: row <= len; begin/end are balanced.
+ unsafe {
+ self.as_mut().begin_insert_rows(&root, r, r);
+ self.as_mut().rust_mut().rows.insert(row, new[row].clone());
+ self.as_mut().end_insert_rows();
+ }
+ }
+ Edit::Change(row) => {
+ self.as_mut().rust_mut().rows[row] = new[row].clone();
+ let index = self.index(row as i32, 0, &root);
+ self.as_mut().data_changed(&index, &index, &QList::default());
+ }
+ }
+ }
+ let count = self.rows.len() as i32;
+ self.as_mut().set_count(count);
+ let revision = self.revision.wrapping_add(1);
+ self.set_revision(revision);
+ }
+
+ pub fn first_for(&self, device: &QString) -> i32 {
+ let device = String::from(device);
+ self.rows.iter().position(|r| r.device.to_string() == device).map_or(-1, |row| row as i32)
+ }
+
+ pub fn data(&self, index: &QModelIndex, role: i32) -> QVariant {
+ let row = usize::try_from(index.row()).ok();
+ let role = usize::try_from(role - USER_ROLE).ok().and_then(|r| ROLES.get(r));
+ match (row.and_then(|r| self.rows.get(r)), role) {
+ (Some(row), Some(role)) => role_value(row, role),
+ _ => QVariant::default(),
+ }
+ }
+
+ pub fn role_names(&self) -> QHash<QHashPair_i32_QByteArray> {
+ let mut roles = QHash::<QHashPair_i32_QByteArray>::default();
+ for (i, name) in ROLES.iter().enumerate() {
+ roles.insert(USER_ROLE + i as i32, QByteArray::from(*name));
+ }
+ roles
+ }
+
+ pub fn row_count(&self, _parent: &QModelIndex) -> i32 {
+ self.rows.len() as i32
+ }
+
+ pub fn command(&self, device: &QString, player: &QString, action: &QString, position_ms: f64) {
+ let Some(device) = super::parse_device(device) else { return };
+ let Some(action) = MediaAction::parse(&String::from(action)) else { return };
+ let position = (action == MediaAction::Seek).then(|| position_ms.max(0.0) as u64);
+ media::command(device, String::from(player), action, position);
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::path::PathBuf;
+
+ use nectarlink_core::{DeviceId, MediaPlayer};
+
+ use super::*;
+
+ #[test]
+ fn roles_describe_the_player() {
+ let row = PlayerView {
+ device: DeviceId([4; 32]),
+ player: MediaPlayer {
+ id: "com.music".into(),
+ app: "Music".into(),
+ title: Some("Song".into()),
+ artist: None,
+ album: None,
+ playing: true,
+ duration: Some(90_000),
+ position: None,
+ actions: vec!["pause".into(), "seek".into()],
+ art_key: None,
+ art: None,
+ },
+ art: Some(PathBuf::from(r"C:\data\cache\art\a.jpg")),
+ at: UNIX_EPOCH + std::time::Duration::from_millis(1_500),
+ };
+ let get = |role| role_value(&row, role);
+ assert_eq!(get("title").value::<QString>().map(|s| String::from(&s)), Some("Song".into()));
+ assert_eq!(get("artist").value::<QString>().map(|s| String::from(&s)), Some(String::new()));
+ assert_eq!(get("duration").value::<f64>(), Some(90_000.0));
+ assert_eq!(get("position").value::<f64>(), Some(-1.0));
+ assert_eq!(get("positionAt").value::<f64>(), Some(1_500.0));
+ assert_eq!(get("canPause").value::<bool>(), Some(true));
+ assert_eq!(get("canPlay").value::<bool>(), Some(false));
+ assert_eq!(
+ get("art").value::<QString>().map(|s| String::from(&s)),
+ Some("file:///C:/data/cache/art/a.jpg".into())
+ );
+ }
+}
diff --git a/desktop/app/src/bridge/mod.rs b/desktop/app/src/bridge/mod.rs
index f167d52..66d9e24 100644
--- a/desktop/app/src/bridge/mod.rs
+++ b/desktop/app/src/bridge/mod.rs
@@ -10,6 +10,8 @@ pub mod app;
#[allow(unsafe_code)]
pub mod devices;
#[allow(unsafe_code)]
+pub mod media;
+#[allow(unsafe_code)]
pub mod native;
#[allow(unsafe_code)]
pub mod notifications;
diff --git a/desktop/app/src/core_host.rs b/desktop/app/src/core_host.rs
index 4e4c7f7..736533a 100644
--- a/desktop/app/src/core_host.rs
+++ b/desktop/app/src/core_host.rs
@@ -94,6 +94,8 @@ async fn run(data_dir: PathBuf, platform: Arc<dyn Platform>) {
let host = host();
let mut config = NodeConfig::new(&data_dir, this_device(), env!("CARGO_PKG_VERSION"));
config.downloads_dir = Some(downloads_dir());
+ // This PC's own players, for phones (crate::win::media_sessions).
+ config.capabilities = vec!["media.control".into()];
let node = match Node::start(config, platform).await {
Ok(node) => node,
Err(e) => {
@@ -136,6 +138,7 @@ async fn run(data_dir: PathBuf, platform: Arc<dyn Platform>) {
match events.recv().await {
Ok(event) => {
crate::notifications::apply(&event);
+ crate::media::on_event(&event);
crate::notifications::update_toasts(&event);
crate::clipboard::on_event(&event);
crate::transfers::on_event(&event);
diff --git a/desktop/app/src/main.rs b/desktop/app/src/main.rs
index ba07113..37f2fb5 100644
--- a/desktop/app/src/main.rs
+++ b/desktop/app/src/main.rs
@@ -15,6 +15,7 @@ mod icons;
mod launch;
mod logging;
mod mark;
+mod media;
mod notifications;
mod palette;
mod qr;
@@ -35,7 +36,7 @@ use std::{
};
use cxx_qt_lib::{QGuiApplication, QQmlApplicationEngine, QString, QUrl};
-use nectarlink_core::Platform;
+use nectarlink_core::{MediaAction, MediaError, Platform};
use crate::{
launch::Request,
@@ -62,6 +63,14 @@ impl Platform for DesktopPlatform {
fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
win::clipboard::write_image(mime, bytes)
}
+ fn media_command(
+ &self,
+ player: &str,
+ action: MediaAction,
+ position: Option<u64>,
+ ) -> Result<(), MediaError> {
+ win::media_sessions::command(player, action, position)
+ }
}
/// Command-line options (also used by shortcuts and autostart).
@@ -202,6 +211,8 @@ fn main() -> ExitCode {
watch_network();
start_toasts();
clipboard::start();
+ win::smtc::start(media::on_flyout);
+ win::media_sessions::start(media::local_changed);
send_to::set_enabled(settings::Settings::load(&data_dir).send_to_menu);
// A test instance (own data folder) leaves the user's menu alone.
if options.data_dir.is_none() {
diff --git a/desktop/app/src/media.rs b/desktop/app/src/media.rs
new file mode 100644
index 0000000..08d9292
--- /dev/null
+++ b/desktop/app/src/media.rs
@@ -0,0 +1,156 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//! Media between this PC and paired phones (docs/protocol/media.md): what
+//! plays on a phone shows in the app and in Windows' media flyout, where
+//! it can be controlled; what plays on this PC goes to the phones (see
+//! `win::media_sessions`).
+
+use std::{
+ collections::HashMap,
+ path::PathBuf,
+ sync::{Mutex, MutexGuard},
+};
+
+use nectarlink_core::{DeviceId, Error, MediaAction, NodeEvent};
+
+use crate::{
+ bridge::app::{describe, show_message},
+ core_host,
+ state::PlayerView,
+ win::smtc,
+};
+
+fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
+ m.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// Artwork files by (device, art key), for players that only send a key.
+static ART: Mutex<Option<HashMap<(DeviceId, String), PathBuf>>> = Mutex::new(None);
+/// Artwork files kept per device (older ones are deleted).
+const KEPT_ART: usize = 12;
+
+fn art_dir() -> PathBuf {
+ core_host::host().data_dir.join("cache").join("art")
+}
+
+/// Saves the artwork in a media event; returns every player's artwork file
+/// by player ID.
+pub fn save_art(event: &NodeEvent) -> HashMap<String, PathBuf> {
+ let NodeEvent::MediaChanged { device, players } = event else { return HashMap::new() };
+ let mut art = lock(&ART);
+ let art = art.get_or_insert_default();
+ let mut found = HashMap::new();
+ for player in players {
+ let Some(key) = &player.art_key else { continue };
+ let slot = (*device, key.clone());
+ if let Some(bytes) = &player.art {
+ let ext = if bytes.starts_with(b"\x89PNG") { "png" } else { "jpg" };
+ // Keys are the sender's; the file name is ours.
+ let name = format!("{}-{:016x}.{ext}", device.short(), fnv(key));
+ let path = art_dir().join(name);
+ let written = std::fs::create_dir_all(art_dir()).and_then(|()| std::fs::write(&path, bytes));
+ match written {
+ Ok(()) => {
+ art.insert(slot.clone(), path);
+ }
+ Err(e) => tracing::warn!(error = %e, "can't save artwork"),
+ }
+ }
+ if let Some(path) = art.get(&slot) {
+ found.insert(player.id.clone(), path.clone());
+ }
+ }
+ prune(art, *device, players.iter().filter_map(|p| p.art_key.clone()).collect());
+ found
+}
+
+/// Deletes a device's artwork beyond the newest few, never what's in use.
+fn prune(art: &mut HashMap<(DeviceId, String), PathBuf>, device: DeviceId, in_use: Vec<String>) {
+ let mut theirs: Vec<(String, PathBuf)> =
+ art.iter().filter(|((d, _), _)| *d == device).map(|((_, k), p)| (k.clone(), p.clone())).collect();
+ if theirs.len() <= KEPT_ART {
+ return;
+ }
+ theirs.sort_by_key(|(_, path)| std::fs::metadata(path).and_then(|m| m.modified()).ok());
+ let excess = theirs.len() - KEPT_ART;
+ for (key, path) in theirs.into_iter().filter(|(k, _)| !in_use.contains(k)).take(excess) {
+ let _ = std::fs::remove_file(&path);
+ art.remove(&(device, key));
+ }
+}
+
+fn fnv(text: &str) -> u64 {
+ text.bytes().fold(0xcbf2_9ce4_8422_2325_u64, |h, b| (h ^ u64::from(b)).wrapping_mul(0x0100_0000_01b3))
+}
+
+/// Keeps Windows' media flyout showing what plays on the phones.
+pub fn update_flyout() {
+ let current = core_host::host().hub.read(|s| {
+ // A player that's playing wins; then the first one shown.
+ let pick = s.media.iter().find(|p| p.player.playing).or_else(|| s.media.first())?;
+ let device = s.name_of(&pick.device).unwrap_or_default();
+ Some(smtc::Shown { device: pick.device, device_name: device, view: pick.clone() })
+ });
+ smtc::show(current);
+}
+
+/// What the flyout or the keyboard's media keys asked for.
+pub fn on_flyout(device: DeviceId, player: String, action: MediaAction, position: Option<u64>) {
+ send(device, player, action, position, false);
+}
+
+/// Runs a command on a phone's player; says so if it didn't work.
+pub fn command(device: DeviceId, player: String, action: MediaAction, position: Option<u64>) {
+ send(device, player, action, position, true);
+}
+
+fn send(device: DeviceId, player: String, action: MediaAction, position: Option<u64>, tell: bool) {
+ let Some(node) = core_host::node() else { return };
+ core_host::spawn(async move {
+ let result = node.media_command(device, player, action, position).await;
+ let message = match result {
+ Ok(()) => return,
+ Err(Error::NotFound) => "That player isn't playing anymore.".to_owned(),
+ Err(Error::Unsupported) => "That player can't do that.".to_owned(),
+ Err(Error::Denied) => "Media is turned off for that device.".to_owned(),
+ Err(e) => describe(&e),
+ };
+ tracing::debug!(message, "media command failed");
+ if tell {
+ show_message(message);
+ }
+ });
+}
+
+/// The hub's view of a player, with when its position was measured.
+pub fn view(device: DeviceId, mut player: nectarlink_core::MediaPlayer, art: Option<PathBuf>) -> PlayerView {
+ player.art = None;
+ PlayerView { device, player, art, at: std::time::SystemTime::now() }
+}
+
+/// Folds a media event into the hub (artwork saved first) and updates the
+/// flyout.
+pub fn on_event(event: &NodeEvent) {
+ let changed = match event {
+ NodeEvent::MediaChanged { device, players } => {
+ let art = save_art(event);
+ let views = players.iter().map(|p| view(*device, p.clone(), art.get(&p.id).cloned())).collect();
+ core_host::host().hub.update(|s| s.set_media(*device, views));
+ true
+ }
+ // Offline or unpaired devices' players are cleared by the hub.
+ NodeEvent::LinkChanged { .. } | NodeEvent::DeviceRemoved(_) => true,
+ _ => false,
+ };
+ if changed {
+ update_flyout();
+ }
+}
+
+/// What plays on this PC changed: tell the phones.
+pub fn local_changed(players: Vec<nectarlink_core::MediaPlayer>) {
+ core_host::spawn(async move {
+ if let Some(node) = core_host::wait_for_node().await {
+ node.media_changed(players).await;
+ }
+ });
+}
diff --git a/desktop/app/src/state.rs b/desktop/app/src/state.rs
index 506f4e3..8ca8e60 100644
--- a/desktop/app/src/state.rs
+++ b/desktop/app/src/state.rs
@@ -11,13 +11,13 @@ use std::{
};
use nectarlink_core::{
- Battery, CapabilityMatrix, DeviceId, DeviceInfo, DiscoveredDevice, LinkState, NodeEvent, Notification,
- PairedDevice, PairingEvent, PairingFailure, PowerLevel, Transfer, TransferState,
+ Battery, CapabilityMatrix, DeviceId, DeviceInfo, DiscoveredDevice, LinkState, MediaPlayer, NodeEvent,
+ Notification, PairedDevice, PairingEvent, PairingFailure, PowerLevel, Transfer, TransferState,
};
/// Which parts of the state changed, so listeners refresh only what they show.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
-pub struct Changes(u8);
+pub struct Changes(u16);
impl Changes {
pub const NONE: Changes = Changes(0);
@@ -29,6 +29,7 @@ impl Changes {
pub const RINGING: Changes = Changes(1 << 5);
pub const NOTIFICATIONS: Changes = Changes(1 << 6);
pub const TRANSFERS: Changes = Changes(1 << 7);
+ pub const MEDIA: Changes = Changes(1 << 8);
pub fn is_empty(self) -> bool {
self.0 == 0
@@ -84,6 +85,17 @@ impl From<PairedDevice> for DeviceView {
}
}
+/// A player on a paired device (docs/protocol/media.md).
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct PlayerView {
+ pub device: DeviceId,
+ /// Without its artwork, which is in `art`.
+ pub player: MediaPlayer,
+ pub art: Option<PathBuf>,
+ /// When `player.position` was current.
+ pub at: SystemTime,
+}
+
/// The pairing screen's state.
#[derive(Debug, Clone, PartialEq, Eq, Default)]
pub enum PairingView {
@@ -169,6 +181,8 @@ pub struct AppState {
pub replies: HashMap<(DeviceId, String), Vec<SentReply>>,
/// File transfers, newest first.
pub transfers: Vec<TransferView>,
+ /// Media players on paired devices, each device's in its order.
+ pub media: Vec<PlayerView>,
}
impl AppState {
@@ -242,9 +256,16 @@ impl AppState {
if self.notifications.len() != notifications {
changes |= Changes::NOTIFICATIONS;
}
+ changes | self.set_media(*id, Vec::new())
+ }
+ NodeEvent::LinkChanged { device, link } => {
+ let mut changes = self.update_device(device, |d| d.link = link.clone());
+ // A device that comes back sends what plays then.
+ if matches!(link, LinkState::Offline { .. }) {
+ changes |= self.set_media(*device, Vec::new());
+ }
changes
}
- NodeEvent::LinkChanged { device, link } => self.update_device(device, |d| d.link = link.clone()),
NodeEvent::PeerInfoChanged { device, info } => {
self.update_device(device, |d| d.info = info.clone())
}
@@ -305,7 +326,22 @@ impl AppState {
// Feedback only (see crate::clipboard); nothing to keep.
NodeEvent::ClipboardReceived { .. } => Changes::NONE,
NodeEvent::Transfer(transfer) => self.update_transfer(transfer.clone(), Instant::now()),
+ // With artwork saved first (see crate::media).
+ NodeEvent::MediaChanged { .. } => Changes::NONE,
+ }
+ }
+
+ /// Replaces a device's players.
+ pub fn set_media(&mut self, device: DeviceId, players: Vec<PlayerView>) -> Changes {
+ let at = self.media.iter().position(|p| p.device == device).unwrap_or(self.media.len());
+ let before: Vec<PlayerView> = self.media.iter().filter(|p| p.device == device).cloned().collect();
+ if before.is_empty() && players.is_empty() {
+ return Changes::NONE;
}
+ self.media.retain(|p| p.device != device);
+ let at = at.min(self.media.len());
+ self.media.splice(at..at, players);
+ Changes::MEDIA
}
/// Adds or updates a transfer, keeping a smoothed speed.
diff --git a/desktop/app/src/win/image.rs b/desktop/app/src/win/image.rs
index 86b9331..6c8f676 100644
--- a/desktop/app/src/win/image.rs
+++ b/desktop/app/src/win/image.rs
@@ -108,6 +108,43 @@ pub fn encode_png(bitmap: &Bitmap) -> Result<Vec<u8>> {
})
}
+/// Scaled down (never up) so neither side is longer than `max`, averaging
+/// the pixels each new one covers.
+pub fn scale_to(bitmap: &Bitmap, max: u32) -> Bitmap {
+ let longest = bitmap.width.max(bitmap.height);
+ if longest <= max || longest == 0 {
+ return bitmap.clone();
+ }
+ let scale = f64::from(max) / f64::from(longest);
+ let width = ((f64::from(bitmap.width) * scale).round() as u32).max(1);
+ let height = ((f64::from(bitmap.height) * scale).round() as u32).max(1);
+ let mut bgra = Vec::with_capacity((width * height * 4) as usize);
+ for y in 0..height {
+ let (y0, y1) = (
+ y * bitmap.height / height,
+ ((y + 1) * bitmap.height / height).max(y * bitmap.height / height + 1),
+ );
+ for x in 0..width {
+ let (x0, x1) = (
+ x * bitmap.width / width,
+ ((x + 1) * bitmap.width / width).max(x * bitmap.width / width + 1),
+ );
+ let mut sum = [0u32; 4];
+ for sy in y0..y1 {
+ for sx in x0..x1 {
+ let i = ((sy * bitmap.width + sx) * 4) as usize;
+ for (total, &value) in sum.iter_mut().zip(&bitmap.bgra[i..i + 4]) {
+ *total += u32::from(value);
+ }
+ }
+ }
+ let n = (y1 - y0) * (x1 - x0);
+ bgra.extend(sum.map(|s| (s / n) as u8));
+ }
+ }
+ Bitmap { width, height, bgra }
+}
+
/// A device-independent bitmap (`CF_DIB` / `CF_DIBV5`) as a BMP file, which
/// WIC can decode: the DIB with a file header in front.
pub fn bmp_from_dib(dib: &[u8]) -> Option<Vec<u8>> {
@@ -195,6 +232,19 @@ mod tests {
assert_eq!(decode(&png).unwrap(), bitmap);
}
+ #[test]
+ fn large_pictures_scale_down_evenly() {
+ let big = Bitmap { width: 400, height: 200, bgra: [10, 20, 30, 255].repeat(400 * 200) };
+ let small = scale_to(&big, 100);
+ assert_eq!((small.width, small.height), (100, 50));
+ assert_eq!(&small.bgra[..4], &[10, 20, 30, 255]);
+ assert_eq!(scale_to(&small, 100), small, "never scaled up");
+ // A checkerboard averages to grey.
+ let checker = checker(4, 4);
+ let half = scale_to(&checker, 2);
+ assert_eq!(half.bgra.len(), 2 * 2 * 4);
+ }
+
#[test]
fn copied_bitmaps_decode() {
let bitmap = checker(7, 4);
diff --git a/desktop/app/src/win/media_sessions.rs b/desktop/app/src/win/media_sessions.rs
new file mode 100644
index 0000000..5a34306
--- /dev/null
+++ b/desktop/app/src/win/media_sessions.rs
@@ -0,0 +1,344 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//! What plays on this PC, for paired phones (docs/protocol/media.md): the
+//! sessions apps report to Windows (Global System Media Transport
+//! Controls), and commands from a phone sent to them.
+//!
+//! Nectarlink's own session (a phone's media in the flyout) is left out, so
+//! a phone never gets its own music back.
+
+use std::{
+ collections::HashMap,
+ sync::{OnceLock, mpsc},
+ time::{Duration, SystemTime, UNIX_EPOCH},
+};
+
+use nectarlink_core::{MediaAction, MediaError, MediaPlayer};
+use windows::{
+ ApplicationModel::AppInfo,
+ Foundation::TypedEventHandler,
+ Media::Control::{
+ GlobalSystemMediaTransportControlsSession, GlobalSystemMediaTransportControlsSessionManager,
+ GlobalSystemMediaTransportControlsSessionPlaybackStatus,
+ },
+ Storage::Streams::DataReader,
+ Win32::System::Com::{COINIT_MULTITHREADED, CoInitializeEx},
+ core::HSTRING,
+};
+
+use super::toast::AUMID;
+
+/// The largest artwork sent as is; larger pictures are scaled down.
+const ART_BYTES: u64 = 256 * 1024;
+/// Changes come in bursts (title, then artwork, then position).
+const SETTLE: Duration = Duration::from_millis(300);
+/// Windows' clock starts in 1601; Unix time in 1970.
+const WINDOWS_TO_UNIX_SECS: i64 = 11_644_473_600;
+
+type OnChange = Box<dyn Fn(Vec<MediaPlayer>) + Send + Sync>;
+
+static MANAGER: OnceLock<GlobalSystemMediaTransportControlsSessionManager> = OnceLock::new();
+
+/// Watches this PC's media; `on_change` gets every player after each
+/// change (on the watcher's thread).
+pub fn start(on_change: impl Fn(Vec<MediaPlayer>) + Send + Sync + 'static) {
+ let on_change: OnChange = Box::new(on_change);
+ let spawned = std::thread::Builder::new().name("media-sessions".into()).spawn(move || {
+ if let Err(e) = run(&on_change) {
+ tracing::warn!(error = %e, "can't watch this PC's media");
+ }
+ });
+ if let Err(e) = spawned {
+ tracing::warn!(error = %e, "can't watch this PC's media");
+ }
+}
+
+fn run(on_change: &OnChange) -> windows::core::Result<()> {
+ // SAFETY: COM for this thread, which lives as long as the app.
+ unsafe { CoInitializeEx(None, COINIT_MULTITHREADED) }.ok()?;
+ let manager = GlobalSystemMediaTransportControlsSessionManager::RequestAsync()?.join()?;
+ let (changed, changes) = mpsc::channel::<()>();
+ let signal = changed.clone();
+ manager.SessionsChanged(&TypedEventHandler::new(move |_, _| {
+ let _ = signal.send(());
+ Ok(())
+ }))?;
+ let _ = MANAGER.set(manager.clone());
+
+ let mut watched: Vec<String> = Vec::new();
+ let mut art = ArtCache::default();
+ let mut last: Option<Vec<MediaPlayer>> = None;
+ loop {
+ // Hook up sessions that are new since last time.
+ let mut present = Vec::new();
+ for session in manager.GetSessions()? {
+ let id = session.SourceAppUserModelId()?.to_string();
+ if !watched.contains(&id) {
+ watch(&session, &changed)?;
+ watched.push(id.clone());
+ }
+ present.push(id);
+ }
+ // A session that ends and comes back is hooked up again.
+ watched.retain(|id| present.contains(id));
+ let players = snapshot(&manager, &mut art);
+ if last.as_ref() != Some(&players) {
+ on_change(players.clone());
+ last = Some(players);
+ }
+ // Wait for a change, then let the burst settle.
+ if changes.recv().is_err() {
+ return Ok(());
+ }
+ while changes.recv_timeout(SETTLE).is_ok() {}
+ }
+}
+
+fn watch(
+ session: &GlobalSystemMediaTransportControlsSession,
+ changed: &mpsc::Sender<()>,
+) -> windows::core::Result<()> {
+ session.MediaPropertiesChanged(&TypedEventHandler::new({
+ let changed = changed.clone();
+ move |_, _| {
+ let _ = changed.send(());
+ Ok(())
+ }
+ }))?;
+ session.PlaybackInfoChanged(&TypedEventHandler::new({
+ let changed = changed.clone();
+ move |_, _| {
+ let _ = changed.send(());
+ Ok(())
+ }
+ }))?;
+ session.TimelinePropertiesChanged(&TypedEventHandler::new({
+ let changed = changed.clone();
+ move |_, _| {
+ let _ = changed.send(());
+ Ok(())
+ }
+ }))?;
+ Ok(())
+}
+
+/// Every session but Nectarlink's own, the one Windows considers current
+/// first.
+fn snapshot(
+ manager: &GlobalSystemMediaTransportControlsSessionManager,
+ art: &mut ArtCache,
+) -> Vec<MediaPlayer> {
+ let current =
+ manager.GetCurrentSession().ok().and_then(|s| s.SourceAppUserModelId().ok()).map(|id| id.to_string());
+ let Ok(sessions) = manager.GetSessions() else { return Vec::new() };
+ let mut players: Vec<MediaPlayer> = sessions
+ .into_iter()
+ .filter_map(|session| match player(&session, art) {
+ Ok(player) => player,
+ Err(e) => {
+ tracing::debug!(error = %e, "can't read a media session");
+ None
+ }
+ })
+ .collect();
+ players.sort_by_key(|p| (Some(&p.id) != current.as_ref(), !p.playing));
+ players
+}
+
+fn player(
+ session: &GlobalSystemMediaTransportControlsSession,
+ art: &mut ArtCache,
+) -> windows::core::Result<Option<MediaPlayer>> {
+ let id = session.SourceAppUserModelId()?.to_string();
+ if id == AUMID {
+ return Ok(None);
+ }
+ let properties = session.TryGetMediaPropertiesAsync()?.join()?;
+ let text =
+ |s: windows::core::Result<HSTRING>| s.ok().map(|s| s.to_string()).filter(|s| !s.trim().is_empty());
+ let title = text(properties.Title());
+ let artist = text(properties.Artist());
+ let album = text(properties.AlbumTitle());
+ if title.is_none() {
+ return Ok(None);
+ }
+ let info = session.GetPlaybackInfo()?;
+ let playing = info.PlaybackStatus()? == GlobalSystemMediaTransportControlsSessionPlaybackStatus::Playing;
+ let controls = info.Controls()?;
+ let mut actions = Vec::new();
+ for (on, action) in [
+ (controls.IsPlayEnabled()?, "play"),
+ (controls.IsPauseEnabled()?, "pause"),
+ (controls.IsNextEnabled()?, "next"),
+ (controls.IsPreviousEnabled()?, "previous"),
+ (controls.IsPlaybackPositionEnabled()?, "seek"),
+ ] {
+ if on {
+ actions.push(action.to_owned());
+ }
+ }
+
+ let timeline = session.GetTimelineProperties()?;
+ let ms = |ticks: i64| u64::try_from(ticks / 10_000).unwrap_or(0);
+ let duration = ms(timeline.EndTime()?.Duration - timeline.StartTime()?.Duration);
+ let (duration, position) = if duration > 0 {
+ let mut position = ms(timeline.Position()?.Duration);
+ // The position is as of the last update; it has moved on since.
+ if playing {
+ let updated = timeline.LastUpdatedTime()?.UniversalTime;
+ let updated_ms = updated / 10_000 - WINDOWS_TO_UNIX_SECS * 1000;
+ let now_ms = SystemTime::now().duration_since(UNIX_EPOCH).map_or(0, |d| d.as_millis() as i64);
+ position = position.saturating_add(u64::try_from(now_ms - updated_ms).unwrap_or(0));
+ }
+ (Some(duration), Some(position.min(duration)))
+ } else {
+ (None, None)
+ };
+
+ let (art_key, art_bytes) = art.get(&id, &title, &artist, &album, || thumbnail(&properties)).unzip();
+ Ok(Some(MediaPlayer {
+ app: app_name(&id),
+ id,
+ title,
+ artist,
+ album,
+ playing,
+ duration,
+ position,
+ actions,
+ art_key,
+ art: art_bytes,
+ }))
+}
+
+/// Artwork per track, read once: reading it takes a moment.
+#[derive(Default)]
+struct ArtCache {
+ by_track: HashMap<String, Option<(String, Vec<u8>)>>,
+}
+
+impl ArtCache {
+ fn get(
+ &mut self,
+ id: &str,
+ title: &Option<String>,
+ artist: &Option<String>,
+ album: &Option<String>,
+ read: impl FnOnce() -> Option<Vec<u8>>,
+ ) -> Option<(String, Vec<u8>)> {
+ let track = format!("{id}\u{1}{title:?}\u{1}{artist:?}\u{1}{album:?}");
+ if !self.by_track.contains_key(&track) {
+ if self.by_track.len() > 32 {
+ self.by_track.clear();
+ }
+ let art = read().map(|bytes| (format!("{:016x}", fnv(&bytes)), bytes));
+ self.by_track.insert(track.clone(), art);
+ }
+ self.by_track.get(&track).cloned().flatten()
+ }
+}
+
+fn fnv(bytes: &[u8]) -> u64 {
+ bytes.iter().fold(0xcbf2_9ce4_8422_2325_u64, |h, b| (h ^ u64::from(*b)).wrapping_mul(0x0100_0000_01b3))
+}
+
+fn thumbnail(
+ properties: &windows::Media::Control::GlobalSystemMediaTransportControlsSessionMediaProperties,
+) -> Option<Vec<u8>> {
+ let read = || -> windows::core::Result<Vec<u8>> {
+ let stream = properties.Thumbnail()?.OpenReadAsync()?.join()?;
+ let size = stream.Size()?;
+ let reader = DataReader::CreateDataReader(&stream)?;
+ let loaded = reader.LoadAsync(u32::try_from(size).unwrap_or(u32::MAX))?.join()?;
+ let mut bytes = vec![0u8; loaded as usize];
+ reader.ReadBytes(&mut bytes)?;
+ Ok(bytes)
+ };
+ let bytes = read().ok().filter(|b| !b.is_empty())?;
+ if bytes.len() as u64 <= ART_BYTES {
+ return Some(bytes);
+ }
+ // Too large to send: a smaller PNG.
+ let bitmap = super::image::decode(&bytes).ok()?;
+ let small = super::image::encode_png(&super::image::scale_to(&bitmap, 360)).ok()?;
+ (small.len() as u64 <= ART_BYTES).then_some(small)
+}
+
+/// The app's name as Windows shows it, or a readable form of its ID.
+fn app_name(id: &str) -> String {
+ let display =
+ AppInfo::GetFromAppUserModelId(&HSTRING::from(id)).and_then(|info| info.DisplayInfo()?.DisplayName());
+ if let Ok(name) = display {
+ let name = name.to_string();
+ if !name.trim().is_empty() {
+ return name;
+ }
+ }
+ readable_name(id)
+}
+
+/// "Spotify.exe" → "Spotify", `C:\…\vlc.exe` → "vlc", "Microsoft.ZuneMusic_8wekyb3d8bbwe!App" → "Microsoft.ZuneMusic".
+fn readable_name(id: &str) -> String {
+ let id = id.rsplit(['\\', '/']).next().unwrap_or(id);
+ let id = id.split('!').next().unwrap_or(id);
+ let id = id.split('_').next().unwrap_or(id);
+ let id = id.strip_suffix(".exe").or_else(|| id.strip_suffix(".EXE")).unwrap_or(id);
+ if id.is_empty() { "Media".into() } else { id.to_owned() }
+}
+
+/// Runs a phone's command on one of this PC's sessions.
+pub fn command(player: &str, action: MediaAction, position: Option<u64>) -> Result<(), MediaError> {
+ let manager = MANAGER.get().ok_or(MediaError::Unsupported)?;
+ let failed = |e: windows::core::Error| MediaError::Failed(e.to_string());
+ let session = manager
+ .GetSessions()
+ .map_err(failed)?
+ .into_iter()
+ .find(|s| s.SourceAppUserModelId().is_ok_and(|id| id == player))
+ .ok_or(MediaError::NotFound)?;
+ let done = match action {
+ MediaAction::Play => session.TryPlayAsync(),
+ MediaAction::Pause => session.TryPauseAsync(),
+ MediaAction::Next => session.TrySkipNextAsync(),
+ MediaAction::Previous => session.TrySkipPreviousAsync(),
+ MediaAction::Seek => {
+ let ticks = i64::try_from(position.unwrap_or(0)).unwrap_or(0).saturating_mul(10_000);
+ session.TryChangePlaybackPositionAsync(ticks)
+ }
+ }
+ .and_then(|op| op.join())
+ .map_err(failed)?;
+ if done { Ok(()) } else { Err(MediaError::Unsupported) }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ #[test]
+ fn app_ids_read_as_names() {
+ assert_eq!(readable_name("Spotify.exe"), "Spotify");
+ assert_eq!(readable_name(r"C:\Program Files\VideoLAN\vlc.exe"), "vlc");
+ assert_eq!(
+ readable_name("Microsoft.ZuneMusic_8wekyb3d8bbwe!Microsoft.ZuneMusic"),
+ "Microsoft.ZuneMusic"
+ );
+ assert_eq!(readable_name("Chrome"), "Chrome");
+ assert_eq!(readable_name(""), "Media");
+ }
+
+ #[test]
+ fn artwork_is_read_once_per_track() {
+ let mut cache = ArtCache::default();
+ let mut reads = 0;
+ let title = Some("Song".to_owned());
+ for _ in 0..3 {
+ let art = cache.get("app", &title, &None, &None, || {
+ reads += 1;
+ Some(vec![1, 2, 3])
+ });
+ assert_eq!(art.map(|(_, b)| b), Some(vec![1, 2, 3]));
+ }
+ assert_eq!(reads, 1);
+ assert!(cache.get("app", &Some("Other".into()), &None, &None, || None).is_none());
+ }
+}
diff --git a/desktop/app/src/win/mod.rs b/desktop/app/src/win/mod.rs
index e4b5430..7f437fb 100644
--- a/desktop/app/src/win/mod.rs
+++ b/desktop/app/src/win/mod.rs
@@ -8,9 +8,11 @@
pub mod clipboard;
pub mod icon;
pub mod image;
+pub mod media_sessions;
pub mod net;
pub mod shortcut;
pub mod single_instance;
+pub mod smtc;
pub mod sound;
pub mod toast;
pub mod tray;
diff --git a/desktop/app/src/win/smtc.rs b/desktop/app/src/win/smtc.rs
new file mode 100644
index 0000000..125ca57
--- /dev/null
+++ b/desktop/app/src/win/smtc.rs
@@ -0,0 +1,288 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//! What plays on a phone, in Windows' own media controls: the flyout by the
+//! volume and Quick Settings, the lock screen, and the keyboard's media
+//! keys (System Media Transport Controls).
+//!
+//! A desktop app gets these controls for one of its windows, so a hidden
+//! window on its own thread owns them. Updates reach that thread through a
+//! queue and a window message; button presses come back on Windows' threads.
+
+use std::{
+ path::PathBuf,
+ sync::{
+ Mutex, MutexGuard, OnceLock,
+ atomic::{AtomicIsize, Ordering},
+ mpsc,
+ },
+ time::SystemTime,
+};
+
+use nectarlink_core::{DeviceId, MediaAction};
+use windows::{
+ Foundation::{TimeSpan, TypedEventHandler},
+ Media::{
+ MediaPlaybackStatus, MediaPlaybackType, PlaybackPositionChangeRequestedEventArgs,
+ SystemMediaTransportControls, SystemMediaTransportControlsButton,
+ SystemMediaTransportControlsButtonPressedEventArgs, SystemMediaTransportControlsTimelineProperties,
+ },
+ Storage::Streams::{DataWriter, InMemoryRandomAccessStream, RandomAccessStreamReference},
+ Win32::{
+ Foundation::{HWND, LPARAM, LRESULT, WPARAM},
+ System::{
+ Com::{COINIT_MULTITHREADED, CoInitializeEx},
+ LibraryLoader::GetModuleHandleW,
+ WinRT::ISystemMediaTransportControlsInterop,
+ },
+ UI::WindowsAndMessaging::{
+ CreateWindowExW, DefWindowProcW, DispatchMessageW, GetMessageW, MSG, PostMessageW,
+ RegisterClassExW, WINDOW_EX_STYLE, WM_APP, WNDCLASSEXW, WS_OVERLAPPED,
+ },
+ },
+ core::{HSTRING, w},
+};
+
+use crate::state::PlayerView;
+
+/// What the controls show: one player on one device.
+#[derive(Debug, Clone)]
+pub struct Shown {
+ pub device: DeviceId,
+ pub device_name: String,
+ pub view: PlayerView,
+}
+
+type OnCommand = Box<dyn Fn(DeviceId, String, MediaAction, Option<u64>) + Send + Sync>;
+
+static WINDOW: AtomicIsize = AtomicIsize::new(0);
+static QUEUE: OnceLock<Mutex<mpsc::Sender<Option<Shown>>>> = OnceLock::new();
+static ON_COMMAND: OnceLock<OnCommand> = OnceLock::new();
+/// The player the controls stand for, for button presses.
+static CURRENT: Mutex<Option<(DeviceId, String)>> = Mutex::new(None);
+
+const WM_UPDATE: u32 = WM_APP + 1;
+
+fn lock<T>(m: &Mutex<T>) -> MutexGuard<'_, T> {
+ m.lock().unwrap_or_else(|e| e.into_inner())
+}
+
+/// Starts the controls (hidden until something plays); `on_command` gets
+/// what the user pressed, on a Windows thread.
+pub fn start(on_command: impl Fn(DeviceId, String, MediaAction, Option<u64>) + Send + Sync + 'static) {
+ if ON_COMMAND.set(Box::new(on_command)).is_err() {
+ return;
+ }
+ let (sender, updates) = mpsc::channel();
+ let _ = QUEUE.set(Mutex::new(sender));
+ let spawned = std::thread::Builder::new().name("media-controls".into()).spawn(move || {
+ if let Err(e) = run(updates) {
+ tracing::warn!(error = %e, "Windows media controls are unavailable");
+ }
+ });
+ if let Err(e) = spawned {
+ tracing::warn!(error = %e, "Windows media controls are unavailable");
+ }
+}
+
+/// Shows a player in the controls, or hides them.
+pub fn show(current: Option<Shown>) {
+ let Some(queue) = QUEUE.get() else { return };
+ if lock(queue).send(current).is_err() {
+ return;
+ }
+ let window = WINDOW.load(Ordering::Acquire);
+ if window != 0 {
+ // SAFETY: posting to our own window; it drains the queue.
+ unsafe {
+ let _ = PostMessageW(Some(HWND(window as *mut _)), WM_UPDATE, WPARAM(0), LPARAM(0));
+ }
+ }
+}
+
+struct Controls {
+ smtc: SystemMediaTransportControls,
+ /// The artwork shown, so it's only loaded when it changes.
+ art: Option<PathBuf>,
+}
+
+static CONTROLS: OnceLock<Mutex<Option<Controls>>> = OnceLock::new();
+static UPDATES: OnceLock<Mutex<mpsc::Receiver<Option<Shown>>>> = OnceLock::new();
+
+fn run(updates: mpsc::Receiver<Option<Shown>>) -> windows::core::Result<()> {
+ // SAFETY: COM for this thread, which lives as long as the app.
+ unsafe { CoInitializeEx(None, COINIT_MULTITHREADED) }.ok()?;
+ // SAFETY: class registration and a window that's never shown; the
+ // window procedure only touches statics.
+ let hwnd = unsafe {
+ let instance = GetModuleHandleW(None)?;
+ let class = WNDCLASSEXW {
+ cbSize: std::mem::size_of::<WNDCLASSEXW>() as u32,
+ lpfnWndProc: Some(window_proc),
+ hInstance: instance.into(),
+ lpszClassName: w!("Nectarlink.Media"),
+ ..Default::default()
+ };
+ RegisterClassExW(&class);
+ CreateWindowExW(
+ WINDOW_EX_STYLE(0),
+ w!("Nectarlink.Media"),
+ w!("Nectarlink media"),
+ WS_OVERLAPPED,
+ 0,
+ 0,
+ 0,
+ 0,
+ None,
+ None,
+ Some(instance.into()),
+ None,
+ )?
+ };
+ let interop =
+ windows::core::factory::<SystemMediaTransportControls, ISystemMediaTransportControlsInterop>()?;
+ // SAFETY: the window is ours and alive for the app's lifetime.
+ let smtc: SystemMediaTransportControls = unsafe { interop.GetForWindow(hwnd)? };
+ smtc.SetIsEnabled(false)?;
+ smtc.ButtonPressed(&TypedEventHandler::<
+ SystemMediaTransportControls,
+ SystemMediaTransportControlsButtonPressedEventArgs,
+ >::new(|_, args| {
+ let Some(args) = args.as_ref() else { return Ok(()) };
+ let action = match args.Button()? {
+ SystemMediaTransportControlsButton::Play => MediaAction::Play,
+ SystemMediaTransportControlsButton::Pause => MediaAction::Pause,
+ SystemMediaTransportControlsButton::Next => MediaAction::Next,
+ SystemMediaTransportControlsButton::Previous => MediaAction::Previous,
+ _ => return Ok(()),
+ };
+ command(action, None);
+ Ok(())
+ }))?;
+ smtc.PlaybackPositionChangeRequested(&TypedEventHandler::<
+ SystemMediaTransportControls,
+ PlaybackPositionChangeRequestedEventArgs,
+ >::new(|_, args| {
+ if let Some(args) = args.as_ref() {
+ let ticks = args.RequestedPlaybackPosition()?.Duration.max(0);
+ command(MediaAction::Seek, Some(u64::try_from(ticks / 10_000).unwrap_or(0)));
+ }
+ Ok(())
+ }))?;
+ let _ = CONTROLS.set(Mutex::new(Some(Controls { smtc, art: None })));
+ let _ = UPDATES.set(Mutex::new(updates));
+ WINDOW.store(hwnd.0 as isize, Ordering::Release);
+ // Anything queued before the window existed.
+ apply_updates();
+
+ let mut msg = MSG::default();
+ // SAFETY: a plain message loop for this thread's window.
+ unsafe {
+ while GetMessageW(&mut msg, None, 0, 0).as_bool() {
+ DispatchMessageW(&msg);
+ }
+ }
+ Ok(())
+}
+
+fn command(action: MediaAction, position: Option<u64>) {
+ let Some((device, player)) = lock(&CURRENT).clone() else { return };
+ if let Some(on_command) = ON_COMMAND.get() {
+ on_command(device, player, action, position);
+ }
+}
+
+unsafe extern "system" fn window_proc(hwnd: HWND, msg: u32, wparam: WPARAM, lparam: LPARAM) -> LRESULT {
+ if msg == WM_UPDATE {
+ apply_updates();
+ return LRESULT(0);
+ }
+ // SAFETY: default handling for everything else.
+ unsafe { DefWindowProcW(hwnd, msg, wparam, lparam) }
+}
+
+/// Applies the newest queued update (older ones are already out of date).
+fn apply_updates() {
+ let Some(updates) = UPDATES.get() else { return };
+ let Some(latest) = lock(updates).try_iter().last() else { return };
+ let Some(controls) = CONTROLS.get() else { return };
+ let mut controls = lock(controls);
+ let Some(controls) = controls.as_mut() else { return };
+ if let Err(e) = apply(controls, latest) {
+ tracing::debug!(error = %e, "can't update the media controls");
+ }
+}
+
+fn apply(controls: &mut Controls, shown: Option<Shown>) -> windows::core::Result<()> {
+ let smtc = &controls.smtc;
+ let Some(shown) = shown else {
+ *lock(&CURRENT) = None;
+ smtc.SetIsEnabled(false)?;
+ smtc.DisplayUpdater()?.ClearAll()?;
+ controls.art = None;
+ return Ok(());
+ };
+ let player = &shown.view.player;
+ *lock(&CURRENT) = Some((shown.device, player.id.clone()));
+ let can = |action: &str| player.actions.iter().any(|a| a == action);
+
+ smtc.SetIsEnabled(true)?;
+ smtc.SetIsPlayEnabled(can("play"))?;
+ smtc.SetIsPauseEnabled(can("pause"))?;
+ smtc.SetIsNextEnabled(can("next"))?;
+ smtc.SetIsPreviousEnabled(can("previous"))?;
+ smtc.SetPlaybackStatus(if player.playing {
+ MediaPlaybackStatus::Playing
+ } else {
+ MediaPlaybackStatus::Paused
+ })?;
+
+ let updater = smtc.DisplayUpdater()?;
+ updater.SetType(MediaPlaybackType::Music)?;
+ let music = updater.MusicProperties()?;
+ music.SetTitle(&HSTRING::from(player.title.as_deref().unwrap_or(&player.app)))?;
+ // The flyout shows two lines; the second says where it plays.
+ let source = format!("{} on {}", player.app, shown.device_name);
+ let artist = match &player.artist {
+ Some(artist) => format!("{artist} · {source}"),
+ None => source,
+ };
+ music.SetArtist(&HSTRING::from(artist))?;
+ music.SetAlbumTitle(&HSTRING::from(player.album.as_deref().unwrap_or_default()))?;
+ if controls.art != shown.view.art {
+ match shown.view.art.as_deref().map(std::fs::read) {
+ Some(Ok(bytes)) => updater.SetThumbnail(&thumbnail(&bytes)?)?,
+ _ => updater.SetThumbnail(None)?,
+ }
+ controls.art = shown.view.art.clone();
+ }
+ updater.Update()?;
+
+ if let Some(duration) = player.duration {
+ let elapsed = if player.playing {
+ SystemTime::now().duration_since(shown.view.at).map_or(0, |d| d.as_millis() as u64)
+ } else {
+ 0
+ };
+ let position = player.position.unwrap_or(0).saturating_add(elapsed).min(duration);
+ let span = |ms: u64| TimeSpan { Duration: i64::try_from(ms).unwrap_or(i64::MAX / 10_000) * 10_000 };
+ let timeline = SystemMediaTransportControlsTimelineProperties::new()?;
+ timeline.SetStartTime(span(0))?;
+ timeline.SetEndTime(span(duration))?;
+ timeline.SetMinSeekTime(span(0))?;
+ timeline.SetMaxSeekTime(span(if can("seek") { duration } else { 0 }))?;
+ timeline.SetPosition(span(position))?;
+ smtc.UpdateTimelineProperties(&timeline)?;
+ }
+ Ok(())
+}
+
+/// Artwork as a stream reference, which the controls take for thumbnails.
+fn thumbnail(bytes: &[u8]) -> windows::core::Result<RandomAccessStreamReference> {
+ let stream = InMemoryRandomAccessStream::new()?;
+ let writer = DataWriter::CreateDataWriter(&stream)?;
+ writer.WriteBytes(bytes)?;
+ writer.StoreAsync()?.join()?;
+ writer.FlushAsync()?.join()?;
+ writer.DetachStream()?;
+ stream.Seek(0)?;
+ RandomAccessStreamReference::CreateFromStream(&stream)
+}
diff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.md
index f988939..541f13e 100644
--- a/docs/protocol/capabilities.md
+++ b/docs/protocol/capabilities.md
@@ -23,7 +23,8 @@ capability is missing.
| `device.ring` | Rings on `device.ring`, even on silent | always offered |
| `files.transfer` | Sends and receives files ([files service](files.md)) | app update |
| `clip.image` | Accepts images on its clipboard ([clipboard service](clipboard.md)) | app update |
-| `media.control` | Exposes its media sessions for remote control (phone: needs notification access) | phone: notification access · desktop: app update |
+| `media.control` | Shares its media players for remote control ([media service](media.md); phone: needs notification access) | phone: notification access · desktop: app update |
+| `media.remote` | Shows and controls other devices' media players | app update |
## Phone
diff --git a/docs/protocol/media.md b/docs/protocol/media.md
new file mode 100644
index 0000000..cf2a752
--- /dev/null
+++ b/docs/protocol/media.md
@@ -0,0 +1,78 @@
+# Media service (protocol v0)
+
+> Status: **draft.** Builds on [protocol v0](v0.md); breaking changes are
+> allowed until v1. License: CC BY 4.0.
+
+What plays on one device shows on the other, which can control it: a
+phone's music in the PC's app and in Windows' media flyout, a PC's in the
+phone's media controls. Both directions use the same messages.
+
+## 1. Capabilities
+
+| ID | Offered by | Meaning |
+|---|---|---|
+| `media.control` | both | Shares its players and takes commands for them (a phone needs notification access to read other apps' players) |
+| `media.remote` | both | Shows other devices' players and sends commands to them |
+
+A device sends its players only to devices that offer `media.remote`, and
+only while the user allows media for that device (the `media` device
+toggle, on by default). It takes commands only while it offers
+`media.control` and allows the sender.
+
+## 2. Messages
+
+All on the control stream.
+
+```
+t = "media.state" b = { players: [Player] } // no reply
+t = "media.sync" b = {} // no reply
+t = "media.command" id = n b = { player: text, action: text, position?: uint }
+t = "ok" re = n
+```
+
+**`media.state`** carries every player the sender shares, the most
+relevant first (what Windows calls the current session, or what's playing,
+then the most recently active). It's sent when a session starts, after
+every change, and again when the user changes what the other device may
+see. An empty list means nothing plays (or the user stopped sharing); the
+receiver clears what it showed. A receiver also clears a device's players
+when the connection ends.
+
+**`media.sync`** asks for a `media.state` now (the user allowed media again
+on the receiving side).
+
+**`media.command`** asks a player to `play`, `pause`, go to the `next` or
+`previous` item, or `seek` to `position` (milliseconds). Errors:
+`NOT_FOUND` when the player is gone, `UNSUPPORTED` when it can't do that,
+`DENIED` when the user turned media off for the sender, `BAD_MESSAGE` for a
+seek without a position.
+
+```
+Player = {
+ id: text, // 1–256 bytes, stable while the player exists
+ app: text, // user-visible app name
+ title?: text, // each text up to 256 characters
+ artist?: text,
+ album?: text,
+ playing: bool,
+ duration?: uint, // milliseconds
+ position?: uint, // milliseconds, when sent
+ actions: [text], // what it takes: play, pause, next, previous, seek
+ art_key?: text, // 1–64 bytes; the same key means the same picture
+ art?: bytes, // JPEG or PNG, up to 256 KiB
+}
+```
+
+While `playing`, the position moves on in real time from when the state
+arrived; senders don't send updates just because time passes.
+
+Artwork goes once per `art_key` and session: later states carry the key
+alone, and receivers keep pictures by key. A device sends at most 8
+players. Receivers drop actions they don't know.
+
+## 3. Rules
+
+- **No echoes.** A device leaves its own stand-in for another device's
+ player (Windows' flyout session, Android's media session) out of what it
+ shares.
+- Implementations **MUST NOT** log titles, artists or albums (v0 §11).
diff --git a/docs/protocol/v0.md b/docs/protocol/v0.md
index 08a766b..02bdf8f 100644
--- a/docs/protocol/v0.md
+++ b/docs/protocol/v0.md
@@ -8,7 +8,8 @@ connect and exchange messages. Version 0 covers the Phase 0 scope: identity,
pairing, the session handshake, capabilities, heartbeat and the first events
(device info, battery, ping). Feature services are specified in separate
documents that build on this one: [notifications](notifications.md),
-[clipboard](clipboard.md) and [files](files.md) so far; mirroring follows.
+[clipboard](clipboard.md), [files](files.md) and [media](media.md) so far;
+mirroring follows.
Key words **MUST**, **SHOULD** and **MAY** are used as in RFC 2119.