Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.
Copy images between PC and phone
Bee1d9bf192aacf
diff --git a/README.md b/README.mdindex 83bf38e..e177a52 100644--- a/README.md+++ b/README.md@@ -74,10 +74,10 @@ place and tested; the user-facing features of Phase 1 come next.a feed in the app, with the app's icon. Reply inline, run their actions(such as "Mark as read") or dismiss them from the PC; what's cleared on oneside is cleared on the other.-- **Clipboard**: text you copy on the PC is ready to paste on the phone. From- the phone, send it with the Quick Settings tile, the share sheet or a button- in the app (Android lets only the app in front read the clipboard).- Passwords that password managers mark private are never sent.+- **Clipboard**: text and images you copy on the PC are ready to paste on the+ phone. From the phone, send them with the Quick Settings tile, the share+ sheet or a button in the app (Android lets only the app in front read the+ clipboard). Passwords that password managers mark private are never sent.- **Files** in both directions, at Wi-Fi speed. On the PC, drop files on thewindow, pick them, or right-click them in File Explorer and choose**Send to** and your phone (a notification follows the transfer); on the@@ -95,8 +95,8 @@ place and tested; the user-facing features of Phase 1 come next.connection service, and an identity key protected by the Android Keystore.- **Command-line client** for development and testing (`nectarlink`).-**Next (Phase 1)**: images on the clipboard, media controls, notification-filters and history, and a Windows installer. Mirroring, messages, calls and the other+**Next (Phase 1)**: media controls, notification filters and history, and a+Windows installer. Mirroring, messages, calls and the otherfeatures in the plan follow in later phases.## Screenshotsdiff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xmlindex ad071b3..d49bd3a 100644--- a/android/app/src/main/AndroidManifest.xml+++ b/android/app/src/main/AndroidManifest.xml@@ -86,7 +86,8 @@android:name=".files.TransferNotifications$CancelReceiver"android:exported="false" />- <!-- Opening received files on Android 8–9 (later versions use MediaStore). -->+ <!-- Opening received files on Android 8–9 (later versions use MediaStore),+ and images from a PC on the clipboard. --><providerandroid:name="androidx.core.content.FileProvider"android:authorities="${applicationId}.files"diff --git a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.ktindex ad69fe9..9b36542 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt@@ -5,33 +5,114 @@ import android.content.ClipDataimport android.content.ClipDescriptionimport android.content.ClipboardManagerimport android.content.Context+import android.graphics.Bitmap+import android.graphics.BitmapFactory+import android.net.Uriimport android.os.Build+import android.util.Log+import androidx.core.content.FileProvider+import java.io.File/*** The phone's clipboard. Android lets an app read it only while that app* is in front (and focused); writing works any time.*/object PhoneClipboard {+ private const val TAG = "PhoneClipboard"++ /** The largest image sent (docs/protocol/clipboard.md). */+ private const val MAX_IMAGE_BYTES = 32L * 1024 * 1024++ /** Where images from a PC are kept while they're on the clipboard. */+ private const val IMAGE_DIR = "clipboard"+sealed interface Read {data class Text(val text: String) : Read+ /** An image, still to be read with [readImage] (the app holds access to it for now). */+ data class Image(val uri: Uri) : Read/** Marked sensitive by the app it came from (a password manager): never sent. */data object Private : Readdata object Empty : Read}+ /** An image ready to send: PNG or JPEG, as the protocol allows. */+ class ImageData(val mime: String, val bytes: ByteArray)++ sealed interface ImageResult {+ class Ready(val image: ImageData) : ImageResult+ data object TooLarge : ImageResult+ data object Unreadable : ImageResult+ }+fun read(context: Context): Read {val clip = manager(context)?.primaryClip ?: return Read.Emptyif (clip.description.isSensitive()) return Read.Private+ val resolver = context.contentResolver+ for (i in 0 until clip.itemCount) {+ val item = clip.getItemAt(i)+ item.text?.toString()?.takeIf(String::isNotBlank)?.let { return Read.Text(it) }+ // Checked before coercing to text, which would turn it into its address.+ val uri = item.uri+ if (uri != null && resolver.getType(uri)?.startsWith("image/") == true) return Read.Image(uri)+ }val text = (0 until clip.itemCount).firstNotNullOfOrNull { clip.getItemAt(it).coerceToText(context)?.toString()?.takeIf(String::isNotBlank) }return text?.let { Read.Text(it) } ?: Read.Empty}+ /**+ * Reads a copied image for sending (blocking; call off the main thread).+ * PNG and JPEG go as they are; other formats are converted to PNG.+ */+ fun readImage(context: Context, uri: Uri): ImageResult = runCatching {+ val resolver = context.contentResolver+ val size = resolver.openAssetFileDescriptor(uri, "r")?.use { it.length } ?: -1+ if (size > MAX_IMAGE_BYTES) return ImageResult.TooLarge+ val bytes = resolver.openInputStream(uri)?.use { input ->+ // Read one byte past the limit to notice a source that didn't say its size.+ val buffer = input.readNBytesCompat(MAX_IMAGE_BYTES.toInt() + 1)+ if (buffer.size > MAX_IMAGE_BYTES) return ImageResult.TooLarge+ buffer+ } ?: return ImageResult.Unreadable+ when (val mime = resolver.getType(uri)) {+ "image/png", "image/jpeg" -> ImageResult.Ready(ImageData(mime, bytes))+ else -> {+ val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size) ?: return ImageResult.Unreadable+ val png = java.io.ByteArrayOutputStream().also { bitmap.compress(Bitmap.CompressFormat.PNG, 100, it) }+ bitmap.recycle()+ if (png.size() > MAX_IMAGE_BYTES) ImageResult.TooLarge else ImageResult.Ready(ImageData("image/png", png.toByteArray()))+ }+ }+ }.getOrElse {+ Log.w(TAG, "can't read the copied image", it)+ ImageResult.Unreadable+ }+/** Puts text a PC sent on the clipboard. */fun write(context: Context, text: String): Boolean =runCatching { manager(context)?.setPrimaryClip(ClipData.newPlainText("Nectarlink", text)) != null }.getOrDefault(false)+ /**+ * Puts an image a PC sent on the clipboard: kept in the app's cache and+ * shared through its FileProvider, so the app that pastes can read it.+ * The previous one is deleted; only the latest is on the clipboard.+ */+ fun writeImage(context: Context, mime: String, bytes: ByteArray): Boolean = runCatching {+ val dir = File(context.cacheDir, IMAGE_DIR).apply { mkdirs() }+ dir.listFiles()?.forEach { it.delete() }+ val extension = if (mime == "image/jpeg") "jpg" else "png"+ // A new name each time: apps cache what they read by address.+ val file = File(dir, "Image ${System.currentTimeMillis()}.$extension")+ file.writeBytes(bytes)+ val uri = FileProvider.getUriForFile(context, "${context.packageName}.files", file)+ val clip = ClipData.newUri(context.contentResolver, "Image", uri)+ manager(context)?.setPrimaryClip(clip) != null+ }.getOrElse {+ Log.w(TAG, "can't put an image on the clipboard", it)+ false+ }+private fun manager(context: Context) = context.getSystemService(ClipboardManager::class.java)private fun ClipDescription.isSensitive(): Boolean {@@ -42,4 +123,16 @@ object PhoneClipboard {}return extras?.getBoolean(key) == true}++ /** `readNBytes`, which needs Android 13. */+ private fun java.io.InputStream.readNBytesCompat(limit: Int): ByteArray {+ val out = java.io.ByteArrayOutputStream()+ val buffer = ByteArray(64 * 1024)+ while (out.size() < limit) {+ val read = read(buffer, 0, minOf(buffer.size, limit - out.size()))+ if (read < 0) break+ out.write(buffer, 0, read)+ }+ return out.toByteArray()+ }}diff --git a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.ktindex 38fbf83..3bd2633 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt@@ -75,6 +75,7 @@ class SendActivity : Activity() {if (!hasFocus || sent || intent?.action != ACTION_SEND_CLIPBOARD) returnwhen (val clip = PhoneClipboard.read(this)) {is PhoneClipboard.Read.Text -> send(clip.text)+ is PhoneClipboard.Read.Image -> sendImage(clip.uri)PhoneClipboard.Read.Private -> finishWith(getString(R.string.clip_private))PhoneClipboard.Read.Empty -> finishWith(getString(R.string.clip_empty))}@@ -86,6 +87,12 @@ class SendActivity : Activity() {scope.launch { finishWith(core.sendClipboard(text)) }}+ private fun sendImage(uri: Uri) {+ sent = true+ val core = (application as NectarlinkApplication).core+ scope.launch { finishWith(core.sendClipboardImage(uri)) }+ }+private fun finishWith(message: String) {sent = trueToast.makeText(applicationContext, message, Toast.LENGTH_SHORT).show()diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.ktindex b367f09..abe4650 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt@@ -7,6 +7,7 @@ import android.service.notification.NotificationListenerServiceimport android.util.Logimport android.net.Uriimport app.nectarlink.android.BuildConfig+import app.nectarlink.android.clipboard.PhoneClipboardimport app.nectarlink.android.files.OutgoingFilesimport app.nectarlink.android.files.ReceivedFilesimport app.nectarlink.android.files.TransferNotifications@@ -37,6 +38,7 @@ import kotlinx.coroutines.flow.asSharedFlowimport kotlinx.coroutines.flow.asStateFlowimport kotlinx.coroutines.flow.updateimport kotlinx.coroutines.launch+import kotlinx.coroutines.withContext/*** The app's single connection to `nectarlink-core`. Owns the node, folds@@ -208,7 +210,22 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener* Sends text to every connected PC; returns what to tell the user.* Waits for the core when the app was just launched to send.*/- suspend fun sendClipboard(text: String): String {+ suspend fun sendClipboard(text: String): String =+ sendToPcs(image = false) { node, pc -> node.sendClipboard(pc, text) }++ /** Sends a copied image to the connected PCs; returns what to tell the user. */+ suspend fun sendClipboardImage(uri: Uri): String {+ startJob?.join()+ if (_state.value.devices.none { it.online }) return context.getString(R.string.clip_no_pc)+ val image = when (val read = withContext(Dispatchers.IO) { PhoneClipboard.readImage(context, uri) }) {+ is PhoneClipboard.ImageResult.Ready -> read.image+ PhoneClipboard.ImageResult.TooLarge -> return context.getString(R.string.clip_image_too_large)+ PhoneClipboard.ImageResult.Unreadable -> return context.getString(R.string.clip_image_unreadable)+ }+ return sendToPcs(image = true) { node, pc -> node.sendClipboardImage(pc, image.mime, image.bytes) }+ }++ private suspend fun sendToPcs(image: Boolean, send: suspend (NectarlinkNode, String) -> Unit): String {startJob?.join()val node = node ?: return context.getString(R.string.clip_no_pc)val pcs = _state.value.devices.filter { it.online }@@ -217,12 +234,16 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListenervar failure: String? = nullfor (pc in pcs) {try {- node.sendClipboard(pc.id, text)+ send(node, pc.id)sentTo += pc.name} catch (e: NectarlinkException) {failure = when (e) {is NectarlinkException.Denied -> context.getString(R.string.clip_off_for, pc.name)- is NectarlinkException.TooLarge -> context.getString(R.string.clip_too_large)+ is NectarlinkException.TooLarge -> context.getString(+ if (image) R.string.clip_image_too_large else R.string.clip_too_large,+ )+ is NectarlinkException.Unsupported ->+ if (image) context.getString(R.string.clip_image_unsupported, pc.name) else describe(e)else -> describe(e)}}diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.ktindex 89664cd..db5805e 100644--- a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt@@ -26,6 +26,9 @@ internal class PhonePlatform(context: Context, private val ringer: Ringer) : Plaoverride fun setClipboard(text: String): Boolean = PhoneClipboard.write(context, text)+ override fun setClipboardImage(mime: String, bytes: ByteArray): Boolean =+ PhoneClipboard.writeImage(context, mime, bytes)+override fun dismissNotification(key: String) {val listener = NotificationListener.instance ?: throw NotificationFailure.Unsupported()listener.dismiss(key)diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xmlindex 525d164..535600b 100644--- a/android/app/src/main/res/values/strings.xml+++ b/android/app/src/main/res/values/strings.xml@@ -81,7 +81,10 @@<string name="clip_off_for">The clipboard is turned off for %1$s.</string><string name="clip_too_large">That’s too much text to send at once.</string><string name="clip_private">What you copied is marked private by its app, so it wasn’t sent.</string>- <string name="clip_empty">There’s no text on the clipboard.</string>+ <string name="clip_empty">There’s no text or image on the clipboard.</string>+ <string name="clip_image_too_large">That image is too large to send.</string>+ <string name="clip_image_unreadable">That image can’t be read.</string>+ <string name="clip_image_unsupported">Update Nectarlink on %1$s to send it images.</string><string name="clip_nothing_shared">There’s no text to send.</string><string name="background_title">Stay reachable in the background</string><string name="background_text">Your phone pauses Nectarlink to save battery, so your PC can’t reach it, and replies and dismissals from the PC time out. Let it run in the background.</string>diff --git a/android/app/src/main/res/xml/file_paths.xml b/android/app/src/main/res/xml/file_paths.xmlindex f9fad49..905e132 100644--- a/android/app/src/main/res/xml/file_paths.xml+++ b/android/app/src/main/res/xml/file_paths.xml@@ -1,6 +1,8 @@<?xml version="1.0" encoding="utf-8"?><!-- SPDX-License-Identifier: GPL-3.0-or-later -->-<!-- Received files on Android 8–9 (see files/ReceivedFiles.kt). -->+<!-- Received files on Android 8–9 (see files/ReceivedFiles.kt), and images+ a PC put on the clipboard (see clipboard/PhoneClipboard.kt). --><paths><external-files-path name="received" path="Download/Nectarlink/" />+ <cache-path name="clipboard" path="clipboard/" /></paths>diff --git a/core/nectarlink-cli/src/main.rs b/core/nectarlink-cli/src/main.rsindex be6d4b1..f6df376 100644--- a/core/nectarlink-cli/src/main.rs+++ b/core/nectarlink-cli/src/main.rs@@ -117,8 +117,15 @@ enum Command {#[arg(required = true)]files: Vec<PathBuf>,},- /// Put text on a paired device's clipboard.- Clip { device: String, text: String },+ /// Put text, or an image with --image, on a paired device's clipboard.+ Clip {+ device: String,+ #[arg(required_unless_present = "image", conflicts_with = "image")]+ text: Option<String>,+ /// A PNG or JPEG file to put on the clipboard instead of text.+ #[arg(long)]+ image: Option<PathBuf>,+ },/// Dismiss a phone's notification (`run` prints the keys).Dismiss { device: String, key: String },/// Run an action of a phone's notification, or reply to it.@@ -176,6 +183,13 @@ impl Platform for TerminalPlatform {println!("Clipboard from a paired device: {text}");Ok(())}+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {+ let extension = if mime == "image/jpeg" { "jpg" } else { "png" };+ let path = std::env::temp_dir().join(format!("nectarlink-clipboard.{extension}"));+ std::fs::write(&path, bytes).map_err(|e| e.to_string())?;+ println!("Clipboard image from a paired device ({} bytes): {}", bytes.len(), path.display());+ Ok(())+ }fn dismiss_notification(&self, key: &str) -> Result<(), NotificationError> {println!("The PC dismissed {key}");Ok(())@@ -295,10 +309,25 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {wait_until_online(node, id).await?;send_files(node, id, files).await?;}- Command::Clip { device, text } => {+ Command::Clip { device, text, image } => {let id = resolve(node, device)?;wait_until_online(node, id).await?;- node.send_clipboard(id, text.clone()).await.context("clipboard failed")?;+ match (text, image) {+ (_, Some(path)) => {+ let mime = match path.extension().and_then(|e| e.to_str()).map(str::to_ascii_lowercase) {+ Some(e) if e == "png" => "image/png",+ Some(e) if e == "jpg" || e == "jpeg" => "image/jpeg",+ _ => anyhow::bail!("only PNG and JPEG images can go on the clipboard"),+ };+ let bytes =+ std::fs::read(path).with_context(|| format!("can't read {}", path.display()))?;+ node.send_clipboard_image(id, mime.into(), bytes).await.context("clipboard failed")?;+ }+ (Some(text), None) => {+ node.send_clipboard(id, text.clone()).await.context("clipboard failed")?+ }+ (None, None) => unreachable!("clap requires text or --image"),+ }println!("Sent.");}Command::Dismiss { device, key } => {diff --git a/core/nectarlink-core/src/clipboard.rs b/core/nectarlink-core/src/clipboard.rsindex 85cf8a6..1ca12e2 100644--- a/core/nectarlink-core/src/clipboard.rs+++ b/core/nectarlink-core/src/clipboard.rs@@ -1,15 +1,24 @@// SPDX-License-Identifier: MPL-2.0-//! The clipboard service (docs/protocol/clipboard.md): copied text sent to-//! a paired device, which puts it on its own clipboard.+//! The clipboard service (docs/protocol/clipboard.md): copied text and+//! images sent to a paired device, which puts them on its own clipboard.+//! Text goes on the control stream; images on a stream of their own.-use std::sync::Arc;+use std::{sync::Arc, time::Duration};+use iroh::endpoint::{RecvStream, SendStream};use nectarlink_protocol::{- Envelope, ErrorCode,- messages::{ClipSet, types},+ DeviceId, Envelope, ErrorCode,+ messages::{ClipImage, ClipSet, StreamHeader, clip, types},+ read_frame, write_frame,};-use crate::{Result, events::NodeEvent, node::Shared, session::Session};+use crate::{Error, Result, error::net, events::NodeEvent, node::Shared, session::Session};++/// How long an image may take to arrive, or to be answered: generous, for+/// a large screenshot on a slow link.+const IMAGE_TIMEOUT: Duration = Duration::from_secs(60);+/// Remote capability for receiving images.+const IMAGE_CAPABILITY: &str = "clip.image";/// The device toggle that allows the clipboard for a device.pub(crate) const TOGGLE: &str = "clipboard";@@ -48,3 +57,99 @@ pub(crate) async fn handle(shared: &Arc<Shared>, session: &Arc<Session>, env: &Esession.send(reply.reply_to(env.id)).await?;Ok(true)}++/// Sends an image on its own stream and waits for the receiver's answer.+pub(crate) async fn send_image(+ shared: &Arc<Shared>,+ session: &Session,+ mime: String,+ bytes: Vec<u8>,+) -> Result<()> {+ let peer = session.peer;+ let image = ClipImage { mime, size: bytes.len() as u64 };+ if image.size > clip::MAX_IMAGE_BYTES {+ return Err(Error::TooLarge);+ }+ if !image.is_valid() {+ return Err(Error::Internal("not a PNG or JPEG image".into()));+ }+ if !shared.toggle_on(&peer, TOGGLE) {+ return Err(Error::Denied);+ }+ let takes_images = shared.store.get_peer(&peer)?.is_some_and(|p| p.caps.contains(IMAGE_CAPABILITY));+ if !takes_images {+ return Err(Error::Unsupported);+ }+ let exchange = async {+ let (mut send, mut recv) = session.conn.open_bi().await.map_err(net)?;+ let header = StreamHeader { svc: clip::SERVICE.into(), op: clip::OP_IMAGE.into(), v: clip::VERSION };+ write_frame(&mut send, &Envelope::new(types::STREAM, &header)?.to_cbor()).await?;+ write_frame(&mut send, &Envelope::new(clip::IMAGE, &image)?.to_cbor()).await?;+ // A receiver that refuses stops reading; its answer still comes.+ let written = send.write_all(&bytes).await;+ let _ = send.finish();+ let reply = read_frame(&mut recv).await?.ok_or(Error::Offline)?;+ Envelope::from_cbor(&reply)?.expect(types::OK)?;+ written.map_err(net)?;+ Ok(())+ };+ tokio::time::timeout(IMAGE_TIMEOUT, exchange).await.map_err(|_| Error::Timeout)?+}++/// Receives an image (the stream header was read) and puts it on the+/// clipboard.+pub(crate) async fn receive_image(+ shared: Arc<Shared>,+ peer: DeviceId,+ mut send: SendStream,+ mut recv: RecvStream,+) {+ let reply = match tokio::time::timeout(IMAGE_TIMEOUT, take_image(&shared, peer, &mut recv)).await {+ Ok(Ok(())) => Envelope::empty(types::OK),+ Ok(Err((code, msg))) => Envelope::error(code, msg),+ Err(_) => Envelope::error(ErrorCode::Internal, "the image took too long"),+ };+ if reply.t != types::OK {+ let _ = recv.stop(0u32.into());+ }+ let _ = write_frame(&mut send, &reply.to_cbor()).await;+ let _ = send.finish();+ let _ = send.stopped().await;+}++async fn take_image(+ shared: &Arc<Shared>,+ peer: DeviceId,+ recv: &mut RecvStream,+) -> std::result::Result<(), (ErrorCode, &'static str)> {+ let image = read_frame(recv)+ .await+ .ok()+ .flatten()+ .and_then(|bytes| Envelope::from_cbor(&bytes).ok())+ .and_then(|env| env.expect_body::<ClipImage>(clip::IMAGE).ok())+ .filter(ClipImage::is_valid)+ .ok_or((ErrorCode::BadMessage, "invalid image"))?;+ if !shared.toggle_on(&peer, TOGGLE) {+ return Err((ErrorCode::Denied, "the clipboard is off for this device"));+ }+ let size = usize::try_from(image.size).map_err(|_| (ErrorCode::BadMessage, "invalid image"))?;+ let mut bytes = vec![0u8; size];+ recv.read_exact(&mut bytes).await.map_err(|_| (ErrorCode::BadMessage, "the image ended early"))?;+ let platform = shared.platform.clone();+ let mime = image.mime;+ match tokio::task::spawn_blocking(move || platform.set_clipboard_image(&mime, &bytes)).await {+ Ok(Ok(())) => {+ shared.emit(NodeEvent::ClipboardReceived { device: peer });+ Ok(())+ }+ Ok(Err(reason)) => {+ tracing::warn!(reason, "can't put an image on the clipboard");+ Err((ErrorCode::Internal, "can't write the clipboard"))+ }+ Err(e) => {+ tracing::warn!(error = %e, "clipboard writer failed");+ Err((ErrorCode::Internal, "can't write the clipboard"))+ }+ }+}diff --git a/core/nectarlink-core/src/config.rs b/core/nectarlink-core/src/config.rsindex b71a31a..ecae6d6 100644--- a/core/nectarlink-core/src/config.rs+++ b/core/nectarlink-core/src/config.rs@@ -93,6 +93,12 @@ pub trait Platform: Send + Sync + 'static {fn set_clipboard(&self, _text: &str) -> Result<(), String> {Err("this device has no clipboard".into())}++ /// Put an image on this device's clipboard (a paired device sent it).+ /// `mime` is `image/png` or `image/jpeg`.+ fn set_clipboard_image(&self, _mime: &str, _bytes: &[u8]) -> Result<(), String> {+ Err("this device has no clipboard".into())+ }}/// A platform that does nothing; useful for tests and headless tools.diff --git a/core/nectarlink-core/src/lib.rs b/core/nectarlink-core/src/lib.rsindex 327c2ad..21f2998 100644--- a/core/nectarlink-core/src/lib.rs+++ b/core/nectarlink-core/src/lib.rs@@ -26,6 +26,9 @@ pub use events::{};pub use features::{CapabilityMatrix, FeatureState};pub use identity::{KeyProtector, PlainKeyProtector, default_protector};+pub use nectarlink_protocol::messages::clip::{+ IMAGE_TYPES as CLIP_IMAGE_TYPES, MAX_IMAGE_BYTES as CLIP_MAX_IMAGE_BYTES,+};pub use nectarlink_protocol::{DeviceId,messages::{diff --git a/core/nectarlink-core/src/node.rs b/core/nectarlink-core/src/node.rsindex a844199..0018be1 100644--- a/core/nectarlink-core/src/node.rs+++ b/core/nectarlink-core/src/node.rs@@ -53,7 +53,8 @@ const EVENT_CAPACITY: usize = 512;const IDLE_TIMEOUT: Duration = Duration::from_secs(8);/// Capabilities every build offers.-const BASE_CAPABILITIES: &[&str] = &["core.ping", "device.battery", "device.ring", "files.transfer"];+const BASE_CAPABILITIES: &[&str] =+ &["core.ping", "device.battery", "device.ring", "files.transfer", "clip.image"];/// This device's mutable description, sent to peers.#[derive(Debug, Clone)]@@ -858,6 +859,16 @@ impl Node {Ok(())}+ /// Puts an image on a paired device's clipboard (`image/png` or+ /// `image/jpeg`), when the user copied one. [`Error::Unsupported`] when+ /// the device's app can't take images, [`Error::Denied`] when the user+ /// turned the clipboard off for it, and [`Error::TooLarge`] beyond+ /// [`crate::CLIP_MAX_IMAGE_BYTES`].+ pub async fn send_clipboard_image(&self, peer: DeviceId, mime: String, bytes: Vec<u8>) -> Result<()> {+ let session = self.connected(&peer)?;+ crate::clipboard::send_image(&self.shared, &session, mime, bytes).await+ }+// ---- Notifications (docs/protocol/notifications.md) ----/// A notification appeared or changed on this phone; sent to everydiff --git a/core/nectarlink-core/src/transfer.rs b/core/nectarlink-core/src/transfer.rsindex cbd7c12..f2a0231 100644--- a/core/nectarlink-core/src/transfer.rs+++ b/core/nectarlink-core/src/transfer.rs@@ -13,7 +13,7 @@ use iroh::endpoint::{ReadError, RecvStream, SendStream, VarInt, WriteError};use nectarlink_protocol::{DeviceId, Envelope, ErrorCode,messages::{- ErrorBody, FileEntry, FilesAccept, FilesOffer, StreamHeader, files, is_valid_file_name, types,+ ErrorBody, FileEntry, FilesAccept, FilesOffer, StreamHeader, clip, files, is_valid_file_name, types,},read_frame, write_frame,};@@ -435,6 +435,9 @@ pub(crate) async fn accept_stream(Some(h) if h.svc == files::SERVICE && h.op == files::OP_SEND && h.v == files::VERSION => {receive(shared, session.peer, send, recv).await;}+ Some(h) if h.svc == clip::SERVICE && h.op == clip::OP_IMAGE && h.v == clip::VERSION => {+ crate::clipboard::receive_image(shared, session.peer, send, recv).await;+ }_ => {let reply = Envelope::error(ErrorCode::Unsupported, "unknown stream");let _ = write_frame(&mut send, &reply.to_cbor()).await;diff --git a/core/nectarlink-core/tests/two_devices.rs b/core/nectarlink-core/tests/two_devices.rsindex 203bc99..6dc8935 100644--- a/core/nectarlink-core/tests/two_devices.rs+++ b/core/nectarlink-core/tests/two_devices.rs@@ -26,6 +26,7 @@ struct RecordingPlatform {dismissed: Mutex<Vec<String>>,actions: Mutex<Vec<(String, String, Option<String>)>>,clipboard: Mutex<Vec<String>>,+ images: Mutex<Vec<(String, Vec<u8>)>>,}impl Platform for RecordingPlatform {@@ -55,6 +56,10 @@ impl Platform for RecordingPlatform {self.clipboard.lock().unwrap().push(text.to_owned());Ok(())}+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {+ self.images.lock().unwrap().push((mime.to_owned(), bytes.to_vec()));+ Ok(())+ }}struct TestDevice {@@ -585,6 +590,56 @@ async fn clipboard_text_goes_both_ways_with_consent() {assert!(matches!(pc.node.send_clipboard(phone_id, huge).await, Err(Error::TooLarge)));}+#[tokio::test(flavor = "multi_thread")]+async fn clipboard_images_go_both_ways_with_consent() {+ let mut pc = device("Desktop", DeviceKind::Desktop).await;+ let mut phone = device("Pixel", DeviceKind::Phone).await;+ pair_qr(&mut pc, &mut phone).await;+ let (pc_id, phone_id) = (pc.node.device_id(), phone.node.device_id());++ // A large screenshot, and a small photo the other way.+ let screenshot = data(9 * 1024 * 1024 + 17, 3);+ with_timeout(+ "pc to phone",+ pc.node.send_clipboard_image(phone_id, "image/png".into(), screenshot.clone()),+ )+ .await+ .expect("sent");+ assert_eq!(*phone.platform.images.lock().unwrap(), [("image/png".to_owned(), screenshot)]);+ wait_for(&mut phone, "clipboard received", |e| match e {+ NodeEvent::ClipboardReceived { device } if *device == pc_id => Some(()),+ _ => None,+ })+ .await;+ let photo = data(40_000, 4);+ with_timeout("phone to pc", phone.node.send_clipboard_image(pc_id, "image/jpeg".into(), photo.clone()))+ .await+ .expect("sent");+ assert_eq!(*pc.platform.images.lock().unwrap(), [("image/jpeg".to_owned(), photo)]);++ // Only PNG and JPEG, and not too large.+ let gif = pc.node.send_clipboard_image(phone_id, "image/gif".into(), vec![1; 10]).await;+ assert!(matches!(gif, Err(Error::Internal(_))), "{gif:?}");+ let max = usize::try_from(nectarlink_core::CLIP_MAX_IMAGE_BYTES).unwrap();+ let huge = pc.node.send_clipboard_image(phone_id, "image/png".into(), vec![0; max + 1]).await;+ assert!(matches!(huge, Err(Error::TooLarge)), "{huge:?}");++ // The receiver turned the clipboard off for the sender; the sender's+ // stream is refused and nothing lands.+ phone.node.set_device_toggle(pc_id, "clipboard", false).unwrap();+ let denied = with_timeout(+ "denied",+ pc.node.send_clipboard_image(phone_id, "image/png".into(), data(2_000_000, 5)),+ )+ .await;+ assert!(matches!(denied, Err(Error::Denied)), "{denied:?}");+ assert_eq!(phone.platform.images.lock().unwrap().len(), 1);+ // And the connection still works.+ with_timeout("still connected", pc.node.send_clipboard(phone_id, "x".into()))+ .await+ .expect_err("still off");+}+/// Deterministic, incompressible-looking test data.fn data(len: usize, seed: u64) -> Vec<u8> {let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;diff --git a/core/nectarlink-ffi/src/lib.rs b/core/nectarlink-ffi/src/lib.rsindex bf402b0..9ec5c5a 100644--- a/core/nectarlink-ffi/src/lib.rs+++ b/core/nectarlink-ffi/src/lib.rs@@ -726,6 +726,9 @@ pub trait Platform: Send + Sync {) -> Result<(), NotificationFailure>;/// A PC sent this text: put it on the clipboard. False if that failed.fn set_clipboard(&self, text: String) -> bool;+ /// A PC sent this image (`image/png` or `image/jpeg`): put it on the+ /// clipboard. False if that failed.+ fn set_clipboard_image(&self, mime: String, bytes: Vec<u8>) -> bool;}/// Encrypts the device key at rest (Android: a Keystore key).@@ -770,6 +773,13 @@ impl core::Platform for PlatformAdapter {fn set_clipboard(&self, text: &str) -> Result<(), String> {if self.0.set_clipboard(text.to_owned()) { Ok(()) } else { Err("the clipboard rejected it".into()) }}+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {+ if self.0.set_clipboard_image(mime.to_owned(), bytes.to_vec()) {+ Ok(())+ } else {+ Err("the clipboard rejected it".into())+ }+ }}struct KeyProtectorAdapter(Arc<dyn KeyProtector>);@@ -979,6 +989,14 @@ impl NectarlinkNode {self.run(async move { Ok(node.send_clipboard(id, text).await?) }).await}+ /// Puts an image (`image/png` or `image/jpeg`) on a paired PC's+ /// clipboard.+ pub async fn send_clipboard_image(&self, id: String, mime: String, bytes: Vec<u8>) -> Result<()> {+ let id = parse_id(&id)?;+ let node = self.node.clone();+ self.run(async move { Ok(node.send_clipboard_image(id, mime, bytes).await?) }).await+ }+// ---- Notifications ----/// A notification appeared or changed on this phone.diff --git a/core/nectarlink-protocol/src/messages.rs b/core/nectarlink-protocol/src/messages.rsindex a893d69..748ca30 100644--- a/core/nectarlink-protocol/src/messages.rs+++ b/core/nectarlink-protocol/src/messages.rs@@ -419,6 +419,33 @@ impl std::fmt::Debug for ClipSet {}}+/// Clipboard images, which travel on their own stream.+pub mod clip {+ pub const SERVICE: &str = "clip";+ pub const OP_IMAGE: &str = "image";+ pub const VERSION: u32 = 1;+ pub const IMAGE: &str = "clip.image";+ /// The largest image sent or accepted.+ pub const MAX_IMAGE_BYTES: u64 = 32 * 1024 * 1024;+ /// Image types a receiver accepts.+ pub const IMAGE_TYPES: &[&str] = &["image/png", "image/jpeg"];+}++/// Body of `clip.image`: what follows on the stream.+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+pub struct ClipImage {+ /// `image/png` or `image/jpeg`.+ pub mime: String,+ pub size: u64,+}++impl ClipImage {+ /// Whether the type and size are within the spec's limits.+ pub fn is_valid(&self) -> bool {+ clip::IMAGE_TYPES.contains(&self.mime.as_str()) && (1..=clip::MAX_IMAGE_BYTES).contains(&self.size)+ }+}+/// A paired device identity as stored in trust stores and exchanged in tests.#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]pub struct PeerIdentity {diff --git a/desktop/app/src/clipboard.rs b/desktop/app/src/clipboard.rsindex a74cfae..924dd31 100644--- a/desktop/app/src/clipboard.rs+++ b/desktop/app/src/clipboard.rs@@ -1,12 +1,13 @@// SPDX-License-Identifier: GPL-3.0-or-later//! The clipboard between this PC and paired phones-//! (docs/protocol/clipboard.md): what the user copies goes to connected-//! phones (unless turned off in Settings), "Send clipboard" sends it on-//! demand, and text from a phone lands on the Windows clipboard.+//! (docs/protocol/clipboard.md): what the user copies (text or an image)+//! goes to connected phones (unless turned off in Settings), "Send+//! clipboard" sends it on demand, and what a phone sends lands on the+//! Windows clipboard.use std::sync::atomic::{AtomicBool, Ordering};-use nectarlink_core::{DeviceId, Error, FeatureState, LinkState, NodeEvent};+use nectarlink_core::{CLIP_MAX_IMAGE_BYTES, DeviceId, Error, FeatureState, LinkState, Node, NodeEvent};use crate::{bridge::app::{describe, show_message},@@ -41,20 +42,56 @@ fn receivers() -> Vec<DeviceId> {})}-/// The user copied text on this PC.-fn on_copy(text: String) {+/// The user copied text or an image on this PC.+fn on_copy(clip: Clip) {if !AUTO_SEND.load(Ordering::Relaxed) {return;}let Some(node) = core_host::node() else { return };- for device in receivers() {- let (node, text) = (node.clone(), text.clone());- core_host::spawn(async move {+ let devices = receivers();+ if devices.is_empty() {+ return;+ }+ core_host::spawn(async move {+ let Some(content) = Content::from_clip(clip).await else { return };+ for device in devices {// Automatic: nothing to tell the user if a phone can't take it.- if let Err(e) = node.send_clipboard(device, text).await {+ if let Err(e) = content.send(&node, device).await {tracing::debug!(device = %device.short(), error = %e, "clipboard not sent");}- });+ }+ });+}++/// What goes to a phone.+enum Content {+ Text(String),+ Png(Vec<u8>),+}++impl Content {+ /// Text as is, an image as PNG (converted off the async threads).+ async fn from_clip(clip: Clip) -> Option<Content> {+ match clip {+ Clip::Text(text) => Some(Content::Text(text)),+ Clip::Image(image) => match tokio::task::spawn_blocking(move || image.to_png()).await {+ Ok(Ok(png)) => Some(Content::Png(png)),+ Ok(Err(reason)) => {+ tracing::warn!(reason, "can't read the copied image");+ None+ }+ Err(_) => None,+ },+ Clip::Private | Clip::Empty => None,+ }+ }++ async fn send(&self, node: &Node, device: DeviceId) -> Result<(), Error> {+ match self {+ Content::Text(text) => node.send_clipboard(device, text.clone()).await,+ Content::Png(png) if png.len() as u64 > CLIP_MAX_IMAGE_BYTES => Err(Error::TooLarge),+ Content::Png(png) => node.send_clipboard_image(device, "image/png".into(), png.clone()).await,+ }}}@@ -62,21 +99,26 @@ fn on_copy(text: String) {pub fn send_now(device: DeviceId) {let Some(node) = core_host::node() else { return };let name = core_host::host().hub.read(|s| s.name_of(&device)).unwrap_or_else(|| "the phone".into());- let text = match clipboard::read() {- Clip::Text(text) => text,+ let clip = clipboard::read();+ let image = matches!(clip, Clip::Image(_));+ match clip {Clip::Private => {- show_message("What you copied is marked private by its app, so it wasn't sent.");- return;- }- Clip::Empty => {- show_message("There's no text on the clipboard.");- return;+ return show_message("What you copied is marked private by its app, so it wasn't sent.");}- };+ Clip::Empty => return show_message("There's no text or image on the clipboard."),+ Clip::Text(_) | Clip::Image(_) => {}+ }core_host::spawn(async move {- match node.send_clipboard(device, text).await {+ let Some(content) = Content::from_clip(clip).await else {+ return show_message("That image can't be read.");+ };+ match content.send(&node, device).await {Ok(()) => show_message(format!("Sent to {name}.")),+ Err(Error::TooLarge) if image => show_message("That image is too large to send."),Err(Error::TooLarge) => show_message("That's too much text to send at once."),+ Err(Error::Unsupported) if image => {+ show_message(format!("Update Nectarlink on {name} to send it images."))+ }Err(Error::Denied) => show_message(format!("The clipboard is turned off for {name}.")),Err(e) => show_message(describe(&e)),}diff --git a/desktop/app/src/main.rs b/desktop/app/src/main.rsindex fc37fdd..ba07113 100644--- a/desktop/app/src/main.rs+++ b/desktop/app/src/main.rs@@ -59,6 +59,9 @@ impl Platform for DesktopPlatform {fn set_clipboard(&self, text: &str) -> Result<(), String> {win::clipboard::write(text)}+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {+ win::clipboard::write_image(mime, bytes)+ }}/// Command-line options (also used by shortcuts and autostart).diff --git a/desktop/app/src/win/clipboard.rs b/desktop/app/src/win/clipboard.rsindex a20098f..a63a3fb 100644--- a/desktop/app/src/win/clipboard.rs+++ b/desktop/app/src/win/clipboard.rs@@ -1,13 +1,16 @@// SPDX-License-Identifier: GPL-3.0-or-later-//! The Windows clipboard: watching for copied text, reading it and writing-//! text a phone sent (docs/protocol/clipboard.md).+//! The Windows clipboard: watching for copied text and images, reading them+//! and writing what a phone sent (docs/protocol/clipboard.md).//!//! A hidden message-only window on its own thread listens for clipboard//! changes and owns what Nectarlink writes, so text that came from a phone//! is never sent back. Content that password managers mark as private is-//! never read for sending.+//! never read for sending. Images are read as PNG when the app offers it,+//! or else as a bitmap (converted to PNG off the clipboard thread), and+//! written as a bitmap with alpha, plus PNG when that's what arrived.use std::{+ hash::{DefaultHasher, Hash, Hasher},sync::{Mutex, OnceLock,atomic::{AtomicIsize, Ordering},@@ -35,31 +38,80 @@ use windows::{core::{HSTRING, w},};-/// Standard clipboard format for UTF-16 text.+/// Standard clipboard formats.+const CF_DIB: u32 = 8;const CF_UNICODETEXT: u32 = 13;+const CF_DIBV5: u32 = 17;+/// Copied bitmaps larger than this aren't read (an 8K screenshot is+/// about 130 MB raw).+const MAX_BITMAP_BYTES: usize = 160 * 1024 * 1024;++/// A copied image, as the app offered it.+#[derive(Clone, PartialEq, Eq)]+pub enum Image {+ Png(Vec<u8>),+ /// A device-independent bitmap (`CF_DIBV5` or `CF_DIB`).+ Bitmap(Vec<u8>),+}++impl std::fmt::Debug for Image {+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {+ let (kind, len) = match self {+ Image::Png(b) => ("Png", b.len()),+ Image::Bitmap(b) => ("Bitmap", b.len()),+ };+ f.debug_struct(kind).field("bytes", &len).finish()+ }+}++impl Image {+ /// As a PNG file (bitmaps are converted; this can take a moment).+ pub fn to_png(&self) -> Result<Vec<u8>, String> {+ match self {+ Image::Png(png) => Ok(png.clone()),+ Image::Bitmap(dib) => {+ let bmp = super::image::bmp_from_dib(dib).ok_or("not a bitmap Windows knows")?;+ let bitmap = super::image::decode(&bmp).map_err(|e| e.to_string())?;+ super::image::encode_png(&bitmap).map_err(|e| e.to_string())+ }+ }+ }+}/// What's on the clipboard, for sending.#[derive(Debug, Clone, PartialEq, Eq)]pub enum Clip {Text(String),+ Image(Image),/// Marked private (a password manager): never sent.Private,- /// Nothing that's text.+ /// Nothing that's text or an image.Empty,}-type OnCopy = Box<dyn Fn(String) + Send + Sync>;+type OnCopy = Box<dyn Fn(Clip) + Send + Sync>;/// The listener window, once started (as an isize: HWND isn't Send).static WINDOW: AtomicIsize = AtomicIsize::new(0);static ON_COPY: OnceLock<OnCopy> = OnceLock::new();-/// The last text reported, so one copy reported several times (apps often-/// write the clipboard in steps) is sent once.-static LAST_REPORTED: Mutex<Option<String>> = Mutex::new(None);+/// A fingerprint of the last content reported, so one copy reported several+/// times (apps often write the clipboard in steps) is sent once.+static LAST_REPORTED: Mutex<Option<u64>> = Mutex::new(None);++fn fingerprint(clip: &Clip) -> Option<u64> {+ let mut hasher = DefaultHasher::new();+ match clip {+ Clip::Text(text) => text.hash(&mut hasher),+ Clip::Image(Image::Png(bytes) | Image::Bitmap(bytes)) => bytes.hash(&mut hasher),+ Clip::Private | Clip::Empty => return None,+ }+ Some(hasher.finish())+}-/// Starts watching the clipboard; `on_copy` gets each newly copied text-/// that isn't private and didn't come from Nectarlink, on the watcher thread.-pub fn start(on_copy: impl Fn(String) + Send + Sync + 'static) {+/// Starts watching the clipboard; `on_copy` gets each newly copied text or+/// image that isn't private and didn't come from Nectarlink, on the watcher+/// thread.+pub fn start(on_copy: impl Fn(Clip) + Send + Sync + 'static) {if ON_COPY.set(Box::new(on_copy)).is_err() {return; // already started}@@ -129,16 +181,17 @@ fn on_clipboard_changed(hwnd: HWND) {if unsafe { GetClipboardOwner() }.is_ok_and(|owner| owner == hwnd) {return;}- let Clip::Text(text) = read_with(Some(hwnd)) else { return };+ let clip = read_with(Some(hwnd));+ let Some(print) = fingerprint(&clip) else { return };{let mut last = LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner());- if last.as_deref() == Some(text.as_str()) {+ if *last == Some(print) {return;}- *last = Some(text.clone());+ *last = Some(print);}if let Some(on_copy) = ON_COPY.get() {- on_copy(text);+ on_copy(clip);}}@@ -176,7 +229,48 @@ pub fn write(text: &str) -> Result<(), String> {return Err(e.to_string());}}- *LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner()) = Some(text.to_owned());+ *LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner()) = fingerprint(&Clip::Text(text.to_owned()));+ Ok(())+}++/// Puts an image a phone sent (`image/png` or `image/jpeg`) on the+/// clipboard: as a bitmap with alpha, which every app pastes, and also as+/// PNG when it is one, which keeps it exact for apps that read PNG.+pub fn write_image(mime: &str, bytes: &[u8]) -> Result<(), String> {+ let owner = window().ok_or("the clipboard watcher isn't running")?;+ let bitmap = super::image::decode(bytes).map_err(|e| e.to_string())?;+ let mut blocks = vec![(CF_DIBV5, super::image::dibv5(&bitmap))];+ if mime == "image/png" {+ blocks.push((format("PNG"), bytes.to_vec()));+ }+ let _open = Opened::open(Some(owner)).ok_or("the clipboard is busy")?;+ // SAFETY: the clipboard is open.+ unsafe { EmptyClipboard() }.map_err(|e| e.to_string())?;+ for (format, data) in blocks {+ set_block(format, &data)?;+ }+ Ok(())+}++/// Copies `data` into a global block and hands it to the clipboard, which+/// must be open.+fn set_block(format: u32, data: &[u8]) -> Result<(), String> {+ // SAFETY: the block is sized for `data`; ownership passes to the+ // clipboard only when SetClipboardData succeeds (it's freed otherwise).+ unsafe {+ let block = GlobalAlloc(GMEM_MOVEABLE, data.len()).map_err(|e| e.to_string())?;+ let target = GlobalLock(block).cast::<u8>();+ if target.is_null() {+ let _ = GlobalFree(Some(block));+ return Err("can't lock memory".into());+ }+ std::ptr::copy_nonoverlapping(data.as_ptr(), target, data.len());+ let _ = GlobalUnlock(block);+ if let Err(e) = SetClipboardData(format, Some(HANDLE(block.0))) {+ let _ = GlobalFree(Some(block));+ return Err(e.to_string());+ }+ }Ok(())}@@ -214,7 +308,9 @@ fn format(name: &str) -> u32 {fn read_with(owner: Option<HWND>) -> Clip {// SAFETY: format availability checks need no open clipboard.let has = |format: u32| format != 0 && unsafe { IsClipboardFormatAvailable(format) }.is_ok();- if !has(CF_UNICODETEXT) {+ let png = format("PNG");+ let text = has(CF_UNICODETEXT);+ if !text && !has(png) && !has(CF_DIBV5) && !has(CF_DIB) {return Clip::Empty;}// Password managers and other apps mark private content this way.@@ -225,6 +321,17 @@ fn read_with(owner: Option<HWND>) -> Clip {if let Some(0) = read_dword(format("CanIncludeInClipboardHistory")) {return Clip::Private;}+ // Apps that copy both (a table from a spreadsheet) mean the text.+ if !text {+ return [(png, true), (CF_DIBV5, false), (CF_DIB, false)]+ .into_iter()+ .filter(|&(format, _)| has(format))+ .find_map(|(format, is_png)| {+ let bytes = read_block(format)?;+ Some(Clip::Image(if is_png { Image::Png(bytes) } else { Image::Bitmap(bytes) }))+ })+ .unwrap_or(Clip::Empty);+ }// SAFETY: the clipboard is open; the data handle is valid until it's// closed, and the text is read within the block's size.let text = unsafe {@@ -244,6 +351,28 @@ fn read_with(owner: Option<HWND>) -> Clip {if text.trim().is_empty() { Clip::Empty } else { Clip::Text(text) }}+/// A format's raw bytes, if present and not too large. The clipboard must+/// be open.+fn read_block(format: u32) -> Option<Vec<u8>> {+ // SAFETY: the caller opened the clipboard; the block is read within its+ // size while locked.+ unsafe {+ let handle = GetClipboardData(format).ok()?;+ let block = HGLOBAL(handle.0);+ let size = GlobalSize(block);+ if size == 0 || size > MAX_BITMAP_BYTES {+ return None;+ }+ let data = GlobalLock(block).cast::<u8>();+ if data.is_null() {+ return None;+ }+ let bytes = std::slice::from_raw_parts(data, size).to_vec();+ let _ = GlobalUnlock(block);+ Some(bytes)+ }+}+/// A DWORD-valued clipboard format, if present. The clipboard must be open.fn read_dword(format: u32) -> Option<u32> {if format == 0 {diff --git a/desktop/app/src/win/image.rs b/desktop/app/src/win/image.rsnew file mode 100644index 0000000..86b9331--- /dev/null+++ b/desktop/app/src/win/image.rs@@ -0,0 +1,225 @@+// SPDX-License-Identifier: GPL-3.0-or-later+//! Image conversions for the clipboard, with WIC: what Windows apps copy+//! (PNG, or a device-independent bitmap) to PNG for sending, and a PNG or+//! JPEG from a phone to pixels for pasting.++use windows::{+ Win32::{+ Graphics::Imaging::{+ CLSID_WICImagingFactory, GUID_ContainerFormatPng, GUID_WICPixelFormat32bppBGRA,+ IWICImagingFactory, WICBitmapDitherTypeNone, WICBitmapEncoderNoCache, WICBitmapPaletteTypeCustom,+ WICDecodeMetadataCacheOnDemand,+ },+ System::Com::{CLSCTX_INPROC_SERVER, CoCreateInstance, STATFLAG_NONAME, STREAM_SEEK_SET},+ UI::Shell::SHCreateMemStream,+ },+ core::{Error, HRESULT, Result},+};++use super::with_com;++/// Straight-alpha BGRA pixels, top row first.+#[derive(Debug, Clone, PartialEq, Eq)]+pub struct Bitmap {+ pub width: u32,+ pub height: u32,+ pub bgra: Vec<u8>,+}++fn invalid(message: &str) -> Error {+ // E_INVALIDARG+ Error::new(HRESULT(0x8007_0057_u32 as i32), message)+}++fn factory() -> Result<IWICImagingFactory> {+ // SAFETY: creating the WIC factory; COM is initialized by the caller.+ unsafe { CoCreateInstance(&CLSID_WICImagingFactory, None, CLSCTX_INPROC_SERVER) }+}++/// Decodes an image file in memory (PNG, JPEG, BMP and the other formats+/// WIC knows).+pub fn decode(bytes: &[u8]) -> Result<Bitmap> {+ with_com(|| {+ // SAFETY: WIC calls on live objects; the memory stream reads from+ // `bytes`, which outlives the decoder, and the pixel buffer is sized+ // for the converter's stride and height.+ unsafe {+ let factory = factory()?;+ let stream = factory.CreateStream()?;+ stream.InitializeFromMemory(bytes)?;+ let decoder =+ factory.CreateDecoderFromStream(&stream, std::ptr::null(), WICDecodeMetadataCacheOnDemand)?;+ let frame = decoder.GetFrame(0)?;+ let converter = factory.CreateFormatConverter()?;+ converter.Initialize(+ &frame,+ &GUID_WICPixelFormat32bppBGRA,+ WICBitmapDitherTypeNone,+ None,+ 0.0,+ WICBitmapPaletteTypeCustom,+ )?;+ let (mut width, mut height) = (0, 0);+ converter.GetSize(&mut width, &mut height)?;+ let len = (width as usize).checked_mul(height as usize).and_then(|n| n.checked_mul(4));+ let mut bgra = vec![0u8; len.ok_or_else(|| invalid("image too large"))?];+ converter.CopyPixels(std::ptr::null(), width * 4, &mut bgra)?;+ Ok(Bitmap { width, height, bgra })+ }+ })+}++/// Encodes pixels as a PNG file.+pub fn encode_png(bitmap: &Bitmap) -> Result<Vec<u8>> {+ with_com(|| {+ // SAFETY: as in `decode`; the bitmap is created from (a copy of)+ // the pixels, and the memory stream is read back within its size.+ unsafe {+ let factory = factory()?;+ let source = factory.CreateBitmapFromMemory(+ bitmap.width,+ bitmap.height,+ &GUID_WICPixelFormat32bppBGRA,+ bitmap.width * 4,+ &bitmap.bgra,+ )?;+ let stream = SHCreateMemStream(None).ok_or_else(|| invalid("no memory stream"))?;+ let encoder = factory.CreateEncoder(&GUID_ContainerFormatPng, std::ptr::null())?;+ encoder.Initialize(&stream, WICBitmapEncoderNoCache)?;+ let mut frame = None;+ encoder.CreateNewFrame(&mut frame, std::ptr::null_mut())?;+ let frame = frame.ok_or_else(|| invalid("no frame"))?;+ frame.Initialize(None)?;+ frame.WriteSource(&source, std::ptr::null())?;+ frame.Commit()?;+ encoder.Commit()?;++ let mut stat = Default::default();+ stream.Stat(&mut stat, STATFLAG_NONAME)?;+ let mut png = vec![0u8; usize::try_from(stat.cbSize).map_err(|_| invalid("too large"))?];+ stream.Seek(0, STREAM_SEEK_SET, None)?;+ let mut read = 0;+ stream+ .Read(png.as_mut_ptr().cast(), u32::try_from(png.len()).unwrap_or(u32::MAX), Some(&mut read))+ .ok()?;+ png.truncate(read as usize);+ Ok(png)+ }+ })+}++/// A device-independent bitmap (`CF_DIB` / `CF_DIBV5`) as a BMP file, which+/// WIC can decode: the DIB with a file header in front.+pub fn bmp_from_dib(dib: &[u8]) -> Option<Vec<u8>> {+ let u32_at = |at: usize| dib.get(at..at + 4).map(|b| u32::from_le_bytes([b[0], b[1], b[2], b[3]]));+ let header = u32_at(0)?;+ let bit_count = dib.get(14..16).map(|b| u16::from_le_bytes([b[0], b[1]]))?;+ let compression = u32_at(16)?;+ let colors_used = u32_at(32)?;+ if !(40..=124).contains(&header) {+ return None;+ }+ // Color masks follow a plain BITMAPINFOHEADER (later headers hold them).+ let masks = match (header, compression) {+ (40, 3) => 12, // BI_BITFIELDS+ (40, 6) => 16, // BI_ALPHABITFIELDS+ _ => 0,+ };+ let palette = match colors_used {+ 0 if bit_count <= 8 => 4u32 << bit_count,+ n => n.checked_mul(4)?,+ };+ let offset = 14 + header.checked_add(masks)?.checked_add(palette)?;+ let size = u32::try_from(14 + dib.len()).ok()?;+ if offset > size {+ return None;+ }+ let mut bmp = Vec::with_capacity(size as usize);+ bmp.extend_from_slice(b"BM");+ bmp.extend_from_slice(&size.to_le_bytes());+ bmp.extend_from_slice(&[0; 4]);+ bmp.extend_from_slice(&offset.to_le_bytes());+ bmp.extend_from_slice(dib);+ Some(bmp)+}++/// Pixels as a `CF_DIBV5` block: a BITMAPV5HEADER (32-bit, straight alpha,+/// sRGB) and the rows bottom-up, which is what apps read most reliably.+pub fn dibv5(bitmap: &Bitmap) -> Vec<u8> {+ let (width, height) = (bitmap.width, bitmap.height);+ let mut dib = Vec::with_capacity(124 + bitmap.bgra.len());+ let mut put = |bytes: &[u8]| dib.extend_from_slice(bytes);+ put(&124u32.to_le_bytes());+ put(&(width as i32).to_le_bytes());+ put(&(height as i32).to_le_bytes());+ put(&1u16.to_le_bytes());+ put(&32u16.to_le_bytes());+ put(&3u32.to_le_bytes()); // BI_BITFIELDS+ put(&(width * height * 4).to_le_bytes());+ put(&2835i32.to_le_bytes()); // 72 dpi+ put(&2835i32.to_le_bytes());+ put(&[0; 8]); // colors used, important+ for mask in [0x00FF_0000u32, 0x0000_FF00, 0x0000_00FF, 0xFF00_0000] {+ put(&mask.to_le_bytes());+ }+ put(&0x7352_4742u32.to_le_bytes()); // LCS_sRGB+ put(&[0; 36 + 12]); // endpoints, gamma+ put(&4u32.to_le_bytes()); // LCS_GM_IMAGES+ put(&[0; 12]); // profile data, size, reserved+ let row = width as usize * 4;+ for line in bitmap.bgra.chunks_exact(row.max(1)).rev() {+ dib.extend_from_slice(line);+ }+ dib+}++#[cfg(test)]+mod tests {+ use super::*;++ fn checker(width: u32, height: u32) -> Bitmap {+ let bgra = (0..width * height)+ .flat_map(|i| {+ let (x, y) = (i % width, i / width);+ if (x + y) % 2 == 0 { [255, 0, 0, 255] } else { [0, 0, 255, 128] }+ })+ .collect();+ Bitmap { width, height, bgra }+ }++ #[test]+ fn png_round_trips_with_alpha() {+ let bitmap = checker(5, 3);+ let png = encode_png(&bitmap).unwrap();+ assert_eq!(&png[1..4], b"PNG");+ assert_eq!(decode(&png).unwrap(), bitmap);+ }++ #[test]+ fn copied_bitmaps_decode() {+ let bitmap = checker(7, 4);+ let dib = dibv5(&bitmap);+ assert_eq!(dib.len(), 124 + 7 * 4 * 4);+ let decoded = decode(&bmp_from_dib(&dib).unwrap()).unwrap();+ assert_eq!((decoded.width, decoded.height), (7, 4));+ // Top-left stays top-left (rows are stored bottom-up).+ assert_eq!(&decoded.bgra[..4], &[255, 0, 0, 255]);++ // A plain 24-bit BITMAPINFOHEADER DIB, as older apps copy.+ let mut old = Vec::new();+ for field in [40u32, 2, 2] {+ old.extend_from_slice(&field.to_le_bytes());+ }+ old.extend_from_slice(&1u16.to_le_bytes());+ old.extend_from_slice(&24u16.to_le_bytes());+ old.extend_from_slice(&[0; 24]);+ // Bottom row: green, green; top row: white, black (rows padded to 4).+ old.extend_from_slice(&[0, 255, 0, 0, 255, 0, 0, 0]);+ old.extend_from_slice(&[255, 255, 255, 0, 0, 0, 0, 0]);+ let decoded = decode(&bmp_from_dib(&old).unwrap()).unwrap();+ assert_eq!(&decoded.bgra[..4], &[255, 255, 255, 255]);+ assert_eq!(&decoded.bgra[8..12], &[0, 255, 0, 255]);++ assert_eq!(bmp_from_dib(&[1, 2, 3]), None);+ }+}diff --git a/desktop/app/src/win/mod.rs b/desktop/app/src/win/mod.rsindex 77be6c2..e4b5430 100644--- a/desktop/app/src/win/mod.rs+++ b/desktop/app/src/win/mod.rs@@ -7,6 +7,7 @@pub mod clipboard;pub mod icon;+pub mod image;pub mod net;pub mod shortcut;pub mod single_instance;@@ -19,6 +20,7 @@ use windows::{Win32::{Foundation::ERROR_SUCCESS,System::{+ Com::{COINIT_APARTMENTTHREADED, CoInitializeEx, CoUninitialize},Power::{GetSystemPowerStatus, SYSTEM_POWER_STATUS},Registry::{HKEY, HKEY_CURRENT_USER, HKEY_LOCAL_MACHINE, RRF_RT_REG_DWORD, RRF_RT_REG_SZ, RegGetValueW,@@ -116,6 +118,19 @@ pub fn reduce_motion() -> bool {ok && !enabled.as_bool()}+/// Runs `f` with COM available on this thread.+pub fn with_com<T>(f: impl FnOnce() -> T) -> T {+ // SAFETY: balanced below; if the thread already has COM in another mode+ // (RPC_E_CHANGED_MODE), it's usable as is and isn't uninitialized here.+ let initialized = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED) }.is_ok();+ let result = f();+ if initialized {+ // SAFETY: pairs with the successful CoInitializeEx above.+ unsafe { CoUninitialize() };+ }+ result+}+#[cfg(test)]mod tests {use super::*;diff --git a/desktop/app/src/win/shortcut.rs b/desktop/app/src/win/shortcut.rsindex 103624a..a9f83c6 100644--- a/desktop/app/src/win/shortcut.rs+++ b/desktop/app/src/win/shortcut.rs@@ -5,15 +5,14 @@ use std::path::{Path, PathBuf};use windows::{Win32::{- System::Com::{- CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED, CoCreateInstance, CoInitializeEx, CoTaskMemFree,- CoUninitialize, IPersistFile,- },+ System::Com::{CLSCTX_INPROC_SERVER, CoCreateInstance, CoTaskMemFree, IPersistFile},UI::Shell::{FOLDERID_SendTo, IShellLinkW, KF_FLAG_DEFAULT, SHGetKnownFolderPath, ShellLink},},core::{HSTRING, Interface},};+use super::with_com;+/// What a shortcut starts.#[derive(Debug, Clone, PartialEq, Eq)]pub struct Shortcut {@@ -23,19 +22,6 @@ pub struct Shortcut {pub description: String,}-/// Runs `f` with COM available on this thread.-fn with_com<T>(f: impl FnOnce() -> T) -> T {- // SAFETY: balanced below; if the thread already has COM in another mode- // (RPC_E_CHANGED_MODE), it's usable as is and isn't uninitialized here.- let initialized = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED) }.is_ok();- let result = f();- if initialized {- // SAFETY: pairs with the successful CoInitializeEx above.- unsafe { CoUninitialize() };- }- result-}-/// Writes (or replaces) a shortcut at `path`, with the target's own icon.pub fn write(path: &Path, shortcut: &Shortcut) -> windows::core::Result<()> {with_com(|| {diff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.mdindex 76234b5..f988939 100644--- a/docs/protocol/capabilities.md+++ b/docs/protocol/capabilities.md@@ -22,6 +22,7 @@ capability is missing.| `device.battery` | Sends `event.battery` | always offered || `device.ring` | Rings on `device.ring`, even on silent | always offered || `files.transfer` | Sends and receives files ([files service](files.md)) | app update |+| `clip.image` | Accepts images on its clipboard ([clipboard service](clipboard.md)) | app update || `media.control` | Exposes its media sessions for remote control (phone: needs notification access) | phone: notification access · desktop: app update |## Phonediff --git a/docs/protocol/clipboard.md b/docs/protocol/clipboard.mdindex 1c59018..d7b0972 100644--- a/docs/protocol/clipboard.md+++ b/docs/protocol/clipboard.md@@ -3,13 +3,12 @@> Status: **draft.** Builds on [protocol v0](v0.md); breaking changes are> allowed until v1. License: CC BY 4.0.-Copied text moves between paired devices: a PC sends what the user copies-(automatically, unless turned off), and a phone sends its clipboard when-the user asks (Android only lets the app in front read the clipboard;-reading it in the background needs the Elevated power level).+Copied text and images move between paired devices: a PC sends what the+user copies (automatically, unless turned off), and a phone sends its+clipboard when the user asks (Android only lets the app in front read the+clipboard; reading it in the background needs the Elevated power level).-Images, rich text and files on the clipboard are not part of this version;-they will travel over bulk streams with file transfer.+Rich text and files on the clipboard are not part of this version.## 1. Capabilities@@ -18,6 +17,7 @@ they will travel over bulk streams with file transfer.| `clip.write` | phone | Accepts `clip.set` || `clip.share` | phone | Sends its clipboard when the user asks || `clip.read.auto` | phone | Sends clipboard changes on its own (Elevated) |+| `clip.image` | both | Accepts images (§3) |A PC always accepts `clip.set` from a phone the user allows.@@ -36,7 +36,34 @@ Errors: `DENIED` when the user turned the clipboard off for the sendingdevice (the `clipboard` device toggle), `INTERNAL` when the clipboardcouldn't be written.-## 3. Rules+## 3. Images++An image travels on its own bidirectional stream (v0 §4), so a large+screenshot never holds up the control stream. Only devices that offer+`clip.image` are sent images. The sender opens the stream and writes:++```+t = "stream" b = { svc: "clip", op: "image", v: 1 }+t = "clip.image" b = { mime: text, size: uint }+```++followed by exactly `size` bytes of the image, then finishes its side.+`mime` is `image/png` or `image/jpeg`; senders convert other formats (a+copied bitmap on Windows, a WebP on Android) to PNG. `size` is 1 byte to+32 MiB.++The receiver puts the image on its clipboard and answers on the same+stream with `ok`, or with `error`: `BAD_MESSAGE` for a type or size out of+bounds, `DENIED` as for `clip.set`, `INTERNAL` when the clipboard couldn't+be written. A receiver that refuses early stops reading (QUIC+`STOP_SENDING`) and still sends its answer. The whole exchange times out+after 60 seconds.++Receivers offer the image the way their platform's apps paste it best:+Windows as a bitmap with alpha (`CF_DIBV5`) plus the `PNG` format when a+PNG arrived; Android as a content URI from the app's own provider.++## 4. Rules- **Consent.** Each device sends and accepts only with the `clipboard`toggle on for the other device.