Skip to content

Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.

Copy images between PC and phone

Bee1d9bf192aacf
diff --git a/README.md b/README.md
index 83bf38e..e177a52 100644
--- a/README.md
+++ b/README.md
@@ -74,10 +74,10 @@ place and tested; the user-facing features of Phase 1 come next.
a feed in the app, with the app's icon. Reply inline, run their actions
(such as "Mark as read") or dismiss them from the PC; what's cleared on one
side is cleared on the other.
-- **Clipboard**: text you copy on the PC is ready to paste on the phone. From
- the phone, send it with the Quick Settings tile, the share sheet or a button
- in the app (Android lets only the app in front read the clipboard).
- Passwords that password managers mark private are never sent.
+- **Clipboard**: text and images you copy on the PC are ready to paste on the
+ phone. From the phone, send them with the Quick Settings tile, the share
+ sheet or a button in the app (Android lets only the app in front read the
+ clipboard). Passwords that password managers mark private are never sent.
- **Files** in both directions, at Wi-Fi speed. On the PC, drop files on the
window, pick them, or right-click them in File Explorer and choose
**Send to** and your phone (a notification follows the transfer); on the
@@ -95,8 +95,8 @@ place and tested; the user-facing features of Phase 1 come next.
connection service, and an identity key protected by the Android Keystore.
- **Command-line client** for development and testing (`nectarlink`).
-**Next (Phase 1)**: images on the clipboard, media controls, notification
-filters and history, and a Windows installer. Mirroring, messages, calls and the other
+**Next (Phase 1)**: media controls, notification filters and history, and a
+Windows installer. Mirroring, messages, calls and the other
features in the plan follow in later phases.
## Screenshots
diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xml
index ad071b3..d49bd3a 100644
--- a/android/app/src/main/AndroidManifest.xml
+++ b/android/app/src/main/AndroidManifest.xml
@@ -86,7 +86,8 @@
android:name=".files.TransferNotifications$CancelReceiver"
android:exported="false" />
- <!-- Opening received files on Android 8–9 (later versions use MediaStore). -->
+ <!-- Opening received files on Android 8–9 (later versions use MediaStore),
+ and images from a PC on the clipboard. -->
<provider
android:name="androidx.core.content.FileProvider"
android:authorities="${applicationId}.files"
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt
index ad69fe9..9b36542 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/PhoneClipboard.kt
@@ -5,33 +5,114 @@ import android.content.ClipData
import android.content.ClipDescription
import android.content.ClipboardManager
import android.content.Context
+import android.graphics.Bitmap
+import android.graphics.BitmapFactory
+import android.net.Uri
import android.os.Build
+import android.util.Log
+import androidx.core.content.FileProvider
+import java.io.File
/**
* The phone's clipboard. Android lets an app read it only while that app
* is in front (and focused); writing works any time.
*/
object PhoneClipboard {
+ private const val TAG = "PhoneClipboard"
+
+ /** The largest image sent (docs/protocol/clipboard.md). */
+ private const val MAX_IMAGE_BYTES = 32L * 1024 * 1024
+
+ /** Where images from a PC are kept while they're on the clipboard. */
+ private const val IMAGE_DIR = "clipboard"
+
sealed interface Read {
data class Text(val text: String) : Read
+ /** An image, still to be read with [readImage] (the app holds access to it for now). */
+ data class Image(val uri: Uri) : Read
/** Marked sensitive by the app it came from (a password manager): never sent. */
data object Private : Read
data object Empty : Read
}
+ /** An image ready to send: PNG or JPEG, as the protocol allows. */
+ class ImageData(val mime: String, val bytes: ByteArray)
+
+ sealed interface ImageResult {
+ class Ready(val image: ImageData) : ImageResult
+ data object TooLarge : ImageResult
+ data object Unreadable : ImageResult
+ }
+
fun read(context: Context): Read {
val clip = manager(context)?.primaryClip ?: return Read.Empty
if (clip.description.isSensitive()) return Read.Private
+ val resolver = context.contentResolver
+ for (i in 0 until clip.itemCount) {
+ val item = clip.getItemAt(i)
+ item.text?.toString()?.takeIf(String::isNotBlank)?.let { return Read.Text(it) }
+ // Checked before coercing to text, which would turn it into its address.
+ val uri = item.uri
+ if (uri != null && resolver.getType(uri)?.startsWith("image/") == true) return Read.Image(uri)
+ }
val text = (0 until clip.itemCount)
.firstNotNullOfOrNull { clip.getItemAt(it).coerceToText(context)?.toString()?.takeIf(String::isNotBlank) }
return text?.let { Read.Text(it) } ?: Read.Empty
}
+ /**
+ * Reads a copied image for sending (blocking; call off the main thread).
+ * PNG and JPEG go as they are; other formats are converted to PNG.
+ */
+ fun readImage(context: Context, uri: Uri): ImageResult = runCatching {
+ val resolver = context.contentResolver
+ val size = resolver.openAssetFileDescriptor(uri, "r")?.use { it.length } ?: -1
+ if (size > MAX_IMAGE_BYTES) return ImageResult.TooLarge
+ val bytes = resolver.openInputStream(uri)?.use { input ->
+ // Read one byte past the limit to notice a source that didn't say its size.
+ val buffer = input.readNBytesCompat(MAX_IMAGE_BYTES.toInt() + 1)
+ if (buffer.size > MAX_IMAGE_BYTES) return ImageResult.TooLarge
+ buffer
+ } ?: return ImageResult.Unreadable
+ when (val mime = resolver.getType(uri)) {
+ "image/png", "image/jpeg" -> ImageResult.Ready(ImageData(mime, bytes))
+ else -> {
+ val bitmap = BitmapFactory.decodeByteArray(bytes, 0, bytes.size) ?: return ImageResult.Unreadable
+ val png = java.io.ByteArrayOutputStream().also { bitmap.compress(Bitmap.CompressFormat.PNG, 100, it) }
+ bitmap.recycle()
+ if (png.size() > MAX_IMAGE_BYTES) ImageResult.TooLarge else ImageResult.Ready(ImageData("image/png", png.toByteArray()))
+ }
+ }
+ }.getOrElse {
+ Log.w(TAG, "can't read the copied image", it)
+ ImageResult.Unreadable
+ }
+
/** Puts text a PC sent on the clipboard. */
fun write(context: Context, text: String): Boolean =
runCatching { manager(context)?.setPrimaryClip(ClipData.newPlainText("Nectarlink", text)) != null }
.getOrDefault(false)
+ /**
+ * Puts an image a PC sent on the clipboard: kept in the app's cache and
+ * shared through its FileProvider, so the app that pastes can read it.
+ * The previous one is deleted; only the latest is on the clipboard.
+ */
+ fun writeImage(context: Context, mime: String, bytes: ByteArray): Boolean = runCatching {
+ val dir = File(context.cacheDir, IMAGE_DIR).apply { mkdirs() }
+ dir.listFiles()?.forEach { it.delete() }
+ val extension = if (mime == "image/jpeg") "jpg" else "png"
+ // A new name each time: apps cache what they read by address.
+ val file = File(dir, "Image ${System.currentTimeMillis()}.$extension")
+ file.writeBytes(bytes)
+ val uri = FileProvider.getUriForFile(context, "${context.packageName}.files", file)
+ val clip = ClipData.newUri(context.contentResolver, "Image", uri)
+ manager(context)?.setPrimaryClip(clip) != null
+ }.getOrElse {
+ Log.w(TAG, "can't put an image on the clipboard", it)
+ false
+ }
+
private fun manager(context: Context) = context.getSystemService(ClipboardManager::class.java)
private fun ClipDescription.isSensitive(): Boolean {
@@ -42,4 +123,16 @@ object PhoneClipboard {
}
return extras?.getBoolean(key) == true
}
+
+ /** `readNBytes`, which needs Android 13. */
+ private fun java.io.InputStream.readNBytesCompat(limit: Int): ByteArray {
+ val out = java.io.ByteArrayOutputStream()
+ val buffer = ByteArray(64 * 1024)
+ while (out.size() < limit) {
+ val read = read(buffer, 0, minOf(buffer.size, limit - out.size()))
+ if (read < 0) break
+ out.write(buffer, 0, read)
+ }
+ return out.toByteArray()
+ }
}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt
index 38fbf83..3bd2633 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/clipboard/SendActivity.kt
@@ -75,6 +75,7 @@ class SendActivity : Activity() {
if (!hasFocus || sent || intent?.action != ACTION_SEND_CLIPBOARD) return
when (val clip = PhoneClipboard.read(this)) {
is PhoneClipboard.Read.Text -> send(clip.text)
+ is PhoneClipboard.Read.Image -> sendImage(clip.uri)
PhoneClipboard.Read.Private -> finishWith(getString(R.string.clip_private))
PhoneClipboard.Read.Empty -> finishWith(getString(R.string.clip_empty))
}
@@ -86,6 +87,12 @@ class SendActivity : Activity() {
scope.launch { finishWith(core.sendClipboard(text)) }
}
+ private fun sendImage(uri: Uri) {
+ sent = true
+ val core = (application as NectarlinkApplication).core
+ scope.launch { finishWith(core.sendClipboardImage(uri)) }
+ }
+
private fun finishWith(message: String) {
sent = true
Toast.makeText(applicationContext, message, Toast.LENGTH_SHORT).show()
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
index b367f09..abe4650 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
@@ -7,6 +7,7 @@ import android.service.notification.NotificationListenerService
import android.util.Log
import android.net.Uri
import app.nectarlink.android.BuildConfig
+import app.nectarlink.android.clipboard.PhoneClipboard
import app.nectarlink.android.files.OutgoingFiles
import app.nectarlink.android.files.ReceivedFiles
import app.nectarlink.android.files.TransferNotifications
@@ -37,6 +38,7 @@ import kotlinx.coroutines.flow.asSharedFlow
import kotlinx.coroutines.flow.asStateFlow
import kotlinx.coroutines.flow.update
import kotlinx.coroutines.launch
+import kotlinx.coroutines.withContext
/**
* The app's single connection to `nectarlink-core`. Owns the node, folds
@@ -208,7 +210,22 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
* Sends text to every connected PC; returns what to tell the user.
* Waits for the core when the app was just launched to send.
*/
- suspend fun sendClipboard(text: String): String {
+ suspend fun sendClipboard(text: String): String =
+ sendToPcs(image = false) { node, pc -> node.sendClipboard(pc, text) }
+
+ /** Sends a copied image to the connected PCs; returns what to tell the user. */
+ suspend fun sendClipboardImage(uri: Uri): String {
+ startJob?.join()
+ if (_state.value.devices.none { it.online }) return context.getString(R.string.clip_no_pc)
+ val image = when (val read = withContext(Dispatchers.IO) { PhoneClipboard.readImage(context, uri) }) {
+ is PhoneClipboard.ImageResult.Ready -> read.image
+ PhoneClipboard.ImageResult.TooLarge -> return context.getString(R.string.clip_image_too_large)
+ PhoneClipboard.ImageResult.Unreadable -> return context.getString(R.string.clip_image_unreadable)
+ }
+ return sendToPcs(image = true) { node, pc -> node.sendClipboardImage(pc, image.mime, image.bytes) }
+ }
+
+ private suspend fun sendToPcs(image: Boolean, send: suspend (NectarlinkNode, String) -> Unit): String {
startJob?.join()
val node = node ?: return context.getString(R.string.clip_no_pc)
val pcs = _state.value.devices.filter { it.online }
@@ -217,12 +234,16 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
var failure: String? = null
for (pc in pcs) {
try {
- node.sendClipboard(pc.id, text)
+ send(node, pc.id)
sentTo += pc.name
} catch (e: NectarlinkException) {
failure = when (e) {
is NectarlinkException.Denied -> context.getString(R.string.clip_off_for, pc.name)
- is NectarlinkException.TooLarge -> context.getString(R.string.clip_too_large)
+ is NectarlinkException.TooLarge -> context.getString(
+ if (image) R.string.clip_image_too_large else R.string.clip_too_large,
+ )
+ is NectarlinkException.Unsupported ->
+ if (image) context.getString(R.string.clip_image_unsupported, pc.name) else describe(e)
else -> describe(e)
}
}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
index 89664cd..db5805e 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/PhonePlatform.kt
@@ -26,6 +26,9 @@ internal class PhonePlatform(context: Context, private val ringer: Ringer) : Pla
override fun setClipboard(text: String): Boolean = PhoneClipboard.write(context, text)
+ override fun setClipboardImage(mime: String, bytes: ByteArray): Boolean =
+ PhoneClipboard.writeImage(context, mime, bytes)
+
override fun dismissNotification(key: String) {
val listener = NotificationListener.instance ?: throw NotificationFailure.Unsupported()
listener.dismiss(key)
diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml
index 525d164..535600b 100644
--- a/android/app/src/main/res/values/strings.xml
+++ b/android/app/src/main/res/values/strings.xml
@@ -81,7 +81,10 @@
<string name="clip_off_for">The clipboard is turned off for %1$s.</string>
<string name="clip_too_large">That’s too much text to send at once.</string>
<string name="clip_private">What you copied is marked private by its app, so it wasn’t sent.</string>
- <string name="clip_empty">There’s no text on the clipboard.</string>
+ <string name="clip_empty">There’s no text or image on the clipboard.</string>
+ <string name="clip_image_too_large">That image is too large to send.</string>
+ <string name="clip_image_unreadable">That image can’t be read.</string>
+ <string name="clip_image_unsupported">Update Nectarlink on %1$s to send it images.</string>
<string name="clip_nothing_shared">There’s no text to send.</string>
<string name="background_title">Stay reachable in the background</string>
<string name="background_text">Your phone pauses Nectarlink to save battery, so your PC can’t reach it, and replies and dismissals from the PC time out. Let it run in the background.</string>
diff --git a/android/app/src/main/res/xml/file_paths.xml b/android/app/src/main/res/xml/file_paths.xml
index f9fad49..905e132 100644
--- a/android/app/src/main/res/xml/file_paths.xml
+++ b/android/app/src/main/res/xml/file_paths.xml
@@ -1,6 +1,8 @@
<?xml version="1.0" encoding="utf-8"?>
<!-- SPDX-License-Identifier: GPL-3.0-or-later -->
-<!-- Received files on Android 8–9 (see files/ReceivedFiles.kt). -->
+<!-- Received files on Android 8–9 (see files/ReceivedFiles.kt), and images
+ a PC put on the clipboard (see clipboard/PhoneClipboard.kt). -->
<paths>
<external-files-path name="received" path="Download/Nectarlink/" />
+ <cache-path name="clipboard" path="clipboard/" />
</paths>
diff --git a/core/nectarlink-cli/src/main.rs b/core/nectarlink-cli/src/main.rs
index be6d4b1..f6df376 100644
--- a/core/nectarlink-cli/src/main.rs
+++ b/core/nectarlink-cli/src/main.rs
@@ -117,8 +117,15 @@ enum Command {
#[arg(required = true)]
files: Vec<PathBuf>,
},
- /// Put text on a paired device's clipboard.
- Clip { device: String, text: String },
+ /// Put text, or an image with --image, on a paired device's clipboard.
+ Clip {
+ device: String,
+ #[arg(required_unless_present = "image", conflicts_with = "image")]
+ text: Option<String>,
+ /// A PNG or JPEG file to put on the clipboard instead of text.
+ #[arg(long)]
+ image: Option<PathBuf>,
+ },
/// Dismiss a phone's notification (`run` prints the keys).
Dismiss { device: String, key: String },
/// Run an action of a phone's notification, or reply to it.
@@ -176,6 +183,13 @@ impl Platform for TerminalPlatform {
println!("Clipboard from a paired device: {text}");
Ok(())
}
+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
+ let extension = if mime == "image/jpeg" { "jpg" } else { "png" };
+ let path = std::env::temp_dir().join(format!("nectarlink-clipboard.{extension}"));
+ std::fs::write(&path, bytes).map_err(|e| e.to_string())?;
+ println!("Clipboard image from a paired device ({} bytes): {}", bytes.len(), path.display());
+ Ok(())
+ }
fn dismiss_notification(&self, key: &str) -> Result<(), NotificationError> {
println!("The PC dismissed {key}");
Ok(())
@@ -295,10 +309,25 @@ async fn run(cli: &Cli, node: &Node) -> Result<()> {
wait_until_online(node, id).await?;
send_files(node, id, files).await?;
}
- Command::Clip { device, text } => {
+ Command::Clip { device, text, image } => {
let id = resolve(node, device)?;
wait_until_online(node, id).await?;
- node.send_clipboard(id, text.clone()).await.context("clipboard failed")?;
+ match (text, image) {
+ (_, Some(path)) => {
+ let mime = match path.extension().and_then(|e| e.to_str()).map(str::to_ascii_lowercase) {
+ Some(e) if e == "png" => "image/png",
+ Some(e) if e == "jpg" || e == "jpeg" => "image/jpeg",
+ _ => anyhow::bail!("only PNG and JPEG images can go on the clipboard"),
+ };
+ let bytes =
+ std::fs::read(path).with_context(|| format!("can't read {}", path.display()))?;
+ node.send_clipboard_image(id, mime.into(), bytes).await.context("clipboard failed")?;
+ }
+ (Some(text), None) => {
+ node.send_clipboard(id, text.clone()).await.context("clipboard failed")?
+ }
+ (None, None) => unreachable!("clap requires text or --image"),
+ }
println!("Sent.");
}
Command::Dismiss { device, key } => {
diff --git a/core/nectarlink-core/src/clipboard.rs b/core/nectarlink-core/src/clipboard.rs
index 85cf8a6..1ca12e2 100644
--- a/core/nectarlink-core/src/clipboard.rs
+++ b/core/nectarlink-core/src/clipboard.rs
@@ -1,15 +1,24 @@
// SPDX-License-Identifier: MPL-2.0
-//! The clipboard service (docs/protocol/clipboard.md): copied text sent to
-//! a paired device, which puts it on its own clipboard.
+//! The clipboard service (docs/protocol/clipboard.md): copied text and
+//! images sent to a paired device, which puts them on its own clipboard.
+//! Text goes on the control stream; images on a stream of their own.
-use std::sync::Arc;
+use std::{sync::Arc, time::Duration};
+use iroh::endpoint::{RecvStream, SendStream};
use nectarlink_protocol::{
- Envelope, ErrorCode,
- messages::{ClipSet, types},
+ DeviceId, Envelope, ErrorCode,
+ messages::{ClipImage, ClipSet, StreamHeader, clip, types},
+ read_frame, write_frame,
};
-use crate::{Result, events::NodeEvent, node::Shared, session::Session};
+use crate::{Error, Result, error::net, events::NodeEvent, node::Shared, session::Session};
+
+/// How long an image may take to arrive, or to be answered: generous, for
+/// a large screenshot on a slow link.
+const IMAGE_TIMEOUT: Duration = Duration::from_secs(60);
+/// Remote capability for receiving images.
+const IMAGE_CAPABILITY: &str = "clip.image";
/// The device toggle that allows the clipboard for a device.
pub(crate) const TOGGLE: &str = "clipboard";
@@ -48,3 +57,99 @@ pub(crate) async fn handle(shared: &Arc<Shared>, session: &Arc<Session>, env: &E
session.send(reply.reply_to(env.id)).await?;
Ok(true)
}
+
+/// Sends an image on its own stream and waits for the receiver's answer.
+pub(crate) async fn send_image(
+ shared: &Arc<Shared>,
+ session: &Session,
+ mime: String,
+ bytes: Vec<u8>,
+) -> Result<()> {
+ let peer = session.peer;
+ let image = ClipImage { mime, size: bytes.len() as u64 };
+ if image.size > clip::MAX_IMAGE_BYTES {
+ return Err(Error::TooLarge);
+ }
+ if !image.is_valid() {
+ return Err(Error::Internal("not a PNG or JPEG image".into()));
+ }
+ if !shared.toggle_on(&peer, TOGGLE) {
+ return Err(Error::Denied);
+ }
+ let takes_images = shared.store.get_peer(&peer)?.is_some_and(|p| p.caps.contains(IMAGE_CAPABILITY));
+ if !takes_images {
+ return Err(Error::Unsupported);
+ }
+ let exchange = async {
+ let (mut send, mut recv) = session.conn.open_bi().await.map_err(net)?;
+ let header = StreamHeader { svc: clip::SERVICE.into(), op: clip::OP_IMAGE.into(), v: clip::VERSION };
+ write_frame(&mut send, &Envelope::new(types::STREAM, &header)?.to_cbor()).await?;
+ write_frame(&mut send, &Envelope::new(clip::IMAGE, &image)?.to_cbor()).await?;
+ // A receiver that refuses stops reading; its answer still comes.
+ let written = send.write_all(&bytes).await;
+ let _ = send.finish();
+ let reply = read_frame(&mut recv).await?.ok_or(Error::Offline)?;
+ Envelope::from_cbor(&reply)?.expect(types::OK)?;
+ written.map_err(net)?;
+ Ok(())
+ };
+ tokio::time::timeout(IMAGE_TIMEOUT, exchange).await.map_err(|_| Error::Timeout)?
+}
+
+/// Receives an image (the stream header was read) and puts it on the
+/// clipboard.
+pub(crate) async fn receive_image(
+ shared: Arc<Shared>,
+ peer: DeviceId,
+ mut send: SendStream,
+ mut recv: RecvStream,
+) {
+ let reply = match tokio::time::timeout(IMAGE_TIMEOUT, take_image(&shared, peer, &mut recv)).await {
+ Ok(Ok(())) => Envelope::empty(types::OK),
+ Ok(Err((code, msg))) => Envelope::error(code, msg),
+ Err(_) => Envelope::error(ErrorCode::Internal, "the image took too long"),
+ };
+ if reply.t != types::OK {
+ let _ = recv.stop(0u32.into());
+ }
+ let _ = write_frame(&mut send, &reply.to_cbor()).await;
+ let _ = send.finish();
+ let _ = send.stopped().await;
+}
+
+async fn take_image(
+ shared: &Arc<Shared>,
+ peer: DeviceId,
+ recv: &mut RecvStream,
+) -> std::result::Result<(), (ErrorCode, &'static str)> {
+ let image = read_frame(recv)
+ .await
+ .ok()
+ .flatten()
+ .and_then(|bytes| Envelope::from_cbor(&bytes).ok())
+ .and_then(|env| env.expect_body::<ClipImage>(clip::IMAGE).ok())
+ .filter(ClipImage::is_valid)
+ .ok_or((ErrorCode::BadMessage, "invalid image"))?;
+ if !shared.toggle_on(&peer, TOGGLE) {
+ return Err((ErrorCode::Denied, "the clipboard is off for this device"));
+ }
+ let size = usize::try_from(image.size).map_err(|_| (ErrorCode::BadMessage, "invalid image"))?;
+ let mut bytes = vec![0u8; size];
+ recv.read_exact(&mut bytes).await.map_err(|_| (ErrorCode::BadMessage, "the image ended early"))?;
+ let platform = shared.platform.clone();
+ let mime = image.mime;
+ match tokio::task::spawn_blocking(move || platform.set_clipboard_image(&mime, &bytes)).await {
+ Ok(Ok(())) => {
+ shared.emit(NodeEvent::ClipboardReceived { device: peer });
+ Ok(())
+ }
+ Ok(Err(reason)) => {
+ tracing::warn!(reason, "can't put an image on the clipboard");
+ Err((ErrorCode::Internal, "can't write the clipboard"))
+ }
+ Err(e) => {
+ tracing::warn!(error = %e, "clipboard writer failed");
+ Err((ErrorCode::Internal, "can't write the clipboard"))
+ }
+ }
+}
diff --git a/core/nectarlink-core/src/config.rs b/core/nectarlink-core/src/config.rs
index b71a31a..ecae6d6 100644
--- a/core/nectarlink-core/src/config.rs
+++ b/core/nectarlink-core/src/config.rs
@@ -93,6 +93,12 @@ pub trait Platform: Send + Sync + 'static {
fn set_clipboard(&self, _text: &str) -> Result<(), String> {
Err("this device has no clipboard".into())
}
+
+ /// Put an image on this device's clipboard (a paired device sent it).
+ /// `mime` is `image/png` or `image/jpeg`.
+ fn set_clipboard_image(&self, _mime: &str, _bytes: &[u8]) -> Result<(), String> {
+ Err("this device has no clipboard".into())
+ }
}
/// A platform that does nothing; useful for tests and headless tools.
diff --git a/core/nectarlink-core/src/lib.rs b/core/nectarlink-core/src/lib.rs
index 327c2ad..21f2998 100644
--- a/core/nectarlink-core/src/lib.rs
+++ b/core/nectarlink-core/src/lib.rs
@@ -26,6 +26,9 @@ pub use events::{
};
pub use features::{CapabilityMatrix, FeatureState};
pub use identity::{KeyProtector, PlainKeyProtector, default_protector};
+pub use nectarlink_protocol::messages::clip::{
+ IMAGE_TYPES as CLIP_IMAGE_TYPES, MAX_IMAGE_BYTES as CLIP_MAX_IMAGE_BYTES,
+};
pub use nectarlink_protocol::{
DeviceId,
messages::{
diff --git a/core/nectarlink-core/src/node.rs b/core/nectarlink-core/src/node.rs
index a844199..0018be1 100644
--- a/core/nectarlink-core/src/node.rs
+++ b/core/nectarlink-core/src/node.rs
@@ -53,7 +53,8 @@ const EVENT_CAPACITY: usize = 512;
const IDLE_TIMEOUT: Duration = Duration::from_secs(8);
/// Capabilities every build offers.
-const BASE_CAPABILITIES: &[&str] = &["core.ping", "device.battery", "device.ring", "files.transfer"];
+const BASE_CAPABILITIES: &[&str] =
+ &["core.ping", "device.battery", "device.ring", "files.transfer", "clip.image"];
/// This device's mutable description, sent to peers.
#[derive(Debug, Clone)]
@@ -858,6 +859,16 @@ impl Node {
Ok(())
}
+ /// Puts an image on a paired device's clipboard (`image/png` or
+ /// `image/jpeg`), when the user copied one. [`Error::Unsupported`] when
+ /// the device's app can't take images, [`Error::Denied`] when the user
+ /// turned the clipboard off for it, and [`Error::TooLarge`] beyond
+ /// [`crate::CLIP_MAX_IMAGE_BYTES`].
+ pub async fn send_clipboard_image(&self, peer: DeviceId, mime: String, bytes: Vec<u8>) -> Result<()> {
+ let session = self.connected(&peer)?;
+ crate::clipboard::send_image(&self.shared, &session, mime, bytes).await
+ }
+
// ---- Notifications (docs/protocol/notifications.md) ----
/// A notification appeared or changed on this phone; sent to every
diff --git a/core/nectarlink-core/src/transfer.rs b/core/nectarlink-core/src/transfer.rs
index cbd7c12..f2a0231 100644
--- a/core/nectarlink-core/src/transfer.rs
+++ b/core/nectarlink-core/src/transfer.rs
@@ -13,7 +13,7 @@ use iroh::endpoint::{ReadError, RecvStream, SendStream, VarInt, WriteError};
use nectarlink_protocol::{
DeviceId, Envelope, ErrorCode,
messages::{
- ErrorBody, FileEntry, FilesAccept, FilesOffer, StreamHeader, files, is_valid_file_name, types,
+ ErrorBody, FileEntry, FilesAccept, FilesOffer, StreamHeader, clip, files, is_valid_file_name, types,
},
read_frame, write_frame,
};
@@ -435,6 +435,9 @@ pub(crate) async fn accept_stream(
Some(h) if h.svc == files::SERVICE && h.op == files::OP_SEND && h.v == files::VERSION => {
receive(shared, session.peer, send, recv).await;
}
+ Some(h) if h.svc == clip::SERVICE && h.op == clip::OP_IMAGE && h.v == clip::VERSION => {
+ crate::clipboard::receive_image(shared, session.peer, send, recv).await;
+ }
_ => {
let reply = Envelope::error(ErrorCode::Unsupported, "unknown stream");
let _ = write_frame(&mut send, &reply.to_cbor()).await;
diff --git a/core/nectarlink-core/tests/two_devices.rs b/core/nectarlink-core/tests/two_devices.rs
index 203bc99..6dc8935 100644
--- a/core/nectarlink-core/tests/two_devices.rs
+++ b/core/nectarlink-core/tests/two_devices.rs
@@ -26,6 +26,7 @@ struct RecordingPlatform {
dismissed: Mutex<Vec<String>>,
actions: Mutex<Vec<(String, String, Option<String>)>>,
clipboard: Mutex<Vec<String>>,
+ images: Mutex<Vec<(String, Vec<u8>)>>,
}
impl Platform for RecordingPlatform {
@@ -55,6 +56,10 @@ impl Platform for RecordingPlatform {
self.clipboard.lock().unwrap().push(text.to_owned());
Ok(())
}
+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
+ self.images.lock().unwrap().push((mime.to_owned(), bytes.to_vec()));
+ Ok(())
+ }
}
struct TestDevice {
@@ -585,6 +590,56 @@ async fn clipboard_text_goes_both_ways_with_consent() {
assert!(matches!(pc.node.send_clipboard(phone_id, huge).await, Err(Error::TooLarge)));
}
+#[tokio::test(flavor = "multi_thread")]
+async fn clipboard_images_go_both_ways_with_consent() {
+ let mut pc = device("Desktop", DeviceKind::Desktop).await;
+ let mut phone = device("Pixel", DeviceKind::Phone).await;
+ pair_qr(&mut pc, &mut phone).await;
+ let (pc_id, phone_id) = (pc.node.device_id(), phone.node.device_id());
+
+ // A large screenshot, and a small photo the other way.
+ let screenshot = data(9 * 1024 * 1024 + 17, 3);
+ with_timeout(
+ "pc to phone",
+ pc.node.send_clipboard_image(phone_id, "image/png".into(), screenshot.clone()),
+ )
+ .await
+ .expect("sent");
+ assert_eq!(*phone.platform.images.lock().unwrap(), [("image/png".to_owned(), screenshot)]);
+ wait_for(&mut phone, "clipboard received", |e| match e {
+ NodeEvent::ClipboardReceived { device } if *device == pc_id => Some(()),
+ _ => None,
+ })
+ .await;
+ let photo = data(40_000, 4);
+ with_timeout("phone to pc", phone.node.send_clipboard_image(pc_id, "image/jpeg".into(), photo.clone()))
+ .await
+ .expect("sent");
+ assert_eq!(*pc.platform.images.lock().unwrap(), [("image/jpeg".to_owned(), photo)]);
+
+ // Only PNG and JPEG, and not too large.
+ let gif = pc.node.send_clipboard_image(phone_id, "image/gif".into(), vec![1; 10]).await;
+ assert!(matches!(gif, Err(Error::Internal(_))), "{gif:?}");
+ let max = usize::try_from(nectarlink_core::CLIP_MAX_IMAGE_BYTES).unwrap();
+ let huge = pc.node.send_clipboard_image(phone_id, "image/png".into(), vec![0; max + 1]).await;
+ assert!(matches!(huge, Err(Error::TooLarge)), "{huge:?}");
+
+ // The receiver turned the clipboard off for the sender; the sender's
+ // stream is refused and nothing lands.
+ phone.node.set_device_toggle(pc_id, "clipboard", false).unwrap();
+ let denied = with_timeout(
+ "denied",
+ pc.node.send_clipboard_image(phone_id, "image/png".into(), data(2_000_000, 5)),
+ )
+ .await;
+ assert!(matches!(denied, Err(Error::Denied)), "{denied:?}");
+ assert_eq!(phone.platform.images.lock().unwrap().len(), 1);
+ // And the connection still works.
+ with_timeout("still connected", pc.node.send_clipboard(phone_id, "x".into()))
+ .await
+ .expect_err("still off");
+}
+
/// Deterministic, incompressible-looking test data.
fn data(len: usize, seed: u64) -> Vec<u8> {
let mut x = seed.wrapping_mul(0x9E37_79B9_7F4A_7C15) | 1;
diff --git a/core/nectarlink-ffi/src/lib.rs b/core/nectarlink-ffi/src/lib.rs
index bf402b0..9ec5c5a 100644
--- a/core/nectarlink-ffi/src/lib.rs
+++ b/core/nectarlink-ffi/src/lib.rs
@@ -726,6 +726,9 @@ pub trait Platform: Send + Sync {
) -> Result<(), NotificationFailure>;
/// A PC sent this text: put it on the clipboard. False if that failed.
fn set_clipboard(&self, text: String) -> bool;
+ /// A PC sent this image (`image/png` or `image/jpeg`): put it on the
+ /// clipboard. False if that failed.
+ fn set_clipboard_image(&self, mime: String, bytes: Vec<u8>) -> bool;
}
/// Encrypts the device key at rest (Android: a Keystore key).
@@ -770,6 +773,13 @@ impl core::Platform for PlatformAdapter {
fn set_clipboard(&self, text: &str) -> Result<(), String> {
if self.0.set_clipboard(text.to_owned()) { Ok(()) } else { Err("the clipboard rejected it".into()) }
}
+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
+ if self.0.set_clipboard_image(mime.to_owned(), bytes.to_vec()) {
+ Ok(())
+ } else {
+ Err("the clipboard rejected it".into())
+ }
+ }
}
struct KeyProtectorAdapter(Arc<dyn KeyProtector>);
@@ -979,6 +989,14 @@ impl NectarlinkNode {
self.run(async move { Ok(node.send_clipboard(id, text).await?) }).await
}
+ /// Puts an image (`image/png` or `image/jpeg`) on a paired PC's
+ /// clipboard.
+ pub async fn send_clipboard_image(&self, id: String, mime: String, bytes: Vec<u8>) -> Result<()> {
+ let id = parse_id(&id)?;
+ let node = self.node.clone();
+ self.run(async move { Ok(node.send_clipboard_image(id, mime, bytes).await?) }).await
+ }
+
// ---- Notifications ----
/// A notification appeared or changed on this phone.
diff --git a/core/nectarlink-protocol/src/messages.rs b/core/nectarlink-protocol/src/messages.rs
index a893d69..748ca30 100644
--- a/core/nectarlink-protocol/src/messages.rs
+++ b/core/nectarlink-protocol/src/messages.rs
@@ -419,6 +419,33 @@ impl std::fmt::Debug for ClipSet {
}
}
+/// Clipboard images, which travel on their own stream.
+pub mod clip {
+ pub const SERVICE: &str = "clip";
+ pub const OP_IMAGE: &str = "image";
+ pub const VERSION: u32 = 1;
+ pub const IMAGE: &str = "clip.image";
+ /// The largest image sent or accepted.
+ pub const MAX_IMAGE_BYTES: u64 = 32 * 1024 * 1024;
+ /// Image types a receiver accepts.
+ pub const IMAGE_TYPES: &[&str] = &["image/png", "image/jpeg"];
+}
+
+/// Body of `clip.image`: what follows on the stream.
+#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+pub struct ClipImage {
+ /// `image/png` or `image/jpeg`.
+ pub mime: String,
+ pub size: u64,
+}
+
+impl ClipImage {
+ /// Whether the type and size are within the spec's limits.
+ pub fn is_valid(&self) -> bool {
+ clip::IMAGE_TYPES.contains(&self.mime.as_str()) && (1..=clip::MAX_IMAGE_BYTES).contains(&self.size)
+ }
+}
+
/// A paired device identity as stored in trust stores and exchanged in tests.
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
pub struct PeerIdentity {
diff --git a/desktop/app/src/clipboard.rs b/desktop/app/src/clipboard.rs
index a74cfae..924dd31 100644
--- a/desktop/app/src/clipboard.rs
+++ b/desktop/app/src/clipboard.rs
@@ -1,12 +1,13 @@
// SPDX-License-Identifier: GPL-3.0-or-later
//! The clipboard between this PC and paired phones
-//! (docs/protocol/clipboard.md): what the user copies goes to connected
-//! phones (unless turned off in Settings), "Send clipboard" sends it on
-//! demand, and text from a phone lands on the Windows clipboard.
+//! (docs/protocol/clipboard.md): what the user copies (text or an image)
+//! goes to connected phones (unless turned off in Settings), "Send
+//! clipboard" sends it on demand, and what a phone sends lands on the
+//! Windows clipboard.
use std::sync::atomic::{AtomicBool, Ordering};
-use nectarlink_core::{DeviceId, Error, FeatureState, LinkState, NodeEvent};
+use nectarlink_core::{CLIP_MAX_IMAGE_BYTES, DeviceId, Error, FeatureState, LinkState, Node, NodeEvent};
use crate::{
bridge::app::{describe, show_message},
@@ -41,20 +42,56 @@ fn receivers() -> Vec<DeviceId> {
})
}
-/// The user copied text on this PC.
-fn on_copy(text: String) {
+/// The user copied text or an image on this PC.
+fn on_copy(clip: Clip) {
if !AUTO_SEND.load(Ordering::Relaxed) {
return;
}
let Some(node) = core_host::node() else { return };
- for device in receivers() {
- let (node, text) = (node.clone(), text.clone());
- core_host::spawn(async move {
+ let devices = receivers();
+ if devices.is_empty() {
+ return;
+ }
+ core_host::spawn(async move {
+ let Some(content) = Content::from_clip(clip).await else { return };
+ for device in devices {
// Automatic: nothing to tell the user if a phone can't take it.
- if let Err(e) = node.send_clipboard(device, text).await {
+ if let Err(e) = content.send(&node, device).await {
tracing::debug!(device = %device.short(), error = %e, "clipboard not sent");
}
- });
+ }
+ });
+}
+
+/// What goes to a phone.
+enum Content {
+ Text(String),
+ Png(Vec<u8>),
+}
+
+impl Content {
+ /// Text as is, an image as PNG (converted off the async threads).
+ async fn from_clip(clip: Clip) -> Option<Content> {
+ match clip {
+ Clip::Text(text) => Some(Content::Text(text)),
+ Clip::Image(image) => match tokio::task::spawn_blocking(move || image.to_png()).await {
+ Ok(Ok(png)) => Some(Content::Png(png)),
+ Ok(Err(reason)) => {
+ tracing::warn!(reason, "can't read the copied image");
+ None
+ }
+ Err(_) => None,
+ },
+ Clip::Private | Clip::Empty => None,
+ }
+ }
+
+ async fn send(&self, node: &Node, device: DeviceId) -> Result<(), Error> {
+ match self {
+ Content::Text(text) => node.send_clipboard(device, text.clone()).await,
+ Content::Png(png) if png.len() as u64 > CLIP_MAX_IMAGE_BYTES => Err(Error::TooLarge),
+ Content::Png(png) => node.send_clipboard_image(device, "image/png".into(), png.clone()).await,
+ }
}
}
@@ -62,21 +99,26 @@ fn on_copy(text: String) {
pub fn send_now(device: DeviceId) {
let Some(node) = core_host::node() else { return };
let name = core_host::host().hub.read(|s| s.name_of(&device)).unwrap_or_else(|| "the phone".into());
- let text = match clipboard::read() {
- Clip::Text(text) => text,
+ let clip = clipboard::read();
+ let image = matches!(clip, Clip::Image(_));
+ match clip {
Clip::Private => {
- show_message("What you copied is marked private by its app, so it wasn't sent.");
- return;
- }
- Clip::Empty => {
- show_message("There's no text on the clipboard.");
- return;
+ return show_message("What you copied is marked private by its app, so it wasn't sent.");
}
- };
+ Clip::Empty => return show_message("There's no text or image on the clipboard."),
+ Clip::Text(_) | Clip::Image(_) => {}
+ }
core_host::spawn(async move {
- match node.send_clipboard(device, text).await {
+ let Some(content) = Content::from_clip(clip).await else {
+ return show_message("That image can't be read.");
+ };
+ match content.send(&node, device).await {
Ok(()) => show_message(format!("Sent to {name}.")),
+ Err(Error::TooLarge) if image => show_message("That image is too large to send."),
Err(Error::TooLarge) => show_message("That's too much text to send at once."),
+ Err(Error::Unsupported) if image => {
+ show_message(format!("Update Nectarlink on {name} to send it images."))
+ }
Err(Error::Denied) => show_message(format!("The clipboard is turned off for {name}.")),
Err(e) => show_message(describe(&e)),
}
diff --git a/desktop/app/src/main.rs b/desktop/app/src/main.rs
index fc37fdd..ba07113 100644
--- a/desktop/app/src/main.rs
+++ b/desktop/app/src/main.rs
@@ -59,6 +59,9 @@ impl Platform for DesktopPlatform {
fn set_clipboard(&self, text: &str) -> Result<(), String> {
win::clipboard::write(text)
}
+ fn set_clipboard_image(&self, mime: &str, bytes: &[u8]) -> Result<(), String> {
+ win::clipboard::write_image(mime, bytes)
+ }
}
/// Command-line options (also used by shortcuts and autostart).
diff --git a/desktop/app/src/win/clipboard.rs b/desktop/app/src/win/clipboard.rs
index a20098f..a63a3fb 100644
--- a/desktop/app/src/win/clipboard.rs
+++ b/desktop/app/src/win/clipboard.rs
@@ -1,13 +1,16 @@
// SPDX-License-Identifier: GPL-3.0-or-later
-//! The Windows clipboard: watching for copied text, reading it and writing
-//! text a phone sent (docs/protocol/clipboard.md).
+//! The Windows clipboard: watching for copied text and images, reading them
+//! and writing what a phone sent (docs/protocol/clipboard.md).
//!
//! A hidden message-only window on its own thread listens for clipboard
//! changes and owns what Nectarlink writes, so text that came from a phone
//! is never sent back. Content that password managers mark as private is
-//! never read for sending.
+//! never read for sending. Images are read as PNG when the app offers it,
+//! or else as a bitmap (converted to PNG off the clipboard thread), and
+//! written as a bitmap with alpha, plus PNG when that's what arrived.
use std::{
+ hash::{DefaultHasher, Hash, Hasher},
sync::{
Mutex, OnceLock,
atomic::{AtomicIsize, Ordering},
@@ -35,31 +38,80 @@ use windows::{
core::{HSTRING, w},
};
-/// Standard clipboard format for UTF-16 text.
+/// Standard clipboard formats.
+const CF_DIB: u32 = 8;
const CF_UNICODETEXT: u32 = 13;
+const CF_DIBV5: u32 = 17;
+/// Copied bitmaps larger than this aren't read (an 8K screenshot is
+/// about 130 MB raw).
+const MAX_BITMAP_BYTES: usize = 160 * 1024 * 1024;
+
+/// A copied image, as the app offered it.
+#[derive(Clone, PartialEq, Eq)]
+pub enum Image {
+ Png(Vec<u8>),
+ /// A device-independent bitmap (`CF_DIBV5` or `CF_DIB`).
+ Bitmap(Vec<u8>),
+}
+
+impl std::fmt::Debug for Image {
+ fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+ let (kind, len) = match self {
+ Image::Png(b) => ("Png", b.len()),
+ Image::Bitmap(b) => ("Bitmap", b.len()),
+ };
+ f.debug_struct(kind).field("bytes", &len).finish()
+ }
+}
+
+impl Image {
+ /// As a PNG file (bitmaps are converted; this can take a moment).
+ pub fn to_png(&self) -> Result<Vec<u8>, String> {
+ match self {
+ Image::Png(png) => Ok(png.clone()),
+ Image::Bitmap(dib) => {
+ let bmp = super::image::bmp_from_dib(dib).ok_or("not a bitmap Windows knows")?;
+ let bitmap = super::image::decode(&bmp).map_err(|e| e.to_string())?;
+ super::image::encode_png(&bitmap).map_err(|e| e.to_string())
+ }
+ }
+ }
+}
/// What's on the clipboard, for sending.
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum Clip {
Text(String),
+ Image(Image),
/// Marked private (a password manager): never sent.
Private,
- /// Nothing that's text.
+ /// Nothing that's text or an image.
Empty,
}
-type OnCopy = Box<dyn Fn(String) + Send + Sync>;
+type OnCopy = Box<dyn Fn(Clip) + Send + Sync>;
/// The listener window, once started (as an isize: HWND isn't Send).
static WINDOW: AtomicIsize = AtomicIsize::new(0);
static ON_COPY: OnceLock<OnCopy> = OnceLock::new();
-/// The last text reported, so one copy reported several times (apps often
-/// write the clipboard in steps) is sent once.
-static LAST_REPORTED: Mutex<Option<String>> = Mutex::new(None);
+/// A fingerprint of the last content reported, so one copy reported several
+/// times (apps often write the clipboard in steps) is sent once.
+static LAST_REPORTED: Mutex<Option<u64>> = Mutex::new(None);
+
+fn fingerprint(clip: &Clip) -> Option<u64> {
+ let mut hasher = DefaultHasher::new();
+ match clip {
+ Clip::Text(text) => text.hash(&mut hasher),
+ Clip::Image(Image::Png(bytes) | Image::Bitmap(bytes)) => bytes.hash(&mut hasher),
+ Clip::Private | Clip::Empty => return None,
+ }
+ Some(hasher.finish())
+}
-/// Starts watching the clipboard; `on_copy` gets each newly copied text
-/// that isn't private and didn't come from Nectarlink, on the watcher thread.
-pub fn start(on_copy: impl Fn(String) + Send + Sync + 'static) {
+/// Starts watching the clipboard; `on_copy` gets each newly copied text or
+/// image that isn't private and didn't come from Nectarlink, on the watcher
+/// thread.
+pub fn start(on_copy: impl Fn(Clip) + Send + Sync + 'static) {
if ON_COPY.set(Box::new(on_copy)).is_err() {
return; // already started
}
@@ -129,16 +181,17 @@ fn on_clipboard_changed(hwnd: HWND) {
if unsafe { GetClipboardOwner() }.is_ok_and(|owner| owner == hwnd) {
return;
}
- let Clip::Text(text) = read_with(Some(hwnd)) else { return };
+ let clip = read_with(Some(hwnd));
+ let Some(print) = fingerprint(&clip) else { return };
{
let mut last = LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner());
- if last.as_deref() == Some(text.as_str()) {
+ if *last == Some(print) {
return;
}
- *last = Some(text.clone());
+ *last = Some(print);
}
if let Some(on_copy) = ON_COPY.get() {
- on_copy(text);
+ on_copy(clip);
}
}
@@ -176,7 +229,48 @@ pub fn write(text: &str) -> Result<(), String> {
return Err(e.to_string());
}
}
- *LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner()) = Some(text.to_owned());
+ *LAST_REPORTED.lock().unwrap_or_else(|e| e.into_inner()) = fingerprint(&Clip::Text(text.to_owned()));
+ Ok(())
+}
+
+/// Puts an image a phone sent (`image/png` or `image/jpeg`) on the
+/// clipboard: as a bitmap with alpha, which every app pastes, and also as
+/// PNG when it is one, which keeps it exact for apps that read PNG.
+pub fn write_image(mime: &str, bytes: &[u8]) -> Result<(), String> {
+ let owner = window().ok_or("the clipboard watcher isn't running")?;
+ let bitmap = super::image::decode(bytes).map_err(|e| e.to_string())?;
+ let mut blocks = vec![(CF_DIBV5, super::image::dibv5(&bitmap))];
+ if mime == "image/png" {
+ blocks.push((format("PNG"), bytes.to_vec()));
+ }
+ let _open = Opened::open(Some(owner)).ok_or("the clipboard is busy")?;
+ // SAFETY: the clipboard is open.
+ unsafe { EmptyClipboard() }.map_err(|e| e.to_string())?;
+ for (format, data) in blocks {
+ set_block(format, &data)?;
+ }
+ Ok(())
+}
+
+/// Copies `data` into a global block and hands it to the clipboard, which
+/// must be open.
+fn set_block(format: u32, data: &[u8]) -> Result<(), String> {
+ // SAFETY: the block is sized for `data`; ownership passes to the
+ // clipboard only when SetClipboardData succeeds (it's freed otherwise).
+ unsafe {
+ let block = GlobalAlloc(GMEM_MOVEABLE, data.len()).map_err(|e| e.to_string())?;
+ let target = GlobalLock(block).cast::<u8>();
+ if target.is_null() {
+ let _ = GlobalFree(Some(block));
+ return Err("can't lock memory".into());
+ }
+ std::ptr::copy_nonoverlapping(data.as_ptr(), target, data.len());
+ let _ = GlobalUnlock(block);
+ if let Err(e) = SetClipboardData(format, Some(HANDLE(block.0))) {
+ let _ = GlobalFree(Some(block));
+ return Err(e.to_string());
+ }
+ }
Ok(())
}
@@ -214,7 +308,9 @@ fn format(name: &str) -> u32 {
fn read_with(owner: Option<HWND>) -> Clip {
// SAFETY: format availability checks need no open clipboard.
let has = |format: u32| format != 0 && unsafe { IsClipboardFormatAvailable(format) }.is_ok();
- if !has(CF_UNICODETEXT) {
+ let png = format("PNG");
+ let text = has(CF_UNICODETEXT);
+ if !text && !has(png) && !has(CF_DIBV5) && !has(CF_DIB) {
return Clip::Empty;
}
// Password managers and other apps mark private content this way.
@@ -225,6 +321,17 @@ fn read_with(owner: Option<HWND>) -> Clip {
if let Some(0) = read_dword(format("CanIncludeInClipboardHistory")) {
return Clip::Private;
}
+ // Apps that copy both (a table from a spreadsheet) mean the text.
+ if !text {
+ return [(png, true), (CF_DIBV5, false), (CF_DIB, false)]
+ .into_iter()
+ .filter(|&(format, _)| has(format))
+ .find_map(|(format, is_png)| {
+ let bytes = read_block(format)?;
+ Some(Clip::Image(if is_png { Image::Png(bytes) } else { Image::Bitmap(bytes) }))
+ })
+ .unwrap_or(Clip::Empty);
+ }
// SAFETY: the clipboard is open; the data handle is valid until it's
// closed, and the text is read within the block's size.
let text = unsafe {
@@ -244,6 +351,28 @@ fn read_with(owner: Option<HWND>) -> Clip {
if text.trim().is_empty() { Clip::Empty } else { Clip::Text(text) }
}
+/// A format's raw bytes, if present and not too large. The clipboard must
+/// be open.
+fn read_block(format: u32) -> Option<Vec<u8>> {
+ // SAFETY: the caller opened the clipboard; the block is read within its
+ // size while locked.
+ unsafe {
+ let handle = GetClipboardData(format).ok()?;
+ let block = HGLOBAL(handle.0);
+ let size = GlobalSize(block);
+ if size == 0 || size > MAX_BITMAP_BYTES {
+ return None;
+ }
+ let data = GlobalLock(block).cast::<u8>();
+ if data.is_null() {
+ return None;
+ }
+ let bytes = std::slice::from_raw_parts(data, size).to_vec();
+ let _ = GlobalUnlock(block);
+ Some(bytes)
+ }
+}
+
/// A DWORD-valued clipboard format, if present. The clipboard must be open.
fn read_dword(format: u32) -> Option<u32> {
if format == 0 {
diff --git a/desktop/app/src/win/image.rs b/desktop/app/src/win/image.rs
new file mode 100644
index 0000000..86b9331
--- /dev/null
+++ b/desktop/app/src/win/image.rs
@@ -0,0 +1,225 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+//! Image conversions for the clipboard, with WIC: what Windows apps copy
+//! (PNG, or a device-independent bitmap) to PNG for sending, and a PNG or
+//! JPEG from a phone to pixels for pasting.
+
+use windows::{
+ Win32::{
+ Graphics::Imaging::{
+ CLSID_WICImagingFactory, GUID_ContainerFormatPng, GUID_WICPixelFormat32bppBGRA,
+ IWICImagingFactory, WICBitmapDitherTypeNone, WICBitmapEncoderNoCache, WICBitmapPaletteTypeCustom,
+ WICDecodeMetadataCacheOnDemand,
+ },
+ System::Com::{CLSCTX_INPROC_SERVER, CoCreateInstance, STATFLAG_NONAME, STREAM_SEEK_SET},
+ UI::Shell::SHCreateMemStream,
+ },
+ core::{Error, HRESULT, Result},
+};
+
+use super::with_com;
+
+/// Straight-alpha BGRA pixels, top row first.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct Bitmap {
+ pub width: u32,
+ pub height: u32,
+ pub bgra: Vec<u8>,
+}
+
+fn invalid(message: &str) -> Error {
+ // E_INVALIDARG
+ Error::new(HRESULT(0x8007_0057_u32 as i32), message)
+}
+
+fn factory() -> Result<IWICImagingFactory> {
+ // SAFETY: creating the WIC factory; COM is initialized by the caller.
+ unsafe { CoCreateInstance(&CLSID_WICImagingFactory, None, CLSCTX_INPROC_SERVER) }
+}
+
+/// Decodes an image file in memory (PNG, JPEG, BMP and the other formats
+/// WIC knows).
+pub fn decode(bytes: &[u8]) -> Result<Bitmap> {
+ with_com(|| {
+ // SAFETY: WIC calls on live objects; the memory stream reads from
+ // `bytes`, which outlives the decoder, and the pixel buffer is sized
+ // for the converter's stride and height.
+ unsafe {
+ let factory = factory()?;
+ let stream = factory.CreateStream()?;
+ stream.InitializeFromMemory(bytes)?;
+ let decoder =
+ factory.CreateDecoderFromStream(&stream, std::ptr::null(), WICDecodeMetadataCacheOnDemand)?;
+ let frame = decoder.GetFrame(0)?;
+ let converter = factory.CreateFormatConverter()?;
+ converter.Initialize(
+ &frame,
+ &GUID_WICPixelFormat32bppBGRA,
+ WICBitmapDitherTypeNone,
+ None,
+ 0.0,
+ WICBitmapPaletteTypeCustom,
+ )?;
+ let (mut width, mut height) = (0, 0);
+ converter.GetSize(&mut width, &mut height)?;
+ let len = (width as usize).checked_mul(height as usize).and_then(|n| n.checked_mul(4));
+ let mut bgra = vec![0u8; len.ok_or_else(|| invalid("image too large"))?];
+ converter.CopyPixels(std::ptr::null(), width * 4, &mut bgra)?;
+ Ok(Bitmap { width, height, bgra })
+ }
+ })
+}
+
+/// Encodes pixels as a PNG file.
+pub fn encode_png(bitmap: &Bitmap) -> Result<Vec<u8>> {
+ with_com(|| {
+ // SAFETY: as in `decode`; the bitmap is created from (a copy of)
+ // the pixels, and the memory stream is read back within its size.
+ unsafe {
+ let factory = factory()?;
+ let source = factory.CreateBitmapFromMemory(
+ bitmap.width,
+ bitmap.height,
+ &GUID_WICPixelFormat32bppBGRA,
+ bitmap.width * 4,
+ &bitmap.bgra,
+ )?;
+ let stream = SHCreateMemStream(None).ok_or_else(|| invalid("no memory stream"))?;
+ let encoder = factory.CreateEncoder(&GUID_ContainerFormatPng, std::ptr::null())?;
+ encoder.Initialize(&stream, WICBitmapEncoderNoCache)?;
+ let mut frame = None;
+ encoder.CreateNewFrame(&mut frame, std::ptr::null_mut())?;
+ let frame = frame.ok_or_else(|| invalid("no frame"))?;
+ frame.Initialize(None)?;
+ frame.WriteSource(&source, std::ptr::null())?;
+ frame.Commit()?;
+ encoder.Commit()?;
+
+ let mut stat = Default::default();
+ stream.Stat(&mut stat, STATFLAG_NONAME)?;
+ let mut png = vec![0u8; usize::try_from(stat.cbSize).map_err(|_| invalid("too large"))?];
+ stream.Seek(0, STREAM_SEEK_SET, None)?;
+ let mut read = 0;
+ stream
+ .Read(png.as_mut_ptr().cast(), u32::try_from(png.len()).unwrap_or(u32::MAX), Some(&mut read))
+ .ok()?;
+ png.truncate(read as usize);
+ Ok(png)
+ }
+ })
+}
+
+/// A device-independent bitmap (`CF_DIB` / `CF_DIBV5`) as a BMP file, which
+/// WIC can decode: the DIB with a file header in front.
+pub fn bmp_from_dib(dib: &[u8]) -> Option<Vec<u8>> {
+ let u32_at = |at: usize| dib.get(at..at + 4).map(|b| u32::from_le_bytes([b[0], b[1], b[2], b[3]]));
+ let header = u32_at(0)?;
+ let bit_count = dib.get(14..16).map(|b| u16::from_le_bytes([b[0], b[1]]))?;
+ let compression = u32_at(16)?;
+ let colors_used = u32_at(32)?;
+ if !(40..=124).contains(&header) {
+ return None;
+ }
+ // Color masks follow a plain BITMAPINFOHEADER (later headers hold them).
+ let masks = match (header, compression) {
+ (40, 3) => 12, // BI_BITFIELDS
+ (40, 6) => 16, // BI_ALPHABITFIELDS
+ _ => 0,
+ };
+ let palette = match colors_used {
+ 0 if bit_count <= 8 => 4u32 << bit_count,
+ n => n.checked_mul(4)?,
+ };
+ let offset = 14 + header.checked_add(masks)?.checked_add(palette)?;
+ let size = u32::try_from(14 + dib.len()).ok()?;
+ if offset > size {
+ return None;
+ }
+ let mut bmp = Vec::with_capacity(size as usize);
+ bmp.extend_from_slice(b"BM");
+ bmp.extend_from_slice(&size.to_le_bytes());
+ bmp.extend_from_slice(&[0; 4]);
+ bmp.extend_from_slice(&offset.to_le_bytes());
+ bmp.extend_from_slice(dib);
+ Some(bmp)
+}
+
+/// Pixels as a `CF_DIBV5` block: a BITMAPV5HEADER (32-bit, straight alpha,
+/// sRGB) and the rows bottom-up, which is what apps read most reliably.
+pub fn dibv5(bitmap: &Bitmap) -> Vec<u8> {
+ let (width, height) = (bitmap.width, bitmap.height);
+ let mut dib = Vec::with_capacity(124 + bitmap.bgra.len());
+ let mut put = |bytes: &[u8]| dib.extend_from_slice(bytes);
+ put(&124u32.to_le_bytes());
+ put(&(width as i32).to_le_bytes());
+ put(&(height as i32).to_le_bytes());
+ put(&1u16.to_le_bytes());
+ put(&32u16.to_le_bytes());
+ put(&3u32.to_le_bytes()); // BI_BITFIELDS
+ put(&(width * height * 4).to_le_bytes());
+ put(&2835i32.to_le_bytes()); // 72 dpi
+ put(&2835i32.to_le_bytes());
+ put(&[0; 8]); // colors used, important
+ for mask in [0x00FF_0000u32, 0x0000_FF00, 0x0000_00FF, 0xFF00_0000] {
+ put(&mask.to_le_bytes());
+ }
+ put(&0x7352_4742u32.to_le_bytes()); // LCS_sRGB
+ put(&[0; 36 + 12]); // endpoints, gamma
+ put(&4u32.to_le_bytes()); // LCS_GM_IMAGES
+ put(&[0; 12]); // profile data, size, reserved
+ let row = width as usize * 4;
+ for line in bitmap.bgra.chunks_exact(row.max(1)).rev() {
+ dib.extend_from_slice(line);
+ }
+ dib
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ fn checker(width: u32, height: u32) -> Bitmap {
+ let bgra = (0..width * height)
+ .flat_map(|i| {
+ let (x, y) = (i % width, i / width);
+ if (x + y) % 2 == 0 { [255, 0, 0, 255] } else { [0, 0, 255, 128] }
+ })
+ .collect();
+ Bitmap { width, height, bgra }
+ }
+
+ #[test]
+ fn png_round_trips_with_alpha() {
+ let bitmap = checker(5, 3);
+ let png = encode_png(&bitmap).unwrap();
+ assert_eq!(&png[1..4], b"PNG");
+ assert_eq!(decode(&png).unwrap(), bitmap);
+ }
+
+ #[test]
+ fn copied_bitmaps_decode() {
+ let bitmap = checker(7, 4);
+ let dib = dibv5(&bitmap);
+ assert_eq!(dib.len(), 124 + 7 * 4 * 4);
+ let decoded = decode(&bmp_from_dib(&dib).unwrap()).unwrap();
+ assert_eq!((decoded.width, decoded.height), (7, 4));
+ // Top-left stays top-left (rows are stored bottom-up).
+ assert_eq!(&decoded.bgra[..4], &[255, 0, 0, 255]);
+
+ // A plain 24-bit BITMAPINFOHEADER DIB, as older apps copy.
+ let mut old = Vec::new();
+ for field in [40u32, 2, 2] {
+ old.extend_from_slice(&field.to_le_bytes());
+ }
+ old.extend_from_slice(&1u16.to_le_bytes());
+ old.extend_from_slice(&24u16.to_le_bytes());
+ old.extend_from_slice(&[0; 24]);
+ // Bottom row: green, green; top row: white, black (rows padded to 4).
+ old.extend_from_slice(&[0, 255, 0, 0, 255, 0, 0, 0]);
+ old.extend_from_slice(&[255, 255, 255, 0, 0, 0, 0, 0]);
+ let decoded = decode(&bmp_from_dib(&old).unwrap()).unwrap();
+ assert_eq!(&decoded.bgra[..4], &[255, 255, 255, 255]);
+ assert_eq!(&decoded.bgra[8..12], &[0, 255, 0, 255]);
+
+ assert_eq!(bmp_from_dib(&[1, 2, 3]), None);
+ }
+}
diff --git a/desktop/app/src/win/mod.rs b/desktop/app/src/win/mod.rs
index 77be6c2..e4b5430 100644
--- a/desktop/app/src/win/mod.rs
+++ b/desktop/app/src/win/mod.rs
@@ -7,6 +7,7 @@
pub mod clipboard;
pub mod icon;
+pub mod image;
pub mod net;
pub mod shortcut;
pub mod single_instance;
@@ -19,6 +20,7 @@ use windows::{
Win32::{
Foundation::ERROR_SUCCESS,
System::{
+ Com::{COINIT_APARTMENTTHREADED, CoInitializeEx, CoUninitialize},
Power::{GetSystemPowerStatus, SYSTEM_POWER_STATUS},
Registry::{
HKEY, HKEY_CURRENT_USER, HKEY_LOCAL_MACHINE, RRF_RT_REG_DWORD, RRF_RT_REG_SZ, RegGetValueW,
@@ -116,6 +118,19 @@ pub fn reduce_motion() -> bool {
ok && !enabled.as_bool()
}
+/// Runs `f` with COM available on this thread.
+pub fn with_com<T>(f: impl FnOnce() -> T) -> T {
+ // SAFETY: balanced below; if the thread already has COM in another mode
+ // (RPC_E_CHANGED_MODE), it's usable as is and isn't uninitialized here.
+ let initialized = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED) }.is_ok();
+ let result = f();
+ if initialized {
+ // SAFETY: pairs with the successful CoInitializeEx above.
+ unsafe { CoUninitialize() };
+ }
+ result
+}
+
#[cfg(test)]
mod tests {
use super::*;
diff --git a/desktop/app/src/win/shortcut.rs b/desktop/app/src/win/shortcut.rs
index 103624a..a9f83c6 100644
--- a/desktop/app/src/win/shortcut.rs
+++ b/desktop/app/src/win/shortcut.rs
@@ -5,15 +5,14 @@ use std::path::{Path, PathBuf};
use windows::{
Win32::{
- System::Com::{
- CLSCTX_INPROC_SERVER, COINIT_APARTMENTTHREADED, CoCreateInstance, CoInitializeEx, CoTaskMemFree,
- CoUninitialize, IPersistFile,
- },
+ System::Com::{CLSCTX_INPROC_SERVER, CoCreateInstance, CoTaskMemFree, IPersistFile},
UI::Shell::{FOLDERID_SendTo, IShellLinkW, KF_FLAG_DEFAULT, SHGetKnownFolderPath, ShellLink},
},
core::{HSTRING, Interface},
};
+use super::with_com;
+
/// What a shortcut starts.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct Shortcut {
@@ -23,19 +22,6 @@ pub struct Shortcut {
pub description: String,
}
-/// Runs `f` with COM available on this thread.
-fn with_com<T>(f: impl FnOnce() -> T) -> T {
- // SAFETY: balanced below; if the thread already has COM in another mode
- // (RPC_E_CHANGED_MODE), it's usable as is and isn't uninitialized here.
- let initialized = unsafe { CoInitializeEx(None, COINIT_APARTMENTTHREADED) }.is_ok();
- let result = f();
- if initialized {
- // SAFETY: pairs with the successful CoInitializeEx above.
- unsafe { CoUninitialize() };
- }
- result
-}
-
/// Writes (or replaces) a shortcut at `path`, with the target's own icon.
pub fn write(path: &Path, shortcut: &Shortcut) -> windows::core::Result<()> {
with_com(|| {
diff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.md
index 76234b5..f988939 100644
--- a/docs/protocol/capabilities.md
+++ b/docs/protocol/capabilities.md
@@ -22,6 +22,7 @@ capability is missing.
| `device.battery` | Sends `event.battery` | always offered |
| `device.ring` | Rings on `device.ring`, even on silent | always offered |
| `files.transfer` | Sends and receives files ([files service](files.md)) | app update |
+| `clip.image` | Accepts images on its clipboard ([clipboard service](clipboard.md)) | app update |
| `media.control` | Exposes its media sessions for remote control (phone: needs notification access) | phone: notification access · desktop: app update |
## Phone
diff --git a/docs/protocol/clipboard.md b/docs/protocol/clipboard.md
index 1c59018..d7b0972 100644
--- a/docs/protocol/clipboard.md
+++ b/docs/protocol/clipboard.md
@@ -3,13 +3,12 @@
> Status: **draft.** Builds on [protocol v0](v0.md); breaking changes are
> allowed until v1. License: CC BY 4.0.
-Copied text moves between paired devices: a PC sends what the user copies
-(automatically, unless turned off), and a phone sends its clipboard when
-the user asks (Android only lets the app in front read the clipboard;
-reading it in the background needs the Elevated power level).
+Copied text and images move between paired devices: a PC sends what the
+user copies (automatically, unless turned off), and a phone sends its
+clipboard when the user asks (Android only lets the app in front read the
+clipboard; reading it in the background needs the Elevated power level).
-Images, rich text and files on the clipboard are not part of this version;
-they will travel over bulk streams with file transfer.
+Rich text and files on the clipboard are not part of this version.
## 1. Capabilities
@@ -18,6 +17,7 @@ they will travel over bulk streams with file transfer.
| `clip.write` | phone | Accepts `clip.set` |
| `clip.share` | phone | Sends its clipboard when the user asks |
| `clip.read.auto` | phone | Sends clipboard changes on its own (Elevated) |
+| `clip.image` | both | Accepts images (§3) |
A PC always accepts `clip.set` from a phone the user allows.
@@ -36,7 +36,34 @@ Errors: `DENIED` when the user turned the clipboard off for the sending
device (the `clipboard` device toggle), `INTERNAL` when the clipboard
couldn't be written.
-## 3. Rules
+## 3. Images
+
+An image travels on its own bidirectional stream (v0 §4), so a large
+screenshot never holds up the control stream. Only devices that offer
+`clip.image` are sent images. The sender opens the stream and writes:
+
+```
+t = "stream" b = { svc: "clip", op: "image", v: 1 }
+t = "clip.image" b = { mime: text, size: uint }
+```
+
+followed by exactly `size` bytes of the image, then finishes its side.
+`mime` is `image/png` or `image/jpeg`; senders convert other formats (a
+copied bitmap on Windows, a WebP on Android) to PNG. `size` is 1 byte to
+32 MiB.
+
+The receiver puts the image on its clipboard and answers on the same
+stream with `ok`, or with `error`: `BAD_MESSAGE` for a type or size out of
+bounds, `DENIED` as for `clip.set`, `INTERNAL` when the clipboard couldn't
+be written. A receiver that refuses early stops reading (QUIC
+`STOP_SENDING`) and still sends its answer. The whole exchange times out
+after 60 seconds.
+
+Receivers offer the image the way their platform's apps paste it best:
+Windows as a bitmap with alpha (`CF_DIBV5`) plus the `PNG` format when a
+PNG arrived; Android as a content URI from the app's own provider.
+
+## 4. Rules
- **Consent.** Each device sends and accepts only with the `clipboard`
toggle on for the other device.