Skip to content

Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.

Docs: app windows in the mirroring protocol and the README

Beed91534c94d34
diff --git a/README.md b/README.md
index 4b61383..0049ab2 100644
--- a/README.md
+++ b/README.md
@@ -100,6 +100,10 @@ tests; there are no releases yet.
debugging** there once for real touch (live dragging) and keys, with
nothing else to install; Nectarlink turns Wireless debugging back on by
itself after a reboot.
+- **Phone apps in windows** on the PC: with Wireless debugging set up
+ (Android 11 and later), pick any of the phone's apps and it opens in a
+ window of its own, running on the phone beside whatever its screen
+ shows; use it with the mouse and keyboard, and open several at once.
- **Texts** on the PC: a Messages page with the phone's conversations
(pictures included), to reply or start a new one; texts go out through
the phone. Search them, copy a one-time code with one click, and open
@@ -136,8 +140,7 @@ tests; there are no releases yet.
against their SHA-256 sums.
- **Command-line client** for development and testing (`nectarlink`).
-**Next**: phone apps in windows of their own on the PC, contacts and the
-call log, talking on calls through the PC (Bluetooth), and the rest of the
+**Next**: contacts and the call log, talking on calls through the PC (Bluetooth), and the rest of the
[plan](docs/PLAN.md).
## Screenshots
diff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.md
index ac955ac..d360217 100644
--- a/docs/protocol/capabilities.md
+++ b/docs/protocol/capabilities.md
@@ -45,7 +45,7 @@ capability is missing.
| `photos.read` | Announces new photos and screenshots and sends them on request ([photos](photos.md)) | photos permission |
| `mirror.capture` | Streams its screen (with the system's capture prompt at Basic; [mirroring](mirror.md)) | app update |
| `mirror.input` | Accepts touch, key and scroll input while mirrored | Assist |
-| `mirror.virtual_display` | Runs apps on a separate virtual display (app windows) | Elevated |
+| `mirror.virtual_display` | Runs apps on displays of their own, shown in windows on the PC ([mirroring](mirror.md)) | Elevated, Android 11+ |
| `mirror.audio` | Streams all audio while mirrored | Elevated |
| `mirror.audio.playback` | Streams audio from apps that allow playback capture | app update |
| `camera.stream` | Streams a camera for webcam use | camera permission |
diff --git a/docs/protocol/mirror.md b/docs/protocol/mirror.md
index 61c2fd7..3a70fc9 100644
--- a/docs/protocol/mirror.md
+++ b/docs/protocol/mirror.md
@@ -6,7 +6,8 @@
A PC shows a phone's screen. The phone's user agrees each time (Android
asks); the phone then streams its screen as H.264 on a stream of its own,
and its sound, when the PC asks, on another (§5). The PC can also use the
-phone with its mouse and keyboard (§4).
+phone with its mouse and keyboard (§4), and open the phone's apps in
+windows of their own (§6).
## 1. Capabilities
@@ -17,6 +18,7 @@ phone with its mouse and keyboard (§4).
| `mirror.input` | phone | Takes the PC's mouse and keyboard while mirrored (Assist: an accessibility service the user turns on; Elevated: injected input) |
| `mirror.audio.playback` | phone | Shares the sound of apps that allow it while mirrored (Android 10+) |
| `mirror.audio` | phone | Shares all its sound while mirrored (planned, Elevated) |
+| `mirror.virtual_display` | phone | Runs apps on displays of their own, shown in windows on the PC (Elevated, Android 11+) |
| `mirror.listen` | PC | Plays a mirrored phone's sound |
Both devices must allow it: the `mirroring` device toggle, on by default,
@@ -27,12 +29,19 @@ on each side.
On the control stream.
```
-t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool }
-t = "mirror.stop" id = n
-t = "mirror.keyframe"
+t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool,
+ ? session: uint, ? app: text }
+t = "mirror.stop" id = n b = { ? session: uint }
+t = "mirror.keyframe" b = { ? session: uint }
+t = "mirror.apps" id = n
+t = "mirror.apps" re = n b = { apps: [{ pkg: text, label: text, ? icon: bytes }] }
t = "ok" re = n
```
+Each mirroring is a **session** the PC numbers: 0 (the default) is the
+phone's screen; any other is an app window (§6). A missing `session`
+means 0, so PCs and phones that predate sessions work as before.
+
- `mirror.start` (PC → phone): show me your screen, its longer side at most
`max_size` pixels, at most `fps` frames a second, aiming for `bitrate`
bits a second, and with `audio` (default false), its sound too. The
@@ -121,7 +130,33 @@ heard. Sharing all sound (`mirror.audio`) is planned for Elevated.
The phone's own speaker keeps playing; the PC can mute its copy.
-## 6. Latency over completeness
+## 6. App windows
+
+With `mirror.virtual_display`, a PC can open one of the phone's apps in a
+window of its own, beside whatever the phone's screen shows:
+
+- `mirror.apps` (PC → phone) lists the apps that can open: launchable ones,
+ by name, each with its package name and a small PNG icon (at most 8 KiB;
+ at most 500 apps, and icons may be left out to keep the answer small).
+ Errors: `DENIED`, `UNSUPPORTED` (no `mirror.virtual_display`).
+- `mirror.start` with `app` (a package name) and a `session` other than 0
+ runs that app on a display of its own and streams it like the screen,
+ with no prompt (Elevated was the user's consent). Its video stream's
+ config packets carry `session` (the config's CBOR gains `? session:
+ uint`). Sound isn't sent for app windows. `BAD_MESSAGE` when `app` and
+ `session` don't go together.
+- `mirror.input` gains `? session: uint`: input for an app window goes to
+ its display. App windows take input whenever the phone offers
+ `mirror.virtual_display`; the screen still needs `mirror.input`.
+- `mirror.stop` and `mirror.keyframe` with a `session` are about that
+ window; stopping one closes the app's display (the app closes with it).
+
+On Android, the Elevated helper (running as the shell user) makes a
+virtual display for each window with an H.264 encoder drawing from it, and
+starts the app there; the phone shows a notification while any app window
+is open, with a button to close them all.
+
+## 7. Latency over completeness
The stream is reliable, so a slow network would otherwise build up delay.
Instead, the phone keeps only a few packets waiting to go out: when that's
@@ -136,7 +171,7 @@ plays silence and refills it when the network stalls, and drops the oldest
sound when more than 200 ms piles up, so sound stays in step with the
picture.
-## 7. Privacy
+## 8. Privacy
The screen and sound are never recorded or logged by either side. Android shows that
the screen is being shared (in the status bar), and the user can stop it