Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.
Docs: app windows in the mirroring protocol and the README
Beed91534c94d34
diff --git a/README.md b/README.mdindex 4b61383..0049ab2 100644--- a/README.md+++ b/README.md@@ -100,6 +100,10 @@ tests; there are no releases yet.debugging** there once for real touch (live dragging) and keys, withnothing else to install; Nectarlink turns Wireless debugging back on byitself after a reboot.+- **Phone apps in windows** on the PC: with Wireless debugging set up+ (Android 11 and later), pick any of the phone's apps and it opens in a+ window of its own, running on the phone beside whatever its screen+ shows; use it with the mouse and keyboard, and open several at once.- **Texts** on the PC: a Messages page with the phone's conversations(pictures included), to reply or start a new one; texts go out throughthe phone. Search them, copy a one-time code with one click, and open@@ -136,8 +140,7 @@ tests; there are no releases yet.against their SHA-256 sums.- **Command-line client** for development and testing (`nectarlink`).-**Next**: phone apps in windows of their own on the PC, contacts and the-call log, talking on calls through the PC (Bluetooth), and the rest of the+**Next**: contacts and the call log, talking on calls through the PC (Bluetooth), and the rest of the[plan](docs/PLAN.md).## Screenshotsdiff --git a/docs/protocol/capabilities.md b/docs/protocol/capabilities.mdindex ac955ac..d360217 100644--- a/docs/protocol/capabilities.md+++ b/docs/protocol/capabilities.md@@ -45,7 +45,7 @@ capability is missing.| `photos.read` | Announces new photos and screenshots and sends them on request ([photos](photos.md)) | photos permission || `mirror.capture` | Streams its screen (with the system's capture prompt at Basic; [mirroring](mirror.md)) | app update || `mirror.input` | Accepts touch, key and scroll input while mirrored | Assist |-| `mirror.virtual_display` | Runs apps on a separate virtual display (app windows) | Elevated |+| `mirror.virtual_display` | Runs apps on displays of their own, shown in windows on the PC ([mirroring](mirror.md)) | Elevated, Android 11+ || `mirror.audio` | Streams all audio while mirrored | Elevated || `mirror.audio.playback` | Streams audio from apps that allow playback capture | app update || `camera.stream` | Streams a camera for webcam use | camera permission |diff --git a/docs/protocol/mirror.md b/docs/protocol/mirror.mdindex 61c2fd7..3a70fc9 100644--- a/docs/protocol/mirror.md+++ b/docs/protocol/mirror.md@@ -6,7 +6,8 @@A PC shows a phone's screen. The phone's user agrees each time (Androidasks); the phone then streams its screen as H.264 on a stream of its own,and its sound, when the PC asks, on another (§5). The PC can also use the-phone with its mouse and keyboard (§4).+phone with its mouse and keyboard (§4), and open the phone's apps in+windows of their own (§6).## 1. Capabilities@@ -17,6 +18,7 @@ phone with its mouse and keyboard (§4).| `mirror.input` | phone | Takes the PC's mouse and keyboard while mirrored (Assist: an accessibility service the user turns on; Elevated: injected input) || `mirror.audio.playback` | phone | Shares the sound of apps that allow it while mirrored (Android 10+) || `mirror.audio` | phone | Shares all its sound while mirrored (planned, Elevated) |+| `mirror.virtual_display` | phone | Runs apps on displays of their own, shown in windows on the PC (Elevated, Android 11+) || `mirror.listen` | PC | Plays a mirrored phone's sound |Both devices must allow it: the `mirroring` device toggle, on by default,@@ -27,12 +29,19 @@ on each side.On the control stream.```-t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool }-t = "mirror.stop" id = n-t = "mirror.keyframe"+t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool,+ ? session: uint, ? app: text }+t = "mirror.stop" id = n b = { ? session: uint }+t = "mirror.keyframe" b = { ? session: uint }+t = "mirror.apps" id = n+t = "mirror.apps" re = n b = { apps: [{ pkg: text, label: text, ? icon: bytes }] }t = "ok" re = n```+Each mirroring is a **session** the PC numbers: 0 (the default) is the+phone's screen; any other is an app window (§6). A missing `session`+means 0, so PCs and phones that predate sessions work as before.+- `mirror.start` (PC → phone): show me your screen, its longer side at most`max_size` pixels, at most `fps` frames a second, aiming for `bitrate`bits a second, and with `audio` (default false), its sound too. The@@ -121,7 +130,33 @@ heard. Sharing all sound (`mirror.audio`) is planned for Elevated.The phone's own speaker keeps playing; the PC can mute its copy.-## 6. Latency over completeness+## 6. App windows++With `mirror.virtual_display`, a PC can open one of the phone's apps in a+window of its own, beside whatever the phone's screen shows:++- `mirror.apps` (PC → phone) lists the apps that can open: launchable ones,+ by name, each with its package name and a small PNG icon (at most 8 KiB;+ at most 500 apps, and icons may be left out to keep the answer small).+ Errors: `DENIED`, `UNSUPPORTED` (no `mirror.virtual_display`).+- `mirror.start` with `app` (a package name) and a `session` other than 0+ runs that app on a display of its own and streams it like the screen,+ with no prompt (Elevated was the user's consent). Its video stream's+ config packets carry `session` (the config's CBOR gains `? session:+ uint`). Sound isn't sent for app windows. `BAD_MESSAGE` when `app` and+ `session` don't go together.+- `mirror.input` gains `? session: uint`: input for an app window goes to+ its display. App windows take input whenever the phone offers+ `mirror.virtual_display`; the screen still needs `mirror.input`.+- `mirror.stop` and `mirror.keyframe` with a `session` are about that+ window; stopping one closes the app's display (the app closes with it).++On Android, the Elevated helper (running as the shell user) makes a+virtual display for each window with an H.264 encoder drawing from it, and+starts the app there; the phone shows a notification while any app window+is open, with a button to close them all.++## 7. Latency over completenessThe stream is reliable, so a slow network would otherwise build up delay.Instead, the phone keeps only a few packets waiting to go out: when that's@@ -136,7 +171,7 @@ plays silence and refills it when the network stalls, and drops the oldestsound when more than 200 ms piles up, so sound stays in step with thepicture.-## 7. Privacy+## 8. PrivacyThe screen and sound are never recorded or logged by either side. Android shows thatthe screen is being shared (in the status bar), and the user can stop it