Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.
Screen mirroring (protocol v0)
Status: draft. Builds on protocol v0; breaking changes are allowed until v1. License: CC BY 4.0.
A PC shows a phone's screen. The phone's user agrees each time (Android asks); the phone then streams its screen as H.264 or, when both sides support it, H.265 on a stream of its own, and its sound, when the PC asks, on another (§5). The PC can also use the phone with its mouse and keyboard (§4), and open the phone's apps in windows of their own (§6).
1. Capabilities
| ID | Offered by | Meaning |
|---|---|---|
mirror.capture |
phone | Shares its screen when its user agrees |
mirror.view |
PC | Shows a phone's screen |
mirror.decode.hevc |
PC or phone | Decodes H.265 screen-mirroring streams |
mirror.input |
phone | Takes the PC's mouse and keyboard while mirrored (Assist: an accessibility service the user turns on; Elevated: injected input) |
mirror.audio.playback |
phone | Shares the sound of apps that allow it while mirrored (Android 10+) |
mirror.audio |
phone | Shares all its sound while mirrored (planned, Elevated) |
mirror.sound |
phone | Shares its sound on its own, without the screen (sound_only, §5) |
mirror.virtual_display |
phone | Runs apps on displays of their own, shown in windows on the PC (Elevated, Android 11+) |
mirror.listen |
PC | Plays a mirrored phone's sound |
Both devices must allow it: the mirroring device toggle, on by default,
on each side.
2. Control messages
On the control stream.
t = "mirror.start" id = n b = { max_size: uint, fps: uint, bitrate: uint, ? audio: bool,
? session: uint, ? app: text,
? stay_awake: bool, ? screen_off: bool, ? watch: text,
? sound_only: bool }
t = "mirror.stop" id = n b = { ? session: uint }
t = "mirror.power" b = { ? stay_awake: bool, ? screen_off: bool }
t = "mirror.keyframe" b = { ? session: uint }
t = "mirror.resize" b = { session: uint, width: uint, height: uint }
t = "mirror.apps" id = n
t = "mirror.apps" re = n b = { apps: [{ pkg: text, label: text, ? icon: bytes }] }
t = "ok" re = n
Each mirroring is a session the PC numbers: 0 (the default) is the
phone's screen; any other is an app window (§6). A missing session
means 0, so PCs and phones that predate sessions work as before.
mirror.start(PC → phone): show me your screen, its longer side at mostmax_sizepixels, at mostfpsframes a second, aiming forbitratebits a second, and withaudio(default false), its sound too. For the phone's screen (session0),stay_awakekeeps the phone's screen from timing out while mirrored, andscreen_off(Elevated) turns the phone's physical display panel off while the mirror keeps running. The phone answersokonce it has asked its user; video follows only if the user agrees, and sound only if the phone can (it may share the screen without it). Errors:DENIED(mirroring is off for the PC),UNSUPPORTED(nomirror.capture).sound_only(PC → phone,session0 withaudio, never withapp,watch,stay_awakeorscreen_off): only the phone's sound, no video stream; the PC plays it without a window. Sent only to a phone that offersmirror.sound; one that doesn't answersUNSUPPORTED, so an older phone is never asked for its screen instead.mirror.stop(session 0) ends it.watch(phone → PC,session0 only, never withapp): Play on phone. Show what this PC player plays instead of the PC's screen, after the PC offered it withmedia.watch(media.md §4).mirror.stop(either way): stop. The PC stops reading the video stream; the phone stops sharing, restores normal display power, and finishes it.mirror.power(PC → phone): updatestay_awakeandscreen_offwhile mirroring the phone's screen (session0). Not answered.mirror.keyframe(PC → phone): the decoder lost its place; send a keyframe next. Not answered.mirror.resize(PC → phone): resize an app window's display (session != 0) towidth × heightpixels. Not answered; the phone sends a newconfigpacket on the video stream once its encoder restarts at the new size.
3. The video stream
The phone opens a bidirectional stream with the header
{ svc: "mirror", op: "video", v: 1 } (v0 §4), then writes packets until
it stops. The PC writes nothing; it finishes its side, or stops reading,
to end the stream.
Each packet is:
u32 length (big-endian; counts everything after it)
u8 kind // 0 config, 1 frame, 2 keyframe
u64 time // capture time in microseconds, big-endian (only differences mean anything)
data
- config: the stream's format, CBOR
{ codec: "h264" | "h265", width: uint, height: uint }. First, and again whenever it changes (the phone turned). - frame: one H.264 or H.265 access unit (Annex B) that depends on earlier ones.
- keyframe: an access unit that starts fresh (an IDR/IRAP picture), with the codec's parameter sets before it (SPS/PPS for H.264; VPS/SPS/PPS for H.265).
Packets are at most 8 MiB. Video is BT.709, limited range.
The PC can also send its screen to the phone using the same packet format.
It uses H.265 when the phone advertises mirror.decode.hevc and the PC's
HEVC encoder starts successfully; otherwise it uses H.264. If the codec changes
during a stream, the sender sends a new config packet.
4. Input
t = "mirror.input" b = { type: "touch", action: "down" | "move" | "up", x: float, y: float }
| { type: "scroll", x: float, y: float, dx: float, dy: float }
| { type: "key", key: text }
| { type: "text", text: text }
| { type: "gamepad", buttons: uint, lx: int, ly: int, rx: int, ry: int, lt: uint, rt: uint }
Sent by the PC, not answered (input is only worth it right away), and
only to phones that offer mirror.input. Positions are fractions of the
screen as the phone shows it now (0 at the left or top, 1 at the right or
bottom), so they hold at any size and rotation.
touch: one finger. At Elevated, the phone injects each down, move and up as it comes, so dragging is live. At Assist, Android plays a gesture whole, so the phone turns each down and up into a tap (barely moved), a long press (held 450 ms or more) or a swipe along the way it went, when it comes up.scroll: the mouse wheel at a point, in notches (positive: down or right); the phone swipes the other way.key:back,home,recents,notifications,enter,backspace,delete,left,right,up,down,tab,copy,cut,select_all,keyboard_onorkeyboard_off. Others are dropped (so phones from beforecopy,cutandselect_allignore them).copy,cutandselect_allact on the focused text field.keyboard_onandkeyboard_offmark when a PC starts or stops typing on the phone (session0) without opening a video stream; the phone shows a banner with a button to stop it (which sendsmirror.stopforsession0 back to the PC).text: typed text (at most 4 KiB), into the focused text field. When the accessibility service (InputService) is running, non-ASCII / Unicode text is inserted directly into the focused editable node; at Elevated without the accessibility service, ASCII/Latin text is typed as key events and non-ASCII text is pasted via the phone's clipboard (KEYCODE_PASTE).gamepad: controller state forwarded from the PC's connected gamepad (e.g. XInput).buttonsis a 16-bit XInput-compatible button bitmask (D-pad up/down/left/right, Start, Back, L3, R3, LB, RB, Guide, A, B, X, Y);lx,ly,rx,ryare signed 16-bit thumbstick axes (-32768..=32767);ltandrtare unsigned 8-bit trigger axes (0..=255). At Elevated, the phone injects matchingKeyEvent(SOURCE_GAMEPAD) andMotionEvent(SOURCE_JOYSTICK) inputs on the target display.
The Windows app maps the left button to a finger, right-click to Back,
middle-click to Home, Ctrl+C to syncing the phone's clipboard back to the PC,
and Ctrl+V to syncing the PC's clipboard (clip.set) and pasting/typing into
the phone. Dragging and dropping files onto the mirror canvas sends them to
the phone via the files service (files.offer). The app also offers a
Type on phone bar on Home (and in the Command Palette) to type on the
phone with the PC keyboard without mirroring its screen.
5. Sound
When the PC asked for it, the phone opens a second stream with the header
{ svc: "mirror", op: "audio", v: 1 }, with packets as in §3:
- config: the sound's format, CBOR
{ codec: "pcm_s16le", rate: uint, channels: uint }: 16-bit little-endian PCM, channels interleaved, 8 to 96 kHz, mono or stereo. First, and again if it changes. - frame: a few milliseconds of sound (Android sends 10 ms of 48 kHz stereo, 1,920 bytes), whole sample frames only.
A PC that doesn't offer mirror.listen resets the stream. The stream ends
with the video: when either side stops mirroring, or the connection drops.
With sound_only there's no video: the sound stream is all there is, and
it ends when either side stops or the connection drops.
On Android, the sound is what apps play (media, games, and apps that don't
say otherwise), captured with the screen share's consent and the
permission to record audio, which the phone asks for the first time a PC
wants the sound. Apps that don't allow playback capture, and calls, aren't
heard. Sharing all sound (mirror.audio) is planned for Elevated.
While mirroring, the phone's own speaker keeps playing with playback
capture, and the PC can mute its copy. Through Elevated the sound comes
from the remote submix, which takes it off the speaker. With only the sound
(sound_only), the phone turns its media volume to 0 for playback capture
and gives it back afterwards: a loopback capture is fixed volume, so the PC
still gets it at full level.
6. App windows
With mirror.virtual_display, a PC can open one of the phone's apps in a
window of its own, beside whatever the phone's screen shows:
mirror.apps(PC → phone) lists the apps that can open: launchable ones, by name, each with its package name and a small PNG icon (at most 8 KiB; at most 500 apps, and icons may be left out to keep the answer small). Errors:DENIED,UNSUPPORTED(nomirror.virtual_display).mirror.startwithapp(a package name) and asessionother than 0 runs that app on a display of its own and streams it like the screen, with no prompt (Elevated was the user's consent). Its video stream's config packets carrysession(the config's CBOR gains? session: uint). Sound isn't sent for app windows.BAD_MESSAGEwhenappandsessiondon't go together.mirror.inputgains? session: uint: input for an app window goes to its display. App windows take input whenever the phone offersmirror.virtual_display; the screen still needsmirror.input.mirror.resizewithsession,widthandheightresizes that window's virtual display (clamped to even dimensions within encoder limits, keeping density sensible) and restarts its H.264 encoder, sending a newconfigpacket and keyframe so the app re-lays out at the new aspect ratio.mirror.stopandmirror.keyframewith asessionare about that window; stopping one closes the app's display (the app closes with it). When the app closes its last activity on the phone, the phone finishes that session's video stream.
On Android, the Elevated helper (running as the shell user) makes a virtual display for each window with its display IME policy set to hide the soft keyboard, attaches an H.264 encoder drawing from it, and starts the app there; the phone shows a notification while any app window is open, with a button to close them all.
7. Latency over completeness
The stream is reliable, so a slow network would otherwise build up delay. Instead, the phone keeps only a few packets waiting to go out: when that's full it drops the picture and every one after it until a keyframe, and asks its encoder for one. The PC likewise decodes everything it gets but shows only the newest picture when it's behind, and asks for a keyframe when a picture doesn't decode.
Sound works the same way without keyframes: a packet that doesn't fit is dropped on its own. The PC plays from a short buffer (about 40 ms), plays silence and refills it when the network stalls, and drops the oldest sound when more than 200 ms piles up, so sound stays in step with the picture.
8. Privacy
The screen and sound are never recorded or logged by either side. Android shows that the screen is being shared (in the status bar), and the user can stop it there, from the phone's notification, or from the PC.