Skip to content

Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.

6.6 KB
Raw Blame

Media service (protocol v0)

Status: draft. Builds on protocol v0; breaking changes are allowed until v1. License: CC BY 4.0.

What plays on one device shows on the other, which can control it: a phone's music in the PC's app and in Windows' media flyout, a PC's in the phone's media controls. Both directions use the same messages.

1. Capabilities

ID Offered by Meaning
media.control both Shares its players and takes commands for them (a phone needs notification access to read other apps' players)
media.remote both Shows other devices' players and sends commands to them
media.watch phone Shows a PC's playback streamed to it (Play on phone, §4)
media.stream PC Streams a player's picture and sound on request (§4)

A device sends its players only to devices that offer media.remote, and only while the user allows media for that device (the media device toggle, on by default). It takes commands only while it offers media.control and allows the sender.

2. Messages

All on the control stream.

t = "media.state"    b = { players: [Player] }          // no reply
t = "media.sync"     b = {}                             // no reply
t = "media.command"  id = n  b = { player: text, action: text, position?: uint }
t = "ok"             re = n

media.state carries every player the sender shares, the most relevant first (what Windows calls the current session, or what's playing, then the most recently active). It's sent when a session starts, after every change, and again when the user changes what the other device may see. An empty list means nothing plays (or the user stopped sharing); the receiver clears what it showed. A receiver also clears a device's players when the connection ends.

media.sync asks for a media.state now (the user allowed media again on the receiving side).

media.command asks a player to play, pause, go to the next or previous item, or seek to position (milliseconds). Errors: NOT_FOUND when the player is gone, UNSUPPORTED when it can't do that, DENIED when the user turned media off for the sender, BAD_MESSAGE for a seek without a position.

Player = {
  id:        text,       // 1 to 256 bytes, stable while the player exists
  app:       text,       // user-visible app name
  title?:    text,       // each text up to 256 characters
  artist?:   text,
  album?:    text,
  playing:   bool,
  duration?: uint,       // milliseconds
  position?: uint,       // milliseconds, when sent
  actions:   [text],     // what it takes: play, pause, next, previous, seek
  art_key?:  text,       // 1 to 64 bytes; the same key means the same picture
  art?:      bytes,      // JPEG or PNG, up to 256 KiB
}

While playing, the position moves on in real time from when the state arrived; senders don't send updates just because time passes.

Artwork goes once per art_key and session: later states carry the key alone, and receivers keep pictures by key. A device sends at most 8 players. Receivers drop actions they don't know.

3. Rules

  • No echoes. A device leaves its own stand-in for another device's player (Windows' flyout session, Android's media session) out of what it shares.
  • Implementations MUST NOT log titles, artists or albums (v0 §11).

4. Play on phone

A PC can move what one of its players plays to a phone. How depends on what plays, and the PC decides:

  • Streaming services that protect their video (Netflix, Prime Video, JioHotstar, Disney+ and the like) come out black when captured. The PC sends the page's address with link.open (actions.md), which Android hands to the service's app, and pauses its own player.
  • A video file a desktop player plays (VLC, Media Player and the like) is offered as a file: the phone reads it from the PC and plays it itself, from where the PC had got to, at its full quality.
  • Everything else is streamed from the PC as it plays, picture and sound, so whatever the PC does to it carries over (a browser's ad blocker on YouTube, for one). The PC keeps playing.

Streaming starts with an offer from the PC:

t = "media.watch"  id = n  b = { player: text, title?: text, file?: WatchFile }   // PC to phone
t = "ok"           re = n

WatchFile = { name: text, size: uint, position: uint }

player is the PC's ID for the player, as in media.state; title is what it plays, up to 256 characters, to show with the offer. The phone shows the offer (Android doesn't let an app open a screen from the background, so usually as a notification) and answers ok. Errors: UNSUPPORTED when the phone doesn't offer media.watch, DENIED when media is off for the PC, BAD_MESSAGE for an empty or too long player or title.

To watch, the phone sends mirror.start with watch set to that player, session 0 and audio true (mirror.md §2). The PC allows it only for a player it offered that phone, while the offer is open: for 10 minutes after the offer or after watching last stopped, so the phone can come back to it, and for as long as the connection lasts. It then streams the part of its screen the player's picture shows in (a browser's video, else its page, else its window) on the mirroring video stream, and its sound as Phone speaker does. Input from the phone is ignored while it watches. Anything else in that mirror.start is as for the PC's screen; without an offer the answer is DENIED, and the PC doesn't ask its user to share the screen.

Playing a file

With file, the phone plays the file itself instead of watching a stream. name is the file's name (1 to 256 characters, no / or \), for the phone's player to tell the format by; size its size in bytes (more than 0); position where the PC had got to, in milliseconds. A PC offers a file only in a format phones play (MP4, MKV, WebM, MOV, 3GP, MPEG-TS), and streams anything else. Once the phone plays it, it pauses the PC's player with media.command.

The phone reads the file on bidirectional streams of their own, one read each:

stream header  { svc: "media", op: "file", v: 1 }
t = "media.file"       id = 1  b = { offset: uint, length: uint }   // phone to PC
t = "media.file.meta"  re = 1  b = { size: uint, length: uint }     // PC to phone
<length bytes>

The PC answers with at most 4 MiB (length is cut to that, and to the end of the file; past the end it is 0), then the bytes, and finishes the stream. It serves only the file of its open offer to that phone (the same 10 minutes as for watching, renewed by every read), so a phone can't read anything else on the PC; without one the answer is DENIED. A newer offer, streamed or not, replaces it.