Skip to content

Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.

docs: BLE protocol between the board and the iOS app

Henryf9948d27e17a
diff --git a/docs/ble-protocol.md b/docs/ble-protocol.md
new file mode 100644
index 0000000..5e6b507
--- /dev/null
+++ b/docs/ble-protocol.md
@@ -0,0 +1,73 @@
+# BLE protocol (ESP32 ⇄ iOS app)
+
+The board is a BLE peripheral named `ESP32 Agent XXXX` (XXXX = last 2 MAC bytes, hex).
+Security: LE Secure Connections, bonding, Just Works. `net` requires an encrypted link,
+so reading it triggers the iOS pairing prompt.
+
+Service `6e400001-a9e8-4b1b-9c39-6d6f8a3c0000`
+
+| Characteristic | UUID suffix | Props | Purpose |
+|---|---|---|---|
+| `cmd` | `...0001` | write, write-no-rsp | one command line, UTF-8, `<id> <command>` |
+| `reply` | `...0002` | notify | `<id> ok` or `<id> err <reason>` |
+| `status` | `...0003` | notify, read | JSON heartbeat, every 1 s (also keeps the app alive in background) |
+| `video` | `...0004` | notify | H.264 chunks, see below |
+| `net` | `...0005` | read, write (encrypted) | network config JSON |
+
+Full UUID = `6e400001-a9e8-4b1b-9c39-6d6f8a3c` + 4-hex suffix, e.g. cmd = `6e400001-a9e8-4b1b-9c39-6d6f8a3c0001`.
+
+## cmd
+`<id>` is a decimal request id chosen by the app; the reply echoes it.
+Existing HID commands, unchanged semantics from the old TCP protocol:
+`ping`, `home`, `move DX DY`, `press`, `release`, `click`, `scroll A B`, `type TEXT`, `key COMBO`.
+New:
+- `stream on|off` — continuously forward every access unit over `video` (default off).
+- `snap` — send SPS/PPS + every AU since the last IDR (the current GOP), then reply `ok`.
+- `res W H` — AirPlay resolution advertised in `/info` (default 480 1040), stored in NVS; takes effect on the next mirroring session.
+- `fps N` — max fps advertised (default 5), stored in NVS.
+- `bench KB` — send KB kilobytes of filler on `video` with the `bench` flag, then reply `ok <ms>`.
+- `unpair` — erase all bonds and restart advertising.
+- `coex [bt|balance|wifi]` — Wi-Fi/BLE radio-share preference (A/B knob, not stored; every boot starts at
+ `balance`). Replies `ok <current>`; bare `coex` just reports it.
+When the link drops, the board releases all buttons/keys (same as before).
+
+## status (JSON, one notify per second, must fit one MTU; keep short keys)
+```json
+{"up":123456,"rst":"power-on","heap":81234,"psram":6200000,"usb":"ok|asleep|off",
+ "net":"ap|sta|none","ip":"192.168.4.1","sta":1,
+ "ap":"idle|connected|streaming","w":480,"h":1040,"kbps":180,"fps":4.8,"idr":2000,"gop":42000,
+ "ble":{"mtu":247,"phy":2,"kbps":350,"drop":0,"rdrop":0,"n":1}}
+```
+`kbps` = AirPlay video bitrate in, `ble.kbps` = video bytes out over BLE, `idr` = ms between IDRs, `gop` = bytes buffered since last IDR.
+`rst` = reset reason of the current boot (`power-on|brownout|watchdog|crash|software|usb|other`), `up` = ms since boot.
+`ble.drop` = video notifies given up, `ble.rdrop` = replies that never reached a subscribed peer (should stay 0),
+`ble.n` = connected centrals.
+
+Send priority: replies and status are queued and always go out before further video; video keeps at most
+~8 ACL packets (~2 KB) in the controller, so a reply waits behind at most that much video. In `stream on`,
+a P-frame more than 800 ms old is skipped and the stream resumes at the next IDR (a slow link degrades to
+IDR-only instead of building latency).
+
+## video
+Each notify: `[seq u16 LE][flags u8][payload]`, payload up to MTU-3-3 bytes.
+flags: bit0 START of access unit, bit1 END of access unit, bit2 IDR, bit3 CONFIG (payload is SPS+PPS),
+bit4 BENCH (filler, ignore), bit5 SNAP (part of a snap reply).
+Payload is Annex-B (`00 00 00 01` start codes). On a START chunk the first 4 payload bytes are a u32 LE
+board timestamp (ms) for latency stats, followed by the Annex-B data.
+`seq` increments per notify; the app counts gaps as drops and waits for the next IDR/CONFIG after a gap.
+
+## net (JSON)
+Read: `{"mode":"ap","ssid":"ESP32Agent-1A2B","pass":"<random 12 chars>","ip":"192.168.4.1"}`
+(SoftAP credentials are generated once at first boot and kept in NVS).
+Write `{"mode":"ap"}` or `{"mode":"sta","ssid":"...","pass":"..."}` — stored in NVS, board restarts Wi-Fi.
+
+## AirPlay
+Receiver name `ESP32 Agent`, mirroring only (no audio), H.264 only, resolution/fps from NVS.
+
+## Deviations (firmware notes)
+- `status`: if the negotiated MTU is too small for the full JSON (MTU-3 < ~255 bytes), the notify carries a
+ reduced object `{"up","rst","usb","net","ap","ble":{"mtu"}}`; a `status` read always returns the last full JSON.
+- `sta` in `status` = number of stations on the board's hotspot (mode `ap`), or 1/0 = connected (mode `sta`).
+- `net` read in `sta` mode returns the station network's `ssid`/`pass`.
+- `snap` while mirroring has stopped returns the last buffered GOP (check `"ap"` in status); `snap` without
+ any video replies `err no video ...`.