Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.
docs: BLE protocol between the board and the iOS app
Henryf9948d27e17a
diff --git a/docs/ble-protocol.md b/docs/ble-protocol.mdnew file mode 100644index 0000000..5e6b507--- /dev/null+++ b/docs/ble-protocol.md@@ -0,0 +1,73 @@+# BLE protocol (ESP32 ⇄ iOS app)++The board is a BLE peripheral named `ESP32 Agent XXXX` (XXXX = last 2 MAC bytes, hex).+Security: LE Secure Connections, bonding, Just Works. `net` requires an encrypted link,+so reading it triggers the iOS pairing prompt.++Service `6e400001-a9e8-4b1b-9c39-6d6f8a3c0000`++| Characteristic | UUID suffix | Props | Purpose |+|---|---|---|---|+| `cmd` | `...0001` | write, write-no-rsp | one command line, UTF-8, `<id> <command>` |+| `reply` | `...0002` | notify | `<id> ok` or `<id> err <reason>` |+| `status` | `...0003` | notify, read | JSON heartbeat, every 1 s (also keeps the app alive in background) |+| `video` | `...0004` | notify | H.264 chunks, see below |+| `net` | `...0005` | read, write (encrypted) | network config JSON |++Full UUID = `6e400001-a9e8-4b1b-9c39-6d6f8a3c` + 4-hex suffix, e.g. cmd = `6e400001-a9e8-4b1b-9c39-6d6f8a3c0001`.++## cmd+`<id>` is a decimal request id chosen by the app; the reply echoes it.+Existing HID commands, unchanged semantics from the old TCP protocol:+`ping`, `home`, `move DX DY`, `press`, `release`, `click`, `scroll A B`, `type TEXT`, `key COMBO`.+New:+- `stream on|off` — continuously forward every access unit over `video` (default off).+- `snap` — send SPS/PPS + every AU since the last IDR (the current GOP), then reply `ok`.+- `res W H` — AirPlay resolution advertised in `/info` (default 480 1040), stored in NVS; takes effect on the next mirroring session.+- `fps N` — max fps advertised (default 5), stored in NVS.+- `bench KB` — send KB kilobytes of filler on `video` with the `bench` flag, then reply `ok <ms>`.+- `unpair` — erase all bonds and restart advertising.+- `coex [bt|balance|wifi]` — Wi-Fi/BLE radio-share preference (A/B knob, not stored; every boot starts at+ `balance`). Replies `ok <current>`; bare `coex` just reports it.+When the link drops, the board releases all buttons/keys (same as before).++## status (JSON, one notify per second, must fit one MTU; keep short keys)+```json+{"up":123456,"rst":"power-on","heap":81234,"psram":6200000,"usb":"ok|asleep|off",+ "net":"ap|sta|none","ip":"192.168.4.1","sta":1,+ "ap":"idle|connected|streaming","w":480,"h":1040,"kbps":180,"fps":4.8,"idr":2000,"gop":42000,+ "ble":{"mtu":247,"phy":2,"kbps":350,"drop":0,"rdrop":0,"n":1}}+```+`kbps` = AirPlay video bitrate in, `ble.kbps` = video bytes out over BLE, `idr` = ms between IDRs, `gop` = bytes buffered since last IDR.+`rst` = reset reason of the current boot (`power-on|brownout|watchdog|crash|software|usb|other`), `up` = ms since boot.+`ble.drop` = video notifies given up, `ble.rdrop` = replies that never reached a subscribed peer (should stay 0),+`ble.n` = connected centrals.++Send priority: replies and status are queued and always go out before further video; video keeps at most+~8 ACL packets (~2 KB) in the controller, so a reply waits behind at most that much video. In `stream on`,+a P-frame more than 800 ms old is skipped and the stream resumes at the next IDR (a slow link degrades to+IDR-only instead of building latency).++## video+Each notify: `[seq u16 LE][flags u8][payload]`, payload up to MTU-3-3 bytes.+flags: bit0 START of access unit, bit1 END of access unit, bit2 IDR, bit3 CONFIG (payload is SPS+PPS),+bit4 BENCH (filler, ignore), bit5 SNAP (part of a snap reply).+Payload is Annex-B (`00 00 00 01` start codes). On a START chunk the first 4 payload bytes are a u32 LE+board timestamp (ms) for latency stats, followed by the Annex-B data.+`seq` increments per notify; the app counts gaps as drops and waits for the next IDR/CONFIG after a gap.++## net (JSON)+Read: `{"mode":"ap","ssid":"ESP32Agent-1A2B","pass":"<random 12 chars>","ip":"192.168.4.1"}`+(SoftAP credentials are generated once at first boot and kept in NVS).+Write `{"mode":"ap"}` or `{"mode":"sta","ssid":"...","pass":"..."}` — stored in NVS, board restarts Wi-Fi.++## AirPlay+Receiver name `ESP32 Agent`, mirroring only (no audio), H.264 only, resolution/fps from NVS.++## Deviations (firmware notes)+- `status`: if the negotiated MTU is too small for the full JSON (MTU-3 < ~255 bytes), the notify carries a+ reduced object `{"up","rst","usb","net","ap","ble":{"mtu"}}`; a `status` read always returns the last full JSON.+- `sta` in `status` = number of stations on the board's hotspot (mode `ap`), or 1/0 = connected (mode `sta`).+- `net` read in `sta` mode returns the station network's `ssid`/`pass`.+- `snap` while mirroring has stopped returns the last buffered GOP (check `"ap"` in status); `snap` without+ any video replies `err no video ...`.