Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.
firmware: watchdog, fast reconnect, never block on a sleeping phone; docs
Henry1fce18425e16
diff --git a/README.md b/README.mdindex 6f1e9c9..eb77612 100644--- a/README.md+++ b/README.md@@ -55,7 +55,7 @@ uses the same paths a person uses — the screen and a pointer — so the phone| Part | Notes ||---|---|-| **ESP32-S3 board** | Tested with a bare [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (~$7, no camera add-on needed). Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. |+| **ESP32-S3 board** | Tested with a bare [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (~$7, no camera add-on needed); attach the antenna that comes with it, as the XIAO has no on-board antenna and Wi-Fi drops often without it. Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. || **iPhone** | USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. || **Computer** | macOS or Linux, on the same network as the phone; runs UxPlay and the server. || **Vision model** | Any OpenAI-compatible chat API that accepts images. Running a model locally is optional. |@@ -106,7 +106,8 @@ Open `http://<computer-ip>:8890` in a browser.1. **Settings → Accessibility → Touch → AssistiveTouch**: turn it on.2. Plug the board into the phone. A round pointer appears; the web page shows *controller connected*.3. **Control Center → Screen Mirroring → iPhone Agent**. The live screen appears in the web page.-4. In the web page, open **Manual controls → Calibrate**, then click exactly on the pointer in the live screen.+4. In the web page, open **Manual controls → Calibrate**. The server moves the pointer, finds it on the live screen+ and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.### 4. Give it a task@@ -118,6 +119,8 @@ Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**,- **Live log** — every plan, thought and action as it happens, with timings.- **Step wall** — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus themodel's reasoning and how long thinking and acting took. Click a card to zoom in.+- **Taps** are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the **Motion** switch+ turns the animation off. Taps the model marked `precise` (pointer checked before tapping) are labelled in the log.- **Emergency stop** (or <kbd>Esc</kbd>) stops the agent and releases the pointer. **Clear history** empties the log.- English and Simplified Chinese interface.@@ -130,6 +133,7 @@ Set in `server/.env` (see [`.env.example`](server/.env.example)):| `LLM_BASE_URL` | `http://localhost:8000/v1` | OpenAI-compatible endpoint || `LLM_MODEL` / `LLM_API_KEY` | — | Model name and API key || `LLM_EXTRA_BODY` | `{}` | JSON merged into every request (e.g. to turn off a model's thinking mode) |+| `LLM_TOKEN_PARAM` | `auto` | Token-limit field: `max_tokens`, `max_completion_tokens` (newer OpenAI models), or `auto` to detect || `ACTOR_*` / `PLANNER_*` | — | Per-role overrides of any `LLM_*` variable || `AIRPLAY_NAME` | `iPhone Agent` | Receiver name in the phone's Screen Mirroring list || `PORT` / `DEVICE_PORT` | `8890` / `7300` | Web UI port / port the board connects to |@@ -169,7 +173,8 @@ Set `HIDAGENT_DEBUG=1` to log every board command with its latency and to save o## Board protocol-The board holds no logic. The server sends one command per line over TCP; the board answers `ok` or `err <reason>`.+The board holds no logic. On connecting it sends `hello <last reset reason> <uptime ms>`; then the server sends one+command per line over TCP and the board answers `ok` or `err <reason>`.| Command | Effect ||---|---|@@ -181,7 +186,9 @@ The board holds no logic. The server sends one command per line over TCP; the bo| `type <text>` | Type ASCII text || `key <combo>` | Press keys, e.g. `enter`, `cmd+h`, `cmd+space` |-If the Wi-Fi link drops, the board releases all buttons and reconnects on its own.+If the Wi-Fi link drops, the board releases all buttons and reconnects on its own. While the phone is locked, its USB+port sleeps: the board then tries to wake it and otherwise answers `err phone usb asleep` instead of waiting. A watchdog+restarts the board if it ever stops responding.## Project layout@@ -194,7 +201,8 @@ server/ agent, web UI, screen capture and board link (Python, FastAPI)## Roadmap- [ ] Test a fully local setup with DiffusionGemma (`server/serve/` has an experimental OpenAI-compatible server)-- [ ] Correct the pointer only when needed, to make taps faster+- [ ] Landscape apps (screen coordinates and calibration assume portrait)+- [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)- [ ] More example tasks across popular apps## Limitations@@ -203,6 +211,7 @@ server/ agent, web UI, screen capture and board link (Python, FastAPI)- Mirroring stops when the phone locks; keep auto-lock off while the agent works.- Keyboard input is ASCII only.- Re-calibrate after changing AssistiveTouch tracking speed.+- Landscape apps are not supported yet: drawings and taps there land in the wrong place.- On Linux machines with several network interfaces, limit Avahi (mDNS) to the interface on the phone's networkif the phone does not list the receiver.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex 9891564..0fd9f2c 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -50,7 +50,7 @@ USB 鼠标加键盘。大脑可以是任何兼容 OpenAI 接口的视觉模型| 部件 | 说明 ||---|---|-| **ESP32-S3 开发板** | 实测使用 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/) 裸板(约 50 元,不需要摄像头扩展板)。任何 USB-C 口直连 S3 原生 USB 的 ESP32-S3 板子都可以。 |+| **ESP32-S3 开发板** | 实测使用 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/) 裸板(约 50 元,不需要摄像头扩展板)。记得接上附带的天线:XIAO 板上没有天线,不接的话 Wi-Fi 会经常断。任何 USB-C 口直连 S3 原生 USB 的 ESP32-S3 板子都可以。 || **iPhone** | USB-C 机型直接插;Lightning 机型用 Lightning 转 USB 相机转换器。板子由手机供电。 || **电脑** | macOS 或 Linux,和手机在同一个局域网;运行 UxPlay 和服务端。 || **视觉模型** | 任何支持图片输入、兼容 OpenAI 接口的对话模型。本地部署是可选的。 |@@ -100,7 +100,8 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent1. **设置 → 辅助功能 → 触控 → 辅助触控**:打开。2. 把板子插到手机上,屏幕上会出现一个圆形指针,网页显示控制器已连接。3. **控制中心 → 屏幕镜像 → iPhone Agent**,网页里出现实时画面。-4. 在网页里打开 **手动操控 → 校准**,然后在实时画面里精确点一下指针的位置。+4. 在网页里打开 **手动操控 → 校准**。服务端会移动指针、在实时画面里找到它,几秒内自动完成校准;+ 如果找不到指针,会提示你在画面里精确点一下指针。### 4. 下达任务@@ -112,6 +113,7 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent- **实时日志**:每一个计划、思考和动作实时显示,带耗时。- **步骤截图墙**:每一步模型做决定时看到的截图,上面画出点按或滑动的位置,附带模型的理由和思考、执行各用了多久;点卡片可以放大。+- **点按**用红色圆圈和扩散的波纹标出,滑动和笔画用黄色轨迹;**动效**开关可以关掉动画。模型标为 `precise`(点之前先确认指针位置)的点按会在日志里注明。- **紧急停止**(或按 <kbd>Esc</kbd>):立即停止 Agent 并松开指针。**清空记录**:清空日志和截图墙。- 界面支持英文和简体中文。@@ -124,6 +126,7 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent| `LLM_BASE_URL` | `http://localhost:8000/v1` | 兼容 OpenAI 的接口地址 || `LLM_MODEL` / `LLM_API_KEY` | — | 模型名称和 API Key || `LLM_EXTRA_BODY` | `{}` | 合并进每次请求的 JSON(例如关闭模型的思考模式) |+| `LLM_TOKEN_PARAM` | `auto` | 输出长度参数名:`max_tokens`、`max_completion_tokens`(新版 OpenAI 模型),或 `auto` 自动识别 || `ACTOR_*` / `PLANNER_*` | — | 单独覆盖执行模型 / 规划模型的任意 `LLM_*` 配置 || `AIRPLAY_NAME` | `iPhone Agent` | 手机"屏幕镜像"列表里显示的名字 || `PORT` / `DEVICE_PORT` | `8890` / `7300` | 网页端口 / 板子连接的端口 |@@ -162,7 +165,7 @@ LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'## 板子协议-板子本身不含任何逻辑。服务端通过 TCP 每行发一条命令,板子回复 `ok` 或 `err <原因>`。+板子本身不含任何逻辑。连上时先发送 `hello <上次重启原因> <开机毫秒数>`,之后服务端通过 TCP 每行发一条命令,板子回复 `ok` 或 `err <原因>`。| 命令 | 作用 ||---|---|@@ -174,7 +177,7 @@ LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'| `type <text>` | 输入 ASCII 文字 || `key <combo>` | 按键,例如 `enter`、`cmd+h`、`cmd+space` |-Wi-Fi 断开时,板子会松开所有按键并自动重连。+Wi-Fi 断开时,板子会松开所有按键并自动重连。手机锁屏时 USB 会休眠:板子会先尝试唤醒手机,唤醒不了就立刻回复 `err phone usb asleep`,不会卡住等待。万一板子没有响应,看门狗会让它自动重启。## 目录结构@@ -187,7 +190,8 @@ server/ Agent、网页界面、屏幕采集和板子连接(Python,FastAP## 路线图- [ ] 测试基于 DiffusionGemma 的纯本地方案(`server/serve/` 里有实验性的 OpenAI 兼容服务)-- [ ] 指针校正改为按需执行,让点按更快+- [ ] 支持横屏 App(目前坐标和校准都按竖屏计算)+- [ ] 非 ASCII 文字输入(例如通过 iOS 拼音键盘输入中文)- [ ] 更多常用 App 的示例任务## 限制@@ -196,6 +200,7 @@ server/ Agent、网页界面、屏幕采集和板子连接(Python,FastAP- 手机锁屏后镜像会断开,Agent 工作时请关闭自动锁定。- 键盘只支持 ASCII 输入。- 修改辅助触控的跟踪速度后需要重新校准。+- 暂不支持横屏 App:在横屏界面里的点按和绘图位置会偏。- 在有多个网卡的 Linux 机器上,如果手机列表里看不到接收端,把 Avahi(mDNS)限制在手机所在网络的网卡上。## 合理使用diff --git a/hardware/hid_bridge/hid_bridge.ino b/hardware/hid_bridge/hid_bridge.inoindex 5a89e6a..6a8e2ff 100644--- a/hardware/hid_bridge/hid_bridge.ino+++ b/hardware/hid_bridge/hid_bridge.ino@@ -6,9 +6,11 @@// scroll <v> <h> | type <ascii text> | key <combo, e.g. cmd+h, enter>#include <WiFi.h>+#include <esp_task_wdt.h>#include "USB.h"#include "USBHIDMouse.h"#include "USBHIDKeyboard.h"+#include "tusb.h"#include "secrets.h" // WIFI_SSID, WIFI_PASS, SERVER_HOST, SERVER_PORTconst int STEP = 10; // max pointer delta per report while moving@@ -18,6 +20,8 @@ const int HOME_REPORTS = 50;#define DEBUG_SERIAL 0 // 1: echo every command over USB serial (needs CDCOnBoot=cdc)#endif+const int WATCHDOG_MS = 8000; // a hung loop (e.g. blocked USB) reboots the board instead of staying offline+USBHIDMouse mouse;USBHIDKeyboard keyboard;WiFiClient server;@@ -61,6 +65,16 @@ bool pressCombo(String combo) {return true;}+// A locked or sleeping phone suspends USB; sending HID reports then can stall. Ask it to wake up instead,+// and report "asleep" right away if it does not, so the board never hangs.+bool usbReady() {+ if (tud_suspended()) {+ tud_remote_wakeup();+ for (int i = 0; i < 50 && tud_suspended(); i++) delay(10);+ }+ return tud_mounted() && !tud_suspended();+}+String run(const String &line) {int sp = line.indexOf(' ');String cmd = sp < 0 ? line : line.substring(0, sp);@@ -69,6 +83,7 @@ String run(const String &line) {sscanf(arg.c_str(), "%ld %ld", &a, &b);if (cmd == "ping") return "ok";+ if (!usbReady()) return "err phone usb asleep (unlock the phone)";if (cmd == "home") { // slam the pointer into the top-left cornerfor (int i = 0; i < HOME_REPORTS; i++) { mouse.move(-127, -127); delay(2); }return "ok";@@ -83,6 +98,18 @@ String run(const String &line) {return "err unknown command";}+const char *resetReason() {+ switch (esp_reset_reason()) {+ case ESP_RST_POWERON: return "power-on";+ case ESP_RST_BROWNOUT: return "brownout";+ case ESP_RST_TASK_WDT: case ESP_RST_INT_WDT: case ESP_RST_WDT: return "watchdog";+ case ESP_RST_PANIC: return "crash";+ case ESP_RST_SW: return "software";+ case ESP_RST_USB: return "usb";+ default: return "other";+ }+}+void setup() {#if DEBUG_SERIALSerial.begin(115200);@@ -91,20 +118,27 @@ void setup() {keyboard.begin();USB.productName("HID Bridge");USB.begin();+ esp_task_wdt_config_t wdt = {.timeout_ms = WATCHDOG_MS, .idle_core_mask = 0, .trigger_panic = true};+ esp_task_wdt_reconfigure(&wdt);+ esp_task_wdt_add(NULL);WiFi.setHostname("hid-bridge");WiFi.setSleep(false); // modem sleep adds ~100 ms per command+ WiFi.setAutoReconnect(true);WiFi.begin(WIFI_SSID, WIFI_PASS);}void loop() {- if (WiFi.status() != WL_CONNECTED) { mouse.release(MOUSE_LEFT); delay(500); return; }+ esp_task_wdt_reset();+ if (WiFi.status() != WL_CONNECTED) { mouse.release(MOUSE_LEFT); delay(50); return; } // auto-reconnectsif (!server.connected()) {mouse.release(MOUSE_LEFT); // a gesture cut off mid-way must not leave the button held (long press)keyboard.releaseAll();- if (!server.connect(SERVER_HOST, SERVER_PORT)) { delay(2000); return; }+ if (!server.connect(SERVER_HOST, SERVER_PORT, 1000)) { delay(200); return; } // retry fast: show up at onceserver.setNoDelay(true);+ server.printf("hello %s %lu\n", resetReason(), millis()); // why the board last restarted, and uptime#if DEBUG_SERIAL- Serial.printf("connected to %s:%d as %s\n", SERVER_HOST, SERVER_PORT, WiFi.localIP().toString().c_str());+ Serial.printf("connected to %s:%d as %s after %lu ms\n", SERVER_HOST, SERVER_PORT,+ WiFi.localIP().toString().c_str(), millis());#endif}if (server.available()) {