Skip to content

Let an AI model use a real iPhone with a $7 ESP32: AirPlay eyes, USB HID hands, any OpenAI-compatible vision model. No Mac, no developer account, no jailbreak.

firmware: watchdog, fast reconnect, never block on a sleeping phone; docs

Henry1fce18425e16
diff --git a/README.md b/README.md
index 6f1e9c9..eb77612 100644
--- a/README.md
+++ b/README.md
@@ -55,7 +55,7 @@ uses the same paths a person uses — the screen and a pointer — so the phone
| Part | Notes |
|---|---|
-| **ESP32-S3 board** | Tested with a bare [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (~$7, no camera add-on needed). Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. |
+| **ESP32-S3 board** | Tested with a bare [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/xiao_esp32s3_getting_started/) (~$7, no camera add-on needed); attach the antenna that comes with it, as the XIAO has no on-board antenna and Wi-Fi drops often without it. Any ESP32-S3 board whose USB-C port is wired to the S3's native USB works. |
| **iPhone** | USB-C models plug in directly. Lightning models need a Lightning to USB Camera Adapter. The phone powers the board. |
| **Computer** | macOS or Linux, on the same network as the phone; runs UxPlay and the server. |
| **Vision model** | Any OpenAI-compatible chat API that accepts images. Running a model locally is optional. |
@@ -106,7 +106,8 @@ Open `http://<computer-ip>:8890` in a browser.
1. **Settings → Accessibility → Touch → AssistiveTouch**: turn it on.
2. Plug the board into the phone. A round pointer appears; the web page shows *controller connected*.
3. **Control Center → Screen Mirroring → iPhone Agent**. The live screen appears in the web page.
-4. In the web page, open **Manual controls → Calibrate**, then click exactly on the pointer in the live screen.
+4. In the web page, open **Manual controls → Calibrate**. The server moves the pointer, finds it on the live screen
+ and calibrates itself in a few seconds; if it cannot see the pointer, it asks you to click exactly on it.
### 4. Give it a task
@@ -118,6 +119,8 @@ Type a goal (for example *"Open Settings and go to Wi-Fi"*) and press **Start**,
- **Live log** — every plan, thought and action as it happens, with timings.
- **Step wall** — the screenshot the model decided on for every step, with its tap or swipe drawn on top, plus the
model's reasoning and how long thinking and acting took. Click a card to zoom in.
+- **Taps** are marked with a red ring and a spreading ripple, swipes and strokes with a yellow path; the **Motion** switch
+ turns the animation off. Taps the model marked `precise` (pointer checked before tapping) are labelled in the log.
- **Emergency stop** (or <kbd>Esc</kbd>) stops the agent and releases the pointer. **Clear history** empties the log.
- English and Simplified Chinese interface.
@@ -130,6 +133,7 @@ Set in `server/.env` (see [`.env.example`](server/.env.example)):
| `LLM_BASE_URL` | `http://localhost:8000/v1` | OpenAI-compatible endpoint |
| `LLM_MODEL` / `LLM_API_KEY` | — | Model name and API key |
| `LLM_EXTRA_BODY` | `{}` | JSON merged into every request (e.g. to turn off a model's thinking mode) |
+| `LLM_TOKEN_PARAM` | `auto` | Token-limit field: `max_tokens`, `max_completion_tokens` (newer OpenAI models), or `auto` to detect |
| `ACTOR_*` / `PLANNER_*` | — | Per-role overrides of any `LLM_*` variable |
| `AIRPLAY_NAME` | `iPhone Agent` | Receiver name in the phone's Screen Mirroring list |
| `PORT` / `DEVICE_PORT` | `8890` / `7300` | Web UI port / port the board connects to |
@@ -169,7 +173,8 @@ Set `HIDAGENT_DEBUG=1` to log every board command with its latency and to save o
## Board protocol
-The board holds no logic. The server sends one command per line over TCP; the board answers `ok` or `err <reason>`.
+The board holds no logic. On connecting it sends `hello <last reset reason> <uptime ms>`; then the server sends one
+command per line over TCP and the board answers `ok` or `err <reason>`.
| Command | Effect |
|---|---|
@@ -181,7 +186,9 @@ The board holds no logic. The server sends one command per line over TCP; the bo
| `type <text>` | Type ASCII text |
| `key <combo>` | Press keys, e.g. `enter`, `cmd+h`, `cmd+space` |
-If the Wi-Fi link drops, the board releases all buttons and reconnects on its own.
+If the Wi-Fi link drops, the board releases all buttons and reconnects on its own. While the phone is locked, its USB
+port sleeps: the board then tries to wake it and otherwise answers `err phone usb asleep` instead of waiting. A watchdog
+restarts the board if it ever stops responding.
## Project layout
@@ -194,7 +201,8 @@ server/ agent, web UI, screen capture and board link (Python, FastAPI)
## Roadmap
- [ ] Test a fully local setup with DiffusionGemma (`server/serve/` has an experimental OpenAI-compatible server)
-- [ ] Correct the pointer only when needed, to make taps faster
+- [ ] Landscape apps (screen coordinates and calibration assume portrait)
+- [ ] Non-ASCII text input (e.g. Chinese through the iOS pinyin keyboard)
- [ ] More example tasks across popular apps
## Limitations
@@ -203,6 +211,7 @@ server/ agent, web UI, screen capture and board link (Python, FastAPI)
- Mirroring stops when the phone locks; keep auto-lock off while the agent works.
- Keyboard input is ASCII only.
- Re-calibrate after changing AssistiveTouch tracking speed.
+- Landscape apps are not supported yet: drawings and taps there land in the wrong place.
- On Linux machines with several network interfaces, limit Avahi (mDNS) to the interface on the phone's network
if the phone does not list the receiver.
diff --git a/README.zh-CN.md b/README.zh-CN.md
index 9891564..0fd9f2c 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -50,7 +50,7 @@ USB 鼠标加键盘。大脑可以是任何兼容 OpenAI 接口的视觉模型
| 部件 | 说明 |
|---|---|
-| **ESP32-S3 开发板** | 实测使用 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/) 裸板(约 50 元,不需要摄像头扩展板)。任何 USB-C 口直连 S3 原生 USB 的 ESP32-S3 板子都可以。 |
+| **ESP32-S3 开发板** | 实测使用 [Seeed Studio XIAO ESP32S3](https://wiki.seeedstudio.com/cn/xiao_esp32s3_getting_started/) 裸板(约 50 元,不需要摄像头扩展板)。记得接上附带的天线:XIAO 板上没有天线,不接的话 Wi-Fi 会经常断。任何 USB-C 口直连 S3 原生 USB 的 ESP32-S3 板子都可以。 |
| **iPhone** | USB-C 机型直接插;Lightning 机型用 Lightning 转 USB 相机转换器。板子由手机供电。 |
| **电脑** | macOS 或 Linux,和手机在同一个局域网;运行 UxPlay 和服务端。 |
| **视觉模型** | 任何支持图片输入、兼容 OpenAI 接口的对话模型。本地部署是可选的。 |
@@ -100,7 +100,8 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent
1. **设置 → 辅助功能 → 触控 → 辅助触控**:打开。
2. 把板子插到手机上,屏幕上会出现一个圆形指针,网页显示控制器已连接。
3. **控制中心 → 屏幕镜像 → iPhone Agent**,网页里出现实时画面。
-4. 在网页里打开 **手动操控 → 校准**,然后在实时画面里精确点一下指针的位置。
+4. 在网页里打开 **手动操控 → 校准**。服务端会移动指针、在实时画面里找到它,几秒内自动完成校准;
+ 如果找不到指针,会提示你在画面里精确点一下指针。
### 4. 下达任务
@@ -112,6 +113,7 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent
- **实时日志**:每一个计划、思考和动作实时显示,带耗时。
- **步骤截图墙**:每一步模型做决定时看到的截图,上面画出点按或滑动的位置,附带模型的理由和思考、执行各用了多久;
点卡片可以放大。
+- **点按**用红色圆圈和扩散的波纹标出,滑动和笔画用黄色轨迹;**动效**开关可以关掉动画。模型标为 `precise`(点之前先确认指针位置)的点按会在日志里注明。
- **紧急停止**(或按 <kbd>Esc</kbd>):立即停止 Agent 并松开指针。**清空记录**:清空日志和截图墙。
- 界面支持英文和简体中文。
@@ -124,6 +126,7 @@ set -a && . ./.env && set +a && .venv/bin/python -m hidagent
| `LLM_BASE_URL` | `http://localhost:8000/v1` | 兼容 OpenAI 的接口地址 |
| `LLM_MODEL` / `LLM_API_KEY` | — | 模型名称和 API Key |
| `LLM_EXTRA_BODY` | `{}` | 合并进每次请求的 JSON(例如关闭模型的思考模式) |
+| `LLM_TOKEN_PARAM` | `auto` | 输出长度参数名:`max_tokens`、`max_completion_tokens`(新版 OpenAI 模型),或 `auto` 自动识别 |
| `ACTOR_*` / `PLANNER_*` | — | 单独覆盖执行模型 / 规划模型的任意 `LLM_*` 配置 |
| `AIRPLAY_NAME` | `iPhone Agent` | 手机"屏幕镜像"列表里显示的名字 |
| `PORT` / `DEVICE_PORT` | `8890` / `7300` | 网页端口 / 板子连接的端口 |
@@ -162,7 +165,7 @@ LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
## 板子协议
-板子本身不含任何逻辑。服务端通过 TCP 每行发一条命令,板子回复 `ok` 或 `err <原因>`。
+板子本身不含任何逻辑。连上时先发送 `hello <上次重启原因> <开机毫秒数>`,之后服务端通过 TCP 每行发一条命令,板子回复 `ok` 或 `err <原因>`。
| 命令 | 作用 |
|---|---|
@@ -174,7 +177,7 @@ LLM_EXTRA_BODY='{"chat_template_kwargs": {"enable_thinking": false}}'
| `type <text>` | 输入 ASCII 文字 |
| `key <combo>` | 按键,例如 `enter`、`cmd+h`、`cmd+space` |
-Wi-Fi 断开时,板子会松开所有按键并自动重连。
+Wi-Fi 断开时,板子会松开所有按键并自动重连。手机锁屏时 USB 会休眠:板子会先尝试唤醒手机,唤醒不了就立刻回复 `err phone usb asleep`,不会卡住等待。万一板子没有响应,看门狗会让它自动重启。
## 目录结构
@@ -187,7 +190,8 @@ server/ Agent、网页界面、屏幕采集和板子连接(Python,FastAP
## 路线图
- [ ] 测试基于 DiffusionGemma 的纯本地方案(`server/serve/` 里有实验性的 OpenAI 兼容服务)
-- [ ] 指针校正改为按需执行,让点按更快
+- [ ] 支持横屏 App(目前坐标和校准都按竖屏计算)
+- [ ] 非 ASCII 文字输入(例如通过 iOS 拼音键盘输入中文)
- [ ] 更多常用 App 的示例任务
## 限制
@@ -196,6 +200,7 @@ server/ Agent、网页界面、屏幕采集和板子连接(Python,FastAP
- 手机锁屏后镜像会断开,Agent 工作时请关闭自动锁定。
- 键盘只支持 ASCII 输入。
- 修改辅助触控的跟踪速度后需要重新校准。
+- 暂不支持横屏 App:在横屏界面里的点按和绘图位置会偏。
- 在有多个网卡的 Linux 机器上,如果手机列表里看不到接收端,把 Avahi(mDNS)限制在手机所在网络的网卡上。
## 合理使用
diff --git a/hardware/hid_bridge/hid_bridge.ino b/hardware/hid_bridge/hid_bridge.ino
index 5a89e6a..6a8e2ff 100644
--- a/hardware/hid_bridge/hid_bridge.ino
+++ b/hardware/hid_bridge/hid_bridge.ino
@@ -6,9 +6,11 @@
// scroll <v> <h> | type <ascii text> | key <combo, e.g. cmd+h, enter>
#include <WiFi.h>
+#include <esp_task_wdt.h>
#include "USB.h"
#include "USBHIDMouse.h"
#include "USBHIDKeyboard.h"
+#include "tusb.h"
#include "secrets.h" // WIFI_SSID, WIFI_PASS, SERVER_HOST, SERVER_PORT
const int STEP = 10; // max pointer delta per report while moving
@@ -18,6 +20,8 @@ const int HOME_REPORTS = 50;
#define DEBUG_SERIAL 0 // 1: echo every command over USB serial (needs CDCOnBoot=cdc)
#endif
+const int WATCHDOG_MS = 8000; // a hung loop (e.g. blocked USB) reboots the board instead of staying offline
+
USBHIDMouse mouse;
USBHIDKeyboard keyboard;
WiFiClient server;
@@ -61,6 +65,16 @@ bool pressCombo(String combo) {
return true;
}
+// A locked or sleeping phone suspends USB; sending HID reports then can stall. Ask it to wake up instead,
+// and report "asleep" right away if it does not, so the board never hangs.
+bool usbReady() {
+ if (tud_suspended()) {
+ tud_remote_wakeup();
+ for (int i = 0; i < 50 && tud_suspended(); i++) delay(10);
+ }
+ return tud_mounted() && !tud_suspended();
+}
+
String run(const String &line) {
int sp = line.indexOf(' ');
String cmd = sp < 0 ? line : line.substring(0, sp);
@@ -69,6 +83,7 @@ String run(const String &line) {
sscanf(arg.c_str(), "%ld %ld", &a, &b);
if (cmd == "ping") return "ok";
+ if (!usbReady()) return "err phone usb asleep (unlock the phone)";
if (cmd == "home") { // slam the pointer into the top-left corner
for (int i = 0; i < HOME_REPORTS; i++) { mouse.move(-127, -127); delay(2); }
return "ok";
@@ -83,6 +98,18 @@ String run(const String &line) {
return "err unknown command";
}
+const char *resetReason() {
+ switch (esp_reset_reason()) {
+ case ESP_RST_POWERON: return "power-on";
+ case ESP_RST_BROWNOUT: return "brownout";
+ case ESP_RST_TASK_WDT: case ESP_RST_INT_WDT: case ESP_RST_WDT: return "watchdog";
+ case ESP_RST_PANIC: return "crash";
+ case ESP_RST_SW: return "software";
+ case ESP_RST_USB: return "usb";
+ default: return "other";
+ }
+}
+
void setup() {
#if DEBUG_SERIAL
Serial.begin(115200);
@@ -91,20 +118,27 @@ void setup() {
keyboard.begin();
USB.productName("HID Bridge");
USB.begin();
+ esp_task_wdt_config_t wdt = {.timeout_ms = WATCHDOG_MS, .idle_core_mask = 0, .trigger_panic = true};
+ esp_task_wdt_reconfigure(&wdt);
+ esp_task_wdt_add(NULL);
WiFi.setHostname("hid-bridge");
WiFi.setSleep(false); // modem sleep adds ~100 ms per command
+ WiFi.setAutoReconnect(true);
WiFi.begin(WIFI_SSID, WIFI_PASS);
}
void loop() {
- if (WiFi.status() != WL_CONNECTED) { mouse.release(MOUSE_LEFT); delay(500); return; }
+ esp_task_wdt_reset();
+ if (WiFi.status() != WL_CONNECTED) { mouse.release(MOUSE_LEFT); delay(50); return; } // auto-reconnects
if (!server.connected()) {
mouse.release(MOUSE_LEFT); // a gesture cut off mid-way must not leave the button held (long press)
keyboard.releaseAll();
- if (!server.connect(SERVER_HOST, SERVER_PORT)) { delay(2000); return; }
+ if (!server.connect(SERVER_HOST, SERVER_PORT, 1000)) { delay(200); return; } // retry fast: show up at once
server.setNoDelay(true);
+ server.printf("hello %s %lu\n", resetReason(), millis()); // why the board last restarted, and uptime
#if DEBUG_SERIAL
- Serial.printf("connected to %s:%d as %s\n", SERVER_HOST, SERVER_PORT, WiFi.localIP().toString().c_str());
+ Serial.printf("connected to %s:%d as %s after %lu ms\n", SERVER_HOST, SERVER_PORT,
+ WiFi.localIP().toString().c_str(), millis());
#endif
}
if (server.available()) {