Business operating system for Claude Code — 57 skills, 21 agents, smart daemon. Unified inbox (WhatsApp/Email/Slack/Telegram), autonomous PR merge, full-AWS monitoring, revenue (Stripe+RevenueCat), e-commerce (Shopify), marketing (Klaviyo/Meta/GA4), voice (Bland/ElevenLabs), APM (Datadog/NewRelic/OTEL), YOLO mode.
Shell agent-teamsai-agentai-automationanthropicautonomous-agentawsbusiness-operationsclaudeclaude-codeclaude-code-plugindevopsecommerceinbox-zeroklaviyomarketingmcpmorning-briefingpr-automationshopifyterminal
docs(iphonetest): document launch_app cmd + working type fallback (#380)
Aurora Capital8ddb1f7e690b
diff --git a/docs/operators/iphonetest.md b/docs/operators/iphonetest.mdindex 02a46f66..fba794c9 100644--- a/docs/operators/iphonetest.md+++ b/docs/operators/iphonetest.md@@ -10,7 +10,9 @@ and every Appium-extension surprise that has been observed live.- Driver script: `my-project/scripts/devicefarm/iphonetest.py` → hands off to`tunnel.py` for `--mode tunnel`. This guide documents the **JSON-RPC surfaceof `tunnel.py` as it exists today**, verified against- [`tunnel.py`](https://github.com/your-org/my-project/blob/main/scripts/devicefarm/tunnel.py).+ [`tunnel.py @ 22cf0cac`](https://github.com/your-org/my-project/blob/22cf0cac5d9b88d066b4badd94b59d480be5285e/scripts/devicefarm/tunnel.py)+ (my-project [PR #6174](https://github.com/your-org/my-project/pull/6174),+ merged 2026-05-29). When the surface drifts, update this file and re-pin the SHA.- Device Farm API surface:[`CreateRemoteAccessSession`](https://docs.aws.amazon.com/devicefarm/latest/APIReference/API_CreateRemoteAccessSession.html).- Appium driver: `appium-xcuitest-driver` —@@ -112,7 +114,8 @@ python3 scripts/devicefarm/stop-session.py --all-active## 4. JSON-RPC command catalogue`tunnel.py --json` reads one JSON object per stdin line and writes one JSON-response per stdout line. The full handlers dict from `tunnel.py:jsonrpc`:+response per stdout line. The full handlers dict from `tunnel.py:jsonrpc`+(quoted verbatim from my-project@`22cf0cac5d9b88d066b4badd94b59d480be5285e`):```pythonhandlers: dict[str, Any] = {@@ -125,6 +128,7 @@ handlers: dict[str, Any] = {"swipe": _swipe,"type": lambda m: t.type_text(m["text"], submit=bool(m.get("submit", False))),"press": lambda m: t.press(m["button"]),+ "launch_app": lambda m: t.launch_app(m.get("bundle_id")),"wait": lambda m: time.sleep(float(m["seconds"])),"status": lambda m: t.status(),"stop": _stop,@@ -229,14 +233,31 @@ Both compile down to `mobile: dragFromToForDuration` on the driver.### `type`-Send keys to the focused element. `submit: true` appends a newline.+Send keys to whichever field is focused. `submit: true` appends a newline+(send-keys `"\n"` on the focused element) after the text lands.```json{"cmd":"type","text":"hello Anna","submit":false}```-Internally this dispatches `mobile: keys` — which the Device Farm-bundled-xcuitest driver **does not expose** on the WDA endpoint. See gotcha §6.1.+Since my-project@`22cf0cac5d9b88d066b4badd94b59d480be5285e`, `type_text` cascades+through the same four strategies `explore.py:execute_action` uses, in this+order (verbatim from `tunnel.py:Tunnel.type_text`):++1. Focused element (`AppiumBy.IOS_PREDICATE, "hasKeyboardFocus == 1"`) →+ `set_value(text)`+2. Same focused element → `send_keys(text)`+3. First-visible TextField or SecureTextField (predicate+ `(type == 'XCUIElementTypeTextField' OR type == 'XCUIElementTypeSecureTextField') AND visible == 1`,+ sorted top-left first) → tap, then `set_value`, falling back to+ `send_keys` on `set_value` failure+4. `mobile: keys` (last resort — usually fails on Device Farm; kept for+ local Appium / simulator hosts)++The first three strategies are why **`type` now works on app-rendered My-Project+UI without first having to focus the field manually** — see the matrix in §5+and gotcha §6.1 for what's still broken (iOS Springboard / Spotlight system+search fields).### `press`@@ -247,6 +268,66 @@ Hardware-button press. Valid values:{"cmd":"press","button":"home"}```+### `launch_app`++Launch (or bring to foreground) an app by bundle id. Device Farm does NOT+auto-launch the freshly-installed IPA after `start` — the device lands on+Springboard. This command is the canonical recovery; call it once right+after `start`. Added in my-project [PR #6174](https://github.com/your-org/my-project/pull/6174)+(merged 2026-05-29, SHA `22cf0cac5d9b88d066b4badd94b59d480be5285e`).++Explicit bundle id form:++```json+{"cmd":"launch_app","bundle_id":"com.staging.my-project"}+```++Auto-resolve form (`bundle_id` omitted — `tunnel.py` reads the IPA's+`package_name` field out of the upload metadata Device Farm holds against+the `app_arn` cached at `start` time, via+`df.get_upload(arn=app_arn)["upload"]["metadata"]["package_name"]`):++```json+{"cmd":"launch_app"}+```++Response on success:++```json+{"ok":true,"bundle_id":"com.staging.my-project","method":"activateApp"}+```++`method` is `activateApp` when `mobile: activateApp` succeeds, or `launchApp`+when `tunnel.py` falls back to the older `mobile: launchApp` (kept for older+Appium driver builds). Quoted verbatim from `tunnel.py:Tunnel.launch_app`:++```python+try:+ self.driver.execute_script("mobile: activateApp", {"bundleId": bundle})+ return {"ok": True, "bundle_id": bundle, "method": "activateApp"}+except Exception as exc_activate:+ try:+ self.driver.execute_script("mobile: launchApp", {"bundleId": bundle})+ return {"ok": True, "bundle_id": bundle, "method": "launchApp"}+ except Exception as exc_launch:+ return {+ "ok": False,+ "error": f"activateApp: {exc_activate!r} | launchApp: {exc_launch!r}",+ }+```++Error paths:++- Called before `start` (no Appium driver yet):+ `{"ok": false, "error": "session not started"}`.+- No `bundle_id` and no cached `app_arn`:+ `{"ok": false, "error": "no bundle_id provided and no app_arn cached on the session"}`.+- IPA upload metadata is missing or has no `package_name`:+ `{"ok": false, "error": "upload metadata has no \`package_name\`; pass bundle_id explicitly"}`+ (or a sibling error covering "no metadata" / "metadata is not valid JSON").+- Both `mobile: activateApp` and `mobile: launchApp` throw:+ `{"ok": false, "error": "activateApp: <repr> | launchApp: <repr>"}`.+### `wait`Sleep without sending anything to the device. Use this to let an animation@@ -295,8 +376,9 @@ Response on clean stop: `{"ok":true,"quit":true}`. On stop failure:## 5. What works vs. what's broken on Device FarmBehavior observed live on iPhone 17 Pro / iOS 26.3.1 driving My-Project staging-(2026-05-29). The Appium driver bundled in Device Farm's managed Appium server-lags upstream `appium-xcuitest-driver` v11.7.4+(2026-05-29, against my-project@`22cf0cac5d9b88d066b4badd94b59d480be5285e`). The+Appium driver bundled in Device Farm's managed Appium server lags upstream+`appium-xcuitest-driver` v11.7.4([reference](https://appium.github.io/appium-xcuitest-driver/latest/reference/execute-methods/)),which is why some `mobile:` extensions the docs list are not actuallyexposed at the WDA endpoint.@@ -305,9 +387,10 @@ exposed at the WDA endpoint.|---|---|---|| `tap` (coords) | ✓ | ✓ || `tap_text` | ✓ (every interactive My-Project element ships an accessibility label per `figma-design-system.md`) | ✗ — Springboard / Spotlight system controls (Cancel, search field, App-Library labels) often have no label exposed to WDA's predicate matcher |-| `type` | ✓ for app-rendered text fields with proper accessibility labels (focused-element `set_value` / `send_keys` fallback path inside the driver kicks in) | ✗ — returns `Unhandled endpoint: /wda/element/0/keyboardInput` for the iOS Spotlight / App-Library search field (DF's bundled driver does not expose `mobile: keys`) |+| `type` | ✓ — the 4-strategy fallback (focused element → `set_value`/`send_keys`, then first-visible TextField/SecureTextField, then `mobile: keys`) lands on any field with `hasKeyboardFocus == 1` OR any visible TextField, with no operator-side pre-focusing required | ✗ — iOS Springboard / Spotlight / App-Library search fields are not exposed in the active app's accessibility tree, so the predicate-based fallbacks find nothing and `mobile: keys` returns `Unhandled endpoint: /wda/element/0/keyboardInput` on DF || `swipe` (both forms) | ✓ | ✓ || `press` | ✓ | ✓ |+| `launch_app` | ✓ — `mobile: activateApp` succeeds against the staging bundle id; `mobile: launchApp` fallback covers older driver builds | n/a (operates on installed-app bundle ids, not system UI) || `screenshot` / `tap` (coords) | ✓ | ✓ — **always usable as the last-resort fallback** |## 6. Workarounds catalog@@ -315,28 +398,34 @@ exposed at the WDA endpoint.Each of these has been hit live; treat the workaround as the primary pathwhen you recognise the symptom.-### 6.1 `type` returns `Unhandled endpoint: /wda/element/0/keyboardInput`+### 6.1 `type` returns `Unhandled endpoint: /wda/element/0/keyboardInput` (system UI only)**Symptom:** `{"cmd":"type","text":"my-project"}` against the iOS Spotlight /App-Library system search field comes back as`{"ok":false,"error":"Unhandled endpoint: /wda/element/0/keyboardInput"}`.**Cause:** Device Farm's bundled `appium-xcuitest-driver` does **not** expose-`mobile: keys` (the extension `tunnel.py:Tunnel.type_text` dispatches). Even-though upstream v11.7.4 documents `mobile: keys`, the managed Appium server-on DF is older / stripped.--**Workaround for app text fields:** the driver's internal fallback to-focused-element `set_value` / `send_keys` works as long as the field has a-proper accessibility label. Focus the field first with `tap_text` (or `tap`-with coords) then re-issue `type`. This is how My-Project staging text inputs-accept input.+`mobile: keys` (the strategy-4 last resort `tunnel.py:Tunnel.type_text`+dispatches when nothing earlier matches). Even though upstream v11.7.4+documents `mobile: keys`, the managed Appium server on DF is older /+stripped. Springboard / Spotlight / App-Library search fields aren't exposed+in the active app's accessibility tree, so strategies 1–3 (focused element ++first-visible TextField) find nothing and the cascade tumbles into+strategy 4 — which fails.++**App-rendered My-Project UI: `type` just works now.** Since my-project@`22cf0cac`,+`tunnel.py:Tunnel.type_text` cascades through the same four strategies+`explore.py:execute_action` uses (focused-element `set_value` →+focused-element `send_keys` → first-visible TextField/SecureTextField with+tap + `set_value`/`send_keys` → `mobile: keys`). You do **not** need to+pre-focus the field with `tap_text` first — strategy 3 finds the topmost+visible TextField on its own. See §4 `type` for the verbatim cascade.**Workaround for iOS system text fields (Spotlight, App Library, Settings-search):** `type` is unusable here. Instead, tap each on-screen keyboard-letter individually by coordinate (read them off the screenshot), or skip-the system search entirely and reach the app via the App-Library route in-§6.3.+search):** `type` is still unusable here. Instead, tap each on-screen+keyboard letter individually by coordinate (read them off the screenshot),+or skip the system search entirely and reach the app via `launch_app`+(§6.3) — which is now the canonical recovery.### 6.2 `tap_text` returns `No element matched text 'Cancel'`@@ -366,31 +455,30 @@ the screenshot shows the iOS home screen — not the My-Project app.**Cause:** `tunnel.py` installs the IPA on session start (`appArn` is passedto `create_remote_access_session`), but the Device Farm runtime does **not**auto-launch it. The bundle is on the device — it's just sitting in App-Library → Recently Added. There is no `cmd` in `tunnel.py` today that-invokes `mobile: launchApp` / `mobile: activateApp` with the staging bundle-id, even though-[both extensions exist upstream](https://appium.github.io/appium-xcuitest-driver/latest/reference/execute-methods/).+Library → Recently Added.-**Workaround:** swipe left to App Library, tap "Recently Added" by coords,-find the My-Project icon.+**Canonical fix (since my-project@`22cf0cac`):** issue `launch_app` once. Either+form works:```bash-./tx.sh '{"cmd":"screenshot","path":"home.png"}'-./tx.sh '{"cmd":"swipe","direction":"left"}' # → today view → app library-./tx.sh '{"cmd":"swipe","direction":"left"}'-./tx.sh '{"cmd":"screenshot","path":"app-library.png"}'-# Read app-library.png, find "Recently Added" stack-./tx.sh '{"cmd":"tap","x":<x>,"y":<y>}'-./tx.sh '{"cmd":"screenshot","path":"recently-added.png"}'-# Tap My-Project icon-./tx.sh '{"cmd":"tap","x":<x>,"y":<y>}'+# Explicit bundle id (recommended — no extra metadata round-trip)+./tx.sh '{"cmd":"launch_app","bundle_id":"com.staging.my-project"}'++# Auto-resolve from IPA upload metadata (tunnel.py reads `package_name`+# off the `app_arn` cached at `start` time)+./tx.sh '{"cmd":"launch_app"}'```-**Followup:** a single-line addition of a `launch_app` / `activate_app` JSON-command in `tunnel.py` (dispatching `mobile: activateApp` with-`bundleId="com.staging.my-project"`) would eliminate this dance. File against-my-project under `scripts/devicefarm` — pure additive change. Until that lands,-the App-Library route is the recipe.+Expect `{"ok":true,"bundle_id":"com.staging.my-project","method":"activateApp"}`.+On older Appium driver builds the response will be+`...,"method":"launchApp"` — same outcome, fallback path. See §4 `launch_app`+for the full error catalogue.++**Fallback (if `launch_app` itself fails):** swipe left to App Library, tap+"Recently Added" by coords, find the My-Project icon (one-liner pattern:+`swipe left → swipe left → screenshot → tap "Recently Added" → screenshot →+tap My-Project icon`). Useful when `activateApp` and `launchApp` both throw+and you need a screenshot-driven recovery while you file the bug.### 6.4 Picker auto-selection (iPhone 17 Pro / iOS 26.3.1)@@ -554,24 +642,23 @@ command.```bashTX="$TMP/tx.sh"-# 1. Take a baseline screenshot-"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/01.png"}'-# → Read 01.png. iOS home screen — app not auto-launched (gotcha §6.3).+# 1. `start` already ran when /iphonetest launched the tunnel — device+# is reserved but sitting on Springboard.-# 2. Swipe to App Library-"$TX" '{"cmd":"swipe","direction":"left"}'-"$TX" '{"cmd":"swipe","direction":"left"}'-"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/02.png"}'+# 2. Launch the app (canonical recovery, since my-project@22cf0cac).+# Bundle id is optional — tunnel.py resolves it from IPA metadata+# when omitted.+"$TX" '{"cmd":"launch_app","bundle_id":"com.staging.my-project"}'+# → expect {"ok":true,"bundle_id":"com.staging.my-project","method":"activateApp"}.-# 3. Tap My-Project icon (coords read from 02.png)-"$TX" '{"cmd":"tap","x":120,"y":240}'-"$TX" '{"cmd":"wait","seconds":2}'-"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/03.png"}'+# 3. Take a baseline screenshot of the app's first screen+"$TX" '{"cmd":"wait","seconds":1}'+"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/01.png"}'# 4. Sign in"$TX" '{"cmd":"tap_text","text":"Continue with Apple"}'"$TX" '{"cmd":"wait","seconds":3}'-"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/04.png"}'+"$TX" '{"cmd":"screenshot","path":"'"$TMP"'/02.png"}'# 5. Onwards — one screenshot per turn, one decision per turn@@ -613,18 +700,20 @@ When you hit one of these in the field, file in the my-project repo and linkback to this guide. Do **not** patch them in the same PR that ships thisdoc — keep the doc PR pure.-- **`launch_app` / `activate_app` JSON command in `tunnel.py`.** Wraps- `mobile: activateApp` with `bundleId="com.staging.my-project"` (or the- resolved production bundle id). Eliminates the App-Library workaround- in §6.3. One-line handler addition.- **`mobile: keys` fallback for system text fields.** Either add aper-key tap-coords helper for the iOS on-screen keyboard, or documentthat My-Project tests should never depend on driving Spotlight / App-Library- search.+ search. `launch_app` (shipped in my-project@`22cf0cac`) removed the most+ common reason an operator needed to drive system search at all, but+ Settings flows still hit it.- **Document the bundled Appium driver version on Device Farm.** A`mobile: getEnv` probe at session-start would let `tunnel.py` log theWDA build version into `status` so future operators don't re-discoverthe upstream-vs-bundled drift from scratch.+- ~~**`launch_app` / `activate_app` JSON command in `tunnel.py`.**~~+ Shipped in my-project [PR #6174](https://github.com/your-org/my-project/pull/6174)+ (merged 2026-05-29, SHA `22cf0cac5d9b88d066b4badd94b59d480be5285e`). See+ §4 `launch_app` and §6.3.Open these as my-project GitHub issues, link from`my-project/.claude/commands/iphonetest.md`, and reference back here.@@ -632,5 +721,7 @@ Open these as my-project GitHub issues, link from---Last verified live: 2026-05-29 on iPhone 17 Pro / iOS 26.3.1 against-My-Project staging IPA. When anything in §5 / §6 changes, update this file and-the upstream slash-command in the same PR-pair.+My-Project staging IPA, pinned to my-project@`22cf0cac5d9b88d066b4badd94b59d480be5285e`+(`scripts/devicefarm/tunnel.py`, my-project PR #6174). When anything in §4 / §5 /+§6 changes, update this file, re-pin the SHA, and update the upstream+slash-command in the same PR-pair.