Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。
feat(screen): switch MIUI's tree helper back on when WeChat hides its UI
booboosui6aa1fbd373ec
diff --git a/CLAUDE.md b/CLAUDE.mdindex 9786814..b2d95fe 100644--- a/CLAUDE.md+++ b/CLAUDE.md@@ -100,6 +100,9 @@ Ported from generic_bridge (phone.sh), whose finding is the founding rule here:- **Stuck-loop guard**: per-location counting, not consecutive-with-no-change (the real dead-loop interleaves menu opens, wrong pages and backs, and apps emit content events constantly — both naive conditions miss it). Gestures within 120 px of each other share a bucket; within 5 minutes, the 4th tap on the same bucket is refused with an error telling the brain to stop, re-observe, and ask the user. Swipes are exempt (list scrolling legitimately repeats); e-taps are exempt (re-tapping a confirmed element is normal operation).- **NO_TREE policy** (empty-list hint): prefer `back`/`home` to exit and retry via a different route over blind coordinate taps — a wrong exit costs one tap, a wrong tap can cost the task.- **Stale-service note**: `ScreenController.goodDumps` records which packages have actually produced a usable element list here. When a dump comes back with *zero nodes* from a package with such a history, `reviveNote()` appends the evidence to the hint — WeChat's tree is genuinely obtainable, but the accessibility connection sometimes goes stale and only a human toggling Andee off/on in 设置 → 无障碍 restores it. It is **evidence, not a rule**: the note states the facts and the brain decides whether to interrupt the user. It is deliberately not keyed to `com.tencent.mm`, and deliberately fires only on `nodes_total == 0` — a self-drawn pane still reports its containers, so mini-programs never trigger it.+- **An app can withhold its tree, and that is not a stale service** (`isWithheld`, measured 2026-10-08). WeChat 8.0.78 on the Xiaomi pad handed accessibility a blank root — no class, 0×0 bounds, no children — and so did the system's own `uiautomator dump` **with no accessibility service enabled at all** and a freshly started WeChat, while Settings dumped 89 nodes. The switch was another service: with MIUI's `com.miui.accessibility/.enhance.tb.MiuiEnhanceTBService` enabled beside Andee the same screen gave 219 nodes, and removing it blanked the root again, reproducibly both ways. That is the whole "sometimes it works" — toggling Andee never mattered, which is why a `withheld` dump gets its own hint *ahead of* `reviveNote` (work from the screenshot) instead of the toggle-Andee note. Occlusion by our card keeps real root bounds and a self-drawn pane keeps its containers, so neither can be mistaken for this.+ - **And MIUI switches that helper by itself, so Andee switches it back** (`screen/TreeHelper.kt`). logcat (`talkback-test`) shows MIUI enabling or disabling `MiuiEnhanceTBService` on its own whenever Andee is re-bound — a reinstall, a crash, a toggle — which is how the tree came and went with nobody touching a setting. So a `withheld` dump calls `TreeHelper.wake`, which **appends** the helper to `enabled_accessibility_services` (`WRITE_SECURE_SETTINGS`, the same grant `ImeSwitch` uses) and `awaitHelper` re-dumps until the app hands over a real root, up to 3 s. It never removes anything, does nothing where the component does not exist or the grant is missing, writes a scrollback row (the user's system settings changed and they are owed the news — same rule as the vault fill), and is throttled to once a minute so a user who turns the helper off on purpose is not overruled on every dump. `tree_helper_enabled` in the result tells the hint to say "retry once".+ - **Without the adb grant there is no way back, and the hint must not pretend there is.** The helper is TalkBack's hidden MIUI companion: logcat shows it started when TalkBack is switched on and stopped when TalkBack is switched off, and it has no switch of its own a user can find. Turning TalkBack on would work and is not an answer — it reads the whole screen aloud and turns every tap into select-then-double-tap. One cheaper route was tried and ruled out: matching the helper's own config (`typeAllMask`, `notificationTimeout=0`, same `FEEDBACK_GENERIC`) left WeChat blank on a cold start, so it recognises the helper by identity, not by what it subscribes to. So a device with no grant gets the screenshot route and a hint that tells the brain **not** to send the user hunting for a setting.### Text input: the service types for itself, ADBKeyboard is the sparediff --git a/README.md b/README.mdindex d0baf9a..7e49215 100644--- a/README.md+++ b/README.md@@ -382,6 +382,32 @@ report success and a filename.*Note:* most vendor ROMs **switch this grant off again** after every system update and everyforce-stop. That is ROM behaviour, and not something the app can prevent.+*WeChat (and other apps) sometimes show a blank screen to it.* Some apps hand **every**+accessibility client an empty placeholder instead of their real UI. This is not Andee's fault:+the system's own `uiautomator dump` gets the same blank page. Measured on a Xiaomi Pad 5 with+MIUI and WeChat 8.0.78, WeChat only shows its real UI tree when MIUI's hidden+**`MiuiEnhanceTBService`** is enabled. That is a TalkBack companion service, and it has no switch+in Settings. With it on, the same chat screen dumps 219 nodes. With it off, it dumps none. MIUI+also turns that service on and off by itself whenever Andee is re-bound (after a reinstall, a+crash, or a toggle). That is why WeChat could be read **sometimes but not always**, and why+toggling Andee never actually fixed it. Matching the helper's accessibility config does not help+either: WeChat checks which service is asking, not what it asked for.++What Andee does about it (`screen/TreeHelper.kt`): when a dump comes back blank like this, Andee+**adds** that service to the enabled list and dumps again. It never removes anything. It does+nothing on devices that don't have that component. It writes a line in the chat history so you+know your system settings changed. It also tries at most once a minute, so if you turn the+service off on purpose, Andee won't keep turning it back on. Writing that list needs one grant+that only adb can give. It is the same grant the ADBKeyboard fallback uses:++```bash+adb shell pm grant net.kuafuai.andee android.permission.WRITE_SECURE_SETTINGS+```++Without the grant, nothing breaks. On the blank screens, Andee works from screenshots and+coordinates instead of the element list. That is slower and less precise, but the task still+gets done.+**Microphone.***Why:* ASR is streaming, so it needs the PCM stream upstreamed.diff --git a/README.zh-CN.md b/README.zh-CN.mdindex c15b01e..94e2767 100644--- a/README.zh-CN.md+++ b/README.zh-CN.md@@ -557,6 +557,26 @@ Manifest 里声明的能力(`res/xml/accessibility_service_config.xml`):**注意**:大部分厂商 ROM 在每次系统更新、每次强制停止应用之后会**自动关掉**这个权限,需要重开。这是 ROM 行为,不是应用能阻止的。+**微信(或别的应用)有时候读不到界面**:有些应用会给**所有**无障碍客户端一个空白的占位界面,+不给真实的 UI。这不是 Andee 的问题:系统自带的 `uiautomator dump` 拿到的也是空白。我们在小米平板 5+(MIUI,微信 8.0.78)上实测,只有开着 MIUI 一个隐藏服务 **`MiuiEnhanceTBService`** 的时候,微信才肯+交出真实的 UI 树。这个服务是 TalkBack 的配套服务,设置里没有它的开关。开着它,同一个聊天页能读到+219 个节点;关掉它,一个都读不到。而且每次 Andee 被重新绑定(重装、崩溃、开关一次),MIUI 都会+**自己**打开或关掉这个服务。这就是"有时候能读到、有时候不行"的原因,也是为什么重开 Andee 从来+没真正解决过问题。把 Andee 的无障碍配置改成和它一样也没用:微信认的是"谁在要",不是"要了什么"。++Andee 的处理方式(`screen/TreeHelper.kt`):读到这种空白界面时,Andee 会把这个服务**追加**到已启用+列表里,然后重新读一次。它只加不删。设备上没有这个组件时,它什么都不做。它会在聊天记录里写一行,+让你知道系统设置被改过了。它最多一分钟尝试一次,所以如果你是故意关掉这个服务的,Andee 不会反复+把它打开。修改这个列表需要一项只能用 adb 授予的权限,和 ADBKeyboard 备用通道用的是同一项:++```bash+adb shell pm grant net.kuafuai.andee android.permission.WRITE_SECURE_SETTINGS+```++不授权也不会坏。遇到空白界面时,Andee 改用截图加坐标来操作,不再依赖元素列表。这样更慢、+也没那么准,但事情照样能做完。+### 三、麦克风**为什么要**:ASR 是流式识别,需要拿 PCM 数据流上行。diff --git a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.ktindex 43b1938..7d7e805 100644--- a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt+++ b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt@@ -17,8 +17,11 @@ import android.util.Logimport android.view.Displayimport android.view.accessibility.AccessibilityNodeInfoimport android.view.accessibility.AccessibilityWindowInfo+import net.kuafuai.andee.Rimport net.kuafuai.andee.device.DeviceState+import net.kuafuai.andee.i18n.AppLocaleimport net.kuafuai.andee.net.ToolSchemas+import net.kuafuai.andee.ui.ChatHistoryimport org.json.JSONArrayimport org.json.JSONObjectimport java.io.ByteArrayOutputStream@@ -1603,6 +1606,8 @@ class ScreenController(waited += SETTLE_POLL_MSout = dumpOnce(verbose, budget)}+ val helped = !verbose && out.optBoolean("withheld") && TreeHelper.wake(service)+ if (helped) out = awaitHelper(out)// Stability pass — the half the old loop never covered: "has content"// is not "has stopped moving". A screen mid-animation (list settling,// pager sliding, splash fading) dumps real elements at positions that@@ -1639,6 +1644,7 @@ class ScreenController(out.put("still_moving", true)}}+ if (helped) out.put("tree_helper_enabled", true)if (!verbose && out.optInt("nodes_useful", 0) == 0 && waited >= SETTLE_BUDGET_MS) {out.put("settling", true).put("settle_waited_ms", waited)}@@ -1725,6 +1731,27 @@ class ScreenController(return out}+ /**+ * Re-dump after [TreeHelper] switched the system's helper on. The setting+ * is written at once but the service binds asynchronously, and the app+ * only hands over its tree once it has — so this polls rather than sleeps.+ * Its own budget: the wait is not walking, and the first dump may already+ * have spent most of [TREE_BUDGET_MS].+ */+ private fun awaitHelper(first: JSONObject): JSONObject {+ runCatching {+ ChatHistory.addAssistant(AppLocale.str(service, R.string.tree_helper_enabled_row))+ }+ var out = first+ val until = SystemClock.uptimeMillis() + HELPER_WAIT_MS+ while (SystemClock.uptimeMillis() < until) {+ try { Thread.sleep(HELPER_POLL_MS) } catch (_: InterruptedException) { break }+ out = dumpOnce(false, WalkBudget(TREE_BUDGET_MS))+ if (!out.optBoolean("withheld")) break+ }+ return out+ }+/*** Signature of a dump's content for the stability pass: which nodes* exist, their ids and their bounds. Bounds are the part that matters —@@ -1750,7 +1777,9 @@ class ScreenController(try {val stats = Stats(budget)val body = buildNode(active, verbose, stats) ?: JSONObject().put("empty", true)- withStats(body, active.packageName?.toString() ?: "", stats)+ withStats(body, active.packageName?.toString() ?: "", stats).also {+ if (isWithheld(active)) it.put("withheld", true)+ }} finally {active.recycle()}@@ -1774,6 +1803,29 @@ class ScreenController(return out}+ /**+ * The app handed accessibility a root with no class, no size and no+ * children — a placeholder, not a pane.+ *+ * Measured on WeChat 8.0.78 (Xiaomi Pad 5, 2026-10-08): the system's own+ * `uiautomator dump` got the same single blank node with **no**+ * accessibility service enabled and a freshly started WeChat, while+ * Settings dumped 89 nodes. With MIUI's `MiuiEnhanceTBService` enabled+ * alongside us the same screen dumped 219 nodes, and removing it blanked the+ * root again — reproducible both ways, and the real "有时候可以有时候不行".+ * So it is the app refusing every client, not our+ * connection going stale — and the two must not share a hint, because the+ * stale one sends the user to toggle Andee, which cannot help here.+ * Occlusion (our card) and self-drawn panes are both distinguishable: they+ * keep real bounds and their containers.+ */+ private fun isWithheld(root: AccessibilityNodeInfo): Boolean = runCatching {+ if (root.childCount != 0 || root.packageName?.toString() == service.packageName) return@runCatching false+ val r = android.graphics.Rect()+ root.getBoundsInScreen(r)+ r.isEmpty && root.className.isNullOrEmpty()+ }.getOrDefault(false)+/*** Fallback for the case above: walk every window we can see and keep the* best one. Only ever entered when the active window gave us nothing, so the@@ -2003,7 +2055,7 @@ class ScreenController(val keep = setOf("ts", "pkg", "nodes_total", "nodes_kept", "nodes_useful", "surface","source", "windows_seen", "windows_detail", "settling", "settle_waited_ms",- "error", "empty",+ "error", "empty", "withheld", "tree_helper_enabled",)val rootKeys = ArrayList(out.keys().asSequence().toList())for (k in rootKeys) if (k !in keep) out.remove(k)@@ -2240,6 +2292,41 @@ class ScreenController("not tell the user to restart anything, and do not tap blindly.",)+ // Ahead of the two [reviveNote] branches: the app worked before,+ // so they would send the user off to toggle Andee — which is+ // exactly what they did, to no effect. See [isWithheld].+ useful == 0 && out.optBoolean("withheld") -> {+ val app = out.optString("pkg").ifEmpty { "This app" }+ val helper = if (out.optBoolean("tree_helper_enabled")) {+ "Andee has just switched that helper on itself and the app has not picked " ++ "it up yet — the next dump may read normally, so retry once before " ++ "giving up on the list; if it is still blank, the user may need to " ++ "reopen the app. "+ } else {+ "Andee could not switch it on here, and the user cannot either: it is a " ++ "hidden companion MIUI only runs alongside TalkBack, and TalkBack " ++ "would read the whole screen aloud and change how every tap works. So " ++ "do not send the user to look for any setting — if it matters, say " ++ "once that in this app you are working from the picture, and carry on. "+ }+ out.put(+ "hint",+ "`$app` is withholding its UI from accessibility: its window hands every " ++ "accessibility client — the system's own uiautomator included — a blank " ++ "placeholder with no size and no children. That is the app's choice, not a " ++ "fault on this device; switching Andee off and on does not change it, so " ++ "do not ask the user to. What it does depend on is which *other* " ++ "accessibility services are on: WeChat was measured exposing its full " ++ "tree only while MIUI's TalkBack helper (MiuiEnhanceTalkback) was also " ++ "enabled. " + helper + "It covers " ++ "this whole app, not just this screen — so backing out to find a readable " ++ "page will not help either. Work from $eyes: find the target in the image " ++ "and use tap_by_coordinates / swipe_by_coordinates (0-1000), checking the " ++ "result in the screenshot each tap returns. type_text and submit_input " ++ "still work on a focused field.",+ )+ }+out.has("error") && total == 0 ->out.put("hint",@@ -2584,6 +2671,10 @@ class ScreenController(private const val FINGERPRINT_BUDGET_MS = 1_500Lprivate const val SEARCH_BUDGET_MS = 1_500L+ /** See [awaitHelper]. Measured: the helper binds in about a second. */+ private const val HELPER_WAIT_MS = 3_000L+ private const val HELPER_POLL_MS = 300L+/** The relative coordinate space the brain speaks: 0..1000 both axes. */private const val NORM_MAX = 1000diff --git a/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.kt b/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.ktnew file mode 100644index 0000000..acd9edb--- /dev/null+++ b/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.kt@@ -0,0 +1,58 @@+package net.kuafuai.andee.screen++import android.content.ComponentName+import android.content.Context+import android.content.pm.PackageManager+import android.os.SystemClock+import android.provider.Settings+import android.util.Log++/**+ * Switches on the one system service that makes WeChat hand over its tree.+ *+ * Measured on a Xiaomi Pad 5 (MIUI, WeChat 8.0.78, 2026-10-08): WeChat gives+ * every accessibility client a blank placeholder root unless MIUI's own+ * `MiuiEnhanceTBService` (the "TalkBack enhance" helper) is enabled — then the+ * same screen dumps 219 nodes. MIUI also flips that service on or off by+ * itself whenever Andee is re-bound (reinstall, crash, toggle), which is the+ * "有时候可以有时候不行" the user saw.+ *+ * Only ever *adds* to the enabled list, never removes, and only when the+ * component exists on this device and `WRITE_SECURE_SETTINGS` was granted —+ * everywhere else [wake] is a no-op returning false. Throttled so a user who+ * switches it off on purpose is not overruled on every dump.+ */+object TreeHelper {+ private const val TAG = "Body"++ private val HELPER = ComponentName(+ "com.miui.accessibility",+ "com.miui.accessibility.enhance.tb.MiuiEnhanceTBService",+ )++ private const val RETRY_AFTER_MS = 60_000L++ @Volatile private var lastTryAt = 0L++ /** True when this call just added the helper to the enabled list. */+ fun wake(context: Context): Boolean {+ if (context.checkSelfPermission(android.Manifest.permission.WRITE_SECURE_SETTINGS) !=+ PackageManager.PERMISSION_GRANTED+ ) return false+ if (runCatching { context.packageManager.getServiceInfo(HELPER, 0) }.isFailure) return false+ val now = SystemClock.uptimeMillis()+ if (lastTryAt != 0L && now - lastTryAt < RETRY_AFTER_MS) return false+ lastTryAt = now+ val cr = context.contentResolver+ val current = Settings.Secure.getString(cr, Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES).orEmpty()+ val flat = HELPER.flattenToString()+ val short = HELPER.flattenToShortString()+ if (current.split(':').any { it.equals(flat, true) || it.equals(short, true) }) return false+ val next = if (current.isBlank()) flat else "$current:$flat"+ return runCatching {+ Settings.Secure.putString(cr, Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES, next)+ }.onFailure { Log.w(TAG, "tree helper: enable failed", it) }+ .getOrDefault(false)+ .also { if (it) Log.i(TAG, "tree helper: enabled $flat") }+ }+}diff --git a/app/src/main/res/values-en/strings.xml b/app/src/main/res/values-en/strings.xmlindex 9b755eb..4656e30 100644--- a/app/src/main/res/values-en/strings.xml+++ b/app/src/main/res/values-en/strings.xml@@ -167,6 +167,7 @@<string name="scenes_chip">Scene · %1$s</string><string name="scenes_entered_row">(Entered scene: %1$s)</string><string name="scenes_exited_row">(Left scene: %1$s)</string>+ <string name="tree_helper_enabled_row">(This app was hiding its screen; turned on the system\'s accessibility enhance service so it shows it again)</string><string name="settings_scenes_browse">Manage scenes</string><!-- The factory-reset confirmation list; see config/FactoryReset.kt. -->diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xmlindex 8c41ecf..2edfa30 100644--- a/app/src/main/res/values/strings.xml+++ b/app/src/main/res/values/strings.xml@@ -147,6 +147,8 @@<!-- Scrollback rows written by the device itself when a scene starts or ends. --><string name="scenes_entered_row">(进入情景:%1$s)</string><string name="scenes_exited_row">(退出情景:%1$s)</string>+ <!-- Scrollback row: the device switched on a system accessibility helper so an app would expose its UI. -->+ <string name="tree_helper_enabled_row">(这个应用把界面藏起来了,已打开系统的「无障碍增强」服务让它交出来)</string><string name="settings_scenes_browse">管理情景</string><!-- The 恢复出厂设置 confirmation list: one line per thing wiped.