Skip to content

armgit / andee

Public Imported from GitHub
0

Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。

feat(screen): switch MIUI's tree helper back on when WeChat hides its UI

booboosui6aa1fbd373ec
diff --git a/CLAUDE.md b/CLAUDE.md
index 9786814..b2d95fe 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -100,6 +100,9 @@ Ported from generic_bridge (phone.sh), whose finding is the founding rule here:
- **Stuck-loop guard**: per-location counting, not consecutive-with-no-change (the real dead-loop interleaves menu opens, wrong pages and backs, and apps emit content events constantly — both naive conditions miss it). Gestures within 120 px of each other share a bucket; within 5 minutes, the 4th tap on the same bucket is refused with an error telling the brain to stop, re-observe, and ask the user. Swipes are exempt (list scrolling legitimately repeats); e-taps are exempt (re-tapping a confirmed element is normal operation).
- **NO_TREE policy** (empty-list hint): prefer `back`/`home` to exit and retry via a different route over blind coordinate taps — a wrong exit costs one tap, a wrong tap can cost the task.
- **Stale-service note**: `ScreenController.goodDumps` records which packages have actually produced a usable element list here. When a dump comes back with *zero nodes* from a package with such a history, `reviveNote()` appends the evidence to the hint — WeChat's tree is genuinely obtainable, but the accessibility connection sometimes goes stale and only a human toggling Andee off/on in 设置 → 无障碍 restores it. It is **evidence, not a rule**: the note states the facts and the brain decides whether to interrupt the user. It is deliberately not keyed to `com.tencent.mm`, and deliberately fires only on `nodes_total == 0` — a self-drawn pane still reports its containers, so mini-programs never trigger it.
+- **An app can withhold its tree, and that is not a stale service** (`isWithheld`, measured 2026-10-08). WeChat 8.0.78 on the Xiaomi pad handed accessibility a blank root — no class, 0×0 bounds, no children — and so did the system's own `uiautomator dump` **with no accessibility service enabled at all** and a freshly started WeChat, while Settings dumped 89 nodes. The switch was another service: with MIUI's `com.miui.accessibility/.enhance.tb.MiuiEnhanceTBService` enabled beside Andee the same screen gave 219 nodes, and removing it blanked the root again, reproducibly both ways. That is the whole "sometimes it works" — toggling Andee never mattered, which is why a `withheld` dump gets its own hint *ahead of* `reviveNote` (work from the screenshot) instead of the toggle-Andee note. Occlusion by our card keeps real root bounds and a self-drawn pane keeps its containers, so neither can be mistaken for this.
+ - **And MIUI switches that helper by itself, so Andee switches it back** (`screen/TreeHelper.kt`). logcat (`talkback-test`) shows MIUI enabling or disabling `MiuiEnhanceTBService` on its own whenever Andee is re-bound — a reinstall, a crash, a toggle — which is how the tree came and went with nobody touching a setting. So a `withheld` dump calls `TreeHelper.wake`, which **appends** the helper to `enabled_accessibility_services` (`WRITE_SECURE_SETTINGS`, the same grant `ImeSwitch` uses) and `awaitHelper` re-dumps until the app hands over a real root, up to 3 s. It never removes anything, does nothing where the component does not exist or the grant is missing, writes a scrollback row (the user's system settings changed and they are owed the news — same rule as the vault fill), and is throttled to once a minute so a user who turns the helper off on purpose is not overruled on every dump. `tree_helper_enabled` in the result tells the hint to say "retry once".
+ - **Without the adb grant there is no way back, and the hint must not pretend there is.** The helper is TalkBack's hidden MIUI companion: logcat shows it started when TalkBack is switched on and stopped when TalkBack is switched off, and it has no switch of its own a user can find. Turning TalkBack on would work and is not an answer — it reads the whole screen aloud and turns every tap into select-then-double-tap. One cheaper route was tried and ruled out: matching the helper's own config (`typeAllMask`, `notificationTimeout=0`, same `FEEDBACK_GENERIC`) left WeChat blank on a cold start, so it recognises the helper by identity, not by what it subscribes to. So a device with no grant gets the screenshot route and a hint that tells the brain **not** to send the user hunting for a setting.
### Text input: the service types for itself, ADBKeyboard is the spare
diff --git a/README.md b/README.md
index d0baf9a..7e49215 100644
--- a/README.md
+++ b/README.md
@@ -382,6 +382,32 @@ report success and a filename.
*Note:* most vendor ROMs **switch this grant off again** after every system update and every
force-stop. That is ROM behaviour, and not something the app can prevent.
+*WeChat (and other apps) sometimes show a blank screen to it.* Some apps hand **every**
+accessibility client an empty placeholder instead of their real UI. This is not Andee's fault:
+the system's own `uiautomator dump` gets the same blank page. Measured on a Xiaomi Pad 5 with
+MIUI and WeChat 8.0.78, WeChat only shows its real UI tree when MIUI's hidden
+**`MiuiEnhanceTBService`** is enabled. That is a TalkBack companion service, and it has no switch
+in Settings. With it on, the same chat screen dumps 219 nodes. With it off, it dumps none. MIUI
+also turns that service on and off by itself whenever Andee is re-bound (after a reinstall, a
+crash, or a toggle). That is why WeChat could be read **sometimes but not always**, and why
+toggling Andee never actually fixed it. Matching the helper's accessibility config does not help
+either: WeChat checks which service is asking, not what it asked for.
+
+What Andee does about it (`screen/TreeHelper.kt`): when a dump comes back blank like this, Andee
+**adds** that service to the enabled list and dumps again. It never removes anything. It does
+nothing on devices that don't have that component. It writes a line in the chat history so you
+know your system settings changed. It also tries at most once a minute, so if you turn the
+service off on purpose, Andee won't keep turning it back on. Writing that list needs one grant
+that only adb can give. It is the same grant the ADBKeyboard fallback uses:
+
+```bash
+adb shell pm grant net.kuafuai.andee android.permission.WRITE_SECURE_SETTINGS
+```
+
+Without the grant, nothing breaks. On the blank screens, Andee works from screenshots and
+coordinates instead of the element list. That is slower and less precise, but the task still
+gets done.
+
**Microphone.**
*Why:* ASR is streaming, so it needs the PCM stream upstreamed.
diff --git a/README.zh-CN.md b/README.zh-CN.md
index c15b01e..94e2767 100644
--- a/README.zh-CN.md
+++ b/README.zh-CN.md
@@ -557,6 +557,26 @@ Manifest 里声明的能力(`res/xml/accessibility_service_config.xml`):
**注意**:大部分厂商 ROM 在每次系统更新、每次强制停止应用之后会**自动关掉**这个权限,需要重开。
这是 ROM 行为,不是应用能阻止的。
+**微信(或别的应用)有时候读不到界面**:有些应用会给**所有**无障碍客户端一个空白的占位界面,
+不给真实的 UI。这不是 Andee 的问题:系统自带的 `uiautomator dump` 拿到的也是空白。我们在小米平板 5
+(MIUI,微信 8.0.78)上实测,只有开着 MIUI 一个隐藏服务 **`MiuiEnhanceTBService`** 的时候,微信才肯
+交出真实的 UI 树。这个服务是 TalkBack 的配套服务,设置里没有它的开关。开着它,同一个聊天页能读到
+219 个节点;关掉它,一个都读不到。而且每次 Andee 被重新绑定(重装、崩溃、开关一次),MIUI 都会
+**自己**打开或关掉这个服务。这就是"有时候能读到、有时候不行"的原因,也是为什么重开 Andee 从来
+没真正解决过问题。把 Andee 的无障碍配置改成和它一样也没用:微信认的是"谁在要",不是"要了什么"。
+
+Andee 的处理方式(`screen/TreeHelper.kt`):读到这种空白界面时,Andee 会把这个服务**追加**到已启用
+列表里,然后重新读一次。它只加不删。设备上没有这个组件时,它什么都不做。它会在聊天记录里写一行,
+让你知道系统设置被改过了。它最多一分钟尝试一次,所以如果你是故意关掉这个服务的,Andee 不会反复
+把它打开。修改这个列表需要一项只能用 adb 授予的权限,和 ADBKeyboard 备用通道用的是同一项:
+
+```bash
+adb shell pm grant net.kuafuai.andee android.permission.WRITE_SECURE_SETTINGS
+```
+
+不授权也不会坏。遇到空白界面时,Andee 改用截图加坐标来操作,不再依赖元素列表。这样更慢、
+也没那么准,但事情照样能做完。
+
### 三、麦克风
**为什么要**:ASR 是流式识别,需要拿 PCM 数据流上行。
diff --git a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
index 43b1938..7d7e805 100644
--- a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
+++ b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
@@ -17,8 +17,11 @@ import android.util.Log
import android.view.Display
import android.view.accessibility.AccessibilityNodeInfo
import android.view.accessibility.AccessibilityWindowInfo
+import net.kuafuai.andee.R
import net.kuafuai.andee.device.DeviceState
+import net.kuafuai.andee.i18n.AppLocale
import net.kuafuai.andee.net.ToolSchemas
+import net.kuafuai.andee.ui.ChatHistory
import org.json.JSONArray
import org.json.JSONObject
import java.io.ByteArrayOutputStream
@@ -1603,6 +1606,8 @@ class ScreenController(
waited += SETTLE_POLL_MS
out = dumpOnce(verbose, budget)
}
+ val helped = !verbose && out.optBoolean("withheld") && TreeHelper.wake(service)
+ if (helped) out = awaitHelper(out)
// Stability pass — the half the old loop never covered: "has content"
// is not "has stopped moving". A screen mid-animation (list settling,
// pager sliding, splash fading) dumps real elements at positions that
@@ -1639,6 +1644,7 @@ class ScreenController(
out.put("still_moving", true)
}
}
+ if (helped) out.put("tree_helper_enabled", true)
if (!verbose && out.optInt("nodes_useful", 0) == 0 && waited >= SETTLE_BUDGET_MS) {
out.put("settling", true).put("settle_waited_ms", waited)
}
@@ -1725,6 +1731,27 @@ class ScreenController(
return out
}
+ /**
+ * Re-dump after [TreeHelper] switched the system's helper on. The setting
+ * is written at once but the service binds asynchronously, and the app
+ * only hands over its tree once it has — so this polls rather than sleeps.
+ * Its own budget: the wait is not walking, and the first dump may already
+ * have spent most of [TREE_BUDGET_MS].
+ */
+ private fun awaitHelper(first: JSONObject): JSONObject {
+ runCatching {
+ ChatHistory.addAssistant(AppLocale.str(service, R.string.tree_helper_enabled_row))
+ }
+ var out = first
+ val until = SystemClock.uptimeMillis() + HELPER_WAIT_MS
+ while (SystemClock.uptimeMillis() < until) {
+ try { Thread.sleep(HELPER_POLL_MS) } catch (_: InterruptedException) { break }
+ out = dumpOnce(false, WalkBudget(TREE_BUDGET_MS))
+ if (!out.optBoolean("withheld")) break
+ }
+ return out
+ }
+
/**
* Signature of a dump's content for the stability pass: which nodes
* exist, their ids and their bounds. Bounds are the part that matters —
@@ -1750,7 +1777,9 @@ class ScreenController(
try {
val stats = Stats(budget)
val body = buildNode(active, verbose, stats) ?: JSONObject().put("empty", true)
- withStats(body, active.packageName?.toString() ?: "", stats)
+ withStats(body, active.packageName?.toString() ?: "", stats).also {
+ if (isWithheld(active)) it.put("withheld", true)
+ }
} finally {
active.recycle()
}
@@ -1774,6 +1803,29 @@ class ScreenController(
return out
}
+ /**
+ * The app handed accessibility a root with no class, no size and no
+ * children — a placeholder, not a pane.
+ *
+ * Measured on WeChat 8.0.78 (Xiaomi Pad 5, 2026-10-08): the system's own
+ * `uiautomator dump` got the same single blank node with **no**
+ * accessibility service enabled and a freshly started WeChat, while
+ * Settings dumped 89 nodes. With MIUI's `MiuiEnhanceTBService` enabled
+ * alongside us the same screen dumped 219 nodes, and removing it blanked the
+ * root again — reproducible both ways, and the real "有时候可以有时候不行".
+ * So it is the app refusing every client, not our
+ * connection going stale — and the two must not share a hint, because the
+ * stale one sends the user to toggle Andee, which cannot help here.
+ * Occlusion (our card) and self-drawn panes are both distinguishable: they
+ * keep real bounds and their containers.
+ */
+ private fun isWithheld(root: AccessibilityNodeInfo): Boolean = runCatching {
+ if (root.childCount != 0 || root.packageName?.toString() == service.packageName) return@runCatching false
+ val r = android.graphics.Rect()
+ root.getBoundsInScreen(r)
+ r.isEmpty && root.className.isNullOrEmpty()
+ }.getOrDefault(false)
+
/**
* Fallback for the case above: walk every window we can see and keep the
* best one. Only ever entered when the active window gave us nothing, so the
@@ -2003,7 +2055,7 @@ class ScreenController(
val keep = setOf(
"ts", "pkg", "nodes_total", "nodes_kept", "nodes_useful", "surface",
"source", "windows_seen", "windows_detail", "settling", "settle_waited_ms",
- "error", "empty",
+ "error", "empty", "withheld", "tree_helper_enabled",
)
val rootKeys = ArrayList(out.keys().asSequence().toList())
for (k in rootKeys) if (k !in keep) out.remove(k)
@@ -2240,6 +2292,41 @@ class ScreenController(
"not tell the user to restart anything, and do not tap blindly.",
)
+ // Ahead of the two [reviveNote] branches: the app worked before,
+ // so they would send the user off to toggle Andee — which is
+ // exactly what they did, to no effect. See [isWithheld].
+ useful == 0 && out.optBoolean("withheld") -> {
+ val app = out.optString("pkg").ifEmpty { "This app" }
+ val helper = if (out.optBoolean("tree_helper_enabled")) {
+ "Andee has just switched that helper on itself and the app has not picked " +
+ "it up yet — the next dump may read normally, so retry once before " +
+ "giving up on the list; if it is still blank, the user may need to " +
+ "reopen the app. "
+ } else {
+ "Andee could not switch it on here, and the user cannot either: it is a " +
+ "hidden companion MIUI only runs alongside TalkBack, and TalkBack " +
+ "would read the whole screen aloud and change how every tap works. So " +
+ "do not send the user to look for any setting — if it matters, say " +
+ "once that in this app you are working from the picture, and carry on. "
+ }
+ out.put(
+ "hint",
+ "`$app` is withholding its UI from accessibility: its window hands every " +
+ "accessibility client — the system's own uiautomator included — a blank " +
+ "placeholder with no size and no children. That is the app's choice, not a " +
+ "fault on this device; switching Andee off and on does not change it, so " +
+ "do not ask the user to. What it does depend on is which *other* " +
+ "accessibility services are on: WeChat was measured exposing its full " +
+ "tree only while MIUI's TalkBack helper (MiuiEnhanceTalkback) was also " +
+ "enabled. " + helper + "It covers " +
+ "this whole app, not just this screen — so backing out to find a readable " +
+ "page will not help either. Work from $eyes: find the target in the image " +
+ "and use tap_by_coordinates / swipe_by_coordinates (0-1000), checking the " +
+ "result in the screenshot each tap returns. type_text and submit_input " +
+ "still work on a focused field.",
+ )
+ }
+
out.has("error") && total == 0 ->
out.put(
"hint",
@@ -2584,6 +2671,10 @@ class ScreenController(
private const val FINGERPRINT_BUDGET_MS = 1_500L
private const val SEARCH_BUDGET_MS = 1_500L
+ /** See [awaitHelper]. Measured: the helper binds in about a second. */
+ private const val HELPER_WAIT_MS = 3_000L
+ private const val HELPER_POLL_MS = 300L
+
/** The relative coordinate space the brain speaks: 0..1000 both axes. */
private const val NORM_MAX = 1000
diff --git a/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.kt b/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.kt
new file mode 100644
index 0000000..acd9edb
--- /dev/null
+++ b/app/src/main/java/net/kuafuai/andee/screen/TreeHelper.kt
@@ -0,0 +1,58 @@
+package net.kuafuai.andee.screen
+
+import android.content.ComponentName
+import android.content.Context
+import android.content.pm.PackageManager
+import android.os.SystemClock
+import android.provider.Settings
+import android.util.Log
+
+/**
+ * Switches on the one system service that makes WeChat hand over its tree.
+ *
+ * Measured on a Xiaomi Pad 5 (MIUI, WeChat 8.0.78, 2026-10-08): WeChat gives
+ * every accessibility client a blank placeholder root unless MIUI's own
+ * `MiuiEnhanceTBService` (the "TalkBack enhance" helper) is enabled — then the
+ * same screen dumps 219 nodes. MIUI also flips that service on or off by
+ * itself whenever Andee is re-bound (reinstall, crash, toggle), which is the
+ * "有时候可以有时候不行" the user saw.
+ *
+ * Only ever *adds* to the enabled list, never removes, and only when the
+ * component exists on this device and `WRITE_SECURE_SETTINGS` was granted —
+ * everywhere else [wake] is a no-op returning false. Throttled so a user who
+ * switches it off on purpose is not overruled on every dump.
+ */
+object TreeHelper {
+ private const val TAG = "Body"
+
+ private val HELPER = ComponentName(
+ "com.miui.accessibility",
+ "com.miui.accessibility.enhance.tb.MiuiEnhanceTBService",
+ )
+
+ private const val RETRY_AFTER_MS = 60_000L
+
+ @Volatile private var lastTryAt = 0L
+
+ /** True when this call just added the helper to the enabled list. */
+ fun wake(context: Context): Boolean {
+ if (context.checkSelfPermission(android.Manifest.permission.WRITE_SECURE_SETTINGS) !=
+ PackageManager.PERMISSION_GRANTED
+ ) return false
+ if (runCatching { context.packageManager.getServiceInfo(HELPER, 0) }.isFailure) return false
+ val now = SystemClock.uptimeMillis()
+ if (lastTryAt != 0L && now - lastTryAt < RETRY_AFTER_MS) return false
+ lastTryAt = now
+ val cr = context.contentResolver
+ val current = Settings.Secure.getString(cr, Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES).orEmpty()
+ val flat = HELPER.flattenToString()
+ val short = HELPER.flattenToShortString()
+ if (current.split(':').any { it.equals(flat, true) || it.equals(short, true) }) return false
+ val next = if (current.isBlank()) flat else "$current:$flat"
+ return runCatching {
+ Settings.Secure.putString(cr, Settings.Secure.ENABLED_ACCESSIBILITY_SERVICES, next)
+ }.onFailure { Log.w(TAG, "tree helper: enable failed", it) }
+ .getOrDefault(false)
+ .also { if (it) Log.i(TAG, "tree helper: enabled $flat") }
+ }
+}
diff --git a/app/src/main/res/values-en/strings.xml b/app/src/main/res/values-en/strings.xml
index 9b755eb..4656e30 100644
--- a/app/src/main/res/values-en/strings.xml
+++ b/app/src/main/res/values-en/strings.xml
@@ -167,6 +167,7 @@
<string name="scenes_chip">Scene · %1$s</string>
<string name="scenes_entered_row">(Entered scene: %1$s)</string>
<string name="scenes_exited_row">(Left scene: %1$s)</string>
+ <string name="tree_helper_enabled_row">(This app was hiding its screen; turned on the system\'s accessibility enhance service so it shows it again)</string>
<string name="settings_scenes_browse">Manage scenes</string>
<!-- The factory-reset confirmation list; see config/FactoryReset.kt. -->
diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml
index 8c41ecf..2edfa30 100644
--- a/app/src/main/res/values/strings.xml
+++ b/app/src/main/res/values/strings.xml
@@ -147,6 +147,8 @@
<!-- Scrollback rows written by the device itself when a scene starts or ends. -->
<string name="scenes_entered_row">(进入情景:%1$s)</string>
<string name="scenes_exited_row">(退出情景:%1$s)</string>
+ <!-- Scrollback row: the device switched on a system accessibility helper so an app would expose its UI. -->
+ <string name="tree_helper_enabled_row">(这个应用把界面藏起来了,已打开系统的「无障碍增强」服务让它交出来)</string>
<string name="settings_scenes_browse">管理情景</string>
<!-- The 恢复出厂设置 confirmation list: one line per thing wiped.