Skip to content

armgit / andee

Public Imported from GitHub
0

Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。

fix(screen): wait for the system to notice the fold before reading the tree

booboosuid3b797951455
diff --git a/CLAUDE.md b/CLAUDE.md
index e8d6f3b..73a6075 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -85,7 +85,7 @@ The mirror-image problem is **accessibility** visibility, and it is not solved b
What makes `hideFromAccessibility()` actually work is **`accessibilityDataSensitive`, not `importantForAccessibility`** — and the difference is invisible until you look. `accessibility_service_config.xml` asks for `flagIncludeNotImportantViews`, which makes `View.includeForAccessibility()` return true no matter what `importantForAccessibility` says. So `NO_HIDE_DESCENDANTS` on its own hides our overlays from every service *except the one that needed it*: measured, with the vault form open, `get_screen_element` returned the whole form, labels and phone number included. `ACCESSIBILITY_DATA_SENSITIVE_YES` (API 34+, inherits down the subtree) is honoured against us because we do not declare `android:isAccessibilityTool` — and is strictly kinder than the blunt flag, since TalkBack does declare it and keeps the ball. Both are set; the old one is the pre-34 fallback. Don't casually drop `flagIncludeNotImportantViews` to "simplify" this — it is what makes unlabelled app views visible at all.
-Hiding the two **focusable** cards (`SettingsUi`, `VaultUi`) then creates a second-order problem: they are the active window, and a hidden active window dumps as zero nodes — the exact signature `reviveNote()` reads as "the accessibility connection went stale, have the user toggle Andee off and on". That message is aimed at a human, so it has to be true. `OwnCard` in `ui/OverlayA11y.kt` (incremented in those two classes' `show`/`hide`; counted, because the vault opens on top of settings) lets `annotate` say the real thing instead — the user has our own card open, nothing is broken, wait or ask them to close it. Same rule as the meeting recorder's `stopped_by` and `device.look`'s grace message.
+Hiding the two **focusable** cards (`SettingsUi`, `VaultUi`) then creates a second-order problem: they are the active window, and a hidden active window dumps as zero nodes — the exact signature `reviveNote()` reads as "the accessibility connection went stale, have the user toggle Andee off and on". That message is aimed at a human, so it has to be true. `OwnCard` in `ui/OverlayA11y.kt` (incremented in those two classes' `show`/`hide`; counted, because the vault opens on top of settings) lets `annotate` say the real thing instead — the user has our own card open, nothing is broken, wait or ask them to close it. Same rule as the meeting recorder's `stopped_by` and `device.look`'s grace message. The *unfolded* card produces the same zero by a different mechanism and wants a different answer — it is not focusable and hides nothing, it simply covers — see `FullscreenCard` under "Getting out of the way is the default".
### Coordinate protocol: the model never does math
@@ -370,7 +370,13 @@ Four things about that spread are not cosmetic:
### Getting out of the way is the default
-`CommandDispatcher.dispatch` folds the card into the corner ball before running **any** `screen.*` command — by prefix, so a new method there inherits it. It is not a tool anymore: `reveal_screen` is deleted, because a brain that forgot to call it left the user staring at our face instead of the app being operated, and its `seconds` deadline meant the assistant deciding how long the user gets to look. The fold has no deadline; the user taps the corner ball to talk (`dispatcher` does not expand), and long-press unfolds (`dispatcher.expand()`). **The service starts *unfolded*** — `onServiceConnected` calls `window.setCompact(false)` after `show()`, so the first thing the user sees is the full card, not a ball in the corner. It used to start folded, on the reasoning that a freshly enabled service should not freeze the tablet with a full-screen touchable window; what answers that worry is `ensureCompact()`, which folds the card before any `screen.*` command drives another app, and it was never touched. The only remaining unfold paths are therefore the service's own birth and a long-press on the ball. If you add a `screen.*` method that does *not* drive the device, keep it out of the agent-driving path.
+`CommandDispatcher.dispatch` folds the card into the corner ball before running **any** `screen.*` command — by prefix, so a new method there inherits it. It is not a tool anymore: `reveal_screen` is deleted, because a brain that forgot to call it left the user staring at our face instead of the app being operated, and its `seconds` deadline meant the assistant deciding how long the user gets to look. The fold has no deadline; the user taps the corner ball to talk (`dispatcher` does not expand), and long-press unfolds (`dispatcher.expand()`). **The service starts *unfolded*** — `onServiceConnected` calls `window.setCompact(false)` after `show()`, so the first thing the user sees is the full card, not a ball in the corner. It used to start folded, on the reasoning that a freshly enabled service should not freeze the tablet with a full-screen touchable window; what answers that worry is `ensureCompact()`, which folds the card before any `screen.*` command drives another app — but that change is also what made the next paragraph reachable, so `ensureCompact()` did not survive it untouched. The only remaining unfold paths are therefore the service's own birth and a long-press on the ball. If you add a `screen.*` method that does *not* drive the device, keep it out of the agent-driving path.
+
+**Folding the card is two events, and waiting for the wrong one reads as a dead accessibility service.** Reported as "之前能抓带微信的ui树,现在抓不到了,好像重启个什么权限就行了" — and the user believed that because our own tool hint told them so. The card is `FLAG_NOT_FOCUSABLE`, so it never takes focus and the app behind it stays `active`, `focused` and rooted; it merely *covers* the display, and every node of an occluded window answers `isVisibleToUser == false`. `ScreenController.buildNode` drops those **before** `st.total++`, so the dump is `nodes_total: 0` from a perfectly healthy screen — byte-for-byte the signature `reviveNote()` reports as a stale connection. Measured on this tablet: card up, Chrome present/`active`/`focused`/`root: true`, 0 nodes; card folded, 75 from the same screen.
+
+The bug was that `ensureCompact()` waited on the fold *request*. `updateViewLayout` only asks; the system recomputes occlusion a frame or two later, **measured at 4 polls / ~140 ms**, against the couple of milliseconds `ensureCompact`'s two `ui.post` turns take to return (`COMPACT_SETTLE_MS` is an `await` cap, never a floor). So the dump ran ~140 ms early, repeatably. It now polls `ScreenController.activeWindowVisible()` to `UNCOVER_BUDGET_MS` (600 ms, several times the measurement because overrunning it costs an empty tree) — a poll and not a longer sleep, for the same reason the typing readback is one: the settle is tens of ms on a free main thread and much longer on a busy one, and paying the worst case on every tool call taxes all of them. It is only reached when a fold actually happened, so the usual path — a card already folded, which is most of a task — pays nothing. `ensureCompact` also asks `window.isCompact()` rather than trusting its `compacted` cache; that one is belt-and-braces, not the fix (`ensureOverlays()`'s only caller follows it with `expand()`, so no live desync path exists today).
+
+A missed settle is survivable rather than silent: `ui.FullscreenCard` (set by `FloatingWindowUi` alone, derived from `root != null && !compact` so a card hidden while unfolded cannot keep claiming to cover) gives `annotate()` a branch ahead of the two `reviveNote` ones that names the card and says nothing is broken. It carries **no** `revive` — that note asks for a permission toggle, and asking for one when nothing is wrong is precisely how this went wrong. A dump that still reaches it is a passive observer (`driving = false` on the 9008 socket, by design) or a fold that failed, and both want naming rather than a guess about accessibility.
Handlers otherwise must **not** block for the length of a timed effect — `reveal_screen` did, and a ten-second tool call reads to the brain as work still in progress, so it sat silent while the user stared at the thing the reveal was for. `screen.wait` is the one deliberate exception, because there the waiting *is* the request and returning early would answer the wrong question. It pays for that by being bounded: `WAIT_MAX_MS` is 20s against `body_hub`'s 30s request timeout, and the element dump plus screenshot that follow the sleep are inside the same request (~2.5s measured, more on a page that never settles). Anything with a duration that is *not* a wait still arms a `Handler` and returns.
diff --git a/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt b/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
index baf77aa..db0cef2 100644
--- a/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
+++ b/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
@@ -1060,11 +1060,17 @@ class CommandDispatcher(
}
private fun ensureCompact() {
- val wait = !compacted
+ // The window, not the cache. [compacted] is written on the main thread
+ // by everything that folds, but the card can be unfolded by paths that
+ // never come back through here — birth, `ensureOverlays`, a card that
+ // was hidden and re-shown — and a cache that says "already folded"
+ // about a fullscreen card leaves it covering the app for the whole tool
+ // call. `isCompact()` is a field read, so asking is free.
+ val wait = !compacted || !window.isCompact()
val settled = CountDownLatch(1)
ui.post {
touchBusy()
- if (!compacted) {
+ if (wait) {
window.setCompact(true)
compacted = true
// Counted down a turn later rather than inline: `setCompact`
@@ -1079,7 +1085,37 @@ class CommandDispatcher(
}
// Bounded, and only a hint: a busy main thread costs us the wait, not
// the command.
- if (wait) runCatching { settled.await(COMPACT_SETTLE_MS, TimeUnit.MILLISECONDS) }
+ if (!wait) return
+ runCatching { settled.await(COMPACT_SETTLE_MS, TimeUnit.MILLISECONDS) }
+ awaitUncovered()
+ }
+
+ /**
+ * Wait for the *system* to notice the fold, which is a separate event from
+ * the fold itself and the one that actually matters.
+ *
+ * `updateViewLayout` only asks; occlusion is recomputed a frame or two
+ * later. Until it is, the app behind the card still reports every node as
+ * `isVisibleToUser == false` and the dump comes back with zero nodes from a
+ * screen that is fine — the same signature as a stale accessibility
+ * connection, which is how this shipped as "WeChat's tree cannot be read
+ * any more" and sent the user off to toggle a permission that was working.
+ *
+ * A poll rather than a longer sleep, for the same reason the typing readback
+ * is one: the settle is a few tens of ms when the main thread is free and
+ * much longer when it is not, and paying the worst case on every tool call
+ * is a tax on all of them. Bounded, and a miss is not fatal — the dump's own
+ * hint can then name the card (see [net.kuafuai.andee.ui.FullscreenCard]).
+ *
+ * Only reached when a fold actually happened, so the usual path — a card
+ * that was already folded, which is most of a task — pays nothing.
+ */
+ private fun awaitUncovered() {
+ val deadline = System.currentTimeMillis() + UNCOVER_BUDGET_MS
+ while (!screen.activeWindowVisible()) {
+ if (System.currentTimeMillis() >= deadline) return
+ runCatching { Thread.sleep(UNCOVER_POLL_MS) }
+ }
}
/**
@@ -1309,6 +1345,17 @@ class CommandDispatcher(
/** How long to let the main thread take to apply the fold. */
const val COMPACT_SETTLE_MS = 200L
+ /**
+ * And how long to then let the *system* take to notice it — see
+ * [awaitUncovered]. Measured on this tablet at 4 polls / ~140 ms,
+ * repeatably, against the handful of milliseconds the fold itself takes
+ * to return: that gap is the whole bug. The budget is several times the
+ * measurement because overrunning it costs an empty tree, and it is paid
+ * once per unfold rather than per tool call.
+ */
+ const val UNCOVER_BUDGET_MS = 600L
+ const val UNCOVER_POLL_MS = 30L
+
/**
* Not a timeout on thinking — [setBusy] has none, on purpose. This only
* catches a brain that walks away mid-task (crash, dropped socket) so
diff --git a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
index 63774dc..a0f1cd4 100644
--- a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
+++ b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt
@@ -1124,6 +1124,32 @@ class ScreenController(
}
}
+ /**
+ * Whether the window in front reports itself as visible to us **right
+ * now** — the one question [net.kuafuai.andee.net.CommandDispatcher]'s fold
+ * has to be able to wait on.
+ *
+ * A fullscreen overlay of ours occludes the app, and every node of an
+ * occluded window answers `isVisibleToUser == false`; [buildNode] drops
+ * those before it counts them, so the whole tree reads as empty. Folding
+ * the card fixes it, but `updateViewLayout` only *asks* — the system
+ * recomputes occlusion a frame or two later, measured here at ~140 ms after
+ * the fold call has already returned. A dump that beats that recomputation
+ * gets zero nodes from a healthy screen.
+ *
+ * False is deliberately also the answer when there is no root at all: the
+ * caller polls to a deadline, and a screen mid-transition is worth the same
+ * short wait as one still being uncovered.
+ */
+ fun activeWindowVisible(): Boolean {
+ val root = runCatching { service.rootInActiveWindow }.getOrNull() ?: return false
+ return try {
+ runCatching { root.isVisibleToUser }.getOrDefault(false)
+ } finally {
+ runCatching { root.recycle() }
+ }
+ }
+
/**
* Dump the active window's UI tree, with enough diagnostics that an empty
* tree can say *why* it is empty.
@@ -1683,16 +1709,20 @@ class ScreenController(
* if the capture failed, say so instead of promising an image that is not
* there.
*
- * The branches are not all the same verdict. Only the first three are
+ * The branches are not all the same verdict. Only the first four are
* genuinely empty; the last has nodes that simply have no names, and since
* [rebuildIndexAndList] started rendering nesting and centres for every
* node rather than sorting the labelled ones to the top, that case still
* has a usable outline and must not be told to tap by coordinate.
*
- * [reviveNote] is appended to the first two only — the two that mean "no
- * nodes AT ALL". That is the whole discrimination: a self-drawn pane still
- * reports its containers, so a mini-program never reaches those branches
- * and never nags the user about restarting anything.
+ * [reviveNote] is appended to the two genuinely-empty branches only — the
+ * two that mean "no nodes AT ALL" and have no other explanation. That is
+ * the whole discrimination: a self-drawn pane still reports its containers,
+ * so a mini-program never reaches those branches and never nags the user
+ * about restarting anything. The two branches ahead of them are the cases
+ * where a window of *ours* explains the zero, and they must not carry it —
+ * that note asks for a permission toggle, and asking for one when nothing
+ * is wrong is how this last went wrong.
*/
private fun annotate(out: JSONObject, verbose: Boolean) {
if (verbose) return
@@ -1722,6 +1752,21 @@ class ScreenController(
"whatever is behind the card.",
)
+ // Same zero, a different window of ours, and a different answer:
+ // the big card is not focusable, so the app below is still active
+ // and still has a root — it is merely *covered*, and an occluded
+ // window reports every node invisible. Nothing about accessibility
+ // is wrong here, so this has to come ahead of [reviveNote] too.
+ total == 0 && net.kuafuai.andee.ui.FullscreenCard.covering ->
+ out.put(
+ "hint",
+ "Andee's own card is unfolded over the whole screen, so the app behind it " +
+ "is covered and reports every node as not visible — that is why this is " +
+ "empty, and nothing is broken. Folding the card is automatic before any " +
+ "screen tool, so retry this call once and it should read normally. Do " +
+ "not tell the user to restart anything, and do not tap blindly.",
+ )
+
out.has("error") && total == 0 ->
out.put(
"hint",
diff --git a/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt b/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
index 2da9525..8fa5bec 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
@@ -399,6 +399,7 @@ class FloatingWindowUi(
wm.addView(v, p)
root = v
params = p
+ syncCover()
applyCompactAppearance()
// Only now can [winYOrigin] be measured, and the resting position
// depends on it — the y above is a first guess that is correct only
@@ -689,6 +690,7 @@ class FloatingWindowUi(
root = null
params = null
ball = null
+ syncCover()
true
} catch (_: Throwable) {
false
@@ -875,11 +877,25 @@ class FloatingWindowUi(
ui.post {
if (this.compact == compact) return@post
this.compact = compact
+ syncCover()
if (compact) enterCompact() else exitCompact()
applyWindow()
}
}
+ /**
+ * Publish "this window is covering the display" for the two readers that
+ * cannot be handed a reference to it — see [FullscreenCard].
+ *
+ * Derived from both facts rather than mirrored off [setCompact], because
+ * [hide] leaves `compact` alone: a card torn down while unfolded would
+ * otherwise keep claiming to cover a screen it is no longer on. Main thread
+ * only, like everything that touches these two fields.
+ */
+ private fun syncCover() {
+ FullscreenCard.covering = root != null && !compact
+ }
+
/**
* Whether the card is folded, read from outside.
*
diff --git a/app/src/main/java/net/kuafuai/andee/ui/OverlayA11y.kt b/app/src/main/java/net/kuafuai/andee/ui/OverlayA11y.kt
index 379ca1f..33fcb53 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/OverlayA11y.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/OverlayA11y.kt
@@ -105,3 +105,32 @@ object OwnCard {
depth.updateAndGet { if (it > 0) it - 1 else 0 }
}
}
+
+/**
+ * Whether the assistant's own card is unfolded over the whole display.
+ *
+ * A second way to get zero nodes, by a different mechanism than [OwnCard] and
+ * with a different answer. The card is `FLAG_NOT_FOCUSABLE`, so it never
+ * becomes the active window and the app behind it stays focused and keeps its
+ * root — but it *covers* the display, and every node of an occluded window
+ * reports `isVisibleToUser == false`. [ScreenController.buildNode] drops those
+ * before it counts them, so `nodes_total` is 0 with a perfectly healthy tree
+ * underneath. Measured on this device: with the card unfolded, Chrome's window
+ * is present, `active`, `focused`, `root: true` and yields 0 nodes; folding the
+ * card returns 75 from the same screen.
+ *
+ * It exists because that is the same signature [ScreenController.reviveNote]
+ * reads as a stale accessibility connection, and it sent a user off to toggle a
+ * permission that was working fine — the dump has to be able to say the real
+ * reason. Normally unreachable now that [net.kuafuai.andee.net.CommandDispatcher]
+ * waits for the fold to land, so a dump that still sees this is either a passive
+ * observer (`driving = false`) or the fold failing, and both want naming rather
+ * than a guess about accessibility.
+ *
+ * Set by [FloatingWindowUi] alone, from every place that changes whether the
+ * card is on screen at full size.
+ */
+object FullscreenCard {
+ @Volatile
+ var covering: Boolean = false
+}