Skip to content

armgit / andee

Public Imported from GitHub
0

Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。

fix(mac): type into apps that hide their focused field, without the shell

booboosui1866d2f30b23
diff --git a/CLAUDE.md b/CLAUDE.md
index 4255904..039fdd0 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -661,6 +661,7 @@ meetings, gear, friends) and lists exactly the tools some provider handles. What
listens, double-tap stops and types, long-press opens the window; a sideways drag on the chat ball switches
character.
- **Notifications come from `usernoted`'s SQLite store, and where it is depends on the macOS version** (`notifCandidates` in `AndeeMac.swift`): macOS 15 keeps it in `~/Library/Group Containers/group.com.apple.usernoted/db2`, earlier systems under `confstr(_CS_DARWIN_USER_DIR)` + `com.apple.notificationcenter/db2` (never derive that from `$TMPDIR` — the JVM replaces it). Only the first was tried, and a store that was not there was reported as missing Full Disk Access, so the user granted it and was told the same thing forever. `andee_notif_latest_id` now answers -2 for "no store anywhere" (`State.NOT_FOUND`, no permission button) and -1 for "exists but refuses us" (`NO_ACCESS`), and reads in process through `libsqlite3` (read-only, busy timeout, since `usernoted` is writing the WAL) instead of forking `/usr/bin/sqlite3`, which swallowed the reason.
+- **`type_text` pastes even when the front app reports no focused element** (`MacScreen.typeBlind`). Mac WeChat answers `focused: false` to accessibility, and refusing there sent the model to the shell (`osascript` + ⌘V), which a notification's turn may not use — so a message Andee could read, it could not answer. The blind path sends ⌘V only (no ⌘A: with focus unknown, select-all may take the chat history), refuses when Andee's own window is active, and returns `unverified` with a note to check the screenshot and not to reach for the shell. Verified in WeChat's 文件传输助手 by reading the box back with ⌘A ⌘C. The pasteboard is still not restored afterwards — that was a deliberate choice (see `andee_set_pasteboard`).
- **A prompt card appears beside the ball** (`PromptWindow`), which may be on a screen the user is not looking
at. A friend's request therefore plays a sound and waves the ball first — the first one timed out unseen.
diff --git a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt
index 5e14472..34acb33 100644
--- a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt
+++ b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt
@@ -252,7 +252,7 @@ class MacScreen(
*/
private fun type(text: String): JSONObject {
val before = Bridge.focusedValue()
- if (!before.optBoolean("focused")) throw IllegalStateException("no field has focus — click into the field first")
+ if (!before.optBoolean("focused")) return typeBlind(text)
// Keys go to the focused app, not to a point on screen: fading our windows changes nothing for them.
var via = "paste"
if (Bridge.setPasteboard(text)) {
@@ -278,6 +278,36 @@ class MacScreen(
}
}
+ /**
+ * The app in front reports no focused element to accessibility — Mac WeChat is the case that forced this.
+ * Refusing here ("no field has focus") was a dead end the model could only leave through the shell
+ * (`osascript` to set the clipboard and press ⌘V), and the shell is closed to a notification's turn, so
+ * a message arrived that Andee could read and could not answer. The keystroke never needed the focused
+ * element: ⌘V goes to whatever the front app has focused, exactly as a person's would.
+ *
+ * So: paste, and only paste. No ⌘A — with no idea where focus is, select-all may select the chat history
+ * rather than the box. Unverified by construction, and the result says what the screenshot must show.
+ */
+ private fun typeBlind(text: String): JSONObject {
+ // ⌘V lands in the frontmost app, and if that is Andee's own window it lands in Andee's own input.
+ if (java.awt.Window.getWindows().any { it.isActive }) {
+ throw IllegalStateException("Andee's own window is in front, so a paste would land in Andee. Click into the other app's input box first.")
+ }
+ val app = Bridge.frontmost().optString("name").ifEmpty { throw IllegalStateException("no app is in front — open the app and click into its input box first") }
+ if (!Bridge.setPasteboard(text)) throw IllegalStateException("could not put the text on the pasteboard")
+ Bridge.key(Bridge.Keys.V, Bridge.Keys.CMD)
+ Thread.sleep(350)
+ return JSONObject().put("performed", true).put("unverified", true).put("via", "paste_blind").put("len", text.length)
+ .put("app", app)
+ .put(
+ "note",
+ "$app does not tell the accessibility layer which box has focus, so I pasted into whatever it has focused " +
+ "(⌘V, once) and cannot read it back. Look at a screenshot: the text must be in the input box before you " +
+ "submit. If the box already had text, mine was added at the cursor, not in place of it. Do NOT type it " +
+ "again and do NOT use the shell to paste — this is the way to type into $app.",
+ )
+ }
+
private fun submit(): JSONObject {
Bridge.key(Bridge.Keys.RETURN)
return JSONObject().put("performed", true).put("via", "return_key")
diff --git a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/shell/ShellPolicy.kt b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/shell/ShellPolicy.kt
index 6327e66..f28f86e 100644
--- a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/shell/ShellPolicy.kt
+++ b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/shell/ShellPolicy.kt
@@ -74,17 +74,22 @@ object ShellPolicy {
fun decide(tier: Tier, turn: Turn, mode: Mode, alwaysAllowed: Boolean, criticalConfirm: Boolean): Decision {
when (turn) {
Turn.NOTIFY -> return Decision.Refuse(
- "This turn started from a notification somebody else sent, so no command can run from it. " +
- "Tell the user what you were about to run and let them ask for it themselves.",
+ "This turn started from a notification somebody else sent, so no command can run from it — not in this turn " +
+ "and not in a todo you set from it. To type into an app (a reply), use type_text, which needs no command. " +
+ "Otherwise tell the user what you were about to run and let them ask for it themselves; do not promise a " +
+ "later turn will run it.",
)
Turn.SILENT -> return Decision.Refuse("A silent review may only use the notebook; it must not run commands.")
Turn.WAKE_OTHER -> if (tier != Tier.SAFE) return Decision.Refuse(
"This todo was not set by the user, and nobody is here to approve a command — only read-only " +
- "commands may run now. Tell the user what you wanted to run when they are back.",
+ "commands may run now. To type into an app, use type_text, which needs no command. Tell the user what " +
+ "you wanted to run when they are back; do not promise another automatic turn will run it.",
)
Turn.WAKE_OWN -> if (mode != Mode.FULL && tier != Tier.SAFE) return Decision.Refuse(
"A todo is running and the user may not be at the Mac to approve anything, so only read-only " +
- "commands may run now. Tell the user what you wanted to run when they are back.",
+ "commands may run now (full access lifts this). To type into an app, use type_text, which needs no " +
+ "command. Tell the user what you wanted to run when they are back; do not promise another automatic " +
+ "turn will run it.",
)
Turn.USER -> Unit
}