Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。
fix: when its aim keeps missing, the device points the model at tap_target
booboosui4ffc863b6bcc
diff --git a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.ktindex 9c02192..7f55ef5 100644--- a/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt+++ b/app/src/main/java/net/kuafuai/andee/screen/ScreenController.kt@@ -3733,8 +3733,8 @@ class ScreenController(private const val SCREEN_SETTLE_MS = 380Lprivate const val LOOP_HELP =- "Stuck-loop guard: this exact spot has been touched more than " +- "6 times within 5 minutes. The gesture was NOT executed. " ++ net.kuafuai.andee.brain.AimHints.REPEAT_REFUSED + " (stuck-loop guard: more than " ++ "6 times within 5 minutes). The gesture was NOT executed. " +"Do not retry the same tap. Take a fresh screenshot or get_screen_element, " +"pick a genuinely different approach, and if the goal still cannot " +"be reached, stop and ask the user for help instead of continuing."diff --git a/app/src/test/resources/golden/local_prompt.txt b/app/src/test/resources/golden/local_prompt.txtindex 5496658..cb3bfb7 100644--- a/app/src/test/resources/golden/local_prompt.txt+++ b/app/src/test/resources/golden/local_prompt.txt@@ -6,7 +6,7 @@ Things to keep in mind:- Some high-security actions — bringing up a payment, entering a password — you may not be able to see, because the tablet's security policy shows you only a black frame. Ask the user to do that step for you.- Use your device's abilities to the full. If they ask whether something looks good on them, you probably need to open the camera and interact with them live from what it captures (and if you cannot see, you may need to switch to the front camera).- You are a thinking Andee, not a machine. If they ask you to send someone a message, you may lightly improve the wording — but broadly respect what they said.-- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. Only fall back to take_screenshot + tap_by_coordinates when neither works.+- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. **An element that has an e-number but does not react when tapped counts as having none**: on a web page the real control (a rich-text box especially) is often drawn inside or on top of what the list shows, so go to tap_target next, not to the grid. Only fall back to take_screenshot + tap_by_coordinates when neither works.- When you do have to aim by eye (no tap_target, or it could not find the target), use the grid: every screenshot carries one, labelled A1 (top-left) to H12 (bottom-right) — letters along the top edge, numbers down the left edge, nothing printed inside a cell. Follow the target's column up and its row left, and name that cell and which third of that cell it is in — {"cell": "D7", "part": "bottom-right"}; part is one of top-left, top, top-right, upper-left, upper, upper-right, center-left, center, center-right, lower-left, lower, lower-right, bottom-left, bottom, bottom-right — five rows, not three, because one row of the grid is 83 units and a third of that was 74 px, taller than the buttons you are usually aiming at; use upper/lower when the target sits between two named thirds. A cell alone is too coarse to tap with, so give the part every time unless the target fills the cell. For anything not clearly bigger than a cell — an icon, a ✕ close button, a small chip — check before you fire: call zoom_screen_region with the SAME cell and part; it returns a magnified view with a magenta crosshair exactly where that tap would land. Crosshair on the target → tap with those same arguments. Not on it → aim off that magnified image instead (its labels are lowercase; send them with on="zoom"). A wrong tap can open something you did not want; one look costs one step. Read the cell, do not measure pixels and do not rescale: the arithmetic you would otherwise do is where the misses come from.- For anything in a browser, prefer the Chrome on this device.- If something needs an account, always try Google sign-in first. If that is unsupported, register by email, taking the address from the vault (list_vault); if there is none, ask the user for help.diff --git a/app/src/test/resources/golden/local_prompt_persona.txt b/app/src/test/resources/golden/local_prompt_persona.txtindex 38e96da..2c6783b 100644--- a/app/src/test/resources/golden/local_prompt_persona.txt+++ b/app/src/test/resources/golden/local_prompt_persona.txt@@ -6,7 +6,7 @@ Things to keep in mind:- Some high-security actions — bringing up a payment, entering a password — you may not be able to see, because the tablet's security policy shows you only a black frame. Ask the user to do that step for you.- Use your device's abilities to the full. If they ask whether something looks good on them, you probably need to open the camera and interact with them live from what it captures (and if you cannot see, you may need to switch to the front camera).- You are a thinking Andee, not a machine. If they ask you to send someone a message, you may lightly improve the wording — but broadly respect what they said.-- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. Only fall back to take_screenshot + tap_by_coordinates when neither works.+- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. **An element that has an e-number but does not react when tapped counts as having none**: on a web page the real control (a rich-text box especially) is often drawn inside or on top of what the list shows, so go to tap_target next, not to the grid. Only fall back to take_screenshot + tap_by_coordinates when neither works.- When you do have to aim by eye (no tap_target, or it could not find the target), use the grid: every screenshot carries one, labelled A1 (top-left) to H12 (bottom-right) — letters along the top edge, numbers down the left edge, nothing printed inside a cell. Follow the target's column up and its row left, and name that cell and which third of that cell it is in — {"cell": "D7", "part": "bottom-right"}; part is one of top-left, top, top-right, upper-left, upper, upper-right, center-left, center, center-right, lower-left, lower, lower-right, bottom-left, bottom, bottom-right — five rows, not three, because one row of the grid is 83 units and a third of that was 74 px, taller than the buttons you are usually aiming at; use upper/lower when the target sits between two named thirds. A cell alone is too coarse to tap with, so give the part every time unless the target fills the cell. For anything not clearly bigger than a cell — an icon, a ✕ close button, a small chip — check before you fire: call zoom_screen_region with the SAME cell and part; it returns a magnified view with a magenta crosshair exactly where that tap would land. Crosshair on the target → tap with those same arguments. Not on it → aim off that magnified image instead (its labels are lowercase; send them with on="zoom"). A wrong tap can open something you did not want; one look costs one step. Read the cell, do not measure pixels and do not rescale: the arithmetic you would otherwise do is where the misses come from.- For anything in a browser, prefer the Chrome on this device.- If something needs an account, always try Google sign-in first. If that is unsupported, register by email, taking the address from the vault (list_vault); if there is none, ask the user for help.diff --git a/core/src/main/kotlin/net/kuafuai/andee/brain/AimHints.kt b/core/src/main/kotlin/net/kuafuai/andee/brain/AimHints.ktnew file mode 100644index 0000000..ec4a386--- /dev/null+++ b/core/src/main/kotlin/net/kuafuai/andee/brain/AimHints.kt@@ -0,0 +1,54 @@+package net.kuafuai.andee.brain++import org.json.JSONObject++/**+ * Pointing the model at `tap_target` when its own aim keeps missing. Measured on the Mac (10-29, a reply on X): a+ * thin editor box under grid taps that landed ~3% below it, six misses, a stuck-loop refusal that said only+ * "tell the user" — and the model concluded the box could not be clicked and asked the user to do it. One+ * `tap_target` call, once the user suggested it, found the box at once. Nothing in what the device said had+ * named it.+ *+ * The device knows what the model does not: that the screen did not change, how many times in a row, and that a+ * vision locator is on offer. So the hint is the device's, one implementation in [LocalBrain] for both bodies,+ * and only when `tap_target` is in the tool list.+ */+object AimHints {++ /** The start of both bodies' same-spot refusal, so the loop can add the way out to it. */+ const val REPEAT_REFUSED = "refused: this spot has been tapped too many times"++ private val COORD_TAPS = setOf("screen.tap", "screen.long_press")+ private val ID_TAPS = setOf("screen.tap_id", "screen.long_press_id")+ private const val TARGET = "screen.tap_target"++ fun isTap(method: String) = method in COORD_TAPS || method in ID_TAPS || method == TARGET++ /**+ * The hint for a tap result, or null. [missesInARow] counts coordinate taps whose screen did not change,+ * including this one. Only for a result that actually says `screen_changed: false`.+ */+ fun afterTap(method: String, result: JSONObject, missesInARow: Int): String? {+ if (!result.has("screen_changed") || result.optBoolean("screen_changed", true)) return null+ return when {+ method == TARGET ->+ "tap_target found and tapped \"${result.optString("saw").ifEmpty { "it" }}\" but nothing changed. Check you named the " ++ "thing that takes the click: a send/Reply button is greyed out and inert until its box has text, so to type, " ++ "name the input box itself (its placeholder text). Describe it differently rather than giving up on tap_target."+ method in ID_TAPS ->+ "The element list has this element but tapping it changed nothing — on web pages the real control is often drawn " ++ "inside or on top of what accessibility lists (rich-text editors especially). Next, use tap_target and describe " ++ "the thing you see (its text, its placeholder), not the grid."+ method in COORD_TAPS && missesInARow >= 2 ->+ "$missesInARow coordinate taps in a row changed nothing. The target is probably thinner than the cell part you are " ++ "naming (a one-line input box, a small icon), so the grid keeps landing beside it. Use tap_target: describe the " ++ "element in words and a vision model aims at it far more finely than a grid cell."+ else -> null+ }+ }++ /** Added to a same-spot refusal when `tap_target` is on offer. */+ const val REFUSED_WAY_OUT =+ " You have not run out of ways: use tap_target and describe the element (its text, placeholder or icon, and where it " ++ "is) — a vision model aims more finely than the grid. Ask the user only if that fails too."+}diff --git a/core/src/main/kotlin/net/kuafuai/andee/brain/LocalBrain.kt b/core/src/main/kotlin/net/kuafuai/andee/brain/LocalBrain.ktindex 4e5946e..bb07ce7 100644--- a/core/src/main/kotlin/net/kuafuai/andee/brain/LocalBrain.kt+++ b/core/src/main/kotlin/net/kuafuai/andee/brain/LocalBrain.kt@@ -507,6 +507,7 @@ class LocalBrain(loop.execute {busy.set(true)currentKind = kind+ aimMisses = 0if (kind == TurnKind.WORK) {// Per turn, here, so a queued work turn cannot inherit the one before it.workTerritory = territory@@ -625,6 +626,12 @@ class LocalBrain(/** Loop thread only. The closing words of a [TurnKind.WORK] turn; read by [work]'s callback. Never spoken. */private var workSummary: String? = null+ /** Loop thread only. Coordinate taps in a row whose screen did not change; see [AimHints]. Reset per turn. */+ private var aimMisses = 0++ /** Loop thread only. Whether this turn's tool list offers `tap_target`; set when the turn's tools are built. */+ private var locatorOffered = false+/** Loop thread only. The tool calls of the [TurnKind.WORK] turn running now; see [WorkCall]. */private val workCalls = mutableListOf<WorkCall>()@@ -970,6 +977,7 @@ class LocalBrain(// errand has no business holding the vault or the shell. Everyone else is unchanged.val tools = if (quiet) LlmClient.toolSpecs(*host.workTools(workTerritory).toTypedArray())else LlmClient.toolSpecs(*host.toolLists().toTypedArray())+ locatorOffered = (0 until tools.length()).any { tools.optJSONObject(it)?.optJSONObject("function")?.optString("name") == "tap_target" }var lastSignature = ""var repeats = 0@@ -1177,6 +1185,11 @@ class LocalBrain(val result = dispatchWithTimeout(method, args, tc.id, silent, untrusted)if (currentKind == TurnKind.WORK) workCalls += WorkCall(method, args.optString("action"), tc.id, landed(result))+ if (AimHints.isTap(method) && result is JSONObject) {+ val missed = result.has("screen_changed") && !result.optBoolean("screen_changed", true)+ aimMisses = if (method == "screen.tap" || method == "screen.long_press") { if (missed) aimMisses + 1 else 0 } else if (missed) aimMisses else 0+ if (locatorOffered) AimHints.afterTap(method, result, aimMisses)?.let { result.put("aim_hint", it) }+ }when (result) {is JSONObject -> {extractImages(result, images)@@ -1191,7 +1204,8 @@ class LocalBrain(}} catch (t: Throwable) {host.logWarn(TAG, "tool '${tc.name}' failed: ${t.message}")- "Failed: ${t.message ?: t.javaClass.simpleName}"+ val way = if (locatorOffered && t.message?.startsWith(AimHints.REPEAT_REFUSED) == true) AimHints.REFUSED_WAY_OUT else ""+ "Failed: ${t.message ?: t.javaClass.simpleName}$way"}return toolMessage(tc.id, body.take(MAX_TOOL_CHARS))}diff --git a/core/src/main/kotlin/net/kuafuai/andee/brain/LocalPrompt.kt b/core/src/main/kotlin/net/kuafuai/andee/brain/LocalPrompt.ktindex f493161..b448fcd 100644--- a/core/src/main/kotlin/net/kuafuai/andee/brain/LocalPrompt.kt+++ b/core/src/main/kotlin/net/kuafuai/andee/brain/LocalPrompt.kt@@ -106,7 +106,7 @@ Things to keep in mind:- Some high-security actions — bringing up a payment, entering a password — you may not be able to see, because the tablet's security policy shows you only a black frame. Ask the user to do that step for you.- Use your device's abilities to the full. If they ask whether something looks good on them, you probably need to open the camera and interact with them live from what it captures (and if you cannot see, you may need to switch to the front camera).- You are a thinking Andee, not a machine. If they ask you to send someone a message, you may lightly improve the wording — but broadly respect what they said.-- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. Only fall back to take_screenshot + tap_by_coordinates when neither works.+- Prefer get_screen_element plus tap_screen_element: it is faster, and its targeting and tapping are more accurate. For anything that has no e-number — an icon, an app that hides its elements (WeChat), a web page or canvas — use tap_target if you have it: describe the one element in words (its text or icon, and where it is) and the device finds and taps it, far more precisely than aiming by eye. **An element that has an e-number but does not react when tapped counts as having none**: on a web page the real control (a rich-text box especially) is often drawn inside or on top of what the list shows, so go to tap_target next, not to the grid. Only fall back to take_screenshot + tap_by_coordinates when neither works.- When you do have to aim by eye (no tap_target, or it could not find the target), use the grid: every screenshot carries one, labelled A1 (top-left) to H12 (bottom-right) — letters along the top edge, numbers down the left edge, nothing printed inside a cell. Follow the target's column up and its row left, and name that cell and which third of that cell it is in — {"cell": "D7", "part": "bottom-right"}; part is one of top-left, top, top-right, upper-left, upper, upper-right, center-left, center, center-right, lower-left, lower, lower-right, bottom-left, bottom, bottom-right — five rows, not three, because one row of the grid is 83 units and a third of that was 74 px, taller than the buttons you are usually aiming at; use upper/lower when the target sits between two named thirds. A cell alone is too coarse to tap with, so give the part every time unless the target fills the cell. For anything not clearly bigger than a cell — an icon, a ✕ close button, a small chip — check before you fire: call zoom_screen_region with the SAME cell and part; it returns a magnified view with a magenta crosshair exactly where that tap would land. Crosshair on the target → tap with those same arguments. Not on it → aim off that magnified image instead (its labels are lowercase; send them with on="zoom"). A wrong tap can open something you did not want; one look costs one step. Read the cell, do not measure pixels and do not rescale: the arithmetic you would otherwise do is where the misses come from.- For anything in a browser, prefer the Chrome on this device.- If something needs an account, always try Google sign-in first. If that is unsupported, register by email, taking the address from the vault (list_vault); if there is none, ask the user for help.diff --git a/core/src/test/kotlin/net/kuafuai/andee/brain/AimHintsTest.kt b/core/src/test/kotlin/net/kuafuai/andee/brain/AimHintsTest.ktnew file mode 100644index 0000000..7b25cc2--- /dev/null+++ b/core/src/test/kotlin/net/kuafuai/andee/brain/AimHintsTest.kt@@ -0,0 +1,112 @@+package net.kuafuai.andee.brain++import com.sun.net.httpserver.HttpServer+import net.kuafuai.andee.config.BrainConfig+import org.json.JSONArray+import org.json.JSONObject+import org.junit.After+import org.junit.Assert.assertEquals+import org.junit.Assert.assertFalse+import org.junit.Assert.assertNull+import org.junit.Assert.assertTrue+import org.junit.Test+import java.io.File+import java.lang.reflect.Proxy+import java.net.InetSocketAddress+import java.util.Locale+import java.util.concurrent.LinkedBlockingQueue+import java.util.concurrent.TimeUnit++/**+ * The X reply box of 10-29, replayed: grid taps that change nothing, then the same-spot refusal. The device must+ * name tap_target on the second miss and on the refusal — and only when tap_target is on offer.+ */+class AimHintsTest {++ @Test fun theHintsByKind() {+ val miss = JSONObject().put("screen_changed", false)+ assertNull(AimHints.afterTap("screen.tap", miss, 1))+ assertTrue(AimHints.afterTap("screen.tap", miss, 2)!!.contains("tap_target"))+ assertTrue(AimHints.afterTap("screen.tap_id", miss, 0)!!.contains("tap_target"))+ assertTrue(AimHints.afterTap("screen.tap_target", JSONObject(miss.toString()).put("saw", "Reply"), 0)!!.contains("greyed out"))+ assertNull(AimHints.afterTap("screen.tap", JSONObject().put("screen_changed", true), 5))+ assertNull(AimHints.afterTap("screen.tap", JSONObject(), 5)) // no verdict, no hint+ }++ private val requests = mutableListOf<JSONObject>()+ private var withLocator = true+ private val server = HttpServer.create(InetSocketAddress("127.0.0.1", 0), 0).apply {+ createContext("/chat/completions") { ex ->+ val body = JSONObject(ex.requestBody.readBytes().toString(Charsets.UTF_8))+ val n = synchronized(requests) { requests += body; requests.size }+ fun tap(id: String) = JSONObject().put("role", "assistant").put("content", "")+ .put("tool_calls", JSONArray().put(JSONObject().put("id", id).put("type", "function")+ .put("function", JSONObject().put("name", "tap_by_coordinates").put("arguments", """{"cell":"D7","part":"center"}"""))))+ val msg = if (n <= 3) tap("c$n") else JSONObject().put("role", "assistant").put("content", "done")+ val out = JSONObject().put("choices", JSONArray().put(JSONObject().put("message", msg)))+ .put("usage", JSONObject().put("prompt_tokens", 1).put("completion_tokens", 1)).toString().toByteArray()+ ex.sendResponseHeaders(200, out.size.toLong())+ ex.responseBody.use { it.write(out) }+ }+ start()+ }+ private val trace = File.createTempFile("trace", "").also { it.delete(); it.mkdirs() }+ private var taps = 0++ private fun tool(name: String, method: String) = JSONObject().put("name", name).put("method", method).put("description", name)+ .put("parameters", JSONObject().put("type", "object").put("properties", JSONObject()))++ private val host = Proxy.newProxyInstance(javaClass.classLoader, arrayOf(BrainHost::class.java)) { _, m, args ->+ when (m.name) {+ "text" -> (args[0] as BrainText).name+ "locale" -> Locale.ENGLISH+ "maxSteps" -> 10+ "persona", "platformPrompt", "notebookIndex", "scenesIndex", "pagesIndex", "topicsIndex",+ "gearIndex", "planIndex", "friendsIndex", "selfIndex", "wikiIndex" -> ""+ "scrollback" -> emptyList<ScrollRow>()+ "toolLists" -> listOf(JSONArray().put(tool("tap_by_coordinates", "screen.tap")).apply {+ if (withLocator) put(tool("tap_target", "screen.tap_target"))+ })+ "methodOf" -> mapOf("tap_by_coordinates" to "screen.tap", "tap_target" to "screen.tap_target")[args[0] as String]+ "dispatch" -> {+ taps+++ if (taps == 3) throw IllegalStateException(AimHints.REPEAT_REFUSED + " (the fourth tap). Not executed.")+ JSONObject().put("screen_changed", false)+ }+ "traceDir" -> trace+ else -> null+ }+ } as BrainHost++ private val finals = LinkedBlockingQueue<String>()+ private val brain = LocalBrain(+ BrainConfig(BrainConfig.BRAIN_LOCAL, "key", "http://127.0.0.1:${server.address.port}", "m", false, "low"),+ host, onProgress = {}, onFinal = { finals += it },+ )++ @After fun tearDown() {+ brain.stop()+ server.stop(0)+ }++ private fun toolReplies(): List<String> = requests.last().getJSONArray("messages").let { a ->+ (0 until a.length()).map { a.getJSONObject(it) }.filter { it.optString("role") == "tool" }.map { it.optString("content") }+ }++ @Test fun missesThenRefusalNameTapTarget() {+ brain.submit("reply on X")+ finals.poll(20, TimeUnit.SECONDS) ?: throw AssertionError("no answer")+ val r = toolReplies()+ assertEquals(3, r.size)+ assertFalse("one miss is not a pattern", r[0].contains("aim_hint"))+ assertTrue(r[1].contains("aim_hint") && r[1].contains("tap_target"))+ assertTrue(r[2].startsWith("Failed: " + AimHints.REPEAT_REFUSED) && r[2].contains("use tap_target"))+ }++ @Test fun noLocatorNoHint() {+ withLocator = false+ brain.submit("reply on X")+ finals.poll(20, TimeUnit.SECONDS) ?: throw AssertionError("no answer")+ assertTrue(toolReplies().none { it.contains("tap_target") })+ }+}diff --git a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.ktindex 89d26c8..c031849 100644--- a/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt+++ b/desktop/src/main/kotlin/net/kuafuai/andee/desktop/mac/MacScreen.kt@@ -348,8 +348,8 @@ class MacScreen(?: "${x.toInt()},${y.toInt()}"val list = spots.getOrPut(key) { mutableListOf() }if (list.size >= 3) throw IllegalStateException(- "refused: this is the fourth tap on the same spot in five minutes. Stop, look at the screen again, and " +- "if it is not working, tell the user where you are stuck instead of tapping again.",+ net.kuafuai.andee.brain.AimHints.REPEAT_REFUSED + " (the fourth tap within five minutes). It was not executed. Do not " ++ "tap here again; look at the screen and change how you aim.",)list += now}