Skip to content

armgit / andee

Public Imported from GitHub
0

Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。

feat(wake): keep listening with the screen off, and stop the tools at the lock screen

booboosuia9d6ebc07c8e
diff --git a/app/src/main/java/net/kuafuai/andee/BodyCoreService.kt b/app/src/main/java/net/kuafuai/andee/BodyCoreService.kt
index e6855c9..eaea2ae 100644
--- a/app/src/main/java/net/kuafuai/andee/BodyCoreService.kt
+++ b/app/src/main/java/net/kuafuai/andee/BodyCoreService.kt
@@ -1284,8 +1284,9 @@ class BodyCoreService : Service() {
* only while every one of these is true, and lets go within ~100 ms of any
* of them turning false.
*
- * - **Screen on.** The user's choice, and it keeps a dark tablet on a
- * bedside table from listening all night.
+ * - **Listening is allowed.** Screen on, or the user left 锁屏时也听 on —
+ * see [wakeListeningAllowed]. This term used to be the screen alone, and
+ * that is the whole reason a phone in a pocket answered nothing.
* - **ASR idle.** The turn's own recorder must never lose a race to the one
* listening for a wake word.
* - **TTS quiet.** Otherwise Andee can hear itself say something close to
@@ -1310,7 +1311,7 @@ class BodyCoreService : Service() {
net.kuafuai.andee.wake.WakeWord.init(
this,
gate = {
- screenOn && !asr.isActive() && !tts.isSpeaking() &&
+ wakeListeningAllowed() && !asr.isActive() && !tts.isSpeaking() &&
!meeting.isActive() && !net.kuafuai.andee.device.CallState.isActive &&
!net.kuafuai.andee.audio.Earcon.isSounding()
},
@@ -1322,6 +1323,24 @@ class BodyCoreService : Service() {
@Volatile
private var screenOn = true
+ /**
+ * Whether the wake listener may hold the microphone right now.
+ *
+ * Screen on is the old rule, and it is still the cheap one: the user is
+ * looking at the device, so listening costs nothing extra and cannot be a
+ * surprise. Screen off is the user's own call — see
+ * [net.kuafuai.andee.config.VoiceConfig.wakeWhileLocked], on unless they
+ * turn it off.
+ *
+ * The reason the second half exists at all: the screen being off is the
+ * normal state of a phone in a pocket, and an assistant that can only be
+ * called while you are already holding it is not one you can call. The cost
+ * — a microphone held continuously and the platform's recording indicator
+ * with it — is stated on the settings row rather than hidden here.
+ */
+ private fun wakeListeningAllowed(): Boolean =
+ screenOn || net.kuafuai.andee.config.VoiceConfig.wakeWhileLocked(this)
+
private val screenReceiver = object : android.content.BroadcastReceiver() {
override fun onReceive(c: android.content.Context?, i: android.content.Intent?) {
screenOn = i?.action != android.content.Intent.ACTION_SCREEN_OFF
diff --git a/app/src/main/java/net/kuafuai/andee/config/VoiceConfig.kt b/app/src/main/java/net/kuafuai/andee/config/VoiceConfig.kt
index 8db317b..bc5723d 100644
--- a/app/src/main/java/net/kuafuai/andee/config/VoiceConfig.kt
+++ b/app/src/main/java/net/kuafuai/andee/config/VoiceConfig.kt
@@ -603,6 +603,39 @@ data class VoiceConfig(
context.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
.getString("ball", "").orEmpty().trim()
+ /**
+ * Whether the wake phrase is still listened for with the screen off.
+ *
+ * **On unless the user turns it off.** That default is the whole point
+ * of the switch: the screen being off is the normal state of a phone in
+ * a pocket, so the older rule — hold the microphone only while the
+ * screen was on — meant the assistant could not be called from exactly
+ * the place a person keeps their phone. The gate in
+ * [net.kuafuai.andee.wake.WakeWord] used to spell that rule out
+ * unconditionally; this is the user's override of it.
+ *
+ * What it costs, and why the settings row that flips it says so out
+ * loud: with the screen off the microphone is held continuously, so the
+ * platform's recording indicator stays lit and the segmenter runs for as
+ * long as the device is awake. Nothing recoverable as speech leaves the
+ * phone either way — see [net.kuafuai.andee.wake.WakeWord] — but
+ * "listening all the time" is a claim the user is entitled to see stated
+ * before they agree to it rather than discover on a status bar.
+ *
+ * Read from the wake supervisor's gate, which reconsiders ten times a
+ * second. Cheap (SharedPreferences answers from an already-loaded map)
+ * and deliberately not cached anywhere: the settings card writes it and
+ * the very next poll has to see it.
+ */
+ fun wakeWhileLocked(context: Context): Boolean =
+ context.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
+ .getBoolean("wake_locked", true)
+
+ fun setWakeWhileLocked(context: Context, on: Boolean) {
+ context.getSharedPreferences(PREFS, Context.MODE_PRIVATE)
+ .edit().putBoolean("wake_locked", on).apply()
+ }
+
/**
* The scene the assistant is in, by [Notebook.Scene.name], or null.
*
diff --git a/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt b/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
index 8496582..4c8dfb6 100644
--- a/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
+++ b/app/src/main/java/net/kuafuai/andee/net/CommandDispatcher.kt
@@ -204,6 +204,24 @@ class CommandDispatcher(
private fun requireScreen(): ScreenController =
screenOrNull() ?: throw IllegalStateException(SCREEN_UNAVAILABLE)
+ /**
+ * Whether the keyguard is up right now.
+ *
+ * A call rather than a field because the answer changes without anything
+ * here being told about it: the user can lock the phone between two tool
+ * calls of the same turn, and the guard in [dispatch] has to be reading the
+ * device, not a memory of it.
+ *
+ * Defaults to *not* locked when the service cannot be reached. A `false`
+ * here only means the screen gets the benefit of the doubt — the alternative
+ * would be an unrelated platform hiccup taking every device-operating tool
+ * away, which is a worse failure than the one this guard exists to stop.
+ */
+ private fun keyguardLocked(): Boolean = runCatching {
+ (appContext.getSystemService(android.content.Context.KEYGUARD_SERVICE)
+ as android.app.KeyguardManager).isKeyguardLocked
+ }.getOrDefault(false)
+
/** Written on the main thread, read by worker threads: card already folded. */
@Volatile
private var compacted = false
@@ -276,6 +294,36 @@ class CommandDispatcher(
"user next time they talk to you.",
)
}
+ // The lock screen is a wall around the screen, and unlike every other
+ // wall in this file it is one that fails *silently*. A gesture aimed at
+ // a locked device is dropped by the input pipeline — `dispatchGesture`
+ // reports nothing, and the lock screen's own `FLAG_SECURE` means there
+ // were never any logs to search. `screen.ui_tree` returns the empty
+ // placeholder the keyguard presents as, which is indistinguishable from
+ // the blank-tree case documented in the README. A launch lands behind
+ // the lock screen where nobody sees it. Every one of those would come
+ // back to the brain as a "completed" it then repeats to the user as
+ // something that happened — the single outcome this class exists to
+ // prevent, spelled out at [requireScreen].
+ //
+ // Read live rather than mirrored into a flag someone has to keep in
+ // step. The caller that matters here is the wake word, and it can fire
+ // at any moment of a turn: the user locking the phone mid-answer is a
+ // normal thing to do, not a state transition to be tracked. One binder
+ // call per tool is nothing beside what the tool itself is about to cost.
+ //
+ // `screen.` by prefix so a new tool in that namespace inherits the
+ // guard without a second edit here, then the handful of names outside it
+ // that are the same act by another road: launching something, opening a
+ // URL, pointing a camera, or drawing a surface the lock screen would
+ // cover anyway.
+ if (keyguardLocked() && (method.startsWith("screen.") || method in LOCKED_BLOCKED)) {
+ throw IllegalStateException(
+ "The phone is locked, so nothing behind the lock screen can be read or " +
+ "touched: $method. Say so, in the user's language and without inventing " +
+ "a result, and do it in the next turn once they have unlocked it.",
+ )
+ }
// Every command below reaches into the real device, and every one of
// them is worth watching, so get out of the way *before* doing it.
// Prefix rather than an explicit list, so adding a tool stays the
@@ -1720,6 +1768,37 @@ class CommandDispatcher(
/** How long to let the main thread take to apply the fold. */
const val COMPACT_SETTLE_MS = 200L
+ /**
+ * What a locked phone refuses, beyond the whole `screen.` namespace —
+ * see the guard at the top of [dispatch], which tests this with a
+ * prefix for `screen.` and by name for these.
+ *
+ * Each one is the same act by another road. The first group reaches the
+ * screen without going near [ScreenController]: `device.launch` and
+ * `device.open_url` start an Activity that would open behind the
+ * keyguard, `device.scan` and `device.look` want a camera the platform
+ * will not hand to a locked device, and `device.permissions.request`
+ * needs an Activity to show a dialog in. The second group —
+ * `ui.ask`, `ui.alert`, `ui.show_html` — draws a surface. A card or a
+ * page under a lock screen is a claim the user cannot check, which is
+ * the same lie as a gesture that never landed.
+ *
+ * Not a prefix anywhere, and that is deliberate: `device.battery`,
+ * `device.volume`, `tts.speak` and the whole `note.` namespace all work
+ * perfectly well with the screen locked, and `tts.speak` is how the
+ * answer gets out at all. Only the names that would fail are here.
+ */
+ val LOCKED_BLOCKED = setOf(
+ "device.launch",
+ "device.open_url",
+ "device.scan",
+ "device.look",
+ "device.permissions.request",
+ "ui.ask",
+ "ui.alert",
+ "ui.show_html",
+ )
+
/**
* And how long to then let the *system* take to notice it — see
* [awaitUncovered]. Measured on this tablet at 4 polls / ~140 ms,
diff --git a/app/src/main/java/net/kuafuai/andee/ui/SettingsUi.kt b/app/src/main/java/net/kuafuai/andee/ui/SettingsUi.kt
index c1b73b7..91c1293 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/SettingsUi.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/SettingsUi.kt
@@ -813,6 +813,37 @@ class SettingsUi(
state.text = summary()
})
panel.addView(row, rowParams(dp(12)))
+
+ // The screen-off half of the wake word, as one button that carries its
+ // own state. Two buttons in a picker would read as a choice to make;
+ // this is a thing that is on or off, and tapping it says which.
+ //
+ // Written straight through rather than on 保存, like the language and
+ // the ball's look: the gate that reads it reconsiders ten times a
+ // second, so the new answer is live before the finger is off the
+ // button, and there is nothing left here for a 保存 to confirm.
+ val lockedBtn = button("", filled = false) {}
+ fun paintLocked() {
+ val on = VoiceConfig.wakeWhileLocked(context)
+ lockedBtn.text = lctx.getString(
+ if (on) R.string.settings_wake_locked_on else R.string.settings_wake_locked_off,
+ )
+ // Same pair [button] picks with, so the on state reads as the
+ // filled chip it would have been had it been born on.
+ lockedBtn.setTextColor(Color.parseColor(if (on) Glass.TITLE else Glass.LABEL))
+ lockedBtn.background =
+ if (on) Glass.tinted(context, dp(24)) else Glass.panel(context, dp(24))
+ }
+ lockedBtn.setOnClickListener {
+ VoiceConfig.setWakeWhileLocked(context, !VoiceConfig.wakeWhileLocked(context))
+ paintLocked()
+ }
+ paintLocked()
+ panel.addView(lockedBtn, rowParams(dp(12)))
+ // The cost, said out loud next to the switch rather than buried in a
+ // README: with the screen off this holds the microphone open, and the
+ // platform says so on the status bar whether we mention it or not.
+ note(panel, lctx.getString(R.string.settings_wake_locked_note))
}
/**
diff --git a/app/src/main/res/values-en/strings.xml b/app/src/main/res/values-en/strings.xml
index 03790d0..97050b1 100644
--- a/app/src/main/res/values-en/strings.xml
+++ b/app/src/main/res/values-en/strings.xml
@@ -95,9 +95,12 @@
<string name="common_low">Low</string>
<string name="settings_brain_local_note">The device runs the agent loop itself, with no hub. Without a key it will not take over.</string>
<string name="settings_section_wake">Wake word</string>
- <string name="settings_wake_enrolled">Enrolled — say your wake word while the screen is on</string>
+ <string name="settings_wake_enrolled">Enrolled — say your wake word and it wakes up</string>
<string name="settings_wake_not_enrolled">Not enrolled — pick a phrase and say it 3 times to wake me by voice, e.g. “Hey Andee”</string>
<string name="settings_wake_record">Record</string>
+ <string name="settings_wake_locked_on">Listen while locked: on</string>
+ <string name="settings_wake_locked_off">Listen while locked: off</string>
+ <string name="settings_wake_locked_note">Andee keeps listening with the screen off. That costs battery: the microphone stays open, and so does the system recording indicator.</string>
<string name="common_clear">Clear</string>
<string name="common_delete">Delete</string>
<string name="settings_section_vault">Vault</string>
@@ -230,7 +233,7 @@
<string name="wake_say_again">Say it again</string>
<string name="wake_received">Got it (%1$d ms). %2$d to go.</string>
<string name="wake_rerecord">Record again</string>
- <string name="wake_done">Done — just say your wake word while the screen is on. (threshold %1$s)</string>
+ <string name="wake_done">Done — just say your wake word. (threshold %1$s)</string>
<!-- device/LookActivity.kt -->
<string name="dev_look_focusing">Focusing…</string>
<string name="dev_look_seen">Got it</string>
@@ -483,7 +486,7 @@
<string name="check_detail_voice_key_missing">Not set. The source ships without a key — open a Volcengine Speech service of your own and put the key in ⚙ → Backend → On-device → “Volcengine API key”. Without it, hearing and speaking cannot connect, and what you get back is a connection timeout that looks like a network fault. Everything else works.</string>
<string name="check_detail_voice_host_ok">%1$s is reachable.</string>
<string name="check_detail_voice_host_bad">Could not reach %1$s: %2$s. Both recognition and synthesis will time out.</string>
- <string name="check_detail_wake_on">Taught, and it wakes while the screen is on.</string>
+ <string name="check_detail_wake_on">Taught, and it wakes when you say it.</string>
<string name="check_detail_wake_off">Not taught yet. Pick a phrase (e.g. “Hey Andee”) and say it 3 times to wake me by voice. Everything still works without it, you just tap the ball to talk.</string>
<string name="check_detail_battery_on">Exempt, so pending reminders are not delayed.</string>
<string name="check_detail_battery_off">Not exempt from battery optimisation. Tap “Open settings” — the system asks “Ignore battery optimisations?”; take the button that ignores them.\nThe vendor’s own “App launch / background running” controls are a separate system; changing them leaves this row red. With the screen off a pending reminder may arrive late, and the wake phrase stops listening.</string>
diff --git a/app/src/main/res/values/strings.xml b/app/src/main/res/values/strings.xml
index 54912b3..260956d 100644
--- a/app/src/main/res/values/strings.xml
+++ b/app/src/main/res/values/strings.xml
@@ -72,9 +72,12 @@
<string name="common_low">低</string>
<string name="settings_brain_local_note">设备自己跑 agent loop,不连 Hub。没有 Key 就不会接管。</string>
<string name="settings_section_wake">语音唤醒</string>
- <string name="settings_wake_enrolled">已录入 —— 亮屏时说出你录的唤醒词就行</string>
+ <string name="settings_wake_enrolled">已录入 —— 说出你录的唤醒词就行</string>
<string name="settings_wake_not_enrolled">未录入 —— 自己选个词录 3 遍就能语音唤醒,例如「嘿 Andee」</string>
<string name="settings_wake_record">录入</string>
+ <string name="settings_wake_locked_on">锁屏时也听:已开启</string>
+ <string name="settings_wake_locked_off">锁屏时也听:已关闭</string>
+ <string name="settings_wake_locked_note">屏幕关闭时也会听。更耗电,麦克风会一直开着,状态栏的录音提示也会一直显示。</string>
<string name="common_clear">清除</string>
<string name="common_delete">删除</string>
<string name="settings_section_vault">保险箱</string>
@@ -213,7 +216,7 @@
<string name="wake_say_again">再说一次</string>
<string name="wake_received">收到(%1$d 毫秒)。还差 %2$d 次。</string>
<string name="wake_rerecord">重录</string>
- <string name="wake_done">好了,亮屏的时候说出你录的唤醒词就行。(阈值 %1$s)</string>
+ <string name="wake_done">好了,说出你录的唤醒词就行。(阈值 %1$s)</string>
<!-- device/LookActivity.kt -->
<string name="dev_look_focusing">正在对焦…</string>
<string name="dev_look_seen">看到了</string>
@@ -565,7 +568,7 @@
<string name="check_detail_voice_key_missing">还没填。源码里不带密钥,得你自己去火山引擎开一个语音服务,把 Key 填进 ⚙ →「后端」→「本机」→「火山 API Key」。没有它,听和说都连不上 —— 报出来的却是连接超时,看着像网络坏了。其余功能照常。</string>
<string name="check_detail_voice_host_ok">%1$s 可达。</string>
<string name="check_detail_voice_host_bad">连不上 %1$s:%2$s。识别和合成都会超时。</string>
- <string name="check_detail_wake_on">已录入,亮屏时喊得醒。</string>
+ <string name="check_detail_wake_on">已录入,喊一声就醒。</string>
<string name="check_detail_wake_off">还没录。自己选个词(例如「嘿 Andee」)录 3 遍就能喊醒。不录也能用,只是得点球。</string>
<string name="check_detail_battery_on">已在白名单,待办闹钟不会被推迟。</string>
<string name="check_detail_battery_off">不在电池优化白名单里。点右边「去开启」,系统会问「是否忽略电池优化?」,选「忽略」就好。\n厂商自己的「应用启动管理 / 后台运行」是另一套,在那里怎么改这项都是红的。息屏后待办提醒会晚到,唤醒词也会停。</string>