Skip to content

armgit / andee

Public Imported from GitHub
0

Andee lives on your phone or Mac and uses any app like you do—no integrations, no root, any model; memory and passwords never leave your device | Andee 住进你的手机或 Mac,像你一样操作任何 App——无需集成、无需 Root、支持任意模型,记忆和密码绝不离开设备。

feat(ball): let the model set its own face, and change it mid-sentence

booboosui17a017cd2001
diff --git a/CLAUDE.md b/CLAUDE.md
index a69acdc..c057843 100644
--- a/CLAUDE.md
+++ b/CLAUDE.md
@@ -145,6 +145,20 @@ Screenshots return base64 PNG over the wire (downscaled to 1280 long side — se
Non-obvious: `AsrController.stopImpl` deliberately blocks on `waitFinished(3000)` because 火山 sends the final transcript ~500 ms *after* the client's `isLast` — closing eagerly loses it.
+### The ball wears the face the model picked (`brain/SpeechMood.kt`)
+
+`Mood`'s own KDoc has called CALM..CONCERNED *"emotions the model picks from"* since it was written, and nothing ever picked one — `HAPPY`, `CURIOUS`, `TENSE` and `ANXIOUS` were unreachable, and the ball wore CALM through every answer it ever gave. The channel is **inline markers in the model's own text**: `[happy] 找到了,在第三页。[calm] 要我打开吗?`. It is text rather than a tool call or a JSON field because the one thing a model reliably does is write the next token, and a face is worth nothing if it arrives a round trip after the sentence it belonged to. `LocalPrompt` §10 states the rule; `SpeechMood.parse` turns a raw reply into a clean string plus `Cue(at, mood)` offsets *into that clean string*.
+
+Five things are load-bearing:
+
+- **Stripping is conservative, and it happens before the text is logged or spoken.** A marker that reaches `ChatHistory` is visible junk in the user's scrollback; one that reaches 火山 is read out loud as the word "happy". `onBrainFinal` is the chokepoint, with `onBrainProgress`, `speakCardText` and `tellAboutNotification` as the other three sites. Three shapes must survive: `[12条]` out of a notification summary, `[END]` out of our own prompt, and the label half of every markdown link — excluded by letters-only, by SHOUTING, and by a following `(` respectively.
+- **The vocabulary is wide, and that is a correction.** It shipped as the six `Mood` names matched exactly, and the first user to ask for a face it did not have broke it: told 「做一个生气的表情」 the model wrote `[angry]`, which matched nothing, so it was printed in the scrollback *and* read aloud *and* changed no face — all three failures at once, from one missing word. Every ordinary name for a feeling now maps to the nearest of the six, in English and Chinese (`[生气]`, `【开心】` — a model answering in Chinese reaches for those long before it reaches for `tense`, whatever the prompt lists). A lowercase bracketed word we still don't know is **dropped silently and changes no face**: the model meant it as a face, never as something to say, and holding the previous face is what a single-marker reply does anyway. Don't narrow this back to an enum — the model is choosing a feeling, not calling an API.
+- **The emotion layers over `SPEAKING`; it does not replace it.** `EmotionState.tick` used to gate the mouth envelope on `mood == Mood.SPEAKING`, so substituting HAPPY into `mood` would have given the user a grinning ball with a shut mouth halfway through its own sentence. The fix is a separate `talking: Boolean` param and a separate `EmotionBallRenderer.emotion` field — the face and the mouth became two questions the moment the model could answer one of them. `FloatingWindowUi.setState` clears the emotion on any non-SPEAKING state, so the channel can never leak into the next conversation.
+- **`TTSSentenceStart` is useless for sync, and finding that out cost a build.** The obvious way to line a cue up with the audio is 火山's own sentence event, which would pin each sentence in both clocks at once. Measured on device: the server sends it **once per `push()`, not once per sentence, and with an empty `text` field**. There is nothing to locate and nothing to locate it with. Pushing each sentence as its own call would get one event each, but then the events race the audio (the server may announce all of them before much sound has arrived) and serialising the pushes risks an audible gap mid-reply. Don't rebuild this on the event.
+- **So the mapping is arithmetic, against two different clocks.** `AudioIO.queuedMs()` is the producer's clock (bytes handed to `feedPcm`) and `AudioIO.playedMs()` is the track's own `playbackHeadPosition`; 火山 streams far faster than real time, so the first runs seconds ahead of the second and **that gap is the point**. `cueTotalMs` is captured at `onDone` (the one moment the producer's clock is final) and the ticker walks `cueText.length * playedMs / cueTotalMs`. Constant characters-per-ms is false in detail — a comma is a pause, a digit is three syllables — but the question is "which of two faces", where a few hundred ms of error is invisible. The ticker gates on `AudioIO.isStreamPlaying()` and deliberately **not** on `client != null`: the socket is nulled at `onDone`, which is near the *start* of what the user hears, so gating on it stopped the walk before any mid-utterance face was due.
+
+Fallbacks all land on the pre-existing behaviour rather than on nothing: no markers at all is the identity, one marker holds for the whole reply, and a `cueTotalMs` that never arrives (stall, error) holds the opening face. `SpeechMoodTest` exists because every way the offset arithmetic can be wrong is invisible on the device — a swallowed space is only audible, and a surviving marker is read aloud as a word nobody notices was not in the answer.
+
### Meeting recorder
`MeetingRecorder` holds the microphone for a whole meeting and hands back a **verbatim transcript**. The device has no model, so it does not summarise: the brain turns the transcript into minutes and puts them on screen with the existing `show_html` — that split is why the class has no notion of a "summary" anywhere in it.
diff --git a/app/src/main/java/net/kuafuai/andee/ScreenBodyService.kt b/app/src/main/java/net/kuafuai/andee/ScreenBodyService.kt
index 5f08ceb..3d0d5e6 100644
--- a/app/src/main/java/net/kuafuai/andee/ScreenBodyService.kt
+++ b/app/src/main/java/net/kuafuai/andee/ScreenBodyService.kt
@@ -7,6 +7,7 @@ import android.os.Looper
import android.view.accessibility.AccessibilityEvent
import net.kuafuai.andee.asr.AsrController
import net.kuafuai.andee.audio.AudioIO
+import net.kuafuai.andee.brain.SpeechMood
import net.kuafuai.andee.i18n.AppLocale
import net.kuafuai.andee.net.BodyWsServer
import net.kuafuai.andee.net.CommandDispatcher
@@ -1811,9 +1812,9 @@ class ScreenBodyService : AccessibilityService() {
dispatcher.beginTask()
window.setState(FloatingWindowUi.State.THINKING)
window.setSubtitle(
- // Drafts of a settling reply may already carry the end marker;
- // it must not flash on the live line either.
- stripMarkdown(END_MARKER.replace(text, "")),
+ // Drafts of a settling reply may already carry the end marker
+ // or an emotion tag; neither must flash on the live line.
+ SpeechMood.strip(stripMarkdown(END_MARKER.replace(text, ""))),
FloatingWindowUi.SubtitleKind.PARTIAL,
)
}
@@ -1826,7 +1827,13 @@ class ScreenBodyService : AccessibilityService() {
// open. Written before the speak below because [followUpWindow]
// reads it seconds later, off the TTS drain — see [expectFollowUp].
expectFollowUp = !END_MARKER.containsMatchIn(text)
- val plain = stripMarkdown(END_MARKER.replace(text, ""))
+ // Markers come out here, with the markdown and before anything
+ // else touches the string. This is the one place the answer is
+ // turned from what the model wrote into what the user gets, and a
+ // tag that survives it reaches *both* the scrollback (visible
+ // junk) and 火山 (read out loud as the word "happy").
+ val script = SpeechMood.parse(stripMarkdown(END_MARKER.replace(text, "")))
+ val plain = script.text
// The finished answer is the end of the task as far as the
// device is concerned: nothing further will move the app.
dispatcher.endTask()
@@ -1865,7 +1872,9 @@ class ScreenBodyService : AccessibilityService() {
// same way it displaces the meeting recorder's. The
// scrollback row above is already the answer; the user
// reads it after they hang up.
- if (!meeting.isActive() && !net.kuafuai.andee.device.CallState.isActive) tts.speak(plain)
+ if (!meeting.isActive() && !net.kuafuai.andee.device.CallState.isActive) {
+ tts.speak(plain, script.cues)
+ }
}
}
@@ -1904,7 +1913,8 @@ class ScreenBodyService : AccessibilityService() {
// reason: a rehearsal line is not a question the assistant asked, so
// the mic must not come up hunting for an answer to it.
expectFollowUp = false
- tts.speak(stripMarkdown(text))
+ val script = SpeechMood.parse(stripMarkdown(text))
+ tts.speak(script.text, script.cues)
return true
}
@@ -2026,13 +2036,16 @@ class ScreenBodyService : AccessibilityService() {
* Main thread only.
*/
private fun tellAboutNotification(n: JSONObject, line: String) {
+ val script = SpeechMood.parse(line)
net.kuafuai.andee.ui.ChatHistory.addUser(
AppLocale.str(this, R.string.svc_notification_auto_line, n.optString("app"))
)
- net.kuafuai.andee.ui.ChatHistory.addAssistant(line)
- lastAnswer = line
+ net.kuafuai.andee.ui.ChatHistory.addAssistant(script.text)
+ lastAnswer = script.text
expectFollowUp = false
- if (!meeting.isActive() && !net.kuafuai.andee.device.CallState.isActive) tts.speak(line)
+ if (!meeting.isActive() && !net.kuafuai.andee.device.CallState.isActive) {
+ tts.speak(script.text, script.cues)
+ }
}
/**
diff --git a/app/src/main/java/net/kuafuai/andee/audio/AudioIO.kt b/app/src/main/java/net/kuafuai/andee/audio/AudioIO.kt
index 4a85e91..96bfc94 100644
--- a/app/src/main/java/net/kuafuai/andee/audio/AudioIO.kt
+++ b/app/src/main/java/net/kuafuai/andee/audio/AudioIO.kt
@@ -285,6 +285,20 @@ class AudioIO {
@Volatile
private var streamBytesWritten: Long = 0
+ /**
+ * Bytes handed to [feedPcm], as against [streamBytesWritten]'s bytes handed
+ * to the track.
+ *
+ * Two counters because there are two clocks and the gap between them is the
+ * whole reason this class has a queue: 火山 delivers an utterance far faster
+ * than it plays, so at any moment "audio we have been given" runs seconds
+ * ahead of "audio the user has heard". Anything lining text up against
+ * sound needs both — the producer's clock to say where a sentence *starts*,
+ * the track's to say when it is *reached*.
+ */
+ @Volatile
+ private var streamBytesQueued: Long = 0
+
/**
* PCM waiting to be written to [streamTrack], and the thread that writes it.
*
@@ -309,6 +323,17 @@ class AudioIO {
@Volatile
private var playing = false
+ /**
+ * Is an utterance still on the speaker?
+ *
+ * True from [startStreamPlayback] until any of the stop paths, all of which
+ * go through [stopStreamPlayback] — including the end of the drain in
+ * [finishStreamPlayback]. That makes it the honest answer to "is the user
+ * still hearing this", which the websocket's own state is not: the socket
+ * finishes long before the sound does.
+ */
+ fun isStreamPlaying(): Boolean = playing
+
/** Queued by [finishStreamPlayback] to mean "no more audio is coming". */
private val endOfStream = ByteArray(0)
@@ -340,6 +365,7 @@ class AudioIO {
.build()
streamPlaySampleRate = sampleRate
streamBytesWritten = 0
+ streamBytesQueued = 0
playQueue.clear()
t.play()
streamTrack = t
@@ -375,9 +401,33 @@ class AudioIO {
*/
fun feedPcm(pcm: ByteArray) {
if (!playing || pcm.isEmpty()) return
+ streamBytesQueued += pcm.size
playQueue.offer(pcm)
}
+ /**
+ * Milliseconds of the current utterance that have been *received*.
+ *
+ * The producer's clock — see [streamBytesQueued]. Sampled at the moment a
+ * sentence is announced, it is that sentence's offset into the utterance.
+ */
+ fun queuedMs(): Long =
+ if (!playing) 0L else streamBytesQueued / BYTES_PER_FRAME * 1000L / streamPlaySampleRate
+
+ /**
+ * Milliseconds of the current utterance the user has actually *heard*.
+ *
+ * Straight off the track's own playback head, so it is real time rather
+ * than an estimate, and it is the only honest answer to "where in the
+ * sentence are we right now".
+ */
+ fun playedMs(): Long {
+ val t = streamTrack ?: return 0L
+ val frames = runCatching { t.playbackHeadPosition.toLong() and 0xFFFFFFFFL }
+ .getOrDefault(0L)
+ return frames * 1000L / streamPlaySampleRate
+ }
+
/**
* Called when the producer is done. Waits for the backlog to reach the track
* and for the track to drain, then releases.
@@ -436,5 +486,8 @@ class AudioIO {
private companion object {
const val TAG = "AudioIO"
+
+ /** 16-bit mono, everywhere in this class. */
+ const val BYTES_PER_FRAME = 2
}
}
diff --git a/app/src/main/java/net/kuafuai/andee/brain/LocalPrompt.kt b/app/src/main/java/net/kuafuai/andee/brain/LocalPrompt.kt
index 72d0e1d..0970149 100644
--- a/app/src/main/java/net/kuafuai/andee/brain/LocalPrompt.kt
+++ b/app/src/main/java/net/kuafuai/andee/brain/LocalPrompt.kt
@@ -300,6 +300,14 @@ Your answer will be read aloud to the user. So: no Markdown headings, no bullet
Before you call a tool, say what you are about to do — one line such as 「我看一下屏幕」 or 「打开微信」, said in their language (§9). That sentence is how the user knows you are not stuck. One is enough; you do not have to narrate every step.
+**You have a face, and you set it as you talk.** The ball in front of the user is you. Write a feeling in square brackets inline, and from that point in the sentence onward the ball wears it — the device strips the tag before speaking, so nobody ever hears the word. Put one at the front of a reply that has a feeling in it, and put another wherever the feeling turns:
+
+`[curious]这是什么东西…… [happy]找到了,在第三页。`
+
+There are six faces: `[calm]`, `[happy]`, `[curious]`, `[tense]`, `[anxious]`, `[concerned]`. Ordinary words for a feeling also work and land on the nearest of the six, in English or in Chinese — `[angry]` and `[生气]` both get the tense face, `[sad]` and `[难过]` both get the concerned one — so write what you mean rather than hunting for the exact label.
+
+Rules. Use them when you mean them: a face on every sentence is a twitch, and a cheerful face on bad news is worse than no face at all. Leave them off entirely and the ball just stays calm, which is the right answer for most replies. `[tense]` and `[anxious]` are for something going wrong, not for working hard. `[concerned]` is for bad news you are delivering about them, not about you. Keep the tag to one lowercase word — a bracket holding anything else is left alone and read out loud as written.
+
**`[END]` is how you close a conversation.** After your reply is read aloud, the device opens its microphone for a follow-up — that is how the user keeps talking without touching the ball. But a mic left open after a settled matter overhears the room: people chat near a tablet that has just finished its job, and answering words that were not addressed to you is butting in, not helpfulness. So when this reply settles the matter and asks the user nothing — a task finished, a fact delivered, thanks or a goodbye answered — end the reply with `[END]` on the last line. The device strips the marker before speaking; nobody ever hears it. Leave it off only when you are waiting on them: you asked a question, offered choices, or said something that clearly invites their answer.
""".trimIndent()
diff --git a/app/src/main/java/net/kuafuai/andee/brain/SpeechMood.kt b/app/src/main/java/net/kuafuai/andee/brain/SpeechMood.kt
new file mode 100644
index 0000000..77442cb
--- /dev/null
+++ b/app/src/main/java/net/kuafuai/andee/brain/SpeechMood.kt
@@ -0,0 +1,204 @@
+package net.kuafuai.andee.brain
+
+import net.kuafuai.andee.ui.ball.Mood
+
+/**
+ * Emotion markers the model writes into its own reply.
+ *
+ * `[happy] 找到了,在第三页。[calm] 要我打开吗?`
+ *
+ * The ball has had six "emotions the model picks from" since it was written —
+ * see [Mood]'s own KDoc — and nothing has ever picked one: four of the six were
+ * unreachable, and the ball wore CALM through every answer it ever gave. This
+ * is the channel that was missing. It is deliberately *in the text* rather than
+ * a tool call or a JSON field, because the one thing the model reliably does is
+ * write the next token, and a face is worth nothing if it arrives a round trip
+ * after the sentence it belonged to.
+ *
+ * Three rules hold this together:
+ *
+ * - **The vocabulary is wide, and it is a vocabulary rather than an enum.**
+ * It started as the six [Mood] names spelled exactly, and that failed the
+ * first time a user asked for a face it did not have: told to look angry,
+ * the model wrote `[angry]`, which matched nothing and was therefore shown
+ * on screen and read aloud as a word. The model is picking a *feeling*, so
+ * every ordinary name for one maps to the nearest of the six — in English
+ * and in Chinese, because this device mostly answers in Chinese.
+ * - **A bracket is only eaten when it is clearly not words.** `[12条]` out of
+ * a notification summary, `[END]` out of our own prompt, and the label half
+ * of every markdown link all have to survive. Letters-only excludes the
+ * first, SHOUTING excludes the second and a following `(` excludes the
+ * third; what is left is a lowercase word the model wrote in brackets,
+ * which is a face attempt whether or not we know the word. An unrecognised
+ * one is dropped silently and changes no face — the previous face holds,
+ * which is what a reply with one marker does anyway.
+ * - **Parsing happens before the text is logged or spoken, never after.** A
+ * marker that reaches [net.kuafuai.andee.ui.ChatHistory] is visible junk in
+ * the user's scrollback, and one that reaches 火山 gets read out loud as
+ * "happy". Both failures are silent in the code and obvious on the device.
+ */
+object SpeechMood {
+
+ /**
+ * What the model asked for, and where in the cleaned text it asked for it.
+ *
+ * [at] is an index into [Script.text] — the string *after* the markers are
+ * gone — because that is the only string anything downstream ever sees.
+ */
+ data class Cue(val at: Int, val mood: Mood)
+
+ /** A reply with its markers removed and recorded. */
+ data class Script(val text: String, val cues: List<Cue>) {
+ /** The face to wear from the first word. Null when the reply opened without one. */
+ val opening: Mood? get() = cues.firstOrNull()?.takeIf { it.at == 0 }?.mood
+ }
+
+ /**
+ * The six faces, by every name a model actually reaches for.
+ *
+ * It started as six words and that was the bug: asked to look angry, the
+ * model wrote `[angry]` — a word with no face behind it — so the tag was
+ * left alone, printed in the scrollback and read out loud as "angry",
+ * which is all three of the failures this class exists to prevent, at
+ * once. The vocabulary is wide because the model is choosing a *feeling*,
+ * not calling an API, and the nearest of six faces is a far better answer
+ * than the word itself in the user's ear.
+ *
+ * Chinese is in here for the same reason. This device answers in Chinese
+ * most of the time, and a model writing Chinese reaches for 「生气」 well
+ * before it reaches for `tense` no matter what the prompt lists.
+ *
+ * The voice-pipeline and work moods (LISTENING, SEARCHING, …) are
+ * deliberately absent: those describe what the *device* is doing and are
+ * owned by the code that does it. A model claiming to be SEARCHING while
+ * it speaks is describing something that is not happening.
+ */
+ private val VOCABULARY: Map<String, Mood> = buildMap {
+ fun List<String>.all(m: Mood) = forEach { put(it, m) }
+
+ listOf("calm", "neutral", "normal", "relaxed", "平静", "冷静", "正常")
+ .all(Mood.CALM)
+ listOf(
+ "happy", "glad", "pleased", "excited", "cheerful", "proud", "delighted",
+ "开心", "高兴", "快乐", "兴奋", "得意",
+ ).all(Mood.HAPPY)
+ listOf(
+ "curious", "interested", "surprised", "puzzled", "confused", "wondering",
+ "好奇", "疑惑", "困惑", "惊讶", "惊奇",
+ ).all(Mood.CURIOUS)
+ listOf(
+ "tense", "angry", "mad", "annoyed", "frustrated", "irritated", "upset",
+ "生气", "愤怒", "恼火", "烦躁", "不满",
+ ).all(Mood.TENSE)
+ listOf(
+ "anxious", "nervous", "worried", "scared", "afraid", "uneasy", "panicked",
+ "焦虑", "紧张", "担心", "不安", "害怕",
+ ).all(Mood.ANXIOUS)
+ listOf(
+ "concerned", "sad", "sorry", "apologetic", "disappointed", "unhappy", "serious",
+ "难过", "抱歉", "遗憾", "失望", "担忧", "严肃",
+ ).all(Mood.CONCERNED)
+ }
+
+ /**
+ * Anything bracket-shaped that *might* be a marker. Deciding whether it
+ * really is one is [parse]'s job — this only has to be cheap and to
+ * exclude the shapes that obviously are not.
+ *
+ * Letters only, so `[12条]` from a notification summary can never match.
+ * Three bracket families: ASCII, fullwidth `[]`, and 【】, which a model
+ * writing Chinese types about as readily as either.
+ */
+ private val MARKER = Regex("[\\[\\uFF3B\\u3010]\\s*([A-Za-z\\u4e00-\\u9fff]{1,12})\\s*[\\]\\uFF3D\\u3011]")
+
+ /**
+ * Is this unknown tag one the model meant as a face, or words it meant the
+ * user to hear?
+ *
+ * Leaving unknown tags alone was the original rule and it is what put
+ * "angry" in the user's ear. But deleting *every* unknown bracket is worse
+ * — `[END]` is a sentinel in our own prompt, and a notification badge
+ * reads `[SALE]`. The discriminator that actually separates them is case:
+ * a model writing a feeling types it in lowercase, and sentinels and
+ * badges are SHOUTED. Markdown link labels are excluded by the `(` that
+ * always follows them.
+ *
+ * A dropped tag changes no face. Holding the previous one is the same
+ * degradation as a reply with a single marker, which is already the
+ * common case.
+ */
+ private fun droppable(word: String, after: Char?): Boolean =
+ word.all { it in 'a'..'z' || it in 'A'..'Z' } &&
+ word != word.uppercase() &&
+ after != '('
+
+ /**
+ * Pull the markers out, keeping where each one fell.
+ *
+ * The fiddly part is the whitespace the model typed *around* the tag, and
+ * it has to be fiddly because the two languages want opposite things.
+ * `[happy] 找到了` must not keep a leading space (it is audible as a pause
+ * at the top of the reply), and `ok [happy] yes` must not lose the one
+ * between the words. So: whitespace on either side of a removed marker
+ * collapses to exactly one space, and to nothing at all when the marker
+ * opened the reply.
+ */
+ fun parse(raw: String): Script {
+ if (raw.isEmpty()) return Script(raw, emptyList())
+ val matches = MARKER.findAll(raw).toList()
+ if (matches.isEmpty()) return Script(raw, emptyList())
+
+ val out = StringBuilder(raw.length)
+ val cues = ArrayList<Cue>(matches.size)
+ var cursor = 0
+ for (m in matches) {
+ val word = m.groupValues[1]
+ val mood = VOCABULARY[word.lowercase()]
+ // Not a face. Either drop it silently or leave it to be spoken —
+ // and "leave it" means leaving the brackets too, so this branch
+ // must not touch [cursor] or the whitespace around the match.
+ if (mood == null &&
+ !droppable(word, raw.getOrNull(m.range.last + 1))
+ ) continue
+
+ out.append(raw, cursor, m.range.first)
+ cursor = m.range.last + 1
+
+ val spaceBefore = out.isNotEmpty() && (out.last() == ' ' || out.last() == '\t')
+ var spaceAfter = false
+ // Only spaces and tabs: a newline is the model's own paragraph
+ // break and belongs to the text, not to the marker.
+ while (cursor < raw.length && (raw[cursor] == ' ' || raw[cursor] == '\t')) {
+ cursor++
+ spaceAfter = true
+ }
+ while (out.isNotEmpty() && (out.last() == ' ' || out.last() == '\t')) {
+ out.setLength(out.length - 1)
+ }
+ // The space is a *separator*, so it is only owed when there is
+ // something on both sides. `好的 [calm]` ends the reply and must
+ // not end it on a space.
+ if (out.isNotEmpty() && cursor < raw.length && (spaceBefore || spaceAfter)) {
+ out.append(' ')
+ }
+
+ if (mood == null) continue // dropped, but it names no face
+
+ // Two markers with nothing between them: the later one wins, which
+ // is what the model meant by writing it second.
+ if (cues.isNotEmpty() && cues.last().at == out.length) cues.removeAt(cues.size - 1)
+ cues.add(Cue(out.length, mood))
+ }
+ out.append(raw, cursor, raw.length)
+
+ val text = out.toString()
+ // A reply that was *nothing but* markers has had its words removed by
+ // something other than us, and speaking "" is a socket opened for
+ // silence. Give the raw text back and keep the faces.
+ if (text.isBlank() && raw.isNotBlank()) return Script(raw, emptyList())
+ return Script(text, cues.filter { it.at <= text.length })
+ }
+
+ /** [parse] for callers that only need the words. */
+ fun strip(raw: String): String = parse(raw).text
+}
diff --git a/app/src/main/java/net/kuafuai/andee/tts/TtsController.kt b/app/src/main/java/net/kuafuai/andee/tts/TtsController.kt
index 9fc0858..95f0d99 100644
--- a/app/src/main/java/net/kuafuai/andee/tts/TtsController.kt
+++ b/app/src/main/java/net/kuafuai/andee/tts/TtsController.kt
@@ -7,6 +7,7 @@ import android.os.SystemClock
import android.util.Log
import net.kuafuai.andee.audio.AudioFocusGate
import net.kuafuai.andee.audio.AudioIO
+import net.kuafuai.andee.brain.SpeechMood
import net.kuafuai.andee.config.VoiceConfig
import net.kuafuai.andee.ui.FloatingWindowUi
import java.util.concurrent.Executors
@@ -123,6 +124,88 @@ class TtsController(
stallWatchdog = null
}
+ // ---- Emotion cues: lining the model's faces up with its own voice ----
+ //
+ // The hard part is that there is no clock in common: the markers are at
+ // character offsets into our string, and the thing the user is actually
+ // hearing is an AudioTrack. The obvious bridge is 火山's own
+ // TTSSentenceStart, which would pin each sentence in both clocks at once —
+ // and it does not work. Measured on device: the server sends that event
+ // **once per [HuoshanTts.push]**, not once per sentence, and with an
+ // **empty** text field, so there is nothing to locate and nothing to
+ // locate it with. Pushing each sentence as its own call would get one
+ // event each, but then the events race the audio (the server may announce
+ // all of them before much sound has arrived) and serialising the pushes
+ // risks an audible gap mid-reply.
+ //
+ // So the mapping is arithmetic instead of protocol: assume the utterance
+ // is read at a constant number of characters per millisecond. That is
+ // false in detail — a comma is a pause, a digit is three syllables — but
+ // the question being answered is "which of two faces", at one-sentence
+ // resolution, where a few hundred milliseconds of error is invisible.
+ // Total length comes from [AudioIO.queuedMs] at [onDone] (the producer's
+ // clock, known within the first fraction of playback because 火山 streams
+ // far faster than real time) and the position from [AudioIO.playedMs]
+ // (the track's own head, so real time). Until the total is known the
+ // opening face holds, which is the same fallback as having no cues.
+ //
+ // Everything below is touched from [ui] only, except [cueText] /
+ // [cueList] / [cueTotalMs], which are written off the ui thread and are
+ // therefore volatile.
+
+ /** The clean utterance; only its length is used, as the character clock's span. */
+ @Volatile
+ private var cueText: String = ""
+
+ @Volatile
+ private var cueList: List<SpeechMood.Cue> = emptyList()
+
+ /**
+ * Length of the whole utterance in ms, or 0 until the server has sent all
+ * of it. Set from [onDone] on the websocket thread, after the last
+ * [onAudio] on that same thread — so it is complete by construction.
+ */
+ @Volatile
+ private var cueTotalMs: Long = 0
+
+ private var cueNext = 0
+ private var cueTicker: Runnable? = null
+
+ private fun startCueTicker() {
+ stopCueTicker()
+ if (cueList.size < 2) return // one face for the whole reply needs no walking
+ val r = object : Runnable {
+ override fun run() {
+ // Deliberately not `client == null`: the socket is nulled at
+ // [onDone], which is near the *start* of what the user hears,
+ // not the end. Gating on it stopped the walk before any
+ // mid-utterance face was due. The speaker is the clock here,
+ // so the speaker decides when the walk is over.
+ if (cancelled || !audio.isStreamPlaying()) {
+ cueTicker = null
+ return
+ }
+ val total = cueTotalMs
+ if (total > 0) {
+ val heard = cueText.length * audio.playedMs() / total
+ while (cueNext < cueList.size && cueList[cueNext].at <= heard) {
+ window.setEmotion(cueList[cueNext].mood)
+ cueNext++
+ }
+ }
+ if (cueNext < cueList.size) ui.postDelayed(this, CUE_CHECK_MS)
+ else cueTicker = null
+ }
+ }
+ cueTicker = r
+ ui.postDelayed(r, CUE_CHECK_MS)
+ }
+
+ private fun stopCueTicker() {
+ cueTicker?.let { ui.removeCallbacks(it) }
+ cueTicker = null
+ }
+
private val executor = Executors.newSingleThreadExecutor { r ->
Thread(r, "TtsController").apply { isDaemon = true }
}
@@ -143,13 +226,21 @@ class TtsController(
fun isSpeaking(): Boolean = client != null
- fun speak(text: String) {
+ /**
+ * Say it, optionally wearing the faces the model asked for.
+ *
+ * [cues] are [SpeechMood.Cue]s into [text] — already stripped of their
+ * markers by the caller, which has to be the caller because the same clean
+ * string is what goes into the scrollback. Empty is the normal case and
+ * behaves exactly as this method did before the channel existed.
+ */
+ fun speak(text: String, cues: List<SpeechMood.Cue> = emptyList()) {
if (text.isBlank()) return
cancelled = false
reported = false
executor.execute {
try {
- speakImpl(text)
+ speakImpl(text, cues)
} catch (t: Throwable) {
Log.e(TAG, "speak failed", t)
window.setState(FloatingWindowUi.State.IDLE)
@@ -179,6 +270,7 @@ class TtsController(
// while an utterance is very much on its way.
cancelled = true
disarmStallWatchdog()
+ stopCueTicker()
val c = client
client = null
runCatching { c?.cancel() }
@@ -189,6 +281,7 @@ class TtsController(
fun cancel() {
executor.execute {
disarmStallWatchdog()
+ stopCueTicker()
runCatching { client?.cancel() }
runCatching { audio.stopStreamPlayback() }
focus.release()
@@ -199,6 +292,7 @@ class TtsController(
fun shutdown() {
disarmStallWatchdog()
+ stopCueTicker()
runCatching { client?.cancel() }
runCatching { audio.stopStreamPlayback() }
focus.release()
@@ -206,11 +300,16 @@ class TtsController(
executor.shutdownNow()
}
- private fun speakImpl(text: String) {
+ private fun speakImpl(text: String, cues: List<SpeechMood.Cue>) {
val cfg = VoiceConfig.load(context)
// Interrupted while this utterance sat in the executor queue.
if (cancelled) return
+ cueText = text
+ cueList = cues
+ cueTotalMs = 0
+ ui.post { cueNext = 0 }
+
// Before the socket opens, not when the first audio lands: a media app
// takes a moment to actually stop, and the whole 火山 handshake is that
// moment. Losing the floor mid-answer (a call arriving) ends the
@@ -235,6 +334,10 @@ class TtsController(
// the same words on screen twice, once settled and once in blue
// right underneath.
window.setState(FloatingWindowUi.State.SPEAKING)
+ // Straight after SPEAKING and on the same handler, so the face
+ // the model opened with is already on when the first word is.
+ cues.firstOrNull()?.takeIf { it.at == 0 }?.let { window.setEmotion(it.mood) }
+ ui.post { startCueTicker() }
armStallWatchdog()
executor.execute {
runCatching {
@@ -257,6 +360,12 @@ class TtsController(
override fun onDone() {
if (cancelled) return
disarmStallWatchdog()
+ // The whole utterance has now been received, so this is its
+ // length — the span the cue walk measures character offsets
+ // against. Read here rather than incrementally because it is
+ // the one moment the producer's clock is known to be final,
+ // and it is read on the same thread the audio arrived on.
+ cueTotalMs = audio.queuedMs()
// SessionFinished means the *server* has stopped sending, not
// that the user has stopped hearing: 火山 delivers audio far
// faster than real time, so most of the utterance is usually
@@ -317,5 +426,16 @@ class TtsController(
// than "until the next turn".
private const val STALL_CHECK_MS = 5_000L
private const val STALL_AFTER_MS = 20_000L
+
+ /**
+ * How often the cue walk asks the track where it is.
+ *
+ * A face arriving 120 ms into the sentence it belongs to is
+ * indistinguishable from one arriving on the first syllable — the
+ * crossfade in [net.kuafuai.andee.ui.ball.EmotionState] takes longer
+ * than that by itself. Anything faster is a wake-up per frame for a
+ * change nobody can see.
+ */
+ private const val CUE_CHECK_MS = 120L
}
}
diff --git a/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt b/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
index 8fa5bec..09e3053 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/FloatingWindowUi.kt
@@ -710,6 +710,12 @@ class FloatingWindowUi(
}
ball?.setMood(mood)
ball?.setListening(s == State.RECORDING)
+ // The model's face belongs to one utterance. Anything that is not
+ // speaking has ended that utterance — including the IDLE the drain
+ // posts — so the channel is dropped here rather than at each of the
+ // places that stop the voice. Missing one of those would leave the
+ // ball wearing an emotion into the next conversation.
+ if (s != State.SPEAKING) ball?.setEmotion(null)
// Two transitions are worth a gesture, and only two: the mic
// opening is the one moment the user is waiting for an acknowledgement
// before speaking, and an error is the one moment a face alone is
@@ -739,6 +745,18 @@ class FloatingWindowUi(
}
}
+ /**
+ * The face the model picked for the reply it is saying right now.
+ *
+ * Posted to the same handler as [setState], which is what keeps the two
+ * ordered: the emotion is applied by [net.kuafuai.andee.tts.TtsController]
+ * immediately after it asks for SPEAKING, and a null arriving from any
+ * other state transition lands after it rather than racing it.
+ */
+ fun setEmotion(m: Mood?) {
+ ui.post { ball?.setEmotion(m) }
+ }
+
/**
* The agent ran a tool against the device. Shows as a nod plus the
* SEARCHING face, which decays on its own — so this is fire-and-forget and
diff --git a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallRenderer.kt b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallRenderer.kt
index 6670f67..002df66 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallRenderer.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallRenderer.kt
@@ -62,6 +62,24 @@ class EmotionBallRenderer : GLSurfaceView.Renderer {
@Volatile
var listening: Boolean = false
+ /**
+ * The face the model asked for, for the reply it is saying right now.
+ *
+ * A *separate* channel from [mood] rather than a value written into it, and
+ * the separation is the whole trick. [mood] carries the voice pipeline's
+ * state, and SPEAKING is what used to drive the mouth; swapping HAPPY in
+ * there would have given the user a grinning ball with a shut mouth
+ * halfway through its own sentence. So this layers on top: it supplies the
+ * *face* while [EmotionState.tick]'s `talking` flag keeps supplying the
+ * *mouth*.
+ *
+ * Only read while [mood] is SPEAKING. Null means the model said nothing
+ * about how it felt, which is the overwhelmingly common case and must land
+ * on exactly the behaviour that existed before this channel did.
+ */
+ @Volatile
+ var emotion: Mood? = null
+
/**
* A task is running — set for its whole duration, not per tool call.
*
@@ -414,7 +432,14 @@ class EmotionBallRenderer : GLSurfaceView.Renderer {
// actually at the ledge — see [atLedge]. Everything else about working
// (the face, the rim) is fine to start immediately; the ball looks
// eager, not broken.
- state.tick(dt, t, resolveMood(dt), listening, working && atLedge)
+ state.tick(
+ dt,
+ t,
+ resolveMood(dt),
+ listening,
+ working && atLedge,
+ talking = mood == Mood.SPEAKING,
+ )
GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT or GLES20.GL_DEPTH_BUFFER_BIT)
clipToLedge()
@@ -510,10 +535,15 @@ class EmotionBallRenderer : GLSurfaceView.Renderer {
if (overlayLeft <= 0f) overlayMood = null
}
- val base = if ((working || toolLeft > 0f) && (mood == Mood.CALM || mood == Mood.THINKING)) {
- Mood.SEARCHING
- } else {
- mood
+ val spoken = if (mood == Mood.SPEAKING) emotion else null
+ val base = when {
+ // An emotion the model picked for this reply outranks the work
+ // faces for the same reason the voice pipeline does: the ball is
+ // mid-sentence with the user, and housekeeping can wait.
+ spoken != null -> spoken
+ (working || toolLeft > 0f) && (mood == Mood.CALM || mood == Mood.THINKING) ->
+ Mood.SEARCHING
+ else -> mood
}
// Holding a signboard: attending to the user beats the work faces
// (the task is *paused* on this question) but not the voice pipeline
diff --git a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallTextureView.kt b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallTextureView.kt
index 308ca10..b1ca940 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallTextureView.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionBallTextureView.kt
@@ -58,6 +58,11 @@ class EmotionBallTextureView @JvmOverloads constructor(
renderer.listening = l
}
+ /** The face the model picked for what it is saying — see [EmotionBallRenderer.emotion]. */
+ fun setEmotion(m: Mood?) {
+ renderer.emotion = m
+ }
+
/** A task is running, for its whole duration — see [EmotionBallRenderer.working]. */
fun setWorking(w: Boolean) {
renderer.working = w
diff --git a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionState.kt b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionState.kt
index 1977576..84a7586 100644
--- a/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionState.kt
+++ b/app/src/main/java/net/kuafuai/andee/ui/ball/EmotionState.kt
@@ -203,7 +203,14 @@ class EmotionState {
actionDur = ACTION_SECONDS[a.ordinal]
}
- fun tick(dt: Float, t: Float, mood: Mood, listening: Boolean, working: Boolean) {
+ fun tick(
+ dt: Float,
+ t: Float,
+ mood: Mood,
+ listening: Boolean,
+ working: Boolean,
+ talking: Boolean = mood == Mood.SPEAKING,
+ ) {
val want = FACE[mood]!!
// Lerp toward the target face. 0.14 s to reach = min(1, dt*7).
@@ -241,7 +248,12 @@ class EmotionState {
// (matches the R3F fallback path when the audio source can't report
// level). Everything else falls back to the mood's resting [Face.gape],
// which is how LISTENING gets a mouth that is open rather than curved.
- val talkTarget = if (mood == Mood.SPEAKING) {
+ //
+ // Gated on [talking], not on `mood == SPEAKING`, because the two stopped
+ // being the same question once the model could pick a face for its own
+ // reply: a HAPPY answer is still an answer being said out loud, and
+ // reading the mood here would have left it grinning in silence.
+ val talkTarget = if (talking) {
0.34f + sin(t * 17f) * 0.16f + sin(t * 29f) * 0.09f
} else want.gape
open += (max(0f, talkTarget) - open) * min(1f, dt * 22f)
diff --git a/app/src/test/java/net/kuafuai/andee/brain/SpeechMoodTest.kt b/app/src/test/java/net/kuafuai/andee/brain/SpeechMoodTest.kt
new file mode 100644
index 0000000..567bee9
--- /dev/null
+++ b/app/src/test/java/net/kuafuai/andee/brain/SpeechMoodTest.kt
@@ -0,0 +1,123 @@
+package net.kuafuai.andee.brain
+
+import net.kuafuai.andee.ui.ball.Mood
+import org.junit.Assert.assertEquals
+import org.junit.Test
+
+/**
+ * [SpeechMood] is offset arithmetic over two alphabets, and every way it can be
+ * wrong is invisible on the device: a cue one character off still lands on the
+ * right sentence, a swallowed space is only audible, and a marker that survives
+ * is read out loud as a word nobody notices was not in the answer. Hence a test.
+ */
+class SpeechMoodTest {
+
+ @Test
+ fun `no markers is the identity`() {
+ val s = SpeechMood.parse("好的,我看一下。")
+ assertEquals("好的,我看一下。", s.text)
+ assertEquals(emptyList<SpeechMood.Cue>(), s.cues)
+ }
+
+ @Test
+ fun `opening marker leaves no leading space`() {
+ val s = SpeechMood.parse("[happy] 找到了。")
+ assertEquals("找到了。", s.text)
+ assertEquals(listOf(SpeechMood.Cue(0, Mood.HAPPY)), s.cues)
+ assertEquals(Mood.HAPPY, s.opening)
+ }
+
+ @Test
+ fun `a marker between English words keeps exactly one space`() {
+ val s = SpeechMood.parse("ok [happy] yes")
+ assertEquals("ok yes", s.text)
+ assertEquals(listOf(SpeechMood.Cue(3, Mood.HAPPY)), s.cues)
+ }
+
+ @Test
+ fun `cue offsets index the cleaned text`() {
+ val s = SpeechMood.parse("[curious]这是什么。[happy]找到了。")
+ assertEquals("这是什么。找到了。", s.text)
+ assertEquals(
+ listOf(SpeechMood.Cue(0, Mood.CURIOUS), SpeechMood.Cue(5, Mood.HAPPY)),
+ s.cues,
+ )
+ assertEquals("找到了。", s.text.substring(5))
+ }
+
+ @Test
+ fun `digit tags, shouted sentinels and markdown labels survive untouched`() {
+ val s = SpeechMood.parse("有 [12条] 未读,[END] 见 [here](http://x)")
+ assertEquals("有 [12条] 未读,[END] 见 [here](http://x)", s.text)
+ assertEquals(emptyList<SpeechMood.Cue>(), s.cues)
+ }
+
+ /**
+ * The bug this whole vocabulary exists for: asked to look angry, the model
+ * wrote the obvious word, which was in neither the six names nor any face.
+ */
+ @Test
+ fun `angry is a face rather than a word to read out`() {
+ val s = SpeechMood.parse("[angry]你怎么又这样。")
+ assertEquals("你怎么又这样。", s.text)
+ assertEquals(listOf(SpeechMood.Cue(0, Mood.TENSE)), s.cues)
+ }
+
+ @Test
+ fun `chinese names and lenticular brackets parse`() {
+ val s = SpeechMood.parse("【生气】真是的。【开心】算了。")
+ assertEquals("真是的。算了。", s.text)
+ assertEquals(
+ listOf(SpeechMood.Cue(0, Mood.TENSE), SpeechMood.Cue(4, Mood.HAPPY)),
+ s.cues,
+ )
+ }
+
+ /**
+ * A lowercase bracketed word is a face attempt even when we don't know it.
+ * Speaking it aloud is the one outcome that is never right; holding the
+ * previous face is what a single-marker reply does anyway.
+ */
+ @Test
+ fun `an unknown lowercase tag is dropped silently and names no face`() {
+ val s = SpeechMood.parse("[happy]好了。[sparkly]你看。")
+ assertEquals("好了。你看。", s.text)
+ assertEquals(listOf(SpeechMood.Cue(0, Mood.HAPPY)), s.cues)
+ }
+
+ @Test
+ fun `fullwidth brackets and mixed case parse`() {
+ val s = SpeechMood.parse("[Calm]好的")
+ assertEquals("好的", s.text)
+ assertEquals(listOf(SpeechMood.Cue(0, Mood.CALM)), s.cues)
+ }
+
+ @Test
+ fun `adjacent markers collapse to the later one`() {
+ val s = SpeechMood.parse("[happy][concerned]出事了")
+ assertEquals("出事了", s.text)
+ assertEquals(listOf(SpeechMood.Cue(0, Mood.CONCERNED)), s.cues)
+ }
+
+ @Test
+ fun `a reply that is only markers keeps its words rather than becoming silence`() {
+ val s = SpeechMood.parse("[happy]")
+ assertEquals("[happy]", s.text)
+ assertEquals(emptyList<SpeechMood.Cue>(), s.cues)
+ }
+
+ @Test
+ fun `a trailing marker does not report a cue past the end`() {
+ val s = SpeechMood.parse("好的 [calm]")
+ assertEquals("好的", s.text)
+ assertEquals(listOf(SpeechMood.Cue(2, Mood.CALM)), s.cues)
+ assertEquals(null, s.opening)
+ }
+
+ @Test
+ fun `newlines belong to the text, not the marker`() {
+ val s = SpeechMood.parse("第一句。\n[happy]第二句。")
+ assertEquals("第一句。\n第二句。", s.text)
+ assertEquals(listOf(SpeechMood.Cue(5, Mood.HAPPY)), s.cues)
+ }
+}