Skip to content

Your Android phone and your Windows PC, working as one. Local-first, end-to-end encrypted, no account.

Voice typing into the PC, and the phone as an air mouse

Bee373e4fbd0171
diff --git a/README.md b/README.md
index 2d55fbd..5fa4af0 100644
--- a/README.md
+++ b/README.md
@@ -128,17 +128,22 @@ tests; there are no releases yet.
clipboard, or save one or many to a folder you pick (`Downloads\Nectarlink`
by default). New photos and screenshots from the phone also pop up on the PC
with a preview, ready to save or copy.
-- **Touchpad, keyboard and presentation remote**: control the PC from the
- phone — a touchpad with one-finger move, tap to click, two-finger tap for
- right-click, two-finger scroll, double-tap-and-hold drag and adjustable
- speed; soft-keyboard typing with a bar of modifier and special keys (`Ctrl`,
- `Alt`, `Shift`, `Win`, `Esc`, `Tab`, arrows, `Home`/`End`, `PgUp`/`PgDn`,
- `Del` and shortcuts); and a presentation mode with Next/Previous slide (plus
- the phone's volume buttons while open), Start/End show, Black screen, and a
- hold-for-laser pointer that draws an accent-colored dot on the PC screen
- following the phone's gyroscope or finger. Off by default on the PC, with a
- one-time prompt the first time a phone asks and a per-phone toggle in
- Settings.
+- **Touchpad, air mouse, keyboard, voice typing and presentation remote**:
+ control the PC from the phone — a touchpad with one-finger move, tap to
+ click, two-finger tap for right-click, two-finger scroll,
+ double-tap-and-hold drag and adjustable speed; an **air mouse** where holding
+ the pad and tilting the phone moves the PC cursor with its gyroscope (with
+ hand-tremor filtering, smoothing and tap-to-click); **voice typing** with a
+ **Dictate** button (tap or hold to talk) that uses the phone's speech
+ recognizer (on-device where available) with live partial results and types
+ into the PC's focused field; soft-keyboard typing with a bar of modifier and
+ special keys (`Ctrl`, `Alt`, `Shift`, `Win`, `Esc`, `Tab`, arrows,
+ `Home`/`End`, `PgUp`/`PgDn`, `Del` and shortcuts); and a presentation mode
+ with Next/Previous slide (plus the phone's volume buttons while open),
+ Start/End show, Black screen, and a hold-for-laser pointer that draws an
+ accent-colored dot on the PC screen following the phone's gyroscope or
+ finger. Off by default on the PC, with a one-time prompt the first time a
+ phone asks and a per-phone toggle in Settings.
- **Voice recorder**: record with the phone's microphone (with a live level
meter, pause and resume, timestamped markers, and recording that keeps going
when the screen turns off) and send it straight to the PC. On the PC it lands
diff --git a/android/app/src/main/AndroidManifest.xml b/android/app/src/main/AndroidManifest.xml
index 62b7b34..342fa9c 100644
--- a/android/app/src/main/AndroidManifest.xml
+++ b/android/app/src/main/AndroidManifest.xml
@@ -18,7 +18,7 @@
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MEDIA_PROJECTION" />
<!-- Voice recorder to PC while the screen is off or the app is in the background. -->
<uses-permission android:name="android.permission.FOREGROUND_SERVICE_MICROPHONE" />
- <!-- Voice recordings for a PC (asked for on the Record screen) and the
+ <!-- Voice recordings and voice typing for a PC (asked in context) and the
phone's sound while mirrored (playback capture, with the screen share's consent). -->
<uses-permission android:name="android.permission.RECORD_AUDIO" />
<!-- Elevated: turning wireless debugging back on after a reboot or a new
@@ -74,16 +74,24 @@
android:name="android.permission.READ_EXTERNAL_STORAGE"
android:maxSdkVersion="32" />
- <!-- Names and icons of the apps whose notifications are mirrored: Android
- hides other apps unless they're declared here. -->
+ <!-- Names and icons of the apps whose notifications are mirrored, and the
+ phone's speech recognizer for voice typing: Android hides other apps
+ unless they're declared here. -->
<queries>
<intent>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent>
+ <intent>
+ <action android:name="android.speech.RecognitionService" />
+ </intent>
+ <intent>
+ <action android:name="android.speech.action.RECOGNIZE_SPEECH" />
+ </intent>
</queries>
<uses-feature android:name="android.hardware.camera.any" android:required="false" />
+ <uses-feature android:name="android.hardware.sensor.gyroscope" android:required="false" />
<application
android:name=".NectarlinkApplication"
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
index 336c00d..86a17aa 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/core/Core.kt
@@ -708,9 +708,13 @@ class Core(context: Context, private val scope: CoroutineScope) : EventListener
fun remoteButton(id: String, button: String, action: String, onStatus: ((RemoteAccess) -> Unit)? = null) =
remoteCommand(onStatus) { it.remoteButton(id, button, action) }
- /** Types Unicode text on a PC. */
+ /** Types Unicode text on a PC, splitting long strings into 256-byte chunks. */
fun remoteText(id: String, text: String, onStatus: ((RemoteAccess) -> Unit)? = null) =
- remoteCommand(onStatus) { it.remoteText(id, text) }
+ remoteCommand(onStatus) { node ->
+ for (chunk in app.nectarlink.android.ui.remote.UtteranceJoiner.chunkUtf8(text)) {
+ node.remoteText(id, chunk)
+ }
+ }
/** Presses a named key or shortcut with optional modifiers on a PC. */
fun remoteKey(id: String, key: String, mods: List<String> = emptyList(), onStatus: ((RemoteAccess) -> Unit)? = null) =
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/ui/MainActivity.kt b/android/app/src/main/kotlin/app/nectarlink/android/ui/MainActivity.kt
index 663e9c2..d90dfd2 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/ui/MainActivity.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/ui/MainActivity.kt
@@ -167,10 +167,12 @@ class MainActivity : ComponentActivity() {
private fun handleRemoteIntent(intent: Intent?) {
val pc = intent?.getStringExtra("remote_pc") ?: return
val mode = intent.getStringExtra("remote_mode")
- requestedRemoteMode = if (mode.equals("presentation", ignoreCase = true)) {
- RemoteMode.Presentation
- } else {
- RemoteMode.Touchpad
+ requestedRemoteMode = when {
+ mode.equals("presentation", ignoreCase = true) -> RemoteMode.Presentation
+ mode.equals("air", ignoreCase = true) ||
+ mode.equals("air_mouse", ignoreCase = true) ||
+ mode.equals("airmouse", ignoreCase = true) -> RemoteMode.AirMouse
+ else -> RemoteMode.Touchpad
}
requestedRemotePc = pc
}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteMath.kt b/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteMath.kt
new file mode 100644
index 0000000..48982cc
--- /dev/null
+++ b/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteMath.kt
@@ -0,0 +1,205 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+package app.nectarlink.android.ui.remote
+
+import kotlin.math.hypot
+
+/**
+ * Maps phone gyroscope angular velocity (`Sensor.TYPE_GYROSCOPE`, in rad/s)
+ * to relative PC pointer motion `(dx, dy)` in logical pixels, with a radial
+ * dead zone against hand tremor and exponential smoothing.
+ */
+class AirMouseFilter(
+ private val deadZoneRadPerSec: Float = DEFAULT_DEAD_ZONE_RAD_PER_SEC,
+ private val smoothingAlpha: Float = DEFAULT_SMOOTHING_ALPHA,
+ private val basePixelsPerRad: Float = DEFAULT_PIXELS_PER_RAD,
+) {
+ var smoothX: Float = 0f
+ private set
+ var smoothY: Float = 0f
+ private set
+ private var lastTimestampNs: Long = 0L
+
+ /**
+ * Processes a gyroscope sample at [timestampNs] and returns the relative
+ * pointer step `(dx, dy)` in logical pixels.
+ */
+ fun onSample(
+ wx: Float,
+ wy: Float,
+ wz: Float,
+ timestampNs: Long,
+ sensitivity: Float,
+ ): Pair<Float, Float> {
+ val dtSec = if (lastTimestampNs == 0L || timestampNs <= lastTimestampNs) {
+ DEFAULT_DT_SEC
+ } else {
+ ((timestampNs - lastTimestampNs) * 1e-9f).coerceIn(MIN_DT_SEC, MAX_DT_SEC)
+ }
+ lastTimestampNs = timestampNs
+ return step(wx, wy, wz, dtSec, sensitivity)
+ }
+
+ /**
+ * Pure step with an explicit time delta [dtSec] (in seconds).
+ *
+ * Axis mapping:
+ * - Horizontal (`dx` > 0 moves right): panning right rotates around -Z
+ * (when held flat) and -Y (when held upright), so `rawX = -(wz + wy)`.
+ * - Vertical (`dy` > 0 moves down): tilting down rotates around -X, so
+ * `rawY = -wx`.
+ */
+ fun step(
+ wx: Float,
+ wy: Float,
+ wz: Float,
+ dtSec: Float,
+ sensitivity: Float = 1.0f,
+ ): Pair<Float, Float> {
+ if (!wx.isFinite() || !wy.isFinite() || !wz.isFinite() || !dtSec.isFinite() || !sensitivity.isFinite()) {
+ return 0f to 0f
+ }
+ val clampedDt = dtSec.coerceIn(MIN_DT_SEC, MAX_DT_SEC)
+ val clampedSens = sensitivity.coerceIn(0.1f, 5.0f)
+
+ val rawX = -(wz + wy)
+ val rawY = -wx
+ val (dzX, dzY) = applyDeadZone(rawX, rawY, deadZoneRadPerSec)
+
+ val alpha = smoothingAlpha.coerceIn(0.05f, 1.0f)
+ smoothX += alpha * (dzX - smoothX)
+ smoothY += alpha * (dzY - smoothY)
+
+ if (dzX == 0f && dzY == 0f && hypot(smoothX, smoothY) < SNAP_ZERO_EPSILON) {
+ smoothX = 0f
+ smoothY = 0f
+ return 0f to 0f
+ }
+
+ val scale = clampedDt * basePixelsPerRad * clampedSens
+ val dx = (smoothX * scale).coerceIn(-MAX_MOVE_DELTA, MAX_MOVE_DELTA)
+ val dy = (smoothY * scale).coerceIn(-MAX_MOVE_DELTA, MAX_MOVE_DELTA)
+ return dx to dy
+ }
+
+ /** Clears smoothed velocity and timestamp state when the pad is released. */
+ fun reset() {
+ smoothX = 0f
+ smoothY = 0f
+ lastTimestampNs = 0L
+ }
+
+ companion object {
+ const val DEFAULT_DEAD_ZONE_RAD_PER_SEC = 0.04f
+ const val DEFAULT_SMOOTHING_ALPHA = 0.35f
+ const val DEFAULT_PIXELS_PER_RAD = 1800f
+ const val DEFAULT_DT_SEC = 0.02f
+ const val MIN_DT_SEC = 0.001f
+ const val MAX_DT_SEC = 0.1f
+ const val MAX_MOVE_DELTA = 4000f
+ private const val SNAP_ZERO_EPSILON = 0.005f
+
+ /**
+ * Applies a continuous radial dead zone of [deadZone] rad/s so hand
+ * tremor produces zero output while motion above the threshold ramps
+ * smoothly from zero.
+ */
+ fun applyDeadZone(
+ rawX: Float,
+ rawY: Float,
+ deadZone: Float = DEFAULT_DEAD_ZONE_RAD_PER_SEC,
+ ): Pair<Float, Float> {
+ val mag = hypot(rawX, rawY)
+ if (mag <= deadZone || mag == 0f) return 0f to 0f
+ val factor = (mag - deadZone) / mag
+ return (rawX * factor) to (rawY * factor)
+ }
+ }
+}
+
+/**
+ * Formats consecutive dictated utterances for typing into a PC field:
+ * strips control characters, trims surrounding whitespace, and prepends a
+ * single space between consecutive non-empty utterances.
+ */
+class UtteranceJoiner {
+ var hasTypedUtterance: Boolean = false
+ private set
+
+ fun next(raw: String): String {
+ val cleaned = sanitizeUtterance(raw)
+ if (cleaned.isEmpty()) return ""
+ val out = if (hasTypedUtterance) " $cleaned" else cleaned
+ hasTypedUtterance = true
+ return out
+ }
+
+ fun reset() {
+ hasTypedUtterance = false
+ }
+
+ companion object {
+ const val MAX_REMOTE_TEXT_BYTES = 256
+
+ /**
+ * Replaces control characters (newlines, tabs, etc.) with spaces and
+ * collapses whitespace runs so the result is valid for `remote.input`
+ * (`type = "text"`).
+ */
+ fun sanitizeUtterance(raw: String): String {
+ if (raw.isEmpty()) return ""
+ val sb = StringBuilder(raw.length)
+ var lastWasSpace = true
+ for (ch in raw) {
+ if (ch.isISOControl() || ch == '\u0000' || ch.isWhitespace()) {
+ if (!lastWasSpace) {
+ sb.append(' ')
+ lastWasSpace = true
+ }
+ } else {
+ sb.append(ch)
+ lastWasSpace = false
+ }
+ }
+ if (sb.isNotEmpty() && sb[sb.length - 1] == ' ') {
+ sb.setLength(sb.length - 1)
+ }
+ return sb.toString()
+ }
+
+ /**
+ * Splits [text] on Unicode code-point boundaries into chunks of at most
+ * [maxBytes] UTF-8 bytes (`docs/protocol/remote.md` §3: 1–256 bytes),
+ * dropping control characters.
+ */
+ fun chunkUtf8(text: String, maxBytes: Int = MAX_REMOTE_TEXT_BYTES): List<String> {
+ require(maxBytes >= 4) { "maxBytes must fit at least one UTF-8 code point" }
+ val filtered = text.filter { !it.isISOControl() && it != '\u0000' }
+ if (filtered.isEmpty()) return emptyList()
+ val chunks = mutableListOf<String>()
+ var start = 0
+ var currentBytes = 0
+ var i = 0
+ while (i < filtered.length) {
+ val codePoint = Character.codePointAt(filtered, i)
+ val charCount = Character.charCount(codePoint)
+ val utf8Bytes = when {
+ codePoint <= 0x7F -> 1
+ codePoint <= 0x7FF -> 2
+ codePoint <= 0xFFFF -> 3
+ else -> 4
+ }
+ if (currentBytes + utf8Bytes > maxBytes && i > start) {
+ chunks.add(filtered.substring(start, i))
+ start = i
+ currentBytes = 0
+ }
+ currentBytes += utf8Bytes
+ i += charCount
+ }
+ if (start < filtered.length) {
+ chunks.add(filtered.substring(start, filtered.length))
+ }
+ return chunks
+ }
+ }
+}
diff --git a/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteScreen.kt b/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteScreen.kt
index 05716e4..132726b 100644
--- a/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteScreen.kt
+++ b/android/app/src/main/kotlin/app/nectarlink/android/ui/remote/RemoteScreen.kt
@@ -1,13 +1,23 @@
// SPDX-License-Identifier: GPL-3.0-or-later
package app.nectarlink.android.ui.remote
+import android.Manifest
import android.app.Activity
import android.content.Context
+import android.content.Intent
+import android.content.pm.PackageManager
import android.hardware.Sensor
import android.hardware.SensorEvent
import android.hardware.SensorEventListener
import android.hardware.SensorManager
+import android.os.Build
+import android.os.Bundle
+import android.speech.RecognitionListener
+import android.speech.RecognizerIntent
+import android.speech.SpeechRecognizer
import android.view.WindowManager
+import androidx.activity.compose.rememberLauncherForActivityResult
+import androidx.activity.result.contract.ActivityResultContracts
import androidx.compose.foundation.BorderStroke
import androidx.compose.foundation.Canvas
import androidx.compose.foundation.gestures.awaitEachGesture
@@ -50,6 +60,7 @@ import androidx.compose.runtime.mutableFloatStateOf
import androidx.compose.runtime.mutableStateListOf
import androidx.compose.runtime.mutableStateOf
import androidx.compose.runtime.remember
+import androidx.compose.runtime.rememberUpdatedState
import androidx.compose.runtime.saveable.rememberSaveable
import androidx.compose.runtime.setValue
import androidx.compose.ui.Alignment
@@ -65,14 +76,17 @@ import androidx.compose.ui.platform.LocalView
import androidx.compose.ui.res.stringResource
import androidx.compose.ui.text.input.ImeAction
import androidx.compose.ui.text.style.TextAlign
+import androidx.compose.ui.text.style.TextOverflow
import androidx.compose.ui.unit.dp
+import androidx.core.content.ContextCompat
import app.nectarlink.android.R
import app.nectarlink.android.core.Core
import app.nectarlink.android.core.Device
import kotlinx.coroutines.delay
+import kotlin.math.abs
import kotlin.math.hypot
-enum class RemoteMode { Touchpad, Presentation }
+enum class RemoteMode { Touchpad, AirMouse, Presentation }
@Composable
fun RemoteScreen(
@@ -102,6 +116,7 @@ fun RemoteScreen(
var mode by rememberSaveable(initialMode) { mutableStateOf(initialMode) }
var access by remember(device.id) { mutableStateOf(Core.RemoteAccess.Allowed) }
+ val utteranceJoiner = remember(device.id) { UtteranceJoiner() }
// Check access on entry and poll while denied so the prompt banner clears
// as soon as the user clicks Allow on the PC.
@@ -149,21 +164,28 @@ fun RemoteScreen(
) {}
}
- // Mode switcher: Touchpad | Presentation
+ // Mode switcher: Touchpad | Air mouse | Presentation
SingleChoiceSegmentedButtonRow(Modifier.fillMaxWidth()) {
SegmentedButton(
selected = mode == RemoteMode.Touchpad,
onClick = { mode = RemoteMode.Touchpad },
- shape = SegmentedButtonDefaults.itemShape(0, 2),
+ shape = SegmentedButtonDefaults.itemShape(0, 3),
) {
- Text(stringResource(R.string.remote_mode_touchpad))
+ Text(stringResource(R.string.remote_mode_touchpad), maxLines = 1)
+ }
+ SegmentedButton(
+ selected = mode == RemoteMode.AirMouse,
+ onClick = { mode = RemoteMode.AirMouse },
+ shape = SegmentedButtonDefaults.itemShape(1, 3),
+ ) {
+ Text(stringResource(R.string.remote_mode_air), maxLines = 1)
}
SegmentedButton(
selected = mode == RemoteMode.Presentation,
onClick = { mode = RemoteMode.Presentation },
- shape = SegmentedButtonDefaults.itemShape(1, 2),
+ shape = SegmentedButtonDefaults.itemShape(2, 3),
) {
- Text(stringResource(R.string.remote_mode_presentation))
+ Text(stringResource(R.string.remote_mode_presentation), maxLines = 1)
}
}
@@ -189,6 +211,16 @@ fun RemoteScreen(
pcName = device.name,
core = core,
sensitivity = sensitivity,
+ utteranceJoiner = utteranceJoiner,
+ onStatus = onStatus,
+ modifier = Modifier.weight(1f),
+ )
+ RemoteMode.AirMouse -> AirMouseView(
+ pcId = device.id,
+ pcName = device.name,
+ core = core,
+ sensitivity = sensitivity,
+ utteranceJoiner = utteranceJoiner,
onStatus = onStatus,
modifier = Modifier.weight(1f),
)
@@ -226,6 +258,7 @@ private fun TouchpadView(
pcName: String,
core: Core,
sensitivity: Float,
+ utteranceJoiner: UtteranceJoiner,
onStatus: (Core.RemoteAccess) -> Unit,
modifier: Modifier = Modifier,
) {
@@ -333,7 +366,7 @@ private fun TouchpadView(
scrollAccumX += avgDx
scrollAccumY += avgDy
val stepPx = 18f
- if (kotlin.math.abs(scrollAccumX) >= stepPx || kotlin.math.abs(scrollAccumY) >= stepPx) {
+ if (abs(scrollAccumX) >= stepPx || abs(scrollAccumY) >= stepPx) {
val notchesX = -(scrollAccumX / 48f)
val notchesY = -(scrollAccumY / 36f)
scrollAccumX = 0f
@@ -425,6 +458,15 @@ private fun TouchpadView(
}
}
+ // Voice typing bar (Dictate button: hold to talk, or tap to start/stop)
+ VoiceTypingBar(
+ pcId = pcId,
+ pcName = pcName,
+ core = core,
+ utteranceJoiner = utteranceJoiner,
+ onStatus = onStatus,
+ )
+
// Special keys & modifiers bar (always accessible in Touchpad mode)
SpecialKeysBar(
activeMods = activeMods,
@@ -474,6 +516,559 @@ private fun TouchpadView(
}
}
+@Composable
+private fun AirMouseView(
+ pcId: String,
+ pcName: String,
+ core: Core,
+ sensitivity: Float,
+ utteranceJoiner: UtteranceJoiner,
+ onStatus: (Core.RemoteAccess) -> Unit,
+ modifier: Modifier = Modifier,
+) {
+ val context = LocalContext.current
+ val sensorManager = remember(context) {
+ context.getSystemService(Context.SENSOR_SERVICE) as? SensorManager
+ }
+ val gyroSensor = remember(sensorManager) {
+ sensorManager?.getDefaultSensor(Sensor.TYPE_GYROSCOPE)
+ }
+ val hasGyro = gyroSensor != null
+
+ var padHeld by remember { mutableStateOf(false) }
+ var aimOffset by remember { mutableStateOf(Offset.Zero) }
+ val filter = remember { AirMouseFilter() }
+ val currentSensitivity by rememberUpdatedState(sensitivity)
+
+ DisposableEffect(padHeld, pcId, hasGyro) {
+ if (!padHeld || gyroSensor == null || sensorManager == null) {
+ filter.reset()
+ aimOffset = Offset.Zero
+ return@DisposableEffect onDispose {}
+ }
+
+ filter.reset()
+ val listener = object : SensorEventListener {
+ override fun onSensorChanged(event: SensorEvent) {
+ if (!padHeld || event.sensor.type != Sensor.TYPE_GYROSCOPE || event.values.size < 3) return
+ val (dx, dy) = filter.onSample(
+ wx = event.values[0],
+ wy = event.values[1],
+ wz = event.values[2],
+ timestampNs = event.timestamp,
+ sensitivity = currentSensitivity,
+ )
+ if (abs(dx) >= 0.05f || abs(dy) >= 0.05f) {
+ core.remoteMove(pcId, dx, dy)
+ val nextX = (aimOffset.x + dx * 0.35f).coerceIn(-120f, 120f)
+ val nextY = (aimOffset.y + dy * 0.35f).coerceIn(-120f, 120f)
+ aimOffset = Offset(nextX, nextY)
+ }
+ }
+
+ override fun onAccuracyChanged(sensor: Sensor?, accuracy: Int) {}
+ }
+
+ sensorManager.registerListener(listener, gyroSensor, SensorManager.SENSOR_DELAY_GAME)
+ onDispose {
+ sensorManager.unregisterListener(listener)
+ filter.reset()
+ aimOffset = Offset.Zero
+ }
+ }
+
+ Column(
+ modifier = modifier.fillMaxWidth(),
+ verticalArrangement = Arrangement.spacedBy(10.dp),
+ ) {
+ if (!hasGyro) {
+ StatusBanner(
+ title = stringResource(R.string.remote_air_no_gyro_title),
+ text = stringResource(R.string.remote_air_no_gyro_text),
+ )
+ }
+
+ val primaryColor = MaterialTheme.colorScheme.primary
+ val outlineColor = MaterialTheme.colorScheme.outlineVariant
+ Surface(
+ shape = MaterialTheme.shapes.extraLarge,
+ color = if (padHeld) MaterialTheme.colorScheme.primaryContainer else MaterialTheme.colorScheme.surfaceContainerHigh,
+ contentColor = if (padHeld) MaterialTheme.colorScheme.onPrimaryContainer else MaterialTheme.colorScheme.onSurface,
+ border = BorderStroke(
+ width = if (padHeld) 2.dp else 1.dp,
+ color = if (padHeld) primaryColor else outlineColor,
+ ),
+ modifier = Modifier
+ .weight(1f)
+ .fillMaxWidth()
+ .pointerInput(pcId, hasGyro) {
+ val tapSlop = viewConfiguration.touchSlop
+ val tapTimeoutMs = 240L
+
+ awaitEachGesture {
+ awaitFirstDown(requireUnconsumed = false)
+ val downTime = System.currentTimeMillis()
+ var maxPointers = 1
+ var totalMoved = 0f
+
+ filter.reset()
+ aimOffset = Offset.Zero
+ padHeld = hasGyro
+
+ while (true) {
+ val event = awaitPointerEvent()
+ val pressed = event.changes.filter { it.pressed }
+ if (pressed.isEmpty()) break
+ if (pressed.size > maxPointers) maxPointers = pressed.size
+ for (c in pressed) {
+ val d = c.positionChange()
+ totalMoved += hypot(d.x, d.y)
+ }
+ }
+
+ padHeld = false
+ filter.reset()
+ aimOffset = Offset.Zero
+
+ val elapsed = System.currentTimeMillis() - downTime
+ if (elapsed <= tapTimeoutMs && totalMoved <= tapSlop * 1.5f) {
+ if (maxPointers == 1) {
+ core.remoteButton(pcId, "left", "click", onStatus)
+ } else if (maxPointers == 2) {
+ core.remoteButton(pcId, "right", "click", onStatus)
+ }
+ }
+ }
+ },
+ ) {
+ Box(Modifier.fillMaxSize()) {
+ Canvas(Modifier.fillMaxSize()) {
+ val center = Offset(size.width / 2f, size.height / 2f)
+ val ringColor = if (padHeld) primaryColor.copy(alpha = 0.35f) else outlineColor.copy(alpha = 0.5f)
+ drawCircle(
+ color = ringColor,
+ radius = 44.dp.toPx(),
+ center = center,
+ )
+ val dotCenter = Offset(
+ x = (center.x + aimOffset.x.dp.toPx() * 0.4f).coerceIn(24.dp.toPx(), size.width - 24.dp.toPx()),
+ y = (center.y + aimOffset.y.dp.toPx() * 0.4f).coerceIn(24.dp.toPx(), size.height - 24.dp.toPx()),
+ )
+ drawCircle(
+ color = if (padHeld) primaryColor else primaryColor.copy(alpha = 0.45f),
+ radius = if (padHeld) 10.dp.toPx() else 6.dp.toPx(),
+ center = dotCenter,
+ )
+ }
+
+ Column(
+ horizontalAlignment = Alignment.CenterHorizontally,
+ modifier = Modifier
+ .align(Alignment.TopCenter)
+ .padding(horizontal = 20.dp, vertical = 18.dp),
+ ) {
+ Text(
+ text = if (!hasGyro) {
+ stringResource(R.string.remote_air_no_gyro_title)
+ } else if (padHeld) {
+ stringResource(R.string.remote_air_active, pcName)
+ } else {
+ stringResource(R.string.remote_air_hold, pcName)
+ },
+ style = MaterialTheme.typography.titleMedium,
+ textAlign = TextAlign.Center,
+ )
+ }
+
+ Text(
+ stringResource(R.string.remote_air_hint),
+ style = MaterialTheme.typography.bodySmall,
+ color = if (padHeld) {
+ MaterialTheme.colorScheme.onPrimaryContainer.copy(alpha = 0.8f)
+ } else {
+ MaterialTheme.colorScheme.onSurfaceVariant
+ },
+ textAlign = TextAlign.Center,
+ modifier = Modifier
+ .align(Alignment.BottomCenter)
+ .padding(horizontal = 20.dp, vertical = 14.dp),
+ )
+ }
+ }
+
+ // Left / Right click buttons under the air mouse pad
+ Row(
+ modifier = Modifier.fillMaxWidth(),
+ horizontalArrangement = Arrangement.spacedBy(8.dp),
+ ) {
+ FilledTonalButton(
+ onClick = { core.remoteButton(pcId, "left", "click", onStatus) },
+ contentPadding = PaddingValues(horizontal = 12.dp, vertical = 8.dp),
+ modifier = Modifier.weight(1f).height(48.dp),
+ ) {
+ Text(stringResource(R.string.remote_left_click), maxLines = 1)
+ }
+ FilledTonalButton(
+ onClick = { core.remoteButton(pcId, "right", "click", onStatus) },
+ contentPadding = PaddingValues(horizontal = 12.dp, vertical = 8.dp),
+ modifier = Modifier.weight(1f).height(48.dp),
+ ) {
+ Text(stringResource(R.string.remote_right_click), maxLines = 1)
+ }
+ }
+
+ // Voice typing bar (Dictate button: hold to talk, or tap to start/stop)
+ VoiceTypingBar(
+ pcId = pcId,
+ pcName = pcName,
+ core = core,
+ utteranceJoiner = utteranceJoiner,
+ onStatus = onStatus,
+ )
+ }
+}
+
+private enum class DictateError {
+ NoRecognizer,
+ PermissionNeeded,
+ NoSpeech,
+ Failed,
+}
+
+private sealed interface DictateState {
+ data class Idle(val lastTyped: String? = null) : DictateState
+ data class Listening(val partial: String = "") : DictateState
+ data class Typing(val text: String) : DictateState
+ data class Error(val kind: DictateError) : DictateState
+}
+
+@Composable
+private fun VoiceTypingBar(
+ pcId: String,
+ pcName: String,
+ core: Core,
+ utteranceJoiner: UtteranceJoiner,
+ onStatus: (Core.RemoteAccess) -> Unit,
+) {
+ val context = LocalContext.current
+ var state by remember(pcId) { mutableStateOf<DictateState>(DictateState.Idle()) }
+ var recognizer by remember { mutableStateOf<SpeechRecognizer?>(null) }
+ var preferOnDevice by remember { mutableStateOf(true) }
+
+ fun destroyRecognizer() {
+ runCatching {
+ recognizer?.cancel()
+ recognizer?.destroy()
+ }
+ recognizer = null
+ }
+
+ DisposableEffect(pcId) {
+ onDispose {
+ destroyRecognizer()
+ }
+ }
+
+ fun startRecognition(allowOnDevice: Boolean = preferOnDevice) {
+ if (ContextCompat.checkSelfPermission(context, Manifest.permission.RECORD_AUDIO) !=
+ PackageManager.PERMISSION_GRANTED
+ ) {
+ state = DictateState.Error(DictateError.PermissionNeeded)
+ return
+ }
+
+ destroyRecognizer()
+
+ val useOnDevice = allowOnDevice &&
+ Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU &&
+ runCatching { SpeechRecognizer.isOnDeviceRecognitionAvailable(context) }.getOrDefault(false)
+
+ val defaultAvailable = runCatching { SpeechRecognizer.isRecognitionAvailable(context) }.getOrDefault(false)
+ if (!useOnDevice && !defaultAvailable) {
+ state = DictateState.Error(DictateError.NoRecognizer)
+ return
+ }
+
+ val sr = runCatching {
+ if (useOnDevice && Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
+ SpeechRecognizer.createOnDeviceSpeechRecognizer(context)
+ } else {
+ SpeechRecognizer.createSpeechRecognizer(context)
+ }
+ }.getOrNull()
+
+ if (sr == null) {
+ state = DictateState.Error(DictateError.NoRecognizer)
+ return
+ }
+
+ var readyForSpeech = false
+ var latestPartial = ""
+
+ sr.setRecognitionListener(object : RecognitionListener {
+ override fun onReadyForSpeech(params: Bundle?) {
+ readyForSpeech = true
+ state = DictateState.Listening(partial = latestPartial)
+ }
+
+ override fun onBeginningOfSpeech() {}
+ override fun onRmsChanged(rmsdB: Float) {}
+ override fun onBufferReceived(buffer: ByteArray?) {}
+ override fun onEndOfSpeech() {}
+
+ override fun onPartialResults(partialResults: Bundle?) {
+ val partial = partialResults
+ ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
+ ?.firstOrNull()
+ ?.let { UtteranceJoiner.sanitizeUtterance(it) }
+ .orEmpty()
+ if (partial.isNotEmpty()) {
+ latestPartial = partial
+ state = DictateState.Listening(partial = partial)
+ }
+ }
+
+ override fun onResults(results: Bundle?) {
+ val raw = results
+ ?.getStringArrayList(SpeechRecognizer.RESULTS_RECOGNITION)
+ ?.firstOrNull()
+ ?.takeIf { it.isNotBlank() }
+ ?: latestPartial
+ val joined = utteranceJoiner.next(raw)
+ if (joined.isEmpty()) {
+ state = DictateState.Error(DictateError.NoSpeech)
+ return
+ }
+ val preview = joined.trimStart()
+ state = DictateState.Typing(preview)
+ core.remoteText(pcId, joined) { access ->
+ onStatus(access)
+ state = DictateState.Idle(lastTyped = preview)
+ }
+ }
+
+ override fun onError(error: Int) {
+ // If the on-device recognizer fails before speech begins (e.g. its
+ // offline model isn't downloaded), fall back to the default service.
+ if (useOnDevice && !readyForSpeech) {
+ preferOnDevice = false
+ if (defaultAvailable) {
+ startRecognition(allowOnDevice = false)
+ return
+ }
+ }
+ if (latestPartial.isNotBlank() &&
+ (error == SpeechRecognizer.ERROR_NO_MATCH || error == SpeechRecognizer.ERROR_SPEECH_TIMEOUT)
+ ) {
+ val joined = utteranceJoiner.next(latestPartial)
+ if (joined.isNotEmpty()) {
+ val preview = joined.trimStart()
+ state = DictateState.Typing(preview)
+ core.remoteText(pcId, joined) { access ->
+ onStatus(access)
+ state = DictateState.Idle(lastTyped = preview)
+ }
+ return
+ }
+ }
+ state = when (error) {
+ SpeechRecognizer.ERROR_INSUFFICIENT_PERMISSIONS ->
+ DictateState.Error(DictateError.PermissionNeeded)
+ SpeechRecognizer.ERROR_NO_MATCH,
+ SpeechRecognizer.ERROR_SPEECH_TIMEOUT ->
+ DictateState.Error(DictateError.NoSpeech)
+ SpeechRecognizer.ERROR_LANGUAGE_NOT_SUPPORTED,
+ SpeechRecognizer.ERROR_LANGUAGE_UNAVAILABLE,
+ SpeechRecognizer.ERROR_CANNOT_CHECK_SUPPORT ->
+ DictateState.Error(DictateError.NoRecognizer)
+ else ->
+ if (!readyForSpeech && error != SpeechRecognizer.ERROR_RECOGNIZER_BUSY) {
+ DictateState.Error(DictateError.NoRecognizer)
+ } else {
+ DictateState.Error(DictateError.Failed)
+ }
+ }
+ }
+
+ override fun onEvent(eventType: Int, params: Bundle?) {}
+ })
+
+ val intent = Intent(RecognizerIntent.ACTION_RECOGNIZE_SPEECH).apply {
+ putExtra(RecognizerIntent.EXTRA_LANGUAGE_MODEL, RecognizerIntent.LANGUAGE_MODEL_FREE_FORM)
+ putExtra(RecognizerIntent.EXTRA_PARTIAL_RESULTS, true)
+ }
+
+ recognizer = sr
+ state = DictateState.Listening(partial = "")
+ runCatching {
+ sr.startListening(intent)
+ }.onFailure {
+ destroyRecognizer()
+ state = DictateState.Error(DictateError.NoRecognizer)
+ }
+ }
+
+ fun stopRecognition() {
+ val sr = recognizer
+ if (sr != null) {
+ runCatching { sr.stopListening() }
+ } else if (state is DictateState.Listening) {
+ state = DictateState.Idle()
+ }
+ }
+
+ val permissionLauncher = rememberLauncherForActivityResult(
+ ActivityResultContracts.RequestPermission(),
+ ) { granted ->
+ if (granted) {
+ startRecognition()
+ } else {
+ state = DictateState.Error(DictateError.PermissionNeeded)
+ }
+ }
+
+ val isListening = state is DictateState.Listening
+ val isError = state is DictateState.Error
+
+ val containerColor = when {
+ isListening -> MaterialTheme.colorScheme.primaryContainer
+ isError -> MaterialTheme.colorScheme.errorContainer
+ else -> MaterialTheme.colorScheme.surfaceContainer
+ }
+ val contentColor = when {
+ isListening -> MaterialTheme.colorScheme.onPrimaryContainer
+ isError -> MaterialTheme.colorScheme.onErrorContainer
+ else -> MaterialTheme.colorScheme.onSurface
+ }
+
+ Surface(
+ shape = MaterialTheme.shapes.large,
+ color = containerColor,
+ contentColor = contentColor,
+ modifier = Modifier.fillMaxWidth(),
+ ) {
+ Row(
+ modifier = Modifier
+ .fillMaxWidth()
+ .padding(horizontal = 12.dp, vertical = 8.dp),
+ verticalAlignment = Alignment.CenterVertically,
+ horizontalArrangement = Arrangement.spacedBy(10.dp),
+ ) {
+ Column(modifier = Modifier.weight(1f)) {
+ val statusText = when (val s = state) {
+ is DictateState.Idle -> if (s.lastTyped != null) {
+ stringResource(R.string.remote_dictate_typed, pcName, s.lastTyped)
+ } else {
+ stringResource(R.string.remote_dictate_hint, pcName)
+ }
+ is DictateState.Listening -> stringResource(R.string.remote_dictate_listening, pcName)
+ is DictateState.Typing -> stringResource(R.string.remote_dictate_typing, pcName)
+ is DictateState.Error -> when (s.kind) {
+ DictateError.NoRecognizer -> stringResource(R.string.remote_dictate_no_recognizer)
+ DictateError.PermissionNeeded -> stringResource(R.string.remote_dictate_permission_needed, pcName)
+ DictateError.NoSpeech -> stringResource(R.string.remote_dictate_no_speech)
+ DictateError.Failed -> stringResource(R.string.remote_dictate_error)
+ }
+ }
+ Text(
+ text = statusText,
+ style = MaterialTheme.typography.bodySmall,
+ maxLines = 2,
+ overflow = TextOverflow.Ellipsis,
+ )
+
+ val partialOrTyping = when (val s = state) {
+ is DictateState.Listening -> s.partial
+ is DictateState.Typing -> s.text
+ else -> ""
+ }
+ if (partialOrTyping.isNotEmpty()) {
+ Spacer(Modifier.height(2.dp))
+ Text(
+ text = partialOrTyping,
+ style = MaterialTheme.typography.bodyMedium,
+ maxLines = 2,
+ overflow = TextOverflow.Ellipsis,
+ )
+ }
+ }
+
+ // Dictate button: supports both tap to start/stop and hold to talk.
+ val currentIsListening by rememberUpdatedState(isListening)
+ val buttonBg = if (isListening) {
+ MaterialTheme.colorScheme.primary
+ } else {
+ MaterialTheme.colorScheme.secondaryContainer
+ }
+ val buttonFg = if (isListening) {
+ MaterialTheme.colorScheme.onPrimary
+ } else {
+ MaterialTheme.colorScheme.onSecondaryContainer
+ }
+
+ Surface(
+ shape = CircleShape,
+ color = buttonBg,
+ contentColor = buttonFg,
+ modifier = Modifier
+ .height(40.dp)
+ .pointerInput(pcId) {
+ val holdThresholdMs = 350L
+ awaitEachGesture {
+ awaitFirstDown(requireUnconsumed = false)
+ val downTime = System.currentTimeMillis()
+ val wasListening = currentIsListening
+
+ if (wasListening) {
+ while (true) {
+ val ev = awaitPointerEvent()
+ if (ev.changes.none { it.pressed }) break
+ }
+ stopRecognition()
+ } else {
+ val hasPerm = ContextCompat.checkSelfPermission(
+ context,
+ Manifest.permission.RECORD_AUDIO,
+ ) == PackageManager.PERMISSION_GRANTED
+
+ if (!hasPerm) {
+ while (true) {
+ val ev = awaitPointerEvent()
+ if (ev.changes.none { it.pressed }) break
+ }
+ permissionLauncher.launch(Manifest.permission.RECORD_AUDIO)
+ } else {
+ startRecognition()
+ while (true) {
+ val ev = awaitPointerEvent()
+ if (ev.changes.none { it.pressed }) break
+ }
+ val heldMs = System.currentTimeMillis() - downTime
+ if (heldMs >= holdThresholdMs) {
+ stopRecognition()
+ }
+ }
+ }
+ }
+ },
+ ) {
+ Box(
+ contentAlignment = Alignment.Center,
+ modifier = Modifier.padding(horizontal = 16.dp),
+ ) {
+ Text(
+ text = stringResource(
+ if (isListening) R.string.remote_dictate_stop else R.string.remote_dictate,
+ ),
+ style = MaterialTheme.typography.labelLarge,
+ maxLines = 1,
+ )
+ }
+ }
+ }
+ }
+}
+
@Composable
private fun SpecialKeysBar(
activeMods: List<String>,
diff --git a/android/app/src/main/res/values/strings.xml b/android/app/src/main/res/values/strings.xml
index 6c7fe20..ace15d6 100644
--- a/android/app/src/main/res/values/strings.xml
+++ b/android/app/src/main/res/values/strings.xml
@@ -224,20 +224,36 @@
<item quantity="other">%1$d items</item>
</plurals>
- <!-- Remote input (touchpad, keyboard & presentation) -->
+ <!-- Remote input (touchpad, air mouse, keyboard, voice typing & presentation) -->
<string name="action_remote_pc">Control PC</string>
<string name="action_back">Back</string>
<string name="settings_remote">Control your PC</string>
<string name="settings_touchpad_sensitivity">Touchpad speed</string>
<string name="settings_touchpad_sensitivity_value">%1$.1f×</string>
<string name="remote_mode_touchpad">Touchpad</string>
+ <string name="remote_mode_air">Air mouse</string>
<string name="remote_mode_presentation">Presentation</string>
<string name="remote_touchpad_hint">Drag to move · Tap to click · Two fingers to right-click or scroll · Double-tap and hold to drag</string>
+ <string name="remote_air_hold">Hold to aim at %1$s</string>
+ <string name="remote_air_active">Moving on %1$s — tilt phone or tap to click</string>
+ <string name="remote_air_hint">Hold the pad and tilt your phone to move the pointer · Tap the pad to click · Release to stop</string>
+ <string name="remote_air_no_gyro_title">No gyroscope on this phone</string>
+ <string name="remote_air_no_gyro_text">Air mouse uses a gyroscope to follow how you tilt the phone. Use Touchpad mode instead.</string>
<string name="remote_left_click">Left</string>
<string name="remote_right_click">Right</string>
<string name="remote_middle_click">Middle</string>
<string name="remote_keyboard">Keyboard</string>
<string name="remote_type_placeholder">Type on %1$s…</string>
+ <string name="remote_dictate">Dictate</string>
+ <string name="remote_dictate_stop">Stop</string>
+ <string name="remote_dictate_hint">Tap or hold Dictate to speak into %1$s</string>
+ <string name="remote_dictate_listening">Listening for %1$s…</string>
+ <string name="remote_dictate_typing">Typing on %1$s…</string>
+ <string name="remote_dictate_typed">Typed on %1$s: “%2$s”</string>
+ <string name="remote_dictate_no_recognizer">This phone doesn\'t have speech recognition installed.</string>
+ <string name="remote_dictate_no_speech">Didn\'t catch anything. Tap or hold Dictate to try again.</string>
+ <string name="remote_dictate_error">Speech recognition couldn\'t finish. Tap or hold Dictate to try again.</string>
+ <string name="remote_dictate_permission_needed">Nectarlink needs microphone access to dictate into %1$s.</string>
<string name="remote_denied_title">Allow control on %1$s</string>
<string name="remote_denied_text">On %1$s, allow control from this phone in Nectarlink\'s prompt or in Settings › Devices.</string>
<string name="remote_offline_text">%1$s isn\'t connected right now.</string>
diff --git a/android/app/src/test/kotlin/app/nectarlink/android/ui/remote/RemoteMathTest.kt b/android/app/src/test/kotlin/app/nectarlink/android/ui/remote/RemoteMathTest.kt
new file mode 100644
index 0000000..4e13f30
--- /dev/null
+++ b/android/app/src/test/kotlin/app/nectarlink/android/ui/remote/RemoteMathTest.kt
@@ -0,0 +1,115 @@
+// SPDX-License-Identifier: GPL-3.0-or-later
+package app.nectarlink.android.ui.remote
+
+import org.junit.Assert.assertEquals
+import org.junit.Assert.assertFalse
+import org.junit.Assert.assertTrue
+import org.junit.Test
+import kotlin.math.abs
+
+class RemoteMathTest {
+ @Test
+ fun deadZoneSuppressesHandTremorAndRampsSmoothly() {
+ val (zeroX, zeroY) = AirMouseFilter.applyDeadZone(0.02f, -0.02f, deadZone = 0.04f)
+ assertEquals(0f, zeroX, 1e-6f)
+ assertEquals(0f, zeroY, 1e-6f)
+
+ val filter = AirMouseFilter(deadZoneRadPerSec = 0.04f, smoothingAlpha = 1.0f)
+ val (dxTremor, dyTremor) = filter.step(wx = 0.02f, wy = 0f, wz = -0.02f, dtSec = 0.02f)
+ assertEquals(0f, dxTremor, 1e-6f)
+ assertEquals(0f, dyTremor, 1e-6f)
+
+ // Just above the 0.04 rad/s threshold produces a small, non-jumping step.
+ val (dxSmall, _) = filter.step(wx = 0f, wy = 0f, wz = -0.05f, dtSec = 0.02f)
+ assertTrue("motion just above dead zone is positive and small", dxSmall > 0f && dxSmall < 1f)
+ }
+
+ @Test
+ fun gyroToMotionMapsAxesAndScalesWithSensitivity() {
+ val filter1 = AirMouseFilter(deadZoneRadPerSec = 0.04f, smoothingAlpha = 1.0f)
+ // Turning right (negative wz or wy) moves cursor right (+dx);
+ // tilting down (negative wx) moves cursor down (+dy).
+ val (dx1, dy1) = filter1.step(wx = -0.5f, wy = 0f, wz = -0.8f, dtSec = 0.02f, sensitivity = 1.0f)
+ assertTrue("panning right produces positive dx", dx1 > 0f)
+ assertTrue("tilting down produces positive dy", dy1 > 0f)
+
+ // Turning left (+wz) and tilting up (+wx) move left (-dx) and up (-dy).
+ val filterOpposite = AirMouseFilter(deadZoneRadPerSec = 0.04f, smoothingAlpha = 1.0f)
+ val (dxLeft, dyUp) = filterOpposite.step(wx = 0.5f, wy = 0f, wz = 0.8f, dtSec = 0.02f, sensitivity = 1.0f)
+ assertEquals(-dx1, dxLeft, 1e-4f)
+ assertEquals(-dy1, dyUp, 1e-4f)
+
+ // Sensitivity scales speed linearly.
+ val filter2 = AirMouseFilter(deadZoneRadPerSec = 0.04f, smoothingAlpha = 1.0f)
+ val (dx2, dy2) = filter2.step(wx = -0.5f, wy = 0f, wz = -0.8f, dtSec = 0.02f, sensitivity = 2.0f)
+ assertEquals(dx1 * 2f, dx2, 1e-3f)
+ assertEquals(dy1 * 2f, dy2, 1e-3f)
+ }
+
+ @Test
+ fun smoothingRampsGraduallyAndResetStopsMotionImmediately() {
+ val filter = AirMouseFilter(deadZoneRadPerSec = 0.04f, smoothingAlpha = 0.35f)
+ val (firstDx, _) = filter.step(wx = 0f, wy = 0f, wz = -1.0f, dtSec = 0.02f, sensitivity = 1.0f)
+ val (secondDx, _) = filter.step(wx = 0f, wy = 0f, wz = -1.0f, dtSec = 0.02f, sensitivity = 1.0f)
+ val (thirdDx, _) = filter.step(wx = 0f, wy = 0f, wz = -1.0f, dtSec = 0.02f, sensitivity = 1.0f)
+
+ assertTrue("first step is positive", firstDx > 0f)
+ assertTrue("second step is larger than first as smoothing ramps up", secondDx > firstDx)
+ assertTrue("third step continues ramping toward steady state", thirdDx > secondDx)
+
+ // Releasing the pad resets smoothed velocity immediately.
+ filter.reset()
+ assertEquals(0f, filter.smoothX, 1e-6f)
+ assertEquals(0f, filter.smoothY, 1e-6f)
+ val (afterReleaseDx, afterReleaseDy) = filter.step(wx = 0f, wy = 0f, wz = 0f, dtSec = 0.02f)
+ assertEquals(0f, afterReleaseDx, 1e-6f)
+ assertEquals(0f, afterReleaseDy, 1e-6f)
+ }
+
+ @Test
+ fun utteranceJoinerSeparatesUtterancesWithSingleSpaceAndPreservesPunctuation() {
+ val joiner = UtteranceJoiner()
+ assertFalse(joiner.hasTypedUtterance)
+
+ // Blank or control-only input is ignored and does not mark an utterance as typed.
+ assertEquals("", joiner.next(" \n\t "))
+ assertFalse(joiner.hasTypedUtterance)
+
+ // First utterance has no leading space; spoken punctuation from the recognizer is kept.
+ assertEquals("Hello, world!", joiner.next(" Hello, world! \n"))
+ assertTrue(joiner.hasTypedUtterance)
+
+ // Subsequent utterances get a single leading space.
+ assertEquals("This is Nectarlink.", joiner.next("This is Nectarlink.").removePrefix(" ").also {
+ // Verify the actual returned string had the single leading space:
+ })
+ val third = joiner.next("How are you?")
+ assertEquals(" How are you?", third)
+
+ // Reset clears utterance history.
+ joiner.reset()
+ assertFalse(joiner.hasTypedUtterance)
+ assertEquals("Fresh start", joiner.next("Fresh start"))
+ }
+
+ @Test
+ fun chunkUtf8RespectsByteLimitAndSurrogatePairs() {
+ val longAscii = "a".repeat(600)
+ val asciiChunks = UtteranceJoiner.chunkUtf8(longAscii, maxBytes = 256)
+ assertEquals(listOf(256, 256, 88), asciiChunks.map { it.toByteArray(Charsets.UTF_8).size })
+ assertEquals(longAscii, asciiChunks.joinToString(""))
+
+ // Emoji (4 UTF-8 bytes each, surrogate pair in UTF-16): 70 emojis = 280 UTF-8 bytes.
+ // A 256-byte chunk fits 64 emojis (256 bytes), leaving 6 emojis (24 bytes) in the second chunk.
+ val emojis = "\uD83D\uDC1D".repeat(70)
+ val emojiChunks = UtteranceJoiner.chunkUtf8(emojis, maxBytes = 256)
+ assertEquals(2, emojiChunks.size)
+ assertTrue(emojiChunks.all { it.toByteArray(Charsets.UTF_8).size <= 256 })
+ assertEquals(emojis, emojiChunks.joinToString(""))
+ // Verify no unpaired surrogates in any chunk.
+ for (chunk in emojiChunks) {
+ assertFalse(chunk.any { it.isSurrogate() && !Character.isDefined(Character.codePointAt(chunk, 0)) })
+ assertEquals(0, abs(chunk.length % 2))
+ }
+ }
+}
diff --git a/docs/PLAN.md b/docs/PLAN.md
index 28bfae1..5d6c21f 100644
--- a/docs/PLAN.md
+++ b/docs/PLAN.md
@@ -200,8 +200,8 @@ current Android. Talking through the PC comes with the HFP item above.
| Feature | Tier |
|---|---|
| Phone as **touchpad + keyboard** for PC, presentation remote (with laser pointer) ✅ | P1 🟢 |
-| **Air mouse**: point the phone and its gyroscope moves the PC cursor, for presentations and couch use | P1 🟢 |
-| **Voice typing into PC**: hold a button on the phone, speak, and the text (from the phone's own speech engine) is typed into the PC's focused field | P1 🟢 |
+| **Air mouse**: point the phone and its gyroscope moves the PC cursor, for presentations and couch use ✅ | P1 🟢 |
+| **Voice typing into PC**: hold a button on the phone, speak, and the text (from the phone's own speech engine) is typed into the PC's focused field ✅ | P1 🟢 |
| Type on the phone with the PC keyboard (without mirroring) | P1 🟣 |
| **Deck**: the phone becomes a Stream-Deck-style macro pad for the PC with live tiles: mic mute, media, app launchers, scripts, OBS scenes, window switching, multiple pages, haptic feedback | P1 🟢 |
| **Flow**: move the PC mouse off the screen edge onto the phone/tablet and back, with clipboard following | P2 🟣 |