Skip to content

armgit / ente

Public Imported from GitHub
0

πŸ’š End-to-end encrypted cloud for everything.

[mobile] Harden background engine startup and teardown

laurenspriem0bb5cec77b49
diff --git a/mobile/apps/photos/plugins/ente_background_manager/README.md b/mobile/apps/photos/plugins/ente_background_manager/README.md
index 6f642a7f2d..07c4f6c640 100644
--- a/mobile/apps/photos/plugins/ente_background_manager/README.md
+++ b/mobile/apps/photos/plugins/ente_background_manager/README.md
@@ -8,7 +8,7 @@ The package is part of the mobile workspace. Photos does not depend on or initia
## Consumer interface
-- `BackgroundManager.configure` supplies a retained top-level dispatcher, task configurations, scheduling enablement, and an outcome handler. Repeated configuration reconciles persistent schedules.
+- `BackgroundManager.configure` supplies a retained top-level dispatcher, task configurations, scheduling enablement, and an outcome handler. Repeated configuration reconciles persistent schedules. Setting `enabled: false` disables future scheduling and requests stopping of the active run.
- `BackgroundManager.executeTask` runs the callback inside the dispatcher and handles readiness, stop delivery, errors, and completion.
- `BackgroundTask` exposes its identifier, elapsed time, optional remaining budget, and a latched stop signal. The callback stops admitting work, drains cleanup, and returns a `BackgroundTaskResult`.
- `BackgroundManager.stopActiveRun` requests stopping of the current invocation and completes after retirement. It preserves future registrations and is an immediate no-op when idle. Use it from the foreground; a background callback should return after cleanup instead of awaiting its own retirement.
@@ -20,12 +20,14 @@ Before supplying `foregroundStopTimeout`, validate the lifetime of the consumer'
## Native lifecycle
-Each platform has one process-local runtime. Admission checks backend eligibility, the active execution slot, and native foreground visibility before creating an engine. Busy or foreground deliveries finish as skips. Every admitted run captures its configuration and dispatcher binding and creates a fresh engine.
+Each platform has one process-local runtime. Admission checks backend eligibility, the active execution slot, and native foreground visibility before creating an engine. Busy or foreground deliveries finish as skips. Every admitted run captures its configuration and dispatcher binding and uses a fresh engine. On Android, a stop received during Flutter initialization retires the run before creating an engine.
-The slot remains occupied during startup, execution, cleanup, and teardown. Native callbacks and timers are tied to a unique invocation. Foreground entry always requests stopping, including during startup. Stops remain latched; repeated visibility changes cannot revive a task or extend its grace. Configuration updates affect later runs only.
+The slot remains occupied during startup, execution, cleanup, and teardown. Native callbacks and timers are tied to a unique invocation. Foreground entry always requests stopping, including during startup. Stops remain latched; repeated visibility changes cannot revive a task or extend its grace. A callback returning `completed` after a stop request is reported as `stopped`. Configuration updates affect later runs only.
Normal completion, startup failure, system interruption, and configured forced teardown share one retirement path. Retirement invalidates timers, detaches the task channel, destroys the background engine, and completes the native invocation once. iOS re-arms normal future opportunities independently of Dart. No plugin retry loop, work queue, engine pool, or persistent event history is used.
+If Android engine destruction throws, the plugin reports a teardown failure and fails pending and subsequent `stopActiveRun` calls. The slot stays occupied until the process restarts because the engine may still be alive; subsequent background deliveries skip.
+
Consumers log their task activity. Skips, stops, forced teardown, and failures are forwarded to an available outcome handler; otherwise the plugin writes one native log line (`EnteBackgroundManager` on Android, `io.ente.background` on iOS).
## Native installation
@@ -38,7 +40,7 @@ On iOS, call `BackgroundManagerPlugin.install` in `AppDelegate` with an eligibil
## Validation
-The Android test runs the real worker, scheduler configuration, method-channel encoding, and lifecycle/timer logic with mocked Flutter engines. It covers foreground and busy admission, ownership during teardown, startup stops, cooperative completion, foreground grace, configuration changes during a run, stale events, bootstrap failure, and preservation of recurring scheduling after active-run stopping.
+The Android test runs the real worker, scheduler configuration, method-channel encoding, and lifecycle/timer logic with mocked Flutter engines. It covers foreground and busy admission, ownership during teardown, teardown failures, startup stops, cooperative completion, foreground grace, configuration changes during a run, stale events, bootstrap failure, and preservation of recurring scheduling after active-run stopping.
Use a temporary Flutter host with a path dependency on this package to run native checks. Match Photos' Android build-tool versions and set the host's minimum SDK to 26. From its `android` directory:
diff --git a/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/BackgroundRuntime.kt b/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/BackgroundRuntime.kt
index 27435add7f..e113649b0a 100644
--- a/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/BackgroundRuntime.kt
+++ b/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/BackgroundRuntime.kt
@@ -56,6 +56,7 @@ internal object BackgroundRuntime {
var channel: MethodChannel? = null
var ready = false
var retiring = false
+ var teardownFailure: Exception? = null
var stopReason: String? = null
var budgetTimer: Runnable? = null
var foregroundTimer: Runnable? = null
@@ -126,6 +127,10 @@ internal object BackgroundRuntime {
result?.success(null)
return
}
+ run.teardownFailure?.let {
+ result?.error("teardown", it.message, null)
+ return
+ }
if (result != null) run.stopResults.add(result)
stop(run, "requested")
}
@@ -329,6 +334,10 @@ internal object BackgroundRuntime {
retire(run, "stopped", "system")
return@ensureInitializationCompleteAsync
}
+ run.stopReason?.let {
+ retire(run, "stopped", it)
+ return@ensureInitializationCompleteAsync
+ }
try {
val callback =
requireNotNull(findCallback(configuration.callbackHandle)) {
@@ -474,18 +483,29 @@ internal object BackgroundRuntime {
run.budgetTimer?.let(main::removeCallbacks)
run.foregroundTimer?.let(main::removeCallbacks)
val terminal = if (outcome == "completed" && run.stopReason != null) "stopped" else outcome
+ var failure: Exception? = null
try {
run.channel?.setMethodCallHandler(null)
+ } catch (exception: Exception) {
+ failure = exception
+ }
+ try {
run.engine?.destroy()
} catch (exception: Exception) {
- report(run.configuration.identifier, "failed", "teardown", exception.message)
- run.stopResults.forEach { it.error("teardown", exception.message, null) }
+ run.teardownFailure = exception
+ failure = exception
+ }
+ if (run.teardownFailure == null) {
+ active = null
+ retireUnselectedSchedule(run.configuration.identifier)
+ }
+ if (failure != null) {
+ report(run.configuration.identifier, "failed", "teardown", failure.message)
+ run.stopResults.forEach { it.error("teardown", failure.message, null) }
run.stopResults.clear()
run.completion(false)
return
}
- active = null
- retireUnselectedSchedule(run.configuration.identifier)
if (terminal != "completed") report(run.configuration.identifier, terminal, reason, error)
run.completion(terminal != "failed" && terminal != "forcedTeardown")
run.stopResults.forEach { it.success(null) }
diff --git a/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/TaskConfiguration.kt b/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/TaskConfiguration.kt
index 0c8927985d..2b285d7339 100644
--- a/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/TaskConfiguration.kt
+++ b/mobile/apps/photos/plugins/ente_background_manager/android/src/main/kotlin/io/ente/background/TaskConfiguration.kt
@@ -39,17 +39,22 @@ internal data class TaskConfiguration(
callbackHandle: Long = json.getLong("callbackHandle"),
): TaskConfiguration {
fun duration(key: String): Long? =
- if (json.isNull(key)) null else json.getLong(key).also { require(it >= 0) }
+ if (json.isNull(key)) null
+ else json.getLong(key).also { require(it >= 0) { "$key must not be negative" } }
val identifier = json.getString("identifier")
val kind = json.getString("kind")
val frequency = json.getLong("frequencyMs")
val flex = duration("flexMs")
- require(identifier.isNotBlank())
- require(kind == "refresh" || kind == "processing")
- require(frequency >= 15 * 60 * 1000L)
- require(flex == null || flex in 5 * 60 * 1000L..frequency)
- require(callbackHandle != 0L)
+ require(identifier.isNotBlank()) { "Task identifier must not be blank" }
+ require(kind == "refresh" || kind == "processing") {
+ "Task kind must be refresh or processing"
+ }
+ require(frequency >= 15 * 60 * 1000L) { "Frequency must be at least 15 minutes" }
+ require(flex == null || flex in 5 * 60 * 1000L..frequency) {
+ "Flex must be at least 5 minutes and no greater than frequency"
+ }
+ require(callbackHandle != 0L) { "A retained background dispatcher is required" }
return TaskConfiguration(
identifier,
kind,
diff --git a/mobile/apps/photos/plugins/ente_background_manager/android/src/test/kotlin/io/ente/background/BackgroundRuntimeTest.kt b/mobile/apps/photos/plugins/ente_background_manager/android/src/test/kotlin/io/ente/background/BackgroundRuntimeTest.kt
index f4d07a7556..a6dea827a6 100644
--- a/mobile/apps/photos/plugins/ente_background_manager/android/src/test/kotlin/io/ente/background/BackgroundRuntimeTest.kt
+++ b/mobile/apps/photos/plugins/ente_background_manager/android/src/test/kotlin/io/ente/background/BackgroundRuntimeTest.kt
@@ -3,6 +3,7 @@ package io.ente.background
import android.app.Activity
import android.app.Application
import android.os.Looper
+import android.os.SystemClock
import androidx.work.Configuration
import androidx.work.Data
import androidx.work.ListenableWorker
@@ -169,9 +170,16 @@ class BackgroundRuntimeTest {
val budgetPolicy = policy(budget = 100)
configure(budgetPolicy)
- val budgetRun = worker(budgetPolicy).startWork()
+ val expiredDuringInitialization = worker(budgetPolicy).startWork()
+ val enginesBeforeExpiredRun = engines.size
main.idleFor(Duration.ofMillis(150))
+ startups.removeFirst().run()
+ assertEquals(ListenableWorker.Result.success(), expiredDuringInitialization.get())
+ assertEquals(enginesBeforeExpiredRun, engines.size)
+
+ val budgetRun = worker(budgetPolicy).startWork()
val budgetChannel = bootstrap()
+ main.idleFor(Duration.ofMillis(150))
val budgetReady = budgetChannel.call("ready") as Map<*, *>
assertEquals("budget", budgetReady["stopReason"])
assertTrue((budgetReady["elapsedMs"] as Long) >= 150)
@@ -182,9 +190,22 @@ class BackgroundRuntimeTest {
assertTrue(budgetRun.isDone)
configure(cooperative)
+ val stoppedDuringInitialization = worker(cooperative).startWork()
+ val enginesBeforeStoppedRun = engines.size
+ val opening = Robolectric.buildActivity(Activity::class.java).create().start().resume()
+ val initializationStop = Reply()
+ BackgroundRuntime.requestStop(initializationStop)
+ assertFalse(initializationStop.done)
+ opening.pause().stop().destroy()
+ startups.removeFirst().run()
+ assertEquals(ListenableWorker.Result.success(), stoppedDuringInitialization.get())
+ assertEquals(enginesBeforeStoppedRun, engines.size)
+ assertTrue(initializationStop.done)
+ assertNull(initializationStop.error)
+
val stoppingDuringStartup = worker(cooperative).startWork()
- val foreground = Robolectric.buildActivity(Activity::class.java).create().start().resume()
val stoppedChannel = bootstrap()
+ val foreground = Robolectric.buildActivity(Activity::class.java).create().start().resume()
val stoppedReady = stoppedChannel.call("ready") as Map<*, *>
assertEquals("foreground", stoppedReady["stopReason"])
foreground.pause().stop().destroy()
@@ -246,8 +267,8 @@ class BackgroundRuntimeTest {
val immediatePolicy = policy(budget = 0, grace = 0)
configure(immediatePolicy)
val immediate = worker(immediatePolicy).startWork()
- main.idle()
val immediateChannel = bootstrap()
+ main.idle()
val immediateReady = immediateChannel.call("ready") as Map<*, *>
assertEquals("budget", immediateReady["stopReason"])
assertFalse(immediate.isDone)
@@ -271,6 +292,54 @@ class BackgroundRuntimeTest {
assertEquals("requested", (lastChannel.sent.single().arguments as Map<*, *>)["reason"])
complete(lastChannel, lastReady)
assertTrue(last.isDone)
+
+ configure(cooperative)
+ val handlerFailureRun = worker(cooperative).startWork()
+ val handlerFailureChannel = bootstrap()
+ val handlerFailureReady = handlerFailureChannel.call("ready") as Map<*, *>
+ val handlerFailureStop = Reply()
+ BackgroundRuntime.requestStop(handlerFailureStop)
+ handlerFailureChannel.failHandlerRemoval = true
+ complete(handlerFailureChannel, handlerFailureReady)
+ assertEquals(ListenableWorker.Result.failure(), handlerFailureRun.get())
+ assertEquals("teardown: Handler removal failed", handlerFailureStop.error)
+ verify(engines.last().first).destroy()
+
+ val afterHandlerFailure = worker(cooperative).startWork()
+ val afterHandlerFailureChannel = bootstrap()
+ val afterHandlerFailureReady = afterHandlerFailureChannel.call("ready") as Map<*, *>
+ complete(handlerFailureChannel, handlerFailureReady)
+ assertFalse(afterHandlerFailure.isDone)
+ complete(afterHandlerFailureChannel, afterHandlerFailureReady)
+ assertEquals(ListenableWorker.Result.success(), afterHandlerFailure.get())
+
+ val teardownWorker = worker(cooperative)
+ var teardownCompletions = 0
+ BackgroundRuntime.start(teardownWorker, SystemClock.elapsedRealtime()) { success ->
+ assertFalse(success)
+ teardownCompletions++
+ }
+ val teardownChannel = bootstrap()
+ val teardownReady = teardownChannel.call("ready") as Map<*, *>
+ val teardownEngine = engines.last().first
+ doThrow(IllegalStateException("Engine destruction failed")).`when`(teardownEngine).destroy()
+ val teardownStop = Reply()
+ BackgroundRuntime.requestStop(teardownStop)
+ complete(teardownChannel, teardownReady)
+ assertEquals("teardown: Engine destruction failed", teardownStop.error)
+ assertEquals(1, teardownCompletions)
+ val laterStop = Reply()
+ BackgroundRuntime.requestStop(laterStop)
+ assertTrue(laterStop.done)
+ assertEquals(teardownStop.error, laterStop.error)
+ val enginesAfterFailure = engines.size
+ assertEquals(ListenableWorker.Result.success(), worker(cooperative).startWork().get())
+ assertTrue(startups.isEmpty())
+ assertEquals(enginesAfterFailure, engines.size)
+ teardownWorker.onStopped()
+ main.idleFor(Duration.ofSeconds(1))
+ assertEquals(1, teardownCompletions)
+ verify(teardownEngine, times(1)).destroy()
}
private class Reply : MethodChannel.Result {
@@ -279,16 +348,19 @@ class BackgroundRuntimeTest {
var value: Any? = null
override fun success(result: Any?) {
+ assertFalse("Reply already completed", done)
value = result
done = true
}
override fun error(code: String, message: String?, details: Any?) {
+ assertFalse("Reply already completed", done)
error = "$code: $message"
done = true
}
override fun notImplemented() {
+ assertFalse("Reply already completed", done)
error = "notImplemented"
done = true
}
@@ -297,6 +369,7 @@ class BackgroundRuntimeTest {
private class Messenger : BinaryMessenger {
private val codec = StandardMethodCodec.INSTANCE
private var handler: BinaryMessenger.BinaryMessageHandler? = null
+ var failHandlerRemoval = false
val sent = ArrayList<MethodCall>()
override fun send(channel: String, message: ByteBuffer?) = send(channel, message, null)
@@ -315,6 +388,7 @@ class BackgroundRuntimeTest {
channel: String,
handler: BinaryMessenger.BinaryMessageHandler?,
) {
+ check(handler != null || !failHandlerRemoval) { "Handler removal failed" }
this.handler = handler
}
diff --git a/mobile/apps/photos/plugins/ente_background_manager/pubspec.yaml b/mobile/apps/photos/plugins/ente_background_manager/pubspec.yaml
index fb1fa4f4f8..74c0519c91 100644
--- a/mobile/apps/photos/plugins/ente_background_manager/pubspec.yaml
+++ b/mobile/apps/photos/plugins/ente_background_manager/pubspec.yaml
@@ -13,9 +13,9 @@ dependencies:
sdk: flutter
dev_dependencies:
+ flutter_lints: 6.0.0
flutter_test:
sdk: flutter
- flutter_lints: 6.0.0
flutter:
plugin: