π End-to-end encrypted cloud for everything.
Dart 2faandroidauthye2eeencryptionend-to-end-encryptionfluttergolanggoogle-photosgoogle-photos-alternativeiosopensourcephotosprivacyprivacy-toolstypescriptzero-knowledge
Make free-up-space deletion crash resilient
ashilknf730a3f01d1a
diff --git a/mobile/apps/photos/changes/crash-safe-free-space.md b/mobile/apps/photos/changes/crash-safe-free-space.mdnew file mode 100644index 0000000000..5f7e21d28d--- /dev/null+++ b/mobile/apps/photos/changes/crash-safe-free-space.md@@ -0,0 +1 @@+- Made freeing device space resilient to stale local media entries.diff --git a/mobile/apps/photos/docs/README.md b/mobile/apps/photos/docs/README.mdindex 47f83af2ec..1009aadf40 100644--- a/mobile/apps/photos/docs/README.md+++ b/mobile/apps/photos/docs/README.md@@ -6,6 +6,10 @@ Documentation and notes about more advanced or infrequently needed details.See [android-cli.md](android-cli.md) for working on the Android app without installing Android Studio.+### Free up space deletion recovery++See [free-up-space-deletion-recovery-plan.md](free-up-space-deletion-recovery-plan.md) for the failure and recovery plan used when deleting backed-up device files.+### VS CodeThe [vscode](vscode) folder contains template launch configuration. Copy into a top-level `.vscode` to use them.diff --git a/mobile/apps/photos/docs/free-up-space-deletion-recovery-plan.md b/mobile/apps/photos/docs/free-up-space-deletion-recovery-plan.mdnew file mode 100644index 0000000000..069e98e5ec--- /dev/null+++ b/mobile/apps/photos/docs/free-up-space-deletion-recovery-plan.md@@ -0,0 +1,89 @@+# Free up space deletion recovery plan++## Goal++Prevent a failed batch from turning into one platform deletion request per file, while keeping the existing recovery flow and UI.++## Result model++Batch deletion must return both the IDs deleted so far and one terminal status:++- `completed`+- `cancelled`+- `failed`++Success must not be inferred from `deletedIDs.isNotEmpty`. Already-missing or shared-media IDs must not hide a failure to delete the remaining platform assets.++## Deletion rules++- Remove the individual-ID fallback on Android and iOS.+- Stop on the first batch exception or user cancellation; do not attempt later batches.+- Remove `_recursivelyReduceBatchSizeAndRetryDeletion`.+- Use one shared platform-deletion helper for every attempt, with a maximum batch size of 1,900 where applicable.+- Preserve any intentional legacy Android batching and empty-result behavior.+- Checkpoint only the IDs returned as deleted for each successful batch in `FilesDB` before starting the next batch.+- Close progress dialogs in `finally` without relying on `dialogKey.currentContext!`.++## Recovery flow++1. **Normal deletion**+ - Delete the requested platform assets in batches.+ - If completed, finish successfully.+ - If cancelled, stop without another automatic attempt.+ - If failed, continue to stale-ID recovery.++2. **Stale-ID recovery**+ - Verify that Ente has full gallery permission before treating an unresolved asset as missing.+ - Check which candidates still exist.+ - Immediately checkpoint already-missing IDs.+ - Delete the remaining valid assets through the same batched helper.+ - If no valid assets remain, the operation is complete.+ - If completed, finish successfully.+ - If cancelled, stop.+ - If failed on Android, continue to MediaStore recovery.+ - If failed on iOS, stop.++3. **MediaStore recovery β Android only**+ - Run `removeAllNoExistsAsset()`.+ - Run local sync and recalculate the freeable candidates, restricted to the+ IDs included in the original confirmed operation.+ - Perform one final batched deletion.+ - This is the final automatic attempt regardless of its outcome.++## Completion and consistency++- Checkpoint successful platform batches, already-missing IDs, and successfully removed shared-media files independently.+- If a database checkpoint fails, stop before deleting another batch.+- A successful operation means all remaining valid candidates were handled, or every candidate was authoritatively found to be already missing.+- A partially successful operation that later fails must remain failed, while retaining its completed checkpoints.+- Add an operation lock so two free-space deletions cannot run concurrently.++## Existing UI behavior++No new UI is required for this fix:++- First-attempt success: deletion progress, followed by the existing success UI.+- Stale-ID recovery: deletion progress, `Loading...`, another deletion progress, then success.+- MediaStore recovery: the previous steps, `Please wait, this will take a while.`, a final deletion progress, then success.+- Final failure: remain on the Free Space page and show `Could not free up space`.+- Cancellation must stop the flow instead of opening another deletion prompt.++The existing success UI uses the original estimated size, and the existing failure UI does not describe partial progress.++## Verification++Cover at least:++- A stale ID at the start, middle, and end of a batch.+- A successful batch followed by failure or cancellation; the successful batch remains checkpointed.+- All candidates already missing.+- More than 1,900 valid candidates after filtering.+- Missing IDs found while deletion of remaining valid IDs fails; this must not report success.+- Limited or revoked gallery permission; invisible assets must not be classified as missing.+- Shared-media success combined with platform deletion failure.+- A database checkpoint failure; no later platform batch is attempted.+- Concurrent free-space attempts.+- Context disposal during a platform request.+- Final Android recovery failure; no further deletion calls occur.++Lifecycle-sync suppression and a richer native `photo_manager` result API are optional follow-up hardening, not required for this initial fix.diff --git a/mobile/apps/photos/lib/services/free_space/deletion_batch_runner.dart b/mobile/apps/photos/lib/services/free_space/deletion_batch_runner.dartnew file mode 100644index 0000000000..ebb9e4a266--- /dev/null+++ b/mobile/apps/photos/lib/services/free_space/deletion_batch_runner.dart@@ -0,0 +1,147 @@+import "dart:math";++import "package:logging/logging.dart";++final _logger = Logger("DeletionBatchRunner");++enum LocalDeletionStatus { completed, cancelled, failed }++class LocalDeletionResult {+ final LocalDeletionStatus status;+ final Set<String> deletedIDs;+ final Set<String> alreadyMissingIDs;+ final bool canAttemptRecovery;++ const LocalDeletionResult({+ required this.status,+ this.deletedIDs = const <String>{},+ this.alreadyMissingIDs = const <String>{},+ this.canAttemptRecovery = false,+ });++ bool get isCompleted => status == LocalDeletionStatus.completed;+ bool get isCancelled => status == LocalDeletionStatus.cancelled;+}++Set<String> retainOriginalDeletionCandidates({+ required Iterable<String> refreshedLocalIDs,+ required Iterable<String> originalLocalIDs,+}) {+ return refreshedLocalIDs.toSet().intersection(originalLocalIDs.toSet());+}++Future<LocalDeletionResult> executeDeletionBatches({+ required List<String> localIDs,+ required int batchSize,+ required Future<List<String>> Function(List<String>) deleteBatch,+ required Future<void> Function(Set<String>) checkpoint,+ bool emptyResultMeansCompleted = false,+ void Function(int completed, int total)? onProgress,+}) async {+ if (batchSize <= 0) {+ throw ArgumentError.value(batchSize, "batchSize", "Must be positive");+ }+ final uniqueLocalIDs = localIDs.toSet().toList();+ final deletedIDs = <String>{};+ if (uniqueLocalIDs.isEmpty) {+ return const LocalDeletionResult(status: LocalDeletionStatus.completed);+ }++ _logger.info("Batch size: $batchSize");+ for (var index = 0; index < uniqueLocalIDs.length; index += batchSize) {+ onProgress?.call(index, uniqueLocalIDs.length);+ final batch = uniqueLocalIDs+ .getRange(index, min(uniqueLocalIDs.length, index + batchSize))+ .toList();+ final attemptedIDs = batch.toSet();+ _logger.info("Trying to delete ${batch.length} files");++ late final List<String> returnedIDs;+ try {+ returnedIDs = await deleteBatch(batch);+ } catch (e, s) {+ _logger.severe("Could not delete batch of ${batch.length} files", e, s);+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: deletedIDs,+ canAttemptRecovery: true,+ );+ }++ final removedIDs = returnedIDs.toSet().intersection(attemptedIDs);+ if (removedIDs.isNotEmpty) {+ try {+ await checkpoint(removedIDs);+ } catch (e, s) {+ _logger.severe("Could not checkpoint deleted batch", e, s);+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: deletedIDs..addAll(removedIDs),+ );+ }+ deletedIDs.addAll(removedIDs);+ }+ _logger.info("Deleted ${removedIDs.length} of ${batch.length} files");++ if (removedIDs.length != attemptedIDs.length) {+ if (removedIDs.isEmpty && emptyResultMeansCompleted) {+ continue;+ }+ _logger.info(+ "Platform deletion returned fewer IDs than requested; stopping",+ );+ return LocalDeletionResult(+ status: LocalDeletionStatus.cancelled,+ deletedIDs: deletedIDs,+ );+ }+ }+ onProgress?.call(uniqueLocalIDs.length, uniqueLocalIDs.length);+ return LocalDeletionResult(+ status: LocalDeletionStatus.completed,+ deletedIDs: deletedIDs,+ );+}++LocalDeletionResult combineDeletionResults(+ LocalDeletionResult sharedMediaResult,+ LocalDeletionResult platformResult, {+ Set<String> alreadyMissingIDs = const <String>{},+}) {+ final deletedIDs = <String>{+ ...sharedMediaResult.deletedIDs,+ ...platformResult.deletedIDs,+ };+ final missingIDs = <String>{+ ...alreadyMissingIDs,+ ...sharedMediaResult.alreadyMissingIDs,+ ...platformResult.alreadyMissingIDs,+ };+ if (platformResult.isCancelled) {+ return LocalDeletionResult(+ status: LocalDeletionStatus.cancelled,+ deletedIDs: deletedIDs,+ alreadyMissingIDs: missingIDs,+ );+ }+ if (sharedMediaResult.status == LocalDeletionStatus.failed ||+ platformResult.status == LocalDeletionStatus.failed) {+ final sharedFailureCanRecover =+ sharedMediaResult.status != LocalDeletionStatus.failed ||+ sharedMediaResult.canAttemptRecovery;+ final platformFailureCanRecover =+ platformResult.status != LocalDeletionStatus.failed ||+ platformResult.canAttemptRecovery;+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: deletedIDs,+ alreadyMissingIDs: missingIDs,+ canAttemptRecovery: sharedFailureCanRecover && platformFailureCanRecover,+ );+ }+ return LocalDeletionResult(+ status: LocalDeletionStatus.completed,+ deletedIDs: deletedIDs,+ alreadyMissingIDs: missingIDs,+ );+}diff --git a/mobile/apps/photos/lib/ui/tools/free_space_page.dart b/mobile/apps/photos/lib/ui/tools/free_space_page.dartindex 8a31aac39f..66bf062098 100644--- a/mobile/apps/photos/lib/ui/tools/free_space_page.dart+++ b/mobile/apps/photos/lib/ui/tools/free_space_page.dart@@ -7,9 +7,12 @@ import 'package:intl/intl.dart';import 'package:logging/logging.dart';import "package:photos/generated/l10n.dart";import 'package:photos/models/freeable_space_info.dart';+import "package:photos/services/free_space/deletion_batch_runner.dart";import "package:photos/ui/notification/toast.dart";import 'package:photos/utils/delete_file_util.dart';+bool _isFreeSpaceDeletionInProgress = false;+class FreeSpacePage extends StatefulWidget {final FreeableSpaceInfo status;final bool clearSpaceForFolder;@@ -25,6 +28,8 @@ class FreeSpacePage extends StatefulWidget {}class _FreeSpacePageState extends State<FreeSpacePage> {+ bool _isFreeingSpace = false;+@overrideWidget build(BuildContext context) {return Scaffold(@@ -90,6 +95,7 @@ class _FreeSpacePageState extends State<FreeSpacePage> {const SizedBox(height: 32),ButtonComponent(shouldSurfaceExecutionStates: false,+ isDisabled: _isFreeingSpace,onTap: () async {await _showConfirmFreeSpaceSheet(context, status);},@@ -102,29 +108,47 @@ class _FreeSpacePageState extends State<FreeSpacePage> {}Future<void> _freeStorage(FreeableSpaceInfo status) async {- bool isSuccess = await deleteLocalFiles(context, status.localIDs);-- if (isSuccess == false) {- if (!mounted) return;- isSuccess = await deleteLocalFilesAfterRemovingAlreadyDeletedIDs(- context,- status.localIDs,- );+ if (_isFreeSpaceDeletionInProgress) {+ return;}-- if (isSuccess == false && Platform.isAndroid) {- if (!mounted) return;- isSuccess = await retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk(- context,- );+ _isFreeSpaceDeletionInProgress = true;+ if (mounted) {+ setState(() => _isFreeingSpace = true);}- if (isSuccess) {- if (!mounted) return;- Navigator.of(context).pop(true);- } else {+ try {+ var result = await deleteLocalFiles(context, status.localIDs);++ if (result.status == LocalDeletionStatus.failed &&+ result.canAttemptRecovery) {+ if (!mounted) return;+ result = await deleteLocalFilesAfterRemovingAlreadyDeletedIDs(+ context,+ status.localIDs,+ );+ }++ if (result.status == LocalDeletionStatus.failed &&+ result.canAttemptRecovery &&+ Platform.isAndroid) {+ if (!mounted) return;+ result = await retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk(+ context,+ originalLocalIDs: status.localIDs,+ );+ }+if (!mounted) return;- showToast(context, AppLocalizations.of(context).couldNotFreeUpSpace);+ if (result.isCompleted) {+ Navigator.of(context).pop(true);+ } else if (!result.isCancelled) {+ showToast(context, AppLocalizations.of(context).couldNotFreeUpSpace);+ }+ } finally {+ _isFreeSpaceDeletionInProgress = false;+ if (mounted) {+ setState(() => _isFreeingSpace = false);+ }}}diff --git a/mobile/apps/photos/lib/utils/delete_file_util.dart b/mobile/apps/photos/lib/utils/delete_file_util.dartindex ff7bc29815..ee9763823f 100644--- a/mobile/apps/photos/lib/utils/delete_file_util.dart+++ b/mobile/apps/photos/lib/utils/delete_file_util.dart@@ -24,6 +24,7 @@ import 'package:photos/models/selected_files.dart';import 'package:photos/module/download/file.dart';import "package:photos/service_locator.dart";import "package:photos/services/files_service.dart";+import "package:photos/services/free_space/deletion_batch_runner.dart";import "package:photos/services/sync/local_sync_service.dart";import 'package:photos/services/sync/remote_sync_service.dart';import 'package:photos/services/sync/sync_service.dart';@@ -370,92 +371,52 @@ Future<bool> emptyTrash(BuildContext context) async {}}-Future<bool> deleteLocalFiles(+Future<LocalDeletionResult> deleteLocalFiles(BuildContext context,List<String> localIDs,) async {- _logger.info("Trying to delete local files ");- final List<String> deletedIDs = [];- final List<String> localAssetIDs = [];- final List<String> localSharedMediaIDs = [];+ _logger.info("Trying to delete local files");+ final localAssetIDs = <String>[];+ final localSharedMediaIDs = <String>[];- // Keep large platform asset deletes in smaller batches. Shared-media sandbox- // IDs are deleted separately above, and only platform asset IDs are sent to- // PhotoManager.editor.deleteWithIds. Android 11+ routes those IDs through- // MediaStore.createDeleteRequest, where our target SDK has a 2000 URI cap:- // https://developer.android.com/reference/android/provider/MediaStore#createDeleteRequest(android.content.ContentResolver,%20java.util.Collection%3Candroid.net.Uri%3E)- // Smaller batches are also safer for large iOS Photos deletes.- const largeCountThreshold = 1900;try {- for (String id in localIDs) {+ for (final id in localIDs.toSet()) {if (id.startsWith(sharedMediaIdentifier)) {localSharedMediaIDs.add(id);} else {localAssetIDs.add(id);}}- deletedIDs.addAll(await _tryDeleteSharedMediaFiles(localSharedMediaIDs));- final tooManyAssets = localAssetIDs.length > largeCountThreshold;- final bool shouldDeleteInBatches =- await isAndroidSDKVersionLowerThan(android11SDKINT) || tooManyAssets;+ final sharedMediaResult = await _deleteAndCheckpointSharedMediaFiles(+ localSharedMediaIDs,+ );+ if (!sharedMediaResult.canAttemptRecovery &&+ sharedMediaResult.status == LocalDeletionStatus.failed) {+ return sharedMediaResult;+ }if (!context.mounted) {_logger.info("Skipping platform asset deletion after the initiating page was disposed",);- } else if (shouldDeleteInBatches) {- if (tooManyAssets) {- _logger.info(- "Too many assets (${localAssetIDs.length}) to delete in one shot, deleting in batches",- );- await _recursivelyReduceBatchSizeAndRetryDeletion(- batchSize: largeCountThreshold,- context: context,- localIDs: localAssetIDs,- deletedIDs: deletedIDs,- );- } else {- _logger.info("Deleting in batches");- deletedIDs.addAll(- await deleteLocalFilesInBatches(context, localAssetIDs),- );- }- } else {- _logger.info("Deleting in one shot");- deletedIDs.addAll(- await _deleteLocalFilesInOneShot(context, localAssetIDs),- );- }- if (deletedIDs.isEmpty && Platform.isIOS) {- _logger.warning(- "Deletion failed in deleteLocalFiles for ${localAssetIDs.length} files, on iOS",+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: sharedMediaResult.deletedIDs,);}- if (deletedIDs.isNotEmpty) {- final deletedFiles = await FilesDB.instance.getLocalFiles(deletedIDs);- await FilesDB.instance.deleteLocalFiles(deletedIDs);- _logger.info(deletedFiles.length.toString() + " files deleted locally");- Bus.instance.fire(- LocalPhotosUpdatedEvent(deletedFiles, source: "deleteLocal"),- );- return true;- } else {- //On android 10, even if files were deleted, deletedIDs is empty.- //This is a workaround so that users are not shown an error message on- //android 10- if (!await isAndroidSDKVersionLowerThan(android11SDKINT)) {- return false;- }- return true;- }+ final platformResult = await _deletePlatformAssets(context, localAssetIDs);+ return combineDeletionResults(sharedMediaResult, platformResult);} catch (e, s) {_logger.severe("Could not delete local files", e, s);- return false;+ return const LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ canAttemptRecovery: true,+ );}}-Future<bool> deleteLocalFilesAfterRemovingAlreadyDeletedIDs(+Future<LocalDeletionResult> deleteLocalFilesAfterRemovingAlreadyDeletedIDs(BuildContext context,List<String> localIDs,) async {@@ -463,21 +424,39 @@ Future<bool> deleteLocalFilesAfterRemovingAlreadyDeletedIDs("Trying to delete local files after removing already deleted IDs",);- final List<String> deletedIDs = [];- final List<String> localAssetIDs = [];- final List<String> localSharedMediaIDs = [];- final List<String> alreadyDeletedIDs = []; // to ignore already deleted files+ final localAssetIDs = <String>[];+ final localSharedMediaIDs = <String>[];+ final alreadyDeletedIDs = <String>{};+ late LocalDeletionResult sharedMediaResult;final dialog = createProgressDialog(context, "Loading...");await dialog.show();try {+ late final PermissionState permissionState;+ try {+ permissionState = await permissionService.getPermissionState();+ } catch (e, s) {+ _logger.severe(+ "Could not verify gallery permission for stale-ID recovery",+ e,+ s,+ );+ return const LocalDeletionResult(status: LocalDeletionStatus.failed);+ }+ if (permissionState != PermissionState.authorized) {+ _logger.warning(+ "Skipping stale-ID recovery because gallery permission is "+ "$permissionState",+ );+ return const LocalDeletionResult(status: LocalDeletionStatus.failed);+ }+final files = await FilesDB.instance.getLocalFiles(localIDs,dedupeByLocalID: true,);for (final file in files) {if (!(await _localFileExist(file))) {- _logger.warning("Already deleted ${file.tag}");alreadyDeletedIDs.add(file.localID!);} else if (file.localID!.startsWith(sharedMediaIdentifier)) {localSharedMediaIDs.add(file.localID!);@@ -485,64 +464,67 @@ Future<bool> deleteLocalFilesAfterRemovingAlreadyDeletedIDs(localAssetIDs.add(file.localID!);}}- deletedIDs.addAll(alreadyDeletedIDs);- deletedIDs.addAll(await _tryDeleteSharedMediaFiles(localSharedMediaIDs));-- await dialog.hide();-- final bool shouldDeleteInBatches = await isAndroidSDKVersionLowerThan(- android11SDKINT,+ _logger.info(+ "Stale-ID recovery found ${alreadyDeletedIDs.length} missing files",);- if (!context.mounted) {- _logger.info(- "Skipping platform asset deletion after the initiating page was disposed",- );- } else if (shouldDeleteInBatches) {- _logger.info("Deleting in batches");- deletedIDs.addAll(- await deleteLocalFilesInBatches(context, localAssetIDs),- );- } else {- _logger.info("Deleting in one shot");- deletedIDs.addAll(- await _deleteLocalFilesInOneShot(context, localAssetIDs),- );- }- if (deletedIDs.isEmpty && Platform.isIOS) {- _logger.warning(- "Deletion failed in deleteLocalFilesAfterRemovingAlreadyDeletedIDs for ${localAssetIDs.length} files, on iOS",+ try {+ await _checkpointRemovedLocalIDs(alreadyDeletedIDs);+ } catch (e, s) {+ _logger.severe("Could not checkpoint already-missing files", e, s);+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ alreadyMissingIDs: alreadyDeletedIDs,);}- if (deletedIDs.isNotEmpty) {- final deletedFiles = await FilesDB.instance.getLocalFiles(deletedIDs);- await FilesDB.instance.deleteLocalFiles(deletedIDs);- _logger.info(deletedFiles.length.toString() + " files deleted locally");- Bus.instance.fire(- LocalPhotosUpdatedEvent(deletedFiles, source: "deleteLocal"),+ sharedMediaResult = await _deleteAndCheckpointSharedMediaFiles(+ localSharedMediaIDs,+ );+ if (!sharedMediaResult.canAttemptRecovery &&+ sharedMediaResult.status == LocalDeletionStatus.failed) {+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: sharedMediaResult.deletedIDs,+ alreadyMissingIDs: alreadyDeletedIDs,);- return true;- } else {- //On android 10, even if files were deleted, deletedIDs is empty.- //This is a workaround so that users are not shown an error message on- //android 10- if (!await isAndroidSDKVersionLowerThan(android11SDKINT)) {- return false;- }- return true;}} catch (e, s) {_logger.severe("Could not delete local files", e, s);- await dialog.hide();- return false;+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ alreadyMissingIDs: alreadyDeletedIDs,+ canAttemptRecovery: true,+ );+ } finally {+ await _hideProgressDialog(dialog);+ }++ if (!context.mounted) {+ _logger.info(+ "Skipping platform asset deletion after the initiating page was disposed",+ );+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: sharedMediaResult.deletedIDs,+ alreadyMissingIDs: alreadyDeletedIDs,+ );}++ final platformResult = await _deletePlatformAssets(context, localAssetIDs);+ return combineDeletionResults(+ sharedMediaResult,+ platformResult,+ alreadyMissingIDs: alreadyDeletedIDs,+ );}/// Only to be used on Android-Future<bool> retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk(- BuildContext context,-) async {+Future<LocalDeletionResult>+retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk(+ BuildContext context, {+ required Iterable<String> originalLocalIDs,+}) async {_logger.info("Retrying free up space after removing assets non-existing in disk",);@@ -551,6 +533,10 @@ Future<bool> retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk(context,context.l10n.pleaseWaitThisWillTakeAWhile,);+ final localAssetIDs = <String>[];+ final localSharedMediaIDs = <String>[];+ late LocalDeletionResult sharedMediaResult;+await dialog.show();try {final stopwatch = Stopwatch()..start();@@ -563,95 +549,132 @@ Future<bool> retryFreeUpSpaceAfterRemovingAssetsNonExistingInDisk();await LocalSyncService.instance.sync();- late final FreeableSpaceInfo status;- final List<String> deletedIDs = [];- final List<String> localAssetIDs = [];- final List<String> localSharedMediaIDs = [];- status = await FilesService.instance.getFreeableSpaceInfo();+ final FreeableSpaceInfo status = await FilesService.instance+ .getFreeableSpaceInfo();- for (String localID in status.localIDs) {+ final scopedLocalIDs = retainOriginalDeletionCandidates(+ refreshedLocalIDs: status.localIDs,+ originalLocalIDs: originalLocalIDs,+ );+ for (final localID in scopedLocalIDs) {if (localID.startsWith(sharedMediaIdentifier)) {localSharedMediaIDs.add(localID);} else {localAssetIDs.add(localID);}}- deletedIDs.addAll(await _tryDeleteSharedMediaFiles(localSharedMediaIDs));-- await dialog.hide();-- final bool shouldDeleteInBatches = await isAndroidSDKVersionLowerThan(- android11SDKINT,+ sharedMediaResult = await _deleteAndCheckpointSharedMediaFiles(+ localSharedMediaIDs,);- if (!context.mounted) {- _logger.info(- "Skipping platform asset deletion after the initiating page was disposed",- );- } else if (shouldDeleteInBatches) {- _logger.info("Deleting in batches");- deletedIDs.addAll(- await deleteLocalFilesInBatches(context, localAssetIDs),- );- } else {- _logger.info("Deleting in one shot");- deletedIDs.addAll(- await _deleteLocalFilesInOneShot(context, localAssetIDs),+ if (!sharedMediaResult.canAttemptRecovery &&+ sharedMediaResult.status == LocalDeletionStatus.failed) {+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: sharedMediaResult.deletedIDs,);}+ } catch (e, s) {+ _logger.severe("Could not complete MediaStore recovery", e, s);+ return const LocalDeletionResult(status: LocalDeletionStatus.failed);+ } finally {+ await _hideProgressDialog(dialog);+ }- if (deletedIDs.isNotEmpty) {- final deletedFiles = await FilesDB.instance.getLocalFiles(deletedIDs);- await FilesDB.instance.deleteLocalFiles(deletedIDs);- _logger.info(deletedFiles.length.toString() + " files deleted locally");- Bus.instance.fire(- LocalPhotosUpdatedEvent(deletedFiles, source: "deleteLocal"),- );- return true;- } else {- //On android 10, even if files were deleted, deletedIDs is empty.- //This is a workaround so that users are not shown an error message on- //android 10- if (!await isAndroidSDKVersionLowerThan(android11SDKINT)) {- return false;- }- return true;- }- } catch (e) {- await dialog.hide();- return false;+ if (!context.mounted) {+ _logger.info(+ "Skipping final platform deletion after the initiating page was disposed",+ );+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ deletedIDs: sharedMediaResult.deletedIDs,+ );+ }++ final platformResult = await _deletePlatformAssets(context, localAssetIDs);+ final result = combineDeletionResults(sharedMediaResult, platformResult);+ return LocalDeletionResult(+ status: result.status,+ deletedIDs: result.deletedIDs,+ alreadyMissingIDs: result.alreadyMissingIDs,+ );+}++Future<LocalDeletionResult> _deletePlatformAssets(+ BuildContext context,+ List<String> localIDs,+) async {+ final uniqueLocalIDs = localIDs.toSet().toList();+ if (uniqueLocalIDs.isEmpty) {+ return const LocalDeletionResult(status: LocalDeletionStatus.completed);+ }+ if (!context.mounted) {+ return const LocalDeletionResult(status: LocalDeletionStatus.failed);+ }++ // Keep Android 11+ requests below MediaStore's 2000 URI limit. The same+ // threshold also avoids oversized Photos requests on iOS.+ const largeCountThreshold = 1900;+ final isLegacyAndroid =+ Platform.isAndroid && await isAndroidSDKVersionLowerThan(android11SDKINT);+ if (!context.mounted) {+ return const LocalDeletionResult(status: LocalDeletionStatus.failed);+ }+ if (isLegacyAndroid) {+ _logger.info("Deleting platform assets in legacy Android batches");+ return deleteLocalFilesInBatches(+ context,+ uniqueLocalIDs,+ emptyResultMeansCompleted: true,+ );}+ if (uniqueLocalIDs.length > largeCountThreshold) {+ _logger.info(+ "Deleting ${uniqueLocalIDs.length} platform assets in bounded batches",+ );+ return deleteLocalFilesInBatches(+ context,+ uniqueLocalIDs,+ minimumParts: 1,+ maximumBatchSize: largeCountThreshold,+ );+ }+ return _deleteLocalFilesInOneShot(context, uniqueLocalIDs);}-Future<List<String>> _deleteLocalFilesInOneShot(+Future<LocalDeletionResult> _deleteLocalFilesInOneShot(BuildContext context,List<String> localIDs,) async {_logger.info('starting _deleteLocalFilesInOneShot for ${localIDs.length}');- final List<String> deletedIDs = [];final dialog = createProgressDialog(context,"Deleting " + localIDs.length.toString() + " backed up files...",);await dialog.show();try {- deletedIDs.addAll(await PhotoManager.editor.deleteWithIds(localIDs));- } catch (e, s) {- _logger.severe("Could not delete files ", e, s);+ final result = await executeDeletionBatches(+ localIDs: localIDs,+ batchSize: localIDs.length,+ deleteBatch: PhotoManager.editor.deleteWithIds,+ checkpoint: _checkpointRemovedLocalIDs,+ );+ _logger.info(+ '_deleteLocalFilesInOneShot deleted ${result.deletedIDs.length} out '+ 'of ${localIDs.length}',+ );+ return result;+ } finally {+ await _hideProgressDialog(dialog);}- _logger.info(- '_deleteLocalFilesInOneShot deleted ${deletedIDs.length} out '- 'of ${localIDs.length}',- );- await dialog.hide();- return deletedIDs;}-Future<List<String>> deleteLocalFilesInBatches(+Future<LocalDeletionResult> deleteLocalFilesInBatches(BuildContext context,List<String> localIDs, {int minimumParts = 10,int minimumBatchSize = 1,int maximumBatchSize = 100,+ bool emptyResultMeansCompleted = false,}) async {final dialogKey = GlobalKey<LinearProgressDialogState>();final dialog = LinearProgressDialog(@@ -667,82 +690,80 @@ Future<List<String>> deleteLocalFilesInBatches(},barrierColor: Colors.black.withValues(alpha: 0.85),);+ await WidgetsBinding.instance.endOfFrame;final batchSize = min(max(minimumBatchSize, (localIDs.length / minimumParts).round()),maximumBatchSize,);- _logger.info("Batch size: $batchSize");- final List<String> deletedIDs = [];- for (int index = 0; index < localIDs.length; index += batchSize) {- if (dialogKey.currentState != null) {- dialogKey.currentState!.setProgress(index / localIDs.length);- }- final ids = localIDs- .getRange(index, min(localIDs.length, index + batchSize))- .toList();- _logger.info("Trying to delete ${ids.length} files");- final int countBefore = deletedIDs.length;- try {- deletedIDs.addAll(await PhotoManager.editor.deleteWithIds(ids));- _logger.info(- "Deleted ${deletedIDs.length - countBefore} of ${ids.length} files",- );- } catch (e, s) {- _logger.severe("Could not delete batch of ${ids.length} files", e, s);- for (final id in ids) {- try {- deletedIDs.addAll(await PhotoManager.editor.deleteWithIds([id]));- _logger.info("Deleted " + id);- } catch (e, s) {- _logger.severe("Could not delete file " + id, e, s);- }- }+ try {+ return await executeDeletionBatches(+ localIDs: localIDs,+ batchSize: batchSize,+ deleteBatch: PhotoManager.editor.deleteWithIds,+ checkpoint: _checkpointRemovedLocalIDs,+ emptyResultMeansCompleted: emptyResultMeansCompleted,+ onProgress: (completed, total) {+ dialogKey.currentState?.setProgress(completed / total);+ },+ );+ } finally {+ final dialogContext = dialogKey.currentContext;+ if (dialogContext != null && dialogContext.mounted) {+ Navigator.of(dialogContext).pop('dialog');}}- Navigator.of(dialogKey.currentContext!).pop('dialog');- return deletedIDs;}-Future<void> _recursivelyReduceBatchSizeAndRetryDeletion({- required int batchSize,- required BuildContext context,- required List<String> localIDs,- required List<String> deletedIDs,- int minimumBatchSizeThresholdToStopRetry = 1900,-}) async {- // TODO: Revisit whether this recursive retry is still needed. The batch- // helper already falls back to single-ID deletes when a batch fails.- if (batchSize < minimumBatchSizeThresholdToStopRetry) {- _logger.warning(- "Batch size is too small ($batchSize), stopping further retries.",- );- throw Exception(- "Batch size is too small ($batchSize), stopping further retries.",+Future<void> _checkpointRemovedLocalIDs(Iterable<String> localIDs) async {+ final uniqueLocalIDs = localIDs.toSet();+ if (uniqueLocalIDs.isEmpty) {+ return;+ }+ final ids = uniqueLocalIDs.toList();+ final deletedFiles = await FilesDB.instance.getLocalFiles(ids);+ await FilesDB.instance.deleteLocalFiles(ids);+ _logger.info("${deletedFiles.length} files deleted locally");+ if (deletedFiles.isNotEmpty) {+ Bus.instance.fire(+ LocalPhotosUpdatedEvent(deletedFiles, source: "deleteLocal"),);}+}++Future<LocalDeletionResult> _deleteAndCheckpointSharedMediaFiles(+ List<String> localIDs,+) async {+ final requestedIDs = localIDs.toSet();+ if (requestedIDs.isEmpty) {+ return const LocalDeletionResult(status: LocalDeletionStatus.completed);+ }+ final deletedIDs = (await _tryDeleteSharedMediaFiles(+ requestedIDs.toList(),+ )).toSet().intersection(requestedIDs);try {- deletedIDs.addAll(- await deleteLocalFilesInBatches(- context,- localIDs,- minimumBatchSize: 1,- maximumBatchSize: batchSize,- minimumParts: 1,- ),- );- } catch (e) {- _logger.warning(- "Failed to delete local files in batches of $batchSize. Reducing batch size and retrying.",- e,- );- if (!context.mounted) return;- await _recursivelyReduceBatchSizeAndRetryDeletion(- batchSize: (batchSize / 2).floor(),- context: context,- localIDs: localIDs,+ await _checkpointRemovedLocalIDs(deletedIDs);+ } catch (e, s) {+ _logger.severe("Could not checkpoint deleted shared-media files", e, s);+ return LocalDeletionResult(+ status: LocalDeletionStatus.failed,deletedIDs: deletedIDs,);}+ return LocalDeletionResult(+ status: deletedIDs.length == requestedIDs.length+ ? LocalDeletionStatus.completed+ : LocalDeletionStatus.failed,+ deletedIDs: deletedIDs,+ canAttemptRecovery: deletedIDs.length != requestedIDs.length,+ );+}++Future<void> _hideProgressDialog(dynamic dialog) async {+ try {+ await dialog.hide();+ } catch (e, s) {+ _logger.warning("Could not hide deletion progress dialog", e, s);+ }}Future<bool> _localFileExist(EnteFile file) {diff --git a/mobile/apps/photos/test/services/free_space/deletion_batch_runner_test.dart b/mobile/apps/photos/test/services/free_space/deletion_batch_runner_test.dartnew file mode 100644index 0000000000..aeb1678160--- /dev/null+++ b/mobile/apps/photos/test/services/free_space/deletion_batch_runner_test.dart@@ -0,0 +1,151 @@+import "package:flutter_test/flutter_test.dart";+import "package:photos/services/free_space/deletion_batch_runner.dart";++void main() {+ test("stops after the first platform exception", () async {+ final attemptedBatches = <List<String>>[];++ final result = await executeDeletionBatches(+ localIDs: ["1", "2", "3", "4"],+ batchSize: 2,+ deleteBatch: (ids) async {+ attemptedBatches.add(List.of(ids));+ throw StateError("failed batch");+ },+ checkpoint: (_) async {},+ );++ expect(result.status, LocalDeletionStatus.failed);+ expect(result.canAttemptRecovery, isTrue);+ expect(attemptedBatches, [+ ["1", "2"],+ ]);+ });++ test("checkpoints an earlier batch before a later failure", () async {+ final attemptedBatches = <List<String>>[];+ final checkpoints = <Set<String>>[];++ final result = await executeDeletionBatches(+ localIDs: ["1", "2", "3", "4", "5"],+ batchSize: 2,+ deleteBatch: (ids) async {+ attemptedBatches.add(List.of(ids));+ if (attemptedBatches.length == 2) {+ throw StateError("failed batch");+ }+ return ids;+ },+ checkpoint: (ids) async => checkpoints.add(Set.of(ids)),+ );++ expect(result.status, LocalDeletionStatus.failed);+ expect(result.deletedIDs, {"1", "2"});+ expect(checkpoints, [+ {"1", "2"},+ ]);+ expect(attemptedBatches, [+ ["1", "2"],+ ["3", "4"],+ ]);+ });++ test("an empty result stops later batches as cancellation", () async {+ final attemptedBatches = <List<String>>[];++ final result = await executeDeletionBatches(+ localIDs: ["1", "2", "3"],+ batchSize: 2,+ deleteBatch: (ids) async {+ attemptedBatches.add(List.of(ids));+ return [];+ },+ checkpoint: (_) async {},+ );++ expect(result.status, LocalDeletionStatus.cancelled);+ expect(attemptedBatches, [+ ["1", "2"],+ ]);+ });++ test("a checkpoint failure prevents another platform batch", () async {+ final attemptedBatches = <List<String>>[];++ final result = await executeDeletionBatches(+ localIDs: ["1", "2", "3"],+ batchSize: 2,+ deleteBatch: (ids) async {+ attemptedBatches.add(List.of(ids));+ return ids;+ },+ checkpoint: (_) async => throw StateError("checkpoint failed"),+ );++ expect(result.status, LocalDeletionStatus.failed);+ expect(result.canAttemptRecovery, isFalse);+ expect(attemptedBatches, [+ ["1", "2"],+ ]);+ });++ test("preserves legacy Android empty-result completion", () async {+ final attemptedBatches = <List<String>>[];++ final result = await executeDeletionBatches(+ localIDs: ["1", "2", "3", "4", "5"],+ batchSize: 2,+ emptyResultMeansCompleted: true,+ deleteBatch: (ids) async {+ attemptedBatches.add(List.of(ids));+ return [];+ },+ checkpoint: (_) async {},+ );++ expect(result.status, LocalDeletionStatus.completed);+ expect(attemptedBatches.map((ids) => ids.length), [2, 2, 1]);+ });++ test("does not exceed the configured batch size", () async {+ final attemptedBatchSizes = <int>[];+ final ids = List.generate(1901, (index) => "$index");++ final result = await executeDeletionBatches(+ localIDs: ids,+ batchSize: 1900,+ deleteBatch: (ids) async {+ attemptedBatchSizes.add(ids.length);+ return ids;+ },+ checkpoint: (_) async {},+ );++ expect(result.status, LocalDeletionStatus.completed);+ expect(attemptedBatchSizes, [1900, 1]);+ });++ test("reconciled stale IDs do not hide a platform failure", () {+ final result = combineDeletionResults(+ const LocalDeletionResult(status: LocalDeletionStatus.completed),+ const LocalDeletionResult(+ status: LocalDeletionStatus.failed,+ canAttemptRecovery: true,+ ),+ alreadyMissingIDs: {"stale-id"},+ );++ expect(result.status, LocalDeletionStatus.failed);+ expect(result.canAttemptRecovery, isTrue);+ expect(result.alreadyMissingIDs, {"stale-id"});+ });++ test("final recovery retains only originally confirmed candidates", () {+ final candidates = retainOriginalDeletionCandidates(+ refreshedLocalIDs: ["folder-file", "unrelated-file"],+ originalLocalIDs: ["folder-file", "already-handled-file"],+ );++ expect(candidates, {"folder-file"});+ });+}