π End-to-end encrypted cloud for everything.
Dart 2faandroidauthye2eeencryptionend-to-end-encryptionfluttergolanggoogle-photosgoogle-photos-alternativeiosopensourcephotosprivacyprivacy-toolstypescriptzero-knowledge
[photos][cli] Add duplicate cleanup commands (#14299)
Aswin Asokdf0dade34c6d
diff --git a/docs/docs/cli/photos.md b/docs/docs/cli/photos.mdindex 15c9b3e785..9576c09d7f 100644--- a/docs/docs/cli/photos.md+++ b/docs/docs/cli/photos.md@@ -107,6 +107,29 @@ ente photos file list --type video --min-size 500MBA file must match every filter you pass. `--sort created` lists the newest files first and `--sort size` the largest first; add `--reverse` for the opposite order. Files with an unknown size sort as the smallest, and ties keep ID order. Without `--sort`, files are listed by ID.+## Find and remove duplicates++```sh+ente photos duplicate list+ente photos duplicate remove+```++`list` shows exact duplicate groups, their files and albums, which copy will be kept, and reclaimable space, largest first. Lists show 100 groups; use `--all` or `--limit N` to see more. `remove` lets you select groups with space and continue with enter, then shows the plan and asks for confirmation. `duplicates` is an alias for `duplicate`.++Select group IDs from the list, preview cleanup, or select all groups:++```sh+ente photos duplicate remove 1580559962386999 --dry-run+ente photos duplicate remove 1580559962386999 --yes --no-input+ente photos duplicate remove --all --dry-run --json+```++Group IDs are the smallest file ID in each group. Removal refreshes metadata first; a group ID that no longer exists stops the command. `--dry-run` previews without changes. `--yes` approves removal; `--no-input` disables prompts and requires IDs or `--all`, plus `--yes` unless previewing.++Duplicates are matched by stored hashes in non-hidden albums you own. The CLI keeps a copy with a caption first, then one with an edited name or date. It preserves album memberships before moving extra copies to Ente's trash. Files are not permanently deleted. If interrupted, list duplicates again before retrying.++With `--json`, `list` returns an array and `remove` returns one object; see [Read as JSON](#read-as-json) for fields. Previews and progress go to stderr.+## Find files by place```sh@@ -151,6 +174,8 @@ Lists return arrays and `view` returns one object. IDs are strings; timestamps a| `album list`, `album view` | `id`, `name`, `type` (`album`, `folder`, `favorites`, `uncategorized`, `defaultHidden`, `quicklink`), `visibility` (`visible`, `archived`, `hidden`), `ownerId`, `updatedAt` || `file list`, `file view` | `id`, `name`, `type` (`image`, `video`, `livephoto`), `ownerId`, `albumIds`, `createdAt`, `updatedAt`; nullable `modifiedAt`, `location` (`latitude`, `longitude`), `caption`, `hash`, `dateTime`, `offsetTime`, `visibility`, `durationSeconds`, `width`, `height`, `sizeBytes` || `file download` | `id`, `output`, `bytes` |+| `duplicate list` | `id`, `sizeBytes`, `reclaimableBytes`, `retainedId`, `files` (`id`, `name`, `albums`: `id`, `name`). `sizeBytes` is the stored encrypted size of one copy; `reclaimableBytes` is the total for the extra copies in that group. |+| `duplicate remove` | `dryRun`, `groupIds`, `retainedIds`, `trashIds`, `reclaimableBytes` (total across selected groups). IDs are strings and byte counts are integers; `trashIds` contains planned removals in a dry run and completed removals otherwise. || `place list` | `name`, `country`, `countryCode`, `latitude` and `longitude` (the city's position), `fileCount` || `place list --countries` | `name`, `code`, `fileCount` || `usage` | `email`, `planExpiresAt`, `planActive`, `totalBytes`, `usedBytes`, `freeBytes`, `planBytes`, `bonusBytes`; nullable `files`, `family` (`usedBytes`, `storageBytes`, `members`: `email`, `status`, `admin`, `usedBytes`, `storageLimitBytes`) |@@ -158,7 +183,7 @@ Lists return arrays and `view` returns one object. IDs are strings; timestamps a## Read offline-`album list`, `album view`, `file list`, `file view` and `place list` work with `--offline` using the metadata an earlier online run synced. An online `album list` syncs albums only; run `file list` online for each album whose files you want to read offline. Places also need the location data that an earlier online `place list`, `--place` or `--country` downloaded. Export, download and usage need the network.+`album list`, `album view`, `file list`, `file view`, `duplicate list` and `place list` work with `--offline` using the metadata an earlier online run synced. An online `album list` syncs albums only; run `file list` online for each album whose files you want to read offline, or `duplicate list` to sync albums eligible for duplicate detection. Places also need the location data that an earlier online `place list`, `--place` or `--country` downloaded. Export, download, duplicate removal and usage need the network.## Limitsdiff --git a/rust/apps/cli/changes/duplicates.md b/rust/apps/cli/changes/duplicates.mdnew file mode 100644index 0000000000..907adf9e59--- /dev/null+++ b/rust/apps/cli/changes/duplicates.md@@ -0,0 +1 @@+- List exact duplicate photos with `photos duplicate list`, preview cleanup, and select groups to move extra copies to trash using the same Rust logic as the web app.diff --git a/rust/apps/cli/src/args.rs b/rust/apps/cli/src/args.rsindex b8f08f9278..08c5303aed 100644--- a/rust/apps/cli/src/args.rs+++ b/rust/apps/cli/src/args.rs@@ -389,6 +389,14 @@ family is null outside a family; otherwise it contains usedBytes, storageBytes aEach member has email, status, admin, usedBytes and storageLimitBytes (null without a cap).")]Usage,+ #[command(+ about = "Find and remove exact duplicate photos and videos",+ visible_alias = "duplicates"+ )]+ Duplicate {+ #[command(subcommand)]+ command: DuplicateCommand,+ },#[command(about = "Manage albums")]Album {#[command(subcommand)]@@ -607,6 +615,49 @@ pub enum FileCommand {},}+const DUPLICATES_HELP: &str = "Uses the same duplicate detection and removal rules as Ente Photos.+Scans files you own in non-hidden albums, including Favorites and archived albums.+Matches stored hashes; a file in several albums is counted once. Files without a hash or known group size are omitted.+Group IDs are the smallest file ID in each group, not row numbers.+Keeps a copy with a caption first, then one with an edited name or date, then the first copy in album-name order.+The retained copy is added to the other copies' albums before extra copies move to trash.+List refreshes metadata first; --offline reads a previous sync. Groups are ordered by reclaimable size.+Remove always refreshes online. Pass group IDs or --all, or omit both to select with space and enter.+Without --dry-run, removal asks for confirmation; --yes approves it and --no-input only disables prompts.+List JSON: an array of groups with id, sizeBytes, reclaimableBytes, retainedId and files.+Each file has id, name and albums (id and name). All IDs are strings; byte counts are integers.+sizeBytes is the stored encrypted size of one copy; reclaimableBytes is the total for the extra copies in the group.+Remove JSON: dryRun, groupIds, retainedIds, trashIds and reclaimableBytes.+Removal reclaimableBytes is the total for all selected groups.+With --dry-run, trashIds are planned; otherwise they were moved to trash. Preview and progress go to stderr.++Examples:+ ente photos duplicate list+ ente photos duplicate remove+ ente photos duplicate remove 1580559962386999 --dry-run+ ente photos duplicate remove 1580559962386999 --yes --no-input --json";++#[derive(Subcommand)]+pub enum DuplicateCommand {+ #[command(about = "List duplicate groups and the copies to keep or trash", after_long_help = DUPLICATES_HELP)]+ List(ListArgs),+ #[command(about = "Select duplicate groups and move extra copies to trash", after_long_help = DUPLICATES_HELP)]+ Remove {+ #[arg(+ value_name = "GROUP",+ conflicts_with = "all",+ help = "Group IDs from duplicates list; omit to select interactively"+ )]+ groups: Vec<i64>,+ #[arg(long, help = "Select every current duplicate group")]+ all: bool,+ #[arg(long, help = "Show the selected removal plan without changing files")]+ dry_run: bool,+ #[arg(long, help = "Approve removal without a confirmation prompt")]+ yes: bool,+ },+}+#[derive(Args)]pub struct FileQuery {#[arg(diff --git a/rust/apps/cli/src/output.rs b/rust/apps/cli/src/output.rsindex e3400f7bb6..8e30aabd99 100644--- a/rust/apps/cli/src/output.rs+++ b/rust/apps/cli/src/output.rs@@ -104,6 +104,29 @@ struct LocationView {longitude: f64,}+#[derive(Serialize)]+#[serde(rename_all = "camelCase")]+pub struct DuplicateGroupView {+ pub id: String,+ pub size_bytes: u64,+ pub reclaimable_bytes: u64,+ pub retained_id: String,+ pub files: Vec<DuplicateFileView>,+}++#[derive(Serialize)]+pub struct DuplicateFileView {+ pub id: String,+ pub name: String,+ pub albums: Vec<DuplicateAlbumView>,+}++#[derive(Serialize)]+pub struct DuplicateAlbumView {+ pub id: String,+ pub name: String,+}+impl FileView {pub fn new(file: File, album_ids: &[i64]) -> Result<Self> {Ok(Self {@@ -326,6 +349,82 @@ pub fn files()}+pub fn duplicates(groups: &[&DuplicateGroupView], args: &ListArgs, as_json: bool) -> Result<()> {+ let limit = args.limit().map_or(usize::MAX, |n| n as usize);+ let shown = &groups[..groups.len().min(limit)];+ if as_json {+ json(&shown)?;+ } else {+ let mut stdout = std::io::stdout().lock();+ if shown.is_empty() {+ writeln!(stdout, "No duplicates found.")?;+ } else {+ duplicate_groups(&mut stdout, shown)?;+ }+ stdout.flush()?;+ }+ if groups.len() > limit {+ eprintln!("Showing first {limit} results; use --limit N or --all to see more.");+ }+ Ok(())+}++pub fn duplicate_label(group: &DuplicateGroupView) -> String {+ format!(+ "{}: {} copies, {} bytes reclaimable β {}",+ group.id,+ group.files.len(),+ group.reclaimable_bytes,+ group+ .files+ .iter()+ .map(|f| escape_controls(&f.name))+ .collect::<Vec<_>>()+ .join(", ")+ )+}++pub fn duplicate_groups(out: &mut impl Write, groups: &[&DuplicateGroupView]) -> Result<()> {+ if groups.is_empty() {+ writeln!(out, "No duplicate groups selected.")?;+ }+ for (index, group) in groups.iter().enumerate() {+ if index > 0 {+ writeln!(out)?;+ }+ writeln!(+ out,+ "Group {}: {} copies, {} bytes reclaimable",+ group.id,+ group.files.len(),+ group.reclaimable_bytes+ )?;+ let rows = group+ .files+ .iter()+ .map(|file| {+ [+ if file.id == group.retained_id {+ "keep"+ } else {+ "trash"+ }+ .to_owned(),+ file.id.clone(),+ escape_controls(&file.name),+ file.albums+ .iter()+ .map(|a| format!("{} ({})", escape_controls(&a.name), a.id))+ .collect::<Vec<_>>()+ .join(", "),+ ]+ })+ .collect::<Vec<_>>();+ write_table(out, ["ACTION", "ID", "NAME", "ALBUMS"], &rows)?;+ }+ Ok(())+}+pub fn cities(cities: impl Iterator<Item = CityView>,args: &ListArgs,diff --git a/rust/apps/cli/src/photos/duplicates.rs b/rust/apps/cli/src/photos/duplicates.rsnew file mode 100644index 0000000000..1b8daf7f9f--- /dev/null+++ b/rust/apps/cli/src/photos/duplicates.rs@@ -0,0 +1,301 @@+use std::collections::HashSet;++use anyhow::{Context, Result, ensure};+use dialoguer::{Confirm, MultiSelect, console::Term};+use ente_core::{Session, crypto::Key};+use ente_photos::{+ collections::{Collection, Visibility},+ dedup::{self, Duplicate, RemovalCollection, RemovalFile},+ files::File,+};+use serde_json::json;++use crate::{+ args::{DuplicateCommand, Options},+ output::{self, DuplicateAlbumView, DuplicateFileView, DuplicateGroupView},+};++use super::replica::Replica;++struct Group {+ view: DuplicateGroupView,+ items: Vec<Duplicate>,+ file_indices: Vec<usize>,+}++struct Snapshot {+ albums: Vec<Collection>,+ files: Vec<File>,+ groups: Vec<Group>,+}++pub fn validate(command: &DuplicateCommand, options: &Options) -> Result<()> {+ if let DuplicateCommand::Remove {+ groups,+ all,+ dry_run,+ yes,+ } = command+ {+ ensure!(+ !options.offline,+ "duplicate removal requires network access"+ );+ ensure!(+ *all || !groups.is_empty() || options.can_prompt(),+ "supply group IDs or --all when running without interactive prompts"+ );+ ensure!(+ *dry_run || *yes || options.can_prompt(),+ "duplicate removal requires confirmation; supply --yes when running without interactive prompts"+ );+ }+ Ok(())+}++pub async fn run(+ command: DuplicateCommand,+ session: &Session,+ replica: &mut Replica<'_>,+ options: &Options,+) -> Result<()> {+ let snapshot = load(session, replica, options.offline).await?;+ let DuplicateCommand::Remove {+ groups,+ all,+ dry_run,+ yes,+ } = command+ else {+ let DuplicateCommand::List(args) = command else {+ unreachable!()+ };+ let views: Vec<_> = snapshot.groups.iter().map(|g| &g.view).collect();+ return output::duplicates(&views, &args, options.json);+ };++ let mut selected = HashSet::new();+ for id in groups {+ let id = id.to_string();+ let index = snapshot+ .groups+ .iter()+ .position(|g| g.view.id == id)+ .with_context(|| {+ format!("no duplicate group matches {id}; run photos duplicate list again")+ })?;+ selected.insert(index);+ }+ if all {+ selected.extend(0..snapshot.groups.len());+ } else if selected.is_empty() && !snapshot.groups.is_empty() {+ let labels: Vec<_> = snapshot+ .groups+ .iter()+ .map(|g| output::duplicate_label(&g.view))+ .collect();+ selected.extend(+ MultiSelect::new()+ .with_prompt("Select duplicate groups (space to toggle, enter to continue)")+ .items(&labels)+ .interact_on(&Term::stderr())?,+ );+ }+ let groups: Vec<_> = snapshot+ .groups+ .into_iter()+ .enumerate()+ .filter_map(|(i, g)| selected.contains(&i).then_some(g))+ .collect();+ let views: Vec<_> = groups.iter().map(|g| &g.view).collect();+ output::duplicate_groups(&mut std::io::stderr().lock(), &views)?;+ let reclaimable_bytes = groups+ .iter()+ .try_fold(0_u64, |total, g| {+ total.checked_add(g.view.reclaimable_bytes)+ })+ .context("duplicate size exceeds the supported range")?;+ let group_ids: Vec<_> = groups.iter().map(|g| g.view.id.clone()).collect();+ let retained_ids: Vec<_> = groups.iter().map(|g| g.view.retained_id.clone()).collect();+ let files: Vec<_> = groups+ .iter()+ .flat_map(|g| g.file_indices.iter().zip(&g.items))+ .map(|(&index, item)| RemovalFile {+ id: item.file_id,+ collection_id: item.collection_ids[0],+ key: Key::from_bytes(*snapshot.files[index].key.as_bytes()),+ })+ .collect();+ let items: Vec<_> = groups.into_iter().map(|g| g.items).collect();+ let plan = dedup::plan_removal(&items);+ if !dry_run && !plan.trash_ids.is_empty() {+ if !yes {+ ensure!(+ Confirm::new()+ .with_prompt(format!(+ "Move {} extra copies to trash?",+ plan.trash_ids.len()+ ))+ .default(false)+ .interact_on(&Term::stderr())?,+ "cancelled"+ );+ }+ let albums: Vec<_> = snapshot+ .albums+ .into_iter()+ .map(|a| RemovalCollection {+ id: a.id,+ key: a.key,+ })+ .collect();+ dedup::execute_removal(session, &plan, &albums, &files, |p| {+ eprintln!("Completed {}/{} duplicate removal steps.", p.completed, p.total);+ })+ .await+ .context("duplicate removal did not finish; some changes may have completed; list duplicates again before retrying")?;+ replica.sync_collections(session).await.context("duplicates were moved to trash, but refreshing albums failed; run photos duplicate list online")?;+ load(session, replica, false).await.context("duplicates were moved to trash, but refreshing files failed; run photos duplicate list online")?;+ }+ output::action(+ options.json,+ &json!({+ "dryRun": dry_run,+ "groupIds": group_ids,+ "retainedIds": retained_ids,+ "trashIds": plan.trash_ids.iter().map(i64::to_string).collect::<Vec<_>>(),+ "reclaimableBytes": reclaimable_bytes,+ }),+ &format!(+ "{} {} extra copies from {} duplicate groups to trash.",+ if dry_run { "Would move" } else { "Moved" },+ plan.trash_ids.len(),+ items.len(),+ ),+ )+}++async fn load(session: &Session, replica: &mut Replica<'_>, offline: bool) -> Result<Snapshot> {+ let mut albums = replica.collections(None, None, |rows| {+ let mut albums = Vec::new();+ for entry in rows {+ let entry = entry?;+ let album = entry.value.with_context(|| {+ format!("cannot scan duplicates: album {} is unavailable", entry.id)+ })?;+ if album.owner_id == session.user_id && !matches!(album.visibility, Visibility::Hidden)+ {+ albums.push(album);+ }+ }+ Ok(albums)+ })?;+ albums.sort_by(|a, b| a.name.cmp(&b.name).then(a.id.cmp(&b.id)));+ if !offline {+ replica.sync_files(session, &albums).await?;+ }+ let mut files = Vec::new();+ let mut candidates = Vec::new();+ for album in &albums {+ ensure!(+ replica.files_synced_to(album.id)?.is_some(),+ "files for album {:?} have not finished syncing; run photos duplicate list online first",+ album.name+ );+ replica.files(Some(album.id), None, None, |rows| {+ for entry in rows {+ let entry = entry?;+ let file = entry.file.with_context(|| {+ format!("cannot scan duplicates: file {} is unavailable", entry.id)+ })?;+ candidates.push(dedup::File {+ id: file.id,+ collection_id: album.id,+ owner_id: file.owner_id,+ hash: file.hash.clone(),+ size: file.size,+ });+ files.push(file);+ }+ Ok(())+ })?;+ }+ let collections: Vec<_> = albums+ .iter()+ .map(|a| dedup::Collection {+ id: a.id,+ owner_id: a.owner_id,+ })+ .collect();+ let mut groups = Vec::new();+ for group in dedup::find_duplicates(session.user_id, &collections, &candidates) {+ let items: Vec<_> = group+ .items+ .iter()+ .map(|item| {+ let file = &files[item.file_index];+ Duplicate {+ file_id: file.id,+ collection_ids: item.collection_ids.clone(),+ caption: file.caption.clone(),+ edited_name: file.edited_name.clone(),+ edited_time: file.edited_time,+ }+ })+ .collect();+ let plan = dedup::plan_removal(std::slice::from_ref(&items));+ let retained = items+ .iter()+ .find(|f| !plan.trash_ids.contains(&f.file_id))+ .context("duplicate group has no retained file")?;+ let id = items+ .iter()+ .map(|f| f.file_id)+ .min()+ .context("empty duplicate group")?;+ let view = DuplicateGroupView {+ id: id.to_string(),+ size_bytes: group.item_size,+ reclaimable_bytes: group+ .item_size+ .checked_mul(plan.trash_ids.len() as u64)+ .context("duplicate size exceeds the supported range")?,+ retained_id: retained.file_id.to_string(),+ files: group+ .items+ .iter()+ .map(|item| {+ let file = &files[item.file_index];+ DuplicateFileView {+ id: file.id.to_string(),+ name: file.name.clone(),+ albums: albums+ .iter()+ .filter(|a| item.collection_ids.contains(&a.id))+ .map(|a| DuplicateAlbumView {+ id: a.id.to_string(),+ name: a.name.clone(),+ })+ .collect(),+ }+ })+ .collect(),+ };+ groups.push(Group {+ view,+ items,+ file_indices: group.items.iter().map(|f| f.file_index).collect(),+ });+ }+ groups.sort_by(|a, b| {+ b.view+ .reclaimable_bytes+ .cmp(&a.view.reclaimable_bytes)+ .then(a.view.id.cmp(&b.view.id))+ });+ Ok(Snapshot {+ albums,+ files,+ groups,+ })+}diff --git a/rust/apps/cli/src/photos/mod.rs b/rust/apps/cli/src/photos/mod.rsindex f8480ff8a5..1717f6d736 100644--- a/rust/apps/cli/src/photos/mod.rs+++ b/rust/apps/cli/src/photos/mod.rs@@ -1,3 +1,4 @@+mod duplicates;mod export;mod places;pub mod replica;@@ -28,6 +29,7 @@ pub async fn run(command: PhotosCommand, selected: Option<&str>, options: &OptioPhotosCommand::Api(args) => return api::run(App::Photos, args, selected, options).await,PhotosCommand::Export(args) => return export::run(args, selected, options).await,PhotosCommand::Usage => return usage::photos(selected, options).await,+ PhotosCommand::Duplicate { ref command } => duplicates::validate(command, options)?,_ => {}}ensure!(@@ -57,6 +59,9 @@ pub async fn run(command: PhotosCommand, selected: Option<&str>, options: &Optiomatch command {PhotosCommand::Api(_) | PhotosCommand::Export(_) | PhotosCommand::Usage => unreachable!(),PhotosCommand::Album { command } => album(command, &replica, options),+ PhotosCommand::Duplicate { command } => {+ duplicates::run(command, &session, &mut replica, options).await+ }PhotosCommand::File { album, command } => {file(command, album.as_deref(), &session, &mut replica, options).await}diff --git a/rust/apps/cli/tests/cli/duplicates.rs b/rust/apps/cli/tests/cli/duplicates.rsnew file mode 100644index 0000000000..cb0a887595--- /dev/null+++ b/rust/apps/cli/tests/cli/duplicates.rs@@ -0,0 +1,193 @@+use super::{files::*, *};++fn fixture(id: i64, key: &Key, hash: &str, public: Value) -> Value {+ let (mut file, _) = remote_file_with(+ id,+ key,+ b"duplicate original",+ &json!({"title": format!("photo-{id}.jpg"), "fileType": 0, "creationTime": 1_700_000_000_000_000i64, "hash": hash}),+ &public,+ );+ file["info"] = json!({"fileSize": 100});+ file+}++#[test]+fn duplicates_list_and_preview_use_owned_visible_memberships_and_shared_retention_rules() {+ let mut server = mockito::Server::new();+ let home = TestHome::new();+ home.seed(&server.url());+ let key = Key::generate();+ let mut hidden = collection(3, "Hidden", &key);+ let metadata = blob::encrypt_json(&json!({"visibility": 2}), &key).unwrap();+ hidden["magicMetadata"] = json!({"data": b64::encode(&metadata.encrypted_data), "header": b64::encode(metadata.decryption_header.as_bytes())});+ let mut favorites = collection(4, "Favorites", &key);+ favorites["type"] = json!("favorites");+ server.mock("GET", "/collections/v2").match_query(mockito::Matcher::Any)+ .with_body(json!({"collections": [collection(1, "Zeta", &key), collection(2, "Alpha", &key), hidden, favorites]}).to_string())+ .expect_at_least(1).create();+ let first = fixture(10, &key, "one", json!({"editedName": "renamed.jpg"}));+ let caption = fixture(11, &key, "one", json!({"caption": 0}));+ let mut foreign = fixture(50, &key, "one", json!({}));+ foreign["ownerID"] = json!(123);+ let mut unknown = fixture(40, &key, "unknown", json!({}));+ unknown.as_object_mut().unwrap().remove("info");+ let mut unknown_copy = unknown.clone();+ unknown_copy["id"] = json!(41);+ let mut live = remote_file_with(30, &key, b"live photo", &json!({"title":"live.heic", "fileType":2, "creationTime":1, "imageHash":"image", "videoHash":"video"}), &json!({})).0;+ live["info"] = json!({"fileSize": 200});+ let mut live_copy = live.clone();+ live_copy["id"] = json!(31);+ page(+ &mut server,+ 1,+ 0,+ json!([+ first,+ fixture(20, &key, "two", json!({})),+ fixture(21, &key, "two", json!({"editedTime":123})),+ foreign,+ unknown,+ unknown_copy,+ live,+ live_copy+ ]),+ false,+ )+ .create();+ page(&mut server, 2, 0, json!([caption]), false).create();+ page(&mut server, 4, 0, json!([first]), false).create();+ let hidden_request = page(&mut server, 3, 0, json!([]), false).expect(0).create();+ let add = server+ .mock("POST", "/collections/add-files")+ .expect(0)+ .create();+ let trash = server.mock("POST", "/files/trash").expect(0).create();+ let groups = home.json(&["photos", "duplicate", "list", "--all"]);+ assert_eq!(groups.as_array().unwrap().len(), 3);+ assert_eq!(+ home.json(&["photos", "duplicates", "list", "--all", "--offline"]),+ groups+ );+ assert_eq!(groups[0]["id"], "30");+ let one = groups+ .as_array()+ .unwrap()+ .iter()+ .find(|g| g["id"] == "10")+ .unwrap();+ assert_eq!(one["retainedId"], "11");+ assert_eq!(one["reclaimableBytes"], 100);+ assert_eq!(one["files"].as_array().unwrap().len(), 2);+ assert_eq!(+ one["files"][1]["albums"],+ json!([{"id":"4", "name":"Favorites"}, {"id":"1", "name":"Zeta"}])+ );+ assert_eq!(+ home.json(&["photos", "duplicate", "list", "--offline", "--all"]),+ groups+ );+ let limited = success(home.run(&["photos", "duplicate", "list", "--offline", "--limit", "1"]));+ let table = String::from_utf8_lossy(&limited.stdout);+ assert!(table.lines().any(|line| {+ line.split_whitespace().collect::<Vec<_>>() == ["ACTION", "ID", "NAME", "ALBUMS"]+ }));+ assert!(+ table+ .lines()+ .any(|line| { line.split_whitespace().take(2).collect::<Vec<_>>() == ["keep", "30"] })+ );+ assert!(String::from_utf8_lossy(&limited.stderr).contains("Showing first 1"));+ let preview = home.json(&[+ "photos",+ "duplicate",+ "remove",+ "10",+ "10",+ "--dry-run",+ "--no-input",+ ]);+ assert_eq!(+ preview,+ json!({"dryRun":true, "groupIds":["10"], "retainedIds":["11"], "trashIds":["10"], "reclaimableBytes":100})+ );+ let all = home.json(&["photos", "duplicate", "remove", "--all", "--dry-run"]);+ assert_eq!(all["trashIds"].as_array().unwrap().len(), 3);+ assert!(+ all["retainedIds"]+ .as_array()+ .unwrap()+ .contains(&json!("21"))+ );+ assert!(+ failure(&home.run(&["photos", "duplicate", "remove", "10", "99", "--yes"]))+ .contains("no duplicate group matches 99")+ );+ hidden_request.assert();+ add.assert();+ trash.assert();+ add.remove();+ let denied = server+ .mock("POST", "/collections/add-files")+ .with_status(403)+ .create();+ let result = home.run(&["photos", "duplicate", "remove", "10", "--yes", "--json"]);+ assert!(failure(&result).contains("duplicate removal did not finish"));+ assert!(result.stdout.is_empty());+ denied.assert();+ trash.assert();+}++#[test]+fn duplicates_refuse_incomplete_metadata_before_removal() {+ let mut server = mockito::Server::new();+ let home = TestHome::new();+ home.seed(&server.url());+ let key = Key::generate();+ server+ .mock("GET", "/collections/v2")+ .match_query(mockito::Matcher::Any)+ .with_body(json!({"collections":[collection(1, "Album", &key)]}).to_string())+ .create();+ let mut broken = fixture(12, &key, "one", json!({}));+ broken["metadata"]["encryptedData"] = json!("broken");+ page(+ &mut server,+ 1,+ 0,+ json!([+ fixture(10, &key, "one", json!({})),+ fixture(11, &key, "one", json!({})),+ broken+ ]),+ false,+ )+ .create();+ let trash = server.mock("POST", "/files/trash").expect(0).create();+ assert!(+ failure(&home.run(&["photos", "duplicate", "remove", "--all", "--yes"]))+ .contains("file 12 is unavailable")+ );+ trash.assert();+}++#[test]+fn duplicates_require_explicit_selection_and_confirmation_without_a_terminal() {+ let home = TestHome::new();+ for (args, expected) in [+ (+ vec!["--all", "--yes", "--offline"],+ "requires network access",+ ),+ (vec!["--yes", "--no-input"], "supply group IDs or --all"),+ (vec!["10", "--no-input"], "supply --yes"),+ (vec!["10", "--all", "--yes"], "cannot be used with"),+ ] {+ let mut command = vec!["photos", "duplicate", "remove"];+ command.extend(args);+ assert!(failure(&home.run(&command)).contains(expected));+ }+ assert_eq!(fs::read_dir(home.dir.path()).unwrap().count(), 0);+ let help = success(home.run(&["help", "photos", "duplicate", "remove"]));+ assert!(String::from_utf8_lossy(&help.stdout).contains("--dry-run"));+}diff --git a/rust/apps/cli/tests/cli/duplicates_museum.rs b/rust/apps/cli/tests/cli/duplicates_museum.rsnew file mode 100644index 0000000000..795662dbfd--- /dev/null+++ b/rust/apps/cli/tests/cli/duplicates_museum.rs@@ -0,0 +1,121 @@+use super::*;++#[test]+fn duplicates_select_remove_and_refresh_against_museum() -> TestResult {+ Museum::run_async(|origin| async move {+ let email = format!("duplicates-{}@example.org", Uuid::new_v4());+ let owner = create_account(&origin, &email).await;+ let (alpha, alpha_key) = create_album(&origin, &owner, "Alpha", "album").await;+ let (zeta, zeta_key) = create_album(&origin, &owner, "Zeta", "album").await;+ let (favorites, favorites_key) =+ create_album(&origin, &owner, "Favorites", "favorites").await;+ let (hidden, hidden_key) = create_album(&origin, &owner, "Hidden", "album").await;+ set_album_metadata(+ &origin,+ &owner,+ hidden,+ &hidden_key,+ "magic-metadata",+ json!({"visibility":2}),+ )+ .await;+ let original = b"same original uploaded separately";+ let mut files = Vec::new();+ for (album, key, hash) in [+ (alpha, &alpha_key, "one"),+ (zeta, &zeta_key, "one"),+ (favorites, &favorites_key, "one"),+ (alpha, &alpha_key, "two"),+ (zeta, &zeta_key, "two"),+ (hidden, &hidden_key, "one"),+ ] {+ files.push(upload_fixture(&origin, &owner, album, key, original, json!({+ "title":format!("{hash}.jpg"), "fileType":0, "creationTime":1_700_000_000_000_000i64, "hash":hash+ })).await);+ }+ let caption = blob::encrypt_json(&json!({"caption": "retain this caption"}), &files[1].1)?;+ reqwest::Client::new().put(format!("{origin}/files/public-magic-metadata"))+ .header("x-auth-token", b64::encode_url_safe(&owner.secrets.token))+ .header("x-client-package", "io.ente.photos")+ .json(&json!({"metadataList":[{"id":files[1].0,"magicMetadata":{+ "version":1,"count":1,"data":b64::encode(&caption.encrypted_data),"header":b64::encode(caption.decryption_header.as_bytes())+ }}]})).send().await?.error_for_status()?;+ let home = TestHome::new();+ login(&home, "photos", &email, &["--host", &origin, "--no-input"]);+ let groups = home.json(&["photos", "duplicate", "list", "--all"]);+ assert_eq!(groups.as_array().unwrap().len(), 2);+ let selected = files[..3].iter().map(|f| f.0).min().unwrap().to_string();+ let retained = files[1].0.to_string();+ let group = groups+ .as_array()+ .unwrap()+ .iter()+ .find(|g| g["id"] == selected)+ .unwrap();+ assert_eq!(group["files"].as_array().unwrap().len(), 3);+ assert_eq!(group["retainedId"], retained);+ let preview = home.json(&[+ "photos",+ "duplicate",+ "remove",+ &selected,+ "--dry-run",+ "--no-input",+ ]);+ assert_eq!(preview["trashIds"].as_array().unwrap().len(), 2);+ assert_eq!(home.json(&["photos", "duplicate", "list", "--all"]), groups);+ let removed = home.json(&[+ "photos",+ "duplicate",+ "remove",+ &selected,+ "--yes",+ "--no-input",+ ]);+ assert_eq!(removed["dryRun"], false);+ assert_eq!(removed["trashIds"], preview["trashIds"]);+ assert_eq!(removed["retainedIds"], json!([retained]));+ let remaining = home.json(&["photos", "duplicate", "list", "--offline"]);+ assert_eq!(remaining.as_array().unwrap().len(), 1);+ assert_ne!(remaining[0]["id"], selected);+ for album in [alpha, zeta, favorites] {+ let album = album.to_string();+ let file = home.json(&["photos", "file", "view", &retained, "--album", &album]);+ assert_eq!(file["caption"], "retain this caption");+ let target = home.dir.path().join(format!("{album}.jpg"));+ success(home.run(&[+ "photos",+ "file",+ "download",+ &retained,+ "--album",+ &album,+ "--output",+ target.to_str().unwrap(),+ ]));+ assert_eq!(fs::read(target)?, original);+ }+ let hidden_files = home.json(&["photos", "file", "list", "--album", &hidden.to_string()]);+ assert_eq!(hidden_files[0]["id"], files[5].0.to_string());+ let trash = home.json(&["photos", "api", "/trash/v2/diff", "--query", "sinceTime=0"]);+ let mut trash_ids: Vec<_> = trash["diff"]+ .as_array()+ .unwrap()+ .iter()+ .map(|item| item["file"]["id"].as_i64().unwrap())+ .collect();+ trash_ids.sort();+ let mut expected = vec![files[0].0, files[2].0];+ expected.sort();+ assert_eq!(trash_ids, expected);+ let all = home.json(&["photos", "duplicate", "remove", "--all", "--yes"]);+ assert_eq!(all["trashIds"].as_array().unwrap().len(), 1);+ assert_eq!(+ home.json(&["photos", "duplicate", "list", "--offline"]),+ json!([])+ );+ let empty = home.json(&["photos", "duplicate", "remove", "--all", "--yes"]);+ assert_eq!(empty["trashIds"], json!([]));+ Ok(())+ })+}diff --git a/rust/apps/cli/tests/cli/files.rs b/rust/apps/cli/tests/cli/files.rsindex f7f160860d..5e614c508f 100644--- a/rust/apps/cli/tests/cli/files.rs+++ b/rust/apps/cli/tests/cli/files.rs@@ -741,7 +741,7 @@ pub(super) fn remote_file(id: i64, collection_key: &Key, original: &[u8]) -> (Va)}-fn remote_file_with(+pub(super) fn remote_file_with(id: i64,collection_key: &Key,original: &[u8],diff --git a/rust/apps/cli/tests/cli/main.rs b/rust/apps/cli/tests/cli/main.rsindex f00050be3e..db2ad72940 100644--- a/rust/apps/cli/tests/cli/main.rs+++ b/rust/apps/cli/tests/cli/main.rs@@ -1191,6 +1191,8 @@ mod export_sources;mod files;+mod duplicates;+#[cfg(feature = "museum")]mod mail;diff --git a/rust/apps/cli/tests/cli/museum.rs b/rust/apps/cli/tests/cli/museum.rsindex 39e713f641..baf63aaa9c 100644--- a/rust/apps/cli/tests/cli/museum.rs+++ b/rust/apps/cli/tests/cli/museum.rs@@ -1480,3 +1480,6 @@ async fn set_album_metadata(#[path = "export.rs"]mod export;++#[path = "duplicates_museum.rs"]+mod duplicates;diff --git a/rust/crates/photos/src/files.rs b/rust/crates/photos/src/files.rsindex 66cfca872a..d0f1f2bef3 100644--- a/rust/crates/photos/src/files.rs+++ b/rust/crates/photos/src/files.rs@@ -25,6 +25,8 @@ pub struct File {pub modified_at_micros: Option<i64>,pub location: Option<Location>,pub caption: Option<String>,+ pub edited_name: Option<String>,+ pub edited_time: Option<i64>,pub hash: Option<String>,pub date_time: Option<String>,pub offset_time: Option<String>,@@ -199,7 +201,9 @@ pub fn interpret(id: remote.id,owner_id: remote.owner_id,updated_at_micros: remote.updation_time,- name: public.edited_name.unwrap_or(metadata.title),+ name: public.edited_name.clone().unwrap_or(metadata.title),+ edited_name: public.edited_name,+ edited_time: public.edited_time,kind,created_at_micros: public.edited_time.unwrap_or(metadata.creation_time),modified_at_micros: metadata.modification_time,