Skip to content

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

onedrive: make --onedrive-delta quicker when not listing from the drive root

Nick Craig-Wood3ac301eff1f7
diff --git a/backend/onedrive/onedrive.go b/backend/onedrive/onedrive.go
index 4673947e7..0df28a265 100644
--- a/backend/onedrive/onedrive.go
+++ b/backend/onedrive/onedrive.go
@@ -438,21 +438,12 @@ Setting this flag speeds up these things greatly:
rclone size onedrive:
rclone rc vfs/refresh recursive=true
-**However** the delta listing API **only** works at the root of the
-drive. If you use it not at the root then it recurses from the root
-and discards all the data that is not under the directory you asked
-for. So it will be correct but may not be very efficient.
-
-This is why this flag is not set as the default.
-
-As a rule of thumb if nearly all of your data is under rclone's root
-directory (the |root/directory| in |onedrive:root/directory|) then
-using this flag will be a big performance win. If your data is
-mostly not under the root then using this flag will be a big
-performance loss.
-
-It is recommended if you are mounting your onedrive at the root
-(or near the root when using crypt) and using rclone |rc vfs/refresh|.
+Rclone asks for the delta listing of the directory being listed. If
+the drive only supports delta listings at the root of the drive (as
+some older OneDrive for Business and SharePoint drives do) then
+rclone lists from the root and discards all the data that is not
+under the directory you asked for. So it will be correct but may not
+be very efficient.
`, "|", "`"),
Advanced: true,
}, {
@@ -1511,10 +1502,6 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
return err
}
- // ListR only works at the root of a onedrive, not on a folder
- // So we have to filter things outside of the root which is
- // inefficient.
-
list := list.NewHelper(callback)
// list a folder conventionally - used for shared folders
@@ -1590,17 +1577,26 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
return nil
}
- opts := rest.Opts{
- Method: "GET",
- Path: "/root/delta",
- Parameters: map[string][]string{
- // "token": {token},
+ listDelta := func(opts rest.Opts) error {
+ opts.Parameters = url.Values{
"$top": {fmt.Sprintf("%d", f.opt.ListChunk)},
- },
+ }
+ var result api.DeltaResponse
+ return f._listAll(ctx, "", false, false, fn, &opts, &result, &result.Value, &result.NextLink)
}
- var result api.DeltaResponse
- err = f._listAll(ctx, "", false, false, fn, &opts, &result, &result.Value, &result.NextLink)
+ err = listDelta(f.newOptsCall(directoryID, "GET", "/delta"))
+ // Some drives only support delta listings at the root of the
+ // drive, in which case list the whole drive and filter out the
+ // items outside dir.
+ var apiErr *api.Error
+ if err != nil && len(seen) == 0 && errors.As(err, &apiErr) {
+ fs.Debugf(f, "Delta listing of directory failed, listing from the root of the drive instead: %v", err)
+ err = listDelta(rest.Opts{
+ Method: "GET",
+ Path: "/root/delta",
+ })
+ }
if err != nil {
return err
}
diff --git a/docs/content/onedrive.md b/docs/content/onedrive.md
index 007e741ca..05f608bd2 100644
--- a/docs/content/onedrive.md
+++ b/docs/content/onedrive.md
@@ -326,10 +326,11 @@ This can be useful with `rclone mount` and [rclone rc vfs/refresh
recursive=true](/rc/#vfs-refresh)) to very quickly fill the mount with
information about all the files.
-The API used for the recursive listing (`ListR`) only supports listing
-from the root of the drive. This will become increasingly inefficient
-the further away you get from the root as rclone will have to discard
-files outside of the directory you are using.
+Rclone asks for the delta listing of the directory being listed. Some
+drives only support delta listings from the root of the drive, in
+which case rclone lists the whole drive and discards the files outside
+of the directory you are using, which is increasingly inefficient the
+further away from the root you get.
Some commands (like `rclone lsf -R`) will use `ListR` by default - you
can turn this off with `--disable ListR` if you need to.