Skip to content

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

yandex: add --yandex-upload-wait to fix 500 errors when uploading

Nick Craig-Wood92fbc85f108b
diff --git a/backend/yandex/yandex.go b/backend/yandex/yandex.go
index beff566b9..70aba8409 100644
--- a/backend/yandex/yandex.go
+++ b/backend/yandex/yandex.go
@@ -85,6 +85,21 @@ func init() {
Default: true,
Advanced: true,
Hide: fs.OptionHideConfigurator,
+ }, {
+ Name: "upload_wait",
+ Help: `Wait this long after an upload before setting the modification time.
+
+Yandex Disk finalizes an upload asynchronously on its servers after
+the upload has completed. If the modification time is set while this
+finalization is still in progress the server returns 500 Internal
+Server Error errors.
+
+If you are getting 500 errors on upload then setting this to 2s is
+normally enough to stop them, at the cost of slowing down uploads.
+
+Yandex support recommend a value of 1.5s - 3s.`,
+ Default: fs.Duration(0),
+ Advanced: true,
}}...),
})
}
@@ -95,6 +110,7 @@ type Options struct {
HardDelete bool `config:"hard_delete"`
Enc encoder.MultiEncoder `config:"encoding"`
SpoofUserAgent bool `config:"spoof_ua"`
+ UploadWait fs.Duration `config:"upload_wait"`
}
// Fs represents a remote yandex
@@ -1131,9 +1147,24 @@ func (o *Object) upload(ctx context.Context, in io.Reader, overwrite bool, mimeT
// Wait for PUT to be committed
if ur.OperationID != "" {
err = o.fs.waitForJob(ctx, rootURL+"/operations/"+ur.OperationID)
+ if err != nil {
+ return err
+ }
}
- return err
+ // Wait for the server to finalize the upload before the file's
+ // metadata is accessed. The operation status above can report
+ // success before the finalization has completed, so give the
+ // server some extra time if configured.
+ if o.fs.opt.UploadWait > 0 {
+ select {
+ case <-ctx.Done():
+ return ctx.Err()
+ case <-time.After(time.Duration(o.fs.opt.UploadWait)):
+ }
+ }
+
+ return nil
}
// Update the already existing object
diff --git a/docs/content/yandex.md b/docs/content/yandex.md
index 158888254..d5e741633 100644
--- a/docs/content/yandex.md
+++ b/docs/content/yandex.md
@@ -273,6 +273,14 @@ to twice the max size of file in GiB should be enough, so if you want
to upload a 30 GiB file set a timeout of `2 * 30 = 60m`, that is
`--timeout 60m`.
+If you get `500 Internal Server Error` errors just after uploads,
+particularly when uploading many files in parallel, then try setting
+`--yandex-upload-wait 2s`. Yandex Disk finalizes uploads
+asynchronously on its servers and can report an upload as complete
+slightly before the file is ready, so accessing the file's metadata
+too soon causes these errors. Yandex support recommend waiting
+1.5s - 3s after each upload.
+
Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription.
Token generation will work without a mail account, but Rclone won't be able to
complete any actions.