"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
yandex: add --yandex-upload-wait to fix 500 errors when uploading
Nick Craig-Wood92fbc85f108b
diff --git a/backend/yandex/yandex.go b/backend/yandex/yandex.goindex beff566b9..70aba8409 100644--- a/backend/yandex/yandex.go+++ b/backend/yandex/yandex.go@@ -85,6 +85,21 @@ func init() {Default: true,Advanced: true,Hide: fs.OptionHideConfigurator,+ }, {+ Name: "upload_wait",+ Help: `Wait this long after an upload before setting the modification time.++Yandex Disk finalizes an upload asynchronously on its servers after+the upload has completed. If the modification time is set while this+finalization is still in progress the server returns 500 Internal+Server Error errors.++If you are getting 500 errors on upload then setting this to 2s is+normally enough to stop them, at the cost of slowing down uploads.++Yandex support recommend a value of 1.5s - 3s.`,+ Default: fs.Duration(0),+ Advanced: true,}}...),})}@@ -95,6 +110,7 @@ type Options struct {HardDelete bool `config:"hard_delete"`Enc encoder.MultiEncoder `config:"encoding"`SpoofUserAgent bool `config:"spoof_ua"`+ UploadWait fs.Duration `config:"upload_wait"`}// Fs represents a remote yandex@@ -1131,9 +1147,24 @@ func (o *Object) upload(ctx context.Context, in io.Reader, overwrite bool, mimeT// Wait for PUT to be committedif ur.OperationID != "" {err = o.fs.waitForJob(ctx, rootURL+"/operations/"+ur.OperationID)+ if err != nil {+ return err+ }}- return err+ // Wait for the server to finalize the upload before the file's+ // metadata is accessed. The operation status above can report+ // success before the finalization has completed, so give the+ // server some extra time if configured.+ if o.fs.opt.UploadWait > 0 {+ select {+ case <-ctx.Done():+ return ctx.Err()+ case <-time.After(time.Duration(o.fs.opt.UploadWait)):+ }+ }++ return nil}// Update the already existing objectdiff --git a/docs/content/yandex.md b/docs/content/yandex.mdindex 158888254..d5e741633 100644--- a/docs/content/yandex.md+++ b/docs/content/yandex.md@@ -273,6 +273,14 @@ to twice the max size of file in GiB should be enough, so if you wantto upload a 30 GiB file set a timeout of `2 * 30 = 60m`, that is`--timeout 60m`.+If you get `500 Internal Server Error` errors just after uploads,+particularly when uploading many files in parallel, then try setting+`--yandex-upload-wait 2s`. Yandex Disk finalizes uploads+asynchronously on its servers and can report an upload as complete+slightly before the file is ready, so accessing the file's metadata+too soon causes these errors. Yandex support recommend waiting+1.5s - 3s after each upload.+Having a Yandex Mail account is mandatory to use the Yandex.Disk subscription.Token generation will work without a mail account, but Rclone won't be able tocomplete any actions.