"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
docs: add section for improving performance for s3
Pawel Paluchada70088b11e6
diff --git a/docs/content/s3.md b/docs/content/s3.mdindex db52f5c26..9031e57fe 100644--- a/docs/content/s3.md+++ b/docs/content/s3.md@@ -401,6 +401,38 @@ there for more details.Setting this flag increases the chance for undetected upload failures.+### Increasing performance++#### Using server-side copy++If you are copying objects between S3 buckets in the same region, you should+use server-side copy.+This is much faster than downloading and re-uploading the objects, as no data is transferred.++For rclone to use server-side copy, you must use the same remote for the source and destination.++ rclone copy s3:source-bucket s3:destination-bucket++When using server-side copy, the performance is limited by the rate at which rclone issues+API requests to S3.+See below for how to increase the number of API requests rclone makes.++#### Increasing the rate of API requests++You can increase the rate of API requests to S3 by increasing the parallelism using `--transfers` and `--checkers`+options.++Rclone uses a very conservative defaults for these settings, as not all providers support high rates of requests.+Depending on your provider, you can increase significantly the number of transfers and checkers.++For example, with AWS S3, if you can increase the number of checkers to values like 200.+If you are doing a server-side copy, you can also increase the number of transfers to 200.++ rclone sync --transfers 200 --checkers 200 --checksum s3:source-bucket s3:destination-bucket++You will need to experiment with these values to find the optimal settings for your setup.++### VersionsWhen bucket versioning is enabled (this can be done with rclone with