"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
docs: add section on performance
Nick Craig-Wood18262df4656e
diff --git a/docs/content/docs.md b/docs/content/docs.mdindex a5f93d008..4398c9c5b 100644--- a/docs/content/docs.md+++ b/docs/content/docs.md@@ -778,6 +778,135 @@ modification time of the source object.Hashes are not included in system metadata as there is a well definedway of reading those already.+## Performance {#performance}++Rclone's defaults are fairly conservative. They are chosen to work on+small machines, slow links and with cloud providers which rate limit+heavily. If you have plenty of bandwidth, memory and a fast storage+system then rclone can usually be made to run a great deal faster by+adjusting a few flags.++The right values depend on the size of your files, your network and+the storage systems at each end, so treat the numbers below as+starting points. Change one thing at a time and measure the result.++### Measuring++Use `rclone copy` with [-P/--progress](#p-progress) to watch the+transfer rate while you are experimenting. For unattended runs log the+stats instead, e.g. `--stats 1m --stats-one-line --stats-log-level+NOTICE`.++A transfer is always limited by something: the network bandwidth, the+rate at which the storage system will accept requests, disk IO, CPU or+memory. Tuning is working out which limit you are hitting and raising+it until you hit the next one. Watch the network, disk, CPU and memory+use of the machine rclone is running on as well as rclone's stats.++### Lots of files++The time taken to transfer small files is dominated by the round trip+time of the requests rather than by the bandwidth, so the way to make+them go faster is to do more of them at once.++[--transfers](#transfers-int) sets the number of files transferred in+parallel. The default is 4. Double this until the transfer rate stops+increasing or the storage system starts returning errors. Values of 32+to 64 are common and values in the hundreds work well with the large+object storage systems (e.g. s3, azureblob, googlecloudstorage, b2)+when transferring lots of small files.++[--checkers](#checkers-int) sets the number of directory listings and+file comparisons done in parallel. The default is 8. Raising this+speeds up syncs where most of the files are already present on the+destination and stops the checking being the bottleneck when+`--transfers` is large. 16 to 64 is a reasonable range.++Not all storage systems like lots of parallel requests. If you see rate+limiting errors (e.g. HTTP 429 or 503 errors) or lots of low level+retries in the log then you have gone too far, so reduce `--transfers`+and `--checkers` or use [--tpslimit](#tpslimit-float) to cap the+request rate. Local disks, especially spinning ones, and network file+systems may get slower rather than faster with too much parallelism.++### Large files++A single stream often won't fill a fast link, so rclone can transfer a+large file as several chunks in parallel. These are called multi-thread+transfers and are used for files bigger than+[--multi-thread-cutoff](#multi-thread-cutoff) on backends which+support them.++The number of chunks transferred in parallel is set by+[--multi-thread-streams](#multi-thread-streams-int) (default 4) and+this works for all backends so increase this first to make large files+transfer faster.++The size of the chunks is set by the backend, usually with a+`--backend-chunk-size` flag, e.g. `--s3-chunk-size`. Backends without+a natural chunk size (e.g. local, smb) use+[--multi-thread-chunk-size](#multi-thread-chunk-size-sizesuffix).+Larger chunks mean fewer requests which increases throughput on big+files. See the docs for the backend you are using for details.++Chunked uploads are usually buffered in memory, so in the worst case+rclone will use++```text+--transfers * upload concurrency * chunk size+```++of memory for buffers. For example `--transfers 256` with an upload+concurrency of 8 and a chunk size of 16 MiB could use up to 32 GiB.+When you raise `--transfers` to speed up small files you may therefore+need to lower the upload concurrency, or set+[--max-buffer-memory](#max-buffer-memory) which limits the total+memory used for buffers and makes transfers wait when the limit is+reached.++[--use-mmap](#use-mmap) makes rclone return buffer memory to the OS as+soon as it has finished with it which is helpful when using lots of+transfers or large chunks.++### A mixture of large and small files++By default rclone transfers files in roughly the order it finds them.+With a mixture of file sizes this means that at times all the+transfers are large files, which fill the bandwidth but make few+requests, and at other times all the transfers are small files, which+make lots of requests but use little of the bandwidth.++Using [--order-by](#order-by-string) `size,mixed,50` makes half of the+transfers take the smallest files in the backlog and half of them take+the largest. This keeps both large and small files moving at once so+the bandwidth and the request rate are both kept full for the whole of+the run.++This works best if rclone has lots of files to choose from so use it+with a big [--max-backlog](#max-backlog-int), e.g. `--max-backlog+2000000`. Each item in the backlog uses in the order of 1 KiB of memory.++### Doing less work++Fewer requests to the storage system means a faster transfer.++- If the source and destination are on the same remote then rclone+ will use [server-side copy](#server-side-copy) where it is supported+ so the data doesn't need to be downloaded and uploaded again.+- [--fast-list](#fast-list) lists bucket-based remotes with fewer+ transactions at the cost of using more memory. This may be faster or+ slower than the default so it is worth testing.+- [--no-traverse](#no-traverse) stops rclone listing the destination+ which helps when copying a small number of files into a large+ destination.+- [--size-only](#size-only) or [--checksum](#c-checksum) avoid reading+ the modification time. On some object storage systems (e.g. s3,+ swift) this needs an extra request for each file. Alternatively use+ [--use-server-modtime](#use-server-modtime) with `--update`.+- [--no-check-dest](#no-check-dest) skips checking the destination+ entirely. Only use this if you know that none of the files are+ there already.+## OptionsRclone has a number of options to control its behaviour. These are