"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
docs: s3: add section on using too much memory #7974
Nick Craig-Wood9866d1c636d0
diff --git a/docs/content/s3.md b/docs/content/s3.mdindex 80e1ca34b..c4ccbab89 100644--- a/docs/content/s3.md+++ b/docs/content/s3.md@@ -5018,6 +5018,28 @@ nodes across the network.For more detailed comparison please check the documentation of the[storj](/storj) backend.+## Memory usage {memory}++The most common cause of rclone using lots of memory is a single+directory with millions of files in. Despite s3 not really having the+concepts of directories, rclone does the sync on a directory by+directory basis to be compatible with normal filing systems.++Rclone loads each directory into memory as rclone objects. Each rclone+object takes 0.5k-1k of memory, so approximately 1GB per 1,000,000+files, and the sync for that directory does not begin until it is+entirely loaded in memory. So the sync can take a long time to start+for large directories.++To sync a directory with 100,000,000 files in you would need approximately+100 GB of memory. At some point the amount of memory becomes difficult+to provide so there is+[a workaround for this](https://github.com/rclone/rclone/wiki/Big-syncs-with-millions-of-files)+which involves a bit of scripting.++At some point rclone will gain a sync mode which is effectively this+workaround but built in to rclone.+## Limitations`rclone about` is not supported by the S3 backend. Backends without@@ -5028,7 +5050,6 @@ remote.See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)-### Synology C2 Object Storage {#synology-c2}[Synology C2 Object Storage](https://c2.synology.com/en-global/object-storage/overview) provides a secure, S3-compatible, and cost-effective cloud storage solution without API request, download fees, and deletion penalty.