"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
s3: speed up multi-thread uploads by checksumming chunks in 1 MiB reads
Roman Butora68480093c4e
diff --git a/backend/s3/s3.go b/backend/s3/s3.goindex d35abdc43..7543d9441 100644--- a/backend/s3/s3.go+++ b/backend/s3/s3.go@@ -4727,8 +4727,23 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader// create checksum of buffer for integrity checking// currently there is no way to calculate the md5 without reading the chunk a 2nd time (1st read is in uploadMultipart)// possible in AWS SDK v2 with trailers?+ //+ // io.Copy reads with a 32 KiB buffer from a reader that implements+ // neither WriterTo nor a memory backed Read - which is what a local+ // file opened by a multi-thread copy is - so give it a bigger one to+ // keep the source reads a useful size. It isn't taken from the pool,+ // where it could wait for ever on --max-buffer-memory. A reader with+ // WriterTo, such as a pool.RW, doesn't need one.m := md5.New()- currentChunkSize, err := io.Copy(m, reader)+ var (+ currentChunkSize int64+ err error+ )+ if _, ok := reader.(io.WriterTo); ok {+ currentChunkSize, err = io.Copy(m, reader)+ } else {+ currentChunkSize, err = io.CopyBuffer(m, reader, make([]byte, multipart.BufferSize))+ }if err != nil {return -1, err}