Skip to content

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

s3: speed up multi-thread uploads by checksumming chunks in 1 MiB reads

Roman Butora68480093c4e
diff --git a/backend/s3/s3.go b/backend/s3/s3.go
index d35abdc43..7543d9441 100644
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -4727,8 +4727,23 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader
// create checksum of buffer for integrity checking
// currently there is no way to calculate the md5 without reading the chunk a 2nd time (1st read is in uploadMultipart)
// possible in AWS SDK v2 with trailers?
+ //
+ // io.Copy reads with a 32 KiB buffer from a reader that implements
+ // neither WriterTo nor a memory backed Read - which is what a local
+ // file opened by a multi-thread copy is - so give it a bigger one to
+ // keep the source reads a useful size. It isn't taken from the pool,
+ // where it could wait for ever on --max-buffer-memory. A reader with
+ // WriterTo, such as a pool.RW, doesn't need one.
m := md5.New()
- currentChunkSize, err := io.Copy(m, reader)
+ var (
+ currentChunkSize int64
+ err error
+ )
+ if _, ok := reader.(io.WriterTo); ok {
+ currentChunkSize, err = io.Copy(m, reader)
+ } else {
+ currentChunkSize, err = io.CopyBuffer(m, reader, make([]byte, multipart.BufferSize))
+ }
if err != nil {
return -1, err
}