"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
serve s3: upload all multipart uploads via the VFS
Nick Craig-Woodb4db289d0a02
diff --git a/cmd/serve/s3/backend.go b/cmd/serve/s3/backend.goindex afa4e2cae..8c135e969 100644--- a/cmd/serve/s3/backend.go+++ b/cmd/serve/s3/backend.go@@ -371,10 +371,10 @@ func (b *s3Backend) PutObject(// (never the object at fp) and any stale VFS state for fp.cleanup := func() {if tmpFp != fp {- b.forgetPath(ctx, tmpFp)+ b.forgetPath(_vfs, tmpFp)_ = _vfs.Remove(tmpFp)} else {- b.forgetPath(ctx, fp)+ b.forgetPath(_vfs, fp)}}diff --git a/cmd/serve/s3/multipart.go b/cmd/serve/s3/multipart.goindex dd38e3c3d..304aaf69f 100644--- a/cmd/serve/s3/multipart.go+++ b/cmd/serve/s3/multipart.go@@ -1,13 +1,17 @@// Multipart upload support for serve s3.//-// Multipart uploads received by serve s3 are streamed, in part-number order,-// into a single PutStream upload to the underlying Fs, so the whole file is-// never buffered in memory. This implements the gofakes3.MultipartBackend+// Multipart uploads received by serve s3 are written, in part-number order,+// through the VFS, exactly like a plain PutObject: to a temporary object+// which is renamed into place on completion, so the object at the key only+// ever changes atomically on success. With the default --vfs-cache-mode off+// the parts stream through the VFS into a single upload to the remote; with+// --vfs-cache-mode writes or above they are buffered in the VFS cache and+// uploaded by its write-back. This implements the gofakes3.MultipartBackend// interface on s3Backend.//-// When streaming is disabled (--disable-multipart-streaming) or the Fs has no-// PutStream, ErrMultipartUploadNotSupported is returned so that gofakes3 falls-// back to buffering the parts in memory.+// When streaming is disabled (--disable-multipart-streaming),+// ErrMultipartUploadNotSupported is returned so that gofakes3 falls back to+// buffering the parts in memory.package s3@@ -23,46 +27,47 @@ import ("sort""strings""sync"- "time""github.com/google/uuid""github.com/ncw/swift/v2""github.com/rclone/gofakes3""github.com/rclone/rclone/fs"- "github.com/rclone/rclone/fs/object""github.com/rclone/rclone/fs/operations""github.com/rclone/rclone/lib/multipart""github.com/rclone/rclone/lib/pool"+ "github.com/rclone/rclone/vfs"+ "github.com/rclone/rclone/vfs/vfscommon")// multipartUploadPrefix is prepended to the leaf name of the temporary object// a streamed multipart upload is written to before it is moved into place.const multipartUploadPrefix = tempObjectPrefix + "multipart_"-// multipartUpload tracks one in-flight S3 multipart upload that is being-// streamed, in part order, into a single PutStream upload to the underlying Fs.+// multipartUpload tracks one in-flight S3 multipart upload. The parts are+// written, in part-number order, into fh - a VFS file handle which either+// streams straight through to the remote (the default) or is backed by the+// VFS cache (when the VFS is caching writes).type multipartUpload struct {bucket, key stringfp string // final object path- streamFp string // path the parts are streamed to (fp when the backend uploads atomically)+ streamFp string // path the parts are written to (fp when the remote has no server-side move or copy)meta map[string]string- pipeW *io.PipeWriter // parts are streamed here, in part-number order+ fh io.WriteCloser // sink the in-order parts are written to+ vfs *vfs.VFS // the VFS fh was created on, used for all later operationsmu sync.Mutexcond *sync.Cond // signalled when buffered shrinks, nextPart advances or the upload closespartMD5s map[int][]byte // raw MD5 sums per part (for the final S3 multipart ETag)partSizes map[int]int64 // observed part sizesclosed bool+ aborted bool // closed by abort, so nothing was committed- putCancel context.CancelFunc // cancels the background PutStream- putDone chan struct{} // closed when the background PutStream returns- putErr error // PutStream result (read only after putDone is closed)- nextPart int // next part number to stream (1-based)- streamBuf map[int]*pool.RW // parts received ahead of nextPart, awaiting their turn- pumping bool // a goroutine is currently writing to the pipe- buffered int64 // bytes of parts admitted but not yet streamed or released- bufferLimit int64 // max buffered before parts ahead of nextPart must wait (<= 0 for no limit)+ nextPart int // next part number to stream (1-based)+ streamBuf map[int]*pool.RW // parts received ahead of nextPart, awaiting their turn+ pumping bool // a goroutine is currently writing to the sink+ buffered int64 // bytes of parts admitted but not yet streamed or released+ bufferLimit int64 // max buffered before parts ahead of nextPart must wait (<= 0 for no limit)}// newMultipartUpload allocates an upload struct.@@ -92,22 +97,26 @@ func (b *s3Backend) loadUpload(uploadID gofakes3.UploadID) (*multipartUpload, erreturn v.(*multipartUpload), nil}-// CreateMultipartUpload begins a new multipart upload that streams the parts,-// in part-number order, into a single PutStream upload to the underlying Fs.+// CreateMultipartUpload begins a new multipart upload.//-// Backends that upload atomically (PartialUploads=false) are streamed-// straight to the final object. An aborted or failed upload never-// makes a partial object visible or disturbs a pre-existing one.-// Backends where a partial upload is visible (PartialUploads=true)-// are instead streamed to a temporary object that is moved into-// place, server-side, on completion, giving the same atomic-// behaviour.+// The parts are written, in part-number order, through the VFS to a temporary+// object which is renamed into place on completion. With the default+// --vfs-cache-mode off the write streams through to the remote as the parts+// arrive; with --vfs-cache-mode writes or above it lands in the VFS cache and+// is uploaded by the write-back. Either way an aborted or failed upload never+// makes a partial object visible at the final path or disturbs a pre-existing+// one.//-// If streaming is disabled (--disable-multipart-streaming), the Fs has no-// PutStream, or a non-atomic Fs can't move/copy objects server-side,-// ErrMultipartUploadNotSupported is returned so that gofakes3 falls back to-// buffering the whole upload in memory; a one-off NOTICE warns about the-// memory use.+// On a remote with no server-side move or copy the parts are written straight+// to the final object instead, trading some atomicity for never buffering in+// memory: the in-flight upload is visible at the key, and a failed or aborted+// upload can leave partial data there when the remote doesn't upload atomically+// or the VFS is caching writes (a write to the cache can't be abandoned).+//+// With --disable-multipart-streaming, ErrMultipartUploadNotSupported is+// returned so that gofakes3 falls back to buffering the whole upload in+// memory; a one-off NOTICE warns about the memory use. A caching VFS needs+// no streaming support, so it ignores the flag.func (b *s3Backend) CreateMultipartUpload(ctx context.Context, bucketName, objectName string, meta map[string]string) (gofakes3.UploadID, error) {_vfs, err := b.s.getVFS(ctx)if err != nil {@@ -117,11 +126,9 @@ func (b *s3Backend) CreateMultipartUpload(ctx context.Context, bucketName, objecreturn "", gofakes3.BucketNotFound(bucketName)}- f := _vfs.Fs()- features := f.Features()- if reason := b.noStreamingReason(f); reason != "" {+ if b.s.opt.DisableMultipartStreaming && _vfs.Opt.CacheMode < vfscommon.CacheModeMinimal {b.warnInMemoryOnce.Do(func() {- fs.Logf(nil, "serve s3: buffering multipart uploads in memory because %s - this may use a lot of memory", reason)+ fs.Logf(nil, "serve s3: buffering multipart uploads in memory because --disable-multipart-streaming is set - this may use a lot of memory")})return "", gofakes3.ErrMultipartUploadNotSupported}@@ -139,47 +146,28 @@ func (b *s3Backend) CreateMultipartUpload(ctx context.Context, bucketName, objecuploadID := gofakes3.UploadID(uuid.New().String())streamFp := fp- // If partial uploads visible, stream to temporary object- if features.PartialUploads {+ // Write to a temporary object moved into place on completion if the+ // remote supports server-side move (if not write directly to the final+ // object). Unlike a plain PutObject all remotes use a temporary object.+ // S3 semantics say the key must not show it until it completes. This is+ // at the cost of a server-side copy and delete where the remote has no+ // server side move.+ if operations.CanServerSideMove(_vfs.Fs()) {streamFp = path.Join(objectDir, multipartUploadPrefix+string(uploadID))}up := newMultipartUpload(bucketName, objectName, fp, streamFp, meta, int64(b.s.opt.MultipartStreamingBufferLimit))-- src := object.NewStaticObjectInfo(streamFp, time.Now(), -1, true, nil, f)- pr, pw := io.Pipe()- // Use a context that outlives this request (it's cancelled on abort) but- // keeps its values.- putCtx, cancel := context.WithCancel(context.WithoutCancel(ctx))- up.pipeW = pw- up.putCancel = cancel- up.putDone = make(chan struct{})- go func() {- _, err := features.PutStream(putCtx, pr, src)- up.putErr = err- _ = pr.CloseWithError(err)- close(up.putDone)- }()+ fh, err := _vfs.Create(streamFp)+ if err != nil {+ return "", err+ }+ up.fh = fh+ up.vfs = _vfsb.multipartUploads.Store(uploadID, up)return uploadID, nil}-// noStreamingReason returns a non-empty reason why streamed multipart uploads-// can't be used for f, or "" if they can.-func (b *s3Backend) noStreamingReason(f fs.Fs) string {- switch {- case b.s.opt.DisableMultipartStreaming:- return "--disable-multipart-streaming is set"- case f.Features().PutStream == nil:- return "this backend doesn't support streaming uploads"- case f.Features().PartialUploads && !operations.CanServerSideMove(f):- return "this backend can't upload atomically and has no server-side move or copy"- default:- return ""- }-}-// UploadPart writes a single part from the S3 client into the streaming upload.func (b *s3Backend) UploadPart(ctx context.Context, bucketName, objectName string, uploadID gofakes3.UploadID, partNumber int, contentLength int64, body io.Reader) (string, error) {up, err := b.loadUpload(uploadID)@@ -222,7 +210,7 @@ func (b *s3Backend) UploadPart(ctx context.Context, bucketName, objectName strin// client sends parts faster than the backend drains them.//// The next part the stream needs (and any retry of an earlier one) is always-// admitted so the pipe can keep draining; so is a single part bigger than the+// admitted so the sink can keep draining; so is a single part bigger than the// limit when the buffer is empty, to guarantee progress. Reserved bytes are// returned with release, or by the pump as the part is streamed.func (up *multipartUpload) waitForTurn(partNumber int, size int64) error {@@ -249,11 +237,11 @@ func (up *multipartUpload) release(size int64) {up.mu.Unlock()}-// streamPart records a part and streams the parts into the pipe in order.+// streamPart records a part and streams the parts into the sink in order.//// Parts must be uploaded in ascending, contiguous part-number order. A part// that arrives ahead of the next expected one is buffered until its turn; the-// parts are then pumped into the pipe in order. Whichever goroutine finds the+// parts are then pumped into the sink in order. Whichever goroutine finds the// next part available does the pumping, so concurrent (but in-order) clients// are tolerated, with the buffering bounded by waitForTurn.//@@ -298,13 +286,21 @@ func (up *multipartUpload) streamPart(partNumber int, size int64, md5Sum []byte,up.streamBuf[partNumber] = rwif up.pumping {- // Another goroutine owns the pipe and will pump this part in turn.+ // Another goroutine owns the sink and will pump this part in turn.up.mu.Unlock()return nil}up.pumping = truefor {+ if up.closed {+ // Aborted while pumping: the sink is closed and any+ // parts still buffered were released by the abort, so+ // report the upload gone rather than success.+ up.pumping = false+ up.mu.Unlock()+ return gofakes3.ErrNoSuchUpload+ }prw, ok := up.streamBuf[up.nextPart]if !ok {up.pumping = false@@ -315,14 +311,21 @@ func (up *multipartUpload) streamPart(partNumber int, size int64, md5Sum []byte,psize := prw.Size()up.mu.Unlock()- err := pipePart(up.pipeW, prw)+ err := pipePart(up.fh, prw)_ = prw.Close()if err != nil {up.mu.Lock()up.pumping = falseup.buffered -= psize+ closed := up.closedup.cond.Broadcast()up.mu.Unlock()+ if closed {+ // The write failed because the upload was aborted while+ // this part was being pumped: report the upload gone+ // rather than the sink's ECLOSED as an internal error.+ return gofakes3.ErrNoSuchUpload+ }return err}@@ -333,7 +336,7 @@ func (up *multipartUpload) streamPart(partNumber int, size int64, md5Sum []byte,}}-// pipePart writes the whole of rw into w (the pipe).+// pipePart writes the whole of rw into w (the sink).func pipePart(w io.Writer, rw *pool.RW) error {if _, err := rw.Seek(0, io.SeekStart); err != nil {return err@@ -342,110 +345,95 @@ func pipePart(w io.Writer, rw *pool.RW) error {return err}-// CompleteMultipartUpload finalises a streamed multipart upload. It closes the-// pipe (so PutStream finishes), registers the new file with the VFS, computes-// the S3-style multipart ETag, and stores the user metadata so HeadObject and+// CompleteMultipartUpload finalises a multipart upload. It closes the sink,+// committing the upload, renames the temporary object into place, computes the+// S3-style multipart ETag, and stores the user metadata so HeadObject and// GetObject see the same fields the in-memory PutObject path produces.func (b *s3Backend) CompleteMultipartUpload(ctx context.Context, bucketName, objectName string, uploadID gofakes3.UploadID, input *gofakes3.CompleteMultipartUploadRequest) (gofakes3.VersionID, string, error) {up, err := b.loadUpload(uploadID)if err != nil {return "", "", err}- defer b.multipartUploads.Delete(uploadID)if err := up.validate(input); err != nil {- _ = up.abort(ctx)- b.forgetPath(ctx, up.streamFp)+ b.multipartUploads.Delete(uploadID)+ _ = up.abort()+ b.discardUpload(up)return "", "", err}- // All parts must have been streamed: contiguous part numbers from 1 with- // nothing left buffered. A leftover means the client used non-contiguous- // part numbers, which the in-order stream can't place.- up.mu.Lock()- streamed := up.nextPart - 1- total := len(up.partSizes)- leftover := len(up.streamBuf)- up.mu.Unlock()- if leftover != 0 || streamed != total {- _ = up.abort(ctx)- b.forgetPath(ctx, up.streamFp)- return "", "", gofakes3.ErrInvalidPart- }-- if err := up.close(ctx); err != nil {- b.forgetPath(ctx, up.streamFp)+ // close commits the upload, failing with the upload left open if the+ // streamed parts don't form the complete object; abort then tears it+ // down. (After a successful or failed commit the abort is a no-op.)+ if err := up.close(); err != nil {+ b.multipartUploads.Delete(uploadID)+ _ = up.abort()+ b.discardUpload(up)return "", "", err}- _vfs, err := b.s.getVFS(ctx)- if err != nil {- return "", "", err- }-- // If the parts were streamed to a temporary object move it into place+ // Rename the temporary object into place: on a caching VFS the+ // write-back then uploads it under the final name, otherwise it is+ // moved server-side on the remote. On failure the upload record is+ // kept, because gofakes3 keeps its own record when the backend errors+ // so that the client can retry the CompleteMultipartUpload: the+ // committed close is idempotent, so the retry just renames again.if up.streamFp != up.fp {- if err := b.moveIntoPlace(ctx, _vfs.Fs(), up.streamFp, up.fp); err != nil {- b.forgetPath(ctx, up.streamFp)- b.forgetPath(ctx, up.fp)+ if err := up.vfs.Rename(up.streamFp, up.fp); err != nil {return "", "", err}- b.forgetPath(ctx, up.streamFp)}- b.forgetPath(ctx, up.fp)+ b.multipartUploads.Delete(uploadID)b.meta.Store(up.fp, up.meta)if val, ok := up.meta["X-Amz-Meta-Mtime"]; ok {if ti, err := swift.FloatStringToTime(val); err == nil {b.storeModtime(up.fp, up.meta, val)- _ = _vfs.Chtimes(up.fp, ti, ti)+ _ = up.vfs.Chtimes(up.fp, ti, ti)}} else if val, ok := up.meta["mtime"]; ok {if ti, err := swift.FloatStringToTime(val); err == nil {b.storeModtime(up.fp, up.meta, val)- _ = _vfs.Chtimes(up.fp, ti, ti)+ _ = up.vfs.Chtimes(up.fp, ti, ti)}}return "", up.multipartETag(input), nil}-// AbortMultipartUpload tears down an in-progress upload, asking the background-// PutStream to discard any data already sent.+// AbortMultipartUpload tears down an in-progress upload, discarding any data+// already received.func (b *s3Backend) AbortMultipartUpload(ctx context.Context, bucketName, objectName string, uploadID gofakes3.UploadID) error {up, err := b.loadUpload(uploadID)if err != nil {return err}defer b.multipartUploads.Delete(uploadID)- err = up.abort(ctx)- // An atomic backend leaves the final object untouched by the aborted- // upload; a non-atomic one only ever wrote the temporary object. Either way- // invalidating streamFp is enough.- b.forgetPath(ctx, up.streamFp)- return err+ if err := up.abort(); err != nil {+ fs.Errorf(up.fp, "aborting multipart upload: %v", err)+ }+ b.discardUpload(up)+ return nil}-// moveIntoPlace moves the temporary object srcFp to its final path dstFp on f,-// server-side, overwriting any object already there.-func (b *s3Backend) moveIntoPlace(ctx context.Context, f fs.Fs, srcFp, dstFp string) error {- srcObj, err := f.NewObject(ctx, srcFp)- if err != nil {- return fmt.Errorf("failed to find uploaded object: %w", err)- }- if _, err := operations.Move(ctx, f, nil, dstFp, srcObj); err != nil {- return fmt.Errorf("failed to move uploaded object into place: %w", err)+// discardUpload cleans up after a failed or aborted upload, removing the+// temporary object and any stale VFS state for it. It never removes the+// object at the final path: when the parts were written straight to the+// final object (a remote with no server-side move or copy) it may hold a+// pre-existing object the abandoned streaming write never disturbed, so+// only the VFS's view of the path is refreshed. (A caching VFS on such a+// remote commits the partial data instead - see abort.)+func (b *s3Backend) discardUpload(up *multipartUpload) {+ b.forgetPath(up.vfs, up.streamFp)+ if up.streamFp == up.fp {+ return}- return nil+ _ = up.vfs.Remove(up.streamFp)}// forgetPath invalidates the parent directory's cached VFS listing so that// subsequent VFS Stat / List calls re-read fp from the underlying Fs.-func (b *s3Backend) forgetPath(ctx context.Context, fp string) {- _vfs, err := b.s.getVFS(ctx)- if err != nil {- return- }+func (b *s3Backend) forgetPath(_vfs *vfs.VFS, fp string) {if root, err := _vfs.Root(); err == nil {root.ForgetPath(fp, fs.EntryObject)}@@ -478,39 +466,59 @@ func (up *multipartUpload) validate(input *gofakes3.CompleteMultipartUploadRequereturn nil}-// close finalises the upload by signalling EOF to the background PutStream and-// waiting for it to finish.-func (up *multipartUpload) close(ctx context.Context) error {+// close finalises the upload, committing it: closing the sink completes the+// streaming upload to the remote, or the write to the VFS cache whose+// write-back then uploads it.+//+// The upload is only committed if the streamed parts form the complete+// object: contiguous part numbers from 1 with nothing left buffered (a+// leftover means the client used non-contiguous part numbers, which the+// in-order stream can't place). Otherwise ErrInvalidPart is returned and+// the upload is left open. The check and the commit share one critical+// section so no late part can slip into the stream between them.+//+// Returns ErrNoSuchUpload if the upload was aborted by a concurrent+// AbortMultipartUpload, in which case nothing has been committed - the+// upload must not be reported as complete.+func (up *multipartUpload) close() error {up.mu.Lock()if up.closed {+ aborted := up.abortedup.mu.Unlock()+ if aborted {+ return gofakes3.ErrNoSuchUpload+ }return nil}+ if len(up.streamBuf) != 0 || up.nextPart-1 != len(up.partSizes) {+ up.mu.Unlock()+ return gofakes3.ErrInvalidPart+ }up.closed = trueup.cond.Broadcast()up.mu.Unlock()- err := up.pipeW.Close()- <-up.putDone- up.putCancel()- if up.putErr != nil {- return up.putErr- }- return err+ return up.fh.Close()}-// errMultipartAborted makes the background PutStream fail when an upload is-// aborted, so it tears down its partial object instead of completing.+// errMultipartAborted is the reason an aborted upload's write is abandoned+// with, so the streaming upload fails instead of committing what it has.var errMultipartAborted = errors.New("serve s3: multipart upload aborted")-// abort cancels the background PutStream and releases any buffered parts.-func (up *multipartUpload) abort(ctx context.Context) error {+// abort tears down the upload and releases any buffered parts. The write is+// abandoned so nothing is committed; a sink which can't abandon (a caching+// one) is closed normally, committing what it has to the cache - the caller+// then removes its temporary file, except on a remote with no server-side+// move or copy, where the parts went straight to the final path and the+// partial data is left to be written back.+func (up *multipartUpload) abort() error {up.mu.Lock()if up.closed {up.mu.Unlock()return nil}up.closed = true+ up.aborted = truestreamBuf := up.streamBufup.streamBuf = nilup.cond.Broadcast()@@ -520,12 +528,11 @@ func (up *multipartUpload) abort(ctx context.Context) error {_ = rw.Close()}- // Fail the background PutStream (so it discards its partial object) and- // wait for it to return.- up.putCancel()- _ = up.pipeW.CloseWithError(errMultipartAborted)- <-up.putDone- return nil+ if aborter, ok := up.fh.(interface{ CloseWithError(error) error }); ok {+ _ = aborter.CloseWithError(errMultipartAborted)+ return nil+ }+ return up.fh.Close()}// multipartETag computes the S3 multipart ETag for the assembled object:diff --git a/cmd/serve/s3/multipart_test.go b/cmd/serve/s3/multipart_test.goindex 67c3a69bb..a0b585b56 100644--- a/cmd/serve/s3/multipart_test.go+++ b/cmd/serve/s3/multipart_test.go@@ -22,6 +22,7 @@ import ("github.com/rclone/rclone/cmd/serve/proxy""github.com/rclone/rclone/fs""github.com/rclone/rclone/fs/object"+ "github.com/rclone/rclone/fs/operations""github.com/rclone/rclone/fstest""github.com/rclone/rclone/lib/multipart""github.com/rclone/rclone/lib/random"@@ -44,9 +45,8 @@ func newMultipartTestServer(t *testing.T, disableStreaming bool) (*minio.Core, f// newMultipartTestServerBacking is like newMultipartTestServer but backed by// the named remote (a fresh local temp directory if empty). ":memory:" gives-// an atomic (PartialUploads=false) backing, so the streamed-straight-to-the--// destination path is exercised as well as the temporary-object path that-// local (PartialUploads=true) uses.+// an atomic (PartialUploads=false) backing, so both flavours of remote go+// through the temporary-object-plus-rename path.func newMultipartTestServerBacking(t *testing.T, backing string, disableStreaming bool) (*minio.Core, fs.Fs, string) {return newMultipartTestServerOpt(t, backing, disableStreaming, nil)}@@ -54,8 +54,22 @@ func newMultipartTestServerBacking(t *testing.T, backing string, disableStreamin// newMultipartTestServerOpt is like newMultipartTestServerBacking but also// applies tweak (if non-nil) to the server Options before starting it.func newMultipartTestServerOpt(t *testing.T, backing string, disableStreaming bool, tweak func(*Options)) (*minio.Core, fs.Fs, string) {+ return newMultipartTestServerVFS(t, backing, disableStreaming, tweak, nil)+}++// newMultipartTestServerVFS is like newMultipartTestServerOpt but also+// overrides the VFS options (nil for the defaults) and disables the named+// features on the backing remote. A backing with features disabled must have+// a unique config string (e.g. a distinct description=) so an active VFS+// wrapping a fully-featured instance of the same remote isn't reused.+func newMultipartTestServerVFS(t *testing.T, backing string, disableStreaming bool, tweak func(*Options), vfsOpt *vfscommon.Options, disableFeatures ...string) (*minio.Core, fs.Fs, string) {fstest.Initialise()ctx := context.Background()+ if len(disableFeatures) > 0 {+ var ci *fs.ConfigInfo+ ctx, ci = fs.AddConfig(ctx)+ ci.DisableFeatures = disableFeatures+ }if backing == "" {backing = t.TempDir()}@@ -65,10 +79,13 @@ func newMultipartTestServerOpt(t *testing.T, backing string, disableStreaming bo// process-wide store, so a fixed name would leak objects between tests.bucket := fmt.Sprintf("test-%d", testBackingCounter.Add(1))require.NoError(t, f.Mkdir(ctx, bucket))+ if vfsOpt == nil {+ vfsOpt = &vfscommon.Opt+ }// The VFS is cached per remote (fs.ConfigString), so a shared ":memory:"// server reuses a VFS whose cached root listing predates the bucket just// created; forget it so the new bucket is visible.- if root, err := vfs.New(ctx, f, &vfscommon.Opt).Root(); err == nil {+ if root, err := vfs.New(ctx, f, vfsOpt).Root(); err == nil {root.ForgetAll()}@@ -81,7 +98,7 @@ func newMultipartTestServerOpt(t *testing.T, backing string, disableStreaming boif tweak != nil {tweak(&opt)}- w, err := newServer(ctx, f, &opt, &vfscommon.Opt, &proxy.Opt)+ w, err := newServer(ctx, f, &opt, vfsOpt, &proxy.Opt)require.NoError(t, err)go func() { _ = w.Serve() }()t.Cleanup(func() { _ = w.Shutdown() })@@ -452,32 +469,43 @@ func TestMultipartBufferLimit(t *testing.T) {assert.Equal(t, want, readObject(t, f, bucket, object))}+// stubSink is a multipartUpload sink which records how it was closed.+type stubSink struct {+ closed bool+ abortErr error // the reason passed to CloseWithError+}++func (s *stubSink) Write(p []byte) (int, error) { return len(p), nil }++func (s *stubSink) Close() error {+ s.closed = true+ return nil+}++func (s *stubSink) CloseWithError(err error) error {+ s.closed = true+ s.abortErr = err+ return nil+}+// TestMultipartAbortDuringUploadPart aborts the upload between a part's// waitForTurn and its streamPart - as happens when the abort arrives while// the part body is still being received from the client - and checks that// streamPart fails cleanly instead of panicking on the torn-down upload.func TestMultipartAbortDuringUploadPart(t *testing.T) {up := newMultipartUpload("bucket", "key", "bucket/key", "bucket/key", nil, 0)-- // Mimic CreateMultipartUpload's background PutStream with a stub consumer.- pr, pw := io.Pipe()- _, cancel := context.WithCancel(context.Background())- defer cancel()- up.pipeW = pw- up.putCancel = cancel- up.putDone = make(chan struct{})- go func() {- _, err := io.Copy(io.Discard, pr)- up.putErr = err- _ = pr.CloseWithError(err)- close(up.putDone)- }()+ sink := &stubSink{}+ up.fh = sink// An UploadPart in progress: the part is admitted, then the abort lands// while its body is still being received.contents := []byte("hello")require.NoError(t, up.waitForTurn(1, int64(len(contents))))- require.NoError(t, up.abort(context.Background()))+ require.NoError(t, up.abort())++ // The abort must abandon the write rather than committing it.+ assert.True(t, sink.closed)+ assert.Equal(t, errMultipartAborted, sink.abortErr)// The UploadPart resumes: it buffers the part and calls streamPart.rw := multipart.NewRW()@@ -493,6 +521,328 @@ func TestMultipartAbortDuringUploadPart(t *testing.T) {up.mu.Unlock()}+// TestMultipartCloseAfterAbort checks that a close racing an abort reports+// the abort: gofakes3 can dispatch CompleteMultipartUpload and+// AbortMultipartUpload for the same uploadID concurrently, and a Complete+// that loses the race must not report success for data that was never+// committed.+func TestMultipartCloseAfterAbort(t *testing.T) {+ up := newMultipartUpload("bucket", "key", "bucket/key", "bucket/key", nil, 0)+ up.fh = &stubSink{}+ require.NoError(t, up.abort())+ require.ErrorIs(t, up.close(), gofakes3.ErrNoSuchUpload)++ // close stays idempotent after a successful close.+ up = newMultipartUpload("bucket", "key", "bucket/key", "bucket/key", nil, 0)+ up.fh = &stubSink{}+ require.NoError(t, up.close())+ require.NoError(t, up.close())+}++// TestMultipartCloseIncomplete checks that close refuses to commit while a+// part is still buffered awaiting an earlier one, leaving the upload open+// for more parts or an abort.+func TestMultipartCloseIncomplete(t *testing.T) {+ up := newMultipartUpload("bucket", "key", "bucket/key", "bucket/key", nil, 0)+ sink := &stubSink{}+ up.fh = sink++ // Part 2 arrives ahead of part 1 so it is buffered, not streamed.+ contents := []byte("hello")+ rw := multipart.NewRW()+ _, err := rw.Write(contents)+ require.NoError(t, err)+ md5Sum := md5.Sum(contents)+ require.NoError(t, up.waitForTurn(2, int64(len(contents))))+ require.NoError(t, up.streamPart(2, int64(len(contents)), md5Sum[:], rw))++ require.ErrorIs(t, up.close(), gofakes3.ErrInvalidPart)+ assert.False(t, sink.closed)++ // The upload is still open so abort tears it down.+ require.NoError(t, up.abort())+ assert.True(t, sink.closed)+}++// failingSink is a sink whose Close fails and which cannot abandon writes,+// like a caching VFS handle whose synchronous write-back fails.+type failingSink struct{}++func (failingSink) Write(p []byte) (int, error) { return len(p), nil }+func (failingSink) Close() error { return errBoom }++// TestMultipartAbortAlwaysSucceeds checks that AbortMultipartUpload reports+// success even when closing the sink fails: the upload is torn down either+// way, and an error reply would leave gofakes3's record of the upload alive+// with ours consumed, so every retried abort would 404 on a ghost upload.+func TestMultipartAbortAlwaysSucceeds(t *testing.T) {+ b, _, bucket := newPutTestBackend(t, "", nil)+ ctx := context.Background()+ _vfs, err := b.s.getVFS(ctx)+ require.NoError(t, err)++ up := newMultipartUpload(bucket, "key", bucket+"/key", bucket+"/"+multipartUploadPrefix+"x", nil, 0)+ up.fh = failingSink{}+ up.vfs = _vfs+ const uploadID = gofakes3.UploadID("failing-close")+ b.multipartUploads.Store(uploadID, up)++ require.NoError(t, b.AbortMultipartUpload(ctx, bucket, "key", uploadID))++ // The upload record is consumed - a second abort is NoSuchUpload.+ require.ErrorIs(t, b.AbortMultipartUpload(ctx, bucket, "key", uploadID), gofakes3.ErrNoSuchUpload)+}++// TestMultipartCompleteRenameFailureKeepsUpload checks that a Complete+// failing after the commit (here: the rename of the temporary object) keeps+// the upload record, so the retried CompleteMultipartUpload which gofakes3+// allows on a backend error finds the upload instead of a NoSuchUpload.+func TestMultipartCompleteRenameFailureKeepsUpload(t *testing.T) {+ b, _, bucket := newPutTestBackend(t, "", nil)+ ctx := context.Background()+ _vfs, err := b.s.getVFS(ctx)+ require.NoError(t, err)++ // A committed upload whose temporary object is missing: the rename+ // fails after the commit succeeded.+ up := newMultipartUpload(bucket, "key", bucket+"/key", bucket+"/"+multipartUploadPrefix+"missing", nil, 0)+ up.fh = &stubSink{}+ up.vfs = _vfs+ const uploadID = gofakes3.UploadID("rename-fails")+ b.multipartUploads.Store(uploadID, up)++ _, _, err = b.CompleteMultipartUpload(ctx, bucket, "key", uploadID, &gofakes3.CompleteMultipartUploadRequest{})+ require.Error(t, err)+ _, err = b.loadUpload(uploadID)+ require.NoError(t, err, "the upload record must survive a retryable Complete failure")+}++// cacheWritesVFSOpt returns VFS options with --vfs-cache-mode writes and the+// given write-back delay.+func cacheWritesVFSOpt(writeBack time.Duration) *vfscommon.Options {+ vfsOpt := vfscommon.Opt+ vfsOpt.CacheMode = vfscommon.CacheModeWrites+ vfsOpt.WriteBack = fs.Duration(writeBack)+ return &vfsOpt+}++// waitForContent waits for bucket/object on the backing Fs to hold want+// (e.g. after the VFS write-back delay).+func waitForContent(t *testing.T, f fs.Fs, bucket, object string, want []byte) {+ ctx := context.Background()+ deadline := time.Now().Add(10 * time.Second)+ for time.Now().Before(deadline) {+ if o, err := f.NewObject(ctx, path.Join(bucket, object)); err == nil {+ if rc, err := o.Open(ctx); err == nil {+ got, err := io.ReadAll(rc)+ _ = rc.Close()+ if err == nil && bytes.Equal(got, want) {+ return+ }+ }+ }+ time.Sleep(50 * time.Millisecond)+ }+ t.Fatalf("object %s/%s never reached the expected content on the backing remote", bucket, object)+}++// TestMultipartCacheModeWrites checks that with --vfs-cache-mode writes a+// multipart upload goes through the VFS cache and is written back to the+// backing remote, with no temporary object left behind. Also run with+// --disable-multipart-streaming, which only affects the streaming path - the+// cache needs no PutStream, so backends without one take this path instead+// of buffering in memory.+func TestMultipartCacheModeWrites(t *testing.T) {+ for _, tc := range []struct {+ name string+ disableStreaming bool+ }{+ {"Streaming", false},+ {"NoStreaming", true},+ } {+ t.Run(tc.name, func(t *testing.T) {+ core, f, bucket := newMultipartTestServerVFS(t, "", tc.disableStreaming, nil, cacheWritesVFSOpt(100*time.Millisecond))+ const object = "cached.bin"+ want, err := multipartUploadParts(t, core, bucket, object, []int{120 * 1024, 100 * 1024, 53 * 1024})+ require.NoError(t, err)+ waitForContent(t, f, bucket, object, want)+ requireOnly(t, f, bucket, object)+ })+ }+}++// TestMultipartCacheModeMinimal checks that --vfs-cache-mode minimal takes+// the cached path just like writes - the parts are written through a+// read-write handle, which the VFS caches from minimal up - including with+// --disable-multipart-streaming set, which only affects the streaming path.+func TestMultipartCacheModeMinimal(t *testing.T) {+ vfsOpt := vfscommon.Opt+ vfsOpt.CacheMode = vfscommon.CacheModeMinimal+ vfsOpt.WriteBack = fs.Duration(100 * time.Millisecond)+ core, f, bucket := newMultipartTestServerVFS(t, "", true, nil, &vfsOpt)+ const object = "cached-minimal.bin"++ want, err := multipartUploadParts(t, core, bucket, object, []int{120 * 1024, 100 * 1024, 53 * 1024})+ require.NoError(t, err)+ waitForContent(t, f, bucket, object, want)+ requireOnly(t, f, bucket, object)+}++// TestMultipartCacheModeWritesAbort checks that with --vfs-cache-mode writes+// an aborted upload is discarded from the cache: nothing reaches the backing+// remote and an existing object at the key survives.+func TestMultipartCacheModeWritesAbort(t *testing.T) {+ core, f, bucket := newMultipartTestServerVFS(t, "", false, nil, cacheWritesVFSOpt(100*time.Millisecond))+ ctx := context.Background()+ const object = "cached-abort.bin"++ existing := []byte(random.String(100))+ _, err := core.PutObject(ctx, bucket, object, bytes.NewReader(existing), int64(len(existing)), "", "", minio.PutObjectOptions{})+ require.NoError(t, err)+ waitForContent(t, f, bucket, object, existing)++ uploadID, err := core.NewMultipartUpload(ctx, bucket, object, minio.PutObjectOptions{})+ require.NoError(t, err)+ data := []byte(random.String(50 * 1024))+ _, err = core.PutObjectPart(ctx, bucket, object, uploadID, 1, bytes.NewReader(data), int64(len(data)), minio.PutObjectPartOptions{})+ require.NoError(t, err)+ require.NoError(t, core.AbortMultipartUpload(ctx, bucket, object, uploadID))++ // Wait out several write-back intervals: the aborted upload must not be+ // written back, neither over the object nor as a temporary object.+ time.Sleep(time.Second)+ assert.Equal(t, existing, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)+}++// TestMultipartCacheModeWritesSupersedesPut checks that a multipart upload+// completed while an earlier PUT to the same key is still in the write-back+// window ends up with the multipart data: both writes go through the same+// cache item, so the earlier PUT's write-back cannot land on top of the+// newer multipart object.+func TestMultipartCacheModeWritesSupersedesPut(t *testing.T) {+ core, f, bucket := newMultipartTestServerVFS(t, "", false, nil, cacheWritesVFSOpt(500*time.Millisecond))+ ctx := context.Background()+ const object = "supersede.bin"++ // PUT an object; it sits in the cache awaiting write-back.+ old := []byte(random.String(100))+ _, err := core.PutObject(ctx, bucket, object, bytes.NewReader(old), int64(len(old)), "", "", minio.PutObjectOptions{})+ require.NoError(t, err)++ // Immediately replace it with a multipart upload to the same key.+ want, err := multipartUploadParts(t, core, bucket, object, []int{60 * 1024, 40 * 1024})+ require.NoError(t, err)++ // After all write-backs settle the multipart data must have won.+ waitForContent(t, f, bucket, object, want)+ time.Sleep(time.Second)+ assert.Equal(t, want, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)+}++// TestMultipartNoPutStream checks that a multipart upload to a remote+// without streaming upload support works with the default cache mode: the+// parts are spooled to a temporary file on local disk and uploaded with a+// known size, rather than being buffered in memory.+func TestMultipartNoPutStream(t *testing.T) {+ core, f, bucket := newMultipartTestServerVFS(t, "", false, nil, nil, "PutStream")+ require.Nil(t, f.Features().PutStream)+ const object = "no-putstream.bin"++ want, err := multipartUploadParts(t, core, bucket, object, []int{120 * 1024, 100 * 1024, 53 * 1024})+ require.NoError(t, err)+ assert.Equal(t, want, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)+}++// TestMultipartNoServerSideMove checks multipart uploads to an atomic remote+// with no server-side move or copy, where the parts stream straight to the+// final object: an upload round-trips, and an aborted upload leaves an+// existing object at the key untouched.+func TestMultipartNoServerSideMove(t *testing.T) {+ // The distinct description= gives this remote its own config string so+ // it doesn't share a VFS with the fully-featured ":memory:" servers.+ core, f, bucket := newMultipartTestServerVFS(t, ":memory,description=no-server-side-move:", false, nil, nil, "Copy")+ require.False(t, operations.CanServerSideMove(f))+ ctx := context.Background()+ const object = "direct.bin"++ existing := []byte(random.String(100))+ _, err := core.PutObject(ctx, bucket, object, bytes.NewReader(existing), int64(len(existing)), "", "", minio.PutObjectOptions{})+ require.NoError(t, err)++ // An aborted upload must leave the existing object untouched.+ uploadID, err := core.NewMultipartUpload(ctx, bucket, object, minio.PutObjectOptions{})+ require.NoError(t, err)+ data := []byte(random.String(50 * 1024))+ _, err = core.PutObjectPart(ctx, bucket, object, uploadID, 1, bytes.NewReader(data), int64(len(data)), minio.PutObjectPartOptions{})+ require.NoError(t, err)+ require.NoError(t, core.AbortMultipartUpload(ctx, bucket, object, uploadID))+ assert.Equal(t, existing, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)++ // A completed upload replaces it.+ want, err := multipartUploadParts(t, core, bucket, object, []int{60 * 1024, 40 * 1024})+ require.NoError(t, err)+ assert.Equal(t, want, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)+}++// TestMultipartNoServerSideMovePartialUploads checks that a multipart upload+// to a remote where partial uploads are visible and which has no server-side+// move or copy round-trips: the parts are written straight to the final+// object rather than being buffered in memory.+func TestMultipartNoServerSideMovePartialUploads(t *testing.T) {+ core, f, bucket := newMultipartTestServerVFS(t, "", false, nil, nil, "Move", "Copy")+ require.False(t, operations.CanServerSideMove(f))+ require.True(t, f.Features().PartialUploads)+ const object = "direct-partial.bin"++ want, err := multipartUploadParts(t, core, bucket, object, []int{60 * 1024, 40 * 1024})+ require.NoError(t, err)+ assert.Equal(t, want, readObject(t, f, bucket, object))+ requireOnly(t, f, bucket, object)+}++// TestMultipartCacheModeWritesNoServerSideMove checks that with+// --vfs-cache-mode writes a remote with no server-side move or copy is still+// written through the cache rather than falling back to buffering the upload+// in memory, and pins the documented trade-offs of that path: the parts go+// into the cache under the final key, so the in-flight upload is visible+// there, and an aborted upload cannot be abandoned once in the cache, so its+// partial data is written back as if it were a completed object.+func TestMultipartCacheModeWritesNoServerSideMove(t *testing.T) {+ // The distinct description= gives this remote its own config string so+ // it doesn't share a VFS with the fully-featured ":memory:" servers.+ core, f, bucket := newMultipartTestServerVFS(t, ":memory,description=no-move-cache:", false, nil, cacheWritesVFSOpt(100*time.Millisecond), "Copy")+ require.False(t, operations.CanServerSideMove(f))+ ctx := context.Background()++ // A round trip goes through the cache under the final key.+ const object = "cached-direct.bin"+ want, err := multipartUploadParts(t, core, bucket, object, []int{120 * 1024, 100 * 1024, 53 * 1024})+ require.NoError(t, err)+ waitForContent(t, f, bucket, object, want)++ // The parts are written to the cache, not buffered in memory, so the+ // in-flight upload is visible at the key.+ const object2 = "cached-direct-inflight.bin"+ uploadID, err := core.NewMultipartUpload(ctx, bucket, object2, minio.PutObjectOptions{})+ require.NoError(t, err)+ data := []byte(random.String(50 * 1024))+ _, err = core.PutObjectPart(ctx, bucket, object2, uploadID, 1, bytes.NewReader(data), int64(len(data)), minio.PutObjectPartOptions{})+ require.NoError(t, err)+ _, err = core.StatObject(ctx, bucket, object2, minio.StatObjectOptions{})+ assert.NoError(t, err, "in-flight upload should be visible at the key")++ // An aborted upload's partial data is committed to the cache and+ // written back.+ require.NoError(t, core.AbortMultipartUpload(ctx, bucket, object2, uploadID))+ waitForContent(t, f, bucket, object2, data)+ requireOnly(t, f, bucket, object, object2)+}+// TestTempObjectsHiddenFromListings checks that the reserved .rclone_temp_// prefix, and the multipart prefix used before it was reserved, are hidden// from S3 listings while remaining visible to rclone itself for cleanup.diff --git a/cmd/serve/s3/serve_s3.md b/cmd/serve/s3/serve_s3.mdindex 2c9152007..23e89ec5d 100644--- a/cmd/serve/s3/serve_s3.md+++ b/cmd/serve/s3/serve_s3.md@@ -108,22 +108,53 @@ S3 listings but must be removed manually.### Multipart uploads-By default `serve s3` **streams** each multipart upload, in part-number-order, into a single `PutStream` upload to the underlying remote, so the-whole file is never buffered in memory - memory use stays bounded by the-parts in flight. The remote then performs its own internal upload (for-example its own multipart upload, still with bounded memory). This works-for any remote that supports `PutStream`, which is nearly all of them,-including through `crypt`.--The upload is atomic so the destination object only ever changes on a+Multipart uploads are written, in part-number order, to a temporary+object which is renamed into place, server-side, on completion, so the+upload is atomic. The object at the key only ever changes on asuccessful completion. A failed or aborted upload never affects any-object already stored under that name. Remotes that upload atomically-already (object stores such as `s3`) are streamed straight to the-destination. On remotes where a partial upload would otherwise be visible-(such as `local`), the parts are streamed to a temporary object that is-moved into place, server-side, on completion; these remotes therefore-also need to support a server-side move or copy.+object already stored under that name and a partly-uploaded object+never becomes visible under it.++With the default `--vfs-cache-mode off` `serve s3` **streams** each+multipart upload, in part-number order, into a single streaming upload+to the underlying remote, so the whole file is never buffered in+memory. Memory use stays bounded by the parts in flight. The remote+then performs its own internal upload (for example its own multipart+upload, still with bounded memory). Remotes that don't support+streaming uploads (those that must know the file size before the+upload starts, such as `onedrive`, `pcloud`, `jottacloud`, `mailru`,+`opendrive`, `putio`, `protondrive` and `zoho`) have the parts spooled+to a temporary file on **local disk** instead, and uploaded with the+size then known on completion, so they need local disk space for the+largest objects in flight rather than memory.++With `--vfs-cache-mode writes` (or `full`) the parts are written to a+temporary file in the VFS cache and uploaded by the VFS write-back -+see [Multipart uploads and the VFS+cache](#multipart-uploads-and-the-vfs-cache) below.++The rename into place needs the remote to support a server-side move+or copy, which nearly all do. It is a cheap rename on most remotes,+but on object stores without a real rename (such as `s3` itself) the+move is performed as a server-side copy and delete of the whole+object, which can take time and API calls for large objects.+Concurrent multipart uploads of the same key (which S3 permits) are+safe. Each writes its own temporary object and the last to complete+wins.++On the few remotes that support neither server side move nor copy, the+parts are written straight to the destination object instead and never+buffered in memory. This is at some cost in atomicity - the incomplete+object is visible under its final name while the upload is in flight,+as it also is for a plain object PUT on such remotes, and concurrent+multipart uploads of the same key write to the same object and can+interleave. A failed or aborted upload still leaves any pre-existing+object untouched provided the remote uploads atomically and the VFS+cache is off; on a remote where partial uploads are visible it may+leave partial data at the key (like a plain PUT there), and with+`--vfs-cache-mode writes` (or `full`) a write to the cache cannot be+abandoned, so an aborted upload's partial data is written back to the+remote as if it had completed.**Features**@@ -138,10 +169,14 @@ also need to support a server-side move or copy.as one continuous stream.- The destination object only ever changes atomically, on completion: anaborted or failed upload leaves any pre-existing object of the same- name untouched, and a partly-uploaded object never becomes visible.-- Backend-agnostic - it only needs the remote to support `PutStream`- (plus a server-side move or copy on remotes that don't upload- atomically).+ name untouched, and a partly-uploaded object never becomes visible+ (except on the few remotes with no server-side move or copy, as+ above).+- Multipart uploads go through the VFS like any other upload, so they+ show in rclone's transfer stats and obey `--bwlimit`.+- Backend-agnostic - it only needs the remote to support a server-side+ move or copy for the rename into place, which nearly all do; a remote+ without streaming upload support spools to local disk as above.**Limitations**@@ -167,14 +202,62 @@ also need to support a server-side move or copy.upload and the client must start it again. (The remote's own uploadstill retries its internal chunks.)- Parts are serialised into one stream, so ingest from the client is- effectively single-threaded, although the remote's own upload still- runs concurrently.-- On remotes that don't upload atomically (such as `local`), the- completed object is moved into place with a server-side operation.- This is a cheap rename on most such remotes. On these remotes, if- `serve s3` is killed part-way through an upload the temporary object- (named with a leading `.rclone_temp_multipart_`) may be left behind;- it is hidden from S3 listings but must be removed manually.+ effectively single-threaded. When streaming, the remote's own upload+ runs concurrently with the parts arriving; with the local disk spool+ or the VFS cache the upload to the remote only starts on completion.+- If `serve s3` is killed part-way through an upload the temporary+ object (named with a leading `.rclone_temp_multipart_`) may be left+ behind; it is hidden from S3 listings but must be removed manually.++#### Multipart uploads and the VFS cache++With `--vfs-cache-mode writes` (or `full`) multipart uploads do not+stream to the remote at all. The parts are written, in part-number+order, to a temporary file in the VFS cache. On completion the file is+renamed into place and uploaded by the VFS write-back, exactly like a+plain object PUT. This needs no streaming upload support from the+remote. The rename normally happens in the cache before the upload has+started, but the VFS requires the remote to support a server-side move+or copy to rename files at all (and uses one if the temporary file has+already been written back, e.g. with `--vfs-write-back 0`). On remotes+without either, the parts are written to the cache directly under the+final key instead: the upload still never touches memory, but it loses+its atomicity - the in-flight upload is visible at the key, and an+aborted upload cannot be abandoned once in the cache, so its partial+data is written back to the remote as if it were a completed object.++Remotes that benefit from `--vfs-cache-mode writes`:++- **Remotes over slow or unreliable links.** A failure in a streamed+ upload aborts the whole multipart upload and the client must start+ again from the first part; a failed write-back upload is retried by+ the VFS (see `--vfs-cache-max-age` and friends) without the client+ being involved. Ingest from the client also runs at local disk speed+ rather than being throttled to the remote's pace.+- **Workloads that read back or overwrite what they just wrote.** The+ completed object stays in the cache, so subsequent `GET`/`HEAD`+ requests are served locally, and plain PUTs and multipart uploads to+ the same key go through the same cache entry so the last write wins+ regardless of upload style.++The trade-offs of the VFS cache:++- The whole object lands on local disk, so the cache (`--cache-dir`)+ needs space for the largest objects in flight; `--vfs-cache-max-size`+ cannot evict files which are still being uploaded.+- The `200 OK` for `CompleteMultipartUpload` means the data is safely+ in the **local cache**, not yet on the remote - the same durability+ the cache gives plain PUTs. If an acknowledgement must mean the data+ has reached the remote (for example WAL archiving), use the default+ `--vfs-cache-mode off`.+- The upload to the remote only starts on completion, rather than+ overlapping with the parts arriving, so the data reaches the remote+ later than with streaming.+- If `serve s3` is killed part-way through an upload, the temporary+ file survives in the cache and the VFS cache recovery uploads it to+ the remote on restart as a temporary object (named with a leading+ `.rclone_temp_multipart_`); as with the streaming path, it is+ hidden from S3 listings but must be removed manually.#### Cleaning up temporary objects@@ -202,20 +285,18 @@ hidden from listings and can be cleaned up the same way.#### Disabling streaming-If you pass `--disable-multipart-streaming`, or the remote doesn't-support `PutStream` (or doesn't upload atomically and can't move or copy-server-side), multipart uploads are instead **buffered in memory**-by the underlying S3 library: every part is held in memory and the whole-object is written out in one go when the upload completes (the previous-behaviour). This removes the in-order/contiguous-part restriction above,-so parts can be uploaded in any order, but **memory use grows with the-size of the upload**, so it is only suitable for small objects. A one-off-`NOTICE` is logged the first time this happens.--Alternatively, if the client is an rclone `s3` remote (like the-`[serves3]` example above), you can set `use_multipart_uploads = false`-on it so it uploads each object as a single stream and skips multipart-uploads altogether.+If you pass `--disable-multipart-streaming`, multipart uploads are+instead **buffered in memory** by the underlying S3 library: every+part is held in memory and the whole object is written out in one go+when the upload completes. This removes the in-order/contiguous-part+restriction above, so parts can be uploaded in any order, but **memory+use grows with the size of the upload**, so it is only suitable for+small objects. A one-off `NOTICE` is logged the first time this+happens. This flag is the only thing that makes multipart uploads+buffer in memory - it is never done because of missing remote+capabilities. Consider `--vfs-cache-mode writes` instead, which+buffers the upload in the VFS cache on disk and takes precedence over+`--disable-multipart-streaming`.### Bugs