"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files
Go azure-blobazure-blob-storageazure-filesbackblaze-b2cloud-storagedropboxencryptionftpfuse-filesystemgogolanggoogle-cloud-storagegoogle-driveonedriveopenstack-swiftrclones3sftpsyncwebdav
s3: add initial --s3-directory-bucket to support AWS Directory Buckets
Nick Craig-Wood192524c00482
diff --git a/backend/s3/s3.go b/backend/s3/s3.goindex dac7d6375..5bc36cc11 100644--- a/backend/s3/s3.go+++ b/backend/s3/s3.go@@ -2606,6 +2606,35 @@ knows about - please make a bug report if not.`,Default: fs.Tristate{},Advanced: true,+ }, {+ Name: "directory_bucket",+ Help: strings.ReplaceAll(`Set to use AWS Directory Buckets++If you are using an AWS Directory Bucket then set this flag.++This will ensure no |Content-Md5| headers are sent and ensure |ETag|+headers are not interpreted as MD5 sums. |X-Amz-Meta-Md5chksum| will+be set on all objects whether single or multipart uploaded.++This also sets |no_check_bucket = true|.++Note that Directory Buckets do not support:++- Versioning+- |Content-Encoding: gzip|++Rclone limitations with Directory Buckets:++- rclone does not support creating Directory Buckets with |rclone mkdir|+- ... or removing them with |rclone rmdir| yet+- Directory Buckets do not appear when doing |rclone lsf| at the top level.+- Rclone can't remove auto created directories yet. In theory this should+ work with |directory_markers = true| but it doesn't.+- Directories don't seem to appear in recursive (ListR) listings.+`, "|", "`"),+ Default: false,+ Advanced: true,+ Provider: "AWS",}, {Name: "sdk_log_mode",Help: strings.ReplaceAll(`Set to debug the SDK@@ -2780,6 +2809,7 @@ type Options struct {UseMultipartUploads fs.Tristate `config:"use_multipart_uploads"`UseUnsignedPayload fs.Tristate `config:"use_unsigned_payload"`SDKLogMode sdkLogMode `config:"sdk_log_mode"`+ DirectoryBucket bool `config:"directory_bucket"`}// Fs represents a remote s3 server@@ -3548,6 +3578,14 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e// MD5 digest of their object data.f.etagIsNotMD5 = true}+ if opt.DirectoryBucket {+ // Objects uploaded to directory buckets appear to have random ETags+ //+ // This doesn't appear to be documented+ f.etagIsNotMD5 = true+ // The normal API doesn't work for creating directory buckets, so don't try+ f.opt.NoCheckBucket = true+ }f.setRoot(root)f.features = (&fs.Features{ReadMimeType: true,@@ -6029,6 +6067,10 @@ func (w *s3ChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, readerSSECustomerKey: w.multiPartUploadInput.SSECustomerKey,SSECustomerKeyMD5: w.multiPartUploadInput.SSECustomerKeyMD5,}+ if w.f.opt.DirectoryBucket {+ // Directory buckets do not support "Content-Md5" header+ uploadPartReq.ContentMD5 = nil+ }var uout *s3.UploadPartOutputerr = w.f.pacer.Call(func() (bool, error) {// rewind the reader on retry and after reading md5@@ -6305,7 +6347,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [if (multipart || o.fs.etagIsNotMD5) && !o.fs.opt.DisableChecksum {// Set the md5sum as metadata on the object if// - a multipart upload- // - the Etag is not an MD5, eg when using SSE/SSE-C+ // - the Etag is not an MD5, eg when using SSE/SSE-C or directory buckets// provided checksums aren't disabledui.req.Metadata[metaMD5Hash] = md5sumBase64}@@ -6320,7 +6362,7 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [if size >= 0 {ui.req.ContentLength = &size}- if md5sumBase64 != "" {+ if md5sumBase64 != "" && !o.fs.opt.DirectoryBucket {ui.req.ContentMD5 = &md5sumBase64}if o.fs.opt.RequesterPays {diff --git a/docs/content/s3.md b/docs/content/s3.mdindex 9031e57fe..2117e95a5 100644--- a/docs/content/s3.md+++ b/docs/content/s3.md@@ -2294,6 +2294,21 @@ You can also do this entirely on the command lineThis is the provider used as main example and described in the [configuration](#configuration) section above.+### AWS Directory Buckets++From rclone v1.69 [Directory Buckets](https://docs.aws.amazon.com/AmazonS3/latest/userguide/directory-buckets-overview.html)+are supported.++You will need to set the `directory_buckets = true` config parameter+or use `--s3-directory-buckets`.++Note that rclone cannot yet:++- Create directory buckets+- List directory buckets++See [the --s3-directory-buckets flag](#s3-directory-buckets) for more info+### AWS Snowball Edge[AWS Snowball](https://aws.amazon.com/snowball/) is a hardware