Skip to content

"rsync for cloud storage" - Google Drive, S3, Dropbox, Backblaze B2, One Drive, Swift, Hubic, Wasabi, Google Cloud Storage, Azure Blob, Azure Files, Yandex Files

s3: make it easier to add new S3 providers

dougalf28c83c6de2f
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
index 9e3a565c5..4a75f8975 100644
--- a/CONTRIBUTING.md
+++ b/CONTRIBUTING.md
@@ -621,44 +621,7 @@ in the web browser and the links (internal and external) all work.
## Adding a new s3 provider
-It is quite easy to add a new S3 provider to rclone.
-
-You'll need to modify the following files
-
-- `backend/s3/s3.go`
- - Add the provider to `providerOption` at the top of the file
- - Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`.
- - Exclude your provider from generic config questions (eg `region` and `endpoint`).
- - Add the provider to the `setQuirks` function - see the documentation there.
-- `docs/content/s3.md`
- - Add the provider at the top of the page.
- - Add a section about the provider linked from there.
- - Make sure this is in alphabetical order in the `Providers` section.
- - Add a transcript of a trial `rclone config` session
- - Edit the transcript to remove things which might change in subsequent versions
- - **Do not** alter or add to the autogenerated parts of `s3.md`
- - **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
-- `README.md` - this is the home page in github
- - Add the provider and a link to the section you wrote in `docs/contents/s3.md`
-- `docs/content/_index.md` - this is the home page of rclone.org
- - Add the provider and a link to the section you wrote in `docs/contents/s3.md`
-
-When adding the provider, endpoints, quirks, docs etc keep them in
-alphabetical order by `Provider` name, but with `AWS` first and
-`Other` last.
-
-Once you've written the docs, run `make serve` and check they look OK
-in the web browser and the links (internal and external) all work.
-
-Once you've written the code, test `rclone config` works to your
-satisfaction, and check the integration tests work `go test -v -remote
-NewS3Provider:`. You may need to adjust the quirks to get them to
-pass. Some providers just can't pass the tests with control characters
-in the names so if these fail and the provider doesn't support
-`urlEncodeListings` in the quirks then ignore them. Note that the
-`SetTier` test may also fail on non AWS providers.
-
-For an example of adding an s3 provider see [eb3082a1](https://github.com/rclone/rclone/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10).
+[Please see the guide in the S3 backend directory](backend/s3/README.md).
## Writing a plugin
diff --git a/backend/s3/README.md b/backend/s3/README.md
new file mode 100644
index 000000000..d4ea43d9a
--- /dev/null
+++ b/backend/s3/README.md
@@ -0,0 +1,194 @@
+## Adding a new s3 provider
+
+It is quite easy to add a new S3 provider to rclone.
+
+You'll then need to do add the following (optional tags are in [] and
+do not get displayed in rclone config if empty):
+
+The process is as follows: Create yaml -> add docs -> run tests ->
+adjust yaml until tests pass.
+
+All tags can be found in `backend/s3/providers.go` Provider Struct.
+Looking through a few of the yaml files as examples should make things
+clear. `AWS.yaml` as the most config. pasting.
+
+### YAML
+
+In `backend/s3/provider/YourProvider.yaml`
+
+- name
+- description
+ - More like the full name often "YourProvider + Object Storage"
+- [Region]
+ - Any regions your provider supports or the defaults (use `region: {}` for this)
+ - Example from AWS.yaml:
+ ```yaml
+ region:
+ us-east-1: |-
+ The default endpoint - a good choice if you are unsure.
+ US Region, Northern Virginia, or Pacific Northwest.
+ Leave location constraint empty.
+ ```
+ - The defaults (as seen in Rclone.yaml):
+ ```yaml
+ region:
+ "": |-
+ Use this if unsure.
+ Will use v4 signatures and an empty region.
+ other-v2-signature: |-
+ Use this only if v4 signatures don't work.
+ E.g. pre Jewel/v10 CEPH.
+ ```
+- [Endpoint]
+ - Any endpoints your provider supports
+ - Example from Mega.yaml
+ ```yaml
+ endpoint:
+ s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam)
+ ```
+- [Location Constraint]
+ - The Location Constraint of your remote, often same as region.
+ - Example from AWS.yaml
+ ```yaml
+ location_constraint:
+ "": Empty for US Region, Northern Virginia, or Pacific Northwest
+ us-east-2: US East (Ohio) Region
+ ```
+- [ACL]
+ - Identical across *most* providers. Select the default with `acl: {}`
+ - Example from AWS.yaml
+ ```yaml
+ acl:
+ private: |-
+ Owner gets FULL_CONTROL.
+ No one else has access rights (default).
+ public-read: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ access.
+ public-read-write: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ and WRITE access.
+ Granting this on a bucket is generally not recommended.
+ authenticated-read: |-
+ Owner gets FULL_CONTROL.
+ The AuthenticatedUsers group gets READ access.
+ bucket-owner-read: |-
+ Object owner gets FULL_CONTROL.
+ Bucket owner gets READ access.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ bucket-owner-full-control: |-
+ Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ ```
+- [Storage Class]
+ - Identical across *most* providers.
+ - Defaults from AWS.yaml
+ ```yaml
+ storage_class:
+ "": Default
+ STANDARD: Standard storage class
+ REDUCED_REDUNDANCY: Reduced redundancy storage class
+ STANDARD_IA: Standard Infrequent Access storage class
+ ONEZONE_IA: One Zone Infrequent Access storage class
+ GLACIER: Glacier Flexible Retrieval storage class
+ DEEP_ARCHIVE: Glacier Deep Archive storage class
+ INTELLIGENT_TIERING: Intelligent-Tiering storage class
+ GLACIER_IR: Glacier Instant Retrieval storage class
+ ```
+- [Server Side Encryption]
+ - Not common, identical across *most* providers.
+ - Defaults from AWS.yaml
+ ```yaml
+ server_side_encryption:
+ "": None
+ AES256: AES256
+ aws:kms: aws:kms
+ ```
+- [Advanced Options]
+ - All advanced options are Boolean - if true the configurator asks about that value, if not it doesn't:
+ ```go
+ BucketACL bool `yaml:"bucket_acl,omitempty"`
+ DirectoryBucket bool `yaml:"directory_bucket,omitempty"`
+ LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"`
+ RequesterPays bool `yaml:"requester_pays,omitempty"`
+ SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"`
+ SSECustomerKey bool `yaml:"sse_customer_key,omitempty"`
+ SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"`
+ SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"`
+ SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"`
+ STSEndpoint bool `yaml:"sts_endpoint,omitempty"`
+ UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"`
+ ```
+ - Example from AWS.yaml:
+ ```yaml
+ bucket_acl: true
+ directory_bucket: true
+ leave_parts_on_error: true
+ requester_pays: true
+ sse_customer_algorithm: true
+ sse_customer_key: true
+ sse_customer_key_base64: true
+ sse_customer_key_md5: true
+ sse_kms_key_id: true
+ sts_endpoint: true
+ use_accelerate_endpoint: true
+ ```
+- Quirks
+ - Quirks are discovered through documentation and running the tests as seen below.
+ - Most quirks are *bool as to have 3 values, `true`, `false` and `dont care`.
+ ```go
+ type Quirks struct {
+ ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2
+ ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style
+ ListURLEncode *bool `yaml:"list_url_encode,omitempty"`
+ UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"`
+ UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"`
+ UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"`
+ MightGzip *bool `yaml:"might_gzip,omitempty"`
+ UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"`
+ UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"`
+ UseXID *bool `yaml:"use_x_id,omitempty"`
+ SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
+ CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
+ MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
+ MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
+ }
+ ```
+ - Example from AWS.yaml
+ ```yaml
+ quirks:
+ might_gzip: false # Never auto gzips objects
+ use_unsigned_payload: false # AWS has trailer support
+ ```
+
+Note that if you omit a section, eg `region` then the user won't be
+asked that question, and if you add an empty section e.g. `region: {}`
+then the defaults from the `Other.yaml` will be used.
+
+### DOCS
+
+- `docs/content/s3.md`
+ - Add the provider at the top of the page.
+ - Add a section about the provider linked from there.
+ - Make sure this is in alphabetical order in the `Providers` section.
+ - Add a transcript of a trial `rclone config` session
+ - Edit the transcript to remove things which might change in subsequent versions
+ - **Do not** alter or add to the autogenerated parts of `s3.md`
+ - Rule of thumb: don't edit anything not mentioned above.
+ - **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
+ - This will make autogenerated changes!
+- `README.md` - this is the home page in github
+ - Add the provider and a link to the section you wrote in `docs/contents/s3.md`
+- `docs/content/_index.md` - this is the home page of rclone.org
+ - Add the provider and a link to the section you wrote in `docs/contents/s3.md`
+- Once you've written the docs, run `make serve` and check they look OK
+ in the web browser and the links (internal and external) all work.
+
+### TESTS
+
+Once you've written the code, test `rclone config` works to your
+satisfaction and looks correct, and check the integration tests work
+`go test -v -remote NewS3Provider:`. You may need to adjust the quirks
+to get them to pass. Some providers just can't pass the tests with
+control characters in the names so if these fail and the provider
+doesn't support `urlEncodeListings` in the quirks then ignore them.
diff --git a/backend/s3/provider/AWS.yaml b/backend/s3/provider/AWS.yaml
new file mode 100644
index 000000000..39c12ae08
--- /dev/null
+++ b/backend/s3/provider/AWS.yaml
@@ -0,0 +1,139 @@
+name: AWS
+description: Amazon Web Services (AWS) S3
+region:
+ us-east-1: |-
+ The default endpoint - a good choice if you are unsure.
+ US Region, Northern Virginia, or Pacific Northwest.
+ Leave location constraint empty.
+ us-east-2: |-
+ US East (Ohio) Region.
+ Needs location constraint us-east-2.
+ us-west-1: |-
+ US West (Northern California) Region.
+ Needs location constraint us-west-1.
+ us-west-2: |-
+ US West (Oregon) Region.
+ Needs location constraint us-west-2.
+ ca-central-1: |-
+ Canada (Central) Region.
+ Needs location constraint ca-central-1.
+ eu-west-1: |-
+ EU (Ireland) Region.
+ Needs location constraint EU or eu-west-1.
+ eu-west-2: |-
+ EU (London) Region.
+ Needs location constraint eu-west-2.
+ eu-west-3: |-
+ EU (Paris) Region.
+ Needs location constraint eu-west-3.
+ eu-north-1: |-
+ EU (Stockholm) Region.
+ Needs location constraint eu-north-1.
+ eu-south-1: |-
+ EU (Milan) Region.
+ Needs location constraint eu-south-1.
+ eu-central-1: |-
+ EU (Frankfurt) Region.
+ Needs location constraint eu-central-1.
+ ap-southeast-1: |-
+ Asia Pacific (Singapore) Region.
+ Needs location constraint ap-southeast-1.
+ ap-southeast-2: |-
+ Asia Pacific (Sydney) Region.
+ Needs location constraint ap-southeast-2.
+ ap-northeast-1: |-
+ Asia Pacific (Tokyo) Region.
+ Needs location constraint ap-northeast-1.
+ ap-northeast-2: |-
+ Asia Pacific (Seoul).
+ Needs location constraint ap-northeast-2.
+ ap-northeast-3: |-
+ Asia Pacific (Osaka-Local).
+ Needs location constraint ap-northeast-3.
+ ap-south-1: |-
+ Asia Pacific (Mumbai).
+ Needs location constraint ap-south-1.
+ ap-east-1: |-
+ Asia Pacific (Hong Kong) Region.
+ Needs location constraint ap-east-1.
+ sa-east-1: |-
+ South America (Sao Paulo) Region.
+ Needs location constraint sa-east-1.
+ il-central-1: |-
+ Israel (Tel Aviv) Region.
+ Needs location constraint il-central-1.
+ me-south-1: |-
+ Middle East (Bahrain) Region.
+ Needs location constraint me-south-1.
+ af-south-1: |-
+ Africa (Cape Town) Region.
+ Needs location constraint af-south-1.
+ cn-north-1: |-
+ China (Beijing) Region.
+ Needs location constraint cn-north-1.
+ cn-northwest-1: |-
+ China (Ningxia) Region.
+ Needs location constraint cn-northwest-1.
+ us-gov-east-1: |-
+ AWS GovCloud (US-East) Region.
+ Needs location constraint us-gov-east-1.
+ us-gov-west-1: |-
+ AWS GovCloud (US) Region.
+ Needs location constraint us-gov-west-1.
+endpoint: {}
+location_constraint:
+ '': Empty for US Region, Northern Virginia, or Pacific Northwest
+ us-east-2: US East (Ohio) Region
+ us-west-1: US West (Northern California) Region
+ us-west-2: US West (Oregon) Region
+ ca-central-1: Canada (Central) Region
+ eu-west-1: EU (Ireland) Region
+ eu-west-2: EU (London) Region
+ eu-west-3: EU (Paris) Region
+ eu-north-1: EU (Stockholm) Region
+ eu-south-1: EU (Milan) Region
+ EU: EU Region
+ ap-southeast-1: Asia Pacific (Singapore) Region
+ ap-southeast-2: Asia Pacific (Sydney) Region
+ ap-northeast-1: Asia Pacific (Tokyo) Region
+ ap-northeast-2: Asia Pacific (Seoul) Region
+ ap-northeast-3: Asia Pacific (Osaka-Local) Region
+ ap-south-1: Asia Pacific (Mumbai) Region
+ ap-east-1: Asia Pacific (Hong Kong) Region
+ sa-east-1: South America (Sao Paulo) Region
+ il-central-1: Israel (Tel Aviv) Region
+ me-south-1: Middle East (Bahrain) Region
+ af-south-1: Africa (Cape Town) Region
+ cn-north-1: China (Beijing) Region
+ cn-northwest-1: China (Ningxia) Region
+ us-gov-east-1: AWS GovCloud (US-East) Region
+ us-gov-west-1: AWS GovCloud (US) Region
+acl: {}
+storage_class:
+ '': Default
+ STANDARD: Standard storage class
+ REDUCED_REDUNDANCY: Reduced redundancy storage class
+ STANDARD_IA: Standard Infrequent Access storage class
+ ONEZONE_IA: One Zone Infrequent Access storage class
+ GLACIER: Glacier Flexible Retrieval storage class
+ DEEP_ARCHIVE: Glacier Deep Archive storage class
+ INTELLIGENT_TIERING: Intelligent-Tiering storage class
+ GLACIER_IR: Glacier Instant Retrieval storage class
+server_side_encryption:
+ '': None
+ AES256: AES256
+ aws:kms: aws:kms
+bucket_acl: true
+directory_bucket: true
+leave_parts_on_error: true
+requester_pays: true
+sse_customer_algorithm: true
+sse_customer_key: true
+sse_customer_key_base64: true
+sse_customer_key_md5: true
+sse_kms_key_id: true
+sts_endpoint: true
+use_accelerate_endpoint: true
+quirks:
+ might_gzip: false # Never auto gzips objects
+ use_unsigned_payload: false # AWS has trailer support which means it adds checksums in the trailer without seeking
diff --git a/backend/s3/provider/Alibaba.yaml b/backend/s3/provider/Alibaba.yaml
new file mode 100644
index 000000000..adb0bf86b
--- /dev/null
+++ b/backend/s3/provider/Alibaba.yaml
@@ -0,0 +1,37 @@
+name: Alibaba
+description: Alibaba Cloud Object Storage System (OSS) formerly Aliyun
+endpoint:
+ oss-accelerate.aliyuncs.com: Global Accelerate
+ oss-accelerate-overseas.aliyuncs.com: Global Accelerate (outside mainland China)
+ oss-cn-hangzhou.aliyuncs.com: East China 1 (Hangzhou)
+ oss-cn-shanghai.aliyuncs.com: East China 2 (Shanghai)
+ oss-cn-qingdao.aliyuncs.com: North China 1 (Qingdao)
+ oss-cn-beijing.aliyuncs.com: North China 2 (Beijing)
+ oss-cn-zhangjiakou.aliyuncs.com: North China 3 (Zhangjiakou)
+ oss-cn-huhehaote.aliyuncs.com: North China 5 (Hohhot)
+ oss-cn-wulanchabu.aliyuncs.com: North China 6 (Ulanqab)
+ oss-cn-shenzhen.aliyuncs.com: South China 1 (Shenzhen)
+ oss-cn-heyuan.aliyuncs.com: South China 2 (Heyuan)
+ oss-cn-guangzhou.aliyuncs.com: South China 3 (Guangzhou)
+ oss-cn-chengdu.aliyuncs.com: West China 1 (Chengdu)
+ oss-cn-hongkong.aliyuncs.com: Hong Kong (Hong Kong)
+ oss-us-west-1.aliyuncs.com: US West 1 (Silicon Valley)
+ oss-us-east-1.aliyuncs.com: US East 1 (Virginia)
+ oss-ap-southeast-1.aliyuncs.com: Southeast Asia Southeast 1 (Singapore)
+ oss-ap-southeast-2.aliyuncs.com: Asia Pacific Southeast 2 (Sydney)
+ oss-ap-southeast-3.aliyuncs.com: Southeast Asia Southeast 3 (Kuala Lumpur)
+ oss-ap-southeast-5.aliyuncs.com: Asia Pacific Southeast 5 (Jakarta)
+ oss-ap-northeast-1.aliyuncs.com: Asia Pacific Northeast 1 (Japan)
+ oss-ap-south-1.aliyuncs.com: Asia Pacific South 1 (Mumbai)
+ oss-eu-central-1.aliyuncs.com: Central Europe 1 (Frankfurt)
+ oss-eu-west-1.aliyuncs.com: West Europe (London)
+ oss-me-east-1.aliyuncs.com: Middle East 1 (Dubai)
+acl: {}
+storage_class:
+ '': Default
+ STANDARD: Standard storage class
+ GLACIER: Archive storage mode
+ STANDARD_IA: Infrequent access storage mode
+bucket_acl: true
+quirks:
+ use_multipart_etag: false # multipar ETags differ from AWS
diff --git a/backend/s3/provider/ArvanCloud.yaml b/backend/s3/provider/ArvanCloud.yaml
new file mode 100644
index 000000000..04d28fef2
--- /dev/null
+++ b/backend/s3/provider/ArvanCloud.yaml
@@ -0,0 +1,19 @@
+name: ArvanCloud
+description: Arvan Cloud Object Storage (AOS)
+endpoint:
+ s3.ir-thr-at1.arvanstorage.ir: |-
+ The default endpoint - a good choice if you are unsure.
+ Tehran Iran (Simin)
+ s3.ir-tbz-sh1.arvanstorage.ir: Tabriz Iran (Shahriar)
+location_constraint:
+ ir-thr-at1: Tehran Iran (Simin)
+ ir-tbz-sh1: Tabriz Iran (Shahriar)
+acl: {}
+storage_class:
+ STANDARD: Standard storage class
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Ceph.yaml b/backend/s3/provider/Ceph.yaml
new file mode 100644
index 000000000..910a00ea3
--- /dev/null
+++ b/backend/s3/provider/Ceph.yaml
@@ -0,0 +1,20 @@
+name: Ceph
+description: Ceph Object Storage
+region: {}
+endpoint: {}
+location_constraint: {}
+acl: {}
+server_side_encryption:
+ '': None
+ AES256: AES256
+ aws:kms: aws:kms
+bucket_acl: true
+sse_customer_algorithm: true
+sse_customer_key: true
+sse_customer_key_base64: true
+sse_customer_key_md5: true
+sse_kms_key_id: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
diff --git a/backend/s3/provider/ChinaMobile.yaml b/backend/s3/provider/ChinaMobile.yaml
new file mode 100644
index 000000000..8209fa4ba
--- /dev/null
+++ b/backend/s3/provider/ChinaMobile.yaml
@@ -0,0 +1,98 @@
+name: ChinaMobile
+description: China Mobile Ecloud Elastic Object Storage (EOS)
+endpoint:
+ eos-wuxi-1.cmecloud.cn: |-
+ The default endpoint - a good choice if you are unsure.
+ East China (Suzhou)
+ eos-jinan-1.cmecloud.cn: East China (Jinan)
+ eos-ningbo-1.cmecloud.cn: East China (Hangzhou)
+ eos-shanghai-1.cmecloud.cn: East China (Shanghai-1)
+ eos-zhengzhou-1.cmecloud.cn: Central China (Zhengzhou)
+ eos-hunan-1.cmecloud.cn: Central China (Changsha-1)
+ eos-zhuzhou-1.cmecloud.cn: Central China (Changsha-2)
+ eos-guangzhou-1.cmecloud.cn: South China (Guangzhou-2)
+ eos-dongguan-1.cmecloud.cn: South China (Guangzhou-3)
+ eos-beijing-1.cmecloud.cn: North China (Beijing-1)
+ eos-beijing-2.cmecloud.cn: North China (Beijing-2)
+ eos-beijing-4.cmecloud.cn: North China (Beijing-3)
+ eos-huhehaote-1.cmecloud.cn: North China (Huhehaote)
+ eos-chengdu-1.cmecloud.cn: Southwest China (Chengdu)
+ eos-chongqing-1.cmecloud.cn: Southwest China (Chongqing)
+ eos-guiyang-1.cmecloud.cn: Southwest China (Guiyang)
+ eos-xian-1.cmecloud.cn: Nouthwest China (Xian)
+ eos-yunnan.cmecloud.cn: Yunnan China (Kunming)
+ eos-yunnan-2.cmecloud.cn: Yunnan China (Kunming-2)
+ eos-tianjin-1.cmecloud.cn: Tianjin China (Tianjin)
+ eos-jilin-1.cmecloud.cn: Jilin China (Changchun)
+ eos-hubei-1.cmecloud.cn: Hubei China (Xiangyan)
+ eos-jiangxi-1.cmecloud.cn: Jiangxi China (Nanchang)
+ eos-gansu-1.cmecloud.cn: Gansu China (Lanzhou)
+ eos-shanxi-1.cmecloud.cn: Shanxi China (Taiyuan)
+ eos-liaoning-1.cmecloud.cn: Liaoning China (Shenyang)
+ eos-hebei-1.cmecloud.cn: Hebei China (Shijiazhuang)
+ eos-fujian-1.cmecloud.cn: Fujian China (Xiamen)
+ eos-guangxi-1.cmecloud.cn: Guangxi China (Nanning)
+ eos-anhui-1.cmecloud.cn: Anhui China (Huainan)
+location_constraint:
+ wuxi1: East China (Suzhou)
+ jinan1: East China (Jinan)
+ ningbo1: East China (Hangzhou)
+ shanghai1: East China (Shanghai-1)
+ zhengzhou1: Central China (Zhengzhou)
+ hunan1: Central China (Changsha-1)
+ zhuzhou1: Central China (Changsha-2)
+ guangzhou1: South China (Guangzhou-2)
+ dongguan1: South China (Guangzhou-3)
+ beijing1: North China (Beijing-1)
+ beijing2: North China (Beijing-2)
+ beijing4: North China (Beijing-3)
+ huhehaote1: North China (Huhehaote)
+ chengdu1: Southwest China (Chengdu)
+ chongqing1: Southwest China (Chongqing)
+ guiyang1: Southwest China (Guiyang)
+ xian1: Northwest China (Xian)
+ yunnan: Yunnan China (Kunming)
+ yunnan2: Yunnan China (Kunming-2)
+ tianjin1: Tianjin China (Tianjin)
+ jilin1: Jilin China (Changchun)
+ hubei1: Hubei China (Xiangyan)
+ jiangxi1: Jiangxi China (Nanchang)
+ gansu1: Gansu China (Lanzhou)
+ shanxi1: Shanxi China (Taiyuan)
+ liaoning1: Liaoning China (Shenyang)
+ hebei1: Hebei China (Shijiazhuang)
+ fujian1: Fujian China (Xiamen)
+ guangxi1: Guangxi China (Nanning)
+ anhui1: Anhui China (Huainan)
+acl:
+ private: |-
+ Owner gets FULL_CONTROL.
+ No one else has access rights (default).
+ public-read: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ access.
+ public-read-write: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ and WRITE access.
+ Granting this on a bucket is generally not recommended.
+ authenticated-read: |-
+ Owner gets FULL_CONTROL.
+ The AuthenticatedUsers group gets READ access.
+storage_class:
+ '': Default
+ STANDARD: Standard storage class
+ GLACIER: Archive storage mode
+ STANDARD_IA: Infrequent access storage mode
+server_side_encryption:
+ '': None
+ AES256: AES256
+bucket_acl: true
+sse_customer_algorithm: true
+sse_customer_key: true
+sse_customer_key_base64: true
+sse_customer_key_md5: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Cloudflare.yaml b/backend/s3/provider/Cloudflare.yaml
new file mode 100644
index 000000000..991a212c0
--- /dev/null
+++ b/backend/s3/provider/Cloudflare.yaml
@@ -0,0 +1,8 @@
+name: Cloudflare
+description: Cloudflare R2 Storage
+region:
+ auto: R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
+endpoint: {}
+quirks:
+ force_path_style: true
+ use_multipart_etag: false # multipart ETags are random
diff --git a/backend/s3/provider/Cubbit.yaml b/backend/s3/provider/Cubbit.yaml
new file mode 100644
index 000000000..19907b272
--- /dev/null
+++ b/backend/s3/provider/Cubbit.yaml
@@ -0,0 +1,10 @@
+name: Cubbit
+description: Cubbit DS3 Object Storage
+region:
+ eu-west-1: Europe West
+endpoint:
+ s3.cubbit.eu: Cubbit DS3 Object Storage endpoint
+acl: {}
+bucket_acl: true
+quirks:
+ use_multipart_etag: false
diff --git a/backend/s3/provider/DigitalOcean.yaml b/backend/s3/provider/DigitalOcean.yaml
new file mode 100644
index 000000000..1a64d47af
--- /dev/null
+++ b/backend/s3/provider/DigitalOcean.yaml
@@ -0,0 +1,20 @@
+name: DigitalOcean
+description: DigitalOcean Spaces
+region: {}
+endpoint:
+ syd1.digitaloceanspaces.com: DigitalOcean Spaces Sydney 1
+ sfo3.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 3
+ sfo2.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 2
+ fra1.digitaloceanspaces.com: DigitalOcean Spaces Frankfurt 1
+ nyc3.digitaloceanspaces.com: DigitalOcean Spaces New York 3
+ ams3.digitaloceanspaces.com: DigitalOcean Spaces Amsterdam 3
+ sgp1.digitaloceanspaces.com: DigitalOcean Spaces Singapore 1
+ lon1.digitaloceanspaces.com: DigitalOcean Spaces London 1
+ tor1.digitaloceanspaces.com: DigitalOcean Spaces Toronto 1
+ blr1.digitaloceanspaces.com: DigitalOcean Spaces Bangalore 1
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Dreamhost.yaml b/backend/s3/provider/Dreamhost.yaml
new file mode 100644
index 000000000..ab5e87612
--- /dev/null
+++ b/backend/s3/provider/Dreamhost.yaml
@@ -0,0 +1,11 @@
+name: Dreamhost
+description: Dreamhost DreamObjects
+region: {}
+endpoint:
+ objects-us-east-1.dream.io: Dream Objects endpoint
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Exaba.yaml b/backend/s3/provider/Exaba.yaml
new file mode 100644
index 000000000..f30a5b698
--- /dev/null
+++ b/backend/s3/provider/Exaba.yaml
@@ -0,0 +1,9 @@
+name: Exaba
+description: Exaba Object Storage
+region: {}
+endpoint: {}
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ force_path_style: true
diff --git a/backend/s3/provider/FileLu.yaml b/backend/s3/provider/FileLu.yaml
new file mode 100644
index 000000000..f725c0ce7
--- /dev/null
+++ b/backend/s3/provider/FileLu.yaml
@@ -0,0 +1,21 @@
+name: FileLu
+description: FileLu S5 (S3-Compatible Object Storage)
+region:
+ global: Global
+ us-east: North America (US-East)
+ eu-central: Europe (EU-Central)
+ ap-southeast: Asia Pacific (AP-Southeast)
+ me-central: Middle East (ME-Central)
+endpoint:
+ s5lu.com: Global FileLu S5 endpoint
+ us.s5lu.com: North America (US-East) region endpoint
+ eu.s5lu.com: Europe (EU-Central) region endpoint
+ ap.s5lu.com: Asia Pacific (AP-Southeast) region endpoint
+ me.s5lu.com: Middle East (ME-Central) region endpoint
+acl: {}
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false
diff --git a/backend/s3/provider/FlashBlade.yaml b/backend/s3/provider/FlashBlade.yaml
new file mode 100644
index 000000000..a4adfd986
--- /dev/null
+++ b/backend/s3/provider/FlashBlade.yaml
@@ -0,0 +1,6 @@
+name: FlashBlade
+description: Pure Storage FlashBlade Object Storage
+endpoint: {}
+quirks:
+ might_gzip: false # never auto-gzip
+ force_path_style: true # supports vhost but defaults to path-style
diff --git a/backend/s3/provider/GCS.yaml b/backend/s3/provider/GCS.yaml
new file mode 100644
index 000000000..4ece3251d
--- /dev/null
+++ b/backend/s3/provider/GCS.yaml
@@ -0,0 +1,20 @@
+name: GCS
+description: Google Cloud Storage
+region: {}
+endpoint:
+ https://storage.googleapis.com: Google Cloud Storage endpoint
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ # Google break request Signature by mutating accept-encoding HTTP header
+ # https://github.com/rclone/rclone/issues/6670
+ use_accept_encoding_gzip: false
+ sign_accept_encoding: false
+ use_already_exists: true # returns BucketNameUnavailable instead of BucketAlreadyExists but good enough!
+ # GCS doesn't like the x-id URL parameter the SDKv2 inserts
+ use_x_id: false
+ # GCS S3 doesn't support multi-part server side copy:
+ # See: https://issuetracker.google.com/issues/323465186
+ # So make cutoff very large which it does seem to support
+ copy_cutoff: 9223372036854775807
diff --git a/backend/s3/provider/Hetzner.yaml b/backend/s3/provider/Hetzner.yaml
new file mode 100644
index 000000000..a7218a3da
--- /dev/null
+++ b/backend/s3/provider/Hetzner.yaml
@@ -0,0 +1,15 @@
+name: Hetzner
+description: Hetzner Object Storage
+region:
+ hel1: Helsinki
+ fsn1: Falkenstein
+ nbg1: Nuremberg
+endpoint:
+ hel1.your-objectstorage.com: Helsinki
+ fsn1.your-objectstorage.com: Falkenstein
+ nbg1.your-objectstorage.com: Nuremberg
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ use_already_exists: false
diff --git a/backend/s3/provider/HuaweiOBS.yaml b/backend/s3/provider/HuaweiOBS.yaml
new file mode 100644
index 000000000..9811ca497
--- /dev/null
+++ b/backend/s3/provider/HuaweiOBS.yaml
@@ -0,0 +1,41 @@
+name: HuaweiOBS
+description: Huawei Object Storage Service
+region:
+ af-south-1: AF-Johannesburg
+ ap-southeast-2: AP-Bangkok
+ ap-southeast-3: AP-Singapore
+ cn-east-3: CN East-Shanghai1
+ cn-east-2: CN East-Shanghai2
+ cn-north-1: CN North-Beijing1
+ cn-north-4: CN North-Beijing4
+ cn-south-1: CN South-Guangzhou
+ ap-southeast-1: CN-Hong Kong
+ sa-argentina-1: LA-Buenos Aires1
+ sa-peru-1: LA-Lima1
+ na-mexico-1: LA-Mexico City1
+ sa-chile-1: LA-Santiago2
+ sa-brazil-1: LA-Sao Paulo1
+ ru-northwest-2: RU-Moscow2
+endpoint:
+ obs.af-south-1.myhuaweicloud.com: AF-Johannesburg
+ obs.ap-southeast-2.myhuaweicloud.com: AP-Bangkok
+ obs.ap-southeast-3.myhuaweicloud.com: AP-Singapore
+ obs.cn-east-3.myhuaweicloud.com: CN East-Shanghai1
+ obs.cn-east-2.myhuaweicloud.com: CN East-Shanghai2
+ obs.cn-north-1.myhuaweicloud.com: CN North-Beijing1
+ obs.cn-north-4.myhuaweicloud.com: CN North-Beijing4
+ obs.cn-south-1.myhuaweicloud.com: CN South-Guangzhou
+ obs.ap-southeast-1.myhuaweicloud.com: CN-Hong Kong
+ obs.sa-argentina-1.myhuaweicloud.com: LA-Buenos Aires1
+ obs.sa-peru-1.myhuaweicloud.com: LA-Lima1
+ obs.na-mexico-1.myhuaweicloud.com: LA-Mexico City1
+ obs.sa-chile-1.myhuaweicloud.com: LA-Santiago2
+ obs.sa-brazil-1.myhuaweicloud.com: LA-Sao Paulo1
+ obs.ru-northwest-2.myhuaweicloud.com: RU-Moscow2
+acl: {}
+bucket_acl: true
+quirks:
+ # Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever.
+ list_url_encode: false
+ list_version: 1
+ use_already_exists: false
diff --git a/backend/s3/provider/IBMCOS.yaml b/backend/s3/provider/IBMCOS.yaml
new file mode 100644
index 000000000..8b6290fc3
--- /dev/null
+++ b/backend/s3/provider/IBMCOS.yaml
@@ -0,0 +1,126 @@
+name: IBMCOS
+description: IBM COS S3
+region: {}
+endpoint:
+ s3.us.cloud-object-storage.appdomain.cloud: US Cross Region Endpoint
+ s3.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Endpoint
+ s3.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Endpoint
+ s3.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Endpoint
+ s3.private.us.cloud-object-storage.appdomain.cloud: US Cross Region Private Endpoint
+ s3.private.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Private Endpoint
+ s3.private.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Private Endpoint
+ s3.private.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Private Endpoint
+ s3.us-east.cloud-object-storage.appdomain.cloud: US Region East Endpoint
+ s3.private.us-east.cloud-object-storage.appdomain.cloud: US Region East Private Endpoint
+ s3.us-south.cloud-object-storage.appdomain.cloud: US Region South Endpoint
+ s3.private.us-south.cloud-object-storage.appdomain.cloud: US Region South Private Endpoint
+ s3.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Endpoint
+ s3.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Endpoint
+ s3.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Endpoint
+ s3.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Endpoint
+ s3.private.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Private Endpoint
+ s3.private.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Private Endpoint
+ s3.private.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Private Endpoint
+ s3.private.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Private Endpoint
+ s3.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Endpoint
+ s3.private.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Private Endpoint
+ s3.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Endpoint
+ s3.private.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Private Endpoint
+ s3.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Endpoint
+ s3.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Endpoint
+ s3.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Endpoint
+ s3.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Endpoint
+ s3.private.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Private Endpoint
+ s3.private.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Private Endpoint
+ s3.private.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Private Endpoint
+ s3.private.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Private Endpoint
+ s3.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Endpoint
+ s3.private.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Private Endpoint
+ s3.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Endpoint
+ s3.private.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Private Endpoint
+ s3.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Endpoint
+ s3.private.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Private Endpoint
+ s3.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Endpoint
+ s3.private.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Private Endpoint
+ s3.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Endpoint
+ s3.private.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Private Endpoint
+ s3.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Endpoint
+ s3.private.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Private Endpoint
+ s3.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Endpoint
+ s3.private.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Private Endpoint
+ s3.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Endpoint
+ s3.private.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Private Endpoint
+ s3.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Endpoint
+ s3.private.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Private Endpoint
+ s3.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Endpoint
+ s3.private.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Private Endpoint
+ s3.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Endpoint
+ s3.private.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Private Endpoint
+ s3.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Endpoint
+ s3.private.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Private Endpoint
+ s3.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Endpoint
+ s3.private.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Private Endpoint
+ s3.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Endpoint
+ s3.private.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Private Endpoint
+ s3.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Endpoint
+ s3.private.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Private Endpoint
+location_constraint:
+ us-standard: US Cross Region Standard
+ us-vault: US Cross Region Vault
+ us-cold: US Cross Region Cold
+ us-flex: US Cross Region Flex
+ us-east-standard: US East Region Standard
+ us-east-vault: US East Region Vault
+ us-east-cold: US East Region Cold
+ us-east-flex: US East Region Flex
+ us-south-standard: US South Region Standard
+ us-south-vault: US South Region Vault
+ us-south-cold: US South Region Cold
+ us-south-flex: US South Region Flex
+ eu-standard: EU Cross Region Standard
+ eu-vault: EU Cross Region Vault
+ eu-cold: EU Cross Region Cold
+ eu-flex: EU Cross Region Flex
+ eu-gb-standard: Great Britain Standard
+ eu-gb-vault: Great Britain Vault
+ eu-gb-cold: Great Britain Cold
+ eu-gb-flex: Great Britain Flex
+ ap-standard: APAC Standard
+ ap-vault: APAC Vault
+ ap-cold: APAC Cold
+ ap-flex: APAC Flex
+ mel01-standard: Melbourne Standard
+ mel01-vault: Melbourne Vault
+ mel01-cold: Melbourne Cold
+ mel01-flex: Melbourne Flex
+ tor01-standard: Toronto Standard
+ tor01-vault: Toronto Vault
+ tor01-cold: Toronto Cold
+ tor01-flex: Toronto Flex
+acl:
+ private: |-
+ Owner gets FULL_CONTROL.
+ No one else has access rights (default).
+ This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
+ public-read: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ access.
+ This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
+ public-read-write: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ and WRITE access.
+ This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
+ authenticated-read: |-
+ Owner gets FULL_CONTROL.
+ The AuthenticatedUsers group gets READ access.
+ Not supported on Buckets.
+ This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
+ibm_api_key: true
+ibm_resource_instance_id: true
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false
+ use_already_exists: false # returns BucketAlreadyExists
diff --git a/backend/s3/provider/IDrive.yaml b/backend/s3/provider/IDrive.yaml
new file mode 100644
index 000000000..ff43add77
--- /dev/null
+++ b/backend/s3/provider/IDrive.yaml
@@ -0,0 +1,7 @@
+name: IDrive
+description: IDrive e2
+acl: {}
+bucket_acl: true
+quirks:
+ force_path_style: true
+ use_already_exists: false
diff --git a/backend/s3/provider/IONOS.yaml b/backend/s3/provider/IONOS.yaml
new file mode 100644
index 000000000..ab455a15b
--- /dev/null
+++ b/backend/s3/provider/IONOS.yaml
@@ -0,0 +1,17 @@
+name: IONOS
+description: IONOS Cloud
+region:
+ de: Frankfurt, Germany
+ eu-central-2: Berlin, Germany
+ eu-south-2: Logrono, Spain
+endpoint:
+ s3-eu-central-1.ionoscloud.com: Frankfurt, Germany
+ s3-eu-central-2.ionoscloud.com: Berlin, Germany
+ s3-eu-south-2.ionoscloud.com: Logrono, Spain
+acl: {}
+bucket_acl: true
+quirks:
+ # listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2
+ force_path_style: true
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Intercolo.yaml b/backend/s3/provider/Intercolo.yaml
new file mode 100644
index 000000000..51d877351
--- /dev/null
+++ b/backend/s3/provider/Intercolo.yaml
@@ -0,0 +1,10 @@
+name: Intercolo
+description: Intercolo Object Storage
+region:
+ de-fra: Frankfurt, Germany
+endpoint:
+ de-fra.i3storage.com: Frankfurt, Germany
+acl: {}
+bucket_acl: true
+quirks:
+ use_unsigned_payload: false # has trailer support
diff --git a/backend/s3/provider/Leviia.yaml b/backend/s3/provider/Leviia.yaml
new file mode 100644
index 000000000..f79dd49fe
--- /dev/null
+++ b/backend/s3/provider/Leviia.yaml
@@ -0,0 +1,11 @@
+name: Leviia
+description: Leviia Object Storage
+region: {}
+endpoint:
+ s3.leviia.com: |-
+ The default endpoint
+ Leviia
+acl: {}
+bucket_acl: true
+quirks:
+ use_already_exists: false
diff --git a/backend/s3/provider/Liara.yaml b/backend/s3/provider/Liara.yaml
new file mode 100644
index 000000000..c6cff97fb
--- /dev/null
+++ b/backend/s3/provider/Liara.yaml
@@ -0,0 +1,15 @@
+name: Liara
+description: Liara Object Storage
+endpoint:
+ storage.iran.liara.space: |-
+ The default endpoint
+ Iran
+acl: {}
+storage_class:
+ STANDARD: Standard storage class
+bucket_acl: true
+quirks:
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false # mulitpart ETags differ from AWS
+ use_already_exists: false
diff --git a/backend/s3/provider/Linode.yaml b/backend/s3/provider/Linode.yaml
new file mode 100644
index 000000000..0923290d0
--- /dev/null
+++ b/backend/s3/provider/Linode.yaml
@@ -0,0 +1,26 @@
+name: Linode
+description: Linode Object Storage
+endpoint:
+ nl-ams-1.linodeobjects.com: Amsterdam (Netherlands), nl-ams-1
+ us-southeast-1.linodeobjects.com: Atlanta, GA (USA), us-southeast-1
+ in-maa-1.linodeobjects.com: Chennai (India), in-maa-1
+ us-ord-1.linodeobjects.com: Chicago, IL (USA), us-ord-1
+ eu-central-1.linodeobjects.com: Frankfurt (Germany), eu-central-1
+ id-cgk-1.linodeobjects.com: Jakarta (Indonesia), id-cgk-1
+ gb-lon-1.linodeobjects.com: London 2 (Great Britain), gb-lon-1
+ us-lax-1.linodeobjects.com: Los Angeles, CA (USA), us-lax-1
+ es-mad-1.linodeobjects.com: Madrid (Spain), es-mad-1
+ au-mel-1.linodeobjects.com: Melbourne (Australia), au-mel-1
+ us-mia-1.linodeobjects.com: Miami, FL (USA), us-mia-1
+ it-mil-1.linodeobjects.com: Milan (Italy), it-mil-1
+ us-east-1.linodeobjects.com: Newark, NJ (USA), us-east-1
+ jp-osa-1.linodeobjects.com: Osaka (Japan), jp-osa-1
+ fr-par-1.linodeobjects.com: Paris (France), fr-par-1
+ br-gru-1.linodeobjects.com: São Paulo (Brazil), br-gru-1
+ us-sea-1.linodeobjects.com: Seattle, WA (USA), us-sea-1
+ ap-south-1.linodeobjects.com: Singapore, ap-south-1
+ sg-sin-1.linodeobjects.com: Singapore 2, sg-sin-1
+ se-sto-1.linodeobjects.com: Stockholm (Sweden), se-sto-1
+ us-iad-1.linodeobjects.com: Washington, DC, (USA), us-iad-1
+acl: {}
+bucket_acl: true
diff --git a/backend/s3/provider/LyveCloud.yaml b/backend/s3/provider/LyveCloud.yaml
new file mode 100644
index 000000000..c24a4ba20
--- /dev/null
+++ b/backend/s3/provider/LyveCloud.yaml
@@ -0,0 +1,12 @@
+name: LyveCloud
+description: Seagate Lyve Cloud
+region: {}
+endpoint:
+ 's3.us-west-1.{account_name}.lyve.seagate.com': US West 1 - California
+ 's3.eu-west-1.{account_name}.lyve.seagate.com': EU West 1 - Ireland
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ use_multipart_etag: false # multipart ETags differ from AWS
+ use_already_exists: false
diff --git a/backend/s3/provider/Magalu.yaml b/backend/s3/provider/Magalu.yaml
new file mode 100644
index 000000000..4f9afe8f9
--- /dev/null
+++ b/backend/s3/provider/Magalu.yaml
@@ -0,0 +1,16 @@
+name: Magalu
+description: Magalu Object Storage
+endpoint:
+ br-se1.magaluobjects.com: São Paulo, SP (BR), br-se1
+ br-ne1.magaluobjects.com: Fortaleza, CE (BR), br-ne1
+acl: {}
+storage_class:
+ STANDARD: Standard storage class
+ GLACIER_IR: Glacier Instant Retrieval storage class
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Mega.yaml b/backend/s3/provider/Mega.yaml
new file mode 100644
index 000000000..0f2a15fa1
--- /dev/null
+++ b/backend/s3/provider/Mega.yaml
@@ -0,0 +1,15 @@
+name: Mega
+description: MEGA S4 Object Storage
+endpoint:
+ s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam)
+ s3.eu-central-2.s4.mega.io: Mega S4 eu-central-2 (Bettembourg)
+ s3.ca-central-1.s4.mega.io: Mega S4 ca-central-1 (Montreal)
+ s3.ca-west-1.s4.mega.io: Mega S4 ca-west-1 (Vancouver)
+bucket_acl: true
+quirks:
+ list_version: 2
+ force_path_style: true
+ list_url_encode: true
+ use_multipart_etag: false
+ use_already_exists: false
+ copy_cutoff: 9223372036854775807
diff --git a/backend/s3/provider/Minio.yaml b/backend/s3/provider/Minio.yaml
new file mode 100644
index 000000000..2b2c42ace
--- /dev/null
+++ b/backend/s3/provider/Minio.yaml
@@ -0,0 +1,18 @@
+name: Minio
+description: Minio Object Storage
+region: {}
+endpoint: {}
+location_constraint: {}
+acl: {}
+server_side_encryption:
+ '': None
+ AES256: AES256
+ aws:kms: aws:kms
+bucket_acl: true
+sse_customer_algorithm: true
+sse_customer_key: true
+sse_customer_key_base64: true
+sse_customer_key_md5: true
+sse_kms_key_id: true
+quirks:
+ force_path_style: true
diff --git a/backend/s3/provider/Netease.yaml b/backend/s3/provider/Netease.yaml
new file mode 100644
index 000000000..88ed59d19
--- /dev/null
+++ b/backend/s3/provider/Netease.yaml
@@ -0,0 +1,12 @@
+name: Netease
+description: Netease Object Storage (NOS)
+region: {}
+endpoint: {}
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ list_version: 1
+ list_url_encode: false
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/OVHcloud.yaml b/backend/s3/provider/OVHcloud.yaml
new file mode 100644
index 000000000..2220f2a94
--- /dev/null
+++ b/backend/s3/provider/OVHcloud.yaml
@@ -0,0 +1,36 @@
+name: OVHcloud
+description: OVHcloud Object Storage
+region:
+ gra: Gravelines, France
+ rbx: Roubaix, France
+ sbg: Strasbourg, France
+ eu-west-par: Paris, France (3AZ)
+ de: Frankfurt, Germany
+ uk: London, United Kingdom
+ waw: Warsaw, Poland
+ bhs: Beauharnois, Canada
+ ca-east-tor: Toronto, Canada
+ sgp: Singapore
+ ap-southeast-syd: Sydney, Australia
+ ap-south-mum: Mumbai, India
+ us-east-va: Vint Hill, Virginia, USA
+ us-west-or: Hillsboro, Oregon, USA
+ rbx-archive: Roubaix, France (Cold Archive)
+endpoint:
+ s3.gra.io.cloud.ovh.net: OVHcloud Gravelines, France
+ s3.rbx.io.cloud.ovh.net: OVHcloud Roubaix, France
+ s3.sbg.io.cloud.ovh.net: OVHcloud Strasbourg, France
+ s3.eu-west-par.io.cloud.ovh.net: OVHcloud Paris, France (3AZ)
+ s3.de.io.cloud.ovh.net: OVHcloud Frankfurt, Germany
+ s3.uk.io.cloud.ovh.net: OVHcloud London, United Kingdom
+ s3.waw.io.cloud.ovh.net: OVHcloud Warsaw, Poland
+ s3.bhs.io.cloud.ovh.net: OVHcloud Beauharnois, Canada
+ s3.ca-east-tor.io.cloud.ovh.net: OVHcloud Toronto, Canada
+ s3.sgp.io.cloud.ovh.net: OVHcloud Singapore
+ s3.ap-southeast-syd.io.cloud.ovh.net: OVHcloud Sydney, Australia
+ s3.ap-south-mum.io.cloud.ovh.net: OVHcloud Mumbai, India
+ s3.us-east-va.io.cloud.ovh.us: OVHcloud Vint Hill, Virginia, USA
+ s3.us-west-or.io.cloud.ovh.us: OVHcloud Hillsboro, Oregon, USA
+ s3.rbx-archive.io.cloud.ovh.net: OVHcloud Roubaix, France (Cold Archive)
+acl: {}
+bucket_acl: true
diff --git a/backend/s3/provider/Other.yaml b/backend/s3/provider/Other.yaml
new file mode 100644
index 000000000..a32c3fdf2
--- /dev/null
+++ b/backend/s3/provider/Other.yaml
@@ -0,0 +1,39 @@
+name: Other
+description: Any other S3 compatible provider
+region:
+ '': |-
+ Use this if unsure.
+ Will use v4 signatures and an empty region.
+ other-v2-signature: |-
+ Use this only if v4 signatures don't work.
+ E.g. pre Jewel/v10 CEPH.
+endpoint: {}
+location_constraint: {}
+acl:
+ private: |-
+ Owner gets FULL_CONTROL.
+ No one else has access rights (default).
+ public-read: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ access.
+ public-read-write: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ and WRITE access.
+ Granting this on a bucket is generally not recommended.
+ authenticated-read: |-
+ Owner gets FULL_CONTROL.
+ The AuthenticatedUsers group gets READ access.
+ bucket-owner-read: |-
+ Object owner gets FULL_CONTROL.
+ Bucket owner gets READ access.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ bucket-owner-full-control: |-
+ Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Outscale.yaml b/backend/s3/provider/Outscale.yaml
new file mode 100644
index 000000000..e107456df
--- /dev/null
+++ b/backend/s3/provider/Outscale.yaml
@@ -0,0 +1,18 @@
+name: Outscale
+description: OUTSCALE Object Storage (OOS)
+region:
+ eu-west-2: Paris, France
+ us-east-2: New Jersey, USA
+ us-west-1: California, USA
+ cloudgouv-eu-west-1: SecNumCloud, Paris, France
+ ap-northeast-1: Tokyo, Japan
+endpoint:
+ oos.eu-west-2.outscale.com: Outscale EU West 2 (Paris)
+ oos.us-east-2.outscale.com: Outscale US east 2 (New Jersey)
+ oos.us-west-1.outscale.com: Outscale EU West 1 (California)
+ oos.cloudgouv-eu-west-1.outscale.com: Outscale SecNumCloud (Paris)
+ oos.ap-northeast-1.outscale.com: Outscale AP Northeast 1 (Japan)
+acl: {}
+bucket_acl: true
+quirks:
+ force_path_style: true
diff --git a/backend/s3/provider/Petabox.yaml b/backend/s3/provider/Petabox.yaml
new file mode 100644
index 000000000..a28922478
--- /dev/null
+++ b/backend/s3/provider/Petabox.yaml
@@ -0,0 +1,19 @@
+name: Petabox
+description: Petabox Object Storage
+region:
+ us-east-1: US East (N. Virginia)
+ eu-central-1: Europe (Frankfurt)
+ ap-southeast-1: Asia Pacific (Singapore)
+ me-south-1: Middle East (Bahrain)
+ sa-east-1: South America (São Paulo)
+endpoint:
+ s3.petabox.io: US East (N. Virginia)
+ s3.us-east-1.petabox.io: US East (N. Virginia)
+ s3.eu-central-1.petabox.io: Europe (Frankfurt)
+ s3.ap-southeast-1.petabox.io: Asia Pacific (Singapore)
+ s3.me-south-1.petabox.io: Middle East (Bahrain)
+ s3.sa-east-1.petabox.io: South America (São Paulo)
+acl: {}
+bucket_acl: true
+quirks:
+ use_already_exists: false
diff --git a/backend/s3/provider/Qiniu.yaml b/backend/s3/provider/Qiniu.yaml
new file mode 100644
index 000000000..a1f8df282
--- /dev/null
+++ b/backend/s3/provider/Qiniu.yaml
@@ -0,0 +1,53 @@
+name: Qiniu
+description: Qiniu Object Storage (Kodo)
+region:
+ cn-east-1: |-
+ The default endpoint - a good choice if you are unsure.
+ East China Region 1.
+ Needs location constraint cn-east-1.
+ cn-east-2: |-
+ East China Region 2.
+ Needs location constraint cn-east-2.
+ cn-north-1: |-
+ North China Region 1.
+ Needs location constraint cn-north-1.
+ cn-south-1: |-
+ South China Region 1.
+ Needs location constraint cn-south-1.
+ us-north-1: |-
+ North America Region.
+ Needs location constraint us-north-1.
+ ap-southeast-1: |-
+ Southeast Asia Region 1.
+ Needs location constraint ap-southeast-1.
+ ap-northeast-1: |-
+ Northeast Asia Region 1.
+ Needs location constraint ap-northeast-1.
+endpoint:
+ s3-cn-east-1.qiniucs.com: East China Endpoint 1
+ s3-cn-east-2.qiniucs.com: East China Endpoint 2
+ s3-cn-north-1.qiniucs.com: North China Endpoint 1
+ s3-cn-south-1.qiniucs.com: South China Endpoint 1
+ s3-us-north-1.qiniucs.com: North America Endpoint 1
+ s3-ap-southeast-1.qiniucs.com: Southeast Asia Endpoint 1
+ s3-ap-northeast-1.qiniucs.com: Northeast Asia Endpoint 1
+location_constraint:
+ cn-east-1: East China Region 1
+ cn-east-2: East China Region 2
+ cn-north-1: North China Region 1
+ cn-south-1: South China Region 1
+ us-north-1: North America Region 1
+ ap-southeast-1: Southeast Asia Region 1
+ ap-northeast-1: Northeast Asia Region 1
+acl: {}
+storage_class:
+ STANDARD: Standard storage class
+ LINE: Infrequent access storage mode
+ GLACIER: Archive storage mode
+ DEEP_ARCHIVE: Deep archive storage mode
+bucket_acl: true
+quirks:
+ use_multipart_etag: false
+ list_url_encode: false
+ force_path_style: true
+ use_already_exists: false
diff --git a/backend/s3/provider/Rabata.yaml b/backend/s3/provider/Rabata.yaml
new file mode 100644
index 000000000..dd5f5d17b
--- /dev/null
+++ b/backend/s3/provider/Rabata.yaml
@@ -0,0 +1,15 @@
+name: Rabata
+description: Rabata Cloud Storage
+region:
+ us-east-1: US East (N. Virginia)
+ eu-west-1: EU (Ireland)
+ eu-west-2: EU (London)
+endpoint:
+ s3.us-east-1.rabata.io: US East (N. Virginia)
+ s3.eu-west-1.rabata.io: EU West (Ireland)
+ s3.eu-west-2.rabata.io: EU West (London)
+location_constraint:
+ us-east-1: US East (N. Virginia)
+ eu-west-1: EU (Ireland)
+ eu-west-2: EU (London)
+# server side copy not supported
diff --git a/backend/s3/provider/RackCorp.yaml b/backend/s3/provider/RackCorp.yaml
new file mode 100644
index 000000000..e2030480e
--- /dev/null
+++ b/backend/s3/provider/RackCorp.yaml
@@ -0,0 +1,67 @@
+name: RackCorp
+description: RackCorp Object Storage
+region:
+ global: Global CDN (All locations) Region
+ au: Australia (All states)
+ au-nsw: NSW (Australia) Region
+ au-qld: QLD (Australia) Region
+ au-vic: VIC (Australia) Region
+ au-wa: Perth (Australia) Region
+ ph: Manila (Philippines) Region
+ th: Bangkok (Thailand) Region
+ hk: HK (Hong Kong) Region
+ mn: Ulaanbaatar (Mongolia) Region
+ kg: Bishkek (Kyrgyzstan) Region
+ id: Jakarta (Indonesia) Region
+ jp: Tokyo (Japan) Region
+ sg: SG (Singapore) Region
+ de: Frankfurt (Germany) Region
+ us: USA (AnyCast) Region
+ us-east-1: New York (USA) Region
+ us-west-1: Freemont (USA) Region
+ nz: Auckland (New Zealand) Region
+endpoint:
+ s3.rackcorp.com: Global (AnyCast) Endpoint
+ au.s3.rackcorp.com: Australia (Anycast) Endpoint
+ au-nsw.s3.rackcorp.com: Sydney (Australia) Endpoint
+ au-qld.s3.rackcorp.com: Brisbane (Australia) Endpoint
+ au-vic.s3.rackcorp.com: Melbourne (Australia) Endpoint
+ au-wa.s3.rackcorp.com: Perth (Australia) Endpoint
+ ph.s3.rackcorp.com: Manila (Philippines) Endpoint
+ th.s3.rackcorp.com: Bangkok (Thailand) Endpoint
+ hk.s3.rackcorp.com: HK (Hong Kong) Endpoint
+ mn.s3.rackcorp.com: Ulaanbaatar (Mongolia) Endpoint
+ kg.s3.rackcorp.com: Bishkek (Kyrgyzstan) Endpoint
+ id.s3.rackcorp.com: Jakarta (Indonesia) Endpoint
+ jp.s3.rackcorp.com: Tokyo (Japan) Endpoint
+ sg.s3.rackcorp.com: SG (Singapore) Endpoint
+ de.s3.rackcorp.com: Frankfurt (Germany) Endpoint
+ us.s3.rackcorp.com: USA (AnyCast) Endpoint
+ us-east-1.s3.rackcorp.com: New York (USA) Endpoint
+ us-west-1.s3.rackcorp.com: Freemont (USA) Endpoint
+ nz.s3.rackcorp.com: Auckland (New Zealand) Endpoint
+location_constraint:
+ global: Global CDN Region
+ au: Australia (All locations)
+ au-nsw: NSW (Australia) Region
+ au-qld: QLD (Australia) Region
+ au-vic: VIC (Australia) Region
+ au-wa: Perth (Australia) Region
+ ph: Manila (Philippines) Region
+ th: Bangkok (Thailand) Region
+ hk: HK (Hong Kong) Region
+ mn: Ulaanbaatar (Mongolia) Region
+ kg: Bishkek (Kyrgyzstan) Region
+ id: Jakarta (Indonesia) Region
+ jp: Tokyo (Japan) Region
+ sg: SG (Singapore) Region
+ de: Frankfurt (Germany) Region
+ us: USA (AnyCast) Region
+ us-east-1: New York (USA) Region
+ us-west-1: Fremont (USA) Region
+ nz: Auckland (New Zealand) Region
+acl: {}
+bucket_acl: true
+quirks:
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Rclone.yaml b/backend/s3/provider/Rclone.yaml
new file mode 100644
index 000000000..6697c7323
--- /dev/null
+++ b/backend/s3/provider/Rclone.yaml
@@ -0,0 +1,11 @@
+name: Rclone
+description: Rclone S3 Server
+endpoint: {}
+quirks:
+ force_path_style: true
+ use_multipart_etag: false
+ use_already_exists: false
+ # rclone serve doesn't support multi-part server side copy:
+ # See: https://github.com/rclone/rclone/issues/7454
+ # So make cutoff very large which it does support
+ copy_cutoff: 9223372036854775807
diff --git a/backend/s3/provider/Scaleway.yaml b/backend/s3/provider/Scaleway.yaml
new file mode 100644
index 000000000..29497b945
--- /dev/null
+++ b/backend/s3/provider/Scaleway.yaml
@@ -0,0 +1,28 @@
+name: Scaleway
+description: Scaleway Object Storage
+region:
+ nl-ams: Amsterdam, The Netherlands
+ fr-par: Paris, France
+ pl-waw: Warsaw, Poland
+endpoint:
+ s3.nl-ams.scw.cloud: Amsterdam Endpoint
+ s3.fr-par.scw.cloud: Paris Endpoint
+ s3.pl-waw.scw.cloud: Warsaw Endpoint
+acl: {}
+storage_class:
+ '': Default.
+ STANDARD: |-
+ The Standard class for any upload.
+ Suitable for on-demand content like streaming or CDN.
+ Available in all regions.
+ GLACIER: |-
+ Archived storage.
+ Prices are lower, but it needs to be restored first to be accessed.
+ Available in FR-PAR and NL-AMS regions.
+ ONEZONE_IA: |-
+ One Zone - Infrequent Access.
+ A good choice for storing secondary backup copies or easily re-creatable data.
+ Available in the FR-PAR region only.
+bucket_acl: true
+quirks:
+ max_upload_parts: 1000
diff --git a/backend/s3/provider/SeaweedFS.yaml b/backend/s3/provider/SeaweedFS.yaml
new file mode 100644
index 000000000..1e377e09a
--- /dev/null
+++ b/backend/s3/provider/SeaweedFS.yaml
@@ -0,0 +1,14 @@
+name: SeaweedFS
+description: SeaweedFS S3
+region: {}
+endpoint:
+ localhost:8333: SeaweedFS S3 localhost
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Selectel.yaml b/backend/s3/provider/Selectel.yaml
new file mode 100644
index 000000000..70cc74bad
--- /dev/null
+++ b/backend/s3/provider/Selectel.yaml
@@ -0,0 +1,8 @@
+name: Selectel
+description: Selectel Object Storage
+region:
+ ru-1: St. Petersburg
+endpoint:
+ s3.ru-1.storage.selcloud.ru: Saint Petersburg
+quirks:
+ list_url_encode: false
diff --git a/backend/s3/provider/Servercore.yaml b/backend/s3/provider/Servercore.yaml
new file mode 100644
index 000000000..cc566e109
--- /dev/null
+++ b/backend/s3/provider/Servercore.yaml
@@ -0,0 +1,17 @@
+name: Servercore
+description: Servercore Object Storage
+region:
+ ru-1: St. Petersburg
+ gis-1: Moscow
+ ru-7: Moscow
+ uz-2: Tashkent, Uzbekistan
+ kz-1: Almaty, Kazakhstan
+endpoint:
+ s3.ru-1.storage.selcloud.ru: Saint Petersburg
+ s3.gis-1.storage.selcloud.ru: Moscow
+ s3.ru-7.storage.selcloud.ru: Moscow
+ s3.uz-2.srvstorage.uz: Tashkent, Uzbekistan
+ s3.kz-1.srvstorage.kz: Almaty, Kazakhstan
+bucket_acl: true
+quirks:
+ list_url_encode: false
diff --git a/backend/s3/provider/SpectraLogic.yaml b/backend/s3/provider/SpectraLogic.yaml
new file mode 100644
index 000000000..8ed135616
--- /dev/null
+++ b/backend/s3/provider/SpectraLogic.yaml
@@ -0,0 +1,5 @@
+name: SpectraLogic
+description: Spectra Logic Black Pearl
+endpoint: {}
+quirks:
+ force_path_style: true # path-style required
diff --git a/backend/s3/provider/StackPath.yaml b/backend/s3/provider/StackPath.yaml
new file mode 100644
index 000000000..9b3723f3f
--- /dev/null
+++ b/backend/s3/provider/StackPath.yaml
@@ -0,0 +1,14 @@
+name: StackPath
+description: StackPath Object Storage
+region: {}
+endpoint:
+ s3.us-east-2.stackpathstorage.com: US East Endpoint
+ s3.us-west-1.stackpathstorage.com: US West Endpoint
+ s3.eu-central-1.stackpathstorage.com: EU Endpoint
+acl: {}
+bucket_acl: true
+quirks:
+ list_version: 1
+ force_path_style: true
+ list_url_encode: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Storj.yaml b/backend/s3/provider/Storj.yaml
new file mode 100644
index 000000000..11697be85
--- /dev/null
+++ b/backend/s3/provider/Storj.yaml
@@ -0,0 +1,11 @@
+name: Storj
+description: Storj (S3 Compatible Gateway)
+endpoint:
+ gateway.storjshare.io: Global Hosted Gateway
+quirks:
+ use_already_exists: false # returns BucketAlreadyExists
+ # Storj doesn't support multi-part server side copy:
+ # https://github.com/storj/roadmap/issues/40
+ # So make cutoff very large which it does support
+ copy_cutoff: 9223372036854775807
+ min_chunk_size: 67108864
diff --git a/backend/s3/provider/Synology.yaml b/backend/s3/provider/Synology.yaml
new file mode 100644
index 000000000..8efeed610
--- /dev/null
+++ b/backend/s3/provider/Synology.yaml
@@ -0,0 +1,18 @@
+name: Synology
+description: Synology C2 Object Storage
+region:
+ eu-001: Europe Region 1
+ eu-002: Europe Region 2
+ us-001: US Region 1
+ us-002: US Region 2
+ tw-001: Asia (Taiwan)
+endpoint:
+ eu-001.s3.synologyc2.net: EU Endpoint 1
+ eu-002.s3.synologyc2.net: EU Endpoint 2
+ us-001.s3.synologyc2.net: US Endpoint 1
+ us-002.s3.synologyc2.net: US Endpoint 2
+ tw-001.s3.synologyc2.net: TW Endpoint 1
+location_constraint: {}
+quirks:
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/TencentCOS.yaml b/backend/s3/provider/TencentCOS.yaml
new file mode 100644
index 000000000..ca7a8d30c
--- /dev/null
+++ b/backend/s3/provider/TencentCOS.yaml
@@ -0,0 +1,52 @@
+name: TencentCOS
+description: Tencent Cloud Object Storage (COS)
+endpoint:
+ cos.ap-beijing.myqcloud.com: Beijing Region
+ cos.ap-nanjing.myqcloud.com: Nanjing Region
+ cos.ap-shanghai.myqcloud.com: Shanghai Region
+ cos.ap-guangzhou.myqcloud.com: Guangzhou Region
+ cos.ap-chengdu.myqcloud.com: Chengdu Region
+ cos.ap-chongqing.myqcloud.com: Chongqing Region
+ cos.ap-hongkong.myqcloud.com: Hong Kong (China) Region
+ cos.ap-singapore.myqcloud.com: Singapore Region
+ cos.ap-mumbai.myqcloud.com: Mumbai Region
+ cos.ap-seoul.myqcloud.com: Seoul Region
+ cos.ap-bangkok.myqcloud.com: Bangkok Region
+ cos.ap-tokyo.myqcloud.com: Tokyo Region
+ cos.na-siliconvalley.myqcloud.com: Silicon Valley Region
+ cos.na-ashburn.myqcloud.com: Virginia Region
+ cos.na-toronto.myqcloud.com: Toronto Region
+ cos.eu-frankfurt.myqcloud.com: Frankfurt Region
+ cos.eu-moscow.myqcloud.com: Moscow Region
+ cos.accelerate.myqcloud.com: Use Tencent COS Accelerate Endpoint
+acl:
+ default: |-
+ Owner gets Full_CONTROL.
+ No one else has access rights (default).
+ public-read: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ access.
+ public-read-write: |-
+ Owner gets FULL_CONTROL.
+ The AllUsers group gets READ and WRITE access.
+ Granting this on a bucket is generally not recommended.
+ authenticated-read: |-
+ Owner gets FULL_CONTROL.
+ The AuthenticatedUsers group gets READ access.
+ bucket-owner-read: |-
+ Object owner gets FULL_CONTROL.
+ Bucket owner gets READ access.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+ bucket-owner-full-control: |-
+ Both the object owner and the bucket owner get FULL_CONTROL over the object.
+ If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
+storage_class:
+ '': Default
+ STANDARD: Standard storage class
+ ARCHIVE: Archive storage mode
+ STANDARD_IA: Infrequent access storage mode
+bucket_acl: true
+quirks:
+ list_version: 1
+ use_multipart_etag: false
+ use_already_exists: false
diff --git a/backend/s3/provider/Wasabi.yaml b/backend/s3/provider/Wasabi.yaml
new file mode 100644
index 000000000..e44e61f76
--- /dev/null
+++ b/backend/s3/provider/Wasabi.yaml
@@ -0,0 +1,21 @@
+name: Wasabi
+description: Wasabi Object Storage
+region: {}
+endpoint:
+ s3.wasabisys.com: Wasabi US East 1 (N. Virginia)
+ s3.us-east-2.wasabisys.com: Wasabi US East 2 (N. Virginia)
+ s3.us-central-1.wasabisys.com: Wasabi US Central 1 (Texas)
+ s3.us-west-1.wasabisys.com: Wasabi US West 1 (Oregon)
+ s3.ca-central-1.wasabisys.com: Wasabi CA Central 1 (Toronto)
+ s3.eu-central-1.wasabisys.com: Wasabi EU Central 1 (Amsterdam)
+ s3.eu-central-2.wasabisys.com: Wasabi EU Central 2 (Frankfurt)
+ s3.eu-west-1.wasabisys.com: Wasabi EU West 1 (London)
+ s3.eu-west-2.wasabisys.com: Wasabi EU West 2 (Paris)
+ s3.eu-south-1.wasabisys.com: Wasabi EU South 1 (Milan)
+ s3.ap-northeast-1.wasabisys.com: Wasabi AP Northeast 1 (Tokyo) endpoint
+ s3.ap-northeast-2.wasabisys.com: Wasabi AP Northeast 2 (Osaka) endpoint
+ s3.ap-southeast-1.wasabisys.com: Wasabi AP Southeast 1 (Singapore)
+ s3.ap-southeast-2.wasabisys.com: Wasabi AP Southeast 2 (Sydney)
+location_constraint: {}
+acl: {}
+bucket_acl: true
diff --git a/backend/s3/provider/Zata.yaml b/backend/s3/provider/Zata.yaml
new file mode 100644
index 000000000..ac8cb1d88
--- /dev/null
+++ b/backend/s3/provider/Zata.yaml
@@ -0,0 +1,14 @@
+name: Zata
+description: Zata (S3 compatible Gateway)
+region:
+ us-east-1: Indore, Madhya Pradesh, India
+endpoint:
+ idr01.zata.ai: South Asia Endpoint
+location_constraint: {}
+acl: {}
+bucket_acl: true
+quirks:
+ use_multipart_etag: false
+ might_gzip: false
+ use_unsigned_payload: false
+ use_already_exists: false
diff --git a/backend/s3/providers.go b/backend/s3/providers.go
new file mode 100644
index 000000000..3d42aebe9
--- /dev/null
+++ b/backend/s3/providers.go
@@ -0,0 +1,236 @@
+package s3
+
+import (
+ "embed"
+ stdfs "io/fs"
+ "os"
+ "sort"
+ "strings"
+
+ "github.com/rclone/rclone/fs"
+ orderedmap "github.com/wk8/go-ordered-map/v2"
+ "gopkg.in/yaml.v3"
+)
+
+// YamlMap is converted to YAML in the correct order
+type YamlMap = *orderedmap.OrderedMap[string, string]
+
+// NewYamlMap creates a new ordered map
+var NewYamlMap = orderedmap.New[string, string]
+
+// Quirks defines all the S3 provider quirks
+type Quirks struct {
+ ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2
+ ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style
+ ListURLEncode *bool `yaml:"list_url_encode,omitempty"`
+ UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"`
+ UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"`
+ UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"`
+ MightGzip *bool `yaml:"might_gzip,omitempty"`
+ UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"`
+ UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"`
+ UseXID *bool `yaml:"use_x_id,omitempty"`
+ SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
+ CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
+ MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
+ MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
+}
+
+// Provider defines the configurable data in each provider.yaml
+type Provider struct {
+ Name string `yaml:"name,omitempty"`
+ Description string `yaml:"description,omitempty"`
+ Region YamlMap `yaml:"region,omitempty"`
+ Endpoint YamlMap `yaml:"endpoint,omitempty"`
+ LocationConstraint YamlMap `yaml:"location_constraint,omitempty"`
+ ACL YamlMap `yaml:"acl,omitempty"`
+ StorageClass YamlMap `yaml:"storage_class,omitempty"`
+ ServerSideEncryption YamlMap `yaml:"server_side_encryption,omitempty"`
+
+ // other
+ IBMApiKey bool `yaml:"ibm_api_key,omitempty"`
+ IBMResourceInstanceID bool `yaml:"ibm_resource_instance_id,omitempty"`
+
+ // advanced
+ BucketACL bool `yaml:"bucket_acl,omitempty"`
+ DirectoryBucket bool `yaml:"directory_bucket,omitempty"`
+ LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"`
+ RequesterPays bool `yaml:"requester_pays,omitempty"`
+ SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"`
+ SSECustomerKey bool `yaml:"sse_customer_key,omitempty"`
+ SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"`
+ SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"`
+ SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"`
+ STSEndpoint bool `yaml:"sts_endpoint,omitempty"`
+ UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"`
+
+ Quirks Quirks `yaml:"quirks,omitempty"`
+}
+
+//go:embed provider/*.yaml
+var providerFS embed.FS
+
+// addProvidersToInfo adds provider information to the fs.RegInfo
+func addProvidersToInfo(info *fs.RegInfo) *fs.RegInfo {
+ providerMap := loadProviders()
+ providerList := constructProviders(info.Options, providerMap)
+ info.Description += strings.TrimSuffix(providerList, ", ")
+ return info
+}
+
+// loadProvider loads a single provider
+//
+// It returns nil if it could not be found except if "Other" which is a fatal error.
+func loadProvider(name string) *Provider {
+ data, err := stdfs.ReadFile(providerFS, "provider/"+name+".yaml")
+ if err != nil {
+ if os.IsNotExist(err) && name != "Other" {
+ return nil
+ }
+ fs.Fatalf(nil, "internal error: failed to load provider %q: %v", name, err)
+ }
+ var p Provider
+ err = yaml.Unmarshal(data, &p)
+ if err != nil {
+ fs.Fatalf(nil, "internal error: failed to unmarshal provider %q: %v", name, err)
+ }
+ return &p
+}
+
+// loadProviders loads provider definitions from embedded YAML files
+func loadProviders() map[string]*Provider {
+ providers, err := stdfs.ReadDir(providerFS, "provider")
+ if err != nil {
+ fs.Fatalf(nil, "internal error: failed to read embedded providers: %v", err)
+ }
+ providerMap := make(map[string]*Provider, len(providers))
+
+ for _, provider := range providers {
+ name, _ := strings.CutSuffix(provider.Name(), ".yaml")
+ p := loadProvider(name)
+ providerMap[p.Name] = p
+ }
+ return providerMap
+}
+
+// constructProviders populates fs.Options with provider-specific examples and information
+func constructProviders(options fs.Options, providerMap map[string]*Provider) string {
+ // Defaults for map options set to {}
+ defaults := providerMap["Other"]
+
+ // sort providers: AWS first, Other last, rest alphabetically
+ providers := make([]*Provider, 0, len(providerMap))
+ for _, p := range providerMap {
+ providers = append(providers, p)
+ }
+ sort.Slice(providers, func(i, j int) bool {
+ if providers[i].Name == "AWS" {
+ return true
+ }
+ if providers[j].Name == "AWS" {
+ return false
+ }
+ if providers[i].Name == "Other" {
+ return false
+ }
+ if providers[j].Name == "Other" {
+ return true
+ }
+ return strings.ToLower(providers[i].Name) < strings.ToLower(providers[j].Name)
+ })
+
+ addProvider := func(sp *string, name string) {
+ if *sp != "" {
+ *sp += ","
+ }
+ *sp += name
+ }
+
+ addBool := func(opt *fs.Option, p *Provider, flag bool) {
+ if flag {
+ addProvider(&opt.Provider, p.Name)
+ }
+ }
+
+ addExample := func(opt *fs.Option, p *Provider, examples, defaultExamples YamlMap) {
+ if examples == nil {
+ return
+ }
+ if examples.Len() == 0 {
+ examples = defaultExamples
+ }
+ addProvider(&opt.Provider, p.Name)
+ OUTER:
+ for pair := examples.Oldest(); pair != nil; pair = pair.Next() {
+ // Find an existing example to add to if possible
+ for i, example := range opt.Examples {
+ if example.Value == pair.Key && example.Help == pair.Value {
+ addProvider(&opt.Examples[i].Provider, p.Name)
+ continue OUTER
+ }
+ }
+ // Otherwise add a new one
+ opt.Examples = append(opt.Examples, fs.OptionExample{
+ Value: pair.Key,
+ Help: pair.Value,
+ Provider: p.Name,
+ })
+ }
+ }
+
+ var providerList string
+
+ for _, p := range providers {
+ for i := range options {
+ opt := &options[i]
+ switch opt.Name {
+ case "provider":
+ opt.Examples = append(opt.Examples, fs.OptionExample{
+ Value: p.Name,
+ Help: p.Description,
+ })
+ providerList += p.Name + ", "
+ case "region":
+ addExample(opt, p, p.Region, defaults.Region)
+ case "endpoint":
+ addExample(opt, p, p.Endpoint, defaults.Endpoint)
+ case "location_constraint":
+ addExample(opt, p, p.LocationConstraint, defaults.LocationConstraint)
+ case "acl":
+ addExample(opt, p, p.ACL, defaults.ACL)
+ case "storage_class":
+ addExample(opt, p, p.StorageClass, defaults.StorageClass)
+ case "server_side_encryption":
+ addExample(opt, p, p.ServerSideEncryption, defaults.ServerSideEncryption)
+ case "bucket_acl":
+ addBool(opt, p, p.BucketACL)
+ case "requester_pays":
+ addBool(opt, p, p.RequesterPays)
+ case "sse_customer_algorithm":
+ addBool(opt, p, p.SSECustomerAlgorithm)
+ case "sse_kms_key_id":
+ addBool(opt, p, p.SSEKmsKeyID)
+ case "sse_customer_key":
+ addBool(opt, p, p.SSECustomerKey)
+ case "sse_customer_key_base64":
+ addBool(opt, p, p.SSECustomerKeyBase64)
+ case "sse_customer_key_md5":
+ addBool(opt, p, p.SSECustomerKeyMd5)
+ case "directory_bucket":
+ addBool(opt, p, p.DirectoryBucket)
+ case "ibm_api_key":
+ addBool(opt, p, p.IBMApiKey)
+ case "ibm_resource_instance_id":
+ addBool(opt, p, p.IBMResourceInstanceID)
+ case "leave_parts_on_error":
+ addBool(opt, p, p.LeavePartsOnError)
+ case "sts_endpoint":
+ addBool(opt, p, p.STSEndpoint)
+ case "use_accelerate_endpoint":
+ addBool(opt, p, p.UseAccelerateEndpoint)
+ }
+ }
+ }
+
+ return strings.TrimSuffix(providerList, ", ")
+}
diff --git a/backend/s3/s3.go b/backend/s3/s3.go
index 9ddae5083..79c618566 100644
--- a/backend/s3/s3.go
+++ b/backend/s3/s3.go
@@ -63,2214 +63,55 @@ import (
"golang.org/x/sync/errgroup"
)
-// The S3 providers
-//
-// Please keep these in alphabetical order, but with AWS first and
-// Other last.
-//
-// NB if you add a new provider here, then add it in the setQuirks
-// function and set the correct quirks. Test the quirks are correct by
-// running the integration tests "go test -v -remote NewS3Provider:".
-//
-// See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#adding-a-new-s3-provider
-// for full information about how to add a new s3 provider.
-var providerOption = fs.Option{
- Name: fs.ConfigProvider,
- Help: "Choose your S3 provider.",
- Examples: []fs.OptionExample{{
- Value: "AWS",
- Help: "Amazon Web Services (AWS) S3",
- }, {
- Value: "Alibaba",
- Help: "Alibaba Cloud Object Storage System (OSS) formerly Aliyun",
- }, {
- Value: "ArvanCloud",
- Help: "Arvan Cloud Object Storage (AOS)",
- }, {
- Value: "Ceph",
- Help: "Ceph Object Storage",
- }, {
- Value: "ChinaMobile",
- Help: "China Mobile Ecloud Elastic Object Storage (EOS)",
- }, {
- Value: "Cloudflare",
- Help: "Cloudflare R2 Storage",
- }, {
- Value: "Cubbit",
- Help: "Cubbit DS3 Object Storage",
- }, {
- Value: "DigitalOcean",
- Help: "DigitalOcean Spaces",
- }, {
- Value: "Dreamhost",
- Help: "Dreamhost DreamObjects",
- }, {
- Value: "Exaba",
- Help: "Exaba Object Storage",
- }, {
- Value: "FileLu",
- Help: "FileLu S5 (S3-Compatible Object Storage)",
- }, {
- Value: "FlashBlade",
- Help: "Pure Storage FlashBlade Object Storage",
- }, {
- Value: "GCS",
- Help: "Google Cloud Storage",
- }, {
- Value: "Hetzner",
- Help: "Hetzner Object Storage",
- }, {
- Value: "HuaweiOBS",
- Help: "Huawei Object Storage Service",
- }, {
- Value: "IBMCOS",
- Help: "IBM COS S3",
- }, {
- Value: "IDrive",
- Help: "IDrive e2",
- }, {
- Value: "Intercolo",
- Help: "Intercolo Object Storage",
- }, {
- Value: "IONOS",
- Help: "IONOS Cloud",
- }, {
- Value: "LyveCloud",
- Help: "Seagate Lyve Cloud",
- }, {
- Value: "Leviia",
- Help: "Leviia Object Storage",
- }, {
- Value: "Liara",
- Help: "Liara Object Storage",
- }, {
- Value: "Linode",
- Help: "Linode Object Storage",
- }, {
- Value: "Magalu",
- Help: "Magalu Object Storage",
- }, {
- Value: "Mega",
- Help: "MEGA S4 Object Storage",
- }, {
- Value: "Minio",
- Help: "Minio Object Storage",
- }, {
- Value: "Netease",
- Help: "Netease Object Storage (NOS)",
- }, {
- Value: "Outscale",
- Help: "OUTSCALE Object Storage (OOS)",
- }, {
- Value: "OVHcloud",
- Help: "OVHcloud Object Storage",
- }, {
- Value: "Petabox",
- Help: "Petabox Object Storage",
- }, {
- Value: "Rabata",
- Help: "Rabata Cloud Storage",
- }, {
- Value: "RackCorp",
- Help: "RackCorp Object Storage",
- }, {
- Value: "Rclone",
- Help: "Rclone S3 Server",
- }, {
- Value: "Scaleway",
- Help: "Scaleway Object Storage",
- }, {
- Value: "SeaweedFS",
- Help: "SeaweedFS S3",
- }, {
- Value: "Selectel",
- Help: "Selectel Object Storage",
- }, {
- Value: "Servercore",
- Help: "Servercore Object Storage",
- }, {
- Value: "SpectraLogic",
- Help: "Spectra Logic Black Pearl",
- }, {
- Value: "StackPath",
- Help: "StackPath Object Storage",
- }, {
- Value: "Storj",
- Help: "Storj (S3 Compatible Gateway)",
- }, {
- Value: "Synology",
- Help: "Synology C2 Object Storage",
- }, {
- Value: "TencentCOS",
- Help: "Tencent Cloud Object Storage (COS)",
- }, {
- Value: "Wasabi",
- Help: "Wasabi Object Storage",
- }, {
- Value: "Qiniu",
- Help: "Qiniu Object Storage (Kodo)",
- }, {
- Value: "Zata",
- Help: "Zata (S3 compatible Gateway)",
- }, {
- Value: "Other",
- Help: "Any other S3 compatible provider",
- }},
-}
-
-var providersList string
-
-// Register with Fs
-func init() {
- var s strings.Builder
- for i, provider := range providerOption.Examples {
- if provider.Value == "Other" {
- _, _ = s.WriteString(" and others")
- } else {
- if i != 0 {
- _, _ = s.WriteString(", ")
- }
- _, _ = s.WriteString(provider.Value)
- }
- }
- providersList = s.String()
- fs.Register(&fs.RegInfo{
- Name: "s3",
- Description: "Amazon S3 Compliant Storage Providers including " + providersList,
- NewFs: NewFs,
- CommandHelp: commandHelp,
- Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
- switch config.State {
- case "":
- return nil, setEndpointValueForIDriveE2(m)
- }
- return nil, fmt.Errorf("unknown state %q", config.State)
- },
- MetadataInfo: &fs.MetadataInfo{
- System: systemMetadataInfo,
- Help: `User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.`,
- },
- Options: []fs.Option{providerOption, {
- Name: "env_auth",
- Help: "Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).\n\nOnly applies if access_key_id and secret_access_key is blank.",
- Default: false,
- Examples: []fs.OptionExample{{
- Value: "false",
- Help: "Enter AWS credentials in the next step.",
- }, {
- Value: "true",
- Help: "Get AWS credentials from the environment (env vars or IAM).",
- }},
- }, {
- Name: "access_key_id",
- Help: "AWS Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
- Sensitive: true,
- }, {
- Name: "secret_access_key",
- Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
- Sensitive: true,
- }, {
- // References:
- // 1. https://docs.aws.amazon.com/general/latest/gr/rande.html
- // 2. https://docs.aws.amazon.com/general/latest/gr/s3.html
- Name: "region",
- Help: "Region to connect to.",
- Provider: "AWS",
- Examples: []fs.OptionExample{{
- Value: "us-east-1",
- Help: "The default endpoint - a good choice if you are unsure.\nUS Region, Northern Virginia, or Pacific Northwest.\nLeave location constraint empty.",
- }, {
- Value: "us-east-2",
- Help: "US East (Ohio) Region.\nNeeds location constraint us-east-2.",
- }, {
- Value: "us-west-1",
- Help: "US West (Northern California) Region.\nNeeds location constraint us-west-1.",
- }, {
- Value: "us-west-2",
- Help: "US West (Oregon) Region.\nNeeds location constraint us-west-2.",
- }, {
- Value: "ca-central-1",
- Help: "Canada (Central) Region.\nNeeds location constraint ca-central-1.",
- }, {
- Value: "eu-west-1",
- Help: "EU (Ireland) Region.\nNeeds location constraint EU or eu-west-1.",
- }, {
- Value: "eu-west-2",
- Help: "EU (London) Region.\nNeeds location constraint eu-west-2.",
- }, {
- Value: "eu-west-3",
- Help: "EU (Paris) Region.\nNeeds location constraint eu-west-3.",
- }, {
- Value: "eu-north-1",
- Help: "EU (Stockholm) Region.\nNeeds location constraint eu-north-1.",
- }, {
- Value: "eu-south-1",
- Help: "EU (Milan) Region.\nNeeds location constraint eu-south-1.",
- }, {
- Value: "eu-central-1",
- Help: "EU (Frankfurt) Region.\nNeeds location constraint eu-central-1.",
- }, {
- Value: "ap-southeast-1",
- Help: "Asia Pacific (Singapore) Region.\nNeeds location constraint ap-southeast-1.",
- }, {
- Value: "ap-southeast-2",
- Help: "Asia Pacific (Sydney) Region.\nNeeds location constraint ap-southeast-2.",
- }, {
- Value: "ap-northeast-1",
- Help: "Asia Pacific (Tokyo) Region.\nNeeds location constraint ap-northeast-1.",
- }, {
- Value: "ap-northeast-2",
- Help: "Asia Pacific (Seoul).\nNeeds location constraint ap-northeast-2.",
- }, {
- Value: "ap-northeast-3",
- Help: "Asia Pacific (Osaka-Local).\nNeeds location constraint ap-northeast-3.",
- }, {
- Value: "ap-south-1",
- Help: "Asia Pacific (Mumbai).\nNeeds location constraint ap-south-1.",
- }, {
- Value: "ap-east-1",
- Help: "Asia Pacific (Hong Kong) Region.\nNeeds location constraint ap-east-1.",
- }, {
- Value: "sa-east-1",
- Help: "South America (Sao Paulo) Region.\nNeeds location constraint sa-east-1.",
- }, {
- Value: "il-central-1",
- Help: "Israel (Tel Aviv) Region.\nNeeds location constraint il-central-1.",
- }, {
- Value: "me-south-1",
- Help: "Middle East (Bahrain) Region.\nNeeds location constraint me-south-1.",
- }, {
- Value: "af-south-1",
- Help: "Africa (Cape Town) Region.\nNeeds location constraint af-south-1.",
- }, {
- Value: "cn-north-1",
- Help: "China (Beijing) Region.\nNeeds location constraint cn-north-1.",
- }, {
- Value: "cn-northwest-1",
- Help: "China (Ningxia) Region.\nNeeds location constraint cn-northwest-1.",
- }, {
- Value: "us-gov-east-1",
- Help: "AWS GovCloud (US-East) Region.\nNeeds location constraint us-gov-east-1.",
- }, {
- Value: "us-gov-west-1",
- Help: "AWS GovCloud (US) Region.\nNeeds location constraint us-gov-west-1.",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to.",
- Provider: "Cloudflare",
- Examples: []fs.OptionExample{{
- Value: "auto",
- Help: "R2 buckets are automatically distributed across Cloudflare's data centers for low latency.",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to.",
- Provider: "Cubbit",
- Examples: []fs.OptionExample{{
- Value: "eu-west-1",
- Help: "Europe West",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to for FileLu S5.",
- Provider: "FileLu",
- Examples: []fs.OptionExample{{
- Value: "global",
- Help: "Global",
- }, {
- Value: "us-east",
- Help: "North America (US-East)",
- }, {
- Value: "eu-central",
- Help: "Europe (EU-Central)",
- }, {
- Value: "ap-southeast",
- Help: "Asia Pacific (AP-Southeast)",
- }, {
- Value: "me-central",
- Help: "Middle East (ME-Central)",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to.",
- Provider: "Hetzner",
- Examples: []fs.OptionExample{{
- Value: "hel1",
- Help: "Helsinki",
- }, {
- Value: "fsn1",
- Help: "Falkenstein",
- }, {
- Value: "nbg1",
- Help: "Nuremberg",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to. - the location where your bucket will be created and your data stored. Need to be same with your endpoint.\n",
- Provider: "HuaweiOBS",
- Examples: []fs.OptionExample{{
- Value: "af-south-1",
- Help: "AF-Johannesburg",
- }, {
- Value: "ap-southeast-2",
- Help: "AP-Bangkok",
- }, {
- Value: "ap-southeast-3",
- Help: "AP-Singapore",
- }, {
- Value: "cn-east-3",
- Help: "CN East-Shanghai1",
- }, {
- Value: "cn-east-2",
- Help: "CN East-Shanghai2",
- }, {
- Value: "cn-north-1",
- Help: "CN North-Beijing1",
- }, {
- Value: "cn-north-4",
- Help: "CN North-Beijing4",
- }, {
- Value: "cn-south-1",
- Help: "CN South-Guangzhou",
- }, {
- Value: "ap-southeast-1",
- Help: "CN-Hong Kong",
- }, {
- Value: "sa-argentina-1",
- Help: "LA-Buenos Aires1",
- }, {
- Value: "sa-peru-1",
- Help: "LA-Lima1",
- }, {
- Value: "na-mexico-1",
- Help: "LA-Mexico City1",
- }, {
- Value: "sa-chile-1",
- Help: "LA-Santiago2",
- }, {
- Value: "sa-brazil-1",
- Help: "LA-Sao Paulo1",
- }, {
- Value: "ru-northwest-2",
- Help: "RU-Moscow2",
- }},
- }, {
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "Intercolo",
- Examples: []fs.OptionExample{{
- Value: "de-fra",
- Help: "Frankfurt, Germany",
- }},
- }, {
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "IONOS",
- Examples: []fs.OptionExample{{
- Value: "de",
- Help: "Frankfurt, Germany",
- }, {
- Value: "eu-central-2",
- Help: "Berlin, Germany",
- }, {
- Value: "eu-south-2",
- Help: "Logrono, Spain",
- }},
- }, {
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "Outscale",
- Examples: []fs.OptionExample{{
- Value: "eu-west-2",
- Help: "Paris, France",
- }, {
- Value: "us-east-2",
- Help: "New Jersey, USA",
- }, {
- Value: "us-west-1",
- Help: "California, USA",
- }, {
- Value: "cloudgouv-eu-west-1",
- Help: "SecNumCloud, Paris, France",
- }, {
- Value: "ap-northeast-1",
- Help: "Tokyo, Japan",
- }},
- }, {
- // References:
- // https://help.ovhcloud.com/csm/en-public-cloud-storage-s3-location?id=kb_article_view&sysparm_article=KB0047384
- // https://support.us.ovhcloud.com/hc/en-us/articles/10667991081107-Endpoints-and-Object-Storage-Geoavailability
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "OVHcloud",
- Examples: []fs.OptionExample{{
- Value: "gra",
- Help: "Gravelines, France",
- }, {
- Value: "rbx",
- Help: "Roubaix, France",
- }, {
- Value: "sbg",
- Help: "Strasbourg, France",
- }, {
- Value: "eu-west-par",
- Help: "Paris, France (3AZ)",
- }, {
- Value: "de",
- Help: "Frankfurt, Germany",
- }, {
- Value: "uk",
- Help: "London, United Kingdom",
- }, {
- Value: "waw",
- Help: "Warsaw, Poland",
- }, {
- Value: "bhs",
- Help: "Beauharnois, Canada",
- }, {
- Value: "ca-east-tor",
- Help: "Toronto, Canada",
- }, {
- Value: "sgp",
- Help: "Singapore",
- }, {
- Value: "ap-southeast-syd",
- Help: "Sydney, Australia",
- }, {
- Value: "ap-south-mum",
- Help: "Mumbai, India",
- }, {
- Value: "us-east-va",
- Help: "Vint Hill, Virginia, USA",
- }, {
- Value: "us-west-or",
- Help: "Hillsboro, Oregon, USA",
- }, {
- Value: "rbx-archive",
- Help: "Roubaix, France (Cold Archive)",
- }},
- }, {
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "Petabox",
- Examples: []fs.OptionExample{{
- Value: "us-east-1",
- Help: "US East (N. Virginia)",
- }, {
- Value: "eu-central-1",
- Help: "Europe (Frankfurt)",
- }, {
- Value: "ap-southeast-1",
- Help: "Asia Pacific (Singapore)",
- }, {
- Value: "me-south-1",
- Help: "Middle East (Bahrain)",
- }, {
- Value: "sa-east-1",
- Help: "South America (São Paulo)",
- }},
- }, {
- // References:
- // https://developer.qiniu.com/kodo/4088/s3-access-domainname
- Name: "region",
- Help: "Region to connect to.",
- Provider: "Qiniu",
- Examples: []fs.OptionExample{{
- Value: "cn-east-1",
- Help: "The default endpoint - a good choice if you are unsure.\nEast China Region 1.\nNeeds location constraint cn-east-1.",
- }, {
- Value: "cn-east-2",
- Help: "East China Region 2.\nNeeds location constraint cn-east-2.",
- }, {
- Value: "cn-north-1",
- Help: "North China Region 1.\nNeeds location constraint cn-north-1.",
- }, {
- Value: "cn-south-1",
- Help: "South China Region 1.\nNeeds location constraint cn-south-1.",
- }, {
- Value: "us-north-1",
- Help: "North America Region.\nNeeds location constraint us-north-1.",
- }, {
- Value: "ap-southeast-1",
- Help: "Southeast Asia Region 1.\nNeeds location constraint ap-southeast-1.",
- }, {
- Value: "ap-northeast-1",
- Help: "Northeast Asia Region 1.\nNeeds location constraint ap-northeast-1.",
- }},
- }, {
- Name: "region",
- Help: "Region where your bucket will be created and your data stored.\n",
- Provider: "Rabata",
- Examples: []fs.OptionExample{{
- Value: "us-east-1",
- Help: "US East (N. Virginia)",
- }, {
- Value: "eu-west-1",
- Help: "EU (Ireland)",
- }, {
- Value: "eu-west-2",
- Help: "EU (London)",
- }},
- }, {
- Name: "region",
- Help: "region - the location where your bucket will be created and your data stored.\n",
- Provider: "RackCorp",
- Examples: []fs.OptionExample{{
- Value: "global",
- Help: "Global CDN (All locations) Region",
- }, {
- Value: "au",
- Help: "Australia (All states)",
- }, {
- Value: "au-nsw",
- Help: "NSW (Australia) Region",
- }, {
- Value: "au-qld",
- Help: "QLD (Australia) Region",
- }, {
- Value: "au-vic",
- Help: "VIC (Australia) Region",
- }, {
- Value: "au-wa",
- Help: "Perth (Australia) Region",
- }, {
- Value: "ph",
- Help: "Manila (Philippines) Region",
- }, {
- Value: "th",
- Help: "Bangkok (Thailand) Region",
- }, {
- Value: "hk",
- Help: "HK (Hong Kong) Region",
- }, {
- Value: "mn",
- Help: "Ulaanbaatar (Mongolia) Region",
- }, {
- Value: "kg",
- Help: "Bishkek (Kyrgyzstan) Region",
- }, {
- Value: "id",
- Help: "Jakarta (Indonesia) Region",
- }, {
- Value: "jp",
- Help: "Tokyo (Japan) Region",
- }, {
- Value: "sg",
- Help: "SG (Singapore) Region",
- }, {
- Value: "de",
- Help: "Frankfurt (Germany) Region",
- }, {
- Value: "us",
- Help: "USA (AnyCast) Region",
- }, {
- Value: "us-east-1",
- Help: "New York (USA) Region",
- }, {
- Value: "us-west-1",
- Help: "Freemont (USA) Region",
- }, {
- Value: "nz",
- Help: "Auckland (New Zealand) Region",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to.",
- Provider: "Scaleway",
- Examples: []fs.OptionExample{{
- Value: "nl-ams",
- Help: "Amsterdam, The Netherlands",
- }, {
- Value: "fr-par",
- Help: "Paris, France",
- }, {
- Value: "pl-waw",
- Help: "Warsaw, Poland",
- }},
- }, {
- // See endpoints for object storage regions: https://docs.selectel.ru/en/cloud/object-storage/manage/domains/#s3-api-domains
- Name: "region",
- Help: "Region where your data stored.\n",
- Provider: "Selectel",
- Examples: []fs.OptionExample{{
- Value: "ru-1",
- Help: "St. Petersburg",
- }},
- }, {
- Name: "region",
- Help: "Region where your is data stored.\n",
- Provider: "Servercore",
- Examples: []fs.OptionExample{{
- Value: "ru-1",
- Help: "St. Petersburg",
- }, {
- Value: "gis-1",
- Help: "Moscow",
- }, {
- Value: "ru-7",
- Help: "Moscow",
- }, {
- Value: "uz-2",
- Help: "Tashkent, Uzbekistan",
- }, {
- Value: "kz-1",
- Help: "Almaty, Kazakhstan",
- }},
- }, {
- Name: "region",
- Help: "Region where your data stored.\n",
- Provider: "Synology",
- Examples: []fs.OptionExample{{
- Value: "eu-001",
- Help: "Europe Region 1",
- }, {
- Value: "eu-002",
- Help: "Europe Region 2",
- }, {
- Value: "us-001",
- Help: "US Region 1",
- }, {
- Value: "us-002",
- Help: "US Region 2",
- }, {
- Value: "tw-001",
- Help: "Asia (Taiwan)",
- }},
- }, {
- Name: "region",
- Help: "Region where you can connect with.\n",
- Provider: "Zata",
- Examples: []fs.OptionExample{{
- Value: "us-east-1",
- Help: "Indore, Madhya Pradesh, India",
- }},
- }, {
- Name: "region",
- Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.",
- Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,Cubbit,FlashBlade,FileLu,Hetzner,HuaweiOBS,IDrive,Intercolo,IONOS,Liara,Linode,Magalu,Mega,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,SpectraLogic,Storj,Synology,TencentCOS,Zata",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Use this if unsure.\nWill use v4 signatures and an empty region.",
- }, {
- Value: "other-v2-signature",
- Help: "Use this only if v4 signatures don't work.\nE.g. pre Jewel/v10 CEPH.",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for S3 API.\n\nLeave blank if using AWS to use the default endpoint for the region.",
- Provider: "AWS",
- }, {
- // oss endpoints: https://help.aliyun.com/document_detail/31837.html
- Name: "endpoint",
- Help: "Endpoint for OSS API.",
- Provider: "Alibaba",
- Examples: []fs.OptionExample{{
- Value: "oss-accelerate.aliyuncs.com",
- Help: "Global Accelerate",
- }, {
- Value: "oss-accelerate-overseas.aliyuncs.com",
- Help: "Global Accelerate (outside mainland China)",
- }, {
- Value: "oss-cn-hangzhou.aliyuncs.com",
- Help: "East China 1 (Hangzhou)",
- }, {
- Value: "oss-cn-shanghai.aliyuncs.com",
- Help: "East China 2 (Shanghai)",
- }, {
- Value: "oss-cn-qingdao.aliyuncs.com",
- Help: "North China 1 (Qingdao)",
- }, {
- Value: "oss-cn-beijing.aliyuncs.com",
- Help: "North China 2 (Beijing)",
- }, {
- Value: "oss-cn-zhangjiakou.aliyuncs.com",
- Help: "North China 3 (Zhangjiakou)",
- }, {
- Value: "oss-cn-huhehaote.aliyuncs.com",
- Help: "North China 5 (Hohhot)",
- }, {
- Value: "oss-cn-wulanchabu.aliyuncs.com",
- Help: "North China 6 (Ulanqab)",
- }, {
- Value: "oss-cn-shenzhen.aliyuncs.com",
- Help: "South China 1 (Shenzhen)",
- }, {
- Value: "oss-cn-heyuan.aliyuncs.com",
- Help: "South China 2 (Heyuan)",
- }, {
- Value: "oss-cn-guangzhou.aliyuncs.com",
- Help: "South China 3 (Guangzhou)",
- }, {
- Value: "oss-cn-chengdu.aliyuncs.com",
- Help: "West China 1 (Chengdu)",
- }, {
- Value: "oss-cn-hongkong.aliyuncs.com",
- Help: "Hong Kong (Hong Kong)",
- }, {
- Value: "oss-us-west-1.aliyuncs.com",
- Help: "US West 1 (Silicon Valley)",
- }, {
- Value: "oss-us-east-1.aliyuncs.com",
- Help: "US East 1 (Virginia)",
- }, {
- Value: "oss-ap-southeast-1.aliyuncs.com",
- Help: "Southeast Asia Southeast 1 (Singapore)",
- }, {
- Value: "oss-ap-southeast-2.aliyuncs.com",
- Help: "Asia Pacific Southeast 2 (Sydney)",
- }, {
- Value: "oss-ap-southeast-3.aliyuncs.com",
- Help: "Southeast Asia Southeast 3 (Kuala Lumpur)",
- }, {
- Value: "oss-ap-southeast-5.aliyuncs.com",
- Help: "Asia Pacific Southeast 5 (Jakarta)",
- }, {
- Value: "oss-ap-northeast-1.aliyuncs.com",
- Help: "Asia Pacific Northeast 1 (Japan)",
- }, {
- Value: "oss-ap-south-1.aliyuncs.com",
- Help: "Asia Pacific South 1 (Mumbai)",
- }, {
- Value: "oss-eu-central-1.aliyuncs.com",
- Help: "Central Europe 1 (Frankfurt)",
- }, {
- Value: "oss-eu-west-1.aliyuncs.com",
- Help: "West Europe (London)",
- }, {
- Value: "oss-me-east-1.aliyuncs.com",
- Help: "Middle East 1 (Dubai)",
- }},
- }, {
- // ArvanCloud endpoints: https://www.arvancloud.ir/en/products/cloud-storage
- Name: "endpoint",
- Help: "Endpoint for Arvan Cloud Object Storage (AOS) API.",
- Provider: "ArvanCloud",
- Examples: []fs.OptionExample{{
- Value: "s3.ir-thr-at1.arvanstorage.ir",
- Help: "The default endpoint - a good choice if you are unsure.\nTehran Iran (Simin)",
- }, {
- Value: "s3.ir-tbz-sh1.arvanstorage.ir",
- Help: "Tabriz Iran (Shahriar)",
- }},
- }, {
- // ChinaMobile endpoints: https://ecloud.10086.cn/op-help-center/doc/article/24534
- Name: "endpoint",
- Help: "Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.",
- Provider: "ChinaMobile",
- Examples: []fs.OptionExample{{
- Value: "eos-wuxi-1.cmecloud.cn",
- Help: "The default endpoint - a good choice if you are unsure.\nEast China (Suzhou)",
- }, {
- Value: "eos-jinan-1.cmecloud.cn",
- Help: "East China (Jinan)",
- }, {
- Value: "eos-ningbo-1.cmecloud.cn",
- Help: "East China (Hangzhou)",
- }, {
- Value: "eos-shanghai-1.cmecloud.cn",
- Help: "East China (Shanghai-1)",
- }, {
- Value: "eos-zhengzhou-1.cmecloud.cn",
- Help: "Central China (Zhengzhou)",
- }, {
- Value: "eos-hunan-1.cmecloud.cn",
- Help: "Central China (Changsha-1)",
- }, {
- Value: "eos-zhuzhou-1.cmecloud.cn",
- Help: "Central China (Changsha-2)",
- }, {
- Value: "eos-guangzhou-1.cmecloud.cn",
- Help: "South China (Guangzhou-2)",
- }, {
- Value: "eos-dongguan-1.cmecloud.cn",
- Help: "South China (Guangzhou-3)",
- }, {
- Value: "eos-beijing-1.cmecloud.cn",
- Help: "North China (Beijing-1)",
- }, {
- Value: "eos-beijing-2.cmecloud.cn",
- Help: "North China (Beijing-2)",
- }, {
- Value: "eos-beijing-4.cmecloud.cn",
- Help: "North China (Beijing-3)",
- }, {
- Value: "eos-huhehaote-1.cmecloud.cn",
- Help: "North China (Huhehaote)",
- }, {
- Value: "eos-chengdu-1.cmecloud.cn",
- Help: "Southwest China (Chengdu)",
- }, {
- Value: "eos-chongqing-1.cmecloud.cn",
- Help: "Southwest China (Chongqing)",
- }, {
- Value: "eos-guiyang-1.cmecloud.cn",
- Help: "Southwest China (Guiyang)",
- }, {
- Value: "eos-xian-1.cmecloud.cn",
- Help: "Nouthwest China (Xian)",
- }, {
- Value: "eos-yunnan.cmecloud.cn",
- Help: "Yunnan China (Kunming)",
- }, {
- Value: "eos-yunnan-2.cmecloud.cn",
- Help: "Yunnan China (Kunming-2)",
- }, {
- Value: "eos-tianjin-1.cmecloud.cn",
- Help: "Tianjin China (Tianjin)",
- }, {
- Value: "eos-jilin-1.cmecloud.cn",
- Help: "Jilin China (Changchun)",
- }, {
- Value: "eos-hubei-1.cmecloud.cn",
- Help: "Hubei China (Xiangyan)",
- }, {
- Value: "eos-jiangxi-1.cmecloud.cn",
- Help: "Jiangxi China (Nanchang)",
- }, {
- Value: "eos-gansu-1.cmecloud.cn",
- Help: "Gansu China (Lanzhou)",
- }, {
- Value: "eos-shanxi-1.cmecloud.cn",
- Help: "Shanxi China (Taiyuan)",
- }, {
- Value: "eos-liaoning-1.cmecloud.cn",
- Help: "Liaoning China (Shenyang)",
- }, {
- Value: "eos-hebei-1.cmecloud.cn",
- Help: "Hebei China (Shijiazhuang)",
- }, {
- Value: "eos-fujian-1.cmecloud.cn",
- Help: "Fujian China (Xiamen)",
- }, {
- Value: "eos-guangxi-1.cmecloud.cn",
- Help: "Guangxi China (Nanning)",
- }, {
- Value: "eos-anhui-1.cmecloud.cn",
- Help: "Anhui China (Huainan)",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Cubbit DS3 Object Storage.",
- Provider: "Cubbit",
- Examples: []fs.OptionExample{{
- Value: "s3.cubbit.eu",
- Help: "Cubbit DS3 Object Storage endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for FileLu S5 Object Storage.\nRequired when using FileLu S5.",
- Provider: "FileLu",
- Examples: []fs.OptionExample{{
- Value: "s5lu.com",
- Help: "Global FileLu S5 endpoint",
- }, {
- Value: "us.s5lu.com",
- Help: "North America (US-East) region endpoint",
- }, {
- Value: "eu.s5lu.com",
- Help: "Europe (EU-Central) region endpoint",
- }, {
- Value: "ap.s5lu.com",
- Help: "Asia Pacific (AP-Southeast) region endpoint",
- }, {
- Value: "me.s5lu.com",
- Help: "Middle East (ME-Central) region endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Google Cloud Storage.",
- Provider: "GCS",
- Examples: []fs.OptionExample{{
- Value: "https://storage.googleapis.com",
- Help: "Google Cloud Storage endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Hetzner Object Storage",
- Provider: "Hetzner",
- Examples: []fs.OptionExample{{
- Value: "hel1.your-objectstorage.com",
- Help: "Helsinki",
- }, {
- Value: "fsn1.your-objectstorage.com",
- Help: "Falkenstein",
- }, {
- Value: "nbg1.your-objectstorage.com",
- Help: "Nuremberg",
- }},
- }, {
- // obs endpoints: https://developer.huaweicloud.com/intl/en-us/endpoint?OBS
- Name: "endpoint",
- Help: "Endpoint for OBS API.",
- Provider: "HuaweiOBS",
- Examples: []fs.OptionExample{{
- Value: "obs.af-south-1.myhuaweicloud.com",
- Help: "AF-Johannesburg",
- }, {
- Value: "obs.ap-southeast-2.myhuaweicloud.com",
- Help: "AP-Bangkok",
- }, {
- Value: "obs.ap-southeast-3.myhuaweicloud.com",
- Help: "AP-Singapore",
- }, {
- Value: "obs.cn-east-3.myhuaweicloud.com",
- Help: "CN East-Shanghai1",
- }, {
- Value: "obs.cn-east-2.myhuaweicloud.com",
- Help: "CN East-Shanghai2",
- }, {
- Value: "obs.cn-north-1.myhuaweicloud.com",
- Help: "CN North-Beijing1",
- }, {
- Value: "obs.cn-north-4.myhuaweicloud.com",
- Help: "CN North-Beijing4",
- }, {
- Value: "obs.cn-south-1.myhuaweicloud.com",
- Help: "CN South-Guangzhou",
- }, {
- Value: "obs.ap-southeast-1.myhuaweicloud.com",
- Help: "CN-Hong Kong",
- }, {
- Value: "obs.sa-argentina-1.myhuaweicloud.com",
- Help: "LA-Buenos Aires1",
- }, {
- Value: "obs.sa-peru-1.myhuaweicloud.com",
- Help: "LA-Lima1",
- }, {
- Value: "obs.na-mexico-1.myhuaweicloud.com",
- Help: "LA-Mexico City1",
- }, {
- Value: "obs.sa-chile-1.myhuaweicloud.com",
- Help: "LA-Santiago2",
- }, {
- Value: "obs.sa-brazil-1.myhuaweicloud.com",
- Help: "LA-Sao Paulo1",
- }, {
- Value: "obs.ru-northwest-2.myhuaweicloud.com",
- Help: "RU-Moscow2",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for IBM COS S3 API.\n\nSpecify if using an IBM COS On Premise.",
- Provider: "IBMCOS",
- Examples: []fs.OptionExample{{
- Value: "s3.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Endpoint",
- }, {
- Value: "s3.dal.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Dallas Endpoint",
- }, {
- Value: "s3.wdc.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Washington DC Endpoint",
- }, {
- Value: "s3.sjc.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region San Jose Endpoint",
- }, {
- Value: "s3.private.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Private Endpoint",
- }, {
- Value: "s3.private.dal.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Dallas Private Endpoint",
- }, {
- Value: "s3.private.wdc.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region Washington DC Private Endpoint",
- }, {
- Value: "s3.private.sjc.us.cloud-object-storage.appdomain.cloud",
- Help: "US Cross Region San Jose Private Endpoint",
- }, {
- Value: "s3.us-east.cloud-object-storage.appdomain.cloud",
- Help: "US Region East Endpoint",
- }, {
- Value: "s3.private.us-east.cloud-object-storage.appdomain.cloud",
- Help: "US Region East Private Endpoint",
- }, {
- Value: "s3.us-south.cloud-object-storage.appdomain.cloud",
- Help: "US Region South Endpoint",
- }, {
- Value: "s3.private.us-south.cloud-object-storage.appdomain.cloud",
- Help: "US Region South Private Endpoint",
- }, {
- Value: "s3.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Endpoint",
- }, {
- Value: "s3.fra.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Frankfurt Endpoint",
- }, {
- Value: "s3.mil.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Milan Endpoint",
- }, {
- Value: "s3.ams.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Amsterdam Endpoint",
- }, {
- Value: "s3.private.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Private Endpoint",
- }, {
- Value: "s3.private.fra.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Frankfurt Private Endpoint",
- }, {
- Value: "s3.private.mil.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Milan Private Endpoint",
- }, {
- Value: "s3.private.ams.eu.cloud-object-storage.appdomain.cloud",
- Help: "EU Cross Region Amsterdam Private Endpoint",
- }, {
- Value: "s3.eu-gb.cloud-object-storage.appdomain.cloud",
- Help: "Great Britain Endpoint",
- }, {
- Value: "s3.private.eu-gb.cloud-object-storage.appdomain.cloud",
- Help: "Great Britain Private Endpoint",
- }, {
- Value: "s3.eu-de.cloud-object-storage.appdomain.cloud",
- Help: "EU Region DE Endpoint",
- }, {
- Value: "s3.private.eu-de.cloud-object-storage.appdomain.cloud",
- Help: "EU Region DE Private Endpoint",
- }, {
- Value: "s3.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Endpoint",
- }, {
- Value: "s3.tok.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Tokyo Endpoint",
- }, {
- Value: "s3.hkg.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Hong Kong Endpoint",
- }, {
- Value: "s3.seo.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Seoul Endpoint",
- }, {
- Value: "s3.private.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Private Endpoint",
- }, {
- Value: "s3.private.tok.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Tokyo Private Endpoint",
- }, {
- Value: "s3.private.hkg.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Hong Kong Private Endpoint",
- }, {
- Value: "s3.private.seo.ap.cloud-object-storage.appdomain.cloud",
- Help: "APAC Cross Regional Seoul Private Endpoint",
- }, {
- Value: "s3.jp-tok.cloud-object-storage.appdomain.cloud",
- Help: "APAC Region Japan Endpoint",
- }, {
- Value: "s3.private.jp-tok.cloud-object-storage.appdomain.cloud",
- Help: "APAC Region Japan Private Endpoint",
- }, {
- Value: "s3.au-syd.cloud-object-storage.appdomain.cloud",
- Help: "APAC Region Australia Endpoint",
- }, {
- Value: "s3.private.au-syd.cloud-object-storage.appdomain.cloud",
- Help: "APAC Region Australia Private Endpoint",
- }, {
- Value: "s3.ams03.cloud-object-storage.appdomain.cloud",
- Help: "Amsterdam Single Site Endpoint",
- }, {
- Value: "s3.private.ams03.cloud-object-storage.appdomain.cloud",
- Help: "Amsterdam Single Site Private Endpoint",
- }, {
- Value: "s3.che01.cloud-object-storage.appdomain.cloud",
- Help: "Chennai Single Site Endpoint",
- }, {
- Value: "s3.private.che01.cloud-object-storage.appdomain.cloud",
- Help: "Chennai Single Site Private Endpoint",
- }, {
- Value: "s3.mel01.cloud-object-storage.appdomain.cloud",
- Help: "Melbourne Single Site Endpoint",
- }, {
- Value: "s3.private.mel01.cloud-object-storage.appdomain.cloud",
- Help: "Melbourne Single Site Private Endpoint",
- }, {
- Value: "s3.osl01.cloud-object-storage.appdomain.cloud",
- Help: "Oslo Single Site Endpoint",
- }, {
- Value: "s3.private.osl01.cloud-object-storage.appdomain.cloud",
- Help: "Oslo Single Site Private Endpoint",
- }, {
- Value: "s3.tor01.cloud-object-storage.appdomain.cloud",
- Help: "Toronto Single Site Endpoint",
- }, {
- Value: "s3.private.tor01.cloud-object-storage.appdomain.cloud",
- Help: "Toronto Single Site Private Endpoint",
- }, {
- Value: "s3.seo01.cloud-object-storage.appdomain.cloud",
- Help: "Seoul Single Site Endpoint",
- }, {
- Value: "s3.private.seo01.cloud-object-storage.appdomain.cloud",
- Help: "Seoul Single Site Private Endpoint",
- }, {
- Value: "s3.mon01.cloud-object-storage.appdomain.cloud",
- Help: "Montreal Single Site Endpoint",
- }, {
- Value: "s3.private.mon01.cloud-object-storage.appdomain.cloud",
- Help: "Montreal Single Site Private Endpoint",
- }, {
- Value: "s3.mex01.cloud-object-storage.appdomain.cloud",
- Help: "Mexico Single Site Endpoint",
- }, {
- Value: "s3.private.mex01.cloud-object-storage.appdomain.cloud",
- Help: "Mexico Single Site Private Endpoint",
- }, {
- Value: "s3.sjc04.cloud-object-storage.appdomain.cloud",
- Help: "San Jose Single Site Endpoint",
- }, {
- Value: "s3.private.sjc04.cloud-object-storage.appdomain.cloud",
- Help: "San Jose Single Site Private Endpoint",
- }, {
- Value: "s3.mil01.cloud-object-storage.appdomain.cloud",
- Help: "Milan Single Site Endpoint",
- }, {
- Value: "s3.private.mil01.cloud-object-storage.appdomain.cloud",
- Help: "Milan Single Site Private Endpoint",
- }, {
- Value: "s3.hkg02.cloud-object-storage.appdomain.cloud",
- Help: "Hong Kong Single Site Endpoint",
- }, {
- Value: "s3.private.hkg02.cloud-object-storage.appdomain.cloud",
- Help: "Hong Kong Single Site Private Endpoint",
- }, {
- Value: "s3.par01.cloud-object-storage.appdomain.cloud",
- Help: "Paris Single Site Endpoint",
- }, {
- Value: "s3.private.par01.cloud-object-storage.appdomain.cloud",
- Help: "Paris Single Site Private Endpoint",
- }, {
- Value: "s3.sng01.cloud-object-storage.appdomain.cloud",
- Help: "Singapore Single Site Endpoint",
- }, {
- Value: "s3.private.sng01.cloud-object-storage.appdomain.cloud",
- Help: "Singapore Single Site Private Endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Intercolo Object Storage.",
- Provider: "Intercolo",
- Examples: []fs.OptionExample{{
- Value: "de-fra.i3storage.com",
- Help: "Frankfurt, Germany",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for IONOS S3 Object Storage.\n\nSpecify the endpoint from the same region.",
- Provider: "IONOS",
- Examples: []fs.OptionExample{{
- Value: "s3-eu-central-1.ionoscloud.com",
- Help: "Frankfurt, Germany",
- }, {
- Value: "s3-eu-central-2.ionoscloud.com",
- Help: "Berlin, Germany",
- }, {
- Value: "s3-eu-south-2.ionoscloud.com",
- Help: "Logrono, Spain",
- }},
- }, {
- // Leviia endpoints: https://www.leviia.com/object-storage/
- Name: "endpoint",
- Help: "Endpoint for Leviia Object Storage API.",
- Provider: "Leviia",
- Examples: []fs.OptionExample{{
- Value: "s3.leviia.com",
- Help: "The default endpoint\nLeviia",
- }},
- }, {
- // Liara endpoints: https://liara.ir/landing/object-storage
- Name: "endpoint",
- Help: "Endpoint for Liara Object Storage API.",
- Provider: "Liara",
- Examples: []fs.OptionExample{{
- Value: "storage.iran.liara.space",
- Help: "The default endpoint\nIran",
- }},
- }, {
- // Linode endpoints: https://techdocs.akamai.com/cloud-computing/docs/object-storage-product-limits#supported-endpoint-types-by-region
- Name: "endpoint",
- Help: "Endpoint for Linode Object Storage API.",
- Provider: "Linode",
- Examples: []fs.OptionExample{{
- Value: "nl-ams-1.linodeobjects.com",
- Help: "Amsterdam (Netherlands), nl-ams-1",
- }, {
- Value: "us-southeast-1.linodeobjects.com",
- Help: "Atlanta, GA (USA), us-southeast-1",
- }, {
- Value: "in-maa-1.linodeobjects.com",
- Help: "Chennai (India), in-maa-1",
- }, {
- Value: "us-ord-1.linodeobjects.com",
- Help: "Chicago, IL (USA), us-ord-1",
- }, {
- Value: "eu-central-1.linodeobjects.com",
- Help: "Frankfurt (Germany), eu-central-1",
- }, {
- Value: "id-cgk-1.linodeobjects.com",
- Help: "Jakarta (Indonesia), id-cgk-1",
- }, {
- Value: "gb-lon-1.linodeobjects.com",
- Help: "London 2 (Great Britain), gb-lon-1",
- }, {
- Value: "us-lax-1.linodeobjects.com",
- Help: "Los Angeles, CA (USA), us-lax-1",
- }, {
- Value: "es-mad-1.linodeobjects.com",
- Help: "Madrid (Spain), es-mad-1",
- }, {
- Value: "au-mel-1.linodeobjects.com",
- Help: "Melbourne (Australia), au-mel-1",
- }, {
- Value: "us-mia-1.linodeobjects.com",
- Help: "Miami, FL (USA), us-mia-1",
- }, {
- Value: "it-mil-1.linodeobjects.com",
- Help: "Milan (Italy), it-mil-1",
- }, {
- Value: "us-east-1.linodeobjects.com",
- Help: "Newark, NJ (USA), us-east-1",
- }, {
- Value: "jp-osa-1.linodeobjects.com",
- Help: "Osaka (Japan), jp-osa-1",
- }, {
- Value: "fr-par-1.linodeobjects.com",
- Help: "Paris (France), fr-par-1",
- }, {
- Value: "br-gru-1.linodeobjects.com",
- Help: "São Paulo (Brazil), br-gru-1",
- }, {
- Value: "us-sea-1.linodeobjects.com",
- Help: "Seattle, WA (USA), us-sea-1",
- }, {
- Value: "ap-south-1.linodeobjects.com",
- Help: "Singapore, ap-south-1",
- }, {
- Value: "sg-sin-1.linodeobjects.com",
- Help: "Singapore 2, sg-sin-1",
- }, {
- Value: "se-sto-1.linodeobjects.com",
- Help: "Stockholm (Sweden), se-sto-1",
- }, {
- Value: "us-iad-1.linodeobjects.com",
- Help: "Washington, DC, (USA), us-iad-1",
- }},
- }, {
- // Lyve Cloud endpoints
- Name: "endpoint",
- Help: "Endpoint for Lyve Cloud S3 API.\nRequired when using an S3 clone. Please type in your LyveCloud endpoint.\nExamples:\n- s3.us-west-1.{account_name}.lyve.seagate.com (US West 1 - California)\n- s3.eu-west-1.{account_name}.lyve.seagate.com (EU West 1 - Ireland)",
- Provider: "LyveCloud",
- Required: true,
- }, {
- // Magalu endpoints: https://docs.magalu.cloud/docs/object-storage/how-to/copy-url
- Name: "endpoint",
- Help: "Endpoint for Magalu Object Storage API.",
- Provider: "Magalu",
- Examples: []fs.OptionExample{{
- Value: "br-se1.magaluobjects.com",
- Help: "São Paulo, SP (BR), br-se1",
- }, {
- Value: "br-ne1.magaluobjects.com",
- Help: "Fortaleza, CE (BR), br-ne1",
- },
- },
- }, {
- Name: "endpoint",
- Help: "Endpoint for OVHcloud Object Storage.",
- Provider: "OVHcloud",
- Examples: []fs.OptionExample{{
- Value: "s3.gra.io.cloud.ovh.net",
- Help: "OVHcloud Gravelines, France",
- Provider: "OVHcloud",
- }, {
- Value: "s3.rbx.io.cloud.ovh.net",
- Help: "OVHcloud Roubaix, France",
- Provider: "OVHcloud",
- }, {
- Value: "s3.sbg.io.cloud.ovh.net",
- Help: "OVHcloud Strasbourg, France",
- Provider: "OVHcloud",
- }, {
- Value: "s3.eu-west-par.io.cloud.ovh.net",
- Help: "OVHcloud Paris, France (3AZ)",
- Provider: "OVHcloud",
- }, {
- Value: "s3.de.io.cloud.ovh.net",
- Help: "OVHcloud Frankfurt, Germany",
- Provider: "OVHcloud",
- }, {
- Value: "s3.uk.io.cloud.ovh.net",
- Help: "OVHcloud London, United Kingdom",
- Provider: "OVHcloud",
- }, {
- Value: "s3.waw.io.cloud.ovh.net",
- Help: "OVHcloud Warsaw, Poland",
- Provider: "OVHcloud",
- }, {
- Value: "s3.bhs.io.cloud.ovh.net",
- Help: "OVHcloud Beauharnois, Canada",
- Provider: "OVHcloud",
- }, {
- Value: "s3.ca-east-tor.io.cloud.ovh.net",
- Help: "OVHcloud Toronto, Canada",
- Provider: "OVHcloud",
- }, {
- Value: "s3.sgp.io.cloud.ovh.net",
- Help: "OVHcloud Singapore",
- Provider: "OVHcloud",
- }, {
- Value: "s3.ap-southeast-syd.io.cloud.ovh.net",
- Help: "OVHcloud Sydney, Australia",
- Provider: "OVHcloud",
- }, {
- Value: "s3.ap-south-mum.io.cloud.ovh.net",
- Help: "OVHcloud Mumbai, India",
- Provider: "OVHcloud",
- }, {
- Value: "s3.us-east-va.io.cloud.ovh.us",
- Help: "OVHcloud Vint Hill, Virginia, USA",
- Provider: "OVHcloud",
- }, {
- Value: "s3.us-west-or.io.cloud.ovh.us",
- Help: "OVHcloud Hillsboro, Oregon, USA",
- Provider: "OVHcloud",
- }, {
- Value: "s3.rbx-archive.io.cloud.ovh.net",
- Help: "OVHcloud Roubaix, France (Cold Archive)",
- Provider: "OVHcloud",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Petabox S3 Object Storage.\n\nSpecify the endpoint from the same region.",
- Provider: "Petabox",
- Required: true,
- Examples: []fs.OptionExample{{
- Value: "s3.petabox.io",
- Help: "US East (N. Virginia)",
- }, {
- Value: "s3.us-east-1.petabox.io",
- Help: "US East (N. Virginia)",
- }, {
- Value: "s3.eu-central-1.petabox.io",
- Help: "Europe (Frankfurt)",
- }, {
- Value: "s3.ap-southeast-1.petabox.io",
- Help: "Asia Pacific (Singapore)",
- }, {
- Value: "s3.me-south-1.petabox.io",
- Help: "Middle East (Bahrain)",
- }, {
- Value: "s3.sa-east-1.petabox.io",
- Help: "South America (São Paulo)",
- }},
- }, {
- // Qiniu endpoints: https://developer.qiniu.com/kodo/4088/s3-access-domainname
- Name: "endpoint",
- Help: "Endpoint for Qiniu Object Storage.",
- Provider: "Qiniu",
- Examples: []fs.OptionExample{{
- Value: "s3-cn-east-1.qiniucs.com",
- Help: "East China Endpoint 1",
- }, {
- Value: "s3-cn-east-2.qiniucs.com",
- Help: "East China Endpoint 2",
- }, {
- Value: "s3-cn-north-1.qiniucs.com",
- Help: "North China Endpoint 1",
- }, {
- Value: "s3-cn-south-1.qiniucs.com",
- Help: "South China Endpoint 1",
- }, {
- Value: "s3-us-north-1.qiniucs.com",
- Help: "North America Endpoint 1",
- }, {
- Value: "s3-ap-southeast-1.qiniucs.com",
- Help: "Southeast Asia Endpoint 1",
- }, {
- Value: "s3-ap-northeast-1.qiniucs.com",
- Help: "Northeast Asia Endpoint 1",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Rabata Object Storage.",
- Provider: "Rabata",
- Examples: []fs.OptionExample{{
- Value: "s3.us-east-1.rabata.io",
- Help: "US East (N. Virginia)",
- }, {
- Value: "s3.eu-west-1.rabata.io",
- Help: "EU West (Ireland)",
- }, {
- Value: "s3.eu-west-2.rabata.io",
- Help: "EU West (London)",
- }},
- }, {
- // RackCorp endpoints: https://www.rackcorp.com/storage/s3storage
- Name: "endpoint",
- Help: "Endpoint for RackCorp Object Storage.",
- Provider: "RackCorp",
- Examples: []fs.OptionExample{{
- Value: "s3.rackcorp.com",
- Help: "Global (AnyCast) Endpoint",
- }, {
- Value: "au.s3.rackcorp.com",
- Help: "Australia (Anycast) Endpoint",
- }, {
- Value: "au-nsw.s3.rackcorp.com",
- Help: "Sydney (Australia) Endpoint",
- }, {
- Value: "au-qld.s3.rackcorp.com",
- Help: "Brisbane (Australia) Endpoint",
- }, {
- Value: "au-vic.s3.rackcorp.com",
- Help: "Melbourne (Australia) Endpoint",
- }, {
- Value: "au-wa.s3.rackcorp.com",
- Help: "Perth (Australia) Endpoint",
- }, {
- Value: "ph.s3.rackcorp.com",
- Help: "Manila (Philippines) Endpoint",
- }, {
- Value: "th.s3.rackcorp.com",
- Help: "Bangkok (Thailand) Endpoint",
- }, {
- Value: "hk.s3.rackcorp.com",
- Help: "HK (Hong Kong) Endpoint",
- }, {
- Value: "mn.s3.rackcorp.com",
- Help: "Ulaanbaatar (Mongolia) Endpoint",
- }, {
- Value: "kg.s3.rackcorp.com",
- Help: "Bishkek (Kyrgyzstan) Endpoint",
- }, {
- Value: "id.s3.rackcorp.com",
- Help: "Jakarta (Indonesia) Endpoint",
- }, {
- Value: "jp.s3.rackcorp.com",
- Help: "Tokyo (Japan) Endpoint",
- }, {
- Value: "sg.s3.rackcorp.com",
- Help: "SG (Singapore) Endpoint",
- }, {
- Value: "de.s3.rackcorp.com",
- Help: "Frankfurt (Germany) Endpoint",
- }, {
- Value: "us.s3.rackcorp.com",
- Help: "USA (AnyCast) Endpoint",
- }, {
- Value: "us-east-1.s3.rackcorp.com",
- Help: "New York (USA) Endpoint",
- }, {
- Value: "us-west-1.s3.rackcorp.com",
- Help: "Freemont (USA) Endpoint",
- }, {
- Value: "nz.s3.rackcorp.com",
- Help: "Auckland (New Zealand) Endpoint",
- }},
- }, {
- // Selectel endpoints: https://docs.selectel.ru/en/cloud/object-storage/manage/domains/#s3-api-domains
- Name: "endpoint",
- Help: "Endpoint for Selectel Object Storage.",
- Provider: "Selectel",
- Examples: []fs.OptionExample{{
- Value: "s3.ru-1.storage.selcloud.ru",
- Help: "Saint Petersburg",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Servercore Object Storage.",
- Provider: "Servercore",
- Examples: []fs.OptionExample{{
- Value: "s3.ru-1.storage.selcloud.ru",
- Help: "Saint Petersburg",
- }, {
- Value: "s3.gis-1.storage.selcloud.ru",
- Help: "Moscow",
- }, {
- Value: "s3.ru-7.storage.selcloud.ru",
- Help: "Moscow",
- }, {
- Value: "s3.uz-2.srvstorage.uz",
- Help: "Tashkent, Uzbekistan",
- }, {
- Value: "s3.kz-1.srvstorage.kz",
- Help: "Almaty, Kazakhstan",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Scaleway Object Storage.",
- Provider: "Scaleway",
- Examples: []fs.OptionExample{{
- Value: "s3.nl-ams.scw.cloud",
- Help: "Amsterdam Endpoint",
- }, {
- Value: "s3.fr-par.scw.cloud",
- Help: "Paris Endpoint",
- }, {
- Value: "s3.pl-waw.scw.cloud",
- Help: "Warsaw Endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for StackPath Object Storage.",
- Provider: "StackPath",
- Examples: []fs.OptionExample{{
- Value: "s3.us-east-2.stackpathstorage.com",
- Help: "US East Endpoint",
- }, {
- Value: "s3.us-west-1.stackpathstorage.com",
- Help: "US West Endpoint",
- }, {
- Value: "s3.eu-central-1.stackpathstorage.com",
- Help: "EU Endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Storj Gateway.",
- Provider: "Storj",
- Examples: []fs.OptionExample{{
- Value: "gateway.storjshare.io",
- Help: "Global Hosted Gateway",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Synology C2 Object Storage API.",
- Provider: "Synology",
- Examples: []fs.OptionExample{{
- Value: "eu-001.s3.synologyc2.net",
- Help: "EU Endpoint 1",
- }, {
- Value: "eu-002.s3.synologyc2.net",
- Help: "EU Endpoint 2",
- }, {
- Value: "us-001.s3.synologyc2.net",
- Help: "US Endpoint 1",
- }, {
- Value: "us-002.s3.synologyc2.net",
- Help: "US Endpoint 2",
- }, {
- Value: "tw-001.s3.synologyc2.net",
- Help: "TW Endpoint 1",
- }},
- }, {
- // cos endpoints: https://intl.cloud.tencent.com/document/product/436/6224
- Name: "endpoint",
- Help: "Endpoint for Tencent COS API.",
- Provider: "TencentCOS",
- Examples: []fs.OptionExample{{
- Value: "cos.ap-beijing.myqcloud.com",
- Help: "Beijing Region",
- }, {
- Value: "cos.ap-nanjing.myqcloud.com",
- Help: "Nanjing Region",
- }, {
- Value: "cos.ap-shanghai.myqcloud.com",
- Help: "Shanghai Region",
- }, {
- Value: "cos.ap-guangzhou.myqcloud.com",
- Help: "Guangzhou Region",
- }, {
- Value: "cos.ap-nanjing.myqcloud.com",
- Help: "Nanjing Region",
- }, {
- Value: "cos.ap-chengdu.myqcloud.com",
- Help: "Chengdu Region",
- }, {
- Value: "cos.ap-chongqing.myqcloud.com",
- Help: "Chongqing Region",
- }, {
- Value: "cos.ap-hongkong.myqcloud.com",
- Help: "Hong Kong (China) Region",
- }, {
- Value: "cos.ap-singapore.myqcloud.com",
- Help: "Singapore Region",
- }, {
- Value: "cos.ap-mumbai.myqcloud.com",
- Help: "Mumbai Region",
- }, {
- Value: "cos.ap-seoul.myqcloud.com",
- Help: "Seoul Region",
- }, {
- Value: "cos.ap-bangkok.myqcloud.com",
- Help: "Bangkok Region",
- }, {
- Value: "cos.ap-tokyo.myqcloud.com",
- Help: "Tokyo Region",
- }, {
- Value: "cos.na-siliconvalley.myqcloud.com",
- Help: "Silicon Valley Region",
- }, {
- Value: "cos.na-ashburn.myqcloud.com",
- Help: "Virginia Region",
- }, {
- Value: "cos.na-toronto.myqcloud.com",
- Help: "Toronto Region",
- }, {
- Value: "cos.eu-frankfurt.myqcloud.com",
- Help: "Frankfurt Region",
- }, {
- Value: "cos.eu-moscow.myqcloud.com",
- Help: "Moscow Region",
- }, {
- Value: "cos.accelerate.myqcloud.com",
- Help: "Use Tencent COS Accelerate Endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for Zata Object Storage.",
- Provider: "Zata",
- Examples: []fs.OptionExample{{
- Value: "idr01.zata.ai",
- Help: "South Asia Endpoint",
- }},
- }, {
- Name: "endpoint",
- Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.",
- Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cubbit,GCS,Hetzner,HuaweiOBS,IBMCOS,IDrive,Intercolo,IONOS,Liara,Linode,LyveCloud,Magalu,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,StackPath,Storj,Synology,TencentCOS,Zata",
- Examples: []fs.OptionExample{{
- Value: "objects-us-east-1.dream.io",
- Help: "Dream Objects endpoint",
- Provider: "Dreamhost",
- }, {
- Value: "syd1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Sydney 1",
- Provider: "DigitalOcean",
- }, {
- Value: "sfo3.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces San Francisco 3",
- Provider: "DigitalOcean",
- }, {
- Value: "sfo2.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces San Francisco 2",
- Provider: "DigitalOcean",
- }, {
- Value: "fra1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Frankfurt 1",
- Provider: "DigitalOcean",
- }, {
- Value: "nyc3.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces New York 3",
- Provider: "DigitalOcean",
- }, {
- Value: "ams3.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Amsterdam 3",
- Provider: "DigitalOcean",
- }, {
- Value: "sgp1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Singapore 1",
- Provider: "DigitalOcean",
- }, {
- Value: "lon1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces London 1",
- Provider: "DigitalOcean",
- }, {
- Value: "tor1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Toronto 1",
- Provider: "DigitalOcean",
- }, {
- Value: "blr1.digitaloceanspaces.com",
- Help: "DigitalOcean Spaces Bangalore 1",
- Provider: "DigitalOcean",
- }, {
- Value: "localhost:8333",
- Help: "SeaweedFS S3 localhost",
- Provider: "SeaweedFS",
- }, {
- Value: "oos.eu-west-2.outscale.com",
- Help: "Outscale EU West 2 (Paris)",
- Provider: "Outscale",
- }, {
- Value: "oos.us-east-2.outscale.com",
- Help: "Outscale US east 2 (New Jersey)",
- Provider: "Outscale",
- }, {
- Value: "oos.us-west-1.outscale.com",
- Help: "Outscale EU West 1 (California)",
- Provider: "Outscale",
- }, {
- Value: "oos.cloudgouv-eu-west-1.outscale.com",
- Help: "Outscale SecNumCloud (Paris)",
- Provider: "Outscale",
- }, {
- Value: "oos.ap-northeast-1.outscale.com",
- Help: "Outscale AP Northeast 1 (Japan)",
- Provider: "Outscale",
- }, {
- Value: "s3.wasabisys.com",
- Help: "Wasabi US East 1 (N. Virginia)",
- Provider: "Wasabi",
- }, {
- Value: "s3.us-east-2.wasabisys.com",
- Help: "Wasabi US East 2 (N. Virginia)",
- Provider: "Wasabi",
- }, {
- Value: "s3.us-central-1.wasabisys.com",
- Help: "Wasabi US Central 1 (Texas)",
- Provider: "Wasabi",
- }, {
- Value: "s3.us-west-1.wasabisys.com",
- Help: "Wasabi US West 1 (Oregon)",
- Provider: "Wasabi",
- }, {
- Value: "s3.ca-central-1.wasabisys.com",
- Help: "Wasabi CA Central 1 (Toronto)",
- Provider: "Wasabi",
- }, {
- Value: "s3.eu-central-1.wasabisys.com",
- Help: "Wasabi EU Central 1 (Amsterdam)",
- Provider: "Wasabi",
- }, {
- Value: "s3.eu-central-2.wasabisys.com",
- Help: "Wasabi EU Central 2 (Frankfurt)",
- Provider: "Wasabi",
- }, {
- Value: "s3.eu-west-1.wasabisys.com",
- Help: "Wasabi EU West 1 (London)",
- Provider: "Wasabi",
- }, {
- Value: "s3.eu-west-2.wasabisys.com",
- Help: "Wasabi EU West 2 (Paris)",
- Provider: "Wasabi",
- }, {
- Value: "s3.eu-south-1.wasabisys.com",
- Help: "Wasabi EU South 1 (Milan)",
- Provider: "Wasabi",
- }, {
- Value: "s3.ap-northeast-1.wasabisys.com",
- Help: "Wasabi AP Northeast 1 (Tokyo) endpoint",
- Provider: "Wasabi",
- }, {
- Value: "s3.ap-northeast-2.wasabisys.com",
- Help: "Wasabi AP Northeast 2 (Osaka) endpoint",
- Provider: "Wasabi",
- }, {
- Value: "s3.ap-southeast-1.wasabisys.com",
- Help: "Wasabi AP Southeast 1 (Singapore)",
- Provider: "Wasabi",
- }, {
- Value: "s3.ap-southeast-2.wasabisys.com",
- Help: "Wasabi AP Southeast 2 (Sydney)",
- Provider: "Wasabi",
- }, {
- Value: "storage.iran.liara.space",
- Help: "Liara Iran endpoint",
- Provider: "Liara",
- }, {
- Value: "s3.ir-thr-at1.arvanstorage.ir",
- Help: "ArvanCloud Tehran Iran (Simin) endpoint",
- Provider: "ArvanCloud",
- }, {
- Value: "s3.ir-tbz-sh1.arvanstorage.ir",
- Help: "ArvanCloud Tabriz Iran (Shahriar) endpoint",
- Provider: "ArvanCloud",
- }, {
- Value: "s3.eu-central-1.s4.mega.io",
- Help: "Mega S4 eu-central-1 (Amsterdam)",
- Provider: "Mega",
- }, {
- Value: "s3.eu-central-2.s4.mega.io",
- Help: "Mega S4 eu-central-2 (Bettembourg)",
- Provider: "Mega",
- }, {
- Value: "s3.ca-central-1.s4.mega.io",
- Help: "Mega S4 ca-central-1 (Montreal)",
- Provider: "Mega",
- }, {
- Value: "s3.ca-west-1.s4.mega.io",
- Help: "Mega S4 ca-west-1 (Vancouver)",
- Provider: "Mega",
- }},
- }, {
- Name: "location_constraint",
- Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.",
- Provider: "AWS",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Empty for US Region, Northern Virginia, or Pacific Northwest",
- }, {
- Value: "us-east-2",
- Help: "US East (Ohio) Region",
- }, {
- Value: "us-west-1",
- Help: "US West (Northern California) Region",
- }, {
- Value: "us-west-2",
- Help: "US West (Oregon) Region",
- }, {
- Value: "ca-central-1",
- Help: "Canada (Central) Region",
- }, {
- Value: "eu-west-1",
- Help: "EU (Ireland) Region",
- }, {
- Value: "eu-west-2",
- Help: "EU (London) Region",
- }, {
- Value: "eu-west-3",
- Help: "EU (Paris) Region",
- }, {
- Value: "eu-north-1",
- Help: "EU (Stockholm) Region",
- }, {
- Value: "eu-south-1",
- Help: "EU (Milan) Region",
- }, {
- Value: "EU",
- Help: "EU Region",
- }, {
- Value: "ap-southeast-1",
- Help: "Asia Pacific (Singapore) Region",
- }, {
- Value: "ap-southeast-2",
- Help: "Asia Pacific (Sydney) Region",
- }, {
- Value: "ap-northeast-1",
- Help: "Asia Pacific (Tokyo) Region",
- }, {
- Value: "ap-northeast-2",
- Help: "Asia Pacific (Seoul) Region",
- }, {
- Value: "ap-northeast-3",
- Help: "Asia Pacific (Osaka-Local) Region",
- }, {
- Value: "ap-south-1",
- Help: "Asia Pacific (Mumbai) Region",
- }, {
- Value: "ap-east-1",
- Help: "Asia Pacific (Hong Kong) Region",
- }, {
- Value: "sa-east-1",
- Help: "South America (Sao Paulo) Region",
- }, {
- Value: "il-central-1",
- Help: "Israel (Tel Aviv) Region",
- }, {
- Value: "me-south-1",
- Help: "Middle East (Bahrain) Region",
- }, {
- Value: "af-south-1",
- Help: "Africa (Cape Town) Region",
- }, {
- Value: "cn-north-1",
- Help: "China (Beijing) Region",
- }, {
- Value: "cn-northwest-1",
- Help: "China (Ningxia) Region",
- }, {
- Value: "us-gov-east-1",
- Help: "AWS GovCloud (US-East) Region",
- }, {
- Value: "us-gov-west-1",
- Help: "AWS GovCloud (US) Region",
- }},
- }, {
- Name: "location_constraint",
- Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.",
- Provider: "ArvanCloud",
- Examples: []fs.OptionExample{{
- Value: "ir-thr-at1",
- Help: "Tehran Iran (Simin)",
- }, {
- Value: "ir-tbz-sh1",
- Help: "Tabriz Iran (Shahriar)",
- }},
+// Register with Fs
+func init() {
+ fs.Register(addProvidersToInfo(&fs.RegInfo{
+ Name: "s3",
+ Description: "Amazon S3 Compliant Storage Providers including ",
+ NewFs: NewFs,
+ CommandHelp: commandHelp,
+ Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) {
+ switch config.State {
+ case "":
+ return nil, setEndpointValueForIDriveE2(m)
+ }
+ return nil, fmt.Errorf("unknown state %q", config.State)
+ },
+ MetadataInfo: &fs.MetadataInfo{
+ System: systemMetadataInfo,
+ Help: `User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.`,
+ },
+ Options: []fs.Option{{
+ Name: fs.ConfigProvider,
+ Help: "Choose your S3 provider.",
}, {
- Name: "location_constraint",
- Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.",
- Provider: "ChinaMobile",
+ Name: "env_auth",
+ Help: "Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).\n\nOnly applies if access_key_id and secret_access_key is blank.",
+ Default: false,
Examples: []fs.OptionExample{{
- Value: "wuxi1",
- Help: "East China (Suzhou)",
- }, {
- Value: "jinan1",
- Help: "East China (Jinan)",
- }, {
- Value: "ningbo1",
- Help: "East China (Hangzhou)",
- }, {
- Value: "shanghai1",
- Help: "East China (Shanghai-1)",
- }, {
- Value: "zhengzhou1",
- Help: "Central China (Zhengzhou)",
- }, {
- Value: "hunan1",
- Help: "Central China (Changsha-1)",
- }, {
- Value: "zhuzhou1",
- Help: "Central China (Changsha-2)",
- }, {
- Value: "guangzhou1",
- Help: "South China (Guangzhou-2)",
- }, {
- Value: "dongguan1",
- Help: "South China (Guangzhou-3)",
- }, {
- Value: "beijing1",
- Help: "North China (Beijing-1)",
- }, {
- Value: "beijing2",
- Help: "North China (Beijing-2)",
- }, {
- Value: "beijing4",
- Help: "North China (Beijing-3)",
- }, {
- Value: "huhehaote1",
- Help: "North China (Huhehaote)",
- }, {
- Value: "chengdu1",
- Help: "Southwest China (Chengdu)",
- }, {
- Value: "chongqing1",
- Help: "Southwest China (Chongqing)",
- }, {
- Value: "guiyang1",
- Help: "Southwest China (Guiyang)",
- }, {
- Value: "xian1",
- Help: "Northwest China (Xian)",
- }, {
- Value: "yunnan",
- Help: "Yunnan China (Kunming)",
- }, {
- Value: "yunnan2",
- Help: "Yunnan China (Kunming-2)",
- }, {
- Value: "tianjin1",
- Help: "Tianjin China (Tianjin)",
- }, {
- Value: "jilin1",
- Help: "Jilin China (Changchun)",
- }, {
- Value: "hubei1",
- Help: "Hubei China (Xiangyan)",
- }, {
- Value: "jiangxi1",
- Help: "Jiangxi China (Nanchang)",
- }, {
- Value: "gansu1",
- Help: "Gansu China (Lanzhou)",
- }, {
- Value: "shanxi1",
- Help: "Shanxi China (Taiyuan)",
- }, {
- Value: "liaoning1",
- Help: "Liaoning China (Shenyang)",
- }, {
- Value: "hebei1",
- Help: "Hebei China (Shijiazhuang)",
- }, {
- Value: "fujian1",
- Help: "Fujian China (Xiamen)",
- }, {
- Value: "guangxi1",
- Help: "Guangxi China (Nanning)",
+ Value: "false",
+ Help: "Enter AWS credentials in the next step.",
}, {
- Value: "anhui1",
- Help: "Anhui China (Huainan)",
+ Value: "true",
+ Help: "Get AWS credentials from the environment (env vars or IAM).",
}},
}, {
- Name: "location_constraint",
- Help: "Location constraint - must match endpoint when using IBM Cloud Public.\n\nFor on-prem COS, do not make a selection from this list, hit enter.",
- Provider: "IBMCOS",
- Examples: []fs.OptionExample{{
- Value: "us-standard",
- Help: "US Cross Region Standard",
- }, {
- Value: "us-vault",
- Help: "US Cross Region Vault",
- }, {
- Value: "us-cold",
- Help: "US Cross Region Cold",
- }, {
- Value: "us-flex",
- Help: "US Cross Region Flex",
- }, {
- Value: "us-east-standard",
- Help: "US East Region Standard",
- }, {
- Value: "us-east-vault",
- Help: "US East Region Vault",
- }, {
- Value: "us-east-cold",
- Help: "US East Region Cold",
- }, {
- Value: "us-east-flex",
- Help: "US East Region Flex",
- }, {
- Value: "us-south-standard",
- Help: "US South Region Standard",
- }, {
- Value: "us-south-vault",
- Help: "US South Region Vault",
- }, {
- Value: "us-south-cold",
- Help: "US South Region Cold",
- }, {
- Value: "us-south-flex",
- Help: "US South Region Flex",
- }, {
- Value: "eu-standard",
- Help: "EU Cross Region Standard",
- }, {
- Value: "eu-vault",
- Help: "EU Cross Region Vault",
- }, {
- Value: "eu-cold",
- Help: "EU Cross Region Cold",
- }, {
- Value: "eu-flex",
- Help: "EU Cross Region Flex",
- }, {
- Value: "eu-gb-standard",
- Help: "Great Britain Standard",
- }, {
- Value: "eu-gb-vault",
- Help: "Great Britain Vault",
- }, {
- Value: "eu-gb-cold",
- Help: "Great Britain Cold",
- }, {
- Value: "eu-gb-flex",
- Help: "Great Britain Flex",
- }, {
- Value: "ap-standard",
- Help: "APAC Standard",
- }, {
- Value: "ap-vault",
- Help: "APAC Vault",
- }, {
- Value: "ap-cold",
- Help: "APAC Cold",
- }, {
- Value: "ap-flex",
- Help: "APAC Flex",
- }, {
- Value: "mel01-standard",
- Help: "Melbourne Standard",
- }, {
- Value: "mel01-vault",
- Help: "Melbourne Vault",
- }, {
- Value: "mel01-cold",
- Help: "Melbourne Cold",
- }, {
- Value: "mel01-flex",
- Help: "Melbourne Flex",
- }, {
- Value: "tor01-standard",
- Help: "Toronto Standard",
- }, {
- Value: "tor01-vault",
- Help: "Toronto Vault",
- }, {
- Value: "tor01-cold",
- Help: "Toronto Cold",
- }, {
- Value: "tor01-flex",
- Help: "Toronto Flex",
- }},
+ Name: "access_key_id",
+ Help: "AWS Access Key ID.\n\nLeave blank for anonymous access or runtime credentials.",
+ Sensitive: true,
}, {
- Name: "location_constraint",
- Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.",
- Provider: "Qiniu",
- Examples: []fs.OptionExample{{
- Value: "cn-east-1",
- Help: "East China Region 1",
- }, {
- Value: "cn-east-2",
- Help: "East China Region 2",
- }, {
- Value: "cn-north-1",
- Help: "North China Region 1",
- }, {
- Value: "cn-south-1",
- Help: "South China Region 1",
- }, {
- Value: "us-north-1",
- Help: "North America Region 1",
- }, {
- Value: "ap-southeast-1",
- Help: "Southeast Asia Region 1",
- }, {
- Value: "ap-northeast-1",
- Help: "Northeast Asia Region 1",
- }},
+ Name: "secret_access_key",
+ Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.",
+ Sensitive: true,
}, {
- Name: "location_constraint",
- Help: "location where your bucket will be created and your data stored.\n",
- Provider: "Rabata",
- Examples: []fs.OptionExample{{
- Value: "us-east-1",
- Help: "US East (N. Virginia)",
- }, {
- Value: "eu-west-1",
- Help: "EU (Ireland)",
- }, {
- Value: "eu-west-2",
- Help: "EU (London)",
- }},
+ Name: "region",
+ Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.",
}, {
- Name: "location_constraint",
- Help: "Location constraint - the location where your bucket will be located and your data stored.\n",
- Provider: "RackCorp",
- Examples: []fs.OptionExample{{
- Value: "global",
- Help: "Global CDN Region",
- }, {
- Value: "au",
- Help: "Australia (All locations)",
- }, {
- Value: "au-nsw",
- Help: "NSW (Australia) Region",
- }, {
- Value: "au-qld",
- Help: "QLD (Australia) Region",
- }, {
- Value: "au-vic",
- Help: "VIC (Australia) Region",
- }, {
- Value: "au-wa",
- Help: "Perth (Australia) Region",
- }, {
- Value: "ph",
- Help: "Manila (Philippines) Region",
- }, {
- Value: "th",
- Help: "Bangkok (Thailand) Region",
- }, {
- Value: "hk",
- Help: "HK (Hong Kong) Region",
- }, {
- Value: "mn",
- Help: "Ulaanbaatar (Mongolia) Region",
- }, {
- Value: "kg",
- Help: "Bishkek (Kyrgyzstan) Region",
- }, {
- Value: "id",
- Help: "Jakarta (Indonesia) Region",
- }, {
- Value: "jp",
- Help: "Tokyo (Japan) Region",
- }, {
- Value: "sg",
- Help: "SG (Singapore) Region",
- }, {
- Value: "de",
- Help: "Frankfurt (Germany) Region",
- }, {
- Value: "us",
- Help: "USA (AnyCast) Region",
- }, {
- Value: "us-east-1",
- Help: "New York (USA) Region",
- }, {
- Value: "us-west-1",
- Help: "Fremont (USA) Region",
- }, {
- Value: "nz",
- Help: "Auckland (New Zealand) Region",
- }},
+ Name: "endpoint",
+ Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.",
}, {
- Name: "location_constraint",
- Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.",
- Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,Cubbit,FlashBlade,FileLu,HuaweiOBS,IBMCOS,IDrive,Intercolo,IONOS,Leviia,Liara,Linode,Magalu,Mega,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,SpectraLogic,StackPath,Storj,TencentCOS",
+ Name: "location_constraint",
+ Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.",
}, {
Name: "acl",
Help: `Canned ACL used when creating buckets and storing or copying objects.
@@ -2285,52 +126,6 @@ doesn't copy the ACL from the source but rather writes a fresh one.
If the acl is an empty string then no X-Amz-Acl: header is added and
the default (private) will be used.
`,
- Provider: "!Cloudflare,FlashBlade,Mega,Rabata,Selectel,Servercore,SpectraLogic,Storj,Synology",
- Examples: []fs.OptionExample{{
- Value: "default",
- Help: "Owner gets Full_CONTROL.\nNo one else has access rights (default).",
- Provider: "TencentCOS",
- }, {
- Value: "private",
- Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).",
- Provider: "!IBMCOS,TencentCOS",
- }, {
- Value: "public-read",
- Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.",
- Provider: "!IBMCOS",
- }, {
- Value: "public-read-write",
- Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nGranting this on a bucket is generally not recommended.",
- Provider: "!IBMCOS",
- }, {
- Value: "authenticated-read",
- Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.",
- Provider: "!IBMCOS",
- }, {
- Value: "bucket-owner-read",
- Help: "Object owner gets FULL_CONTROL.\nBucket owner gets READ access.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
- Provider: "!ChinaMobile,IBMCOS",
- }, {
- Value: "bucket-owner-full-control",
- Help: "Both the object owner and the bucket owner get FULL_CONTROL over the object.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.",
- Provider: "!ChinaMobile,IBMCOS",
- }, {
- Value: "private",
- Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.",
- Provider: "IBMCOS",
- }, {
- Value: "public-read",
- Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.",
- Provider: "IBMCOS",
- }, {
- Value: "public-read-write",
- Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nThis acl is available on IBM Cloud (Infra), On-Premise IBM COS.",
- Provider: "IBMCOS",
- }, {
- Value: "authenticated-read",
- Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.\nNot supported on Buckets.\nThis acl is available on IBM Cloud (Infra) and On-Premise IBM COS.",
- Provider: "IBMCOS",
- }},
}, {
Name: "bucket_acl",
Help: `Canned ACL used when creating buckets.
@@ -2343,7 +138,6 @@ isn't set then "acl" is used instead.
If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl:
header is added and the default (private) will be used.
`,
- Provider: "!Cloudflare,FlashBlade,Rabata,Selectel,SpectraLogic,Storj,Synology",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "private",
@@ -2361,28 +155,14 @@ header is added and the default (private) will be used.
}, {
Name: "requester_pays",
Help: "Enables requester pays option when interacting with S3 bucket.",
- Provider: "AWS",
Default: false,
Advanced: true,
}, {
- Name: "server_side_encryption",
- Help: "The server-side encryption algorithm used when storing this object in S3.",
- Provider: "AWS,Ceph,ChinaMobile,Minio",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "None",
- }, {
- Value: "AES256",
- Help: "AES256",
- }, {
- Value: "aws:kms",
- Help: "aws:kms",
- Provider: "!ChinaMobile",
- }},
+ Name: "server_side_encryption",
+ Help: "The server-side encryption algorithm used when storing this object in S3.",
}, {
Name: "sse_customer_algorithm",
Help: "If using SSE-C, the server-side encryption algorithm used when storing this object in S3.",
- Provider: "AWS,Ceph,ChinaMobile,Minio",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -2392,9 +172,8 @@ header is added and the default (private) will be used.
Help: "AES256",
}},
}, {
- Name: "sse_kms_key_id",
- Help: "If using KMS ID you must provide the ARN of Key.",
- Provider: "AWS,Ceph,Minio",
+ Name: "sse_kms_key_id",
+ Help: "If using KMS ID you must provide the ARN of Key.",
Examples: []fs.OptionExample{{
Value: "",
Help: "None",
@@ -2408,7 +187,6 @@ header is added and the default (private) will be used.
Help: `To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key-base64.`,
- Provider: "AWS,Ceph,ChinaMobile,Minio",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -2420,7 +198,6 @@ Alternatively you can provide --sse-customer-key-base64.`,
Help: `If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data.
Alternatively you can provide --sse-customer-key.`,
- Provider: "AWS,Ceph,ChinaMobile,Minio",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -2433,7 +210,6 @@ Alternatively you can provide --sse-customer-key.`,
If you leave it blank, this is calculated automatically from the sse_customer_key provided.
`,
- Provider: "AWS,Ceph,ChinaMobile,Minio",
Advanced: true,
Examples: []fs.OptionExample{{
Value: "",
@@ -2441,157 +217,8 @@ If you leave it blank, this is calculated automatically from the sse_customer_ke
}},
Sensitive: true,
}, {
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in S3.",
- Provider: "AWS",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Default",
- }, {
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "REDUCED_REDUNDANCY",
- Help: "Reduced redundancy storage class",
- }, {
- Value: "STANDARD_IA",
- Help: "Standard Infrequent Access storage class",
- }, {
- Value: "ONEZONE_IA",
- Help: "One Zone Infrequent Access storage class",
- }, {
- Value: "GLACIER",
- Help: "Glacier Flexible Retrieval storage class",
- }, {
- Value: "DEEP_ARCHIVE",
- Help: "Glacier Deep Archive storage class",
- }, {
- Value: "INTELLIGENT_TIERING",
- Help: "Intelligent-Tiering storage class",
- }, {
- Value: "GLACIER_IR",
- Help: "Glacier Instant Retrieval storage class",
- }},
- }, {
- // Mapping from here: https://www.arvancloud.ir/en/products/cloud-storage
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in ArvanCloud.",
- Provider: "ArvanCloud",
- Examples: []fs.OptionExample{{
- Value: "STANDARD",
- Help: "Standard storage class",
- }},
- }, {
- // Mapping from here: https://www.alibabacloud.com/help/doc-detail/64919.htm
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in OSS.",
- Provider: "Alibaba",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Default",
- }, {
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "GLACIER",
- Help: "Archive storage mode",
- }, {
- Value: "STANDARD_IA",
- Help: "Infrequent access storage mode",
- }},
- }, {
- // Mapping from here: https://ecloud.10086.cn/op-help-center/doc/article/24495
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in ChinaMobile.",
- Provider: "ChinaMobile",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Default",
- }, {
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "GLACIER",
- Help: "Archive storage mode",
- }, {
- Value: "STANDARD_IA",
- Help: "Infrequent access storage mode",
- }},
- }, {
- // Mapping from here: https://liara.ir/landing/object-storage
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in Liara",
- Provider: "Liara",
- Examples: []fs.OptionExample{{
- Value: "STANDARD",
- Help: "Standard storage class",
- }},
- }, {
- // Mapping from here: https://docs.magalu.cloud/docs/storage/object-storage/Classes-de-Armazenamento/standard
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in Magalu.",
- Provider: "Magalu",
- Examples: []fs.OptionExample{{
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "GLACIER_IR",
- Help: "Glacier Instant Retrieval storage class",
- }},
- }, {
- // Mapping from here: https://developer.qiniu.com/kodo/5906/storage-type
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in Qiniu.",
- Provider: "Qiniu",
- Examples: []fs.OptionExample{{
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "LINE",
- Help: "Infrequent access storage mode",
- }, {
- Value: "GLACIER",
- Help: "Archive storage mode",
- }, {
- Value: "DEEP_ARCHIVE",
- Help: "Deep archive storage mode",
- }},
- }, {
- // Mapping from here: https://www.scaleway.com/en/docs/storage/object/quickstart/
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in S3.",
- Provider: "Scaleway",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Default.",
- }, {
- Value: "STANDARD",
- Help: "The Standard class for any upload.\nSuitable for on-demand content like streaming or CDN.\nAvailable in all regions.",
- }, {
- Value: "GLACIER",
- Help: "Archived storage.\nPrices are lower, but it needs to be restored first to be accessed.\nAvailable in FR-PAR and NL-AMS regions.",
- }, {
- Value: "ONEZONE_IA",
- Help: "One Zone - Infrequent Access.\nA good choice for storing secondary backup copies or easily re-creatable data.\nAvailable in the FR-PAR region only.",
- }},
- }, {
- // Mapping from here: https://intl.cloud.tencent.com/document/product/436/30925
- Name: "storage_class",
- Help: "The storage class to use when storing new objects in Tencent COS.",
- Provider: "TencentCOS",
- Examples: []fs.OptionExample{{
- Value: "",
- Help: "Default",
- }, {
- Value: "STANDARD",
- Help: "Standard storage class",
- }, {
- Value: "ARCHIVE",
- Help: "Archive storage mode",
- }, {
- Value: "STANDARD_IA",
- Help: "Infrequent access storage mode",
- }},
+ Name: "storage_class",
+ Help: "The storage class to use when storing new objects in S3.",
}, {
Name: "upload_cutoff",
Help: `Cutoff for switching to chunked upload.
@@ -2745,8 +372,7 @@ See [AWS Docs on Dualstack Endpoints](https://docs.aws.amazon.com/AmazonS3/lates
Default: false,
Advanced: true,
}, {
- Name: "use_accelerate_endpoint",
- Provider: "AWS",
+ Name: "use_accelerate_endpoint",
Help: `If true use the AWS S3 accelerated endpoint.
See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)`,
@@ -2758,8 +384,7 @@ See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/
Default: false,
Advanced: true,
}, {
- Name: "leave_parts_on_error",
- Provider: "AWS",
+ Name: "leave_parts_on_error",
Help: `If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery.
It should be set to true for resuming uploads across different sessions.
@@ -3045,7 +670,6 @@ In this case, you might want to try disabling this option.
}, {
Name: "sts_endpoint",
Help: "Endpoint for STS (deprecated).\n\nLeave blank if using AWS to use the default endpoint for the region.",
- Provider: "AWS",
Advanced: true,
Hide: fs.OptionHideBoth,
}, {
@@ -3143,7 +767,6 @@ Rclone limitations with Directory Buckets:
`, "|", "`"),
Default: false,
Advanced: true,
- Provider: "AWS",
}, {
Name: "sdk_log_mode",
Help: strings.ReplaceAll(`Set to debug the SDK
@@ -3165,18 +788,14 @@ use |-vv| to see the debug level logs.
`, "|", "`"),
Default: sdkLogMode(0),
Advanced: true,
+ }, {
+ Name: "ibm_api_key",
+ Help: "IBM API Key to be used to obtain IAM token",
+ }, {
+ Name: "ibm_resource_instance_id",
+ Help: "IBM service instance id",
},
- {
- Name: "ibm_api_key",
- Help: "IBM API Key to be used to obtain IAM token",
- Provider: "IBMCOS",
- },
- {
- Name: "ibm_resource_instance_id",
- Help: "IBM service instance id",
- Provider: "IBMCOS",
- },
- }})
+ }}))
}
// Constants
@@ -3612,7 +1231,7 @@ func (s3logger) Logf(classification logging.Classification, format string, v ...
}
// s3Connection makes a connection to s3
-func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Client *s3.Client, err error) {
+func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Client *s3.Client, provider *Provider, err error) {
ci := fs.GetConfig(ctx)
var awsConfig aws.Config
// Make the default static auth
@@ -3637,7 +1256,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
}
awsConfig, err = awsconfig.LoadDefaultConfig(ctx, configOpts...)
if err != nil {
- return nil, fmt.Errorf("couldn't load configuration with env_auth=true: %w", err)
+ return nil, nil, fmt.Errorf("couldn't load configuration with env_auth=true: %w", err)
}
} else {
@@ -3650,9 +1269,9 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
awsConfig.Credentials = aws.AnonymousCredentials{}
fs.Debugf(nil, "Using anonymous credentials - did you mean to set env_auth=true?")
case opt.AccessKeyID == "":
- return nil, errors.New("access_key_id not found")
+ return nil, nil, errors.New("access_key_id not found")
case opt.SecretAccessKey == "":
- return nil, errors.New("secret_access_key not found")
+ return nil, nil, errors.New("secret_access_key not found")
default:
// static credentials are already set
}
@@ -3661,7 +1280,14 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
if opt.Region == "" {
opt.Region = "us-east-1"
}
- setQuirks(opt)
+
+ provider = loadProvider(opt.Provider)
+ if provider == nil {
+ fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider)
+ provider = loadProvider("Other")
+ }
+
+ setQuirks(opt, provider)
awsConfig.RetryMaxAttempts = ci.LowLevelRetries
awsConfig.HTTPClient = client
@@ -3686,7 +1312,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
if opt.STSEndpoint != "" {
// FIXME not sure if anyone is using this
// Haven't figured out how to do it with the v2 SDK
- return nil, errors.New("--s3-sts-endpoint is no longer supported with the v2 SDK - please make an issue")
+ return nil, nil, errors.New("--s3-sts-endpoint is no longer supported with the v2 SDK - please make an issue")
}
if opt.Endpoint != "" {
if !strings.HasPrefix(opt.Endpoint, "http") {
@@ -3724,7 +1350,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
}
c := s3.NewFromConfig(awsConfig, options...)
- return c, nil
+ return c, provider, nil
}
func checkUploadChunkSize(cs fs.SizeSuffix) error {
@@ -3817,291 +1443,69 @@ func setEndpointValueForIDriveE2(m configmap.Mapper) (err error) {
// Run the integration tests to check you have the quirks correct.
//
// go test -v -remote NewS3Provider:
-func setQuirks(opt *Options) {
- var (
- listObjectsV2 = true // Always use ListObjectsV2 instead of ListObjects
- virtualHostStyle = true // Use bucket.provider.com instead of putting the bucket in the URL
- urlEncodeListings = true // URL encode the listings to help with control characters
- useMultipartEtag = true // Set if Etags for multipart uploads are compatible with AWS
- useAcceptEncodingGzip = true // Set Accept-Encoding: gzip
- mightGzip = true // assume all providers might use content encoding gzip until proven otherwise
- useAlreadyExists = true // Set if provider returns AlreadyOwnedByYou or no error if you try to remake your own bucket
- useMultipartUploads = true // Set if provider supports multipart uploads
- useUnsignedPayload = true // Do we need to use unsigned payloads to avoid seeking in PutObject
- useXID = true // Add x-id URL parameter into requests
- signAcceptEncoding = true // If we should include AcceptEncoding in the signature
- )
- switch opt.Provider {
- case "AWS":
- // No quirks
- mightGzip = false // Never auto gzips objects
- useUnsignedPayload = false // AWS has trailer support which means it adds checksums in the trailer without seeking
- case "Alibaba":
- useMultipartEtag = false // Alibaba seems to calculate multipart Etags differently from AWS
- useAlreadyExists = true // returns 200 OK
- case "Hetzner":
- useAlreadyExists = false
- case "HuaweiOBS":
- // Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever.
- urlEncodeListings = false
- listObjectsV2 = false
- useAlreadyExists = false // untested
- case "Ceph":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useAlreadyExists = true
- case "ChinaMobile":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "Cloudflare":
- virtualHostStyle = false
- useMultipartEtag = false // currently multipart Etags are random
- case "Cubbit":
- // no quirks
- useMultipartEtag = false // Cubbit calculates multipart Etags differently from AWS
- case "ArvanCloud":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "DigitalOcean":
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "Dreamhost":
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "FlashBlade":
- mightGzip = false // Never auto gzips objects
- virtualHostStyle = false // supports vhost but defaults to paths
- case "FileLu":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false
- case "IBMCOS":
- listObjectsV2 = false // untested
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false // untested
- useAlreadyExists = false // returns BucketAlreadyExists
- case "IDrive":
- virtualHostStyle = false
- useAlreadyExists = false // untested
- case "Intercolo":
- // no quirks
- useUnsignedPayload = false // Intercolo has trailer support
- case "IONOS":
- // listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2
- virtualHostStyle = false
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "Petabox":
- useAlreadyExists = false // untested
- case "Liara":
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false
- useAlreadyExists = false // untested
- case "Linode":
- useAlreadyExists = true // returns 200 OK
- case "LyveCloud":
- useMultipartEtag = false // LyveCloud seems to calculate multipart Etags differently from AWS
- useAlreadyExists = false // untested
- case "Magalu":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false
- useAlreadyExists = false
- case "Mega":
- listObjectsV2 = true
- virtualHostStyle = false
- urlEncodeListings = true
- useMultipartEtag = false
- useAlreadyExists = false
- // Multipart server side copies not supported
- opt.CopyCutoff = math.MaxInt64
- case "Minio":
- virtualHostStyle = false
- case "Netease":
- listObjectsV2 = false // untested
- urlEncodeListings = false
- useMultipartEtag = false // untested
- useAlreadyExists = false // untested
- case "Outscale":
- virtualHostStyle = false
- case "OVHcloud":
- // No quirks
- case "Rabata":
- // server side copy not supported
- case "RackCorp":
- // No quirks
- useMultipartEtag = false // untested
- useAlreadyExists = false // untested
- case "Rclone":
- listObjectsV2 = true
- urlEncodeListings = true
- virtualHostStyle = false
- useMultipartEtag = false
- useAlreadyExists = false
- // useMultipartUploads = false - set this manually
- // rclone serve doesn't support multi-part server side copy:
- // See: https://github.com/rclone/rclone/issues/7454
- // So make cutoff very large which it does support
- opt.CopyCutoff = math.MaxInt64
- case "Scaleway":
- // Scaleway can only have 1000 parts in an upload
- if opt.MaxUploadParts > 1000 {
- opt.MaxUploadParts = 1000
- }
- urlEncodeListings = true
- useAlreadyExists = true
- case "Selectel":
- urlEncodeListings = false
- case "Servercore":
- urlEncodeListings = false
- case "SeaweedFS":
- listObjectsV2 = false // untested
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false // untested
- useAlreadyExists = false // untested
- case "SpectraLogic":
- virtualHostStyle = false // path-style required
- case "StackPath":
- listObjectsV2 = false // untested
- virtualHostStyle = false
- urlEncodeListings = false
- useAlreadyExists = false // untested
- case "Storj":
- // Force chunk size to >= 64 MiB
- if opt.ChunkSize < 64*fs.Mebi {
- opt.ChunkSize = 64 * fs.Mebi
- }
- useAlreadyExists = false // returns BucketAlreadyExists
- // Storj doesn't support multi-part server side copy:
- // https://github.com/storj/roadmap/issues/40
- // So make cutoff very large which it does support
- opt.CopyCutoff = math.MaxInt64
- case "Synology":
- useMultipartEtag = false
- useAlreadyExists = false // untested
- case "TencentCOS":
- listObjectsV2 = false // untested
- useMultipartEtag = false // untested
- useAlreadyExists = false // untested
- case "Wasabi":
- useAlreadyExists = true // returns 200 OK
- case "Leviia":
- useAlreadyExists = false // untested
- case "Qiniu":
- useMultipartEtag = false
- urlEncodeListings = false
- virtualHostStyle = false
- useAlreadyExists = false // untested
- case "Zata":
- useMultipartEtag = false
- mightGzip = false
- useUnsignedPayload = false
- useAlreadyExists = false
- case "Exaba":
- virtualHostStyle = false
- case "GCS":
- // Google break request Signature by mutating accept-encoding HTTP header
- // https://github.com/rclone/rclone/issues/6670
- useAcceptEncodingGzip = false
- signAcceptEncoding = false
- useAlreadyExists = true // returns BucketNameUnavailable instead of BucketAlreadyExists but good enough!
- // GCS S3 doesn't support multi-part server side copy:
- // See: https://issuetracker.google.com/issues/323465186
- // So make cutoff very large which it does seem to support
- opt.CopyCutoff = math.MaxInt64
- // GCS doesn't like the x-id URL parameter the SDKv2 inserts
- useXID = false
- default: //nolint:gocritic // Don't include gocritic when running golangci-lint to avoid defaultCaseOrder: consider to make `default` case as first or as last case
- fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider)
- fallthrough
- case "Other":
- listObjectsV2 = false
- virtualHostStyle = false
- urlEncodeListings = false
- useMultipartEtag = false
- useAlreadyExists = false
+func setQuirks(opt *Options, provider *Provider) {
+ // Set tristate to the ultimate default value or the override
+ // in provider.Quirks. Pass in the ultimate default as value.
+ set := func(tristate *fs.Tristate, value bool, override *bool) {
+ if override != nil {
+ value = *override
+ }
+ if !tristate.Valid {
+ tristate.Valid = true
+ tristate.Value = value
+ }
}
- // Path Style vs Virtual Host style
+ // Set Path Style vs Virtual Host style
+ var virtualHostStyle = true // Default use bucket.provider.com instead of putting the bucket in the URL
+ if provider.Quirks.ForcePathStyle != nil {
+ // "don't force path style" so inverted
+ virtualHostStyle = !*provider.Quirks.ForcePathStyle
+ }
if virtualHostStyle || opt.UseAccelerateEndpoint {
opt.ForcePathStyle = false
}
- // Set to see if we need to URL encode listings
- if !opt.ListURLEncode.Valid {
- opt.ListURLEncode.Valid = true
- opt.ListURLEncode.Value = urlEncodeListings
- }
-
// Set the correct list version if not manually set
if opt.ListVersion == 0 {
- if listObjectsV2 {
- opt.ListVersion = 2
+ if provider.Quirks.ListVersion != nil {
+ opt.ListVersion = *provider.Quirks.ListVersion
} else {
- opt.ListVersion = 1
+ opt.ListVersion = 2
}
}
- // Set the correct use multipart Etag for error checking if not manually set
- if !opt.UseMultipartEtag.Valid {
- opt.UseMultipartEtag.Valid = true
- opt.UseMultipartEtag.Value = useMultipartEtag
- }
-
- // set MightGzip if not manually set
- if !opt.MightGzip.Valid {
- opt.MightGzip.Valid = true
- opt.MightGzip.Value = mightGzip
+ // Set the copy cutoff if not manually set
+ // Check equality with strings as config values get round tripped via strings
+ if opt.CopyCutoff.String() == fs.SizeSuffix(maxSizeForCopy).String() {
+ if provider.Quirks.CopyCutoff != nil {
+ opt.CopyCutoff = fs.SizeSuffix(*provider.Quirks.CopyCutoff)
+ }
}
- // set UseAcceptEncodingGzip if not manually set
- if !opt.UseAcceptEncodingGzip.Valid {
- opt.UseAcceptEncodingGzip.Valid = true
- opt.UseAcceptEncodingGzip.Value = useAcceptEncodingGzip
+ // Clip the max upload parts to the quirk
+ if provider.Quirks.MaxUploadParts != nil {
+ opt.MaxUploadParts = min(opt.MaxUploadParts, *provider.Quirks.MaxUploadParts)
}
- // Has the provider got AlreadyOwnedByYou error?
- if !opt.UseAlreadyExists.Valid {
- opt.UseAlreadyExists.Valid = true
- opt.UseAlreadyExists.Value = useAlreadyExists
+ // Clip the chunk size to the quirk
+ if provider.Quirks.MinChunkSize != nil {
+ opt.ChunkSize = max(opt.ChunkSize, fs.SizeSuffix(*provider.Quirks.MinChunkSize))
}
- // Set the correct use multipart uploads if not manually set
- if !opt.UseMultipartUploads.Valid {
- opt.UseMultipartUploads.Valid = true
- opt.UseMultipartUploads.Value = useMultipartUploads
- }
+ // Set new style Tristate quirks
+ set(&opt.ListURLEncode, true, provider.Quirks.ListURLEncode)
+ set(&opt.UseMultipartEtag, true, provider.Quirks.UseMultipartEtag)
+ set(&opt.UseAcceptEncodingGzip, true, provider.Quirks.UseAcceptEncodingGzip)
+ set(&opt.MightGzip, true, provider.Quirks.MightGzip)
+ set(&opt.UseAlreadyExists, true, provider.Quirks.UseAlreadyExists)
+ set(&opt.UseMultipartUploads, true, provider.Quirks.UseMultipartUploads)
if !opt.UseMultipartUploads.Value {
opt.UploadCutoff = math.MaxInt64
}
-
- // Set the correct use multipart uploads if not manually set
- if !opt.UseUnsignedPayload.Valid {
- opt.UseUnsignedPayload.Valid = true
- opt.UseUnsignedPayload.Value = useUnsignedPayload
- }
-
- // Set the correct use UseXID if not manually set
- if !opt.UseXID.Valid {
- opt.UseXID.Valid = true
- opt.UseXID.Value = useXID
- }
-
- // Set the correct SignAcceptEncoding if not manually set
- if !opt.SignAcceptEncoding.Valid {
- opt.SignAcceptEncoding.Valid = true
- opt.SignAcceptEncoding.Value = signAcceptEncoding
- }
+ set(&opt.UseUnsignedPayload, true, provider.Quirks.UseUnsignedPayload)
+ set(&opt.UseXID, true, provider.Quirks.UseXID)
+ set(&opt.SignAcceptEncoding, true, provider.Quirks.SignAcceptEncoding)
}
// setRoot changes the root of the Fs
@@ -4155,7 +1559,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:])
}
srv := getClient(ctx, opt)
- c, err := s3Connection(ctx, opt, srv)
+ c, provider, err := s3Connection(ctx, opt, srv)
if err != nil {
return nil, err
}
@@ -4205,26 +1609,16 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
UserMetadata: true,
BucketBased: true,
BucketBasedRootOK: true,
- SetTier: true,
- GetTier: true,
+ SetTier: provider.StorageClass.Len() > 0,
+ GetTier: provider.StorageClass.Len() > 0,
SlowModTime: true,
}).Fill(ctx, f)
- if opt.Provider == "Storj" {
- f.features.SetTier = false
- f.features.GetTier = false
- }
- if opt.Provider == "IDrive" {
- f.features.SetTier = false
- }
if opt.Provider == "AWS" {
f.features.DoubleSlash = true
}
if opt.Provider == "Rabata" {
f.features.Copy = nil
}
- if opt.Provider == "Hetzner" {
- f.features.SetTier = false
- }
if opt.DirectoryMarkers {
f.features.CanHaveEmptyDirectories = true
}
@@ -4392,7 +1786,7 @@ func (f *Fs) updateRegionForBucket(ctx context.Context, bucket string) error {
// Make a new session with the new region
oldRegion := f.opt.Region
f.opt.Region = region
- c, err := s3Connection(f.ctx, &f.opt, f.srv)
+ c, _, err := s3Connection(f.ctx, &f.opt, f.srv)
if err != nil {
return fmt.Errorf("creating new session failed: %w", err)
}
@@ -5801,7 +3195,7 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str
if err != nil {
return nil, fmt.Errorf("reading config: %w", err)
}
- c, err := s3Connection(f.ctx, &newOpt, f.srv)
+ c, _, err := s3Connection(f.ctx, &newOpt, f.srv)
if err != nil {
return nil, fmt.Errorf("updating session: %w", err)
}
diff --git a/backend/s3/s3_test.go b/backend/s3/s3_test.go
index 6d00d8761..973314c5c 100644
--- a/backend/s3/s3_test.go
+++ b/backend/s3/s3_test.go
@@ -62,14 +62,14 @@ func TestAWSDualStackOption(t *testing.T) {
// test enabled
ctx, opt, client := SetupS3Test(t)
opt.UseDualStack = true
- s3Conn, err := s3Connection(ctx, opt, client)
+ s3Conn, _, err := s3Connection(ctx, opt, client)
require.NoError(t, err)
assert.Equal(t, aws.DualStackEndpointStateEnabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
}
{
// test default case
ctx, opt, client := SetupS3Test(t)
- s3Conn, err := s3Connection(ctx, opt, client)
+ s3Conn, _, err := s3Connection(ctx, opt, client)
require.NoError(t, err)
assert.Equal(t, aws.DualStackEndpointStateDisabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
}
diff --git a/fs/config/ui.go b/fs/config/ui.go
index 6e31923fe..1a897aefe 100644
--- a/fs/config/ui.go
+++ b/fs/config/ui.go
@@ -3,6 +3,7 @@
package config
import (
+ "bufio"
"context"
"errors"
"fmt"
@@ -12,6 +13,7 @@ import (
"sort"
"strconv"
"strings"
+ "sync"
"unicode/utf8"
"github.com/peterh/liner"
@@ -25,8 +27,24 @@ import (
"golang.org/x/text/unicode/norm"
)
+var (
+ stdinBufOnce sync.Once
+ stdinBuf *bufio.Reader
+)
+
// ReadLine reads an unlimited length line from stdin with a prompt.
var ReadLine = func(prompt string) string {
+ if !terminal.IsTerminal(int(os.Stdout.Fd())) {
+ stdinBufOnce.Do(func() {
+ stdinBuf = bufio.NewReader(os.Stdin)
+ })
+ line, err := stdinBuf.ReadString('\n')
+ if err != nil && (line == "" || err != io.EOF) {
+ fs.Fatalf(nil, "Failed to read line: %v", err)
+ }
+ return strings.TrimSpace(line)
+ }
+
l := liner.NewLiner()
defer func() {
_ = l.Close()
diff --git a/go.mod b/go.mod
index a7f6ea9b7..6033ce8b8 100644
--- a/go.mod
+++ b/go.mod
@@ -75,6 +75,7 @@ require (
github.com/unknwon/goconfig v1.0.0
github.com/willscott/go-nfs v0.0.3
github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471
+ github.com/wk8/go-ordered-map/v2 v2.1.8
github.com/xanzy/ssh-agent v0.3.3
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78
github.com/yunify/qingstor-sdk-go/v3 v3.2.0
@@ -126,10 +127,12 @@ require (
github.com/aws/aws-sdk-go-v2/service/sso v1.29.4 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.0 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.38.5 // indirect
+ github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect
github.com/bradenaw/juniper v0.15.3 // indirect
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect
+ github.com/buger/jsonparser v1.1.1 // indirect
github.com/calebcase/tmpfile v1.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect
@@ -185,6 +188,7 @@ require (
github.com/lpar/date v1.0.0 // indirect
github.com/lucasb-eyer/go-colorful v1.3.0 // indirect
github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54 // indirect
+ github.com/mailru/easyjson v0.7.7 // indirect
github.com/mattn/go-isatty v0.0.20 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
diff --git a/go.sum b/go.sum
index f3a86c7dc..853032209 100644
--- a/go.sum
+++ b/go.sum
@@ -148,6 +148,8 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.38.5 h1:+LVB0xBqEgjQoqr9bGZbRzvg212B
github.com/aws/aws-sdk-go-v2/service/sts v1.38.5/go.mod h1:xoaxeqnnUaZjPjaICgIy5B+MHCSb/ZSOn4MvkFNOUA0=
github.com/aws/smithy-go v1.23.0 h1:8n6I3gXzWJB2DxBDnfxgBaSX6oe0d/t10qGz7OKqMCE=
github.com/aws/smithy-go v1.23.0/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI=
+github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk=
+github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg=
github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM=
github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw=
github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc h1:biVzkmvwrH8WK8raXaxBx6fRVTlJILwEwQGL1I/ByEI=
@@ -158,6 +160,8 @@ github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 h1:GKTyiRCL6zVf5wWaq
github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8/go.mod h1:spo1JLcs67NmW1aVLEgtA8Yy1elc+X8y5SRW1sFW4Og=
github.com/buengese/sgzip v0.1.1 h1:ry+T8l1mlmiWEsDrH/YHZnCVWD2S3im1KLsyO+8ZmTU=
github.com/buengese/sgzip v0.1.1/go.mod h1:i5ZiXGF3fhV7gL1xaRRL1nDnmpNj0X061FQzOS8VMas=
+github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs=
+github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0=
github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0=
github.com/bytedance/sonic v1.13.2 h1:8/H1FempDZqC4VqjptGo14QQlJx8VdZJegxs6wwfqpQ=
github.com/bytedance/sonic v1.13.2/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4=
@@ -402,6 +406,7 @@ github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 h1:ZxO6Qr2GOXPdcW80
github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3/go.mod h1:dvLUr/8Fs9a2OBrEnCC5duphbkz/k/mSy5OkXg3PAgI=
github.com/josephspurrier/goversioninfo v1.5.0 h1:9TJtORoyf4YMoWSOo/cXFN9A/lB3PniJ91OxIH6e7Zg=
github.com/josephspurrier/goversioninfo v1.5.0/go.mod h1:6MoTvFZ6GKJkzcdLnU5T/RGYUbHQbKpYeNP0AgQLd2o=
+github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y=
github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM=
github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo=
github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU=
@@ -447,6 +452,8 @@ github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQ
github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54 h1:mFWunSatvkQQDhpdyuFAYwyAan3hzCuma+Pz8sqvOfg=
github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg=
+github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0=
+github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc=
github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE=
github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY=
@@ -598,8 +605,6 @@ github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
-github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5 h1:Sa+sR8aaAMFwxhXWENEnE6ZpqhZ9d7u1RT2722Rw6hc=
-github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5/go.mod h1:UdZiFUFu6e2WjjtjxivwXWcwc1N/8zgbkBR9QNucUOY=
github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c h1:BLopNCyqewbE8+BtlIp/Juzu8AJGxz0gHdGADnsblVc=
github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c/go.mod h1:ykucQyiE9Q2qx1wLlEtZkkNn1IURib/2O+Mvd25i1Fo=
github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502/go.mod h1:p9lPsd+cx33L3H9nNoecRRxPssFKUwwI50I3pZ0yT+8=
@@ -623,6 +628,8 @@ github.com/willscott/go-nfs-client v0.0.0-20240104095149-b44639837b00 h1:U0DnHRZ
github.com/willscott/go-nfs-client v0.0.0-20240104095149-b44639837b00/go.mod h1:Tq++Lr/FgiS3X48q5FETemXiSLGuYMQT2sPjYNPJSwA=
github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471 h1:aSOo0k+aLWdhUQiUxzv4cZ7cUp3OLP+Qx7cjs6OUxME=
github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471/go.mod h1:uxjoF2jEYT3+x+vC2KJddEGdk/LU8pRowXmyVMHSV5I=
+github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc=
+github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw=
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw=
github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=