jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-12-06 00:03:32 +00:00
Code Issues Projects Releases Wiki Activity

s3: make it easier to add new S3 providers

Browse Source 
Before this change, you had to modify a fragile data-structure
containing all providers. This often led to things being out of order,
duplicates and conflicts whilst merging. As well as the changes for
one provider being in different places across the file.

After this change, new providers are defined in an easy to edit YAML file,
one per provider.

The config output has been tested before and after for all providers
and any changes are cosmetic only.
This commit is contained in:
dougal
2025-10-14 17:50:28 +01:00
committed by Nick Craig-Wood
parent 2cf44e584c
commit f28c83c6de
54 changed files with 1769 additions and 2734 deletions
Download Patch File Download Diff File Expand all files Collapse all files

194
backend/s3/README.md Normal file
View File

@@ -0,0 +1,194 @@
## Adding a new s3 provider
It is quite easy to add a new S3 provider to rclone.
You'll then need to do add the following (optional tags are in [] and
do not get displayed in rclone config if empty):
The process is as follows: Create yaml -> add docs -> run tests ->
adjust yaml until tests pass.
All tags can be found in `backend/s3/providers.go` Provider Struct.
Looking through a few of the yaml files as examples should make things
clear. `AWS.yaml` as the most config. pasting.
### YAML
In `backend/s3/provider/YourProvider.yaml`
- name
- description
- More like the full name often "YourProvider + Object Storage"
- [Region]
- Any regions your provider supports or the defaults (use `region: {}` for this)
- Example from AWS.yaml:
```yaml
region:
us-east-1: |-
The default endpoint - a good choice if you are unsure.
US Region, Northern Virginia, or Pacific Northwest.
Leave location constraint empty.
```
- The defaults (as seen in Rclone.yaml):
```yaml
region:
"": |-
Use this if unsure.
Will use v4 signatures and an empty region.
other-v2-signature: |-
Use this only if v4 signatures don't work.
E.g. pre Jewel/v10 CEPH.
```
- [Endpoint]
- Any endpoints your provider supports
- Example from Mega.yaml
```yaml
endpoint:
s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam)
```
- [Location Constraint]
- The Location Constraint of your remote, often same as region.
- Example from AWS.yaml
```yaml
location_constraint:
"": Empty for US Region, Northern Virginia, or Pacific Northwest
us-east-2: US East (Ohio) Region
```
- [ACL]
- Identical across *most* providers. Select the default with `acl: {}`
- Example from AWS.yaml
```yaml
acl:
private: |-
Owner gets FULL_CONTROL.
No one else has access rights (default).
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
public-read-write: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ and WRITE access.
Granting this on a bucket is generally not recommended.
authenticated-read: |-
Owner gets FULL_CONTROL.
The AuthenticatedUsers group gets READ access.
bucket-owner-read: |-
Object owner gets FULL_CONTROL.
Bucket owner gets READ access.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
bucket-owner-full-control: |-
Both the object owner and the bucket owner get FULL_CONTROL over the object.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
```
- [Storage Class]
- Identical across *most* providers.
- Defaults from AWS.yaml
```yaml
storage_class:
"": Default
STANDARD: Standard storage class
REDUCED_REDUNDANCY: Reduced redundancy storage class
STANDARD_IA: Standard Infrequent Access storage class
ONEZONE_IA: One Zone Infrequent Access storage class
GLACIER: Glacier Flexible Retrieval storage class
DEEP_ARCHIVE: Glacier Deep Archive storage class
INTELLIGENT_TIERING: Intelligent-Tiering storage class
GLACIER_IR: Glacier Instant Retrieval storage class
```
- [Server Side Encryption]
- Not common, identical across *most* providers.
- Defaults from AWS.yaml
```yaml
server_side_encryption:
"": None
AES256: AES256
aws:kms: aws:kms
```
- [Advanced Options]
- All advanced options are Boolean - if true the configurator asks about that value, if not it doesn't:
```go
BucketACL bool `yaml:"bucket_acl,omitempty"`
DirectoryBucket bool `yaml:"directory_bucket,omitempty"`
LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"`
RequesterPays bool `yaml:"requester_pays,omitempty"`
SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"`
SSECustomerKey bool `yaml:"sse_customer_key,omitempty"`
SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"`
SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"`
SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"`
STSEndpoint bool `yaml:"sts_endpoint,omitempty"`
UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"`
```
- Example from AWS.yaml:
```yaml
bucket_acl: true
directory_bucket: true
leave_parts_on_error: true
requester_pays: true
sse_customer_algorithm: true
sse_customer_key: true
sse_customer_key_base64: true
sse_customer_key_md5: true
sse_kms_key_id: true
sts_endpoint: true
use_accelerate_endpoint: true
```
- Quirks
- Quirks are discovered through documentation and running the tests as seen below.
- Most quirks are *bool as to have 3 values, `true`, `false` and `dont care`.
```go
type Quirks struct {
ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2
ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style
ListURLEncode *bool `yaml:"list_url_encode,omitempty"`
UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"`
UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"`
UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"`
MightGzip *bool `yaml:"might_gzip,omitempty"`
UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"`
UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"`
UseXID *bool `yaml:"use_x_id,omitempty"`
SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
}
```
- Example from AWS.yaml
```yaml
quirks:
might_gzip: false # Never auto gzips objects
use_unsigned_payload: false # AWS has trailer support
```
Note that if you omit a section, eg `region` then the user won't be
asked that question, and if you add an empty section e.g. `region: {}`
then the defaults from the `Other.yaml` will be used.
### DOCS
- `docs/content/s3.md`
- Add the provider at the top of the page.
- Add a section about the provider linked from there.
- Make sure this is in alphabetical order in the `Providers` section.
- Add a transcript of a trial `rclone config` session
- Edit the transcript to remove things which might change in subsequent versions
- **Do not** alter or add to the autogenerated parts of `s3.md`
- Rule of thumb: don't edit anything not mentioned above.
- **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3`
- This will make autogenerated changes!
- `README.md` - this is the home page in github
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
- `docs/content/_index.md` - this is the home page of rclone.org
- Add the provider and a link to the section you wrote in `docs/contents/s3.md`
- Once you've written the docs, run `make serve` and check they look OK
in the web browser and the links (internal and external) all work.
### TESTS
Once you've written the code, test `rclone config` works to your
satisfaction and looks correct, and check the integration tests work
`go test -v -remote NewS3Provider:`. You may need to adjust the quirks
to get them to pass. Some providers just can't pass the tests with
control characters in the names so if these fail and the provider
doesn't support `urlEncodeListings` in the quirks then ignore them.

139
backend/s3/provider/AWS.yaml Normal file
View File

@@ -0,0 +1,139 @@
name: AWS
description: Amazon Web Services (AWS) S3
region:
us-east-1: |-
The default endpoint - a good choice if you are unsure.
US Region, Northern Virginia, or Pacific Northwest.
Leave location constraint empty.
us-east-2: |-
US East (Ohio) Region.
Needs location constraint us-east-2.
us-west-1: |-
US West (Northern California) Region.
Needs location constraint us-west-1.
us-west-2: |-
US West (Oregon) Region.
Needs location constraint us-west-2.
ca-central-1: |-
Canada (Central) Region.
Needs location constraint ca-central-1.
eu-west-1: |-
EU (Ireland) Region.
Needs location constraint EU or eu-west-1.
eu-west-2: |-
EU (London) Region.
Needs location constraint eu-west-2.
eu-west-3: |-
EU (Paris) Region.
Needs location constraint eu-west-3.
eu-north-1: |-
EU (Stockholm) Region.
Needs location constraint eu-north-1.
eu-south-1: |-
EU (Milan) Region.
Needs location constraint eu-south-1.
eu-central-1: |-
EU (Frankfurt) Region.
Needs location constraint eu-central-1.
ap-southeast-1: |-
Asia Pacific (Singapore) Region.
Needs location constraint ap-southeast-1.
ap-southeast-2: |-
Asia Pacific (Sydney) Region.
Needs location constraint ap-southeast-2.
ap-northeast-1: |-
Asia Pacific (Tokyo) Region.
Needs location constraint ap-northeast-1.
ap-northeast-2: |-
Asia Pacific (Seoul).
Needs location constraint ap-northeast-2.
ap-northeast-3: |-
Asia Pacific (Osaka-Local).
Needs location constraint ap-northeast-3.
ap-south-1: |-
Asia Pacific (Mumbai).
Needs location constraint ap-south-1.
ap-east-1: |-
Asia Pacific (Hong Kong) Region.
Needs location constraint ap-east-1.
sa-east-1: |-
South America (Sao Paulo) Region.
Needs location constraint sa-east-1.
il-central-1: |-
Israel (Tel Aviv) Region.
Needs location constraint il-central-1.
me-south-1: |-
Middle East (Bahrain) Region.
Needs location constraint me-south-1.
af-south-1: |-
Africa (Cape Town) Region.
Needs location constraint af-south-1.
cn-north-1: |-
China (Beijing) Region.
Needs location constraint cn-north-1.
cn-northwest-1: |-
China (Ningxia) Region.
Needs location constraint cn-northwest-1.
us-gov-east-1: |-
AWS GovCloud (US-East) Region.
Needs location constraint us-gov-east-1.
us-gov-west-1: |-
AWS GovCloud (US) Region.
Needs location constraint us-gov-west-1.
endpoint: {}
location_constraint:
'': Empty for US Region, Northern Virginia, or Pacific Northwest
us-east-2: US East (Ohio) Region
us-west-1: US West (Northern California) Region
us-west-2: US West (Oregon) Region
ca-central-1: Canada (Central) Region
eu-west-1: EU (Ireland) Region
eu-west-2: EU (London) Region
eu-west-3: EU (Paris) Region
eu-north-1: EU (Stockholm) Region
eu-south-1: EU (Milan) Region
EU: EU Region
ap-southeast-1: Asia Pacific (Singapore) Region
ap-southeast-2: Asia Pacific (Sydney) Region
ap-northeast-1: Asia Pacific (Tokyo) Region
ap-northeast-2: Asia Pacific (Seoul) Region
ap-northeast-3: Asia Pacific (Osaka-Local) Region
ap-south-1: Asia Pacific (Mumbai) Region
ap-east-1: Asia Pacific (Hong Kong) Region
sa-east-1: South America (Sao Paulo) Region
il-central-1: Israel (Tel Aviv) Region
me-south-1: Middle East (Bahrain) Region
af-south-1: Africa (Cape Town) Region
cn-north-1: China (Beijing) Region
cn-northwest-1: China (Ningxia) Region
us-gov-east-1: AWS GovCloud (US-East) Region
us-gov-west-1: AWS GovCloud (US) Region
acl: {}
storage_class:
'': Default
STANDARD: Standard storage class
REDUCED_REDUNDANCY: Reduced redundancy storage class
STANDARD_IA: Standard Infrequent Access storage class
ONEZONE_IA: One Zone Infrequent Access storage class
GLACIER: Glacier Flexible Retrieval storage class
DEEP_ARCHIVE: Glacier Deep Archive storage class
INTELLIGENT_TIERING: Intelligent-Tiering storage class
GLACIER_IR: Glacier Instant Retrieval storage class
server_side_encryption:
'': None
AES256: AES256
aws:kms: aws:kms
bucket_acl: true
directory_bucket: true
leave_parts_on_error: true
requester_pays: true
sse_customer_algorithm: true
sse_customer_key: true
sse_customer_key_base64: true
sse_customer_key_md5: true
sse_kms_key_id: true
sts_endpoint: true
use_accelerate_endpoint: true
quirks:
might_gzip: false # Never auto gzips objects
use_unsigned_payload: false # AWS has trailer support which means it adds checksums in the trailer without seeking

37
backend/s3/provider/Alibaba.yaml Normal file
View File

@@ -0,0 +1,37 @@
name: Alibaba
description: Alibaba Cloud Object Storage System (OSS) formerly Aliyun
endpoint:
oss-accelerate.aliyuncs.com: Global Accelerate
oss-accelerate-overseas.aliyuncs.com: Global Accelerate (outside mainland China)
oss-cn-hangzhou.aliyuncs.com: East China 1 (Hangzhou)
oss-cn-shanghai.aliyuncs.com: East China 2 (Shanghai)
oss-cn-qingdao.aliyuncs.com: North China 1 (Qingdao)
oss-cn-beijing.aliyuncs.com: North China 2 (Beijing)
oss-cn-zhangjiakou.aliyuncs.com: North China 3 (Zhangjiakou)
oss-cn-huhehaote.aliyuncs.com: North China 5 (Hohhot)
oss-cn-wulanchabu.aliyuncs.com: North China 6 (Ulanqab)
oss-cn-shenzhen.aliyuncs.com: South China 1 (Shenzhen)
oss-cn-heyuan.aliyuncs.com: South China 2 (Heyuan)
oss-cn-guangzhou.aliyuncs.com: South China 3 (Guangzhou)
oss-cn-chengdu.aliyuncs.com: West China 1 (Chengdu)
oss-cn-hongkong.aliyuncs.com: Hong Kong (Hong Kong)
oss-us-west-1.aliyuncs.com: US West 1 (Silicon Valley)
oss-us-east-1.aliyuncs.com: US East 1 (Virginia)
oss-ap-southeast-1.aliyuncs.com: Southeast Asia Southeast 1 (Singapore)
oss-ap-southeast-2.aliyuncs.com: Asia Pacific Southeast 2 (Sydney)
oss-ap-southeast-3.aliyuncs.com: Southeast Asia Southeast 3 (Kuala Lumpur)
oss-ap-southeast-5.aliyuncs.com: Asia Pacific Southeast 5 (Jakarta)
oss-ap-northeast-1.aliyuncs.com: Asia Pacific Northeast 1 (Japan)
oss-ap-south-1.aliyuncs.com: Asia Pacific South 1 (Mumbai)
oss-eu-central-1.aliyuncs.com: Central Europe 1 (Frankfurt)
oss-eu-west-1.aliyuncs.com: West Europe (London)
oss-me-east-1.aliyuncs.com: Middle East 1 (Dubai)
acl: {}
storage_class:
'': Default
STANDARD: Standard storage class
GLACIER: Archive storage mode
STANDARD_IA: Infrequent access storage mode
bucket_acl: true
quirks:
use_multipart_etag: false # multipar ETags differ from AWS

19
backend/s3/provider/ArvanCloud.yaml Normal file
View File

@@ -0,0 +1,19 @@
name: ArvanCloud
description: Arvan Cloud Object Storage (AOS)
endpoint:
s3.ir-thr-at1.arvanstorage.ir: |-
The default endpoint - a good choice if you are unsure.
Tehran Iran (Simin)
s3.ir-tbz-sh1.arvanstorage.ir: Tabriz Iran (Shahriar)
location_constraint:
ir-thr-at1: Tehran Iran (Simin)
ir-tbz-sh1: Tabriz Iran (Shahriar)
acl: {}
storage_class:
STANDARD: Standard storage class
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_already_exists: false

20
backend/s3/provider/Ceph.yaml Normal file
View File

@@ -0,0 +1,20 @@
name: Ceph
description: Ceph Object Storage
region: {}
endpoint: {}
location_constraint: {}
acl: {}
server_side_encryption:
'': None
AES256: AES256
aws:kms: aws:kms
bucket_acl: true
sse_customer_algorithm: true
sse_customer_key: true
sse_customer_key_base64: true
sse_customer_key_md5: true
sse_kms_key_id: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false

98
backend/s3/provider/ChinaMobile.yaml Normal file
View File

@@ -0,0 +1,98 @@
name: ChinaMobile
description: China Mobile Ecloud Elastic Object Storage (EOS)
endpoint:
eos-wuxi-1.cmecloud.cn: |-
The default endpoint - a good choice if you are unsure.
East China (Suzhou)
eos-jinan-1.cmecloud.cn: East China (Jinan)
eos-ningbo-1.cmecloud.cn: East China (Hangzhou)
eos-shanghai-1.cmecloud.cn: East China (Shanghai-1)
eos-zhengzhou-1.cmecloud.cn: Central China (Zhengzhou)
eos-hunan-1.cmecloud.cn: Central China (Changsha-1)
eos-zhuzhou-1.cmecloud.cn: Central China (Changsha-2)
eos-guangzhou-1.cmecloud.cn: South China (Guangzhou-2)
eos-dongguan-1.cmecloud.cn: South China (Guangzhou-3)
eos-beijing-1.cmecloud.cn: North China (Beijing-1)
eos-beijing-2.cmecloud.cn: North China (Beijing-2)
eos-beijing-4.cmecloud.cn: North China (Beijing-3)
eos-huhehaote-1.cmecloud.cn: North China (Huhehaote)
eos-chengdu-1.cmecloud.cn: Southwest China (Chengdu)
eos-chongqing-1.cmecloud.cn: Southwest China (Chongqing)
eos-guiyang-1.cmecloud.cn: Southwest China (Guiyang)
eos-xian-1.cmecloud.cn: Nouthwest China (Xian)
eos-yunnan.cmecloud.cn: Yunnan China (Kunming)
eos-yunnan-2.cmecloud.cn: Yunnan China (Kunming-2)
eos-tianjin-1.cmecloud.cn: Tianjin China (Tianjin)
eos-jilin-1.cmecloud.cn: Jilin China (Changchun)
eos-hubei-1.cmecloud.cn: Hubei China (Xiangyan)
eos-jiangxi-1.cmecloud.cn: Jiangxi China (Nanchang)
eos-gansu-1.cmecloud.cn: Gansu China (Lanzhou)
eos-shanxi-1.cmecloud.cn: Shanxi China (Taiyuan)
eos-liaoning-1.cmecloud.cn: Liaoning China (Shenyang)
eos-hebei-1.cmecloud.cn: Hebei China (Shijiazhuang)
eos-fujian-1.cmecloud.cn: Fujian China (Xiamen)
eos-guangxi-1.cmecloud.cn: Guangxi China (Nanning)
eos-anhui-1.cmecloud.cn: Anhui China (Huainan)
location_constraint:
wuxi1: East China (Suzhou)
jinan1: East China (Jinan)
ningbo1: East China (Hangzhou)
shanghai1: East China (Shanghai-1)
zhengzhou1: Central China (Zhengzhou)
hunan1: Central China (Changsha-1)
zhuzhou1: Central China (Changsha-2)
guangzhou1: South China (Guangzhou-2)
dongguan1: South China (Guangzhou-3)
beijing1: North China (Beijing-1)
beijing2: North China (Beijing-2)
beijing4: North China (Beijing-3)
huhehaote1: North China (Huhehaote)
chengdu1: Southwest China (Chengdu)
chongqing1: Southwest China (Chongqing)
guiyang1: Southwest China (Guiyang)
xian1: Northwest China (Xian)
yunnan: Yunnan China (Kunming)
yunnan2: Yunnan China (Kunming-2)
tianjin1: Tianjin China (Tianjin)
jilin1: Jilin China (Changchun)
hubei1: Hubei China (Xiangyan)
jiangxi1: Jiangxi China (Nanchang)
gansu1: Gansu China (Lanzhou)
shanxi1: Shanxi China (Taiyuan)
liaoning1: Liaoning China (Shenyang)
hebei1: Hebei China (Shijiazhuang)
fujian1: Fujian China (Xiamen)
guangxi1: Guangxi China (Nanning)
anhui1: Anhui China (Huainan)
acl:
private: |-
Owner gets FULL_CONTROL.
No one else has access rights (default).
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
public-read-write: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ and WRITE access.
Granting this on a bucket is generally not recommended.
authenticated-read: |-
Owner gets FULL_CONTROL.
The AuthenticatedUsers group gets READ access.
storage_class:
'': Default
STANDARD: Standard storage class
GLACIER: Archive storage mode
STANDARD_IA: Infrequent access storage mode
server_side_encryption:
'': None
AES256: AES256
bucket_acl: true
sse_customer_algorithm: true
sse_customer_key: true
sse_customer_key_base64: true
sse_customer_key_md5: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_already_exists: false

8
backend/s3/provider/Cloudflare.yaml Normal file
View File

@@ -0,0 +1,8 @@
name: Cloudflare
description: Cloudflare R2 Storage
region:
auto: R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
endpoint: {}
quirks:
force_path_style: true
use_multipart_etag: false # multipart ETags are random

10
backend/s3/provider/Cubbit.yaml Normal file
View File

@@ -0,0 +1,10 @@
name: Cubbit
description: Cubbit DS3 Object Storage
region:
eu-west-1: Europe West
endpoint:
s3.cubbit.eu: Cubbit DS3 Object Storage endpoint
acl: {}
bucket_acl: true
quirks:
use_multipart_etag: false

20
backend/s3/provider/DigitalOcean.yaml Normal file
View File

@@ -0,0 +1,20 @@
name: DigitalOcean
description: DigitalOcean Spaces
region: {}
endpoint:
syd1.digitaloceanspaces.com: DigitalOcean Spaces Sydney 1
sfo3.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 3
sfo2.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 2
fra1.digitaloceanspaces.com: DigitalOcean Spaces Frankfurt 1
nyc3.digitaloceanspaces.com: DigitalOcean Spaces New York 3
ams3.digitaloceanspaces.com: DigitalOcean Spaces Amsterdam 3
sgp1.digitaloceanspaces.com: DigitalOcean Spaces Singapore 1
lon1.digitaloceanspaces.com: DigitalOcean Spaces London 1
tor1.digitaloceanspaces.com: DigitalOcean Spaces Toronto 1
blr1.digitaloceanspaces.com: DigitalOcean Spaces Bangalore 1
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
list_url_encode: false
use_already_exists: false

11
backend/s3/provider/Dreamhost.yaml Normal file
View File

@@ -0,0 +1,11 @@
name: Dreamhost
description: Dreamhost DreamObjects
region: {}
endpoint:
objects-us-east-1.dream.io: Dream Objects endpoint
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
list_url_encode: false
use_already_exists: false

9
backend/s3/provider/Exaba.yaml Normal file
View File

@@ -0,0 +1,9 @@
name: Exaba
description: Exaba Object Storage
region: {}
endpoint: {}
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
force_path_style: true

21
backend/s3/provider/FileLu.yaml Normal file
View File

@@ -0,0 +1,21 @@
name: FileLu
description: FileLu S5 (S3-Compatible Object Storage)
region:
global: Global
us-east: North America (US-East)
eu-central: Europe (EU-Central)
ap-southeast: Asia Pacific (AP-Southeast)
me-central: Middle East (ME-Central)
endpoint:
s5lu.com: Global FileLu S5 endpoint
us.s5lu.com: North America (US-East) region endpoint
eu.s5lu.com: Europe (EU-Central) region endpoint
ap.s5lu.com: Asia Pacific (AP-Southeast) region endpoint
me.s5lu.com: Middle East (ME-Central) region endpoint
acl: {}
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_multipart_etag: false

6
backend/s3/provider/FlashBlade.yaml Normal file
View File

@@ -0,0 +1,6 @@
name: FlashBlade
description: Pure Storage FlashBlade Object Storage
endpoint: {}
quirks:
might_gzip: false # never auto-gzip
force_path_style: true # supports vhost but defaults to path-style

20
backend/s3/provider/GCS.yaml Normal file
View File

@@ -0,0 +1,20 @@
name: GCS
description: Google Cloud Storage
region: {}
endpoint:
https://storage.googleapis.com: Google Cloud Storage endpoint
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
# Google break request Signature by mutating accept-encoding HTTP header
# https://github.com/rclone/rclone/issues/6670
use_accept_encoding_gzip: false
sign_accept_encoding: false
use_already_exists: true # returns BucketNameUnavailable instead of BucketAlreadyExists but good enough!
# GCS doesn't like the x-id URL parameter the SDKv2 inserts
use_x_id: false
# GCS S3 doesn't support multi-part server side copy:
# See: https://issuetracker.google.com/issues/323465186
# So make cutoff very large which it does seem to support
copy_cutoff: 9223372036854775807

15
backend/s3/provider/Hetzner.yaml Normal file
View File

@@ -0,0 +1,15 @@
name: Hetzner
description: Hetzner Object Storage
region:
hel1: Helsinki
fsn1: Falkenstein
nbg1: Nuremberg
endpoint:
hel1.your-objectstorage.com: Helsinki
fsn1.your-objectstorage.com: Falkenstein
nbg1.your-objectstorage.com: Nuremberg
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
use_already_exists: false

41
backend/s3/provider/HuaweiOBS.yaml Normal file
View File

@@ -0,0 +1,41 @@
name: HuaweiOBS
description: Huawei Object Storage Service
region:
af-south-1: AF-Johannesburg
ap-southeast-2: AP-Bangkok
ap-southeast-3: AP-Singapore
cn-east-3: CN East-Shanghai1
cn-east-2: CN East-Shanghai2
cn-north-1: CN North-Beijing1
cn-north-4: CN North-Beijing4
cn-south-1: CN South-Guangzhou
ap-southeast-1: CN-Hong Kong
sa-argentina-1: LA-Buenos Aires1
sa-peru-1: LA-Lima1
na-mexico-1: LA-Mexico City1
sa-chile-1: LA-Santiago2
sa-brazil-1: LA-Sao Paulo1
ru-northwest-2: RU-Moscow2
endpoint:
obs.af-south-1.myhuaweicloud.com: AF-Johannesburg
obs.ap-southeast-2.myhuaweicloud.com: AP-Bangkok
obs.ap-southeast-3.myhuaweicloud.com: AP-Singapore
obs.cn-east-3.myhuaweicloud.com: CN East-Shanghai1
obs.cn-east-2.myhuaweicloud.com: CN East-Shanghai2
obs.cn-north-1.myhuaweicloud.com: CN North-Beijing1
obs.cn-north-4.myhuaweicloud.com: CN North-Beijing4
obs.cn-south-1.myhuaweicloud.com: CN South-Guangzhou
obs.ap-southeast-1.myhuaweicloud.com: CN-Hong Kong
obs.sa-argentina-1.myhuaweicloud.com: LA-Buenos Aires1
obs.sa-peru-1.myhuaweicloud.com: LA-Lima1
obs.na-mexico-1.myhuaweicloud.com: LA-Mexico City1
obs.sa-chile-1.myhuaweicloud.com: LA-Santiago2
obs.sa-brazil-1.myhuaweicloud.com: LA-Sao Paulo1
obs.ru-northwest-2.myhuaweicloud.com: RU-Moscow2
acl: {}
bucket_acl: true
quirks:
# Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever.
list_url_encode: false
list_version: 1
use_already_exists: false

126
backend/s3/provider/IBMCOS.yaml Normal file
View File

@@ -0,0 +1,126 @@
name: IBMCOS
description: IBM COS S3
region: {}
endpoint:
s3.us.cloud-object-storage.appdomain.cloud: US Cross Region Endpoint
s3.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Endpoint
s3.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Endpoint
s3.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Endpoint
s3.private.us.cloud-object-storage.appdomain.cloud: US Cross Region Private Endpoint
s3.private.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Private Endpoint
s3.private.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Private Endpoint
s3.private.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Private Endpoint
s3.us-east.cloud-object-storage.appdomain.cloud: US Region East Endpoint
s3.private.us-east.cloud-object-storage.appdomain.cloud: US Region East Private Endpoint
s3.us-south.cloud-object-storage.appdomain.cloud: US Region South Endpoint
s3.private.us-south.cloud-object-storage.appdomain.cloud: US Region South Private Endpoint
s3.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Endpoint
s3.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Endpoint
s3.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Endpoint
s3.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Endpoint
s3.private.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Private Endpoint
s3.private.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Private Endpoint
s3.private.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Private Endpoint
s3.private.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Private Endpoint
s3.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Endpoint
s3.private.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Private Endpoint
s3.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Endpoint
s3.private.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Private Endpoint
s3.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Endpoint
s3.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Endpoint
s3.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Endpoint
s3.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Endpoint
s3.private.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Private Endpoint
s3.private.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Private Endpoint
s3.private.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Private Endpoint
s3.private.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Private Endpoint
s3.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Endpoint
s3.private.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Private Endpoint
s3.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Endpoint
s3.private.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Private Endpoint
s3.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Endpoint
s3.private.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Private Endpoint
s3.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Endpoint
s3.private.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Private Endpoint
s3.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Endpoint
s3.private.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Private Endpoint
s3.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Endpoint
s3.private.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Private Endpoint
s3.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Endpoint
s3.private.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Private Endpoint
s3.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Endpoint
s3.private.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Private Endpoint
s3.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Endpoint
s3.private.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Private Endpoint
s3.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Endpoint
s3.private.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Private Endpoint
s3.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Endpoint
s3.private.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Private Endpoint
s3.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Endpoint
s3.private.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Private Endpoint
s3.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Endpoint
s3.private.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Private Endpoint
s3.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Endpoint
s3.private.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Private Endpoint
s3.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Endpoint
s3.private.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Private Endpoint
location_constraint:
us-standard: US Cross Region Standard
us-vault: US Cross Region Vault
us-cold: US Cross Region Cold
us-flex: US Cross Region Flex
us-east-standard: US East Region Standard
us-east-vault: US East Region Vault
us-east-cold: US East Region Cold
us-east-flex: US East Region Flex
us-south-standard: US South Region Standard
us-south-vault: US South Region Vault
us-south-cold: US South Region Cold
us-south-flex: US South Region Flex
eu-standard: EU Cross Region Standard
eu-vault: EU Cross Region Vault
eu-cold: EU Cross Region Cold
eu-flex: EU Cross Region Flex
eu-gb-standard: Great Britain Standard
eu-gb-vault: Great Britain Vault
eu-gb-cold: Great Britain Cold
eu-gb-flex: Great Britain Flex
ap-standard: APAC Standard
ap-vault: APAC Vault
ap-cold: APAC Cold
ap-flex: APAC Flex
mel01-standard: Melbourne Standard
mel01-vault: Melbourne Vault
mel01-cold: Melbourne Cold
mel01-flex: Melbourne Flex
tor01-standard: Toronto Standard
tor01-vault: Toronto Vault
tor01-cold: Toronto Cold
tor01-flex: Toronto Flex
acl:
private: |-
Owner gets FULL_CONTROL.
No one else has access rights (default).
This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.
public-read-write: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ and WRITE access.
This acl is available on IBM Cloud (Infra), On-Premise IBM COS.
authenticated-read: |-
Owner gets FULL_CONTROL.
The AuthenticatedUsers group gets READ access.
Not supported on Buckets.
This acl is available on IBM Cloud (Infra) and On-Premise IBM COS.
ibm_api_key: true
ibm_resource_instance_id: true
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_multipart_etag: false
use_already_exists: false # returns BucketAlreadyExists

7
backend/s3/provider/IDrive.yaml Normal file
View File

@@ -0,0 +1,7 @@
name: IDrive
description: IDrive e2
acl: {}
bucket_acl: true
quirks:
force_path_style: true
use_already_exists: false

17
backend/s3/provider/IONOS.yaml Normal file
View File

@@ -0,0 +1,17 @@
name: IONOS
description: IONOS Cloud
region:
de: Frankfurt, Germany
eu-central-2: Berlin, Germany
eu-south-2: Logrono, Spain
endpoint:
s3-eu-central-1.ionoscloud.com: Frankfurt, Germany
s3-eu-central-2.ionoscloud.com: Berlin, Germany
s3-eu-south-2.ionoscloud.com: Logrono, Spain
acl: {}
bucket_acl: true
quirks:
# listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2
force_path_style: true
list_url_encode: false
use_already_exists: false

10
backend/s3/provider/Intercolo.yaml Normal file
View File

@@ -0,0 +1,10 @@
name: Intercolo
description: Intercolo Object Storage
region:
de-fra: Frankfurt, Germany
endpoint:
de-fra.i3storage.com: Frankfurt, Germany
acl: {}
bucket_acl: true
quirks:
use_unsigned_payload: false # has trailer support

11
backend/s3/provider/Leviia.yaml Normal file
View File

@@ -0,0 +1,11 @@
name: Leviia
description: Leviia Object Storage
region: {}
endpoint:
s3.leviia.com: |-
The default endpoint
Leviia
acl: {}
bucket_acl: true
quirks:
use_already_exists: false

15
backend/s3/provider/Liara.yaml Normal file
View File

@@ -0,0 +1,15 @@
name: Liara
description: Liara Object Storage
endpoint:
storage.iran.liara.space: |-
The default endpoint
Iran
acl: {}
storage_class:
STANDARD: Standard storage class
bucket_acl: true
quirks:
force_path_style: true
list_url_encode: false
use_multipart_etag: false # mulitpart ETags differ from AWS
use_already_exists: false

26
backend/s3/provider/Linode.yaml Normal file
View File

@@ -0,0 +1,26 @@
name: Linode
description: Linode Object Storage
endpoint:
nl-ams-1.linodeobjects.com: Amsterdam (Netherlands), nl-ams-1
us-southeast-1.linodeobjects.com: Atlanta, GA (USA), us-southeast-1
in-maa-1.linodeobjects.com: Chennai (India), in-maa-1
us-ord-1.linodeobjects.com: Chicago, IL (USA), us-ord-1
eu-central-1.linodeobjects.com: Frankfurt (Germany), eu-central-1
id-cgk-1.linodeobjects.com: Jakarta (Indonesia), id-cgk-1
gb-lon-1.linodeobjects.com: London 2 (Great Britain), gb-lon-1
us-lax-1.linodeobjects.com: Los Angeles, CA (USA), us-lax-1
es-mad-1.linodeobjects.com: Madrid (Spain), es-mad-1
au-mel-1.linodeobjects.com: Melbourne (Australia), au-mel-1
us-mia-1.linodeobjects.com: Miami, FL (USA), us-mia-1
it-mil-1.linodeobjects.com: Milan (Italy), it-mil-1
us-east-1.linodeobjects.com: Newark, NJ (USA), us-east-1
jp-osa-1.linodeobjects.com: Osaka (Japan), jp-osa-1
fr-par-1.linodeobjects.com: Paris (France), fr-par-1
br-gru-1.linodeobjects.com: São Paulo (Brazil), br-gru-1
us-sea-1.linodeobjects.com: Seattle, WA (USA), us-sea-1
ap-south-1.linodeobjects.com: Singapore, ap-south-1
sg-sin-1.linodeobjects.com: Singapore 2, sg-sin-1
se-sto-1.linodeobjects.com: Stockholm (Sweden), se-sto-1
us-iad-1.linodeobjects.com: Washington, DC, (USA), us-iad-1
acl: {}
bucket_acl: true

12
backend/s3/provider/LyveCloud.yaml Normal file
View File

@@ -0,0 +1,12 @@
name: LyveCloud
description: Seagate Lyve Cloud
region: {}
endpoint:
's3.us-west-1.{account_name}.lyve.seagate.com': US West 1 - California
's3.eu-west-1.{account_name}.lyve.seagate.com': EU West 1 - Ireland
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
use_multipart_etag: false # multipart ETags differ from AWS
use_already_exists: false

16
backend/s3/provider/Magalu.yaml Normal file
View File

@@ -0,0 +1,16 @@
name: Magalu
description: Magalu Object Storage
endpoint:
br-se1.magaluobjects.com: São Paulo, SP (BR), br-se1
br-ne1.magaluobjects.com: Fortaleza, CE (BR), br-ne1
acl: {}
storage_class:
STANDARD: Standard storage class
GLACIER_IR: Glacier Instant Retrieval storage class
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_multipart_etag: false
use_already_exists: false

15
backend/s3/provider/Mega.yaml Normal file
View File

@@ -0,0 +1,15 @@
name: Mega
description: MEGA S4 Object Storage
endpoint:
s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam)
s3.eu-central-2.s4.mega.io: Mega S4 eu-central-2 (Bettembourg)
s3.ca-central-1.s4.mega.io: Mega S4 ca-central-1 (Montreal)
s3.ca-west-1.s4.mega.io: Mega S4 ca-west-1 (Vancouver)
bucket_acl: true
quirks:
list_version: 2
force_path_style: true
list_url_encode: true
use_multipart_etag: false
use_already_exists: false
copy_cutoff: 9223372036854775807

18
backend/s3/provider/Minio.yaml Normal file
View File

@@ -0,0 +1,18 @@
name: Minio
description: Minio Object Storage
region: {}
endpoint: {}
location_constraint: {}
acl: {}
server_side_encryption:
'': None
AES256: AES256
aws:kms: aws:kms
bucket_acl: true
sse_customer_algorithm: true
sse_customer_key: true
sse_customer_key_base64: true
sse_customer_key_md5: true
sse_kms_key_id: true
quirks:
force_path_style: true

12
backend/s3/provider/Netease.yaml Normal file
View File

@@ -0,0 +1,12 @@
name: Netease
description: Netease Object Storage (NOS)
region: {}
endpoint: {}
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
list_version: 1
list_url_encode: false
use_multipart_etag: false
use_already_exists: false

36
backend/s3/provider/OVHcloud.yaml Normal file
View File

@@ -0,0 +1,36 @@
name: OVHcloud
description: OVHcloud Object Storage
region:
gra: Gravelines, France
rbx: Roubaix, France
sbg: Strasbourg, France
eu-west-par: Paris, France (3AZ)
de: Frankfurt, Germany
uk: London, United Kingdom
waw: Warsaw, Poland
bhs: Beauharnois, Canada
ca-east-tor: Toronto, Canada
sgp: Singapore
ap-southeast-syd: Sydney, Australia
ap-south-mum: Mumbai, India
us-east-va: Vint Hill, Virginia, USA
us-west-or: Hillsboro, Oregon, USA
rbx-archive: Roubaix, France (Cold Archive)
endpoint:
s3.gra.io.cloud.ovh.net: OVHcloud Gravelines, France
s3.rbx.io.cloud.ovh.net: OVHcloud Roubaix, France
s3.sbg.io.cloud.ovh.net: OVHcloud Strasbourg, France
s3.eu-west-par.io.cloud.ovh.net: OVHcloud Paris, France (3AZ)
s3.de.io.cloud.ovh.net: OVHcloud Frankfurt, Germany
s3.uk.io.cloud.ovh.net: OVHcloud London, United Kingdom
s3.waw.io.cloud.ovh.net: OVHcloud Warsaw, Poland
s3.bhs.io.cloud.ovh.net: OVHcloud Beauharnois, Canada
s3.ca-east-tor.io.cloud.ovh.net: OVHcloud Toronto, Canada
s3.sgp.io.cloud.ovh.net: OVHcloud Singapore
s3.ap-southeast-syd.io.cloud.ovh.net: OVHcloud Sydney, Australia
s3.ap-south-mum.io.cloud.ovh.net: OVHcloud Mumbai, India
s3.us-east-va.io.cloud.ovh.us: OVHcloud Vint Hill, Virginia, USA
s3.us-west-or.io.cloud.ovh.us: OVHcloud Hillsboro, Oregon, USA
s3.rbx-archive.io.cloud.ovh.net: OVHcloud Roubaix, France (Cold Archive)
acl: {}
bucket_acl: true

39
backend/s3/provider/Other.yaml Normal file
View File

@@ -0,0 +1,39 @@
name: Other
description: Any other S3 compatible provider
region:
'': |-
Use this if unsure.
Will use v4 signatures and an empty region.
other-v2-signature: |-
Use this only if v4 signatures don't work.
E.g. pre Jewel/v10 CEPH.
endpoint: {}
location_constraint: {}
acl:
private: |-
Owner gets FULL_CONTROL.
No one else has access rights (default).
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
public-read-write: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ and WRITE access.
Granting this on a bucket is generally not recommended.
authenticated-read: |-
Owner gets FULL_CONTROL.
The AuthenticatedUsers group gets READ access.
bucket-owner-read: |-
Object owner gets FULL_CONTROL.
Bucket owner gets READ access.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
bucket-owner-full-control: |-
Both the object owner and the bucket owner get FULL_CONTROL over the object.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_multipart_etag: false
use_already_exists: false

18
backend/s3/provider/Outscale.yaml Normal file
View File

@@ -0,0 +1,18 @@
name: Outscale
description: OUTSCALE Object Storage (OOS)
region:
eu-west-2: Paris, France
us-east-2: New Jersey, USA
us-west-1: California, USA
cloudgouv-eu-west-1: SecNumCloud, Paris, France
ap-northeast-1: Tokyo, Japan
endpoint:
oos.eu-west-2.outscale.com: Outscale EU West 2 (Paris)
oos.us-east-2.outscale.com: Outscale US east 2 (New Jersey)
oos.us-west-1.outscale.com: Outscale EU West 1 (California)
oos.cloudgouv-eu-west-1.outscale.com: Outscale SecNumCloud (Paris)
oos.ap-northeast-1.outscale.com: Outscale AP Northeast 1 (Japan)
acl: {}
bucket_acl: true
quirks:
force_path_style: true

19
backend/s3/provider/Petabox.yaml Normal file
View File

@@ -0,0 +1,19 @@
name: Petabox
description: Petabox Object Storage
region:
us-east-1: US East (N. Virginia)
eu-central-1: Europe (Frankfurt)
ap-southeast-1: Asia Pacific (Singapore)
me-south-1: Middle East (Bahrain)
sa-east-1: South America (São Paulo)
endpoint:
s3.petabox.io: US East (N. Virginia)
s3.us-east-1.petabox.io: US East (N. Virginia)
s3.eu-central-1.petabox.io: Europe (Frankfurt)
s3.ap-southeast-1.petabox.io: Asia Pacific (Singapore)
s3.me-south-1.petabox.io: Middle East (Bahrain)
s3.sa-east-1.petabox.io: South America (São Paulo)
acl: {}
bucket_acl: true
quirks:
use_already_exists: false

53
backend/s3/provider/Qiniu.yaml Normal file
View File

@@ -0,0 +1,53 @@
name: Qiniu
description: Qiniu Object Storage (Kodo)
region:
cn-east-1: |-
The default endpoint - a good choice if you are unsure.
East China Region 1.
Needs location constraint cn-east-1.
cn-east-2: |-
East China Region 2.
Needs location constraint cn-east-2.
cn-north-1: |-
North China Region 1.
Needs location constraint cn-north-1.
cn-south-1: |-
South China Region 1.
Needs location constraint cn-south-1.
us-north-1: |-
North America Region.
Needs location constraint us-north-1.
ap-southeast-1: |-
Southeast Asia Region 1.
Needs location constraint ap-southeast-1.
ap-northeast-1: |-
Northeast Asia Region 1.
Needs location constraint ap-northeast-1.
endpoint:
s3-cn-east-1.qiniucs.com: East China Endpoint 1
s3-cn-east-2.qiniucs.com: East China Endpoint 2
s3-cn-north-1.qiniucs.com: North China Endpoint 1
s3-cn-south-1.qiniucs.com: South China Endpoint 1
s3-us-north-1.qiniucs.com: North America Endpoint 1
s3-ap-southeast-1.qiniucs.com: Southeast Asia Endpoint 1
s3-ap-northeast-1.qiniucs.com: Northeast Asia Endpoint 1
location_constraint:
cn-east-1: East China Region 1
cn-east-2: East China Region 2
cn-north-1: North China Region 1
cn-south-1: South China Region 1
us-north-1: North America Region 1
ap-southeast-1: Southeast Asia Region 1
ap-northeast-1: Northeast Asia Region 1
acl: {}
storage_class:
STANDARD: Standard storage class
LINE: Infrequent access storage mode
GLACIER: Archive storage mode
DEEP_ARCHIVE: Deep archive storage mode
bucket_acl: true
quirks:
use_multipart_etag: false
list_url_encode: false
force_path_style: true
use_already_exists: false

15
backend/s3/provider/Rabata.yaml Normal file
View File

@@ -0,0 +1,15 @@
name: Rabata
description: Rabata Cloud Storage
region:
us-east-1: US East (N. Virginia)
eu-west-1: EU (Ireland)
eu-west-2: EU (London)
endpoint:
s3.us-east-1.rabata.io: US East (N. Virginia)
s3.eu-west-1.rabata.io: EU West (Ireland)
s3.eu-west-2.rabata.io: EU West (London)
location_constraint:
us-east-1: US East (N. Virginia)
eu-west-1: EU (Ireland)
eu-west-2: EU (London)
# server side copy not supported

67
backend/s3/provider/RackCorp.yaml Normal file
View File

@@ -0,0 +1,67 @@
name: RackCorp
description: RackCorp Object Storage
region:
global: Global CDN (All locations) Region
au: Australia (All states)
au-nsw: NSW (Australia) Region
au-qld: QLD (Australia) Region
au-vic: VIC (Australia) Region
au-wa: Perth (Australia) Region
ph: Manila (Philippines) Region
th: Bangkok (Thailand) Region
hk: HK (Hong Kong) Region
mn: Ulaanbaatar (Mongolia) Region
kg: Bishkek (Kyrgyzstan) Region
id: Jakarta (Indonesia) Region
jp: Tokyo (Japan) Region
sg: SG (Singapore) Region
de: Frankfurt (Germany) Region
us: USA (AnyCast) Region
us-east-1: New York (USA) Region
us-west-1: Freemont (USA) Region
nz: Auckland (New Zealand) Region
endpoint:
s3.rackcorp.com: Global (AnyCast) Endpoint
au.s3.rackcorp.com: Australia (Anycast) Endpoint
au-nsw.s3.rackcorp.com: Sydney (Australia) Endpoint
au-qld.s3.rackcorp.com: Brisbane (Australia) Endpoint
au-vic.s3.rackcorp.com: Melbourne (Australia) Endpoint
au-wa.s3.rackcorp.com: Perth (Australia) Endpoint
ph.s3.rackcorp.com: Manila (Philippines) Endpoint
th.s3.rackcorp.com: Bangkok (Thailand) Endpoint
hk.s3.rackcorp.com: HK (Hong Kong) Endpoint
mn.s3.rackcorp.com: Ulaanbaatar (Mongolia) Endpoint
kg.s3.rackcorp.com: Bishkek (Kyrgyzstan) Endpoint
id.s3.rackcorp.com: Jakarta (Indonesia) Endpoint
jp.s3.rackcorp.com: Tokyo (Japan) Endpoint
sg.s3.rackcorp.com: SG (Singapore) Endpoint
de.s3.rackcorp.com: Frankfurt (Germany) Endpoint
us.s3.rackcorp.com: USA (AnyCast) Endpoint
us-east-1.s3.rackcorp.com: New York (USA) Endpoint
us-west-1.s3.rackcorp.com: Freemont (USA) Endpoint
nz.s3.rackcorp.com: Auckland (New Zealand) Endpoint
location_constraint:
global: Global CDN Region
au: Australia (All locations)
au-nsw: NSW (Australia) Region
au-qld: QLD (Australia) Region
au-vic: VIC (Australia) Region
au-wa: Perth (Australia) Region
ph: Manila (Philippines) Region
th: Bangkok (Thailand) Region
hk: HK (Hong Kong) Region
mn: Ulaanbaatar (Mongolia) Region
kg: Bishkek (Kyrgyzstan) Region
id: Jakarta (Indonesia) Region
jp: Tokyo (Japan) Region
sg: SG (Singapore) Region
de: Frankfurt (Germany) Region
us: USA (AnyCast) Region
us-east-1: New York (USA) Region
us-west-1: Fremont (USA) Region
nz: Auckland (New Zealand) Region
acl: {}
bucket_acl: true
quirks:
use_multipart_etag: false
use_already_exists: false

11
backend/s3/provider/Rclone.yaml Normal file
View File

@@ -0,0 +1,11 @@
name: Rclone
description: Rclone S3 Server
endpoint: {}
quirks:
force_path_style: true
use_multipart_etag: false
use_already_exists: false
# rclone serve doesn't support multi-part server side copy:
# See: https://github.com/rclone/rclone/issues/7454
# So make cutoff very large which it does support
copy_cutoff: 9223372036854775807

28
backend/s3/provider/Scaleway.yaml Normal file
View File

@@ -0,0 +1,28 @@
name: Scaleway
description: Scaleway Object Storage
region:
nl-ams: Amsterdam, The Netherlands
fr-par: Paris, France
pl-waw: Warsaw, Poland
endpoint:
s3.nl-ams.scw.cloud: Amsterdam Endpoint
s3.fr-par.scw.cloud: Paris Endpoint
s3.pl-waw.scw.cloud: Warsaw Endpoint
acl: {}
storage_class:
'': Default.
STANDARD: |-
The Standard class for any upload.
Suitable for on-demand content like streaming or CDN.
Available in all regions.
GLACIER: |-
Archived storage.
Prices are lower, but it needs to be restored first to be accessed.
Available in FR-PAR and NL-AMS regions.
ONEZONE_IA: |-
One Zone - Infrequent Access.
A good choice for storing secondary backup copies or easily re-creatable data.
Available in the FR-PAR region only.
bucket_acl: true
quirks:
max_upload_parts: 1000

14
backend/s3/provider/SeaweedFS.yaml Normal file
View File

@@ -0,0 +1,14 @@
name: SeaweedFS
description: SeaweedFS S3
region: {}
endpoint:
localhost:8333: SeaweedFS S3 localhost
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_multipart_etag: false
use_already_exists: false

8
backend/s3/provider/Selectel.yaml Normal file
View File

@@ -0,0 +1,8 @@
name: Selectel
description: Selectel Object Storage
region:
ru-1: St. Petersburg
endpoint:
s3.ru-1.storage.selcloud.ru: Saint Petersburg
quirks:
list_url_encode: false

17
backend/s3/provider/Servercore.yaml Normal file
View File

@@ -0,0 +1,17 @@
name: Servercore
description: Servercore Object Storage
region:
ru-1: St. Petersburg
gis-1: Moscow
ru-7: Moscow
uz-2: Tashkent, Uzbekistan
kz-1: Almaty, Kazakhstan
endpoint:
s3.ru-1.storage.selcloud.ru: Saint Petersburg
s3.gis-1.storage.selcloud.ru: Moscow
s3.ru-7.storage.selcloud.ru: Moscow
s3.uz-2.srvstorage.uz: Tashkent, Uzbekistan
s3.kz-1.srvstorage.kz: Almaty, Kazakhstan
bucket_acl: true
quirks:
list_url_encode: false

5
backend/s3/provider/SpectraLogic.yaml Normal file
View File

@@ -0,0 +1,5 @@
name: SpectraLogic
description: Spectra Logic Black Pearl
endpoint: {}
quirks:
force_path_style: true # path-style required

14
backend/s3/provider/StackPath.yaml Normal file
View File

@@ -0,0 +1,14 @@
name: StackPath
description: StackPath Object Storage
region: {}
endpoint:
s3.us-east-2.stackpathstorage.com: US East Endpoint
s3.us-west-1.stackpathstorage.com: US West Endpoint
s3.eu-central-1.stackpathstorage.com: EU Endpoint
acl: {}
bucket_acl: true
quirks:
list_version: 1
force_path_style: true
list_url_encode: false
use_already_exists: false

11
backend/s3/provider/Storj.yaml Normal file
View File

@@ -0,0 +1,11 @@
name: Storj
description: Storj (S3 Compatible Gateway)
endpoint:
gateway.storjshare.io: Global Hosted Gateway
quirks:
use_already_exists: false # returns BucketAlreadyExists
# Storj doesn't support multi-part server side copy:
# https://github.com/storj/roadmap/issues/40
# So make cutoff very large which it does support
copy_cutoff: 9223372036854775807
min_chunk_size: 67108864

18
backend/s3/provider/Synology.yaml Normal file
View File

@@ -0,0 +1,18 @@
name: Synology
description: Synology C2 Object Storage
region:
eu-001: Europe Region 1
eu-002: Europe Region 2
us-001: US Region 1
us-002: US Region 2
tw-001: Asia (Taiwan)
endpoint:
eu-001.s3.synologyc2.net: EU Endpoint 1
eu-002.s3.synologyc2.net: EU Endpoint 2
us-001.s3.synologyc2.net: US Endpoint 1
us-002.s3.synologyc2.net: US Endpoint 2
tw-001.s3.synologyc2.net: TW Endpoint 1
location_constraint: {}
quirks:
use_multipart_etag: false
use_already_exists: false

52
backend/s3/provider/TencentCOS.yaml Normal file
View File

@@ -0,0 +1,52 @@
name: TencentCOS
description: Tencent Cloud Object Storage (COS)
endpoint:
cos.ap-beijing.myqcloud.com: Beijing Region
cos.ap-nanjing.myqcloud.com: Nanjing Region
cos.ap-shanghai.myqcloud.com: Shanghai Region
cos.ap-guangzhou.myqcloud.com: Guangzhou Region
cos.ap-chengdu.myqcloud.com: Chengdu Region
cos.ap-chongqing.myqcloud.com: Chongqing Region
cos.ap-hongkong.myqcloud.com: Hong Kong (China) Region
cos.ap-singapore.myqcloud.com: Singapore Region
cos.ap-mumbai.myqcloud.com: Mumbai Region
cos.ap-seoul.myqcloud.com: Seoul Region
cos.ap-bangkok.myqcloud.com: Bangkok Region
cos.ap-tokyo.myqcloud.com: Tokyo Region
cos.na-siliconvalley.myqcloud.com: Silicon Valley Region
cos.na-ashburn.myqcloud.com: Virginia Region
cos.na-toronto.myqcloud.com: Toronto Region
cos.eu-frankfurt.myqcloud.com: Frankfurt Region
cos.eu-moscow.myqcloud.com: Moscow Region
cos.accelerate.myqcloud.com: Use Tencent COS Accelerate Endpoint
acl:
default: |-
Owner gets Full_CONTROL.
No one else has access rights (default).
public-read: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ access.
public-read-write: |-
Owner gets FULL_CONTROL.
The AllUsers group gets READ and WRITE access.
Granting this on a bucket is generally not recommended.
authenticated-read: |-
Owner gets FULL_CONTROL.
The AuthenticatedUsers group gets READ access.
bucket-owner-read: |-
Object owner gets FULL_CONTROL.
Bucket owner gets READ access.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
bucket-owner-full-control: |-
Both the object owner and the bucket owner get FULL_CONTROL over the object.
If you specify this canned ACL when creating a bucket, Amazon S3 ignores it.
storage_class:
'': Default
STANDARD: Standard storage class
ARCHIVE: Archive storage mode
STANDARD_IA: Infrequent access storage mode
bucket_acl: true
quirks:
list_version: 1
use_multipart_etag: false
use_already_exists: false

21
backend/s3/provider/Wasabi.yaml Normal file
View File

@@ -0,0 +1,21 @@
name: Wasabi
description: Wasabi Object Storage
region: {}
endpoint:
s3.wasabisys.com: Wasabi US East 1 (N. Virginia)
s3.us-east-2.wasabisys.com: Wasabi US East 2 (N. Virginia)
s3.us-central-1.wasabisys.com: Wasabi US Central 1 (Texas)
s3.us-west-1.wasabisys.com: Wasabi US West 1 (Oregon)
s3.ca-central-1.wasabisys.com: Wasabi CA Central 1 (Toronto)
s3.eu-central-1.wasabisys.com: Wasabi EU Central 1 (Amsterdam)
s3.eu-central-2.wasabisys.com: Wasabi EU Central 2 (Frankfurt)
s3.eu-west-1.wasabisys.com: Wasabi EU West 1 (London)
s3.eu-west-2.wasabisys.com: Wasabi EU West 2 (Paris)
s3.eu-south-1.wasabisys.com: Wasabi EU South 1 (Milan)
s3.ap-northeast-1.wasabisys.com: Wasabi AP Northeast 1 (Tokyo) endpoint
s3.ap-northeast-2.wasabisys.com: Wasabi AP Northeast 2 (Osaka) endpoint
s3.ap-southeast-1.wasabisys.com: Wasabi AP Southeast 1 (Singapore)
s3.ap-southeast-2.wasabisys.com: Wasabi AP Southeast 2 (Sydney)
location_constraint: {}
acl: {}
bucket_acl: true

14
backend/s3/provider/Zata.yaml Normal file
View File

@@ -0,0 +1,14 @@
name: Zata
description: Zata (S3 compatible Gateway)
region:
us-east-1: Indore, Madhya Pradesh, India
endpoint:
idr01.zata.ai: South Asia Endpoint
location_constraint: {}
acl: {}
bucket_acl: true
quirks:
use_multipart_etag: false
might_gzip: false
use_unsigned_payload: false
use_already_exists: false

236
backend/s3/providers.go Normal file
View File

@@ -0,0 +1,236 @@
package s3
import (
"embed"
stdfs "io/fs"
"os"
"sort"
"strings"
"github.com/rclone/rclone/fs"
orderedmap "github.com/wk8/go-ordered-map/v2"
"gopkg.in/yaml.v3"
)
// YamlMap is converted to YAML in the correct order
type YamlMap = *orderedmap.OrderedMap[string, string]
// NewYamlMap creates a new ordered map
var NewYamlMap = orderedmap.New[string, string]
// Quirks defines all the S3 provider quirks
type Quirks struct {
ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2
ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style
ListURLEncode *bool `yaml:"list_url_encode,omitempty"`
UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"`
UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"`
UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"`
MightGzip *bool `yaml:"might_gzip,omitempty"`
UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"`
UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"`
UseXID *bool `yaml:"use_x_id,omitempty"`
SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"`
CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"`
MaxUploadParts *int `yaml:"max_upload_parts,omitempty"`
MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"`
}
// Provider defines the configurable data in each provider.yaml
type Provider struct {
Name string `yaml:"name,omitempty"`
Description string `yaml:"description,omitempty"`
Region YamlMap `yaml:"region,omitempty"`
Endpoint YamlMap `yaml:"endpoint,omitempty"`
LocationConstraint YamlMap `yaml:"location_constraint,omitempty"`
ACL YamlMap `yaml:"acl,omitempty"`
StorageClass YamlMap `yaml:"storage_class,omitempty"`
ServerSideEncryption YamlMap `yaml:"server_side_encryption,omitempty"`
// other
IBMApiKey bool `yaml:"ibm_api_key,omitempty"`
IBMResourceInstanceID bool `yaml:"ibm_resource_instance_id,omitempty"`
// advanced
BucketACL bool `yaml:"bucket_acl,omitempty"`
DirectoryBucket bool `yaml:"directory_bucket,omitempty"`
LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"`
RequesterPays bool `yaml:"requester_pays,omitempty"`
SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"`
SSECustomerKey bool `yaml:"sse_customer_key,omitempty"`
SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"`
SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"`
SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"`
STSEndpoint bool `yaml:"sts_endpoint,omitempty"`
UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"`
Quirks Quirks `yaml:"quirks,omitempty"`
}
//go:embed provider/*.yaml
var providerFS embed.FS
// addProvidersToInfo adds provider information to the fs.RegInfo
func addProvidersToInfo(info *fs.RegInfo) *fs.RegInfo {
providerMap := loadProviders()
providerList := constructProviders(info.Options, providerMap)
info.Description += strings.TrimSuffix(providerList, ", ")
return info
}
// loadProvider loads a single provider
//
// It returns nil if it could not be found except if "Other" which is a fatal error.
func loadProvider(name string) *Provider {
data, err := stdfs.ReadFile(providerFS, "provider/"+name+".yaml")
if err != nil {
if os.IsNotExist(err) && name != "Other" {
return nil
}
fs.Fatalf(nil, "internal error: failed to load provider %q: %v", name, err)
}
var p Provider
err = yaml.Unmarshal(data, &p)
if err != nil {
fs.Fatalf(nil, "internal error: failed to unmarshal provider %q: %v", name, err)
}
return &p
}
// loadProviders loads provider definitions from embedded YAML files
func loadProviders() map[string]*Provider {
providers, err := stdfs.ReadDir(providerFS, "provider")
if err != nil {
fs.Fatalf(nil, "internal error: failed to read embedded providers: %v", err)
}
providerMap := make(map[string]*Provider, len(providers))
for _, provider := range providers {
name, _ := strings.CutSuffix(provider.Name(), ".yaml")
p := loadProvider(name)
providerMap[p.Name] = p
}
return providerMap
}
// constructProviders populates fs.Options with provider-specific examples and information
func constructProviders(options fs.Options, providerMap map[string]*Provider) string {
// Defaults for map options set to {}
defaults := providerMap["Other"]
// sort providers: AWS first, Other last, rest alphabetically
providers := make([]*Provider, 0, len(providerMap))
for _, p := range providerMap {
providers = append(providers, p)
}
sort.Slice(providers, func(i, j int) bool {
if providers[i].Name == "AWS" {
return true
}
if providers[j].Name == "AWS" {
return false
}
if providers[i].Name == "Other" {
return false
}
if providers[j].Name == "Other" {
return true
}
return strings.ToLower(providers[i].Name) < strings.ToLower(providers[j].Name)
})
addProvider := func(sp *string, name string) {
if *sp != "" {
*sp += ","
}
*sp += name
}
addBool := func(opt *fs.Option, p *Provider, flag bool) {
if flag {
addProvider(&opt.Provider, p.Name)
}
}
addExample := func(opt *fs.Option, p *Provider, examples, defaultExamples YamlMap) {
if examples == nil {
return
}
if examples.Len() == 0 {
examples = defaultExamples
}
addProvider(&opt.Provider, p.Name)
OUTER:
for pair := examples.Oldest(); pair != nil; pair = pair.Next() {
// Find an existing example to add to if possible
for i, example := range opt.Examples {
if example.Value == pair.Key && example.Help == pair.Value {
addProvider(&opt.Examples[i].Provider, p.Name)
continue OUTER
}
}
// Otherwise add a new one
opt.Examples = append(opt.Examples, fs.OptionExample{
Value: pair.Key,
Help: pair.Value,
Provider: p.Name,
})
}
}
var providerList string
for _, p := range providers {
for i := range options {
opt := &options[i]
switch opt.Name {
case "provider":
opt.Examples = append(opt.Examples, fs.OptionExample{
Value: p.Name,
Help: p.Description,
})
providerList += p.Name + ", "
case "region":
addExample(opt, p, p.Region, defaults.Region)
case "endpoint":
addExample(opt, p, p.Endpoint, defaults.Endpoint)
case "location_constraint":
addExample(opt, p, p.LocationConstraint, defaults.LocationConstraint)
case "acl":
addExample(opt, p, p.ACL, defaults.ACL)
case "storage_class":
addExample(opt, p, p.StorageClass, defaults.StorageClass)
case "server_side_encryption":
addExample(opt, p, p.ServerSideEncryption, defaults.ServerSideEncryption)
case "bucket_acl":
addBool(opt, p, p.BucketACL)
case "requester_pays":
addBool(opt, p, p.RequesterPays)
case "sse_customer_algorithm":
addBool(opt, p, p.SSECustomerAlgorithm)
case "sse_kms_key_id":
addBool(opt, p, p.SSEKmsKeyID)
case "sse_customer_key":
addBool(opt, p, p.SSECustomerKey)
case "sse_customer_key_base64":
addBool(opt, p, p.SSECustomerKeyBase64)
case "sse_customer_key_md5":
addBool(opt, p, p.SSECustomerKeyMd5)
case "directory_bucket":
addBool(opt, p, p.DirectoryBucket)
case "ibm_api_key":
addBool(opt, p, p.IBMApiKey)
case "ibm_resource_instance_id":
addBool(opt, p, p.IBMResourceInstanceID)
case "leave_parts_on_error":
addBool(opt, p, p.LeavePartsOnError)
case "sts_endpoint":
addBool(opt, p, p.STSEndpoint)
case "use_accelerate_endpoint":
addBool(opt, p, p.UseAccelerateEndpoint)
}
}
}
return strings.TrimSuffix(providerList, ", ")
}

2778
backend/s3/s3.go
View File

File diff suppressed because it is too large Load Diff

4
backend/s3/s3_test.go
View File

@@ -62,14 +62,14 @@ func TestAWSDualStackOption(t *testing.T) {
// test enabled
ctx, opt, client := SetupS3Test(t)
opt.UseDualStack = true
s3Conn, err := s3Connection(ctx, opt, client)
s3Conn, _, err := s3Connection(ctx, opt, client)
require.NoError(t, err)
assert.Equal(t, aws.DualStackEndpointStateEnabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
}
{
// test default case
ctx, opt, client := SetupS3Test(t)
s3Conn, err := s3Connection(ctx, opt, client)
s3Conn, _, err := s3Connection(ctx, opt, client)
require.NoError(t, err)
assert.Equal(t, aws.DualStackEndpointStateDisabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint)
}