diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 9e3a565c5..4a75f8975 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -621,44 +621,7 @@ in the web browser and the links (internal and external) all work. ## Adding a new s3 provider -It is quite easy to add a new S3 provider to rclone. - -You'll need to modify the following files - -- `backend/s3/s3.go` - - Add the provider to `providerOption` at the top of the file - - Add endpoints and other config for your provider gated on the provider in `fs.RegInfo`. - - Exclude your provider from generic config questions (eg `region` and `endpoint`). - - Add the provider to the `setQuirks` function - see the documentation there. -- `docs/content/s3.md` - - Add the provider at the top of the page. - - Add a section about the provider linked from there. - - Make sure this is in alphabetical order in the `Providers` section. - - Add a transcript of a trial `rclone config` session - - Edit the transcript to remove things which might change in subsequent versions - - **Do not** alter or add to the autogenerated parts of `s3.md` - - **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3` -- `README.md` - this is the home page in github - - Add the provider and a link to the section you wrote in `docs/contents/s3.md` -- `docs/content/_index.md` - this is the home page of rclone.org - - Add the provider and a link to the section you wrote in `docs/contents/s3.md` - -When adding the provider, endpoints, quirks, docs etc keep them in -alphabetical order by `Provider` name, but with `AWS` first and -`Other` last. - -Once you've written the docs, run `make serve` and check they look OK -in the web browser and the links (internal and external) all work. - -Once you've written the code, test `rclone config` works to your -satisfaction, and check the integration tests work `go test -v -remote -NewS3Provider:`. You may need to adjust the quirks to get them to -pass. Some providers just can't pass the tests with control characters -in the names so if these fail and the provider doesn't support -`urlEncodeListings` in the quirks then ignore them. Note that the -`SetTier` test may also fail on non AWS providers. - -For an example of adding an s3 provider see [eb3082a1](https://github.com/rclone/rclone/commit/eb3082a1ebdb76d5625f14cedec3f5154a5e7b10). +[Please see the guide in the S3 backend directory](backend/s3/README.md). ## Writing a plugin diff --git a/backend/s3/README.md b/backend/s3/README.md new file mode 100644 index 000000000..d4ea43d9a --- /dev/null +++ b/backend/s3/README.md @@ -0,0 +1,194 @@ +## Adding a new s3 provider + +It is quite easy to add a new S3 provider to rclone. + +You'll then need to do add the following (optional tags are in [] and +do not get displayed in rclone config if empty): + +The process is as follows: Create yaml -> add docs -> run tests -> +adjust yaml until tests pass. + +All tags can be found in `backend/s3/providers.go` Provider Struct. +Looking through a few of the yaml files as examples should make things +clear. `AWS.yaml` as the most config. pasting. + +### YAML + +In `backend/s3/provider/YourProvider.yaml` + +- name +- description + - More like the full name often "YourProvider + Object Storage" +- [Region] + - Any regions your provider supports or the defaults (use `region: {}` for this) + - Example from AWS.yaml: + ```yaml + region: + us-east-1: |- + The default endpoint - a good choice if you are unsure. + US Region, Northern Virginia, or Pacific Northwest. + Leave location constraint empty. + ``` + - The defaults (as seen in Rclone.yaml): + ```yaml + region: + "": |- + Use this if unsure. + Will use v4 signatures and an empty region. + other-v2-signature: |- + Use this only if v4 signatures don't work. + E.g. pre Jewel/v10 CEPH. + ``` +- [Endpoint] + - Any endpoints your provider supports + - Example from Mega.yaml + ```yaml + endpoint: + s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam) + ``` +- [Location Constraint] + - The Location Constraint of your remote, often same as region. + - Example from AWS.yaml + ```yaml + location_constraint: + "": Empty for US Region, Northern Virginia, or Pacific Northwest + us-east-2: US East (Ohio) Region + ``` +- [ACL] + - Identical across *most* providers. Select the default with `acl: {}` + - Example from AWS.yaml + ```yaml + acl: + private: |- + Owner gets FULL_CONTROL. + No one else has access rights (default). + public-read: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ access. + public-read-write: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ and WRITE access. + Granting this on a bucket is generally not recommended. + authenticated-read: |- + Owner gets FULL_CONTROL. + The AuthenticatedUsers group gets READ access. + bucket-owner-read: |- + Object owner gets FULL_CONTROL. + Bucket owner gets READ access. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. + bucket-owner-full-control: |- + Both the object owner and the bucket owner get FULL_CONTROL over the object. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. + ``` +- [Storage Class] + - Identical across *most* providers. + - Defaults from AWS.yaml + ```yaml + storage_class: + "": Default + STANDARD: Standard storage class + REDUCED_REDUNDANCY: Reduced redundancy storage class + STANDARD_IA: Standard Infrequent Access storage class + ONEZONE_IA: One Zone Infrequent Access storage class + GLACIER: Glacier Flexible Retrieval storage class + DEEP_ARCHIVE: Glacier Deep Archive storage class + INTELLIGENT_TIERING: Intelligent-Tiering storage class + GLACIER_IR: Glacier Instant Retrieval storage class + ``` +- [Server Side Encryption] + - Not common, identical across *most* providers. + - Defaults from AWS.yaml + ```yaml + server_side_encryption: + "": None + AES256: AES256 + aws:kms: aws:kms + ``` +- [Advanced Options] + - All advanced options are Boolean - if true the configurator asks about that value, if not it doesn't: + ```go + BucketACL bool `yaml:"bucket_acl,omitempty"` + DirectoryBucket bool `yaml:"directory_bucket,omitempty"` + LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"` + RequesterPays bool `yaml:"requester_pays,omitempty"` + SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"` + SSECustomerKey bool `yaml:"sse_customer_key,omitempty"` + SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"` + SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"` + SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"` + STSEndpoint bool `yaml:"sts_endpoint,omitempty"` + UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"` + ``` + - Example from AWS.yaml: + ```yaml + bucket_acl: true + directory_bucket: true + leave_parts_on_error: true + requester_pays: true + sse_customer_algorithm: true + sse_customer_key: true + sse_customer_key_base64: true + sse_customer_key_md5: true + sse_kms_key_id: true + sts_endpoint: true + use_accelerate_endpoint: true + ``` +- Quirks + - Quirks are discovered through documentation and running the tests as seen below. + - Most quirks are *bool as to have 3 values, `true`, `false` and `dont care`. + ```go + type Quirks struct { + ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2 + ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style + ListURLEncode *bool `yaml:"list_url_encode,omitempty"` + UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"` + UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"` + UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"` + MightGzip *bool `yaml:"might_gzip,omitempty"` + UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"` + UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"` + UseXID *bool `yaml:"use_x_id,omitempty"` + SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"` + CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"` + MaxUploadParts *int `yaml:"max_upload_parts,omitempty"` + MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"` + } + ``` + - Example from AWS.yaml + ```yaml + quirks: + might_gzip: false # Never auto gzips objects + use_unsigned_payload: false # AWS has trailer support + ``` + +Note that if you omit a section, eg `region` then the user won't be +asked that question, and if you add an empty section e.g. `region: {}` +then the defaults from the `Other.yaml` will be used. + +### DOCS + +- `docs/content/s3.md` + - Add the provider at the top of the page. + - Add a section about the provider linked from there. + - Make sure this is in alphabetical order in the `Providers` section. + - Add a transcript of a trial `rclone config` session + - Edit the transcript to remove things which might change in subsequent versions + - **Do not** alter or add to the autogenerated parts of `s3.md` + - Rule of thumb: don't edit anything not mentioned above. + - **Do not** run `make backenddocs` or `bin/make_backend_docs.py s3` + - This will make autogenerated changes! +- `README.md` - this is the home page in github + - Add the provider and a link to the section you wrote in `docs/contents/s3.md` +- `docs/content/_index.md` - this is the home page of rclone.org + - Add the provider and a link to the section you wrote in `docs/contents/s3.md` +- Once you've written the docs, run `make serve` and check they look OK + in the web browser and the links (internal and external) all work. + +### TESTS + +Once you've written the code, test `rclone config` works to your +satisfaction and looks correct, and check the integration tests work +`go test -v -remote NewS3Provider:`. You may need to adjust the quirks +to get them to pass. Some providers just can't pass the tests with +control characters in the names so if these fail and the provider +doesn't support `urlEncodeListings` in the quirks then ignore them. diff --git a/backend/s3/provider/AWS.yaml b/backend/s3/provider/AWS.yaml new file mode 100644 index 000000000..39c12ae08 --- /dev/null +++ b/backend/s3/provider/AWS.yaml @@ -0,0 +1,139 @@ +name: AWS +description: Amazon Web Services (AWS) S3 +region: + us-east-1: |- + The default endpoint - a good choice if you are unsure. + US Region, Northern Virginia, or Pacific Northwest. + Leave location constraint empty. + us-east-2: |- + US East (Ohio) Region. + Needs location constraint us-east-2. + us-west-1: |- + US West (Northern California) Region. + Needs location constraint us-west-1. + us-west-2: |- + US West (Oregon) Region. + Needs location constraint us-west-2. + ca-central-1: |- + Canada (Central) Region. + Needs location constraint ca-central-1. + eu-west-1: |- + EU (Ireland) Region. + Needs location constraint EU or eu-west-1. + eu-west-2: |- + EU (London) Region. + Needs location constraint eu-west-2. + eu-west-3: |- + EU (Paris) Region. + Needs location constraint eu-west-3. + eu-north-1: |- + EU (Stockholm) Region. + Needs location constraint eu-north-1. + eu-south-1: |- + EU (Milan) Region. + Needs location constraint eu-south-1. + eu-central-1: |- + EU (Frankfurt) Region. + Needs location constraint eu-central-1. + ap-southeast-1: |- + Asia Pacific (Singapore) Region. + Needs location constraint ap-southeast-1. + ap-southeast-2: |- + Asia Pacific (Sydney) Region. + Needs location constraint ap-southeast-2. + ap-northeast-1: |- + Asia Pacific (Tokyo) Region. + Needs location constraint ap-northeast-1. + ap-northeast-2: |- + Asia Pacific (Seoul). + Needs location constraint ap-northeast-2. + ap-northeast-3: |- + Asia Pacific (Osaka-Local). + Needs location constraint ap-northeast-3. + ap-south-1: |- + Asia Pacific (Mumbai). + Needs location constraint ap-south-1. + ap-east-1: |- + Asia Pacific (Hong Kong) Region. + Needs location constraint ap-east-1. + sa-east-1: |- + South America (Sao Paulo) Region. + Needs location constraint sa-east-1. + il-central-1: |- + Israel (Tel Aviv) Region. + Needs location constraint il-central-1. + me-south-1: |- + Middle East (Bahrain) Region. + Needs location constraint me-south-1. + af-south-1: |- + Africa (Cape Town) Region. + Needs location constraint af-south-1. + cn-north-1: |- + China (Beijing) Region. + Needs location constraint cn-north-1. + cn-northwest-1: |- + China (Ningxia) Region. + Needs location constraint cn-northwest-1. + us-gov-east-1: |- + AWS GovCloud (US-East) Region. + Needs location constraint us-gov-east-1. + us-gov-west-1: |- + AWS GovCloud (US) Region. + Needs location constraint us-gov-west-1. +endpoint: {} +location_constraint: + '': Empty for US Region, Northern Virginia, or Pacific Northwest + us-east-2: US East (Ohio) Region + us-west-1: US West (Northern California) Region + us-west-2: US West (Oregon) Region + ca-central-1: Canada (Central) Region + eu-west-1: EU (Ireland) Region + eu-west-2: EU (London) Region + eu-west-3: EU (Paris) Region + eu-north-1: EU (Stockholm) Region + eu-south-1: EU (Milan) Region + EU: EU Region + ap-southeast-1: Asia Pacific (Singapore) Region + ap-southeast-2: Asia Pacific (Sydney) Region + ap-northeast-1: Asia Pacific (Tokyo) Region + ap-northeast-2: Asia Pacific (Seoul) Region + ap-northeast-3: Asia Pacific (Osaka-Local) Region + ap-south-1: Asia Pacific (Mumbai) Region + ap-east-1: Asia Pacific (Hong Kong) Region + sa-east-1: South America (Sao Paulo) Region + il-central-1: Israel (Tel Aviv) Region + me-south-1: Middle East (Bahrain) Region + af-south-1: Africa (Cape Town) Region + cn-north-1: China (Beijing) Region + cn-northwest-1: China (Ningxia) Region + us-gov-east-1: AWS GovCloud (US-East) Region + us-gov-west-1: AWS GovCloud (US) Region +acl: {} +storage_class: + '': Default + STANDARD: Standard storage class + REDUCED_REDUNDANCY: Reduced redundancy storage class + STANDARD_IA: Standard Infrequent Access storage class + ONEZONE_IA: One Zone Infrequent Access storage class + GLACIER: Glacier Flexible Retrieval storage class + DEEP_ARCHIVE: Glacier Deep Archive storage class + INTELLIGENT_TIERING: Intelligent-Tiering storage class + GLACIER_IR: Glacier Instant Retrieval storage class +server_side_encryption: + '': None + AES256: AES256 + aws:kms: aws:kms +bucket_acl: true +directory_bucket: true +leave_parts_on_error: true +requester_pays: true +sse_customer_algorithm: true +sse_customer_key: true +sse_customer_key_base64: true +sse_customer_key_md5: true +sse_kms_key_id: true +sts_endpoint: true +use_accelerate_endpoint: true +quirks: + might_gzip: false # Never auto gzips objects + use_unsigned_payload: false # AWS has trailer support which means it adds checksums in the trailer without seeking diff --git a/backend/s3/provider/Alibaba.yaml b/backend/s3/provider/Alibaba.yaml new file mode 100644 index 000000000..adb0bf86b --- /dev/null +++ b/backend/s3/provider/Alibaba.yaml @@ -0,0 +1,37 @@ +name: Alibaba +description: Alibaba Cloud Object Storage System (OSS) formerly Aliyun +endpoint: + oss-accelerate.aliyuncs.com: Global Accelerate + oss-accelerate-overseas.aliyuncs.com: Global Accelerate (outside mainland China) + oss-cn-hangzhou.aliyuncs.com: East China 1 (Hangzhou) + oss-cn-shanghai.aliyuncs.com: East China 2 (Shanghai) + oss-cn-qingdao.aliyuncs.com: North China 1 (Qingdao) + oss-cn-beijing.aliyuncs.com: North China 2 (Beijing) + oss-cn-zhangjiakou.aliyuncs.com: North China 3 (Zhangjiakou) + oss-cn-huhehaote.aliyuncs.com: North China 5 (Hohhot) + oss-cn-wulanchabu.aliyuncs.com: North China 6 (Ulanqab) + oss-cn-shenzhen.aliyuncs.com: South China 1 (Shenzhen) + oss-cn-heyuan.aliyuncs.com: South China 2 (Heyuan) + oss-cn-guangzhou.aliyuncs.com: South China 3 (Guangzhou) + oss-cn-chengdu.aliyuncs.com: West China 1 (Chengdu) + oss-cn-hongkong.aliyuncs.com: Hong Kong (Hong Kong) + oss-us-west-1.aliyuncs.com: US West 1 (Silicon Valley) + oss-us-east-1.aliyuncs.com: US East 1 (Virginia) + oss-ap-southeast-1.aliyuncs.com: Southeast Asia Southeast 1 (Singapore) + oss-ap-southeast-2.aliyuncs.com: Asia Pacific Southeast 2 (Sydney) + oss-ap-southeast-3.aliyuncs.com: Southeast Asia Southeast 3 (Kuala Lumpur) + oss-ap-southeast-5.aliyuncs.com: Asia Pacific Southeast 5 (Jakarta) + oss-ap-northeast-1.aliyuncs.com: Asia Pacific Northeast 1 (Japan) + oss-ap-south-1.aliyuncs.com: Asia Pacific South 1 (Mumbai) + oss-eu-central-1.aliyuncs.com: Central Europe 1 (Frankfurt) + oss-eu-west-1.aliyuncs.com: West Europe (London) + oss-me-east-1.aliyuncs.com: Middle East 1 (Dubai) +acl: {} +storage_class: + '': Default + STANDARD: Standard storage class + GLACIER: Archive storage mode + STANDARD_IA: Infrequent access storage mode +bucket_acl: true +quirks: + use_multipart_etag: false # multipar ETags differ from AWS diff --git a/backend/s3/provider/ArvanCloud.yaml b/backend/s3/provider/ArvanCloud.yaml new file mode 100644 index 000000000..04d28fef2 --- /dev/null +++ b/backend/s3/provider/ArvanCloud.yaml @@ -0,0 +1,19 @@ +name: ArvanCloud +description: Arvan Cloud Object Storage (AOS) +endpoint: + s3.ir-thr-at1.arvanstorage.ir: |- + The default endpoint - a good choice if you are unsure. + Tehran Iran (Simin) + s3.ir-tbz-sh1.arvanstorage.ir: Tabriz Iran (Shahriar) +location_constraint: + ir-thr-at1: Tehran Iran (Simin) + ir-tbz-sh1: Tabriz Iran (Shahriar) +acl: {} +storage_class: + STANDARD: Standard storage class +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Ceph.yaml b/backend/s3/provider/Ceph.yaml new file mode 100644 index 000000000..910a00ea3 --- /dev/null +++ b/backend/s3/provider/Ceph.yaml @@ -0,0 +1,20 @@ +name: Ceph +description: Ceph Object Storage +region: {} +endpoint: {} +location_constraint: {} +acl: {} +server_side_encryption: + '': None + AES256: AES256 + aws:kms: aws:kms +bucket_acl: true +sse_customer_algorithm: true +sse_customer_key: true +sse_customer_key_base64: true +sse_customer_key_md5: true +sse_kms_key_id: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false diff --git a/backend/s3/provider/ChinaMobile.yaml b/backend/s3/provider/ChinaMobile.yaml new file mode 100644 index 000000000..8209fa4ba --- /dev/null +++ b/backend/s3/provider/ChinaMobile.yaml @@ -0,0 +1,98 @@ +name: ChinaMobile +description: China Mobile Ecloud Elastic Object Storage (EOS) +endpoint: + eos-wuxi-1.cmecloud.cn: |- + The default endpoint - a good choice if you are unsure. + East China (Suzhou) + eos-jinan-1.cmecloud.cn: East China (Jinan) + eos-ningbo-1.cmecloud.cn: East China (Hangzhou) + eos-shanghai-1.cmecloud.cn: East China (Shanghai-1) + eos-zhengzhou-1.cmecloud.cn: Central China (Zhengzhou) + eos-hunan-1.cmecloud.cn: Central China (Changsha-1) + eos-zhuzhou-1.cmecloud.cn: Central China (Changsha-2) + eos-guangzhou-1.cmecloud.cn: South China (Guangzhou-2) + eos-dongguan-1.cmecloud.cn: South China (Guangzhou-3) + eos-beijing-1.cmecloud.cn: North China (Beijing-1) + eos-beijing-2.cmecloud.cn: North China (Beijing-2) + eos-beijing-4.cmecloud.cn: North China (Beijing-3) + eos-huhehaote-1.cmecloud.cn: North China (Huhehaote) + eos-chengdu-1.cmecloud.cn: Southwest China (Chengdu) + eos-chongqing-1.cmecloud.cn: Southwest China (Chongqing) + eos-guiyang-1.cmecloud.cn: Southwest China (Guiyang) + eos-xian-1.cmecloud.cn: Nouthwest China (Xian) + eos-yunnan.cmecloud.cn: Yunnan China (Kunming) + eos-yunnan-2.cmecloud.cn: Yunnan China (Kunming-2) + eos-tianjin-1.cmecloud.cn: Tianjin China (Tianjin) + eos-jilin-1.cmecloud.cn: Jilin China (Changchun) + eos-hubei-1.cmecloud.cn: Hubei China (Xiangyan) + eos-jiangxi-1.cmecloud.cn: Jiangxi China (Nanchang) + eos-gansu-1.cmecloud.cn: Gansu China (Lanzhou) + eos-shanxi-1.cmecloud.cn: Shanxi China (Taiyuan) + eos-liaoning-1.cmecloud.cn: Liaoning China (Shenyang) + eos-hebei-1.cmecloud.cn: Hebei China (Shijiazhuang) + eos-fujian-1.cmecloud.cn: Fujian China (Xiamen) + eos-guangxi-1.cmecloud.cn: Guangxi China (Nanning) + eos-anhui-1.cmecloud.cn: Anhui China (Huainan) +location_constraint: + wuxi1: East China (Suzhou) + jinan1: East China (Jinan) + ningbo1: East China (Hangzhou) + shanghai1: East China (Shanghai-1) + zhengzhou1: Central China (Zhengzhou) + hunan1: Central China (Changsha-1) + zhuzhou1: Central China (Changsha-2) + guangzhou1: South China (Guangzhou-2) + dongguan1: South China (Guangzhou-3) + beijing1: North China (Beijing-1) + beijing2: North China (Beijing-2) + beijing4: North China (Beijing-3) + huhehaote1: North China (Huhehaote) + chengdu1: Southwest China (Chengdu) + chongqing1: Southwest China (Chongqing) + guiyang1: Southwest China (Guiyang) + xian1: Northwest China (Xian) + yunnan: Yunnan China (Kunming) + yunnan2: Yunnan China (Kunming-2) + tianjin1: Tianjin China (Tianjin) + jilin1: Jilin China (Changchun) + hubei1: Hubei China (Xiangyan) + jiangxi1: Jiangxi China (Nanchang) + gansu1: Gansu China (Lanzhou) + shanxi1: Shanxi China (Taiyuan) + liaoning1: Liaoning China (Shenyang) + hebei1: Hebei China (Shijiazhuang) + fujian1: Fujian China (Xiamen) + guangxi1: Guangxi China (Nanning) + anhui1: Anhui China (Huainan) +acl: + private: |- + Owner gets FULL_CONTROL. + No one else has access rights (default). + public-read: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ access. + public-read-write: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ and WRITE access. + Granting this on a bucket is generally not recommended. + authenticated-read: |- + Owner gets FULL_CONTROL. + The AuthenticatedUsers group gets READ access. +storage_class: + '': Default + STANDARD: Standard storage class + GLACIER: Archive storage mode + STANDARD_IA: Infrequent access storage mode +server_side_encryption: + '': None + AES256: AES256 +bucket_acl: true +sse_customer_algorithm: true +sse_customer_key: true +sse_customer_key_base64: true +sse_customer_key_md5: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Cloudflare.yaml b/backend/s3/provider/Cloudflare.yaml new file mode 100644 index 000000000..991a212c0 --- /dev/null +++ b/backend/s3/provider/Cloudflare.yaml @@ -0,0 +1,8 @@ +name: Cloudflare +description: Cloudflare R2 Storage +region: + auto: R2 buckets are automatically distributed across Cloudflare's data centers for low latency. +endpoint: {} +quirks: + force_path_style: true + use_multipart_etag: false # multipart ETags are random diff --git a/backend/s3/provider/Cubbit.yaml b/backend/s3/provider/Cubbit.yaml new file mode 100644 index 000000000..19907b272 --- /dev/null +++ b/backend/s3/provider/Cubbit.yaml @@ -0,0 +1,10 @@ +name: Cubbit +description: Cubbit DS3 Object Storage +region: + eu-west-1: Europe West +endpoint: + s3.cubbit.eu: Cubbit DS3 Object Storage endpoint +acl: {} +bucket_acl: true +quirks: + use_multipart_etag: false diff --git a/backend/s3/provider/DigitalOcean.yaml b/backend/s3/provider/DigitalOcean.yaml new file mode 100644 index 000000000..1a64d47af --- /dev/null +++ b/backend/s3/provider/DigitalOcean.yaml @@ -0,0 +1,20 @@ +name: DigitalOcean +description: DigitalOcean Spaces +region: {} +endpoint: + syd1.digitaloceanspaces.com: DigitalOcean Spaces Sydney 1 + sfo3.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 3 + sfo2.digitaloceanspaces.com: DigitalOcean Spaces San Francisco 2 + fra1.digitaloceanspaces.com: DigitalOcean Spaces Frankfurt 1 + nyc3.digitaloceanspaces.com: DigitalOcean Spaces New York 3 + ams3.digitaloceanspaces.com: DigitalOcean Spaces Amsterdam 3 + sgp1.digitaloceanspaces.com: DigitalOcean Spaces Singapore 1 + lon1.digitaloceanspaces.com: DigitalOcean Spaces London 1 + tor1.digitaloceanspaces.com: DigitalOcean Spaces Toronto 1 + blr1.digitaloceanspaces.com: DigitalOcean Spaces Bangalore 1 +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Dreamhost.yaml b/backend/s3/provider/Dreamhost.yaml new file mode 100644 index 000000000..ab5e87612 --- /dev/null +++ b/backend/s3/provider/Dreamhost.yaml @@ -0,0 +1,11 @@ +name: Dreamhost +description: Dreamhost DreamObjects +region: {} +endpoint: + objects-us-east-1.dream.io: Dream Objects endpoint +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Exaba.yaml b/backend/s3/provider/Exaba.yaml new file mode 100644 index 000000000..f30a5b698 --- /dev/null +++ b/backend/s3/provider/Exaba.yaml @@ -0,0 +1,9 @@ +name: Exaba +description: Exaba Object Storage +region: {} +endpoint: {} +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + force_path_style: true diff --git a/backend/s3/provider/FileLu.yaml b/backend/s3/provider/FileLu.yaml new file mode 100644 index 000000000..f725c0ce7 --- /dev/null +++ b/backend/s3/provider/FileLu.yaml @@ -0,0 +1,21 @@ +name: FileLu +description: FileLu S5 (S3-Compatible Object Storage) +region: + global: Global + us-east: North America (US-East) + eu-central: Europe (EU-Central) + ap-southeast: Asia Pacific (AP-Southeast) + me-central: Middle East (ME-Central) +endpoint: + s5lu.com: Global FileLu S5 endpoint + us.s5lu.com: North America (US-East) region endpoint + eu.s5lu.com: Europe (EU-Central) region endpoint + ap.s5lu.com: Asia Pacific (AP-Southeast) region endpoint + me.s5lu.com: Middle East (ME-Central) region endpoint +acl: {} +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_multipart_etag: false diff --git a/backend/s3/provider/FlashBlade.yaml b/backend/s3/provider/FlashBlade.yaml new file mode 100644 index 000000000..a4adfd986 --- /dev/null +++ b/backend/s3/provider/FlashBlade.yaml @@ -0,0 +1,6 @@ +name: FlashBlade +description: Pure Storage FlashBlade Object Storage +endpoint: {} +quirks: + might_gzip: false # never auto-gzip + force_path_style: true # supports vhost but defaults to path-style diff --git a/backend/s3/provider/GCS.yaml b/backend/s3/provider/GCS.yaml new file mode 100644 index 000000000..4ece3251d --- /dev/null +++ b/backend/s3/provider/GCS.yaml @@ -0,0 +1,20 @@ +name: GCS +description: Google Cloud Storage +region: {} +endpoint: + https://storage.googleapis.com: Google Cloud Storage endpoint +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + # Google break request Signature by mutating accept-encoding HTTP header + # https://github.com/rclone/rclone/issues/6670 + use_accept_encoding_gzip: false + sign_accept_encoding: false + use_already_exists: true # returns BucketNameUnavailable instead of BucketAlreadyExists but good enough! + # GCS doesn't like the x-id URL parameter the SDKv2 inserts + use_x_id: false + # GCS S3 doesn't support multi-part server side copy: + # See: https://issuetracker.google.com/issues/323465186 + # So make cutoff very large which it does seem to support + copy_cutoff: 9223372036854775807 diff --git a/backend/s3/provider/Hetzner.yaml b/backend/s3/provider/Hetzner.yaml new file mode 100644 index 000000000..a7218a3da --- /dev/null +++ b/backend/s3/provider/Hetzner.yaml @@ -0,0 +1,15 @@ +name: Hetzner +description: Hetzner Object Storage +region: + hel1: Helsinki + fsn1: Falkenstein + nbg1: Nuremberg +endpoint: + hel1.your-objectstorage.com: Helsinki + fsn1.your-objectstorage.com: Falkenstein + nbg1.your-objectstorage.com: Nuremberg +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + use_already_exists: false diff --git a/backend/s3/provider/HuaweiOBS.yaml b/backend/s3/provider/HuaweiOBS.yaml new file mode 100644 index 000000000..9811ca497 --- /dev/null +++ b/backend/s3/provider/HuaweiOBS.yaml @@ -0,0 +1,41 @@ +name: HuaweiOBS +description: Huawei Object Storage Service +region: + af-south-1: AF-Johannesburg + ap-southeast-2: AP-Bangkok + ap-southeast-3: AP-Singapore + cn-east-3: CN East-Shanghai1 + cn-east-2: CN East-Shanghai2 + cn-north-1: CN North-Beijing1 + cn-north-4: CN North-Beijing4 + cn-south-1: CN South-Guangzhou + ap-southeast-1: CN-Hong Kong + sa-argentina-1: LA-Buenos Aires1 + sa-peru-1: LA-Lima1 + na-mexico-1: LA-Mexico City1 + sa-chile-1: LA-Santiago2 + sa-brazil-1: LA-Sao Paulo1 + ru-northwest-2: RU-Moscow2 +endpoint: + obs.af-south-1.myhuaweicloud.com: AF-Johannesburg + obs.ap-southeast-2.myhuaweicloud.com: AP-Bangkok + obs.ap-southeast-3.myhuaweicloud.com: AP-Singapore + obs.cn-east-3.myhuaweicloud.com: CN East-Shanghai1 + obs.cn-east-2.myhuaweicloud.com: CN East-Shanghai2 + obs.cn-north-1.myhuaweicloud.com: CN North-Beijing1 + obs.cn-north-4.myhuaweicloud.com: CN North-Beijing4 + obs.cn-south-1.myhuaweicloud.com: CN South-Guangzhou + obs.ap-southeast-1.myhuaweicloud.com: CN-Hong Kong + obs.sa-argentina-1.myhuaweicloud.com: LA-Buenos Aires1 + obs.sa-peru-1.myhuaweicloud.com: LA-Lima1 + obs.na-mexico-1.myhuaweicloud.com: LA-Mexico City1 + obs.sa-chile-1.myhuaweicloud.com: LA-Santiago2 + obs.sa-brazil-1.myhuaweicloud.com: LA-Sao Paulo1 + obs.ru-northwest-2.myhuaweicloud.com: RU-Moscow2 +acl: {} +bucket_acl: true +quirks: + # Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever. + list_url_encode: false + list_version: 1 + use_already_exists: false diff --git a/backend/s3/provider/IBMCOS.yaml b/backend/s3/provider/IBMCOS.yaml new file mode 100644 index 000000000..8b6290fc3 --- /dev/null +++ b/backend/s3/provider/IBMCOS.yaml @@ -0,0 +1,126 @@ +name: IBMCOS +description: IBM COS S3 +region: {} +endpoint: + s3.us.cloud-object-storage.appdomain.cloud: US Cross Region Endpoint + s3.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Endpoint + s3.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Endpoint + s3.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Endpoint + s3.private.us.cloud-object-storage.appdomain.cloud: US Cross Region Private Endpoint + s3.private.dal.us.cloud-object-storage.appdomain.cloud: US Cross Region Dallas Private Endpoint + s3.private.wdc.us.cloud-object-storage.appdomain.cloud: US Cross Region Washington DC Private Endpoint + s3.private.sjc.us.cloud-object-storage.appdomain.cloud: US Cross Region San Jose Private Endpoint + s3.us-east.cloud-object-storage.appdomain.cloud: US Region East Endpoint + s3.private.us-east.cloud-object-storage.appdomain.cloud: US Region East Private Endpoint + s3.us-south.cloud-object-storage.appdomain.cloud: US Region South Endpoint + s3.private.us-south.cloud-object-storage.appdomain.cloud: US Region South Private Endpoint + s3.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Endpoint + s3.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Endpoint + s3.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Endpoint + s3.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Endpoint + s3.private.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Private Endpoint + s3.private.fra.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Frankfurt Private Endpoint + s3.private.mil.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Milan Private Endpoint + s3.private.ams.eu.cloud-object-storage.appdomain.cloud: EU Cross Region Amsterdam Private Endpoint + s3.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Endpoint + s3.private.eu-gb.cloud-object-storage.appdomain.cloud: Great Britain Private Endpoint + s3.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Endpoint + s3.private.eu-de.cloud-object-storage.appdomain.cloud: EU Region DE Private Endpoint + s3.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Endpoint + s3.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Endpoint + s3.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Endpoint + s3.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Endpoint + s3.private.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Private Endpoint + s3.private.tok.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Tokyo Private Endpoint + s3.private.hkg.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Hong Kong Private Endpoint + s3.private.seo.ap.cloud-object-storage.appdomain.cloud: APAC Cross Regional Seoul Private Endpoint + s3.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Endpoint + s3.private.jp-tok.cloud-object-storage.appdomain.cloud: APAC Region Japan Private Endpoint + s3.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Endpoint + s3.private.au-syd.cloud-object-storage.appdomain.cloud: APAC Region Australia Private Endpoint + s3.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Endpoint + s3.private.ams03.cloud-object-storage.appdomain.cloud: Amsterdam Single Site Private Endpoint + s3.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Endpoint + s3.private.che01.cloud-object-storage.appdomain.cloud: Chennai Single Site Private Endpoint + s3.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Endpoint + s3.private.mel01.cloud-object-storage.appdomain.cloud: Melbourne Single Site Private Endpoint + s3.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Endpoint + s3.private.osl01.cloud-object-storage.appdomain.cloud: Oslo Single Site Private Endpoint + s3.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Endpoint + s3.private.tor01.cloud-object-storage.appdomain.cloud: Toronto Single Site Private Endpoint + s3.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Endpoint + s3.private.seo01.cloud-object-storage.appdomain.cloud: Seoul Single Site Private Endpoint + s3.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Endpoint + s3.private.mon01.cloud-object-storage.appdomain.cloud: Montreal Single Site Private Endpoint + s3.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Endpoint + s3.private.mex01.cloud-object-storage.appdomain.cloud: Mexico Single Site Private Endpoint + s3.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Endpoint + s3.private.sjc04.cloud-object-storage.appdomain.cloud: San Jose Single Site Private Endpoint + s3.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Endpoint + s3.private.mil01.cloud-object-storage.appdomain.cloud: Milan Single Site Private Endpoint + s3.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Endpoint + s3.private.hkg02.cloud-object-storage.appdomain.cloud: Hong Kong Single Site Private Endpoint + s3.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Endpoint + s3.private.par01.cloud-object-storage.appdomain.cloud: Paris Single Site Private Endpoint + s3.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Endpoint + s3.private.sng01.cloud-object-storage.appdomain.cloud: Singapore Single Site Private Endpoint +location_constraint: + us-standard: US Cross Region Standard + us-vault: US Cross Region Vault + us-cold: US Cross Region Cold + us-flex: US Cross Region Flex + us-east-standard: US East Region Standard + us-east-vault: US East Region Vault + us-east-cold: US East Region Cold + us-east-flex: US East Region Flex + us-south-standard: US South Region Standard + us-south-vault: US South Region Vault + us-south-cold: US South Region Cold + us-south-flex: US South Region Flex + eu-standard: EU Cross Region Standard + eu-vault: EU Cross Region Vault + eu-cold: EU Cross Region Cold + eu-flex: EU Cross Region Flex + eu-gb-standard: Great Britain Standard + eu-gb-vault: Great Britain Vault + eu-gb-cold: Great Britain Cold + eu-gb-flex: Great Britain Flex + ap-standard: APAC Standard + ap-vault: APAC Vault + ap-cold: APAC Cold + ap-flex: APAC Flex + mel01-standard: Melbourne Standard + mel01-vault: Melbourne Vault + mel01-cold: Melbourne Cold + mel01-flex: Melbourne Flex + tor01-standard: Toronto Standard + tor01-vault: Toronto Vault + tor01-cold: Toronto Cold + tor01-flex: Toronto Flex +acl: + private: |- + Owner gets FULL_CONTROL. + No one else has access rights (default). + This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS. + public-read: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ access. + This acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS. + public-read-write: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ and WRITE access. + This acl is available on IBM Cloud (Infra), On-Premise IBM COS. + authenticated-read: |- + Owner gets FULL_CONTROL. + The AuthenticatedUsers group gets READ access. + Not supported on Buckets. + This acl is available on IBM Cloud (Infra) and On-Premise IBM COS. +ibm_api_key: true +ibm_resource_instance_id: true +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_multipart_etag: false + use_already_exists: false # returns BucketAlreadyExists diff --git a/backend/s3/provider/IDrive.yaml b/backend/s3/provider/IDrive.yaml new file mode 100644 index 000000000..ff43add77 --- /dev/null +++ b/backend/s3/provider/IDrive.yaml @@ -0,0 +1,7 @@ +name: IDrive +description: IDrive e2 +acl: {} +bucket_acl: true +quirks: + force_path_style: true + use_already_exists: false diff --git a/backend/s3/provider/IONOS.yaml b/backend/s3/provider/IONOS.yaml new file mode 100644 index 000000000..ab455a15b --- /dev/null +++ b/backend/s3/provider/IONOS.yaml @@ -0,0 +1,17 @@ +name: IONOS +description: IONOS Cloud +region: + de: Frankfurt, Germany + eu-central-2: Berlin, Germany + eu-south-2: Logrono, Spain +endpoint: + s3-eu-central-1.ionoscloud.com: Frankfurt, Germany + s3-eu-central-2.ionoscloud.com: Berlin, Germany + s3-eu-south-2.ionoscloud.com: Logrono, Spain +acl: {} +bucket_acl: true +quirks: + # listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2 + force_path_style: true + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Intercolo.yaml b/backend/s3/provider/Intercolo.yaml new file mode 100644 index 000000000..51d877351 --- /dev/null +++ b/backend/s3/provider/Intercolo.yaml @@ -0,0 +1,10 @@ +name: Intercolo +description: Intercolo Object Storage +region: + de-fra: Frankfurt, Germany +endpoint: + de-fra.i3storage.com: Frankfurt, Germany +acl: {} +bucket_acl: true +quirks: + use_unsigned_payload: false # has trailer support diff --git a/backend/s3/provider/Leviia.yaml b/backend/s3/provider/Leviia.yaml new file mode 100644 index 000000000..f79dd49fe --- /dev/null +++ b/backend/s3/provider/Leviia.yaml @@ -0,0 +1,11 @@ +name: Leviia +description: Leviia Object Storage +region: {} +endpoint: + s3.leviia.com: |- + The default endpoint + Leviia +acl: {} +bucket_acl: true +quirks: + use_already_exists: false diff --git a/backend/s3/provider/Liara.yaml b/backend/s3/provider/Liara.yaml new file mode 100644 index 000000000..c6cff97fb --- /dev/null +++ b/backend/s3/provider/Liara.yaml @@ -0,0 +1,15 @@ +name: Liara +description: Liara Object Storage +endpoint: + storage.iran.liara.space: |- + The default endpoint + Iran +acl: {} +storage_class: + STANDARD: Standard storage class +bucket_acl: true +quirks: + force_path_style: true + list_url_encode: false + use_multipart_etag: false # mulitpart ETags differ from AWS + use_already_exists: false diff --git a/backend/s3/provider/Linode.yaml b/backend/s3/provider/Linode.yaml new file mode 100644 index 000000000..0923290d0 --- /dev/null +++ b/backend/s3/provider/Linode.yaml @@ -0,0 +1,26 @@ +name: Linode +description: Linode Object Storage +endpoint: + nl-ams-1.linodeobjects.com: Amsterdam (Netherlands), nl-ams-1 + us-southeast-1.linodeobjects.com: Atlanta, GA (USA), us-southeast-1 + in-maa-1.linodeobjects.com: Chennai (India), in-maa-1 + us-ord-1.linodeobjects.com: Chicago, IL (USA), us-ord-1 + eu-central-1.linodeobjects.com: Frankfurt (Germany), eu-central-1 + id-cgk-1.linodeobjects.com: Jakarta (Indonesia), id-cgk-1 + gb-lon-1.linodeobjects.com: London 2 (Great Britain), gb-lon-1 + us-lax-1.linodeobjects.com: Los Angeles, CA (USA), us-lax-1 + es-mad-1.linodeobjects.com: Madrid (Spain), es-mad-1 + au-mel-1.linodeobjects.com: Melbourne (Australia), au-mel-1 + us-mia-1.linodeobjects.com: Miami, FL (USA), us-mia-1 + it-mil-1.linodeobjects.com: Milan (Italy), it-mil-1 + us-east-1.linodeobjects.com: Newark, NJ (USA), us-east-1 + jp-osa-1.linodeobjects.com: Osaka (Japan), jp-osa-1 + fr-par-1.linodeobjects.com: Paris (France), fr-par-1 + br-gru-1.linodeobjects.com: São Paulo (Brazil), br-gru-1 + us-sea-1.linodeobjects.com: Seattle, WA (USA), us-sea-1 + ap-south-1.linodeobjects.com: Singapore, ap-south-1 + sg-sin-1.linodeobjects.com: Singapore 2, sg-sin-1 + se-sto-1.linodeobjects.com: Stockholm (Sweden), se-sto-1 + us-iad-1.linodeobjects.com: Washington, DC, (USA), us-iad-1 +acl: {} +bucket_acl: true diff --git a/backend/s3/provider/LyveCloud.yaml b/backend/s3/provider/LyveCloud.yaml new file mode 100644 index 000000000..c24a4ba20 --- /dev/null +++ b/backend/s3/provider/LyveCloud.yaml @@ -0,0 +1,12 @@ +name: LyveCloud +description: Seagate Lyve Cloud +region: {} +endpoint: + 's3.us-west-1.{account_name}.lyve.seagate.com': US West 1 - California + 's3.eu-west-1.{account_name}.lyve.seagate.com': EU West 1 - Ireland +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + use_multipart_etag: false # multipart ETags differ from AWS + use_already_exists: false diff --git a/backend/s3/provider/Magalu.yaml b/backend/s3/provider/Magalu.yaml new file mode 100644 index 000000000..4f9afe8f9 --- /dev/null +++ b/backend/s3/provider/Magalu.yaml @@ -0,0 +1,16 @@ +name: Magalu +description: Magalu Object Storage +endpoint: + br-se1.magaluobjects.com: São Paulo, SP (BR), br-se1 + br-ne1.magaluobjects.com: Fortaleza, CE (BR), br-ne1 +acl: {} +storage_class: + STANDARD: Standard storage class + GLACIER_IR: Glacier Instant Retrieval storage class +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/Mega.yaml b/backend/s3/provider/Mega.yaml new file mode 100644 index 000000000..0f2a15fa1 --- /dev/null +++ b/backend/s3/provider/Mega.yaml @@ -0,0 +1,15 @@ +name: Mega +description: MEGA S4 Object Storage +endpoint: + s3.eu-central-1.s4.mega.io: Mega S4 eu-central-1 (Amsterdam) + s3.eu-central-2.s4.mega.io: Mega S4 eu-central-2 (Bettembourg) + s3.ca-central-1.s4.mega.io: Mega S4 ca-central-1 (Montreal) + s3.ca-west-1.s4.mega.io: Mega S4 ca-west-1 (Vancouver) +bucket_acl: true +quirks: + list_version: 2 + force_path_style: true + list_url_encode: true + use_multipart_etag: false + use_already_exists: false + copy_cutoff: 9223372036854775807 diff --git a/backend/s3/provider/Minio.yaml b/backend/s3/provider/Minio.yaml new file mode 100644 index 000000000..2b2c42ace --- /dev/null +++ b/backend/s3/provider/Minio.yaml @@ -0,0 +1,18 @@ +name: Minio +description: Minio Object Storage +region: {} +endpoint: {} +location_constraint: {} +acl: {} +server_side_encryption: + '': None + AES256: AES256 + aws:kms: aws:kms +bucket_acl: true +sse_customer_algorithm: true +sse_customer_key: true +sse_customer_key_base64: true +sse_customer_key_md5: true +sse_kms_key_id: true +quirks: + force_path_style: true diff --git a/backend/s3/provider/Netease.yaml b/backend/s3/provider/Netease.yaml new file mode 100644 index 000000000..88ed59d19 --- /dev/null +++ b/backend/s3/provider/Netease.yaml @@ -0,0 +1,12 @@ +name: Netease +description: Netease Object Storage (NOS) +region: {} +endpoint: {} +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + list_version: 1 + list_url_encode: false + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/OVHcloud.yaml b/backend/s3/provider/OVHcloud.yaml new file mode 100644 index 000000000..2220f2a94 --- /dev/null +++ b/backend/s3/provider/OVHcloud.yaml @@ -0,0 +1,36 @@ +name: OVHcloud +description: OVHcloud Object Storage +region: + gra: Gravelines, France + rbx: Roubaix, France + sbg: Strasbourg, France + eu-west-par: Paris, France (3AZ) + de: Frankfurt, Germany + uk: London, United Kingdom + waw: Warsaw, Poland + bhs: Beauharnois, Canada + ca-east-tor: Toronto, Canada + sgp: Singapore + ap-southeast-syd: Sydney, Australia + ap-south-mum: Mumbai, India + us-east-va: Vint Hill, Virginia, USA + us-west-or: Hillsboro, Oregon, USA + rbx-archive: Roubaix, France (Cold Archive) +endpoint: + s3.gra.io.cloud.ovh.net: OVHcloud Gravelines, France + s3.rbx.io.cloud.ovh.net: OVHcloud Roubaix, France + s3.sbg.io.cloud.ovh.net: OVHcloud Strasbourg, France + s3.eu-west-par.io.cloud.ovh.net: OVHcloud Paris, France (3AZ) + s3.de.io.cloud.ovh.net: OVHcloud Frankfurt, Germany + s3.uk.io.cloud.ovh.net: OVHcloud London, United Kingdom + s3.waw.io.cloud.ovh.net: OVHcloud Warsaw, Poland + s3.bhs.io.cloud.ovh.net: OVHcloud Beauharnois, Canada + s3.ca-east-tor.io.cloud.ovh.net: OVHcloud Toronto, Canada + s3.sgp.io.cloud.ovh.net: OVHcloud Singapore + s3.ap-southeast-syd.io.cloud.ovh.net: OVHcloud Sydney, Australia + s3.ap-south-mum.io.cloud.ovh.net: OVHcloud Mumbai, India + s3.us-east-va.io.cloud.ovh.us: OVHcloud Vint Hill, Virginia, USA + s3.us-west-or.io.cloud.ovh.us: OVHcloud Hillsboro, Oregon, USA + s3.rbx-archive.io.cloud.ovh.net: OVHcloud Roubaix, France (Cold Archive) +acl: {} +bucket_acl: true diff --git a/backend/s3/provider/Other.yaml b/backend/s3/provider/Other.yaml new file mode 100644 index 000000000..a32c3fdf2 --- /dev/null +++ b/backend/s3/provider/Other.yaml @@ -0,0 +1,39 @@ +name: Other +description: Any other S3 compatible provider +region: + '': |- + Use this if unsure. + Will use v4 signatures and an empty region. + other-v2-signature: |- + Use this only if v4 signatures don't work. + E.g. pre Jewel/v10 CEPH. +endpoint: {} +location_constraint: {} +acl: + private: |- + Owner gets FULL_CONTROL. + No one else has access rights (default). + public-read: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ access. + public-read-write: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ and WRITE access. + Granting this on a bucket is generally not recommended. + authenticated-read: |- + Owner gets FULL_CONTROL. + The AuthenticatedUsers group gets READ access. + bucket-owner-read: |- + Object owner gets FULL_CONTROL. + Bucket owner gets READ access. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. + bucket-owner-full-control: |- + Both the object owner and the bucket owner get FULL_CONTROL over the object. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/Outscale.yaml b/backend/s3/provider/Outscale.yaml new file mode 100644 index 000000000..e107456df --- /dev/null +++ b/backend/s3/provider/Outscale.yaml @@ -0,0 +1,18 @@ +name: Outscale +description: OUTSCALE Object Storage (OOS) +region: + eu-west-2: Paris, France + us-east-2: New Jersey, USA + us-west-1: California, USA + cloudgouv-eu-west-1: SecNumCloud, Paris, France + ap-northeast-1: Tokyo, Japan +endpoint: + oos.eu-west-2.outscale.com: Outscale EU West 2 (Paris) + oos.us-east-2.outscale.com: Outscale US east 2 (New Jersey) + oos.us-west-1.outscale.com: Outscale EU West 1 (California) + oos.cloudgouv-eu-west-1.outscale.com: Outscale SecNumCloud (Paris) + oos.ap-northeast-1.outscale.com: Outscale AP Northeast 1 (Japan) +acl: {} +bucket_acl: true +quirks: + force_path_style: true diff --git a/backend/s3/provider/Petabox.yaml b/backend/s3/provider/Petabox.yaml new file mode 100644 index 000000000..a28922478 --- /dev/null +++ b/backend/s3/provider/Petabox.yaml @@ -0,0 +1,19 @@ +name: Petabox +description: Petabox Object Storage +region: + us-east-1: US East (N. Virginia) + eu-central-1: Europe (Frankfurt) + ap-southeast-1: Asia Pacific (Singapore) + me-south-1: Middle East (Bahrain) + sa-east-1: South America (São Paulo) +endpoint: + s3.petabox.io: US East (N. Virginia) + s3.us-east-1.petabox.io: US East (N. Virginia) + s3.eu-central-1.petabox.io: Europe (Frankfurt) + s3.ap-southeast-1.petabox.io: Asia Pacific (Singapore) + s3.me-south-1.petabox.io: Middle East (Bahrain) + s3.sa-east-1.petabox.io: South America (São Paulo) +acl: {} +bucket_acl: true +quirks: + use_already_exists: false diff --git a/backend/s3/provider/Qiniu.yaml b/backend/s3/provider/Qiniu.yaml new file mode 100644 index 000000000..a1f8df282 --- /dev/null +++ b/backend/s3/provider/Qiniu.yaml @@ -0,0 +1,53 @@ +name: Qiniu +description: Qiniu Object Storage (Kodo) +region: + cn-east-1: |- + The default endpoint - a good choice if you are unsure. + East China Region 1. + Needs location constraint cn-east-1. + cn-east-2: |- + East China Region 2. + Needs location constraint cn-east-2. + cn-north-1: |- + North China Region 1. + Needs location constraint cn-north-1. + cn-south-1: |- + South China Region 1. + Needs location constraint cn-south-1. + us-north-1: |- + North America Region. + Needs location constraint us-north-1. + ap-southeast-1: |- + Southeast Asia Region 1. + Needs location constraint ap-southeast-1. + ap-northeast-1: |- + Northeast Asia Region 1. + Needs location constraint ap-northeast-1. +endpoint: + s3-cn-east-1.qiniucs.com: East China Endpoint 1 + s3-cn-east-2.qiniucs.com: East China Endpoint 2 + s3-cn-north-1.qiniucs.com: North China Endpoint 1 + s3-cn-south-1.qiniucs.com: South China Endpoint 1 + s3-us-north-1.qiniucs.com: North America Endpoint 1 + s3-ap-southeast-1.qiniucs.com: Southeast Asia Endpoint 1 + s3-ap-northeast-1.qiniucs.com: Northeast Asia Endpoint 1 +location_constraint: + cn-east-1: East China Region 1 + cn-east-2: East China Region 2 + cn-north-1: North China Region 1 + cn-south-1: South China Region 1 + us-north-1: North America Region 1 + ap-southeast-1: Southeast Asia Region 1 + ap-northeast-1: Northeast Asia Region 1 +acl: {} +storage_class: + STANDARD: Standard storage class + LINE: Infrequent access storage mode + GLACIER: Archive storage mode + DEEP_ARCHIVE: Deep archive storage mode +bucket_acl: true +quirks: + use_multipart_etag: false + list_url_encode: false + force_path_style: true + use_already_exists: false diff --git a/backend/s3/provider/Rabata.yaml b/backend/s3/provider/Rabata.yaml new file mode 100644 index 000000000..dd5f5d17b --- /dev/null +++ b/backend/s3/provider/Rabata.yaml @@ -0,0 +1,15 @@ +name: Rabata +description: Rabata Cloud Storage +region: + us-east-1: US East (N. Virginia) + eu-west-1: EU (Ireland) + eu-west-2: EU (London) +endpoint: + s3.us-east-1.rabata.io: US East (N. Virginia) + s3.eu-west-1.rabata.io: EU West (Ireland) + s3.eu-west-2.rabata.io: EU West (London) +location_constraint: + us-east-1: US East (N. Virginia) + eu-west-1: EU (Ireland) + eu-west-2: EU (London) +# server side copy not supported diff --git a/backend/s3/provider/RackCorp.yaml b/backend/s3/provider/RackCorp.yaml new file mode 100644 index 000000000..e2030480e --- /dev/null +++ b/backend/s3/provider/RackCorp.yaml @@ -0,0 +1,67 @@ +name: RackCorp +description: RackCorp Object Storage +region: + global: Global CDN (All locations) Region + au: Australia (All states) + au-nsw: NSW (Australia) Region + au-qld: QLD (Australia) Region + au-vic: VIC (Australia) Region + au-wa: Perth (Australia) Region + ph: Manila (Philippines) Region + th: Bangkok (Thailand) Region + hk: HK (Hong Kong) Region + mn: Ulaanbaatar (Mongolia) Region + kg: Bishkek (Kyrgyzstan) Region + id: Jakarta (Indonesia) Region + jp: Tokyo (Japan) Region + sg: SG (Singapore) Region + de: Frankfurt (Germany) Region + us: USA (AnyCast) Region + us-east-1: New York (USA) Region + us-west-1: Freemont (USA) Region + nz: Auckland (New Zealand) Region +endpoint: + s3.rackcorp.com: Global (AnyCast) Endpoint + au.s3.rackcorp.com: Australia (Anycast) Endpoint + au-nsw.s3.rackcorp.com: Sydney (Australia) Endpoint + au-qld.s3.rackcorp.com: Brisbane (Australia) Endpoint + au-vic.s3.rackcorp.com: Melbourne (Australia) Endpoint + au-wa.s3.rackcorp.com: Perth (Australia) Endpoint + ph.s3.rackcorp.com: Manila (Philippines) Endpoint + th.s3.rackcorp.com: Bangkok (Thailand) Endpoint + hk.s3.rackcorp.com: HK (Hong Kong) Endpoint + mn.s3.rackcorp.com: Ulaanbaatar (Mongolia) Endpoint + kg.s3.rackcorp.com: Bishkek (Kyrgyzstan) Endpoint + id.s3.rackcorp.com: Jakarta (Indonesia) Endpoint + jp.s3.rackcorp.com: Tokyo (Japan) Endpoint + sg.s3.rackcorp.com: SG (Singapore) Endpoint + de.s3.rackcorp.com: Frankfurt (Germany) Endpoint + us.s3.rackcorp.com: USA (AnyCast) Endpoint + us-east-1.s3.rackcorp.com: New York (USA) Endpoint + us-west-1.s3.rackcorp.com: Freemont (USA) Endpoint + nz.s3.rackcorp.com: Auckland (New Zealand) Endpoint +location_constraint: + global: Global CDN Region + au: Australia (All locations) + au-nsw: NSW (Australia) Region + au-qld: QLD (Australia) Region + au-vic: VIC (Australia) Region + au-wa: Perth (Australia) Region + ph: Manila (Philippines) Region + th: Bangkok (Thailand) Region + hk: HK (Hong Kong) Region + mn: Ulaanbaatar (Mongolia) Region + kg: Bishkek (Kyrgyzstan) Region + id: Jakarta (Indonesia) Region + jp: Tokyo (Japan) Region + sg: SG (Singapore) Region + de: Frankfurt (Germany) Region + us: USA (AnyCast) Region + us-east-1: New York (USA) Region + us-west-1: Fremont (USA) Region + nz: Auckland (New Zealand) Region +acl: {} +bucket_acl: true +quirks: + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/Rclone.yaml b/backend/s3/provider/Rclone.yaml new file mode 100644 index 000000000..6697c7323 --- /dev/null +++ b/backend/s3/provider/Rclone.yaml @@ -0,0 +1,11 @@ +name: Rclone +description: Rclone S3 Server +endpoint: {} +quirks: + force_path_style: true + use_multipart_etag: false + use_already_exists: false + # rclone serve doesn't support multi-part server side copy: + # See: https://github.com/rclone/rclone/issues/7454 + # So make cutoff very large which it does support + copy_cutoff: 9223372036854775807 diff --git a/backend/s3/provider/Scaleway.yaml b/backend/s3/provider/Scaleway.yaml new file mode 100644 index 000000000..29497b945 --- /dev/null +++ b/backend/s3/provider/Scaleway.yaml @@ -0,0 +1,28 @@ +name: Scaleway +description: Scaleway Object Storage +region: + nl-ams: Amsterdam, The Netherlands + fr-par: Paris, France + pl-waw: Warsaw, Poland +endpoint: + s3.nl-ams.scw.cloud: Amsterdam Endpoint + s3.fr-par.scw.cloud: Paris Endpoint + s3.pl-waw.scw.cloud: Warsaw Endpoint +acl: {} +storage_class: + '': Default. + STANDARD: |- + The Standard class for any upload. + Suitable for on-demand content like streaming or CDN. + Available in all regions. + GLACIER: |- + Archived storage. + Prices are lower, but it needs to be restored first to be accessed. + Available in FR-PAR and NL-AMS regions. + ONEZONE_IA: |- + One Zone - Infrequent Access. + A good choice for storing secondary backup copies or easily re-creatable data. + Available in the FR-PAR region only. +bucket_acl: true +quirks: + max_upload_parts: 1000 diff --git a/backend/s3/provider/SeaweedFS.yaml b/backend/s3/provider/SeaweedFS.yaml new file mode 100644 index 000000000..1e377e09a --- /dev/null +++ b/backend/s3/provider/SeaweedFS.yaml @@ -0,0 +1,14 @@ +name: SeaweedFS +description: SeaweedFS S3 +region: {} +endpoint: + localhost:8333: SeaweedFS S3 localhost +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/Selectel.yaml b/backend/s3/provider/Selectel.yaml new file mode 100644 index 000000000..70cc74bad --- /dev/null +++ b/backend/s3/provider/Selectel.yaml @@ -0,0 +1,8 @@ +name: Selectel +description: Selectel Object Storage +region: + ru-1: St. Petersburg +endpoint: + s3.ru-1.storage.selcloud.ru: Saint Petersburg +quirks: + list_url_encode: false diff --git a/backend/s3/provider/Servercore.yaml b/backend/s3/provider/Servercore.yaml new file mode 100644 index 000000000..cc566e109 --- /dev/null +++ b/backend/s3/provider/Servercore.yaml @@ -0,0 +1,17 @@ +name: Servercore +description: Servercore Object Storage +region: + ru-1: St. Petersburg + gis-1: Moscow + ru-7: Moscow + uz-2: Tashkent, Uzbekistan + kz-1: Almaty, Kazakhstan +endpoint: + s3.ru-1.storage.selcloud.ru: Saint Petersburg + s3.gis-1.storage.selcloud.ru: Moscow + s3.ru-7.storage.selcloud.ru: Moscow + s3.uz-2.srvstorage.uz: Tashkent, Uzbekistan + s3.kz-1.srvstorage.kz: Almaty, Kazakhstan +bucket_acl: true +quirks: + list_url_encode: false diff --git a/backend/s3/provider/SpectraLogic.yaml b/backend/s3/provider/SpectraLogic.yaml new file mode 100644 index 000000000..8ed135616 --- /dev/null +++ b/backend/s3/provider/SpectraLogic.yaml @@ -0,0 +1,5 @@ +name: SpectraLogic +description: Spectra Logic Black Pearl +endpoint: {} +quirks: + force_path_style: true # path-style required diff --git a/backend/s3/provider/StackPath.yaml b/backend/s3/provider/StackPath.yaml new file mode 100644 index 000000000..9b3723f3f --- /dev/null +++ b/backend/s3/provider/StackPath.yaml @@ -0,0 +1,14 @@ +name: StackPath +description: StackPath Object Storage +region: {} +endpoint: + s3.us-east-2.stackpathstorage.com: US East Endpoint + s3.us-west-1.stackpathstorage.com: US West Endpoint + s3.eu-central-1.stackpathstorage.com: EU Endpoint +acl: {} +bucket_acl: true +quirks: + list_version: 1 + force_path_style: true + list_url_encode: false + use_already_exists: false diff --git a/backend/s3/provider/Storj.yaml b/backend/s3/provider/Storj.yaml new file mode 100644 index 000000000..11697be85 --- /dev/null +++ b/backend/s3/provider/Storj.yaml @@ -0,0 +1,11 @@ +name: Storj +description: Storj (S3 Compatible Gateway) +endpoint: + gateway.storjshare.io: Global Hosted Gateway +quirks: + use_already_exists: false # returns BucketAlreadyExists + # Storj doesn't support multi-part server side copy: + # https://github.com/storj/roadmap/issues/40 + # So make cutoff very large which it does support + copy_cutoff: 9223372036854775807 + min_chunk_size: 67108864 diff --git a/backend/s3/provider/Synology.yaml b/backend/s3/provider/Synology.yaml new file mode 100644 index 000000000..8efeed610 --- /dev/null +++ b/backend/s3/provider/Synology.yaml @@ -0,0 +1,18 @@ +name: Synology +description: Synology C2 Object Storage +region: + eu-001: Europe Region 1 + eu-002: Europe Region 2 + us-001: US Region 1 + us-002: US Region 2 + tw-001: Asia (Taiwan) +endpoint: + eu-001.s3.synologyc2.net: EU Endpoint 1 + eu-002.s3.synologyc2.net: EU Endpoint 2 + us-001.s3.synologyc2.net: US Endpoint 1 + us-002.s3.synologyc2.net: US Endpoint 2 + tw-001.s3.synologyc2.net: TW Endpoint 1 +location_constraint: {} +quirks: + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/TencentCOS.yaml b/backend/s3/provider/TencentCOS.yaml new file mode 100644 index 000000000..ca7a8d30c --- /dev/null +++ b/backend/s3/provider/TencentCOS.yaml @@ -0,0 +1,52 @@ +name: TencentCOS +description: Tencent Cloud Object Storage (COS) +endpoint: + cos.ap-beijing.myqcloud.com: Beijing Region + cos.ap-nanjing.myqcloud.com: Nanjing Region + cos.ap-shanghai.myqcloud.com: Shanghai Region + cos.ap-guangzhou.myqcloud.com: Guangzhou Region + cos.ap-chengdu.myqcloud.com: Chengdu Region + cos.ap-chongqing.myqcloud.com: Chongqing Region + cos.ap-hongkong.myqcloud.com: Hong Kong (China) Region + cos.ap-singapore.myqcloud.com: Singapore Region + cos.ap-mumbai.myqcloud.com: Mumbai Region + cos.ap-seoul.myqcloud.com: Seoul Region + cos.ap-bangkok.myqcloud.com: Bangkok Region + cos.ap-tokyo.myqcloud.com: Tokyo Region + cos.na-siliconvalley.myqcloud.com: Silicon Valley Region + cos.na-ashburn.myqcloud.com: Virginia Region + cos.na-toronto.myqcloud.com: Toronto Region + cos.eu-frankfurt.myqcloud.com: Frankfurt Region + cos.eu-moscow.myqcloud.com: Moscow Region + cos.accelerate.myqcloud.com: Use Tencent COS Accelerate Endpoint +acl: + default: |- + Owner gets Full_CONTROL. + No one else has access rights (default). + public-read: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ access. + public-read-write: |- + Owner gets FULL_CONTROL. + The AllUsers group gets READ and WRITE access. + Granting this on a bucket is generally not recommended. + authenticated-read: |- + Owner gets FULL_CONTROL. + The AuthenticatedUsers group gets READ access. + bucket-owner-read: |- + Object owner gets FULL_CONTROL. + Bucket owner gets READ access. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. + bucket-owner-full-control: |- + Both the object owner and the bucket owner get FULL_CONTROL over the object. + If you specify this canned ACL when creating a bucket, Amazon S3 ignores it. +storage_class: + '': Default + STANDARD: Standard storage class + ARCHIVE: Archive storage mode + STANDARD_IA: Infrequent access storage mode +bucket_acl: true +quirks: + list_version: 1 + use_multipart_etag: false + use_already_exists: false diff --git a/backend/s3/provider/Wasabi.yaml b/backend/s3/provider/Wasabi.yaml new file mode 100644 index 000000000..e44e61f76 --- /dev/null +++ b/backend/s3/provider/Wasabi.yaml @@ -0,0 +1,21 @@ +name: Wasabi +description: Wasabi Object Storage +region: {} +endpoint: + s3.wasabisys.com: Wasabi US East 1 (N. Virginia) + s3.us-east-2.wasabisys.com: Wasabi US East 2 (N. Virginia) + s3.us-central-1.wasabisys.com: Wasabi US Central 1 (Texas) + s3.us-west-1.wasabisys.com: Wasabi US West 1 (Oregon) + s3.ca-central-1.wasabisys.com: Wasabi CA Central 1 (Toronto) + s3.eu-central-1.wasabisys.com: Wasabi EU Central 1 (Amsterdam) + s3.eu-central-2.wasabisys.com: Wasabi EU Central 2 (Frankfurt) + s3.eu-west-1.wasabisys.com: Wasabi EU West 1 (London) + s3.eu-west-2.wasabisys.com: Wasabi EU West 2 (Paris) + s3.eu-south-1.wasabisys.com: Wasabi EU South 1 (Milan) + s3.ap-northeast-1.wasabisys.com: Wasabi AP Northeast 1 (Tokyo) endpoint + s3.ap-northeast-2.wasabisys.com: Wasabi AP Northeast 2 (Osaka) endpoint + s3.ap-southeast-1.wasabisys.com: Wasabi AP Southeast 1 (Singapore) + s3.ap-southeast-2.wasabisys.com: Wasabi AP Southeast 2 (Sydney) +location_constraint: {} +acl: {} +bucket_acl: true diff --git a/backend/s3/provider/Zata.yaml b/backend/s3/provider/Zata.yaml new file mode 100644 index 000000000..ac8cb1d88 --- /dev/null +++ b/backend/s3/provider/Zata.yaml @@ -0,0 +1,14 @@ +name: Zata +description: Zata (S3 compatible Gateway) +region: + us-east-1: Indore, Madhya Pradesh, India +endpoint: + idr01.zata.ai: South Asia Endpoint +location_constraint: {} +acl: {} +bucket_acl: true +quirks: + use_multipart_etag: false + might_gzip: false + use_unsigned_payload: false + use_already_exists: false diff --git a/backend/s3/providers.go b/backend/s3/providers.go new file mode 100644 index 000000000..3d42aebe9 --- /dev/null +++ b/backend/s3/providers.go @@ -0,0 +1,236 @@ +package s3 + +import ( + "embed" + stdfs "io/fs" + "os" + "sort" + "strings" + + "github.com/rclone/rclone/fs" + orderedmap "github.com/wk8/go-ordered-map/v2" + "gopkg.in/yaml.v3" +) + +// YamlMap is converted to YAML in the correct order +type YamlMap = *orderedmap.OrderedMap[string, string] + +// NewYamlMap creates a new ordered map +var NewYamlMap = orderedmap.New[string, string] + +// Quirks defines all the S3 provider quirks +type Quirks struct { + ListVersion *int `yaml:"list_version,omitempty"` // 1 or 2 + ForcePathStyle *bool `yaml:"force_path_style,omitempty"` // true = path-style + ListURLEncode *bool `yaml:"list_url_encode,omitempty"` + UseMultipartEtag *bool `yaml:"use_multipart_etag,omitempty"` + UseAlreadyExists *bool `yaml:"use_already_exists,omitempty"` + UseAcceptEncodingGzip *bool `yaml:"use_accept_encoding_gzip,omitempty"` + MightGzip *bool `yaml:"might_gzip,omitempty"` + UseMultipartUploads *bool `yaml:"use_multipart_uploads,omitempty"` + UseUnsignedPayload *bool `yaml:"use_unsigned_payload,omitempty"` + UseXID *bool `yaml:"use_x_id,omitempty"` + SignAcceptEncoding *bool `yaml:"sign_accept_encoding,omitempty"` + CopyCutoff *int64 `yaml:"copy_cutoff,omitempty"` + MaxUploadParts *int `yaml:"max_upload_parts,omitempty"` + MinChunkSize *int64 `yaml:"min_chunk_size,omitempty"` +} + +// Provider defines the configurable data in each provider.yaml +type Provider struct { + Name string `yaml:"name,omitempty"` + Description string `yaml:"description,omitempty"` + Region YamlMap `yaml:"region,omitempty"` + Endpoint YamlMap `yaml:"endpoint,omitempty"` + LocationConstraint YamlMap `yaml:"location_constraint,omitempty"` + ACL YamlMap `yaml:"acl,omitempty"` + StorageClass YamlMap `yaml:"storage_class,omitempty"` + ServerSideEncryption YamlMap `yaml:"server_side_encryption,omitempty"` + + // other + IBMApiKey bool `yaml:"ibm_api_key,omitempty"` + IBMResourceInstanceID bool `yaml:"ibm_resource_instance_id,omitempty"` + + // advanced + BucketACL bool `yaml:"bucket_acl,omitempty"` + DirectoryBucket bool `yaml:"directory_bucket,omitempty"` + LeavePartsOnError bool `yaml:"leave_parts_on_error,omitempty"` + RequesterPays bool `yaml:"requester_pays,omitempty"` + SSECustomerAlgorithm bool `yaml:"sse_customer_algorithm,omitempty"` + SSECustomerKey bool `yaml:"sse_customer_key,omitempty"` + SSECustomerKeyBase64 bool `yaml:"sse_customer_key_base64,omitempty"` + SSECustomerKeyMd5 bool `yaml:"sse_customer_key_md5,omitempty"` + SSEKmsKeyID bool `yaml:"sse_kms_key_id,omitempty"` + STSEndpoint bool `yaml:"sts_endpoint,omitempty"` + UseAccelerateEndpoint bool `yaml:"use_accelerate_endpoint,omitempty"` + + Quirks Quirks `yaml:"quirks,omitempty"` +} + +//go:embed provider/*.yaml +var providerFS embed.FS + +// addProvidersToInfo adds provider information to the fs.RegInfo +func addProvidersToInfo(info *fs.RegInfo) *fs.RegInfo { + providerMap := loadProviders() + providerList := constructProviders(info.Options, providerMap) + info.Description += strings.TrimSuffix(providerList, ", ") + return info +} + +// loadProvider loads a single provider +// +// It returns nil if it could not be found except if "Other" which is a fatal error. +func loadProvider(name string) *Provider { + data, err := stdfs.ReadFile(providerFS, "provider/"+name+".yaml") + if err != nil { + if os.IsNotExist(err) && name != "Other" { + return nil + } + fs.Fatalf(nil, "internal error: failed to load provider %q: %v", name, err) + } + var p Provider + err = yaml.Unmarshal(data, &p) + if err != nil { + fs.Fatalf(nil, "internal error: failed to unmarshal provider %q: %v", name, err) + } + return &p +} + +// loadProviders loads provider definitions from embedded YAML files +func loadProviders() map[string]*Provider { + providers, err := stdfs.ReadDir(providerFS, "provider") + if err != nil { + fs.Fatalf(nil, "internal error: failed to read embedded providers: %v", err) + } + providerMap := make(map[string]*Provider, len(providers)) + + for _, provider := range providers { + name, _ := strings.CutSuffix(provider.Name(), ".yaml") + p := loadProvider(name) + providerMap[p.Name] = p + } + return providerMap +} + +// constructProviders populates fs.Options with provider-specific examples and information +func constructProviders(options fs.Options, providerMap map[string]*Provider) string { + // Defaults for map options set to {} + defaults := providerMap["Other"] + + // sort providers: AWS first, Other last, rest alphabetically + providers := make([]*Provider, 0, len(providerMap)) + for _, p := range providerMap { + providers = append(providers, p) + } + sort.Slice(providers, func(i, j int) bool { + if providers[i].Name == "AWS" { + return true + } + if providers[j].Name == "AWS" { + return false + } + if providers[i].Name == "Other" { + return false + } + if providers[j].Name == "Other" { + return true + } + return strings.ToLower(providers[i].Name) < strings.ToLower(providers[j].Name) + }) + + addProvider := func(sp *string, name string) { + if *sp != "" { + *sp += "," + } + *sp += name + } + + addBool := func(opt *fs.Option, p *Provider, flag bool) { + if flag { + addProvider(&opt.Provider, p.Name) + } + } + + addExample := func(opt *fs.Option, p *Provider, examples, defaultExamples YamlMap) { + if examples == nil { + return + } + if examples.Len() == 0 { + examples = defaultExamples + } + addProvider(&opt.Provider, p.Name) + OUTER: + for pair := examples.Oldest(); pair != nil; pair = pair.Next() { + // Find an existing example to add to if possible + for i, example := range opt.Examples { + if example.Value == pair.Key && example.Help == pair.Value { + addProvider(&opt.Examples[i].Provider, p.Name) + continue OUTER + } + } + // Otherwise add a new one + opt.Examples = append(opt.Examples, fs.OptionExample{ + Value: pair.Key, + Help: pair.Value, + Provider: p.Name, + }) + } + } + + var providerList string + + for _, p := range providers { + for i := range options { + opt := &options[i] + switch opt.Name { + case "provider": + opt.Examples = append(opt.Examples, fs.OptionExample{ + Value: p.Name, + Help: p.Description, + }) + providerList += p.Name + ", " + case "region": + addExample(opt, p, p.Region, defaults.Region) + case "endpoint": + addExample(opt, p, p.Endpoint, defaults.Endpoint) + case "location_constraint": + addExample(opt, p, p.LocationConstraint, defaults.LocationConstraint) + case "acl": + addExample(opt, p, p.ACL, defaults.ACL) + case "storage_class": + addExample(opt, p, p.StorageClass, defaults.StorageClass) + case "server_side_encryption": + addExample(opt, p, p.ServerSideEncryption, defaults.ServerSideEncryption) + case "bucket_acl": + addBool(opt, p, p.BucketACL) + case "requester_pays": + addBool(opt, p, p.RequesterPays) + case "sse_customer_algorithm": + addBool(opt, p, p.SSECustomerAlgorithm) + case "sse_kms_key_id": + addBool(opt, p, p.SSEKmsKeyID) + case "sse_customer_key": + addBool(opt, p, p.SSECustomerKey) + case "sse_customer_key_base64": + addBool(opt, p, p.SSECustomerKeyBase64) + case "sse_customer_key_md5": + addBool(opt, p, p.SSECustomerKeyMd5) + case "directory_bucket": + addBool(opt, p, p.DirectoryBucket) + case "ibm_api_key": + addBool(opt, p, p.IBMApiKey) + case "ibm_resource_instance_id": + addBool(opt, p, p.IBMResourceInstanceID) + case "leave_parts_on_error": + addBool(opt, p, p.LeavePartsOnError) + case "sts_endpoint": + addBool(opt, p, p.STSEndpoint) + case "use_accelerate_endpoint": + addBool(opt, p, p.UseAccelerateEndpoint) + } + } + } + + return strings.TrimSuffix(providerList, ", ") +} diff --git a/backend/s3/s3.go b/backend/s3/s3.go index 9ddae5083..79c618566 100644 --- a/backend/s3/s3.go +++ b/backend/s3/s3.go @@ -63,180 +63,11 @@ import ( "golang.org/x/sync/errgroup" ) -// The S3 providers -// -// Please keep these in alphabetical order, but with AWS first and -// Other last. -// -// NB if you add a new provider here, then add it in the setQuirks -// function and set the correct quirks. Test the quirks are correct by -// running the integration tests "go test -v -remote NewS3Provider:". -// -// See https://github.com/rclone/rclone/blob/master/CONTRIBUTING.md#adding-a-new-s3-provider -// for full information about how to add a new s3 provider. -var providerOption = fs.Option{ - Name: fs.ConfigProvider, - Help: "Choose your S3 provider.", - Examples: []fs.OptionExample{{ - Value: "AWS", - Help: "Amazon Web Services (AWS) S3", - }, { - Value: "Alibaba", - Help: "Alibaba Cloud Object Storage System (OSS) formerly Aliyun", - }, { - Value: "ArvanCloud", - Help: "Arvan Cloud Object Storage (AOS)", - }, { - Value: "Ceph", - Help: "Ceph Object Storage", - }, { - Value: "ChinaMobile", - Help: "China Mobile Ecloud Elastic Object Storage (EOS)", - }, { - Value: "Cloudflare", - Help: "Cloudflare R2 Storage", - }, { - Value: "Cubbit", - Help: "Cubbit DS3 Object Storage", - }, { - Value: "DigitalOcean", - Help: "DigitalOcean Spaces", - }, { - Value: "Dreamhost", - Help: "Dreamhost DreamObjects", - }, { - Value: "Exaba", - Help: "Exaba Object Storage", - }, { - Value: "FileLu", - Help: "FileLu S5 (S3-Compatible Object Storage)", - }, { - Value: "FlashBlade", - Help: "Pure Storage FlashBlade Object Storage", - }, { - Value: "GCS", - Help: "Google Cloud Storage", - }, { - Value: "Hetzner", - Help: "Hetzner Object Storage", - }, { - Value: "HuaweiOBS", - Help: "Huawei Object Storage Service", - }, { - Value: "IBMCOS", - Help: "IBM COS S3", - }, { - Value: "IDrive", - Help: "IDrive e2", - }, { - Value: "Intercolo", - Help: "Intercolo Object Storage", - }, { - Value: "IONOS", - Help: "IONOS Cloud", - }, { - Value: "LyveCloud", - Help: "Seagate Lyve Cloud", - }, { - Value: "Leviia", - Help: "Leviia Object Storage", - }, { - Value: "Liara", - Help: "Liara Object Storage", - }, { - Value: "Linode", - Help: "Linode Object Storage", - }, { - Value: "Magalu", - Help: "Magalu Object Storage", - }, { - Value: "Mega", - Help: "MEGA S4 Object Storage", - }, { - Value: "Minio", - Help: "Minio Object Storage", - }, { - Value: "Netease", - Help: "Netease Object Storage (NOS)", - }, { - Value: "Outscale", - Help: "OUTSCALE Object Storage (OOS)", - }, { - Value: "OVHcloud", - Help: "OVHcloud Object Storage", - }, { - Value: "Petabox", - Help: "Petabox Object Storage", - }, { - Value: "Rabata", - Help: "Rabata Cloud Storage", - }, { - Value: "RackCorp", - Help: "RackCorp Object Storage", - }, { - Value: "Rclone", - Help: "Rclone S3 Server", - }, { - Value: "Scaleway", - Help: "Scaleway Object Storage", - }, { - Value: "SeaweedFS", - Help: "SeaweedFS S3", - }, { - Value: "Selectel", - Help: "Selectel Object Storage", - }, { - Value: "Servercore", - Help: "Servercore Object Storage", - }, { - Value: "SpectraLogic", - Help: "Spectra Logic Black Pearl", - }, { - Value: "StackPath", - Help: "StackPath Object Storage", - }, { - Value: "Storj", - Help: "Storj (S3 Compatible Gateway)", - }, { - Value: "Synology", - Help: "Synology C2 Object Storage", - }, { - Value: "TencentCOS", - Help: "Tencent Cloud Object Storage (COS)", - }, { - Value: "Wasabi", - Help: "Wasabi Object Storage", - }, { - Value: "Qiniu", - Help: "Qiniu Object Storage (Kodo)", - }, { - Value: "Zata", - Help: "Zata (S3 compatible Gateway)", - }, { - Value: "Other", - Help: "Any other S3 compatible provider", - }}, -} - -var providersList string - // Register with Fs func init() { - var s strings.Builder - for i, provider := range providerOption.Examples { - if provider.Value == "Other" { - _, _ = s.WriteString(" and others") - } else { - if i != 0 { - _, _ = s.WriteString(", ") - } - _, _ = s.WriteString(provider.Value) - } - } - providersList = s.String() - fs.Register(&fs.RegInfo{ + fs.Register(addProvidersToInfo(&fs.RegInfo{ Name: "s3", - Description: "Amazon S3 Compliant Storage Providers including " + providersList, + Description: "Amazon S3 Compliant Storage Providers including ", NewFs: NewFs, CommandHelp: commandHelp, Config: func(ctx context.Context, name string, m configmap.Mapper, config fs.ConfigIn) (*fs.ConfigOut, error) { @@ -250,7 +81,10 @@ func init() { System: systemMetadataInfo, Help: `User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.`, }, - Options: []fs.Option{providerOption, { + Options: []fs.Option{{ + Name: fs.ConfigProvider, + Help: "Choose your S3 provider.", + }, { Name: "env_auth", Help: "Get AWS credentials from runtime (environment variables or EC2/ECS meta data if no env vars).\n\nOnly applies if access_key_id and secret_access_key is blank.", Default: false, @@ -270,2007 +104,14 @@ func init() { Help: "AWS Secret Access Key (password).\n\nLeave blank for anonymous access or runtime credentials.", Sensitive: true, }, { - // References: - // 1. https://docs.aws.amazon.com/general/latest/gr/rande.html - // 2. https://docs.aws.amazon.com/general/latest/gr/s3.html - Name: "region", - Help: "Region to connect to.", - Provider: "AWS", - Examples: []fs.OptionExample{{ - Value: "us-east-1", - Help: "The default endpoint - a good choice if you are unsure.\nUS Region, Northern Virginia, or Pacific Northwest.\nLeave location constraint empty.", - }, { - Value: "us-east-2", - Help: "US East (Ohio) Region.\nNeeds location constraint us-east-2.", - }, { - Value: "us-west-1", - Help: "US West (Northern California) Region.\nNeeds location constraint us-west-1.", - }, { - Value: "us-west-2", - Help: "US West (Oregon) Region.\nNeeds location constraint us-west-2.", - }, { - Value: "ca-central-1", - Help: "Canada (Central) Region.\nNeeds location constraint ca-central-1.", - }, { - Value: "eu-west-1", - Help: "EU (Ireland) Region.\nNeeds location constraint EU or eu-west-1.", - }, { - Value: "eu-west-2", - Help: "EU (London) Region.\nNeeds location constraint eu-west-2.", - }, { - Value: "eu-west-3", - Help: "EU (Paris) Region.\nNeeds location constraint eu-west-3.", - }, { - Value: "eu-north-1", - Help: "EU (Stockholm) Region.\nNeeds location constraint eu-north-1.", - }, { - Value: "eu-south-1", - Help: "EU (Milan) Region.\nNeeds location constraint eu-south-1.", - }, { - Value: "eu-central-1", - Help: "EU (Frankfurt) Region.\nNeeds location constraint eu-central-1.", - }, { - Value: "ap-southeast-1", - Help: "Asia Pacific (Singapore) Region.\nNeeds location constraint ap-southeast-1.", - }, { - Value: "ap-southeast-2", - Help: "Asia Pacific (Sydney) Region.\nNeeds location constraint ap-southeast-2.", - }, { - Value: "ap-northeast-1", - Help: "Asia Pacific (Tokyo) Region.\nNeeds location constraint ap-northeast-1.", - }, { - Value: "ap-northeast-2", - Help: "Asia Pacific (Seoul).\nNeeds location constraint ap-northeast-2.", - }, { - Value: "ap-northeast-3", - Help: "Asia Pacific (Osaka-Local).\nNeeds location constraint ap-northeast-3.", - }, { - Value: "ap-south-1", - Help: "Asia Pacific (Mumbai).\nNeeds location constraint ap-south-1.", - }, { - Value: "ap-east-1", - Help: "Asia Pacific (Hong Kong) Region.\nNeeds location constraint ap-east-1.", - }, { - Value: "sa-east-1", - Help: "South America (Sao Paulo) Region.\nNeeds location constraint sa-east-1.", - }, { - Value: "il-central-1", - Help: "Israel (Tel Aviv) Region.\nNeeds location constraint il-central-1.", - }, { - Value: "me-south-1", - Help: "Middle East (Bahrain) Region.\nNeeds location constraint me-south-1.", - }, { - Value: "af-south-1", - Help: "Africa (Cape Town) Region.\nNeeds location constraint af-south-1.", - }, { - Value: "cn-north-1", - Help: "China (Beijing) Region.\nNeeds location constraint cn-north-1.", - }, { - Value: "cn-northwest-1", - Help: "China (Ningxia) Region.\nNeeds location constraint cn-northwest-1.", - }, { - Value: "us-gov-east-1", - Help: "AWS GovCloud (US-East) Region.\nNeeds location constraint us-gov-east-1.", - }, { - Value: "us-gov-west-1", - Help: "AWS GovCloud (US) Region.\nNeeds location constraint us-gov-west-1.", - }}, + Name: "region", + Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.", }, { - Name: "region", - Help: "Region to connect to.", - Provider: "Cloudflare", - Examples: []fs.OptionExample{{ - Value: "auto", - Help: "R2 buckets are automatically distributed across Cloudflare's data centers for low latency.", - }}, + Name: "endpoint", + Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.", }, { - Name: "region", - Help: "Region to connect to.", - Provider: "Cubbit", - Examples: []fs.OptionExample{{ - Value: "eu-west-1", - Help: "Europe West", - }}, - }, { - Name: "region", - Help: "Region to connect to for FileLu S5.", - Provider: "FileLu", - Examples: []fs.OptionExample{{ - Value: "global", - Help: "Global", - }, { - Value: "us-east", - Help: "North America (US-East)", - }, { - Value: "eu-central", - Help: "Europe (EU-Central)", - }, { - Value: "ap-southeast", - Help: "Asia Pacific (AP-Southeast)", - }, { - Value: "me-central", - Help: "Middle East (ME-Central)", - }}, - }, { - Name: "region", - Help: "Region to connect to.", - Provider: "Hetzner", - Examples: []fs.OptionExample{{ - Value: "hel1", - Help: "Helsinki", - }, { - Value: "fsn1", - Help: "Falkenstein", - }, { - Value: "nbg1", - Help: "Nuremberg", - }}, - }, { - Name: "region", - Help: "Region to connect to. - the location where your bucket will be created and your data stored. Need to be same with your endpoint.\n", - Provider: "HuaweiOBS", - Examples: []fs.OptionExample{{ - Value: "af-south-1", - Help: "AF-Johannesburg", - }, { - Value: "ap-southeast-2", - Help: "AP-Bangkok", - }, { - Value: "ap-southeast-3", - Help: "AP-Singapore", - }, { - Value: "cn-east-3", - Help: "CN East-Shanghai1", - }, { - Value: "cn-east-2", - Help: "CN East-Shanghai2", - }, { - Value: "cn-north-1", - Help: "CN North-Beijing1", - }, { - Value: "cn-north-4", - Help: "CN North-Beijing4", - }, { - Value: "cn-south-1", - Help: "CN South-Guangzhou", - }, { - Value: "ap-southeast-1", - Help: "CN-Hong Kong", - }, { - Value: "sa-argentina-1", - Help: "LA-Buenos Aires1", - }, { - Value: "sa-peru-1", - Help: "LA-Lima1", - }, { - Value: "na-mexico-1", - Help: "LA-Mexico City1", - }, { - Value: "sa-chile-1", - Help: "LA-Santiago2", - }, { - Value: "sa-brazil-1", - Help: "LA-Sao Paulo1", - }, { - Value: "ru-northwest-2", - Help: "RU-Moscow2", - }}, - }, { - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "Intercolo", - Examples: []fs.OptionExample{{ - Value: "de-fra", - Help: "Frankfurt, Germany", - }}, - }, { - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "IONOS", - Examples: []fs.OptionExample{{ - Value: "de", - Help: "Frankfurt, Germany", - }, { - Value: "eu-central-2", - Help: "Berlin, Germany", - }, { - Value: "eu-south-2", - Help: "Logrono, Spain", - }}, - }, { - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "Outscale", - Examples: []fs.OptionExample{{ - Value: "eu-west-2", - Help: "Paris, France", - }, { - Value: "us-east-2", - Help: "New Jersey, USA", - }, { - Value: "us-west-1", - Help: "California, USA", - }, { - Value: "cloudgouv-eu-west-1", - Help: "SecNumCloud, Paris, France", - }, { - Value: "ap-northeast-1", - Help: "Tokyo, Japan", - }}, - }, { - // References: - // https://help.ovhcloud.com/csm/en-public-cloud-storage-s3-location?id=kb_article_view&sysparm_article=KB0047384 - // https://support.us.ovhcloud.com/hc/en-us/articles/10667991081107-Endpoints-and-Object-Storage-Geoavailability - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "OVHcloud", - Examples: []fs.OptionExample{{ - Value: "gra", - Help: "Gravelines, France", - }, { - Value: "rbx", - Help: "Roubaix, France", - }, { - Value: "sbg", - Help: "Strasbourg, France", - }, { - Value: "eu-west-par", - Help: "Paris, France (3AZ)", - }, { - Value: "de", - Help: "Frankfurt, Germany", - }, { - Value: "uk", - Help: "London, United Kingdom", - }, { - Value: "waw", - Help: "Warsaw, Poland", - }, { - Value: "bhs", - Help: "Beauharnois, Canada", - }, { - Value: "ca-east-tor", - Help: "Toronto, Canada", - }, { - Value: "sgp", - Help: "Singapore", - }, { - Value: "ap-southeast-syd", - Help: "Sydney, Australia", - }, { - Value: "ap-south-mum", - Help: "Mumbai, India", - }, { - Value: "us-east-va", - Help: "Vint Hill, Virginia, USA", - }, { - Value: "us-west-or", - Help: "Hillsboro, Oregon, USA", - }, { - Value: "rbx-archive", - Help: "Roubaix, France (Cold Archive)", - }}, - }, { - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "Petabox", - Examples: []fs.OptionExample{{ - Value: "us-east-1", - Help: "US East (N. Virginia)", - }, { - Value: "eu-central-1", - Help: "Europe (Frankfurt)", - }, { - Value: "ap-southeast-1", - Help: "Asia Pacific (Singapore)", - }, { - Value: "me-south-1", - Help: "Middle East (Bahrain)", - }, { - Value: "sa-east-1", - Help: "South America (São Paulo)", - }}, - }, { - // References: - // https://developer.qiniu.com/kodo/4088/s3-access-domainname - Name: "region", - Help: "Region to connect to.", - Provider: "Qiniu", - Examples: []fs.OptionExample{{ - Value: "cn-east-1", - Help: "The default endpoint - a good choice if you are unsure.\nEast China Region 1.\nNeeds location constraint cn-east-1.", - }, { - Value: "cn-east-2", - Help: "East China Region 2.\nNeeds location constraint cn-east-2.", - }, { - Value: "cn-north-1", - Help: "North China Region 1.\nNeeds location constraint cn-north-1.", - }, { - Value: "cn-south-1", - Help: "South China Region 1.\nNeeds location constraint cn-south-1.", - }, { - Value: "us-north-1", - Help: "North America Region.\nNeeds location constraint us-north-1.", - }, { - Value: "ap-southeast-1", - Help: "Southeast Asia Region 1.\nNeeds location constraint ap-southeast-1.", - }, { - Value: "ap-northeast-1", - Help: "Northeast Asia Region 1.\nNeeds location constraint ap-northeast-1.", - }}, - }, { - Name: "region", - Help: "Region where your bucket will be created and your data stored.\n", - Provider: "Rabata", - Examples: []fs.OptionExample{{ - Value: "us-east-1", - Help: "US East (N. Virginia)", - }, { - Value: "eu-west-1", - Help: "EU (Ireland)", - }, { - Value: "eu-west-2", - Help: "EU (London)", - }}, - }, { - Name: "region", - Help: "region - the location where your bucket will be created and your data stored.\n", - Provider: "RackCorp", - Examples: []fs.OptionExample{{ - Value: "global", - Help: "Global CDN (All locations) Region", - }, { - Value: "au", - Help: "Australia (All states)", - }, { - Value: "au-nsw", - Help: "NSW (Australia) Region", - }, { - Value: "au-qld", - Help: "QLD (Australia) Region", - }, { - Value: "au-vic", - Help: "VIC (Australia) Region", - }, { - Value: "au-wa", - Help: "Perth (Australia) Region", - }, { - Value: "ph", - Help: "Manila (Philippines) Region", - }, { - Value: "th", - Help: "Bangkok (Thailand) Region", - }, { - Value: "hk", - Help: "HK (Hong Kong) Region", - }, { - Value: "mn", - Help: "Ulaanbaatar (Mongolia) Region", - }, { - Value: "kg", - Help: "Bishkek (Kyrgyzstan) Region", - }, { - Value: "id", - Help: "Jakarta (Indonesia) Region", - }, { - Value: "jp", - Help: "Tokyo (Japan) Region", - }, { - Value: "sg", - Help: "SG (Singapore) Region", - }, { - Value: "de", - Help: "Frankfurt (Germany) Region", - }, { - Value: "us", - Help: "USA (AnyCast) Region", - }, { - Value: "us-east-1", - Help: "New York (USA) Region", - }, { - Value: "us-west-1", - Help: "Freemont (USA) Region", - }, { - Value: "nz", - Help: "Auckland (New Zealand) Region", - }}, - }, { - Name: "region", - Help: "Region to connect to.", - Provider: "Scaleway", - Examples: []fs.OptionExample{{ - Value: "nl-ams", - Help: "Amsterdam, The Netherlands", - }, { - Value: "fr-par", - Help: "Paris, France", - }, { - Value: "pl-waw", - Help: "Warsaw, Poland", - }}, - }, { - // See endpoints for object storage regions: https://docs.selectel.ru/en/cloud/object-storage/manage/domains/#s3-api-domains - Name: "region", - Help: "Region where your data stored.\n", - Provider: "Selectel", - Examples: []fs.OptionExample{{ - Value: "ru-1", - Help: "St. Petersburg", - }}, - }, { - Name: "region", - Help: "Region where your is data stored.\n", - Provider: "Servercore", - Examples: []fs.OptionExample{{ - Value: "ru-1", - Help: "St. Petersburg", - }, { - Value: "gis-1", - Help: "Moscow", - }, { - Value: "ru-7", - Help: "Moscow", - }, { - Value: "uz-2", - Help: "Tashkent, Uzbekistan", - }, { - Value: "kz-1", - Help: "Almaty, Kazakhstan", - }}, - }, { - Name: "region", - Help: "Region where your data stored.\n", - Provider: "Synology", - Examples: []fs.OptionExample{{ - Value: "eu-001", - Help: "Europe Region 1", - }, { - Value: "eu-002", - Help: "Europe Region 2", - }, { - Value: "us-001", - Help: "US Region 1", - }, { - Value: "us-002", - Help: "US Region 2", - }, { - Value: "tw-001", - Help: "Asia (Taiwan)", - }}, - }, { - Name: "region", - Help: "Region where you can connect with.\n", - Provider: "Zata", - Examples: []fs.OptionExample{{ - Value: "us-east-1", - Help: "Indore, Madhya Pradesh, India", - }}, - }, { - Name: "region", - Help: "Region to connect to.\n\nLeave blank if you are using an S3 clone and you don't have a region.", - Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,Cubbit,FlashBlade,FileLu,Hetzner,HuaweiOBS,IDrive,Intercolo,IONOS,Liara,Linode,Magalu,Mega,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,SpectraLogic,Storj,Synology,TencentCOS,Zata", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Use this if unsure.\nWill use v4 signatures and an empty region.", - }, { - Value: "other-v2-signature", - Help: "Use this only if v4 signatures don't work.\nE.g. pre Jewel/v10 CEPH.", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for S3 API.\n\nLeave blank if using AWS to use the default endpoint for the region.", - Provider: "AWS", - }, { - // oss endpoints: https://help.aliyun.com/document_detail/31837.html - Name: "endpoint", - Help: "Endpoint for OSS API.", - Provider: "Alibaba", - Examples: []fs.OptionExample{{ - Value: "oss-accelerate.aliyuncs.com", - Help: "Global Accelerate", - }, { - Value: "oss-accelerate-overseas.aliyuncs.com", - Help: "Global Accelerate (outside mainland China)", - }, { - Value: "oss-cn-hangzhou.aliyuncs.com", - Help: "East China 1 (Hangzhou)", - }, { - Value: "oss-cn-shanghai.aliyuncs.com", - Help: "East China 2 (Shanghai)", - }, { - Value: "oss-cn-qingdao.aliyuncs.com", - Help: "North China 1 (Qingdao)", - }, { - Value: "oss-cn-beijing.aliyuncs.com", - Help: "North China 2 (Beijing)", - }, { - Value: "oss-cn-zhangjiakou.aliyuncs.com", - Help: "North China 3 (Zhangjiakou)", - }, { - Value: "oss-cn-huhehaote.aliyuncs.com", - Help: "North China 5 (Hohhot)", - }, { - Value: "oss-cn-wulanchabu.aliyuncs.com", - Help: "North China 6 (Ulanqab)", - }, { - Value: "oss-cn-shenzhen.aliyuncs.com", - Help: "South China 1 (Shenzhen)", - }, { - Value: "oss-cn-heyuan.aliyuncs.com", - Help: "South China 2 (Heyuan)", - }, { - Value: "oss-cn-guangzhou.aliyuncs.com", - Help: "South China 3 (Guangzhou)", - }, { - Value: "oss-cn-chengdu.aliyuncs.com", - Help: "West China 1 (Chengdu)", - }, { - Value: "oss-cn-hongkong.aliyuncs.com", - Help: "Hong Kong (Hong Kong)", - }, { - Value: "oss-us-west-1.aliyuncs.com", - Help: "US West 1 (Silicon Valley)", - }, { - Value: "oss-us-east-1.aliyuncs.com", - Help: "US East 1 (Virginia)", - }, { - Value: "oss-ap-southeast-1.aliyuncs.com", - Help: "Southeast Asia Southeast 1 (Singapore)", - }, { - Value: "oss-ap-southeast-2.aliyuncs.com", - Help: "Asia Pacific Southeast 2 (Sydney)", - }, { - Value: "oss-ap-southeast-3.aliyuncs.com", - Help: "Southeast Asia Southeast 3 (Kuala Lumpur)", - }, { - Value: "oss-ap-southeast-5.aliyuncs.com", - Help: "Asia Pacific Southeast 5 (Jakarta)", - }, { - Value: "oss-ap-northeast-1.aliyuncs.com", - Help: "Asia Pacific Northeast 1 (Japan)", - }, { - Value: "oss-ap-south-1.aliyuncs.com", - Help: "Asia Pacific South 1 (Mumbai)", - }, { - Value: "oss-eu-central-1.aliyuncs.com", - Help: "Central Europe 1 (Frankfurt)", - }, { - Value: "oss-eu-west-1.aliyuncs.com", - Help: "West Europe (London)", - }, { - Value: "oss-me-east-1.aliyuncs.com", - Help: "Middle East 1 (Dubai)", - }}, - }, { - // ArvanCloud endpoints: https://www.arvancloud.ir/en/products/cloud-storage - Name: "endpoint", - Help: "Endpoint for Arvan Cloud Object Storage (AOS) API.", - Provider: "ArvanCloud", - Examples: []fs.OptionExample{{ - Value: "s3.ir-thr-at1.arvanstorage.ir", - Help: "The default endpoint - a good choice if you are unsure.\nTehran Iran (Simin)", - }, { - Value: "s3.ir-tbz-sh1.arvanstorage.ir", - Help: "Tabriz Iran (Shahriar)", - }}, - }, { - // ChinaMobile endpoints: https://ecloud.10086.cn/op-help-center/doc/article/24534 - Name: "endpoint", - Help: "Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.", - Provider: "ChinaMobile", - Examples: []fs.OptionExample{{ - Value: "eos-wuxi-1.cmecloud.cn", - Help: "The default endpoint - a good choice if you are unsure.\nEast China (Suzhou)", - }, { - Value: "eos-jinan-1.cmecloud.cn", - Help: "East China (Jinan)", - }, { - Value: "eos-ningbo-1.cmecloud.cn", - Help: "East China (Hangzhou)", - }, { - Value: "eos-shanghai-1.cmecloud.cn", - Help: "East China (Shanghai-1)", - }, { - Value: "eos-zhengzhou-1.cmecloud.cn", - Help: "Central China (Zhengzhou)", - }, { - Value: "eos-hunan-1.cmecloud.cn", - Help: "Central China (Changsha-1)", - }, { - Value: "eos-zhuzhou-1.cmecloud.cn", - Help: "Central China (Changsha-2)", - }, { - Value: "eos-guangzhou-1.cmecloud.cn", - Help: "South China (Guangzhou-2)", - }, { - Value: "eos-dongguan-1.cmecloud.cn", - Help: "South China (Guangzhou-3)", - }, { - Value: "eos-beijing-1.cmecloud.cn", - Help: "North China (Beijing-1)", - }, { - Value: "eos-beijing-2.cmecloud.cn", - Help: "North China (Beijing-2)", - }, { - Value: "eos-beijing-4.cmecloud.cn", - Help: "North China (Beijing-3)", - }, { - Value: "eos-huhehaote-1.cmecloud.cn", - Help: "North China (Huhehaote)", - }, { - Value: "eos-chengdu-1.cmecloud.cn", - Help: "Southwest China (Chengdu)", - }, { - Value: "eos-chongqing-1.cmecloud.cn", - Help: "Southwest China (Chongqing)", - }, { - Value: "eos-guiyang-1.cmecloud.cn", - Help: "Southwest China (Guiyang)", - }, { - Value: "eos-xian-1.cmecloud.cn", - Help: "Nouthwest China (Xian)", - }, { - Value: "eos-yunnan.cmecloud.cn", - Help: "Yunnan China (Kunming)", - }, { - Value: "eos-yunnan-2.cmecloud.cn", - Help: "Yunnan China (Kunming-2)", - }, { - Value: "eos-tianjin-1.cmecloud.cn", - Help: "Tianjin China (Tianjin)", - }, { - Value: "eos-jilin-1.cmecloud.cn", - Help: "Jilin China (Changchun)", - }, { - Value: "eos-hubei-1.cmecloud.cn", - Help: "Hubei China (Xiangyan)", - }, { - Value: "eos-jiangxi-1.cmecloud.cn", - Help: "Jiangxi China (Nanchang)", - }, { - Value: "eos-gansu-1.cmecloud.cn", - Help: "Gansu China (Lanzhou)", - }, { - Value: "eos-shanxi-1.cmecloud.cn", - Help: "Shanxi China (Taiyuan)", - }, { - Value: "eos-liaoning-1.cmecloud.cn", - Help: "Liaoning China (Shenyang)", - }, { - Value: "eos-hebei-1.cmecloud.cn", - Help: "Hebei China (Shijiazhuang)", - }, { - Value: "eos-fujian-1.cmecloud.cn", - Help: "Fujian China (Xiamen)", - }, { - Value: "eos-guangxi-1.cmecloud.cn", - Help: "Guangxi China (Nanning)", - }, { - Value: "eos-anhui-1.cmecloud.cn", - Help: "Anhui China (Huainan)", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Cubbit DS3 Object Storage.", - Provider: "Cubbit", - Examples: []fs.OptionExample{{ - Value: "s3.cubbit.eu", - Help: "Cubbit DS3 Object Storage endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for FileLu S5 Object Storage.\nRequired when using FileLu S5.", - Provider: "FileLu", - Examples: []fs.OptionExample{{ - Value: "s5lu.com", - Help: "Global FileLu S5 endpoint", - }, { - Value: "us.s5lu.com", - Help: "North America (US-East) region endpoint", - }, { - Value: "eu.s5lu.com", - Help: "Europe (EU-Central) region endpoint", - }, { - Value: "ap.s5lu.com", - Help: "Asia Pacific (AP-Southeast) region endpoint", - }, { - Value: "me.s5lu.com", - Help: "Middle East (ME-Central) region endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Google Cloud Storage.", - Provider: "GCS", - Examples: []fs.OptionExample{{ - Value: "https://storage.googleapis.com", - Help: "Google Cloud Storage endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Hetzner Object Storage", - Provider: "Hetzner", - Examples: []fs.OptionExample{{ - Value: "hel1.your-objectstorage.com", - Help: "Helsinki", - }, { - Value: "fsn1.your-objectstorage.com", - Help: "Falkenstein", - }, { - Value: "nbg1.your-objectstorage.com", - Help: "Nuremberg", - }}, - }, { - // obs endpoints: https://developer.huaweicloud.com/intl/en-us/endpoint?OBS - Name: "endpoint", - Help: "Endpoint for OBS API.", - Provider: "HuaweiOBS", - Examples: []fs.OptionExample{{ - Value: "obs.af-south-1.myhuaweicloud.com", - Help: "AF-Johannesburg", - }, { - Value: "obs.ap-southeast-2.myhuaweicloud.com", - Help: "AP-Bangkok", - }, { - Value: "obs.ap-southeast-3.myhuaweicloud.com", - Help: "AP-Singapore", - }, { - Value: "obs.cn-east-3.myhuaweicloud.com", - Help: "CN East-Shanghai1", - }, { - Value: "obs.cn-east-2.myhuaweicloud.com", - Help: "CN East-Shanghai2", - }, { - Value: "obs.cn-north-1.myhuaweicloud.com", - Help: "CN North-Beijing1", - }, { - Value: "obs.cn-north-4.myhuaweicloud.com", - Help: "CN North-Beijing4", - }, { - Value: "obs.cn-south-1.myhuaweicloud.com", - Help: "CN South-Guangzhou", - }, { - Value: "obs.ap-southeast-1.myhuaweicloud.com", - Help: "CN-Hong Kong", - }, { - Value: "obs.sa-argentina-1.myhuaweicloud.com", - Help: "LA-Buenos Aires1", - }, { - Value: "obs.sa-peru-1.myhuaweicloud.com", - Help: "LA-Lima1", - }, { - Value: "obs.na-mexico-1.myhuaweicloud.com", - Help: "LA-Mexico City1", - }, { - Value: "obs.sa-chile-1.myhuaweicloud.com", - Help: "LA-Santiago2", - }, { - Value: "obs.sa-brazil-1.myhuaweicloud.com", - Help: "LA-Sao Paulo1", - }, { - Value: "obs.ru-northwest-2.myhuaweicloud.com", - Help: "RU-Moscow2", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for IBM COS S3 API.\n\nSpecify if using an IBM COS On Premise.", - Provider: "IBMCOS", - Examples: []fs.OptionExample{{ - Value: "s3.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Endpoint", - }, { - Value: "s3.dal.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Dallas Endpoint", - }, { - Value: "s3.wdc.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Washington DC Endpoint", - }, { - Value: "s3.sjc.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region San Jose Endpoint", - }, { - Value: "s3.private.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Private Endpoint", - }, { - Value: "s3.private.dal.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Dallas Private Endpoint", - }, { - Value: "s3.private.wdc.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region Washington DC Private Endpoint", - }, { - Value: "s3.private.sjc.us.cloud-object-storage.appdomain.cloud", - Help: "US Cross Region San Jose Private Endpoint", - }, { - Value: "s3.us-east.cloud-object-storage.appdomain.cloud", - Help: "US Region East Endpoint", - }, { - Value: "s3.private.us-east.cloud-object-storage.appdomain.cloud", - Help: "US Region East Private Endpoint", - }, { - Value: "s3.us-south.cloud-object-storage.appdomain.cloud", - Help: "US Region South Endpoint", - }, { - Value: "s3.private.us-south.cloud-object-storage.appdomain.cloud", - Help: "US Region South Private Endpoint", - }, { - Value: "s3.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Endpoint", - }, { - Value: "s3.fra.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Frankfurt Endpoint", - }, { - Value: "s3.mil.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Milan Endpoint", - }, { - Value: "s3.ams.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Amsterdam Endpoint", - }, { - Value: "s3.private.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Private Endpoint", - }, { - Value: "s3.private.fra.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Frankfurt Private Endpoint", - }, { - Value: "s3.private.mil.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Milan Private Endpoint", - }, { - Value: "s3.private.ams.eu.cloud-object-storage.appdomain.cloud", - Help: "EU Cross Region Amsterdam Private Endpoint", - }, { - Value: "s3.eu-gb.cloud-object-storage.appdomain.cloud", - Help: "Great Britain Endpoint", - }, { - Value: "s3.private.eu-gb.cloud-object-storage.appdomain.cloud", - Help: "Great Britain Private Endpoint", - }, { - Value: "s3.eu-de.cloud-object-storage.appdomain.cloud", - Help: "EU Region DE Endpoint", - }, { - Value: "s3.private.eu-de.cloud-object-storage.appdomain.cloud", - Help: "EU Region DE Private Endpoint", - }, { - Value: "s3.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Endpoint", - }, { - Value: "s3.tok.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Tokyo Endpoint", - }, { - Value: "s3.hkg.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Hong Kong Endpoint", - }, { - Value: "s3.seo.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Seoul Endpoint", - }, { - Value: "s3.private.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Private Endpoint", - }, { - Value: "s3.private.tok.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Tokyo Private Endpoint", - }, { - Value: "s3.private.hkg.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Hong Kong Private Endpoint", - }, { - Value: "s3.private.seo.ap.cloud-object-storage.appdomain.cloud", - Help: "APAC Cross Regional Seoul Private Endpoint", - }, { - Value: "s3.jp-tok.cloud-object-storage.appdomain.cloud", - Help: "APAC Region Japan Endpoint", - }, { - Value: "s3.private.jp-tok.cloud-object-storage.appdomain.cloud", - Help: "APAC Region Japan Private Endpoint", - }, { - Value: "s3.au-syd.cloud-object-storage.appdomain.cloud", - Help: "APAC Region Australia Endpoint", - }, { - Value: "s3.private.au-syd.cloud-object-storage.appdomain.cloud", - Help: "APAC Region Australia Private Endpoint", - }, { - Value: "s3.ams03.cloud-object-storage.appdomain.cloud", - Help: "Amsterdam Single Site Endpoint", - }, { - Value: "s3.private.ams03.cloud-object-storage.appdomain.cloud", - Help: "Amsterdam Single Site Private Endpoint", - }, { - Value: "s3.che01.cloud-object-storage.appdomain.cloud", - Help: "Chennai Single Site Endpoint", - }, { - Value: "s3.private.che01.cloud-object-storage.appdomain.cloud", - Help: "Chennai Single Site Private Endpoint", - }, { - Value: "s3.mel01.cloud-object-storage.appdomain.cloud", - Help: "Melbourne Single Site Endpoint", - }, { - Value: "s3.private.mel01.cloud-object-storage.appdomain.cloud", - Help: "Melbourne Single Site Private Endpoint", - }, { - Value: "s3.osl01.cloud-object-storage.appdomain.cloud", - Help: "Oslo Single Site Endpoint", - }, { - Value: "s3.private.osl01.cloud-object-storage.appdomain.cloud", - Help: "Oslo Single Site Private Endpoint", - }, { - Value: "s3.tor01.cloud-object-storage.appdomain.cloud", - Help: "Toronto Single Site Endpoint", - }, { - Value: "s3.private.tor01.cloud-object-storage.appdomain.cloud", - Help: "Toronto Single Site Private Endpoint", - }, { - Value: "s3.seo01.cloud-object-storage.appdomain.cloud", - Help: "Seoul Single Site Endpoint", - }, { - Value: "s3.private.seo01.cloud-object-storage.appdomain.cloud", - Help: "Seoul Single Site Private Endpoint", - }, { - Value: "s3.mon01.cloud-object-storage.appdomain.cloud", - Help: "Montreal Single Site Endpoint", - }, { - Value: "s3.private.mon01.cloud-object-storage.appdomain.cloud", - Help: "Montreal Single Site Private Endpoint", - }, { - Value: "s3.mex01.cloud-object-storage.appdomain.cloud", - Help: "Mexico Single Site Endpoint", - }, { - Value: "s3.private.mex01.cloud-object-storage.appdomain.cloud", - Help: "Mexico Single Site Private Endpoint", - }, { - Value: "s3.sjc04.cloud-object-storage.appdomain.cloud", - Help: "San Jose Single Site Endpoint", - }, { - Value: "s3.private.sjc04.cloud-object-storage.appdomain.cloud", - Help: "San Jose Single Site Private Endpoint", - }, { - Value: "s3.mil01.cloud-object-storage.appdomain.cloud", - Help: "Milan Single Site Endpoint", - }, { - Value: "s3.private.mil01.cloud-object-storage.appdomain.cloud", - Help: "Milan Single Site Private Endpoint", - }, { - Value: "s3.hkg02.cloud-object-storage.appdomain.cloud", - Help: "Hong Kong Single Site Endpoint", - }, { - Value: "s3.private.hkg02.cloud-object-storage.appdomain.cloud", - Help: "Hong Kong Single Site Private Endpoint", - }, { - Value: "s3.par01.cloud-object-storage.appdomain.cloud", - Help: "Paris Single Site Endpoint", - }, { - Value: "s3.private.par01.cloud-object-storage.appdomain.cloud", - Help: "Paris Single Site Private Endpoint", - }, { - Value: "s3.sng01.cloud-object-storage.appdomain.cloud", - Help: "Singapore Single Site Endpoint", - }, { - Value: "s3.private.sng01.cloud-object-storage.appdomain.cloud", - Help: "Singapore Single Site Private Endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Intercolo Object Storage.", - Provider: "Intercolo", - Examples: []fs.OptionExample{{ - Value: "de-fra.i3storage.com", - Help: "Frankfurt, Germany", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for IONOS S3 Object Storage.\n\nSpecify the endpoint from the same region.", - Provider: "IONOS", - Examples: []fs.OptionExample{{ - Value: "s3-eu-central-1.ionoscloud.com", - Help: "Frankfurt, Germany", - }, { - Value: "s3-eu-central-2.ionoscloud.com", - Help: "Berlin, Germany", - }, { - Value: "s3-eu-south-2.ionoscloud.com", - Help: "Logrono, Spain", - }}, - }, { - // Leviia endpoints: https://www.leviia.com/object-storage/ - Name: "endpoint", - Help: "Endpoint for Leviia Object Storage API.", - Provider: "Leviia", - Examples: []fs.OptionExample{{ - Value: "s3.leviia.com", - Help: "The default endpoint\nLeviia", - }}, - }, { - // Liara endpoints: https://liara.ir/landing/object-storage - Name: "endpoint", - Help: "Endpoint for Liara Object Storage API.", - Provider: "Liara", - Examples: []fs.OptionExample{{ - Value: "storage.iran.liara.space", - Help: "The default endpoint\nIran", - }}, - }, { - // Linode endpoints: https://techdocs.akamai.com/cloud-computing/docs/object-storage-product-limits#supported-endpoint-types-by-region - Name: "endpoint", - Help: "Endpoint for Linode Object Storage API.", - Provider: "Linode", - Examples: []fs.OptionExample{{ - Value: "nl-ams-1.linodeobjects.com", - Help: "Amsterdam (Netherlands), nl-ams-1", - }, { - Value: "us-southeast-1.linodeobjects.com", - Help: "Atlanta, GA (USA), us-southeast-1", - }, { - Value: "in-maa-1.linodeobjects.com", - Help: "Chennai (India), in-maa-1", - }, { - Value: "us-ord-1.linodeobjects.com", - Help: "Chicago, IL (USA), us-ord-1", - }, { - Value: "eu-central-1.linodeobjects.com", - Help: "Frankfurt (Germany), eu-central-1", - }, { - Value: "id-cgk-1.linodeobjects.com", - Help: "Jakarta (Indonesia), id-cgk-1", - }, { - Value: "gb-lon-1.linodeobjects.com", - Help: "London 2 (Great Britain), gb-lon-1", - }, { - Value: "us-lax-1.linodeobjects.com", - Help: "Los Angeles, CA (USA), us-lax-1", - }, { - Value: "es-mad-1.linodeobjects.com", - Help: "Madrid (Spain), es-mad-1", - }, { - Value: "au-mel-1.linodeobjects.com", - Help: "Melbourne (Australia), au-mel-1", - }, { - Value: "us-mia-1.linodeobjects.com", - Help: "Miami, FL (USA), us-mia-1", - }, { - Value: "it-mil-1.linodeobjects.com", - Help: "Milan (Italy), it-mil-1", - }, { - Value: "us-east-1.linodeobjects.com", - Help: "Newark, NJ (USA), us-east-1", - }, { - Value: "jp-osa-1.linodeobjects.com", - Help: "Osaka (Japan), jp-osa-1", - }, { - Value: "fr-par-1.linodeobjects.com", - Help: "Paris (France), fr-par-1", - }, { - Value: "br-gru-1.linodeobjects.com", - Help: "São Paulo (Brazil), br-gru-1", - }, { - Value: "us-sea-1.linodeobjects.com", - Help: "Seattle, WA (USA), us-sea-1", - }, { - Value: "ap-south-1.linodeobjects.com", - Help: "Singapore, ap-south-1", - }, { - Value: "sg-sin-1.linodeobjects.com", - Help: "Singapore 2, sg-sin-1", - }, { - Value: "se-sto-1.linodeobjects.com", - Help: "Stockholm (Sweden), se-sto-1", - }, { - Value: "us-iad-1.linodeobjects.com", - Help: "Washington, DC, (USA), us-iad-1", - }}, - }, { - // Lyve Cloud endpoints - Name: "endpoint", - Help: "Endpoint for Lyve Cloud S3 API.\nRequired when using an S3 clone. Please type in your LyveCloud endpoint.\nExamples:\n- s3.us-west-1.{account_name}.lyve.seagate.com (US West 1 - California)\n- s3.eu-west-1.{account_name}.lyve.seagate.com (EU West 1 - Ireland)", - Provider: "LyveCloud", - Required: true, - }, { - // Magalu endpoints: https://docs.magalu.cloud/docs/object-storage/how-to/copy-url - Name: "endpoint", - Help: "Endpoint for Magalu Object Storage API.", - Provider: "Magalu", - Examples: []fs.OptionExample{{ - Value: "br-se1.magaluobjects.com", - Help: "São Paulo, SP (BR), br-se1", - }, { - Value: "br-ne1.magaluobjects.com", - Help: "Fortaleza, CE (BR), br-ne1", - }, - }, - }, { - Name: "endpoint", - Help: "Endpoint for OVHcloud Object Storage.", - Provider: "OVHcloud", - Examples: []fs.OptionExample{{ - Value: "s3.gra.io.cloud.ovh.net", - Help: "OVHcloud Gravelines, France", - Provider: "OVHcloud", - }, { - Value: "s3.rbx.io.cloud.ovh.net", - Help: "OVHcloud Roubaix, France", - Provider: "OVHcloud", - }, { - Value: "s3.sbg.io.cloud.ovh.net", - Help: "OVHcloud Strasbourg, France", - Provider: "OVHcloud", - }, { - Value: "s3.eu-west-par.io.cloud.ovh.net", - Help: "OVHcloud Paris, France (3AZ)", - Provider: "OVHcloud", - }, { - Value: "s3.de.io.cloud.ovh.net", - Help: "OVHcloud Frankfurt, Germany", - Provider: "OVHcloud", - }, { - Value: "s3.uk.io.cloud.ovh.net", - Help: "OVHcloud London, United Kingdom", - Provider: "OVHcloud", - }, { - Value: "s3.waw.io.cloud.ovh.net", - Help: "OVHcloud Warsaw, Poland", - Provider: "OVHcloud", - }, { - Value: "s3.bhs.io.cloud.ovh.net", - Help: "OVHcloud Beauharnois, Canada", - Provider: "OVHcloud", - }, { - Value: "s3.ca-east-tor.io.cloud.ovh.net", - Help: "OVHcloud Toronto, Canada", - Provider: "OVHcloud", - }, { - Value: "s3.sgp.io.cloud.ovh.net", - Help: "OVHcloud Singapore", - Provider: "OVHcloud", - }, { - Value: "s3.ap-southeast-syd.io.cloud.ovh.net", - Help: "OVHcloud Sydney, Australia", - Provider: "OVHcloud", - }, { - Value: "s3.ap-south-mum.io.cloud.ovh.net", - Help: "OVHcloud Mumbai, India", - Provider: "OVHcloud", - }, { - Value: "s3.us-east-va.io.cloud.ovh.us", - Help: "OVHcloud Vint Hill, Virginia, USA", - Provider: "OVHcloud", - }, { - Value: "s3.us-west-or.io.cloud.ovh.us", - Help: "OVHcloud Hillsboro, Oregon, USA", - Provider: "OVHcloud", - }, { - Value: "s3.rbx-archive.io.cloud.ovh.net", - Help: "OVHcloud Roubaix, France (Cold Archive)", - Provider: "OVHcloud", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Petabox S3 Object Storage.\n\nSpecify the endpoint from the same region.", - Provider: "Petabox", - Required: true, - Examples: []fs.OptionExample{{ - Value: "s3.petabox.io", - Help: "US East (N. Virginia)", - }, { - Value: "s3.us-east-1.petabox.io", - Help: "US East (N. Virginia)", - }, { - Value: "s3.eu-central-1.petabox.io", - Help: "Europe (Frankfurt)", - }, { - Value: "s3.ap-southeast-1.petabox.io", - Help: "Asia Pacific (Singapore)", - }, { - Value: "s3.me-south-1.petabox.io", - Help: "Middle East (Bahrain)", - }, { - Value: "s3.sa-east-1.petabox.io", - Help: "South America (São Paulo)", - }}, - }, { - // Qiniu endpoints: https://developer.qiniu.com/kodo/4088/s3-access-domainname - Name: "endpoint", - Help: "Endpoint for Qiniu Object Storage.", - Provider: "Qiniu", - Examples: []fs.OptionExample{{ - Value: "s3-cn-east-1.qiniucs.com", - Help: "East China Endpoint 1", - }, { - Value: "s3-cn-east-2.qiniucs.com", - Help: "East China Endpoint 2", - }, { - Value: "s3-cn-north-1.qiniucs.com", - Help: "North China Endpoint 1", - }, { - Value: "s3-cn-south-1.qiniucs.com", - Help: "South China Endpoint 1", - }, { - Value: "s3-us-north-1.qiniucs.com", - Help: "North America Endpoint 1", - }, { - Value: "s3-ap-southeast-1.qiniucs.com", - Help: "Southeast Asia Endpoint 1", - }, { - Value: "s3-ap-northeast-1.qiniucs.com", - Help: "Northeast Asia Endpoint 1", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Rabata Object Storage.", - Provider: "Rabata", - Examples: []fs.OptionExample{{ - Value: "s3.us-east-1.rabata.io", - Help: "US East (N. Virginia)", - }, { - Value: "s3.eu-west-1.rabata.io", - Help: "EU West (Ireland)", - }, { - Value: "s3.eu-west-2.rabata.io", - Help: "EU West (London)", - }}, - }, { - // RackCorp endpoints: https://www.rackcorp.com/storage/s3storage - Name: "endpoint", - Help: "Endpoint for RackCorp Object Storage.", - Provider: "RackCorp", - Examples: []fs.OptionExample{{ - Value: "s3.rackcorp.com", - Help: "Global (AnyCast) Endpoint", - }, { - Value: "au.s3.rackcorp.com", - Help: "Australia (Anycast) Endpoint", - }, { - Value: "au-nsw.s3.rackcorp.com", - Help: "Sydney (Australia) Endpoint", - }, { - Value: "au-qld.s3.rackcorp.com", - Help: "Brisbane (Australia) Endpoint", - }, { - Value: "au-vic.s3.rackcorp.com", - Help: "Melbourne (Australia) Endpoint", - }, { - Value: "au-wa.s3.rackcorp.com", - Help: "Perth (Australia) Endpoint", - }, { - Value: "ph.s3.rackcorp.com", - Help: "Manila (Philippines) Endpoint", - }, { - Value: "th.s3.rackcorp.com", - Help: "Bangkok (Thailand) Endpoint", - }, { - Value: "hk.s3.rackcorp.com", - Help: "HK (Hong Kong) Endpoint", - }, { - Value: "mn.s3.rackcorp.com", - Help: "Ulaanbaatar (Mongolia) Endpoint", - }, { - Value: "kg.s3.rackcorp.com", - Help: "Bishkek (Kyrgyzstan) Endpoint", - }, { - Value: "id.s3.rackcorp.com", - Help: "Jakarta (Indonesia) Endpoint", - }, { - Value: "jp.s3.rackcorp.com", - Help: "Tokyo (Japan) Endpoint", - }, { - Value: "sg.s3.rackcorp.com", - Help: "SG (Singapore) Endpoint", - }, { - Value: "de.s3.rackcorp.com", - Help: "Frankfurt (Germany) Endpoint", - }, { - Value: "us.s3.rackcorp.com", - Help: "USA (AnyCast) Endpoint", - }, { - Value: "us-east-1.s3.rackcorp.com", - Help: "New York (USA) Endpoint", - }, { - Value: "us-west-1.s3.rackcorp.com", - Help: "Freemont (USA) Endpoint", - }, { - Value: "nz.s3.rackcorp.com", - Help: "Auckland (New Zealand) Endpoint", - }}, - }, { - // Selectel endpoints: https://docs.selectel.ru/en/cloud/object-storage/manage/domains/#s3-api-domains - Name: "endpoint", - Help: "Endpoint for Selectel Object Storage.", - Provider: "Selectel", - Examples: []fs.OptionExample{{ - Value: "s3.ru-1.storage.selcloud.ru", - Help: "Saint Petersburg", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Servercore Object Storage.", - Provider: "Servercore", - Examples: []fs.OptionExample{{ - Value: "s3.ru-1.storage.selcloud.ru", - Help: "Saint Petersburg", - }, { - Value: "s3.gis-1.storage.selcloud.ru", - Help: "Moscow", - }, { - Value: "s3.ru-7.storage.selcloud.ru", - Help: "Moscow", - }, { - Value: "s3.uz-2.srvstorage.uz", - Help: "Tashkent, Uzbekistan", - }, { - Value: "s3.kz-1.srvstorage.kz", - Help: "Almaty, Kazakhstan", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Scaleway Object Storage.", - Provider: "Scaleway", - Examples: []fs.OptionExample{{ - Value: "s3.nl-ams.scw.cloud", - Help: "Amsterdam Endpoint", - }, { - Value: "s3.fr-par.scw.cloud", - Help: "Paris Endpoint", - }, { - Value: "s3.pl-waw.scw.cloud", - Help: "Warsaw Endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for StackPath Object Storage.", - Provider: "StackPath", - Examples: []fs.OptionExample{{ - Value: "s3.us-east-2.stackpathstorage.com", - Help: "US East Endpoint", - }, { - Value: "s3.us-west-1.stackpathstorage.com", - Help: "US West Endpoint", - }, { - Value: "s3.eu-central-1.stackpathstorage.com", - Help: "EU Endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Storj Gateway.", - Provider: "Storj", - Examples: []fs.OptionExample{{ - Value: "gateway.storjshare.io", - Help: "Global Hosted Gateway", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Synology C2 Object Storage API.", - Provider: "Synology", - Examples: []fs.OptionExample{{ - Value: "eu-001.s3.synologyc2.net", - Help: "EU Endpoint 1", - }, { - Value: "eu-002.s3.synologyc2.net", - Help: "EU Endpoint 2", - }, { - Value: "us-001.s3.synologyc2.net", - Help: "US Endpoint 1", - }, { - Value: "us-002.s3.synologyc2.net", - Help: "US Endpoint 2", - }, { - Value: "tw-001.s3.synologyc2.net", - Help: "TW Endpoint 1", - }}, - }, { - // cos endpoints: https://intl.cloud.tencent.com/document/product/436/6224 - Name: "endpoint", - Help: "Endpoint for Tencent COS API.", - Provider: "TencentCOS", - Examples: []fs.OptionExample{{ - Value: "cos.ap-beijing.myqcloud.com", - Help: "Beijing Region", - }, { - Value: "cos.ap-nanjing.myqcloud.com", - Help: "Nanjing Region", - }, { - Value: "cos.ap-shanghai.myqcloud.com", - Help: "Shanghai Region", - }, { - Value: "cos.ap-guangzhou.myqcloud.com", - Help: "Guangzhou Region", - }, { - Value: "cos.ap-nanjing.myqcloud.com", - Help: "Nanjing Region", - }, { - Value: "cos.ap-chengdu.myqcloud.com", - Help: "Chengdu Region", - }, { - Value: "cos.ap-chongqing.myqcloud.com", - Help: "Chongqing Region", - }, { - Value: "cos.ap-hongkong.myqcloud.com", - Help: "Hong Kong (China) Region", - }, { - Value: "cos.ap-singapore.myqcloud.com", - Help: "Singapore Region", - }, { - Value: "cos.ap-mumbai.myqcloud.com", - Help: "Mumbai Region", - }, { - Value: "cos.ap-seoul.myqcloud.com", - Help: "Seoul Region", - }, { - Value: "cos.ap-bangkok.myqcloud.com", - Help: "Bangkok Region", - }, { - Value: "cos.ap-tokyo.myqcloud.com", - Help: "Tokyo Region", - }, { - Value: "cos.na-siliconvalley.myqcloud.com", - Help: "Silicon Valley Region", - }, { - Value: "cos.na-ashburn.myqcloud.com", - Help: "Virginia Region", - }, { - Value: "cos.na-toronto.myqcloud.com", - Help: "Toronto Region", - }, { - Value: "cos.eu-frankfurt.myqcloud.com", - Help: "Frankfurt Region", - }, { - Value: "cos.eu-moscow.myqcloud.com", - Help: "Moscow Region", - }, { - Value: "cos.accelerate.myqcloud.com", - Help: "Use Tencent COS Accelerate Endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for Zata Object Storage.", - Provider: "Zata", - Examples: []fs.OptionExample{{ - Value: "idr01.zata.ai", - Help: "South Asia Endpoint", - }}, - }, { - Name: "endpoint", - Help: "Endpoint for S3 API.\n\nRequired when using an S3 clone.", - Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cubbit,GCS,Hetzner,HuaweiOBS,IBMCOS,IDrive,Intercolo,IONOS,Liara,Linode,LyveCloud,Magalu,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,StackPath,Storj,Synology,TencentCOS,Zata", - Examples: []fs.OptionExample{{ - Value: "objects-us-east-1.dream.io", - Help: "Dream Objects endpoint", - Provider: "Dreamhost", - }, { - Value: "syd1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Sydney 1", - Provider: "DigitalOcean", - }, { - Value: "sfo3.digitaloceanspaces.com", - Help: "DigitalOcean Spaces San Francisco 3", - Provider: "DigitalOcean", - }, { - Value: "sfo2.digitaloceanspaces.com", - Help: "DigitalOcean Spaces San Francisco 2", - Provider: "DigitalOcean", - }, { - Value: "fra1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Frankfurt 1", - Provider: "DigitalOcean", - }, { - Value: "nyc3.digitaloceanspaces.com", - Help: "DigitalOcean Spaces New York 3", - Provider: "DigitalOcean", - }, { - Value: "ams3.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Amsterdam 3", - Provider: "DigitalOcean", - }, { - Value: "sgp1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Singapore 1", - Provider: "DigitalOcean", - }, { - Value: "lon1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces London 1", - Provider: "DigitalOcean", - }, { - Value: "tor1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Toronto 1", - Provider: "DigitalOcean", - }, { - Value: "blr1.digitaloceanspaces.com", - Help: "DigitalOcean Spaces Bangalore 1", - Provider: "DigitalOcean", - }, { - Value: "localhost:8333", - Help: "SeaweedFS S3 localhost", - Provider: "SeaweedFS", - }, { - Value: "oos.eu-west-2.outscale.com", - Help: "Outscale EU West 2 (Paris)", - Provider: "Outscale", - }, { - Value: "oos.us-east-2.outscale.com", - Help: "Outscale US east 2 (New Jersey)", - Provider: "Outscale", - }, { - Value: "oos.us-west-1.outscale.com", - Help: "Outscale EU West 1 (California)", - Provider: "Outscale", - }, { - Value: "oos.cloudgouv-eu-west-1.outscale.com", - Help: "Outscale SecNumCloud (Paris)", - Provider: "Outscale", - }, { - Value: "oos.ap-northeast-1.outscale.com", - Help: "Outscale AP Northeast 1 (Japan)", - Provider: "Outscale", - }, { - Value: "s3.wasabisys.com", - Help: "Wasabi US East 1 (N. Virginia)", - Provider: "Wasabi", - }, { - Value: "s3.us-east-2.wasabisys.com", - Help: "Wasabi US East 2 (N. Virginia)", - Provider: "Wasabi", - }, { - Value: "s3.us-central-1.wasabisys.com", - Help: "Wasabi US Central 1 (Texas)", - Provider: "Wasabi", - }, { - Value: "s3.us-west-1.wasabisys.com", - Help: "Wasabi US West 1 (Oregon)", - Provider: "Wasabi", - }, { - Value: "s3.ca-central-1.wasabisys.com", - Help: "Wasabi CA Central 1 (Toronto)", - Provider: "Wasabi", - }, { - Value: "s3.eu-central-1.wasabisys.com", - Help: "Wasabi EU Central 1 (Amsterdam)", - Provider: "Wasabi", - }, { - Value: "s3.eu-central-2.wasabisys.com", - Help: "Wasabi EU Central 2 (Frankfurt)", - Provider: "Wasabi", - }, { - Value: "s3.eu-west-1.wasabisys.com", - Help: "Wasabi EU West 1 (London)", - Provider: "Wasabi", - }, { - Value: "s3.eu-west-2.wasabisys.com", - Help: "Wasabi EU West 2 (Paris)", - Provider: "Wasabi", - }, { - Value: "s3.eu-south-1.wasabisys.com", - Help: "Wasabi EU South 1 (Milan)", - Provider: "Wasabi", - }, { - Value: "s3.ap-northeast-1.wasabisys.com", - Help: "Wasabi AP Northeast 1 (Tokyo) endpoint", - Provider: "Wasabi", - }, { - Value: "s3.ap-northeast-2.wasabisys.com", - Help: "Wasabi AP Northeast 2 (Osaka) endpoint", - Provider: "Wasabi", - }, { - Value: "s3.ap-southeast-1.wasabisys.com", - Help: "Wasabi AP Southeast 1 (Singapore)", - Provider: "Wasabi", - }, { - Value: "s3.ap-southeast-2.wasabisys.com", - Help: "Wasabi AP Southeast 2 (Sydney)", - Provider: "Wasabi", - }, { - Value: "storage.iran.liara.space", - Help: "Liara Iran endpoint", - Provider: "Liara", - }, { - Value: "s3.ir-thr-at1.arvanstorage.ir", - Help: "ArvanCloud Tehran Iran (Simin) endpoint", - Provider: "ArvanCloud", - }, { - Value: "s3.ir-tbz-sh1.arvanstorage.ir", - Help: "ArvanCloud Tabriz Iran (Shahriar) endpoint", - Provider: "ArvanCloud", - }, { - Value: "s3.eu-central-1.s4.mega.io", - Help: "Mega S4 eu-central-1 (Amsterdam)", - Provider: "Mega", - }, { - Value: "s3.eu-central-2.s4.mega.io", - Help: "Mega S4 eu-central-2 (Bettembourg)", - Provider: "Mega", - }, { - Value: "s3.ca-central-1.s4.mega.io", - Help: "Mega S4 ca-central-1 (Montreal)", - Provider: "Mega", - }, { - Value: "s3.ca-west-1.s4.mega.io", - Help: "Mega S4 ca-west-1 (Vancouver)", - Provider: "Mega", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.", - Provider: "AWS", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Empty for US Region, Northern Virginia, or Pacific Northwest", - }, { - Value: "us-east-2", - Help: "US East (Ohio) Region", - }, { - Value: "us-west-1", - Help: "US West (Northern California) Region", - }, { - Value: "us-west-2", - Help: "US West (Oregon) Region", - }, { - Value: "ca-central-1", - Help: "Canada (Central) Region", - }, { - Value: "eu-west-1", - Help: "EU (Ireland) Region", - }, { - Value: "eu-west-2", - Help: "EU (London) Region", - }, { - Value: "eu-west-3", - Help: "EU (Paris) Region", - }, { - Value: "eu-north-1", - Help: "EU (Stockholm) Region", - }, { - Value: "eu-south-1", - Help: "EU (Milan) Region", - }, { - Value: "EU", - Help: "EU Region", - }, { - Value: "ap-southeast-1", - Help: "Asia Pacific (Singapore) Region", - }, { - Value: "ap-southeast-2", - Help: "Asia Pacific (Sydney) Region", - }, { - Value: "ap-northeast-1", - Help: "Asia Pacific (Tokyo) Region", - }, { - Value: "ap-northeast-2", - Help: "Asia Pacific (Seoul) Region", - }, { - Value: "ap-northeast-3", - Help: "Asia Pacific (Osaka-Local) Region", - }, { - Value: "ap-south-1", - Help: "Asia Pacific (Mumbai) Region", - }, { - Value: "ap-east-1", - Help: "Asia Pacific (Hong Kong) Region", - }, { - Value: "sa-east-1", - Help: "South America (Sao Paulo) Region", - }, { - Value: "il-central-1", - Help: "Israel (Tel Aviv) Region", - }, { - Value: "me-south-1", - Help: "Middle East (Bahrain) Region", - }, { - Value: "af-south-1", - Help: "Africa (Cape Town) Region", - }, { - Value: "cn-north-1", - Help: "China (Beijing) Region", - }, { - Value: "cn-northwest-1", - Help: "China (Ningxia) Region", - }, { - Value: "us-gov-east-1", - Help: "AWS GovCloud (US-East) Region", - }, { - Value: "us-gov-west-1", - Help: "AWS GovCloud (US) Region", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.", - Provider: "ArvanCloud", - Examples: []fs.OptionExample{{ - Value: "ir-thr-at1", - Help: "Tehran Iran (Simin)", - }, { - Value: "ir-tbz-sh1", - Help: "Tabriz Iran (Shahriar)", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must match endpoint.\n\nUsed when creating buckets only.", - Provider: "ChinaMobile", - Examples: []fs.OptionExample{{ - Value: "wuxi1", - Help: "East China (Suzhou)", - }, { - Value: "jinan1", - Help: "East China (Jinan)", - }, { - Value: "ningbo1", - Help: "East China (Hangzhou)", - }, { - Value: "shanghai1", - Help: "East China (Shanghai-1)", - }, { - Value: "zhengzhou1", - Help: "Central China (Zhengzhou)", - }, { - Value: "hunan1", - Help: "Central China (Changsha-1)", - }, { - Value: "zhuzhou1", - Help: "Central China (Changsha-2)", - }, { - Value: "guangzhou1", - Help: "South China (Guangzhou-2)", - }, { - Value: "dongguan1", - Help: "South China (Guangzhou-3)", - }, { - Value: "beijing1", - Help: "North China (Beijing-1)", - }, { - Value: "beijing2", - Help: "North China (Beijing-2)", - }, { - Value: "beijing4", - Help: "North China (Beijing-3)", - }, { - Value: "huhehaote1", - Help: "North China (Huhehaote)", - }, { - Value: "chengdu1", - Help: "Southwest China (Chengdu)", - }, { - Value: "chongqing1", - Help: "Southwest China (Chongqing)", - }, { - Value: "guiyang1", - Help: "Southwest China (Guiyang)", - }, { - Value: "xian1", - Help: "Northwest China (Xian)", - }, { - Value: "yunnan", - Help: "Yunnan China (Kunming)", - }, { - Value: "yunnan2", - Help: "Yunnan China (Kunming-2)", - }, { - Value: "tianjin1", - Help: "Tianjin China (Tianjin)", - }, { - Value: "jilin1", - Help: "Jilin China (Changchun)", - }, { - Value: "hubei1", - Help: "Hubei China (Xiangyan)", - }, { - Value: "jiangxi1", - Help: "Jiangxi China (Nanchang)", - }, { - Value: "gansu1", - Help: "Gansu China (Lanzhou)", - }, { - Value: "shanxi1", - Help: "Shanxi China (Taiyuan)", - }, { - Value: "liaoning1", - Help: "Liaoning China (Shenyang)", - }, { - Value: "hebei1", - Help: "Hebei China (Shijiazhuang)", - }, { - Value: "fujian1", - Help: "Fujian China (Xiamen)", - }, { - Value: "guangxi1", - Help: "Guangxi China (Nanning)", - }, { - Value: "anhui1", - Help: "Anhui China (Huainan)", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must match endpoint when using IBM Cloud Public.\n\nFor on-prem COS, do not make a selection from this list, hit enter.", - Provider: "IBMCOS", - Examples: []fs.OptionExample{{ - Value: "us-standard", - Help: "US Cross Region Standard", - }, { - Value: "us-vault", - Help: "US Cross Region Vault", - }, { - Value: "us-cold", - Help: "US Cross Region Cold", - }, { - Value: "us-flex", - Help: "US Cross Region Flex", - }, { - Value: "us-east-standard", - Help: "US East Region Standard", - }, { - Value: "us-east-vault", - Help: "US East Region Vault", - }, { - Value: "us-east-cold", - Help: "US East Region Cold", - }, { - Value: "us-east-flex", - Help: "US East Region Flex", - }, { - Value: "us-south-standard", - Help: "US South Region Standard", - }, { - Value: "us-south-vault", - Help: "US South Region Vault", - }, { - Value: "us-south-cold", - Help: "US South Region Cold", - }, { - Value: "us-south-flex", - Help: "US South Region Flex", - }, { - Value: "eu-standard", - Help: "EU Cross Region Standard", - }, { - Value: "eu-vault", - Help: "EU Cross Region Vault", - }, { - Value: "eu-cold", - Help: "EU Cross Region Cold", - }, { - Value: "eu-flex", - Help: "EU Cross Region Flex", - }, { - Value: "eu-gb-standard", - Help: "Great Britain Standard", - }, { - Value: "eu-gb-vault", - Help: "Great Britain Vault", - }, { - Value: "eu-gb-cold", - Help: "Great Britain Cold", - }, { - Value: "eu-gb-flex", - Help: "Great Britain Flex", - }, { - Value: "ap-standard", - Help: "APAC Standard", - }, { - Value: "ap-vault", - Help: "APAC Vault", - }, { - Value: "ap-cold", - Help: "APAC Cold", - }, { - Value: "ap-flex", - Help: "APAC Flex", - }, { - Value: "mel01-standard", - Help: "Melbourne Standard", - }, { - Value: "mel01-vault", - Help: "Melbourne Vault", - }, { - Value: "mel01-cold", - Help: "Melbourne Cold", - }, { - Value: "mel01-flex", - Help: "Melbourne Flex", - }, { - Value: "tor01-standard", - Help: "Toronto Standard", - }, { - Value: "tor01-vault", - Help: "Toronto Vault", - }, { - Value: "tor01-cold", - Help: "Toronto Cold", - }, { - Value: "tor01-flex", - Help: "Toronto Flex", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must be set to match the Region.\n\nUsed when creating buckets only.", - Provider: "Qiniu", - Examples: []fs.OptionExample{{ - Value: "cn-east-1", - Help: "East China Region 1", - }, { - Value: "cn-east-2", - Help: "East China Region 2", - }, { - Value: "cn-north-1", - Help: "North China Region 1", - }, { - Value: "cn-south-1", - Help: "South China Region 1", - }, { - Value: "us-north-1", - Help: "North America Region 1", - }, { - Value: "ap-southeast-1", - Help: "Southeast Asia Region 1", - }, { - Value: "ap-northeast-1", - Help: "Northeast Asia Region 1", - }}, - }, { - Name: "location_constraint", - Help: "location where your bucket will be created and your data stored.\n", - Provider: "Rabata", - Examples: []fs.OptionExample{{ - Value: "us-east-1", - Help: "US East (N. Virginia)", - }, { - Value: "eu-west-1", - Help: "EU (Ireland)", - }, { - Value: "eu-west-2", - Help: "EU (London)", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - the location where your bucket will be located and your data stored.\n", - Provider: "RackCorp", - Examples: []fs.OptionExample{{ - Value: "global", - Help: "Global CDN Region", - }, { - Value: "au", - Help: "Australia (All locations)", - }, { - Value: "au-nsw", - Help: "NSW (Australia) Region", - }, { - Value: "au-qld", - Help: "QLD (Australia) Region", - }, { - Value: "au-vic", - Help: "VIC (Australia) Region", - }, { - Value: "au-wa", - Help: "Perth (Australia) Region", - }, { - Value: "ph", - Help: "Manila (Philippines) Region", - }, { - Value: "th", - Help: "Bangkok (Thailand) Region", - }, { - Value: "hk", - Help: "HK (Hong Kong) Region", - }, { - Value: "mn", - Help: "Ulaanbaatar (Mongolia) Region", - }, { - Value: "kg", - Help: "Bishkek (Kyrgyzstan) Region", - }, { - Value: "id", - Help: "Jakarta (Indonesia) Region", - }, { - Value: "jp", - Help: "Tokyo (Japan) Region", - }, { - Value: "sg", - Help: "SG (Singapore) Region", - }, { - Value: "de", - Help: "Frankfurt (Germany) Region", - }, { - Value: "us", - Help: "USA (AnyCast) Region", - }, { - Value: "us-east-1", - Help: "New York (USA) Region", - }, { - Value: "us-west-1", - Help: "Fremont (USA) Region", - }, { - Value: "nz", - Help: "Auckland (New Zealand) Region", - }}, - }, { - Name: "location_constraint", - Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.", - Provider: "!AWS,Alibaba,ArvanCloud,ChinaMobile,Cloudflare,Cubbit,FlashBlade,FileLu,HuaweiOBS,IBMCOS,IDrive,Intercolo,IONOS,Leviia,Liara,Linode,Magalu,Mega,Outscale,OVHcloud,Petabox,Qiniu,Rabata,RackCorp,Scaleway,Selectel,Servercore,SpectraLogic,StackPath,Storj,TencentCOS", + Name: "location_constraint", + Help: "Location constraint - must be set to match the Region.\n\nLeave blank if not sure. Used when creating buckets only.", }, { Name: "acl", Help: `Canned ACL used when creating buckets and storing or copying objects. @@ -2285,52 +126,6 @@ doesn't copy the ACL from the source but rather writes a fresh one. If the acl is an empty string then no X-Amz-Acl: header is added and the default (private) will be used. `, - Provider: "!Cloudflare,FlashBlade,Mega,Rabata,Selectel,Servercore,SpectraLogic,Storj,Synology", - Examples: []fs.OptionExample{{ - Value: "default", - Help: "Owner gets Full_CONTROL.\nNo one else has access rights (default).", - Provider: "TencentCOS", - }, { - Value: "private", - Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).", - Provider: "!IBMCOS,TencentCOS", - }, { - Value: "public-read", - Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.", - Provider: "!IBMCOS", - }, { - Value: "public-read-write", - Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nGranting this on a bucket is generally not recommended.", - Provider: "!IBMCOS", - }, { - Value: "authenticated-read", - Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.", - Provider: "!IBMCOS", - }, { - Value: "bucket-owner-read", - Help: "Object owner gets FULL_CONTROL.\nBucket owner gets READ access.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.", - Provider: "!ChinaMobile,IBMCOS", - }, { - Value: "bucket-owner-full-control", - Help: "Both the object owner and the bucket owner get FULL_CONTROL over the object.\nIf you specify this canned ACL when creating a bucket, Amazon S3 ignores it.", - Provider: "!ChinaMobile,IBMCOS", - }, { - Value: "private", - Help: "Owner gets FULL_CONTROL.\nNo one else has access rights (default).\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise COS.", - Provider: "IBMCOS", - }, { - Value: "public-read", - Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ access.\nThis acl is available on IBM Cloud (Infra), IBM Cloud (Storage), On-Premise IBM COS.", - Provider: "IBMCOS", - }, { - Value: "public-read-write", - Help: "Owner gets FULL_CONTROL.\nThe AllUsers group gets READ and WRITE access.\nThis acl is available on IBM Cloud (Infra), On-Premise IBM COS.", - Provider: "IBMCOS", - }, { - Value: "authenticated-read", - Help: "Owner gets FULL_CONTROL.\nThe AuthenticatedUsers group gets READ access.\nNot supported on Buckets.\nThis acl is available on IBM Cloud (Infra) and On-Premise IBM COS.", - Provider: "IBMCOS", - }}, }, { Name: "bucket_acl", Help: `Canned ACL used when creating buckets. @@ -2343,7 +138,6 @@ isn't set then "acl" is used instead. If the "acl" and "bucket_acl" are empty strings then no X-Amz-Acl: header is added and the default (private) will be used. `, - Provider: "!Cloudflare,FlashBlade,Rabata,Selectel,SpectraLogic,Storj,Synology", Advanced: true, Examples: []fs.OptionExample{{ Value: "private", @@ -2361,28 +155,14 @@ header is added and the default (private) will be used. }, { Name: "requester_pays", Help: "Enables requester pays option when interacting with S3 bucket.", - Provider: "AWS", Default: false, Advanced: true, }, { - Name: "server_side_encryption", - Help: "The server-side encryption algorithm used when storing this object in S3.", - Provider: "AWS,Ceph,ChinaMobile,Minio", - Examples: []fs.OptionExample{{ - Value: "", - Help: "None", - }, { - Value: "AES256", - Help: "AES256", - }, { - Value: "aws:kms", - Help: "aws:kms", - Provider: "!ChinaMobile", - }}, + Name: "server_side_encryption", + Help: "The server-side encryption algorithm used when storing this object in S3.", }, { Name: "sse_customer_algorithm", Help: "If using SSE-C, the server-side encryption algorithm used when storing this object in S3.", - Provider: "AWS,Ceph,ChinaMobile,Minio", Advanced: true, Examples: []fs.OptionExample{{ Value: "", @@ -2392,9 +172,8 @@ header is added and the default (private) will be used. Help: "AES256", }}, }, { - Name: "sse_kms_key_id", - Help: "If using KMS ID you must provide the ARN of Key.", - Provider: "AWS,Ceph,Minio", + Name: "sse_kms_key_id", + Help: "If using KMS ID you must provide the ARN of Key.", Examples: []fs.OptionExample{{ Value: "", Help: "None", @@ -2408,7 +187,6 @@ header is added and the default (private) will be used. Help: `To use SSE-C you may provide the secret encryption key used to encrypt/decrypt your data. Alternatively you can provide --sse-customer-key-base64.`, - Provider: "AWS,Ceph,ChinaMobile,Minio", Advanced: true, Examples: []fs.OptionExample{{ Value: "", @@ -2420,7 +198,6 @@ Alternatively you can provide --sse-customer-key-base64.`, Help: `If using SSE-C you must provide the secret encryption key encoded in base64 format to encrypt/decrypt your data. Alternatively you can provide --sse-customer-key.`, - Provider: "AWS,Ceph,ChinaMobile,Minio", Advanced: true, Examples: []fs.OptionExample{{ Value: "", @@ -2433,7 +210,6 @@ Alternatively you can provide --sse-customer-key.`, If you leave it blank, this is calculated automatically from the sse_customer_key provided. `, - Provider: "AWS,Ceph,ChinaMobile,Minio", Advanced: true, Examples: []fs.OptionExample{{ Value: "", @@ -2441,157 +217,8 @@ If you leave it blank, this is calculated automatically from the sse_customer_ke }}, Sensitive: true, }, { - Name: "storage_class", - Help: "The storage class to use when storing new objects in S3.", - Provider: "AWS", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Default", - }, { - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "REDUCED_REDUNDANCY", - Help: "Reduced redundancy storage class", - }, { - Value: "STANDARD_IA", - Help: "Standard Infrequent Access storage class", - }, { - Value: "ONEZONE_IA", - Help: "One Zone Infrequent Access storage class", - }, { - Value: "GLACIER", - Help: "Glacier Flexible Retrieval storage class", - }, { - Value: "DEEP_ARCHIVE", - Help: "Glacier Deep Archive storage class", - }, { - Value: "INTELLIGENT_TIERING", - Help: "Intelligent-Tiering storage class", - }, { - Value: "GLACIER_IR", - Help: "Glacier Instant Retrieval storage class", - }}, - }, { - // Mapping from here: https://www.arvancloud.ir/en/products/cloud-storage - Name: "storage_class", - Help: "The storage class to use when storing new objects in ArvanCloud.", - Provider: "ArvanCloud", - Examples: []fs.OptionExample{{ - Value: "STANDARD", - Help: "Standard storage class", - }}, - }, { - // Mapping from here: https://www.alibabacloud.com/help/doc-detail/64919.htm - Name: "storage_class", - Help: "The storage class to use when storing new objects in OSS.", - Provider: "Alibaba", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Default", - }, { - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "GLACIER", - Help: "Archive storage mode", - }, { - Value: "STANDARD_IA", - Help: "Infrequent access storage mode", - }}, - }, { - // Mapping from here: https://ecloud.10086.cn/op-help-center/doc/article/24495 - Name: "storage_class", - Help: "The storage class to use when storing new objects in ChinaMobile.", - Provider: "ChinaMobile", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Default", - }, { - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "GLACIER", - Help: "Archive storage mode", - }, { - Value: "STANDARD_IA", - Help: "Infrequent access storage mode", - }}, - }, { - // Mapping from here: https://liara.ir/landing/object-storage - Name: "storage_class", - Help: "The storage class to use when storing new objects in Liara", - Provider: "Liara", - Examples: []fs.OptionExample{{ - Value: "STANDARD", - Help: "Standard storage class", - }}, - }, { - // Mapping from here: https://docs.magalu.cloud/docs/storage/object-storage/Classes-de-Armazenamento/standard - Name: "storage_class", - Help: "The storage class to use when storing new objects in Magalu.", - Provider: "Magalu", - Examples: []fs.OptionExample{{ - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "GLACIER_IR", - Help: "Glacier Instant Retrieval storage class", - }}, - }, { - // Mapping from here: https://developer.qiniu.com/kodo/5906/storage-type - Name: "storage_class", - Help: "The storage class to use when storing new objects in Qiniu.", - Provider: "Qiniu", - Examples: []fs.OptionExample{{ - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "LINE", - Help: "Infrequent access storage mode", - }, { - Value: "GLACIER", - Help: "Archive storage mode", - }, { - Value: "DEEP_ARCHIVE", - Help: "Deep archive storage mode", - }}, - }, { - // Mapping from here: https://www.scaleway.com/en/docs/storage/object/quickstart/ - Name: "storage_class", - Help: "The storage class to use when storing new objects in S3.", - Provider: "Scaleway", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Default.", - }, { - Value: "STANDARD", - Help: "The Standard class for any upload.\nSuitable for on-demand content like streaming or CDN.\nAvailable in all regions.", - }, { - Value: "GLACIER", - Help: "Archived storage.\nPrices are lower, but it needs to be restored first to be accessed.\nAvailable in FR-PAR and NL-AMS regions.", - }, { - Value: "ONEZONE_IA", - Help: "One Zone - Infrequent Access.\nA good choice for storing secondary backup copies or easily re-creatable data.\nAvailable in the FR-PAR region only.", - }}, - }, { - // Mapping from here: https://intl.cloud.tencent.com/document/product/436/30925 - Name: "storage_class", - Help: "The storage class to use when storing new objects in Tencent COS.", - Provider: "TencentCOS", - Examples: []fs.OptionExample{{ - Value: "", - Help: "Default", - }, { - Value: "STANDARD", - Help: "Standard storage class", - }, { - Value: "ARCHIVE", - Help: "Archive storage mode", - }, { - Value: "STANDARD_IA", - Help: "Infrequent access storage mode", - }}, + Name: "storage_class", + Help: "The storage class to use when storing new objects in S3.", }, { Name: "upload_cutoff", Help: `Cutoff for switching to chunked upload. @@ -2745,8 +372,7 @@ See [AWS Docs on Dualstack Endpoints](https://docs.aws.amazon.com/AmazonS3/lates Default: false, Advanced: true, }, { - Name: "use_accelerate_endpoint", - Provider: "AWS", + Name: "use_accelerate_endpoint", Help: `If true use the AWS S3 accelerated endpoint. See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/dev/transfer-acceleration-examples.html)`, @@ -2758,8 +384,7 @@ See: [AWS S3 Transfer acceleration](https://docs.aws.amazon.com/AmazonS3/latest/ Default: false, Advanced: true, }, { - Name: "leave_parts_on_error", - Provider: "AWS", + Name: "leave_parts_on_error", Help: `If true avoid calling abort upload on a failure, leaving all successfully uploaded parts on S3 for manual recovery. It should be set to true for resuming uploads across different sessions. @@ -3045,7 +670,6 @@ In this case, you might want to try disabling this option. }, { Name: "sts_endpoint", Help: "Endpoint for STS (deprecated).\n\nLeave blank if using AWS to use the default endpoint for the region.", - Provider: "AWS", Advanced: true, Hide: fs.OptionHideBoth, }, { @@ -3143,7 +767,6 @@ Rclone limitations with Directory Buckets: `, "|", "`"), Default: false, Advanced: true, - Provider: "AWS", }, { Name: "sdk_log_mode", Help: strings.ReplaceAll(`Set to debug the SDK @@ -3165,18 +788,14 @@ use |-vv| to see the debug level logs. `, "|", "`"), Default: sdkLogMode(0), Advanced: true, + }, { + Name: "ibm_api_key", + Help: "IBM API Key to be used to obtain IAM token", + }, { + Name: "ibm_resource_instance_id", + Help: "IBM service instance id", }, - { - Name: "ibm_api_key", - Help: "IBM API Key to be used to obtain IAM token", - Provider: "IBMCOS", - }, - { - Name: "ibm_resource_instance_id", - Help: "IBM service instance id", - Provider: "IBMCOS", - }, - }}) + }})) } // Constants @@ -3612,7 +1231,7 @@ func (s3logger) Logf(classification logging.Classification, format string, v ... } // s3Connection makes a connection to s3 -func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Client *s3.Client, err error) { +func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Client *s3.Client, provider *Provider, err error) { ci := fs.GetConfig(ctx) var awsConfig aws.Config // Make the default static auth @@ -3637,7 +1256,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli } awsConfig, err = awsconfig.LoadDefaultConfig(ctx, configOpts...) if err != nil { - return nil, fmt.Errorf("couldn't load configuration with env_auth=true: %w", err) + return nil, nil, fmt.Errorf("couldn't load configuration with env_auth=true: %w", err) } } else { @@ -3650,9 +1269,9 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli awsConfig.Credentials = aws.AnonymousCredentials{} fs.Debugf(nil, "Using anonymous credentials - did you mean to set env_auth=true?") case opt.AccessKeyID == "": - return nil, errors.New("access_key_id not found") + return nil, nil, errors.New("access_key_id not found") case opt.SecretAccessKey == "": - return nil, errors.New("secret_access_key not found") + return nil, nil, errors.New("secret_access_key not found") default: // static credentials are already set } @@ -3661,7 +1280,14 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli if opt.Region == "" { opt.Region = "us-east-1" } - setQuirks(opt) + + provider = loadProvider(opt.Provider) + if provider == nil { + fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider) + provider = loadProvider("Other") + } + + setQuirks(opt, provider) awsConfig.RetryMaxAttempts = ci.LowLevelRetries awsConfig.HTTPClient = client @@ -3686,7 +1312,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli if opt.STSEndpoint != "" { // FIXME not sure if anyone is using this // Haven't figured out how to do it with the v2 SDK - return nil, errors.New("--s3-sts-endpoint is no longer supported with the v2 SDK - please make an issue") + return nil, nil, errors.New("--s3-sts-endpoint is no longer supported with the v2 SDK - please make an issue") } if opt.Endpoint != "" { if !strings.HasPrefix(opt.Endpoint, "http") { @@ -3724,7 +1350,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli } c := s3.NewFromConfig(awsConfig, options...) - return c, nil + return c, provider, nil } func checkUploadChunkSize(cs fs.SizeSuffix) error { @@ -3817,291 +1443,69 @@ func setEndpointValueForIDriveE2(m configmap.Mapper) (err error) { // Run the integration tests to check you have the quirks correct. // // go test -v -remote NewS3Provider: -func setQuirks(opt *Options) { - var ( - listObjectsV2 = true // Always use ListObjectsV2 instead of ListObjects - virtualHostStyle = true // Use bucket.provider.com instead of putting the bucket in the URL - urlEncodeListings = true // URL encode the listings to help with control characters - useMultipartEtag = true // Set if Etags for multipart uploads are compatible with AWS - useAcceptEncodingGzip = true // Set Accept-Encoding: gzip - mightGzip = true // assume all providers might use content encoding gzip until proven otherwise - useAlreadyExists = true // Set if provider returns AlreadyOwnedByYou or no error if you try to remake your own bucket - useMultipartUploads = true // Set if provider supports multipart uploads - useUnsignedPayload = true // Do we need to use unsigned payloads to avoid seeking in PutObject - useXID = true // Add x-id URL parameter into requests - signAcceptEncoding = true // If we should include AcceptEncoding in the signature - ) - switch opt.Provider { - case "AWS": - // No quirks - mightGzip = false // Never auto gzips objects - useUnsignedPayload = false // AWS has trailer support which means it adds checksums in the trailer without seeking - case "Alibaba": - useMultipartEtag = false // Alibaba seems to calculate multipart Etags differently from AWS - useAlreadyExists = true // returns 200 OK - case "Hetzner": - useAlreadyExists = false - case "HuaweiOBS": - // Huawei OBS PFS is not support listObjectV2, and if turn on the urlEncodeListing, marker will not work and keep list same page forever. - urlEncodeListings = false - listObjectsV2 = false - useAlreadyExists = false // untested - case "Ceph": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useAlreadyExists = true - case "ChinaMobile": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useAlreadyExists = false // untested - case "Cloudflare": - virtualHostStyle = false - useMultipartEtag = false // currently multipart Etags are random - case "Cubbit": - // no quirks - useMultipartEtag = false // Cubbit calculates multipart Etags differently from AWS - case "ArvanCloud": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useAlreadyExists = false // untested - case "DigitalOcean": - urlEncodeListings = false - useAlreadyExists = false // untested - case "Dreamhost": - urlEncodeListings = false - useAlreadyExists = false // untested - case "FlashBlade": - mightGzip = false // Never auto gzips objects - virtualHostStyle = false // supports vhost but defaults to paths - case "FileLu": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false - case "IBMCOS": - listObjectsV2 = false // untested - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false // untested - useAlreadyExists = false // returns BucketAlreadyExists - case "IDrive": - virtualHostStyle = false - useAlreadyExists = false // untested - case "Intercolo": - // no quirks - useUnsignedPayload = false // Intercolo has trailer support - case "IONOS": - // listObjectsV2 supported - https://api.ionos.com/docs/s3/#Basic-Operations-get-Bucket-list-type-2 - virtualHostStyle = false - urlEncodeListings = false - useAlreadyExists = false // untested - case "Petabox": - useAlreadyExists = false // untested - case "Liara": - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false - useAlreadyExists = false // untested - case "Linode": - useAlreadyExists = true // returns 200 OK - case "LyveCloud": - useMultipartEtag = false // LyveCloud seems to calculate multipart Etags differently from AWS - useAlreadyExists = false // untested - case "Magalu": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false - useAlreadyExists = false - case "Mega": - listObjectsV2 = true - virtualHostStyle = false - urlEncodeListings = true - useMultipartEtag = false - useAlreadyExists = false - // Multipart server side copies not supported - opt.CopyCutoff = math.MaxInt64 - case "Minio": - virtualHostStyle = false - case "Netease": - listObjectsV2 = false // untested - urlEncodeListings = false - useMultipartEtag = false // untested - useAlreadyExists = false // untested - case "Outscale": - virtualHostStyle = false - case "OVHcloud": - // No quirks - case "Rabata": - // server side copy not supported - case "RackCorp": - // No quirks - useMultipartEtag = false // untested - useAlreadyExists = false // untested - case "Rclone": - listObjectsV2 = true - urlEncodeListings = true - virtualHostStyle = false - useMultipartEtag = false - useAlreadyExists = false - // useMultipartUploads = false - set this manually - // rclone serve doesn't support multi-part server side copy: - // See: https://github.com/rclone/rclone/issues/7454 - // So make cutoff very large which it does support - opt.CopyCutoff = math.MaxInt64 - case "Scaleway": - // Scaleway can only have 1000 parts in an upload - if opt.MaxUploadParts > 1000 { - opt.MaxUploadParts = 1000 +func setQuirks(opt *Options, provider *Provider) { + // Set tristate to the ultimate default value or the override + // in provider.Quirks. Pass in the ultimate default as value. + set := func(tristate *fs.Tristate, value bool, override *bool) { + if override != nil { + value = *override } - urlEncodeListings = true - useAlreadyExists = true - case "Selectel": - urlEncodeListings = false - case "Servercore": - urlEncodeListings = false - case "SeaweedFS": - listObjectsV2 = false // untested - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false // untested - useAlreadyExists = false // untested - case "SpectraLogic": - virtualHostStyle = false // path-style required - case "StackPath": - listObjectsV2 = false // untested - virtualHostStyle = false - urlEncodeListings = false - useAlreadyExists = false // untested - case "Storj": - // Force chunk size to >= 64 MiB - if opt.ChunkSize < 64*fs.Mebi { - opt.ChunkSize = 64 * fs.Mebi + if !tristate.Valid { + tristate.Valid = true + tristate.Value = value } - useAlreadyExists = false // returns BucketAlreadyExists - // Storj doesn't support multi-part server side copy: - // https://github.com/storj/roadmap/issues/40 - // So make cutoff very large which it does support - opt.CopyCutoff = math.MaxInt64 - case "Synology": - useMultipartEtag = false - useAlreadyExists = false // untested - case "TencentCOS": - listObjectsV2 = false // untested - useMultipartEtag = false // untested - useAlreadyExists = false // untested - case "Wasabi": - useAlreadyExists = true // returns 200 OK - case "Leviia": - useAlreadyExists = false // untested - case "Qiniu": - useMultipartEtag = false - urlEncodeListings = false - virtualHostStyle = false - useAlreadyExists = false // untested - case "Zata": - useMultipartEtag = false - mightGzip = false - useUnsignedPayload = false - useAlreadyExists = false - case "Exaba": - virtualHostStyle = false - case "GCS": - // Google break request Signature by mutating accept-encoding HTTP header - // https://github.com/rclone/rclone/issues/6670 - useAcceptEncodingGzip = false - signAcceptEncoding = false - useAlreadyExists = true // returns BucketNameUnavailable instead of BucketAlreadyExists but good enough! - // GCS S3 doesn't support multi-part server side copy: - // See: https://issuetracker.google.com/issues/323465186 - // So make cutoff very large which it does seem to support - opt.CopyCutoff = math.MaxInt64 - // GCS doesn't like the x-id URL parameter the SDKv2 inserts - useXID = false - default: //nolint:gocritic // Don't include gocritic when running golangci-lint to avoid defaultCaseOrder: consider to make `default` case as first or as last case - fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider) - fallthrough - case "Other": - listObjectsV2 = false - virtualHostStyle = false - urlEncodeListings = false - useMultipartEtag = false - useAlreadyExists = false } - // Path Style vs Virtual Host style + // Set Path Style vs Virtual Host style + var virtualHostStyle = true // Default use bucket.provider.com instead of putting the bucket in the URL + if provider.Quirks.ForcePathStyle != nil { + // "don't force path style" so inverted + virtualHostStyle = !*provider.Quirks.ForcePathStyle + } if virtualHostStyle || opt.UseAccelerateEndpoint { opt.ForcePathStyle = false } - // Set to see if we need to URL encode listings - if !opt.ListURLEncode.Valid { - opt.ListURLEncode.Valid = true - opt.ListURLEncode.Value = urlEncodeListings - } - // Set the correct list version if not manually set if opt.ListVersion == 0 { - if listObjectsV2 { - opt.ListVersion = 2 + if provider.Quirks.ListVersion != nil { + opt.ListVersion = *provider.Quirks.ListVersion } else { - opt.ListVersion = 1 + opt.ListVersion = 2 } } - // Set the correct use multipart Etag for error checking if not manually set - if !opt.UseMultipartEtag.Valid { - opt.UseMultipartEtag.Valid = true - opt.UseMultipartEtag.Value = useMultipartEtag + // Set the copy cutoff if not manually set + // Check equality with strings as config values get round tripped via strings + if opt.CopyCutoff.String() == fs.SizeSuffix(maxSizeForCopy).String() { + if provider.Quirks.CopyCutoff != nil { + opt.CopyCutoff = fs.SizeSuffix(*provider.Quirks.CopyCutoff) + } } - // set MightGzip if not manually set - if !opt.MightGzip.Valid { - opt.MightGzip.Valid = true - opt.MightGzip.Value = mightGzip + // Clip the max upload parts to the quirk + if provider.Quirks.MaxUploadParts != nil { + opt.MaxUploadParts = min(opt.MaxUploadParts, *provider.Quirks.MaxUploadParts) } - // set UseAcceptEncodingGzip if not manually set - if !opt.UseAcceptEncodingGzip.Valid { - opt.UseAcceptEncodingGzip.Valid = true - opt.UseAcceptEncodingGzip.Value = useAcceptEncodingGzip + // Clip the chunk size to the quirk + if provider.Quirks.MinChunkSize != nil { + opt.ChunkSize = max(opt.ChunkSize, fs.SizeSuffix(*provider.Quirks.MinChunkSize)) } - // Has the provider got AlreadyOwnedByYou error? - if !opt.UseAlreadyExists.Valid { - opt.UseAlreadyExists.Valid = true - opt.UseAlreadyExists.Value = useAlreadyExists - } - - // Set the correct use multipart uploads if not manually set - if !opt.UseMultipartUploads.Valid { - opt.UseMultipartUploads.Valid = true - opt.UseMultipartUploads.Value = useMultipartUploads - } + // Set new style Tristate quirks + set(&opt.ListURLEncode, true, provider.Quirks.ListURLEncode) + set(&opt.UseMultipartEtag, true, provider.Quirks.UseMultipartEtag) + set(&opt.UseAcceptEncodingGzip, true, provider.Quirks.UseAcceptEncodingGzip) + set(&opt.MightGzip, true, provider.Quirks.MightGzip) + set(&opt.UseAlreadyExists, true, provider.Quirks.UseAlreadyExists) + set(&opt.UseMultipartUploads, true, provider.Quirks.UseMultipartUploads) if !opt.UseMultipartUploads.Value { opt.UploadCutoff = math.MaxInt64 } - - // Set the correct use multipart uploads if not manually set - if !opt.UseUnsignedPayload.Valid { - opt.UseUnsignedPayload.Valid = true - opt.UseUnsignedPayload.Value = useUnsignedPayload - } - - // Set the correct use UseXID if not manually set - if !opt.UseXID.Valid { - opt.UseXID.Valid = true - opt.UseXID.Value = useXID - } - - // Set the correct SignAcceptEncoding if not manually set - if !opt.SignAcceptEncoding.Valid { - opt.SignAcceptEncoding.Valid = true - opt.SignAcceptEncoding.Value = signAcceptEncoding - } + set(&opt.UseUnsignedPayload, true, provider.Quirks.UseUnsignedPayload) + set(&opt.UseXID, true, provider.Quirks.UseXID) + set(&opt.SignAcceptEncoding, true, provider.Quirks.SignAcceptEncoding) } // setRoot changes the root of the Fs @@ -4155,7 +1559,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:]) } srv := getClient(ctx, opt) - c, err := s3Connection(ctx, opt, srv) + c, provider, err := s3Connection(ctx, opt, srv) if err != nil { return nil, err } @@ -4205,26 +1609,16 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e UserMetadata: true, BucketBased: true, BucketBasedRootOK: true, - SetTier: true, - GetTier: true, + SetTier: provider.StorageClass.Len() > 0, + GetTier: provider.StorageClass.Len() > 0, SlowModTime: true, }).Fill(ctx, f) - if opt.Provider == "Storj" { - f.features.SetTier = false - f.features.GetTier = false - } - if opt.Provider == "IDrive" { - f.features.SetTier = false - } if opt.Provider == "AWS" { f.features.DoubleSlash = true } if opt.Provider == "Rabata" { f.features.Copy = nil } - if opt.Provider == "Hetzner" { - f.features.SetTier = false - } if opt.DirectoryMarkers { f.features.CanHaveEmptyDirectories = true } @@ -4392,7 +1786,7 @@ func (f *Fs) updateRegionForBucket(ctx context.Context, bucket string) error { // Make a new session with the new region oldRegion := f.opt.Region f.opt.Region = region - c, err := s3Connection(f.ctx, &f.opt, f.srv) + c, _, err := s3Connection(f.ctx, &f.opt, f.srv) if err != nil { return fmt.Errorf("creating new session failed: %w", err) } @@ -5801,7 +3195,7 @@ func (f *Fs) Command(ctx context.Context, name string, arg []string, opt map[str if err != nil { return nil, fmt.Errorf("reading config: %w", err) } - c, err := s3Connection(f.ctx, &newOpt, f.srv) + c, _, err := s3Connection(f.ctx, &newOpt, f.srv) if err != nil { return nil, fmt.Errorf("updating session: %w", err) } diff --git a/backend/s3/s3_test.go b/backend/s3/s3_test.go index 6d00d8761..973314c5c 100644 --- a/backend/s3/s3_test.go +++ b/backend/s3/s3_test.go @@ -62,14 +62,14 @@ func TestAWSDualStackOption(t *testing.T) { // test enabled ctx, opt, client := SetupS3Test(t) opt.UseDualStack = true - s3Conn, err := s3Connection(ctx, opt, client) + s3Conn, _, err := s3Connection(ctx, opt, client) require.NoError(t, err) assert.Equal(t, aws.DualStackEndpointStateEnabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint) } { // test default case ctx, opt, client := SetupS3Test(t) - s3Conn, err := s3Connection(ctx, opt, client) + s3Conn, _, err := s3Connection(ctx, opt, client) require.NoError(t, err) assert.Equal(t, aws.DualStackEndpointStateDisabled, s3Conn.Options().EndpointOptions.UseDualStackEndpoint) } diff --git a/fs/config/ui.go b/fs/config/ui.go index 6e31923fe..1a897aefe 100644 --- a/fs/config/ui.go +++ b/fs/config/ui.go @@ -3,6 +3,7 @@ package config import ( + "bufio" "context" "errors" "fmt" @@ -12,6 +13,7 @@ import ( "sort" "strconv" "strings" + "sync" "unicode/utf8" "github.com/peterh/liner" @@ -25,8 +27,24 @@ import ( "golang.org/x/text/unicode/norm" ) +var ( + stdinBufOnce sync.Once + stdinBuf *bufio.Reader +) + // ReadLine reads an unlimited length line from stdin with a prompt. var ReadLine = func(prompt string) string { + if !terminal.IsTerminal(int(os.Stdout.Fd())) { + stdinBufOnce.Do(func() { + stdinBuf = bufio.NewReader(os.Stdin) + }) + line, err := stdinBuf.ReadString('\n') + if err != nil && (line == "" || err != io.EOF) { + fs.Fatalf(nil, "Failed to read line: %v", err) + } + return strings.TrimSpace(line) + } + l := liner.NewLiner() defer func() { _ = l.Close() diff --git a/go.mod b/go.mod index a7f6ea9b7..6033ce8b8 100644 --- a/go.mod +++ b/go.mod @@ -75,6 +75,7 @@ require ( github.com/unknwon/goconfig v1.0.0 github.com/willscott/go-nfs v0.0.3 github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471 + github.com/wk8/go-ordered-map/v2 v2.1.8 github.com/xanzy/ssh-agent v0.3.3 github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 github.com/yunify/qingstor-sdk-go/v3 v3.2.0 @@ -126,10 +127,12 @@ require ( github.com/aws/aws-sdk-go-v2/service/sso v1.29.4 // indirect github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.0 // indirect github.com/aws/aws-sdk-go-v2/service/sts v1.38.5 // indirect + github.com/bahlo/generic-list-go v0.2.0 // indirect github.com/beorn7/perks v1.0.1 // indirect github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc // indirect github.com/bradenaw/juniper v0.15.3 // indirect github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 // indirect + github.com/buger/jsonparser v1.1.1 // indirect github.com/calebcase/tmpfile v1.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect github.com/chilts/sid v0.0.0-20190607042430-660e94789ec9 // indirect @@ -185,6 +188,7 @@ require ( github.com/lpar/date v1.0.0 // indirect github.com/lucasb-eyer/go-colorful v1.3.0 // indirect github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54 // indirect + github.com/mailru/easyjson v0.7.7 // indirect github.com/mattn/go-isatty v0.0.20 // indirect github.com/minio/crc64nvme v1.1.1 // indirect github.com/minio/md5-simd v1.1.2 // indirect diff --git a/go.sum b/go.sum index f3a86c7dc..853032209 100644 --- a/go.sum +++ b/go.sum @@ -148,6 +148,8 @@ github.com/aws/aws-sdk-go-v2/service/sts v1.38.5 h1:+LVB0xBqEgjQoqr9bGZbRzvg212B github.com/aws/aws-sdk-go-v2/service/sts v1.38.5/go.mod h1:xoaxeqnnUaZjPjaICgIy5B+MHCSb/ZSOn4MvkFNOUA0= github.com/aws/smithy-go v1.23.0 h1:8n6I3gXzWJB2DxBDnfxgBaSX6oe0d/t10qGz7OKqMCE= github.com/aws/smithy-go v1.23.0/go.mod h1:t1ufH5HMublsJYulve2RKmHDC15xu1f26kHCp/HgceI= +github.com/bahlo/generic-list-go v0.2.0 h1:5sz/EEAK+ls5wF+NeqDpk5+iNdMDXrh3z3nPnH1Wvgk= +github.com/bahlo/generic-list-go v0.2.0/go.mod h1:2KvAjgMlE5NNynlg/5iLrrCCZ2+5xWbdbCW3pNTGyYg= github.com/beorn7/perks v1.0.1 h1:VlbKKnNfV8bJzeqoa4cOKqO6bYr3WgKZxO8Z16+hsOM= github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6rlkpw= github.com/boombuler/barcode v1.0.1-0.20190219062509-6c824513bacc h1:biVzkmvwrH8WK8raXaxBx6fRVTlJILwEwQGL1I/ByEI= @@ -158,6 +160,8 @@ github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8 h1:GKTyiRCL6zVf5wWaq github.com/bradfitz/iter v0.0.0-20191230175014-e8f45d346db8/go.mod h1:spo1JLcs67NmW1aVLEgtA8Yy1elc+X8y5SRW1sFW4Og= github.com/buengese/sgzip v0.1.1 h1:ry+T8l1mlmiWEsDrH/YHZnCVWD2S3im1KLsyO+8ZmTU= github.com/buengese/sgzip v0.1.1/go.mod h1:i5ZiXGF3fhV7gL1xaRRL1nDnmpNj0X061FQzOS8VMas= +github.com/buger/jsonparser v1.1.1 h1:2PnMjfWD7wBILjqQbt530v576A/cAbQvEW9gGIpYMUs= +github.com/buger/jsonparser v1.1.1/go.mod h1:6RYKKt7H4d4+iWqouImQ9R2FZql3VbhNgx27UK13J/0= github.com/bwesterb/go-ristretto v1.2.0/go.mod h1:fUIoIZaG73pV5biE2Blr2xEzDoMj7NFEuV9ekS419A0= github.com/bytedance/sonic v1.13.2 h1:8/H1FempDZqC4VqjptGo14QQlJx8VdZJegxs6wwfqpQ= github.com/bytedance/sonic v1.13.2/go.mod h1:o68xyaF9u2gvVBuGHPlUVCy+ZfmNNO5ETf1+KgkJhz4= @@ -402,6 +406,7 @@ github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3 h1:ZxO6Qr2GOXPdcW80 github.com/jlaffaye/ftp v0.2.1-0.20240918233326-1b970516f5d3/go.mod h1:dvLUr/8Fs9a2OBrEnCC5duphbkz/k/mSy5OkXg3PAgI= github.com/josephspurrier/goversioninfo v1.5.0 h1:9TJtORoyf4YMoWSOo/cXFN9A/lB3PniJ91OxIH6e7Zg= github.com/josephspurrier/goversioninfo v1.5.0/go.mod h1:6MoTvFZ6GKJkzcdLnU5T/RGYUbHQbKpYeNP0AgQLd2o= +github.com/josharian/intern v1.0.0/go.mod h1:5DoeVV0s6jJacbCEi61lwdGj/aVlrQvzHFFd8Hwg//Y= github.com/json-iterator/go v1.1.12 h1:PV8peI4a0ysnczrg+LtxykD8LfKY9ML6u2jnxaEnrnM= github.com/json-iterator/go v1.1.12/go.mod h1:e30LSqwooZae/UwlEbR2852Gd8hjQvJoHmT4TnhNGBo= github.com/jstemmer/go-junit-report v0.0.0-20190106144839-af01ea7f8024/go.mod h1:6v2b51hI/fHJwM22ozAgKL4VKDeJcHhJFhtBdhmNjmU= @@ -447,6 +452,8 @@ github.com/lucasb-eyer/go-colorful v1.3.0 h1:2/yBRLdWBZKrf7gB40FoiKfAWYQ0lqNcbuQ github.com/lucasb-eyer/go-colorful v1.3.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0= github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54 h1:mFWunSatvkQQDhpdyuFAYwyAan3hzCuma+Pz8sqvOfg= github.com/lufia/plan9stats v0.0.0-20250827001030-24949be3fa54/go.mod h1:autxFIvghDt3jPTLoqZ9OZ7s9qTGNAWmYCjVFWPX/zg= +github.com/mailru/easyjson v0.7.7 h1:UGYAvKxe3sBsEDzO8ZeWOSlIQfWFlxbzLZe7hwFURr0= +github.com/mailru/easyjson v0.7.7/go.mod h1:xzfreul335JAWq5oZzymOObrkdz5UnU4kGfJJLY9Nlc= github.com/mattn/go-colorable v0.1.14 h1:9A9LHSqF/7dyVVX6g0U9cwm9pG3kP9gSzcuIPHPsaIE= github.com/mattn/go-colorable v0.1.14/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8= github.com/mattn/go-isatty v0.0.20 h1:xfD0iDuEKnDkl03q4limB+vH+GxLEtL/jb4xVJSWWEY= @@ -598,8 +605,6 @@ github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo= github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= -github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5 h1:Sa+sR8aaAMFwxhXWENEnE6ZpqhZ9d7u1RT2722Rw6hc= -github.com/t3rm1n4l/go-mega v0.0.0-20241213151442-a19cff0ec7b5/go.mod h1:UdZiFUFu6e2WjjtjxivwXWcwc1N/8zgbkBR9QNucUOY= github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c h1:BLopNCyqewbE8+BtlIp/Juzu8AJGxz0gHdGADnsblVc= github.com/t3rm1n4l/go-mega v0.0.0-20250926104142-ccb8d3498e6c/go.mod h1:ykucQyiE9Q2qx1wLlEtZkkNn1IURib/2O+Mvd25i1Fo= github.com/tailscale/depaware v0.0.0-20210622194025-720c4b409502/go.mod h1:p9lPsd+cx33L3H9nNoecRRxPssFKUwwI50I3pZ0yT+8= @@ -623,6 +628,8 @@ github.com/willscott/go-nfs-client v0.0.0-20240104095149-b44639837b00 h1:U0DnHRZ github.com/willscott/go-nfs-client v0.0.0-20240104095149-b44639837b00/go.mod h1:Tq++Lr/FgiS3X48q5FETemXiSLGuYMQT2sPjYNPJSwA= github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471 h1:aSOo0k+aLWdhUQiUxzv4cZ7cUp3OLP+Qx7cjs6OUxME= github.com/winfsp/cgofuse v1.6.1-0.20250813110601-7d90b0992471/go.mod h1:uxjoF2jEYT3+x+vC2KJddEGdk/LU8pRowXmyVMHSV5I= +github.com/wk8/go-ordered-map/v2 v2.1.8 h1:5h/BUHu93oj4gIdvHHHGsScSTMijfx5PeYkE/fJgbpc= +github.com/wk8/go-ordered-map/v2 v2.1.8/go.mod h1:5nJHM5DyteebpVlHnWMV0rPz6Zp7+xBAnxjb1X5vnTw= github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM= github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw= github.com/youmark/pkcs8 v0.0.0-20240726163527-a2c0da244d78 h1:ilQV1hzziu+LLM3zUTJ0trRztfwgjqKnBWNtSRkbmwM=