jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-12-16 00:04:40 +00:00
Code Issues Projects Releases Wiki Activity

Version v1.59.0

Browse Source
This commit is contained in:
Nick Craig-Wood
2022-07-09 18:08:20 +01:00
parent 1c4ee2feee
commit 00a684d877
101 changed files with 21596 additions and 3818 deletions
Download Patch File Download Diff File Expand all files Collapse all files

310
docs/content/s3.md
View File

@@ -13,7 +13,7 @@ The S3 backend can be used with a number of different providers:
{{< provider name="Ceph" home="http://ceph.com/" config="/s3/#ceph" >}}
{{< provider name="China Mobile Ecloud Elastic Object Storage (EOS)" home="https://ecloud.10086.cn/home/product-introduction/eos/" config="/s3/#china-mobile-ecloud-eos" >}}
{{< provider name="Cloudflare R2" home="https://blog.cloudflare.com/r2-open-beta/" config="/s3/#cloudflare-r2" >}}
{{< provider name="Arvan Cloud Object Storage (AOS)" home="https://www.arvancloud.com/en/products/cloud-storage" config="/s3/#arvan-cloud-object-storage-aos" >}}
{{< provider name="Arvan Cloud Object Storage (AOS)" home="https://www.arvancloud.com/en/products/cloud-storage" config="/s3/#arvan-cloud" >}}
{{< provider name="DigitalOcean Spaces" home="https://www.digitalocean.com/products/object-storage/" config="/s3/#digitalocean-spaces" >}}
{{< provider name="Dreamhost" home="https://www.dreamhost.com/cloud/storage/" config="/s3/#dreamhost" >}}
{{< provider name="Huawei OBS" home="https://www.huaweicloud.com/intl/en-us/product/obs.html" config="/s3/#huawei-obs" >}}
@@ -571,7 +571,7 @@ A simple solution is to set the `--s3-upload-cutoff 0` and force all the files t
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/s3/s3.go then run make backenddocs" >}}
### Standard options
Here are the standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS).
Here are the Standard options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi).
#### --s3-provider
@@ -592,6 +592,8 @@ Properties:
- Ceph Object Storage
- "ChinaMobile"
- China Mobile Ecloud Elastic Object Storage (EOS)
- "Cloudflare"
- Cloudflare R2 Storage
- "ArvanCloud"
- Arvan Cloud Object Storage (AOS)
- "DigitalOcean"
@@ -828,6 +830,67 @@ Properties:
- Amsterdam, The Netherlands
- "fr-par"
- Paris, France
- "pl-waw"
- Warsaw, Poland
#### --s3-region
Region to connect to. - the location where your bucket will be created and your data stored. Need bo be same with your endpoint.
Properties:
- Config: region
- Env Var: RCLONE_S3_REGION
- Provider: HuaweiOBS
- Type: string
- Required: false
- Examples:
- "af-south-1"
- AF-Johannesburg
- "ap-southeast-2"
- AP-Bangkok
- "ap-southeast-3"
- AP-Singapore
- "cn-east-3"
- CN East-Shanghai1
- "cn-east-2"
- CN East-Shanghai2
- "cn-north-1"
- CN North-Beijing1
- "cn-north-4"
- CN North-Beijing4
- "cn-south-1"
- CN South-Guangzhou
- "ap-southeast-1"
- CN-Hong Kong
- "sa-argentina-1"
- LA-Buenos Aires1
- "sa-peru-1"
- LA-Lima1
- "na-mexico-1"
- LA-Mexico City1
- "sa-chile-1"
- LA-Santiago2
- "sa-brazil-1"
- LA-Sao Paulo1
- "ru-northwest-2"
- RU-Moscow2
#### --s3-region
Region to connect to.
Properties:
- Config: region
- Env Var: RCLONE_S3_REGION
- Provider: Cloudflare
- Type: string
- Required: false
- Examples:
- "auto"
- R2 buckets are automatically distributed across Cloudflare's data centers for low latency.
#### --s3-region
@@ -839,7 +902,7 @@ Properties:
- Config: region
- Env Var: RCLONE_S3_REGION
- Provider: !AWS,Alibaba,ChinaMobile,ArvanCloud,RackCorp,Scaleway,Storj,TencentCOS,HuaweiOBS,IDrive
- Provider: !AWS,Alibaba,ChinaMobile,Cloudflare,ArvanCloud,RackCorp,Scaleway,Storj,TencentCOS,HuaweiOBS,IDrive
- Type: string
- Required: false
- Examples:
@@ -868,6 +931,8 @@ Properties:
Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: ChinaMobile
@@ -925,7 +990,7 @@ Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
- Gansu China (Lanzhou)
- "eos-shanxi-1.cmecloud.cn"
- Shanxi China (Taiyuan)
- eos-liaoning-1.cmecloud.cn"
- "eos-liaoning-1.cmecloud.cn"
- Liaoning China (Shenyang)
- "eos-hebei-1.cmecloud.cn"
- Hebei China (Shijiazhuang)
@@ -940,6 +1005,8 @@ Endpoint for China Mobile Ecloud Elastic Object Storage (EOS) API.
Endpoint for Arvan Cloud Object Storage (AOS) API.
Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: ArvanCloud
@@ -952,50 +1019,6 @@ Endpoint for Arvan Cloud Object Storage (AOS) API.
- "s3.ir-tbz-sh1.arvanstorage.com"
- Tabriz Iran (Shahriar)
#### --s3-endpoint
Endpoint for Huawei Cloud Object Storage Service (OBS) API.
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: HuaweiOBS
- Type: string
- Required: false
- Examples:
- "obs.af-south-1.myhuaweicloud.com"
- AF-Johannesburg Endpoint
- "obs.ap-southeast-2.myhuaweicloud.com"
- AP-Bangkok Endpoint
- "obs.ap-southeast-3.myhuaweicloud.com"
- AP-Singapore Endpoint
- "obs.cn-east-3.myhuaweicloud.com"
- CN East-Shanghai1 Endpoint
- "obs.cn-east-2.myhuaweicloud.com"
- CN East-Shanghai2 Endpoint
- "obs.cn-north-1.myhuaweicloud.com"
- CN North-Beijing1 Endpoint
- "obs.cn-north-4.myhuaweicloud.com"
- CN North-Beijing4 Endpoint
- "obs.cn-south-1.myhuaweicloud.com"
- CN South-Guangzhou Endpoint
- "obs.ap-southeast-1.myhuaweicloud.com"
- CN-Hong Kong Endpoint
- "obs.sa-argentina-1.myhuaweicloud.com"
- LA-Buenos Aires1 Endpoint
- "obs.sa-peru-1.myhuaweicloud.com"
- LA-Lima1 Endpoint
- "obs.na-mexico-1.myhuaweicloud.com"
- LA-Mexico City1 Endpoint
- "obs.sa-chile-1.myhuaweicloud.com"
- LA-Santiago2 Endpoint
- "obs.sa-brazil-1.myhuaweicloud.com"
- LA-Sao Paulo1 Endpoint
- "obs.ru-northwest-2.myhuaweicloud.com"
- RU-Moscow2 Endpoint
#### --s3-endpoint
Endpoint for IBM COS S3 API.
@@ -1200,6 +1223,49 @@ Properties:
#### --s3-endpoint
Endpoint for OBS API.
Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: HuaweiOBS
- Type: string
- Required: false
- Examples:
- "obs.af-south-1.myhuaweicloud.com"
- AF-Johannesburg
- "obs.ap-southeast-2.myhuaweicloud.com"
- AP-Bangkok
- "obs.ap-southeast-3.myhuaweicloud.com"
- AP-Singapore
- "obs.cn-east-3.myhuaweicloud.com"
- CN East-Shanghai1
- "obs.cn-east-2.myhuaweicloud.com"
- CN East-Shanghai2
- "obs.cn-north-1.myhuaweicloud.com"
- CN North-Beijing1
- "obs.cn-north-4.myhuaweicloud.com"
- CN North-Beijing4
- "obs.cn-south-1.myhuaweicloud.com"
- CN South-Guangzhou
- "obs.ap-southeast-1.myhuaweicloud.com"
- CN-Hong Kong
- "obs.sa-argentina-1.myhuaweicloud.com"
- LA-Buenos Aires1
- "obs.sa-peru-1.myhuaweicloud.com"
- LA-Lima1
- "obs.na-mexico-1.myhuaweicloud.com"
- LA-Mexico City1
- "obs.sa-chile-1.myhuaweicloud.com"
- LA-Santiago2
- "obs.sa-brazil-1.myhuaweicloud.com"
- LA-Sao Paulo1
- "obs.ru-northwest-2.myhuaweicloud.com"
- RU-Moscow2
#### --s3-endpoint
Endpoint for Scaleway Object Storage.
Properties:
@@ -1214,6 +1280,8 @@ Properties:
- Amsterdam Endpoint
- "s3.fr-par.scw.cloud"
- Paris Endpoint
- "s3.pl-waw.scw.cloud"
- Warsaw Endpoint
#### --s3-endpoint
@@ -1365,7 +1433,7 @@ Properties:
- Config: endpoint
- Env Var: RCLONE_S3_ENDPOINT
- Provider: !AWS,IBMCOS,IDrive,TencentCOS,Alibaba,ChinaMobile,ArvanCloud,Scaleway,StackPath,Storj,RackCorp,HuaweiOBS
- Provider: !AWS,IBMCOS,IDrive,TencentCOS,HuaweiOBS,Alibaba,ChinaMobile,ArvanCloud,Scaleway,StackPath,Storj,RackCorp
- Type: string
- Required: false
- Examples:
@@ -1465,6 +1533,100 @@ Properties:
#### --s3-location-constraint
Location constraint - must match endpoint.
Used when creating buckets only.
Properties:
- Config: location_constraint
- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
- Provider: ChinaMobile
- Type: string
- Required: false
- Examples:
- "wuxi1"
- East China (Suzhou)
- "jinan1"
- East China (Jinan)
- "ningbo1"
- East China (Hangzhou)
- "shanghai1"
- East China (Shanghai-1)
- "zhengzhou1"
- Central China (Zhengzhou)
- "hunan1"
- Central China (Changsha-1)
- "zhuzhou1"
- Central China (Changsha-2)
- "guangzhou1"
- South China (Guangzhou-2)
- "dongguan1"
- South China (Guangzhou-3)
- "beijing1"
- North China (Beijing-1)
- "beijing2"
- North China (Beijing-2)
- "beijing4"
- North China (Beijing-3)
- "huhehaote1"
- North China (Huhehaote)
- "chengdu1"
- Southwest China (Chengdu)
- "chongqing1"
- Southwest China (Chongqing)
- "guiyang1"
- Southwest China (Guiyang)
- "xian1"
- Nouthwest China (Xian)
- "yunnan"
- Yunnan China (Kunming)
- "yunnan2"
- Yunnan China (Kunming-2)
- "tianjin1"
- Tianjin China (Tianjin)
- "jilin1"
- Jilin China (Changchun)
- "hubei1"
- Hubei China (Xiangyan)
- "jiangxi1"
- Jiangxi China (Nanchang)
- "gansu1"
- Gansu China (Lanzhou)
- "shanxi1"
- Shanxi China (Taiyuan)
- "liaoning1"
- Liaoning China (Shenyang)
- "hebei1"
- Hebei China (Shijiazhuang)
- "fujian1"
- Fujian China (Xiamen)
- "guangxi1"
- Guangxi China (Nanning)
- "anhui1"
- Anhui China (Huainan)
#### --s3-location-constraint
Location constraint - must match endpoint.
Used when creating buckets only.
Properties:
- Config: location_constraint
- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
- Provider: ArvanCloud
- Type: string
- Required: false
- Examples:
- "ir-thr-at1"
- Tehran Iran (Asiatech)
- "ir-tbz-sh1"
- Tabriz Iran (Shahriar)
#### --s3-location-constraint
Location constraint - must match endpoint when using IBM Cloud Public.
For on-prem COS, do not make a selection from this list, hit enter.
@@ -1604,7 +1766,7 @@ Properties:
- Config: location_constraint
- Env Var: RCLONE_S3_LOCATION_CONSTRAINT
- Provider: !AWS,IBMCOS,IDrive,Alibaba,ChinaMobile,ArvanCloud,RackCorp,Scaleway,StackPath,Storj,TencentCOS,HuaweiOBS
- Provider: !AWS,IBMCOS,IDrive,Alibaba,HuaweiOBS,ChinaMobile,Cloudflare,ArvanCloud,RackCorp,Scaleway,StackPath,Storj,TencentCOS
- Type: string
- Required: false
@@ -1623,7 +1785,7 @@ Properties:
- Config: acl
- Env Var: RCLONE_S3_ACL
- Provider: !Storj
- Provider: !Storj,Cloudflare
- Type: string
- Required: false
- Examples:
@@ -1676,7 +1838,7 @@ Properties:
- Config: server_side_encryption
- Env Var: RCLONE_S3_SERVER_SIDE_ENCRYPTION
- Provider: AWS,Ceph,ChinaMobile,ArvanCloud,Minio
- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -1760,6 +1922,8 @@ Properties:
The storage class to use when storing new objects in ChinaMobile.
Properties:
- Config: storage_class
- Env Var: RCLONE_S3_STORAGE_CLASS
- Provider: ChinaMobile
@@ -1779,6 +1943,8 @@ The storage class to use when storing new objects in ChinaMobile.
The storage class to use when storing new objects in ArvanCloud.
Properties:
- Config: storage_class
- Env Var: RCLONE_S3_STORAGE_CLASS
- Provider: ArvanCloud
@@ -1832,7 +1998,7 @@ Properties:
### Advanced options
Here are the advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, ChinaMobile, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, Lyve Cloud, Minio, RackCorp, SeaweedFS, and Tencent COS).
Here are the Advanced options specific to s3 (Amazon S3 Compliant Storage Providers including AWS, Alibaba, Ceph, China Mobile, Cloudflare, ArvanCloud, Digital Ocean, Dreamhost, Huawei OBS, IBM COS, IDrive e2, Lyve Cloud, Minio, Netease, RackCorp, Scaleway, SeaweedFS, StackPath, Storj, Tencent COS and Wasabi).
#### --s3-bucket-acl
@@ -1884,7 +2050,7 @@ Properties:
- Config: sse_customer_algorithm
- Env Var: RCLONE_S3_SSE_CUSTOMER_ALGORITHM
- Provider: AWS,Ceph,ChinaMobile,ArvanCloud,Minio
- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -1901,7 +2067,7 @@ Properties:
- Config: sse_customer_key
- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY
- Provider: AWS,Ceph,ChinaMobile,ArvanCloud,Minio
- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -1919,7 +2085,7 @@ Properties:
- Config: sse_customer_key_md5
- Env Var: RCLONE_S3_SSE_CUSTOMER_KEY_MD5
- Provider: AWS,Ceph,ChinaMobile,ArvanCloud,Minio
- Provider: AWS,Ceph,ChinaMobile,Minio
- Type: string
- Required: false
- Examples:
@@ -1964,6 +2130,13 @@ most 10,000 chunks, this means that by default the maximum size of
a file you can stream upload is 48 GiB. If you wish to stream upload
larger files then you will need to increase chunk_size.
Increasing the chunk size decreases the accuracy of the progress
statistics displayed with "-P" flag. Rclone treats chunk as sent when
it's buffered by the AWS SDK, when in fact it may still be uploading.
A bigger chunk size means a bigger AWS SDK buffer and progress
reporting more deviating from the truth.
Properties:
- Config: chunk_size
@@ -2369,6 +2542,26 @@ Properties:
- Type: Tristate
- Default: unset
#### --s3-use-presigned-request
Whether to use a presigned request or PutObject for single part uploads
If this is false rclone will use PutObject from the AWS SDK to upload
an object.
Versions of rclone < 1.59 use presigned requests to upload a single
part object and setting this flag to true will re-enable that
functionality. This shouldn't be necessary except in exceptional
circumstances or for testing.
Properties:
- Config: use_presigned_request
- Env Var: RCLONE_S3_USE_PRESIGNED_REQUEST
- Type: bool
- Default: false
### Metadata
User metadata is stored as x-amz-meta- keys. S3 metadata keys are case insensitive and are always returned in lower case.
@@ -2386,7 +2579,6 @@ Here are the possible system metadata items for the s3 backend.
| mtime | Time of last modification, read from rclone metadata | RFC 3339 | 2006-01-02T15:04:05.999999999Z07:00 | N |
| tier | Tier of the object | string | GLACIER | **Y** |
See the [metadata](/docs/#metadata) docs for more info.
## Backend commands
@@ -2399,7 +2591,7 @@ Run them with
The help below will explain what arguments each command takes.
See [the "rclone backend" command](/commands/rclone_backend/) for more
See the [backend](/commands/rclone_backend/) command for more
info on how to pass options and arguments.
These can be run on a running backend using the rc command
@@ -3991,7 +4183,7 @@ d) Delete this remote
y/e/d> y
```
### ArvanCloud
### ArvanCloud {#arvan-cloud}
[ArvanCloud](https://www.arvancloud.com/en/products/cloud-storage) ArvanCloud Object Storage goes beyond the limited traditional file storage.
It gives you access to backup and archived files and allows sharing.