jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2026-01-08 03:23:26 +00:00
Code Issues Projects Releases Wiki Activity

dropbox: add --dropbox-batch-mode flag to speed up uploading #5156

Browse Source 
This adds 3 upload modes for dropbox off, sync and async and makes
sync the default.

This should improve uploads (especially for small files) greatly.
This commit is contained in:
Nick Craig-Wood
2020-09-07 11:08:46 +01:00
parent 77cda6773c
commit 2e4b65f888
3 changed files with 533 additions and 20 deletions
Download Patch File Download Diff File Expand all files Collapse all files

61
docs/content/dropbox.md
View File

@@ -100,7 +100,7 @@ Dropbox supports [its own hash
type](https://www.dropbox.com/developers/reference/content-hash) which
is checked for all transfers.
#### Restricted filename characters
### Restricted filename characters
| Character | Value | Replacement |
| --------- |:-----:|:-----------:|
@@ -119,6 +119,65 @@ These only get replaced if they are the last character in the name:
Invalid UTF-8 bytes will also be [replaced](/overview/#invalid-utf8),
as they can't be used in JSON strings.
### Batch mode uploads {#batch-mode}
Using batch mode uploads is very important for performance when using
the Dropbox API. See [the dropbox performance guide](https://developers.dropbox.com/dbx-performance-guide)
for more info.
There are 3 modes rclone can use for uploads.
#### --dropbox-batch-mode off
In this mode rclone will not use upload batching. This was the default
before rclone v1.55. It has the disadvantage that it is very likely to
encounter `too_many_requests` errors like this
NOTICE: too_many_requests/.: Too many requests or write operations. Trying again in 15 seconds.
When rclone receives these it has to wait for 15s or sometimes 300s
before continuing which really slows down transfers.
This will happen especially if `--transfers` is large, so this mode
isn't recommended except for compatibility or investigating problems.
#### --dropbox-batch-mode sync
In this mode rclone will batch up uploads to the size specified by
`--dropbox-batch-size` and commit them together.
Using this mode means you can use a much higher `--transfers`
parameter (32 or 64 works fine) without receiving `too_many_requests`
errors.
This mode ensures full data integrity.
Note that there may be a pause when quitting rclone while rclone
finishes up the last batch using this mode.
#### --dropbox-batch-mode async
In this mode rclone will batch up uploads to the size specified by
`--dropbox-batch-size` and commit them together.
However it will not wait for the status of the batch to be returned to
the caller. This means rclone can use a much bigger batch size (much
bigger than `--transfers`), at the cost of not being able to check the
status of the upload.
This provides the maximum possible upload speed especially with lots
of small files, however rclone can't check the file got uploaded
properly using this mode.
If you are using this mode then using "rclone check" after the
transfer completes is recommended. Or you could do an initial transfer
with `--dropbox-batch-mode async` then do a final transfer with
`--dropbox-batch-mode sync` (the default).
Note that there may be a pause when quitting rclone while rclone
finishes up the last batch using this mode.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/dropbox/dropbox.go then run make backenddocs" >}}
### Standard Options