mirror of
https://github.com/rclone/rclone.git
synced 2026-01-12 13:34:01 +00:00
Compare commits
1 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
094cc27621 |
3
.gitignore
vendored
3
.gitignore
vendored
@@ -4,3 +4,6 @@ rclone
|
||||
rclonetest/rclonetest
|
||||
build
|
||||
docs/public
|
||||
README.html
|
||||
README.txt
|
||||
rclone.1
|
||||
|
||||
22
.travis.yml
22
.travis.yml
@@ -1,21 +1,11 @@
|
||||
language: go
|
||||
sudo: false
|
||||
osx_image: xcode7.3
|
||||
|
||||
os:
|
||||
- linux
|
||||
- osx
|
||||
|
||||
go:
|
||||
- 1.5.4
|
||||
- 1.6.3
|
||||
- 1.7
|
||||
|
||||
# - tip
|
||||
|
||||
install:
|
||||
- make build_dep
|
||||
- 1.1.2
|
||||
- 1.2.2
|
||||
- 1.3
|
||||
- tip
|
||||
|
||||
script:
|
||||
- make check
|
||||
- make quicktest
|
||||
- go get ./...
|
||||
- go test -v ./...
|
||||
|
||||
162
CONTRIBUTING.md
162
CONTRIBUTING.md
@@ -1,162 +0,0 @@
|
||||
# Contributing to rclone #
|
||||
|
||||
This is a short guide on how to contribute things to rclone.
|
||||
|
||||
## Reporting a bug ##
|
||||
|
||||
Bug reports are welcome. Check your issue exists with the latest
|
||||
version first. Please add when submitting:
|
||||
|
||||
* Rclone version (eg output from `rclone -V`)
|
||||
* Which OS you are using and how many bits (eg Windows 7, 64 bit)
|
||||
* The command you were trying to run (eg `rclone copy /tmp remote:tmp`)
|
||||
* A log of the command with the `-v` flag (eg output from `rclone -v copy /tmp remote:tmp`)
|
||||
* if the log contains secrets then edit the file with a text editor first to obscure them
|
||||
|
||||
## Submitting a pull request ##
|
||||
|
||||
If you find a bug that you'd like to fix, or a new feature that you'd
|
||||
like to implement then please submit a pull request via Github.
|
||||
|
||||
If it is a big feature then make an issue first so it can be discussed.
|
||||
|
||||
You'll need a Go environment set up with GOPATH set. See [the Go
|
||||
getting started docs](https://golang.org/doc/install) for more info.
|
||||
|
||||
First in your web browser press the fork button on [rclone's Github
|
||||
page](https://github.com/ncw/rclone).
|
||||
|
||||
Now in your terminal
|
||||
|
||||
go get github.com/ncw/rclone
|
||||
cd $GOPATH/src/github.com/ncw/rclone
|
||||
git remote rename origin upstream
|
||||
git remote add origin git@github.com:YOURUSER/rclone.git
|
||||
|
||||
Make a branch to add your new feature
|
||||
|
||||
git checkout -b my-new-feature
|
||||
|
||||
And get hacking.
|
||||
|
||||
When ready - run the unit tests for the code you changed
|
||||
|
||||
go test -v
|
||||
|
||||
Note that you make need to make a test remote, eg `TestSwift` for some
|
||||
of the unit tests.
|
||||
|
||||
Note the top level Makefile targets
|
||||
|
||||
* make check
|
||||
* make test
|
||||
|
||||
Both of these will be run by Travis when you make a pull request but
|
||||
you can do this yourself locally too.
|
||||
|
||||
Make sure you
|
||||
|
||||
* Add documentation for a new feature
|
||||
* Add unit tests for a new feature
|
||||
* squash commits down to one per feature
|
||||
* rebase to master `git rebase master`
|
||||
|
||||
When you are done with that
|
||||
|
||||
git push origin my-new-feature
|
||||
|
||||
Go to the Github website and click [Create pull
|
||||
request](https://help.github.com/articles/creating-a-pull-request/).
|
||||
|
||||
You patch will get reviewed and you might get asked to fix some stuff.
|
||||
|
||||
If so, then make the changes in the same branch, squash the commits,
|
||||
rebase it to master then push it to Github with `--force`.
|
||||
|
||||
## Testing ##
|
||||
|
||||
rclone's tests are run from the go testing framework, so at the top
|
||||
level you can run this to run all the tests.
|
||||
|
||||
go test -v ./...
|
||||
|
||||
rclone contains a mixture of unit tests and integration tests.
|
||||
Because it is difficult (and in some respects pointless) to test cloud
|
||||
storage systems by mocking all their interfaces, rclone unit tests can
|
||||
run against any of the backends. This is done by making specially
|
||||
named remotes in the default config file.
|
||||
|
||||
If you wanted to test changes in the `drive` backend, then you would
|
||||
need to make a remote called `TestDrive`.
|
||||
|
||||
You can then run the unit tests in the drive directory. These tests
|
||||
are skipped if `TestDrive:` isn't defined.
|
||||
|
||||
cd drive
|
||||
go test -v
|
||||
|
||||
You can then run the integration tests which tests all of rclone's
|
||||
operations. Normally these get run against the local filing system,
|
||||
but they can be run against any of the remotes.
|
||||
|
||||
cd ../fs
|
||||
go test -v -remote TestDrive:
|
||||
go test -v -remote TestDrive: -subdir
|
||||
|
||||
If you want to run all the integration tests against all the remotes,
|
||||
then run in that directory
|
||||
|
||||
go run test_all.go
|
||||
|
||||
## Making a release ##
|
||||
|
||||
There are separate instructions for making a release in the RELEASE.md
|
||||
file - doing the first few steps is useful before making a
|
||||
contribution.
|
||||
|
||||
* go get -u -f -v ./...
|
||||
* make check
|
||||
* make test
|
||||
* make tag
|
||||
|
||||
## Writing a new backend ##
|
||||
|
||||
Choose a name. The docs here will use `remote` as an example.
|
||||
|
||||
Note that in rclone terminology a file system backend is called a
|
||||
remote or an fs.
|
||||
|
||||
Research
|
||||
|
||||
* Look at the interfaces defined in `fs/fs.go`
|
||||
* Study one or more of the existing remotes
|
||||
|
||||
Getting going
|
||||
|
||||
* Create `remote/remote.go` (copy this from a similar fs)
|
||||
* Add your fs to the imports in `fs/all/all.go`
|
||||
|
||||
Unit tests
|
||||
|
||||
* Create a config entry called `TestRemote` for the unit tests to use
|
||||
* Add your fs to the end of `fstest/fstests/gen_tests.go`
|
||||
* generate `remote/remote_test.go` unit tests `cd fstest/fstests; go generate`
|
||||
* Make sure all tests pass with `go test -v`
|
||||
|
||||
Integration tests
|
||||
|
||||
* Add your fs to `fs/test_all.go`
|
||||
* Make sure integration tests pass with
|
||||
* `cd fs`
|
||||
* `go test -v -remote TestRemote:` and
|
||||
* `go test -v -remote TestRemote: -subdir`
|
||||
|
||||
Add your fs to the docs
|
||||
|
||||
* `README.md` - main Github page
|
||||
* `docs/content/remote.md` - main docs page
|
||||
* `docs/content/overview.md` - overview docs
|
||||
* `docs/content/docs.md` - list of remotes in config section
|
||||
* `docs/content/about.md` - front page of rclone.org
|
||||
* `docs/layouts/chrome/navbar.html` - add it to the website navigation
|
||||
* `make_manual.py` - add the page to the `docs` constant
|
||||
@@ -1,14 +0,0 @@
|
||||
When filing an issue, please include the following information if
|
||||
possible as well as a description of the problem. Make sure you are
|
||||
using the [latest version of rclone](http://rclone.org/downloads/).
|
||||
|
||||
> What is your rclone version (eg output from `rclone -V`)
|
||||
|
||||
> Which OS you are using and how many bits (eg Windows 7, 64 bit)
|
||||
|
||||
> Which cloud storage system are you using? (eg Google Drive)
|
||||
|
||||
> The command you were trying to run (eg `rclone copy /tmp remote:tmp`)
|
||||
|
||||
> A log from the command with the `-v` flag (eg output from `rclone -v copy /tmp remote:tmp`)
|
||||
|
||||
3169
MANUAL.html
3169
MANUAL.html
File diff suppressed because it is too large
Load Diff
4625
MANUAL.txt
4625
MANUAL.txt
File diff suppressed because it is too large
Load Diff
83
Makefile
83
Makefile
@@ -1,106 +1,59 @@
|
||||
SHELL = /bin/bash
|
||||
TAG := $(shell git describe --tags)
|
||||
LAST_TAG := $(shell git describe --tags --abbrev=0)
|
||||
NEW_TAG := $(shell echo $(LAST_TAG) | perl -lpe 's/v//; $$_ += 0.01; $$_ = sprintf("v%.2f", $$_)')
|
||||
|
||||
rclone:
|
||||
rclone: *.go */*.go
|
||||
@go version
|
||||
go install -v ./...
|
||||
go build
|
||||
|
||||
# Full suite of integration tests
|
||||
test: rclone
|
||||
go test ./...
|
||||
cd fs && go run test_all.go
|
||||
doc: rclone.1 README.html README.txt
|
||||
|
||||
# Quick test
|
||||
quicktest:
|
||||
go test ./...
|
||||
go test -cpu=2 -race ./...
|
||||
rclone.1: README.md
|
||||
pandoc -s --from markdown --to man README.md -o rclone.1
|
||||
|
||||
# Do source code quality checks
|
||||
check: rclone
|
||||
go vet ./...
|
||||
errcheck ./...
|
||||
goimports -d . | grep . ; test $$? -eq 1
|
||||
golint ./... | grep -E -v '(StorageUrl|CdnUrl)' ; test $$? -eq 1
|
||||
README.html: README.md
|
||||
pandoc -s --from markdown_github --to html README.md -o README.html
|
||||
|
||||
# Get the build dependencies
|
||||
build_dep:
|
||||
go get -t ./...
|
||||
go get -u github.com/kisielk/errcheck
|
||||
go get -u golang.org/x/tools/cmd/goimports
|
||||
go get -u github.com/golang/lint/golint
|
||||
|
||||
# Update dependencies
|
||||
update:
|
||||
go get -t -u -f -v ./...
|
||||
|
||||
doc: rclone.1 MANUAL.html MANUAL.txt
|
||||
|
||||
rclone.1: MANUAL.md
|
||||
pandoc -s --from markdown --to man MANUAL.md -o rclone.1
|
||||
|
||||
MANUAL.md: make_manual.py docs/content/*.md commanddocs
|
||||
./make_manual.py
|
||||
|
||||
MANUAL.html: MANUAL.md
|
||||
pandoc -s --from markdown --to html MANUAL.md -o MANUAL.html
|
||||
|
||||
MANUAL.txt: MANUAL.md
|
||||
pandoc -s --from markdown --to plain MANUAL.md -o MANUAL.txt
|
||||
|
||||
commanddocs: rclone
|
||||
rclone gendocs docs/content/commands/
|
||||
README.txt: README.md
|
||||
pandoc -s --from markdown_github --to plain README.md -o README.txt
|
||||
|
||||
install: rclone
|
||||
install -d ${DESTDIR}/usr/bin
|
||||
install -t ${DESTDIR}/usr/bin ${GOPATH}/bin/rclone
|
||||
install -t ${DESTDIR}/usr/bin rclone
|
||||
|
||||
clean:
|
||||
go clean ./...
|
||||
find . -name \*~ | xargs -r rm -f
|
||||
rm -rf build docs/public
|
||||
rm -f rclone rclonetest/rclonetest
|
||||
rm -f rclone rclonetest/rclonetest rclone.1 README.html README.txt
|
||||
|
||||
website:
|
||||
cd docs && hugo
|
||||
|
||||
upload_website: website
|
||||
rclone -v sync docs/public memstore:www-rclone-org
|
||||
./rclone -v sync docs/public memstore:www-rclone-org
|
||||
|
||||
upload:
|
||||
rclone -v copy build/ memstore:downloads-rclone-org
|
||||
|
||||
upload_github:
|
||||
./upload-github $(TAG)
|
||||
./rclone -v copy build/ memstore:downloads-rclone-org
|
||||
|
||||
cross: doc
|
||||
./cross-compile $(TAG)
|
||||
|
||||
beta:
|
||||
./cross-compile $(TAG)β
|
||||
rm build/*-current-*
|
||||
rclone -v copy build/ memstore:pub-rclone-org/$(TAG)β
|
||||
@echo Beta release ready at http://pub.rclone.org/$(TAG)%CE%B2/
|
||||
|
||||
serve: website
|
||||
serve:
|
||||
cd docs && hugo server -v -w
|
||||
|
||||
tag: doc
|
||||
tag:
|
||||
@echo "Old tag is $(LAST_TAG)"
|
||||
@echo "New tag is $(NEW_TAG)"
|
||||
echo -e "package fs\n\n// Version of rclone\nvar Version = \"$(NEW_TAG)-DEV\"\n" | gofmt > fs/version.go
|
||||
echo -e "package fs\n const Version = \"$(NEW_TAG)\"\n" | gofmt > fs/version.go
|
||||
perl -lpe 's/VERSION/${NEW_TAG}/g; s/DATE/'`date -I`'/g;' docs/content/downloads.md.in > docs/content/downloads.md
|
||||
git tag $(NEW_TAG)
|
||||
@echo "Add this to changelog in docs/content/changelog.md"
|
||||
@echo "Add this to changelog in README.md"
|
||||
@echo " * $(NEW_TAG) -" `date -I`
|
||||
@git log $(LAST_TAG)..$(NEW_TAG) --oneline
|
||||
@echo "Then commit the changes"
|
||||
@echo git commit -m \"Version $(NEW_TAG)\" -a -v
|
||||
@echo git commit -m "Version $(NEW_TAG)" -a -v
|
||||
@echo "And finally run make retag before make cross etc"
|
||||
|
||||
retag:
|
||||
git tag -f $(LAST_TAG)
|
||||
|
||||
gen_tests:
|
||||
cd fstest/fstests && go generate
|
||||
|
||||
321
README.md
321
README.md
@@ -1,15 +1,12 @@
|
||||
% rclone(1) User Manual
|
||||
% Nick Craig-Wood
|
||||
% Jul 7, 2014
|
||||
|
||||
Rclone
|
||||
======
|
||||
|
||||
[](http://rclone.org/)
|
||||
|
||||
[Website](http://rclone.org) |
|
||||
[Documentation](http://rclone.org/docs/) |
|
||||
[Contributing](CONTRIBUTING.md) |
|
||||
[Changelog](http://rclone.org/changelog/) |
|
||||
[Installation](http://rclone.org/install/) |
|
||||
[G+](https://google.com/+RcloneOrg)
|
||||
|
||||
|
||||
[](https://travis-ci.org/ncw/rclone) [](https://ci.appveyor.com/project/ncw/rclone) [](https://godoc.org/github.com/ncw/rclone)
|
||||
|
||||
Rclone is a command line program to sync files and directories to and from
|
||||
|
||||
* Google Drive
|
||||
@@ -17,30 +14,312 @@ Rclone is a command line program to sync files and directories to and from
|
||||
* Openstack Swift / Rackspace cloud files / Memset Memstore
|
||||
* Dropbox
|
||||
* Google Cloud Storage
|
||||
* Amazon Drive
|
||||
* Microsoft One Drive
|
||||
* Hubic
|
||||
* Backblaze B2
|
||||
* Yandex Disk
|
||||
* The local filesystem
|
||||
|
||||
Features
|
||||
|
||||
* MD5/SHA1 hashes checked at all times for file integrity
|
||||
* MD5SUMs checked at all times for file integrity
|
||||
* Timestamps preserved on files
|
||||
* Partial syncs supported on a whole file basis
|
||||
* Copy mode to just copy new/changed files
|
||||
* Sync (one way) mode to make a directory identical
|
||||
* Check mode to check for file hash equality
|
||||
* Can sync to and from network, eg two different cloud accounts
|
||||
* Sync mode to make a directory identical
|
||||
* Check mode to check all MD5SUMs
|
||||
* Can sync to and from network, eg two different Drive accounts
|
||||
|
||||
See the home page for installation, usage, documentation, changelog
|
||||
and configuration walkthroughs.
|
||||
See the Home page for more documentation and configuration walkthroughs.
|
||||
|
||||
* http://rclone.org/
|
||||
|
||||
Install
|
||||
-------
|
||||
|
||||
Rclone is a Go program and comes as a single binary file.
|
||||
|
||||
Download the binary for your OS from
|
||||
|
||||
* http://rclone.org/downloads/
|
||||
|
||||
Or alternatively if you have Go installed use
|
||||
|
||||
go install github.com/ncw/rclone
|
||||
|
||||
and this will build the binary in `$GOPATH/bin`.
|
||||
|
||||
Configure
|
||||
---------
|
||||
|
||||
First you'll need to configure rclone. As the object storage systems
|
||||
have quite complicated authentication these are kept in a config file
|
||||
`.rclone.conf` in your home directory by default. (You can use the
|
||||
`--config` option to choose a different config file.)
|
||||
|
||||
The easiest way to make the config is to run rclone with the config
|
||||
option, Eg
|
||||
|
||||
rclone config
|
||||
|
||||
Usage
|
||||
-----
|
||||
|
||||
Rclone syncs a directory tree from local to remote.
|
||||
|
||||
Its basic syntax is
|
||||
|
||||
Syntax: [options] subcommand <parameters> <parameters...>
|
||||
|
||||
See below for how to specify the source and destination paths.
|
||||
|
||||
Subcommands
|
||||
-----------
|
||||
|
||||
rclone copy source:path dest:path
|
||||
|
||||
Copy the source to the destination. Doesn't transfer
|
||||
unchanged files, testing first by modification time then by
|
||||
MD5SUM. Doesn't delete files from the destination.
|
||||
|
||||
rclone sync source:path dest:path
|
||||
|
||||
Sync the source to the destination. Doesn't transfer
|
||||
unchanged files, testing first by modification time then by
|
||||
MD5SUM. Deletes any files that exist in source that don't
|
||||
exist in destination. Since this can cause data loss, test
|
||||
first with the `--dry-run` flag.
|
||||
|
||||
rclone ls [remote:path]
|
||||
|
||||
List all the objects in the the path with sizes.
|
||||
|
||||
rclone lsl [remote:path]
|
||||
|
||||
List all the objects in the the path with sizes and timestamps.
|
||||
|
||||
rclone lsd [remote:path]
|
||||
|
||||
List all directories/objects/buckets in the the path.
|
||||
|
||||
rclone mkdir remote:path
|
||||
|
||||
Make the path if it doesn't already exist
|
||||
|
||||
rclone rmdir remote:path
|
||||
|
||||
Remove the path. Note that you can't remove a path with
|
||||
objects in it, use purge for that.
|
||||
|
||||
rclone purge remote:path
|
||||
|
||||
Remove the path and all of its contents.
|
||||
|
||||
rclone check source:path dest:path
|
||||
|
||||
Checks the files in the source and destination match. It
|
||||
compares sizes and MD5SUMs and prints a report of files which
|
||||
don't match. It doesn't alter the source or destination.
|
||||
|
||||
rclone md5sum remote:path
|
||||
|
||||
Produces an md5sum file for all the objects in the path. This is in
|
||||
the same format as the standard md5sum tool produces.
|
||||
|
||||
General options:
|
||||
|
||||
```
|
||||
--checkers=8: Number of checkers to run in parallel.
|
||||
--config="~/.rclone.conf": Config file.
|
||||
-n, --dry-run=false: Do a trial run with no permanent changes
|
||||
--modify-window=1ns: Max time diff to be considered the same
|
||||
-q, --quiet=false: Print as little stuff as possible
|
||||
--stats=1m0s: Interval to print stats
|
||||
--transfers=4: Number of file transfers to run in parallel.
|
||||
-v, --verbose=false: Print lots more stuff
|
||||
```
|
||||
|
||||
Developer options:
|
||||
|
||||
```
|
||||
--cpuprofile="": Write cpu profile to file
|
||||
```
|
||||
|
||||
Local Filesystem
|
||||
----------------
|
||||
|
||||
Paths are specified as normal filesystem paths, so
|
||||
|
||||
rclone sync /home/source /tmp/destination
|
||||
|
||||
Will sync `/home/source` to `/tmp/destination`
|
||||
|
||||
Swift / Rackspace cloudfiles / Memset Memstore
|
||||
----------------------------------------------
|
||||
|
||||
Paths are specified as remote:container (or remote: for the `lsd`
|
||||
command.) You may put subdirectories in too, eg
|
||||
`remote:container/path/to/dir`.
|
||||
|
||||
So to copy a local directory to a swift container called backup:
|
||||
|
||||
rclone sync /home/source swift:backup
|
||||
|
||||
The modified time is stored as metadata on the object as
|
||||
`X-Object-Meta-Mtime` as floating point since the epoch.
|
||||
|
||||
This is a defacto standard (used in the official python-swiftclient
|
||||
amongst others) for storing the modification time (as read using
|
||||
os.Stat) for an object.
|
||||
|
||||
Amazon S3
|
||||
---------
|
||||
|
||||
Paths are specified as remote:bucket. You may put subdirectories in
|
||||
too, eg `remote:bucket/path/to/dir`.
|
||||
|
||||
So to copy a local directory to a s3 container called backup
|
||||
|
||||
rclone sync /home/source s3:backup
|
||||
|
||||
The modified time is stored as metadata on the object as
|
||||
`X-Amz-Meta-Mtime` as floating point since the epoch.
|
||||
|
||||
Google drive
|
||||
------------
|
||||
|
||||
Paths are specified as remote:path Drive paths may be as deep as required.
|
||||
|
||||
The initial setup for drive involves getting a token from Google drive
|
||||
which you need to do in your browser. `rclone config` walks you
|
||||
through it.
|
||||
|
||||
To copy a local directory to a drive directory called backup
|
||||
|
||||
rclone copy /home/source remote:backup
|
||||
|
||||
Google drive stores modification times accurate to 1 ms natively.
|
||||
|
||||
Dropbox
|
||||
-------
|
||||
|
||||
Paths are specified as remote:path Dropbox paths may be as deep as required.
|
||||
|
||||
The initial setup for dropbox involves getting a token from Dropbox
|
||||
which you need to do in your browser. `rclone config` walks you
|
||||
through it.
|
||||
|
||||
To copy a local directory to a drive directory called backup
|
||||
|
||||
rclone copy /home/source dropbox:backup
|
||||
|
||||
Md5sums and timestamps in RFC3339 format accurate to 1ns are stored in
|
||||
a Dropbox datastore called "rclone". Dropbox datastores are limited
|
||||
to 100,000 rows so this is the maximum number of files rclone can
|
||||
manage on Dropbox.
|
||||
|
||||
Google Cloud Storage
|
||||
--------------------
|
||||
|
||||
Paths are specified as remote:path Google Cloud Storage paths may be
|
||||
as deep as required.
|
||||
|
||||
The initial setup for Google Cloud Storage involves getting a token
|
||||
from Google which you need to do in your browser. `rclone config`
|
||||
walks you through it.
|
||||
|
||||
To copy a local directory to a google cloud storage directory called backup
|
||||
|
||||
rclone copy /home/source remote:backup
|
||||
|
||||
Google google cloud storage stores md5sums natively and rclone stores
|
||||
modification times as metadata on the object, under the "mtime" key in
|
||||
RFC3339 format accurate to 1ns.
|
||||
|
||||
Single file copies
|
||||
------------------
|
||||
|
||||
Rclone can copy single files
|
||||
|
||||
rclone src:path/to/file dst:path/dir
|
||||
|
||||
Or
|
||||
|
||||
rclone src:path/to/file dst:path/to/file
|
||||
|
||||
Note that you can't rename the file if you are copying from one file to another.
|
||||
|
||||
License
|
||||
-------
|
||||
|
||||
This is free software under the terms of MIT the license (check the
|
||||
COPYING file included in this package).
|
||||
|
||||
Bugs
|
||||
----
|
||||
|
||||
* Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
|
||||
* quota is 100.0 requests/second/user
|
||||
* Empty directories left behind with Local and Drive
|
||||
* eg purging a local directory with subdirectories doesn't work
|
||||
|
||||
Changelog
|
||||
---------
|
||||
* v1.02 - 2014-07-19
|
||||
* Implement Dropbox remote
|
||||
* Implement Google Cloud Storage remote
|
||||
* Verify Md5sums and Sizes after copies
|
||||
* Remove times from "ls" command - lists sizes only
|
||||
* Add add "lsl" - lists times and sizes
|
||||
* Add "md5sum" command
|
||||
* v1.01 - 2014-07-04
|
||||
* drive: fix transfer of big files using up lots of memory
|
||||
* v1.00 - 2014-07-03
|
||||
* drive: fix whole second dates
|
||||
* v0.99 - 2014-06-26
|
||||
* Fix --dry-run not working
|
||||
* Make compatible with go 1.1
|
||||
* v0.98 - 2014-05-30
|
||||
* s3: Treat missing Content-Length as 0 for some ceph installations
|
||||
* rclonetest: add file with a space in
|
||||
* v0.97 - 2014-05-05
|
||||
* Implement copying of single files
|
||||
* s3 & swift: support paths inside containers/buckets
|
||||
* v0.96 - 2014-04-24
|
||||
* drive: Fix multiple files of same name being created
|
||||
* drive: Use o.Update and fs.Put to optimise transfers
|
||||
* Add version number, -V and --version
|
||||
* v0.95 - 2014-03-28
|
||||
* rclone.org: website, docs and graphics
|
||||
* drive: fix path parsing
|
||||
* v0.94 - 2014-03-27
|
||||
* Change remote format one last time
|
||||
* GNU style flags
|
||||
* v0.93 - 2014-03-16
|
||||
* drive: store token in config file
|
||||
* cross compile other versions
|
||||
* set strict permissions on config file
|
||||
* v0.92 - 2014-03-15
|
||||
* Config fixes and --config option
|
||||
* v0.91 - 2014-03-15
|
||||
* Make config file
|
||||
* v0.90 - 2013-06-27
|
||||
* Project named rclone
|
||||
* v0.00 - 2012-11-18
|
||||
* Project started
|
||||
|
||||
|
||||
Contact and support
|
||||
-------------------
|
||||
|
||||
The project website is at:
|
||||
|
||||
* https://github.com/ncw/rclone
|
||||
|
||||
There you can file bug reports, ask for help or send pull requests.
|
||||
|
||||
Authors
|
||||
-------
|
||||
|
||||
* Nick Craig-Wood <nick@craig-wood.com>
|
||||
|
||||
Contributors
|
||||
------------
|
||||
|
||||
* Your name goes here!
|
||||
|
||||
24
RELEASE.md
24
RELEASE.md
@@ -1,24 +0,0 @@
|
||||
Required software for making a release
|
||||
* [github-release](https://github.com/aktau/github-release) for uploading packages
|
||||
* [gox](https://github.com/mitchellh/gox) for cross compiling
|
||||
* Run `gox -build-toolchain`
|
||||
* This assumes you have your own source checkout
|
||||
* pandoc for making the html and man pages
|
||||
* errcheck - go get github.com/kisielk/errcheck
|
||||
* golint - go get github.com/golang/lint
|
||||
|
||||
Making a release
|
||||
* make update
|
||||
* make check
|
||||
* make test
|
||||
* make tag
|
||||
* edit docs/content/changelog.md
|
||||
* make doc
|
||||
* git commit -a -v
|
||||
* make retag
|
||||
* # Set the GOPATH for a gox enabled compiler - . ~/bin/go-cross - not required for go >= 1.5
|
||||
* make cross
|
||||
* make upload
|
||||
* make upload_website
|
||||
* git push --tags origin master
|
||||
* make upload_github
|
||||
@@ -1,854 +0,0 @@
|
||||
// Package amazonclouddrive provides an interface to the Amazon Cloud
|
||||
// Drive object storage system.
|
||||
package amazonclouddrive
|
||||
|
||||
/*
|
||||
|
||||
FIXME make searching for directory in id and file in id more efficient
|
||||
- use the name: search parameter - remember the escaping rules
|
||||
- use Folder GetNode and GetFile
|
||||
|
||||
FIXME make the default for no files and no dirs be (FILE & FOLDER) so
|
||||
we ignore assets completely!
|
||||
*/
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"github.com/ncw/go-acd"
|
||||
"github.com/ncw/rclone/dircache"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/oauthutil"
|
||||
"github.com/ncw/rclone/pacer"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/spf13/pflag"
|
||||
"golang.org/x/oauth2"
|
||||
)
|
||||
|
||||
const (
|
||||
rcloneClientID = "amzn1.application-oa2-client.6bf18d2d1f5b485c94c8988bb03ad0e7"
|
||||
rcloneEncryptedClientSecret = "ZP12wYlGw198FtmqfOxyNAGXU3fwVcQdmt--ba1d00wJnUs0LOzvVyXVDbqhbcUqnr5Vd1QejwWmiv1Ep7UJG1kUQeuBP5n9goXWd5MrAf0"
|
||||
folderKind = "FOLDER"
|
||||
fileKind = "FILE"
|
||||
assetKind = "ASSET"
|
||||
statusAvailable = "AVAILABLE"
|
||||
timeFormat = time.RFC3339 // 2014-03-07T22:31:12.173Z
|
||||
minSleep = 20 * time.Millisecond
|
||||
warnFileSize = 50 << 30 // Display warning for files larger than this size
|
||||
)
|
||||
|
||||
// Globals
|
||||
var (
|
||||
// Flags
|
||||
tempLinkThreshold = fs.SizeSuffix(9 << 30) // Download files bigger than this via the tempLink
|
||||
uploadWaitTime = pflag.DurationP("acd-upload-wait-time", "", 2*60*time.Second, "Time to wait after a failed complete upload to see if it appears.")
|
||||
// Description of how to auth for this app
|
||||
acdConfig = &oauth2.Config{
|
||||
Scopes: []string{"clouddrive:read_all", "clouddrive:write"},
|
||||
Endpoint: oauth2.Endpoint{
|
||||
AuthURL: "https://www.amazon.com/ap/oa",
|
||||
TokenURL: "https://api.amazon.com/auth/o2/token",
|
||||
},
|
||||
ClientID: rcloneClientID,
|
||||
ClientSecret: fs.MustReveal(rcloneEncryptedClientSecret),
|
||||
RedirectURL: oauthutil.RedirectURL,
|
||||
}
|
||||
)
|
||||
|
||||
// Register with Fs
|
||||
func init() {
|
||||
fs.Register(&fs.RegInfo{
|
||||
Name: "amazon cloud drive",
|
||||
Description: "Amazon Drive",
|
||||
NewFs: NewFs,
|
||||
Config: func(name string) {
|
||||
err := oauthutil.Config("amazon cloud drive", name, acdConfig)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to configure token: %v", err)
|
||||
}
|
||||
},
|
||||
Options: []fs.Option{{
|
||||
Name: fs.ConfigClientID,
|
||||
Help: "Amazon Application Client Id - leave blank normally.",
|
||||
}, {
|
||||
Name: fs.ConfigClientSecret,
|
||||
Help: "Amazon Application Client Secret - leave blank normally.",
|
||||
}},
|
||||
})
|
||||
pflag.VarP(&tempLinkThreshold, "acd-templink-threshold", "", "Files >= this size will be downloaded via their tempLink.")
|
||||
}
|
||||
|
||||
// Fs represents a remote acd server
|
||||
type Fs struct {
|
||||
name string // name of this remote
|
||||
c *acd.Client // the connection to the acd server
|
||||
noAuthClient *http.Client // unauthenticated http client
|
||||
root string // the path we are working on
|
||||
dirCache *dircache.DirCache // Map of directory path to directory id
|
||||
pacer *pacer.Pacer // pacer for API calls
|
||||
ts *oauthutil.TokenSource // token source for oauth
|
||||
uploads int32 // number of uploads in progress - atomic access required
|
||||
}
|
||||
|
||||
// Object describes a acd object
|
||||
//
|
||||
// Will definitely have info but maybe not meta
|
||||
type Object struct {
|
||||
fs *Fs // what this object is part of
|
||||
remote string // The remote path
|
||||
info *acd.Node // Info from the acd object if known
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------
|
||||
|
||||
// Name of the remote (as passed into NewFs)
|
||||
func (f *Fs) Name() string {
|
||||
return f.name
|
||||
}
|
||||
|
||||
// Root of the remote (as passed into NewFs)
|
||||
func (f *Fs) Root() string {
|
||||
return f.root
|
||||
}
|
||||
|
||||
// String converts this Fs to a string
|
||||
func (f *Fs) String() string {
|
||||
return fmt.Sprintf("amazon drive root '%s'", f.root)
|
||||
}
|
||||
|
||||
// Pattern to match a acd path
|
||||
var matcher = regexp.MustCompile(`^([^/]*)(.*)$`)
|
||||
|
||||
// parsePath parses an acd 'url'
|
||||
func parsePath(path string) (root string) {
|
||||
root = strings.Trim(path, "/")
|
||||
return
|
||||
}
|
||||
|
||||
// retryErrorCodes is a slice of error codes that we will retry
|
||||
var retryErrorCodes = []int{
|
||||
400, // Bad request (seen in "Next token is expired")
|
||||
401, // Unauthorized (seen in "Token has expired")
|
||||
408, // Request Timeout
|
||||
429, // Rate exceeded.
|
||||
500, // Get occasional 500 Internal Server Error
|
||||
503, // Service Unavailable
|
||||
504, // Gateway Time-out
|
||||
}
|
||||
|
||||
// shouldRetry returns a boolean as to whether this resp and err
|
||||
// deserve to be retried. It returns the err as a convenience
|
||||
func (f *Fs) shouldRetry(resp *http.Response, err error) (bool, error) {
|
||||
if resp != nil {
|
||||
if resp.StatusCode == 401 {
|
||||
f.ts.Invalidate()
|
||||
fs.Log(f, "401 error received - invalidating token")
|
||||
return true, err
|
||||
}
|
||||
// Work around receiving this error sporadically on authentication
|
||||
//
|
||||
// HTTP code 403: "403 Forbidden", reponse body: {"message":"Authorization header requires 'Credential' parameter. Authorization header requires 'Signature' parameter. Authorization header requires 'SignedHeaders' parameter. Authorization header requires existence of either a 'X-Amz-Date' or a 'Date' header. Authorization=Bearer"}
|
||||
if resp.StatusCode == 403 && strings.Contains(err.Error(), "Authorization header requires") {
|
||||
fs.Log(f, "403 \"Authorization header requires...\" error received - retry")
|
||||
return true, err
|
||||
}
|
||||
}
|
||||
return fs.ShouldRetry(err) || fs.ShouldRetryHTTP(resp, retryErrorCodes), err
|
||||
}
|
||||
|
||||
// NewFs constructs an Fs from the path, container:path
|
||||
func NewFs(name, root string) (fs.Fs, error) {
|
||||
root = parsePath(root)
|
||||
oAuthClient, ts, err := oauthutil.NewClient(name, acdConfig)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to configure Amazon Drive: %v", err)
|
||||
}
|
||||
|
||||
c := acd.NewClient(oAuthClient)
|
||||
c.UserAgent = fs.UserAgent
|
||||
f := &Fs{
|
||||
name: name,
|
||||
root: root,
|
||||
c: c,
|
||||
pacer: pacer.New().SetMinSleep(minSleep).SetPacer(pacer.AmazonCloudDrivePacer),
|
||||
noAuthClient: fs.Config.Client(),
|
||||
ts: ts,
|
||||
}
|
||||
|
||||
// Update endpoints
|
||||
var resp *http.Response
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
_, resp, err = f.c.Account.GetEndpoints()
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to get endpoints")
|
||||
}
|
||||
|
||||
// Get rootID
|
||||
rootInfo, err := f.getRootInfo()
|
||||
if err != nil || rootInfo.Id == nil {
|
||||
return nil, errors.Wrap(err, "failed to get root")
|
||||
}
|
||||
|
||||
// Renew the token in the background
|
||||
go f.renewToken()
|
||||
|
||||
f.dirCache = dircache.New(root, *rootInfo.Id, f)
|
||||
|
||||
// Find the current root
|
||||
err = f.dirCache.FindRoot(false)
|
||||
if err != nil {
|
||||
// Assume it is a file
|
||||
newRoot, remote := dircache.SplitPath(root)
|
||||
newF := *f
|
||||
newF.dirCache = dircache.New(newRoot, *rootInfo.Id, &newF)
|
||||
newF.root = newRoot
|
||||
// Make new Fs which is the parent
|
||||
err = newF.dirCache.FindRoot(false)
|
||||
if err != nil {
|
||||
// No root so return old f
|
||||
return f, nil
|
||||
}
|
||||
_, err := newF.newObjectWithInfo(remote, nil)
|
||||
if err != nil {
|
||||
if err == fs.ErrorObjectNotFound {
|
||||
// File doesn't exist so return old f
|
||||
return f, nil
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
// return an error with an fs which points to the parent
|
||||
return &newF, fs.ErrorIsFile
|
||||
}
|
||||
return f, nil
|
||||
}
|
||||
|
||||
// getRootInfo gets the root folder info
|
||||
func (f *Fs) getRootInfo() (rootInfo *acd.Folder, err error) {
|
||||
var resp *http.Response
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
rootInfo, resp, err = f.c.Nodes.GetRoot()
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
return rootInfo, err
|
||||
}
|
||||
|
||||
// Renew the token - runs in the background
|
||||
//
|
||||
// Renews the token whenever it expires. Useful when there are lots
|
||||
// of uploads in progress and the token doesn't get renewed. Amazon
|
||||
// seem to cancel your uploads if you don't renew your token for 2hrs.
|
||||
func (f *Fs) renewToken() {
|
||||
expiry := f.ts.OnExpiry()
|
||||
for {
|
||||
<-expiry
|
||||
uploads := atomic.LoadInt32(&f.uploads)
|
||||
if uploads != 0 {
|
||||
fs.Debug(f, "Token expired - %d uploads in progress - refreshing", uploads)
|
||||
// Do a transaction
|
||||
_, err := f.getRootInfo()
|
||||
if err == nil {
|
||||
fs.Debug(f, "Token refresh successful")
|
||||
} else {
|
||||
fs.ErrorLog(f, "Token refresh failed: %v", err)
|
||||
}
|
||||
} else {
|
||||
fs.Debug(f, "Token expired but no uploads in progress - doing nothing")
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (f *Fs) startUpload() {
|
||||
atomic.AddInt32(&f.uploads, 1)
|
||||
}
|
||||
|
||||
func (f *Fs) stopUpload() {
|
||||
atomic.AddInt32(&f.uploads, -1)
|
||||
}
|
||||
|
||||
// Return an Object from a path
|
||||
//
|
||||
// If it can't be found it returns the error fs.ErrorObjectNotFound.
|
||||
func (f *Fs) newObjectWithInfo(remote string, info *acd.Node) (fs.Object, error) {
|
||||
o := &Object{
|
||||
fs: f,
|
||||
remote: remote,
|
||||
}
|
||||
if info != nil {
|
||||
// Set info but not meta
|
||||
o.info = info
|
||||
} else {
|
||||
err := o.readMetaData() // reads info and meta, returning an error
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
}
|
||||
return o, nil
|
||||
}
|
||||
|
||||
// NewObject finds the Object at remote. If it can't be found
|
||||
// it returns the error fs.ErrorObjectNotFound.
|
||||
func (f *Fs) NewObject(remote string) (fs.Object, error) {
|
||||
return f.newObjectWithInfo(remote, nil)
|
||||
}
|
||||
|
||||
// FindLeaf finds a directory of name leaf in the folder with ID pathID
|
||||
func (f *Fs) FindLeaf(pathID, leaf string) (pathIDOut string, found bool, err error) {
|
||||
//fs.Debug(f, "FindLeaf(%q, %q)", pathID, leaf)
|
||||
folder := acd.FolderFromId(pathID, f.c.Nodes)
|
||||
var resp *http.Response
|
||||
var subFolder *acd.Folder
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
subFolder, resp, err = folder.GetFolder(leaf)
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
if err == acd.ErrorNodeNotFound {
|
||||
//fs.Debug(f, "...Not found")
|
||||
return "", false, nil
|
||||
}
|
||||
//fs.Debug(f, "...Error %v", err)
|
||||
return "", false, err
|
||||
}
|
||||
if subFolder.Status != nil && *subFolder.Status != statusAvailable {
|
||||
fs.Debug(f, "Ignoring folder %q in state %q", leaf, *subFolder.Status)
|
||||
time.Sleep(1 * time.Second) // FIXME wait for problem to go away!
|
||||
return "", false, nil
|
||||
}
|
||||
//fs.Debug(f, "...Found(%q, %v)", *subFolder.Id, leaf)
|
||||
return *subFolder.Id, true, nil
|
||||
}
|
||||
|
||||
// CreateDir makes a directory with pathID as parent and name leaf
|
||||
func (f *Fs) CreateDir(pathID, leaf string) (newID string, err error) {
|
||||
//fmt.Printf("CreateDir(%q, %q)\n", pathID, leaf)
|
||||
folder := acd.FolderFromId(pathID, f.c.Nodes)
|
||||
var resp *http.Response
|
||||
var info *acd.Folder
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
info, resp, err = folder.CreateFolder(leaf)
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
//fmt.Printf("...Error %v\n", err)
|
||||
return "", err
|
||||
}
|
||||
//fmt.Printf("...Id %q\n", *info.Id)
|
||||
return *info.Id, nil
|
||||
}
|
||||
|
||||
// list the objects into the function supplied
|
||||
//
|
||||
// If directories is set it only sends directories
|
||||
// User function to process a File item from listAll
|
||||
//
|
||||
// Should return true to finish processing
|
||||
type listAllFn func(*acd.Node) bool
|
||||
|
||||
// Lists the directory required calling the user function on each item found
|
||||
//
|
||||
// If the user fn ever returns true then it early exits with found = true
|
||||
func (f *Fs) listAll(dirID string, title string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
|
||||
query := "parents:" + dirID
|
||||
if directoriesOnly {
|
||||
query += " AND kind:" + folderKind
|
||||
} else if filesOnly {
|
||||
query += " AND kind:" + fileKind
|
||||
} else {
|
||||
// FIXME none of these work
|
||||
//query += " AND kind:(" + fileKind + " OR " + folderKind + ")"
|
||||
//query += " AND (kind:" + fileKind + " OR kind:" + folderKind + ")"
|
||||
}
|
||||
opts := acd.NodeListOptions{
|
||||
Filters: query,
|
||||
}
|
||||
var nodes []*acd.Node
|
||||
var out []*acd.Node
|
||||
//var resp *http.Response
|
||||
for {
|
||||
var resp *http.Response
|
||||
err = f.pacer.CallNoRetry(func() (bool, error) {
|
||||
nodes, resp, err = f.c.Nodes.GetNodes(&opts)
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if nodes == nil {
|
||||
break
|
||||
}
|
||||
for _, node := range nodes {
|
||||
if node.Name != nil && node.Id != nil && node.Kind != nil && node.Status != nil {
|
||||
// Ignore nodes if not AVAILABLE
|
||||
if *node.Status != statusAvailable {
|
||||
continue
|
||||
}
|
||||
// Store the nodes up in case we have to retry the listing
|
||||
out = append(out, node)
|
||||
}
|
||||
}
|
||||
}
|
||||
// Send the nodes now
|
||||
for _, node := range out {
|
||||
if fn(node) {
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// ListDir reads the directory specified by the job into out, returning any more jobs
|
||||
func (f *Fs) ListDir(out fs.ListOpts, job dircache.ListDirJob) (jobs []dircache.ListDirJob, err error) {
|
||||
fs.Debug(f, "Reading %q", job.Path)
|
||||
maxTries := fs.Config.LowLevelRetries
|
||||
for tries := 1; tries <= maxTries; tries++ {
|
||||
_, err = f.listAll(job.DirID, "", false, false, func(node *acd.Node) bool {
|
||||
remote := job.Path + *node.Name
|
||||
switch *node.Kind {
|
||||
case folderKind:
|
||||
if out.IncludeDirectory(remote) {
|
||||
dir := &fs.Dir{
|
||||
Name: remote,
|
||||
Bytes: -1,
|
||||
Count: -1,
|
||||
}
|
||||
dir.When, _ = time.Parse(timeFormat, *node.ModifiedDate) // FIXME
|
||||
if out.AddDir(dir) {
|
||||
return true
|
||||
}
|
||||
if job.Depth > 0 {
|
||||
jobs = append(jobs, dircache.ListDirJob{DirID: *node.Id, Path: remote + "/", Depth: job.Depth - 1})
|
||||
}
|
||||
}
|
||||
case fileKind:
|
||||
o, err := f.newObjectWithInfo(remote, node)
|
||||
if err != nil {
|
||||
out.SetError(err)
|
||||
return true
|
||||
}
|
||||
if out.Add(o) {
|
||||
return true
|
||||
}
|
||||
default:
|
||||
// ignore ASSET etc
|
||||
}
|
||||
return false
|
||||
})
|
||||
if fs.IsRetryError(err) {
|
||||
fs.Debug(f, "Directory listing error for %q: %v - low level retry %d/%d", job.Path, err, tries, maxTries)
|
||||
continue
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
break
|
||||
}
|
||||
fs.Debug(f, "Finished reading %q", job.Path)
|
||||
return jobs, err
|
||||
}
|
||||
|
||||
// List walks the path returning iles and directories into out
|
||||
func (f *Fs) List(out fs.ListOpts, dir string) {
|
||||
f.dirCache.List(f, out, dir)
|
||||
}
|
||||
|
||||
// checkUpload checks to see if an error occurred after the file was
|
||||
// completely uploaded.
|
||||
//
|
||||
// If it was then it waits for a while to see if the file really
|
||||
// exists and is the right size and returns an updated info.
|
||||
//
|
||||
// If the file wasn't found or was the wrong size then it returns the
|
||||
// original error.
|
||||
//
|
||||
// This is a workaround for Amazon sometimes returning
|
||||
//
|
||||
// * 408 REQUEST_TIMEOUT
|
||||
// * 504 GATEWAY_TIMEOUT
|
||||
// * 500 Internal server error
|
||||
//
|
||||
// At the end of large uploads. The speculation is that the timeout
|
||||
// is waiting for the sha1 hashing to complete and the file may well
|
||||
// be properly uploaded.
|
||||
func (f *Fs) checkUpload(in io.Reader, src fs.ObjectInfo, inInfo *acd.File, inErr error) (fixedError bool, info *acd.File, err error) {
|
||||
// Return if no error - all is well
|
||||
if inErr == nil {
|
||||
return false, inInfo, inErr
|
||||
}
|
||||
const sleepTime = 5 * time.Second // sleep between tries
|
||||
retries := int(*uploadWaitTime / sleepTime) // number of retries
|
||||
if retries <= 0 {
|
||||
retries = 1
|
||||
}
|
||||
buf := make([]byte, 1)
|
||||
n, err := in.Read(buf)
|
||||
if !(n == 0 && err == io.EOF) {
|
||||
fs.Debug(src, "Upload error detected but didn't finish upload (n=%d, err=%v): %v", n, err, inErr)
|
||||
return false, inInfo, inErr
|
||||
}
|
||||
fs.Debug(src, "Error detected after finished upload - waiting to see if object was uploaded correctly: %v", inErr)
|
||||
remote := src.Remote()
|
||||
for i := 1; i <= retries; i++ {
|
||||
o, err := f.NewObject(remote)
|
||||
if err == fs.ErrorObjectNotFound {
|
||||
fs.Debug(src, "Object not found - waiting (%d/%d)", i, retries)
|
||||
} else if err != nil {
|
||||
fs.Debug(src, "Object returned error - waiting (%d/%d): %v", i, retries, err)
|
||||
} else {
|
||||
if src.Size() == o.Size() {
|
||||
fs.Debug(src, "Object found with correct size - returning with no error")
|
||||
info = &acd.File{
|
||||
Node: o.(*Object).info,
|
||||
}
|
||||
return true, info, nil
|
||||
}
|
||||
fs.Debug(src, "Object found but wrong size %d vs %d - waiting (%d/%d)", src.Size(), o.Size(), i, retries)
|
||||
}
|
||||
time.Sleep(sleepTime)
|
||||
}
|
||||
fs.Debug(src, "Finished waiting for object - returning original error: %v", inErr)
|
||||
return false, inInfo, inErr
|
||||
}
|
||||
|
||||
// Put the object into the container
|
||||
//
|
||||
// Copy the reader in to the new object which is returned
|
||||
//
|
||||
// The new object may have been created if an error is returned
|
||||
func (f *Fs) Put(in io.Reader, src fs.ObjectInfo) (fs.Object, error) {
|
||||
remote := src.Remote()
|
||||
size := src.Size()
|
||||
// Temporary Object under construction
|
||||
o := &Object{
|
||||
fs: f,
|
||||
remote: remote,
|
||||
}
|
||||
// Check if object already exists
|
||||
err := o.readMetaData()
|
||||
switch err {
|
||||
case nil:
|
||||
return o, o.Update(in, src)
|
||||
case fs.ErrorObjectNotFound:
|
||||
// Not found so create it
|
||||
default:
|
||||
return nil, err
|
||||
}
|
||||
// If not create it
|
||||
leaf, directoryID, err := f.dirCache.FindPath(remote, true)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if size > warnFileSize {
|
||||
fs.Debug(f, "Warning: file %q may fail because it is too big. Use --max-size=%dGB to skip large files.", remote, warnFileSize>>30)
|
||||
}
|
||||
folder := acd.FolderFromId(directoryID, o.fs.c.Nodes)
|
||||
var info *acd.File
|
||||
var resp *http.Response
|
||||
err = f.pacer.CallNoRetry(func() (bool, error) {
|
||||
f.startUpload()
|
||||
if src.Size() != 0 {
|
||||
info, resp, err = folder.Put(in, leaf)
|
||||
} else {
|
||||
info, resp, err = folder.PutSized(in, size, leaf)
|
||||
}
|
||||
f.stopUpload()
|
||||
var ok bool
|
||||
ok, info, err = f.checkUpload(in, src, info, err)
|
||||
if ok {
|
||||
return false, nil
|
||||
}
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
o.info = info.Node
|
||||
return o, nil
|
||||
}
|
||||
|
||||
// Mkdir creates the container if it doesn't exist
|
||||
func (f *Fs) Mkdir() error {
|
||||
return f.dirCache.FindRoot(true)
|
||||
}
|
||||
|
||||
// purgeCheck remotes the root directory, if check is set then it
|
||||
// refuses to do so if it has anything in
|
||||
func (f *Fs) purgeCheck(check bool) error {
|
||||
if f.root == "" {
|
||||
return errors.New("can't purge root directory")
|
||||
}
|
||||
dc := f.dirCache
|
||||
err := dc.FindRoot(false)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
rootID := dc.RootID()
|
||||
|
||||
if check {
|
||||
// check directory is empty
|
||||
empty := true
|
||||
_, err = f.listAll(rootID, "", false, false, func(node *acd.Node) bool {
|
||||
switch *node.Kind {
|
||||
case folderKind:
|
||||
empty = false
|
||||
return true
|
||||
case fileKind:
|
||||
empty = false
|
||||
return true
|
||||
default:
|
||||
fs.Debug("Found ASSET %s", *node.Id)
|
||||
}
|
||||
return false
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !empty {
|
||||
return errors.New("directory not empty")
|
||||
}
|
||||
}
|
||||
|
||||
node := acd.NodeFromId(rootID, f.c.Nodes)
|
||||
var resp *http.Response
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
resp, err = node.Trash()
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
f.dirCache.ResetRoot()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Rmdir deletes the root folder
|
||||
//
|
||||
// Returns an error if it isn't empty
|
||||
func (f *Fs) Rmdir() error {
|
||||
return f.purgeCheck(true)
|
||||
}
|
||||
|
||||
// Precision return the precision of this Fs
|
||||
func (f *Fs) Precision() time.Duration {
|
||||
return fs.ModTimeNotSupported
|
||||
}
|
||||
|
||||
// Hashes returns the supported hash sets.
|
||||
func (f *Fs) Hashes() fs.HashSet {
|
||||
return fs.HashSet(fs.HashMD5)
|
||||
}
|
||||
|
||||
// Copy src to this remote using server side copy operations.
|
||||
//
|
||||
// This is stored with the remote path given
|
||||
//
|
||||
// It returns the destination Object and a possible error
|
||||
//
|
||||
// Will only be called if src.Fs().Name() == f.Name()
|
||||
//
|
||||
// If it isn't possible then return fs.ErrorCantCopy
|
||||
//func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
|
||||
// srcObj, ok := src.(*Object)
|
||||
// if !ok {
|
||||
// fs.Debug(src, "Can't copy - not same remote type")
|
||||
// return nil, fs.ErrorCantCopy
|
||||
// }
|
||||
// srcFs := srcObj.fs
|
||||
// _, err := f.c.ObjectCopy(srcFs.container, srcFs.root+srcObj.remote, f.container, f.root+remote, nil)
|
||||
// if err != nil {
|
||||
// return nil, err
|
||||
// }
|
||||
// return f.NewObject(remote), nil
|
||||
//}
|
||||
|
||||
// Purge deletes all the files and the container
|
||||
//
|
||||
// Optional interface: Only implement this if you have a way of
|
||||
// deleting all the files quicker than just running Remove() on the
|
||||
// result of List()
|
||||
func (f *Fs) Purge() error {
|
||||
return f.purgeCheck(false)
|
||||
}
|
||||
|
||||
// ------------------------------------------------------------
|
||||
|
||||
// Fs returns the parent Fs
|
||||
func (o *Object) Fs() fs.Info {
|
||||
return o.fs
|
||||
}
|
||||
|
||||
// Return a string version
|
||||
func (o *Object) String() string {
|
||||
if o == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
return o.remote
|
||||
}
|
||||
|
||||
// Remote returns the remote path
|
||||
func (o *Object) Remote() string {
|
||||
return o.remote
|
||||
}
|
||||
|
||||
// Hash returns the Md5sum of an object returning a lowercase hex string
|
||||
func (o *Object) Hash(t fs.HashType) (string, error) {
|
||||
if t != fs.HashMD5 {
|
||||
return "", fs.ErrHashUnsupported
|
||||
}
|
||||
if o.info.ContentProperties.Md5 != nil {
|
||||
return *o.info.ContentProperties.Md5, nil
|
||||
}
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// Size returns the size of an object in bytes
|
||||
func (o *Object) Size() int64 {
|
||||
return int64(*o.info.ContentProperties.Size)
|
||||
}
|
||||
|
||||
// readMetaData gets the metadata if it hasn't already been fetched
|
||||
//
|
||||
// it also sets the info
|
||||
//
|
||||
// If it can't be found it returns the error fs.ErrorObjectNotFound.
|
||||
func (o *Object) readMetaData() (err error) {
|
||||
if o.info != nil {
|
||||
return nil
|
||||
}
|
||||
leaf, directoryID, err := o.fs.dirCache.FindPath(o.remote, false)
|
||||
if err != nil {
|
||||
if err == fs.ErrorDirNotFound {
|
||||
return fs.ErrorObjectNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
folder := acd.FolderFromId(directoryID, o.fs.c.Nodes)
|
||||
var resp *http.Response
|
||||
var info *acd.File
|
||||
err = o.fs.pacer.Call(func() (bool, error) {
|
||||
info, resp, err = folder.GetFile(leaf)
|
||||
return o.fs.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
if err == acd.ErrorNodeNotFound {
|
||||
return fs.ErrorObjectNotFound
|
||||
}
|
||||
return err
|
||||
}
|
||||
o.info = info.Node
|
||||
return nil
|
||||
}
|
||||
|
||||
// ModTime returns the modification time of the object
|
||||
//
|
||||
//
|
||||
// It attempts to read the objects mtime and if that isn't present the
|
||||
// LastModified returned in the http headers
|
||||
func (o *Object) ModTime() time.Time {
|
||||
err := o.readMetaData()
|
||||
if err != nil {
|
||||
fs.Log(o, "Failed to read metadata: %v", err)
|
||||
return time.Now()
|
||||
}
|
||||
modTime, err := time.Parse(timeFormat, *o.info.ModifiedDate)
|
||||
if err != nil {
|
||||
fs.Log(o, "Failed to read mtime from object: %v", err)
|
||||
return time.Now()
|
||||
}
|
||||
return modTime
|
||||
}
|
||||
|
||||
// SetModTime sets the modification time of the local fs object
|
||||
func (o *Object) SetModTime(modTime time.Time) error {
|
||||
// FIXME not implemented
|
||||
return fs.ErrorCantSetModTime
|
||||
}
|
||||
|
||||
// Storable returns a boolean showing whether this object storable
|
||||
func (o *Object) Storable() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
// Open an object for read
|
||||
func (o *Object) Open() (in io.ReadCloser, err error) {
|
||||
bigObject := o.Size() >= int64(tempLinkThreshold)
|
||||
if bigObject {
|
||||
fs.Debug(o, "Dowloading large object via tempLink")
|
||||
}
|
||||
file := acd.File{Node: o.info}
|
||||
var resp *http.Response
|
||||
err = o.fs.pacer.Call(func() (bool, error) {
|
||||
if !bigObject {
|
||||
in, resp, err = file.Open()
|
||||
} else {
|
||||
in, resp, err = file.OpenTempURL(o.fs.noAuthClient)
|
||||
}
|
||||
return o.fs.shouldRetry(resp, err)
|
||||
})
|
||||
return in, err
|
||||
}
|
||||
|
||||
// Update the object with the contents of the io.Reader, modTime and size
|
||||
//
|
||||
// The new object may have been created if an error is returned
|
||||
func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
|
||||
size := src.Size()
|
||||
file := acd.File{Node: o.info}
|
||||
var info *acd.File
|
||||
var resp *http.Response
|
||||
var err error
|
||||
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
|
||||
o.fs.startUpload()
|
||||
if size != 0 {
|
||||
info, resp, err = file.OverwriteSized(in, size)
|
||||
} else {
|
||||
info, resp, err = file.Overwrite(in)
|
||||
}
|
||||
o.fs.stopUpload()
|
||||
var ok bool
|
||||
ok, info, err = o.fs.checkUpload(in, src, info, err)
|
||||
if ok {
|
||||
return false, nil
|
||||
}
|
||||
return o.fs.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
o.info = info.Node
|
||||
return nil
|
||||
}
|
||||
|
||||
// Remove an object
|
||||
func (o *Object) Remove() error {
|
||||
var resp *http.Response
|
||||
var err error
|
||||
err = o.fs.pacer.Call(func() (bool, error) {
|
||||
resp, err = o.info.Trash()
|
||||
return o.fs.shouldRetry(resp, err)
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// Check the interfaces are satisfied
|
||||
var (
|
||||
_ fs.Fs = (*Fs)(nil)
|
||||
_ fs.Purger = (*Fs)(nil)
|
||||
// _ fs.Copier = (*Fs)(nil)
|
||||
// _ fs.Mover = (*Fs)(nil)
|
||||
// _ fs.DirMover = (*Fs)(nil)
|
||||
_ fs.Object = (*Object)(nil)
|
||||
)
|
||||
@@ -1,58 +0,0 @@
|
||||
// Test AmazonCloudDrive filesystem interface
|
||||
//
|
||||
// Automatically generated - DO NOT EDIT
|
||||
// Regenerate with: make gen_tests
|
||||
package amazonclouddrive_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/amazonclouddrive"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/fstest/fstests"
|
||||
)
|
||||
|
||||
func TestSetup(t *testing.T) {
|
||||
fstests.NilObject = fs.Object((*amazonclouddrive.Object)(nil))
|
||||
fstests.RemoteName = "TestAmazonCloudDrive:"
|
||||
}
|
||||
|
||||
// Generic tests for the Fs
|
||||
func TestInit(t *testing.T) { fstests.TestInit(t) }
|
||||
func TestFsString(t *testing.T) { fstests.TestFsString(t) }
|
||||
func TestFsRmdirEmpty(t *testing.T) { fstests.TestFsRmdirEmpty(t) }
|
||||
func TestFsRmdirNotFound(t *testing.T) { fstests.TestFsRmdirNotFound(t) }
|
||||
func TestFsMkdir(t *testing.T) { fstests.TestFsMkdir(t) }
|
||||
func TestFsListEmpty(t *testing.T) { fstests.TestFsListEmpty(t) }
|
||||
func TestFsListDirEmpty(t *testing.T) { fstests.TestFsListDirEmpty(t) }
|
||||
func TestFsNewObjectNotFound(t *testing.T) { fstests.TestFsNewObjectNotFound(t) }
|
||||
func TestFsPutFile1(t *testing.T) { fstests.TestFsPutFile1(t) }
|
||||
func TestFsPutFile2(t *testing.T) { fstests.TestFsPutFile2(t) }
|
||||
func TestFsUpdateFile1(t *testing.T) { fstests.TestFsUpdateFile1(t) }
|
||||
func TestFsListDirFile2(t *testing.T) { fstests.TestFsListDirFile2(t) }
|
||||
func TestFsListDirRoot(t *testing.T) { fstests.TestFsListDirRoot(t) }
|
||||
func TestFsListSubdir(t *testing.T) { fstests.TestFsListSubdir(t) }
|
||||
func TestFsListLevel2(t *testing.T) { fstests.TestFsListLevel2(t) }
|
||||
func TestFsListFile1(t *testing.T) { fstests.TestFsListFile1(t) }
|
||||
func TestFsNewObject(t *testing.T) { fstests.TestFsNewObject(t) }
|
||||
func TestFsListFile1and2(t *testing.T) { fstests.TestFsListFile1and2(t) }
|
||||
func TestFsCopy(t *testing.T) { fstests.TestFsCopy(t) }
|
||||
func TestFsMove(t *testing.T) { fstests.TestFsMove(t) }
|
||||
func TestFsDirMove(t *testing.T) { fstests.TestFsDirMove(t) }
|
||||
func TestFsRmdirFull(t *testing.T) { fstests.TestFsRmdirFull(t) }
|
||||
func TestFsPrecision(t *testing.T) { fstests.TestFsPrecision(t) }
|
||||
func TestObjectString(t *testing.T) { fstests.TestObjectString(t) }
|
||||
func TestObjectFs(t *testing.T) { fstests.TestObjectFs(t) }
|
||||
func TestObjectRemote(t *testing.T) { fstests.TestObjectRemote(t) }
|
||||
func TestObjectHashes(t *testing.T) { fstests.TestObjectHashes(t) }
|
||||
func TestObjectModTime(t *testing.T) { fstests.TestObjectModTime(t) }
|
||||
func TestObjectSetModTime(t *testing.T) { fstests.TestObjectSetModTime(t) }
|
||||
func TestObjectSize(t *testing.T) { fstests.TestObjectSize(t) }
|
||||
func TestObjectOpen(t *testing.T) { fstests.TestObjectOpen(t) }
|
||||
func TestObjectUpdate(t *testing.T) { fstests.TestObjectUpdate(t) }
|
||||
func TestObjectStorable(t *testing.T) { fstests.TestObjectStorable(t) }
|
||||
func TestFsIsFile(t *testing.T) { fstests.TestFsIsFile(t) }
|
||||
func TestFsIsFileNotFound(t *testing.T) { fstests.TestFsIsFileNotFound(t) }
|
||||
func TestObjectRemove(t *testing.T) { fstests.TestObjectRemove(t) }
|
||||
func TestObjectPurge(t *testing.T) { fstests.TestObjectPurge(t) }
|
||||
func TestFinalise(t *testing.T) { fstests.TestFinalise(t) }
|
||||
20
appveyor.yml
20
appveyor.yml
@@ -1,20 +0,0 @@
|
||||
version: "{build}"
|
||||
|
||||
os: Windows Server 2012 R2
|
||||
|
||||
clone_folder: c:\gopath\src\github.com\ncw\rclone
|
||||
|
||||
environment:
|
||||
GOPATH: c:\gopath
|
||||
|
||||
install:
|
||||
- echo %PATH%
|
||||
- echo %GOPATH%
|
||||
- go version
|
||||
- go env
|
||||
- go get -t -d ./...
|
||||
|
||||
build_script:
|
||||
- go vet ./...
|
||||
- go test -cpu=2 ./...
|
||||
- go test -cpu=2 -short -race ./...
|
||||
299
b2/api/types.go
299
b2/api/types.go
@@ -1,299 +0,0 @@
|
||||
package api
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"path"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// Error describes a B2 error response
|
||||
type Error struct {
|
||||
Status int `json:"status"` // The numeric HTTP status code. Always matches the status in the HTTP response.
|
||||
Code string `json:"code"` // A single-identifier code that identifies the error.
|
||||
Message string `json:"message"` // A human-readable message, in English, saying what went wrong.
|
||||
}
|
||||
|
||||
// Error statisfies the error interface
|
||||
func (e *Error) Error() string {
|
||||
return fmt.Sprintf("%s (%d %s)", e.Message, e.Status, e.Code)
|
||||
}
|
||||
|
||||
// Fatal statisfies the Fatal interface
|
||||
//
|
||||
// It indicates which errors should be treated as fatal
|
||||
func (e *Error) Fatal() bool {
|
||||
return e.Status == 403 // 403 errors shouldn't be retried
|
||||
}
|
||||
|
||||
var _ fs.Fataler = (*Error)(nil)
|
||||
|
||||
// Account describes a B2 account
|
||||
type Account struct {
|
||||
ID string `json:"accountId"` // The identifier for the account.
|
||||
}
|
||||
|
||||
// Bucket describes a B2 bucket
|
||||
type Bucket struct {
|
||||
ID string `json:"bucketId"`
|
||||
AccountID string `json:"accountId"`
|
||||
Name string `json:"bucketName"`
|
||||
Type string `json:"bucketType"`
|
||||
}
|
||||
|
||||
// Timestamp is a UTC time when this file was uploaded. It is a base
|
||||
// 10 number of milliseconds since midnight, January 1, 1970 UTC. This
|
||||
// fits in a 64 bit integer such as the type "long" in the programming
|
||||
// language Java. It is intended to be compatible with Java's time
|
||||
// long. For example, it can be passed directly into the java call
|
||||
// Date.setTime(long time).
|
||||
type Timestamp time.Time
|
||||
|
||||
// MarshalJSON turns a Timestamp into JSON (in UTC)
|
||||
func (t *Timestamp) MarshalJSON() (out []byte, err error) {
|
||||
timestamp := (*time.Time)(t).UTC().UnixNano()
|
||||
return []byte(strconv.FormatInt(timestamp/1E6, 10)), nil
|
||||
}
|
||||
|
||||
// UnmarshalJSON turns JSON into a Timestamp
|
||||
func (t *Timestamp) UnmarshalJSON(data []byte) error {
|
||||
timestamp, err := strconv.ParseInt(string(data), 10, 64)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
*t = Timestamp(time.Unix(timestamp/1E3, (timestamp%1E3)*1E6).UTC())
|
||||
return nil
|
||||
}
|
||||
|
||||
const versionFormat = "-v2006-01-02-150405.000"
|
||||
|
||||
// AddVersion adds the timestamp as a version string into the filename passed in.
|
||||
func (t Timestamp) AddVersion(remote string) string {
|
||||
ext := path.Ext(remote)
|
||||
base := remote[:len(remote)-len(ext)]
|
||||
s := (time.Time)(t).Format(versionFormat)
|
||||
// Replace the '.' with a '-'
|
||||
s = strings.Replace(s, ".", "-", -1)
|
||||
return base + s + ext
|
||||
}
|
||||
|
||||
// RemoveVersion removes the timestamp from a filename as a version string.
|
||||
//
|
||||
// It returns the new file name and a timestamp, or the old filename
|
||||
// and a zero timestamp.
|
||||
func RemoveVersion(remote string) (t Timestamp, newRemote string) {
|
||||
newRemote = remote
|
||||
ext := path.Ext(remote)
|
||||
base := remote[:len(remote)-len(ext)]
|
||||
if len(base) < len(versionFormat) {
|
||||
return
|
||||
}
|
||||
versionStart := len(base) - len(versionFormat)
|
||||
// Check it ends in -xxx
|
||||
if base[len(base)-4] != '-' {
|
||||
return
|
||||
}
|
||||
// Replace with .xxx for parsing
|
||||
base = base[:len(base)-4] + "." + base[len(base)-3:]
|
||||
newT, err := time.Parse(versionFormat, base[versionStart:])
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
return Timestamp(newT), base[:versionStart] + ext
|
||||
}
|
||||
|
||||
// IsZero returns true if the timestamp is unitialised
|
||||
func (t Timestamp) IsZero() bool {
|
||||
return (time.Time)(t).IsZero()
|
||||
}
|
||||
|
||||
// Equal compares two timestamps
|
||||
//
|
||||
// If either are !IsZero then it returns false
|
||||
func (t Timestamp) Equal(s Timestamp) bool {
|
||||
if (time.Time)(t).IsZero() {
|
||||
return false
|
||||
}
|
||||
if (time.Time)(s).IsZero() {
|
||||
return false
|
||||
}
|
||||
return (time.Time)(t).Equal((time.Time)(s))
|
||||
}
|
||||
|
||||
// File is info about a file
|
||||
type File struct {
|
||||
ID string `json:"fileId"` // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
|
||||
Name string `json:"fileName"` // The name of this file, which can be used with b2_download_file_by_name.
|
||||
Action string `json:"action"` // Either "upload" or "hide". "upload" means a file that was uploaded to B2 Cloud Storage. "hide" means a file version marking the file as hidden, so that it will not show up in b2_list_file_names. The result of b2_list_file_names will contain only "upload". The result of b2_list_file_versions may have both.
|
||||
Size int64 `json:"size"` // The number of bytes in the file.
|
||||
UploadTimestamp Timestamp `json:"uploadTimestamp"` // This is a UTC time when this file was uploaded.
|
||||
SHA1 string `json:"contentSha1"` // The SHA1 of the bytes stored in the file.
|
||||
ContentType string `json:"contentType"` // The MIME type of the file.
|
||||
Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
|
||||
}
|
||||
|
||||
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
|
||||
type AuthorizeAccountResponse struct {
|
||||
AccountID string `json:"accountId"` // The identifier for the account.
|
||||
AuthorizationToken string `json:"authorizationToken"` // An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
|
||||
APIURL string `json:"apiUrl"` // The base URL to use for all API calls except for uploading and downloading files.
|
||||
DownloadURL string `json:"downloadUrl"` // The base URL to use for downloading files.
|
||||
}
|
||||
|
||||
// ListBucketsResponse is as returned from the b2_list_buckets call
|
||||
type ListBucketsResponse struct {
|
||||
Buckets []Bucket `json:"buckets"`
|
||||
}
|
||||
|
||||
// ListFileNamesRequest is as passed to b2_list_file_names or b2_list_file_versions
|
||||
type ListFileNamesRequest struct {
|
||||
BucketID string `json:"bucketId"` // required - The bucket to look for file names in.
|
||||
StartFileName string `json:"startFileName,omitempty"` // optional - The first file name to return. If there is a file with this name, it will be returned in the list. If not, the first file name after this the first one after this name.
|
||||
MaxFileCount int `json:"maxFileCount,omitempty"` // optional - The maximum number of files to return from this call. The default value is 100, and the maximum allowed is 1000.
|
||||
StartFileID string `json:"startFileId,omitempty"` // optional - What to pass in to startFileId for the next search to continue where this one left off.
|
||||
}
|
||||
|
||||
// ListFileNamesResponse is as received from b2_list_file_names or b2_list_file_versions
|
||||
type ListFileNamesResponse struct {
|
||||
Files []File `json:"files"` // An array of objects, each one describing one file.
|
||||
NextFileName *string `json:"nextFileName"` // What to pass in to startFileName for the next search to continue where this one left off, or null if there are no more files.
|
||||
NextFileID *string `json:"nextFileId"` // What to pass in to startFileId for the next search to continue where this one left off, or null if there are no more files.
|
||||
}
|
||||
|
||||
// GetUploadURLRequest is passed to b2_get_upload_url
|
||||
type GetUploadURLRequest struct {
|
||||
BucketID string `json:"bucketId"` // The ID of the bucket that you want to upload to.
|
||||
}
|
||||
|
||||
// GetUploadURLResponse is received from b2_get_upload_url
|
||||
type GetUploadURLResponse struct {
|
||||
BucketID string `json:"bucketId"` // The unique ID of the bucket.
|
||||
UploadURL string `json:"uploadUrl"` // The URL that can be used to upload files to this bucket, see b2_upload_file.
|
||||
AuthorizationToken string `json:"authorizationToken"` // The authorizationToken that must be used when uploading files to this bucket, see b2_upload_file.
|
||||
}
|
||||
|
||||
// FileInfo is received from b2_upload_file, b2_get_file_info and b2_finish_large_file
|
||||
type FileInfo struct {
|
||||
ID string `json:"fileId"` // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
|
||||
Name string `json:"fileName"` // The name of this file, which can be used with b2_download_file_by_name.
|
||||
Action string `json:"action"` // Either "upload" or "hide". "upload" means a file that was uploaded to B2 Cloud Storage. "hide" means a file version marking the file as hidden, so that it will not show up in b2_list_file_names. The result of b2_list_file_names will contain only "upload". The result of b2_list_file_versions may have both.
|
||||
AccountID string `json:"accountId"` // Your account ID.
|
||||
BucketID string `json:"bucketId"` // The bucket that the file is in.
|
||||
Size int64 `json:"contentLength"` // The number of bytes stored in the file.
|
||||
UploadTimestamp Timestamp `json:"uploadTimestamp"` // This is a UTC time when this file was uploaded.
|
||||
SHA1 string `json:"contentSha1"` // The SHA1 of the bytes stored in the file.
|
||||
ContentType string `json:"contentType"` // The MIME type of the file.
|
||||
Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
|
||||
}
|
||||
|
||||
// CreateBucketRequest is used to create a bucket
|
||||
type CreateBucketRequest struct {
|
||||
AccountID string `json:"accountId"`
|
||||
Name string `json:"bucketName"`
|
||||
Type string `json:"bucketType"`
|
||||
}
|
||||
|
||||
// DeleteBucketRequest is used to create a bucket
|
||||
type DeleteBucketRequest struct {
|
||||
ID string `json:"bucketId"`
|
||||
AccountID string `json:"accountId"`
|
||||
}
|
||||
|
||||
// DeleteFileRequest is used to delete a file version
|
||||
type DeleteFileRequest struct {
|
||||
ID string `json:"fileId"` // The ID of the file, as returned by b2_upload_file, b2_list_file_names, or b2_list_file_versions.
|
||||
Name string `json:"fileName"` // The name of this file.
|
||||
}
|
||||
|
||||
// HideFileRequest is used to delete a file
|
||||
type HideFileRequest struct {
|
||||
BucketID string `json:"bucketId"` // The bucket containing the file to hide.
|
||||
Name string `json:"fileName"` // The name of the file to hide.
|
||||
}
|
||||
|
||||
// GetFileInfoRequest is used to return a FileInfo struct with b2_get_file_info
|
||||
type GetFileInfoRequest struct {
|
||||
ID string `json:"fileId"` // The ID of the file, as returned by b2_upload_file, b2_list_file_names, or b2_list_file_versions.
|
||||
}
|
||||
|
||||
// StartLargeFileRequest (b2_start_large_file) Prepares for uploading the parts of a large file.
|
||||
//
|
||||
// If the original source of the file being uploaded has a last
|
||||
// modified time concept, Backblaze recommends using
|
||||
// src_last_modified_millis as the name, and a string holding the base
|
||||
// 10 number number of milliseconds since midnight, January 1, 1970
|
||||
// UTC. This fits in a 64 bit integer such as the type "long" in the
|
||||
// programming language Java. It is intended to be compatible with
|
||||
// Java's time long. For example, it can be passed directly into the
|
||||
// Java call Date.setTime(long time).
|
||||
//
|
||||
// If the caller knows the SHA1 of the entire large file being
|
||||
// uploaded, Backblaze recommends using large_file_sha1 as the name,
|
||||
// and a 40 byte hex string representing the SHA1.
|
||||
//
|
||||
// Example: { "src_last_modified_millis" : "1452802803026", "large_file_sha1" : "a3195dc1e7b46a2ff5da4b3c179175b75671e80d", "color": "blue" }
|
||||
type StartLargeFileRequest struct {
|
||||
BucketID string `json:"bucketId"` //The ID of the bucket that the file will go in.
|
||||
Name string `json:"fileName"` // The name of the file. See Files for requirements on file names.
|
||||
ContentType string `json:"contentType"` // The MIME type of the content of the file, which will be returned in the Content-Type header when downloading the file. Use the Content-Type b2/x-auto to automatically set the stored Content-Type post upload. In the case where a file extension is absent or the lookup fails, the Content-Type is set to application/octet-stream.
|
||||
Info map[string]string `json:"fileInfo"` // A JSON object holding the name/value pairs for the custom file info.
|
||||
}
|
||||
|
||||
// StartLargeFileResponse is the response to StartLargeFileRequest
|
||||
type StartLargeFileResponse struct {
|
||||
ID string `json:"fileId"` // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
|
||||
Name string `json:"fileName"` // The name of this file, which can be used with b2_download_file_by_name.
|
||||
AccountID string `json:"accountId"` // The identifier for the account.
|
||||
BucketID string `json:"bucketId"` // The unique ID of the bucket.
|
||||
ContentType string `json:"contentType"` // The MIME type of the file.
|
||||
Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
|
||||
UploadTimestamp Timestamp `json:"uploadTimestamp"` // This is a UTC time when this file was uploaded.
|
||||
}
|
||||
|
||||
// GetUploadPartURLRequest is passed to b2_get_upload_part_url
|
||||
type GetUploadPartURLRequest struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
}
|
||||
|
||||
// GetUploadPartURLResponse is received from b2_get_upload_url
|
||||
type GetUploadPartURLResponse struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
UploadURL string `json:"uploadUrl"` // The URL that can be used to upload files to this bucket, see b2_upload_part.
|
||||
AuthorizationToken string `json:"authorizationToken"` // The authorizationToken that must be used when uploading files to this bucket, see b2_upload_part.
|
||||
}
|
||||
|
||||
// UploadPartResponse is the response to b2_upload_part
|
||||
type UploadPartResponse struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
PartNumber int64 `json:"partNumber"` // Which part this is (starting from 1)
|
||||
Size int64 `json:"contentLength"` // The number of bytes stored in the file.
|
||||
SHA1 string `json:"contentSha1"` // The SHA1 of the bytes stored in the file.
|
||||
}
|
||||
|
||||
// FinishLargeFileRequest is passed to b2_finish_large_file
|
||||
//
|
||||
// The response is a FileInfo object (with extra AccountID and BucketID fields which we ignore).
|
||||
//
|
||||
// Large files do not have a SHA1 checksum. The value will always be "none".
|
||||
type FinishLargeFileRequest struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
SHA1s []string `json:"partSha1Array"` // A JSON array of hex SHA1 checksums of the parts of the large file. This is a double-check that the right parts were uploaded in the right order, and that none were missed. Note that the part numbers start at 1, and the SHA1 of the part 1 is the first string in the array, at index 0.
|
||||
}
|
||||
|
||||
// CancelLargeFileRequest is passed to b2_finish_large_file
|
||||
//
|
||||
// The response is a CancelLargeFileResponse
|
||||
type CancelLargeFileRequest struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
}
|
||||
|
||||
// CancelLargeFileResponse is the response to CancelLargeFileRequest
|
||||
type CancelLargeFileResponse struct {
|
||||
ID string `json:"fileId"` // The unique identifier of the file being uploaded.
|
||||
Name string `json:"fileName"` // The name of this file.
|
||||
AccountID string `json:"accountId"` // The identifier for the account.
|
||||
BucketID string `json:"bucketId"` // The unique ID of the bucket.
|
||||
}
|
||||
@@ -1,87 +0,0 @@
|
||||
package api_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/ncw/rclone/b2/api"
|
||||
"github.com/ncw/rclone/fstest"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
var (
|
||||
emptyT api.Timestamp
|
||||
t0 = api.Timestamp(fstest.Time("1970-01-01T01:01:01.123456789Z"))
|
||||
t0r = api.Timestamp(fstest.Time("1970-01-01T01:01:01.123000000Z"))
|
||||
t1 = api.Timestamp(fstest.Time("2001-02-03T04:05:06.123000000Z"))
|
||||
)
|
||||
|
||||
func TestTimestampMarshalJSON(t *testing.T) {
|
||||
resB, err := t0.MarshalJSON()
|
||||
res := string(resB)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "3661123", res)
|
||||
|
||||
resB, err = t1.MarshalJSON()
|
||||
res = string(resB)
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, "981173106123", res)
|
||||
}
|
||||
|
||||
func TestTimestampUnmarshalJSON(t *testing.T) {
|
||||
var tActual api.Timestamp
|
||||
err := tActual.UnmarshalJSON([]byte("981173106123"))
|
||||
require.NoError(t, err)
|
||||
assert.Equal(t, (time.Time)(t1), (time.Time)(tActual))
|
||||
}
|
||||
|
||||
func TestTimestampAddVersion(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
t api.Timestamp
|
||||
in string
|
||||
expected string
|
||||
}{
|
||||
{t0, "potato.txt", "potato-v1970-01-01-010101-123.txt"},
|
||||
{t1, "potato", "potato-v2001-02-03-040506-123"},
|
||||
{t1, "", "-v2001-02-03-040506-123"},
|
||||
} {
|
||||
actual := test.t.AddVersion(test.in)
|
||||
assert.Equal(t, test.expected, actual, test.in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTimestampRemoveVersion(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expectedT api.Timestamp
|
||||
expectedRemote string
|
||||
}{
|
||||
{"potato.txt", emptyT, "potato.txt"},
|
||||
{"potato-v1970-01-01-010101-123.txt", t0r, "potato.txt"},
|
||||
{"potato-v2001-02-03-040506-123", t1, "potato"},
|
||||
{"-v2001-02-03-040506-123", t1, ""},
|
||||
{"potato-v2A01-02-03-040506-123", emptyT, "potato-v2A01-02-03-040506-123"},
|
||||
{"potato-v2001-02-03-040506=123", emptyT, "potato-v2001-02-03-040506=123"},
|
||||
} {
|
||||
actualT, actualRemote := api.RemoveVersion(test.in)
|
||||
assert.Equal(t, test.expectedT, actualT, test.in)
|
||||
assert.Equal(t, test.expectedRemote, actualRemote, test.in)
|
||||
}
|
||||
}
|
||||
|
||||
func TestTimestampIsZero(t *testing.T) {
|
||||
assert.True(t, emptyT.IsZero())
|
||||
assert.False(t, t0.IsZero())
|
||||
assert.False(t, t1.IsZero())
|
||||
}
|
||||
|
||||
func TestTimestampEqual(t *testing.T) {
|
||||
assert.False(t, emptyT.Equal(emptyT))
|
||||
assert.False(t, t0.Equal(emptyT))
|
||||
assert.False(t, emptyT.Equal(t0))
|
||||
assert.False(t, t0.Equal(t1))
|
||||
assert.False(t, t1.Equal(t0))
|
||||
assert.True(t, t0.Equal(t0))
|
||||
assert.True(t, t1.Equal(t1))
|
||||
}
|
||||
@@ -1,284 +0,0 @@
|
||||
package b2
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/ncw/rclone/fstest"
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// Test b2 string encoding
|
||||
// https://www.backblaze.com/b2/docs/string_encoding.html
|
||||
|
||||
var encodeTest = []struct {
|
||||
fullyEncoded string
|
||||
minimallyEncoded string
|
||||
plainText string
|
||||
}{
|
||||
{fullyEncoded: "%20", minimallyEncoded: "+", plainText: " "},
|
||||
{fullyEncoded: "%21", minimallyEncoded: "!", plainText: "!"},
|
||||
{fullyEncoded: "%22", minimallyEncoded: "%22", plainText: "\""},
|
||||
{fullyEncoded: "%23", minimallyEncoded: "%23", plainText: "#"},
|
||||
{fullyEncoded: "%24", minimallyEncoded: "$", plainText: "$"},
|
||||
{fullyEncoded: "%25", minimallyEncoded: "%25", plainText: "%"},
|
||||
{fullyEncoded: "%26", minimallyEncoded: "%26", plainText: "&"},
|
||||
{fullyEncoded: "%27", minimallyEncoded: "'", plainText: "'"},
|
||||
{fullyEncoded: "%28", minimallyEncoded: "(", plainText: "("},
|
||||
{fullyEncoded: "%29", minimallyEncoded: ")", plainText: ")"},
|
||||
{fullyEncoded: "%2A", minimallyEncoded: "*", plainText: "*"},
|
||||
{fullyEncoded: "%2B", minimallyEncoded: "%2B", plainText: "+"},
|
||||
{fullyEncoded: "%2C", minimallyEncoded: "%2C", plainText: ","},
|
||||
{fullyEncoded: "%2D", minimallyEncoded: "-", plainText: "-"},
|
||||
{fullyEncoded: "%2E", minimallyEncoded: ".", plainText: "."},
|
||||
{fullyEncoded: "%2F", minimallyEncoded: "/", plainText: "/"},
|
||||
{fullyEncoded: "%30", minimallyEncoded: "0", plainText: "0"},
|
||||
{fullyEncoded: "%31", minimallyEncoded: "1", plainText: "1"},
|
||||
{fullyEncoded: "%32", minimallyEncoded: "2", plainText: "2"},
|
||||
{fullyEncoded: "%33", minimallyEncoded: "3", plainText: "3"},
|
||||
{fullyEncoded: "%34", minimallyEncoded: "4", plainText: "4"},
|
||||
{fullyEncoded: "%35", minimallyEncoded: "5", plainText: "5"},
|
||||
{fullyEncoded: "%36", minimallyEncoded: "6", plainText: "6"},
|
||||
{fullyEncoded: "%37", minimallyEncoded: "7", plainText: "7"},
|
||||
{fullyEncoded: "%38", minimallyEncoded: "8", plainText: "8"},
|
||||
{fullyEncoded: "%39", minimallyEncoded: "9", plainText: "9"},
|
||||
{fullyEncoded: "%3A", minimallyEncoded: ":", plainText: ":"},
|
||||
{fullyEncoded: "%3B", minimallyEncoded: ";", plainText: ";"},
|
||||
{fullyEncoded: "%3C", minimallyEncoded: "%3C", plainText: "<"},
|
||||
{fullyEncoded: "%3D", minimallyEncoded: "=", plainText: "="},
|
||||
{fullyEncoded: "%3E", minimallyEncoded: "%3E", plainText: ">"},
|
||||
{fullyEncoded: "%3F", minimallyEncoded: "%3F", plainText: "?"},
|
||||
{fullyEncoded: "%40", minimallyEncoded: "@", plainText: "@"},
|
||||
{fullyEncoded: "%41", minimallyEncoded: "A", plainText: "A"},
|
||||
{fullyEncoded: "%42", minimallyEncoded: "B", plainText: "B"},
|
||||
{fullyEncoded: "%43", minimallyEncoded: "C", plainText: "C"},
|
||||
{fullyEncoded: "%44", minimallyEncoded: "D", plainText: "D"},
|
||||
{fullyEncoded: "%45", minimallyEncoded: "E", plainText: "E"},
|
||||
{fullyEncoded: "%46", minimallyEncoded: "F", plainText: "F"},
|
||||
{fullyEncoded: "%47", minimallyEncoded: "G", plainText: "G"},
|
||||
{fullyEncoded: "%48", minimallyEncoded: "H", plainText: "H"},
|
||||
{fullyEncoded: "%49", minimallyEncoded: "I", plainText: "I"},
|
||||
{fullyEncoded: "%4A", minimallyEncoded: "J", plainText: "J"},
|
||||
{fullyEncoded: "%4B", minimallyEncoded: "K", plainText: "K"},
|
||||
{fullyEncoded: "%4C", minimallyEncoded: "L", plainText: "L"},
|
||||
{fullyEncoded: "%4D", minimallyEncoded: "M", plainText: "M"},
|
||||
{fullyEncoded: "%4E", minimallyEncoded: "N", plainText: "N"},
|
||||
{fullyEncoded: "%4F", minimallyEncoded: "O", plainText: "O"},
|
||||
{fullyEncoded: "%50", minimallyEncoded: "P", plainText: "P"},
|
||||
{fullyEncoded: "%51", minimallyEncoded: "Q", plainText: "Q"},
|
||||
{fullyEncoded: "%52", minimallyEncoded: "R", plainText: "R"},
|
||||
{fullyEncoded: "%53", minimallyEncoded: "S", plainText: "S"},
|
||||
{fullyEncoded: "%54", minimallyEncoded: "T", plainText: "T"},
|
||||
{fullyEncoded: "%55", minimallyEncoded: "U", plainText: "U"},
|
||||
{fullyEncoded: "%56", minimallyEncoded: "V", plainText: "V"},
|
||||
{fullyEncoded: "%57", minimallyEncoded: "W", plainText: "W"},
|
||||
{fullyEncoded: "%58", minimallyEncoded: "X", plainText: "X"},
|
||||
{fullyEncoded: "%59", minimallyEncoded: "Y", plainText: "Y"},
|
||||
{fullyEncoded: "%5A", minimallyEncoded: "Z", plainText: "Z"},
|
||||
{fullyEncoded: "%5B", minimallyEncoded: "%5B", plainText: "["},
|
||||
{fullyEncoded: "%5C", minimallyEncoded: "%5C", plainText: "\\"},
|
||||
{fullyEncoded: "%5D", minimallyEncoded: "%5D", plainText: "]"},
|
||||
{fullyEncoded: "%5E", minimallyEncoded: "%5E", plainText: "^"},
|
||||
{fullyEncoded: "%5F", minimallyEncoded: "_", plainText: "_"},
|
||||
{fullyEncoded: "%60", minimallyEncoded: "%60", plainText: "`"},
|
||||
{fullyEncoded: "%61", minimallyEncoded: "a", plainText: "a"},
|
||||
{fullyEncoded: "%62", minimallyEncoded: "b", plainText: "b"},
|
||||
{fullyEncoded: "%63", minimallyEncoded: "c", plainText: "c"},
|
||||
{fullyEncoded: "%64", minimallyEncoded: "d", plainText: "d"},
|
||||
{fullyEncoded: "%65", minimallyEncoded: "e", plainText: "e"},
|
||||
{fullyEncoded: "%66", minimallyEncoded: "f", plainText: "f"},
|
||||
{fullyEncoded: "%67", minimallyEncoded: "g", plainText: "g"},
|
||||
{fullyEncoded: "%68", minimallyEncoded: "h", plainText: "h"},
|
||||
{fullyEncoded: "%69", minimallyEncoded: "i", plainText: "i"},
|
||||
{fullyEncoded: "%6A", minimallyEncoded: "j", plainText: "j"},
|
||||
{fullyEncoded: "%6B", minimallyEncoded: "k", plainText: "k"},
|
||||
{fullyEncoded: "%6C", minimallyEncoded: "l", plainText: "l"},
|
||||
{fullyEncoded: "%6D", minimallyEncoded: "m", plainText: "m"},
|
||||
{fullyEncoded: "%6E", minimallyEncoded: "n", plainText: "n"},
|
||||
{fullyEncoded: "%6F", minimallyEncoded: "o", plainText: "o"},
|
||||
{fullyEncoded: "%70", minimallyEncoded: "p", plainText: "p"},
|
||||
{fullyEncoded: "%71", minimallyEncoded: "q", plainText: "q"},
|
||||
{fullyEncoded: "%72", minimallyEncoded: "r", plainText: "r"},
|
||||
{fullyEncoded: "%73", minimallyEncoded: "s", plainText: "s"},
|
||||
{fullyEncoded: "%74", minimallyEncoded: "t", plainText: "t"},
|
||||
{fullyEncoded: "%75", minimallyEncoded: "u", plainText: "u"},
|
||||
{fullyEncoded: "%76", minimallyEncoded: "v", plainText: "v"},
|
||||
{fullyEncoded: "%77", minimallyEncoded: "w", plainText: "w"},
|
||||
{fullyEncoded: "%78", minimallyEncoded: "x", plainText: "x"},
|
||||
{fullyEncoded: "%79", minimallyEncoded: "y", plainText: "y"},
|
||||
{fullyEncoded: "%7A", minimallyEncoded: "z", plainText: "z"},
|
||||
{fullyEncoded: "%7B", minimallyEncoded: "%7B", plainText: "{"},
|
||||
{fullyEncoded: "%7C", minimallyEncoded: "%7C", plainText: "|"},
|
||||
{fullyEncoded: "%7D", minimallyEncoded: "%7D", plainText: "}"},
|
||||
{fullyEncoded: "%7E", minimallyEncoded: "~", plainText: "~"},
|
||||
{fullyEncoded: "%7F", minimallyEncoded: "%7F", plainText: "\u007f"},
|
||||
{fullyEncoded: "%E8%87%AA%E7%94%B1", minimallyEncoded: "%E8%87%AA%E7%94%B1", plainText: "自由"},
|
||||
{fullyEncoded: "%F0%90%90%80", minimallyEncoded: "%F0%90%90%80", plainText: "𐐀"},
|
||||
}
|
||||
|
||||
func TestUrlEncode(t *testing.T) {
|
||||
for _, test := range encodeTest {
|
||||
got := urlEncode(test.plainText)
|
||||
if got != test.minimallyEncoded && got != test.fullyEncoded {
|
||||
t.Errorf("urlEncode(%q) got %q wanted %q or %q", test.plainText, got, test.minimallyEncoded, test.fullyEncoded)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestTimeString(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in time.Time
|
||||
want string
|
||||
}{
|
||||
{fstest.Time("1970-01-01T00:00:00.000000000Z"), "0"},
|
||||
{fstest.Time("2001-02-03T04:05:10.123123123Z"), "981173110123"},
|
||||
{fstest.Time("2001-02-03T05:05:10.123123123+01:00"), "981173110123"},
|
||||
} {
|
||||
got := timeString(test.in)
|
||||
if test.want != got {
|
||||
t.Logf("%v: want %v got %v", test.in, test.want, got)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestParseTimeString(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
want time.Time
|
||||
wantError string
|
||||
}{
|
||||
{"0", fstest.Time("1970-01-01T00:00:00.000000000Z"), ""},
|
||||
{"981173110123", fstest.Time("2001-02-03T04:05:10.123000000Z"), ""},
|
||||
{"", time.Time{}, ""},
|
||||
{"potato", time.Time{}, `strconv.ParseInt: parsing "potato": invalid syntax`},
|
||||
} {
|
||||
o := Object{}
|
||||
err := o.parseTimeString(test.in)
|
||||
got := o.modTime
|
||||
var gotError string
|
||||
if err != nil {
|
||||
gotError = err.Error()
|
||||
}
|
||||
if test.want != got {
|
||||
t.Logf("%v: want %v got %v", test.in, test.want, got)
|
||||
}
|
||||
if test.wantError != gotError {
|
||||
t.Logf("%v: want error %v got error %v", test.in, test.wantError, gotError)
|
||||
}
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
func TestSendDir(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
lastDir string
|
||||
remote string
|
||||
level int
|
||||
dirNames []string
|
||||
newLastDir string
|
||||
}{
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "test.txt",
|
||||
level: 100,
|
||||
dirNames: nil,
|
||||
newLastDir: "",
|
||||
},
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "potato/test.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"potato"},
|
||||
newLastDir: "potato",
|
||||
},
|
||||
{
|
||||
lastDir: "potato",
|
||||
remote: "potato/test.txt",
|
||||
level: 100,
|
||||
dirNames: nil,
|
||||
newLastDir: "potato",
|
||||
},
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "potato/sausage/test.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"potato", "potato/sausage"},
|
||||
newLastDir: "potato/sausage",
|
||||
},
|
||||
{
|
||||
lastDir: "potato",
|
||||
remote: "potato/sausage/test.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"potato/sausage"},
|
||||
newLastDir: "potato/sausage",
|
||||
},
|
||||
{
|
||||
lastDir: "potato/sausage",
|
||||
remote: "potato/sausage/test.txt",
|
||||
level: 100,
|
||||
dirNames: nil,
|
||||
newLastDir: "potato/sausage",
|
||||
},
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"a", "a/b", "a/b/c", "a/b/c/d", "a/b/c/d/e"},
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
{
|
||||
lastDir: "a/b/c/d/e",
|
||||
remote: "a/b/c/d/E/f.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"a/b/c/d/E"},
|
||||
newLastDir: "a/b/c/d/E",
|
||||
},
|
||||
{
|
||||
lastDir: "a/b/c/d/e",
|
||||
remote: "a/b/C/D/E/f.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"a/b/C", "a/b/C/D", "a/b/C/D/E"},
|
||||
newLastDir: "a/b/C/D/E",
|
||||
},
|
||||
{
|
||||
lastDir: "a/b/c",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 100,
|
||||
dirNames: []string{"a/b/c/d", "a/b/c/d/e"},
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 1,
|
||||
dirNames: []string{"a"},
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
{
|
||||
lastDir: "a/b/c",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 1,
|
||||
dirNames: nil,
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
{
|
||||
lastDir: "",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 3,
|
||||
dirNames: []string{"a", "a/b", "a/b/c"},
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
{
|
||||
lastDir: "a/b/C/D/E",
|
||||
remote: "a/b/c/d/e/f.txt",
|
||||
level: 3,
|
||||
dirNames: []string{"a/b/c"},
|
||||
newLastDir: "a/b/c/d/e",
|
||||
},
|
||||
} {
|
||||
dirNames, newLastDir := sendDir(test.lastDir, test.remote, test.level)
|
||||
assert.Equal(t, test.dirNames, dirNames, "dirNames fail for sendDir(%q,%q,%v)", test.lastDir, test.remote, test.level)
|
||||
assert.Equal(t, test.newLastDir, newLastDir, "newLastDir fail for sendDir(%q,%q,%v)", test.lastDir, test.remote, test.level)
|
||||
}
|
||||
}
|
||||
@@ -1,58 +0,0 @@
|
||||
// Test B2 filesystem interface
|
||||
//
|
||||
// Automatically generated - DO NOT EDIT
|
||||
// Regenerate with: make gen_tests
|
||||
package b2_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/b2"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/fstest/fstests"
|
||||
)
|
||||
|
||||
func TestSetup(t *testing.T) {
|
||||
fstests.NilObject = fs.Object((*b2.Object)(nil))
|
||||
fstests.RemoteName = "TestB2:"
|
||||
}
|
||||
|
||||
// Generic tests for the Fs
|
||||
func TestInit(t *testing.T) { fstests.TestInit(t) }
|
||||
func TestFsString(t *testing.T) { fstests.TestFsString(t) }
|
||||
func TestFsRmdirEmpty(t *testing.T) { fstests.TestFsRmdirEmpty(t) }
|
||||
func TestFsRmdirNotFound(t *testing.T) { fstests.TestFsRmdirNotFound(t) }
|
||||
func TestFsMkdir(t *testing.T) { fstests.TestFsMkdir(t) }
|
||||
func TestFsListEmpty(t *testing.T) { fstests.TestFsListEmpty(t) }
|
||||
func TestFsListDirEmpty(t *testing.T) { fstests.TestFsListDirEmpty(t) }
|
||||
func TestFsNewObjectNotFound(t *testing.T) { fstests.TestFsNewObjectNotFound(t) }
|
||||
func TestFsPutFile1(t *testing.T) { fstests.TestFsPutFile1(t) }
|
||||
func TestFsPutFile2(t *testing.T) { fstests.TestFsPutFile2(t) }
|
||||
func TestFsUpdateFile1(t *testing.T) { fstests.TestFsUpdateFile1(t) }
|
||||
func TestFsListDirFile2(t *testing.T) { fstests.TestFsListDirFile2(t) }
|
||||
func TestFsListDirRoot(t *testing.T) { fstests.TestFsListDirRoot(t) }
|
||||
func TestFsListSubdir(t *testing.T) { fstests.TestFsListSubdir(t) }
|
||||
func TestFsListLevel2(t *testing.T) { fstests.TestFsListLevel2(t) }
|
||||
func TestFsListFile1(t *testing.T) { fstests.TestFsListFile1(t) }
|
||||
func TestFsNewObject(t *testing.T) { fstests.TestFsNewObject(t) }
|
||||
func TestFsListFile1and2(t *testing.T) { fstests.TestFsListFile1and2(t) }
|
||||
func TestFsCopy(t *testing.T) { fstests.TestFsCopy(t) }
|
||||
func TestFsMove(t *testing.T) { fstests.TestFsMove(t) }
|
||||
func TestFsDirMove(t *testing.T) { fstests.TestFsDirMove(t) }
|
||||
func TestFsRmdirFull(t *testing.T) { fstests.TestFsRmdirFull(t) }
|
||||
func TestFsPrecision(t *testing.T) { fstests.TestFsPrecision(t) }
|
||||
func TestObjectString(t *testing.T) { fstests.TestObjectString(t) }
|
||||
func TestObjectFs(t *testing.T) { fstests.TestObjectFs(t) }
|
||||
func TestObjectRemote(t *testing.T) { fstests.TestObjectRemote(t) }
|
||||
func TestObjectHashes(t *testing.T) { fstests.TestObjectHashes(t) }
|
||||
func TestObjectModTime(t *testing.T) { fstests.TestObjectModTime(t) }
|
||||
func TestObjectSetModTime(t *testing.T) { fstests.TestObjectSetModTime(t) }
|
||||
func TestObjectSize(t *testing.T) { fstests.TestObjectSize(t) }
|
||||
func TestObjectOpen(t *testing.T) { fstests.TestObjectOpen(t) }
|
||||
func TestObjectUpdate(t *testing.T) { fstests.TestObjectUpdate(t) }
|
||||
func TestObjectStorable(t *testing.T) { fstests.TestObjectStorable(t) }
|
||||
func TestFsIsFile(t *testing.T) { fstests.TestFsIsFile(t) }
|
||||
func TestFsIsFileNotFound(t *testing.T) { fstests.TestFsIsFileNotFound(t) }
|
||||
func TestObjectRemove(t *testing.T) { fstests.TestObjectRemove(t) }
|
||||
func TestObjectPurge(t *testing.T) { fstests.TestObjectPurge(t) }
|
||||
func TestFinalise(t *testing.T) { fstests.TestFinalise(t) }
|
||||
301
b2/upload.go
301
b2/upload.go
@@ -1,301 +0,0 @@
|
||||
// Upload large files for b2
|
||||
//
|
||||
// Docs - https://www.backblaze.com/b2/docs/large_files.html
|
||||
|
||||
package b2
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha1"
|
||||
"fmt"
|
||||
"io"
|
||||
"sync"
|
||||
|
||||
"github.com/ncw/rclone/b2/api"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/rest"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// largeUpload is used to control the upload of large files which need chunking
|
||||
type largeUpload struct {
|
||||
f *Fs // parent Fs
|
||||
o *Object // object being uploaded
|
||||
in io.Reader // read the data from here
|
||||
id string // ID of the file being uploaded
|
||||
size int64 // total size
|
||||
parts int64 // calculated number of parts
|
||||
sha1s []string // slice of SHA1s for each part
|
||||
uploadMu sync.Mutex // lock for upload variable
|
||||
uploads []*api.GetUploadPartURLResponse // result of get upload URL calls
|
||||
}
|
||||
|
||||
// newLargeUpload starts an upload of object o from in with metadata in src
|
||||
func (f *Fs) newLargeUpload(o *Object, in io.Reader, src fs.ObjectInfo) (up *largeUpload, err error) {
|
||||
remote := o.remote
|
||||
size := src.Size()
|
||||
parts := size / int64(chunkSize)
|
||||
if size%int64(chunkSize) != 0 {
|
||||
parts++
|
||||
}
|
||||
if parts > maxParts {
|
||||
return nil, errors.Errorf("%q too big (%d bytes) makes too many parts %d > %d - increase --b2-chunk-size", remote, size, parts, maxParts)
|
||||
}
|
||||
modTime := src.ModTime()
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/b2_start_large_file",
|
||||
}
|
||||
bucketID, err := f.getBucketID()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var request = api.StartLargeFileRequest{
|
||||
BucketID: bucketID,
|
||||
Name: o.fs.root + remote,
|
||||
ContentType: fs.MimeType(src),
|
||||
Info: map[string]string{
|
||||
timeKey: timeString(modTime),
|
||||
},
|
||||
}
|
||||
// Set the SHA1 if known
|
||||
if calculatedSha1, err := src.Hash(fs.HashSHA1); err == nil && calculatedSha1 != "" {
|
||||
request.Info[sha1Key] = calculatedSha1
|
||||
}
|
||||
var response api.StartLargeFileResponse
|
||||
err = f.pacer.Call(func() (bool, error) {
|
||||
resp, err := f.srv.CallJSON(&opts, &request, &response)
|
||||
return f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
up = &largeUpload{
|
||||
f: f,
|
||||
o: o,
|
||||
in: in,
|
||||
id: response.ID,
|
||||
size: size,
|
||||
parts: parts,
|
||||
sha1s: make([]string, parts),
|
||||
}
|
||||
return up, nil
|
||||
}
|
||||
|
||||
// getUploadURL returns the upload info with the UploadURL and the AuthorizationToken
|
||||
//
|
||||
// This should be returned with returnUploadURL when finished
|
||||
func (up *largeUpload) getUploadURL() (upload *api.GetUploadPartURLResponse, err error) {
|
||||
up.uploadMu.Lock()
|
||||
defer up.uploadMu.Unlock()
|
||||
if len(up.uploads) == 0 {
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/b2_get_upload_part_url",
|
||||
}
|
||||
var request = api.GetUploadPartURLRequest{
|
||||
ID: up.id,
|
||||
}
|
||||
err := up.f.pacer.Call(func() (bool, error) {
|
||||
resp, err := up.f.srv.CallJSON(&opts, &request, &upload)
|
||||
return up.f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to get upload URL")
|
||||
}
|
||||
} else {
|
||||
upload, up.uploads = up.uploads[0], up.uploads[1:]
|
||||
}
|
||||
return upload, nil
|
||||
}
|
||||
|
||||
// returnUploadURL returns the UploadURL to the cache
|
||||
func (up *largeUpload) returnUploadURL(upload *api.GetUploadPartURLResponse) {
|
||||
if upload == nil {
|
||||
return
|
||||
}
|
||||
up.uploadMu.Lock()
|
||||
up.uploads = append(up.uploads, upload)
|
||||
up.uploadMu.Unlock()
|
||||
}
|
||||
|
||||
// clearUploadURL clears the current UploadURL and the AuthorizationToken
|
||||
func (up *largeUpload) clearUploadURL() {
|
||||
up.uploadMu.Lock()
|
||||
up.uploads = nil
|
||||
up.uploadMu.Unlock()
|
||||
}
|
||||
|
||||
// Transfer a chunk
|
||||
func (up *largeUpload) transferChunk(part int64, body []byte) error {
|
||||
calculatedSHA1 := fmt.Sprintf("%x", sha1.Sum(body))
|
||||
up.sha1s[part-1] = calculatedSHA1
|
||||
size := int64(len(body))
|
||||
err := up.f.pacer.Call(func() (bool, error) {
|
||||
fs.Debug(up.o, "Sending chunk %d length %d", part, len(body))
|
||||
|
||||
// Get upload URL
|
||||
upload, err := up.getUploadURL()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
|
||||
// Authorization
|
||||
//
|
||||
// An upload authorization token, from b2_get_upload_part_url.
|
||||
//
|
||||
// X-Bz-Part-Number
|
||||
//
|
||||
// A number from 1 to 10000. The parts uploaded for one file
|
||||
// must have contiguous numbers, starting with 1.
|
||||
//
|
||||
// Content-Length
|
||||
//
|
||||
// The number of bytes in the file being uploaded. Note that
|
||||
// this header is required; you cannot leave it out and just
|
||||
// use chunked encoding. The minimum size of every part but
|
||||
// the last one is 100MB.
|
||||
//
|
||||
// X-Bz-Content-Sha1
|
||||
//
|
||||
// The SHA1 checksum of the this part of the file. B2 will
|
||||
// check this when the part is uploaded, to make sure that the
|
||||
// data arrived correctly. The same SHA1 checksum must be
|
||||
// passed to b2_finish_large_file.
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Absolute: true,
|
||||
Path: upload.UploadURL,
|
||||
Body: fs.AccountPart(up.o, bytes.NewBuffer(body)),
|
||||
ExtraHeaders: map[string]string{
|
||||
"Authorization": upload.AuthorizationToken,
|
||||
"X-Bz-Part-Number": fmt.Sprintf("%d", part),
|
||||
sha1Header: calculatedSHA1,
|
||||
},
|
||||
ContentLength: &size,
|
||||
}
|
||||
|
||||
var response api.UploadPartResponse
|
||||
|
||||
resp, err := up.f.srv.CallJSON(&opts, nil, &response)
|
||||
retry, err := up.f.shouldRetryNoReauth(resp, err)
|
||||
// On retryable error clear PartUploadURL
|
||||
if retry {
|
||||
fs.Debug(up.o, "Clearing part upload URL because of error: %v", err)
|
||||
upload = nil
|
||||
}
|
||||
up.returnUploadURL(upload)
|
||||
return retry, err
|
||||
})
|
||||
if err != nil {
|
||||
fs.Debug(up.o, "Error sending chunk %d: %v", part, err)
|
||||
} else {
|
||||
fs.Debug(up.o, "Done sending chunk %d", part)
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// finish closes off the large upload
|
||||
func (up *largeUpload) finish() error {
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/b2_finish_large_file",
|
||||
}
|
||||
var request = api.FinishLargeFileRequest{
|
||||
ID: up.id,
|
||||
SHA1s: up.sha1s,
|
||||
}
|
||||
var response api.FileInfo
|
||||
err := up.f.pacer.Call(func() (bool, error) {
|
||||
resp, err := up.f.srv.CallJSON(&opts, &request, &response)
|
||||
return up.f.shouldRetry(resp, err)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return up.o.decodeMetaDataFileInfo(&response)
|
||||
}
|
||||
|
||||
// cancel aborts the large upload
|
||||
func (up *largeUpload) cancel() error {
|
||||
opts := rest.Opts{
|
||||
Method: "POST",
|
||||
Path: "/b2_cancel_large_file",
|
||||
}
|
||||
var request = api.CancelLargeFileRequest{
|
||||
ID: up.id,
|
||||
}
|
||||
var response api.CancelLargeFileResponse
|
||||
err := up.f.pacer.Call(func() (bool, error) {
|
||||
resp, err := up.f.srv.CallJSON(&opts, &request, &response)
|
||||
return up.f.shouldRetry(resp, err)
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// Upload uploads the chunks from the input
|
||||
func (up *largeUpload) Upload() error {
|
||||
fs.Debug(up.o, "Starting upload of large file in %d chunks (id %q)", up.parts, up.id)
|
||||
remaining := up.size
|
||||
errs := make(chan error, 1)
|
||||
var wg sync.WaitGroup
|
||||
var err error
|
||||
fs.AccountByPart(up.o) // Cancel whole file accounting before reading
|
||||
outer:
|
||||
for part := int64(1); part <= up.parts; part++ {
|
||||
// Check any errors
|
||||
select {
|
||||
case err = <-errs:
|
||||
break outer
|
||||
default:
|
||||
}
|
||||
|
||||
reqSize := remaining
|
||||
if reqSize >= int64(chunkSize) {
|
||||
reqSize = int64(chunkSize)
|
||||
}
|
||||
|
||||
// Read the chunk
|
||||
buf := make([]byte, reqSize)
|
||||
_, err = io.ReadFull(up.in, buf)
|
||||
if err != nil {
|
||||
break outer
|
||||
}
|
||||
|
||||
// Transfer the chunk
|
||||
// Get upload Token
|
||||
up.f.getUploadToken()
|
||||
wg.Add(1)
|
||||
go func(part int64, buf []byte) {
|
||||
defer up.f.returnUploadToken()
|
||||
defer wg.Done()
|
||||
err := up.transferChunk(part, buf)
|
||||
if err != nil {
|
||||
select {
|
||||
case errs <- err:
|
||||
default:
|
||||
}
|
||||
}
|
||||
}(part, buf)
|
||||
|
||||
remaining -= reqSize
|
||||
}
|
||||
wg.Wait()
|
||||
if err == nil {
|
||||
select {
|
||||
case err = <-errs:
|
||||
default:
|
||||
}
|
||||
}
|
||||
if err != nil {
|
||||
fs.Debug(up.o, "Cancelling large file upload due to error: %v", err)
|
||||
cancelErr := up.cancel()
|
||||
if cancelErr != nil {
|
||||
fs.ErrorLog(up.o, "Failed to cancel large file upload: %v", cancelErr)
|
||||
}
|
||||
return err
|
||||
}
|
||||
// Check any errors
|
||||
fs.Debug(up.o, "Finishing large file upload")
|
||||
return up.finish()
|
||||
}
|
||||
@@ -1,31 +0,0 @@
|
||||
// Package all imports all the commands
|
||||
package all
|
||||
|
||||
import (
|
||||
// Active commands
|
||||
_ "github.com/ncw/rclone/cmd"
|
||||
_ "github.com/ncw/rclone/cmd/authorize"
|
||||
_ "github.com/ncw/rclone/cmd/cat"
|
||||
_ "github.com/ncw/rclone/cmd/check"
|
||||
_ "github.com/ncw/rclone/cmd/cleanup"
|
||||
_ "github.com/ncw/rclone/cmd/config"
|
||||
_ "github.com/ncw/rclone/cmd/copy"
|
||||
_ "github.com/ncw/rclone/cmd/dedupe"
|
||||
_ "github.com/ncw/rclone/cmd/delete"
|
||||
_ "github.com/ncw/rclone/cmd/genautocomplete"
|
||||
_ "github.com/ncw/rclone/cmd/gendocs"
|
||||
_ "github.com/ncw/rclone/cmd/ls"
|
||||
_ "github.com/ncw/rclone/cmd/lsd"
|
||||
_ "github.com/ncw/rclone/cmd/lsl"
|
||||
_ "github.com/ncw/rclone/cmd/md5sum"
|
||||
_ "github.com/ncw/rclone/cmd/memtest"
|
||||
_ "github.com/ncw/rclone/cmd/mkdir"
|
||||
_ "github.com/ncw/rclone/cmd/mount"
|
||||
_ "github.com/ncw/rclone/cmd/move"
|
||||
_ "github.com/ncw/rclone/cmd/purge"
|
||||
_ "github.com/ncw/rclone/cmd/rmdir"
|
||||
_ "github.com/ncw/rclone/cmd/sha1sum"
|
||||
_ "github.com/ncw/rclone/cmd/size"
|
||||
_ "github.com/ncw/rclone/cmd/sync"
|
||||
_ "github.com/ncw/rclone/cmd/version"
|
||||
)
|
||||
@@ -1,24 +0,0 @@
|
||||
package authorize
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(authorizeCmd)
|
||||
}
|
||||
|
||||
var authorizeCmd = &cobra.Command{
|
||||
Use: "authorize",
|
||||
Short: `Remote authorization.`,
|
||||
Long: `
|
||||
Remote authorization. Used to authorize a remote or headless
|
||||
rclone from a machine with a browser - use as instructed by
|
||||
rclone config.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 3, command, args)
|
||||
fs.Authorize(args)
|
||||
},
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
package cat
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(catCmd)
|
||||
}
|
||||
|
||||
var catCmd = &cobra.Command{
|
||||
Use: "cat remote:path",
|
||||
Short: `Concatenates any files and sends them to stdout.`,
|
||||
Long: `
|
||||
rclone cat sends any files to standard output.
|
||||
|
||||
You can use it like this to output a single file
|
||||
|
||||
rclone cat remote:path/to/file
|
||||
|
||||
Or like this to output any file in dir or subdirectories.
|
||||
|
||||
rclone cat remote:path/to/dir
|
||||
|
||||
Or like this to output any .txt files in dir or subdirectories.
|
||||
|
||||
rclone --include "*.txt" cat remote:path/to/dir
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.Cat(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,30 +0,0 @@
|
||||
package check
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(checkCmd)
|
||||
}
|
||||
|
||||
var checkCmd = &cobra.Command{
|
||||
Use: "check source:path dest:path",
|
||||
Short: `Checks the files in the source and destination match.`,
|
||||
Long: `
|
||||
Checks the files in the source and destination match. It
|
||||
compares sizes and MD5SUMs and prints a report of files which
|
||||
don't match. It doesn't alter the source or destination.
|
||||
|
||||
` + "`" + `--size-only` + "`" + ` may be used to only compare the sizes, not the MD5SUMs.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(2, 2, command, args)
|
||||
fsrc, fdst := cmd.NewFsSrcDst(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.Check(fdst, fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,27 +0,0 @@
|
||||
package cleanup
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(cleanupCmd)
|
||||
}
|
||||
|
||||
var cleanupCmd = &cobra.Command{
|
||||
Use: "cleanup remote:path",
|
||||
Short: `Clean up the remote if possible`,
|
||||
Long: `
|
||||
Clean up the remote if possible. Empty the trash or delete old file
|
||||
versions. Not supported by all remotes.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.CleanUp(fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
293
cmd/cmd.go
293
cmd/cmd.go
@@ -1,293 +0,0 @@
|
||||
// Package cmd implemnts the rclone command
|
||||
//
|
||||
// It is in a sub package so it's internals can be re-used elsewhere
|
||||
package cmd
|
||||
|
||||
// FIXME only attach the remote flags when using a remote???
|
||||
// would probably mean bringing all the flags in to here? Or define some flagsets in fs...
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"path"
|
||||
"runtime"
|
||||
"runtime/pprof"
|
||||
"time"
|
||||
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/spf13/pflag"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// Globals
|
||||
var (
|
||||
// Flags
|
||||
cpuProfile = pflag.StringP("cpuprofile", "", "", "Write cpu profile to file")
|
||||
memProfile = pflag.String("memprofile", "", "Write memory profile to file")
|
||||
statsInterval = pflag.DurationP("stats", "", time.Minute*1, "Interval to print stats (0 to disable)")
|
||||
version bool
|
||||
logFile = pflag.StringP("log-file", "", "", "Log everything to this file")
|
||||
retries = pflag.IntP("retries", "", 3, "Retry operations this many times if they fail")
|
||||
)
|
||||
|
||||
// Root is the main rclone command
|
||||
var Root = &cobra.Command{
|
||||
Use: "rclone",
|
||||
Short: "Sync files and directories to and from local and remote object stores - " + fs.Version,
|
||||
Long: `
|
||||
Rclone is a command line program to sync files and directories to and
|
||||
from various cloud storage systems, such as:
|
||||
|
||||
* Google Drive
|
||||
* Amazon S3
|
||||
* Openstack Swift / Rackspace cloud files / Memset Memstore
|
||||
* Dropbox
|
||||
* Google Cloud Storage
|
||||
* Amazon Drive
|
||||
* Microsoft One Drive
|
||||
* Hubic
|
||||
* Backblaze B2
|
||||
* Yandex Disk
|
||||
* The local filesystem
|
||||
|
||||
Features
|
||||
|
||||
* MD5/SHA1 hashes checked at all times for file integrity
|
||||
* Timestamps preserved on files
|
||||
* Partial syncs supported on a whole file basis
|
||||
* Copy mode to just copy new/changed files
|
||||
* Sync (one way) mode to make a directory identical
|
||||
* Check mode to check for file hash equality
|
||||
* Can sync to and from network, eg two different cloud accounts
|
||||
|
||||
See the home page for installation, usage, documentation, changelog
|
||||
and configuration walkthroughs.
|
||||
|
||||
* http://rclone.org/
|
||||
`,
|
||||
}
|
||||
|
||||
// runRoot implements the main rclone command with no subcommands
|
||||
func runRoot(cmd *cobra.Command, args []string) {
|
||||
if version {
|
||||
ShowVersion()
|
||||
os.Exit(0)
|
||||
} else {
|
||||
_ = Root.Usage()
|
||||
fmt.Fprintf(os.Stderr, "Command not found.\n")
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
func init() {
|
||||
Root.Run = runRoot
|
||||
Root.Flags().BoolVarP(&version, "version", "V", false, "Print the version number")
|
||||
cobra.OnInitialize(initConfig)
|
||||
}
|
||||
|
||||
// ShowVersion prints the version to stdout
|
||||
func ShowVersion() {
|
||||
fmt.Printf("rclone %s\n", fs.Version)
|
||||
}
|
||||
|
||||
// newFsSrc creates a src Fs from a name
|
||||
//
|
||||
// This can point to a file
|
||||
func newFsSrc(remote string) fs.Fs {
|
||||
fsInfo, configName, fsPath, err := fs.ParseRemote(remote)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatalf("Failed to create file system for %q: %v", remote, err)
|
||||
}
|
||||
f, err := fsInfo.NewFs(configName, fsPath)
|
||||
if err == fs.ErrorIsFile {
|
||||
if !fs.Config.Filter.InActive() {
|
||||
fs.Stats.Error()
|
||||
log.Fatalf("Can't limit to single files when using filters: %v", remote)
|
||||
}
|
||||
// Limit transfers to this file
|
||||
err = fs.Config.Filter.AddFile(path.Base(fsPath))
|
||||
// Set --no-traverse as only one file
|
||||
fs.Config.NoTraverse = true
|
||||
}
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatalf("Failed to create file system for %q: %v", remote, err)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// newFsDst creates a dst Fs from a name
|
||||
//
|
||||
// This must point to a directory
|
||||
func newFsDst(remote string) fs.Fs {
|
||||
f, err := fs.NewFs(remote)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatalf("Failed to create file system for %q: %v", remote, err)
|
||||
}
|
||||
return f
|
||||
}
|
||||
|
||||
// NewFsSrcDst creates a new src and dst fs from the arguments
|
||||
func NewFsSrcDst(args []string) (fs.Fs, fs.Fs) {
|
||||
fsrc, fdst := newFsSrc(args[0]), newFsDst(args[1])
|
||||
fs.CalculateModifyWindow(fdst, fsrc)
|
||||
return fsrc, fdst
|
||||
}
|
||||
|
||||
// NewFsSrc creates a new src fs from the arguments
|
||||
func NewFsSrc(args []string) fs.Fs {
|
||||
fsrc := newFsSrc(args[0])
|
||||
fs.CalculateModifyWindow(fsrc)
|
||||
return fsrc
|
||||
}
|
||||
|
||||
// NewFsDst creates a new dst fs from the arguments
|
||||
//
|
||||
// Dst fs-es can't point to single files
|
||||
func NewFsDst(args []string) fs.Fs {
|
||||
fdst := newFsDst(args[0])
|
||||
fs.CalculateModifyWindow(fdst)
|
||||
return fdst
|
||||
}
|
||||
|
||||
// Run the function with stats and retries if required
|
||||
func Run(Retry bool, cmd *cobra.Command, f func() error) {
|
||||
var err error
|
||||
stopStats := startStats()
|
||||
for try := 1; try <= *retries; try++ {
|
||||
err = f()
|
||||
if !Retry || (err == nil && !fs.Stats.Errored()) {
|
||||
break
|
||||
}
|
||||
if fs.IsFatalError(err) {
|
||||
fs.Log(nil, "Fatal error received - not attempting retries")
|
||||
break
|
||||
}
|
||||
if fs.IsNoRetryError(err) {
|
||||
fs.Log(nil, "Can't retry this error - not attempting retries")
|
||||
break
|
||||
}
|
||||
if err != nil {
|
||||
fs.Log(nil, "Attempt %d/%d failed with %d errors and: %v", try, *retries, fs.Stats.GetErrors(), err)
|
||||
} else {
|
||||
fs.Log(nil, "Attempt %d/%d failed with %d errors", try, *retries, fs.Stats.GetErrors())
|
||||
}
|
||||
if try < *retries {
|
||||
fs.Stats.ResetErrors()
|
||||
}
|
||||
}
|
||||
close(stopStats)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to %s: %v", cmd.Name(), err)
|
||||
}
|
||||
if !fs.Config.Quiet || fs.Stats.Errored() || *statsInterval > 0 {
|
||||
fs.Log(nil, "%s", fs.Stats)
|
||||
}
|
||||
if fs.Config.Verbose {
|
||||
fs.Debug(nil, "Go routines at exit %d\n", runtime.NumGoroutine())
|
||||
}
|
||||
if fs.Stats.Errored() {
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// CheckArgs checks there are enough arguments and prints a message if not
|
||||
func CheckArgs(MinArgs, MaxArgs int, cmd *cobra.Command, args []string) {
|
||||
if len(args) < MinArgs {
|
||||
_ = cmd.Usage()
|
||||
fmt.Fprintf(os.Stderr, "Command %s needs %d arguments mininum\n", cmd.Name(), MinArgs)
|
||||
os.Exit(1)
|
||||
} else if len(args) > MaxArgs {
|
||||
_ = cmd.Usage()
|
||||
fmt.Fprintf(os.Stderr, "Command %s needs %d arguments maximum\n", cmd.Name(), MaxArgs)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// startStats prints the stats every statsInterval
|
||||
//
|
||||
// It returns a channel which should be closed to stop the stats.
|
||||
func startStats() chan struct{} {
|
||||
stopStats := make(chan struct{})
|
||||
if *statsInterval > 0 {
|
||||
go func() {
|
||||
ticker := time.NewTicker(*statsInterval)
|
||||
for {
|
||||
select {
|
||||
case <-ticker.C:
|
||||
fs.Stats.Log()
|
||||
case <-stopStats:
|
||||
ticker.Stop()
|
||||
return
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
||||
return stopStats
|
||||
}
|
||||
|
||||
// initConfig is run by cobra after initialising the flags
|
||||
func initConfig() {
|
||||
// Log file output
|
||||
if *logFile != "" {
|
||||
f, err := os.OpenFile(*logFile, os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0640)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to open log file: %v", err)
|
||||
}
|
||||
_, err = f.Seek(0, os.SEEK_END)
|
||||
if err != nil {
|
||||
fs.ErrorLog(nil, "Failed to seek log file to end: %v", err)
|
||||
}
|
||||
log.SetOutput(f)
|
||||
fs.DebugLogger.SetOutput(f)
|
||||
redirectStderr(f)
|
||||
}
|
||||
|
||||
// Load the rest of the config now we have started the logger
|
||||
fs.LoadConfig()
|
||||
|
||||
// Write the args for debug purposes
|
||||
fs.Debug("rclone", "Version %q starting with parameters %q", fs.Version, os.Args)
|
||||
|
||||
// Setup CPU profiling if desired
|
||||
if *cpuProfile != "" {
|
||||
fs.Log(nil, "Creating CPU profile %q\n", *cpuProfile)
|
||||
f, err := os.Create(*cpuProfile)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatal(err)
|
||||
}
|
||||
err = pprof.StartCPUProfile(f)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatal(err)
|
||||
}
|
||||
defer pprof.StopCPUProfile()
|
||||
}
|
||||
|
||||
// Setup memory profiling if desired
|
||||
if *memProfile != "" {
|
||||
defer func() {
|
||||
fs.Log(nil, "Saving Memory profile %q\n", *memProfile)
|
||||
f, err := os.Create(*memProfile)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatal(err)
|
||||
}
|
||||
err = pprof.WriteHeapProfile(f)
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatal(err)
|
||||
}
|
||||
err = f.Close()
|
||||
if err != nil {
|
||||
fs.Stats.Error()
|
||||
log.Fatal(err)
|
||||
}
|
||||
}()
|
||||
}
|
||||
}
|
||||
@@ -1,20 +0,0 @@
|
||||
package config
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(configCmd)
|
||||
}
|
||||
|
||||
var configCmd = &cobra.Command{
|
||||
Use: "config",
|
||||
Short: `Enter an interactive configuration session.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(0, 0, command, args)
|
||||
fs.EditConfig()
|
||||
},
|
||||
}
|
||||
@@ -1,63 +0,0 @@
|
||||
package copy
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(copyCmd)
|
||||
}
|
||||
|
||||
var copyCmd = &cobra.Command{
|
||||
Use: "copy source:path dest:path",
|
||||
Short: `Copy files from source to dest, skipping already copied`,
|
||||
Long: `
|
||||
Copy the source to the destination. Doesn't transfer
|
||||
unchanged files, testing by size and modification time or
|
||||
MD5SUM. Doesn't delete files from the destination.
|
||||
|
||||
Note that it is always the contents of the directory that is synced,
|
||||
not the directory so when source:path is a directory, it's the
|
||||
contents of source:path that are copied, not the directory name and
|
||||
contents.
|
||||
|
||||
If dest:path doesn't exist, it is created and the source:path contents
|
||||
go there.
|
||||
|
||||
For example
|
||||
|
||||
rclone copy source:sourcepath dest:destpath
|
||||
|
||||
Let's say there are two files in sourcepath
|
||||
|
||||
sourcepath/one.txt
|
||||
sourcepath/two.txt
|
||||
|
||||
This copies them to
|
||||
|
||||
destpath/one.txt
|
||||
destpath/two.txt
|
||||
|
||||
Not to
|
||||
|
||||
destpath/sourcepath/one.txt
|
||||
destpath/sourcepath/two.txt
|
||||
|
||||
If you are familiar with ` + "`" + `rsync` + "`" + `, rclone always works as if you had
|
||||
written a trailing / - meaning "copy the contents of this directory".
|
||||
This applies to all commands and whether you are talking about the
|
||||
source or destination.
|
||||
|
||||
See the ` + "`" + `--no-traverse` + "`" + ` option for controlling whether rclone lists
|
||||
the destination directory or not.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(2, 2, command, args)
|
||||
fsrc, fdst := cmd.NewFsSrcDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.CopyDir(fdst, fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,113 +0,0 @@
|
||||
package dedupe
|
||||
|
||||
import (
|
||||
"log"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
var (
|
||||
dedupeMode = fs.DeduplicateInteractive
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(dedupeCmd)
|
||||
dedupeCmd.Flags().VarP(&dedupeMode, "dedupe-mode", "", "Dedupe mode interactive|skip|first|newest|oldest|rename.")
|
||||
}
|
||||
|
||||
var dedupeCmd = &cobra.Command{
|
||||
Use: "dedupe [mode] remote:path",
|
||||
Short: `Interactively find duplicate files delete/rename them.`,
|
||||
Long: `
|
||||
By default ` + "`" + `dedup` + "`" + ` interactively finds duplicate files and offers to
|
||||
delete all but one or rename them to be different. Only useful with
|
||||
Google Drive which can have duplicate file names.
|
||||
|
||||
The ` + "`" + `dedupe` + "`" + ` command will delete all but one of any identical (same
|
||||
md5sum) files it finds without confirmation. This means that for most
|
||||
duplicated files the ` + "`" + `dedupe` + "`" + ` command will not be interactive. You
|
||||
can use ` + "`" + `--dry-run` + "`" + ` to see what would happen without doing anything.
|
||||
|
||||
Here is an example run.
|
||||
|
||||
Before - with duplicates
|
||||
|
||||
$ rclone lsl drive:dupes
|
||||
6048320 2016-03-05 16:23:16.798000000 one.txt
|
||||
6048320 2016-03-05 16:23:11.775000000 one.txt
|
||||
564374 2016-03-05 16:23:06.731000000 one.txt
|
||||
6048320 2016-03-05 16:18:26.092000000 one.txt
|
||||
6048320 2016-03-05 16:22:46.185000000 two.txt
|
||||
1744073 2016-03-05 16:22:38.104000000 two.txt
|
||||
564374 2016-03-05 16:22:52.118000000 two.txt
|
||||
|
||||
Now the ` + "`" + `dedupe` + "`" + ` session
|
||||
|
||||
$ rclone dedupe drive:dupes
|
||||
2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
|
||||
one.txt: Found 4 duplicates - deleting identical copies
|
||||
one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
|
||||
one.txt: 2 duplicates remain
|
||||
1: 6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
|
||||
2: 564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
|
||||
s) Skip and do nothing
|
||||
k) Keep just one (choose which in next step)
|
||||
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
||||
s/k/r> k
|
||||
Enter the number of the file to keep> 1
|
||||
one.txt: Deleted 1 extra copies
|
||||
two.txt: Found 3 duplicates - deleting identical copies
|
||||
two.txt: 3 duplicates remain
|
||||
1: 564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
|
||||
2: 6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
|
||||
3: 1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
|
||||
s) Skip and do nothing
|
||||
k) Keep just one (choose which in next step)
|
||||
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
||||
s/k/r> r
|
||||
two-1.txt: renamed from: two.txt
|
||||
two-2.txt: renamed from: two.txt
|
||||
two-3.txt: renamed from: two.txt
|
||||
|
||||
The result being
|
||||
|
||||
$ rclone lsl drive:dupes
|
||||
6048320 2016-03-05 16:23:16.798000000 one.txt
|
||||
564374 2016-03-05 16:22:52.118000000 two-1.txt
|
||||
6048320 2016-03-05 16:22:46.185000000 two-2.txt
|
||||
1744073 2016-03-05 16:22:38.104000000 two-3.txt
|
||||
|
||||
Dedupe can be run non interactively using the ` + "`" + `--dedupe-mode` + "`" + ` flag or by using an extra parameter with the same value
|
||||
|
||||
* ` + "`" + `--dedupe-mode interactive` + "`" + ` - interactive as above.
|
||||
* ` + "`" + `--dedupe-mode skip` + "`" + ` - removes identical files then skips anything left.
|
||||
* ` + "`" + `--dedupe-mode first` + "`" + ` - removes identical files then keeps the first one.
|
||||
* ` + "`" + `--dedupe-mode newest` + "`" + ` - removes identical files then keeps the newest one.
|
||||
* ` + "`" + `--dedupe-mode oldest` + "`" + ` - removes identical files then keeps the oldest one.
|
||||
* ` + "`" + `--dedupe-mode rename` + "`" + ` - removes identical files then renames the rest to be different.
|
||||
|
||||
For example to rename all the identically named photos in your Google Photos directory, do
|
||||
|
||||
rclone dedupe --dedupe-mode rename "drive:Google Photos"
|
||||
|
||||
Or
|
||||
|
||||
rclone dedupe rename "drive:Google Photos"
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 2, command, args)
|
||||
if len(args) > 1 {
|
||||
err := dedupeMode.Set(args[0])
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
args = args[1:]
|
||||
}
|
||||
fdst := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.Deduplicate(fdst, dedupeMode)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,41 +0,0 @@
|
||||
package delete
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(deleteCmd)
|
||||
}
|
||||
|
||||
var deleteCmd = &cobra.Command{
|
||||
Use: "delete remote:path",
|
||||
Short: `Remove the contents of path.`,
|
||||
Long: `
|
||||
Remove the contents of path. Unlike ` + "`" + `purge` + "`" + ` it obeys include/exclude
|
||||
filters so can be used to selectively delete files.
|
||||
|
||||
Eg delete all files bigger than 100MBytes
|
||||
|
||||
Check what would be deleted first (use either)
|
||||
|
||||
rclone --min-size 100M lsl remote:path
|
||||
rclone --dry-run --min-size 100M delete remote:path
|
||||
|
||||
Then delete
|
||||
|
||||
rclone --min-size 100M delete remote:path
|
||||
|
||||
That reads "delete everything with a minimum size of 100 MB", hence
|
||||
delete all files bigger than 100MBytes.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.Delete(fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,44 +0,0 @@
|
||||
package genautocomplete
|
||||
|
||||
import (
|
||||
"log"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(genautocompleteCmd)
|
||||
}
|
||||
|
||||
var genautocompleteCmd = &cobra.Command{
|
||||
Use: "genautocomplete [output_file]",
|
||||
Short: `Output bash completion script for rclone.`,
|
||||
Long: `
|
||||
Generates a bash shell autocompletion script for rclone.
|
||||
|
||||
This writes to /etc/bash_completion.d/rclone by default so will
|
||||
probably need to be run with sudo or as root, eg
|
||||
|
||||
sudo rclone genautocomplete
|
||||
|
||||
Logout and login again to use the autocompletion scripts, or source
|
||||
them directly
|
||||
|
||||
. /etc/bash_completion
|
||||
|
||||
If you supply a command line argument the script will be written
|
||||
there.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(0, 1, command, args)
|
||||
out := "/etc/bash_completion.d/rclone"
|
||||
if len(args) > 0 {
|
||||
out = args[0]
|
||||
}
|
||||
err := cmd.Root.GenBashCompletionFile(out)
|
||||
if err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
},
|
||||
}
|
||||
@@ -1,55 +0,0 @@
|
||||
package gendocs
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/spf13/cobra/doc"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(gendocsCmd)
|
||||
}
|
||||
|
||||
const gendocFrontmatterTemplate = `---
|
||||
date: %s
|
||||
title: "%s"
|
||||
slug: %s
|
||||
url: %s
|
||||
---
|
||||
`
|
||||
|
||||
var gendocsCmd = &cobra.Command{
|
||||
Use: "gendocs output_directory",
|
||||
Short: `Output markdown docs for rclone to the directory supplied.`,
|
||||
Long: `
|
||||
This produces markdown docs for the rclone commands to the directory
|
||||
supplied. These are in a format suitable for hugo to render into the
|
||||
rclone.org website.`,
|
||||
RunE: func(command *cobra.Command, args []string) error {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
out := args[0]
|
||||
err := os.MkdirAll(out, 0777)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
now := time.Now().Format(time.RFC3339)
|
||||
prepender := func(filename string) string {
|
||||
name := filepath.Base(filename)
|
||||
base := strings.TrimSuffix(name, path.Ext(name))
|
||||
url := "/commands/" + strings.ToLower(base) + "/"
|
||||
return fmt.Sprintf(gendocFrontmatterTemplate, now, strings.Replace(base, "_", " ", -1), base, url)
|
||||
}
|
||||
linkHandler := func(name string) string {
|
||||
base := strings.TrimSuffix(name, path.Ext(name))
|
||||
return "/commands/" + strings.ToLower(base) + "/"
|
||||
}
|
||||
return doc.GenMarkdownTreeCustom(cmd.Root, out, prepender, linkHandler)
|
||||
},
|
||||
}
|
||||
25
cmd/ls/ls.go
25
cmd/ls/ls.go
@@ -1,25 +0,0 @@
|
||||
package ls
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(lsCmd)
|
||||
}
|
||||
|
||||
var lsCmd = &cobra.Command{
|
||||
Use: "ls remote:path",
|
||||
Short: `List all the objects in the the path with size and path.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.List(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,25 +0,0 @@
|
||||
package lsd
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(lsdCmd)
|
||||
}
|
||||
|
||||
var lsdCmd = &cobra.Command{
|
||||
Use: "lsd remote:path",
|
||||
Short: `List all directories/containers/buckets in the the path.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.ListDir(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,25 +0,0 @@
|
||||
package lsl
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(lslCmd)
|
||||
}
|
||||
|
||||
var lslCmd = &cobra.Command{
|
||||
Use: "lsl remote:path",
|
||||
Short: `List all the objects path with modification time, size and path.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.ListLong(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
package md5sum
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(md5sumCmd)
|
||||
}
|
||||
|
||||
var md5sumCmd = &cobra.Command{
|
||||
Use: "md5sum remote:path",
|
||||
Short: `Produces an md5sum file for all the objects in the path.`,
|
||||
Long: `
|
||||
Produces an md5sum file for all the objects in the path. This
|
||||
is in the same format as the standard md5sum tool produces.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.Md5sum(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,49 +0,0 @@
|
||||
package memtest
|
||||
|
||||
import (
|
||||
"runtime"
|
||||
"sync"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(memtestCmd)
|
||||
}
|
||||
|
||||
var memtestCmd = &cobra.Command{
|
||||
Use: "memtest remote:path",
|
||||
Short: `Load all the objects at remote:path and report memory stats.`,
|
||||
Hidden: true,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
objects, _, err := fs.Count(fsrc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
objs := make([]fs.Object, 0, objects)
|
||||
var before, after runtime.MemStats
|
||||
runtime.GC()
|
||||
runtime.ReadMemStats(&before)
|
||||
var mu sync.Mutex
|
||||
err = fs.ListFn(fsrc, func(o fs.Object) {
|
||||
mu.Lock()
|
||||
objs = append(objs, o)
|
||||
mu.Unlock()
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
runtime.GC()
|
||||
runtime.ReadMemStats(&after)
|
||||
usedMemory := after.Alloc - before.Alloc
|
||||
fs.Log(nil, "%d objects took %d bytes, %.1f bytes/object", len(objs), usedMemory, float64(usedMemory)/float64(len(objs)))
|
||||
fs.Log(nil, "System memory changed from %d to %d bytes a change of %d bytes", before.Sys, after.Sys, after.Sys-before.Sys)
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,23 +0,0 @@
|
||||
package mkdir
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(mkdirCmd)
|
||||
}
|
||||
|
||||
var mkdirCmd = &cobra.Command{
|
||||
Use: "mkdir remote:path",
|
||||
Short: `Make the path if it doesn't already exist.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fdst := cmd.NewFsDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.Mkdir(fdst)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,57 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// info to create a new object
|
||||
type createInfo struct {
|
||||
f fs.Fs
|
||||
remote string
|
||||
}
|
||||
|
||||
func newCreateInfo(f fs.Fs, remote string) *createInfo {
|
||||
return &createInfo{
|
||||
f: f,
|
||||
remote: remote,
|
||||
}
|
||||
}
|
||||
|
||||
// Fs returns read only access to the Fs that this object is part of
|
||||
func (ci *createInfo) Fs() fs.Info {
|
||||
return ci.f
|
||||
}
|
||||
|
||||
// Remote returns the remote path
|
||||
func (ci *createInfo) Remote() string {
|
||||
return ci.remote
|
||||
}
|
||||
|
||||
// Hash returns the selected checksum of the file
|
||||
// If no checksum is available it returns ""
|
||||
func (ci *createInfo) Hash(fs.HashType) (string, error) {
|
||||
return "", fs.ErrHashUnsupported
|
||||
}
|
||||
|
||||
// ModTime returns the modification date of the file
|
||||
// It should return a best guess if one isn't available
|
||||
func (ci *createInfo) ModTime() time.Time {
|
||||
return time.Now()
|
||||
}
|
||||
|
||||
// Size returns the size of the file
|
||||
func (ci *createInfo) Size() int64 {
|
||||
// FIXME this means this won't work with all remotes...
|
||||
return 0
|
||||
}
|
||||
|
||||
// Storable says whether this object can be stored
|
||||
func (ci *createInfo) Storable() bool {
|
||||
return true
|
||||
}
|
||||
|
||||
var _ fs.ObjectInfo = (*createInfo)(nil)
|
||||
377
cmd/mount/dir.go
377
cmd/mount/dir.go
@@ -1,377 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"bazil.org/fuse"
|
||||
fusefs "bazil.org/fuse/fs"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/pkg/errors"
|
||||
"golang.org/x/net/context"
|
||||
)
|
||||
|
||||
// DirEntry describes the contents of a directory entry
|
||||
//
|
||||
// It can be a file or a directory
|
||||
//
|
||||
// node may be nil, but o may not
|
||||
type DirEntry struct {
|
||||
o fs.BasicInfo
|
||||
node fusefs.Node
|
||||
}
|
||||
|
||||
// Dir represents a directory entry
|
||||
type Dir struct {
|
||||
f fs.Fs
|
||||
path string
|
||||
mu sync.RWMutex // protects the following
|
||||
read bool
|
||||
items map[string]*DirEntry
|
||||
}
|
||||
|
||||
func newDir(f fs.Fs, path string) *Dir {
|
||||
return &Dir{
|
||||
f: f,
|
||||
path: path,
|
||||
}
|
||||
}
|
||||
|
||||
// addObject adds a new object or directory to the directory
|
||||
//
|
||||
// note that we add new objects rather than updating old ones
|
||||
func (d *Dir) addObject(o fs.BasicInfo, node fusefs.Node) *DirEntry {
|
||||
item := &DirEntry{
|
||||
o: o,
|
||||
node: node,
|
||||
}
|
||||
d.mu.Lock()
|
||||
d.items[path.Base(o.Remote())] = item
|
||||
d.mu.Unlock()
|
||||
return item
|
||||
}
|
||||
|
||||
// delObject removes an object from the directory
|
||||
func (d *Dir) delObject(leaf string) {
|
||||
d.mu.Lock()
|
||||
delete(d.items, leaf)
|
||||
d.mu.Unlock()
|
||||
}
|
||||
|
||||
// read the directory
|
||||
func (d *Dir) readDir() error {
|
||||
d.mu.Lock()
|
||||
defer d.mu.Unlock()
|
||||
if d.read {
|
||||
return nil
|
||||
}
|
||||
objs, dirs, err := fs.NewLister().SetLevel(1).Start(d.f, d.path).GetAll()
|
||||
if err == fs.ErrorDirNotFound {
|
||||
// We treat directory not found as empty because we
|
||||
// create directories on the fly
|
||||
} else if err != nil {
|
||||
return err
|
||||
}
|
||||
// Cache the items by name
|
||||
d.items = make(map[string]*DirEntry, len(objs)+len(dirs))
|
||||
for _, obj := range objs {
|
||||
name := path.Base(obj.Remote())
|
||||
d.items[name] = &DirEntry{
|
||||
o: obj,
|
||||
node: nil,
|
||||
}
|
||||
}
|
||||
for _, dir := range dirs {
|
||||
name := path.Base(dir.Remote())
|
||||
d.items[name] = &DirEntry{
|
||||
o: dir,
|
||||
node: nil,
|
||||
}
|
||||
}
|
||||
d.read = true
|
||||
return nil
|
||||
}
|
||||
|
||||
// lookup a single item in the directory
|
||||
//
|
||||
// returns fuse.ENOENT if not found.
|
||||
func (d *Dir) lookup(leaf string) (*DirEntry, error) {
|
||||
err := d.readDir()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
d.mu.RLock()
|
||||
item, ok := d.items[leaf]
|
||||
d.mu.RUnlock()
|
||||
if !ok {
|
||||
return nil, fuse.ENOENT
|
||||
}
|
||||
return item, nil
|
||||
}
|
||||
|
||||
// Check to see if a directory is empty
|
||||
func (d *Dir) isEmpty() (bool, error) {
|
||||
err := d.readDir()
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
d.mu.RLock()
|
||||
defer d.mu.RUnlock()
|
||||
return len(d.items) == 0, nil
|
||||
}
|
||||
|
||||
// Check interface satsified
|
||||
var _ fusefs.Node = (*Dir)(nil)
|
||||
|
||||
// Attr updates the attribes of a directory
|
||||
func (d *Dir) Attr(ctx context.Context, a *fuse.Attr) error {
|
||||
fs.Debug(d.path, "Dir.Attr")
|
||||
a.Mode = os.ModeDir | dirPerms
|
||||
// FIXME include Valid so get some caching? Also mtime
|
||||
return nil
|
||||
}
|
||||
|
||||
// lookupNode calls lookup then makes sure the node is not nil in the DirEntry
|
||||
func (d *Dir) lookupNode(leaf string) (item *DirEntry, err error) {
|
||||
item, err = d.lookup(leaf)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if item.node != nil {
|
||||
return item, nil
|
||||
}
|
||||
var node fusefs.Node
|
||||
switch x := item.o.(type) {
|
||||
case fs.Object:
|
||||
node, err = newFile(d, x), nil
|
||||
case *fs.Dir:
|
||||
node, err = newDir(d.f, x.Remote()), nil
|
||||
default:
|
||||
err = errors.Errorf("unknown type %T", item)
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
item = d.addObject(item.o, node)
|
||||
return item, err
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.NodeRequestLookuper = (*Dir)(nil)
|
||||
|
||||
// Lookup looks up a specific entry in the receiver.
|
||||
//
|
||||
// Lookup should return a Node corresponding to the entry. If the
|
||||
// name does not exist in the directory, Lookup should return ENOENT.
|
||||
//
|
||||
// Lookup need not to handle the names "." and "..".
|
||||
func (d *Dir) Lookup(ctx context.Context, req *fuse.LookupRequest, resp *fuse.LookupResponse) (node fusefs.Node, err error) {
|
||||
path := path.Join(d.path, req.Name)
|
||||
fs.Debug(path, "Dir.Lookup")
|
||||
item, err := d.lookupNode(req.Name)
|
||||
if err != nil {
|
||||
if err != fuse.ENOENT {
|
||||
fs.ErrorLog(path, "Dir.Lookup error: %v", err)
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
fs.Debug(path, "Dir.Lookup OK")
|
||||
return item.node, nil
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.HandleReadDirAller = (*Dir)(nil)
|
||||
|
||||
// ReadDirAll reads the contents of the directory
|
||||
func (d *Dir) ReadDirAll(ctx context.Context) (dirents []fuse.Dirent, err error) {
|
||||
fs.Debug(d.path, "Dir.ReadDirAll")
|
||||
err = d.readDir()
|
||||
if err != nil {
|
||||
fs.Debug(d.path, "Dir.ReadDirAll error: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
d.mu.RLock()
|
||||
defer d.mu.RUnlock()
|
||||
for _, item := range d.items {
|
||||
var dirent fuse.Dirent
|
||||
switch x := item.o.(type) {
|
||||
case fs.Object:
|
||||
dirent = fuse.Dirent{
|
||||
// Inode FIXME ???
|
||||
Type: fuse.DT_File,
|
||||
Name: path.Base(x.Remote()),
|
||||
}
|
||||
case *fs.Dir:
|
||||
dirent = fuse.Dirent{
|
||||
// Inode FIXME ???
|
||||
Type: fuse.DT_Dir,
|
||||
Name: path.Base(x.Remote()),
|
||||
}
|
||||
default:
|
||||
err = errors.Errorf("unknown type %T", item)
|
||||
fs.ErrorLog(d.path, "Dir.ReadDirAll error: %v", err)
|
||||
return nil, err
|
||||
}
|
||||
dirents = append(dirents, dirent)
|
||||
}
|
||||
fs.Debug(d.path, "Dir.ReadDirAll OK with %d entries", len(dirents))
|
||||
return dirents, nil
|
||||
}
|
||||
|
||||
var _ fusefs.NodeCreater = (*Dir)(nil)
|
||||
|
||||
// Create makes a new file
|
||||
func (d *Dir) Create(ctx context.Context, req *fuse.CreateRequest, resp *fuse.CreateResponse) (fusefs.Node, fusefs.Handle, error) {
|
||||
path := path.Join(d.path, req.Name)
|
||||
fs.Debug(path, "Dir.Create")
|
||||
src := newCreateInfo(d.f, path)
|
||||
// This gets added to the directory when the file is written
|
||||
file := newFile(d, nil)
|
||||
fh, err := newWriteFileHandle(d, file, src)
|
||||
if err != nil {
|
||||
fs.ErrorLog(path, "Dir.Create error: %v", err)
|
||||
return nil, nil, err
|
||||
}
|
||||
fs.Debug(path, "Dir.Create OK")
|
||||
return file, fh, nil
|
||||
}
|
||||
|
||||
var _ fusefs.NodeMkdirer = (*Dir)(nil)
|
||||
|
||||
// Mkdir creates a new directory
|
||||
func (d *Dir) Mkdir(ctx context.Context, req *fuse.MkdirRequest) (fusefs.Node, error) {
|
||||
// We just pretend to have created the directory - rclone will
|
||||
// actually create the directory if we write files into it
|
||||
path := path.Join(d.path, req.Name)
|
||||
fs.Debug(path, "Dir.Mkdir")
|
||||
fsDir := &fs.Dir{
|
||||
Name: path,
|
||||
When: time.Now(),
|
||||
}
|
||||
dir := newDir(d.f, path)
|
||||
d.addObject(fsDir, dir)
|
||||
fs.Debug(path, "Dir.Mkdir OK")
|
||||
return dir, nil
|
||||
}
|
||||
|
||||
var _ fusefs.NodeRemover = (*Dir)(nil)
|
||||
|
||||
// Remove removes the entry with the given name from
|
||||
// the receiver, which must be a directory. The entry to be removed
|
||||
// may correspond to a file (unlink) or to a directory (rmdir).
|
||||
func (d *Dir) Remove(ctx context.Context, req *fuse.RemoveRequest) error {
|
||||
path := path.Join(d.path, req.Name)
|
||||
fs.Debug(path, "Dir.Remove")
|
||||
item, err := d.lookupNode(req.Name)
|
||||
if err != nil {
|
||||
fs.ErrorLog(path, "Dir.Remove error: %v", err)
|
||||
return err
|
||||
}
|
||||
switch x := item.o.(type) {
|
||||
case fs.Object:
|
||||
err = x.Remove()
|
||||
if err != nil {
|
||||
fs.ErrorLog(path, "Dir.Remove file error: %v", err)
|
||||
return err
|
||||
}
|
||||
case *fs.Dir:
|
||||
// Do nothing for deleting directory - rclone can't
|
||||
// currently remote a random directory
|
||||
//
|
||||
// Check directory is empty first though
|
||||
dir := item.node.(*Dir)
|
||||
empty, err := dir.isEmpty()
|
||||
if err != nil {
|
||||
fs.ErrorLog(path, "Dir.Remove dir error: %v", err)
|
||||
return err
|
||||
}
|
||||
if !empty {
|
||||
// return fuse.ENOTEMPTY - doesn't exist though so use EEXIST
|
||||
fs.ErrorLog(path, "Dir.Remove not empty")
|
||||
return fuse.EEXIST
|
||||
}
|
||||
default:
|
||||
fs.ErrorLog(path, "Dir.Remove unknown type %T", item)
|
||||
return errors.Errorf("unknown type %T", item)
|
||||
}
|
||||
// Remove the item from the directory listing
|
||||
d.delObject(req.Name)
|
||||
fs.Debug(path, "Dir.Remove OK")
|
||||
return nil
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.NodeRenamer = (*Dir)(nil)
|
||||
|
||||
// Rename the file
|
||||
func (d *Dir) Rename(ctx context.Context, req *fuse.RenameRequest, newDir fusefs.Node) error {
|
||||
oldPath := path.Join(d.path, req.OldName)
|
||||
destDir, ok := newDir.(*Dir)
|
||||
if !ok {
|
||||
err := errors.Errorf("Unknown Dir type %T", newDir)
|
||||
fs.ErrorLog(oldPath, "Dir.Rename error: %v", err)
|
||||
return err
|
||||
}
|
||||
newPath := path.Join(destDir.path, req.NewName)
|
||||
fs.Debug(oldPath, "Dir.Rename to %q", newPath)
|
||||
oldItem, err := d.lookupNode(req.OldName)
|
||||
if err != nil {
|
||||
fs.ErrorLog(oldPath, "Dir.Rename error: %v", err)
|
||||
return err
|
||||
}
|
||||
var newObj fs.BasicInfo
|
||||
switch x := oldItem.o.(type) {
|
||||
case fs.Object:
|
||||
oldObject := x
|
||||
do, ok := d.f.(fs.Mover)
|
||||
if !ok {
|
||||
err := errors.Errorf("Fs %q can't Move files", d.f)
|
||||
fs.ErrorLog(oldPath, "Dir.Rename error: %v", err)
|
||||
return err
|
||||
}
|
||||
newObject, err := do.Move(oldObject, newPath)
|
||||
if err != nil {
|
||||
fs.ErrorLog(oldPath, "Dir.Rename error: %v", err)
|
||||
return err
|
||||
}
|
||||
newObj = newObject
|
||||
case *fs.Dir:
|
||||
oldDir := oldItem.node.(*Dir)
|
||||
empty, err := oldDir.isEmpty()
|
||||
if err != nil {
|
||||
fs.ErrorLog(oldPath, "Dir.Rename dir error: %v", err)
|
||||
return err
|
||||
}
|
||||
if !empty {
|
||||
// return fuse.ENOTEMPTY - doesn't exist though so use EEXIST
|
||||
fs.ErrorLog(oldPath, "Dir.Rename can't rename non empty directory")
|
||||
return fuse.EEXIST
|
||||
}
|
||||
newObj = &fs.Dir{
|
||||
Name: newPath,
|
||||
When: time.Now(),
|
||||
}
|
||||
default:
|
||||
err = errors.Errorf("unknown type %T", oldItem)
|
||||
fs.ErrorLog(d.path, "Dir.ReadDirAll error: %v", err)
|
||||
return err
|
||||
}
|
||||
|
||||
// Show moved - delete from old dir and add to new
|
||||
d.delObject(req.OldName)
|
||||
destDir.addObject(newObj, nil)
|
||||
|
||||
// FIXME need to flush the dir also
|
||||
|
||||
// FIXME use DirMover to move a directory?
|
||||
// or maybe use MoveDir which can move anything
|
||||
// fallback to Copy/Delete if no Move?
|
||||
// if dir is empty then can move it
|
||||
|
||||
fs.ErrorLog(newPath, "Dir.Rename renamed from %q", oldPath)
|
||||
return nil
|
||||
}
|
||||
@@ -1,133 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"os"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestDirLs(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.checkDir(t, "")
|
||||
|
||||
run.mkdir(t, "a directory")
|
||||
run.createFile(t, "a file", "hello")
|
||||
|
||||
run.checkDir(t, "a directory/|a file 5")
|
||||
|
||||
run.rmdir(t, "a directory")
|
||||
run.rm(t, "a file")
|
||||
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
|
||||
func TestDirCreateAndRemoveDir(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.mkdir(t, "dir")
|
||||
run.mkdir(t, "dir/subdir")
|
||||
run.checkDir(t, "dir/|dir/subdir/")
|
||||
|
||||
// Check we can't delete a directory with stuff in
|
||||
err := os.Remove(run.path("dir"))
|
||||
assert.Error(t, err, "file exists")
|
||||
|
||||
// Now delete subdir then dir - should produce no errors
|
||||
run.rmdir(t, "dir/subdir")
|
||||
run.checkDir(t, "dir/")
|
||||
run.rmdir(t, "dir")
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
|
||||
func TestDirCreateAndRemoveFile(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.mkdir(t, "dir")
|
||||
run.createFile(t, "dir/file", "potato")
|
||||
run.checkDir(t, "dir/|dir/file 6")
|
||||
|
||||
// Check we can't delete a directory with stuff in
|
||||
err := os.Remove(run.path("dir"))
|
||||
assert.Error(t, err, "file exists")
|
||||
|
||||
// Now delete file
|
||||
run.rm(t, "dir/file")
|
||||
|
||||
run.checkDir(t, "dir/")
|
||||
run.rmdir(t, "dir")
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
|
||||
func TestDirRenameFile(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.mkdir(t, "dir")
|
||||
run.createFile(t, "file", "potato")
|
||||
run.checkDir(t, "dir/|file 6")
|
||||
|
||||
err := os.Rename(run.path("file"), run.path("dir/file2"))
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/file2 6")
|
||||
|
||||
err = os.Rename(run.path("dir/file2"), run.path("dir/file3"))
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/file3 6")
|
||||
|
||||
run.rm(t, "dir/file3")
|
||||
run.rmdir(t, "dir")
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
|
||||
func TestDirRenameEmptyDir(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.mkdir(t, "dir")
|
||||
run.mkdir(t, "dir1")
|
||||
run.checkDir(t, "dir/|dir1/")
|
||||
|
||||
err := os.Rename(run.path("dir1"), run.path("dir/dir2"))
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/dir2/")
|
||||
|
||||
err = os.Rename(run.path("dir/dir2"), run.path("dir/dir3"))
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/dir3/")
|
||||
|
||||
run.rmdir(t, "dir/dir3")
|
||||
run.rmdir(t, "dir")
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
|
||||
func TestDirRenameFullDir(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.mkdir(t, "dir")
|
||||
run.mkdir(t, "dir1")
|
||||
run.createFile(t, "dir1/potato.txt", "maris piper")
|
||||
run.checkDir(t, "dir/|dir1/|dir1/potato.txt 11")
|
||||
|
||||
err := os.Rename(run.path("dir1"), run.path("dir/dir2"))
|
||||
require.Error(t, err, "file exists")
|
||||
// Can't currently rename directories with stuff in
|
||||
/*
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/dir2/|dir/dir2/potato.txt 11")
|
||||
|
||||
err = os.Rename(run.path("dir/dir2"), run.path("dir/dir3"))
|
||||
require.NoError(t, err)
|
||||
run.checkDir(t, "dir/|dir/dir3/|dir/dir3/potato.txt 11")
|
||||
|
||||
run.rm(t, "dir/dir3/potato.txt")
|
||||
run.rmdir(t, "dir/dir3")
|
||||
*/
|
||||
|
||||
run.rm(t, "dir1/potato.txt")
|
||||
run.rmdir(t, "dir1")
|
||||
run.rmdir(t, "dir")
|
||||
run.checkDir(t, "")
|
||||
}
|
||||
@@ -1,142 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"bazil.org/fuse"
|
||||
fusefs "bazil.org/fuse/fs"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/pkg/errors"
|
||||
"golang.org/x/net/context"
|
||||
)
|
||||
|
||||
// File represents a file
|
||||
type File struct {
|
||||
d *Dir // parent directory - read only
|
||||
size int64 // size of file - read and written with atomic
|
||||
mu sync.RWMutex // protects the following
|
||||
o fs.Object // NB o may be nil if file is being written
|
||||
writers int // number of writers for this file
|
||||
}
|
||||
|
||||
// newFile creates a new File
|
||||
func newFile(d *Dir, o fs.Object) *File {
|
||||
return &File{
|
||||
d: d,
|
||||
o: o,
|
||||
}
|
||||
}
|
||||
|
||||
// addWriters increments or decrements the writers
|
||||
func (f *File) addWriters(n int) {
|
||||
f.mu.Lock()
|
||||
f.writers += n
|
||||
f.mu.Unlock()
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.Node = (*File)(nil)
|
||||
|
||||
// Attr fills out the attributes for the file
|
||||
func (f *File) Attr(ctx context.Context, a *fuse.Attr) error {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
fs.Debug(f.o, "File.Attr")
|
||||
a.Mode = filePerms
|
||||
// if o is nil it isn't valid yet, so return the size so far
|
||||
if f.o == nil {
|
||||
a.Size = uint64(atomic.LoadInt64(&f.size))
|
||||
} else {
|
||||
a.Size = uint64(f.o.Size())
|
||||
if !noModTime {
|
||||
modTime := f.o.ModTime()
|
||||
a.Atime = modTime
|
||||
a.Mtime = modTime
|
||||
a.Ctime = modTime
|
||||
a.Crtime = modTime
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// Update the size while writing
|
||||
func (f *File) written(n int64) {
|
||||
atomic.AddInt64(&f.size, n)
|
||||
}
|
||||
|
||||
// Update the object when written
|
||||
func (f *File) setObject(o fs.Object) {
|
||||
f.mu.Lock()
|
||||
defer f.mu.Unlock()
|
||||
f.o = o
|
||||
f.d.addObject(o, f)
|
||||
}
|
||||
|
||||
// Wait for f.o to become non nil for a short time returning it or an
|
||||
// error
|
||||
//
|
||||
// Call without the mutex held
|
||||
func (f *File) waitForValidObject() (o fs.Object, err error) {
|
||||
for i := 0; i < 50; i++ {
|
||||
f.mu.Lock()
|
||||
o = f.o
|
||||
writers := f.writers
|
||||
f.mu.Unlock()
|
||||
if o != nil {
|
||||
return o, nil
|
||||
}
|
||||
if writers == 0 {
|
||||
return nil, errors.New("can't open file - writer failed")
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
return nil, fuse.ENOENT
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.NodeOpener = (*File)(nil)
|
||||
|
||||
// Open the file for read or write
|
||||
func (f *File) Open(ctx context.Context, req *fuse.OpenRequest, resp *fuse.OpenResponse) (fusefs.Handle, error) {
|
||||
// if o is nil it isn't valid yet
|
||||
o, err := f.waitForValidObject()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
fs.Debug(o, "File.Open")
|
||||
|
||||
// Files aren't seekable
|
||||
resp.Flags |= fuse.OpenNonSeekable
|
||||
|
||||
switch {
|
||||
case req.Flags.IsReadOnly():
|
||||
return newReadFileHandle(o)
|
||||
case req.Flags.IsWriteOnly():
|
||||
src := newCreateInfo(f.d.f, o.Remote())
|
||||
fh, err := newWriteFileHandle(f.d, f, src)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return fh, nil
|
||||
case req.Flags.IsReadWrite():
|
||||
return nil, errors.New("can't open read and write")
|
||||
}
|
||||
|
||||
/*
|
||||
// File was opened in append-only mode, all writes will go to end
|
||||
// of file. OS X does not provide this information.
|
||||
OpenAppend OpenFlags = syscall.O_APPEND
|
||||
OpenCreate OpenFlags = syscall.O_CREAT
|
||||
OpenDirectory OpenFlags = syscall.O_DIRECTORY
|
||||
OpenExclusive OpenFlags = syscall.O_EXCL
|
||||
OpenNonblock OpenFlags = syscall.O_NONBLOCK
|
||||
OpenSync OpenFlags = syscall.O_SYNC
|
||||
OpenTruncate OpenFlags = syscall.O_TRUNC
|
||||
*/
|
||||
return nil, errors.New("can't figure out how to open")
|
||||
}
|
||||
@@ -1,67 +0,0 @@
|
||||
// FUSE main Fs
|
||||
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"bazil.org/fuse"
|
||||
fusefs "bazil.org/fuse/fs"
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// Default permissions
|
||||
const (
|
||||
dirPerms = 0755
|
||||
filePerms = 0644
|
||||
)
|
||||
|
||||
// FS represents the top level filing system
|
||||
type FS struct {
|
||||
f fs.Fs
|
||||
}
|
||||
|
||||
// Check interface satistfied
|
||||
var _ fusefs.FS = (*FS)(nil)
|
||||
|
||||
// Root returns the root node
|
||||
func (f *FS) Root() (fusefs.Node, error) {
|
||||
fs.Debug(f.f, "Root()")
|
||||
return newDir(f.f, ""), nil
|
||||
}
|
||||
|
||||
// mount the file system
|
||||
//
|
||||
// The mount point will be ready when this returns.
|
||||
//
|
||||
// returns an error, and an error channel for the serve process to
|
||||
// report an error when fusermount is called.
|
||||
func mount(f fs.Fs, mountpoint string) (<-chan error, error) {
|
||||
c, err := fuse.Mount(mountpoint)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
filesys := &FS{
|
||||
f: f,
|
||||
}
|
||||
|
||||
// Serve the mount point in the background returning error to errChan
|
||||
errChan := make(chan error, 1)
|
||||
go func() {
|
||||
err := fusefs.Serve(c, filesys)
|
||||
closeErr := c.Close()
|
||||
if err == nil {
|
||||
err = closeErr
|
||||
}
|
||||
errChan <- err
|
||||
}()
|
||||
|
||||
// check if the mount process has an error to report
|
||||
<-c.Ready
|
||||
if err := c.MountError; err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return errChan, nil
|
||||
}
|
||||
@@ -1,264 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
// Test suite for rclonefs
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"flag"
|
||||
"fmt"
|
||||
"io/ioutil"
|
||||
"log"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
_ "github.com/ncw/rclone/fs/all"
|
||||
"github.com/ncw/rclone/fstest"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
// Globals
|
||||
var (
|
||||
RemoteName = flag.String("remote", "", "Remote to test with, defaults to local filesystem")
|
||||
SubDir = flag.Bool("subdir", false, "Set to test with a sub directory")
|
||||
Verbose = flag.Bool("verbose", false, "Set to enable logging")
|
||||
DumpHeaders = flag.Bool("dump-headers", false, "Set to dump headers (needs -verbose)")
|
||||
DumpBodies = flag.Bool("dump-bodies", false, "Set to dump bodies (needs -verbose)")
|
||||
Individual = flag.Bool("individual", false, "Make individual bucket/container/directory for each test - much slower")
|
||||
LowLevelRetries = flag.Int("low-level-retries", 10, "Number of low level retries")
|
||||
)
|
||||
|
||||
// TestMain drives the tests
|
||||
func TestMain(m *testing.M) {
|
||||
flag.Parse()
|
||||
run = newRun()
|
||||
rc := m.Run()
|
||||
run.Finalise()
|
||||
os.Exit(rc)
|
||||
}
|
||||
|
||||
// Run holds the remotes for a test run
|
||||
type Run struct {
|
||||
mountPath string
|
||||
fremote fs.Fs
|
||||
fremoteName string
|
||||
cleanRemote func()
|
||||
umountResult <-chan error
|
||||
skip bool
|
||||
}
|
||||
|
||||
// run holds the master Run data
|
||||
var run *Run
|
||||
|
||||
// newRun initialise the remote mount for testing and returns a run
|
||||
// object.
|
||||
//
|
||||
// r.fremote is an empty remote Fs
|
||||
//
|
||||
// Finalise() will tidy them away when done.
|
||||
func newRun() *Run {
|
||||
r := &Run{
|
||||
umountResult: make(chan error, 1),
|
||||
}
|
||||
|
||||
// Never ask for passwords, fail instead.
|
||||
// If your local config is encrypted set environment variable
|
||||
// "RCLONE_CONFIG_PASS=hunter2" (or your password)
|
||||
*fs.AskPassword = false
|
||||
fs.LoadConfig()
|
||||
fs.Config.Verbose = *Verbose
|
||||
fs.Config.Quiet = !*Verbose
|
||||
fs.Config.DumpHeaders = *DumpHeaders
|
||||
fs.Config.DumpBodies = *DumpBodies
|
||||
fs.Config.LowLevelRetries = *LowLevelRetries
|
||||
var err error
|
||||
r.fremote, r.fremoteName, r.cleanRemote, err = fstest.RandomRemote(*RemoteName, *SubDir)
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to open remote %q: %v", *RemoteName, err)
|
||||
}
|
||||
|
||||
r.mountPath, err = ioutil.TempDir("", "rclonefs-mount")
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to create mount dir: %v", err)
|
||||
}
|
||||
|
||||
// Mount it up
|
||||
r.mount()
|
||||
|
||||
return r
|
||||
}
|
||||
|
||||
func (r *Run) mount() {
|
||||
log.Printf("mount %q %q", r.fremote, r.mountPath)
|
||||
var err error
|
||||
r.umountResult, err = mount(r.fremote, r.mountPath)
|
||||
if err != nil {
|
||||
log.Printf("mount failed: %v", err)
|
||||
r.skip = true
|
||||
}
|
||||
log.Printf("mount OK")
|
||||
}
|
||||
|
||||
func (r *Run) umount() {
|
||||
if r.skip {
|
||||
log.Printf("FUSE not found so skipping umount")
|
||||
return
|
||||
}
|
||||
log.Printf("Calling fusermount -u %q", r.mountPath)
|
||||
err := exec.Command("fusermount", "-u", r.mountPath).Run()
|
||||
if err != nil {
|
||||
log.Printf("fusermount failed: %v", err)
|
||||
}
|
||||
log.Printf("Waiting for umount")
|
||||
err = <-r.umountResult
|
||||
if err != nil {
|
||||
log.Fatalf("umount failed: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Run) skipIfNoFUSE(t *testing.T) {
|
||||
if r.skip {
|
||||
t.Skip("FUSE not found so skipping test")
|
||||
}
|
||||
}
|
||||
|
||||
// Finalise cleans the remote and unmounts
|
||||
func (r *Run) Finalise() {
|
||||
r.umount()
|
||||
r.cleanRemote()
|
||||
err := os.RemoveAll(r.mountPath)
|
||||
if err != nil {
|
||||
log.Printf("Failed to clean mountPath %q: %v", r.mountPath, err)
|
||||
}
|
||||
}
|
||||
|
||||
func (r *Run) path(filepath string) string {
|
||||
return path.Join(run.mountPath, filepath)
|
||||
}
|
||||
|
||||
type dirMap map[string]struct{}
|
||||
|
||||
// Create a dirMap from a string
|
||||
func newDirMap(dirString string) (dm dirMap) {
|
||||
dm = make(dirMap)
|
||||
for _, entry := range strings.Split(dirString, "|") {
|
||||
if entry != "" {
|
||||
dm[entry] = struct{}{}
|
||||
}
|
||||
}
|
||||
return dm
|
||||
}
|
||||
|
||||
// Returns a dirmap with only the files in
|
||||
func (dm dirMap) filesOnly() dirMap {
|
||||
newDm := make(dirMap)
|
||||
for name := range dm {
|
||||
if !strings.HasSuffix(name, "/") {
|
||||
newDm[name] = struct{}{}
|
||||
}
|
||||
}
|
||||
return newDm
|
||||
}
|
||||
|
||||
// reads the local tree into dir
|
||||
func (r *Run) readLocal(t *testing.T, dir dirMap, filepath string) {
|
||||
realPath := r.path(filepath)
|
||||
files, err := ioutil.ReadDir(realPath)
|
||||
require.NoError(t, err)
|
||||
for _, fi := range files {
|
||||
name := path.Join(filepath, fi.Name())
|
||||
if fi.IsDir() {
|
||||
dir[name+"/"] = struct{}{}
|
||||
r.readLocal(t, dir, name)
|
||||
assert.Equal(t, fi.Mode().Perm(), os.FileMode(dirPerms))
|
||||
} else {
|
||||
dir[fmt.Sprintf("%s %d", name, fi.Size())] = struct{}{}
|
||||
assert.Equal(t, fi.Mode().Perm(), os.FileMode(filePerms))
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reads the remote tree into dir
|
||||
func (r *Run) readRemote(t *testing.T, dir dirMap, filepath string) {
|
||||
objs, dirs, err := fs.NewLister().SetLevel(1).Start(r.fremote, filepath).GetAll()
|
||||
if err == fs.ErrorDirNotFound {
|
||||
return
|
||||
}
|
||||
require.NoError(t, err)
|
||||
for _, obj := range objs {
|
||||
dir[fmt.Sprintf("%s %d", obj.Remote(), obj.Size())] = struct{}{}
|
||||
}
|
||||
for _, d := range dirs {
|
||||
name := d.Remote()
|
||||
dir[name+"/"] = struct{}{}
|
||||
r.readRemote(t, dir, name)
|
||||
}
|
||||
}
|
||||
|
||||
// checkDir checks the local and remote against the string passed in
|
||||
func (r *Run) checkDir(t *testing.T, dirString string) {
|
||||
dm := newDirMap(dirString)
|
||||
localDm := make(dirMap)
|
||||
r.readLocal(t, localDm, "")
|
||||
remoteDm := make(dirMap)
|
||||
r.readRemote(t, remoteDm, "")
|
||||
// Ignore directories for remote compare
|
||||
assert.Equal(t, dm.filesOnly(), remoteDm.filesOnly(), "expected vs remote")
|
||||
assert.Equal(t, dm, localDm, "expected vs fuse mount")
|
||||
}
|
||||
|
||||
func (r *Run) createFile(t *testing.T, filepath string, contents string) {
|
||||
filepath = r.path(filepath)
|
||||
err := ioutil.WriteFile(filepath, []byte(contents), 0600)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (r *Run) readFile(t *testing.T, filepath string) string {
|
||||
filepath = r.path(filepath)
|
||||
result, err := ioutil.ReadFile(filepath)
|
||||
require.NoError(t, err)
|
||||
return string(result)
|
||||
}
|
||||
|
||||
func (r *Run) mkdir(t *testing.T, filepath string) {
|
||||
filepath = r.path(filepath)
|
||||
err := os.Mkdir(filepath, 0700)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (r *Run) rm(t *testing.T, filepath string) {
|
||||
filepath = r.path(filepath)
|
||||
err := os.Remove(filepath)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
func (r *Run) rmdir(t *testing.T, filepath string) {
|
||||
filepath = r.path(filepath)
|
||||
err := os.Remove(filepath)
|
||||
require.NoError(t, err)
|
||||
}
|
||||
|
||||
// Check that the Fs is mounted by seeing if the mountpoint is
|
||||
// in the mount output
|
||||
func TestMount(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
out, err := exec.Command("mount").Output()
|
||||
require.NoError(t, err)
|
||||
assert.Contains(t, string(out), run.mountPath)
|
||||
}
|
||||
|
||||
// Check root directory is present and correct
|
||||
func TestRoot(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
fi, err := os.Lstat(run.mountPath)
|
||||
require.NoError(t, err)
|
||||
assert.True(t, fi.IsDir())
|
||||
assert.Equal(t, fi.Mode().Perm(), os.FileMode(dirPerms))
|
||||
}
|
||||
@@ -1,118 +0,0 @@
|
||||
// Package mount implents a FUSE mounting system for rclone remotes.
|
||||
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"bazil.org/fuse"
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
// Globals
|
||||
var (
|
||||
noModTime = false
|
||||
debugFUSE = false
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(mountCmd)
|
||||
mountCmd.Flags().BoolVarP(&noModTime, "no-modtime", "", false, "Don't read the modification time (can speed things up).")
|
||||
mountCmd.Flags().BoolVarP(&debugFUSE, "debug-fuse", "", false, "Debug the FUSE internals - needs -v.")
|
||||
}
|
||||
|
||||
var mountCmd = &cobra.Command{
|
||||
Use: "mount remote:path /path/to/mountpoint",
|
||||
Short: `Mount the remote as a mountpoint. **EXPERIMENTAL**`,
|
||||
Long: `
|
||||
rclone mount allows Linux, FreeBSD and macOS to mount any of Rclone's
|
||||
cloud storage systems as a file system with FUSE.
|
||||
|
||||
This is **EXPERIMENTAL** - use with care.
|
||||
|
||||
First set up your remote using ` + "`rclone config`" + `. Check it works with ` + "`rclone ls`" + ` etc.
|
||||
|
||||
Start the mount like this
|
||||
|
||||
rclone mount remote:path/to/files /path/to/local/mount &
|
||||
|
||||
Stop the mount with
|
||||
|
||||
fusermount -u /path/to/local/mount
|
||||
|
||||
Or with OS X
|
||||
|
||||
umount -u /path/to/local/mount
|
||||
|
||||
### Limitations ###
|
||||
|
||||
This can only read files seqentially, or write files sequentially. It
|
||||
can't read and write or seek in files.
|
||||
|
||||
rclonefs inherits rclone's directory handling. In rclone's world
|
||||
directories don't really exist. This means that empty directories
|
||||
will have a tendency to disappear once they fall out of the directory
|
||||
cache.
|
||||
|
||||
The bucket based FSes (eg swift, s3, google compute storage, b2) won't
|
||||
work from the root - you will need to specify a bucket, or a path
|
||||
within the bucket. So ` + "`swift:`" + ` won't work whereas ` + "`swift:bucket`" + ` will
|
||||
as will ` + "`swift:bucket/path`" + `.
|
||||
|
||||
Only supported on Linux, FreeBSD and OS X at the moment.
|
||||
|
||||
### rclone mount vs rclone sync/copy ##
|
||||
|
||||
File systems expect things to be 100% reliable, whereas cloud storage
|
||||
systems are a long way from 100% reliable. The rclone sync/copy
|
||||
commands cope with this with lots of retries. However rclone mount
|
||||
can't use retries in the same way without making local copies of the
|
||||
uploads. This might happen in the future, but for the moment rclone
|
||||
mount won't do that, so will be less reliable than the rclone command.
|
||||
|
||||
### Bugs ###
|
||||
|
||||
* All the remotes should work for read, but some may not for write
|
||||
* those which need to know the size in advance won't - eg B2
|
||||
* maybe should pass in size as -1 to mean work it out
|
||||
|
||||
### TODO ###
|
||||
|
||||
* Check hashes on upload/download
|
||||
* Preserve timestamps
|
||||
* Move directories
|
||||
`,
|
||||
RunE: func(command *cobra.Command, args []string) error {
|
||||
cmd.CheckArgs(2, 2, command, args)
|
||||
fdst := cmd.NewFsDst(args)
|
||||
return Mount(fdst, args[1])
|
||||
},
|
||||
}
|
||||
|
||||
// Mount mounts the remote at mountpoint.
|
||||
//
|
||||
// If noModTime is set then it
|
||||
func Mount(f fs.Fs, mountpoint string) error {
|
||||
if debugFUSE {
|
||||
fuse.Debug = func(msg interface{}) {
|
||||
fs.Debug("fuse", "%v", msg)
|
||||
}
|
||||
}
|
||||
|
||||
// Mount it
|
||||
errChan, err := mount(f, mountpoint)
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to mount FUSE fs")
|
||||
}
|
||||
|
||||
// Wait for umount
|
||||
err = <-errChan
|
||||
if err != nil {
|
||||
return errors.Wrap(err, "failed to umount FUSE fs")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
// Build for mount for unsupported platforms to stop go complaining
|
||||
// about "no buildable Go source files "
|
||||
|
||||
// +build !linux,!darwin,!freebsd
|
||||
|
||||
package mount
|
||||
@@ -1,130 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"io"
|
||||
"sync"
|
||||
|
||||
"bazil.org/fuse"
|
||||
fusefs "bazil.org/fuse/fs"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"golang.org/x/net/context"
|
||||
)
|
||||
|
||||
// ReadFileHandle is an open for read file handle on a File
|
||||
type ReadFileHandle struct {
|
||||
mu sync.Mutex
|
||||
closed bool // set if handle has been closed
|
||||
r io.ReadCloser
|
||||
o fs.Object
|
||||
readCalled bool // set if read has been called
|
||||
}
|
||||
|
||||
func newReadFileHandle(o fs.Object) (*ReadFileHandle, error) {
|
||||
r, err := o.Open()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &ReadFileHandle{
|
||||
r: r,
|
||||
o: o,
|
||||
}, nil
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.Handle = (*ReadFileHandle)(nil)
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.HandleReader = (*ReadFileHandle)(nil)
|
||||
|
||||
// Read from the file handle
|
||||
func (fh *ReadFileHandle) Read(ctx context.Context, req *fuse.ReadRequest, resp *fuse.ReadResponse) error {
|
||||
fs.Debug(fh.o, "ReadFileHandle.Open")
|
||||
if fh.closed {
|
||||
fs.ErrorLog(fh.o, "ReadFileHandle.Read error: %v", errClosedFileHandle)
|
||||
return errClosedFileHandle
|
||||
}
|
||||
fh.readCalled = true
|
||||
// We don't actually enforce Offset to match where previous read
|
||||
// ended. Maybe we should, but that would mean'd we need to track
|
||||
// it. The kernel *should* do it for us, based on the
|
||||
// fuse.OpenNonSeekable flag.
|
||||
//
|
||||
// One exception to the above is if we fail to fully populate a
|
||||
// page cache page; a read into page cache is always page aligned.
|
||||
// Make sure we never serve a partial read, to avoid that.
|
||||
buf := make([]byte, req.Size)
|
||||
n, err := io.ReadFull(fh.r, buf)
|
||||
if err == io.ErrUnexpectedEOF || err == io.EOF {
|
||||
err = nil
|
||||
}
|
||||
resp.Data = buf[:n]
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.o, "ReadFileHandle.Open error: %v", err)
|
||||
} else {
|
||||
fs.Debug(fh.o, "ReadFileHandle.Open OK")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// close the file handle returning errClosedFileHandle if it has been
|
||||
// closed already.
|
||||
//
|
||||
// Must be called with fh.mu held
|
||||
func (fh *ReadFileHandle) close() error {
|
||||
if fh.closed {
|
||||
return errClosedFileHandle
|
||||
}
|
||||
fh.closed = true
|
||||
return fh.r.Close()
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.HandleFlusher = (*ReadFileHandle)(nil)
|
||||
|
||||
// Flush is called each time the file or directory is closed.
|
||||
// Because there can be multiple file descriptors referring to a
|
||||
// single opened file, Flush can be called multiple times.
|
||||
func (fh *ReadFileHandle) Flush(ctx context.Context, req *fuse.FlushRequest) error {
|
||||
fh.mu.Lock()
|
||||
defer fh.mu.Unlock()
|
||||
fs.Debug(fh.o, "ReadFileHandle.Flush")
|
||||
// If Read hasn't been called then ignore the Flush - Release
|
||||
// will pick it up
|
||||
if !fh.readCalled {
|
||||
fs.Debug(fh.o, "ReadFileHandle.Flush ignoring flush on unread handle")
|
||||
return nil
|
||||
|
||||
}
|
||||
err := fh.close()
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.o, "ReadFileHandle.Flush error: %v", err)
|
||||
return err
|
||||
}
|
||||
fs.Debug(fh.o, "ReadFileHandle.Flush OK")
|
||||
return nil
|
||||
}
|
||||
|
||||
var _ fusefs.HandleReleaser = (*ReadFileHandle)(nil)
|
||||
|
||||
// Release is called when we are finished with the file handle
|
||||
//
|
||||
// It isn't called directly from userspace so the error is ignored by
|
||||
// the kernel
|
||||
func (fh *ReadFileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) error {
|
||||
fh.mu.Lock()
|
||||
defer fh.mu.Unlock()
|
||||
if fh.closed {
|
||||
fs.Debug(fh.o, "ReadFileHandle.Release nothing to do")
|
||||
return nil
|
||||
}
|
||||
fs.Debug(fh.o, "ReadFileHandle.Release closing")
|
||||
err := fh.close()
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.o, "ReadFileHandle.Release error: %v", err)
|
||||
} else {
|
||||
fs.Debug(fh.o, "ReadFileHandle.Release OK")
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -1,79 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"io"
|
||||
"os"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// Read by byte including don't read any bytes
|
||||
func TestReadByByte(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
var data = []byte("hellohello")
|
||||
run.createFile(t, "testfile", string(data))
|
||||
run.checkDir(t, "testfile 10")
|
||||
|
||||
for i := 0; i < len(data); i++ {
|
||||
fd, err := os.Open(run.path("testfile"))
|
||||
assert.NoError(t, err)
|
||||
for j := 0; j < i; j++ {
|
||||
buf := make([]byte, 1)
|
||||
n, err := io.ReadFull(fd, buf)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 1, n)
|
||||
assert.Equal(t, buf[0], data[j])
|
||||
}
|
||||
err = fd.Close()
|
||||
assert.NoError(t, err)
|
||||
}
|
||||
|
||||
run.rm(t, "testfile")
|
||||
}
|
||||
|
||||
// Test double close
|
||||
func TestReadFileDoubleClose(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.createFile(t, "testdoubleclose", "hello")
|
||||
|
||||
in, err := os.Open(run.path("testdoubleclose"))
|
||||
assert.NoError(t, err)
|
||||
fd := in.Fd()
|
||||
|
||||
fd1, err := syscall.Dup(int(fd))
|
||||
assert.NoError(t, err)
|
||||
|
||||
fd2, err := syscall.Dup(int(fd))
|
||||
assert.NoError(t, err)
|
||||
|
||||
// close one of the dups - should produce no error
|
||||
err = syscall.Close(fd1)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// read from the file
|
||||
buf := make([]byte, 1)
|
||||
_, err = in.Read(buf)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// close it
|
||||
err = in.Close()
|
||||
assert.NoError(t, err)
|
||||
|
||||
// read from the other dup - should produce no error as this
|
||||
// file is now buffered
|
||||
n, err := syscall.Read(fd2, buf)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 1, n)
|
||||
|
||||
// close the dup - should produce an error
|
||||
err = syscall.Close(fd2)
|
||||
assert.Error(t, err, "input/output error")
|
||||
|
||||
run.rm(t, "testdoubleclose")
|
||||
}
|
||||
@@ -1,157 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io"
|
||||
"sync"
|
||||
|
||||
"bazil.org/fuse"
|
||||
fusefs "bazil.org/fuse/fs"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"golang.org/x/net/context"
|
||||
)
|
||||
|
||||
var errClosedFileHandle = errors.New("Attempt to use closed file handle")
|
||||
|
||||
// WriteFileHandle is an open for write handle on a File
|
||||
type WriteFileHandle struct {
|
||||
mu sync.Mutex
|
||||
closed bool // set if handle has been closed
|
||||
remote string
|
||||
pipeReader *io.PipeReader
|
||||
pipeWriter *io.PipeWriter
|
||||
o fs.Object
|
||||
result chan error
|
||||
file *File
|
||||
writeCalled bool // set the first time Write() is called
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.Handle = (*WriteFileHandle)(nil)
|
||||
|
||||
func newWriteFileHandle(d *Dir, f *File, src fs.ObjectInfo) (*WriteFileHandle, error) {
|
||||
fh := &WriteFileHandle{
|
||||
remote: src.Remote(),
|
||||
result: make(chan error, 1),
|
||||
file: f,
|
||||
}
|
||||
fh.pipeReader, fh.pipeWriter = io.Pipe()
|
||||
go func() {
|
||||
o, err := d.f.Put(fh.pipeReader, src)
|
||||
fh.o = o
|
||||
fh.result <- err
|
||||
}()
|
||||
fh.file.addWriters(1)
|
||||
return fh, nil
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.HandleWriter = (*WriteFileHandle)(nil)
|
||||
|
||||
// Write data to the file handle
|
||||
func (fh *WriteFileHandle) Write(ctx context.Context, req *fuse.WriteRequest, resp *fuse.WriteResponse) error {
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Write len=%d", len(req.Data))
|
||||
fh.mu.Lock()
|
||||
defer fh.mu.Unlock()
|
||||
if fh.closed {
|
||||
fs.ErrorLog(fh.remote, "WriteFileHandle.Write error: %v", errClosedFileHandle)
|
||||
return errClosedFileHandle
|
||||
}
|
||||
fh.writeCalled = true
|
||||
// FIXME should probably check the file isn't being seeked?
|
||||
n, err := fh.pipeWriter.Write(req.Data)
|
||||
resp.Size = n
|
||||
fh.file.written(int64(n))
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.remote, "WriteFileHandle.Write error: %v", err)
|
||||
return err
|
||||
}
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Write OK (%d bytes written)", n)
|
||||
return nil
|
||||
}
|
||||
|
||||
// close the file handle returning errClosedFileHandle if it has been
|
||||
// closed already.
|
||||
//
|
||||
// Must be called with fh.mu held
|
||||
func (fh *WriteFileHandle) close() error {
|
||||
if fh.closed {
|
||||
return errClosedFileHandle
|
||||
}
|
||||
fh.closed = true
|
||||
fh.file.addWriters(-1)
|
||||
writeCloseErr := fh.pipeWriter.Close()
|
||||
err := <-fh.result
|
||||
readCloseErr := fh.pipeReader.Close()
|
||||
if err == nil {
|
||||
fh.file.setObject(fh.o)
|
||||
err = writeCloseErr
|
||||
}
|
||||
if err == nil {
|
||||
err = readCloseErr
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
// Check interface satisfied
|
||||
var _ fusefs.HandleFlusher = (*WriteFileHandle)(nil)
|
||||
|
||||
// Flush is called on each close() of a file descriptor. So if a
|
||||
// filesystem wants to return write errors in close() and the file has
|
||||
// cached dirty data, this is a good place to write back data and
|
||||
// return any errors. Since many applications ignore close() errors
|
||||
// this is not always useful.
|
||||
//
|
||||
// NOTE: The flush() method may be called more than once for each
|
||||
// open(). This happens if more than one file descriptor refers to an
|
||||
// opened file due to dup(), dup2() or fork() calls. It is not
|
||||
// possible to determine if a flush is final, so each flush should be
|
||||
// treated equally. Multiple write-flush sequences are relatively
|
||||
// rare, so this shouldn't be a problem.
|
||||
//
|
||||
// Filesystems shouldn't assume that flush will always be called after
|
||||
// some writes, or that if will be called at all.
|
||||
func (fh *WriteFileHandle) Flush(ctx context.Context, req *fuse.FlushRequest) error {
|
||||
fh.mu.Lock()
|
||||
defer fh.mu.Unlock()
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Flush")
|
||||
// If Write hasn't been called then ignore the Flush - Release
|
||||
// will pick it up
|
||||
if !fh.writeCalled {
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Flush ignoring flush on unwritten handle")
|
||||
return nil
|
||||
|
||||
}
|
||||
err := fh.close()
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.remote, "WriteFileHandle.Flush error: %v", err)
|
||||
} else {
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Flush OK")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
var _ fusefs.HandleReleaser = (*WriteFileHandle)(nil)
|
||||
|
||||
// Release is called when we are finished with the file handle
|
||||
//
|
||||
// It isn't called directly from userspace so the error is ignored by
|
||||
// the kernel
|
||||
func (fh *WriteFileHandle) Release(ctx context.Context, req *fuse.ReleaseRequest) error {
|
||||
fh.mu.Lock()
|
||||
defer fh.mu.Unlock()
|
||||
if fh.closed {
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Release nothing to do")
|
||||
return nil
|
||||
}
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Release closing")
|
||||
err := fh.close()
|
||||
if err != nil {
|
||||
fs.ErrorLog(fh.remote, "WriteFileHandle.Release error: %v", err)
|
||||
} else {
|
||||
fs.Debug(fh.remote, "WriteFileHandle.Release OK")
|
||||
}
|
||||
return err
|
||||
}
|
||||
@@ -1,103 +0,0 @@
|
||||
// +build linux darwin freebsd
|
||||
|
||||
package mount
|
||||
|
||||
import (
|
||||
"os"
|
||||
"syscall"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
// Test writing a file with no write()'s to it
|
||||
func TestWriteFileNoWrite(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
fd, err := os.Create(run.path("testnowrite"))
|
||||
assert.NoError(t, err)
|
||||
|
||||
err = fd.Close()
|
||||
assert.NoError(t, err)
|
||||
|
||||
run.checkDir(t, "testnowrite 0")
|
||||
|
||||
run.rm(t, "testnowrite")
|
||||
}
|
||||
|
||||
// Test open file in directory listing
|
||||
func FIXMETestWriteOpenFileInDirListing(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
fd, err := os.Create(run.path("testnowrite"))
|
||||
assert.NoError(t, err)
|
||||
|
||||
run.checkDir(t, "testnowrite 0")
|
||||
|
||||
err = fd.Close()
|
||||
assert.NoError(t, err)
|
||||
|
||||
run.rm(t, "testnowrite")
|
||||
}
|
||||
|
||||
// Test writing a file and reading it back
|
||||
func TestWriteFileWrite(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.createFile(t, "testwrite", "data")
|
||||
run.checkDir(t, "testwrite 4")
|
||||
contents := run.readFile(t, "testwrite")
|
||||
assert.Equal(t, "data", contents)
|
||||
run.rm(t, "testwrite")
|
||||
}
|
||||
|
||||
// Test overwriting a file
|
||||
func TestWriteFileOverwrite(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
run.createFile(t, "testwrite", "data")
|
||||
run.checkDir(t, "testwrite 4")
|
||||
run.createFile(t, "testwrite", "potato")
|
||||
contents := run.readFile(t, "testwrite")
|
||||
assert.Equal(t, "potato", contents)
|
||||
run.rm(t, "testwrite")
|
||||
}
|
||||
|
||||
// Test double close
|
||||
func TestWriteFileDoubleClose(t *testing.T) {
|
||||
run.skipIfNoFUSE(t)
|
||||
|
||||
out, err := os.Create(run.path("testdoubleclose"))
|
||||
assert.NoError(t, err)
|
||||
fd := out.Fd()
|
||||
|
||||
fd1, err := syscall.Dup(int(fd))
|
||||
assert.NoError(t, err)
|
||||
|
||||
fd2, err := syscall.Dup(int(fd))
|
||||
assert.NoError(t, err)
|
||||
|
||||
// close one of the dups - should produce no error
|
||||
err = syscall.Close(fd1)
|
||||
assert.NoError(t, err)
|
||||
|
||||
// write to the file
|
||||
buf := []byte("hello")
|
||||
n, err := out.Write(buf)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 5, n)
|
||||
|
||||
// close it
|
||||
err = out.Close()
|
||||
assert.NoError(t, err)
|
||||
|
||||
// write to the other dup - should produce an error
|
||||
n, err = syscall.Write(fd2, buf)
|
||||
assert.Error(t, err, "input/output error")
|
||||
|
||||
// close the dup - should produce an error
|
||||
err = syscall.Close(fd2)
|
||||
assert.Error(t, err, "input/output error")
|
||||
|
||||
run.rm(t, "testdoubleclose")
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
package move
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(moveCmd)
|
||||
}
|
||||
|
||||
var moveCmd = &cobra.Command{
|
||||
Use: "move source:path dest:path",
|
||||
Short: `Move files from source to dest.`,
|
||||
Long: `
|
||||
Moves the contents of the source directory to the destination
|
||||
directory. Rclone will error if the source and destination overlap.
|
||||
|
||||
If no filters are in use and if possible this will server side move
|
||||
` + "`" + `source:path` + "`" + ` into ` + "`" + `dest:path` + "`" + `. After this ` + "`" + `source:path` + "`" + ` will no
|
||||
longer longer exist.
|
||||
|
||||
Otherwise for each file in ` + "`" + `source:path` + "`" + ` selected by the filters (if
|
||||
any) this will move it into ` + "`" + `dest:path` + "`" + `. If possible a server side
|
||||
move will be used, otherwise it will copy it (server side if possible)
|
||||
into ` + "`" + `dest:path` + "`" + ` then delete the original (if no errors on copy) in
|
||||
` + "`" + `source:path` + "`" + `.
|
||||
|
||||
**Important**: Since this can cause data loss, test first with the
|
||||
--dry-run flag.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(2, 2, command, args)
|
||||
fsrc, fdst := cmd.NewFsSrcDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.MoveDir(fdst, fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,28 +0,0 @@
|
||||
package purge
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(purgeCmd)
|
||||
}
|
||||
|
||||
var purgeCmd = &cobra.Command{
|
||||
Use: "purge remote:path",
|
||||
Short: `Remove the path and all of its contents.`,
|
||||
Long: `
|
||||
Remove the path and all of its contents. Note that this does not obey
|
||||
include/exclude filters - everything will be removed. Use ` + "`" + `delete` + "`" + ` if
|
||||
you want to selectively delete files.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fdst := cmd.NewFsDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.Purge(fdst)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,16 +0,0 @@
|
||||
// Log the panic to the log file - for oses which can't do this
|
||||
|
||||
// +build !windows,!darwin,!dragonfly,!freebsd,!linux,!nacl,!netbsd,!openbsd
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// redirectStderr to the file passed in
|
||||
func redirectStderr(f *os.File) {
|
||||
fs.ErrorLog(nil, "Can't redirect stderr to file")
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
// Log the panic under unix to the log file
|
||||
|
||||
// +build darwin dragonfly freebsd linux nacl netbsd openbsd
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
// redirectStderr to the file passed in
|
||||
func redirectStderr(f *os.File) {
|
||||
err := syscall.Dup2(int(f.Fd()), int(os.Stderr.Fd()))
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to redirect stderr to file: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,39 +0,0 @@
|
||||
// Log the panic under windows to the log file
|
||||
//
|
||||
// Code from minix, via
|
||||
//
|
||||
// http://play.golang.org/p/kLtct7lSUg
|
||||
|
||||
// +build windows
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"log"
|
||||
"os"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
var (
|
||||
kernel32 = syscall.MustLoadDLL("kernel32.dll")
|
||||
procSetStdHandle = kernel32.MustFindProc("SetStdHandle")
|
||||
)
|
||||
|
||||
func setStdHandle(stdhandle int32, handle syscall.Handle) error {
|
||||
r0, _, e1 := syscall.Syscall(procSetStdHandle.Addr(), 2, uintptr(stdhandle), uintptr(handle), 0)
|
||||
if r0 == 0 {
|
||||
if e1 != 0 {
|
||||
return error(e1)
|
||||
}
|
||||
return syscall.EINVAL
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// redirectStderr to the file passed in
|
||||
func redirectStderr(f *os.File) {
|
||||
err := setStdHandle(syscall.STD_ERROR_HANDLE, syscall.Handle(f.Fd()))
|
||||
if err != nil {
|
||||
log.Fatalf("Failed to redirect stderr to file: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -1,26 +0,0 @@
|
||||
package rmdir
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(rmdirCmd)
|
||||
}
|
||||
|
||||
var rmdirCmd = &cobra.Command{
|
||||
Use: "rmdir remote:path",
|
||||
Short: `Remove the path if empty.`,
|
||||
Long: `
|
||||
Remove the path. Note that you can't remove a path with
|
||||
objects in it, use purge for that.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fdst := cmd.NewFsDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.Rmdir(fdst)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,29 +0,0 @@
|
||||
package sha1sum
|
||||
|
||||
import (
|
||||
"os"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(sha1sumCmd)
|
||||
}
|
||||
|
||||
var sha1sumCmd = &cobra.Command{
|
||||
Use: "sha1sum remote:path",
|
||||
Short: `Produces an sha1sum file for all the objects in the path.`,
|
||||
Long: `
|
||||
Produces an sha1sum file for all the objects in the path. This
|
||||
is in the same format as the standard sha1sum tool produces.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
return fs.Sha1sum(fsrc, os.Stdout)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,31 +0,0 @@
|
||||
package size
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(sizeCmd)
|
||||
}
|
||||
|
||||
var sizeCmd = &cobra.Command{
|
||||
Use: "size remote:path",
|
||||
Short: `Prints the total size and number of objects in remote:path.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(1, 1, command, args)
|
||||
fsrc := cmd.NewFsSrc(args)
|
||||
cmd.Run(false, command, func() error {
|
||||
objects, size, err := fs.Count(fsrc)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
fmt.Printf("Total objects: %d\n", objects)
|
||||
fmt.Printf("Total size: %s (%d Bytes)\n", fs.SizeSuffix(size).Unit("Bytes"), size)
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,43 +0,0 @@
|
||||
package sync
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(syncCmd)
|
||||
}
|
||||
|
||||
var syncCmd = &cobra.Command{
|
||||
Use: "sync source:path dest:path",
|
||||
Short: `Make source and dest identical, modifying destination only.`,
|
||||
Long: `
|
||||
Sync the source to the destination, changing the destination
|
||||
only. Doesn't transfer unchanged files, testing by size and
|
||||
modification time or MD5SUM. Destination is updated to match
|
||||
source, including deleting files if necessary.
|
||||
|
||||
**Important**: Since this can cause data loss, test first with the
|
||||
` + "`" + `--dry-run` + "`" + ` flag to see exactly what would be copied and deleted.
|
||||
|
||||
Note that files in the destination won't be deleted if there were any
|
||||
errors at any point.
|
||||
|
||||
It is always the contents of the directory that is synced, not the
|
||||
directory so when source:path is a directory, it's the contents of
|
||||
source:path that are copied, not the directory name and contents. See
|
||||
extended explanation in the ` + "`" + `copy` + "`" + ` command above if unsure.
|
||||
|
||||
If dest:path doesn't exist, it is created and the source:path contents
|
||||
go there.
|
||||
`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(2, 2, command, args)
|
||||
fsrc, fdst := cmd.NewFsSrcDst(args)
|
||||
cmd.Run(true, command, func() error {
|
||||
return fs.Sync(fdst, fsrc)
|
||||
})
|
||||
},
|
||||
}
|
||||
@@ -1,19 +0,0 @@
|
||||
package version
|
||||
|
||||
import (
|
||||
"github.com/ncw/rclone/cmd"
|
||||
"github.com/spf13/cobra"
|
||||
)
|
||||
|
||||
func init() {
|
||||
cmd.Root.AddCommand(versionCmd)
|
||||
}
|
||||
|
||||
var versionCmd = &cobra.Command{
|
||||
Use: "version",
|
||||
Short: `Show the version number.`,
|
||||
Run: func(command *cobra.Command, args []string) {
|
||||
cmd.CheckArgs(0, 0, command, args)
|
||||
cmd.ShowVersion()
|
||||
},
|
||||
}
|
||||
@@ -1,6 +0,0 @@
|
||||
//+build !go1.5
|
||||
|
||||
package cmd
|
||||
|
||||
// Upgrade to Go version 1.5 to compile rclone.
|
||||
func init() { Go_version_1_5_required_for_compilation() }
|
||||
@@ -1,9 +1,9 @@
|
||||
#!/bin/bash
|
||||
#!/bin/sh
|
||||
|
||||
set -e
|
||||
|
||||
# This uses gox from https://github.com/mitchellh/gox
|
||||
# Make sure you've run gox -build-toolchain - not required for go >= 1.5
|
||||
# Make sure you've run gox -build-toolchain
|
||||
|
||||
if [ "$1" == "" ]; then
|
||||
echo "Syntax: $0 Version"
|
||||
@@ -13,36 +13,7 @@ VERSION="$1"
|
||||
|
||||
rm -rf build
|
||||
|
||||
# Disable CGO and dynamic builds on all platforms (including build patform)
|
||||
export CGO_ENABLED=0
|
||||
|
||||
# Arch pairs we build for
|
||||
# gox -osarch-list for definitive list
|
||||
|
||||
OSARCH="\
|
||||
windows/386
|
||||
windows/amd64
|
||||
darwin/386
|
||||
darwin/amd64
|
||||
linux/386
|
||||
linux/amd64
|
||||
linux/arm
|
||||
freebsd/386
|
||||
freebsd/amd64
|
||||
freebsd/arm
|
||||
netbsd/386
|
||||
netbsd/amd64
|
||||
netbsd/arm
|
||||
openbsd/386
|
||||
openbsd/amd64
|
||||
plan9/386
|
||||
plan9/amd64
|
||||
solaris/amd64"
|
||||
|
||||
# Make space separated
|
||||
OSARCH=${OSARCH//$'\n'/ }
|
||||
|
||||
gox --ldflags "-s -X github.com/ncw/rclone/fs.Version=${VERSION}" -output "build/{{.Dir}}-${VERSION}-{{.OS}}-{{.Arch}}/{{.Dir}}" -osarch "${OSARCH}"
|
||||
gox -output "build/{{.Dir}}-${VERSION}-{{.OS}}-{{.Arch}}/{{.Dir}}"
|
||||
|
||||
mv build/rclone-${VERSION}-darwin-amd64 build/rclone-${VERSION}-osx-amd64
|
||||
mv build/rclone-${VERSION}-darwin-386 build/rclone-${VERSION}-osx-386
|
||||
@@ -50,12 +21,10 @@ mv build/rclone-${VERSION}-darwin-386 build/rclone-${VERSION}-osx-386
|
||||
cd build
|
||||
|
||||
for d in `ls`; do
|
||||
cp -a ../MANUAL.txt $d/README.txt
|
||||
cp -a ../MANUAL.html $d/README.html
|
||||
cp -a ../README.txt $d/
|
||||
cp -a ../README.html $d/
|
||||
cp -a ../rclone.1 $d/
|
||||
zip -r9 $d.zip $d
|
||||
d_current=${d/-${VERSION}/-current}
|
||||
ln $d.zip $d_current.zip
|
||||
rm -rf $d
|
||||
done
|
||||
|
||||
|
||||
608
crypt/cipher.go
608
crypt/cipher.go
@@ -1,608 +0,0 @@
|
||||
package crypt
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/aes"
|
||||
gocipher "crypto/cipher"
|
||||
"crypto/rand"
|
||||
"encoding/base32"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
"sync"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/ncw/rclone/crypt/pkcs7"
|
||||
"github.com/pkg/errors"
|
||||
|
||||
"golang.org/x/crypto/nacl/secretbox"
|
||||
"golang.org/x/crypto/scrypt"
|
||||
|
||||
"github.com/rfjakob/eme"
|
||||
)
|
||||
|
||||
// Constancs
|
||||
const (
|
||||
nameCipherBlockSize = aes.BlockSize
|
||||
fileMagic = "RCLONE\x00\x00"
|
||||
fileMagicSize = len(fileMagic)
|
||||
fileNonceSize = 24
|
||||
fileHeaderSize = fileMagicSize + fileNonceSize
|
||||
blockHeaderSize = secretbox.Overhead
|
||||
blockDataSize = 64 * 1024
|
||||
blockSize = blockHeaderSize + blockDataSize
|
||||
encryptedSuffix = ".bin" // when file name encryption is off we add this suffix to make sure the cloud provider doesn't process the file
|
||||
)
|
||||
|
||||
// Errors returned by cipher
|
||||
var (
|
||||
ErrorBadDecryptUTF8 = errors.New("bad decryption - utf-8 invalid")
|
||||
ErrorBadDecryptControlChar = errors.New("bad decryption - contains control chars")
|
||||
ErrorNotAMultipleOfBlocksize = errors.New("not a multiple of blocksize")
|
||||
ErrorTooShortAfterDecode = errors.New("too short after base32 decode")
|
||||
ErrorEncryptedFileTooShort = errors.New("file is too short to be encrypted")
|
||||
ErrorEncryptedFileBadHeader = errors.New("file has truncated block header")
|
||||
ErrorEncryptedBadMagic = errors.New("not an encrypted file - bad magic string")
|
||||
ErrorEncryptedBadBlock = errors.New("failed to authenticate decrypted block - bad password?")
|
||||
ErrorBadBase32Encoding = errors.New("bad base32 filename encoding")
|
||||
ErrorFileClosed = errors.New("file already closed")
|
||||
ErrorNotAnEncryptedFile = errors.New("not an encrypted file - no \"" + encryptedSuffix + "\" suffix")
|
||||
defaultSalt = []byte{0xA8, 0x0D, 0xF4, 0x3A, 0x8F, 0xBD, 0x03, 0x08, 0xA7, 0xCA, 0xB8, 0x3E, 0x58, 0x1F, 0x86, 0xB1}
|
||||
)
|
||||
|
||||
// Global variables
|
||||
var (
|
||||
fileMagicBytes = []byte(fileMagic)
|
||||
)
|
||||
|
||||
// Cipher is used to swap out the encryption implementations
|
||||
type Cipher interface {
|
||||
// EncryptFileName encrypts a file path
|
||||
EncryptFileName(string) string
|
||||
// DecryptFileName decrypts a file path, returns error if decrypt was invalid
|
||||
DecryptFileName(string) (string, error)
|
||||
// EncryptDirName encrypts a directory path
|
||||
EncryptDirName(string) string
|
||||
// DecryptDirName decrypts a directory path, returns error if decrypt was invalid
|
||||
DecryptDirName(string) (string, error)
|
||||
// EncryptData
|
||||
EncryptData(io.Reader) (io.Reader, error)
|
||||
// DecryptData
|
||||
DecryptData(io.ReadCloser) (io.ReadCloser, error)
|
||||
// EncryptedSize calculates the size of the data when encrypted
|
||||
EncryptedSize(int64) int64
|
||||
// DecryptedSize calculates the size of the data when decrypted
|
||||
DecryptedSize(int64) (int64, error)
|
||||
}
|
||||
|
||||
// NameEncryptionMode is the type of file name encryption in use
|
||||
type NameEncryptionMode int
|
||||
|
||||
// NameEncryptionMode levels
|
||||
const (
|
||||
NameEncryptionOff NameEncryptionMode = iota
|
||||
NameEncryptionStandard
|
||||
)
|
||||
|
||||
// NewNameEncryptionMode turns a string into a NameEncryptionMode
|
||||
func NewNameEncryptionMode(s string) (mode NameEncryptionMode, err error) {
|
||||
s = strings.ToLower(s)
|
||||
switch s {
|
||||
case "off":
|
||||
mode = NameEncryptionOff
|
||||
case "standard":
|
||||
mode = NameEncryptionStandard
|
||||
default:
|
||||
err = errors.Errorf("Unknown file name encryption mode %q", s)
|
||||
}
|
||||
return mode, err
|
||||
}
|
||||
|
||||
// String turns mode into a human readable string
|
||||
func (mode NameEncryptionMode) String() (out string) {
|
||||
switch mode {
|
||||
case NameEncryptionOff:
|
||||
out = "off"
|
||||
case NameEncryptionStandard:
|
||||
out = "standard"
|
||||
default:
|
||||
out = fmt.Sprintf("Unknown mode #%d", mode)
|
||||
}
|
||||
return out
|
||||
}
|
||||
|
||||
type cipher struct {
|
||||
dataKey [32]byte // Key for secretbox
|
||||
nameKey [32]byte // 16,24 or 32 bytes
|
||||
nameTweak [nameCipherBlockSize]byte // used to tweak the name crypto
|
||||
block gocipher.Block
|
||||
mode NameEncryptionMode
|
||||
buffers sync.Pool // encrypt/decrypt buffers
|
||||
cryptoRand io.Reader // read crypto random numbers from here
|
||||
}
|
||||
|
||||
// newCipher initialises the cipher. If salt is "" then it uses a built in salt val
|
||||
func newCipher(mode NameEncryptionMode, password, salt string) (*cipher, error) {
|
||||
c := &cipher{
|
||||
mode: mode,
|
||||
cryptoRand: rand.Reader,
|
||||
}
|
||||
c.buffers.New = func() interface{} {
|
||||
return make([]byte, blockSize)
|
||||
}
|
||||
err := c.Key(password, salt)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return c, nil
|
||||
}
|
||||
|
||||
// Key creates all the internal keys from the password passed in using
|
||||
// scrypt.
|
||||
//
|
||||
// If salt is "" we use a fixed salt just to make attackers lives
|
||||
// slighty harder than using no salt.
|
||||
//
|
||||
// Note that empty passsword makes all 0x00 keys which is used in the
|
||||
// tests.
|
||||
func (c *cipher) Key(password, salt string) (err error) {
|
||||
const keySize = len(c.dataKey) + len(c.nameKey) + len(c.nameTweak)
|
||||
var saltBytes = defaultSalt
|
||||
if salt != "" {
|
||||
saltBytes = []byte(salt)
|
||||
}
|
||||
var key []byte
|
||||
if password == "" {
|
||||
key = make([]byte, keySize)
|
||||
} else {
|
||||
key, err = scrypt.Key([]byte(password), saltBytes, 16384, 8, 1, keySize)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
copy(c.dataKey[:], key)
|
||||
copy(c.nameKey[:], key[len(c.dataKey):])
|
||||
copy(c.nameTweak[:], key[len(c.dataKey)+len(c.nameKey):])
|
||||
// Key the name cipher
|
||||
c.block, err = aes.NewCipher(c.nameKey[:])
|
||||
return err
|
||||
}
|
||||
|
||||
// getBlock gets a block from the pool of size blockSize
|
||||
func (c *cipher) getBlock() []byte {
|
||||
return c.buffers.Get().([]byte)
|
||||
}
|
||||
|
||||
// putBlock returns a block to the pool of size blockSize
|
||||
func (c *cipher) putBlock(buf []byte) {
|
||||
if len(buf) != blockSize {
|
||||
panic("bad blocksize returned to pool")
|
||||
}
|
||||
c.buffers.Put(buf)
|
||||
}
|
||||
|
||||
// check to see if the byte string is valid with no control characters
|
||||
// from 0x00 to 0x1F and is a valid UTF-8 string
|
||||
func checkValidString(buf []byte) error {
|
||||
for i := range buf {
|
||||
c := buf[i]
|
||||
if c >= 0x00 && c < 0x20 || c == 0x7F {
|
||||
return ErrorBadDecryptControlChar
|
||||
}
|
||||
}
|
||||
if !utf8.Valid(buf) {
|
||||
return ErrorBadDecryptUTF8
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// encodeFileName encodes a filename using a modified version of
|
||||
// standard base32 as described in RFC4648
|
||||
//
|
||||
// The standard encoding is modified in two ways
|
||||
// * it becomes lower case (no-one likes upper case filenames!)
|
||||
// * we strip the padding character `=`
|
||||
func encodeFileName(in []byte) string {
|
||||
encoded := base32.HexEncoding.EncodeToString(in)
|
||||
encoded = strings.TrimRight(encoded, "=")
|
||||
return strings.ToLower(encoded)
|
||||
}
|
||||
|
||||
// decodeFileName decodes a filename as encoded by encodeFileName
|
||||
func decodeFileName(in string) ([]byte, error) {
|
||||
if strings.HasSuffix(in, "=") {
|
||||
return nil, ErrorBadBase32Encoding
|
||||
}
|
||||
// First figure out how many padding characters to add
|
||||
roundUpToMultipleOf8 := (len(in) + 7) &^ 7
|
||||
equals := roundUpToMultipleOf8 - len(in)
|
||||
in = strings.ToUpper(in) + "========"[:equals]
|
||||
return base32.HexEncoding.DecodeString(in)
|
||||
}
|
||||
|
||||
// encryptSegment encrypts a path segment
|
||||
//
|
||||
// This uses EME with AES
|
||||
//
|
||||
// EME (ECB-Mix-ECB) is a wide-block encryption mode presented in the
|
||||
// 2003 paper "A Parallelizable Enciphering Mode" by Halevi and
|
||||
// Rogaway.
|
||||
//
|
||||
// This makes for determinstic encryption which is what we want - the
|
||||
// same filename must encrypt to the same thing.
|
||||
//
|
||||
// This means that
|
||||
// * filenames with the same name will encrypt the same
|
||||
// * filenames which start the same won't have a common prefix
|
||||
func (c *cipher) encryptSegment(plaintext string) string {
|
||||
if plaintext == "" {
|
||||
return ""
|
||||
}
|
||||
paddedPlaintext := pkcs7.Pad(nameCipherBlockSize, []byte(plaintext))
|
||||
ciphertext := eme.Transform(c.block, c.nameTweak[:], paddedPlaintext, eme.DirectionEncrypt)
|
||||
return encodeFileName(ciphertext)
|
||||
}
|
||||
|
||||
// decryptSegment decrypts a path segment
|
||||
func (c *cipher) decryptSegment(ciphertext string) (string, error) {
|
||||
if ciphertext == "" {
|
||||
return "", nil
|
||||
}
|
||||
rawCiphertext, err := decodeFileName(ciphertext)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if len(rawCiphertext)%nameCipherBlockSize != 0 {
|
||||
return "", ErrorNotAMultipleOfBlocksize
|
||||
}
|
||||
if len(rawCiphertext) == 0 {
|
||||
// not possible if decodeFilename() working correctly
|
||||
return "", ErrorTooShortAfterDecode
|
||||
}
|
||||
paddedPlaintext := eme.Transform(c.block, c.nameTweak[:], rawCiphertext, eme.DirectionDecrypt)
|
||||
plaintext, err := pkcs7.Unpad(nameCipherBlockSize, paddedPlaintext)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
err = checkValidString(plaintext)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(plaintext), err
|
||||
}
|
||||
|
||||
// encryptFileName encrypts a file path
|
||||
func (c *cipher) encryptFileName(in string) string {
|
||||
segments := strings.Split(in, "/")
|
||||
for i := range segments {
|
||||
segments[i] = c.encryptSegment(segments[i])
|
||||
}
|
||||
return strings.Join(segments, "/")
|
||||
}
|
||||
|
||||
// EncryptFileName encrypts a file path
|
||||
func (c *cipher) EncryptFileName(in string) string {
|
||||
if c.mode == NameEncryptionOff {
|
||||
return in + encryptedSuffix
|
||||
}
|
||||
return c.encryptFileName(in)
|
||||
}
|
||||
|
||||
// EncryptDirName encrypts a directory path
|
||||
func (c *cipher) EncryptDirName(in string) string {
|
||||
if c.mode == NameEncryptionOff {
|
||||
return in
|
||||
}
|
||||
return c.encryptFileName(in)
|
||||
}
|
||||
|
||||
// decryptFileName decrypts a file path
|
||||
func (c *cipher) decryptFileName(in string) (string, error) {
|
||||
segments := strings.Split(in, "/")
|
||||
for i := range segments {
|
||||
var err error
|
||||
segments[i], err = c.decryptSegment(segments[i])
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
}
|
||||
return strings.Join(segments, "/"), nil
|
||||
}
|
||||
|
||||
// DecryptFileName decrypts a file path
|
||||
func (c *cipher) DecryptFileName(in string) (string, error) {
|
||||
if c.mode == NameEncryptionOff {
|
||||
remainingLength := len(in) - len(encryptedSuffix)
|
||||
if remainingLength > 0 && strings.HasSuffix(in, encryptedSuffix) {
|
||||
return in[:remainingLength], nil
|
||||
}
|
||||
return "", ErrorNotAnEncryptedFile
|
||||
}
|
||||
return c.decryptFileName(in)
|
||||
}
|
||||
|
||||
// DecryptDirName decrypts a directory path
|
||||
func (c *cipher) DecryptDirName(in string) (string, error) {
|
||||
if c.mode == NameEncryptionOff {
|
||||
return in, nil
|
||||
}
|
||||
return c.decryptFileName(in)
|
||||
}
|
||||
|
||||
// nonce is an NACL secretbox nonce
|
||||
type nonce [fileNonceSize]byte
|
||||
|
||||
// pointer returns the nonce as a *[24]byte for secretbox
|
||||
func (n *nonce) pointer() *[fileNonceSize]byte {
|
||||
return (*[fileNonceSize]byte)(n)
|
||||
}
|
||||
|
||||
// fromReader fills the nonce from an io.Reader - normally the OSes
|
||||
// crypto random number generator
|
||||
func (n *nonce) fromReader(in io.Reader) error {
|
||||
read, err := io.ReadFull(in, (*n)[:])
|
||||
if read != fileNonceSize {
|
||||
return errors.Wrap(err, "short read of nonce")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// fromBuf fills the nonce from the buffer passed in
|
||||
func (n *nonce) fromBuf(buf []byte) {
|
||||
read := copy((*n)[:], buf)
|
||||
if read != fileNonceSize {
|
||||
panic("buffer to short to read nonce")
|
||||
}
|
||||
}
|
||||
|
||||
// increment to add 1 to the nonce
|
||||
func (n *nonce) increment() {
|
||||
for i := 0; i < len(*n); i++ {
|
||||
digit := (*n)[i]
|
||||
newDigit := digit + 1
|
||||
(*n)[i] = newDigit
|
||||
if newDigit >= digit {
|
||||
// exit if no carry
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// encrypter encrypts an io.Reader on the fly
|
||||
type encrypter struct {
|
||||
in io.Reader
|
||||
c *cipher
|
||||
nonce nonce
|
||||
buf []byte
|
||||
readBuf []byte
|
||||
bufIndex int
|
||||
bufSize int
|
||||
err error
|
||||
}
|
||||
|
||||
// newEncrypter creates a new file handle encrypting on the fly
|
||||
func (c *cipher) newEncrypter(in io.Reader) (*encrypter, error) {
|
||||
fh := &encrypter{
|
||||
in: in,
|
||||
c: c,
|
||||
buf: c.getBlock(),
|
||||
readBuf: c.getBlock(),
|
||||
bufSize: fileHeaderSize,
|
||||
}
|
||||
// Initialise nonce
|
||||
err := fh.nonce.fromReader(c.cryptoRand)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
// Copy magic into buffer
|
||||
copy(fh.buf, fileMagicBytes)
|
||||
// Copy nonce into buffer
|
||||
copy(fh.buf[fileMagicSize:], fh.nonce[:])
|
||||
return fh, nil
|
||||
}
|
||||
|
||||
// Read as per io.Reader
|
||||
func (fh *encrypter) Read(p []byte) (n int, err error) {
|
||||
if fh.err != nil {
|
||||
return 0, fh.err
|
||||
}
|
||||
if fh.bufIndex >= fh.bufSize {
|
||||
// Read data
|
||||
// FIXME should overlap the reads with a go-routine and 2 buffers?
|
||||
readBuf := fh.readBuf[:blockDataSize]
|
||||
n, err = io.ReadFull(fh.in, readBuf)
|
||||
if err == io.EOF {
|
||||
// ReadFull only returns n=0 and EOF
|
||||
return fh.finish(io.EOF)
|
||||
} else if err == io.ErrUnexpectedEOF {
|
||||
// Next read will return EOF
|
||||
} else if err != nil {
|
||||
return fh.finish(err)
|
||||
}
|
||||
// Write nonce to start of block
|
||||
copy(fh.buf, fh.nonce[:])
|
||||
// Encrypt the block using the nonce
|
||||
block := fh.buf
|
||||
secretbox.Seal(block[:0], readBuf[:n], fh.nonce.pointer(), &fh.c.dataKey)
|
||||
fh.bufIndex = 0
|
||||
fh.bufSize = blockHeaderSize + n
|
||||
fh.nonce.increment()
|
||||
}
|
||||
n = copy(p, fh.buf[fh.bufIndex:fh.bufSize])
|
||||
fh.bufIndex += n
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// finish sets the final error and tidies up
|
||||
func (fh *encrypter) finish(err error) (int, error) {
|
||||
if fh.err != nil {
|
||||
return 0, fh.err
|
||||
}
|
||||
fh.err = err
|
||||
fh.c.putBlock(fh.buf)
|
||||
fh.c.putBlock(fh.readBuf)
|
||||
return 0, err
|
||||
}
|
||||
|
||||
// Encrypt data encrypts the data stream
|
||||
func (c *cipher) EncryptData(in io.Reader) (io.Reader, error) {
|
||||
out, err := c.newEncrypter(in)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// decrypter decrypts an io.ReaderCloser on the fly
|
||||
type decrypter struct {
|
||||
rc io.ReadCloser
|
||||
nonce nonce
|
||||
c *cipher
|
||||
buf []byte
|
||||
readBuf []byte
|
||||
bufIndex int
|
||||
bufSize int
|
||||
err error
|
||||
}
|
||||
|
||||
// newDecrypter creates a new file handle decrypting on the fly
|
||||
func (c *cipher) newDecrypter(rc io.ReadCloser) (*decrypter, error) {
|
||||
fh := &decrypter{
|
||||
rc: rc,
|
||||
c: c,
|
||||
buf: c.getBlock(),
|
||||
readBuf: c.getBlock(),
|
||||
}
|
||||
// Read file header (magic + nonce)
|
||||
readBuf := fh.readBuf[:fileHeaderSize]
|
||||
_, err := io.ReadFull(fh.rc, readBuf)
|
||||
if err == io.EOF || err == io.ErrUnexpectedEOF {
|
||||
// This read from 0..fileHeaderSize-1 bytes
|
||||
return nil, fh.finishAndClose(ErrorEncryptedFileTooShort)
|
||||
} else if err != nil {
|
||||
return nil, fh.finishAndClose(err)
|
||||
}
|
||||
// check the magic
|
||||
if !bytes.Equal(readBuf[:fileMagicSize], fileMagicBytes) {
|
||||
return nil, fh.finishAndClose(ErrorEncryptedBadMagic)
|
||||
}
|
||||
// retreive the nonce
|
||||
fh.nonce.fromBuf(readBuf[fileMagicSize:])
|
||||
return fh, nil
|
||||
}
|
||||
|
||||
// Read as per io.Reader
|
||||
func (fh *decrypter) Read(p []byte) (n int, err error) {
|
||||
if fh.err != nil {
|
||||
return 0, fh.err
|
||||
}
|
||||
if fh.bufIndex >= fh.bufSize {
|
||||
// Read data
|
||||
// FIXME should overlap the reads with a go-routine and 2 buffers?
|
||||
readBuf := fh.readBuf
|
||||
n, err = io.ReadFull(fh.rc, readBuf)
|
||||
if err == io.EOF {
|
||||
// ReadFull only returns n=0 and EOF
|
||||
return 0, fh.finish(io.EOF)
|
||||
} else if err == io.ErrUnexpectedEOF {
|
||||
// Next read will return EOF
|
||||
} else if err != nil {
|
||||
return 0, fh.finish(err)
|
||||
}
|
||||
// Check header + 1 byte exists
|
||||
if n <= blockHeaderSize {
|
||||
return 0, fh.finish(ErrorEncryptedFileBadHeader)
|
||||
}
|
||||
// Decrypt the block using the nonce
|
||||
block := fh.buf
|
||||
_, ok := secretbox.Open(block[:0], readBuf[:n], fh.nonce.pointer(), &fh.c.dataKey)
|
||||
if !ok {
|
||||
return 0, fh.finish(ErrorEncryptedBadBlock)
|
||||
}
|
||||
fh.bufIndex = 0
|
||||
fh.bufSize = n - blockHeaderSize
|
||||
fh.nonce.increment()
|
||||
}
|
||||
n = copy(p, fh.buf[fh.bufIndex:fh.bufSize])
|
||||
fh.bufIndex += n
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// finish sets the final error and tidies up
|
||||
func (fh *decrypter) finish(err error) error {
|
||||
if fh.err != nil {
|
||||
return fh.err
|
||||
}
|
||||
fh.err = err
|
||||
fh.c.putBlock(fh.buf)
|
||||
fh.c.putBlock(fh.readBuf)
|
||||
return err
|
||||
}
|
||||
|
||||
// Close
|
||||
func (fh *decrypter) Close() error {
|
||||
// Check already closed
|
||||
if fh.err == ErrorFileClosed {
|
||||
return fh.err
|
||||
}
|
||||
// Closed before reading EOF so not finish()ed yet
|
||||
if fh.err == nil {
|
||||
_ = fh.finish(io.EOF)
|
||||
}
|
||||
// Show file now closed
|
||||
fh.err = ErrorFileClosed
|
||||
return fh.rc.Close()
|
||||
}
|
||||
|
||||
// finishAndClose does finish then Close()
|
||||
//
|
||||
// Used when we are returning a nil fh from new
|
||||
func (fh *decrypter) finishAndClose(err error) error {
|
||||
_ = fh.finish(err)
|
||||
_ = fh.Close()
|
||||
return err
|
||||
}
|
||||
|
||||
// DecryptData decrypts the data stream
|
||||
func (c *cipher) DecryptData(rc io.ReadCloser) (io.ReadCloser, error) {
|
||||
out, err := c.newDecrypter(rc)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
// EncryptedSize calculates the size of the data when encrypted
|
||||
func (c *cipher) EncryptedSize(size int64) int64 {
|
||||
blocks, residue := size/blockDataSize, size%blockDataSize
|
||||
encryptedSize := int64(fileHeaderSize) + blocks*(blockHeaderSize+blockDataSize)
|
||||
if residue != 0 {
|
||||
encryptedSize += blockHeaderSize + residue
|
||||
}
|
||||
return encryptedSize
|
||||
}
|
||||
|
||||
// DecryptedSize calculates the size of the data when decrypted
|
||||
func (c *cipher) DecryptedSize(size int64) (int64, error) {
|
||||
size -= int64(fileHeaderSize)
|
||||
if size < 0 {
|
||||
return 0, ErrorEncryptedFileTooShort
|
||||
}
|
||||
blocks, residue := size/blockSize, size%blockSize
|
||||
decryptedSize := blocks * blockDataSize
|
||||
if residue != 0 {
|
||||
residue -= blockHeaderSize
|
||||
if residue <= 0 {
|
||||
return 0, ErrorEncryptedFileBadHeader
|
||||
}
|
||||
}
|
||||
decryptedSize += residue
|
||||
return decryptedSize, nil
|
||||
}
|
||||
|
||||
// check interfaces
|
||||
var (
|
||||
_ Cipher = (*cipher)(nil)
|
||||
_ io.ReadCloser = (*decrypter)(nil)
|
||||
_ io.Reader = (*encrypter)(nil)
|
||||
)
|
||||
@@ -1,843 +0,0 @@
|
||||
package crypt
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/base32"
|
||||
"fmt"
|
||||
"io"
|
||||
"io/ioutil"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/crypt/pkcs7"
|
||||
"github.com/pkg/errors"
|
||||
"github.com/stretchr/testify/assert"
|
||||
"github.com/stretchr/testify/require"
|
||||
)
|
||||
|
||||
func TestNewNameEncryptionMode(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expected NameEncryptionMode
|
||||
expectedErr string
|
||||
}{
|
||||
{"off", NameEncryptionOff, ""},
|
||||
{"standard", NameEncryptionStandard, ""},
|
||||
{"potato", NameEncryptionMode(0), "Unknown file name encryption mode \"potato\""},
|
||||
} {
|
||||
actual, actualErr := NewNameEncryptionMode(test.in)
|
||||
assert.Equal(t, actual, test.expected)
|
||||
if test.expectedErr == "" {
|
||||
assert.NoError(t, actualErr)
|
||||
} else {
|
||||
assert.Error(t, actualErr, test.expectedErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewNameEncryptionModeString(t *testing.T) {
|
||||
assert.Equal(t, NameEncryptionOff.String(), "off")
|
||||
assert.Equal(t, NameEncryptionStandard.String(), "standard")
|
||||
assert.Equal(t, NameEncryptionMode(2).String(), "Unknown mode #2")
|
||||
}
|
||||
|
||||
func TestValidString(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expected error
|
||||
}{
|
||||
{"", nil},
|
||||
{"\x01", ErrorBadDecryptControlChar},
|
||||
{"a\x02", ErrorBadDecryptControlChar},
|
||||
{"abc\x03", ErrorBadDecryptControlChar},
|
||||
{"abc\x04def", ErrorBadDecryptControlChar},
|
||||
{"\x05d", ErrorBadDecryptControlChar},
|
||||
{"\x06def", ErrorBadDecryptControlChar},
|
||||
{"\x07", ErrorBadDecryptControlChar},
|
||||
{"\x08", ErrorBadDecryptControlChar},
|
||||
{"\x09", ErrorBadDecryptControlChar},
|
||||
{"\x0A", ErrorBadDecryptControlChar},
|
||||
{"\x0B", ErrorBadDecryptControlChar},
|
||||
{"\x0C", ErrorBadDecryptControlChar},
|
||||
{"\x0D", ErrorBadDecryptControlChar},
|
||||
{"\x0E", ErrorBadDecryptControlChar},
|
||||
{"\x0F", ErrorBadDecryptControlChar},
|
||||
{"\x10", ErrorBadDecryptControlChar},
|
||||
{"\x11", ErrorBadDecryptControlChar},
|
||||
{"\x12", ErrorBadDecryptControlChar},
|
||||
{"\x13", ErrorBadDecryptControlChar},
|
||||
{"\x14", ErrorBadDecryptControlChar},
|
||||
{"\x15", ErrorBadDecryptControlChar},
|
||||
{"\x16", ErrorBadDecryptControlChar},
|
||||
{"\x17", ErrorBadDecryptControlChar},
|
||||
{"\x18", ErrorBadDecryptControlChar},
|
||||
{"\x19", ErrorBadDecryptControlChar},
|
||||
{"\x1A", ErrorBadDecryptControlChar},
|
||||
{"\x1B", ErrorBadDecryptControlChar},
|
||||
{"\x1C", ErrorBadDecryptControlChar},
|
||||
{"\x1D", ErrorBadDecryptControlChar},
|
||||
{"\x1E", ErrorBadDecryptControlChar},
|
||||
{"\x1F", ErrorBadDecryptControlChar},
|
||||
{"\x20", nil},
|
||||
{"\x7E", nil},
|
||||
{"\x7F", ErrorBadDecryptControlChar},
|
||||
{"£100", nil},
|
||||
{`hello? sausage/êé/Hello, 世界/ " ' @ < > & ?/z.txt`, nil},
|
||||
{"£100", nil},
|
||||
// Following tests from http://www.php.net/manual/en/reference.pcre.pattern.modifiers.php#54805
|
||||
{"a", nil}, // Valid ASCII
|
||||
{"\xc3\xb1", nil}, // Valid 2 Octet Sequence
|
||||
{"\xc3\x28", ErrorBadDecryptUTF8}, // Invalid 2 Octet Sequence
|
||||
{"\xa0\xa1", ErrorBadDecryptUTF8}, // Invalid Sequence Identifier
|
||||
{"\xe2\x82\xa1", nil}, // Valid 3 Octet Sequence
|
||||
{"\xe2\x28\xa1", ErrorBadDecryptUTF8}, // Invalid 3 Octet Sequence (in 2nd Octet)
|
||||
{"\xe2\x82\x28", ErrorBadDecryptUTF8}, // Invalid 3 Octet Sequence (in 3rd Octet)
|
||||
{"\xf0\x90\x8c\xbc", nil}, // Valid 4 Octet Sequence
|
||||
{"\xf0\x28\x8c\xbc", ErrorBadDecryptUTF8}, // Invalid 4 Octet Sequence (in 2nd Octet)
|
||||
{"\xf0\x90\x28\xbc", ErrorBadDecryptUTF8}, // Invalid 4 Octet Sequence (in 3rd Octet)
|
||||
{"\xf0\x28\x8c\x28", ErrorBadDecryptUTF8}, // Invalid 4 Octet Sequence (in 4th Octet)
|
||||
{"\xf8\xa1\xa1\xa1\xa1", ErrorBadDecryptUTF8}, // Valid 5 Octet Sequence (but not Unicode!)
|
||||
{"\xfc\xa1\xa1\xa1\xa1\xa1", ErrorBadDecryptUTF8}, // Valid 6 Octet Sequence (but not Unicode!)
|
||||
} {
|
||||
actual := checkValidString([]byte(test.in))
|
||||
assert.Equal(t, actual, test.expected, fmt.Sprintf("in=%q", test.in))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncodeFileName(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expected string
|
||||
}{
|
||||
{"", ""},
|
||||
{"1", "64"},
|
||||
{"12", "64p0"},
|
||||
{"123", "64p36"},
|
||||
{"1234", "64p36d0"},
|
||||
{"12345", "64p36d1l"},
|
||||
{"123456", "64p36d1l6o"},
|
||||
{"1234567", "64p36d1l6org"},
|
||||
{"12345678", "64p36d1l6orjg"},
|
||||
{"123456789", "64p36d1l6orjge8"},
|
||||
{"1234567890", "64p36d1l6orjge9g"},
|
||||
{"12345678901", "64p36d1l6orjge9g64"},
|
||||
{"123456789012", "64p36d1l6orjge9g64p0"},
|
||||
{"1234567890123", "64p36d1l6orjge9g64p36"},
|
||||
{"12345678901234", "64p36d1l6orjge9g64p36d0"},
|
||||
{"123456789012345", "64p36d1l6orjge9g64p36d1l"},
|
||||
{"1234567890123456", "64p36d1l6orjge9g64p36d1l6o"},
|
||||
} {
|
||||
actual := encodeFileName([]byte(test.in))
|
||||
assert.Equal(t, actual, test.expected, fmt.Sprintf("in=%q", test.in))
|
||||
recovered, err := decodeFileName(test.expected)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, string(recovered), test.in, fmt.Sprintf("reverse=%q", test.expected))
|
||||
in := strings.ToUpper(test.expected)
|
||||
recovered, err = decodeFileName(in)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, string(recovered), test.in, fmt.Sprintf("reverse=%q", in))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecodeFileName(t *testing.T) {
|
||||
// We've tested decoding the valid ones above, now concentrate on the invalid ones
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expectedErr error
|
||||
}{
|
||||
{"64=", ErrorBadBase32Encoding},
|
||||
{"!", base32.CorruptInputError(0)},
|
||||
{"hello=hello", base32.CorruptInputError(5)},
|
||||
} {
|
||||
actual, actualErr := decodeFileName(test.in)
|
||||
assert.Equal(t, test.expectedErr, actualErr, fmt.Sprintf("in=%q got actual=%q, err = %v %T", test.in, actual, actualErr, actualErr))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncryptSegment(t *testing.T) {
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expected string
|
||||
}{
|
||||
{"", ""},
|
||||
{"1", "p0e52nreeaj0a5ea7s64m4j72s"},
|
||||
{"12", "l42g6771hnv3an9cgc8cr2n1ng"},
|
||||
{"123", "qgm4avr35m5loi1th53ato71v0"},
|
||||
{"1234", "8ivr2e9plj3c3esisjpdisikos"},
|
||||
{"12345", "rh9vu63q3o29eqmj4bg6gg7s44"},
|
||||
{"123456", "bn717l3alepn75b2fb2ejmi4b4"},
|
||||
{"1234567", "n6bo9jmb1qe3b1ogtj5qkf19k8"},
|
||||
{"12345678", "u9t24j7uaq94dh5q53m3s4t9ok"},
|
||||
{"123456789", "37hn305g6j12d1g0kkrl7ekbs4"},
|
||||
{"1234567890", "ot8d91eplaglb62k2b1trm2qv0"},
|
||||
{"12345678901", "h168vvrgb53qnrtvvmb378qrcs"},
|
||||
{"123456789012", "s3hsdf9e29ithrqbjqu01t8q2s"},
|
||||
{"1234567890123", "cf3jimlv1q2oc553mv7s3mh3eo"},
|
||||
{"12345678901234", "moq0uqdlqrblrc5pa5u5c7hq9g"},
|
||||
{"123456789012345", "eeam3li4rnommi3a762h5n7meg"},
|
||||
{"1234567890123456", "mijbj0frqf6ms7frcr6bd9h0env53jv96pjaaoirk7forcgpt70g"},
|
||||
} {
|
||||
actual := c.encryptSegment(test.in)
|
||||
assert.Equal(t, test.expected, actual, fmt.Sprintf("Testing %q", test.in))
|
||||
recovered, err := c.decryptSegment(test.expected)
|
||||
assert.NoError(t, err, fmt.Sprintf("Testing reverse %q", test.expected))
|
||||
assert.Equal(t, test.in, recovered, fmt.Sprintf("Testing reverse %q", test.expected))
|
||||
in := strings.ToUpper(test.expected)
|
||||
recovered, err = c.decryptSegment(in)
|
||||
assert.NoError(t, err, fmt.Sprintf("Testing reverse %q", in))
|
||||
assert.Equal(t, test.in, recovered, fmt.Sprintf("Testing reverse %q", in))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecryptSegment(t *testing.T) {
|
||||
// We've tested the forwards above, now concentrate on the errors
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
for _, test := range []struct {
|
||||
in string
|
||||
expectedErr error
|
||||
}{
|
||||
{"64=", ErrorBadBase32Encoding},
|
||||
{"!", base32.CorruptInputError(0)},
|
||||
{encodeFileName([]byte("a")), ErrorNotAMultipleOfBlocksize},
|
||||
{encodeFileName([]byte("123456789abcdef")), ErrorNotAMultipleOfBlocksize},
|
||||
{encodeFileName([]byte("123456789abcdef0")), pkcs7.ErrorPaddingTooLong},
|
||||
{c.encryptSegment("\x01"), ErrorBadDecryptControlChar},
|
||||
{c.encryptSegment("\xc3\x28"), ErrorBadDecryptUTF8},
|
||||
} {
|
||||
actual, actualErr := c.decryptSegment(test.in)
|
||||
assert.Equal(t, test.expectedErr, actualErr, fmt.Sprintf("in=%q got actual=%q, err = %v %T", test.in, actual, actualErr, actualErr))
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncryptFileName(t *testing.T) {
|
||||
// First standard mode
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s", c.EncryptFileName("1"))
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng", c.EncryptFileName("1/12"))
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0", c.EncryptFileName("1/12/123"))
|
||||
// Now off mode
|
||||
c, _ = newCipher(NameEncryptionOff, "", "")
|
||||
assert.Equal(t, "1/12/123.bin", c.EncryptFileName("1/12/123"))
|
||||
}
|
||||
|
||||
func TestDecryptFileName(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
mode NameEncryptionMode
|
||||
in string
|
||||
expected string
|
||||
expectedErr error
|
||||
}{
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s", "1", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng", "1/12", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeAJ0A5EA7S64M4J72S/L42G6771HNv3an9cgc8cr2n1ng", "1/12", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0", "1/12/123", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1/qgm4avr35m5loi1th53ato71v0", "", ErrorNotAMultipleOfBlocksize},
|
||||
{NameEncryptionOff, "1/12/123.bin", "1/12/123", nil},
|
||||
{NameEncryptionOff, "1/12/123.bix", "", ErrorNotAnEncryptedFile},
|
||||
{NameEncryptionOff, ".bin", "", ErrorNotAnEncryptedFile},
|
||||
} {
|
||||
c, _ := newCipher(test.mode, "", "")
|
||||
actual, actualErr := c.DecryptFileName(test.in)
|
||||
what := fmt.Sprintf("Testing %q (mode=%v)", test.in, test.mode)
|
||||
assert.Equal(t, test.expected, actual, what)
|
||||
assert.Equal(t, test.expectedErr, actualErr, what)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncryptDirName(t *testing.T) {
|
||||
// First standard mode
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s", c.EncryptDirName("1"))
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng", c.EncryptDirName("1/12"))
|
||||
assert.Equal(t, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0", c.EncryptDirName("1/12/123"))
|
||||
// Now off mode
|
||||
c, _ = newCipher(NameEncryptionOff, "", "")
|
||||
assert.Equal(t, "1/12/123", c.EncryptDirName("1/12/123"))
|
||||
}
|
||||
|
||||
func TestDecryptDirName(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
mode NameEncryptionMode
|
||||
in string
|
||||
expected string
|
||||
expectedErr error
|
||||
}{
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s", "1", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng", "1/12", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeAJ0A5EA7S64M4J72S/L42G6771HNv3an9cgc8cr2n1ng", "1/12", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1ng/qgm4avr35m5loi1th53ato71v0", "1/12/123", nil},
|
||||
{NameEncryptionStandard, "p0e52nreeaj0a5ea7s64m4j72s/l42g6771hnv3an9cgc8cr2n1/qgm4avr35m5loi1th53ato71v0", "", ErrorNotAMultipleOfBlocksize},
|
||||
{NameEncryptionOff, "1/12/123.bin", "1/12/123.bin", nil},
|
||||
{NameEncryptionOff, "1/12/123", "1/12/123", nil},
|
||||
{NameEncryptionOff, ".bin", ".bin", nil},
|
||||
} {
|
||||
c, _ := newCipher(test.mode, "", "")
|
||||
actual, actualErr := c.DecryptDirName(test.in)
|
||||
what := fmt.Sprintf("Testing %q (mode=%v)", test.in, test.mode)
|
||||
assert.Equal(t, test.expected, actual, what)
|
||||
assert.Equal(t, test.expectedErr, actualErr, what)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEncryptedSize(t *testing.T) {
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
for _, test := range []struct {
|
||||
in int64
|
||||
expected int64
|
||||
}{
|
||||
{0, 32},
|
||||
{1, 32 + 16 + 1},
|
||||
{65536, 32 + 16 + 65536},
|
||||
{65537, 32 + 16 + 65536 + 16 + 1},
|
||||
{1 << 20, 32 + 16*(16+65536)},
|
||||
{(1 << 20) + 65535, 32 + 16*(16+65536) + 16 + 65535},
|
||||
{1 << 30, 32 + 16384*(16+65536)},
|
||||
{(1 << 40) + 1, 32 + 16777216*(16+65536) + 16 + 1},
|
||||
} {
|
||||
actual := c.EncryptedSize(test.in)
|
||||
assert.Equal(t, test.expected, actual, fmt.Sprintf("Testing %d", test.in))
|
||||
recovered, err := c.DecryptedSize(test.expected)
|
||||
assert.NoError(t, err, fmt.Sprintf("Testing reverse %d", test.expected))
|
||||
assert.Equal(t, test.in, recovered, fmt.Sprintf("Testing reverse %d", test.expected))
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecryptedSize(t *testing.T) {
|
||||
// Test the errors since we tested the reverse above
|
||||
c, _ := newCipher(NameEncryptionStandard, "", "")
|
||||
for _, test := range []struct {
|
||||
in int64
|
||||
expectedErr error
|
||||
}{
|
||||
{0, ErrorEncryptedFileTooShort},
|
||||
{0, ErrorEncryptedFileTooShort},
|
||||
{1, ErrorEncryptedFileTooShort},
|
||||
{7, ErrorEncryptedFileTooShort},
|
||||
{32 + 1, ErrorEncryptedFileBadHeader},
|
||||
{32 + 16, ErrorEncryptedFileBadHeader},
|
||||
{32 + 16 + 65536 + 1, ErrorEncryptedFileBadHeader},
|
||||
{32 + 16 + 65536 + 16, ErrorEncryptedFileBadHeader},
|
||||
} {
|
||||
_, actualErr := c.DecryptedSize(test.in)
|
||||
assert.Equal(t, test.expectedErr, actualErr, fmt.Sprintf("Testing %d", test.in))
|
||||
}
|
||||
}
|
||||
|
||||
func TestNoncePointer(t *testing.T) {
|
||||
var x nonce
|
||||
assert.Equal(t, (*[24]byte)(&x), x.pointer())
|
||||
}
|
||||
|
||||
func TestNonceFromReader(t *testing.T) {
|
||||
var x nonce
|
||||
buf := bytes.NewBufferString("123456789abcdefghijklmno")
|
||||
err := x.fromReader(buf)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, nonce{'1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o'}, x)
|
||||
buf = bytes.NewBufferString("123456789abcdefghijklmn")
|
||||
err = x.fromReader(buf)
|
||||
assert.Error(t, err, "short read of nonce")
|
||||
}
|
||||
|
||||
func TestNonceFromBuf(t *testing.T) {
|
||||
var x nonce
|
||||
buf := []byte("123456789abcdefghijklmnoXXXXXXXX")
|
||||
x.fromBuf(buf)
|
||||
assert.Equal(t, nonce{'1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o'}, x)
|
||||
buf = []byte("0123456789abcdefghijklmn")
|
||||
x.fromBuf(buf)
|
||||
assert.Equal(t, nonce{'0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n'}, x)
|
||||
buf = []byte("0123456789abcdefghijklm")
|
||||
assert.Panics(t, func() { x.fromBuf(buf) })
|
||||
}
|
||||
|
||||
func TestNonceIncrement(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in nonce
|
||||
out nonce
|
||||
}{
|
||||
{
|
||||
nonce{0x00},
|
||||
nonce{0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF},
|
||||
nonce{0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x01},
|
||||
},
|
||||
{
|
||||
nonce{0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF, 0xFF},
|
||||
nonce{0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00},
|
||||
},
|
||||
} {
|
||||
x := test.in
|
||||
x.increment()
|
||||
assert.Equal(t, test.out, x)
|
||||
}
|
||||
}
|
||||
|
||||
// randomSource can read or write a random sequence
|
||||
type randomSource struct {
|
||||
counter int64
|
||||
size int64
|
||||
}
|
||||
|
||||
func newRandomSource(size int64) *randomSource {
|
||||
return &randomSource{
|
||||
size: size,
|
||||
}
|
||||
}
|
||||
|
||||
func (r *randomSource) next() byte {
|
||||
r.counter++
|
||||
return byte(r.counter % 257)
|
||||
}
|
||||
|
||||
func (r *randomSource) Read(p []byte) (n int, err error) {
|
||||
for i := range p {
|
||||
if r.counter >= r.size {
|
||||
err = io.EOF
|
||||
break
|
||||
}
|
||||
p[i] = r.next()
|
||||
n++
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func (r *randomSource) Write(p []byte) (n int, err error) {
|
||||
for i := range p {
|
||||
if p[i] != r.next() {
|
||||
return 0, errors.Errorf("Error in stream at %d", r.counter)
|
||||
}
|
||||
}
|
||||
return len(p), nil
|
||||
}
|
||||
|
||||
func (r *randomSource) Close() error { return nil }
|
||||
|
||||
// Check interfaces
|
||||
var (
|
||||
_ io.ReadCloser = (*randomSource)(nil)
|
||||
_ io.WriteCloser = (*randomSource)(nil)
|
||||
)
|
||||
|
||||
// Test test infrastructure first!
|
||||
func TestRandomSource(t *testing.T) {
|
||||
source := newRandomSource(1E8)
|
||||
sink := newRandomSource(1E8)
|
||||
n, err := io.Copy(sink, source)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, int64(1E8), n)
|
||||
|
||||
source = newRandomSource(1E8)
|
||||
buf := make([]byte, 16)
|
||||
_, _ = source.Read(buf)
|
||||
sink = newRandomSource(1E8)
|
||||
n, err = io.Copy(sink, source)
|
||||
assert.Error(t, err, "Error in stream")
|
||||
}
|
||||
|
||||
type zeroes struct{}
|
||||
|
||||
func (z *zeroes) Read(p []byte) (n int, err error) {
|
||||
for i := range p {
|
||||
p[i] = 0
|
||||
n++
|
||||
}
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// Test encrypt decrypt with different buffer sizes
|
||||
func testEncryptDecrypt(t *testing.T, bufSize int, copySize int64) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
c.cryptoRand = &zeroes{} // zero out the nonce
|
||||
buf := make([]byte, bufSize)
|
||||
source := newRandomSource(copySize)
|
||||
encrypted, err := c.newEncrypter(source)
|
||||
assert.NoError(t, err)
|
||||
decrypted, err := c.newDecrypter(ioutil.NopCloser(encrypted))
|
||||
assert.NoError(t, err)
|
||||
sink := newRandomSource(copySize)
|
||||
n, err := io.CopyBuffer(sink, decrypted, buf)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, copySize, n)
|
||||
blocks := copySize / blockSize
|
||||
if (copySize % blockSize) != 0 {
|
||||
blocks++
|
||||
}
|
||||
var expectedNonce = nonce{byte(blocks), byte(blocks >> 8), byte(blocks >> 16), byte(blocks >> 32)}
|
||||
assert.Equal(t, expectedNonce, encrypted.nonce)
|
||||
assert.Equal(t, expectedNonce, decrypted.nonce)
|
||||
}
|
||||
|
||||
func TestEncryptDecrypt1(t *testing.T) {
|
||||
testEncryptDecrypt(t, 1, 1E7)
|
||||
}
|
||||
|
||||
func TestEncryptDecrypt32(t *testing.T) {
|
||||
testEncryptDecrypt(t, 32, 1E8)
|
||||
}
|
||||
|
||||
func TestEncryptDecrypt4096(t *testing.T) {
|
||||
testEncryptDecrypt(t, 4096, 1E8)
|
||||
}
|
||||
|
||||
func TestEncryptDecrypt65536(t *testing.T) {
|
||||
testEncryptDecrypt(t, 65536, 1E8)
|
||||
}
|
||||
|
||||
func TestEncryptDecrypt65537(t *testing.T) {
|
||||
testEncryptDecrypt(t, 65537, 1E8)
|
||||
}
|
||||
|
||||
var (
|
||||
file0 = []byte{
|
||||
0x52, 0x43, 0x4c, 0x4f, 0x4e, 0x45, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
|
||||
0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
|
||||
}
|
||||
file1 = []byte{
|
||||
0x52, 0x43, 0x4c, 0x4f, 0x4e, 0x45, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
|
||||
0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
|
||||
0x09, 0x5b, 0x44, 0x6c, 0xd6, 0x23, 0x7b, 0xbc, 0xb0, 0x8d, 0x09, 0xfb, 0x52, 0x4c, 0xe5, 0x65,
|
||||
0xAA,
|
||||
}
|
||||
file16 = []byte{
|
||||
0x52, 0x43, 0x4c, 0x4f, 0x4e, 0x45, 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08,
|
||||
0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18,
|
||||
0xb9, 0xc4, 0x55, 0x2a, 0x27, 0x10, 0x06, 0x29, 0x18, 0x96, 0x0a, 0x3e, 0x60, 0x8c, 0x29, 0xb9,
|
||||
0xaa, 0x8a, 0x5e, 0x1e, 0x16, 0x5b, 0x6d, 0x07, 0x5d, 0xe4, 0xe9, 0xbb, 0x36, 0x7f, 0xd6, 0xd4,
|
||||
}
|
||||
)
|
||||
|
||||
func TestEncryptData(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
in []byte
|
||||
expected []byte
|
||||
}{
|
||||
{[]byte{}, file0},
|
||||
{[]byte{1}, file1},
|
||||
{[]byte{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16}, file16},
|
||||
} {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
|
||||
|
||||
// Check encode works
|
||||
buf := bytes.NewBuffer(test.in)
|
||||
encrypted, err := c.EncryptData(buf)
|
||||
assert.NoError(t, err)
|
||||
out, err := ioutil.ReadAll(encrypted)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, test.expected, out)
|
||||
|
||||
// Check we can decode the data properly too...
|
||||
buf = bytes.NewBuffer(out)
|
||||
decrypted, err := c.DecryptData(ioutil.NopCloser(buf))
|
||||
assert.NoError(t, err)
|
||||
out, err = ioutil.ReadAll(decrypted)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, test.in, out)
|
||||
}
|
||||
}
|
||||
|
||||
func TestNewEncrypter(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
|
||||
|
||||
z := &zeroes{}
|
||||
|
||||
fh, err := c.newEncrypter(z)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, nonce{0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18}, fh.nonce)
|
||||
assert.Equal(t, []byte{'R', 'C', 'L', 'O', 'N', 'E', 0x00, 0x00, 0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08, 0x09, 0x0a, 0x0b, 0x0c, 0x0d, 0x0e, 0x0f, 0x10, 0x11, 0x12, 0x13, 0x14, 0x15, 0x16, 0x17, 0x18}, fh.buf[:32])
|
||||
|
||||
// Test error path
|
||||
c.cryptoRand = bytes.NewBufferString("123456789abcdefghijklmn")
|
||||
fh, err = c.newEncrypter(z)
|
||||
assert.Nil(t, fh)
|
||||
assert.Error(t, err, "short read of nonce")
|
||||
|
||||
}
|
||||
|
||||
type errorReader struct {
|
||||
err error
|
||||
}
|
||||
|
||||
func (er errorReader) Read(p []byte) (n int, err error) {
|
||||
return 0, er.err
|
||||
}
|
||||
|
||||
type closeDetector struct {
|
||||
io.Reader
|
||||
closed int
|
||||
}
|
||||
|
||||
func newCloseDetector(in io.Reader) *closeDetector {
|
||||
return &closeDetector{
|
||||
Reader: in,
|
||||
}
|
||||
}
|
||||
|
||||
func (c *closeDetector) Close() error {
|
||||
c.closed++
|
||||
return nil
|
||||
}
|
||||
|
||||
func TestNewDecrypter(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
c.cryptoRand = newRandomSource(1E8) // nodge the crypto rand generator
|
||||
|
||||
cd := newCloseDetector(bytes.NewBuffer(file0))
|
||||
fh, err := c.newDecrypter(cd)
|
||||
assert.NoError(t, err)
|
||||
// check nonce is in place
|
||||
assert.Equal(t, file0[8:32], fh.nonce[:])
|
||||
assert.Equal(t, 0, cd.closed)
|
||||
|
||||
// Test error paths
|
||||
for i := range file0 {
|
||||
cd := newCloseDetector(bytes.NewBuffer(file0[:i]))
|
||||
fh, err = c.newDecrypter(cd)
|
||||
assert.Nil(t, fh)
|
||||
assert.Error(t, err, ErrorEncryptedFileTooShort.Error())
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
}
|
||||
|
||||
er := &errorReader{errors.New("potato")}
|
||||
cd = newCloseDetector(er)
|
||||
fh, err = c.newDecrypter(cd)
|
||||
assert.Nil(t, fh)
|
||||
assert.Error(t, err, "potato")
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
|
||||
// bad magic
|
||||
file0copy := make([]byte, len(file0))
|
||||
copy(file0copy, file0)
|
||||
for i := range fileMagic {
|
||||
file0copy[i] ^= 0x1
|
||||
cd := newCloseDetector(bytes.NewBuffer(file0copy))
|
||||
fh, err := c.newDecrypter(cd)
|
||||
assert.Nil(t, fh)
|
||||
assert.Error(t, err, ErrorEncryptedBadMagic.Error())
|
||||
file0copy[i] ^= 0x1
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecrypterRead(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Test truncating the header
|
||||
for i := 1; i < blockHeaderSize; i++ {
|
||||
cd := newCloseDetector(bytes.NewBuffer(file1[:len(file1)-i]))
|
||||
fh, err := c.newDecrypter(cd)
|
||||
assert.NoError(t, err)
|
||||
_, err = ioutil.ReadAll(fh)
|
||||
assert.Error(t, err, ErrorEncryptedFileBadHeader.Error())
|
||||
assert.Equal(t, 0, cd.closed)
|
||||
}
|
||||
|
||||
// Test producing an error on the file on Read the underlying file
|
||||
in1 := bytes.NewBuffer(file1)
|
||||
in2 := &errorReader{errors.New("potato")}
|
||||
in := io.MultiReader(in1, in2)
|
||||
cd := newCloseDetector(in)
|
||||
fh, err := c.newDecrypter(cd)
|
||||
assert.NoError(t, err)
|
||||
_, err = ioutil.ReadAll(fh)
|
||||
assert.Error(t, err, "potato")
|
||||
assert.Equal(t, 0, cd.closed)
|
||||
|
||||
// Test corrupting the input
|
||||
// shouldn't be able to corrupt any byte without some sort of error
|
||||
file16copy := make([]byte, len(file16))
|
||||
copy(file16copy, file16)
|
||||
for i := range file16copy {
|
||||
file16copy[i] ^= 0xFF
|
||||
fh, err := c.newDecrypter(ioutil.NopCloser(bytes.NewBuffer(file16copy)))
|
||||
if i < fileMagicSize {
|
||||
assert.Error(t, err, ErrorEncryptedBadMagic.Error())
|
||||
assert.Nil(t, fh)
|
||||
} else {
|
||||
assert.NoError(t, err)
|
||||
_, err = ioutil.ReadAll(fh)
|
||||
assert.Error(t, err, ErrorEncryptedFileBadHeader.Error())
|
||||
}
|
||||
file16copy[i] ^= 0xFF
|
||||
}
|
||||
}
|
||||
|
||||
func TestDecrypterClose(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
|
||||
cd := newCloseDetector(bytes.NewBuffer(file16))
|
||||
fh, err := c.newDecrypter(cd)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 0, cd.closed)
|
||||
|
||||
// close before reading
|
||||
assert.Equal(t, nil, fh.err)
|
||||
err = fh.Close()
|
||||
assert.Equal(t, ErrorFileClosed, fh.err)
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
|
||||
// double close
|
||||
err = fh.Close()
|
||||
assert.Error(t, err, ErrorFileClosed.Error())
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
|
||||
// try again reading the file this time
|
||||
cd = newCloseDetector(bytes.NewBuffer(file1))
|
||||
fh, err = c.newDecrypter(cd)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, 0, cd.closed)
|
||||
|
||||
// close after reading
|
||||
out, err := ioutil.ReadAll(fh)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []byte{1}, out)
|
||||
assert.Equal(t, io.EOF, fh.err)
|
||||
err = fh.Close()
|
||||
assert.Equal(t, ErrorFileClosed, fh.err)
|
||||
assert.Equal(t, 1, cd.closed)
|
||||
}
|
||||
|
||||
func TestPutGetBlock(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
|
||||
block := c.getBlock()
|
||||
c.putBlock(block)
|
||||
c.putBlock(block)
|
||||
|
||||
assert.Panics(t, func() { c.putBlock(block[:len(block)-1]) })
|
||||
}
|
||||
|
||||
func TestKey(t *testing.T) {
|
||||
c, err := newCipher(NameEncryptionStandard, "", "")
|
||||
assert.NoError(t, err)
|
||||
|
||||
// Check zero keys OK
|
||||
assert.Equal(t, [32]byte{}, c.dataKey)
|
||||
assert.Equal(t, [32]byte{}, c.nameKey)
|
||||
assert.Equal(t, [16]byte{}, c.nameTweak)
|
||||
|
||||
require.NoError(t, c.Key("potato", ""))
|
||||
assert.Equal(t, [32]byte{0x74, 0x55, 0xC7, 0x1A, 0xB1, 0x7C, 0x86, 0x5B, 0x84, 0x71, 0xF4, 0x7B, 0x79, 0xAC, 0xB0, 0x7E, 0xB3, 0x1D, 0x56, 0x78, 0xB8, 0x0C, 0x7E, 0x2E, 0xAF, 0x4F, 0xC8, 0x06, 0x6A, 0x9E, 0xE4, 0x68}, c.dataKey)
|
||||
assert.Equal(t, [32]byte{0x76, 0x5D, 0xA2, 0x7A, 0xB1, 0x5D, 0x77, 0xF9, 0x57, 0x96, 0x71, 0x1F, 0x7B, 0x93, 0xAD, 0x63, 0xBB, 0xB4, 0x84, 0x07, 0x2E, 0x71, 0x80, 0xA8, 0xD1, 0x7A, 0x9B, 0xBE, 0xC1, 0x42, 0x70, 0xD0}, c.nameKey)
|
||||
assert.Equal(t, [16]byte{0xC1, 0x8D, 0x59, 0x32, 0xF5, 0x5B, 0x28, 0x28, 0xC5, 0xE1, 0xE8, 0x72, 0x15, 0x52, 0x03, 0x10}, c.nameTweak)
|
||||
|
||||
require.NoError(t, c.Key("Potato", ""))
|
||||
assert.Equal(t, [32]byte{0xAE, 0xEA, 0x6A, 0xD3, 0x47, 0xDF, 0x75, 0xB9, 0x63, 0xCE, 0x12, 0xF5, 0x76, 0x23, 0xE9, 0x46, 0xD4, 0x2E, 0xD8, 0xBF, 0x3E, 0x92, 0x8B, 0x39, 0x24, 0x37, 0x94, 0x13, 0x3E, 0x5E, 0xF7, 0x5E}, c.dataKey)
|
||||
assert.Equal(t, [32]byte{0x54, 0xF7, 0x02, 0x6E, 0x8A, 0xFC, 0x56, 0x0A, 0x86, 0x63, 0x6A, 0xAB, 0x2C, 0x9C, 0x51, 0x62, 0xE5, 0x1A, 0x12, 0x23, 0x51, 0x83, 0x6E, 0xAF, 0x50, 0x42, 0x0F, 0x98, 0x1C, 0x86, 0x0A, 0x19}, c.nameKey)
|
||||
assert.Equal(t, [16]byte{0xF8, 0xC1, 0xB6, 0x27, 0x2D, 0x52, 0x9B, 0x4A, 0x8F, 0xDA, 0xEB, 0x42, 0x4A, 0x28, 0xDD, 0xF3}, c.nameTweak)
|
||||
|
||||
require.NoError(t, c.Key("potato", "sausage"))
|
||||
assert.Equal(t, [32]uint8{0x8e, 0x9b, 0x6b, 0x99, 0xf8, 0x69, 0x4, 0x67, 0xa0, 0x71, 0xf9, 0xcb, 0x92, 0xd0, 0xaa, 0x78, 0x7f, 0x8f, 0xf1, 0x78, 0xbe, 0xc9, 0x6f, 0x99, 0x9f, 0xd5, 0x20, 0x6e, 0x64, 0x4a, 0x1b, 0x50}, c.dataKey)
|
||||
assert.Equal(t, [32]uint8{0x3e, 0xa9, 0x5e, 0xf6, 0x81, 0x78, 0x2d, 0xc9, 0xd9, 0x95, 0x5d, 0x22, 0x5b, 0xfd, 0x44, 0x2c, 0x6f, 0x5d, 0x68, 0x97, 0xb0, 0x29, 0x1, 0x5c, 0x6f, 0x46, 0x2e, 0x2a, 0x9d, 0xae, 0x2c, 0xe3}, c.nameKey)
|
||||
assert.Equal(t, [16]uint8{0xf1, 0x7f, 0xd7, 0x14, 0x1d, 0x65, 0x27, 0x4f, 0x36, 0x3f, 0xc2, 0xa0, 0x4d, 0xd2, 0x14, 0x8a}, c.nameTweak)
|
||||
|
||||
require.NoError(t, c.Key("potato", "Sausage"))
|
||||
assert.Equal(t, [32]uint8{0xda, 0x81, 0x8c, 0x67, 0xef, 0x11, 0xf, 0xc8, 0xd5, 0xc8, 0x62, 0x4b, 0x7f, 0xe2, 0x9e, 0x35, 0x35, 0xb0, 0x8d, 0x79, 0x84, 0x89, 0xac, 0xcb, 0xa0, 0xff, 0x2, 0x72, 0x3, 0x1a, 0x5e, 0x64}, c.dataKey)
|
||||
assert.Equal(t, [32]uint8{0x2, 0x81, 0x7e, 0x7b, 0xea, 0x99, 0x81, 0x5a, 0xd0, 0x2d, 0xb9, 0x64, 0x48, 0xb0, 0x28, 0x27, 0x7c, 0x20, 0xb4, 0xd4, 0xa4, 0x68, 0xad, 0x4e, 0x5c, 0x29, 0xf, 0x79, 0xef, 0xee, 0xdb, 0x3b}, c.nameKey)
|
||||
assert.Equal(t, [16]uint8{0x9a, 0xb5, 0xb, 0x3d, 0xcb, 0x60, 0x59, 0x55, 0xa5, 0x4d, 0xe6, 0xb6, 0x47, 0x3, 0x23, 0xe2}, c.nameTweak)
|
||||
|
||||
require.NoError(t, c.Key("", ""))
|
||||
assert.Equal(t, [32]byte{}, c.dataKey)
|
||||
assert.Equal(t, [32]byte{}, c.nameKey)
|
||||
assert.Equal(t, [16]byte{}, c.nameTweak)
|
||||
}
|
||||
430
crypt/crypt.go
430
crypt/crypt.go
@@ -1,430 +0,0 @@
|
||||
// Package crypt provides wrappers for Fs and Object which implement encryption
|
||||
package crypt
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"path"
|
||||
"sync"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// Register with Fs
|
||||
func init() {
|
||||
fs.Register(&fs.RegInfo{
|
||||
Name: "crypt",
|
||||
Description: "Encrypt/Decrypt a remote",
|
||||
NewFs: NewFs,
|
||||
Options: []fs.Option{{
|
||||
Name: "remote",
|
||||
Help: "Remote to encrypt/decrypt.",
|
||||
}, {
|
||||
Name: "filename_encryption",
|
||||
Help: "How to encrypt the filenames.",
|
||||
Examples: []fs.OptionExample{
|
||||
{
|
||||
Value: "off",
|
||||
Help: "Don't encrypt the file names. Adds a \".bin\" extension only.",
|
||||
}, {
|
||||
Value: "standard",
|
||||
Help: "Encrypt the filenames see the docs for the details.",
|
||||
},
|
||||
},
|
||||
}, {
|
||||
Name: "password",
|
||||
Help: "Password or pass phrase for encryption.",
|
||||
IsPassword: true,
|
||||
}, {
|
||||
Name: "password2",
|
||||
Help: "Password or pass phrase for salt. Optional but recommended.\nShould be different to the previous password.",
|
||||
IsPassword: true,
|
||||
Optional: true,
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
// NewFs contstructs an Fs from the path, container:path
|
||||
func NewFs(name, rpath string) (fs.Fs, error) {
|
||||
mode, err := NewNameEncryptionMode(fs.ConfigFile.MustValue(name, "filename_encryption", "standard"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
password := fs.ConfigFile.MustValue(name, "password", "")
|
||||
if password == "" {
|
||||
return nil, errors.New("password not set in config file")
|
||||
}
|
||||
password, err = fs.Reveal(password)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to decrypt password")
|
||||
}
|
||||
salt := fs.ConfigFile.MustValue(name, "password2", "")
|
||||
if salt != "" {
|
||||
salt, err = fs.Reveal(salt)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to decrypt password2")
|
||||
}
|
||||
}
|
||||
cipher, err := newCipher(mode, password, salt)
|
||||
if err != nil {
|
||||
return nil, errors.Wrap(err, "failed to make cipher")
|
||||
}
|
||||
remote := fs.ConfigFile.MustValue(name, "remote")
|
||||
// Look for a file first
|
||||
remotePath := path.Join(remote, cipher.EncryptFileName(rpath))
|
||||
wrappedFs, err := fs.NewFs(remotePath)
|
||||
// if that didn't produce a file, look for a directory
|
||||
if err != fs.ErrorIsFile {
|
||||
remotePath = path.Join(remote, cipher.EncryptDirName(rpath))
|
||||
wrappedFs, err = fs.NewFs(remotePath)
|
||||
}
|
||||
if err != fs.ErrorIsFile && err != nil {
|
||||
return nil, errors.Wrapf(err, "failed to make remote %q to wrap", remotePath)
|
||||
}
|
||||
f := &Fs{
|
||||
Fs: wrappedFs,
|
||||
cipher: cipher,
|
||||
mode: mode,
|
||||
}
|
||||
return f, err
|
||||
}
|
||||
|
||||
// Fs represents a wrapped fs.Fs
|
||||
type Fs struct {
|
||||
fs.Fs
|
||||
cipher Cipher
|
||||
mode NameEncryptionMode
|
||||
}
|
||||
|
||||
// String returns a description of the FS
|
||||
func (f *Fs) String() string {
|
||||
return fmt.Sprintf("Encrypted %s", f.Fs.String())
|
||||
}
|
||||
|
||||
// List the Fs into a channel
|
||||
func (f *Fs) List(opts fs.ListOpts, dir string) {
|
||||
f.Fs.List(f.newListOpts(opts, dir), f.cipher.EncryptDirName(dir))
|
||||
}
|
||||
|
||||
// NewObject finds the Object at remote.
|
||||
func (f *Fs) NewObject(remote string) (fs.Object, error) {
|
||||
o, err := f.Fs.NewObject(f.cipher.EncryptFileName(remote))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.newObject(o), nil
|
||||
}
|
||||
|
||||
// Put in to the remote path with the modTime given of the given size
|
||||
//
|
||||
// May create the object even if it returns an error - if so
|
||||
// will return the object and the error, otherwise will return
|
||||
// nil and the error
|
||||
func (f *Fs) Put(in io.Reader, src fs.ObjectInfo) (fs.Object, error) {
|
||||
wrappedIn, err := f.cipher.EncryptData(in)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
o, err := f.Fs.Put(wrappedIn, f.newObjectInfo(src))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.newObject(o), nil
|
||||
}
|
||||
|
||||
// Hashes returns the supported hash sets.
|
||||
func (f *Fs) Hashes() fs.HashSet {
|
||||
return fs.HashSet(fs.HashNone)
|
||||
}
|
||||
|
||||
// Purge all files in the root and the root directory
|
||||
//
|
||||
// Implement this if you have a way of deleting all the files
|
||||
// quicker than just running Remove() on the result of List()
|
||||
//
|
||||
// Return an error if it doesn't exist
|
||||
func (f *Fs) Purge() error {
|
||||
do, ok := f.Fs.(fs.Purger)
|
||||
if !ok {
|
||||
return fs.ErrorCantPurge
|
||||
}
|
||||
return do.Purge()
|
||||
}
|
||||
|
||||
// Copy src to this remote using server side copy operations.
|
||||
//
|
||||
// This is stored with the remote path given
|
||||
//
|
||||
// It returns the destination Object and a possible error
|
||||
//
|
||||
// Will only be called if src.Fs().Name() == f.Name()
|
||||
//
|
||||
// If it isn't possible then return fs.ErrorCantCopy
|
||||
func (f *Fs) Copy(src fs.Object, remote string) (fs.Object, error) {
|
||||
do, ok := f.Fs.(fs.Copier)
|
||||
if !ok {
|
||||
return nil, fs.ErrorCantCopy
|
||||
}
|
||||
o, ok := src.(*Object)
|
||||
if !ok {
|
||||
return nil, fs.ErrorCantCopy
|
||||
}
|
||||
oResult, err := do.Copy(o.Object, f.cipher.EncryptFileName(remote))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.newObject(oResult), nil
|
||||
}
|
||||
|
||||
// Move src to this remote using server side move operations.
|
||||
//
|
||||
// This is stored with the remote path given
|
||||
//
|
||||
// It returns the destination Object and a possible error
|
||||
//
|
||||
// Will only be called if src.Fs().Name() == f.Name()
|
||||
//
|
||||
// If it isn't possible then return fs.ErrorCantMove
|
||||
func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
|
||||
do, ok := f.Fs.(fs.Mover)
|
||||
if !ok {
|
||||
return nil, fs.ErrorCantMove
|
||||
}
|
||||
o, ok := src.(*Object)
|
||||
if !ok {
|
||||
return nil, fs.ErrorCantMove
|
||||
}
|
||||
oResult, err := do.Move(o.Object, f.cipher.EncryptFileName(remote))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return f.newObject(oResult), nil
|
||||
}
|
||||
|
||||
// DirMove moves src to this remote using server side move
|
||||
// operations.
|
||||
//
|
||||
// Will only be called if src.Fs().Name() == f.Name()
|
||||
//
|
||||
// If it isn't possible then return fs.ErrorCantDirMove
|
||||
//
|
||||
// If destination exists then return fs.ErrorDirExists
|
||||
func (f *Fs) DirMove(src fs.Fs) error {
|
||||
do, ok := f.Fs.(fs.DirMover)
|
||||
if !ok {
|
||||
return fs.ErrorCantDirMove
|
||||
}
|
||||
srcFs, ok := src.(*Fs)
|
||||
if !ok {
|
||||
fs.Debug(srcFs, "Can't move directory - not same remote type")
|
||||
return fs.ErrorCantDirMove
|
||||
}
|
||||
return do.DirMove(srcFs.Fs)
|
||||
}
|
||||
|
||||
// UnWrap returns the Fs that this Fs is wrapping
|
||||
func (f *Fs) UnWrap() fs.Fs {
|
||||
return f.Fs
|
||||
}
|
||||
|
||||
// Object describes a wrapped for being read from the Fs
|
||||
//
|
||||
// This decrypts the remote name and decrypts the data
|
||||
type Object struct {
|
||||
fs.Object
|
||||
f *Fs
|
||||
}
|
||||
|
||||
func (f *Fs) newObject(o fs.Object) *Object {
|
||||
return &Object{
|
||||
Object: o,
|
||||
f: f,
|
||||
}
|
||||
}
|
||||
|
||||
// Fs returns read only access to the Fs that this object is part of
|
||||
func (o *Object) Fs() fs.Info {
|
||||
return o.f
|
||||
}
|
||||
|
||||
// Return a string version
|
||||
func (o *Object) String() string {
|
||||
if o == nil {
|
||||
return "<nil>"
|
||||
}
|
||||
return o.Remote()
|
||||
}
|
||||
|
||||
// Remote returns the remote path
|
||||
func (o *Object) Remote() string {
|
||||
remote := o.Object.Remote()
|
||||
decryptedName, err := o.f.cipher.DecryptFileName(remote)
|
||||
if err != nil {
|
||||
fs.Debug(remote, "Undecryptable file name: %v", err)
|
||||
return remote
|
||||
}
|
||||
return decryptedName
|
||||
}
|
||||
|
||||
// Size returns the size of the file
|
||||
func (o *Object) Size() int64 {
|
||||
size, err := o.f.cipher.DecryptedSize(o.Object.Size())
|
||||
if err != nil {
|
||||
fs.Debug(o, "Bad size for decrypt: %v", err)
|
||||
}
|
||||
return size
|
||||
}
|
||||
|
||||
// Hash returns the selected checksum of the file
|
||||
// If no checksum is available it returns ""
|
||||
func (o *Object) Hash(hash fs.HashType) (string, error) {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
// Open opens the file for read. Call Close() on the returned io.ReadCloser
|
||||
func (o *Object) Open() (io.ReadCloser, error) {
|
||||
in, err := o.Object.Open()
|
||||
if err != nil {
|
||||
return in, err
|
||||
}
|
||||
return o.f.cipher.DecryptData(in)
|
||||
}
|
||||
|
||||
// Update in to the object with the modTime given of the given size
|
||||
func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
|
||||
wrappedIn, err := o.f.cipher.EncryptData(in)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return o.Object.Update(wrappedIn, o.f.newObjectInfo(src))
|
||||
}
|
||||
|
||||
// newDir returns a dir with the Name decrypted
|
||||
func (f *Fs) newDir(dir *fs.Dir) *fs.Dir {
|
||||
new := *dir
|
||||
remote := dir.Name
|
||||
decryptedRemote, err := f.cipher.DecryptDirName(remote)
|
||||
if err != nil {
|
||||
fs.Debug(remote, "Undecryptable dir name: %v", err)
|
||||
} else {
|
||||
new.Name = decryptedRemote
|
||||
}
|
||||
return &new
|
||||
}
|
||||
|
||||
// ObjectInfo describes a wrapped fs.ObjectInfo for being the source
|
||||
//
|
||||
// This encrypts the remote name and adjusts the size
|
||||
type ObjectInfo struct {
|
||||
fs.ObjectInfo
|
||||
f *Fs
|
||||
}
|
||||
|
||||
func (f *Fs) newObjectInfo(src fs.ObjectInfo) *ObjectInfo {
|
||||
return &ObjectInfo{
|
||||
ObjectInfo: src,
|
||||
f: f,
|
||||
}
|
||||
}
|
||||
|
||||
// Fs returns read only access to the Fs that this object is part of
|
||||
func (o *ObjectInfo) Fs() fs.Info {
|
||||
return o.f
|
||||
}
|
||||
|
||||
// Remote returns the remote path
|
||||
func (o *ObjectInfo) Remote() string {
|
||||
return o.f.cipher.EncryptFileName(o.ObjectInfo.Remote())
|
||||
}
|
||||
|
||||
// Size returns the size of the file
|
||||
func (o *ObjectInfo) Size() int64 {
|
||||
return o.f.cipher.EncryptedSize(o.ObjectInfo.Size())
|
||||
}
|
||||
|
||||
// ListOpts wraps a listopts decrypting the directory listing and
|
||||
// replacing the Objects
|
||||
type ListOpts struct {
|
||||
fs.ListOpts
|
||||
f *Fs
|
||||
dir string // dir we are listing
|
||||
mu sync.Mutex // to protect dirs
|
||||
dirs map[string]struct{} // keep track of synthetic directory objects added
|
||||
}
|
||||
|
||||
// Make a ListOpts wrapper
|
||||
func (f *Fs) newListOpts(lo fs.ListOpts, dir string) *ListOpts {
|
||||
if dir != "" {
|
||||
dir += "/"
|
||||
}
|
||||
return &ListOpts{
|
||||
ListOpts: lo,
|
||||
f: f,
|
||||
dir: dir,
|
||||
dirs: make(map[string]struct{}),
|
||||
}
|
||||
|
||||
}
|
||||
|
||||
// Level gets the recursion level for this listing.
|
||||
//
|
||||
// Fses may ignore this, but should implement it for improved efficiency if possible.
|
||||
//
|
||||
// Level 1 means list just the contents of the directory
|
||||
//
|
||||
// Each returned item must have less than level `/`s in.
|
||||
func (lo *ListOpts) Level() int {
|
||||
return lo.ListOpts.Level()
|
||||
}
|
||||
|
||||
// Add an object to the output.
|
||||
// If the function returns true, the operation has been aborted.
|
||||
// Multiple goroutines can safely add objects concurrently.
|
||||
func (lo *ListOpts) Add(obj fs.Object) (abort bool) {
|
||||
remote := obj.Remote()
|
||||
_, err := lo.f.cipher.DecryptFileName(remote)
|
||||
if err != nil {
|
||||
fs.Debug(remote, "Skipping undecryptable file name: %v", err)
|
||||
return lo.ListOpts.IsFinished()
|
||||
}
|
||||
return lo.ListOpts.Add(lo.f.newObject(obj))
|
||||
}
|
||||
|
||||
// AddDir adds a directory to the output.
|
||||
// If the function returns true, the operation has been aborted.
|
||||
// Multiple goroutines can safely add objects concurrently.
|
||||
func (lo *ListOpts) AddDir(dir *fs.Dir) (abort bool) {
|
||||
remote := dir.Name
|
||||
_, err := lo.f.cipher.DecryptDirName(remote)
|
||||
if err != nil {
|
||||
fs.Debug(remote, "Skipping undecryptable dir name: %v", err)
|
||||
return lo.ListOpts.IsFinished()
|
||||
}
|
||||
return lo.ListOpts.AddDir(lo.f.newDir(dir))
|
||||
}
|
||||
|
||||
// IncludeDirectory returns whether this directory should be
|
||||
// included in the listing (and recursed into or not).
|
||||
func (lo *ListOpts) IncludeDirectory(remote string) bool {
|
||||
decryptedRemote, err := lo.f.cipher.DecryptDirName(remote)
|
||||
if err != nil {
|
||||
fs.Debug(remote, "Not including undecryptable directory name: %v", err)
|
||||
return false
|
||||
}
|
||||
return lo.ListOpts.IncludeDirectory(decryptedRemote)
|
||||
}
|
||||
|
||||
// Check the interfaces are satisfied
|
||||
var (
|
||||
_ fs.Fs = (*Fs)(nil)
|
||||
_ fs.Purger = (*Fs)(nil)
|
||||
_ fs.Copier = (*Fs)(nil)
|
||||
_ fs.Mover = (*Fs)(nil)
|
||||
_ fs.DirMover = (*Fs)(nil)
|
||||
// _ fs.PutUncheckeder = (*Fs)(nil)
|
||||
_ fs.UnWrapper = (*Fs)(nil)
|
||||
_ fs.ObjectInfo = (*ObjectInfo)(nil)
|
||||
_ fs.Object = (*Object)(nil)
|
||||
_ fs.ListOpts = (*ListOpts)(nil)
|
||||
)
|
||||
@@ -1,59 +0,0 @@
|
||||
// Test Crypt filesystem interface
|
||||
//
|
||||
// Automatically generated - DO NOT EDIT
|
||||
// Regenerate with: make gen_tests
|
||||
package crypt_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/crypt"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/fstest/fstests"
|
||||
_ "github.com/ncw/rclone/local"
|
||||
)
|
||||
|
||||
func TestSetup2(t *testing.T) {
|
||||
fstests.NilObject = fs.Object((*crypt.Object)(nil))
|
||||
fstests.RemoteName = "TestCrypt2:"
|
||||
}
|
||||
|
||||
// Generic tests for the Fs
|
||||
func TestInit2(t *testing.T) { fstests.TestInit(t) }
|
||||
func TestFsString2(t *testing.T) { fstests.TestFsString(t) }
|
||||
func TestFsRmdirEmpty2(t *testing.T) { fstests.TestFsRmdirEmpty(t) }
|
||||
func TestFsRmdirNotFound2(t *testing.T) { fstests.TestFsRmdirNotFound(t) }
|
||||
func TestFsMkdir2(t *testing.T) { fstests.TestFsMkdir(t) }
|
||||
func TestFsListEmpty2(t *testing.T) { fstests.TestFsListEmpty(t) }
|
||||
func TestFsListDirEmpty2(t *testing.T) { fstests.TestFsListDirEmpty(t) }
|
||||
func TestFsNewObjectNotFound2(t *testing.T) { fstests.TestFsNewObjectNotFound(t) }
|
||||
func TestFsPutFile12(t *testing.T) { fstests.TestFsPutFile1(t) }
|
||||
func TestFsPutFile22(t *testing.T) { fstests.TestFsPutFile2(t) }
|
||||
func TestFsUpdateFile12(t *testing.T) { fstests.TestFsUpdateFile1(t) }
|
||||
func TestFsListDirFile22(t *testing.T) { fstests.TestFsListDirFile2(t) }
|
||||
func TestFsListDirRoot2(t *testing.T) { fstests.TestFsListDirRoot(t) }
|
||||
func TestFsListSubdir2(t *testing.T) { fstests.TestFsListSubdir(t) }
|
||||
func TestFsListLevel22(t *testing.T) { fstests.TestFsListLevel2(t) }
|
||||
func TestFsListFile12(t *testing.T) { fstests.TestFsListFile1(t) }
|
||||
func TestFsNewObject2(t *testing.T) { fstests.TestFsNewObject(t) }
|
||||
func TestFsListFile1and22(t *testing.T) { fstests.TestFsListFile1and2(t) }
|
||||
func TestFsCopy2(t *testing.T) { fstests.TestFsCopy(t) }
|
||||
func TestFsMove2(t *testing.T) { fstests.TestFsMove(t) }
|
||||
func TestFsDirMove2(t *testing.T) { fstests.TestFsDirMove(t) }
|
||||
func TestFsRmdirFull2(t *testing.T) { fstests.TestFsRmdirFull(t) }
|
||||
func TestFsPrecision2(t *testing.T) { fstests.TestFsPrecision(t) }
|
||||
func TestObjectString2(t *testing.T) { fstests.TestObjectString(t) }
|
||||
func TestObjectFs2(t *testing.T) { fstests.TestObjectFs(t) }
|
||||
func TestObjectRemote2(t *testing.T) { fstests.TestObjectRemote(t) }
|
||||
func TestObjectHashes2(t *testing.T) { fstests.TestObjectHashes(t) }
|
||||
func TestObjectModTime2(t *testing.T) { fstests.TestObjectModTime(t) }
|
||||
func TestObjectSetModTime2(t *testing.T) { fstests.TestObjectSetModTime(t) }
|
||||
func TestObjectSize2(t *testing.T) { fstests.TestObjectSize(t) }
|
||||
func TestObjectOpen2(t *testing.T) { fstests.TestObjectOpen(t) }
|
||||
func TestObjectUpdate2(t *testing.T) { fstests.TestObjectUpdate(t) }
|
||||
func TestObjectStorable2(t *testing.T) { fstests.TestObjectStorable(t) }
|
||||
func TestFsIsFile2(t *testing.T) { fstests.TestFsIsFile(t) }
|
||||
func TestFsIsFileNotFound2(t *testing.T) { fstests.TestFsIsFileNotFound(t) }
|
||||
func TestObjectRemove2(t *testing.T) { fstests.TestObjectRemove(t) }
|
||||
func TestObjectPurge2(t *testing.T) { fstests.TestObjectPurge(t) }
|
||||
func TestFinalise2(t *testing.T) { fstests.TestFinalise(t) }
|
||||
@@ -1,27 +0,0 @@
|
||||
package crypt_test
|
||||
|
||||
import (
|
||||
"os"
|
||||
"path/filepath"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/fstest/fstests"
|
||||
)
|
||||
|
||||
// Create the TestCrypt: remote
|
||||
func init() {
|
||||
tempdir := filepath.Join(os.TempDir(), "rclone-crypt-test-standard")
|
||||
name := "TestCrypt"
|
||||
tempdir2 := filepath.Join(os.TempDir(), "rclone-crypt-test-off")
|
||||
name2 := name + "2"
|
||||
fstests.ExtraConfig = []fstests.ExtraConfigItem{
|
||||
{Name: name, Key: "type", Value: "crypt"},
|
||||
{Name: name, Key: "remote", Value: tempdir},
|
||||
{Name: name, Key: "password", Value: fs.MustObscure("potato")},
|
||||
{Name: name, Key: "filename_encryption", Value: "standard"},
|
||||
{Name: name2, Key: "type", Value: "crypt"},
|
||||
{Name: name2, Key: "remote", Value: tempdir2},
|
||||
{Name: name2, Key: "password", Value: fs.MustObscure("potato2")},
|
||||
{Name: name2, Key: "filename_encryption", Value: "off"},
|
||||
}
|
||||
}
|
||||
@@ -1,59 +0,0 @@
|
||||
// Test Crypt filesystem interface
|
||||
//
|
||||
// Automatically generated - DO NOT EDIT
|
||||
// Regenerate with: make gen_tests
|
||||
package crypt_test
|
||||
|
||||
import (
|
||||
"testing"
|
||||
|
||||
"github.com/ncw/rclone/crypt"
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/ncw/rclone/fstest/fstests"
|
||||
_ "github.com/ncw/rclone/local"
|
||||
)
|
||||
|
||||
func TestSetup(t *testing.T) {
|
||||
fstests.NilObject = fs.Object((*crypt.Object)(nil))
|
||||
fstests.RemoteName = "TestCrypt:"
|
||||
}
|
||||
|
||||
// Generic tests for the Fs
|
||||
func TestInit(t *testing.T) { fstests.TestInit(t) }
|
||||
func TestFsString(t *testing.T) { fstests.TestFsString(t) }
|
||||
func TestFsRmdirEmpty(t *testing.T) { fstests.TestFsRmdirEmpty(t) }
|
||||
func TestFsRmdirNotFound(t *testing.T) { fstests.TestFsRmdirNotFound(t) }
|
||||
func TestFsMkdir(t *testing.T) { fstests.TestFsMkdir(t) }
|
||||
func TestFsListEmpty(t *testing.T) { fstests.TestFsListEmpty(t) }
|
||||
func TestFsListDirEmpty(t *testing.T) { fstests.TestFsListDirEmpty(t) }
|
||||
func TestFsNewObjectNotFound(t *testing.T) { fstests.TestFsNewObjectNotFound(t) }
|
||||
func TestFsPutFile1(t *testing.T) { fstests.TestFsPutFile1(t) }
|
||||
func TestFsPutFile2(t *testing.T) { fstests.TestFsPutFile2(t) }
|
||||
func TestFsUpdateFile1(t *testing.T) { fstests.TestFsUpdateFile1(t) }
|
||||
func TestFsListDirFile2(t *testing.T) { fstests.TestFsListDirFile2(t) }
|
||||
func TestFsListDirRoot(t *testing.T) { fstests.TestFsListDirRoot(t) }
|
||||
func TestFsListSubdir(t *testing.T) { fstests.TestFsListSubdir(t) }
|
||||
func TestFsListLevel2(t *testing.T) { fstests.TestFsListLevel2(t) }
|
||||
func TestFsListFile1(t *testing.T) { fstests.TestFsListFile1(t) }
|
||||
func TestFsNewObject(t *testing.T) { fstests.TestFsNewObject(t) }
|
||||
func TestFsListFile1and2(t *testing.T) { fstests.TestFsListFile1and2(t) }
|
||||
func TestFsCopy(t *testing.T) { fstests.TestFsCopy(t) }
|
||||
func TestFsMove(t *testing.T) { fstests.TestFsMove(t) }
|
||||
func TestFsDirMove(t *testing.T) { fstests.TestFsDirMove(t) }
|
||||
func TestFsRmdirFull(t *testing.T) { fstests.TestFsRmdirFull(t) }
|
||||
func TestFsPrecision(t *testing.T) { fstests.TestFsPrecision(t) }
|
||||
func TestObjectString(t *testing.T) { fstests.TestObjectString(t) }
|
||||
func TestObjectFs(t *testing.T) { fstests.TestObjectFs(t) }
|
||||
func TestObjectRemote(t *testing.T) { fstests.TestObjectRemote(t) }
|
||||
func TestObjectHashes(t *testing.T) { fstests.TestObjectHashes(t) }
|
||||
func TestObjectModTime(t *testing.T) { fstests.TestObjectModTime(t) }
|
||||
func TestObjectSetModTime(t *testing.T) { fstests.TestObjectSetModTime(t) }
|
||||
func TestObjectSize(t *testing.T) { fstests.TestObjectSize(t) }
|
||||
func TestObjectOpen(t *testing.T) { fstests.TestObjectOpen(t) }
|
||||
func TestObjectUpdate(t *testing.T) { fstests.TestObjectUpdate(t) }
|
||||
func TestObjectStorable(t *testing.T) { fstests.TestObjectStorable(t) }
|
||||
func TestFsIsFile(t *testing.T) { fstests.TestFsIsFile(t) }
|
||||
func TestFsIsFileNotFound(t *testing.T) { fstests.TestFsIsFileNotFound(t) }
|
||||
func TestObjectRemove(t *testing.T) { fstests.TestObjectRemove(t) }
|
||||
func TestObjectPurge(t *testing.T) { fstests.TestObjectPurge(t) }
|
||||
func TestFinalise(t *testing.T) { fstests.TestFinalise(t) }
|
||||
@@ -1,63 +0,0 @@
|
||||
// Package pkcs7 implements PKCS#7 padding
|
||||
//
|
||||
// This is a standard way of encoding variable length buffers into
|
||||
// buffers which are a multiple of an underlying crypto block size.
|
||||
package pkcs7
|
||||
|
||||
import "github.com/pkg/errors"
|
||||
|
||||
// Errors Unpad can return
|
||||
var (
|
||||
ErrorPaddingNotFound = errors.New("Bad PKCS#7 padding - not padded")
|
||||
ErrorPaddingNotAMultiple = errors.New("Bad PKCS#7 padding - not a multiple of blocksize")
|
||||
ErrorPaddingTooLong = errors.New("Bad PKCS#7 padding - too long")
|
||||
ErrorPaddingTooShort = errors.New("Bad PKCS#7 padding - too short")
|
||||
ErrorPaddingNotAllTheSame = errors.New("Bad PKCS#7 padding - not all the same")
|
||||
)
|
||||
|
||||
// Pad buf using PKCS#7 to a multiple of n.
|
||||
//
|
||||
// Appends the padding to buf - make a copy of it first if you don't
|
||||
// want it modified.
|
||||
func Pad(n int, buf []byte) []byte {
|
||||
if n <= 1 || n >= 256 {
|
||||
panic("bad multiple")
|
||||
}
|
||||
length := len(buf)
|
||||
padding := n - (length % n)
|
||||
for i := 0; i < padding; i++ {
|
||||
buf = append(buf, byte(padding))
|
||||
}
|
||||
if (len(buf) % n) != 0 {
|
||||
panic("padding failed")
|
||||
}
|
||||
return buf
|
||||
}
|
||||
|
||||
// Unpad buf using PKCS#7 from a multiple of n returning a slice of
|
||||
// buf or an error if malformed.
|
||||
func Unpad(n int, buf []byte) ([]byte, error) {
|
||||
if n <= 1 || n >= 256 {
|
||||
panic("bad multiple")
|
||||
}
|
||||
length := len(buf)
|
||||
if length == 0 {
|
||||
return nil, ErrorPaddingNotFound
|
||||
}
|
||||
if (length % n) != 0 {
|
||||
return nil, ErrorPaddingNotAMultiple
|
||||
}
|
||||
padding := int(buf[length-1])
|
||||
if padding > n {
|
||||
return nil, ErrorPaddingTooLong
|
||||
}
|
||||
if padding == 0 {
|
||||
return nil, ErrorPaddingTooShort
|
||||
}
|
||||
for i := 0; i < padding; i++ {
|
||||
if buf[length-1-i] != byte(padding) {
|
||||
return nil, ErrorPaddingNotAllTheSame
|
||||
}
|
||||
}
|
||||
return buf[:length-padding], nil
|
||||
}
|
||||
@@ -1,73 +0,0 @@
|
||||
package pkcs7
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"testing"
|
||||
|
||||
"github.com/stretchr/testify/assert"
|
||||
)
|
||||
|
||||
func TestPad(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
n int
|
||||
in string
|
||||
expected string
|
||||
}{
|
||||
{8, "", "\x08\x08\x08\x08\x08\x08\x08\x08"},
|
||||
{8, "1", "1\x07\x07\x07\x07\x07\x07\x07"},
|
||||
{8, "12", "12\x06\x06\x06\x06\x06\x06"},
|
||||
{8, "123", "123\x05\x05\x05\x05\x05"},
|
||||
{8, "1234", "1234\x04\x04\x04\x04"},
|
||||
{8, "12345", "12345\x03\x03\x03"},
|
||||
{8, "123456", "123456\x02\x02"},
|
||||
{8, "1234567", "1234567\x01"},
|
||||
{8, "abcdefgh", "abcdefgh\x08\x08\x08\x08\x08\x08\x08\x08"},
|
||||
{8, "abcdefgh1", "abcdefgh1\x07\x07\x07\x07\x07\x07\x07"},
|
||||
{8, "abcdefgh12", "abcdefgh12\x06\x06\x06\x06\x06\x06"},
|
||||
{8, "abcdefgh123", "abcdefgh123\x05\x05\x05\x05\x05"},
|
||||
{8, "abcdefgh1234", "abcdefgh1234\x04\x04\x04\x04"},
|
||||
{8, "abcdefgh12345", "abcdefgh12345\x03\x03\x03"},
|
||||
{8, "abcdefgh123456", "abcdefgh123456\x02\x02"},
|
||||
{8, "abcdefgh1234567", "abcdefgh1234567\x01"},
|
||||
{8, "abcdefgh12345678", "abcdefgh12345678\x08\x08\x08\x08\x08\x08\x08\x08"},
|
||||
{16, "", "\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10\x10"},
|
||||
{16, "a", "a\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f\x0f"},
|
||||
} {
|
||||
actual := Pad(test.n, []byte(test.in))
|
||||
assert.Equal(t, test.expected, string(actual), fmt.Sprintf("Pad %d %q", test.n, test.in))
|
||||
recovered, err := Unpad(test.n, actual)
|
||||
assert.NoError(t, err)
|
||||
assert.Equal(t, []byte(test.in), recovered, fmt.Sprintf("Unpad %d %q", test.n, test.in))
|
||||
}
|
||||
assert.Panics(t, func() { Pad(1, []byte("")) }, "bad multiple")
|
||||
assert.Panics(t, func() { Pad(256, []byte("")) }, "bad multiple")
|
||||
}
|
||||
|
||||
func TestUnpad(t *testing.T) {
|
||||
// We've tested the OK decoding in TestPad, now test the error cases
|
||||
for _, test := range []struct {
|
||||
n int
|
||||
in string
|
||||
err error
|
||||
}{
|
||||
{8, "", ErrorPaddingNotFound},
|
||||
{8, "1", ErrorPaddingNotAMultiple},
|
||||
{8, "12", ErrorPaddingNotAMultiple},
|
||||
{8, "123", ErrorPaddingNotAMultiple},
|
||||
{8, "1234", ErrorPaddingNotAMultiple},
|
||||
{8, "12345", ErrorPaddingNotAMultiple},
|
||||
{8, "123456", ErrorPaddingNotAMultiple},
|
||||
{8, "1234567", ErrorPaddingNotAMultiple},
|
||||
{8, "1234567\xFF", ErrorPaddingTooLong},
|
||||
{8, "1234567\x09", ErrorPaddingTooLong},
|
||||
{8, "1234567\x00", ErrorPaddingTooShort},
|
||||
{8, "123456\x01\x02", ErrorPaddingNotAllTheSame},
|
||||
{8, "\x07\x08\x08\x08\x08\x08\x08\x08", ErrorPaddingNotAllTheSame},
|
||||
} {
|
||||
result, actualErr := Unpad(test.n, []byte(test.in))
|
||||
assert.Equal(t, test.err, actualErr, fmt.Sprintf("Unpad %d %q", test.n, test.in))
|
||||
assert.Equal(t, result, []byte(nil))
|
||||
}
|
||||
assert.Panics(t, func() { _, _ = Unpad(1, []byte("")) }, "bad multiple")
|
||||
assert.Panics(t, func() { _, _ = Unpad(256, []byte("")) }, "bad multiple")
|
||||
}
|
||||
@@ -1,267 +0,0 @@
|
||||
// Package dircache provides a simple cache for caching directory to path lookups
|
||||
package dircache
|
||||
|
||||
// _methods are called without the lock
|
||||
|
||||
import (
|
||||
"log"
|
||||
"strings"
|
||||
"sync"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
"github.com/pkg/errors"
|
||||
)
|
||||
|
||||
// DirCache caches paths to directory IDs and vice versa
|
||||
type DirCache struct {
|
||||
cacheMu sync.RWMutex
|
||||
cache map[string]string
|
||||
invCache map[string]string
|
||||
mu sync.Mutex
|
||||
fs DirCacher // Interface to find and make stuff
|
||||
trueRootID string // ID of the absolute root
|
||||
root string // the path we are working on
|
||||
rootID string // ID of the root directory
|
||||
rootParentID string // ID of the root's parent directory
|
||||
foundRoot bool // Whether we have found the root or not
|
||||
}
|
||||
|
||||
// DirCacher describes an interface for doing the low level directory work
|
||||
type DirCacher interface {
|
||||
FindLeaf(pathID, leaf string) (pathIDOut string, found bool, err error)
|
||||
CreateDir(pathID, leaf string) (newID string, err error)
|
||||
}
|
||||
|
||||
// New makes a DirCache
|
||||
//
|
||||
// The cache is safe for concurrent use
|
||||
func New(root string, trueRootID string, fs DirCacher) *DirCache {
|
||||
d := &DirCache{
|
||||
trueRootID: trueRootID,
|
||||
root: root,
|
||||
fs: fs,
|
||||
}
|
||||
d.Flush()
|
||||
d.ResetRoot()
|
||||
return d
|
||||
}
|
||||
|
||||
// Get an ID given a path
|
||||
func (dc *DirCache) Get(path string) (id string, ok bool) {
|
||||
dc.cacheMu.RLock()
|
||||
id, ok = dc.cache[path]
|
||||
dc.cacheMu.RUnlock()
|
||||
return
|
||||
}
|
||||
|
||||
// GetInv gets a path given an ID
|
||||
func (dc *DirCache) GetInv(id string) (path string, ok bool) {
|
||||
dc.cacheMu.RLock()
|
||||
path, ok = dc.invCache[id]
|
||||
dc.cacheMu.RUnlock()
|
||||
return
|
||||
}
|
||||
|
||||
// Put a path, id into the map
|
||||
func (dc *DirCache) Put(path, id string) {
|
||||
dc.cacheMu.Lock()
|
||||
dc.cache[path] = id
|
||||
dc.invCache[id] = path
|
||||
dc.cacheMu.Unlock()
|
||||
}
|
||||
|
||||
// Flush the map of all data
|
||||
func (dc *DirCache) Flush() {
|
||||
dc.cacheMu.Lock()
|
||||
dc.cache = make(map[string]string)
|
||||
dc.invCache = make(map[string]string)
|
||||
dc.cacheMu.Unlock()
|
||||
}
|
||||
|
||||
// SplitPath splits a path into directory, leaf
|
||||
//
|
||||
// Path shouldn't start or end with a /
|
||||
//
|
||||
// If there are no slashes then directory will be "" and leaf = path
|
||||
func SplitPath(path string) (directory, leaf string) {
|
||||
lastSlash := strings.LastIndex(path, "/")
|
||||
if lastSlash >= 0 {
|
||||
directory = path[:lastSlash]
|
||||
leaf = path[lastSlash+1:]
|
||||
} else {
|
||||
directory = ""
|
||||
leaf = path
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
// FindDir finds the directory passed in returning the directory ID
|
||||
// starting from pathID
|
||||
//
|
||||
// Path shouldn't start or end with a /
|
||||
//
|
||||
// If create is set it will make the directory if not found
|
||||
//
|
||||
// Algorithm:
|
||||
// Look in the cache for the path, if found return the pathID
|
||||
// If not found strip the last path off the path and recurse
|
||||
// Now have a parent directory id, so look in the parent for self and return it
|
||||
func (dc *DirCache) FindDir(path string, create bool) (pathID string, err error) {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
return dc._findDir(path, create)
|
||||
}
|
||||
|
||||
// Look for the root and in the cache - safe to call without the mu
|
||||
func (dc *DirCache) _findDirInCache(path string) string {
|
||||
// fmt.Println("Finding",path,"create",create,"cache",cache)
|
||||
// If it is the root, then return it
|
||||
if path == "" {
|
||||
// fmt.Println("Root")
|
||||
return dc.rootID
|
||||
}
|
||||
|
||||
// If it is in the cache then return it
|
||||
pathID, ok := dc.Get(path)
|
||||
if ok {
|
||||
// fmt.Println("Cache hit on", path)
|
||||
return pathID
|
||||
}
|
||||
|
||||
return ""
|
||||
}
|
||||
|
||||
// Unlocked findDir - must have mu
|
||||
func (dc *DirCache) _findDir(path string, create bool) (pathID string, err error) {
|
||||
pathID = dc._findDirInCache(path)
|
||||
if pathID != "" {
|
||||
return pathID, nil
|
||||
}
|
||||
|
||||
// Split the path into directory, leaf
|
||||
directory, leaf := SplitPath(path)
|
||||
|
||||
// Recurse and find pathID for parent directory
|
||||
parentPathID, err := dc._findDir(directory, create)
|
||||
if err != nil {
|
||||
return "", err
|
||||
|
||||
}
|
||||
|
||||
// Find the leaf in parentPathID
|
||||
pathID, found, err := dc.fs.FindLeaf(parentPathID, leaf)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
|
||||
// If not found create the directory if required or return an error
|
||||
if !found {
|
||||
if create {
|
||||
pathID, err = dc.fs.CreateDir(parentPathID, leaf)
|
||||
if err != nil {
|
||||
return "", errors.Wrap(err, "failed to make directory")
|
||||
}
|
||||
} else {
|
||||
return "", fs.ErrorDirNotFound
|
||||
}
|
||||
}
|
||||
|
||||
// Store the leaf directory in the cache
|
||||
dc.Put(path, pathID)
|
||||
|
||||
// fmt.Println("Dir", path, "is", pathID)
|
||||
return pathID, nil
|
||||
}
|
||||
|
||||
// FindPath finds the leaf and directoryID from a path
|
||||
//
|
||||
// If create is set parent directories will be created if they don't exist
|
||||
func (dc *DirCache) FindPath(path string, create bool) (leaf, directoryID string, err error) {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
directory, leaf := SplitPath(path)
|
||||
directoryID, err = dc._findDir(directory, create)
|
||||
return
|
||||
}
|
||||
|
||||
// FindRoot finds the root directory if not already found
|
||||
//
|
||||
// Resets the root directory
|
||||
//
|
||||
// If create is set it will make the directory if not found
|
||||
func (dc *DirCache) FindRoot(create bool) error {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
if dc.foundRoot {
|
||||
return nil
|
||||
}
|
||||
rootID, err := dc._findDir(dc.root, create)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
dc.foundRoot = true
|
||||
dc.rootID = rootID
|
||||
|
||||
// Find the parent of the root while we still have the root
|
||||
// directory tree cached
|
||||
rootParentPath, _ := SplitPath(dc.root)
|
||||
dc.rootParentID, _ = dc.Get(rootParentPath)
|
||||
|
||||
// Reset the tree based on dc.root
|
||||
dc.Flush()
|
||||
// Put the root directory in
|
||||
dc.Put("", dc.rootID)
|
||||
return nil
|
||||
}
|
||||
|
||||
// FoundRoot returns whether the root directory has been found yet
|
||||
//
|
||||
// Call this from FindLeaf or CreateDir only
|
||||
func (dc *DirCache) FoundRoot() bool {
|
||||
return dc.foundRoot
|
||||
}
|
||||
|
||||
// RootID returns the ID of the root directory
|
||||
//
|
||||
// This should be called after FindRoot
|
||||
func (dc *DirCache) RootID() string {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
if !dc.foundRoot {
|
||||
log.Fatalf("Internal Error: RootID() called before FindRoot")
|
||||
}
|
||||
return dc.rootID
|
||||
}
|
||||
|
||||
// RootParentID returns the ID of the parent of the root directory
|
||||
//
|
||||
// This should be called after FindRoot
|
||||
func (dc *DirCache) RootParentID() (string, error) {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
if !dc.foundRoot {
|
||||
return "", errors.New("internal error: RootID() called before FindRoot")
|
||||
}
|
||||
if dc.rootParentID == "" {
|
||||
return "", errors.New("internal error: didn't find rootParentID")
|
||||
}
|
||||
if dc.rootID == dc.trueRootID {
|
||||
return "", errors.New("is root directory")
|
||||
}
|
||||
return dc.rootParentID, nil
|
||||
}
|
||||
|
||||
// ResetRoot resets the root directory to the absolute root and clears
|
||||
// the DirCache
|
||||
func (dc *DirCache) ResetRoot() {
|
||||
dc.mu.Lock()
|
||||
defer dc.mu.Unlock()
|
||||
dc.foundRoot = false
|
||||
dc.Flush()
|
||||
|
||||
// Put the true root in
|
||||
dc.rootID = dc.trueRootID
|
||||
|
||||
// Put the root directory in
|
||||
dc.Put("", dc.rootID)
|
||||
}
|
||||
@@ -1,82 +0,0 @@
|
||||
// Listing utility functions for fses which use dircache
|
||||
|
||||
package dircache
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/ncw/rclone/fs"
|
||||
)
|
||||
|
||||
// ListDirJob describe a directory listing that needs to be done
|
||||
type ListDirJob struct {
|
||||
DirID string
|
||||
Path string
|
||||
Depth int
|
||||
}
|
||||
|
||||
// ListDirer describes the interface necessary to use ListDir
|
||||
type ListDirer interface {
|
||||
// ListDir reads the directory specified by the job into out, returning any more jobs
|
||||
ListDir(out fs.ListOpts, job ListDirJob) (jobs []ListDirJob, err error)
|
||||
}
|
||||
|
||||
// listDir lists the directory using a recursive list from the root
|
||||
//
|
||||
// It does this in parallel, calling f.ListDir to do the actual reading
|
||||
func listDir(f ListDirer, out fs.ListOpts, dirID string, path string) {
|
||||
// Start some directory listing go routines
|
||||
var wg sync.WaitGroup // sync closing of go routines
|
||||
var traversing sync.WaitGroup // running directory traversals
|
||||
buffer := out.Buffer()
|
||||
in := make(chan ListDirJob, buffer)
|
||||
for i := 0; i < buffer; i++ {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for job := range in {
|
||||
jobs, err := f.ListDir(out, job)
|
||||
if err != nil {
|
||||
out.SetError(err)
|
||||
fs.Debug(f, "Error reading %s: %s", path, err)
|
||||
} else {
|
||||
traversing.Add(len(jobs))
|
||||
go func() {
|
||||
// Now we have traversed this directory, send these
|
||||
// jobs off for traversal in the background
|
||||
for _, job := range jobs {
|
||||
in <- job
|
||||
}
|
||||
}()
|
||||
}
|
||||
traversing.Done()
|
||||
}
|
||||
}()
|
||||
}
|
||||
|
||||
// Start the process
|
||||
traversing.Add(1)
|
||||
in <- ListDirJob{DirID: dirID, Path: path, Depth: out.Level() - 1}
|
||||
traversing.Wait()
|
||||
close(in)
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
// List walks the path returning iles and directories into out
|
||||
func (dc *DirCache) List(f ListDirer, out fs.ListOpts, dir string) {
|
||||
defer out.Finished()
|
||||
err := dc.FindRoot(false)
|
||||
if err != nil {
|
||||
out.SetError(err)
|
||||
return
|
||||
}
|
||||
id, err := dc.FindDir(dir, false)
|
||||
if err != nil {
|
||||
out.SetError(err)
|
||||
return
|
||||
}
|
||||
if dir != "" {
|
||||
dir += "/"
|
||||
}
|
||||
listDir(f, out, id, dir)
|
||||
}
|
||||
@@ -1,15 +1,11 @@
|
||||
{
|
||||
"indexes": {
|
||||
"tag": "tags",
|
||||
"group": "groups",
|
||||
"menu": "menu"
|
||||
},
|
||||
"baseurl": "http://rclone.org",
|
||||
"title": "rclone - rsync for cloud storage",
|
||||
"description": "rclone - rsync for cloud storage: google drive, s3, swift, cloudfiles, dropbox, memstore...",
|
||||
"canonifyurls": true,
|
||||
"blackfriday": {
|
||||
"smartDashes": false,
|
||||
"plainIDAnchors": true
|
||||
}
|
||||
}
|
||||
{
|
||||
"indexes": {
|
||||
"tag": "tags",
|
||||
"group": "groups",
|
||||
"menu": "menu"
|
||||
},
|
||||
"baseurl": "http://rclone.org",
|
||||
"title": "rclone - rsync for cloud storage",
|
||||
"description": "rclone - rsync for cloud storage: google drive, s3, swift, cloudfiles, dropbox, memstore...",
|
||||
"canonifyurls": true
|
||||
}
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
---
|
||||
title: "Rclone"
|
||||
description: "rclone syncs files to and from Google Drive, S3, Swift, Cloudfiles, Dropbox, Google Cloud Storage and Amazon Drive."
|
||||
description: "rclone syncs files to and from Google Drive, S3, Swift, Cloudfiles, Dropbox and Google Cloud Storage."
|
||||
type: page
|
||||
date: "2015-09-06"
|
||||
date: "2014-07-17"
|
||||
groups: ["about"]
|
||||
---
|
||||
|
||||
@@ -18,22 +18,17 @@ Rclone is a command line program to sync files and directories to and from
|
||||
* Openstack Swift / Rackspace cloud files / Memset Memstore
|
||||
* Dropbox
|
||||
* Google Cloud Storage
|
||||
* Amazon Drive
|
||||
* Microsoft One Drive
|
||||
* Hubic
|
||||
* Backblaze B2
|
||||
* Yandex Disk
|
||||
* The local filesystem
|
||||
|
||||
Features
|
||||
|
||||
* MD5/SHA1 hashes checked at all times for file integrity
|
||||
* MD5SUMs checked at all times for file integrity
|
||||
* Timestamps preserved on files
|
||||
* Partial syncs supported on a whole file basis
|
||||
* Copy mode to just copy new/changed files
|
||||
* Sync (one way) mode to make a directory identical
|
||||
* Check mode to check for file hash equality
|
||||
* Can sync to and from network, eg two different cloud accounts
|
||||
* Sync mode to make a directory identical
|
||||
* Check mode to check all MD5SUMs
|
||||
* Can sync to and from network, eg two different Drive accounts
|
||||
|
||||
Links
|
||||
|
||||
|
||||
@@ -1,158 +0,0 @@
|
||||
---
|
||||
title: "Amazon Drive"
|
||||
description: "Rclone docs for Amazon Drive"
|
||||
date: "2016-07-11"
|
||||
---
|
||||
|
||||
<i class="fa fa-amazon"></i> Amazon Drive
|
||||
-----------------------------------------
|
||||
|
||||
Paths are specified as `remote:path`
|
||||
|
||||
Paths may be as deep as required, eg `remote:directory/subdirectory`.
|
||||
|
||||
The initial setup for Amazon Drive involves getting a token from
|
||||
Amazon which you need to do in your browser. `rclone config` walks
|
||||
you through it.
|
||||
|
||||
Here is an example of how to make a remote called `remote`. First run:
|
||||
|
||||
rclone config
|
||||
|
||||
This will guide you through an interactive setup process:
|
||||
|
||||
```
|
||||
n) New remote
|
||||
d) Delete remote
|
||||
q) Quit config
|
||||
e/n/d/q> n
|
||||
name> remote
|
||||
Type of storage to configure.
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Amazon Drive
|
||||
\ "amazon cloud drive"
|
||||
2 / Amazon S3 (also Dreamhost, Ceph)
|
||||
\ "s3"
|
||||
3 / Backblaze B2
|
||||
\ "b2"
|
||||
4 / Dropbox
|
||||
\ "dropbox"
|
||||
5 / Google Cloud Storage (this is not Google Drive)
|
||||
\ "google cloud storage"
|
||||
6 / Google Drive
|
||||
\ "drive"
|
||||
7 / Hubic
|
||||
\ "hubic"
|
||||
8 / Local Disk
|
||||
\ "local"
|
||||
9 / Microsoft OneDrive
|
||||
\ "onedrive"
|
||||
10 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
||||
\ "swift"
|
||||
11 / Yandex Disk
|
||||
\ "yandex"
|
||||
Storage> 1
|
||||
Amazon Application Client Id - leave blank normally.
|
||||
client_id>
|
||||
Amazon Application Client Secret - leave blank normally.
|
||||
client_secret>
|
||||
Remote config
|
||||
If your browser doesn't open automatically go to the following link: http://127.0.0.1:53682/auth
|
||||
Log in and authorize rclone for access
|
||||
Waiting for code...
|
||||
Got code
|
||||
--------------------
|
||||
[remote]
|
||||
client_id =
|
||||
client_secret =
|
||||
token = {"access_token":"xxxxxxxxxxxxxxxxxxxxxxx","token_type":"bearer","refresh_token":"xxxxxxxxxxxxxxxxxx","expiry":"2015-09-06T16:07:39.658438471+01:00"}
|
||||
--------------------
|
||||
y) Yes this is OK
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
```
|
||||
|
||||
See the [remote setup docs](/remote_setup/) for how to set it up on a
|
||||
machine with no Internet browser available.
|
||||
|
||||
Note that rclone runs a webserver on your local machine to collect the
|
||||
token as returned from Amazon. This only runs from the moment it
|
||||
opens your browser to the moment you get back the verification
|
||||
code. This is on `http://127.0.0.1:53682/` and this it may require
|
||||
you to unblock it temporarily if you are running a host firewall.
|
||||
|
||||
Once configured you can then use `rclone` like this,
|
||||
|
||||
List directories in top level of your Amazon Drive
|
||||
|
||||
rclone lsd remote:
|
||||
|
||||
List all the files in your Amazon Drive
|
||||
|
||||
rclone ls remote:
|
||||
|
||||
To copy a local directory to an Amazon Drive directory called backup
|
||||
|
||||
rclone copy /home/source remote:backup
|
||||
|
||||
### Modified time and MD5SUMs ###
|
||||
|
||||
Amazon Drive doesn't allow modification times to be changed via
|
||||
the API so these won't be accurate or used for syncing.
|
||||
|
||||
It does store MD5SUMs so for a more accurate sync, you can use the
|
||||
`--checksum` flag.
|
||||
|
||||
### Deleting files ###
|
||||
|
||||
Any files you delete with rclone will end up in the trash. Amazon
|
||||
don't provide an API to permanently delete files, nor to empty the
|
||||
trash, so you will have to do that with one of Amazon's apps or via
|
||||
the Amazon Drive website.
|
||||
|
||||
### Specific options ###
|
||||
|
||||
Here are the command line options specific to this cloud storage
|
||||
system.
|
||||
|
||||
#### --acd-templink-threshold=SIZE ####
|
||||
|
||||
Files this size or more will be downloaded via their `tempLink`. This
|
||||
is to work around a problem with Amazon Drive which blocks downloads
|
||||
of files bigger than about 10GB. The default for this is 9GB which
|
||||
shouldn't need to be changed.
|
||||
|
||||
To download files above this threshold, rclone requests a `tempLink`
|
||||
which downloads the file through a temporary URL directly from the
|
||||
underlying S3 storage.
|
||||
|
||||
#### --acd-upload-wait-time=TIME ####
|
||||
|
||||
Sometimes Amazon Drive gives an error when a file has been fully
|
||||
uploaded but the file appears anyway after a little while. This
|
||||
controls the time rclone waits - 2 minutes by default. You might want
|
||||
to increase the time if you are having problems with very big files.
|
||||
Upload with the `-v` flag for more info.
|
||||
|
||||
### Limitations ###
|
||||
|
||||
Note that Amazon Drive is case insensitive so you can't have a
|
||||
file called "Hello.doc" and one called "hello.doc".
|
||||
|
||||
Amazon Drive has rate limiting so you may notice errors in the
|
||||
sync (429 errors). rclone will automatically retry the sync up to 3
|
||||
times by default (see `--retries` flag) which should hopefully work
|
||||
around this problem.
|
||||
|
||||
Amazon Drive has an internal limit of file sizes that can be uploaded
|
||||
to the service. This limit is not officially published, but all files
|
||||
larger than this will fail.
|
||||
|
||||
At the time of writing (Jan 2016) is in the area of 50GB per file.
|
||||
This means that larger files are likely to fail.
|
||||
|
||||
Unfortunatly there is no way for rclone to see that this failure is
|
||||
because of file size, so it will retry the operation, as any other
|
||||
failure. To avoid this problem, use `--max-size=50GB` option to limit
|
||||
the maximum size of uploaded files.
|
||||
@@ -1,39 +0,0 @@
|
||||
---
|
||||
title: "Authors"
|
||||
description: "Rclone Authors and Contributors"
|
||||
date: "2016-04-22"
|
||||
---
|
||||
|
||||
Authors
|
||||
-------
|
||||
|
||||
* Nick Craig-Wood <nick@craig-wood.com>
|
||||
|
||||
Contributors
|
||||
------------
|
||||
|
||||
* Alex Couper <amcouper@gmail.com>
|
||||
* Leonid Shalupov <leonid@shalupov.com>
|
||||
* Shimon Doodkin <helpmepro1@gmail.com>
|
||||
* Colin Nicholson <colin@colinn.com>
|
||||
* Klaus Post <klauspost@gmail.com>
|
||||
* Sergey Tolmachev <tolsi.ru@gmail.com>
|
||||
* Adriano Aurélio Meirelles <adriano@atinge.com>
|
||||
* C. Bess <cbess@users.noreply.github.com>
|
||||
* Dmitry Burdeev <dibu28@gmail.com>
|
||||
* Joseph Spurrier <github@josephspurrier.com>
|
||||
* Björn Harrtell <bjorn@wololo.org>
|
||||
* Xavier Lucas <xavier.lucas@corp.ovh.com>
|
||||
* Werner Beroux <werner@beroux.com>
|
||||
* Brian Stengaard <brian@stengaard.eu>
|
||||
* Jakub Gedeon <jgedeon@sofi.com>
|
||||
* Jim Tittsler <jwt@onjapan.net>
|
||||
* Michal Witkowski <michal@improbable.io>
|
||||
* Fabian Ruff <fabian.ruff@sap.com>
|
||||
* Leigh Klotz <klotz@quixey.com>
|
||||
* Romain Lapray <lapray.romain@gmail.com>
|
||||
* Justin R. Wilson <jrw972@gmail.com>
|
||||
* Antonio Messina <antonio.s.messina@gmail.com>
|
||||
* Stefan G. Weichinger <office@oops.co.at>
|
||||
* Per Cederberg <cederberg@gmail.com>
|
||||
* Radek Šenfeld <rush@logic.cz>
|
||||
@@ -1,248 +0,0 @@
|
||||
---
|
||||
title: "B2"
|
||||
description: "Backblaze B2"
|
||||
date: "2016-06-15"
|
||||
---
|
||||
|
||||
<i class="fa fa-fire"></i>Backblaze B2
|
||||
----------------------------------------
|
||||
|
||||
B2 is [Backblaze's cloud storage system](https://www.backblaze.com/b2/).
|
||||
|
||||
Paths are specified as `remote:bucket` (or `remote:` for the `lsd`
|
||||
command.) You may put subdirectories in too, eg `remote:bucket/path/to/dir`.
|
||||
|
||||
Here is an example of making a b2 configuration. First run
|
||||
|
||||
rclone config
|
||||
|
||||
This will guide you through an interactive setup process. You will
|
||||
need your account number (a short hex number) and key (a long hex
|
||||
number) which you can get from the b2 control panel.
|
||||
|
||||
```
|
||||
No remotes found - make a new one
|
||||
n) New remote
|
||||
q) Quit config
|
||||
n/q> n
|
||||
name> remote
|
||||
Type of storage to configure.
|
||||
Choose a number from below, or type in your own value
|
||||
1 / Amazon Drive
|
||||
\ "amazon cloud drive"
|
||||
2 / Amazon S3 (also Dreamhost, Ceph)
|
||||
\ "s3"
|
||||
3 / Backblaze B2
|
||||
\ "b2"
|
||||
4 / Dropbox
|
||||
\ "dropbox"
|
||||
5 / Google Cloud Storage (this is not Google Drive)
|
||||
\ "google cloud storage"
|
||||
6 / Google Drive
|
||||
\ "drive"
|
||||
7 / Hubic
|
||||
\ "hubic"
|
||||
8 / Local Disk
|
||||
\ "local"
|
||||
9 / Microsoft OneDrive
|
||||
\ "onedrive"
|
||||
10 / Openstack Swift (Rackspace Cloud Files, Memset Memstore, OVH)
|
||||
\ "swift"
|
||||
11 / Yandex Disk
|
||||
\ "yandex"
|
||||
Storage> 3
|
||||
Account ID
|
||||
account> 123456789abc
|
||||
Application Key
|
||||
key> 0123456789abcdef0123456789abcdef0123456789
|
||||
Endpoint for the service - leave blank normally.
|
||||
endpoint>
|
||||
Remote config
|
||||
--------------------
|
||||
[remote]
|
||||
account = 123456789abc
|
||||
key = 0123456789abcdef0123456789abcdef0123456789
|
||||
endpoint =
|
||||
--------------------
|
||||
y) Yes this is OK
|
||||
e) Edit this remote
|
||||
d) Delete this remote
|
||||
y/e/d> y
|
||||
```
|
||||
|
||||
This remote is called `remote` and can now be used like this
|
||||
|
||||
See all buckets
|
||||
|
||||
rclone lsd remote:
|
||||
|
||||
Make a new bucket
|
||||
|
||||
rclone mkdir remote:bucket
|
||||
|
||||
List the contents of a bucket
|
||||
|
||||
rclone ls remote:bucket
|
||||
|
||||
Sync `/home/local/directory` to the remote bucket, deleting any
|
||||
excess files in the bucket.
|
||||
|
||||
rclone sync /home/local/directory remote:bucket
|
||||
|
||||
### Modified time ###
|
||||
|
||||
The modified time is stored as metadata on the object as
|
||||
`X-Bz-Info-src_last_modified_millis` as milliseconds since 1970-01-01
|
||||
in the Backblaze standard. Other tools should be able to use this as
|
||||
a modified time.
|
||||
|
||||
Modified times are used in syncing and are fully supported except in
|
||||
the case of updating a modification time on an existing object. In
|
||||
this case the object will be uploaded again as B2 doesn't have an API
|
||||
method to set the modification time independent of doing an upload.
|
||||
|
||||
### SHA1 checksums ###
|
||||
|
||||
The SHA1 checksums of the files are checked on upload and download and
|
||||
will be used in the syncing process.
|
||||
|
||||
Large files which are uploaded in chunks will store their SHA1 on the
|
||||
object as `X-Bz-Info-large_file_sha1` as recommended by Backblaze.
|
||||
|
||||
### Transfers ###
|
||||
|
||||
Backblaze recommends that you do lots of transfers simultaneously for
|
||||
maximum speed. In tests from my SSD equiped laptop the optimum
|
||||
setting is about `--transfers 32` though higher numbers may be used
|
||||
for a slight speed improvement. The optimum number for you may vary
|
||||
depending on your hardware, how big the files are, how much you want
|
||||
to load your computer, etc. The default of `--transfers 4` is
|
||||
definitely too low for Backblaze B2 though.
|
||||
|
||||
Note that uploading big files (bigger than 200 MB by default) will use
|
||||
a 96 MB RAM buffer by default. There can be at most `--transfers` of
|
||||
these in use at any moment, so this sets the upper limit on the memory
|
||||
used.
|
||||
|
||||
### Versions ###
|
||||
|
||||
When rclone uploads a new version of a file it creates a [new version
|
||||
of it](https://www.backblaze.com/b2/docs/file_versions.html).
|
||||
Likewise when you delete a file, the old version will still be
|
||||
available.
|
||||
|
||||
Old versions of files are visible using the `--b2-versions` flag.
|
||||
|
||||
If you wish to remove all the old versions then you can use the
|
||||
`rclone cleanup remote:bucket` command which will delete all the old
|
||||
versions of files, leaving the current ones intact. You can also
|
||||
supply a path and only old versions under that path will be deleted,
|
||||
eg `rclone cleanup remote:bucket/path/to/stuff`.
|
||||
|
||||
When you `purge` a bucket, the current and the old versions will be
|
||||
deleted then the bucket will be deleted.
|
||||
|
||||
However `delete` will cause the current versions of the files to
|
||||
become hidden old versions.
|
||||
|
||||
Here is a session showing the listing and and retreival of an old
|
||||
version followed by a `cleanup` of the old versions.
|
||||
|
||||
Show current version and all the versions with `--b2-versions` flag.
|
||||
|
||||
```
|
||||
$ rclone -q ls b2:cleanup-test
|
||||
9 one.txt
|
||||
|
||||
$ rclone -q --b2-versions ls b2:cleanup-test
|
||||
9 one.txt
|
||||
8 one-v2016-07-04-141032-000.txt
|
||||
16 one-v2016-07-04-141003-000.txt
|
||||
15 one-v2016-07-02-155621-000.txt
|
||||
```
|
||||
|
||||
Retreive an old verson
|
||||
|
||||
```
|
||||
$ rclone -q --b2-versions copy b2:cleanup-test/one-v2016-07-04-141003-000.txt /tmp
|
||||
|
||||
$ ls -l /tmp/one-v2016-07-04-141003-000.txt
|
||||
-rw-rw-r-- 1 ncw ncw 16 Jul 2 17:46 /tmp/one-v2016-07-04-141003-000.txt
|
||||
```
|
||||
|
||||
Clean up all the old versions and show that they've gone.
|
||||
|
||||
```
|
||||
$ rclone -q cleanup b2:cleanup-test
|
||||
|
||||
$ rclone -q ls b2:cleanup-test
|
||||
9 one.txt
|
||||
|
||||
$ rclone -q --b2-versions ls b2:cleanup-test
|
||||
9 one.txt
|
||||
```
|
||||
|
||||
### Specific options ###
|
||||
|
||||
Here are the command line options specific to this cloud storage
|
||||
system.
|
||||
|
||||
#### --b2-chunk-size valuee=SIZE ####
|
||||
|
||||
When uploading large files chunk the file into this size. Note that
|
||||
these chunks are buffered in memory and there might a maximum of
|
||||
`--transfers` chunks in progress at once. 100,000,000 Bytes is the
|
||||
minimim size (default 96M).
|
||||
|
||||
#### --b2-upload-cutoff=SIZE ####
|
||||
|
||||
Cutoff for switching to chunked upload (default 190.735 MiB == 200
|
||||
MB). Files above this size will be uploaded in chunks of
|
||||
`--b2-chunk-size`.
|
||||
|
||||
This value should be set no larger than 4.657GiB (== 5GB) as this is
|
||||
the largest file size that can be uploaded.
|
||||
|
||||
|
||||
#### --b2-test-mode=FLAG ####
|
||||
|
||||
This is for debugging purposes only.
|
||||
|
||||
Setting FLAG to one of the strings below will cause b2 to return
|
||||
specific errors for debugging purposes.
|
||||
|
||||
* `fail_some_uploads`
|
||||
* `expire_some_account_authorization_tokens`
|
||||
* `force_cap_exceeded`
|
||||
|
||||
These will be set in the `X-Bz-Test-Mode` header which is documented
|
||||
in the [b2 integrations
|
||||
checklist](https://www.backblaze.com/b2/docs/integration_checklist.html).
|
||||
|
||||
#### --b2-versions ####
|
||||
|
||||
When set rclone will show and act on older versions of files. For example
|
||||
|
||||
Listing without `--b2-versions`
|
||||
|
||||
```
|
||||
$ rclone -q ls b2:cleanup-test
|
||||
9 one.txt
|
||||
```
|
||||
|
||||
And with
|
||||
|
||||
```
|
||||
$ rclone -q --b2-versions ls b2:cleanup-test
|
||||
9 one.txt
|
||||
8 one-v2016-07-04-141032-000.txt
|
||||
16 one-v2016-07-04-141003-000.txt
|
||||
15 one-v2016-07-02-155621-000.txt
|
||||
```
|
||||
|
||||
Showing that the current version is unchanged but older versions can
|
||||
be seen. These have the UTC date that they were uploaded to the
|
||||
server to the nearest millisecond appended to them.
|
||||
|
||||
Note that when using `--b2-versions` no file write operations are
|
||||
permitted, so you can't upload files or delete them.
|
||||
@@ -1,31 +0,0 @@
|
||||
---
|
||||
title: "Bugs"
|
||||
description: "Rclone Bugs and Limitations"
|
||||
date: "2014-06-16"
|
||||
---
|
||||
|
||||
Bugs and Limitations
|
||||
--------------------
|
||||
|
||||
### Empty directories are left behind / not created ##
|
||||
|
||||
With remotes that have a concept of directory, eg Local and Drive,
|
||||
empty directories may be left behind, or not created when one was
|
||||
expected.
|
||||
|
||||
This is because rclone doesn't have a concept of a directory - it only
|
||||
works on objects. Most of the object storage systems can't actually
|
||||
store a directory so there is nowhere for rclone to store anything
|
||||
about directories.
|
||||
|
||||
You can work round this to some extent with the`purge` command which
|
||||
will delete everything under the path, **inluding** empty directories.
|
||||
|
||||
This may be fixed at some point in
|
||||
[Issue #100](https://github.com/ncw/rclone/issues/100)
|
||||
|
||||
### Directory timestamps aren't preserved ##
|
||||
|
||||
For the same reason as the above, rclone doesn't have a concept of a
|
||||
directory - it only works on objects, therefore it can't preserve the
|
||||
timestamps of directories.
|
||||
@@ -1,432 +0,0 @@
|
||||
---
|
||||
title: "Documentation"
|
||||
description: "Rclone Changelog"
|
||||
date: "2016-08-24"
|
||||
---
|
||||
|
||||
Changelog
|
||||
---------
|
||||
|
||||
* v1.33 - 2016-08-24
|
||||
* New Features
|
||||
* Implement encryption
|
||||
* data encrypted in NACL secretbox format
|
||||
* with optional file name encryption
|
||||
* New commands
|
||||
* rclone mount - implements FUSE mounting of remotes (EXPERIMENTAL)
|
||||
* works on Linux, FreeBSD and OS X (need testers for the last 2!)
|
||||
* rclone cat - outputs remote file or files to the terminal
|
||||
* rclone genautocomplete - command to make a bash completion script for rclone
|
||||
* Editing a remote using `rclone config` now goes through the wizard
|
||||
* Compile with go 1.7 - this fixes rclone on macOS Sierra and on 386 processors
|
||||
* Use cobra for sub commands and docs generation
|
||||
* drive
|
||||
* Document how to make your own client_id
|
||||
* s3
|
||||
* User-configurable Amazon S3 ACL (thanks Radek Šenfeld)
|
||||
* b2
|
||||
* Fix stats accounting for upload - no more jumping to 100% done
|
||||
* On cleanup delete hide marker if it is the current file
|
||||
* New B2 API endpoint (thanks Per Cederberg)
|
||||
* Set maximum backoff to 5 Minutes
|
||||
* onedrive
|
||||
* Fix URL escaping in file names - eg uploading files with `+` in them.
|
||||
* amazon cloud drive
|
||||
* Fix token expiry during large uploads
|
||||
* Work around 408 REQUEST_TIMEOUT and 504 GATEWAY_TIMEOUT errors
|
||||
* local
|
||||
* Fix filenames with invalid UTF-8 not being uploaded
|
||||
* Fix problem with some UTF-8 characters on OS X
|
||||
* v1.32 - 2016-07-13
|
||||
* Backblaze B2
|
||||
* Fix upload of files large files not in root
|
||||
* v1.31 - 2016-07-13
|
||||
* New Features
|
||||
* Reduce memory on sync by about 50%
|
||||
* Implement --no-traverse flag to stop copy traversing the destination remote.
|
||||
* This can be used to reduce memory usage down to the smallest possible.
|
||||
* Useful to copy a small number of files into a large destination folder.
|
||||
* Implement cleanup command for emptying trash / removing old versions of files
|
||||
* Currently B2 only
|
||||
* Single file handling improved
|
||||
* Now copied with --files-from
|
||||
* Automatically sets --no-traverse when copying a single file
|
||||
* Info on using installing with ansible - thanks Stefan Weichinger
|
||||
* Implement --no-update-modtime flag to stop rclone fixing the remote modified times.
|
||||
* Bug Fixes
|
||||
* Fix move command - stop it running for overlapping Fses - this was causing data loss.
|
||||
* Local
|
||||
* Fix incomplete hashes - this was causing problems for B2.
|
||||
* Amazon Drive
|
||||
* Rename Amazon Cloud Drive to Amazon Drive - no changes to config file needed.
|
||||
* Swift
|
||||
* Add support for non-default project domain - thanks Antonio Messina.
|
||||
* S3
|
||||
* Add instructions on how to use rclone with minio.
|
||||
* Add ap-northeast-2 (Seoul) and ap-south-1 (Mumbai) regions.
|
||||
* Skip setting the modified time for objects > 5GB as it isn't possible.
|
||||
* Backblaze B2
|
||||
* Add --b2-versions flag so old versions can be listed and retreived.
|
||||
* Treat 403 errors (eg cap exceeded) as fatal.
|
||||
* Implement cleanup command for deleting old file versions.
|
||||
* Make error handling compliant with B2 integrations notes.
|
||||
* Fix handling of token expiry.
|
||||
* Implement --b2-test-mode to set `X-Bz-Test-Mode` header.
|
||||
* Set cutoff for chunked upload to 200MB as per B2 guidelines.
|
||||
* Make upload multi-threaded.
|
||||
* Dropbox
|
||||
* Don't retry 461 errors.
|
||||
* v1.30 - 2016-06-18
|
||||
* New Features
|
||||
* Directory listing code reworked for more features and better error reporting (thanks to Klaus Post for help). This enables
|
||||
* Directory include filtering for efficiency
|
||||
* --max-depth parameter
|
||||
* Better error reporting
|
||||
* More to come
|
||||
* Retry more errors
|
||||
* Add --ignore-size flag - for uploading images to onedrive
|
||||
* Log -v output to stdout by default
|
||||
* Display the transfer stats in more human readable form
|
||||
* Make 0 size files specifiable with `--max-size 0b`
|
||||
* Add `b` suffix so we can specify bytes in --bwlimit, --min-size etc
|
||||
* Use "password:" instead of "password>" prompt - thanks Klaus Post and Leigh Klotz
|
||||
* Bug Fixes
|
||||
* Fix retry doing one too many retries
|
||||
* Local
|
||||
* Fix problems with OS X and UTF-8 characters
|
||||
* Amazon Drive
|
||||
* Check a file exists before uploading to help with 408 Conflict errors
|
||||
* Reauth on 401 errors - this has been causing a lot of problems
|
||||
* Work around spurious 403 errors
|
||||
* Restart directory listings on error
|
||||
* Google Drive
|
||||
* Check a file exists before uploading to help with duplicates
|
||||
* Fix retry of multipart uploads
|
||||
* Backblaze B2
|
||||
* Implement large file uploading
|
||||
* S3
|
||||
* Add AES256 server-side encryption for - thanks Justin R. Wilson
|
||||
* Google Cloud Storage
|
||||
* Make sure we don't use conflicting content types on upload
|
||||
* Add service account support - thanks Michal Witkowski
|
||||
* Swift
|
||||
* Add auth version parameter
|
||||
* Add domain option for openstack (v3 auth) - thanks Fabian Ruff
|
||||
* v1.29 - 2016-04-18
|
||||
* New Features
|
||||
* Implement `-I, --ignore-times` for unconditional upload
|
||||
* Improve `dedupe`command
|
||||
* Now removes identical copies without asking
|
||||
* Now obeys `--dry-run`
|
||||
* Implement `--dedupe-mode` for non interactive running
|
||||
* `--dedupe-mode interactive` - interactive the default.
|
||||
* `--dedupe-mode skip` - removes identical files then skips anything left.
|
||||
* `--dedupe-mode first` - removes identical files then keeps the first one.
|
||||
* `--dedupe-mode newest` - removes identical files then keeps the newest one.
|
||||
* `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
|
||||
* `--dedupe-mode rename` - removes identical files then renames the rest to be different.
|
||||
* Bug fixes
|
||||
* Make rclone check obey the `--size-only` flag.
|
||||
* Use "application/octet-stream" if discovered mime type is invalid.
|
||||
* Fix missing "quit" option when there are no remotes.
|
||||
* Google Drive
|
||||
* Increase default chunk size to 8 MB - increases upload speed of big files
|
||||
* Speed up directory listings and make more reliable
|
||||
* Add missing retries for Move and DirMove - increases reliability
|
||||
* Preserve mime type on file update
|
||||
* Backblaze B2
|
||||
* Enable mod time syncing
|
||||
* This means that B2 will now check modification times
|
||||
* It will upload new files to update the modification times
|
||||
* (there isn't an API to just set the mod time.)
|
||||
* If you want the old behaviour use `--size-only`.
|
||||
* Update API to new version
|
||||
* Fix parsing of mod time when not in metadata
|
||||
* Swift/Hubic
|
||||
* Don't return an MD5SUM for static large objects
|
||||
* S3
|
||||
* Fix uploading files bigger than 50GB
|
||||
* v1.28 - 2016-03-01
|
||||
* New Features
|
||||
* Configuration file encryption - thanks Klaus Post
|
||||
* Improve `rclone config` adding more help and making it easier to understand
|
||||
* Implement `-u`/`--update` so creation times can be used on all remotes
|
||||
* Implement `--low-level-retries` flag
|
||||
* Optionally disable gzip compression on downloads with `--no-gzip-encoding`
|
||||
* Bug fixes
|
||||
* Don't make directories if `--dry-run` set
|
||||
* Fix and document the `move` command
|
||||
* Fix redirecting stderr on unix-like OSes when using `--log-file`
|
||||
* Fix `delete` command to wait until all finished - fixes missing deletes.
|
||||
* Backblaze B2
|
||||
* Use one upload URL per go routine fixes `more than one upload using auth token`
|
||||
* Add pacing, retries and reauthentication - fixes token expiry problems
|
||||
* Upload without using a temporary file from local (and remotes which support SHA1)
|
||||
* Fix reading metadata for all files when it shouldn't have been
|
||||
* Drive
|
||||
* Fix listing drive documents at root
|
||||
* Disable copy and move for Google docs
|
||||
* Swift
|
||||
* Fix uploading of chunked files with non ASCII characters
|
||||
* Allow setting of `storage_url` in the config - thanks Xavier Lucas
|
||||
* S3
|
||||
* Allow IAM role and credentials from environment variables - thanks Brian Stengaard
|
||||
* Allow low privilege users to use S3 (check if directory exists during Mkdir) - thanks Jakub Gedeon
|
||||
* Amazon Drive
|
||||
* Retry on more things to make directory listings more reliable
|
||||
* v1.27 - 2016-01-31
|
||||
* New Features
|
||||
* Easier headless configuration with `rclone authorize`
|
||||
* Add support for multiple hash types - we now check SHA1 as well as MD5 hashes.
|
||||
* `delete` command which does obey the filters (unlike `purge`)
|
||||
* `dedupe` command to deduplicate a remote. Useful with Google Drive.
|
||||
* Add `--ignore-existing` flag to skip all files that exist on destination.
|
||||
* Add `--delete-before`, `--delete-during`, `--delete-after` flags.
|
||||
* Add `--memprofile` flag to debug memory use.
|
||||
* Warn the user about files with same name but different case
|
||||
* Make `--include` rules add their implict exclude * at the end of the filter list
|
||||
* Deprecate compiling with go1.3
|
||||
* Amazon Drive
|
||||
* Fix download of files > 10 GB
|
||||
* Fix directory traversal ("Next token is expired") for large directory listings
|
||||
* Remove 409 conflict from error codes we will retry - stops very long pauses
|
||||
* Backblaze B2
|
||||
* SHA1 hashes now checked by rclone core
|
||||
* Drive
|
||||
* Add `--drive-auth-owner-only` to only consider files owned by the user - thanks Björn Harrtell
|
||||
* Export Google documents
|
||||
* Dropbox
|
||||
* Make file exclusion error controllable with -q
|
||||
* Swift
|
||||
* Fix upload from unprivileged user.
|
||||
* S3
|
||||
* Fix updating of mod times of files with `+` in.
|
||||
* Local
|
||||
* Add local file system option to disable UNC on Windows.
|
||||
* v1.26 - 2016-01-02
|
||||
* New Features
|
||||
* Yandex storage backend - thank you Dmitry Burdeev ("dibu")
|
||||
* Implement Backblaze B2 storage backend
|
||||
* Add --min-age and --max-age flags - thank you Adriano Aurélio Meirelles
|
||||
* Make ls/lsl/md5sum/size/check obey includes and excludes
|
||||
* Fixes
|
||||
* Fix crash in http logging
|
||||
* Upload releases to github too
|
||||
* Swift
|
||||
* Fix sync for chunked files
|
||||
* One Drive
|
||||
* Re-enable server side copy
|
||||
* Don't mask HTTP error codes with JSON decode error
|
||||
* S3
|
||||
* Fix corrupting Content-Type on mod time update (thanks Joseph Spurrier)
|
||||
* v1.25 - 2015-11-14
|
||||
* New features
|
||||
* Implement Hubic storage system
|
||||
* Fixes
|
||||
* Fix deletion of some excluded files without --delete-excluded
|
||||
* This could have deleted files unexpectedly on sync
|
||||
* Always check first with `--dry-run`!
|
||||
* Swift
|
||||
* Stop SetModTime losing metadata (eg X-Object-Manifest)
|
||||
* This could have caused data loss for files > 5GB in size
|
||||
* Use ContentType from Object to avoid lookups in listings
|
||||
* One Drive
|
||||
* disable server side copy as it seems to be broken at Microsoft
|
||||
* v1.24 - 2015-11-07
|
||||
* New features
|
||||
* Add support for Microsoft One Drive
|
||||
* Add `--no-check-certificate` option to disable server certificate verification
|
||||
* Add async readahead buffer for faster transfer of big files
|
||||
* Fixes
|
||||
* Allow spaces in remotes and check remote names for validity at creation time
|
||||
* Allow '&' and disallow ':' in Windows filenames.
|
||||
* Swift
|
||||
* Ignore directory marker objects where appropriate - allows working with Hubic
|
||||
* Don't delete the container if fs wasn't at root
|
||||
* S3
|
||||
* Don't delete the bucket if fs wasn't at root
|
||||
* Google Cloud Storage
|
||||
* Don't delete the bucket if fs wasn't at root
|
||||
* v1.23 - 2015-10-03
|
||||
* New features
|
||||
* Implement `rclone size` for measuring remotes
|
||||
* Fixes
|
||||
* Fix headless config for drive and gcs
|
||||
* Tell the user they should try again if the webserver method failed
|
||||
* Improve output of `--dump-headers`
|
||||
* S3
|
||||
* Allow anonymous access to public buckets
|
||||
* Swift
|
||||
* Stop chunked operations logging "Failed to read info: Object Not Found"
|
||||
* Use Content-Length on uploads for extra reliability
|
||||
* v1.22 - 2015-09-28
|
||||
* Implement rsync like include and exclude flags
|
||||
* swift
|
||||
* Support files > 5GB - thanks Sergey Tolmachev
|
||||
* v1.21 - 2015-09-22
|
||||
* New features
|
||||
* Display individual transfer progress
|
||||
* Make lsl output times in localtime
|
||||
* Fixes
|
||||
* Fix allowing user to override credentials again in Drive, GCS and ACD
|
||||
* Amazon Drive
|
||||
* Implement compliant pacing scheme
|
||||
* Google Drive
|
||||
* Make directory reads concurrent for increased speed.
|
||||
* v1.20 - 2015-09-15
|
||||
* New features
|
||||
* Amazon Drive support
|
||||
* Oauth support redone - fix many bugs and improve usability
|
||||
* Use "golang.org/x/oauth2" as oauth libary of choice
|
||||
* Improve oauth usability for smoother initial signup
|
||||
* drive, googlecloudstorage: optionally use auto config for the oauth token
|
||||
* Implement --dump-headers and --dump-bodies debug flags
|
||||
* Show multiple matched commands if abbreviation too short
|
||||
* Implement server side move where possible
|
||||
* local
|
||||
* Always use UNC paths internally on Windows - fixes a lot of bugs
|
||||
* dropbox
|
||||
* force use of our custom transport which makes timeouts work
|
||||
* Thanks to Klaus Post for lots of help with this release
|
||||
* v1.19 - 2015-08-28
|
||||
* New features
|
||||
* Server side copies for s3/swift/drive/dropbox/gcs
|
||||
* Move command - uses server side copies if it can
|
||||
* Implement --retries flag - tries 3 times by default
|
||||
* Build for plan9/amd64 and solaris/amd64 too
|
||||
* Fixes
|
||||
* Make a current version download with a fixed URL for scripting
|
||||
* Ignore rmdir in limited fs rather than throwing error
|
||||
* dropbox
|
||||
* Increase chunk size to improve upload speeds massively
|
||||
* Issue an error message when trying to upload bad file name
|
||||
* v1.18 - 2015-08-17
|
||||
* drive
|
||||
* Add `--drive-use-trash` flag so rclone trashes instead of deletes
|
||||
* Add "Forbidden to download" message for files with no downloadURL
|
||||
* dropbox
|
||||
* Remove datastore
|
||||
* This was deprecated and it caused a lot of problems
|
||||
* Modification times and MD5SUMs no longer stored
|
||||
* Fix uploading files > 2GB
|
||||
* s3
|
||||
* use official AWS SDK from github.com/aws/aws-sdk-go
|
||||
* **NB** will most likely require you to delete and recreate remote
|
||||
* enable multipart upload which enables files > 5GB
|
||||
* tested with Ceph / RadosGW / S3 emulation
|
||||
* many thanks to Sam Liston and Brian Haymore at the [Utah
|
||||
Center for High Performance Computing](https://www.chpc.utah.edu/) for a Ceph test account
|
||||
* misc
|
||||
* Show errors when reading the config file
|
||||
* Do not print stats in quiet mode - thanks Leonid Shalupov
|
||||
* Add FAQ
|
||||
* Fix created directories not obeying umask
|
||||
* Linux installation instructions - thanks Shimon Doodkin
|
||||
* v1.17 - 2015-06-14
|
||||
* dropbox: fix case insensitivity issues - thanks Leonid Shalupov
|
||||
* v1.16 - 2015-06-09
|
||||
* Fix uploading big files which was causing timeouts or panics
|
||||
* Don't check md5sum after download with --size-only
|
||||
* v1.15 - 2015-06-06
|
||||
* Add --checksum flag to only discard transfers by MD5SUM - thanks Alex Couper
|
||||
* Implement --size-only flag to sync on size not checksum & modtime
|
||||
* Expand docs and remove duplicated information
|
||||
* Document rclone's limitations with directories
|
||||
* dropbox: update docs about case insensitivity
|
||||
* v1.14 - 2015-05-21
|
||||
* local: fix encoding of non utf-8 file names - fixes a duplicate file problem
|
||||
* drive: docs about rate limiting
|
||||
* google cloud storage: Fix compile after API change in "google.golang.org/api/storage/v1"
|
||||
* v1.13 - 2015-05-10
|
||||
* Revise documentation (especially sync)
|
||||
* Implement --timeout and --conntimeout
|
||||
* s3: ignore etags from multipart uploads which aren't md5sums
|
||||
* v1.12 - 2015-03-15
|
||||
* drive: Use chunked upload for files above a certain size
|
||||
* drive: add --drive-chunk-size and --drive-upload-cutoff parameters
|
||||
* drive: switch to insert from update when a failed copy deletes the upload
|
||||
* core: Log duplicate files if they are detected
|
||||
* v1.11 - 2015-03-04
|
||||
* swift: add region parameter
|
||||
* drive: fix crash on failed to update remote mtime
|
||||
* In remote paths, change native directory separators to /
|
||||
* Add synchronization to ls/lsl/lsd output to stop corruptions
|
||||
* Ensure all stats/log messages to go stderr
|
||||
* Add --log-file flag to log everything (including panics) to file
|
||||
* Make it possible to disable stats printing with --stats=0
|
||||
* Implement --bwlimit to limit data transfer bandwidth
|
||||
* v1.10 - 2015-02-12
|
||||
* s3: list an unlimited number of items
|
||||
* Fix getting stuck in the configurator
|
||||
* v1.09 - 2015-02-07
|
||||
* windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
|
||||
* local: Fix directory separators on Windows
|
||||
* drive: fix rate limit exceeded errors
|
||||
* v1.08 - 2015-02-04
|
||||
* drive: fix subdirectory listing to not list entire drive
|
||||
* drive: Fix SetModTime
|
||||
* dropbox: adapt code to recent library changes
|
||||
* v1.07 - 2014-12-23
|
||||
* google cloud storage: fix memory leak
|
||||
* v1.06 - 2014-12-12
|
||||
* Fix "Couldn't find home directory" on OSX
|
||||
* swift: Add tenant parameter
|
||||
* Use new location of Google API packages
|
||||
* v1.05 - 2014-08-09
|
||||
* Improved tests and consequently lots of minor fixes
|
||||
* core: Fix race detected by go race detector
|
||||
* core: Fixes after running errcheck
|
||||
* drive: reset root directory on Rmdir and Purge
|
||||
* fs: Document that Purger returns error on empty directory, test and fix
|
||||
* google cloud storage: fix ListDir on subdirectory
|
||||
* google cloud storage: re-read metadata in SetModTime
|
||||
* s3: make reading metadata more reliable to work around eventual consistency problems
|
||||
* s3: strip trailing / from ListDir()
|
||||
* swift: return directories without / in ListDir
|
||||
* v1.04 - 2014-07-21
|
||||
* google cloud storage: Fix crash on Update
|
||||
* v1.03 - 2014-07-20
|
||||
* swift, s3, dropbox: fix updated files being marked as corrupted
|
||||
* Make compile with go 1.1 again
|
||||
* v1.02 - 2014-07-19
|
||||
* Implement Dropbox remote
|
||||
* Implement Google Cloud Storage remote
|
||||
* Verify Md5sums and Sizes after copies
|
||||
* Remove times from "ls" command - lists sizes only
|
||||
* Add add "lsl" - lists times and sizes
|
||||
* Add "md5sum" command
|
||||
* v1.01 - 2014-07-04
|
||||
* drive: fix transfer of big files using up lots of memory
|
||||
* v1.00 - 2014-07-03
|
||||
* drive: fix whole second dates
|
||||
* v0.99 - 2014-06-26
|
||||
* Fix --dry-run not working
|
||||
* Make compatible with go 1.1
|
||||
* v0.98 - 2014-05-30
|
||||
* s3: Treat missing Content-Length as 0 for some ceph installations
|
||||
* rclonetest: add file with a space in
|
||||
* v0.97 - 2014-05-05
|
||||
* Implement copying of single files
|
||||
* s3 & swift: support paths inside containers/buckets
|
||||
* v0.96 - 2014-04-24
|
||||
* drive: Fix multiple files of same name being created
|
||||
* drive: Use o.Update and fs.Put to optimise transfers
|
||||
* Add version number, -V and --version
|
||||
* v0.95 - 2014-03-28
|
||||
* rclone.org: website, docs and graphics
|
||||
* drive: fix path parsing
|
||||
* v0.94 - 2014-03-27
|
||||
* Change remote format one last time
|
||||
* GNU style flags
|
||||
* v0.93 - 2014-03-16
|
||||
* drive: store token in config file
|
||||
* cross compile other versions
|
||||
* set strict permissions on config file
|
||||
* v0.92 - 2014-03-15
|
||||
* Config fixes and --config option
|
||||
* v0.91 - 2014-03-15
|
||||
* Make config file
|
||||
* v0.90 - 2013-06-27
|
||||
* Project named rclone
|
||||
* v0.00 - 2012-11-18
|
||||
* Project started
|
||||
@@ -1,143 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone"
|
||||
slug: rclone
|
||||
url: /commands/rclone/
|
||||
---
|
||||
## rclone
|
||||
|
||||
Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Rclone is a command line program to sync files and directories to and
|
||||
from various cloud storage systems, such as:
|
||||
|
||||
* Google Drive
|
||||
* Amazon S3
|
||||
* Openstack Swift / Rackspace cloud files / Memset Memstore
|
||||
* Dropbox
|
||||
* Google Cloud Storage
|
||||
* Amazon Drive
|
||||
* Microsoft One Drive
|
||||
* Hubic
|
||||
* Backblaze B2
|
||||
* Yandex Disk
|
||||
* The local filesystem
|
||||
|
||||
Features
|
||||
|
||||
* MD5/SHA1 hashes checked at all times for file integrity
|
||||
* Timestamps preserved on files
|
||||
* Partial syncs supported on a whole file basis
|
||||
* Copy mode to just copy new/changed files
|
||||
* Sync (one way) mode to make a directory identical
|
||||
* Check mode to check for file hash equality
|
||||
* Can sync to and from network, eg two different cloud accounts
|
||||
|
||||
See the home page for installation, usage, documentation, changelog
|
||||
and configuration walkthroughs.
|
||||
|
||||
* http://rclone.org/
|
||||
|
||||
|
||||
```
|
||||
rclone
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
-V, --version Print the version number
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone authorize](/commands/rclone_authorize/) - Remote authorization.
|
||||
* [rclone cat](/commands/rclone_cat/) - Concatenates any files and sends them to stdout.
|
||||
* [rclone check](/commands/rclone_check/) - Checks the files in the source and destination match.
|
||||
* [rclone cleanup](/commands/rclone_cleanup/) - Clean up the remote if possible
|
||||
* [rclone config](/commands/rclone_config/) - Enter an interactive configuration session.
|
||||
* [rclone copy](/commands/rclone_copy/) - Copy files from source to dest, skipping already copied
|
||||
* [rclone dedupe](/commands/rclone_dedupe/) - Interactively find duplicate files delete/rename them.
|
||||
* [rclone delete](/commands/rclone_delete/) - Remove the contents of path.
|
||||
* [rclone genautocomplete](/commands/rclone_genautocomplete/) - Output bash completion script for rclone.
|
||||
* [rclone gendocs](/commands/rclone_gendocs/) - Output markdown docs for rclone to the directory supplied.
|
||||
* [rclone ls](/commands/rclone_ls/) - List all the objects in the the path with size and path.
|
||||
* [rclone lsd](/commands/rclone_lsd/) - List all directories/containers/buckets in the the path.
|
||||
* [rclone lsl](/commands/rclone_lsl/) - List all the objects path with modification time, size and path.
|
||||
* [rclone md5sum](/commands/rclone_md5sum/) - Produces an md5sum file for all the objects in the path.
|
||||
* [rclone mkdir](/commands/rclone_mkdir/) - Make the path if it doesn't already exist.
|
||||
* [rclone mount](/commands/rclone_mount/) - Mount the remote as a mountpoint. **EXPERIMENTAL**
|
||||
* [rclone move](/commands/rclone_move/) - Move files from source to dest.
|
||||
* [rclone purge](/commands/rclone_purge/) - Remove the path and all of its contents.
|
||||
* [rclone rmdir](/commands/rclone_rmdir/) - Remove the path if empty.
|
||||
* [rclone sha1sum](/commands/rclone_sha1sum/) - Produces an sha1sum file for all the objects in the path.
|
||||
* [rclone size](/commands/rclone_size/) - Prints the total size and number of objects in remote:path.
|
||||
* [rclone sync](/commands/rclone_sync/) - Make source and dest identical, modifying destination only.
|
||||
* [rclone version](/commands/rclone_version/) - Show the version number.
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone authorize"
|
||||
slug: rclone_authorize
|
||||
url: /commands/rclone_authorize/
|
||||
---
|
||||
## rclone authorize
|
||||
|
||||
Remote authorization.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Remote authorization. Used to authorize a remote or headless
|
||||
rclone from a machine with a browser - use as instructed by
|
||||
rclone config.
|
||||
|
||||
```
|
||||
rclone authorize
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,96 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone check"
|
||||
slug: rclone_check
|
||||
url: /commands/rclone_check/
|
||||
---
|
||||
## rclone check
|
||||
|
||||
Checks the files in the source and destination match.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Checks the files in the source and destination match. It
|
||||
compares sizes and MD5SUMs and prints a report of files which
|
||||
don't match. It doesn't alter the source or destination.
|
||||
|
||||
`--size-only` may be used to only compare the sizes, not the MD5SUMs.
|
||||
|
||||
|
||||
```
|
||||
rclone check source:path dest:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone cleanup"
|
||||
slug: rclone_cleanup
|
||||
url: /commands/rclone_cleanup/
|
||||
---
|
||||
## rclone cleanup
|
||||
|
||||
Clean up the remote if possible
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Clean up the remote if possible. Empty the trash or delete old file
|
||||
versions. Not supported by all remotes.
|
||||
|
||||
|
||||
```
|
||||
rclone cleanup remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone config"
|
||||
slug: rclone_config
|
||||
url: /commands/rclone_config/
|
||||
---
|
||||
## rclone config
|
||||
|
||||
Enter an interactive configuration session.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
Enter an interactive configuration session.
|
||||
|
||||
```
|
||||
rclone config
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,129 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone copy"
|
||||
slug: rclone_copy
|
||||
url: /commands/rclone_copy/
|
||||
---
|
||||
## rclone copy
|
||||
|
||||
Copy files from source to dest, skipping already copied
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Copy the source to the destination. Doesn't transfer
|
||||
unchanged files, testing by size and modification time or
|
||||
MD5SUM. Doesn't delete files from the destination.
|
||||
|
||||
Note that it is always the contents of the directory that is synced,
|
||||
not the directory so when source:path is a directory, it's the
|
||||
contents of source:path that are copied, not the directory name and
|
||||
contents.
|
||||
|
||||
If dest:path doesn't exist, it is created and the source:path contents
|
||||
go there.
|
||||
|
||||
For example
|
||||
|
||||
rclone copy source:sourcepath dest:destpath
|
||||
|
||||
Let's say there are two files in sourcepath
|
||||
|
||||
sourcepath/one.txt
|
||||
sourcepath/two.txt
|
||||
|
||||
This copies them to
|
||||
|
||||
destpath/one.txt
|
||||
destpath/two.txt
|
||||
|
||||
Not to
|
||||
|
||||
destpath/sourcepath/one.txt
|
||||
destpath/sourcepath/two.txt
|
||||
|
||||
If you are familiar with `rsync`, rclone always works as if you had
|
||||
written a trailing / - meaning "copy the contents of this directory".
|
||||
This applies to all commands and whether you are talking about the
|
||||
source or destination.
|
||||
|
||||
See the `--no-traverse` option for controlling whether rclone lists
|
||||
the destination directory or not.
|
||||
|
||||
|
||||
```
|
||||
rclone copy source:path dest:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,171 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone dedupe"
|
||||
slug: rclone_dedupe
|
||||
url: /commands/rclone_dedupe/
|
||||
---
|
||||
## rclone dedupe
|
||||
|
||||
Interactively find duplicate files delete/rename them.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
By default `dedup` interactively finds duplicate files and offers to
|
||||
delete all but one or rename them to be different. Only useful with
|
||||
Google Drive which can have duplicate file names.
|
||||
|
||||
The `dedupe` command will delete all but one of any identical (same
|
||||
md5sum) files it finds without confirmation. This means that for most
|
||||
duplicated files the `dedupe` command will not be interactive. You
|
||||
can use `--dry-run` to see what would happen without doing anything.
|
||||
|
||||
Here is an example run.
|
||||
|
||||
Before - with duplicates
|
||||
|
||||
$ rclone lsl drive:dupes
|
||||
6048320 2016-03-05 16:23:16.798000000 one.txt
|
||||
6048320 2016-03-05 16:23:11.775000000 one.txt
|
||||
564374 2016-03-05 16:23:06.731000000 one.txt
|
||||
6048320 2016-03-05 16:18:26.092000000 one.txt
|
||||
6048320 2016-03-05 16:22:46.185000000 two.txt
|
||||
1744073 2016-03-05 16:22:38.104000000 two.txt
|
||||
564374 2016-03-05 16:22:52.118000000 two.txt
|
||||
|
||||
Now the `dedupe` session
|
||||
|
||||
$ rclone dedupe drive:dupes
|
||||
2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
|
||||
one.txt: Found 4 duplicates - deleting identical copies
|
||||
one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
|
||||
one.txt: 2 duplicates remain
|
||||
1: 6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
|
||||
2: 564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
|
||||
s) Skip and do nothing
|
||||
k) Keep just one (choose which in next step)
|
||||
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
||||
s/k/r> k
|
||||
Enter the number of the file to keep> 1
|
||||
one.txt: Deleted 1 extra copies
|
||||
two.txt: Found 3 duplicates - deleting identical copies
|
||||
two.txt: 3 duplicates remain
|
||||
1: 564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
|
||||
2: 6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
|
||||
3: 1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
|
||||
s) Skip and do nothing
|
||||
k) Keep just one (choose which in next step)
|
||||
r) Rename all to be different (by changing file.jpg to file-1.jpg)
|
||||
s/k/r> r
|
||||
two-1.txt: renamed from: two.txt
|
||||
two-2.txt: renamed from: two.txt
|
||||
two-3.txt: renamed from: two.txt
|
||||
|
||||
The result being
|
||||
|
||||
$ rclone lsl drive:dupes
|
||||
6048320 2016-03-05 16:23:16.798000000 one.txt
|
||||
564374 2016-03-05 16:22:52.118000000 two-1.txt
|
||||
6048320 2016-03-05 16:22:46.185000000 two-2.txt
|
||||
1744073 2016-03-05 16:22:38.104000000 two-3.txt
|
||||
|
||||
Dedupe can be run non interactively using the `--dedupe-mode` flag or by using an extra parameter with the same value
|
||||
|
||||
* `--dedupe-mode interactive` - interactive as above.
|
||||
* `--dedupe-mode skip` - removes identical files then skips anything left.
|
||||
* `--dedupe-mode first` - removes identical files then keeps the first one.
|
||||
* `--dedupe-mode newest` - removes identical files then keeps the newest one.
|
||||
* `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
|
||||
* `--dedupe-mode rename` - removes identical files then renames the rest to be different.
|
||||
|
||||
For example to rename all the identically named photos in your Google Photos directory, do
|
||||
|
||||
rclone dedupe --dedupe-mode rename "drive:Google Photos"
|
||||
|
||||
Or
|
||||
|
||||
rclone dedupe rename "drive:Google Photos"
|
||||
|
||||
|
||||
```
|
||||
rclone dedupe [mode] remote:path
|
||||
```
|
||||
|
||||
### Options
|
||||
|
||||
```
|
||||
--dedupe-mode string Dedupe mode interactive|skip|first|newest|oldest|rename.
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,107 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone delete"
|
||||
slug: rclone_delete
|
||||
url: /commands/rclone_delete/
|
||||
---
|
||||
## rclone delete
|
||||
|
||||
Remove the contents of path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Remove the contents of path. Unlike `purge` it obeys include/exclude
|
||||
filters so can be used to selectively delete files.
|
||||
|
||||
Eg delete all files bigger than 100MBytes
|
||||
|
||||
Check what would be deleted first (use either)
|
||||
|
||||
rclone --min-size 100M lsl remote:path
|
||||
rclone --dry-run --min-size 100M delete remote:path
|
||||
|
||||
Then delete
|
||||
|
||||
rclone --min-size 100M delete remote:path
|
||||
|
||||
That reads "delete everything with a minimum size of 100 MB", hence
|
||||
delete all files bigger than 100MBytes.
|
||||
|
||||
|
||||
```
|
||||
rclone delete remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,105 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone genautocomplete"
|
||||
slug: rclone_genautocomplete
|
||||
url: /commands/rclone_genautocomplete/
|
||||
---
|
||||
## rclone genautocomplete
|
||||
|
||||
Output bash completion script for rclone.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Generates a bash shell autocompletion script for rclone.
|
||||
|
||||
This writes to /etc/bash_completion.d/rclone by default so will
|
||||
probably need to be run with sudo or as root, eg
|
||||
|
||||
sudo rclone genautocomplete
|
||||
|
||||
Logout and login again to use the autocompletion scripts, or source
|
||||
them directly
|
||||
|
||||
. /etc/bash_completion
|
||||
|
||||
If you supply a command line argument the script will be written
|
||||
there.
|
||||
|
||||
|
||||
```
|
||||
rclone genautocomplete [output_file]
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone gendocs"
|
||||
slug: rclone_gendocs
|
||||
url: /commands/rclone_gendocs/
|
||||
---
|
||||
## rclone gendocs
|
||||
|
||||
Output markdown docs for rclone to the directory supplied.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
This produces markdown docs for the rclone commands to the directory
|
||||
supplied. These are in a format suitable for hugo to render into the
|
||||
rclone.org website.
|
||||
|
||||
```
|
||||
rclone gendocs output_directory
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone ls"
|
||||
slug: rclone_ls
|
||||
url: /commands/rclone_ls/
|
||||
---
|
||||
## rclone ls
|
||||
|
||||
List all the objects in the the path with size and path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
List all the objects in the the path with size and path.
|
||||
|
||||
```
|
||||
rclone ls remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone lsd"
|
||||
slug: rclone_lsd
|
||||
url: /commands/rclone_lsd/
|
||||
---
|
||||
## rclone lsd
|
||||
|
||||
List all directories/containers/buckets in the the path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
List all directories/containers/buckets in the the path.
|
||||
|
||||
```
|
||||
rclone lsd remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone lsl"
|
||||
slug: rclone_lsl
|
||||
url: /commands/rclone_lsl/
|
||||
---
|
||||
## rclone lsl
|
||||
|
||||
List all the objects path with modification time, size and path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
List all the objects path with modification time, size and path.
|
||||
|
||||
```
|
||||
rclone lsl remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone md5sum"
|
||||
slug: rclone_md5sum
|
||||
url: /commands/rclone_md5sum/
|
||||
---
|
||||
## rclone md5sum
|
||||
|
||||
Produces an md5sum file for all the objects in the path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Produces an md5sum file for all the objects in the path. This
|
||||
is in the same format as the standard md5sum tool produces.
|
||||
|
||||
|
||||
```
|
||||
rclone md5sum remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone mkdir"
|
||||
slug: rclone_mkdir
|
||||
url: /commands/rclone_mkdir/
|
||||
---
|
||||
## rclone mkdir
|
||||
|
||||
Make the path if it doesn't already exist.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
Make the path if it doesn't already exist.
|
||||
|
||||
```
|
||||
rclone mkdir remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,106 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone move"
|
||||
slug: rclone_move
|
||||
url: /commands/rclone_move/
|
||||
---
|
||||
## rclone move
|
||||
|
||||
Move files from source to dest.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Moves the contents of the source directory to the destination
|
||||
directory. Rclone will error if the source and destination overlap.
|
||||
|
||||
If no filters are in use and if possible this will server side move
|
||||
`source:path` into `dest:path`. After this `source:path` will no
|
||||
longer longer exist.
|
||||
|
||||
Otherwise for each file in `source:path` selected by the filters (if
|
||||
any) this will move it into `dest:path`. If possible a server side
|
||||
move will be used, otherwise it will copy it (server side if possible)
|
||||
into `dest:path` then delete the original (if no errors on copy) in
|
||||
`source:path`.
|
||||
|
||||
**Important**: Since this can cause data loss, test first with the
|
||||
--dry-run flag.
|
||||
|
||||
|
||||
```
|
||||
rclone move source:path dest:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,94 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone purge"
|
||||
slug: rclone_purge
|
||||
url: /commands/rclone_purge/
|
||||
---
|
||||
## rclone purge
|
||||
|
||||
Remove the path and all of its contents.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Remove the path and all of its contents. Note that this does not obey
|
||||
include/exclude filters - everything will be removed. Use `delete` if
|
||||
you want to selectively delete files.
|
||||
|
||||
|
||||
```
|
||||
rclone purge remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,92 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone rmdir"
|
||||
slug: rclone_rmdir
|
||||
url: /commands/rclone_rmdir/
|
||||
---
|
||||
## rclone rmdir
|
||||
|
||||
Remove the path if empty.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Remove the path. Note that you can't remove a path with
|
||||
objects in it, use purge for that.
|
||||
|
||||
```
|
||||
rclone rmdir remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,93 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone sha1sum"
|
||||
slug: rclone_sha1sum
|
||||
url: /commands/rclone_sha1sum/
|
||||
---
|
||||
## rclone sha1sum
|
||||
|
||||
Produces an sha1sum file for all the objects in the path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Produces an sha1sum file for all the objects in the path. This
|
||||
is in the same format as the standard sha1sum tool produces.
|
||||
|
||||
|
||||
```
|
||||
rclone sha1sum remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone size"
|
||||
slug: rclone_size
|
||||
url: /commands/rclone_size/
|
||||
---
|
||||
## rclone size
|
||||
|
||||
Prints the total size and number of objects in remote:path.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
Prints the total size and number of objects in remote:path.
|
||||
|
||||
```
|
||||
rclone size remote:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,109 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone sync"
|
||||
slug: rclone_sync
|
||||
url: /commands/rclone_sync/
|
||||
---
|
||||
## rclone sync
|
||||
|
||||
Make source and dest identical, modifying destination only.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
|
||||
Sync the source to the destination, changing the destination
|
||||
only. Doesn't transfer unchanged files, testing by size and
|
||||
modification time or MD5SUM. Destination is updated to match
|
||||
source, including deleting files if necessary.
|
||||
|
||||
**Important**: Since this can cause data loss, test first with the
|
||||
`--dry-run` flag to see exactly what would be copied and deleted.
|
||||
|
||||
Note that files in the destination won't be deleted if there were any
|
||||
errors at any point.
|
||||
|
||||
It is always the contents of the directory that is synced, not the
|
||||
directory so when source:path is a directory, it's the contents of
|
||||
source:path that are copied, not the directory name and contents. See
|
||||
extended explanation in the `copy` command above if unsure.
|
||||
|
||||
If dest:path doesn't exist, it is created and the source:path contents
|
||||
go there.
|
||||
|
||||
|
||||
```
|
||||
rclone sync source:path dest:path
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
@@ -1,90 +0,0 @@
|
||||
---
|
||||
date: 2016-08-24T23:01:36+01:00
|
||||
title: "rclone version"
|
||||
slug: rclone_version
|
||||
url: /commands/rclone_version/
|
||||
---
|
||||
## rclone version
|
||||
|
||||
Show the version number.
|
||||
|
||||
### Synopsis
|
||||
|
||||
|
||||
Show the version number.
|
||||
|
||||
```
|
||||
rclone version
|
||||
```
|
||||
|
||||
### Options inherited from parent commands
|
||||
|
||||
```
|
||||
--acd-templink-threshold int Files >= this size will be downloaded via their tempLink.
|
||||
--acd-upload-wait-time duration Time to wait after a failed complete upload to see if it appears. (default 2m0s)
|
||||
--ask-password Allow prompt for password for encrypted configuration. (default true)
|
||||
--b2-chunk-size int Upload chunk size. Must fit in memory.
|
||||
--b2-test-mode string A flag string for X-Bz-Test-Mode header.
|
||||
--b2-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--b2-versions Include old versions in directory listings.
|
||||
--bwlimit int Bandwidth limit in kBytes/s, or use suffix b|k|M|G
|
||||
--checkers int Number of checkers to run in parallel. (default 8)
|
||||
-c, --checksum Skip based on checksum & size, not mod-time & size
|
||||
--config string Config file. (default "/home/ncw/.rclone.conf")
|
||||
--contimeout duration Connect timeout (default 1m0s)
|
||||
--cpuprofile string Write cpu profile to file
|
||||
--delete-after When synchronizing, delete files on destination after transfering
|
||||
--delete-before When synchronizing, delete files on destination before transfering
|
||||
--delete-during When synchronizing, delete files during transfer (default)
|
||||
--delete-excluded Delete files on dest excluded from sync
|
||||
--drive-auth-owner-only Only consider files owned by the authenticated user. Requires drive-full-list.
|
||||
--drive-chunk-size int Upload chunk size. Must a power of 2 >= 256k.
|
||||
--drive-formats string Comma separated list of preferred formats for downloading Google docs. (default "docx,xlsx,pptx,svg")
|
||||
--drive-full-list Use a full listing for directory list. More data but usually quicker. (obsolete)
|
||||
--drive-upload-cutoff int Cutoff for switching to chunked upload
|
||||
--drive-use-trash Send files to the trash instead of deleting permanently.
|
||||
--dropbox-chunk-size int Upload chunk size. Max 150M.
|
||||
-n, --dry-run Do a trial run with no permanent changes
|
||||
--dump-bodies Dump HTTP headers and bodies - may contain sensitive info
|
||||
--dump-filters Dump the filters to the output
|
||||
--dump-headers Dump HTTP headers - may contain sensitive info
|
||||
--exclude string Exclude files matching pattern
|
||||
--exclude-from string Read exclude patterns from file
|
||||
--files-from string Read list of source-file names from file
|
||||
-f, --filter string Add a file-filtering rule
|
||||
--filter-from string Read filtering patterns from a file
|
||||
--ignore-existing Skip all files that exist on destination
|
||||
--ignore-size Ignore size when skipping use mod-time or checksum.
|
||||
-I, --ignore-times Don't skip files that match size and time - transfer all files
|
||||
--include string Include files matching pattern
|
||||
--include-from string Read include patterns from file
|
||||
--log-file string Log everything to this file
|
||||
--low-level-retries int Number of low level retries to do. (default 10)
|
||||
--max-age string Don't transfer any file older than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--max-depth int If set limits the recursion depth to this. (default -1)
|
||||
--max-size int Don't transfer any file larger than this in k or suffix b|k|M|G
|
||||
--memprofile string Write memory profile to file
|
||||
--min-age string Don't transfer any file younger than this in s or suffix ms|s|m|h|d|w|M|y
|
||||
--min-size int Don't transfer any file smaller than this in k or suffix b|k|M|G
|
||||
--modify-window duration Max time diff to be considered the same (default 1ns)
|
||||
--no-check-certificate Do not verify the server SSL certificate. Insecure.
|
||||
--no-gzip-encoding Don't set Accept-Encoding: gzip.
|
||||
--no-traverse Don't traverse destination file system on copy.
|
||||
--no-update-modtime Don't update destination mod-time if files identical.
|
||||
--onedrive-chunk-size int Above this size files will be chunked - must be multiple of 320k.
|
||||
--onedrive-upload-cutoff int Cutoff for switching to chunked upload - must be <= 100MB
|
||||
-q, --quiet Print as little stuff as possible
|
||||
--retries int Retry operations this many times if they fail (default 3)
|
||||
--size-only Skip based on size only, not mod-time or checksum
|
||||
--stats duration Interval to print stats (0 to disable) (default 1m0s)
|
||||
--swift-chunk-size int Above this size files will be chunked into a _segments container.
|
||||
--timeout duration IO idle timeout (default 5m0s)
|
||||
--transfers int Number of file transfers to run in parallel. (default 4)
|
||||
-u, --update Skip files that are newer on the destination.
|
||||
-v, --verbose Print lots more stuff
|
||||
```
|
||||
|
||||
### SEE ALSO
|
||||
* [rclone](/commands/rclone/) - Sync files and directories to and from local and remote object stores - v1.33-DEV
|
||||
|
||||
###### Auto generated by spf13/cobra on 24-Aug-2016
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user