1
0
mirror of https://github.com/rclone/rclone.git synced 2025-12-06 00:03:32 +00:00

Compare commits

...

14 Commits

Author SHA1 Message Date
Nick Craig-Wood
f1a9d821e4 Version v1.49.2 2019-09-08 16:48:54 +01:00
Nick Craig-Wood
5fe78936d5 test_all: write index.json and add branch, commit and Go version to report 2019-09-08 11:38:18 +01:00
Nick Craig-Wood
4f3eee8d65 build: make sure we add version info to test_all build 2019-09-08 11:38:11 +01:00
Nick Craig-Wood
f2c05bc239 operations: fix -u/--update with google photos / files of unknown size
Before this change if -u/--update was in effect we compared the size
of the files to see if the transfer should go ahead.  This was
comparing -1 with an actual size so the transfer always proceeded.

After this change we use the existing `sizeDiffers` function which
does the correct comparison with -1 for files of unknown length.

See: https://forum.rclone.org/t/sync-with-google-photos-to-local-drive-will-result-in-recoping/11605
2019-09-06 10:11:59 +01:00
Nick Craig-Wood
b463032901 accounting: fix locking in Transfer to avoid deadlock with --progress
Before this change, using -P occasionally deadlocked on the Transfer
mutex when Transfer.Done() was called with a non nil error and the
StatsInfo mutex since they mutually call each other.

This was fixed by making sure that the Transfer mutex is always
released before calling any StatsInfo methods.

This improves on: 6f87267b34

Fixes #3505
2019-09-06 10:10:53 +01:00
Nick Craig-Wood
358decb933 rc: fix docs for config/create /update /password 2019-09-03 08:33:56 +01:00
Nick Craig-Wood
cefa2df3b2 docs: add info on how to build and use the docker images 2019-09-02 14:31:19 +01:00
Alfonso Montero
52efb7e6d0 Add Docker workflow support #3460
* Use a multi-stage build to reduce final image size.
* Run 'quicktest' make target before building.
* Built binary won't run on Alpine unless statically linked.
2019-09-02 14:31:10 +01:00
Nick Craig-Wood
01fa6835c7 gcs: fix need for elevated permissions on SetModTime - fixes #3493
Before this change we used PATCH on the object to update the metadata.

Apparently this requires the "full_control" scope which Google were
unhappy with in their oauth review.

This changes it to update the metadata by copying the object ontop of
itself (which is the way s3 works).  This can be done with normal
permissions.
2019-09-02 12:04:45 +01:00
Cnly
8adf22e294 docs: fix template argument for mktemp in install.sh 2019-09-02 12:04:33 +01:00
Nick Craig-Wood
45f7c687e2 Version v1.49.1 2019-08-28 17:51:23 +01:00
Nick Craig-Wood
a05dd6fc27 config: Fix generated passwords being stored as empty password - Fixes #3492 2019-08-28 14:24:18 +01:00
Nick Craig-Wood
642cb03121 googlephotos,onedrive: fix crash on error response - fixes #3491
This fixes a crash on the google photos backend when an error is
returned from the rest.Call function.

This turned out to be a mis-understanding of the rest docs so
- improved rest.Call docs
- fixed mis-understanding in google photos backend
- fixed similar mis-understading in onedrive backend
2019-08-28 14:24:08 +01:00
Chaitanya
da4dfdc3ec rcd: Added missing parameter for web-gui info logs. 2019-08-28 14:24:04 +01:00
95 changed files with 666 additions and 151 deletions

21
Dockerfile Normal file
View File

@@ -0,0 +1,21 @@
FROM golang AS builder
COPY . /go/src/github.com/rclone/rclone/
WORKDIR /go/src/github.com/rclone/rclone/
RUN make quicktest
RUN \
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
make
RUN ./rclone version
# Begin final image
FROM alpine:latest
RUN apk --no-cache add ca-certificates
WORKDIR /root/
COPY --from=builder /go/src/github.com/rclone/rclone/rclone .
ENTRYPOINT [ "./rclone" ]

65
MANUAL.html generated
View File

@@ -17,7 +17,7 @@
<header>
<h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p>
<p class="date">Aug 26, 2019</p>
<p class="date">Sep 08, 2019</p>
</header>
<h1 id="rclone---rsync-for-cloud-storage">Rclone - rsync for cloud storage</h1>
<p>Rclone is a command line program to sync files and directories to and from:</p>
@@ -134,6 +134,20 @@ sudo mv rclone /usr/local/bin/</code></pre>
<pre><code>cd .. &amp;&amp; rm -rf rclone-*-osx-amd64 rclone-current-osx-amd64.zip</code></pre>
<p>Run <code>rclone config</code> to setup. See <a href="https://rclone.org/docs/">rclone config docs</a> for more details.</p>
<pre><code>rclone config</code></pre>
<h2 id="install-with-docker">Install with docker</h2>
<p>The rclone maintains a <a href="https://hub.docker.com/r/rclone/rclone">docker image for rclone</a>. These images are autobuilt by docker hub from the rclone source based on a minimal Alpine linux image.</p>
<p>The <code>:latest</code> tag will always point to the latest stable release. You can use the <code>:beta</code> tag to get the latest build from master. You can also use version tags, eg <code>:1.49.1</code>, <code>:1.49</code> or <code>:1</code>.</p>
<pre><code>$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9</code></pre>
<p>You will probably want to mount rclones config file directory or file from the host, or configure rclone with environment variables.</p>
<p>Eg to share your local config with the container</p>
<pre><code>docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes</code></pre>
<h2 id="install-from-source">Install from source</h2>
<p>Make sure you have at least <a href="https://golang.org/">Go</a> 1.7 installed. <a href="https://golang.org/dl/">Download go</a> if necessary. The latest release is recommended. Then</p>
<pre><code>git clone https://github.com/rclone/rclone.git
@@ -3301,12 +3315,16 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
<li>type - type of the new remote</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_create/">config create command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
<h3 id="configdelete-delete-a-remote-in-the-config-file.-configdelete">config/delete: Delete a remote in the config file. {#config/delete}</h3>
<p>Parameters: - name - name of remote to delete</p>
<p>Parameters:</p>
<ul>
<li>name - name of remote to delete</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_delete/">config delete command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
<h3 id="configdump-dumps-the-config-file.-configdump">config/dump: Dumps the config file. {#config/dump}</h3>
@@ -3326,6 +3344,7 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_password/">config password command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
@@ -3337,6 +3356,7 @@ rclone rc cache/expire remote=/ withData=true</code></pre>
<p>This takes the following parameters</p>
<ul>
<li>name - name of remote</li>
<li>parameters - a map of { “key”: “value” } pairs</li>
</ul>
<p>See the <a href="https://rclone.org/commands/rclone_config_update/">config update command</a> command for more information on the above.</p>
<p>Authentication is required for this call.</p>
@@ -4590,7 +4610,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.49.0&quot;)
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.49.2&quot;)
-v, --verbose count Print lots more stuff (repeat for more)</code></pre>
<h2 id="backend-flags">Backend Flags</h2>
<p>These flags are available for every command. They control the backends and may be set in the config file.</p>
@@ -12288,6 +12308,41 @@ $ tree /tmp/b
</ul>
<!--- autogenerated options stop -->
<h1 id="changelog">Changelog</h1>
<h2 id="v1.49.2---2019-09-08">v1.49.2 - 2019-09-08</h2>
<ul>
<li>New Features
<ul>
<li>build: Add Docker workflow support (Alfonso Montero)</li>
</ul></li>
<li>Bug Fixes
<ul>
<li>accounting: Fix locking in Transfer to avoid deadlock with progress (Nick Craig-Wood)</li>
<li>docs: Fix template argument for mktemp in install.sh (Cnly)</li>
<li>operations: Fix -u/update with google photos / files of unknown size (Nick Craig-Wood)</li>
<li>rc: Fix docs for config/create /update /password (Nick Craig-Wood)</li>
</ul></li>
<li>Google Cloud Storage
<ul>
<li>Fix need for elevated permissions on SetModTime (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.1---2019-08-28">v1.49.1 - 2019-08-28</h2>
<p>Point release to fix config bug and google photos backend.</p>
<ul>
<li>Bug Fixes
<ul>
<li>config: Fix generated passwords being stored as empty password (Nick Craig-Wood)</li>
<li>rcd: Added missing parameter for web-gui info logs. (Chaitanya)</li>
</ul></li>
<li>Googlephotos
<ul>
<li>Fix crash on error response (Nick Craig-Wood)</li>
</ul></li>
<li>Onedrive
<ul>
<li>Fix crash on error response (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.49.0---2019-08-26">v1.49.0 - 2019-08-26</h2>
<ul>
<li>New backends
@@ -12302,8 +12357,10 @@ $ tree /tmp/b
<li>Experimental <a href="https://rclone.org/gui/">web GUI</a> (Chaitanya Bankanhal)</li>
<li>Implement <code>--compare-dest</code> &amp; <code>--copy-dest</code> (yparitcher)</li>
<li>Implement <code>--suffix</code> without <code>--backup-dir</code> for backup to current dir (yparitcher)</li>
<li><code>config reconnect</code> to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood)</li>
<li><code>config userinfo</code> to discover which user you are logged in as. (Nick Craig-Wood)</li>
<li><code>config disconnect</code> to disconnect you (log out) from the backend. (Nick Craig-Wood)</li>
<li>Add <code>--use-json-log</code> for JSON logging (justinalin)</li>
<li>Add <code>config reconnect</code>, <code>config userinfo</code> and <code>config disconnect</code> subcommands. (Nick Craig-Wood)</li>
<li>Add context propagation to rclone (Aleksandar Jankovic)</li>
<li>Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic)</li>
<li>Add Higher units for ETA (AbelThar)</li>

66
MANUAL.md generated
View File

@@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Aug 26, 2019
% Sep 08, 2019
# Rclone - rsync for cloud storage
@@ -151,6 +151,36 @@ Run `rclone config` to setup. See [rclone config docs](https://rclone.org/docs/)
rclone config
## Install with docker ##
The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
These images are autobuilt by docker hub from the rclone source based
on a minimal Alpine linux image.
The `:latest` tag will always point to the latest stable release. You
can use the `:beta` tag to get the latest build from master. You can
also use version tags, eg `:1.49.1`, `:1.49` or `:1`.
```
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
```
You will probably want to mount rclone's config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
```
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
```
## Install from source ##
Make sure you have at least [Go](https://golang.org/) 1.7
@@ -7010,6 +7040,7 @@ Show statistics for the cache remote.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
- type - type of the new remote
@@ -7020,6 +7051,7 @@ Authentication is required for this call.
### config/delete: Delete a remote in the config file. {#config/delete}
Parameters:
- name - name of remote to delete
See the [config delete command](https://rclone.org/commands/rclone_config_delete/) command for more information on the above.
@@ -7060,6 +7092,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config password command](https://rclone.org/commands/rclone_config_password/) command for more information on the above.
@@ -7080,6 +7113,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config update command](https://rclone.org/commands/rclone_config_update/) command for more information on the above.
@@ -8245,7 +8279,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.2")
-v, --verbose count Print lots more stuff (repeat for more)
```
@@ -18466,6 +18500,30 @@ to override the default choice.
# Changelog
## v1.49.2 - 2019-09-08
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
* Googlephotos
* Fix crash on error response (Nick Craig-Wood)
* Onedrive
* Fix crash on error response (Nick Craig-Wood)
## v1.49.0 - 2019-08-26
* New backends
@@ -18477,8 +18535,10 @@ to override the default choice.
* Experimental [web GUI](https://rclone.org/gui/) (Chaitanya Bankanhal)
* Implement `--compare-dest` & `--copy-dest` (yparitcher)
* Implement `--suffix` without `--backup-dir` for backup to current dir (yparitcher)
* `config reconnect` to re-login (re-run the oauth login) for the backend. (Nick Craig-Wood)
* `config userinfo` to discover which user you are logged in as. (Nick Craig-Wood)
* `config disconnect` to disconnect you (log out) from the backend. (Nick Craig-Wood)
* Add `--use-json-log` for JSON logging (justinalin)
* Add `config reconnect`, `config userinfo` and `config disconnect` subcommands. (Nick Craig-Wood)
* Add context propagation to rclone (Aleksandar Jankovic)
* Reworking internal statistics interfaces so they work with rc jobs (Aleksandar Jankovic)
* Add Higher units for ETA (AbelThar)

77
MANUAL.txt generated
View File

@@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Aug 26, 2019
Sep 08, 2019
@@ -164,6 +164,33 @@ Run rclone config to setup. See rclone config docs for more details.
rclone config
Install with docker
The rclone maintains a docker image for rclone. These images are
autobuilt by docker hub from the rclone source based on a minimal Alpine
linux image.
The :latest tag will always point to the latest stable release. You can
use the :beta tag to get the latest build from master. You can also use
version tags, eg :1.49.1, :1.49 or :1.
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
You will probably want to mount rclones config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
Install from source
Make sure you have at least Go 1.7 installed. Download go if necessary.
@@ -6650,6 +6677,7 @@ config/create: create the config for a remote. {#config/create}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
- type - type of the new remote
See the config create command command for more information on the above.
@@ -6658,7 +6686,9 @@ Authentication is required for this call.
config/delete: Delete a remote in the config file. {#config/delete}
Parameters: - name - name of remote to delete
Parameters:
- name - name of remote to delete
See the config delete command command for more information on the above.
@@ -6695,6 +6725,7 @@ config/password: password the config for a remote. {#config/password}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
See the config password command command for more information on the
above.
@@ -6715,6 +6746,7 @@ config/update: update the config for a remote. {#config/update}
This takes the following parameters
- name - name of remote
- parameters - a map of { “key”: “value” } pairs
See the config update command command for more information on the above.
@@ -7824,7 +7856,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.2")
-v, --verbose count Print lots more stuff (repeat for more)
@@ -17923,6 +17955,37 @@ override the default choice.
CHANGELOG
v1.49.2 - 2019-09-08
- New Features
- build: Add Docker workflow support (Alfonso Montero)
- Bug Fixes
- accounting: Fix locking in Transfer to avoid deadlock with
progress (Nick Craig-Wood)
- docs: Fix template argument for mktemp in install.sh (Cnly)
- operations: Fix -u/update with google photos / files of unknown
size (Nick Craig-Wood)
- rc: Fix docs for config/create /update /password (Nick
Craig-Wood)
- Google Cloud Storage
- Fix need for elevated permissions on SetModTime (Nick
Craig-Wood)
v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
- Bug Fixes
- config: Fix generated passwords being stored as empty password
(Nick Craig-Wood)
- rcd: Added missing parameter for web-gui info logs. (Chaitanya)
- Googlephotos
- Fix crash on error response (Nick Craig-Wood)
- Onedrive
- Fix crash on error response (Nick Craig-Wood)
v1.49.0 - 2019-08-26
- New backends
@@ -17935,9 +17998,13 @@ v1.49.0 - 2019-08-26
- Implement --compare-dest & --copy-dest (yparitcher)
- Implement --suffix without --backup-dir for backup to current
dir (yparitcher)
- config reconnect to re-login (re-run the oauth login) for the
backend. (Nick Craig-Wood)
- config userinfo to discover which user you are logged in as.
(Nick Craig-Wood)
- config disconnect to disconnect you (log out) from the backend.
(Nick Craig-Wood)
- Add --use-json-log for JSON logging (justinalin)
- Add config reconnect, config userinfo and config disconnect
subcommands. (Nick Craig-Wood)
- Add context propagation to rclone (Aleksandar Jankovic)
- Reworking internal statistics interfaces so they work with rc
jobs (Aleksandar Jankovic)

View File

@@ -30,13 +30,15 @@ BUILDTAGS=-tags "$(GOTAGS)"
LINTTAGS=--build-tags "$(GOTAGS)"
endif
.PHONY: rclone vars version
.PHONY: rclone test_all vars version
rclone:
touch fs/version.go
go install -v --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS)
cp -av `go env GOPATH`/bin/rclone .
test_all:
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
vars:
@echo SHELL="'$(SHELL)'"
@echo BRANCH="'$(BRANCH)'"
@@ -50,8 +52,7 @@ version:
@echo '$(TAG)'
# Full suite of integration tests
test: rclone
go install --ldflags "-s -X github.com/rclone/rclone/fs.Version=$(TAG)" $(BUILDTAGS) github.com/rclone/rclone/fstest/test_all
test: rclone test_all
-test_all 2>&1 | tee test_all.log
@echo "Written logs in test_all.log"

View File

@@ -69,3 +69,16 @@ horrendous bug, then
* make TAG=v1.43.1 upload_github
* NB this overwrites the current beta so after the release, rebuild the last travis build
* Announce!
## Making a manual build of docker
The rclone docker image should autobuild on docker hub. If it doesn't
or needs to be updated then rebuild like this.
```
docker build -t rclone/rclone:1.49.1 -t rclone/rclone:1.49 -t rclone/rclone:1 -t rclone/rclone:latest .
docker push rclone/rclone:1.49.1
docker push rclone/rclone:1.49
docker push rclone/rclone:1
docker push rclone/rclone:latest
```

View File

@@ -824,7 +824,11 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
var newObject *storage.Object
err = f.pacer.Call(func() (bool, error) {
newObject, err = f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil).Do()
copyObject := f.svc.Objects.Copy(srcBucket, srcPath, dstBucket, dstPath, nil)
if !f.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(f.opt.ObjectACL)
}
newObject, err = copyObject.Do()
return shouldRetry(err)
})
if err != nil {
@@ -907,15 +911,9 @@ func (o *Object) setMetaData(info *storage.Object) {
}
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
if !o.modTime.IsZero() {
return nil
}
// readObjectInfo reads the definition for an object
func (o *Object) readObjectInfo() (object *storage.Object, err error) {
bucket, bucketPath := o.split()
var object *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
object, err = o.fs.svc.Objects.Get(bucket, bucketPath).Do()
return shouldRetry(err)
@@ -923,9 +921,23 @@ func (o *Object) readMetaData() (err error) {
if err != nil {
if gErr, ok := err.(*googleapi.Error); ok {
if gErr.Code == http.StatusNotFound {
return fs.ErrorObjectNotFound
return nil, fs.ErrorObjectNotFound
}
}
return nil, err
}
return object, nil
}
// readMetaData gets the metadata if it hasn't already been fetched
//
// it also sets the info
func (o *Object) readMetaData() (err error) {
if !o.modTime.IsZero() {
return nil
}
object, err := o.readObjectInfo()
if err != nil {
return err
}
o.setMetaData(object)
@@ -954,16 +966,27 @@ func metadataFromModTime(modTime time.Time) map[string]string {
// SetModTime sets the modification time of the local fs object
func (o *Object) SetModTime(ctx context.Context, modTime time.Time) (err error) {
// This only adds metadata so will perserve other metadata
bucket, bucketPath := o.split()
object := storage.Object{
Bucket: bucket,
Name: bucketPath,
Metadata: metadataFromModTime(modTime),
// read the complete existing object first
object, err := o.readObjectInfo()
if err != nil {
return err
}
// Add the mtime to the existing metadata
mtime := modTime.Format(timeFormatOut)
if object.Metadata == nil {
object.Metadata = make(map[string]string, 1)
}
object.Metadata[metaMtime] = mtime
// Copy the object to itself to update the metadata
// Using PATCH requires too many permissions
bucket, bucketPath := o.split()
var newObject *storage.Object
err = o.fs.pacer.Call(func() (bool, error) {
newObject, err = o.fs.svc.Objects.Patch(bucket, bucketPath, &object).Do()
copyObject := o.fs.svc.Objects.Copy(bucket, bucketPath, bucket, bucketPath, object)
if !o.fs.opt.BucketPolicyOnly {
copyObject.DestinationPredefinedAcl(o.fs.opt.ObjectACL)
}
newObject, err = copyObject.Do()
return shouldRetry(err)
})
if err != nil {

View File

@@ -956,7 +956,6 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
resp, err = o.fs.srv.Call(&opts)
if err != nil {
_ = resp.Body.Close()
return shouldRetry(resp, err)
}
token, err = rest.ReadBody(resp)

View File

@@ -1464,22 +1464,24 @@ func (o *Object) uploadFragment(url string, start int64, totalSize int64, chunk
}
// var response api.UploadFragmentResponse
var resp *http.Response
var body []byte
err = o.fs.pacer.Call(func() (bool, error) {
_, _ = chunk.Seek(0, io.SeekStart)
resp, err = o.fs.srv.Call(&opts)
if resp != nil {
defer fs.CheckClose(resp.Body, &err)
if err != nil {
return shouldRetry(resp, err)
}
retry, err := shouldRetry(resp, err)
if !retry && resp != nil {
if resp.StatusCode == 200 || resp.StatusCode == 201 {
// we are done :)
// read the item
info = &api.Item{}
return false, json.NewDecoder(resp.Body).Decode(info)
}
body, err = rest.ReadBody(resp)
if err != nil {
return shouldRetry(resp, err)
}
return retry, err
if resp.StatusCode == 200 || resp.StatusCode == 201 {
// we are done :)
// read the item
info = &api.Item{}
return false, json.Unmarshal(body, info)
}
return false, nil
})
return info, err
}

View File

@@ -61,7 +61,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
}
if rcflags.Opt.HTTPOptions.BasicUser == "" {
rcflags.Opt.HTTPOptions.BasicUser = "gui"
fs.Infof("Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
fs.Infof(nil, "Using default username: %s \n", rcflags.Opt.HTTPOptions.BasicUser)
}
if rcflags.Opt.HTTPOptions.BasicPass == "" {
randomPass, err := random.Password(128)
@@ -69,7 +69,7 @@ See the [rc documentation](/rc/) for more info on the rc flags.
log.Fatalf("Failed to make password: %v", err)
}
rcflags.Opt.HTTPOptions.BasicPass = randomPass
fs.Infof("No password specified. Using random password: %s \n", randomPass)
fs.Infof(nil, "No password specified. Using random password: %s \n", randomPass)
}
rcflags.Opt.Serve = true
}

View File

@@ -1,11 +1,35 @@
---
title: "Documentation"
description: "Rclone Changelog"
date: "2019-08-26"
date: "2019-09-08"
---
# Changelog
## v1.49.2 - 2019-09-08
* New Features
* build: Add Docker workflow support (Alfonso Montero)
* Bug Fixes
* accounting: Fix locking in Transfer to avoid deadlock with --progress (Nick Craig-Wood)
* docs: Fix template argument for mktemp in install.sh (Cnly)
* operations: Fix -u/--update with google photos / files of unknown size (Nick Craig-Wood)
* rc: Fix docs for config/create /update /password (Nick Craig-Wood)
* Google Cloud Storage
* Fix need for elevated permissions on SetModTime (Nick Craig-Wood)
## v1.49.1 - 2019-08-28
Point release to fix config bug and google photos backend.
* Bug Fixes
* config: Fix generated passwords being stored as empty password (Nick Craig-Wood)
* rcd: Added missing parameter for web-gui info logs. (Chaitanya)
* Googlephotos
* Fix crash on error response (Nick Craig-Wood)
* Onedrive
* Fix crash on error response (Nick Craig-Wood)
## v1.49.0 - 2019-08-26
* New backends

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone"
slug: rclone
url: /commands/rclone/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone about"
slug: rclone_about
url: /commands/rclone_about/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone authorize"
slug: rclone_authorize
url: /commands/rclone_authorize/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone cachestats"
slug: rclone_cachestats
url: /commands/rclone_cachestats/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone cat"
slug: rclone_cat
url: /commands/rclone_cat/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone check"
slug: rclone_check
url: /commands/rclone_check/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone cleanup"
slug: rclone_cleanup
url: /commands/rclone_cleanup/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config"
slug: rclone_config
url: /commands/rclone_config/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config create"
slug: rclone_config_create
url: /commands/rclone_config_create/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config delete"
slug: rclone_config_delete
url: /commands/rclone_config_delete/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config disconnect"
slug: rclone_config_disconnect
url: /commands/rclone_config_disconnect/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config dump"
slug: rclone_config_dump
url: /commands/rclone_config_dump/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config edit"
slug: rclone_config_edit
url: /commands/rclone_config_edit/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config file"
slug: rclone_config_file
url: /commands/rclone_config_file/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config password"
slug: rclone_config_password
url: /commands/rclone_config_password/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config providers"
slug: rclone_config_providers
url: /commands/rclone_config_providers/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config reconnect"
slug: rclone_config_reconnect
url: /commands/rclone_config_reconnect/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config show"
slug: rclone_config_show
url: /commands/rclone_config_show/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config update"
slug: rclone_config_update
url: /commands/rclone_config_update/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone config userinfo"
slug: rclone_config_userinfo
url: /commands/rclone_config_userinfo/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone copy"
slug: rclone_copy
url: /commands/rclone_copy/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone copyto"
slug: rclone_copyto
url: /commands/rclone_copyto/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone copyurl"
slug: rclone_copyurl
url: /commands/rclone_copyurl/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone cryptcheck"
slug: rclone_cryptcheck
url: /commands/rclone_cryptcheck/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone cryptdecode"
slug: rclone_cryptdecode
url: /commands/rclone_cryptdecode/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone dbhashsum"
slug: rclone_dbhashsum
url: /commands/rclone_dbhashsum/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone dedupe"
slug: rclone_dedupe
url: /commands/rclone_dedupe/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone delete"
slug: rclone_delete
url: /commands/rclone_delete/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone deletefile"
slug: rclone_deletefile
url: /commands/rclone_deletefile/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone genautocomplete"
slug: rclone_genautocomplete
url: /commands/rclone_genautocomplete/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone genautocomplete bash"
slug: rclone_genautocomplete_bash
url: /commands/rclone_genautocomplete_bash/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone genautocomplete zsh"
slug: rclone_genautocomplete_zsh
url: /commands/rclone_genautocomplete_zsh/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone gendocs"
slug: rclone_gendocs
url: /commands/rclone_gendocs/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone hashsum"
slug: rclone_hashsum
url: /commands/rclone_hashsum/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone link"
slug: rclone_link
url: /commands/rclone_link/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone listremotes"
slug: rclone_listremotes
url: /commands/rclone_listremotes/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone ls"
slug: rclone_ls
url: /commands/rclone_ls/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone lsd"
slug: rclone_lsd
url: /commands/rclone_lsd/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone lsf"
slug: rclone_lsf
url: /commands/rclone_lsf/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone lsjson"
slug: rclone_lsjson
url: /commands/rclone_lsjson/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone lsl"
slug: rclone_lsl
url: /commands/rclone_lsl/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone md5sum"
slug: rclone_md5sum
url: /commands/rclone_md5sum/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone mkdir"
slug: rclone_mkdir
url: /commands/rclone_mkdir/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone mount"
slug: rclone_mount
url: /commands/rclone_mount/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone move"
slug: rclone_move
url: /commands/rclone_move/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone moveto"
slug: rclone_moveto
url: /commands/rclone_moveto/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone ncdu"
slug: rclone_ncdu
url: /commands/rclone_ncdu/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone obscure"
slug: rclone_obscure
url: /commands/rclone_obscure/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone purge"
slug: rclone_purge
url: /commands/rclone_purge/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone rc"
slug: rclone_rc
url: /commands/rclone_rc/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone rcat"
slug: rclone_rcat
url: /commands/rclone_rcat/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone rcd"
slug: rclone_rcd
url: /commands/rclone_rcd/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone rmdir"
slug: rclone_rmdir
url: /commands/rclone_rmdir/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone rmdirs"
slug: rclone_rmdirs
url: /commands/rclone_rmdirs/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve"
slug: rclone_serve
url: /commands/rclone_serve/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve dlna"
slug: rclone_serve_dlna
url: /commands/rclone_serve_dlna/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve ftp"
slug: rclone_serve_ftp
url: /commands/rclone_serve_ftp/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve http"
slug: rclone_serve_http
url: /commands/rclone_serve_http/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve restic"
slug: rclone_serve_restic
url: /commands/rclone_serve_restic/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve sftp"
slug: rclone_serve_sftp
url: /commands/rclone_serve_sftp/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone serve webdav"
slug: rclone_serve_webdav
url: /commands/rclone_serve_webdav/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone settier"
slug: rclone_settier
url: /commands/rclone_settier/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone sha1sum"
slug: rclone_sha1sum
url: /commands/rclone_sha1sum/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone size"
slug: rclone_size
url: /commands/rclone_size/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone sync"
slug: rclone_sync
url: /commands/rclone_sync/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone touch"
slug: rclone_touch
url: /commands/rclone_touch/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone tree"
slug: rclone_tree
url: /commands/rclone_tree/

View File

@@ -1,5 +1,5 @@
---
date: 2019-08-26T15:19:45+01:00
date: 2019-09-08T16:48:33+01:00
title: "rclone version"
slug: rclone_version
url: /commands/rclone_version/

View File

@@ -1,7 +1,7 @@
---
title: "Global Flags"
description: "Rclone Global Flags"
date: "2019-08-26T15:19:45+01:00"
date: "2019-09-08T16:48:33+01:00"
---
# Global Flags
@@ -127,7 +127,7 @@ These flags are available for every command.
--use-json-log Use json log format.
--use-mmap Use mmap allocator (see docs).
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.0")
--user-agent string Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.49.2")
-v, --verbose count Print lots more stuff (repeat for more)
```

View File

@@ -81,6 +81,36 @@ Run `rclone config` to setup. See [rclone config docs](/docs/) for more details.
rclone config
## Install with docker ##
The rclone maintains a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
These images are autobuilt by docker hub from the rclone source based
on a minimal Alpine linux image.
The `:latest` tag will always point to the latest stable release. You
can use the `:beta` tag to get the latest build from master. You can
also use version tags, eg `:1.49.1`, `:1.49` or `:1`.
```
$ docker pull rclone/rclone:latest
latest: Pulling from rclone/rclone
Digest: sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
...
$ docker run --rm rclone/rclone:latest version
rclone v1.49.1
- os/arch: linux/amd64
- go version: go1.12.9
```
You will probably want to mount rclone's config file directory or file
from the host, or configure rclone with environment variables.
Eg to share your local config with the container
```
docker run -v ~/.config/rclone:/root/.config/rclone rclone/rclone:latest listremotes
```
## Install from source ##
Make sure you have at least [Go](https://golang.org/) 1.7

View File

@@ -25,7 +25,7 @@ fi
#create tmp directory and move to it with macOS compatibility fallback
tmp_dir=`mktemp -d 2>/dev/null || mktemp -d -t 'rclone-install'`; cd $tmp_dir
tmp_dir=`mktemp -d 2>/dev/null || mktemp -d -t 'rclone-install.XXXXXXXXXX'`; cd $tmp_dir
#make sure unzip tool is available and choose one to work with

View File

@@ -308,6 +308,7 @@ Show statistics for the cache remote.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
- type - type of the new remote
@@ -318,6 +319,7 @@ Authentication is required for this call.
### config/delete: Delete a remote in the config file. {#config/delete}
Parameters:
- name - name of remote to delete
See the [config delete command](/commands/rclone_config_delete/) command for more information on the above.
@@ -358,6 +360,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config password command](/commands/rclone_config_password/) command for more information on the above.
@@ -378,6 +381,7 @@ Authentication is required for this call.
This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
See the [config update command](/commands/rclone_config_update/) command for more information on the above.

View File

@@ -1 +1 @@
v1.49.0
v1.49.2

View File

@@ -48,6 +48,10 @@ type Transfer struct {
checking bool
// Protects all below
//
// NB to avoid deadlocks we must release this lock before
// calling any methods on Transfer.stats. This is because
// StatsInfo calls back into Transfer.
mu sync.RWMutex
acc *Account
err error
@@ -79,20 +83,26 @@ func newTransferRemoteSize(stats *StatsInfo, remote string, size int64, checking
// Done ends the transfer.
// Must be called after transfer is finished to run proper cleanups.
func (tr *Transfer) Done(err error) {
tr.mu.Lock()
if err != nil {
tr.stats.Error(err)
tr.mu.Lock()
tr.err = err
tr.mu.Unlock()
}
if tr.acc != nil {
if err := tr.acc.Close(); err != nil {
tr.mu.RLock()
acc := tr.acc
tr.mu.RUnlock()
if acc != nil {
if err := acc.Close(); err != nil {
fs.LogLevelPrintf(fs.Config.StatsLogLevel, nil, "can't close account: %+v\n", err)
}
}
tr.mu.Lock()
tr.completedAt = time.Now()
tr.mu.Unlock()
if tr.checking {

View File

@@ -89,6 +89,9 @@ var (
// For security reasons, the temp file is deleted once the configKey is successfully loaded.
// This can be used to pass the configKey to a child process.
PassConfigKeyForDaemonization = false
// Password can be used to configure the random password generator
Password = random.Password
)
func init() {
@@ -854,6 +857,7 @@ func ChooseOption(o *fs.Option, name string) string {
actions = append(actions, "nNo leave this optional password blank")
}
var password string
var err error
switch i := Command(actions); i {
case 'y':
password = ChangePassword("the")
@@ -861,7 +865,7 @@ func ChooseOption(o *fs.Option, name string) string {
for {
fmt.Printf("Password strength in bits.\n64 is just about memorable\n128 is secure\n1024 is the maximum\n")
bits := ChooseNumber("Bits", 64, 1024)
password, err := random.Password(bits)
password, err = Password(bits)
if err != nil {
log.Fatalf("Failed to make password: %v", err)
}

View File

@@ -30,6 +30,7 @@ func testConfigFile(t *testing.T, configFileName string) func() {
oldConfig := fs.Config
oldConfigFile := configFile
oldReadLine := ReadLine
oldPassword := Password
os.Stdout = nil
ConfigPath = path
fs.Config = &fs.ConfigInfo{}
@@ -63,6 +64,7 @@ func testConfigFile(t *testing.T, configFileName string) func() {
os.Stdout = oldOsStdout
ConfigPath = oldConfigPath
ReadLine = oldReadLine
Password = oldPassword
fs.Config = oldConfig
configFile = oldConfigFile
@@ -71,24 +73,28 @@ func testConfigFile(t *testing.T, configFileName string) func() {
}
}
func TestCRUD(t *testing.T) {
defer testConfigFile(t, "crud.conf")()
// expect script for creating remote
// makeReadLine makes a simple readLine which returns a fixed list of
// strings
func makeReadLine(answers []string) func() string {
i := 0
ReadLine = func() string {
answers := []string{
"config_test_remote", // type
"true", // bool value
"y", // type my own password
"secret", // password
"secret", // repeat
"y", // looks good, save
}
return func() string {
i = i + 1
return answers[i-1]
}
}
func TestCRUD(t *testing.T) {
defer testConfigFile(t, "crud.conf")()
// script for creating remote
ReadLine = makeReadLine([]string{
"config_test_remote", // type
"true", // bool value
"y", // type my own password
"secret", // password
"secret", // repeat
"y", // looks good, save
})
NewRemote("test")
assert.Equal(t, []string{"test"}, configFile.GetSectionList())
@@ -97,7 +103,11 @@ func TestCRUD(t *testing.T) {
assert.Equal(t, "secret", obscure.MustReveal(FileGet("test", "pass")))
// normal rename, test → asdf
ReadLine = func() string { return "asdf" }
ReadLine = makeReadLine([]string{
"asdf",
"asdf",
"asdf",
})
RenameRemote("test")
assert.Equal(t, []string{"asdf"}, configFile.GetSectionList())
@@ -118,6 +128,41 @@ func TestCRUD(t *testing.T) {
assert.Equal(t, []string{}, configFile.GetSectionList())
}
func TestChooseOption(t *testing.T) {
defer testConfigFile(t, "crud.conf")()
// script for creating remote
ReadLine = makeReadLine([]string{
"config_test_remote", // type
"false", // bool value
"x", // bad choice
"g", // generate password
"1024", // very big
"y", // password OK
"y", // looks good, save
})
Password = func(bits int) (string, error) {
assert.Equal(t, 1024, bits)
return "not very random password", nil
}
NewRemote("test")
assert.Equal(t, "false", FileGet("test", "bool"))
assert.Equal(t, "not very random password", obscure.MustReveal(FileGet("test", "pass")))
// script for creating remote
ReadLine = makeReadLine([]string{
"config_test_remote", // type
"true", // bool value
"n", // not required
"y", // looks good, save
})
NewRemote("test")
assert.Equal(t, "true", FileGet("test", "bool"))
assert.Equal(t, "", FileGet("test", "pass"))
}
func TestCreateUpatePasswordRemote(t *testing.T) {
defer testConfigFile(t, "update.conf")()

View File

@@ -120,6 +120,7 @@ func init() {
Help: `This takes the following parameters
- name - name of remote
- parameters - a map of \{ "key": "value" \} pairs
` + extraHelp + `
See the [config ` + name + ` command](/commands/rclone_config_` + name + `/) command for more information on the above.`,
@@ -161,6 +162,7 @@ func init() {
AuthRequired: true,
Help: `
Parameters:
- name - name of remote to delete
See the [config delete command](/commands/rclone_config_delete/) command for more information on the above.

View File

@@ -1522,7 +1522,7 @@ func NeedTransfer(ctx context.Context, dst, src fs.Object) bool {
case dt <= -modifyWindow:
fs.Debugf(src, "Destination is older than source, transferring")
default:
if src.Size() == dst.Size() {
if !sizeDiffers(src, dst) {
fs.Debugf(src, "Destination mod time is within %v of source and sizes identical, skipping", modifyWindow)
return false
}

View File

@@ -1,4 +1,4 @@
package fs
// Version of rclone
var Version = "v1.49.0"
var Version = "v1.49.2"

View File

@@ -3,6 +3,7 @@
package main
import (
"encoding/json"
"fmt"
"html/template"
"io/ioutil"
@@ -10,6 +11,8 @@ import (
"os"
"os/exec"
"path"
"regexp"
"runtime"
"sort"
"time"
@@ -32,6 +35,11 @@ type Report struct {
Previous string // previous test name if known
IndexHTML string // path to the index.html file
URL string // online version
Branch string // rclone branch
Commit string // rclone commit
GOOS string // Go OS
GOARCH string // Go Arch
GoVersion string // Go Version
}
// ReportRun is used in the templates to report on a test run
@@ -40,11 +48,24 @@ type ReportRun struct {
Runs Runs
}
// Parse version numbers
// v1.49.0
// v1.49.0-031-g2298834e-beta
// v1.49.0-032-g20793a5f-sharefile-beta
// match 1 is commit number
// match 2 is branch name
var parseVersion = regexp.MustCompile(`^v(?:[0-9.]+)-(?:\d+)-g([0-9a-f]+)(?:-(.*))?-beta$`)
// FIXME take -issue or -pr parameter...
// NewReport initialises and returns a Report
func NewReport() *Report {
r := &Report{
StartTime: time.Now(),
Version: fs.Version,
GOOS: runtime.GOOS,
GOARCH: runtime.GOARCH,
GoVersion: runtime.Version(),
}
r.DateTime = r.StartTime.Format(timeFormat)
@@ -64,6 +85,16 @@ func NewReport() *Report {
// Online version
r.URL = *urlBase + r.DateTime + "/index.html"
// Get branch/commit out of version
parts := parseVersion.FindStringSubmatch(r.Version)
if len(parts) >= 3 {
r.Commit = parts[1]
r.Branch = parts[2]
}
if r.Branch == "" {
r.Branch = "master"
}
return r
}
@@ -116,6 +147,18 @@ func (r *Report) LogSummary() {
}
}
// LogJSON writes the summary to index.json in LogDir
func (r *Report) LogJSON() {
out, err := json.MarshalIndent(r, "", "\t")
if err != nil {
log.Fatalf("Failed to marshal data for index.json: %v", err)
}
err = ioutil.WriteFile(path.Join(r.LogDir, "index.json"), out, 0666)
if err != nil {
log.Fatalf("Failed to write index.json: %v", err)
}
}
// LogHTML writes the summary to index.html in LogDir
func (r *Report) LogHTML() {
r.IndexHTML = path.Join(r.LogDir, "index.html")
@@ -195,6 +238,9 @@ a:focus {
<table>
<tr><th>Version</th><td>{{ .Version }}</td></tr>
<tr><th>Test</th><td><a href="{{ .URL }}">{{ .DateTime}}</a></td></tr>
<tr><th>Branch</th><td><a href="https://github.com/rclone/rclone/tree/{{ .Branch }}">{{ .Branch }}</a></td></tr>
{{ if .Commit}}<tr><th>Commit</th><td><a href="https://github.com/rclone/rclone/commit/{{ .Commit }}">{{ .Commit }}</a></td></tr>{{ end }}
<tr><th>Go</th><td>{{ .GoVersion }} {{ .GOOS }}/{{ .GOARCH }}</td></tr>
<tr><th>Duration</th><td>{{ .Duration }}</td></tr>
{{ if .Previous}}<tr><th>Previous</th><td><a href="../{{ .Previous }}/index.html">{{ .Previous }}</a></td></tr>{{ end }}
<tr><th>Up</th><td><a href="../">Older Tests</a></td></tr>

View File

@@ -145,6 +145,7 @@ func main() {
// Log and exit
report.End()
report.LogSummary()
report.LogJSON()
report.LogHTML()
report.EmailHTML()
report.Upload()

2
go.mod
View File

@@ -67,3 +67,5 @@ require (
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 // indirect
gopkg.in/yaml.v2 v2.2.2
)
go 1.13

View File

@@ -46,7 +46,7 @@ func ReadBody(resp *http.Response) (result []byte, err error) {
}
// defaultErrorHandler doesn't attempt to parse the http body, just
// returns it in the error message
// returns it in the error message closing resp.Body
func defaultErrorHandler(resp *http.Response) (err error) {
body, err := ReadBody(resp)
if err != nil {
@@ -178,9 +178,11 @@ func ClientWithNoRedirects(c *http.Client) *http.Client {
// Call makes the call and returns the http.Response
//
// if err != nil then resp.Body will need to be closed unless
// if err == nil then resp.Body will need to be closed unless
// opt.NoResponse is set
//
// if err != nil then resp.Body will have been closed
//
// it will return resp if at all possible, even if err is set
func (api *Client) Call(opts *Opts) (resp *http.Response, err error) {
api.mu.RLock()

116
rclone.1 generated
View File

@@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 2.2.1
.\"
.TH "rclone" "1" "Aug 26, 2019" "User Manual" ""
.TH "rclone" "1" "Sep 08, 2019" "User Manual" ""
.hy
.SH Rclone \- rsync for cloud storage
.PP
@@ -264,6 +264,41 @@ See rclone config docs (https://rclone.org/docs/) for more details.
rclone\ config
\f[]
.fi
.SS Install with docker
.PP
The rclone maintains a docker image for
rclone (https://hub.docker.com/r/rclone/rclone).
These images are autobuilt by docker hub from the rclone source based on
a minimal Alpine linux image.
.PP
The \f[C]:latest\f[] tag will always point to the latest stable release.
You can use the \f[C]:beta\f[] tag to get the latest build from master.
You can also use version tags, eg \f[C]:1.49.1\f[], \f[C]:1.49\f[] or
\f[C]:1\f[].
.IP
.nf
\f[C]
$\ docker\ pull\ rclone/rclone:latest
latest:\ Pulling\ from\ rclone/rclone
Digest:\ sha256:0e0ced72671989bb837fea8e88578b3fc48371aa45d209663683e24cfdaa0e11
\&...
$\ docker\ run\ \-\-rm\ rclone/rclone:latest\ version
rclone\ v1.49.1
\-\ os/arch:\ linux/amd64
\-\ go\ version:\ go1.12.9
\f[]
.fi
.PP
You will probably want to mount rclone's config file directory or file
from the host, or configure rclone with environment variables.
.PP
Eg to share your local config with the container
.IP
.nf
\f[C]
docker\ run\ \ \-v\ ~/.config/rclone:/root/.config/rclone\ rclone/rclone:latest\ listremotes
\f[]
.fi
.SS Install from source
.PP
Make sure you have at least Go (https://golang.org/) 1.7 installed.
@@ -8181,6 +8216,8 @@ This takes the following parameters
.IP \[bu] 2
name \- name of remote
.IP \[bu] 2
parameters \- a map of { \[lq]key\[rq]: \[lq]value\[rq] } pairs
.IP \[bu] 2
type \- type of the new remote
.PP
See the config create
@@ -8190,7 +8227,9 @@ more information on the above.
Authentication is required for this call.
.SS config/delete: Delete a remote in the config file. {#config/delete}
.PP
Parameters: \- name \- name of remote to delete
Parameters:
.IP \[bu] 2
name \- name of remote to delete
.PP
See the config delete
command (https://rclone.org/commands/rclone_config_delete/) command for
@@ -8233,6 +8272,8 @@ Authentication is required for this call.
This takes the following parameters
.IP \[bu] 2
name \- name of remote
.IP \[bu] 2
parameters \- a map of { \[lq]key\[rq]: \[lq]value\[rq] } pairs
.PP
See the config password
command (https://rclone.org/commands/rclone_config_password/) command
@@ -8254,6 +8295,8 @@ Authentication is required for this call.
This takes the following parameters
.IP \[bu] 2
name \- name of remote
.IP \[bu] 2
parameters \- a map of { \[lq]key\[rq]: \[lq]value\[rq] } pairs
.PP
See the config update
command (https://rclone.org/commands/rclone_config_update/) command for
@@ -10462,7 +10505,7 @@ These flags are available for every command.
\ \ \ \ \ \ \-\-use\-json\-log\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Use\ json\ log\ format.
\ \ \ \ \ \ \-\-use\-mmap\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Use\ mmap\ allocator\ (see\ docs).
\ \ \ \ \ \ \-\-use\-server\-modtime\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Use\ server\ modified\ time\ instead\ of\ object\ metadata
\ \ \ \ \ \ \-\-user\-agent\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Set\ the\ user\-agent\ to\ a\ specified\ string.\ The\ default\ is\ rclone/\ version\ (default\ "rclone/v1.49.0")
\ \ \ \ \ \ \-\-user\-agent\ string\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Set\ the\ user\-agent\ to\ a\ specified\ string.\ The\ default\ is\ rclone/\ version\ (default\ "rclone/v1.49.2")
\ \ \-v,\ \-\-verbose\ count\ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ \ Print\ lots\ more\ stuff\ (repeat\ for\ more)
\f[]
.fi
@@ -23648,6 +23691,58 @@ Type: bool
.IP \[bu] 2
Default: false
.SH Changelog
.SS v1.49.2 \- 2019\-09\-08
.IP \[bu] 2
New Features
.RS 2
.IP \[bu] 2
build: Add Docker workflow support (Alfonso Montero)
.RE
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
accounting: Fix locking in Transfer to avoid deadlock with \[en]progress
(Nick Craig\-Wood)
.IP \[bu] 2
docs: Fix template argument for mktemp in install.sh (Cnly)
.IP \[bu] 2
operations: Fix \-u/\[en]update with google photos / files of unknown
size (Nick Craig\-Wood)
.IP \[bu] 2
rc: Fix docs for config/create /update /password (Nick Craig\-Wood)
.RE
.IP \[bu] 2
Google Cloud Storage
.RS 2
.IP \[bu] 2
Fix need for elevated permissions on SetModTime (Nick Craig\-Wood)
.RE
.SS v1.49.1 \- 2019\-08\-28
.PP
Point release to fix config bug and google photos backend.
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
config: Fix generated passwords being stored as empty password (Nick
Craig\-Wood)
.IP \[bu] 2
rcd: Added missing parameter for web\-gui info logs.
(Chaitanya)
.RE
.IP \[bu] 2
Googlephotos
.RS 2
.IP \[bu] 2
Fix crash on error response (Nick Craig\-Wood)
.RE
.IP \[bu] 2
Onedrive
.RS 2
.IP \[bu] 2
Fix crash on error response (Nick Craig\-Wood)
.RE
.SS v1.49.0 \- 2019\-08\-26
.IP \[bu] 2
New backends
@@ -23673,12 +23768,19 @@ Implement \f[C]\-\-compare\-dest\f[] & \f[C]\-\-copy\-dest\f[]
Implement \f[C]\-\-suffix\f[] without \f[C]\-\-backup\-dir\f[] for
backup to current dir (yparitcher)
.IP \[bu] 2
Add \f[C]\-\-use\-json\-log\f[] for JSON logging (justinalin)
.IP \[bu] 2
Add \f[C]config\ reconnect\f[], \f[C]config\ userinfo\f[] and
\f[C]config\ disconnect\f[] subcommands.
\f[C]config\ reconnect\f[] to re\-login (re\-run the oauth login) for
the backend.
(Nick Craig\-Wood)
.IP \[bu] 2
\f[C]config\ userinfo\f[] to discover which user you are logged in as.
(Nick Craig\-Wood)
.IP \[bu] 2
\f[C]config\ disconnect\f[] to disconnect you (log out) from the
backend.
(Nick Craig\-Wood)
.IP \[bu] 2
Add \f[C]\-\-use\-json\-log\f[] for JSON logging (justinalin)
.IP \[bu] 2
Add context propagation to rclone (Aleksandar Jankovic)
.IP \[bu] 2
Reworking internal statistics interfaces so they work with rc jobs