Fix frontmatter

Version v1.29
Make test more reliable
2026-02-08 12:39:57 +00:00 · 2016-04-18 18:55:07 +01:00 · 2016-04-18 18:30:29 +01:00 · 2016-04-18 17:48:52 +01:00 · 2016-04-12 21:41:24 +01:00 · 2016-04-12 21:33:55 +01:00
44 changed files with 1914 additions and 507 deletions
--- a/.travis.yml
+++ b/.travis.yml
@@ -7,14 +7,13 @@ os:
 # - osx

 go:
-  - 1.4.2
  - 1.5.3
-
-# - tip
+  - 1.6
+  - tip

 install:
 - go get -t ./...
- - '[[ `go version` =~ go1.[0-4][^0-9] ]] || go get -u github.com/kisielk/errcheck'
+ - go get -u github.com/kisielk/errcheck
 - go get -u golang.org/x/tools/cmd/goimports
 - go get -u github.com/golang/lint/golint

--- a/MANUAL.html
+++ b/MANUAL.html
@@ -12,7 +12,7 @@
 <div id="header">
 <h1 class="title">rclone(1) User Manual</h1>
 <h2 class="author">Nick Craig-Wood</h2>
-<h3 class="date">Mar 01, 2016</h3>
+<h3 class="date">Apr 18, 2016</h3>
 </div>
 <h1 id="rclone">Rclone</h1>
 <p><a href="http://rclone.org/"><img src="http://rclone.org/img/rclone-120x120.png" alt="Logo" /></a></p>
@@ -51,10 +51,10 @@
 <h2 id="install">Install</h2>
 <p>Rclone is a Go program and comes as a single binary file.</p>
 <p><a href="http://rclone.org/downloads/">Download</a> the relevant binary.</p>
-<p>Or alternatively if you have Go installed use</p>
+<p>Or alternatively if you have Go 1.5+ installed use</p>
 <pre><code>go get github.com/ncw/rclone</code></pre>
-<p>and this will build the binary in <code>$GOPATH/bin</code>. If you have built rclone before then you will want to update its dependencies first with this (remove <code>-f</code> if using go &lt; 1.4)</p>
-<pre><code>go get -u -v -f github.com/ncw/rclone/...</code></pre>
+<p>and this will build the binary in <code>$GOPATH/bin</code>. If you have built rclone before then you will want to update its dependencies first with this</p>
+<pre><code>go get -u -v github.com/ncw/rclone/...</code></pre>
 <p>See the <a href="http://rclone.org/docs/">Usage section</a> of the docs for how to use rclone, or run <code>rclone -h</code>.</p>
 <h2 id="linux-binary-downloaded-files-install-example">linux binary downloaded files install example</h2>
 <pre><code>unzip rclone-v1.17-linux-amd64.zip
@@ -148,36 +148,63 @@ rclone --dry-run --min-size 100M delete remote:path</code></pre>
 <p>That reads &quot;delete everything with a minimum size of 100 MB&quot;, hence delete all files bigger than 100MBytes.</p>
 <h3 id="rclone-check-sourcepath-destpath">rclone check source:path dest:path</h3>
 <p>Checks the files in the source and destination match. It compares sizes and MD5SUMs and prints a report of files which don't match. It doesn't alter the source or destination.</p>
+<p><code>--size-only</code> may be used to only compare the sizes, not the MD5SUMs.</p>
 <h3 id="rclone-dedupe-remotepath">rclone dedupe remote:path</h3>
-<p>Interactively find duplicate files and offer to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.</p>
+<p>By default <code>dedup</code> interactively finds duplicate files and offers to delete all but one or rename them to be different. Only useful with Google Drive which can have duplicate file names.</p>
+<p>The <code>dedupe</code> command will delete all but one of any identical (same md5sum) files it finds without confirmation. This means that for most duplicated files the <code>dedupe</code> command will not be interactive. You can use <code>--dry-run</code> to see what would happen without doing anything.</p>
+<p>Here is an example run.</p>
+<p>Before - with duplicates</p>
+<pre><code>$ rclone lsl drive:dupes
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+  6048320 2016-03-05 16:23:11.775000000 one.txt
+   564374 2016-03-05 16:23:06.731000000 one.txt
+  6048320 2016-03-05 16:18:26.092000000 one.txt
+  6048320 2016-03-05 16:22:46.185000000 two.txt
+  1744073 2016-03-05 16:22:38.104000000 two.txt
+   564374 2016-03-05 16:22:52.118000000 two.txt</code></pre>
+<p>Now the <code>dedupe</code> session</p>
 <pre><code>$ rclone dedupe drive:dupes
-2016/01/31 14:13:11 Google drive root &#39;dupes&#39;: Looking for duplicates
-two.txt: Found 3 duplicates
-  1:       564374 bytes, 2016-01-31 14:07:22.159000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
-  2:      1744073 bytes, 2016-01-31 14:07:12.490000000, md5sum 851957f7fb6f0bc4ce76be966d336802
-  3:      6048320 bytes, 2016-01-31 14:07:02.111000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+2016/03/05 16:24:37 Google drive root &#39;dupes&#39;: Looking for duplicates using interactive mode.
+one.txt: Found 4 duplicates - deleting identical copies
+one.txt: Deleting 2/3 identical duplicates (md5sum &quot;1eedaa9fe86fd4b8632e2ac549403b36&quot;)
+one.txt: 2 duplicates remain
+  1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+s) Skip and do nothing
+k) Keep just one (choose which in next step)
+r) Rename all to be different (by changing file.jpg to file-1.jpg)
+s/k/r&gt; k
+Enter the number of the file to keep&gt; 1
+one.txt: Deleted 1 extra copies
+two.txt: Found 3 duplicates - deleting identical copies
+two.txt: 3 duplicates remain
+  1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+  2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
 s) Skip and do nothing
 k) Keep just one (choose which in next step)
 r) Rename all to be different (by changing file.jpg to file-1.jpg)
 s/k/r&gt; r
 two-1.txt: renamed from: two.txt
 two-2.txt: renamed from: two.txt
-two-3.txt: renamed from: two.txt
-one.txt: Found 2 duplicates
-  1:         6579 bytes, 2016-01-31 14:05:01.235000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-  2:         6579 bytes, 2016-01-31 12:50:30.318000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-s) Skip and do nothing
-k) Keep just one (choose which in next step)
-r) Rename all to be different (by changing file.jpg to file-1.jpg)
-s/k/r&gt; k
-Enter the number of the file to keep&gt; 2
-one.txt: Deleted 1 extra copies</code></pre>
+two-3.txt: renamed from: two.txt</code></pre>
 <p>The result being</p>
 <pre><code>$ rclone lsl drive:dupes
-   564374 2016-01-31 14:07:22.159000000 two-1.txt
-  1744073 2016-01-31 14:07:12.490000000 two-2.txt
-  6048320 2016-01-31 14:07:02.111000000 two-3.txt
-     6579 2016-01-31 12:50:30.318000000 one.txt</code></pre>
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+   564374 2016-03-05 16:22:52.118000000 two-1.txt
+  6048320 2016-03-05 16:22:46.185000000 two-2.txt
+  1744073 2016-03-05 16:22:38.104000000 two-3.txt</code></pre>
+<p>Dedupe can be run non interactively using the <code>--dedupe-mode</code> flag.</p>
+<ul>
+<li><code>--dedupe-mode interactive</code> - interactive as above.</li>
+<li><code>--dedupe-mode skip</code> - removes identical files then skips anything left.</li>
+<li><code>--dedupe-mode first</code> - removes identical files then keeps the first one.</li>
+<li><code>--dedupe-mode newest</code> - removes identical files then keeps the newest one.</li>
+<li><code>--dedupe-mode oldest</code> - removes identical files then keeps the oldest one.</li>
+<li><code>--dedupe-mode rename</code> - removes identical files then renames the rest to be different.</li>
+</ul>
+<p>For example to rename all the identically named photos in your Google Photos directory, do</p>
+<pre><code>rclone dedupe --dedupe-mode rename &quot;drive:Google Photos&quot;</code></pre>
 <h3 id="rclone-config">rclone config</h3>
 <p>Enter an interactive configuration session.</p>
 <h3 id="rclone-help">rclone help</h3>
@@ -216,11 +243,16 @@ rclone sync /path/to/files remote:current-backup</code></pre>
 <h3 id="contimeouttime">--contimeout=TIME</h3>
 <p>Set the connection timeout. This should be in go time format which looks like <code>5s</code> for 5 seconds, <code>10m</code> for 10 minutes, or <code>3h30m</code>.</p>
 <p>The connection timeout is the amount of time rclone will wait for a connection to go through to a remote object storage system. It is <code>1m</code> by default.</p>
+<h3 id="dedupe-mode-mode">--dedupe-mode MODE</h3>
+<p>Mode to run dedupe command in. One of <code>interactive</code>, <code>skip</code>, <code>first</code>, <code>newest</code>, <code>oldest</code>, <code>rename</code>. The default is <code>interactive</code>. See the dedupe command for more information as to what these options mean.</p>
 <h3 id="n---dry-run">-n, --dry-run</h3>
 <p>Do a trial run with no permanent changes. Use this to see what rclone would do without actually doing it. Useful when setting up the <code>sync</code> command which deletes files in the destination.</p>
 <h3 id="ignore-existing">--ignore-existing</h3>
 <p>Using this option will make rclone unconditionally skip all files that exist on the destination, no matter the content of these files.</p>
 <p>While this isn't a generally recommended option, it can be useful in cases where your files change due to encryption. However, it cannot correct partial transfers in case a transfer was interrupted.</p>
+<h3 id="i---ignore-times">-I, --ignore-times</h3>
+<p>Using this option will cause rclone to unconditionally upload all files regardless of the state of files on the destination.</p>
+<p>Normally rclone would skip any files that have the same modification time and are the same size (or have the same checksum if using <code>--checksum</code>).</p>
 <h3 id="log-filefile">--log-file=FILE</h3>
 <p>Log all of rclone's output to FILE. This is not active by default. This can be useful for tracking down problems with syncs in combination with the <code>-v</code> flag.</p>
 <h3 id="low-level-retries-number">--low-level-retries NUMBER</h3>
@@ -406,6 +438,7 @@ y/e/d&gt;</code></pre>
 /file.jpg - matches &quot;file.jpg&quot; in the root directory of the remote
          - doesn&#39;t match &quot;afile.jpg&quot;
          - doesn&#39;t match &quot;directory/file.jpg&quot;</code></pre>
+<p><strong>Important</strong> Note that you must use <code>/</code> in patterns and not <code>\</code> even if running on Windows.</p>
 <p>A <code>*</code> matches anything but not a <code>/</code>.</p>
 <pre><code>*.jpg  - matches &quot;file.jpg&quot;
       - matches &quot;directory/file.jpg&quot;
@@ -626,7 +659,7 @@ file2.jpg</code></pre>
 <tr class="odd">
 <td align="left">Backblaze B2</td>
 <td align="center">SHA1</td>
-<td align="center">Partial</td>
+<td align="center">Yes</td>
 <td align="center">No</td>
 <td align="center">No</td>
 </tr>
@@ -652,7 +685,6 @@ file2.jpg</code></pre>
 <h3 id="modtime">ModTime</h3>
 <p>The cloud storage system supports setting modification times on objects. If it does then this enables a using the modification times as part of the sync. If not then only the size will be checked by default, though the MD5SUM can be checked with the <code>--checksum</code> flag.</p>
 <p>All cloud storage systems support some kind of date on the object and these will be set when transferring from the cloud storage system.</p>
-<p>Backblaze B2 preserves file modification times on files uploaded and downloaded, but doesn't use them to decide which objects to sync.</p>
 <h3 id="case-insensitive">Case Insensitive</h3>
 <p>If a cloud storage systems is case sensitive then it is possible to have two files which differ only in case, eg <code>file.txt</code> and <code>FILE.txt</code>. If a cloud storage system is case insensitive then that isn't possible.</p>
 <p>This can cause problems when syncing between a case insensitive system and a case sensitive system. The symptom of this is that no matter how many times you run the sync it never completes fully.</p>
@@ -665,7 +697,7 @@ file2.jpg</code></pre>
 <p>Most of the time this doesn't cause any problems as people tend to avoid files whose name differs only by case even on case sensitive systems.</p>
 <h3 id="duplicate-files">Duplicate files</h3>
 <p>If a cloud storage system allows duplicate files then it can have two objects with the same name.</p>
-<p>This confuses rclone greatly when syncing.</p>
+<p>This confuses rclone greatly when syncing - use the <code>rclone dedupe</code> command to rename or remove duplicates.</p>
 <h2 id="google-drive">Google Drive</h2>
 <p>Paths are specified as <code>drive:path</code></p>
 <p>Drive paths may be as deep as required, eg <code>drive:directory/subdirectory</code>.</p>
@@ -750,11 +782,13 @@ y/e/d&gt; y</code></pre>
 <h3 id="specific-options">Specific options</h3>
 <p>Here are the command line options specific to this cloud storage system.</p>
 <h4 id="drive-chunk-sizesize">--drive-chunk-size=SIZE</h4>
-<p>Upload chunk size. Must a power of 2 &gt;= 256k. Default value is 256kB.</p>
+<p>Upload chunk size. Must a power of 2 &gt;= 256k. Default value is 8 MB.</p>
+<p>Making this larger will improve performance, but note that each chunk is buffered in memory one per transfer.</p>
+<p>Reducing this will reduce memory usage but decrease performance.</p>
 <h4 id="drive-full-list">--drive-full-list</h4>
-<p>Use a full listing for directory list. More data but usually quicker. On by default, disable with <code>--full-drive-list=false</code>.</p>
+<p>No longer does anything - kept for backwards compatibility.</p>
 <h4 id="drive-upload-cutoffsize">--drive-upload-cutoff=SIZE</h4>
-<p>File size cutoff for switching to chunked upload. Default is 256kB.</p>
+<p>File size cutoff for switching to chunked upload. Default is 8 MB.</p>
 <h4 id="drive-use-trash">--drive-use-trash</h4>
 <p>Send files to the trash instead of deleting permanently. Defaults to off, namely deleting files permanently.</p>
 <h4 id="drive-auth-owner-only">--drive-auth-owner-only</h4>
@@ -1169,6 +1203,8 @@ y/e/d&gt; y</code></pre>
 <h3 id="modified-time-2">Modified time</h3>
 <p>The modified time is stored as metadata on the object as <code>X-Object-Meta-Mtime</code> as floating point since the epoch accurate to 1 ns.</p>
 <p>This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time for an object.</p>
+<h3 id="limitations-1">Limitations</h3>
+<p>The Swift API doesn't return a correct MD5SUM for segmented files (Dynamic or Static Large Objects) so rclone won't check or use the MD5SUM for these.</p>
 <h2 id="dropbox">Dropbox</h2>
 <p>Paths are specified as <code>remote:path</code></p>
 <p>Dropbox paths may be as deep as required, eg <code>remote:directory/subdirectory</code>.</p>
@@ -1244,9 +1280,10 @@ y/e/d&gt; y</code></pre>
 <p>Here are the command line options specific to this cloud storage system.</p>
 <h4 id="dropbox-chunk-sizesize">--dropbox-chunk-size=SIZE</h4>
 <p>Upload chunk size. Max 150M. The default is 128MB. Note that this isn't buffered into memory.</p>
-<h3 id="limitations-1">Limitations</h3>
+<h3 id="limitations-2">Limitations</h3>
 <p>Note that Dropbox is case insensitive so you can't have a file called &quot;Hello.doc&quot; and one called &quot;hello.doc&quot;.</p>
 <p>There are some file names such as <code>thumbs.db</code> which Dropbox can't store. There is a full list of them in the <a href="https://www.dropbox.com/en/help/145">&quot;Ignored Files&quot; section of this document</a>. Rclone will issue an error message <code>File name disallowed - not uploading</code> if it attempt to upload one of those file names, but the sync won't fail.</p>
+<p>If you have more than 10,000 files in a directory then <code>rclone purge dropbox:dir</code> will return the error <code>Failed to purge: There are too many files involved in this operation</code>. As a work-around do an <code>rclone delete dropbix:dir</code> followed by an <code>rclone rmdir dropbox:dir</code>.</p>
 <h2 id="google-cloud-storage">Google Cloud Storage</h2>
 <p>Paths are specified as <code>remote:bucket</code> (or <code>remote:</code> for the <code>lsd</code> command.) You may put subdirectories in too, eg <code>remote:bucket/path/to/dir</code>.</p>
 <p>The initial setup for google cloud storage involves getting a token from Google Cloud Storage which you need to do in your browser. <code>rclone config</code> walks you through it.</p>
@@ -1430,7 +1467,7 @@ y/e/d&gt; y</code></pre>
 <h4 id="acd-templink-thresholdsize">--acd-templink-threshold=SIZE</h4>
 <p>Files this size or more will be downloaded via their <code>tempLink</code>. This is to work around a problem with Amazon Cloud Drive which blocks downloads of files bigger than about 10GB. The default for this is 9GB which shouldn't need to be changed.</p>
 <p>To download files above this threshold, rclone requests a <code>tempLink</code> which downloads the file through a temporary URL directly from the underlying S3 storage.</p>
-<h3 id="limitations-2">Limitations</h3>
+<h3 id="limitations-3">Limitations</h3>
 <p>Note that Amazon cloud drive is case insensitive so you can't have a file called &quot;Hello.doc&quot; and one called &quot;hello.doc&quot;.</p>
 <p>Amazon cloud drive has rate limiting so you may notice errors in the sync (429 errors). rclone will automatically retry the sync up to 3 times by default (see <code>--retries</code> flag) which should hopefully work around this problem.</p>
 <p>Amazon cloud drive has an internal limit of file sizes that can be uploaded to the service. This limit is not officially published, but all files larger than this will fail.</p>
@@ -1518,7 +1555,7 @@ y/e/d&gt; y</code></pre>
 <p>Above this size files will be chunked - must be multiple of 320k. The default is 10MB. Note that the chunks will be buffered into memory.</p>
 <h4 id="onedrive-upload-cutoffsize">--onedrive-upload-cutoff=SIZE</h4>
 <p>Cutoff for switching to chunked upload - must be &lt;= 100MB. The default is 10MB.</p>
-<h3 id="limitations-3">Limitations</h3>
+<h3 id="limitations-4">Limitations</h3>
 <p>Note that One Drive is case insensitive so you can't have a file called &quot;Hello.doc&quot; and one called &quot;hello.doc&quot;.</p>
 <p>Rclone only supports your default One Drive, and doesn't work with One Drive for business. Both these issues may be fixed at some point depending on user demand!</p>
 <p>There are quite a few characters that can't be in One Drive file names. These can't occur on Windows platforms, but on non-Windows platforms they are common. Rclone will map these names to and from an identical looking unicode equivalent. For example if a file has a <code>?</code> in it will be mapped to <code>？</code> instead.</p>
@@ -1596,8 +1633,9 @@ y/e/d&gt; y</code></pre>
 <p>The modified time is stored as metadata on the object as <code>X-Object-Meta-Mtime</code> as floating point since the epoch accurate to 1 ns.</p>
 <p>This is a defacto standard (used in the official python-swiftclient amongst others) for storing the modification time for an object.</p>
 <p>Note that Hubic wraps the Swift backend, so most of the properties of are the same.</p>
-<h3 id="limitations-4">Limitations</h3>
-<p>Code to refresh the OpenStack token isn't done yet which may cause problems with very long transfers.</p>
+<h3 id="limitations-5">Limitations</h3>
+<p>This uses the normal OpenStack Swift mechanism to refresh the Swift API credentials and ignores the expires field returned by the Hubic API.</p>
+<p>The Swift API doesn't return a correct MD5SUM for segmented files (Dynamic or Static Large Objects) so rclone won't check or use the MD5SUM for these.</p>
 <h2 id="backblaze-b2">Backblaze B2</h2>
 <p>B2 is <a href="https://www.backblaze.com/b2/">Backblaze's cloud storage system</a>.</p>
 <p>Paths are specified as <code>remote:bucket</code> (or <code>remote:</code> for the <code>lsd</code> command.) You may put subdirectories in too, eg <code>remote:bucket/path/to/dir</code>.</p>
@@ -1662,15 +1700,17 @@ y/e/d&gt; y</code></pre>
 <pre><code>rclone sync /home/local/directory remote:bucket</code></pre>
 <h3 id="modified-time-5">Modified time</h3>
 <p>The modified time is stored as metadata on the object as <code>X-Bz-Info-src_last_modified_millis</code> as milliseconds since 1970-01-01 in the Backblaze standard. Other tools should be able to use this as a modified time.</p>
-<p>Modified times are set on upload, read on download and shown in listings. They are not used in syncing as unfortunately B2 doesn't have an API method to set them independently of doing an upload.</p>
+<p>Modified times are used in syncing and are fully supported except in the case of updating a modification time on an existing object. In this case the object will be uploaded again as B2 doesn't have an API method to set the modification time independent of doing an upload.</p>
 <h3 id="sha1-checksums">SHA1 checksums</h3>
 <p>The SHA1 checksums of the files are checked on upload and download and will be used in the syncing process. You can use the <code>--checksum</code> flag.</p>
 <h3 id="versions">Versions</h3>
 <p>When rclone uploads a new version of a file it creates a <a href="https://www.backblaze.com/b2/docs/file_versions.html">new version of it</a>. Likewise when you delete a file, the old version will still be available.</p>
 <p>The old versions of files are visible in the B2 web interface, but not via rclone yet.</p>
 <p>Rclone doesn't provide any way of managing old versions (downloading them or deleting them) at the moment. When you <code>purge</code> a bucket, all the old versions will be deleted.</p>
+<h3 id="transfers">Transfers</h3>
+<p>Backblaze recommends that you do lots of transfers simultaneously for maximum speed. In tests from my SSD equiped laptop the optimum setting is about <code>--transfers 32</code> though higher numbers may be used for a slight speed improvement. The optimum number for you may vary depending on your hardware, how big the files are, how much you want to load your computer, etc. The default of <code>--transfers 4</code> is definitely too low for Backblaze B2 though.</p>
 <h3 id="api">API</h3>
-<p>Here are <a href="https://gist.github.com/ncw/166dabf352b399f1cc1c">some notes I made on the backblaze API</a> while integrating it with rclone which detail the changes I'd like to see. With a couple of small tweaks Backblaze could enable rclone to not make a temporary copy of files when doing cloud to cloud copies and fully support modification times.</p>
+<p>Here are <a href="https://gist.github.com/ncw/166dabf352b399f1cc1c">some notes I made on the backblaze API</a> while integrating it with rclone which detail the changes I'd like to see.</p>
 <h2 id="yandex-disk">Yandex Disk</h2>
 <p><a href="https://disk.yandex.com">Yandex Disk</a> is a cloud storage solution created by <a href="http://yandex.com">Yandex</a>.</p>
 <p>Yandex paths may be as deep as required, eg <code>remote:directory/subdirectory</code>.</p>
@@ -1774,6 +1814,76 @@ nounc = true</code></pre>
 <p>This will use UNC paths on <code>c:\src</code> but not on <code>z:\dst</code>. Of course this will cause problems if the absolute path length of a file exceeds 258 characters on z, so only use this option if you have to.</p>
 <h2 id="changelog">Changelog</h2>
 <ul>
+<li>v1.29 - 2016-04-18
+<ul>
+<li>New Features</li>
+<li>Implement <code>-I, --ignore-times</code> for unconditional upload</li>
+<li>Improve <code>dedupe</code>command
+<ul>
+<li>Now removes identical copies without asking</li>
+<li>Now obeys <code>--dry-run</code></li>
+<li>Implement <code>--dedupe-mode</code> for non interactive running</li>
+<li><code>--dedupe-mode interactive</code> - interactive the default.</li>
+<li><code>--dedupe-mode skip</code> - removes identical files then skips anything left.</li>
+<li><code>--dedupe-mode first</code> - removes identical files then keeps the first one.</li>
+<li><code>--dedupe-mode newest</code> - removes identical files then keeps the newest one.</li>
+<li><code>--dedupe-mode oldest</code> - removes identical files then keeps the oldest one.</li>
+<li><code>--dedupe-mode rename</code> - removes identical files then renames the rest to be different.</li>
+</ul></li>
+<li>Bug fixes</li>
+<li>Make rclone check obey the <code>--size-only</code> flag.</li>
+<li>Use &quot;application/octet-stream&quot; if discovered mime type is invalid.</li>
+<li>Fix missing &quot;quit&quot; option when there are no remotes.</li>
+<li>Google Drive</li>
+<li>Increase default chunk size to 8 MB - increases upload speed of big files</li>
+<li>Speed up directory listings and make more reliable</li>
+<li>Add missing retries for Move and DirMove - increases reliability</li>
+<li>Preserve mime type on file update</li>
+<li>Backblaze B2</li>
+<li>Enable mod time syncing
+<ul>
+<li>This means that B2 will now check modification times</li>
+<li>It will upload new files to update the modification times</li>
+<li>(there isn't an API to just set the mod time.)</li>
+<li>If you want the old behaviour use <code>--size-only</code>.</li>
+</ul></li>
+<li>Update API to new version</li>
+<li>Fix parsing of mod time when not in metadata</li>
+<li>Swift/Hubic</li>
+<li>Don't return an MD5SUM for static large objects</li>
+<li>S3</li>
+<li>Fix uploading files bigger than 50GB</li>
+</ul></li>
+<li>v1.28 - 2016-03-01
+<ul>
+<li>New Features</li>
+<li>Configuration file encryption - thanks Klaus Post</li>
+<li>Improve <code>rclone config</code> adding more help and making it easier to understand</li>
+<li>Implement <code>-u</code>/<code>--update</code> so creation times can be used on all remotes</li>
+<li>Implement <code>--low-level-retries</code> flag</li>
+<li>Optionally disable gzip compression on downloads with <code>--no-gzip-encoding</code></li>
+<li>Bug fixes</li>
+<li>Don't make directories if <code>--dry-run</code> set</li>
+<li>Fix and document the <code>move</code> command</li>
+<li>Fix redirecting stderr on unix-like OSes when using <code>--log-file</code></li>
+<li>Fix <code>delete</code> command to wait until all finished - fixes missing deletes.</li>
+<li>Backblaze B2</li>
+<li>Use one upload URL per go routine fixes <code>more than one upload using auth token</code></li>
+<li>Add pacing, retries and reauthentication - fixes token expiry problems</li>
+<li>Upload without using a temporary file from local (and remotes which support SHA1)</li>
+<li>Fix reading metadata for all files when it shouldn't have been</li>
+<li>Drive</li>
+<li>Fix listing drive documents at root</li>
+<li>Disable copy and move for Google docs</li>
+<li>Swift</li>
+<li>Fix uploading of chunked files with non ASCII characters</li>
+<li>Allow setting of <code>storage_url</code> in the config - thanks Xavier Lucas</li>
+<li>S3</li>
+<li>Allow IAM role and credentials from environment variables - thanks Brian Stengaard</li>
+<li>Allow low privilege users to use S3 (check if directory exists during Mkdir) - thanks Jakub Gedeon</li>
+<li>Amazon Cloud Drive</li>
+<li>Retry on more things to make directory listings more reliable</li>
+</ul></li>
 <li>v1.27 - 2016-01-31
 <ul>
 <li>New Features</li>
@@ -2174,6 +2284,8 @@ ntpclient -s -h pool.ntp.org</code></pre>
 <h3 id="rclone-gives-failed-to-load-config-file-function-not-implemented-error">Rclone gives Failed to load config file: function not implemented error</h3>
 <p>Likely this means that you are running rclone on Linux version not supported by the go runtime, ie earlier than version 2.6.23.</p>
 <p>See the <a href="https://golang.org/doc/install">system requirements section in the go install docs</a> for full details.</p>
+<h3 id="all-my-uploaded-docxxlsxpptx-files-appear-as-archivezip">All my uploaded docx/xlsx/pptx files appear as archive/zip</h3>
+<p>This is caused by uploading these files from a Windows computer which hasn't got the Microsoft Office suite installed. The easiest way to fix is to install the Word viewer and the Microsoft Office Compatibility Pack for Word, Excel, and PowerPoint 2007 and later versions' file formats</p>
 <h2 id="license">License</h2>
 <p>This is free software under the terms of MIT the license (check the COPYING file included with the source code).</p>
 <pre><code>Copyright (C) 2012 by Nick Craig-Wood http://www.craig-wood.com/nick/
--- a/MANUAL.md
+++ b/MANUAL.md
@@ -1,6 +1,6 @@
 % rclone(1) User Manual
 % Nick Craig-Wood
-% Mar 01, 2016
+% Apr 18, 2016

 Rclone
 ======
@@ -45,15 +45,15 @@ Rclone is a Go program and comes as a single binary file.

 [Download](http://rclone.org/downloads/) the relevant binary.

-Or alternatively if you have Go installed use
+Or alternatively if you have Go 1.5+ installed use

    go get github.com/ncw/rclone

 and this will build the binary in `$GOPATH/bin`.  If you have built
 rclone before then you will want to update its dependencies first with
-this (remove `-f` if using go < 1.4)
+this

-    go get -u -v -f github.com/ncw/rclone/...
+    go get -u -v github.com/ncw/rclone/...

 See the [Usage section](http://rclone.org/docs/) of the docs for how to use rclone, or
 run `rclone -h`.
@@ -260,19 +260,55 @@ Checks the files in the source and destination match.  It
 compares sizes and MD5SUMs and prints a report of files which
 don't match.  It doesn't alter the source or destination.

+`--size-only` may be used to only compare the sizes, not the MD5SUMs.
+
 ### rclone dedupe remote:path ###

-Interactively find duplicate files and offer to delete all but one or
-rename them to be different. Only useful with Google Drive which can
-have duplicate file names.
+By default `dedup` interactively finds duplicate files and offers to
+delete all but one or rename them to be different. Only useful with
+Google Drive which can have duplicate file names.
+
+The `dedupe` command will delete all but one of any identical (same
+md5sum) files it finds without confirmation.  This means that for most
+duplicated files the `dedupe` command will not be interactive.  You
+can use `--dry-run` to see what would happen without doing anything.
+
+Here is an example run.
+
+Before - with duplicates
+
+```
+$ rclone lsl drive:dupes
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+  6048320 2016-03-05 16:23:11.775000000 one.txt
+   564374 2016-03-05 16:23:06.731000000 one.txt
+  6048320 2016-03-05 16:18:26.092000000 one.txt
+  6048320 2016-03-05 16:22:46.185000000 two.txt
+  1744073 2016-03-05 16:22:38.104000000 two.txt
+   564374 2016-03-05 16:22:52.118000000 two.txt
+```
+
+Now the `dedupe` session

 ```
 $ rclone dedupe drive:dupes
-2016/01/31 14:13:11 Google drive root 'dupes': Looking for duplicates
-two.txt: Found 3 duplicates
-  1:       564374 bytes, 2016-01-31 14:07:22.159000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
-  2:      1744073 bytes, 2016-01-31 14:07:12.490000000, md5sum 851957f7fb6f0bc4ce76be966d336802
-  3:      6048320 bytes, 2016-01-31 14:07:02.111000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
+one.txt: Found 4 duplicates - deleting identical copies
+one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
+one.txt: 2 duplicates remain
+  1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+s) Skip and do nothing
+k) Keep just one (choose which in next step)
+r) Rename all to be different (by changing file.jpg to file-1.jpg)
+s/k/r> k
+Enter the number of the file to keep> 1
+one.txt: Deleted 1 extra copies
+two.txt: Found 3 duplicates - deleting identical copies
+two.txt: 3 duplicates remain
+  1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+  2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
 s) Skip and do nothing
 k) Keep just one (choose which in next step)
 r) Rename all to be different (by changing file.jpg to file-1.jpg)
@@ -280,27 +316,31 @@ s/k/r> r
 two-1.txt: renamed from: two.txt
 two-2.txt: renamed from: two.txt
 two-3.txt: renamed from: two.txt
-one.txt: Found 2 duplicates
-  1:         6579 bytes, 2016-01-31 14:05:01.235000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-  2:         6579 bytes, 2016-01-31 12:50:30.318000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-s) Skip and do nothing
-k) Keep just one (choose which in next step)
-r) Rename all to be different (by changing file.jpg to file-1.jpg)
-s/k/r> k
-Enter the number of the file to keep> 2
-one.txt: Deleted 1 extra copies
 ```

 The result being

 ```
 $ rclone lsl drive:dupes
-   564374 2016-01-31 14:07:22.159000000 two-1.txt
-  1744073 2016-01-31 14:07:12.490000000 two-2.txt
-  6048320 2016-01-31 14:07:02.111000000 two-3.txt
-     6579 2016-01-31 12:50:30.318000000 one.txt
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+   564374 2016-03-05 16:22:52.118000000 two-1.txt
+  6048320 2016-03-05 16:22:46.185000000 two-2.txt
+  1744073 2016-03-05 16:22:38.104000000 two-3.txt
 ```

+Dedupe can be run non interactively using the `--dedupe-mode` flag.
+
+  * `--dedupe-mode interactive` - interactive as above.
+  * `--dedupe-mode skip` - removes identical files then skips anything left.
+  * `--dedupe-mode first` - removes identical files then keeps the first one.
+  * `--dedupe-mode newest` - removes identical files then keeps the newest one.
+  * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
+  * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
+
+For example to rename all the identically named photos in your Google Photos directory, do
+
+    rclone dedupe --dedupe-mode rename "drive:Google Photos"
+
 ### rclone config ###

 Enter an interactive configuration session.
@@ -410,6 +450,10 @@ The connection timeout is the amount of time rclone will wait for a
 connection to go through to a remote object storage system.  It is
 `1m` by default.

+### --dedupe-mode MODE ###
+
+Mode to run dedupe command in.  One of `interactive`, `skip`, `first`, `newest`, `oldest`, `rename`.  The default is `interactive`.  See the dedupe command for more information as to what these options mean.
+
 ### -n, --dry-run ###

 Do a trial run with no permanent changes.  Use this to see what rclone
@@ -425,6 +469,15 @@ While this isn't a generally recommended option, it can be useful
 in cases where your files change due to encryption. However, it cannot
 correct partial transfers in case a transfer was interrupted.

+### -I, --ignore-times ###
+
+Using this option will cause rclone to unconditionally upload all
+files regardless of the state of files on the destination.
+
+Normally rclone would skip any files that have the same
+modification time and are the same size (or have the same checksum if
+using `--checksum`).
+
 ### --log-file=FILE ###

 Log all of rclone's output to FILE.  This is not active by default.
@@ -824,6 +877,9 @@ If it doesn't start with `/` then it is matched starting at the
              - doesn't match "afile.jpg"
              - doesn't match "directory/file.jpg"

+**Important** Note that you must use `/` in patterns and not `\` even
+if running on Windows.
+
 A `*` matches anything but not a `/`.

    *.jpg  - matches "file.jpg"
@@ -1113,7 +1169,7 @@ Here is an overview of the major features of each cloud storage system.
 | Amazon Cloud Drive     | MD5     | No      | Yes              | No              |
 | Microsoft One Drive    | SHA1    | Yes     | Yes              | No              |
 | Hubic                  | MD5     | Yes     | No               | No              |
-| Backblaze B2           | SHA1    | Partial | No               | No              |
+| Backblaze B2           | SHA1    | Yes     | No               | No              |
 | Yandex Disk            | MD5     | Yes     | No               | No              |
 | The local filesystem   | All     | Yes     | Depends          | No              |

@@ -1137,9 +1193,6 @@ default, though the MD5SUM can be checked with the `--checksum` flag.
 All cloud storage systems support some kind of date on the object and
 these will be set when transferring from the cloud storage system.

-Backblaze B2 preserves file modification times on files uploaded and
-downloaded, but doesn't use them to decide which objects to sync.
-
 ### Case Insensitive ###

 If a cloud storage systems is case sensitive then it is possible to
@@ -1166,7 +1219,8 @@ systems.
 If a cloud storage system allows duplicate files then it can have two
 objects with the same name.

-This confuses rclone greatly when syncing.
+This confuses rclone greatly when syncing - use the `rclone dedupe`
+command to rename or remove duplicates.

 Google Drive
 -----------------------------------------
@@ -1293,16 +1347,20 @@ system.

 #### --drive-chunk-size=SIZE ####

-Upload chunk size. Must a power of 2 >= 256k. Default value is 256kB.
+Upload chunk size. Must a power of 2 >= 256k. Default value is 8 MB.
+
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.

 #### --drive-full-list ####

-Use a full listing for directory list. More data but usually
-quicker. On by default, disable with `--full-drive-list=false`.
+No longer does anything - kept for backwards compatibility.

 #### --drive-upload-cutoff=SIZE ####

-File size cutoff for switching to chunked upload.  Default is 256kB.
+File size cutoff for switching to chunked upload.  Default is 8 MB.

 #### --drive-use-trash ####

@@ -1764,6 +1822,12 @@ ns.
 This is a defacto standard (used in the official python-swiftclient
 amongst others) for storing the modification time for an object.

+### Limitations ###
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.
+
 Dropbox
 ---------------------------------

@@ -1883,6 +1947,11 @@ of this document](https://www.dropbox.com/en/help/145).  Rclone will
 issue an error message `File name disallowed - not uploading` if it
 attempt to upload one of those file names, but the sync won't fail.

+If you have more than 10,000 files in a directory then `rclone purge
+dropbox:dir` will return the error `Failed to purge: There are too
+many files involved in this operation`.  As a work-around do an
+`rclone delete dropbix:dir` followed by an `rclone rmdir dropbox:dir`.
+
 Google Cloud Storage
 -------------------------------------------------

@@ -2424,8 +2493,13 @@ are the same.

 ### Limitations ###

-Code to refresh the OpenStack token isn't done yet which may cause
-problems with very long transfers.
+This uses the normal OpenStack Swift mechanism to refresh the Swift
+API credentials and ignores the expires field returned by the Hubic
+API.
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.

 Backblaze B2
 ----------------------------------------
@@ -2519,9 +2593,10 @@ The modified time is stored as metadata on the object as
 in the Backblaze standard.  Other tools should be able to use this as
 a modified time.

-Modified times are set on upload, read on download and shown in
-listings.  They are not used in syncing as unfortunately B2 doesn't
-have an API method to set them independently of doing an upload.
+Modified times are used in syncing and are fully supported except in
+the case of updating a modification time on an existing object.  In
+this case the object will be uploaded again as B2 doesn't have an API
+method to set the modification time independent of doing an upload.

 ### SHA1 checksums ###

@@ -2542,14 +2617,21 @@ Rclone doesn't provide any way of managing old versions (downloading
 them or deleting them) at the moment.  When you `purge` a bucket, all
 the old versions will be deleted.

+### Transfers ###
+
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed.  In tests from my SSD equiped laptop the optimum
+setting is about `--transfers 32` though higher numbers may be used
+for a slight speed improvement. The optimum number for you may vary
+depending on your hardware, how big the files are, how much you want
+to load your computer, etc.  The default of `--transfers 4` is
+definitely too low for Backblaze B2 though.
+
 ### API ###

 Here are [some notes I made on the backblaze
 API](https://gist.github.com/ncw/166dabf352b399f1cc1c) while
 integrating it with rclone which detail the changes I'd like to see.
-With a couple of small tweaks Backblaze could enable rclone to not
-make a temporary copy of files when doing cloud to cloud copies and
-fully support modification times.

 Yandex Disk
 ----------------------------------------
@@ -2733,6 +2815,68 @@ file exceeds 258 characters on z, so only use this option if you have to.
 Changelog
 ---------

+  * v1.29 - 2016-04-18
+    * New Features
+      * Implement `-I, --ignore-times` for unconditional upload
+      * Improve `dedupe`command
+        * Now removes identical copies without asking
+        * Now obeys `--dry-run`
+        * Implement `--dedupe-mode` for non interactive running
+          * `--dedupe-mode interactive` - interactive the default.
+          * `--dedupe-mode skip` - removes identical files then skips anything left.
+          * `--dedupe-mode first` - removes identical files then keeps the first one.
+          * `--dedupe-mode newest` - removes identical files then keeps the newest one.
+          * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
+          * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
+    * Bug fixes
+      * Make rclone check obey the `--size-only` flag.
+      * Use "application/octet-stream" if discovered mime type is invalid.
+      * Fix missing "quit" option when there are no remotes.
+    * Google Drive
+      * Increase default chunk size to 8 MB - increases upload speed of big files
+      * Speed up directory listings and make more reliable
+      * Add missing retries for Move and DirMove - increases reliability
+      * Preserve mime type on file update
+    * Backblaze B2
+      * Enable mod time syncing
+        * This means that B2 will now check modification times
+        * It will upload new files to update the modification times
+        * (there isn't an API to just set the mod time.)
+        * If you want the old behaviour use `--size-only`.
+      * Update API to new version
+      * Fix parsing of mod time when not in metadata
+    * Swift/Hubic
+      * Don't return an MD5SUM for static large objects
+    * S3
+      * Fix uploading files bigger than 50GB
+  * v1.28 - 2016-03-01
+    * New Features
+      * Configuration file encryption - thanks Klaus Post
+      * Improve `rclone config` adding more help and making it easier to understand
+      * Implement `-u`/`--update` so creation times can be used on all remotes
+      * Implement `--low-level-retries` flag
+      * Optionally disable gzip compression on downloads with `--no-gzip-encoding`
+    * Bug fixes
+      * Don't make directories if `--dry-run` set
+      * Fix and document the `move` command
+      * Fix redirecting stderr on unix-like OSes when using `--log-file`
+      * Fix `delete` command to wait until all finished - fixes missing deletes.
+    * Backblaze B2
+      * Use one upload URL per go routine fixes `more than one upload using auth token`
+      * Add pacing, retries and reauthentication - fixes token expiry problems
+      * Upload without using a temporary file from local (and remotes which support SHA1)
+      * Fix reading metadata for all files when it shouldn't have been
+    * Drive
+      * Fix listing drive documents at root
+      * Disable copy and move for Google docs
+    * Swift
+      * Fix uploading of chunked files with non ASCII characters
+      * Allow setting of `storage_url` in the config - thanks Xavier Lucas
+    * S3
+      * Allow IAM role and credentials from environment variables - thanks Brian Stengaard
+      * Allow low privilege users to use S3 (check if directory exists during Mkdir) - thanks Jakub Gedeon
+    * Amazon Cloud Drive
+      * Retry on more things to make directory listings more reliable
  * v1.27 - 2016-01-31
    * New Features
      * Easier headless configuration with `rclone authorize`
@@ -3150,6 +3294,14 @@ supported by the go runtime, ie earlier than version 2.6.23.
 See the [system requirements section in the go install
 docs](https://golang.org/doc/install) for full details.

+### All my uploaded docx/xlsx/pptx files appear as archive/zip ###
+
+This is caused by uploading these files from a Windows computer which
+hasn't got the Microsoft Office suite installed.  The easiest way to
+fix is to install the Word viewer and the Microsoft Office
+Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
+versions' file formats
+
 License
 -------

--- a/MANUAL.txt
+++ b/MANUAL.txt
@@ -1,6 +1,6 @@
 rclone(1) User Manual
 Nick Craig-Wood
-Mar 01, 2016
+Apr 18, 2016



@@ -48,15 +48,14 @@ Rclone is a Go program and comes as a single binary file.

 Download the relevant binary.

-Or alternatively if you have Go installed use
+Or alternatively if you have Go 1.5+ installed use

    go get github.com/ncw/rclone

 and this will build the binary in $GOPATH/bin. If you have built rclone
 before then you will want to update its dependencies first with this
-(remove -f if using go < 1.4)

-    go get -u -v -f github.com/ncw/rclone/...
+    go get -u -v github.com/ncw/rclone/...

 See the Usage section of the docs for how to use rclone, or run
 rclone -h.
@@ -262,18 +261,52 @@ Checks the files in the source and destination match. It compares sizes
 and MD5SUMs and prints a report of files which don't match. It doesn't
 alter the source or destination.

+--size-only may be used to only compare the sizes, not the MD5SUMs.
+
 rclone dedupe remote:path

-Interactively find duplicate files and offer to delete all but one or
-rename them to be different. Only useful with Google Drive which can
-have duplicate file names.
+By default dedup interactively finds duplicate files and offers to
+delete all but one or rename them to be different. Only useful with
+Google Drive which can have duplicate file names.
+
+The dedupe command will delete all but one of any identical (same
+md5sum) files it finds without confirmation. This means that for most
+duplicated files the dedupe command will not be interactive. You can use
+--dry-run to see what would happen without doing anything.
+
+Here is an example run.
+
+Before - with duplicates
+
+    $ rclone lsl drive:dupes
+      6048320 2016-03-05 16:23:16.798000000 one.txt
+      6048320 2016-03-05 16:23:11.775000000 one.txt
+       564374 2016-03-05 16:23:06.731000000 one.txt
+      6048320 2016-03-05 16:18:26.092000000 one.txt
+      6048320 2016-03-05 16:22:46.185000000 two.txt
+      1744073 2016-03-05 16:22:38.104000000 two.txt
+       564374 2016-03-05 16:22:52.118000000 two.txt
+
+Now the dedupe session

    $ rclone dedupe drive:dupes
-    2016/01/31 14:13:11 Google drive root 'dupes': Looking for duplicates
-    two.txt: Found 3 duplicates
-      1:       564374 bytes, 2016-01-31 14:07:22.159000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
-      2:      1744073 bytes, 2016-01-31 14:07:12.490000000, md5sum 851957f7fb6f0bc4ce76be966d336802
-      3:      6048320 bytes, 2016-01-31 14:07:02.111000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+    2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
+    one.txt: Found 4 duplicates - deleting identical copies
+    one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
+    one.txt: 2 duplicates remain
+      1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+      2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+    s) Skip and do nothing
+    k) Keep just one (choose which in next step)
+    r) Rename all to be different (by changing file.jpg to file-1.jpg)
+    s/k/r> k
+    Enter the number of the file to keep> 1
+    one.txt: Deleted 1 extra copies
+    two.txt: Found 3 duplicates - deleting identical copies
+    two.txt: 3 duplicates remain
+      1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+      2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+      3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
    s) Skip and do nothing
    k) Keep just one (choose which in next step)
    r) Rename all to be different (by changing file.jpg to file-1.jpg)
@@ -281,23 +314,33 @@ have duplicate file names.
    two-1.txt: renamed from: two.txt
    two-2.txt: renamed from: two.txt
    two-3.txt: renamed from: two.txt
-    one.txt: Found 2 duplicates
-      1:         6579 bytes, 2016-01-31 14:05:01.235000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-      2:         6579 bytes, 2016-01-31 12:50:30.318000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-    s) Skip and do nothing
-    k) Keep just one (choose which in next step)
-    r) Rename all to be different (by changing file.jpg to file-1.jpg)
-    s/k/r> k
-    Enter the number of the file to keep> 2
-    one.txt: Deleted 1 extra copies

 The result being

    $ rclone lsl drive:dupes
-       564374 2016-01-31 14:07:22.159000000 two-1.txt
-      1744073 2016-01-31 14:07:12.490000000 two-2.txt
-      6048320 2016-01-31 14:07:02.111000000 two-3.txt
-         6579 2016-01-31 12:50:30.318000000 one.txt
+      6048320 2016-03-05 16:23:16.798000000 one.txt
+       564374 2016-03-05 16:22:52.118000000 two-1.txt
+      6048320 2016-03-05 16:22:46.185000000 two-2.txt
+      1744073 2016-03-05 16:22:38.104000000 two-3.txt
+
+Dedupe can be run non interactively using the --dedupe-mode flag.
+
+-   --dedupe-mode interactive - interactive as above.
+-   --dedupe-mode skip - removes identical files then skips anything
+    left.
+-   --dedupe-mode first - removes identical files then keeps the first
+    one.
+-   --dedupe-mode newest - removes identical files then keeps the newest
+    one.
+-   --dedupe-mode oldest - removes identical files then keeps the oldest
+    one.
+-   --dedupe-mode rename - removes identical files then renames the rest
+    to be different.
+
+For example to rename all the identically named photos in your Google
+Photos directory, do
+
+    rclone dedupe --dedupe-mode rename "drive:Google Photos"

 rclone config

@@ -407,6 +450,12 @@ The connection timeout is the amount of time rclone will wait for a
 connection to go through to a remote object storage system. It is 1m by
 default.

+--dedupe-mode MODE
+
+Mode to run dedupe command in. One of interactive, skip, first, newest,
+oldest, rename. The default is interactive. See the dedupe command for
+more information as to what these options mean.
+
 -n, --dry-run

 Do a trial run with no permanent changes. Use this to see what rclone
@@ -422,6 +471,15 @@ While this isn't a generally recommended option, it can be useful in
 cases where your files change due to encryption. However, it cannot
 correct partial transfers in case a transfer was interrupted.

+-I, --ignore-times
+
+Using this option will cause rclone to unconditionally upload all files
+regardless of the state of files on the destination.
+
+Normally rclone would skip any files that have the same modification
+time and are the same size (or have the same checksum if using
+--checksum).
+
 --log-file=FILE

 Log all of rclone's output to FILE. This is not active by default. This
@@ -814,6 +872,9 @@ will only match a complete path element:
              - doesn't match "afile.jpg"
              - doesn't match "directory/file.jpg"

+IMPORTANT Note that you must use / in patterns and not \ even if running
+on Windows.
+
 A * matches anything but not a /.

    *.jpg  - matches "file.jpg"
@@ -1107,7 +1168,7 @@ Here is an overview of the major features of each cloud storage system.
  Amazon Cloud Drive      MD5      No            Yes                No
  Microsoft One Drive     SHA1     Yes           Yes                No
  Hubic                   MD5      Yes            No                No
-  Backblaze B2            SHA1   Partial          No                No
+  Backblaze B2            SHA1     Yes            No                No
  Yandex Disk             MD5      Yes            No                No
  The local filesystem    All      Yes         Depends              No

@@ -1131,9 +1192,6 @@ the MD5SUM can be checked with the --checksum flag.
 All cloud storage systems support some kind of date on the object and
 these will be set when transferring from the cloud storage system.

-Backblaze B2 preserves file modification times on files uploaded and
-downloaded, but doesn't use them to decide which objects to sync.
-
 Case Insensitive

 If a cloud storage systems is case sensitive then it is possible to have
@@ -1160,7 +1218,8 @@ Duplicate files
 If a cloud storage system allows duplicate files then it can have two
 objects with the same name.

-This confuses rclone greatly when syncing.
+This confuses rclone greatly when syncing - use the rclone dedupe
+command to rename or remove duplicates.


 Google Drive
@@ -1284,16 +1343,20 @@ Here are the command line options specific to this cloud storage system.

 --drive-chunk-size=SIZE

-Upload chunk size. Must a power of 2 >= 256k. Default value is 256kB.
+Upload chunk size. Must a power of 2 >= 256k. Default value is 8 MB.
+
+Making this larger will improve performance, but note that each chunk is
+buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.

 --drive-full-list

-Use a full listing for directory list. More data but usually quicker. On
-by default, disable with --full-drive-list=false.
+No longer does anything - kept for backwards compatibility.

 --drive-upload-cutoff=SIZE

-File size cutoff for switching to chunked upload. Default is 256kB.
+File size cutoff for switching to chunked upload. Default is 8 MB.

 --drive-use-trash

@@ -1745,6 +1808,12 @@ X-Object-Meta-Mtime as floating point since the epoch accurate to 1 ns.
 This is a defacto standard (used in the official python-swiftclient
 amongst others) for storing the modification time for an object.

+Limitations
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.
+

 Dropbox

@@ -1861,6 +1930,12 @@ document. Rclone will issue an error message
 File name disallowed - not uploading if it attempt to upload one of
 those file names, but the sync won't fail.

+If you have more than 10,000 files in a directory then
+rclone purge dropbox:dir will return the error
+Failed to purge: There are too many files involved in this operation. As
+a work-around do an rclone delete dropbix:dir followed by an
+rclone rmdir dropbox:dir.
+

 Google Cloud Storage

@@ -2392,8 +2467,12 @@ are the same.

 Limitations

-Code to refresh the OpenStack token isn't done yet which may cause
-problems with very long transfers.
+This uses the normal OpenStack Swift mechanism to refresh the Swift API
+credentials and ignores the expires field returned by the Hubic API.
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.


 Backblaze B2
@@ -2485,9 +2564,10 @@ X-Bz-Info-src_last_modified_millis as milliseconds since 1970-01-01 in
 the Backblaze standard. Other tools should be able to use this as a
 modified time.

-Modified times are set on upload, read on download and shown in
-listings. They are not used in syncing as unfortunately B2 doesn't have
-an API method to set them independently of doing an upload.
+Modified times are used in syncing and are fully supported except in the
+case of updating a modification time on an existing object. In this case
+the object will be uploaded again as B2 doesn't have an API method to
+set the modification time independent of doing an upload.

 SHA1 checksums

@@ -2507,13 +2587,20 @@ Rclone doesn't provide any way of managing old versions (downloading
 them or deleting them) at the moment. When you purge a bucket, all the
 old versions will be deleted.

+Transfers
+
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed. In tests from my SSD equiped laptop the optimum setting
+is about --transfers 32 though higher numbers may be used for a slight
+speed improvement. The optimum number for you may vary depending on your
+hardware, how big the files are, how much you want to load your
+computer, etc. The default of --transfers 4 is definitely too low for
+Backblaze B2 though.
+
 API

 Here are some notes I made on the backblaze API while integrating it
-with rclone which detail the changes I'd like to see. With a couple of
-small tweaks Backblaze could enable rclone to not make a temporary copy
-of files when doing cloud to cloud copies and fully support modification
-times.
+with rclone which detail the changes I'd like to see.


 Yandex Disk
@@ -2692,6 +2779,84 @@ characters on z, so only use this option if you have to.

 Changelog

+-   v1.29 - 2016-04-18
+    -   New Features
+    -   Implement -I, --ignore-times for unconditional upload
+    -   Improve dedupecommand
+        -   Now removes identical copies without asking
+        -   Now obeys --dry-run
+        -   Implement --dedupe-mode for non interactive running
+        -   --dedupe-mode interactive - interactive the default.
+        -   --dedupe-mode skip - removes identical files then skips
+            anything left.
+        -   --dedupe-mode first - removes identical files then keeps the
+            first one.
+        -   --dedupe-mode newest - removes identical files then keeps
+            the newest one.
+        -   --dedupe-mode oldest - removes identical files then keeps
+            the oldest one.
+        -   --dedupe-mode rename - removes identical files then renames
+            the rest to be different.
+    -   Bug fixes
+    -   Make rclone check obey the --size-only flag.
+    -   Use "application/octet-stream" if discovered mime type is
+        invalid.
+    -   Fix missing "quit" option when there are no remotes.
+    -   Google Drive
+    -   Increase default chunk size to 8 MB - increases upload speed of
+        big files
+    -   Speed up directory listings and make more reliable
+    -   Add missing retries for Move and DirMove - increases reliability
+    -   Preserve mime type on file update
+    -   Backblaze B2
+    -   Enable mod time syncing
+        -   This means that B2 will now check modification times
+        -   It will upload new files to update the modification times
+        -   (there isn't an API to just set the mod time.)
+        -   If you want the old behaviour use --size-only.
+    -   Update API to new version
+    -   Fix parsing of mod time when not in metadata
+    -   Swift/Hubic
+    -   Don't return an MD5SUM for static large objects
+    -   S3
+    -   Fix uploading files bigger than 50GB
+-   v1.28 - 2016-03-01
+    -   New Features
+    -   Configuration file encryption - thanks Klaus Post
+    -   Improve rclone config adding more help and making it easier to
+        understand
+    -   Implement -u/--update so creation times can be used on all
+        remotes
+    -   Implement --low-level-retries flag
+    -   Optionally disable gzip compression on downloads with
+        --no-gzip-encoding
+    -   Bug fixes
+    -   Don't make directories if --dry-run set
+    -   Fix and document the move command
+    -   Fix redirecting stderr on unix-like OSes when using --log-file
+    -   Fix delete command to wait until all finished - fixes missing
+        deletes.
+    -   Backblaze B2
+    -   Use one upload URL per go routine fixes
+        more than one upload using auth token
+    -   Add pacing, retries and reauthentication - fixes token expiry
+        problems
+    -   Upload without using a temporary file from local (and remotes
+        which support SHA1)
+    -   Fix reading metadata for all files when it shouldn't have been
+    -   Drive
+    -   Fix listing drive documents at root
+    -   Disable copy and move for Google docs
+    -   Swift
+    -   Fix uploading of chunked files with non ASCII characters
+    -   Allow setting of storage_url in the config - thanks Xavier Lucas
+    -   S3
+    -   Allow IAM role and credentials from environment variables -
+        thanks Brian Stengaard
+    -   Allow low privilege users to use S3 (check if directory exists
+        during Mkdir) - thanks Jakub Gedeon
+    -   Amazon Cloud Drive
+    -   Retry on more things to make directory listings more reliable
 -   v1.27 - 2016-01-31
    -   New Features
    -   Easier headless configuration with rclone authorize
@@ -3126,6 +3291,14 @@ supported by the go runtime, ie earlier than version 2.6.23.
 See the system requirements section in the go install docs for full
 details.

+All my uploaded docx/xlsx/pptx files appear as archive/zip
+
+This is caused by uploading these files from a Windows computer which
+hasn't got the Microsoft Office suite installed. The easiest way to fix
+is to install the Word viewer and the Microsoft Office Compatibility
+Pack for Word, Excel, and PowerPoint 2007 and later versions' file
+formats
+

 License

--- a/2
+++ b/2
@@ -13,7 +13,7 @@ test:	rclone

 check:	rclone
 	go vet ./...
-	[[ `go version` =~ go1.[0-4][^0-9] ]] || errcheck ./...
+	errcheck ./...
 	goimports -d . | grep . ; test $$? -eq 1
 	golint ./... | grep -E -v '(StorageUrl|CdnUrl)' ; test $$? -eq 1

--- a/RELEASE.md
+++ b/RELEASE.md
@@ -13,6 +13,7 @@ Making a release
  * make test
  * make tag
  * edit docs/content/changelog.md
+  * make doc
  * git commit -a -v
  * make retag
  * # Set the GOPATH for a gox enabled compiler - . ~/bin/go-cross - not required for go >= 1.5
--- a/amazonclouddrive/amazonclouddrive.go
+++ b/amazonclouddrive/amazonclouddrive.go
@@ -745,9 +745,9 @@ func (o *Object) ModTime() time.Time {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	// FIXME not implemented
-	return
+	return fs.ErrorCantSetModTime
 }

 // Storable returns a boolean showing whether this object storable
--- a/appveyor.yml
+++ b/appveyor.yml
@@ -8,7 +8,6 @@ environment:
  GOPATH: c:\gopath

 install:
-  - go get golang.org/x/tools/cmd/vet
  - echo %PATH%
  - echo %GOPATH%
  - go version
--- a/b2/api/types.go
+++ b/b2/api/types.go
@@ -57,11 +57,14 @@ func (t *Timestamp) UnmarshalJSON(data []byte) error {

 // File is info about a file
 type File struct {
-	ID              string    `json:"fileId"`          // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
-	Name            string    `json:"fileName"`        // The name of this file, which can be used with b2_download_file_by_name.
-	Action          string    `json:"action"`          // Either "upload" or "hide". "upload" means a file that was uploaded to B2 Cloud Storage. "hide" means a file version marking the file as hidden, so that it will not show up in b2_list_file_names. The result of b2_list_file_names will contain only "upload". The result of b2_list_file_versions may have both.
-	Size            int64     `json:"size"`            // The number of bytes in the file.
-	UploadTimestamp Timestamp `json:"uploadTimestamp"` // This is a UTC time when this file was uploaded.
+	ID              string            `json:"fileId"`          // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
+	Name            string            `json:"fileName"`        // The name of this file, which can be used with b2_download_file_by_name.
+	Action          string            `json:"action"`          // Either "upload" or "hide". "upload" means a file that was uploaded to B2 Cloud Storage. "hide" means a file version marking the file as hidden, so that it will not show up in b2_list_file_names. The result of b2_list_file_names will contain only "upload". The result of b2_list_file_versions may have both.
+	Size            int64             `json:"size"`            // The number of bytes in the file.
+	UploadTimestamp Timestamp         `json:"uploadTimestamp"` // This is a UTC time when this file was uploaded.
+	SHA1            string            `json:"contentSha1"`     // The SHA1 of the bytes stored in the file.
+	ContentType     string            `json:"contentType"`     // The MIME type of the file.
+	Info            map[string]string `json:"fileInfo"`        // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
 }

 // AuthorizeAccountResponse is as returned from the b2_authorize_account call
@@ -108,6 +111,7 @@ type GetUploadURLResponse struct {
 type FileInfo struct {
 	ID          string            `json:"fileId"`        // The unique identifier for this version of this file. Used with b2_get_file_info, b2_download_file_by_id, and b2_delete_file_version.
 	Name        string            `json:"fileName"`      // The name of this file, which can be used with b2_download_file_by_name.
+	Action      string            `json:"action"`        // Either "upload" or "hide". "upload" means a file that was uploaded to B2 Cloud Storage. "hide" means a file version marking the file as hidden, so that it will not show up in b2_list_file_names. The result of b2_list_file_names will contain only "upload". The result of b2_list_file_versions may have both.
 	AccountID   string            `json:"accountId"`     // Your account ID.
 	BucketID    string            `json:"bucketId"`      // The bucket that the file is in.
 	Size        int64             `json:"contentLength"` // The number of bytes stored in the file.
--- a/b2/b2.go
+++ b/b2/b2.go
@@ -80,14 +80,13 @@ type Fs struct {
 }

 // Object describes a b2 object
-//
-// Will definitely have info
 type Object struct {
 	fs      *Fs       // what this object is part of
 	remote  string    // The remote path
-	info    api.File  // Info from the b2 object if known
+	id      string    // b2 id of the file
 	modTime time.Time // The modified time of the object if known
 	sha1    string    // SHA-1 hash if known
+	size    int64     // Size of the object
 }

 // ------------------------------------------------------------
@@ -308,8 +307,11 @@ func (f *Fs) newFsObjectWithInfo(remote string, info *api.File) fs.Object {
 		remote: remote,
 	}
 	if info != nil {
-		// Set info but not headers
-		o.info = *info
+		err := o.decodeMetaData(info)
+		if err != nil {
+			fs.Debug(o, "Failed to decode metadata: %s", err)
+			return nil
+		}
 	} else {
 		err := o.readMetaData() // reads info and headers, returning an error
 		if err != nil {
@@ -375,9 +377,6 @@ func (f *Fs) list(prefix string, limit int, hidden bool, fn listFn) error {
 			return f.shouldRetry(resp, err)
 		})
 		if err != nil {
-			if err == errEndList {
-				return nil
-			}
 			return err
 		}
 		for i := range response.Files {
@@ -388,6 +387,9 @@ func (f *Fs) list(prefix string, limit int, hidden bool, fn listFn) error {
 			}
 			err = fn(file.Name[len(f.root):], file)
 			if err != nil {
+				if err == errEndList {
+					return nil
+				}
 				return err
 			}
 		}
@@ -613,7 +615,7 @@ func (f *Fs) Rmdir() error {

 // Precision of the remote
 func (f *Fs) Precision() time.Duration {
-	return fs.ModTimeNotSupported
+	return time.Millisecond
 }

 // deleteByID deletes a file version given Name and ID
@@ -711,8 +713,8 @@ func (o *Object) Hash(t fs.HashType) (string, error) {
 		return "", fs.ErrHashUnsupported
 	}
 	if o.sha1 == "" {
-		// Error is logged in readFileMetadata
-		err := o.readFileMetadata()
+		// Error is logged in readMetaData
+		err := o.readMetaData()
 		if err != nil {
 			return "", err
 		}
@@ -722,26 +724,50 @@ func (o *Object) Hash(t fs.HashType) (string, error) {

 // Size returns the size of an object in bytes
 func (o *Object) Size() int64 {
-	return o.info.Size
+	return o.size
+}
+
+// decodeMetaData sets the metadata in the object from info
+//
+// Sets
+//  o.id
+//  o.modTime
+//  o.size
+//  o.sha1
+func (o *Object) decodeMetaData(info *api.File) (err error) {
+	o.id = info.ID
+	o.sha1 = info.SHA1
+	o.size = info.Size
+	// Use the UploadTimestamp if can't get file info
+	o.modTime = time.Time(info.UploadTimestamp)
+	return o.parseTimeString(info.Info[timeKey])
 }

 // readMetaData gets the metadata if it hasn't already been fetched
 //
-// it also sets the info
+// Sets
+//  o.id
+//  o.modTime
+//  o.size
+//  o.sha1
 func (o *Object) readMetaData() (err error) {
-	if o.info.ID != "" {
+	if o.id != "" {
 		return nil
 	}
+	var info *api.File
 	err = o.fs.list(o.remote, 1, false, func(remote string, object *api.File) error {
 		if remote == o.remote {
-			o.info = *object
+			info = object
 		}
 		return errEndList // read only 1 item
 	})
-	if o.info.ID != "" {
-		return nil
+	if err != nil {
+		return err
 	}
-	return fmt.Errorf("Object %q not found", o.remote)
+	if info == nil {
+		return fmt.Errorf("Object %q not found", o.remote)
+	}
+	return o.decodeMetaData(info)
 }

 // timeString returns modTime as the number of milliseconds
@@ -751,16 +777,19 @@ func timeString(modTime time.Time) string {
 }

 // parseTimeString converts a decimal string number of milliseconds
-// elapsed since January 1, 1970 UTC into a time.Time
-func parseTimeString(timeString string) (result time.Time, err error) {
+// elapsed since January 1, 1970 UTC into a time.Time and stores it in
+// the modTime variable.
+func (o *Object) parseTimeString(timeString string) (err error) {
 	if timeString == "" {
-		return result, fmt.Errorf("%q not found in metadata", timeKey)
+		return nil
 	}
 	unixMilliseconds, err := strconv.ParseInt(timeString, 10, 64)
 	if err != nil {
-		return result, err
+		fs.Debug(o, "Failed to parse mod time string %q: %v", timeString, err)
+		return err
 	}
-	return time.Unix(unixMilliseconds/1E3, (unixMilliseconds%1E3)*1E6).UTC(), nil
+	o.modTime = time.Unix(unixMilliseconds/1E3, (unixMilliseconds%1E3)*1E6).UTC()
+	return nil
 }

 // ModTime returns the modification time of the object
@@ -770,72 +799,15 @@ func parseTimeString(timeString string) (result time.Time, err error) {
 //
 // SHA-1 will also be updated once the request has completed.
 func (o *Object) ModTime() (result time.Time) {
-	// The error is logged in readFileMetadata
-	_ = o.readFileMetadata()
+	// The error is logged in readMetaData
+	_ = o.readMetaData()
 	return o.modTime
 }

-// readFileMetadata attempts to read the modified time and
-// SHA-1 hash of the remote object.
-//
-// If the objects mtime and if that isn't present the
-// LastModified returned in the http headers.
-//
-// It is safe to call this function multiple times, and the
-// result is cached between calls.
-func (o *Object) readFileMetadata() error {
-	// Return if already know it
-	if !o.modTime.IsZero() && o.sha1 != "" {
-		return nil
-	}
-
-	// Set modtime to now, as default value.
-	o.modTime = time.Now()
-
-	// Read metadata (we need the ID)
-	err := o.readMetaData()
-	if err != nil {
-		fs.Debug(o, "Failed to get file metadata: %v", err)
-		return err
-	}
-
-	// Use the UploadTimestamp if can't get file info
-	o.modTime = time.Time(o.info.UploadTimestamp)
-
-	// Now read the metadata for the modified time
-	opts := rest.Opts{
-		Method: "POST",
-		Path:   "/b2_get_file_info",
-	}
-	var request = api.GetFileInfoRequest{
-		ID: o.info.ID,
-	}
-	var response api.FileInfo
-	err = o.fs.pacer.Call(func() (bool, error) {
-		resp, err := o.fs.srv.CallJSON(&opts, &request, &response)
-		return o.fs.shouldRetry(resp, err)
-	})
-	if err != nil {
-		fs.Debug(o, "Failed to get file info: %v", err)
-		return err
-	}
-	o.sha1 = response.SHA1
-
-	// Parse the result
-	timeString := response.Info[timeKey]
-	parsed, err := parseTimeString(timeString)
-	if err != nil {
-		fs.Debug(o, "Failed to parse mod time string %q: %v", timeString, err)
-		return err
-	}
-	o.modTime = parsed
-
-	return nil
-}
-
 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	// Not possible with B2
+	return fs.ErrorCantSetModTime
 }

 // Storable returns if this object is storable
@@ -920,12 +892,10 @@ func (o *Object) Open() (in io.ReadCloser, err error) {
 	}

 	// Parse the time out of the headers if possible
-	timeString := resp.Header.Get(timeHeader)
-	parsed, err := parseTimeString(timeString)
+	err = o.parseTimeString(resp.Header.Get(timeHeader))
 	if err != nil {
-		fs.Debug(o, "Failed to parse mod time string %q: %v", timeString, err)
-	} else {
-		o.modTime = parsed
+		_ = resp.Body.Close()
+		return nil, err
 	}
 	if o.sha1 == "" {
 		o.sha1 = resp.Header.Get(sha1Header)
@@ -1095,12 +1065,11 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) (err error) {
 	if err != nil {
 		return err
 	}
-	o.info.ID = response.ID
-	o.info.Name = response.Name
-	o.info.Action = "upload"
-	o.info.Size = response.Size
-	o.info.UploadTimestamp = api.Timestamp(time.Now()) // FIXME not quite right
+	o.id = response.ID
 	o.sha1 = response.SHA1
+	o.size = response.Size
+	o.modTime = modTime
+	_ = o.parseTimeString(response.Info[timeKey])
 	return nil
 }

--- a/b2/b2_internal_test.go
+++ b/b2/b2_internal_test.go
@@ -149,10 +149,12 @@ func TestParseTimeString(t *testing.T) {
 	}{
 		{"0", fstest.Time("1970-01-01T00:00:00.000000000Z"), ""},
 		{"981173110123", fstest.Time("2001-02-03T04:05:10.123000000Z"), ""},
-		{"", time.Time{}, `"src_last_modified_millis" not found in metadata`},
+		{"", time.Time{}, ""},
 		{"potato", time.Time{}, `strconv.ParseInt: parsing "potato": invalid syntax`},
 	} {
-		got, err := parseTimeString(test.in)
+		o := Object{}
+		err := o.parseTimeString(test.in)
+		got := o.modTime
 		var gotError string
 		if err != nil {
 			gotError = err.Error()
--- a/docs/content/b2.md
+++ b/docs/content/b2.md
@@ -96,9 +96,10 @@ The modified time is stored as metadata on the object as
 in the Backblaze standard.  Other tools should be able to use this as
 a modified time.

-Modified times are set on upload, read on download and shown in
-listings.  They are not used in syncing as unfortunately B2 doesn't
-have an API method to set them independently of doing an upload.
+Modified times are used in syncing and are fully supported except in
+the case of updating a modification time on an existing object.  In
+this case the object will be uploaded again as B2 doesn't have an API
+method to set the modification time independent of doing an upload.

 ### SHA1 checksums ###

@@ -119,11 +120,18 @@ Rclone doesn't provide any way of managing old versions (downloading
 them or deleting them) at the moment.  When you `purge` a bucket, all
 the old versions will be deleted.

+### Transfers ###
+
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed.  In tests from my SSD equiped laptop the optimum
+setting is about `--transfers 32` though higher numbers may be used
+for a slight speed improvement. The optimum number for you may vary
+depending on your hardware, how big the files are, how much you want
+to load your computer, etc.  The default of `--transfers 4` is
+definitely too low for Backblaze B2 though.
+
 ### API ###

 Here are [some notes I made on the backblaze
 API](https://gist.github.com/ncw/166dabf352b399f1cc1c) while
 integrating it with rclone which detail the changes I'd like to see.
-With a couple of small tweaks Backblaze could enable rclone to not
-make a temporary copy of files when doing cloud to cloud copies and
-fully support modification times.
--- a/docs/content/changelog.md
+++ b/docs/content/changelog.md
@@ -1,12 +1,46 @@
 ---
 title: "Documentation"
 description: "Rclone Changelog"
-date: "2016-03-01"
+date: "2016-04-18"
 ---

 Changelog
 ---------

+  * v1.29 - 2016-04-18
+    * New Features
+      * Implement `-I, --ignore-times` for unconditional upload
+      * Improve `dedupe`command
+        * Now removes identical copies without asking
+        * Now obeys `--dry-run`
+        * Implement `--dedupe-mode` for non interactive running
+          * `--dedupe-mode interactive` - interactive the default.
+          * `--dedupe-mode skip` - removes identical files then skips anything left.
+          * `--dedupe-mode first` - removes identical files then keeps the first one.
+          * `--dedupe-mode newest` - removes identical files then keeps the newest one.
+          * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
+          * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
+    * Bug fixes
+      * Make rclone check obey the `--size-only` flag.
+      * Use "application/octet-stream" if discovered mime type is invalid.
+      * Fix missing "quit" option when there are no remotes.
+    * Google Drive
+      * Increase default chunk size to 8 MB - increases upload speed of big files
+      * Speed up directory listings and make more reliable
+      * Add missing retries for Move and DirMove - increases reliability
+      * Preserve mime type on file update
+    * Backblaze B2
+      * Enable mod time syncing
+        * This means that B2 will now check modification times
+        * It will upload new files to update the modification times
+        * (there isn't an API to just set the mod time.)
+        * If you want the old behaviour use `--size-only`.
+      * Update API to new version
+      * Fix parsing of mod time when not in metadata
+    * Swift/Hubic
+      * Don't return an MD5SUM for static large objects
+    * S3
+      * Fix uploading files bigger than 50GB
  * v1.28 - 2016-03-01
    * New Features
      * Configuration file encryption - thanks Klaus Post
--- a/docs/content/docs.md
+++ b/docs/content/docs.md
@@ -192,19 +192,55 @@ Checks the files in the source and destination match.  It
 compares sizes and MD5SUMs and prints a report of files which
 don't match.  It doesn't alter the source or destination.

+`--size-only` may be used to only compare the sizes, not the MD5SUMs.
+
 ### rclone dedupe remote:path ###

-Interactively find duplicate files and offer to delete all but one or
-rename them to be different. Only useful with Google Drive which can
-have duplicate file names.
+By default `dedup` interactively finds duplicate files and offers to
+delete all but one or rename them to be different. Only useful with
+Google Drive which can have duplicate file names.
+
+The `dedupe` command will delete all but one of any identical (same
+md5sum) files it finds without confirmation.  This means that for most
+duplicated files the `dedupe` command will not be interactive.  You
+can use `--dry-run` to see what would happen without doing anything.
+
+Here is an example run.
+
+Before - with duplicates
+
+```
+$ rclone lsl drive:dupes
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+  6048320 2016-03-05 16:23:11.775000000 one.txt
+   564374 2016-03-05 16:23:06.731000000 one.txt
+  6048320 2016-03-05 16:18:26.092000000 one.txt
+  6048320 2016-03-05 16:22:46.185000000 two.txt
+  1744073 2016-03-05 16:22:38.104000000 two.txt
+   564374 2016-03-05 16:22:52.118000000 two.txt
+```
+
+Now the `dedupe` session

 ```
 $ rclone dedupe drive:dupes
-2016/01/31 14:13:11 Google drive root 'dupes': Looking for duplicates
-two.txt: Found 3 duplicates
-  1:       564374 bytes, 2016-01-31 14:07:22.159000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
-  2:      1744073 bytes, 2016-01-31 14:07:12.490000000, md5sum 851957f7fb6f0bc4ce76be966d336802
-  3:      6048320 bytes, 2016-01-31 14:07:02.111000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+2016/03/05 16:24:37 Google drive root 'dupes': Looking for duplicates using interactive mode.
+one.txt: Found 4 duplicates - deleting identical copies
+one.txt: Deleting 2/3 identical duplicates (md5sum "1eedaa9fe86fd4b8632e2ac549403b36")
+one.txt: 2 duplicates remain
+  1:      6048320 bytes, 2016-03-05 16:23:16.798000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  2:       564374 bytes, 2016-03-05 16:23:06.731000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+s) Skip and do nothing
+k) Keep just one (choose which in next step)
+r) Rename all to be different (by changing file.jpg to file-1.jpg)
+s/k/r> k
+Enter the number of the file to keep> 1
+one.txt: Deleted 1 extra copies
+two.txt: Found 3 duplicates - deleting identical copies
+two.txt: 3 duplicates remain
+  1:       564374 bytes, 2016-03-05 16:22:52.118000000, md5sum 7594e7dc9fc28f727c42ee3e0749de81
+  2:      6048320 bytes, 2016-03-05 16:22:46.185000000, md5sum 1eedaa9fe86fd4b8632e2ac549403b36
+  3:      1744073 bytes, 2016-03-05 16:22:38.104000000, md5sum 851957f7fb6f0bc4ce76be966d336802
 s) Skip and do nothing
 k) Keep just one (choose which in next step)
 r) Rename all to be different (by changing file.jpg to file-1.jpg)
@@ -212,27 +248,31 @@ s/k/r> r
 two-1.txt: renamed from: two.txt
 two-2.txt: renamed from: two.txt
 two-3.txt: renamed from: two.txt
-one.txt: Found 2 duplicates
-  1:         6579 bytes, 2016-01-31 14:05:01.235000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-  2:         6579 bytes, 2016-01-31 12:50:30.318000000, md5sum 2b76c776249409d925ae7ccd49aea59b
-s) Skip and do nothing
-k) Keep just one (choose which in next step)
-r) Rename all to be different (by changing file.jpg to file-1.jpg)
-s/k/r> k
-Enter the number of the file to keep> 2
-one.txt: Deleted 1 extra copies
 ```

 The result being

 ```
 $ rclone lsl drive:dupes
-   564374 2016-01-31 14:07:22.159000000 two-1.txt
-  1744073 2016-01-31 14:07:12.490000000 two-2.txt
-  6048320 2016-01-31 14:07:02.111000000 two-3.txt
-     6579 2016-01-31 12:50:30.318000000 one.txt
+  6048320 2016-03-05 16:23:16.798000000 one.txt
+   564374 2016-03-05 16:22:52.118000000 two-1.txt
+  6048320 2016-03-05 16:22:46.185000000 two-2.txt
+  1744073 2016-03-05 16:22:38.104000000 two-3.txt
 ```

+Dedupe can be run non interactively using the `--dedupe-mode` flag.
+
+  * `--dedupe-mode interactive` - interactive as above.
+  * `--dedupe-mode skip` - removes identical files then skips anything left.
+  * `--dedupe-mode first` - removes identical files then keeps the first one.
+  * `--dedupe-mode newest` - removes identical files then keeps the newest one.
+  * `--dedupe-mode oldest` - removes identical files then keeps the oldest one.
+  * `--dedupe-mode rename` - removes identical files then renames the rest to be different.
+
+For example to rename all the identically named photos in your Google Photos directory, do
+
+    rclone dedupe --dedupe-mode rename "drive:Google Photos"
+
 ### rclone config ###

 Enter an interactive configuration session.
@@ -342,6 +382,10 @@ The connection timeout is the amount of time rclone will wait for a
 connection to go through to a remote object storage system.  It is
 `1m` by default.

+### --dedupe-mode MODE ###
+
+Mode to run dedupe command in.  One of `interactive`, `skip`, `first`, `newest`, `oldest`, `rename`.  The default is `interactive`.  See the dedupe command for more information as to what these options mean.
+
 ### -n, --dry-run ###

 Do a trial run with no permanent changes.  Use this to see what rclone
@@ -357,6 +401,15 @@ While this isn't a generally recommended option, it can be useful
 in cases where your files change due to encryption. However, it cannot
 correct partial transfers in case a transfer was interrupted.

+### -I, --ignore-times ###
+
+Using this option will cause rclone to unconditionally upload all
+files regardless of the state of files on the destination.
+
+Normally rclone would skip any files that have the same
+modification time and are the same size (or have the same checksum if
+using `--checksum`).
+
 ### --log-file=FILE ###

 Log all of rclone's output to FILE.  This is not active by default.
--- a/docs/content/downloads.md
+++ b/docs/content/downloads.md
@@ -2,40 +2,40 @@
 title: "Rclone downloads"
 description: "Download rclone binaries for your OS."
 type: page
-date: "2016-03-01"
+date: "2016-04-18"
 ---

-Rclone Download v1.28
+Rclone Download v1.29
 =====================

  * Windows
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-windows-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-windows-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-windows-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-windows-amd64.zip)
  * OSX
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-osx-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-osx-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-osx-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-osx-amd64.zip)
  * Linux
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-linux-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-linux-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.28-linux-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-linux-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-linux-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.29-linux-arm.zip)
  * FreeBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-freebsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-freebsd-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.28-freebsd-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-freebsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-freebsd-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.29-freebsd-arm.zip)
  * NetBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-netbsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-netbsd-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.28-netbsd-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-netbsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-netbsd-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.29-netbsd-arm.zip)
  * OpenBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-openbsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-openbsd-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-openbsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-openbsd-amd64.zip)
  * Plan 9
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.28-plan9-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-plan9-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.29-plan9-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-plan9-amd64.zip)
  * Solaris
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.28-solaris-amd64.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.29-solaris-amd64.zip)

-You can also find a [mirror of the downloads on github](https://github.com/ncw/rclone/releases/tag/v1.28).
+You can also find a [mirror of the downloads on github](https://github.com/ncw/rclone/releases/tag/v1.29).

 Downloads for scripting
 =======================
--- a/docs/content/drive.md
+++ b/docs/content/drive.md
@@ -1,7 +1,7 @@
 ---
 title: "Google drive"
 description: "Rclone docs for Google drive"
-date: "2015-09-12"
+date: "2016-04-12"
 ---

 <i class="fa fa-google"></i> Google Drive
@@ -129,16 +129,20 @@ system.

 #### --drive-chunk-size=SIZE ####

-Upload chunk size. Must a power of 2 >= 256k. Default value is 256kB.
+Upload chunk size. Must a power of 2 >= 256k. Default value is 8 MB.
+
+Making this larger will improve performance, but note that each chunk
+is buffered in memory one per transfer.
+
+Reducing this will reduce memory usage but decrease performance.

 #### --drive-full-list ####

-Use a full listing for directory list. More data but usually
-quicker. On by default, disable with `--full-drive-list=false`.
+No longer does anything - kept for backwards compatibility.

 #### --drive-upload-cutoff=SIZE ####

-File size cutoff for switching to chunked upload.  Default is 256kB.
+File size cutoff for switching to chunked upload.  Default is 8 MB.

 #### --drive-use-trash ####

--- a/docs/content/dropbox.md
+++ b/docs/content/dropbox.md
@@ -122,3 +122,8 @@ store.  There is a full list of them in the ["Ignored Files" section
 of this document](https://www.dropbox.com/en/help/145).  Rclone will
 issue an error message `File name disallowed - not uploading` if it
 attempt to upload one of those file names, but the sync won't fail.
+
+If you have more than 10,000 files in a directory then `rclone purge
+dropbox:dir` will return the error `Failed to purge: There are too
+many files involved in this operation`.  As a work-around do an
+`rclone delete dropbix:dir` followed by an `rclone rmdir dropbox:dir`.
--- a/docs/content/faq.md
+++ b/docs/content/faq.md
@@ -137,3 +137,11 @@ supported by the go runtime, ie earlier than version 2.6.23.

 See the [system requirements section in the go install
 docs](https://golang.org/doc/install) for full details.
+
+### All my uploaded docx/xlsx/pptx files appear as archive/zip ###
+
+This is caused by uploading these files from a Windows computer which
+hasn't got the Microsoft Office suite installed.  The easiest way to
+fix is to install the Word viewer and the Microsoft Office
+Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
+versions' file formats
--- a/docs/content/filtering.md
+++ b/docs/content/filtering.md
@@ -37,6 +37,9 @@ If it doesn't start with `/` then it is matched starting at the
              - doesn't match "afile.jpg"
              - doesn't match "directory/file.jpg"

+**Important** Note that you must use `/` in patterns and not `\` even
+if running on Windows.
+
 A `*` matches anything but not a `/`.

    *.jpg  - matches "file.jpg"
--- a/docs/content/hubic.md
+++ b/docs/content/hubic.md
@@ -1,7 +1,7 @@
 ---
 title: "Hubic"
 description: "Rclone docs for Hubic"
-date: "2015-11-08"
+date: "2016-03-16"
 ---

 <i class="fa fa-space-shuttle"></i> Hubic
@@ -115,5 +115,10 @@ are the same.

 ### Limitations ###

-Code to refresh the OpenStack token isn't done yet which may cause
-problems with very long transfers.
+This uses the normal OpenStack Swift mechanism to refresh the Swift
+API credentials and ignores the expires field returned by the Hubic
+API.
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.
--- a/docs/content/install.md
+++ b/docs/content/install.md
@@ -1,7 +1,7 @@
 ---
 title: "Install"
 description: "Rclone Installation"
-date: "2015-06-12"
+date: "2016-03-28"
 ---

 Install
@@ -11,15 +11,15 @@ Rclone is a Go program and comes as a single binary file.

 [Download](/downloads/) the relevant binary.

-Or alternatively if you have Go installed use
+Or alternatively if you have Go 1.5+ installed use

    go get github.com/ncw/rclone

 and this will build the binary in `$GOPATH/bin`.  If you have built
 rclone before then you will want to update its dependencies first with
-this (remove `-f` if using go < 1.4)
+this

-    go get -u -v -f github.com/ncw/rclone/...
+    go get -u -v github.com/ncw/rclone/...

 See the [Usage section](/docs/) of the docs for how to use rclone, or
 run `rclone -h`.
--- a/docs/content/overview.md
+++ b/docs/content/overview.md
@@ -25,7 +25,7 @@ Here is an overview of the major features of each cloud storage system.
 | Amazon Cloud Drive     | MD5     | No      | Yes              | No              |
 | Microsoft One Drive    | SHA1    | Yes     | Yes              | No              |
 | Hubic                  | MD5     | Yes     | No               | No              |
-| Backblaze B2           | SHA1    | Partial | No               | No              |
+| Backblaze B2           | SHA1    | Yes     | No               | No              |
 | Yandex Disk            | MD5     | Yes     | No               | No              |
 | The local filesystem   | All     | Yes     | Depends          | No              |

@@ -49,9 +49,6 @@ default, though the MD5SUM can be checked with the `--checksum` flag.
 All cloud storage systems support some kind of date on the object and
 these will be set when transferring from the cloud storage system.

-Backblaze B2 preserves file modification times on files uploaded and
-downloaded, but doesn't use them to decide which objects to sync.
-
 ### Case Insensitive ###

 If a cloud storage systems is case sensitive then it is possible to
@@ -78,4 +75,5 @@ systems.
 If a cloud storage system allows duplicate files then it can have two
 objects with the same name.

-This confuses rclone greatly when syncing.
+This confuses rclone greatly when syncing - use the `rclone dedupe`
+command to rename or remove duplicates.
--- a/docs/content/swift.md
+++ b/docs/content/swift.md
@@ -131,3 +131,9 @@ ns.

 This is a defacto standard (used in the official python-swiftclient
 amongst others) for storing the modification time for an object.
+
+### Limitations ###
+
+The Swift API doesn't return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won't check or use the
+MD5SUM for these.
--- a/drive/drive.go
+++ b/drive/drive.go
@@ -36,8 +36,9 @@ const (
 	timeFormatIn                = time.RFC3339
 	timeFormatOut               = "2006-01-02T15:04:05.000000000Z07:00"
 	minSleep                    = 10 * time.Millisecond
-	maxSleep                    = 2 * time.Second
-	decayConstant               = 2 // bigger for slower decay, exponential
+	maxSleep                    = 2000 * time.Millisecond
+	decayConstant               = 0 // bigger for slower decay, exponential
+	attackConstant              = 0 // bigger for slower attack, exponential
 	defaultExtensions           = "docx,xlsx,pptx,svg"
 )

@@ -50,7 +51,7 @@ var (
 	driveExtensions    = pflag.StringP("drive-formats", "", defaultExtensions, "Comma separated list of preferred formats for downloading Google docs.")
 	// chunkSize is the size of the chunks created during a resumable upload and should be a power of two.
 	// 1<<18 is the minimum size supported by the Google uploader, and there is no maximum.
-	chunkSize         = fs.SizeSuffix(256 * 1024)
+	chunkSize         = fs.SizeSuffix(8 * 1024 * 1024)
 	driveUploadCutoff = chunkSize
 	// Description of how to auth for this app
 	driveConfig = &oauth2.Config{
@@ -291,7 +292,7 @@ func NewFs(name, path string) (fs.Fs, error) {
 	f := &Fs{
 		name:  name,
 		root:  root,
-		pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant),
+		pacer: pacer.New().SetMinSleep(minSleep).SetMaxSleep(maxSleep).SetDecayConstant(decayConstant).SetAttackConstant(attackConstant),
 	}

 	// Create a new authorized Drive client.
@@ -771,7 +772,11 @@ func (f *Fs) Move(src fs.Object, remote string) (fs.Object, error) {
 	}

 	// Do the move
-	info, err := f.svc.Files.Patch(srcObj.id, dstInfo).SetModifiedDate(true).Do()
+	var info *drive.File
+	err = f.pacer.Call(func() (bool, error) {
+		info, err = f.svc.Files.Patch(srcObj.id, dstInfo).SetModifiedDate(true).Do()
+		return shouldRetry(err)
+	})
 	if err != nil {
 		return nil, err
 	}
@@ -813,7 +818,10 @@ func (f *Fs) DirMove(src fs.Fs) error {
 		Title:   leaf,
 		Parents: []*drive.ParentReference{{Id: directoryID}},
 	}
-	_, err = f.svc.Files.Patch(srcFs.dirCache.RootID(), &patch).Do()
+	err = f.pacer.Call(func() (bool, error) {
+		_, err = f.svc.Files.Patch(srcFs.dirCache.RootID(), &patch).Do()
+		return shouldRetry(err)
+	})
 	if err != nil {
 		return err
 	}
@@ -928,12 +936,10 @@ func (o *Object) ModTime() time.Time {
 }

 // SetModTime sets the modification time of the drive fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	err := o.readMetaData()
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to read metadata: %s", err)
-		return
+		return err
 	}
 	// New metadata
 	updateInfo := &drive.File{
@@ -946,12 +952,11 @@ func (o *Object) SetModTime(modTime time.Time) {
 		return shouldRetry(err)
 	})
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to update remote mtime: %s", err)
-		return
+		return err
 	}
 	// Update info from read data
 	o.setMetaData(info)
+	return nil
 }

 // Storable returns a boolean as to whether this object is storable
@@ -1044,6 +1049,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 	}
 	updateInfo := &drive.File{
 		Id:           o.id,
+		MimeType:     fs.MimeType(o),
 		ModifiedDate: modTime.Format(timeFormatOut),
 	}

--- a/dropbox/dropbox.go
+++ b/dropbox/dropbox.go
@@ -636,9 +636,9 @@ func (o *Object) ModTime() time.Time {
 // SetModTime sets the modification time of the local fs object
 //
 // Commits the datastore
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	// FIXME not implemented
-	return
+	return fs.ErrorCantSetModTime
 }

 // Storable returns whether this object is storable
--- a/fs/config.go
+++ b/fs/config.go
@@ -69,6 +69,7 @@ var (
 	configFile      = pflag.StringP("config", "", ConfigPath, "Config file.")
 	checkSum        = pflag.BoolP("checksum", "c", false, "Skip based on checksum & size, not mod-time & size")
 	sizeOnly        = pflag.BoolP("size-only", "", false, "Skip based on size only, not mod-time or checksum")
+	ignoreTimes     = pflag.BoolP("ignore-times", "I", false, "Don't skip files that match size and time - transfer all files")
 	ignoreExisting  = pflag.BoolP("ignore-existing", "", false, "Skip all files that exist on destination")
 	dryRun          = pflag.BoolP("dry-run", "n", false, "Do a trial run with no permanent changes")
 	connectTimeout  = pflag.DurationP("contimeout", "", 60*time.Second, "Connect timeout")
@@ -83,6 +84,7 @@ var (
 	lowLevelRetries = pflag.IntP("low-level-retries", "", 10, "Number of low level retries to do.")
 	updateOlder     = pflag.BoolP("update", "u", false, "Skip files that are newer on the destination.")
 	noGzip          = pflag.BoolP("no-gzip-encoding", "", false, "Don't set Accept-Encoding: gzip.")
+	dedupeMode      = pflag.StringP("dedupe-mode", "", "interactive", "Dedupe mode interactive|skip|first|newest|oldest|rename.")
 	bwLimit         SizeSuffix

 	// Key to use for password en/decryption.
@@ -187,6 +189,7 @@ type ConfigInfo struct {
 	DryRun             bool
 	CheckSum           bool
 	SizeOnly           bool
+	IgnoreTimes        bool
 	IgnoreExisting     bool
 	ModifyWindow       time.Duration
 	Checkers           int
@@ -203,6 +206,7 @@ type ConfigInfo struct {
 	LowLevelRetries    int
 	UpdateOlder        bool // Skip files that are newer on the destination
 	NoGzip             bool // Disable compression
+	DedupeMode         DeduplicateMode
 }

 // Transport returns an http.RoundTripper with the correct timeouts
@@ -297,6 +301,7 @@ func LoadConfig() {
 	Config.ConnectTimeout = *connectTimeout
 	Config.CheckSum = *checkSum
 	Config.SizeOnly = *sizeOnly
+	Config.IgnoreTimes = *ignoreTimes
 	Config.IgnoreExisting = *ignoreExisting
 	Config.DumpHeaders = *dumpHeaders
 	Config.DumpBodies = *dumpBodies
@@ -311,6 +316,23 @@ func LoadConfig() {
 	Config.DeleteDuring = *deleteDuring
 	Config.DeleteAfter = *deleteAfter

+	switch strings.ToLower(*dedupeMode) {
+	case "interactive":
+		Config.DedupeMode = DeduplicateInteractive
+	case "skip":
+		Config.DedupeMode = DeduplicateSkip
+	case "first":
+		Config.DedupeMode = DeduplicateFirst
+	case "newest":
+		Config.DedupeMode = DeduplicateNewest
+	case "oldest":
+		Config.DedupeMode = DeduplicateOldest
+	case "rename":
+		Config.DedupeMode = DeduplicateRename
+	default:
+		log.Fatalf(`Unknown mode for --dedupe-mode %q.`, *dedupeMode)
+	}
+
 	switch {
 	case *deleteBefore && (*deleteDuring || *deleteAfter),
 		*deleteDuring && *deleteAfter:
@@ -788,7 +810,7 @@ func EditConfig() {
 			fmt.Printf("\n")
 		} else {
 			fmt.Printf("No remotes found - make a new one\n")
-			what = append(what[1:2], what[3])
+			what = append(what[1:2], what[3:]...)
 		}
 		switch i := Command(what); i {
 		case 'e':
--- a/fs/fs.go
+++ b/fs/fs.go
@@ -31,6 +31,7 @@ var (
 	ErrorCantMove             = fmt.Errorf("Can't move object - incompatible remotes")
 	ErrorCantDirMove          = fmt.Errorf("Can't move directory - incompatible remotes")
 	ErrorDirExists            = fmt.Errorf("Can't copy directory - destination already exists")
+	ErrorCantSetModTime       = fmt.Errorf("Can't set modified time")
 )

 // RegInfo provides information about a filesystem
@@ -142,7 +143,7 @@ type Object interface {
 	String() string

 	// SetModTime sets the metadata on the object to set the modification date
-	SetModTime(time.Time)
+	SetModTime(time.Time) error

 	// Open opens the file for read.  Call Close() on the returned io.ReadCloser
 	Open() (io.ReadCloser, error)
--- a/fs/make_test_files.go
+++ b/fs/make_test_files.go
@@ -0,0 +1,146 @@
+// +build ignore
+
+// Build a directory structure with the required number of files in
+//
+// Run with go run make_test_files.go [flag] <directory>
+package main
+
+import (
+	cryptrand "crypto/rand"
+	"flag"
+	"io"
+	"log"
+	"math/rand"
+	"os"
+	"path/filepath"
+)
+
+var (
+	// Flags
+	numberOfFiles            = flag.Int("n", 1000, "Number of files to create")
+	averageFilesPerDirectory = flag.Int("files-per-directory", 10, "Average number of files per directory")
+	maxDepth                 = flag.Int("max-depth", 10, "Maximum depth of directory heirachy")
+	minFileSize              = flag.Int64("min-size", 0, "Minimum size of file to create")
+	maxFileSize              = flag.Int64("max-size", 100, "Maximum size of files to create")
+	minFileNameLength        = flag.Int("min-name-length", 4, "Minimum size of file to create")
+	maxFileNameLength        = flag.Int("max-name-length", 12, "Maximum size of files to create")
+
+	directoriesToCreate int
+	totalDirectories    int
+	fileNames           = map[string]struct{}{} // keep a note of which file name we've used already
+)
+
+// randomString create a random string for test purposes
+func randomString(n int) string {
+	const (
+		vowel     = "aeiou"
+		consonant = "bcdfghjklmnpqrstvwxyz"
+		digit     = "0123456789"
+	)
+	pattern := []string{consonant, vowel, consonant, vowel, consonant, vowel, consonant, digit}
+	out := make([]byte, n)
+	p := 0
+	for i := range out {
+		source := pattern[p]
+		p = (p + 1) % len(pattern)
+		out[i] = source[rand.Intn(len(source))]
+	}
+	return string(out)
+}
+
+// fileName creates a unique random file or directory name
+func fileName() (name string) {
+	for {
+		length := rand.Intn(*maxFileNameLength-*minFileNameLength) + *minFileNameLength
+		name = randomString(length)
+		if _, found := fileNames[name]; !found {
+			break
+		}
+	}
+	fileNames[name] = struct{}{}
+	return name
+}
+
+// dir is a directory in the directory heirachy being built up
+type dir struct {
+	name     string
+	depth    int
+	children []*dir
+	parent   *dir
+}
+
+// Create a random directory heirachy under d
+func (d *dir) createDirectories() {
+	for totalDirectories < directoriesToCreate {
+		newDir := &dir{
+			name:   fileName(),
+			depth:  d.depth + 1,
+			parent: d,
+		}
+		d.children = append(d.children, newDir)
+		totalDirectories++
+		switch rand.Intn(4) {
+		case 0:
+			if d.depth < *maxDepth {
+				newDir.createDirectories()
+			}
+		case 1:
+			return
+		}
+	}
+	return
+}
+
+// list the directory heirachy
+func (d *dir) list(path string, output []string) []string {
+	dirPath := path + "/" + d.name
+	output = append(output, dirPath)
+	for _, subDir := range d.children {
+		output = subDir.list(dirPath, output)
+	}
+	return output
+}
+
+// writeFile writes a random file at dir/name
+func writeFile(dir, name string) {
+	err := os.MkdirAll(dir, 0777)
+	if err != nil {
+		log.Fatalf("Failed to make directory %q: %v", dir, err)
+	}
+	path := filepath.Join(dir, name)
+	fd, err := os.Create(path)
+	if err != nil {
+		log.Fatalf("Failed to open file %q: %v", path, err)
+	}
+	size := rand.Int63n(*maxFileSize-*minFileSize) + *minFileSize
+	_, err = io.CopyN(fd, cryptrand.Reader, size)
+	if err != nil {
+		log.Fatalf("Failed to write %v bytes to file %q: %v", size, path, err)
+	}
+	err = fd.Close()
+	if err != nil {
+		log.Fatalf("Failed to close file %q: %v", path, err)
+	}
+}
+
+func main() {
+	flag.Parse()
+	args := flag.Args()
+	if len(args) != 1 {
+		log.Fatalf("Require 1 directory argument")
+	}
+	outputDirectory := args[0]
+	log.Printf("Output dir %q", outputDirectory)
+
+	directoriesToCreate = *numberOfFiles / *averageFilesPerDirectory
+	log.Printf("directoriesToCreate %v", directoriesToCreate)
+	root := &dir{name: outputDirectory, depth: 1}
+	for totalDirectories < directoriesToCreate {
+		root.createDirectories()
+	}
+	dirs := root.list("", []string{})
+	for i := 0; i < *numberOfFiles; i++ {
+		dir := dirs[rand.Intn(len(dirs))]
+		writeFile(dir, fileName())
+	}
+}
--- a/fs/operations.go
+++ b/fs/operations.go
@@ -5,8 +5,10 @@ package fs
 import (
 	"fmt"
 	"io"
+	"log"
 	"mime"
 	"path"
+	"sort"
 	"strings"
 	"sync"
 	"sync/atomic"
@@ -143,7 +145,14 @@ func Equal(src, dst Object) bool {
 	if !Config.CheckSum {
 		// Size and hash the same but mtime different so update the
 		// mtime of the dst object here
-		dst.SetModTime(srcModTime)
+		err := dst.SetModTime(srcModTime)
+		if err == ErrorCantSetModTime {
+			Debug(src, "src and dst identical but can't set mod time without re-uploading")
+			return false
+		} else if err != nil {
+			Stats.Error()
+			ErrorLog(dst, "Failed to read set modification time: %s", err)
+		}
 	}

 	if hash == HashNone {
@@ -157,7 +166,7 @@ func Equal(src, dst Object) bool {
 // MimeType returns a guess at the mime type from the extension
 func MimeType(o Object) string {
 	mimeType := mime.TypeByExtension(path.Ext(o.Remote()))
-	if mimeType == "" {
+	if !strings.ContainsRune(mimeType, '/') {
 		mimeType = "application/octet-stream"
 	}
 	return mimeType
@@ -306,6 +315,12 @@ func checkOne(pair ObjectPair, out ObjectPairChan) {
 		Debug(src, "Destination exists, skipping")
 		return
 	}
+	// If we should upload unconditionally
+	if Config.IgnoreTimes {
+		Debug(src, "Uploading unconditionally as --ignore-times is in use")
+		out <- pair
+		return
+	}
 	// If UpdateOlder is in effect, skip if dst is newer than src
 	if Config.UpdateOlder {
 		srcModTime := src.ModTime()
@@ -405,6 +420,23 @@ func PairMover(in ObjectPairChan, fdst Fs, wg *sync.WaitGroup) {
 	}
 }

+// DeleteFile deletes a single file respecting --dry-run and accumulating stats and errors.
+func DeleteFile(dst Object) {
+	if Config.DryRun {
+		Log(dst, "Not deleting as --dry-run")
+	} else {
+		Stats.Checking(dst)
+		err := dst.Remove()
+		Stats.DoneChecking(dst)
+		if err != nil {
+			Stats.Error()
+			ErrorLog(dst, "Couldn't delete: %s", err)
+		} else {
+			Debug(dst, "Deleted")
+		}
+	}
+}
+
 // DeleteFiles removes all the files passed in the channel
 func DeleteFiles(toBeDeleted ObjectsChan) {
 	var wg sync.WaitGroup
@@ -413,19 +445,7 @@ func DeleteFiles(toBeDeleted ObjectsChan) {
 		go func() {
 			defer wg.Done()
 			for dst := range toBeDeleted {
-				if Config.DryRun {
-					Log(dst, "Not deleting as --dry-run")
-				} else {
-					Stats.Checking(dst)
-					err := dst.Remove()
-					Stats.DoneChecking(dst)
-					if err != nil {
-						Stats.Error()
-						ErrorLog(dst, "Couldn't delete: %s", err)
-					} else {
-						Debug(dst, "Deleted")
-					}
-				}
+				DeleteFile(dst)
 			}
 		}()
 	}
@@ -664,6 +684,33 @@ func MoveDir(fdst, fsrc Fs) error {
 	return nil
 }

+// checkIdentical checks to see if dst and src are identical
+//
+// it returns true if differences were found
+func checkIdentical(dst, src Object) bool {
+	Stats.Checking(src)
+	defer Stats.DoneChecking(src)
+	if src.Size() != dst.Size() {
+		Stats.Error()
+		ErrorLog(src, "Sizes differ")
+		return true
+	}
+	if !Config.SizeOnly {
+		same, _, err := CheckHashes(src, dst)
+		if err != nil {
+			// CheckHashes will log and count errors
+			return true
+		}
+		if !same {
+			Stats.Error()
+			ErrorLog(src, "Md5sums differ")
+			return true
+		}
+	}
+	Debug(src, "OK")
+	return false
+}
+
 // Check the files in fsrc and fdst according to Size and hash
 func Check(fdst, fsrc Fs) error {
 	differences := int32(0)
@@ -733,26 +780,9 @@ func Check(fdst, fsrc Fs) error {
 		go func() {
 			defer checkerWg.Done()
 			for check := range checks {
-				dst, src := check[0], check[1]
-				Stats.Checking(src)
-				if src.Size() != dst.Size() {
-					Stats.DoneChecking(src)
-					Stats.Error()
-					ErrorLog(src, "Sizes differ")
+				if checkIdentical(check[0], check[1]) {
 					atomic.AddInt32(&differences, 1)
-					continue
 				}
-				same, _, err := CheckHashes(src, dst)
-				Stats.DoneChecking(src)
-				if err != nil {
-					continue
-				}
-				if !same {
-					Stats.Error()
-					atomic.AddInt32(&differences, 1)
-					ErrorLog(src, "Md5sums differ")
-				}
-				Debug(src, "OK")
 			}
 		}()
 	}
@@ -958,15 +988,132 @@ func Delete(f Fs) error {
 	return err
 }

+// dedupeRename renames the objs slice to different names
+func dedupeRename(remote string, objs []Object) {
+	f := objs[0].Fs()
+	mover, ok := f.(Mover)
+	if !ok {
+		log.Fatalf("Fs %v doesn't support Move", f)
+	}
+	ext := path.Ext(remote)
+	base := remote[:len(remote)-len(ext)]
+	for i, o := range objs {
+		newName := fmt.Sprintf("%s-%d%s", base, i+1, ext)
+		if !Config.DryRun {
+			newObj, err := mover.Move(o, newName)
+			if err != nil {
+				Stats.Error()
+				ErrorLog(o, "Failed to rename: %v", err)
+				continue
+			}
+			Log(newObj, "renamed from: %v", o)
+		} else {
+			Log(remote, "Not renaming to %q as --dry-run", newName)
+		}
+	}
+}
+
+// dedupeDeleteAllButOne deletes all but the one in keep
+func dedupeDeleteAllButOne(keep int, remote string, objs []Object) {
+	for i, o := range objs {
+		if i == keep {
+			continue
+		}
+		DeleteFile(o)
+	}
+	Log(remote, "Deleted %d extra copies", len(objs)-1)
+}
+
+// dedupeDeleteIdentical deletes all but one of identical (by hash) copies
+func dedupeDeleteIdentical(remote string, objs []Object) []Object {
+	// See how many of these duplicates are identical
+	byHash := make(map[string][]Object, len(objs))
+	for _, o := range objs {
+		md5sum, err := o.Hash(HashMD5)
+		if err == nil {
+			byHash[md5sum] = append(byHash[md5sum], o)
+		}
+	}
+
+	// Delete identical duplicates, refilling obj with the ones remaining
+	objs = nil
+	for md5sum, hashObjs := range byHash {
+		if len(hashObjs) > 1 {
+			Log(remote, "Deleting %d/%d identical duplicates (md5sum %q)", len(hashObjs)-1, len(hashObjs), md5sum)
+			for _, o := range hashObjs[1:] {
+				DeleteFile(o)
+			}
+		}
+		objs = append(objs, hashObjs[0])
+	}
+
+	return objs
+}
+
+// dedupeInteractive interactively dedupes the slice of objects
+func dedupeInteractive(remote string, objs []Object) {
+	fmt.Printf("%s: %d duplicates remain\n", remote, len(objs))
+	for i, o := range objs {
+		md5sum, err := o.Hash(HashMD5)
+		if err != nil {
+			md5sum = err.Error()
+		}
+		fmt.Printf("  %d: %12d bytes, %s, md5sum %32s\n", i+1, o.Size(), o.ModTime().Format("2006-01-02 15:04:05.000000000"), md5sum)
+	}
+	switch Command([]string{"sSkip and do nothing", "kKeep just one (choose which in next step)", "rRename all to be different (by changing file.jpg to file-1.jpg)"}) {
+	case 's':
+	case 'k':
+		keep := ChooseNumber("Enter the number of the file to keep", 1, len(objs))
+		dedupeDeleteAllButOne(keep-1, remote, objs)
+	case 'r':
+		dedupeRename(remote, objs)
+	}
+}
+
+type objectsSortedByModTime []Object
+
+func (objs objectsSortedByModTime) Len() int      { return len(objs) }
+func (objs objectsSortedByModTime) Swap(i, j int) { objs[i], objs[j] = objs[j], objs[i] }
+func (objs objectsSortedByModTime) Less(i, j int) bool {
+	return objs[i].ModTime().Before(objs[j].ModTime())
+}
+
+// DeduplicateMode is how the dedupe command chooses what to do
+type DeduplicateMode int
+
+// Deduplicate modes
+const (
+	DeduplicateInteractive DeduplicateMode = iota // interactively ask the user
+	DeduplicateSkip                               // skip all conflicts
+	DeduplicateFirst                              // choose the first object
+	DeduplicateNewest                             // choose the newest object
+	DeduplicateOldest                             // choose the oldest object
+	DeduplicateRename                             // rename the objects
+)
+
+func (mode DeduplicateMode) String() string {
+	switch mode {
+	case DeduplicateInteractive:
+		return "interactive"
+	case DeduplicateSkip:
+		return "skip"
+	case DeduplicateFirst:
+		return "first"
+	case DeduplicateNewest:
+		return "newest"
+	case DeduplicateOldest:
+		return "oldest"
+	case DeduplicateRename:
+		return "rename"
+	}
+	return "unknown"
+}
+
 // Deduplicate interactively finds duplicate files and offers to
 // delete all but one or rename them to be different. Only useful with
 // Google Drive which can have duplicate file names.
-func Deduplicate(f Fs) error {
-	mover, ok := f.(Mover)
-	if !ok {
-		return fmt.Errorf("%v can't Move files", f)
-	}
-	Log(f, "Looking for duplicates")
+func Deduplicate(f Fs, mode DeduplicateMode) error {
+	Log(f, "Looking for duplicates using %v mode.", mode)
 	files := map[string][]Object{}
 	for o := range f.List() {
 		remote := o.Remote()
@@ -974,43 +1121,29 @@ func Deduplicate(f Fs) error {
 	}
 	for remote, objs := range files {
 		if len(objs) > 1 {
-			fmt.Printf("%s: Found %d duplicates\n", remote, len(objs))
-			for i, o := range objs {
-				md5sum, err := o.Hash(HashMD5)
-				if err != nil {
-					md5sum = err.Error()
-				}
-				fmt.Printf("  %d: %12d bytes, %s, md5sum %32s\n", i+1, o.Size(), o.ModTime().Format("2006-01-02 15:04:05.000000000"), md5sum)
+			Log(remote, "Found %d duplicates - deleting identical copies", len(objs))
+			objs = dedupeDeleteIdentical(remote, objs)
+			if len(objs) <= 1 {
+				Log(remote, "All duplicates removed")
+				continue
 			}
-			switch Command([]string{"sSkip and do nothing", "kKeep just one (choose which in next step)", "rRename all to be different (by changing file.jpg to file-1.jpg)"}) {
-			case 's':
-			case 'k':
-				keep := ChooseNumber("Enter the number of the file to keep", 1, len(objs))
-				deleted := 0
-				for i, o := range objs {
-					if i+1 == keep {
-						continue
-					}
-					err := o.Remove()
-					if err != nil {
-						ErrorLog(o, "Failed to delete: %v", err)
-						continue
-					}
-					deleted++
-				}
-				fmt.Printf("%s: Deleted %d extra copies\n", remote, deleted)
-			case 'r':
-				ext := path.Ext(remote)
-				base := remote[:len(remote)-len(ext)]
-				for i, o := range objs {
-					newName := fmt.Sprintf("%s-%d%s", base, i+1, ext)
-					newObj, err := mover.Move(o, newName)
-					if err != nil {
-						ErrorLog(o, "Failed to rename: %v", err)
-						continue
-					}
-					fmt.Printf("%v: renamed from: %v\n", newObj, o)
-				}
+			switch mode {
+			case DeduplicateInteractive:
+				dedupeInteractive(remote, objs)
+			case DeduplicateFirst:
+				dedupeDeleteAllButOne(0, remote, objs)
+			case DeduplicateNewest:
+				sort.Sort(objectsSortedByModTime(objs)) // sort oldest first
+				dedupeDeleteAllButOne(len(objs)-1, remote, objs)
+			case DeduplicateOldest:
+				sort.Sort(objectsSortedByModTime(objs)) // sort oldest first
+				dedupeDeleteAllButOne(0, remote, objs)
+			case DeduplicateRename:
+				dedupeRename(remote, objs)
+			case DeduplicateSkip:
+				// skip
+			default:
+				//skip
 			}
 		}
 	}
--- a/fs/operations_test.go
+++ b/fs/operations_test.go
@@ -432,6 +432,43 @@ func TestSyncSizeOnly(t *testing.T) {
 	fstest.CheckItems(t, r.fremote, file1)
 }

+func TestSyncIgnoreTimes(t *testing.T) {
+	r := NewRun(t)
+	defer r.Finalise()
+	file1 := r.WriteBoth("existing", "potato", t1)
+	fstest.CheckItems(t, r.fremote, file1)
+
+	fs.Stats.ResetCounters()
+	err := fs.Sync(r.fremote, r.flocal)
+	if err != nil {
+		t.Fatalf("Sync failed: %v", err)
+	}
+
+	// We should have transferred exactly 0 files because the
+	// files were identical.
+	if fs.Stats.GetTransfers() != 0 {
+		t.Fatalf("Sync 1: want 0 transfer, got %d", fs.Stats.GetTransfers())
+	}
+
+	fs.Config.IgnoreTimes = true
+	defer func() { fs.Config.IgnoreTimes = false }()
+
+	fs.Stats.ResetCounters()
+	err = fs.Sync(r.fremote, r.flocal)
+	if err != nil {
+		t.Fatalf("Sync failed: %v", err)
+	}
+
+	// We should have transferred exactly one file even though the
+	// files were identical.
+	if fs.Stats.GetTransfers() != 1 {
+		t.Fatalf("Sync 2: want 1 transfer, got %d", fs.Stats.GetTransfers())
+	}
+
+	fstest.CheckItems(t, r.flocal, file1)
+	fstest.CheckItems(t, r.fremote, file1)
+}
+
 func TestSyncIgnoreExisting(t *testing.T) {
 	r := NewRun(t)
 	defer r.Finalise()
@@ -967,11 +1004,179 @@ func TestCheck(t *testing.T) {
 	fstest.CheckItems(t, r.fremote, file1, file3)
 	check(3, 2)

-	r.WriteObject("potato2", "------------------------------------------------------------", t1)
-	fstest.CheckItems(t, r.fremote, file1, file2, file3)
+	file2r := file2
+	if fs.Config.SizeOnly {
+		file2r = r.WriteObject("potato2", "--Some-Differences-But-Size-Only-Is-Enabled-----------------", t1)
+	} else {
+		r.WriteObject("potato2", "------------------------------------------------------------", t1)
+	}
+	fstest.CheckItems(t, r.fremote, file1, file2r, file3)
 	check(4, 1)

 	r.WriteFile("empty space", "", t2)
 	fstest.CheckItems(t, r.flocal, file1, file2, file3)
 	check(5, 0)
 }
+
+func TestCheckSizeOnly(t *testing.T) {
+	fs.Config.SizeOnly = true
+	defer func() { fs.Config.SizeOnly = false }()
+	TestCheck(t)
+}
+
+func (r *Run) checkWithDuplicates(t *testing.T, items ...fstest.Item) {
+	objects, size, err := fs.Count(r.fremote)
+	if err != nil {
+		t.Fatalf("Error listing: %v", err)
+	}
+	if objects != int64(len(items)) {
+		t.Fatalf("Error listing want %d objects, got %d", len(items), objects)
+	}
+	wantSize := int64(0)
+	for _, item := range items {
+		wantSize += item.Size
+	}
+	if wantSize != size {
+		t.Fatalf("Error listing want %d size, got %d", wantSize, size)
+	}
+}
+
+func TestDeduplicateInteractive(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one", "This is one", t1)
+	file2 := r.WriteObject("one", "This is one", t1)
+	file3 := r.WriteObject("one", "This is one", t1)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateInteractive)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	fstest.CheckItems(t, r.fremote, file1)
+}
+
+func TestDeduplicateSkip(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one", "This is one", t1)
+	file2 := r.WriteObject("one", "This is one", t1)
+	file3 := r.WriteObject("one", "This is another one", t1)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateSkip)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	r.checkWithDuplicates(t, file1, file3)
+}
+
+func TestDeduplicateFirst(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one", "This is one", t1)
+	file2 := r.WriteObject("one", "This is one A", t1)
+	file3 := r.WriteObject("one", "This is one BB", t1)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateFirst)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	objects, size, err := fs.Count(r.fremote)
+	if err != nil {
+		t.Fatalf("Error listing: %v", err)
+	}
+	if objects != 1 {
+		t.Errorf("Expecting 1 object got %v", objects)
+	}
+	if size != file1.Size && size != file2.Size && size != file3.Size {
+		t.Errorf("Size not one of the object sizes %d", size)
+	}
+}
+
+func TestDeduplicateNewest(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one", "This is one", t1)
+	file2 := r.WriteObject("one", "This is one too", t2)
+	file3 := r.WriteObject("one", "This is another one", t3)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateNewest)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	fstest.CheckItems(t, r.fremote, file3)
+}
+
+func TestDeduplicateOldest(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one", "This is one", t1)
+	file2 := r.WriteObject("one", "This is one too", t2)
+	file3 := r.WriteObject("one", "This is another one", t3)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateOldest)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	fstest.CheckItems(t, r.fremote, file1)
+}
+
+func TestDeduplicateRename(t *testing.T) {
+	if *RemoteName != "TestDrive:" {
+		t.Skip("Can only test deduplicate on google drive")
+	}
+	r := NewRun(t)
+	defer r.Finalise()
+
+	file1 := r.WriteObject("one.txt", "This is one", t1)
+	file2 := r.WriteObject("one.txt", "This is one too", t2)
+	file3 := r.WriteObject("one.txt", "This is another one", t3)
+	r.checkWithDuplicates(t, file1, file2, file3)
+
+	err := fs.Deduplicate(r.fremote, fs.DeduplicateRename)
+	if err != nil {
+		t.Fatalf("fs.Deduplicate returned error: %v", err)
+	}
+
+	for o := range r.fremote.List() {
+		remote := o.Remote()
+		if remote != "one-1.txt" &&
+			remote != "one-2.txt" &&
+			remote != "one-3.txt" {
+			t.Errorf("Bad file name after rename %q", remote)
+		}
+		size := o.Size()
+		if size != file1.Size && size != file2.Size && size != file3.Size {
+			t.Errorf("Size not one of the object sizes %d", size)
+		}
+	}
+}
--- a/fs/version.go
+++ b/fs/version.go
@@ -1,4 +1,4 @@
 package fs

 // Version of rclone
-const Version = "v1.28"
+const Version = "v1.29"
--- a/fstest/fstests/fstests.go
+++ b/fstest/fstests/fstests.go
@@ -497,7 +497,13 @@ func TestObjectSetModTime(t *testing.T) {
 	skipIfNotOk(t)
 	newModTime := fstest.Time("2011-12-13T14:15:16.999999999Z")
 	obj := findObject(t, file1.Path)
-	obj.SetModTime(newModTime)
+	err := obj.SetModTime(newModTime)
+	if err == fs.ErrorCantSetModTime {
+		t.Log(err)
+		return
+	} else if err != nil {
+		t.Fatal(err)
+	}
 	file1.ModTime = newModTime
 	file1.CheckModTime(t, obj, obj.ModTime(), remote.Precision())
 	// And make a new object and read it from there too
--- a/googlecloudstorage/googlecloudstorage.go
+++ b/googlecloudstorage/googlecloudstorage.go
@@ -567,7 +567,7 @@ func metadataFromModTime(modTime time.Time) map[string]string {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	// This only adds metadata so will perserve other metadata
 	object := storage.Object{
 		Bucket:   o.fs.bucket,
@@ -576,10 +576,10 @@ func (o *Object) SetModTime(modTime time.Time) {
 	}
 	newObject, err := o.fs.svc.Objects.Patch(o.fs.bucket, o.fs.root+o.remote, &object).Do()
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to update remote mtime: %s", err)
+		return err
 	}
 	o.setMetaData(newObject)
+	return nil
 }

 // Storable returns a boolean as to whether this object is storable
--- a/local/local.go
+++ b/local/local.go
@@ -42,6 +42,7 @@ type Fs struct {
 	root        string              // The root directory
 	precisionOk sync.Once           // Whether we need to read the precision
 	precision   time.Duration       // precision of local filesystem
+	wmu         sync.Mutex          // used for locking access to 'warned'.
 	warned      map[string]struct{} // whether we have warned about this string
 	nounc       bool                // Skip UNC conversion on Windows
 }
@@ -175,10 +176,12 @@ func (f *Fs) List() fs.ObjectsChan {
 // Any invalid UTF-8 characters will be replaced with utf8.RuneError
 func (f *Fs) cleanUtf8(name string) string {
 	if !utf8.ValidString(name) {
+		f.wmu.Lock()
 		if _, ok := f.warned[name]; !ok {
 			fs.Debug(f, "Replacing invalid UTF-8 characters in %q", name)
 			f.warned[name] = struct{}{}
 		}
+		f.wmu.Unlock()
 		name = string([]rune(name))
 	}
 	if runtime.GOOS == "windows" {
@@ -492,18 +495,13 @@ func (o *Object) ModTime() time.Time {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	err := os.Chtimes(o.path, modTime, modTime)
 	if err != nil {
-		fs.Debug(o, "Failed to set mtime on file: %s", err)
-		return
+		return err
 	}
 	// Re-read metadata
-	err = o.lstat()
-	if err != nil {
-		fs.Debug(o, "Failed to stat: %s", err)
-		return
-	}
+	return o.lstat()
 }

 // Storable returns a boolean showing if this object is storable
@@ -598,7 +596,10 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 	o.hashes = hash.Sums()

 	// Set the mtime
-	o.SetModTime(src.ModTime())
+	err = o.SetModTime(src.ModTime())
+	if err != nil {
+		return err
+	}

 	// ReRead info now that we have finished
 	return o.lstat()
@@ -706,10 +707,12 @@ func cleanWindowsName(f *Fs, name string) string {
 	}, name)

 	if name2 != original && f != nil {
+		f.wmu.Lock()
 		if _, ok := f.warned[name]; !ok {
 			fs.Debug(f, "Replacing invalid characters in %q to %q", name, name2)
 			f.warned[name] = struct{}{}
 		}
+		f.wmu.Unlock()
 	}
 	return name2
 }
--- a/oauthutil/oauthutil.go
+++ b/oauthutil/oauthutil.go
@@ -319,15 +319,17 @@ func (s *authServer) Start() {
 		Addr:    s.bindAddress,
 		Handler: mux,
 	}
+	server.SetKeepAlivesEnabled(false)
 	mux.HandleFunc("/favicon.ico", func(w http.ResponseWriter, req *http.Request) {
 		http.Error(w, "", 404)
 		return
 	})
 	mux.HandleFunc("/auth", func(w http.ResponseWriter, req *http.Request) {
-		http.Redirect(w, req, s.authURL, 307)
+		http.Redirect(w, req, s.authURL, http.StatusTemporaryRedirect)
 		return
 	})
 	mux.HandleFunc("/", func(w http.ResponseWriter, req *http.Request) {
+		w.Header().Set("Content-Type", "text/html")
 		fs.Debug(nil, "Received request on auth server")
 		code := req.FormValue("code")
 		if code != "" {
@@ -347,8 +349,9 @@ func (s *authServer) Start() {
 			return
 		}
 		fs.Debug(nil, "No code found on request")
-		fmt.Fprintf(w, "<h1>Failed!</h1>\nNo code found.")
-		http.Error(w, "", 500)
+		w.WriteHeader(500)
+		fmt.Fprintf(w, "<h1>Failed!</h1>\nNo code found returned by remote server.")
+
 	})

 	var err error
--- a/onedrive/onedrive.go
+++ b/onedrive/onedrive.go
@@ -802,13 +802,13 @@ func (o *Object) setModTime(modTime time.Time) (*api.Item, error) {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	info, err := o.setModTime(modTime)
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to update remote mtime: %v", err)
+		return err
 	}
 	o.setMetaData(info)
+	return nil
 }

 // Storable returns a boolean showing whether this object storable
--- a/pacer/pacer.go
+++ b/pacer/pacer.go
@@ -15,6 +15,7 @@ type Pacer struct {
 	minSleep           time.Duration // minimum sleep time
 	maxSleep           time.Duration // maximum sleep time
 	decayConstant      uint          // decay constant
+	attackConstant     uint          // attack constant
 	pacer              chan struct{} // To pace the operations
 	sleepTime          time.Duration // Time to sleep for each transaction
 	retries            int           // Max number of retries
@@ -58,11 +59,12 @@ type Paced func() (bool, error)
 // New returns a Pacer with sensible defaults
 func New() *Pacer {
 	p := &Pacer{
-		minSleep:      10 * time.Millisecond,
-		maxSleep:      2 * time.Second,
-		decayConstant: 2,
-		retries:       fs.Config.LowLevelRetries,
-		pacer:         make(chan struct{}, 1),
+		minSleep:       10 * time.Millisecond,
+		maxSleep:       2 * time.Second,
+		decayConstant:  2,
+		attackConstant: 1,
+		retries:        fs.Config.LowLevelRetries,
+		pacer:          make(chan struct{}, 1),
 	}
 	p.sleepTime = p.minSleep
 	p.SetPacer(DefaultPacer)
@@ -116,7 +118,7 @@ func (p *Pacer) SetMaxConnections(n int) *Pacer {
 // This is the speed the time falls back to the minimum after errors
 // have occurred.
 //
-// bigger for slower decay, exponential
+// bigger for slower decay, exponential. 1 is halve, 0 is go straight to minimum
 func (p *Pacer) SetDecayConstant(decay uint) *Pacer {
 	p.mu.Lock()
 	defer p.mu.Unlock()
@@ -124,6 +126,19 @@ func (p *Pacer) SetDecayConstant(decay uint) *Pacer {
 	return p
 }

+// SetAttackConstant sets the attack constant for the pacer
+//
+// This is the speed the time grows from the minimum after errors have
+// occurred.
+//
+// bigger for slower attack, 1 is double, 0 is go straight to maximum
+func (p *Pacer) SetAttackConstant(attack uint) *Pacer {
+	p.mu.Lock()
+	defer p.mu.Unlock()
+	p.attackConstant = attack
+	return p
+}
+
 // SetRetries sets the max number of tries for Call
 func (p *Pacer) SetRetries(retries int) *Pacer {
 	p.mu.Lock()
@@ -185,7 +200,11 @@ func (p *Pacer) beginCall() {
 func (p *Pacer) defaultPacer(retry bool) {
 	oldSleepTime := p.sleepTime
 	if retry {
-		p.sleepTime *= 2
+		if p.attackConstant == 0 {
+			p.sleepTime = p.maxSleep
+		} else {
+			p.sleepTime = (p.sleepTime << p.attackConstant) / ((1 << p.attackConstant) - 1)
+		}
 		if p.sleepTime > p.maxSleep {
 			p.sleepTime = p.maxSleep
 		}
--- a/pacer/pacer_test.go
+++ b/pacer/pacer_test.go
@@ -27,6 +27,9 @@ func TestNew(t *testing.T) {
 	if p.decayConstant != 2 {
 		t.Errorf("decayConstant")
 	}
+	if p.attackConstant != 1 {
+		t.Errorf("attackConstant")
+	}
 	if cap(p.pacer) != 1 {
 		t.Errorf("pacer 1")
 	}
@@ -85,6 +88,58 @@ func TestSetDecayConstant(t *testing.T) {
 	}
 }

+func TestDecay(t *testing.T) {
+	p := New().SetMinSleep(time.Microsecond).SetPacer(DefaultPacer).SetMaxSleep(time.Second)
+	for _, test := range []struct {
+		in             time.Duration
+		attackConstant uint
+		want           time.Duration
+	}{
+		{8 * time.Millisecond, 1, 4 * time.Millisecond},
+		{1 * time.Millisecond, 0, time.Microsecond},
+		{1 * time.Millisecond, 2, (3 * time.Millisecond) / 4},
+		{1 * time.Millisecond, 3, (7 * time.Millisecond) / 8},
+	} {
+		p.sleepTime = test.in
+		p.SetDecayConstant(test.attackConstant)
+		p.defaultPacer(false)
+		got := p.sleepTime
+		if got != test.want {
+			t.Errorf("bad sleep want %v got %v", test.want, got)
+		}
+	}
+}
+
+func TestSetAttackConstant(t *testing.T) {
+	p := New().SetAttackConstant(19)
+	if p.attackConstant != 19 {
+		t.Errorf("didn't set")
+	}
+}
+
+func TestAttack(t *testing.T) {
+	p := New().SetMinSleep(time.Microsecond).SetPacer(DefaultPacer).SetMaxSleep(time.Second)
+	for _, test := range []struct {
+		in             time.Duration
+		attackConstant uint
+		want           time.Duration
+	}{
+		{1 * time.Millisecond, 1, 2 * time.Millisecond},
+		{1 * time.Millisecond, 0, time.Second},
+		{1 * time.Millisecond, 2, (4 * time.Millisecond) / 3},
+		{1 * time.Millisecond, 3, (8 * time.Millisecond) / 7},
+	} {
+		p.sleepTime = test.in
+		p.SetAttackConstant(test.attackConstant)
+		p.defaultPacer(true)
+		got := p.sleepTime
+		if got != test.want {
+			t.Errorf("bad sleep want %v got %v", test.want, got)
+		}
+	}
+
+}
+
 func TestSetRetries(t *testing.T) {
 	p := New().SetRetries(18)
 	if p.retries != 18 {
--- a/rclone.1
+++ b/rclone.1
@@ -1,5 +1,5 @@
 .\"t
-.TH "rclone" "1" "Mar 01, 2016" "User Manual" ""
+.TH "rclone" "1" "Apr 18, 2016" "User Manual" ""
 .SH Rclone
 .PP
 [IMAGE: Logo (http://rclone.org/img/rclone-120x120.png)] (http://rclone.org/)
@@ -63,7 +63,7 @@ Rclone is a Go program and comes as a single binary file.
 .PP
 Download (http://rclone.org/downloads/) the relevant binary.
 .PP
-Or alternatively if you have Go installed use
+Or alternatively if you have Go 1.5+ installed use
 .IP
 .nf
 \f[C]
@@ -73,11 +73,11 @@ go\ get\ github.com/ncw/rclone
 .PP
 and this will build the binary in \f[C]$GOPATH/bin\f[].
 If you have built rclone before then you will want to update its
-dependencies first with this (remove \f[C]\-f\f[] if using go < 1.4)
+dependencies first with this
 .IP
 .nf
 \f[C]
-go\ get\ \-u\ \-v\ \-f\ github.com/ncw/rclone/...
+go\ get\ \-u\ \-v\ github.com/ncw/rclone/...
 \f[]
 .fi
 .PP
@@ -320,20 +320,61 @@ Checks the files in the source and destination match.
 It compares sizes and MD5SUMs and prints a report of files which
 don\[aq]t match.
 It doesn\[aq]t alter the source or destination.
+.PP
+\f[C]\-\-size\-only\f[] may be used to only compare the sizes, not the
+MD5SUMs.
 .SS rclone dedupe remote:path
 .PP
-Interactively find duplicate files and offer to delete all but one or
-rename them to be different.
+By default \f[C]dedup\f[] interactively finds duplicate files and offers
+to delete all but one or rename them to be different.
 Only useful with Google Drive which can have duplicate file names.
+.PP
+The \f[C]dedupe\f[] command will delete all but one of any identical
+(same md5sum) files it finds without confirmation.
+This means that for most duplicated files the \f[C]dedupe\f[] command
+will not be interactive.
+You can use \f[C]\-\-dry\-run\f[] to see what would happen without doing
+anything.
+.PP
+Here is an example run.
+.PP
+Before \- with duplicates
+.IP
+.nf
+\f[C]
+$\ rclone\ lsl\ drive:dupes
+\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
+\ \ 6048320\ 2016\-03\-05\ 16:23:11.775000000\ one.txt
+\ \ \ 564374\ 2016\-03\-05\ 16:23:06.731000000\ one.txt
+\ \ 6048320\ 2016\-03\-05\ 16:18:26.092000000\ one.txt
+\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two.txt
+\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two.txt
+\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two.txt
+\f[]
+.fi
+.PP
+Now the \f[C]dedupe\f[] session
 .IP
 .nf
 \f[C]
 $\ rclone\ dedupe\ drive:dupes
-2016/01/31\ 14:13:11\ Google\ drive\ root\ \[aq]dupes\[aq]:\ Looking\ for\ duplicates
-two.txt:\ Found\ 3\ duplicates
-\ \ 1:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-01\-31\ 14:07:22.159000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
-\ \ 2:\ \ \ \ \ \ 1744073\ bytes,\ 2016\-01\-31\ 14:07:12.490000000,\ md5sum\ 851957f7fb6f0bc4ce76be966d336802
-\ \ 3:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-01\-31\ 14:07:02.111000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
+2016/03/05\ 16:24:37\ Google\ drive\ root\ \[aq]dupes\[aq]:\ Looking\ for\ duplicates\ using\ interactive\ mode.
+one.txt:\ Found\ 4\ duplicates\ \-\ deleting\ identical\ copies
+one.txt:\ Deleting\ 2/3\ identical\ duplicates\ (md5sum\ "1eedaa9fe86fd4b8632e2ac549403b36")
+one.txt:\ 2\ duplicates\ remain
+\ \ 1:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:23:16.798000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
+\ \ 2:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:23:06.731000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
+s)\ Skip\ and\ do\ nothing
+k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
+r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
+s/k/r>\ k
+Enter\ the\ number\ of\ the\ file\ to\ keep>\ 1
+one.txt:\ Deleted\ 1\ extra\ copies
+two.txt:\ Found\ 3\ duplicates\ \-\ deleting\ identical\ copies
+two.txt:\ 3\ duplicates\ remain
+\ \ 1:\ \ \ \ \ \ \ 564374\ bytes,\ 2016\-03\-05\ 16:22:52.118000000,\ md5sum\ 7594e7dc9fc28f727c42ee3e0749de81
+\ \ 2:\ \ \ \ \ \ 6048320\ bytes,\ 2016\-03\-05\ 16:22:46.185000000,\ md5sum\ 1eedaa9fe86fd4b8632e2ac549403b36
+\ \ 3:\ \ \ \ \ \ 1744073\ bytes,\ 2016\-03\-05\ 16:22:38.104000000,\ md5sum\ 851957f7fb6f0bc4ce76be966d336802
 s)\ Skip\ and\ do\ nothing
 k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
 r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
@@ -341,15 +382,6 @@ s/k/r>\ r
 two\-1.txt:\ renamed\ from:\ two.txt
 two\-2.txt:\ renamed\ from:\ two.txt
 two\-3.txt:\ renamed\ from:\ two.txt
-one.txt:\ Found\ 2\ duplicates
-\ \ 1:\ \ \ \ \ \ \ \ \ 6579\ bytes,\ 2016\-01\-31\ 14:05:01.235000000,\ md5sum\ 2b76c776249409d925ae7ccd49aea59b
-\ \ 2:\ \ \ \ \ \ \ \ \ 6579\ bytes,\ 2016\-01\-31\ 12:50:30.318000000,\ md5sum\ 2b76c776249409d925ae7ccd49aea59b
-s)\ Skip\ and\ do\ nothing
-k)\ Keep\ just\ one\ (choose\ which\ in\ next\ step)
-r)\ Rename\ all\ to\ be\ different\ (by\ changing\ file.jpg\ to\ file\-1.jpg)
-s/k/r>\ k
-Enter\ the\ number\ of\ the\ file\ to\ keep>\ 2
-one.txt:\ Deleted\ 1\ extra\ copies
 \f[]
 .fi
 .PP
@@ -358,10 +390,39 @@ The result being
 .nf
 \f[C]
 $\ rclone\ lsl\ drive:dupes
-\ \ \ 564374\ 2016\-01\-31\ 14:07:22.159000000\ two\-1.txt
-\ \ 1744073\ 2016\-01\-31\ 14:07:12.490000000\ two\-2.txt
-\ \ 6048320\ 2016\-01\-31\ 14:07:02.111000000\ two\-3.txt
-\ \ \ \ \ 6579\ 2016\-01\-31\ 12:50:30.318000000\ one.txt
+\ \ 6048320\ 2016\-03\-05\ 16:23:16.798000000\ one.txt
+\ \ \ 564374\ 2016\-03\-05\ 16:22:52.118000000\ two\-1.txt
+\ \ 6048320\ 2016\-03\-05\ 16:22:46.185000000\ two\-2.txt
+\ \ 1744073\ 2016\-03\-05\ 16:22:38.104000000\ two\-3.txt
+\f[]
+.fi
+.PP
+Dedupe can be run non interactively using the \f[C]\-\-dedupe\-mode\f[]
+flag.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive as above.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
+anything left.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
+the first one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
+the newest one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
+the oldest one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
+renames the rest to be different.
+.PP
+For example to rename all the identically named photos in your Google
+Photos directory, do
+.IP
+.nf
+\f[C]
+rclone\ dedupe\ \-\-dedupe\-mode\ rename\ "drive:Google\ Photos"
 \f[]
 .fi
 .SS rclone config
@@ -478,6 +539,14 @@ seconds, \f[C]10m\f[] for 10 minutes, or \f[C]3h30m\f[].
 The connection timeout is the amount of time rclone will wait for a
 connection to go through to a remote object storage system.
 It is \f[C]1m\f[] by default.
+.SS \-\-dedupe\-mode MODE
+.PP
+Mode to run dedupe command in.
+One of \f[C]interactive\f[], \f[C]skip\f[], \f[C]first\f[],
+\f[C]newest\f[], \f[C]oldest\f[], \f[C]rename\f[].
+The default is \f[C]interactive\f[].
+See the dedupe command for more information as to what these options
+mean.
 .SS \-n, \-\-dry\-run
 .PP
 Do a trial run with no permanent changes.
@@ -493,6 +562,14 @@ While this isn\[aq]t a generally recommended option, it can be useful in
 cases where your files change due to encryption.
 However, it cannot correct partial transfers in case a transfer was
 interrupted.
+.SS \-I, \-\-ignore\-times
+.PP
+Using this option will cause rclone to unconditionally upload all files
+regardless of the state of files on the destination.
+.PP
+Normally rclone would skip any files that have the same modification
+time and are the same size (or have the same checksum if using
+\f[C]\-\-checksum\f[]).
 .SS \-\-log\-file=FILE
 .PP
 Log all of rclone\[aq]s output to FILE.
@@ -919,6 +996,9 @@ file.jpg\ \ \-\ matches\ "file.jpg"
 \f[]
 .fi
 .PP
+\f[B]Important\f[] Note that you must use \f[C]/\f[] in patterns and not
+\f[C]\\\f[] even if running on Windows.
+.PP
 A \f[C]*\f[] matches anything but not a \f[C]/\f[].
 .IP
 .nf
@@ -1376,7 +1456,7 @@ Backblaze B2
 T}@T{
 SHA1
 T}@T{
-Partial
+Yes
 T}@T{
 No
 T}@T{
@@ -1427,9 +1507,6 @@ can be checked with the \f[C]\-\-checksum\f[] flag.
 .PP
 All cloud storage systems support some kind of date on the object and
 these will be set when transferring from the cloud storage system.
-.PP
-Backblaze B2 preserves file modification times on files uploaded and
-downloaded, but doesn\[aq]t use them to decide which objects to sync.
 .SS Case Insensitive
 .PP
 If a cloud storage systems is case sensitive then it is possible to have
@@ -1461,7 +1538,8 @@ systems.
 If a cloud storage system allows duplicate files then it can have two
 objects with the same name.
 .PP
-This confuses rclone greatly when syncing.
+This confuses rclone greatly when syncing \- use the
+\f[C]rclone\ dedupe\f[] command to rename or remove duplicates.
 .SS Google Drive
 .PP
 Paths are specified as \f[C]drive:path\f[]
@@ -1603,16 +1681,19 @@ Here are the command line options specific to this cloud storage system.
 .PP
 Upload chunk size.
 Must a power of 2 >= 256k.
-Default value is 256kB.
+Default value is 8 MB.
+.PP
+Making this larger will improve performance, but note that each chunk is
+buffered in memory one per transfer.
+.PP
+Reducing this will reduce memory usage but decrease performance.
 .SS \-\-drive\-full\-list
 .PP
-Use a full listing for directory list.
-More data but usually quicker.
-On by default, disable with \f[C]\-\-full\-drive\-list=false\f[].
+No longer does anything \- kept for backwards compatibility.
 .SS \-\-drive\-upload\-cutoff=SIZE
 .PP
 File size cutoff for switching to chunked upload.
-Default is 256kB.
+Default is 8 MB.
 .SS \-\-drive\-use\-trash
 .PP
 Send files to the trash instead of deleting permanently.
@@ -2249,6 +2330,11 @@ accurate to 1 ns.
 .PP
 This is a defacto standard (used in the official python\-swiftclient
 amongst others) for storing the modification time for an object.
+.SS Limitations
+.PP
+The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
+MD5SUM for these.
 .SS Dropbox
 .PP
 Paths are specified as \f[C]remote:path\f[]
@@ -2386,6 +2472,12 @@ document (https://www.dropbox.com/en/help/145).
 Rclone will issue an error message
 \f[C]File\ name\ disallowed\ \-\ not\ uploading\f[] if it attempt to
 upload one of those file names, but the sync won\[aq]t fail.
+.PP
+If you have more than 10,000 files in a directory then
+\f[C]rclone\ purge\ dropbox:dir\f[] will return the error
+\f[C]Failed\ to\ purge:\ There\ are\ too\ many\ files\ involved\ in\ this\ operation\f[].
+As a work\-around do an \f[C]rclone\ delete\ dropbix:dir\f[] followed by
+an \f[C]rclone\ rmdir\ dropbox:dir\f[].
 .SS Google Cloud Storage
 .PP
 Paths are specified as \f[C]remote:bucket\f[] (or \f[C]remote:\f[] for
@@ -3001,8 +3093,12 @@ Note that Hubic wraps the Swift backend, so most of the properties of
 are the same.
 .SS Limitations
 .PP
-Code to refresh the OpenStack token isn\[aq]t done yet which may cause
-problems with very long transfers.
+This uses the normal OpenStack Swift mechanism to refresh the Swift API
+credentials and ignores the expires field returned by the Hubic API.
+.PP
+The Swift API doesn\[aq]t return a correct MD5SUM for segmented files
+(Dynamic or Static Large Objects) so rclone won\[aq]t check or use the
+MD5SUM for these.
 .SS Backblaze B2
 .PP
 B2 is Backblaze\[aq]s cloud storage
@@ -3118,10 +3214,10 @@ The modified time is stored as metadata on the object as
 1970\-01\-01 in the Backblaze standard.
 Other tools should be able to use this as a modified time.
 .PP
-Modified times are set on upload, read on download and shown in
-listings.
-They are not used in syncing as unfortunately B2 doesn\[aq]t have an API
-method to set them independently of doing an upload.
+Modified times are used in syncing and are fully supported except in the
+case of updating a modification time on an existing object.
+In this case the object will be uploaded again as B2 doesn\[aq]t have an
+API method to set the modification time independent of doing an upload.
 .SS SHA1 checksums
 .PP
 The SHA1 checksums of the files are checked on upload and download and
@@ -3140,14 +3236,22 @@ via rclone yet.
 Rclone doesn\[aq]t provide any way of managing old versions (downloading
 them or deleting them) at the moment.
 When you \f[C]purge\f[] a bucket, all the old versions will be deleted.
+.SS Transfers
+.PP
+Backblaze recommends that you do lots of transfers simultaneously for
+maximum speed.
+In tests from my SSD equiped laptop the optimum setting is about
+\f[C]\-\-transfers\ 32\f[] though higher numbers may be used for a
+slight speed improvement.
+The optimum number for you may vary depending on your hardware, how big
+the files are, how much you want to load your computer, etc.
+The default of \f[C]\-\-transfers\ 4\f[] is definitely too low for
+Backblaze B2 though.
 .SS API
 .PP
 Here are some notes I made on the backblaze
 API (https://gist.github.com/ncw/166dabf352b399f1cc1c) while integrating
 it with rclone which detail the changes I\[aq]d like to see.
-With a couple of small tweaks Backblaze could enable rclone to not make
-a temporary copy of files when doing cloud to cloud copies and fully
-support modification times.
 .SS Yandex Disk
 .PP
 Yandex Disk (https://disk.yandex.com) is a cloud storage solution
@@ -3365,6 +3469,154 @@ Of course this will cause problems if the absolute path length of a file
 exceeds 258 characters on z, so only use this option if you have to.
 .SS Changelog
 .IP \[bu] 2
+v1.29 \- 2016\-04\-18
+.RS 2
+.IP \[bu] 2
+New Features
+.IP \[bu] 2
+Implement \f[C]\-I,\ \-\-ignore\-times\f[] for unconditional upload
+.IP \[bu] 2
+Improve \f[C]dedupe\f[]command
+.RS 2
+.IP \[bu] 2
+Now removes identical copies without asking
+.IP \[bu] 2
+Now obeys \f[C]\-\-dry\-run\f[]
+.IP \[bu] 2
+Implement \f[C]\-\-dedupe\-mode\f[] for non interactive running
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ interactive\f[] \- interactive the default.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ skip\f[] \- removes identical files then skips
+anything left.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ first\f[] \- removes identical files then keeps
+the first one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ newest\f[] \- removes identical files then keeps
+the newest one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ oldest\f[] \- removes identical files then keeps
+the oldest one.
+.IP \[bu] 2
+\f[C]\-\-dedupe\-mode\ rename\f[] \- removes identical files then
+renames the rest to be different.
+.RE
+.IP \[bu] 2
+Bug fixes
+.IP \[bu] 2
+Make rclone check obey the \f[C]\-\-size\-only\f[] flag.
+.IP \[bu] 2
+Use "application/octet\-stream" if discovered mime type is invalid.
+.IP \[bu] 2
+Fix missing "quit" option when there are no remotes.
+.IP \[bu] 2
+Google Drive
+.IP \[bu] 2
+Increase default chunk size to 8 MB \- increases upload speed of big
+files
+.IP \[bu] 2
+Speed up directory listings and make more reliable
+.IP \[bu] 2
+Add missing retries for Move and DirMove \- increases reliability
+.IP \[bu] 2
+Preserve mime type on file update
+.IP \[bu] 2
+Backblaze B2
+.IP \[bu] 2
+Enable mod time syncing
+.RS 2
+.IP \[bu] 2
+This means that B2 will now check modification times
+.IP \[bu] 2
+It will upload new files to update the modification times
+.IP \[bu] 2
+(there isn\[aq]t an API to just set the mod time.)
+.IP \[bu] 2
+If you want the old behaviour use \f[C]\-\-size\-only\f[].
+.RE
+.IP \[bu] 2
+Update API to new version
+.IP \[bu] 2
+Fix parsing of mod time when not in metadata
+.IP \[bu] 2
+Swift/Hubic
+.IP \[bu] 2
+Don\[aq]t return an MD5SUM for static large objects
+.IP \[bu] 2
+S3
+.IP \[bu] 2
+Fix uploading files bigger than 50GB
+.RE
+.IP \[bu] 2
+v1.28 \- 2016\-03\-01
+.RS 2
+.IP \[bu] 2
+New Features
+.IP \[bu] 2
+Configuration file encryption \- thanks Klaus Post
+.IP \[bu] 2
+Improve \f[C]rclone\ config\f[] adding more help and making it easier to
+understand
+.IP \[bu] 2
+Implement \f[C]\-u\f[]/\f[C]\-\-update\f[] so creation times can be used
+on all remotes
+.IP \[bu] 2
+Implement \f[C]\-\-low\-level\-retries\f[] flag
+.IP \[bu] 2
+Optionally disable gzip compression on downloads with
+\f[C]\-\-no\-gzip\-encoding\f[]
+.IP \[bu] 2
+Bug fixes
+.IP \[bu] 2
+Don\[aq]t make directories if \f[C]\-\-dry\-run\f[] set
+.IP \[bu] 2
+Fix and document the \f[C]move\f[] command
+.IP \[bu] 2
+Fix redirecting stderr on unix\-like OSes when using
+\f[C]\-\-log\-file\f[]
+.IP \[bu] 2
+Fix \f[C]delete\f[] command to wait until all finished \- fixes missing
+deletes.
+.IP \[bu] 2
+Backblaze B2
+.IP \[bu] 2
+Use one upload URL per go routine fixes
+\f[C]more\ than\ one\ upload\ using\ auth\ token\f[]
+.IP \[bu] 2
+Add pacing, retries and reauthentication \- fixes token expiry problems
+.IP \[bu] 2
+Upload without using a temporary file from local (and remotes which
+support SHA1)
+.IP \[bu] 2
+Fix reading metadata for all files when it shouldn\[aq]t have been
+.IP \[bu] 2
+Drive
+.IP \[bu] 2
+Fix listing drive documents at root
+.IP \[bu] 2
+Disable copy and move for Google docs
+.IP \[bu] 2
+Swift
+.IP \[bu] 2
+Fix uploading of chunked files with non ASCII characters
+.IP \[bu] 2
+Allow setting of \f[C]storage_url\f[] in the config \- thanks Xavier
+Lucas
+.IP \[bu] 2
+S3
+.IP \[bu] 2
+Allow IAM role and credentials from environment variables \- thanks
+Brian Stengaard
+.IP \[bu] 2
+Allow low privilege users to use S3 (check if directory exists during
+Mkdir) \- thanks Jakub Gedeon
+.IP \[bu] 2
+Amazon Cloud Drive
+.IP \[bu] 2
+Retry on more things to make directory listings more reliable
+.RE
+.IP \[bu] 2
 v1.27 \- 2016\-01\-31
 .RS 2
 .IP \[bu] 2
@@ -4162,6 +4414,13 @@ supported by the go runtime, ie earlier than version 2.6.23.
 .PP
 See the system requirements section in the go install
 docs (https://golang.org/doc/install) for full details.
+.SS All my uploaded docx/xlsx/pptx files appear as archive/zip
+.PP
+This is caused by uploading these files from a Windows computer which
+hasn\[aq]t got the Microsoft Office suite installed.
+The easiest way to fix is to install the Word viewer and the Microsoft
+Office Compatibility Pack for Word, Excel, and PowerPoint 2007 and later
+versions\[aq] file formats
 .SS License
 .PP
 This is free software under the terms of MIT the license (check the
--- a/rclone.go
+++ b/rclone.go
@@ -248,7 +248,7 @@ var Commands = []Command{
        but one or rename them to be different. Only useful with
        Google Drive which can have duplicate file names.`,
 		Run: func(fdst, fsrc fs.Fs) error {
-			return fs.Deduplicate(fdst)
+			return fs.Deduplicate(fdst, fs.Config.DedupeMode)
 		},
 		MinArgs: 1,
 		MaxArgs: 1,
--- a/s3/s3.go
+++ b/s3/s3.go
@@ -702,12 +702,10 @@ func (o *Object) ModTime() time.Time {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	err := o.readMetaData()
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to read metadata: %s", err)
-		return
+		return err
 	}
 	o.meta[metaMtime] = aws.String(swift.TimeToFloatString(modTime))

@@ -728,10 +726,7 @@ func (o *Object) SetModTime(modTime time.Time) {
 		MetadataDirective: &directive,
 	}
 	_, err = o.fs.c.CopyObject(&req)
-	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to update remote mtime: %s", err)
-	}
+	return err
 }

 // Storable raturns a boolean indicating if this object is storable
@@ -761,6 +756,14 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 		u.Concurrency = 2
 		u.LeavePartsOnError = false
 		u.S3 = o.fs.c
+		u.PartSize = s3manager.MinUploadPartSize
+		size := src.Size()
+
+		// Adjust PartSize until the number of parts is small enough.
+		if size/u.PartSize >= s3manager.MaxUploadParts {
+			// Calculate partition size rounded up to the nearest MB
+			u.PartSize = (((size / s3manager.MaxUploadParts) >> 20) + 1) << 20
+		}
 	})

 	// Set the mtime in the meta data
--- a/swift/swift.go
+++ b/swift/swift.go
@@ -79,11 +79,11 @@ func init() {

 // Fs represents a remote swift server
 type Fs struct {
-	name              string           // name of this remote
-	c                 swift.Connection // the connection to the swift server
-	container         string           // the container we are working on
-	segmentsContainer string           // container to store the segments (if any) in
-	root              string           // the path we are working on if any
+	name              string            // name of this remote
+	c                 *swift.Connection // the connection to the swift server
+	container         string            // the container we are working on
+	segmentsContainer string            // container to store the segments (if any) in
+	root              string            // the path we are working on if any
 }

 // Object describes a swift object
@@ -175,7 +175,7 @@ func NewFsWithConnection(name, root string, c *swift.Connection) (fs.Fs, error)
 	}
 	f := &Fs{
 		name:              name,
-		c:                 *c,
+		c:                 c,
 		container:         container,
 		segmentsContainer: container + "_segments",
 		root:              directory,
@@ -223,7 +223,7 @@ func (f *Fs) newFsObjectWithInfo(remote string, info *swift.Object) fs.Object {
 		fs:     f,
 		remote: remote,
 	}
-	// Note that due to a quirk of swift, manifest files are
+	// Note that due to a quirk of swift, dynamic large objects are
 	// returned as 0 bytes in the listing.  Correct this here by
 	// making sure we read the full metadata for all 0 byte files.
 	if info != nil && info.Bytes == 0 {
@@ -320,7 +320,7 @@ func (f *Fs) listFiles(ignoreStorable bool) fs.ObjectsChan {
 			defer close(out)
 			f.list(false, func(remote string, object *swift.Object) error {
 				if o := f.newFsObjectWithInfo(remote, object); o != nil {
-					// Storable does a full metadata read on 0 size objects which might be manifest files
+					// Storable does a full metadata read on 0 size objects which might be dynamic large objects
 					storable := o.Storable()
 					if storable || ignoreStorable {
 						out <- o
@@ -480,19 +480,24 @@ func (o *Object) Hash(t fs.HashType) (string, error) {
 	if t != fs.HashMD5 {
 		return "", fs.ErrHashUnsupported
 	}
-	isManifest, err := o.isManifestFile()
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
 	if err != nil {
 		return "", err
 	}
-	if isManifest {
-		fs.Debug(o, "Returning empty Md5sum for swift manifest file")
+	isStaticLargeObject, err := o.isStaticLargeObject()
+	if err != nil {
+		return "", err
+	}
+	if isDynamicLargeObject || isStaticLargeObject {
+		fs.Debug(o, "Returning empty Md5sum for swift large object")
 		return "", nil
 	}
 	return strings.ToLower(o.info.Hash), nil
 }

-// isManifestFile checks for manifest header
-func (o *Object) isManifestFile() (bool, error) {
+// hasHeader checks for the header passed in returning false if the
+// object isn't found.
+func (o *Object) hasHeader(header string) (bool, error) {
 	err := o.readMetaData()
 	if err != nil {
 		if err == swift.ObjectNotFound {
@@ -500,8 +505,18 @@ func (o *Object) isManifestFile() (bool, error) {
 		}
 		return false, err
 	}
-	_, isManifestFile := (*o.headers)["X-Object-Manifest"]
-	return isManifestFile, nil
+	_, isDynamicLargeObject := (*o.headers)[header]
+	return isDynamicLargeObject, nil
+}
+
+// isDynamicLargeObject checks for X-Object-Manifest header
+func (o *Object) isDynamicLargeObject() (bool, error) {
+	return o.hasHeader("X-Object-Manifest")
+}
+
+// isStaticLargeObjectFile checks for the X-Static-Large-Object header
+func (o *Object) isStaticLargeObject() (bool, error) {
+	return o.hasHeader("X-Static-Large-Object")
 }

 // Size returns the size of an object in bytes
@@ -545,12 +560,10 @@ func (o *Object) ModTime() time.Time {
 }

 // SetModTime sets the modification time of the local fs object
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	err := o.readMetaData()
 	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to read metadata: %s", err)
-		return
+		return err
 	}
 	meta := o.headers.ObjectMetadata()
 	meta.SetModTime(modTime)
@@ -564,11 +577,7 @@ func (o *Object) SetModTime(modTime time.Time) {
 			newHeaders[k] = v
 		}
 	}
-	err = o.fs.c.ObjectUpdate(o.fs.container, o.fs.root+o.remote, newHeaders)
-	if err != nil {
-		fs.Stats.Error()
-		fs.ErrorLog(o, "Failed to update remote mtime: %s", err)
-	}
+	return o.fs.c.ObjectUpdate(o.fs.container, o.fs.root+o.remote, newHeaders)
 }

 // Storable returns if this object is storable
@@ -677,8 +686,8 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 	size := src.Size()
 	modTime := src.ModTime()

-	// Note whether this has a manifest before starting
-	isManifest, err := o.isManifestFile()
+	// Note whether this is a dynamic large object before starting
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
 	if err != nil {
 		return err
 	}
@@ -701,8 +710,8 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 		}
 	}

-	// If file was a manifest then remove old/all segments
-	if isManifest {
+	// If file was a dynamic large object then remove old/all segments
+	if isDynamicLargeObject {
 		err = o.removeSegments(uniquePrefix)
 		if err != nil {
 			fs.Log(o, "Failed to remove old segments - carrying on with upload: %v", err)
@@ -716,7 +725,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {

 // Remove an object
 func (o *Object) Remove() error {
-	isManifestFile, err := o.isManifestFile()
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
 	if err != nil {
 		return err
 	}
@@ -726,7 +735,7 @@ func (o *Object) Remove() error {
 		return err
 	}
 	// ...then segments if required
-	if isManifestFile {
+	if isDynamicLargeObject {
 		err = o.removeSegments("")
 		if err != nil {
 			return err
--- a/versioncheck.go
+++ b/versioncheck.go
@@ -0,0 +1,6 @@
+//+build !go1.5
+
+package main
+
+// Upgrade to Go version 1.5 to compile rclone.
+func init() { Go_version_1_5_required_for_compilation() }
--- a/yandex/yandex.go
+++ b/yandex/yandex.go
@@ -452,14 +452,10 @@ func (o *Object) Remove() error {
 // SetModTime sets the modification time of the local fs object
 //
 // Commits the datastore
-func (o *Object) SetModTime(modTime time.Time) {
+func (o *Object) SetModTime(modTime time.Time) error {
 	remote := o.remotePath()
 	//set custom_property 'rclone_modified' of object to modTime
-	err := o.fs.yd.SetCustomProperty(remote, "rclone_modified", modTime.Format(time.RFC3339Nano))
-	if err != nil {
-		return
-	}
-	return
+	return o.fs.yd.SetCustomProperty(remote, "rclone_modified", modTime.Format(time.RFC3339Nano))
 }

 // Storable returns whether this object is storable
@@ -496,7 +492,7 @@ func (o *Object) Update(in io.Reader, src fs.ObjectInfo) error {
 		o.modTime = modTime
 		o.md5sum = "" // according to unit tests after put the md5 is empty.
 		//and set modTime of uploaded file
-		o.SetModTime(modTime)
+		err = o.SetModTime(modTime)
 	}
 	return err
 }
Author	SHA1	Message	Date
Nick Craig-Wood	bb75d80d33	Fix frontmatter	2016-04-18 18:55:07 +01:00
Nick Craig-Wood	157d7d45f5	Version v1.29	2016-04-18 18:30:29 +01:00
Nick Craig-Wood	b5cba73cc3	Make test more reliable	2016-04-18 17:48:52 +01:00
Nick Craig-Wood	dd36264aad	Add FAQ All my uploaded docx/xlsx/pptx files appear as archive/zip Fixes #417	2016-04-12 21:41:24 +01:00
Nick Craig-Wood	ddb47758f3	drive: increase default chunk size to 8 MB and document - fixes #397	2016-04-12 21:33:55 +01:00
Nick Craig-Wood	9539bbf78a	Fix appveyor build after vet removal from tools repo	2016-04-07 20:07:00 +01:00
Nick Craig-Wood	0f8e7c3843	Make rclone check obey the --size-only flag - fixes #419	2016-04-07 15:01:45 +01:00
Nick Craig-Wood	b835330714	Use "application/octet-stream" if mime.TypeByExtension returns invalid type Fixes #424	2016-04-07 14:32:01 +01:00
Nick Craig-Wood	310db14ed6	Notes on `--transfers` and B2	2016-04-04 17:58:36 +01:00
Klaus Post	7f2e9d9a6b	Require go v1.5 for compilation Google cloud package requires go v1.5 to compile, so we need to require the same for rclone. Fixes #408	2016-04-04 17:34:39 +01:00
Nick Craig-Wood	6cc9c09610	drive: preserve mime type on file update - fixes #417	2016-04-04 16:58:42 +01:00
Nick Craig-Wood	93c60c34e1	b2: Fix incorrect value of Precision - should be 1ms not 1s	2016-03-24 15:23:27 +00:00
Klaus Post	02c11dd4a7	Don't de-reference swift connection The connection object contains a mutex, so it is good practice not to dereference it to a value. Reported by Go tip "go vet".	2016-03-23 17:09:05 +00:00
Klaus Post	40dc575aa4	Update Travis CI - Only use golint if version is > Go 1.4 - Add Go 1.6 and tip as test targets.	2016-03-23 17:07:26 +00:00
Klaus Post	f8101771c9	Disable keepalive to keep server from serving stale results. Fixes issue #402 Bonus fix: Fix "multiple header writes" warning when no code is received.	2016-03-23 16:57:56 +00:00
Klaus Post	8f4d6973fb	Fix missing "quit" option when there are no remotes.	2016-03-23 16:57:56 +00:00
Nick Craig-Wood	ced3a4bc19	Implement -I, --ignore-times for unconditional upload - fixes #311	2016-03-22 17:02:27 +00:00
Nick Craig-Wood	cb22583212	b2: Enable mod time syncing - fixes #348	2016-03-22 15:56:44 +00:00
Nick Craig-Wood	414b35ea56	Change the interface of SetModTime to return an error - #348	2016-03-22 15:56:44 +00:00
Nick Craig-Wood	f469905d07	dropbox: Note 10,000 files limitation on purge - fixes #374	2016-03-22 14:46:43 +00:00
Nick Craig-Wood	20f4b2c91d	b2: update API to new version - fixes #393 * Make reading mod time and SHA1 much more efficient * removes an HTTP transaction to increase speed * Reduce memory usage of the objects	2016-03-22 14:39:56 +00:00
Nick Craig-Wood	37543bd1d9	b2: Fix parsing of mod time when not in metadata This files this error `Failed to parse mod time string "": "src_last_modified_millis" not found in metadata`.	2016-03-22 10:26:37 +00:00
Nick Craig-Wood	0dc0052e93	Note that filters must use / not \ - #394	2016-03-19 17:40:54 +00:00
Nick Craig-Wood	bd27473762	swift: Don't return an MD5SUM for static large objects - #392 * rename isManifest to isDynamicLargeObject for clarity	2016-03-17 17:36:20 +00:00
Nick Craig-Wood	9dccf91da7	swift/hubic: document segmented object MD5SUM limitations - fixes #392	2016-03-16 17:39:44 +00:00
Nick Craig-Wood	a1323eb204	s3: Fix uploading files bigger than 50GB - fixes #386	2016-03-10 16:48:55 +00:00
Klaus Post	e57c4406f3	Add mutex to "warned" map. Fixes #385	2016-03-10 15:51:56 +01:00
Nick Craig-Wood	fdd4b4ee22	drive: Add missing retries for Move and DirMove	2016-03-06 18:15:01 +00:00
Nick Craig-Wood	8ef551bf9c	Make dedupe remove identical copies without asking and add non interactive mode - fixes #338 * Now removes identical copies without asking * Now obeys `--dry-run` * Implement `--dedupe-mode` for non interactive running * `--dedupe-mode interactive` - interactive the default. * `--dedupe-mode skip` - removes identical files then skips anything left. * `--dedupe-mode first` - removes identical files then keeps the first one. * `--dedupe-mode newest` - removes identical files then keeps the newest one. * `--dedupe-mode oldest` - removes identical files then keeps the oldest one. * `--dedupe-mode rename` - removes identical files then renames the rest to be different. * Add tests which will only run on Google Drive.	2016-03-06 18:15:01 +00:00
Nick Craig-Wood	2119fb4314	drive: tweak pacer to speed up directory listings and make more reliable	2016-03-06 18:15:01 +00:00
Nick Craig-Wood	0166544319	Add Attack constant to pacer	2016-03-05 20:29:05 +00:00
Nick Craig-Wood	874a64e5f6	A script to make a directory heirarchy for testing	2016-03-05 20:26:15 +00:00
Nick Craig-Wood	e0c03a11ab	Commit missing docs changes and adjust RELEASE.md to make sure it doesn't happen again	2016-03-01 17:42:27 +00:00