fs: dump config for an Fs before creation - FIXME TESTING DO NOT MERGE

See: https://forum.rclone.org/t/authenticating-to-box-using-connection-strings-and-config-file-fails/23333/7

.github/ISSUE_TEMPLATE/Bug.md

@@ -5,19 +5,31 @@ about: Report a problem with rclone
 
 <!--
 
-Welcome :-) We understand you are having a problem with rclone; we want to help you with that!
+We understand you are having a problem with rclone; we want to help you with that!
 
-If you've just got a question or aren't sure if you've found a bug then please use the rclone forum:
+**STOP and READ**
+**YOUR POST WILL BE REMOVED IF IT IS LOW QUALITY**:
+Please show the effort you've put in to solving the problem and please be specific.
+People are volunteering their time to help! Low effort posts are not likely to get good answers!
+
+If you think you might have found a bug, try to replicate it with the latest beta (or stable).
+The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
+
+If you can still replicate it or just got a question then please use the rclone forum:
 
     https://forum.rclone.org/
 
-instead of filing an issue for a quick response.
+for a quick response instead of filing an issue on this repo.
 
-If you think you might have found a bug, please can you try to replicate it with the latest beta?
+If nothing else helps, then please fill in the info below which helps us help you.
 
-    https://beta.rclone.org/
+**DO NOT REDACT** any information except passwords/keys/personal info.
 
-If you can still replicate it with the latest beta, then please fill in the info below which makes our lives much easier.  A log with -vv will make our day :-)
+You should use 3 backticks to begin and end your paste to make it readable.
+
+Make sure to include a log obtained with '-vv'.
+
+You can also use '-vv --log-file bug.log' and a service such as https://pastebin.com or https://gist.github.com/
 
 Thank you
 
@@ -25,6 +37,11 @@ The Rclone Developers
 
 -->
 
+
+#### The associated forum post URL from `https://forum.rclone.org`
+
+
+
 #### What is the problem you are having with rclone?
 
 

.github/ISSUE_TEMPLATE/Feature.md

@@ -7,12 +7,16 @@ about: Suggest a new feature or enhancement for rclone
 
 Welcome :-)
 
-So you've got an idea to improve rclone? We love that! You'll be glad to hear we've incorporated hundreds of ideas from contributors already.
+So you've got an idea to improve rclone? We love that!
+You'll be glad to hear we've incorporated hundreds of ideas from contributors already.
 
-Here is a checklist of things to do:
+Probably the latest beta (or stable) release has your feature, so try to update your rclone.
+The update instructions are available at https://rclone.org/commands/rclone_selfupdate/
 
-  1. Please search the old issues first for your idea and +1 or comment on an existing issue if possible.
+If it still isn't there, here is a checklist of things to do:
-  2. Discuss on the forum first: https://forum.rclone.org/
+
+  1. Search the old issues for your idea and +1 or comment on an existing issue if possible.
+  2. Discuss on the forum: https://forum.rclone.org/
   3. Make a feature request issue (this is the right place!).
   4. Be prepared to get involved making the feature :-)
 
@@ -23,6 +27,10 @@ The Rclone Developers
 -->
 
 
+#### The associated forum post URL from `https://forum.rclone.org`
+
+
+
 #### What is your current rclone version (output from `rclone version`)?
 
 

.github/workflows/build.yml

@@ -221,8 +221,6 @@ jobs:
      steps:
        - name: Checkout
          uses: actions/checkout@v2
-         with:
-           fetch-depth: 0
 
        # Upgrade together with NDK version
        - name: Set up Go 1.14

MANUAL.html

@@ -17,7 +17,7 @@
 <header id="title-block-header">
 <h1 class="title">rclone(1) User Manual</h1>
 <p class="author">Nick Craig-Wood</p>
-<p class="date">Apr 26, 2021</p>
+<p class="date">Mar 31, 2021</p>
 </header>
 <h1 id="rclone-syncs-your-files-to-cloud-storage">Rclone syncs your files to cloud storage</h1>
 <p><img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" ></p>
@@ -1475,7 +1475,7 @@ rclone mount remote:path/to/files * --volname \\cloud\remote</code></pre>
 <p>The permissions on each entry will be set according to <a href="#options">options</a> <code>--dir-perms</code> and <code>--file-perms</code>, which takes a value in traditional <a href="https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation">numeric notation</a>, where the default corresponds to <code>--file-perms 0666 --dir-perms 0777</code>.</p>
 <p>Note that the mapping of permissions is not always trivial, and the result you see in Windows Explorer may not be exactly like you expected. For example, when setting a value that includes write access, this will be mapped to individual permissions "write attributes", "write data" and "append data", but not "write extended attributes". Windows will then show this as basic permission "Special" instead of "Write", because "Write" includes the "write extended attributes" permission.</p>
 <p>If you set POSIX permissions for only allowing access to the owner, using <code>--file-perms 0600 --dir-perms 0700</code>, the user group and the built-in "Everyone" group will still be given some special permissions, such as "read attributes" and "read permissions", in Windows. This is done for compatibility reasons, e.g. to allow users without additional permissions to be able to read basic metadata about files like in UNIX. One case that may arise is that other programs (incorrectly) interprets this as the file being accessible by everyone. For example an SSH client may warn about "unprotected private key file".</p>
-<p>WinFsp 2021 (version 1.9) introduces a new FUSE option "FileSecurity", that allows the complete specification of file security descriptors using <a href="https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format">SDDL</a>. With this you can work around issues such as the mentioned "unprotected private key file" by specifying <code>-o FileSecurity="D:P(A;;FA;;;OW)"</code>, for file all access (FA) to the owner (OW).</p>
+<p>WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity", that allows the complete specification of file security descriptors using <a href="https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format">SDDL</a>. With this you can work around issues such as the mentioned "unprotected private key file" by specifying <code>-o FileSecurity="D:P(A;;FA;;;OW)"</code>, for file all access (FA) to the owner (OW).</p>
 <h3 id="windows-caveats">Windows caveats</h3>
 <p>Note that drives created as Administrator are not visible by other accounts (including the account that was elevated as Administrator). So if you start a Windows drive from an Administrative Command Prompt and then try to access the same drive from Explorer (which does not run as Administrator), you will not be able to see the new drive.</p>
 <p>The easiest way around this is to start the drive from a normal command prompt. It is also possible to start a drive from the SYSTEM account (using <a href="https://github.com/billziss-gh/winfsp/wiki/WinFsp-Service-Architecture">the WinFsp.Launcher infrastructure</a>) which creates drives accessible for everyone on the system or alternatively using <a href="https://nssm.cc/usage">the nssm service manager</a>.</p>
@@ -1805,7 +1805,7 @@ ffmpeg - | rclone rcat remote:path/to/file</code></pre>
 <p>This command downloads the latest release of rclone and replaces the currently running binary. The download is verified with a hashsum and cryptographically signed signature.</p>
 <p>If used without flags (or with implied <code>--stable</code> flag), this command will install the latest stable release. However, some issues may be fixed (or features added) only in the latest beta release. In such cases you should run the command with the <code>--beta</code> flag, i.e. <code>rclone selfupdate --beta</code>. You can check in advance what version would be installed by adding the <code>--check</code> flag, then repeat the command without it when you are satisfied.</p>
 <p>Sometimes the rclone team may recommend you a concrete beta or stable rclone release to troubleshoot your issue or add a bleeding edge feature. The <code>--version VER</code> flag, if given, will update to the concrete version instead of the latest one. If you omit micro version from <code>VER</code> (for example <code>1.53</code>), the latest matching micro version will be used.</p>
-<p>Upon successful update rclone will print a message that contains a previous version number. You will need it if you later decide to revert your update for some reason. Then you'll have to note the previous version and run the following command: <code>rclone selfupdate [--beta] OLDVER</code>. If the old version contains only dots and digits (for example <code>v1.54.0</code>) then it's a stable release so you won't need the <code>--beta</code> flag. Beta releases have an additional information similar to <code>v1.54.0-beta.5111.06f1c0c61</code>. (if you are a developer and use a locally built rclone, the version number will end with <code>-DEV</code>, you will have to rebuild it as it obviously can't be distributed).</p>
+<p>Upon successful update rclone will print a message that contains a previous version number. You will need it if you later decide to revert your update for some reason. Then you'll have to note the previous version and run the following command: <code>rclone selfupdate [--beta] OLDVER</code>. If the old version contains only dots and digits (for example <code>v1.54.0</code>) then it's a stable release so you won't need the <code>--beta</code> flag. Beta releases have an additional information similar to <code>v1.54.0-beta.5111.06f1c0c61</code>. (if you are a developer and use a locally built rclone, the version number will end with <code>-DEV</code>, you will have to rebuild it as it obvisously can't be distributed).</p>
 <p>If you previously installed rclone via a package manager, the package may include local documentation or configure services. You may wish to update with the flag <code>--package deb</code> or <code>--package rpm</code> (whichever is correct for your OS) to update these too. This command with the default <code>--package zip</code> will update only the rclone executable so the local manual may become inaccurate after it.</p>
 <p>The <code>rclone mount</code> command (https://rclone.org/commands/rclone_mount/) may or may not support extended FUSE options depending on the build and OS. <code>selfupdate</code> will refuse to update if the capability would be discarded.</p>
 <p>Note: Windows forbids deletion of a currently running executable so this command will rename the old executable to 'rclone.old.exe' upon success.</p>
@@ -3378,7 +3378,7 @@ rclone sync -i /path/to/files remote:current-backup</code></pre>
 <p>Normally the config file is in your home directory as a file called <code>.config/rclone/rclone.conf</code> (or <code>.rclone.conf</code> if created with an older version). If <code>$XDG_CONFIG_HOME</code> is set it will be at <code>$XDG_CONFIG_HOME/rclone/rclone.conf</code>.</p>
 <p>If there is a file <code>rclone.conf</code> in the same directory as the rclone executable it will be preferred. This file must be created manually for Rclone to use it, it will never be created automatically.</p>
 <p>If you run <code>rclone config file</code> you will see where the default location is for you.</p>
-<p>Use this flag to override the config location, e.g. <code>rclone --config=".myconfig" config</code>.</p>
+<p>Use this flag to override the config location, e.g. <code>rclone --config=".myconfig" .config</code>.</p>
 <p>If the location is set to empty string <code>""</code> or the special value <code>/notfound</code>, or the os null device represented by value <code>NUL</code> on Windows and <code>/dev/null</code> on Unix systems, then rclone will keep the config file in memory only.</p>
 <p>The file format is basic <a href="https://en.wikipedia.org/wiki/INI_file#Format">INI</a>: Sections of text, led by a <code>[section]</code> header and followed by <code>key=value</code> entries on separate lines. In rclone each remote is represented by its own section, where the section name defines the name of the remote. Options are specified as the <code>key=value</code> entries, where the key is the option name without the <code>--backend-</code> prefix, in lowercase and with <code>_</code> instead of <code>-</code>. E.g. option <code>--mega-hard-delete</code> corresponds to key <code>hard_delete</code>. Only backend options can be specified. A special, and required, key <code>type</code> identifies the <a href="https://rclone.org/overview/">storage system</a>, where the value is the internal lowercase name as returned by command <code>rclone help backends</code>. Comments are indicated by <code>;</code> or <code>#</code> at the beginning of a line.</p>
 <p>Example:</p>
@@ -6806,7 +6806,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
       --use-json-log                         Use json log format.
       --use-mmap                             Use mmap allocator (see docs).
       --use-server-modtime                   Use server modified time instead of object metadata
-      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.55.1&quot;)
+      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default &quot;rclone/v1.55.0&quot;)
   -v, --verbose count                        Print lots more stuff (repeat for more)</code></pre>
 <h2 id="backend-flags">Backend Flags</h2>
 <p>These flags are available for every command. They control the backends and may be set in the config file.</p>
@@ -20355,65 +20355,6 @@ $ tree /tmp/b
 <li>"error": return an error based on option value</li>
 </ul>
 <h1 id="changelog">Changelog</h1>
-<h2 id="v1.55.1---2021-04-26">v1.55.1 - 2021-04-26</h2>
-<p><a href="https://github.com/rclone/rclone/compare/v1.55.0...v1.55.1">See commits</a></p>
-<ul>
-<li>Bug Fixes
-<ul>
-<li>selfupdate
-<ul>
-<li>Dont detect FUSE if build is static (Ivan Andreev)</li>
-<li>Add build tag noselfupdate (Ivan Andreev)</li>
-</ul></li>
-<li>sync: Fix incorrect error reported by graceful cutoff (Nick Craig-Wood)</li>
-<li>install.sh: fix macOS arm64 download (Nick Craig-Wood)</li>
-<li>build: Fix version numbers in android branch builds (Nick Craig-Wood)</li>
-<li>docs
-<ul>
-<li>Contributing.md: update setup instructions for go1.16 (Nick Gaya)</li>
-<li>WinFsp 2021 is out of beta (albertony)</li>
-<li>Minor cleanup of space around code section (albertony)</li>
-<li>Fixed some typos (albertony)</li>
-</ul></li>
-</ul></li>
-<li>VFS
-<ul>
-<li>Fix a code path which allows dirty data to be removed causing data loss (Nick Craig-Wood)</li>
-</ul></li>
-<li>Compress
-<ul>
-<li>Fix compressed name regexp (buengese)</li>
-</ul></li>
-<li>Drive
-<ul>
-<li>Fix backend copyid of google doc to directory (Nick Craig-Wood)</li>
-<li>Don't open browser when service account... (Ansh Mittal)</li>
-</ul></li>
-<li>Dropbox
-<ul>
-<li>Add missing team_data.member scope for use with --impersonate (Nick Craig-Wood)</li>
-<li>Fix About after scopes changes - rclone config reconnect needed (Nick Craig-Wood)</li>
-<li>Fix Unable to decrypt returned paths from changeNotify (Nick Craig-Wood)</li>
-</ul></li>
-<li>FTP
-<ul>
-<li>Fix implicit TLS (Ivan Andreev)</li>
-</ul></li>
-<li>Onedrive
-<ul>
-<li>Work around for random "Unable to initialize RPS" errors (OleFrost)</li>
-</ul></li>
-<li>SFTP
-<ul>
-<li>Revert sftp library to v1.12.0 from v1.13.0 to fix performance regression (Nick Craig-Wood)</li>
-<li>Fix Update ReadFrom failed: failed to send packet: EOF errors (Nick Craig-Wood)</li>
-</ul></li>
-<li>Zoho
-<ul>
-<li>Fix error when region isn't set (buengese)</li>
-<li>Do not ask for mountpoint twice when using headless setup (buengese)</li>
-</ul></li>
-</ul>
 <h2 id="v1.55.0---2021-03-31">v1.55.0 - 2021-03-31</h2>
 <p><a href="https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0">See commits</a></p>
 <ul>

MANUAL.md

@@ -1,6 +1,6 @@
 % rclone(1) User Manual
 % Nick Craig-Wood
-% Apr 26, 2021
+% Mar 31, 2021
 
 # Rclone syncs your files to cloud storage
 
@@ -2931,7 +2931,7 @@ metadata about files like in UNIX. One case that may arise is that other program
 (incorrectly) interprets this as the file being accessible by everyone. For example
 an SSH client may warn about "unprotected private key file".
 
-WinFsp 2021 (version 1.9) introduces a new FUSE option "FileSecurity",
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity",
 that allows the complete specification of file security descriptors using
 [SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
 With this you can work around issues such as the mentioned "unprotected private key file"
@@ -3751,7 +3751,7 @@ If the old version contains only dots and digits (for example `v1.54.0`)
 then it's a stable release so you won't need the `--beta` flag. Beta releases
 have an additional information similar to `v1.54.0-beta.5111.06f1c0c61`.
 (if you are a developer and use a locally built rclone, the version number
-will end with `-DEV`, you will have to rebuild it as it obviously can't
+will end with `-DEV`, you will have to rebuild it as it obvisously can't
 be distributed).
 
 If you previously installed rclone via a package manager, the package may
@@ -6977,7 +6977,7 @@ If you run `rclone config file` you will see where the default
 location is for you.
 
 Use this flag to override the config location, e.g. `rclone
---config=".myconfig" config`.
+--config=".myconfig" .config`.
 
 If the location is set to empty string `""` or the special value
 `/notfound`, or the os null device represented by value `NUL` on
@@ -11750,7 +11750,7 @@ These flags are available for every command.
       --use-json-log                         Use json log format.
       --use-mmap                             Use mmap allocator (see docs).
       --use-server-modtime                   Use server modified time instead of object metadata
-      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.1")
+      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
   -v, --verbose count                        Print lots more stuff (repeat for more)
 ```
 
@@ -28837,44 +28837,6 @@ Options:
 
 # Changelog
 
-## v1.55.1 - 2021-04-26
-
-[See commits](https://github.com/rclone/rclone/compare/v1.55.0...v1.55.1)
-
-* Bug Fixes
-    * selfupdate
-        * Dont detect FUSE if build is static (Ivan Andreev)
-        * Add build tag noselfupdate (Ivan Andreev)
-    * sync: Fix incorrect error reported by graceful cutoff (Nick Craig-Wood)
-    * install.sh: fix macOS arm64 download (Nick Craig-Wood)
-    * build: Fix version numbers in android branch builds (Nick Craig-Wood)
-    * docs
-        * Contributing.md: update setup instructions for go1.16 (Nick Gaya)
-        * WinFsp 2021 is out of beta (albertony)
-        * Minor cleanup of space around code section (albertony)
-        * Fixed some typos (albertony)
-* VFS
-    * Fix a code path which allows dirty data to be removed causing data loss (Nick Craig-Wood)
-* Compress
-    * Fix compressed name regexp (buengese)
-* Drive
-    * Fix backend copyid of google doc to directory (Nick Craig-Wood)
-    * Don't open browser when service account... (Ansh Mittal)
-* Dropbox
-    * Add missing team_data.member scope for use with --impersonate (Nick Craig-Wood)
-    * Fix About after scopes changes - rclone config reconnect needed (Nick Craig-Wood)
-    * Fix Unable to decrypt returned paths from changeNotify (Nick Craig-Wood)
-* FTP
-    * Fix implicit TLS (Ivan Andreev)
-* Onedrive
-    * Work around for random "Unable to initialize RPS" errors (OleFrost)
-* SFTP
-    * Revert sftp library to v1.12.0 from v1.13.0 to fix performance regression (Nick Craig-Wood)
-    * Fix Update ReadFrom failed: failed to send packet: EOF errors (Nick Craig-Wood)
-* Zoho
-    * Fix error when region isn't set (buengese)
-    * Do not ask for mountpoint twice when using headless setup (buengese)
-
 ## v1.55.0 - 2021-03-31
 
 [See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)

MANUAL.txt

@@ -1,6 +1,6 @@
 rclone(1) User Manual
 Nick Craig-Wood
-Apr 26, 2021
+Mar 31, 2021
 
 
 
@@ -2036,10 +2036,9 @@ Synopsis
 Download a URL's content and copy it to the destination without saving
 it in temporary storage.
 
-Setting --auto-filename will cause the file name to be retrieved from
+Setting --auto-filenamewill cause the file name to be retrieved from the
-the from URL (after any redirections) and used in the destination path.
+from URL (after any redirections) and used in the destination path. With
-With --print-filename in addition, the resuling file name will be
+--print-filename in addition, the resuling file name will be printed.
-printed.
 
 Setting --no-clobber will prevent overwriting file on the destination if
 there is one with the same name.
@@ -2969,10 +2968,10 @@ One case that may arise is that other programs (incorrectly) interprets
 this as the file being accessible by everyone. For example an SSH client
 may warn about "unprotected private key file".
 
-WinFsp 2021 (version 1.9) introduces a new FUSE option "FileSecurity",
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option
-that allows the complete specification of file security descriptors
+"FileSecurity", that allows the complete specification of file security
-using SDDL. With this you can work around issues such as the mentioned
+descriptors using SDDL. With this you can work around issues such as the
-"unprotected private key file" by specifying
+mentioned "unprotected private key file" by specifying
 -o FileSecurity="D:P(A;;FA;;;OW)", for file all access (FA) to the owner
 (OW).
 
@@ -3798,7 +3797,7 @@ and digits (for example v1.54.0) then it's a stable release so you won't
 need the --beta flag. Beta releases have an additional information
 similar to v1.54.0-beta.5111.06f1c0c61. (if you are a developer and use
 a locally built rclone, the version number will end with -DEV, you will
-have to rebuild it as it obviously can't be distributed).
+have to rebuild it as it obvisously can't be distributed).
 
 If you previously installed rclone via a package manager, the package
 may include local documentation or configure services. You may wish to
@@ -7124,7 +7123,7 @@ If you run rclone config file you will see where the default location is
 for you.
 
 Use this flag to override the config location, e.g.
-rclone --config=".myconfig" config.
+rclone --config=".myconfig" .config.
 
 If the location is set to empty string "" or the special value
 /notfound, or the os null device represented by value NUL on Windows and
@@ -11863,7 +11862,7 @@ These flags are available for every command.
           --use-json-log                         Use json log format.
           --use-mmap                             Use mmap allocator (see docs).
           --use-server-modtime                   Use server modified time instead of object metadata
-          --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.1")
+          --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
       -v, --verbose count                        Print lots more stuff (repeat for more)
 
 
@@ -28815,56 +28814,6 @@ Options:
 CHANGELOG
 
 
-v1.55.1 - 2021-04-26
-
-See commits
-
--   Bug Fixes
-    -   selfupdate
-        -   Dont detect FUSE if build is static (Ivan Andreev)
-        -   Add build tag noselfupdate (Ivan Andreev)
-    -   sync: Fix incorrect error reported by graceful cutoff (Nick
-        Craig-Wood)
-    -   install.sh: fix macOS arm64 download (Nick Craig-Wood)
-    -   build: Fix version numbers in android branch builds (Nick
-        Craig-Wood)
-    -   docs
-        -   Contributing.md: update setup instructions for go1.16 (Nick
-            Gaya)
-        -   WinFsp 2021 is out of beta (albertony)
-        -   Minor cleanup of space around code section (albertony)
-        -   Fixed some typos (albertony)
--   VFS
-    -   Fix a code path which allows dirty data to be removed causing
-        data loss (Nick Craig-Wood)
--   Compress
-    -   Fix compressed name regexp (buengese)
--   Drive
-    -   Fix backend copyid of google doc to directory (Nick Craig-Wood)
-    -   Don't open browser when service account... (Ansh Mittal)
--   Dropbox
-    -   Add missing team_data.member scope for use with --impersonate
-        (Nick Craig-Wood)
-    -   Fix About after scopes changes - rclone config reconnect needed
-        (Nick Craig-Wood)
-    -   Fix Unable to decrypt returned paths from changeNotify (Nick
-        Craig-Wood)
--   FTP
-    -   Fix implicit TLS (Ivan Andreev)
--   Onedrive
-    -   Work around for random "Unable to initialize RPS" errors
-        (OleFrost)
--   SFTP
-    -   Revert sftp library to v1.12.0 from v1.13.0 to fix performance
-        regression (Nick Craig-Wood)
-    -   Fix Update ReadFrom failed: failed to send packet: EOF errors
-        (Nick Craig-Wood)
--   Zoho
-    -   Fix error when region isn't set (buengese)
-    -   Do not ask for mountpoint twice when using headless setup
-        (buengese)
-
-
 v1.55.0 - 2021-03-31
 
 See commits

VERSION

@@ -1 +1 @@
-v1.55.1
+v1.56.0

backend/compress/compress.go

@@ -53,7 +53,7 @@ const (
 	Gzip         = 2
 )
 
-var nameRegexp = regexp.MustCompile("^(.+?)\\.([A-Za-z0-9-_]{11})$")
+var nameRegexp = regexp.MustCompile("^(.+?)\\.([A-Za-z0-9+_]{11})$")
 
 // Register with Fs
 func init() {

backend/drive/drive.go

@@ -199,7 +199,7 @@ func init() {
 				m.Set("root_folder_id", "appDataFolder")
 			}
 
-			if opt.ServiceAccountFile == "" && opt.ServiceAccountCredentials == "" {
+			if opt.ServiceAccountFile == "" {
 				err = oauthutil.Config(ctx, "drive", name, m, driveConfig, nil)
 				if err != nil {
 					log.Fatalf("Failed to configure token: %v", err)
@@ -2959,12 +2959,12 @@ func (f *Fs) makeShortcut(ctx context.Context, srcPath string, dstFs *Fs, dstPat
 }
 
 // List all team drives
-func (f *Fs) listTeamDrives(ctx context.Context) (drives []*drive.TeamDrive, err error) {
+func (f *Fs) listTeamDrives(ctx context.Context) (drives []*drive.Drive, err error) {
-	drives = []*drive.TeamDrive{}
+	drives = []*drive.Drive{}
-	listTeamDrives := f.svc.Teamdrives.List().PageSize(100)
+	listTeamDrives := f.svc.Drives.List().PageSize(100)
 	var defaultFs Fs // default Fs with default Options
 	for {
-		var teamDrives *drive.TeamDriveList
+		var teamDrives *drive.DriveList
 		err = f.pacer.Call(func() (bool, error) {
 			teamDrives, err = listTeamDrives.Context(ctx).Do()
 			return defaultFs.shouldRetry(ctx, err)
@@ -2972,7 +2972,7 @@ func (f *Fs) listTeamDrives(ctx context.Context) (drives []*drive.TeamDrive, err
 		if err != nil {
 			return drives, errors.Wrap(err, "listing Team Drives failed")
 		}
-		drives = append(drives, teamDrives.TeamDrives...)
+		drives = append(drives, teamDrives.Drives...)
 		if teamDrives.NextPageToken == "" {
 			break
 		}

backend/dropbox/dropbox.go

@@ -99,10 +99,8 @@ var (
 			"files.content.write",
 			"files.content.read",
 			"sharing.write",
-			"account_info.read", // needed for About
 			// "file_requests.write",
 			// "members.read", // needed for impersonate - but causes app to need to be approved by Dropbox Team Admin during the flow
-			// "team_data.member"
 		},
 		// Endpoint: oauth2.Endpoint{
 		// 	AuthURL:  "https://www.dropbox.com/1/oauth2/authorize",
@@ -132,8 +130,8 @@ func getOauthConfig(m configmap.Mapper) *oauth2.Config {
 	}
 	// Make a copy of the config
 	config := *dropboxConfig
-	// Make a copy of the scopes with extra scopes requires appended
+	// Make a copy of the scopes with "members.read" appended
-	config.Scopes = append(config.Scopes, "members.read", "team_data.member")
+	config.Scopes = append(config.Scopes, "members.read")
 	return &config
 }
 
@@ -1086,13 +1084,30 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
 	fs.Debugf(f, "attempting to share '%s' (absolute path: %s)", remote, absPath)
 	createArg := sharing.CreateSharedLinkWithSettingsArg{
 		Path: absPath,
-		// FIXME this gives settings_error/not_authorized/.. errors
+		Settings: &sharing.SharedLinkSettings{
-		// and the expires setting isn't in the documentation so remove
+			RequestedVisibility: &sharing.RequestedVisibility{
-		// for now.
+				Tagged: dropbox.Tagged{Tag: sharing.RequestedVisibilityPublic},
-		// Settings: &sharing.SharedLinkSettings{
+			},
-		// 	Expires: time.Now().Add(time.Duration(expire)).UTC().Round(time.Second),
+			Audience: &sharing.LinkAudience{
-		// },
+				Tagged: dropbox.Tagged{Tag: sharing.LinkAudiencePublic},
+			},
+			Access: &sharing.RequestedLinkAccessLevel{
+				Tagged: dropbox.Tagged{Tag: sharing.RequestedLinkAccessLevelViewer},
+			},
+		},
 	}
+	if expire < fs.DurationOff {
+		expiryTime := time.Now().Add(time.Duration(expire)).UTC().Round(time.Second)
+		createArg.Settings.Expires = expiryTime
+	}
+	// FIXME note we can't set Settings for non enterprise dropbox
+	// because of https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75
+	// however this only goes wrong when we set Expires, so as a
+	// work-around remove Settings unless expire is set.
+	if expire == fs.DurationOff {
+		createArg.Settings = nil
+	}
+
 	var linkRes sharing.IsSharedLinkMetadata
 	err = f.pacer.Call(func() (bool, error) {
 		linkRes, err = f.sharing.CreateSharedLinkWithSettings(&createArg)
@@ -1336,13 +1351,13 @@ func (f *Fs) changeNotifyRunner(ctx context.Context, notifyFunc func(string, fs.
 			switch info := entry.(type) {
 			case *files.FolderMetadata:
 				entryType = fs.EntryDirectory
-				entryPath = strings.TrimPrefix(info.PathDisplay, f.slashRootSlash)
+				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
 			case *files.FileMetadata:
 				entryType = fs.EntryObject
-				entryPath = strings.TrimPrefix(info.PathDisplay, f.slashRootSlash)
+				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
 			case *files.DeletedMetadata:
 				entryType = fs.EntryObject
-				entryPath = strings.TrimPrefix(info.PathDisplay, f.slashRootSlash)
+				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
 			default:
 				fs.Errorf(entry, "dropbox ChangeNotify: ignoring unknown EntryType %T", entry)
 				continue

backend/fichier/fichier.go

@@ -348,8 +348,10 @@ func (f *Fs) putUnchecked(ctx context.Context, in io.Reader, remote string, size
 		return nil, err
 	}
 
-	if len(fileUploadResponse.Links) != 1 {
+	if len(fileUploadResponse.Links) == 0 {
-		return nil, errors.New("unexpected amount of files")
+		return nil, errors.New("upload response not found")
+	} else if len(fileUploadResponse.Links) > 1 {
+		fs.Debugf(remote, "Multiple upload responses found, using the first")
 	}
 
 	link := fileUploadResponse.Links[0]

backend/ftp/ftp.go

@@ -241,6 +241,23 @@ func (dl *debugLog) Write(p []byte) (n int, err error) {
 	return len(p), nil
 }
 
+type dialCtx struct {
+	f   *Fs
+	ctx context.Context
+}
+
+// dial a new connection with fshttp dialer
+func (d *dialCtx) dial(network, address string) (net.Conn, error) {
+	conn, err := fshttp.NewDialer(d.ctx).Dial(network, address)
+	if err != nil {
+		return nil, err
+	}
+	if d.f.tlsConf != nil {
+		conn = tls.Client(conn, d.f.tlsConf)
+	}
+	return conn, err
+}
+
 // shouldRetry returns a boolean as to whether this err deserve to be
 // retried.  It returns the err as a convenience
 func shouldRetry(ctx context.Context, err error) (bool, error) {
@@ -260,22 +277,9 @@ func shouldRetry(ctx context.Context, err error) (bool, error) {
 // Open a new connection to the FTP server.
 func (f *Fs) ftpConnection(ctx context.Context) (c *ftp.ServerConn, err error) {
 	fs.Debugf(f, "Connecting to FTP server")
-
+	dCtx := dialCtx{f, ctx}
-	// Make ftp library dial with fshttp dialer optionally using TLS
+	ftpConfig := []ftp.DialOption{ftp.DialWithDialFunc(dCtx.dial)}
-	dial := func(network, address string) (conn net.Conn, err error) {
+	if f.opt.ExplicitTLS {
-		conn, err = fshttp.NewDialer(ctx).Dial(network, address)
-		if f.tlsConf != nil && err == nil {
-			conn = tls.Client(conn, f.tlsConf)
-		}
-		return
-	}
-	ftpConfig := []ftp.DialOption{ftp.DialWithDialFunc(dial)}
-
-	if f.opt.TLS {
-		// Our dialer takes care of TLS but ftp library also needs tlsConf
-		// as a trigger for sending PSBZ and PROT options to server.
-		ftpConfig = append(ftpConfig, ftp.DialWithTLS(f.tlsConf))
-	} else if f.opt.ExplicitTLS {
 		ftpConfig = append(ftpConfig, ftp.DialWithExplicitTLS(f.tlsConf))
 		// Initial connection needs to be cleartext for explicit TLS
 		conn, err := fshttp.NewDialer(ctx).Dial("tcp", f.dialAddr)

backend/onedrive/onedrive.go

@@ -361,6 +361,11 @@ This will only work if you are copying between two OneDrive *Personal* drives AN
 the files to copy are already shared between them.  In other cases, rclone will
 fall back to normal copy (which will be slightly slower).`,
 			Advanced: true,
+		}, {
+			Name:     "list_chunk",
+			Help:     "Size of listing chunk.",
+			Default:  1000,
+			Advanced: true,
 		}, {
 			Name:    "no_versions",
 			Default: false,
@@ -468,6 +473,7 @@ type Options struct {
 	DriveType               string               `config:"drive_type"`
 	ExposeOneNoteFiles      bool                 `config:"expose_onenote_files"`
 	ServerSideAcrossConfigs bool                 `config:"server_side_across_configs"`
+	ListChunk               int64                `config:"list_chunk"`
 	NoVersions              bool                 `config:"no_versions"`
 	LinkScope               string               `config:"link_scope"`
 	LinkType                string               `config:"link_type"`
@@ -560,9 +566,6 @@ func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, err
 			if len(resp.Header["Www-Authenticate"]) == 1 && strings.Index(resp.Header["Www-Authenticate"][0], "expired_token") >= 0 {
 				retry = true
 				fs.Debugf(nil, "Should retry: %v", err)
-			} else if err != nil && strings.Contains(err.Error(), "Unable to initialize RPS") {
-				retry = true
-				fs.Debugf(nil, "HTTP 401: Unable to initialize RPS. Trying again.")
 			}
 		case 429: // Too Many Requests.
 			// see https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
@@ -899,7 +902,7 @@ type listAllFn func(*api.Item) bool
 func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
 	// Top parameter asks for bigger pages of data
 	// https://dev.onedrive.com/odata/optional-query-parameters.htm
-	opts := f.newOptsCall(dirID, "GET", "/children?$top=1000")
+	opts := f.newOptsCall(dirID, "GET", fmt.Sprintf("/children?$top=%d", f.opt.ListChunk))
 OUTER:
 	for {
 		var result api.ListChildrenResponse
@@ -1426,7 +1429,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
 		Password: f.opt.LinkPassword,
 	}
 
-	if expire < fs.Duration(time.Hour*24*365*100) {
+	if expire < fs.DurationOff {
 		expiry := time.Now().Add(time.Duration(expire))
 		share.Expiry = &expiry
 	}

cmd/cmd.go

@@ -75,8 +75,19 @@ const (
 
 // ShowVersion prints the version to stdout
 func ShowVersion() {
+	osVersion, osKernel := buildinfo.GetOSVersion()
+	if osVersion == "" {
+		osVersion = "unknown"
+	}
+	if osKernel == "" {
+		osKernel = "unknown"
+	}
+
 	linking, tagString := buildinfo.GetLinkingAndTags()
+
 	fmt.Printf("rclone %s\n", fs.Version)
+	fmt.Printf("- os/version: %s\n", osVersion)
+	fmt.Printf("- os/kernel: %s\n", osKernel)
 	fmt.Printf("- os/type: %s\n", runtime.GOOS)
 	fmt.Printf("- os/arch: %s\n", runtime.GOARCH)
 	fmt.Printf("- go/version: %s\n", runtime.Version())

cmd/link/link.go

@@ -3,7 +3,6 @@ package link
 import (
 	"context"
 	"fmt"
-	"time"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/rclone/rclone/fs"
@@ -13,7 +12,7 @@ import (
 )
 
 var (
-	expire = fs.Duration(time.Hour * 24 * 365 * 100)
+	expire = fs.DurationOff
 	unlink = false
 )
 

cmd/version/version.go

@@ -29,13 +29,16 @@ var commandDefinition = &cobra.Command{
 	Use:   "version",
 	Short: `Show the version number.`,
 	Long: `
-Show the rclone version number, the go version, the build target OS and
+Show the rclone version number, the go version, the build target
-architecture, build tags and the type of executable (static or dynamic).
+OS and architecture, the runtime OS and kernel version and bitness,
+build tags and the type of executable (static or dynamic).
 
 For example:
 
     $ rclone version
-    rclone v1.54
+    rclone v1.55.0
+    - os/version: ubuntu 18.04 (64 bit)
+    - os/kernel: 4.15.0-136-generic (x86_64)
     - os/type: linux
     - os/arch: amd64
     - go/version: go1.16

docs/content/authors.md

@@ -478,3 +478,4 @@ put them back in again.` >}}
   * Manish Kumar <krmanish260@gmail.com>
   * x0b <x0bdev@gmail.com>
   * CERN through the CS3MESH4EOSC Project
+  * Nick Gaya <nicholasgaya+github@gmail.com>

docs/content/changelog.md

@@ -5,44 +5,6 @@ description: "Rclone Changelog"
 
 # Changelog
 
-## v1.55.1 - 2021-04-26
-
-[See commits](https://github.com/rclone/rclone/compare/v1.55.0...v1.55.1)
-
-* Bug Fixes
-    * selfupdate
-        * Dont detect FUSE if build is static (Ivan Andreev)
-        * Add build tag noselfupdate (Ivan Andreev)
-    * sync: Fix incorrect error reported by graceful cutoff (Nick Craig-Wood)
-    * install.sh: fix macOS arm64 download (Nick Craig-Wood)
-    * build: Fix version numbers in android branch builds (Nick Craig-Wood)
-    * docs
-        * Contributing.md: update setup instructions for go1.16 (Nick Gaya)
-        * WinFsp 2021 is out of beta (albertony)
-        * Minor cleanup of space around code section (albertony)
-        * Fixed some typos (albertony)
-* VFS
-    * Fix a code path which allows dirty data to be removed causing data loss (Nick Craig-Wood)
-* Compress
-    * Fix compressed name regexp (buengese)
-* Drive
-    * Fix backend copyid of google doc to directory (Nick Craig-Wood)
-    * Don't open browser when service account... (Ansh Mittal)
-* Dropbox
-    * Add missing team_data.member scope for use with --impersonate (Nick Craig-Wood)
-    * Fix About after scopes changes - rclone config reconnect needed (Nick Craig-Wood)
-    * Fix Unable to decrypt returned paths from changeNotify (Nick Craig-Wood)
-* FTP
-    * Fix implicit TLS (Ivan Andreev)
-* Onedrive
-    * Work around for random "Unable to initialize RPS" errors (OleFrost)
-* SFTP
-    * Revert sftp library to v1.12.0 from v1.13.0 to fix performance regression (Nick Craig-Wood)
-    * Fix Update ReadFrom failed: failed to send packet: EOF errors (Nick Craig-Wood)
-* Zoho
-    * Fix error when region isn't set (buengese)
-    * Do not ask for mountpoint twice when using headless setup (buengese)
-
 ## v1.55.0 - 2021-03-31
 
 [See commits](https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)

docs/content/commands/rclone_mount.md

@@ -184,7 +184,7 @@ metadata about files like in UNIX. One case that may arise is that other program
 (incorrectly) interprets this as the file being accessible by everyone. For example
 an SSH client may warn about "unprotected private key file".
 
-WinFsp 2021 (version 1.9) introduces a new FUSE option "FileSecurity",
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity",
 that allows the complete specification of file security descriptors using
 [SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
 With this you can work around issues such as the mentioned "unprotected private key file"

docs/content/commands/rclone_selfupdate.md

@@ -37,7 +37,7 @@ If the old version contains only dots and digits (for example `v1.54.0`)
 then it's a stable release so you won't need the `--beta` flag. Beta releases
 have an additional information similar to `v1.54.0-beta.5111.06f1c0c61`.
 (if you are a developer and use a locally built rclone, the version number
-will end with `-DEV`, you will have to rebuild it as it obviously can't
+will end with `-DEV`, you will have to rebuild it as it obvisously can't
 be distributed).
 
 If you previously installed rclone via a package manager, the package may

docs/content/dropbox.md

@@ -285,6 +285,12 @@ dropbox:dir` will return the error `Failed to purge: There are too
 many files involved in this operation`.  As a work-around do an
 `rclone delete dropbox:dir` followed by an `rclone rmdir dropbox:dir`.
 
+When using `rclone link` you'll need to set `--expire` if using a
+non-personal account otherwise the visibility may not be correct.
+(Note that `--expire` isn't supported on personal accounts). See the
+[forum discussion](https://forum.rclone.org/t/rclone-link-dropbox-permissions/23211) and the 
+[dropbox SDK issue](https://github.com/dropbox/dropbox-sdk-go-unofficial/issues/75).
+
 ### Get your own Dropbox App ID ###
 
 When you use rclone with Dropbox in its default configuration you are using rclone's App ID. This is shared between all the rclone users.

docs/content/flags.md

@@ -153,7 +153,7 @@ These flags are available for every command.
       --use-json-log                         Use json log format.
       --use-mmap                             Use mmap allocator (see docs).
       --use-server-modtime                   Use server modified time instead of object metadata
-      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.1")
+      --user-agent string                    Set the user-agent to a specified string. The default is rclone/ version (default "rclone/v1.55.0")
   -v, --verbose count                        Print lots more stuff (repeat for more)
 ```
 
@@ -415,6 +415,7 @@ and may be set in the config file.
       --onedrive-link-password string                            Set the password for links created by the link command.
       --onedrive-link-scope string                               Set the scope of the links created by the link command. (default "anonymous")
       --onedrive-link-type string                                Set the type of the links created by the link command. (default "view")
+      --onedrive-list-chunk int                                  Size of listing chunk. (default 1000)
       --onedrive-no-versions                                     Remove all versions on modifying operations
       --onedrive-region string                                   Choose national cloud region for OneDrive. (default "global")
       --onedrive-server-side-across-configs                      Allow server-side operations (e.g. copy) to work across different onedrive configs.

docs/content/onedrive.md

@@ -325,6 +325,15 @@ fall back to normal copy (which will be slightly slower).
 - Type:        bool
 - Default:     false
 
+#### --onedrive-list-chunk
+
+Size of listing chunk.
+
+- Config:      list_chunk
+- Env Var:     RCLONE_ONEDRIVE_LIST_CHUNK
+- Type:        int
+- Default:     1000
+
 #### --onedrive-no-versions
 
 Remove all versions on modifying operations

docs/layouts/partials/version.html

@@ -1 +1 @@
-v1.55.1
+v1.56.0

fs/config/authorize.go

@@ -2,9 +2,11 @@ package config
 
 import (
 	"context"
+	"fmt"
 
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/fs"
+	"github.com/rclone/rclone/fs/config/configmap"
 )
 
 // Authorize is for remote authorization of headless machines.
@@ -16,33 +18,61 @@ import (
 func Authorize(ctx context.Context, args []string, noAutoBrowser bool) error {
 	ctx = suppressConfirm(ctx)
 	switch len(args) {
-	case 1, 3:
+	case 1, 2, 3:
 	default:
 		return errors.Errorf("invalid number of arguments: %d", len(args))
 	}
-	newType := args[0]
+	Type := args[0] // FIXME could read this from input
-	f := fs.MustFind(newType)
+	ri, err := fs.Find(Type)
-	if f.Config == nil {
+	if err != nil {
-		return errors.Errorf("can't authorize fs %q", newType)
+		return err
+	}
+	if ri.Config == nil {
+		return errors.Errorf("can't authorize fs %q", Type)
 	}
-	// Name used for temporary fs
-	name := "**temp-fs**"
 
-	// Make sure we delete it
+	// Config map for remote
-	defer DeleteRemote(name)
+	inM := configmap.Simple{}
 
 	// Indicate that we are running rclone authorize
-	Data.SetValue(name, ConfigAuthorize, "true")
+	inM[ConfigAuthorize] = "true"
 	if noAutoBrowser {
-		Data.SetValue(name, ConfigAuthNoBrowser, "true")
+		inM[ConfigAuthNoBrowser] = "true"
 	}
 
-	if len(args) == 3 {
+	// Add extra parameters if supplied
-		Data.SetValue(name, ConfigClientID, args[1])
+	if len(args) == 2 {
-		Data.SetValue(name, ConfigClientSecret, args[2])
+		err := inM.Decode(args[1])
+		if err != nil {
+			return err
+		}
+	} else if len(args) == 3 {
+		inM[ConfigClientID] = args[1]
+		inM[ConfigClientSecret] = args[2]
 	}
 
-	m := fs.ConfigMap(f, name, nil)
+	// Name used for temporary remote
-	f.Config(ctx, name, m)
+	name := "**temp-fs**"
+
+	m := fs.ConfigMap(ri, name, inM)
+	outM := configmap.Simple{}
+	m.ClearSetters()
+	m.AddSetter(outM)
+	m.AddGetter(outM, configmap.PriorityNormal)
+
+	ri.Config(ctx, name, m)
+
+	// Print the code for the user to paste
+	out := outM["token"]
+
+	// If received a config blob, then return one
+	if len(args) == 2 {
+		out, err = outM.Encode()
+		if err != nil {
+			return err
+		}
+	}
+	fmt.Printf("Paste the following into your remote machine --->\n%s\n<---End paste\n", out)
+
 	return nil
 }

fs/config/configmap/configmap.go

@@ -2,8 +2,24 @@
 package configmap
 
 import (
+	"encoding/base64"
+	"encoding/json"
 	"sort"
 	"strings"
+	"unicode"
+
+	"github.com/pkg/errors"
+)
+
+// Priority of getters
+type Priority int8
+
+// Priority levels for AddGetter
+const (
+	PriorityNormal  Priority = iota
+	PriorityConfig           // use for reading from the config
+	PriorityDefault          // use for default values
+	PriorityMax
 )
 
 // Getter provides an interface to get config items
@@ -30,8 +46,12 @@ type Mapper interface {
 // Getter interfaces.
 type Map struct {
 	setters []Setter
-	getters  []Getter
+	getters []getprio
-	override []Getter
+}
+
+type getprio struct {
+	getter   Getter
+	priority Priority
 }
 
 // New returns an empty Map
@@ -39,18 +59,12 @@ func New() *Map {
 	return &Map{}
 }
 
-// AddGetter appends a getter onto the end of the getters
+// AddGetter appends a getter onto the end of the getters in priority order
-func (c *Map) AddGetter(getter Getter) *Map {
+func (c *Map) AddGetter(getter Getter, priority Priority) *Map {
-	c.getters = append(c.getters, getter)
+	c.getters = append(c.getters, getprio{getter, priority})
-	return c
+	sort.SliceStable(c.getters, func(i, j int) bool {
-}
+		return c.getters[i].priority < c.getters[j].priority
-
+	})
-// AddOverrideGetter appends a getter onto the end of the getters
-//
-// It also appends it onto the override getters for GetOverride
-func (c *Map) AddOverrideGetter(getter Getter) *Map {
-	c.getters = append(c.getters, getter)
-	c.override = append(c.override, getter)
 	return c
 }
 
@@ -60,12 +74,34 @@ func (c *Map) AddSetter(setter Setter) *Map {
 	return c
 }
 
-// get gets an item with the key passed in and return the value from
+// ClearSetters removes all the setters set so far
-// the first getter. If the item is found then it returns true,
+func (c *Map) ClearSetters() *Map {
-// otherwise false.
+	c.setters = nil
-func (c *Map) get(key string, getters []Getter) (value string, ok bool) {
+	return c
-	for _, do := range getters {
+}
-		value, ok = do.Get(key)
+
+// ClearGetters removes all the getters with the priority given
+func (c *Map) ClearGetters(priority Priority) *Map {
+	getters := c.getters[:0]
+	for _, item := range c.getters {
+		if item.priority != priority {
+			getters = append(getters, item)
+		}
+	}
+	c.getters = getters
+	return c
+}
+
+// GetPriority gets an item with the key passed in and return the
+// value from the first getter to return a result with priority <=
+// maxPriority. If the item is found then it returns true, otherwise
+// false.
+func (c *Map) GetPriority(key string, maxPriority Priority) (value string, ok bool) {
+	for _, item := range c.getters {
+		if item.priority > maxPriority {
+			break
+		}
+		value, ok = item.getter.Get(key)
 		if ok {
 			return value, ok
 		}
@@ -77,14 +113,7 @@ func (c *Map) get(key string, getters []Getter) (value string, ok bool) {
 // the first getter. If the item is found then it returns true,
 // otherwise false.
 func (c *Map) Get(key string) (value string, ok bool) {
-	return c.get(key, c.getters)
+	return c.GetPriority(key, PriorityMax)
-}
-
-// GetOverride gets an item with the key passed in and return the
-// value from the first override getter. If the item is found then it
-// returns true, otherwise false.
-func (c *Map) GetOverride(key string) (value string, ok bool) {
-	return c.get(key, c.override)
 }
 
 // Set sets an item into all the stored setters.
@@ -135,3 +164,38 @@ func (c Simple) String() string {
 	}
 	return out.String()
 }
+
+// Encode from c into a string suitable for putting on the command line
+func (c Simple) Encode() (string, error) {
+	if len(c) == 0 {
+		return "", nil
+	}
+	buf, err := json.Marshal(c)
+	if err != nil {
+		return "", errors.Wrap(err, "encode simple map")
+	}
+	return base64.RawStdEncoding.EncodeToString(buf), nil
+}
+
+// Decode an Encode~d string in into c
+func (c Simple) Decode(in string) error {
+	// Remove all whitespace from the input string
+	in = strings.Map(func(r rune) rune {
+		if unicode.IsSpace(r) {
+			return -1
+		}
+		return r
+	}, in)
+	if len(in) == 0 {
+		return nil
+	}
+	decodedM, err := base64.RawStdEncoding.DecodeString(in)
+	if err != nil {
+		return errors.Wrap(err, "decode simple map")
+	}
+	err = json.Unmarshal(decodedM, &c)
+	if err != nil {
+		return errors.Wrap(err, "parse simple map")
+	}
+	return nil
+}

fs/config/configmap/configmap_test.go

@@ -1,9 +1,11 @@
 package configmap
 
 import (
+	"encoding/base64"
 	"testing"
 
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
 var (
@@ -27,7 +29,7 @@ func TestConfigMapGet(t *testing.T) {
 		"config1": "one",
 	}
 
-	m.AddGetter(m1)
+	m.AddGetter(m1, PriorityNormal)
 
 	value, found = m.Get("config1")
 	assert.Equal(t, "one", value)
@@ -42,7 +44,7 @@ func TestConfigMapGet(t *testing.T) {
 		"config2": "two2",
 	}
 
-	m.AddGetter(m2)
+	m.AddGetter(m2, PriorityNormal)
 
 	value, found = m.Get("config1")
 	assert.Equal(t, "one", value)
@@ -88,56 +90,160 @@ func TestConfigMapSet(t *testing.T) {
 		"config1": "beetroot",
 		"config2": "potato",
 	}, m2)
+
+	m.ClearSetters()
+
+	// Check that nothing gets set
+	m.Set("config1", "BEETROOT")
+
+	assert.Equal(t, Simple{
+		"config1": "beetroot",
+		"config2": "potato",
+	}, m1)
+	assert.Equal(t, Simple{
+		"config1": "beetroot",
+		"config2": "potato",
+	}, m2)
+
 }
 
-func TestConfigMapGetOverride(t *testing.T) {
+func TestConfigMapGetPriority(t *testing.T) {
 	m := New()
 
-	value, found := m.GetOverride("config1")
+	value, found := m.GetPriority("config1", PriorityMax)
 	assert.Equal(t, "", value)
 	assert.Equal(t, false, found)
 
-	value, found = m.GetOverride("config2")
+	value, found = m.GetPriority("config2", PriorityMax)
 	assert.Equal(t, "", value)
 	assert.Equal(t, false, found)
 
 	m1 := Simple{
 		"config1": "one",
+		"config3": "three",
 	}
 
-	m.AddOverrideGetter(m1)
+	m.AddGetter(m1, PriorityConfig)
 
-	value, found = m.GetOverride("config1")
+	value, found = m.GetPriority("config1", PriorityNormal)
+	assert.Equal(t, "", value)
+	assert.Equal(t, false, found)
+
+	value, found = m.GetPriority("config2", PriorityNormal)
+	assert.Equal(t, "", value)
+	assert.Equal(t, false, found)
+
+	value, found = m.GetPriority("config3", PriorityNormal)
+	assert.Equal(t, "", value)
+	assert.Equal(t, false, found)
+
+	value, found = m.GetPriority("config1", PriorityConfig)
 	assert.Equal(t, "one", value)
 	assert.Equal(t, true, found)
 
-	value, found = m.GetOverride("config2")
+	value, found = m.GetPriority("config2", PriorityConfig)
 	assert.Equal(t, "", value)
 	assert.Equal(t, false, found)
 
+	value, found = m.GetPriority("config3", PriorityConfig)
+	assert.Equal(t, "three", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config1", PriorityMax)
+	assert.Equal(t, "one", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config2", PriorityMax)
+	assert.Equal(t, "", value)
+	assert.Equal(t, false, found)
+
+	value, found = m.GetPriority("config3", PriorityMax)
+	assert.Equal(t, "three", value)
+	assert.Equal(t, true, found)
+
 	m2 := Simple{
 		"config1": "one2",
 		"config2": "two2",
 	}
 
-	m.AddGetter(m2)
+	m.AddGetter(m2, PriorityNormal)
 
-	value, found = m.GetOverride("config1")
+	value, found = m.GetPriority("config1", PriorityNormal)
-	assert.Equal(t, "one", value)
+	assert.Equal(t, "one2", value)
 	assert.Equal(t, true, found)
 
-	value, found = m.GetOverride("config2")
+	value, found = m.GetPriority("config2", PriorityNormal)
-	assert.Equal(t, "", value)
-	assert.Equal(t, false, found)
-
-	value, found = m.Get("config1")
-	assert.Equal(t, "one", value)
-	assert.Equal(t, true, found)
-
-	value, found = m.Get("config2")
 	assert.Equal(t, "two2", value)
 	assert.Equal(t, true, found)
 
+	value, found = m.GetPriority("config3", PriorityNormal)
+	assert.Equal(t, "", value)
+	assert.Equal(t, false, found)
+
+	value, found = m.GetPriority("config1", PriorityConfig)
+	assert.Equal(t, "one2", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config2", PriorityConfig)
+	assert.Equal(t, "two2", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config3", PriorityConfig)
+	assert.Equal(t, "three", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config1", PriorityMax)
+	assert.Equal(t, "one2", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config2", PriorityMax)
+	assert.Equal(t, "two2", value)
+	assert.Equal(t, true, found)
+
+	value, found = m.GetPriority("config3", PriorityMax)
+	assert.Equal(t, "three", value)
+	assert.Equal(t, true, found)
+}
+
+func TestConfigMapClearGetters(t *testing.T) {
+	m := New()
+	m1 := Simple{}
+	m2 := Simple{}
+	m3 := Simple{}
+	m.AddGetter(m1, PriorityNormal)
+	m.AddGetter(m2, PriorityDefault)
+	m.AddGetter(m3, PriorityConfig)
+	assert.Equal(t, []getprio{
+		{m1, PriorityNormal},
+		{m3, PriorityConfig},
+		{m2, PriorityDefault},
+	}, m.getters)
+	m.ClearGetters(PriorityConfig)
+	assert.Equal(t, []getprio{
+		{m1, PriorityNormal},
+		{m2, PriorityDefault},
+	}, m.getters)
+	m.ClearGetters(PriorityNormal)
+	assert.Equal(t, []getprio{
+		{m2, PriorityDefault},
+	}, m.getters)
+	m.ClearGetters(PriorityDefault)
+	assert.Equal(t, []getprio{}, m.getters)
+	m.ClearGetters(PriorityDefault)
+	assert.Equal(t, []getprio{}, m.getters)
+}
+
+func TestConfigMapClearSetters(t *testing.T) {
+	m := New()
+	m1 := Simple{}
+	m2 := Simple{}
+	m3 := Simple{}
+	m.AddSetter(m1)
+	m.AddSetter(m2)
+	m.AddSetter(m3)
+	assert.Equal(t, []Setter{m1, m2, m3}, m.setters)
+	m.ClearSetters()
+	assert.Equal(t, []Setter(nil), m.setters)
 }
 
 func TestSimpleString(t *testing.T) {
@@ -163,3 +269,91 @@ func TestSimpleString(t *testing.T) {
 		"apple":   "",
 	}.String())
 }
+
+func TestSimpleEncode(t *testing.T) {
+	for _, test := range []struct {
+		in   Simple
+		want string
+	}{
+		{
+			in:   Simple{},
+			want: "",
+		},
+		{
+			in: Simple{
+				"one": "potato",
+			},
+			want: "eyJvbmUiOiJwb3RhdG8ifQ",
+		},
+		{
+			in: Simple{
+				"one": "potato",
+				"two": "",
+			},
+			want: "eyJvbmUiOiJwb3RhdG8iLCJ0d28iOiIifQ",
+		},
+	} {
+		got, err := test.in.Encode()
+		require.NoError(t, err)
+		assert.Equal(t, test.want, got)
+		gotM := Simple{}
+		err = gotM.Decode(got)
+		require.NoError(t, err)
+		assert.Equal(t, test.in, gotM)
+	}
+}
+
+func TestSimpleDecode(t *testing.T) {
+	for _, test := range []struct {
+		in      string
+		want    Simple
+		wantErr string
+	}{
+		{
+			in:   "",
+			want: Simple{},
+		},
+		{
+			in: "eyJvbmUiOiJwb3RhdG8ifQ",
+			want: Simple{
+				"one": "potato",
+			},
+		},
+		{
+			in: "   e yJvbm  UiOiJwb\r\n 3Rhd\tG8ifQ\n\n ",
+			want: Simple{
+				"one": "potato",
+			},
+		},
+		{
+			in: "eyJvbmUiOiJwb3RhdG8iLCJ0d28iOiIifQ",
+			want: Simple{
+				"one": "potato",
+				"two": "",
+			},
+		},
+		{
+			in:      "!!!!!",
+			want:    Simple{},
+			wantErr: "decode simple map",
+		},
+		{
+			in:   base64.RawStdEncoding.EncodeToString([]byte(`null`)),
+			want: Simple{},
+		},
+		{
+			in:      base64.RawStdEncoding.EncodeToString([]byte(`rubbish`)),
+			want:    Simple{},
+			wantErr: "parse simple map",
+		},
+	} {
+		got := Simple{}
+		err := got.Decode(test.in)
+		assert.Equal(t, test.want, got, test.in)
+		if test.wantErr == "" {
+			require.NoError(t, err, test.in)
+		} else {
+			assert.Contains(t, err.Error(), test.wantErr, test.in)
+		}
+	}
+}

fs/fs.go

@@ -125,6 +125,38 @@ func (os Options) Get(name string) *Option {
 	return nil
 }
 
+// Overridden discovers which config items have been overridden in the
+// configmap passed in, either by the config string, command line
+// flags or environment variables
+func (os Options) Overridden(m *configmap.Map) configmap.Simple {
+	var overridden = configmap.Simple{}
+	for i := range os {
+		opt := &os[i]
+		value, isSet := m.GetPriority(opt.Name, configmap.PriorityNormal)
+		if isSet {
+			overridden.Set(opt.Name, value)
+		}
+	}
+	return overridden
+}
+
+// NonDefault discovers which config values aren't at their default
+func (os Options) NonDefault(m configmap.Getter) configmap.Simple {
+	var nonDefault = configmap.Simple{}
+	for i := range os {
+		opt := &os[i]
+		value, isSet := m.Get(opt.Name)
+		if !isSet {
+			continue
+		}
+		defaultValue := fmt.Sprint(opt.Default)
+		if value != defaultValue {
+			nonDefault.Set(opt.Name, value)
+		}
+	}
+	return nonDefault
+}
+
 // OptionVisibility controls whether the options are visible in the
 // configurator or the command line.
 type OptionVisibility byte
@@ -1321,28 +1353,28 @@ func ConfigMap(fsInfo *RegInfo, configName string, connectionStringConfig config
 
 	// Config from connection string
 	if len(connectionStringConfig) > 0 {
-		config.AddOverrideGetter(connectionStringConfig)
+		config.AddGetter(connectionStringConfig, configmap.PriorityNormal)
 	}
 
 	// flag values
 	if fsInfo != nil {
-		config.AddOverrideGetter(&regInfoValues{fsInfo, false})
+		config.AddGetter(&regInfoValues{fsInfo, false}, configmap.PriorityNormal)
 	}
 
 	// remote specific environment vars
-	config.AddOverrideGetter(configEnvVars(configName))
+	config.AddGetter(configEnvVars(configName), configmap.PriorityNormal)
 
 	// backend specific environment vars
 	if fsInfo != nil {
-		config.AddOverrideGetter(optionEnvVars{fsInfo: fsInfo})
+		config.AddGetter(optionEnvVars{fsInfo: fsInfo}, configmap.PriorityNormal)
 	}
 
 	// config file
-	config.AddGetter(getConfigFile(configName))
+	config.AddGetter(getConfigFile(configName), configmap.PriorityConfig)
 
 	// default values
 	if fsInfo != nil {
-		config.AddGetter(&regInfoValues{fsInfo, true})
+		config.AddGetter(&regInfoValues{fsInfo, true}, configmap.PriorityDefault)
 	}
 
 	// Set Config
@@ -1381,17 +1413,7 @@ func NewFs(ctx context.Context, path string) (Fs, error) {
 	if err != nil {
 		return nil, err
 	}
-	// Now discover which config items have been overridden,
+	overridden := fsInfo.Options.Overridden(config)
-	// either by the config string, command line flags or
-	// environment variables
-	var overridden = configmap.Simple{}
-	for i := range fsInfo.Options {
-		opt := &fsInfo.Options[i]
-		value, isSet := config.GetOverride(opt.Name)
-		if isSet {
-			overridden.Set(opt.Name, value)
-		}
-	}
 	if len(overridden) > 0 {
 		extraConfig := overridden.String()
 		//Debugf(nil, "detected overriden config %q", extraConfig)
@@ -1409,6 +1431,11 @@ func NewFs(ctx context.Context, path string) (Fs, error) {
 		// These need to work as filesystem names as the VFS cache will use them
 		configName += suffix
 	}
+	Debugf(nil, "Config dump:")
+	nonDefaultConfig := fsInfo.Options.NonDefault(config)
+	for k, v := range nonDefaultConfig {
+		Debugf(nil, "  %s = %q", k, v)
+	}
 	return fsInfo.NewFs(ctx, configName, fsPath, config)
 }
 

fs/fs_test.go

@@ -190,6 +190,43 @@ func TestOptionsGet(t *testing.T) {
 	assert.Nil(t, opt)
 }
 
+func TestOptionsOveridden(t *testing.T) {
+	m := configmap.New()
+	m1 := configmap.Simple{
+		"nounc":      "m1",
+		"copy_links": "m1",
+	}
+	m.AddGetter(m1, configmap.PriorityNormal)
+	m2 := configmap.Simple{
+		"nounc":            "m2",
+		"case_insensitive": "m2",
+	}
+	m.AddGetter(m2, configmap.PriorityConfig)
+	m3 := configmap.Simple{
+		"nounc": "m3",
+	}
+	m.AddGetter(m3, configmap.PriorityDefault)
+	got := testOptions.Overridden(m)
+	assert.Equal(t, configmap.Simple{
+		"copy_links": "m1",
+		"nounc":      "m1",
+	}, got)
+}
+
+func TestOptionsNonDefault(t *testing.T) {
+	m := configmap.Simple{}
+	got := testOptions.NonDefault(m)
+	assert.Equal(t, configmap.Simple{}, got)
+
+	m["case_insensitive"] = "false"
+	got = testOptions.NonDefault(m)
+	assert.Equal(t, configmap.Simple{}, got)
+
+	m["case_insensitive"] = "true"
+	got = testOptions.NonDefault(m)
+	assert.Equal(t, configmap.Simple{"case_insensitive": "true"}, got)
+}
+
 func TestOptionMarshalJSON(t *testing.T) {
 	out, err := json.MarshalIndent(&caseInsensitiveOption, "", "")
 	assert.NoError(t, err)

fs/sync/sync.go

@@ -155,7 +155,9 @@ func newSyncCopyMove(ctx context.Context, fdst, fsrc fs.Fs, deleteMode fs.Delete
 	// Input context - cancel this for graceful stop
 	s.inCtx, s.inCancel = context.WithCancel(s.ctx)
 	if s.noTraverse && s.deleteMode != fs.DeleteModeOff {
+		if !fi.HaveFilesFrom() {
 			fs.Errorf(nil, "Ignoring --no-traverse with sync")
+		}
 		s.noTraverse = false
 	}
 	s.trackRenamesStrategy, err = parseTrackRenamesStrategy(ci.TrackRenamesStrategy)

fs/version.go

@@ -1,4 +1,4 @@
 package fs
 
 // Version of rclone
-var Version = "v1.55.1-DEV"
+var Version = "v1.56.0-DEV"

go.mod

@@ -53,6 +53,7 @@ require (
 	github.com/rivo/uniseg v0.2.0 // indirect
 	github.com/russross/blackfriday/v2 v2.1.0 // indirect
 	github.com/sevlyar/go-daemon v0.1.5
+	github.com/shirou/gopsutil/v3 v3.21.3
 	github.com/sirupsen/logrus v1.7.0
 	github.com/skratchdot/open-golang v0.0.0-20200116055534-eef842397966
 	github.com/spf13/cobra v1.1.1

go.sum

@@ -67,6 +67,8 @@ github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAE
 github.com/RoaringBitmap/roaring v0.4.7/go.mod h1:8khRDP4HmeXns4xIj9oGrKSz7XTQiJx2zgh7AcNke4w=
 github.com/Shopify/sarama v1.19.0/go.mod h1:FVkBWblsNy7DGZRfXLU0O9RCGt5g3g3yEuWXgklEdEo=
 github.com/Shopify/toxiproxy v2.1.4+incompatible/go.mod h1:OXgGpZ6Cli1/URJOF1DMxUHB2q5Ap20/P/eIdh4G0pI=
+github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d h1:G0m3OIz70MZUWq3EgK3CesDbo8upS2Vm9/P3FtgI+Jk=
+github.com/StackExchange/wmi v0.0.0-20190523213315-cbe66965904d/go.mod h1:3eOhrUMpNV+6aFIbp5/iudMxNCF27Vw2OZgy4xEx0Fg=
 github.com/Unknwon/goconfig v0.0.0-20200908083735-df7de6a44db8 h1:1TrMV1HmBApBbM+Hy7RCKZD6UlYWYIPPfoeXomG7+zE=
 github.com/Unknwon/goconfig v0.0.0-20200908083735-df7de6a44db8/go.mod h1:wngxua9XCNjvHjDiTiV26DaKDT+0c63QR6H5hjVUUxw=
 github.com/VividCortex/gohistogram v1.0.0/go.mod h1:Pf5mBqqDxYaXu3hDrrU+w6nw50o/4+TcAqDqk/vUH7g=
@@ -209,6 +211,8 @@ github.com/go-kit/kit v0.10.0/go.mod h1:xUsJbQ/Fp4kEt7AFgCuvyX4a71u8h9jB8tj/ORgO
 github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
 github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
 github.com/go-logfmt/logfmt v0.5.0/go.mod h1:wCYkCAKZfumFQihp8CzCvQ3paCTfi41vtzG1KdI/P7A=
+github.com/go-ole/go-ole v1.2.4 h1:nNBDSCOigTSiarFpYE9J/KtEA1IOW4CNeqT9TQDqCxI=
+github.com/go-ole/go-ole v1.2.4/go.mod h1:XCwSNxSkXRo4vlyPy93sltvi/qJq0jqQhjqQNIwKuxM=
 github.com/go-sql-driver/mysql v1.4.0/go.mod h1:zAC/RDZ24gD3HViQzih4MyKcchzm+sOG5ZlKdlhCg5w=
 github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
 github.com/gogo/googleapis v1.1.0/go.mod h1:gf4bu3Q80BeJ6H1S1vYPm8/ELATdvryBaNFGgqEef3s=
@@ -502,6 +506,8 @@ github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINE
 github.com/pkg/profile v1.2.1/go.mod h1:hJw3o1OdXxsrSjjVksARp5W95eeEaEfptyVZyv6JUPA=
 github.com/pkg/sftp v1.12.0 h1:/f3b24xrDhkhddlaobPe2JgBqfdt+gC/NYl0QY9IOuI=
 github.com/pkg/sftp v1.12.0/go.mod h1:fUqqXB5vEgVCZ131L+9say31RAri6aF6KDViawhxKK8=
+github.com/pkg/sftp v1.13.0 h1:Riw6pgOKK41foc1I1Uu03CjvbLZDXeGpInycM4shXoI=
+github.com/pkg/sftp v1.13.0/go.mod h1:41g+FIPlQUTDCveupEmEA65IoiQFrtgCeDopC4ajGIM=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
 github.com/posener/complete v1.1.1/go.mod h1:em0nMJCgc9GFtwrmVmEMR/ZL6WyhyjMBndrE9hABlRI=
@@ -558,6 +564,8 @@ github.com/samuel/go-zookeeper v0.0.0-20190923202752-2cc03de413da/go.mod h1:gi+0
 github.com/sean-/seed v0.0.0-20170313163322-e2103e2c3529/go.mod h1:DxrIzT+xaE7yg65j358z/aeFdxmN0P9QXhEzd20vsDc=
 github.com/sevlyar/go-daemon v0.1.5 h1:Zy/6jLbM8CfqJ4x4RPr7MJlSKt90f00kNM1D401C+Qk=
 github.com/sevlyar/go-daemon v0.1.5/go.mod h1:6dJpPatBT9eUwM5VCw9Bt6CdX9Tk6UWvhW3MebLDRKE=
+github.com/shirou/gopsutil/v3 v3.21.3 h1:wgcdAHZS2H6qy4JFewVTtqfiYxFzCeEJod/mLztdPG8=
+github.com/shirou/gopsutil/v3 v3.21.3/go.mod h1:ghfMypLDrFSWN2c9cDYFLHyynQ+QUht0cv/18ZqVczw=
 github.com/shurcooL/sanitized_anchor_name v1.0.0/go.mod h1:1NzhyTcUVG4SuEtjjoZeVRXNmyL/1OwPU0+IJeTBvfc=
 github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
 github.com/sirupsen/logrus v1.4.1/go.mod h1:ni0Sbl8bgC9z8RoU9G6nDWqqs/fq4eDPysMBDgk/93Q=
@@ -610,6 +618,10 @@ github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69
 github.com/t3rm1n4l/go-mega v0.0.0-20200416171014-ffad7fcb44b8 h1:IGJQmLBLYBdAknj21W3JsVof0yjEXfy1Q0K3YZebDOg=
 github.com/t3rm1n4l/go-mega v0.0.0-20200416171014-ffad7fcb44b8/go.mod h1:XWL4vDyd3JKmJx+hZWUVgCNmmhZ2dTBcaNDcxH465s0=
 github.com/tinylib/msgp v1.0.2/go.mod h1:+d+yLhGm8mzTaHzB+wgMYrodPfmZrzkirds8fDWklFE=
+github.com/tklauser/go-sysconf v0.3.4 h1:HT8SVixZd3IzLdfs/xlpq0jeSfTX57g1v6wB1EuzV7M=
+github.com/tklauser/go-sysconf v0.3.4/go.mod h1:Cl2c8ZRWfHD5IrfHo9VN+FX9kCFjIOyVklgXycLB6ek=
+github.com/tklauser/numcpus v0.2.1 h1:ct88eFm+Q7m2ZfXJdan1xYoXKlmwsfP+k88q05KvlZc=
+github.com/tklauser/numcpus v0.2.1/go.mod h1:9aU+wOc6WjUIZEwWMP62PL/41d65P+iks1gBkr4QyP8=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
 github.com/tv42/httpunix v0.0.0-20191220191345-2ba4b9c3382c h1:u6SKchux2yDvFQnDHS3lPnIRmfVJ5Sxy3ao2SIdysLQ=
@@ -849,6 +861,7 @@ golang.org/x/sys v0.0.0-20210104204734-6f8348627aad/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20210112091331-59c308dcf3cc/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210119212857-b64e53b001e4/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210217105451-b926d437f341/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210313110737-8e9fff1a3a18 h1:jxr7/dEo+rR29uEBoLSWJ1tRHCFAMwFbGUU9nRqzpds=
 golang.org/x/sys v0.0.0-20210313110737-8e9fff1a3a18/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=

lib/buildinfo/osversion.go

@@ -0,0 +1,34 @@
+// +build !openbsd,!windows
+
+package buildinfo
+
+import (
+	"strings"
+
+	"github.com/shirou/gopsutil/v3/host"
+)
+
+// GetOSVersion returns OS version, kernel and bitness
+func GetOSVersion() (osVersion, osKernel string) {
+	if platform, _, version, err := host.PlatformInformation(); err == nil && platform != "" {
+		osVersion = platform
+		if version != "" {
+			osVersion += " " + version
+		}
+	}
+
+	if version, err := host.KernelVersion(); err == nil && version != "" {
+		osKernel = version
+	}
+
+	if arch, err := host.KernelArch(); err == nil && arch != "" {
+		if strings.HasSuffix(arch, "64") && osVersion != "" {
+			osVersion += " (64 bit)"
+		}
+		if osKernel != "" {
+			osKernel += " (" + arch + ")"
+		}
+	}
+
+	return
+}

lib/buildinfo/osversion_openbsd.go

@@ -0,0 +1,13 @@
+// +build openbsd
+
+package buildinfo
+
+// gopsutil v3.21.3 fails to build on openbsd:
+// Error: .../go/pkg/mod/github.com/tklauser/go-sysconf@v0.3.4/sysconf_openbsd.go:22:28: undefined: unix.RLIMIT_NPROC
+// Error: .../go/pkg/mod/github.com/shirou/gopsutil/v3@v3.21.3/process/process.go:163:15: undefined: pidsWithContext
+// and so on...
+
+// GetOSVersion returns OS version, kernel and bitness
+func GetOSVersion() (osVersion, osKernel string) {
+	return "OpenBSD", ""
+}

lib/buildinfo/osversion_windows.go

@@ -0,0 +1,131 @@
+// +build !openbsd !windows
+
+package buildinfo
+
+import (
+	"fmt"
+	"regexp"
+	"strings"
+	"unsafe"
+
+	"github.com/shirou/gopsutil/v3/host"
+	"golang.org/x/sys/windows"
+	"golang.org/x/sys/windows/registry"
+)
+
+// GetOSVersion returns OS version, kernel and bitness
+// On Windows it performs additional output enhancements.
+func GetOSVersion() (osVersion, osKernel string) {
+	if platform, _, version, err := host.PlatformInformation(); err == nil && platform != "" {
+		osVersion = platform
+		if version != "" {
+			osVersion += " " + version
+		}
+	}
+
+	if version, err := host.KernelVersion(); err == nil && version != "" {
+		osKernel = version
+
+		// Prevent duplication of output on Windows
+		if strings.Contains(osVersion, osKernel) {
+			deduped := strings.TrimSpace(strings.Replace(osVersion, osKernel, "", 1))
+			if deduped != "" {
+				osVersion = deduped
+			}
+		}
+
+		// Simplify kernel output: `RELEASE.BUILD Build BUILD` -> `RELEASE.BUILD`
+		match := regexp.MustCompile(`^([\d\.]+?\.)(\d+) Build (\d+)$`).FindStringSubmatch(osKernel)
+		if len(match) == 4 && match[2] == match[3] {
+			osKernel = match[1] + match[2]
+		}
+	}
+
+	friendlyName := getRegistryVersionString("ReleaseId")
+	if osVersion != "" && friendlyName != "" {
+		osVersion += " " + friendlyName
+	}
+
+	updateRevision := getRegistryVersionInt("UBR")
+	if osKernel != "" && updateRevision != 0 {
+		osKernel += fmt.Sprintf(".%d", updateRevision)
+	}
+
+	if arch, err := host.KernelArch(); err == nil && arch != "" {
+		if strings.HasSuffix(arch, "64") && osVersion != "" {
+			osVersion += " (64 bit)"
+		}
+		if osKernel != "" {
+			osKernel += " (" + arch + ")"
+		}
+	}
+
+	return
+}
+
+var regVersionKeyUTF16 = windows.StringToUTF16Ptr(`SOFTWARE\Microsoft\Windows NT\CurrentVersion`)
+
+func getRegistryVersionString(name string) string {
+	var (
+		err     error
+		handle  windows.Handle
+		bufLen  uint32
+		valType uint32
+	)
+
+	err = windows.RegOpenKeyEx(windows.HKEY_LOCAL_MACHINE, regVersionKeyUTF16, 0, windows.KEY_READ|windows.KEY_WOW64_64KEY, &handle)
+	if err != nil {
+		return ""
+	}
+	defer func() {
+		_ = windows.RegCloseKey(handle)
+	}()
+
+	nameUTF16 := windows.StringToUTF16Ptr(name)
+	err = windows.RegQueryValueEx(handle, nameUTF16, nil, &valType, nil, &bufLen)
+	if err != nil {
+		return ""
+	}
+
+	regBuf := make([]uint16, bufLen/2+1)
+	err = windows.RegQueryValueEx(handle, nameUTF16, nil, &valType, (*byte)(unsafe.Pointer(&regBuf[0])), &bufLen)
+	if err != nil {
+		return ""
+	}
+
+	return windows.UTF16ToString(regBuf[:])
+}
+
+func getRegistryVersionInt(name string) int {
+	var (
+		err     error
+		handle  windows.Handle
+		bufLen  uint32
+		valType uint32
+	)
+
+	err = windows.RegOpenKeyEx(windows.HKEY_LOCAL_MACHINE, regVersionKeyUTF16, 0, windows.KEY_READ|windows.KEY_WOW64_64KEY, &handle)
+	if err != nil {
+		return 0
+	}
+	defer func() {
+		_ = windows.RegCloseKey(handle)
+	}()
+
+	nameUTF16 := windows.StringToUTF16Ptr(name)
+	err = windows.RegQueryValueEx(handle, nameUTF16, nil, &valType, nil, &bufLen)
+	if err != nil {
+		return 0
+	}
+
+	if valType != registry.DWORD || bufLen != 4 {
+		return 0
+	}
+	var val32 uint32
+	err = windows.RegQueryValueEx(handle, nameUTF16, nil, &valType, (*byte)(unsafe.Pointer(&val32)), &bufLen)
+	if err != nil {
+		return 0
+	}
+
+	return int(val32)
+}

lib/oauthutil/oauthutil.go

@@ -447,19 +447,46 @@ Execute the following on the machine with the web browser (same rclone
 version recommended):
 
 `)
-			if changed {
+			// Find the configuration
-				fmt.Printf("\trclone authorize %q -- %q %q\n", id, oauthConfig.ClientID, oauthConfig.ClientSecret)
+			ri, err := fs.Find(id)
+			if err != nil {
+				return errors.Wrap(err, "oauthutil authorize")
+			}
+			// Find the overridden options
+			inM := ri.Options.NonDefault(m)
+			delete(inM, config.ConfigToken) // delete token as we are refreshing it
+			for k, v := range inM {
+				fs.Debugf(nil, "sending %s = %q", k, v)
+			}
+			// Encode them into a string
+			mCopyString, err := inM.Encode()
+			if err != nil {
+				return errors.Wrap(err, "oauthutil authorize encode")
+			}
+			// Write what the user has to do
+			if len(mCopyString) > 0 {
+				fmt.Printf("\trclone authorize %q %q\n", id, mCopyString)
 			} else {
 				fmt.Printf("\trclone authorize %q\n", id)
 			}
 			fmt.Println("\nThen paste the result below:")
+			// Read the updates to the config
+			var outM configmap.Simple
+			for {
+				outM = configmap.Simple{}
 				code := config.ReadNonEmptyLine("result> ")
-			token := &oauth2.Token{}
+				err = outM.Decode(code)
-			err := json.Unmarshal([]byte(code), token)
+				if err == nil {
-			if err != nil {
+					break
-				return err
 				}
-			return PutToken(name, m, token, true)
+				fmt.Printf("Couldn't decode response - try again (make sure you are using a matching version of rclone on both sides: %v\n", err)
+			}
+			// Save the config updates
+			for k, v := range outM {
+				m.Set(k, v)
+				fs.Debugf(nil, "received %s = %q", k, v)
+			}
+			return nil
 		}
 	}
 
@@ -526,14 +553,6 @@ version recommended):
 		return errors.Wrap(err, "failed to get token")
 	}
 
-	// Print code if we are doing a manual auth
-	if authorizeOnly {
-		result, err := json.Marshal(token)
-		if err != nil {
-			return errors.Wrap(err, "failed to marshal token")
-		}
-		fmt.Printf("Paste the following into your remote machine --->\n%s\n<---End paste\n", result)
-	}
 	return PutToken(name, m, token, true)
 }
 

rclone.1

@@ -1,7 +1,7 @@
 .\"t
 .\" Automatically generated by Pandoc 2.5
 .\"
-.TH "rclone" "1" "Apr 26, 2021" "User Manual" ""
+.TH "rclone" "1" "Mar 31, 2021" "User Manual" ""
 .hy
 .SH Rclone syncs your files to cloud storage
 .PP
@@ -3677,7 +3677,7 @@ this as the file being accessible by everyone.
 For example an SSH client may warn about \[dq]unprotected private key
 file\[dq].
 .PP
-WinFsp 2021 (version 1.9) introduces a new FUSE option
+WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option
 \[dq]FileSecurity\[dq], that allows the complete specification of file
 security descriptors using
 SDDL (https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
@@ -4632,7 +4632,7 @@ Beta releases have an additional information similar to
 \f[C]v1.54.0\-beta.5111.06f1c0c61\f[R].
 (if you are a developer and use a locally built rclone, the version
 number will end with \f[C]\-DEV\f[R], you will have to rebuild it as it
-obviously can\[aq]t be distributed).
+obvisously can\[aq]t be distributed).
 .PP
 If you previously installed rclone via a package manager, the package
 may include local documentation or configure services.
@@ -8720,7 +8720,7 @@ If you run \f[C]rclone config file\f[R] you will see where the default
 location is for you.
 .PP
 Use this flag to override the config location, e.g.
-\f[C]rclone \-\-config=\[dq].myconfig\[dq] config\f[R].
+\f[C]rclone \-\-config=\[dq].myconfig\[dq] .config\f[R].
 .PP
 If the location is set to empty string \f[C]\[dq]\[dq]\f[R] or the
 special value \f[C]/notfound\f[R], or the os null device represented by
@@ -15925,7 +15925,7 @@ These flags are available for every command.
       \-\-use\-json\-log                         Use json log format.
       \-\-use\-mmap                             Use mmap allocator (see docs).
       \-\-use\-server\-modtime                   Use server modified time instead of object metadata
-      \-\-user\-agent string                    Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.55.1\[dq])
+      \-\-user\-agent string                    Set the user\-agent to a specified string. The default is rclone/ version (default \[dq]rclone/v1.55.0\[dq])
   \-v, \-\-verbose count                        Print lots more stuff (repeat for more)
 \f[R]
 .fi
@@ -39340,105 +39340,6 @@ Options:
 .IP \[bu] 2
 \[dq]error\[dq]: return an error based on option value
 .SH Changelog
-.SS v1.55.1 \- 2021\-04\-26
-.PP
-See commits (https://github.com/rclone/rclone/compare/v1.55.0...v1.55.1)
-.IP \[bu] 2
-Bug Fixes
-.RS 2
-.IP \[bu] 2
-selfupdate
-.RS 2
-.IP \[bu] 2
-Dont detect FUSE if build is static (Ivan Andreev)
-.IP \[bu] 2
-Add build tag noselfupdate (Ivan Andreev)
-.RE
-.IP \[bu] 2
-sync: Fix incorrect error reported by graceful cutoff (Nick Craig\-Wood)
-.IP \[bu] 2
-install.sh: fix macOS arm64 download (Nick Craig\-Wood)
-.IP \[bu] 2
-build: Fix version numbers in android branch builds (Nick Craig\-Wood)
-.IP \[bu] 2
-docs
-.RS 2
-.IP \[bu] 2
-Contributing.md: update setup instructions for go1.16 (Nick Gaya)
-.IP \[bu] 2
-WinFsp 2021 is out of beta (albertony)
-.IP \[bu] 2
-Minor cleanup of space around code section (albertony)
-.IP \[bu] 2
-Fixed some typos (albertony)
-.RE
-.RE
-.IP \[bu] 2
-VFS
-.RS 2
-.IP \[bu] 2
-Fix a code path which allows dirty data to be removed causing data loss
-(Nick Craig\-Wood)
-.RE
-.IP \[bu] 2
-Compress
-.RS 2
-.IP \[bu] 2
-Fix compressed name regexp (buengese)
-.RE
-.IP \[bu] 2
-Drive
-.RS 2
-.IP \[bu] 2
-Fix backend copyid of google doc to directory (Nick Craig\-Wood)
-.IP \[bu] 2
-Don\[aq]t open browser when service account...
-(Ansh Mittal)
-.RE
-.IP \[bu] 2
-Dropbox
-.RS 2
-.IP \[bu] 2
-Add missing team_data.member scope for use with \-\-impersonate (Nick
-Craig\-Wood)
-.IP \[bu] 2
-Fix About after scopes changes \- rclone config reconnect needed (Nick
-Craig\-Wood)
-.IP \[bu] 2
-Fix Unable to decrypt returned paths from changeNotify (Nick
-Craig\-Wood)
-.RE
-.IP \[bu] 2
-FTP
-.RS 2
-.IP \[bu] 2
-Fix implicit TLS (Ivan Andreev)
-.RE
-.IP \[bu] 2
-Onedrive
-.RS 2
-.IP \[bu] 2
-Work around for random \[dq]Unable to initialize RPS\[dq] errors
-(OleFrost)
-.RE
-.IP \[bu] 2
-SFTP
-.RS 2
-.IP \[bu] 2
-Revert sftp library to v1.12.0 from v1.13.0 to fix performance
-regression (Nick Craig\-Wood)
-.IP \[bu] 2
-Fix Update ReadFrom failed: failed to send packet: EOF errors (Nick
-Craig\-Wood)
-.RE
-.IP \[bu] 2
-Zoho
-.RS 2
-.IP \[bu] 2
-Fix error when region isn\[aq]t set (buengese)
-.IP \[bu] 2
-Do not ask for mountpoint twice when using headless setup (buengese)
-.RE
 .SS v1.55.0 \- 2021\-03\-31
 .PP
 See commits (https://github.com/rclone/rclone/compare/v1.54.0...v1.55.0)

vfs/vfscache/item.go

@@ -774,9 +774,6 @@ func (item *Item) reload(ctx context.Context) error {
 // check the fingerprint of an object and update the item or delete
 // the cached file accordingly
 //
-// If we have local modifications then they take precedence
-// over a change in the remote
-//
 // It ensures the file is the correct size for the object
 //
 // call with lock held
@@ -784,12 +781,8 @@ func (item *Item) _checkObject(o fs.Object) error {
 	if o == nil {
 		if item.info.Fingerprint != "" {
 			// no remote object && local object
-			// remove local object unless dirty
+			// remove local object
-			if !item.info.Dirty {
 			item._remove("stale (remote deleted)")
-			} else {
-				fs.Debugf(item.name, "vfs cache: remote object has gone but local object modified - keeping it")
-			}
 		} else {
 			// no remote object && no local object
 			// OK
@@ -800,12 +793,8 @@ func (item *Item) _checkObject(o fs.Object) error {
 		if item.info.Fingerprint != "" {
 			// remote object && local object
 			if remoteFingerprint != item.info.Fingerprint {
-				if !item.info.Dirty {
 				fs.Debugf(item.name, "vfs cache: removing cached entry as stale (remote fingerprint %q != cached fingerprint %q)", remoteFingerprint, item.info.Fingerprint)
 				item._remove("stale (remote is different)")
-				} else {
-					fs.Debugf(item.name, "vfs cache: remote object has changed but local object modified - keeping it (remote fingerprint %q != cached fingerprint %q)", remoteFingerprint, item.info.Fingerprint)
-				}
 			}
 		} else {
 			// remote object && no local object