webdav: don't delete files on error - fixes #3263

Previous to this change rclone deleted a file if the upload failed
half way through.

After this change we only delete the upload if the
--webdav-delete-on-error flag is set.

We set this in the serve webdav tests to make them pass.

.github/workflows/build.yml

@@ -19,12 +19,12 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        job_name: ['linux', 'mac_amd64', 'mac_arm64', 'windows_amd64', 'windows_386', 'other_os', 'go1.13', 'go1.14', 'go1.15']
+        job_name: ['linux', 'mac', 'windows_amd64', 'windows_386', 'other_os', 'go1.12', 'go1.13', 'go1.14', 'go1.16']
 
         include:
           - job_name: linux
             os: ubuntu-latest
-            go: '1.16.x'
+            go: '1.15.x'
             gotags: cmount
             build_flags: '-include "^linux/"'
             check: true
@@ -32,25 +32,18 @@ jobs:
             racequicktest: true
             deploy: true
 
-          - job_name: mac_amd64
+          - job_name: mac
             os: macOS-latest
-            go: '1.16.x'
+            go: '1.15.x'
             gotags: 'cmount'
             build_flags: '-include "^darwin/amd64" -cgo'
             quicktest: true
             racequicktest: true
             deploy: true
 
-          - job_name: mac_arm64
-            os: macOS-latest
-            go: '1.16.x'
-            gotags: 'cmount'
-            build_flags: '-include "^darwin/arm64" -cgo -macos-arch arm64 -macos-sdk macosx11.1 -cgo-cflags=-I/usr/local/include -cgo-ldflags=-L/usr/local/lib'
-            deploy: true
-
           - job_name: windows_amd64
             os: windows-latest
-            go: '1.16.x'
+            go: '1.15.x'
             gotags: cmount
             build_flags: '-include "^windows/amd64" -cgo'
             build_args: '-buildmode exe'
@@ -60,7 +53,7 @@ jobs:
 
           - job_name: windows_386
             os: windows-latest
-            go: '1.16.x'
+            go: '1.15.x'
             gotags: cmount
             goarch: '386'
             cgo: '1'
@@ -71,11 +64,16 @@ jobs:
 
           - job_name: other_os
             os: ubuntu-latest
-            go: '1.16.x'
-            build_flags: '-exclude "^(windows/|darwin/|linux/)"'
+            go: '1.15.x'
+            build_flags: '-exclude "^(windows/|darwin/amd64|linux/)"'
             compile_all: true
             deploy: true
 
+          - job_name: go1.12
+            os: ubuntu-latest
+            go: '1.12.x'
+            quicktest: true
+
           - job_name: go1.13
             os: ubuntu-latest
             go: '1.13.x'
@@ -87,9 +85,9 @@ jobs:
             quicktest: true
             racequicktest: true
 
-          - job_name: go1.15
+          - job_name: go1.16
             os: ubuntu-latest
-            go: '1.15.x'
+            go: '1.16.0-beta1'
             quicktest: true
             racequicktest: true
 
@@ -131,7 +129,7 @@ jobs:
         shell: bash
         run: |
           brew update
-          brew install --cask macfuse
+          brew install --cask osxfuse
         if: matrix.os == 'macOS-latest'
 
       - name: Install Libraries on Windows

CONTRIBUTING.md

@@ -72,7 +72,7 @@ Make sure you
 
 When you are done with that
 
-    git push -u origin my-new-feature
+    git push origin my-new-feature
 
 Go to the GitHub website and click [Create pull
 request](https://help.github.com/articles/creating-a-pull-request/).
@@ -99,7 +99,7 @@ rclone's tests are run from the go testing framework, so at the top
 level you can run this to run all the tests.
 
     go test -v ./...
-
+    
 rclone contains a mixture of unit tests and integration tests.
 Because it is difficult (and in some respects pointless) to test cloud
 storage systems by mocking all their interfaces, rclone unit tests can
@@ -115,8 +115,8 @@ are skipped if `TestDrive:` isn't defined.
     cd backend/drive
     go test -v
 
-You can then run the integration tests which test all of rclone's
-operations.  Normally these get run against the local file system,
+You can then run the integration tests which tests all of rclone's
+operations.  Normally these get run against the local filing system,
 but they can be run against any of the remotes.
 
     cd fs/sync
@@ -127,7 +127,7 @@ but they can be run against any of the remotes.
     go test -v -remote TestDrive:
 
 If you want to use the integration test framework to run these tests
-altogether with an HTML report and test retries then from the
+all together with an HTML report and test retries then from the
 project root:
 
     go install github.com/rclone/rclone/fstest/test_all
@@ -202,7 +202,7 @@ for the flag help, the remainder is shown to the user in `rclone
 config` and is added to the docs with `make backenddocs`.
 
 The only documentation you need to edit are the `docs/content/*.md`
-files.  The `MANUAL.*`, `rclone.1`, web site, etc. are all auto generated
+files.  The MANUAL.*, rclone.1, web site, etc. are all auto generated
 from those during the release process.  See the `make doc` and `make
 website` targets in the Makefile if you are interested in how.  You
 don't need to run these when adding a feature.
@@ -265,7 +265,7 @@ rclone uses the [go
 modules](https://tip.golang.org/cmd/go/#hdr-Modules__module_versions__and_more)
 support in go1.11 and later to manage its dependencies.
 
-rclone can be built with modules outside of the `GOPATH`.
+rclone can be built with modules outside of the GOPATH
 
 To add a dependency `github.com/ncw/new_dependency` see the
 instructions below.  These will fetch the dependency and add it to
@@ -333,8 +333,8 @@ Getting going
   * Try to implement as many optional methods as possible as it makes the remote more usable.
   * Use lib/encoder to make sure we can encode any path name and `rclone info` to help determine the encodings needed
     * `rclone purge -v TestRemote:rclone-info`
-    * `rclone test info --all --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
-    * `go run cmd/test/info/internal/build_csv/main.go -o remote.csv remote.json`
+    * `rclone info --remote-encoding None -vv --write-json remote.json TestRemote:rclone-info`
+    * `go run cmd/info/internal/build_csv/main.go -o remote.csv remote.json`
     * open `remote.csv` in a spreadsheet and examine
 
 Unit tests
@@ -400,7 +400,7 @@ Usage
    - If this variable doesn't exist, plugin support is disabled.
    - Plugins must be compiled against the exact version of rclone to work.
      (The rclone used during building the plugin must be the same as the source of rclone)
-
+     
 Building
 
 To turn your existing additions into a Go plugin, move them to an external repository

Dockerfile

@@ -16,8 +16,6 @@ RUN apk --no-cache add ca-certificates fuse tzdata && \
 
 COPY --from=builder /go/src/github.com/rclone/rclone/rclone /usr/local/bin/
 
-RUN addgroup -g 1009 rclone && adduser -u 1009 -Ds /bin/sh -G rclone rclone
-
 ENTRYPOINT [ "rclone" ]
 
 WORKDIR /data

Makefile

@@ -93,7 +93,8 @@ build_dep:
 
 # Get the release dependencies we only install on linux
 release_dep_linux:
-	cd /tmp && go get github.com/goreleaser/nfpm/v2/...
+	cd /tmp && go get github.com/goreleaser/nfpm/...
+	cd /tmp && go get github.com/github-release/github-release
 
 # Get the release dependencies we only install on Windows
 release_dep_windows:

RELEASE.md

@@ -4,7 +4,7 @@ This file describes how to make the various kinds of releases
 
 ## Extra required software for making a release
 
-  * [gh the github cli](https://github.com/cli/cli) for uploading packages
+  * [github-release](https://github.com/aktau/github-release) for uploading packages
   * pandoc for making the html and man pages
 
 ## Making a release

VERSION

@@ -1 +1 @@
-v1.55.0
+v1.54.0

backend/azureblob/azureblob.go

@@ -1,6 +1,6 @@
 // Package azureblob provides an interface to the Microsoft Azure blob object storage system
 
-// +build !plan9,!solaris,!js,go1.14
+// +build !plan9,!solaris,!js,go1.13
 
 package azureblob
 
@@ -49,7 +49,6 @@ const (
 	storageDefaultBaseURL = "blob.core.windows.net"
 	defaultChunkSize      = 4 * fs.MebiByte
 	maxChunkSize          = 100 * fs.MebiByte
-	uploadConcurrency     = 4
 	defaultAccessTier     = azblob.AccessTierNone
 	maxTryTimeout         = time.Hour * 24 * 365 //max time of an azure web request response window (whether or not data is flowing)
 	// Default storage account, key and blob endpoint for emulator support,
@@ -1493,48 +1492,6 @@ func increment(xs []byte) {
 	}
 }
 
-// poolWrapper wraps a pool.Pool as an azblob.TransferManager
-type poolWrapper struct {
-	pool     *pool.Pool
-	bufToken chan struct{}
-	runToken chan struct{}
-}
-
-// newPoolWrapper creates an azblob.TransferManager that will use a
-// pool.Pool with maximum concurrency as specified.
-func (f *Fs) newPoolWrapper(concurrency int) azblob.TransferManager {
-	return &poolWrapper{
-		pool:     f.pool,
-		bufToken: make(chan struct{}, concurrency),
-		runToken: make(chan struct{}, concurrency),
-	}
-}
-
-// Get implements TransferManager.Get().
-func (pw *poolWrapper) Get() []byte {
-	pw.bufToken <- struct{}{}
-	return pw.pool.Get()
-}
-
-// Put implements TransferManager.Put().
-func (pw *poolWrapper) Put(b []byte) {
-	pw.pool.Put(b)
-	<-pw.bufToken
-}
-
-// Run implements TransferManager.Run().
-func (pw *poolWrapper) Run(f func()) {
-	pw.runToken <- struct{}{}
-	go func() {
-		f()
-		<-pw.runToken
-	}()
-}
-
-// Close implements TransferManager.Close().
-func (pw *poolWrapper) Close() {
-}
-
 // Update the object with the contents of the io.Reader, modTime and size
 //
 // The new object may have been created if an error is returned
@@ -1581,10 +1538,9 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 
 	putBlobOptions := azblob.UploadStreamToBlockBlobOptions{
 		BufferSize:      int(o.fs.opt.ChunkSize),
-		MaxBuffers:      uploadConcurrency,
+		MaxBuffers:      4,
 		Metadata:        o.meta,
 		BlobHTTPHeaders: httpHeaders,
-		TransferManager: o.fs.newPoolWrapper(uploadConcurrency),
 	}
 
 	// Don't retry, return a retry error instead

backend/azureblob/azureblob_internal_test.go

@@ -1,4 +1,4 @@
-// +build !plan9,!solaris,!js,go1.14
+// +build !plan9,!solaris,!js,go1.13
 
 package azureblob
 

backend/azureblob/azureblob_test.go

@@ -1,6 +1,6 @@
 // Test AzureBlob filesystem interface
 
-// +build !plan9,!solaris,!js,go1.14
+// +build !plan9,!solaris,!js,go1.13
 
 package azureblob
 

backend/azureblob/azureblob_unsupported.go

@@ -1,6 +1,6 @@
 // Build for azureblob for unsupported platforms to stop go complaining
 // about "no buildable Go source files "
 
-// +build plan9 solaris js !go1.14
+// +build plan9 solaris js !go1.13
 
 package azureblob

backend/azureblob/imds.go

@@ -1,4 +1,4 @@
-// +build !plan9,!solaris,!js,go1.14
+// +build !plan9,!solaris,!js,go1.13
 
 package azureblob
 

backend/azureblob/imds_test.go

@@ -1,4 +1,4 @@
-// +build !plan9,!solaris,!js,go1.14
+// +build !plan9,!solaris,!js,go1.13
 
 package azureblob
 

backend/b2/b2.go

@@ -403,10 +403,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 	if err != nil {
 		return nil, err
 	}
-	if opt.UploadCutoff < opt.ChunkSize {
-		opt.UploadCutoff = opt.ChunkSize
-		fs.Infof(nil, "b2: raising upload cutoff to chunk size: %v", opt.UploadCutoff)
-	}
 	err = checkUploadCutoff(opt, opt.UploadCutoff)
 	if err != nil {
 		return nil, errors.Wrap(err, "b2: upload cutoff")
@@ -479,9 +475,12 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		f.setRoot(newRoot)
 		_, err := f.NewObject(ctx, leaf)
 		if err != nil {
-			// File doesn't exist so return old f
-			f.setRoot(oldRoot)
-			return f, nil
+			if err == fs.ErrorObjectNotFound {
+				// File doesn't exist so return old f
+				f.setRoot(oldRoot)
+				return f, nil
+			}
+			return nil, err
 		}
 		// return an error with an fs which points to the parent
 		return f, fs.ErrorIsFile
@@ -709,7 +708,7 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
 			remote := file.Name[len(prefix):]
 			// Check for directory
 			isDirectory := remote == "" || strings.HasSuffix(remote, "/")
-			if isDirectory && len(remote) > 1 {
+			if isDirectory {
 				remote = remote[:len(remote)-1]
 			}
 			if addBucket {

backend/box/box.go

@@ -514,7 +514,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
 	// Find the leaf in pathID
 	found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
-		if strings.EqualFold(item.Name, leaf) {
+		if item.Name == leaf {
 			pathIDOut = item.ID
 			return true
 		}

backend/cache/cache_internal_test.go

@@ -1034,7 +1034,7 @@ func (r *run) updateObjectRemote(t *testing.T, f fs.Fs, remote string, data1 []b
 	objInfo1 := object.NewStaticObjectInfo(remote, time.Now(), int64(len(data1)), true, nil, f)
 	objInfo2 := object.NewStaticObjectInfo(remote, time.Now(), int64(len(data2)), true, nil, f)
 
-	_, err = f.Put(context.Background(), in1, objInfo1)
+	obj, err = f.Put(context.Background(), in1, objInfo1)
 	require.NoError(t, err)
 	obj, err = f.NewObject(context.Background(), remote)
 	require.NoError(t, err)

backend/chunker/chunker.go

@@ -47,8 +47,7 @@ import (
 // The following types of chunks are supported:
 // data and control, active and temporary.
 // Chunk type is identified by matching chunk file name
-// based on the chunk name format configured by user and transaction
-// style being used.
+// based on the chunk name format configured by user.
 //
 // Both data and control chunks can be either temporary (aka hidden)
 // or active (non-temporary aka normal aka permanent).
@@ -64,12 +63,6 @@ import (
 // which is transparently converted to the new format. In its maximum
 // length of 13 decimals it makes a 7-digit base-36 number.
 //
-// When transactions is set to the norename style, data chunks will
-// keep their temporary chunk names (with the transacion identifier
-// suffix). To distinguish them from temporary chunks, the txn field
-// of the metadata file is set to match the transaction identifier of
-// the data chunks.
-//
 // Chunker can tell data chunks from control chunks by the characters
 // located in the "hash placeholder" position of configured format.
 // Data chunks have decimal digits there.
@@ -108,7 +101,7 @@ const maxMetadataSize = 1023
 const maxMetadataSizeWritten = 255
 
 // Current/highest supported metadata format.
-const metadataVersion = 2
+const metadataVersion = 1
 
 // optimizeFirstChunk enables the following optimization in the Put:
 // If a single chunk is expected, put the first chunk using the
@@ -231,31 +224,6 @@ It has the following fields: ver, size, nchunks, md5, sha1.`,
 					Help:  "Warn user, skip incomplete file and proceed.",
 				},
 			},
-		}, {
-			Name:     "transactions",
-			Advanced: true,
-			Default:  "rename",
-			Help:     `Choose how chunker should handle temporary files during transactions.`,
-			Hide:     fs.OptionHideCommandLine,
-			Examples: []fs.OptionExample{
-				{
-					Value: "rename",
-					Help:  "Rename temporary files after a successful transaction.",
-				}, {
-					Value: "norename",
-					Help: `Leave temporary file names and write transaction ID to metadata file.
-Metadata is required for no rename transactions (meta format cannot be "none").
-If you are using norename transactions you should be careful not to downgrade Rclone
-as older versions of Rclone don't support this transaction style and will misinterpret
-files manipulated by norename transactions.
-This method is EXPERIMENTAL, don't use on production systems.`,
-				}, {
-					Value: "auto",
-					Help: `Rename or norename will be used depending on capabilities of the backend.
-If meta format is set to "none", rename transactions will always be used.
-This method is EXPERIMENTAL, don't use on production systems.`,
-				},
-			},
 		}},
 	})
 }
@@ -303,7 +271,7 @@ func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs,
 	cache.PinUntilFinalized(f.base, f)
 	f.dirSort = true // processEntries requires that meta Objects prerun data chunks atm.
 
-	if err := f.configure(opt.NameFormat, opt.MetaFormat, opt.HashType, opt.Transactions); err != nil {
+	if err := f.configure(opt.NameFormat, opt.MetaFormat, opt.HashType); err != nil {
 		return nil, err
 	}
 
@@ -341,14 +309,13 @@ func NewFs(ctx context.Context, name, rpath string, m configmap.Mapper) (fs.Fs,
 
 // Options defines the configuration for this backend
 type Options struct {
-	Remote       string        `config:"remote"`
-	ChunkSize    fs.SizeSuffix `config:"chunk_size"`
-	NameFormat   string        `config:"name_format"`
-	StartFrom    int           `config:"start_from"`
-	MetaFormat   string        `config:"meta_format"`
-	HashType     string        `config:"hash_type"`
-	FailHard     bool          `config:"fail_hard"`
-	Transactions string        `config:"transactions"`
+	Remote     string        `config:"remote"`
+	ChunkSize  fs.SizeSuffix `config:"chunk_size"`
+	NameFormat string        `config:"name_format"`
+	StartFrom  int           `config:"start_from"`
+	MetaFormat string        `config:"meta_format"`
+	HashType   string        `config:"hash_type"`
+	FailHard   bool          `config:"fail_hard"`
 }
 
 // Fs represents a wrapped fs.Fs
@@ -370,13 +337,12 @@ type Fs struct {
 	opt          Options        // copy of Options
 	features     *fs.Features   // optional features
 	dirSort      bool           // reserved for future, ignored
-	useNoRename  bool           // can be set with the transactions option
 }
 
 // configure sets up chunker for given name format, meta format and hash type.
 // It also seeds the source of random transaction identifiers.
 // configure must be called only from NewFs or by unit tests.
-func (f *Fs) configure(nameFormat, metaFormat, hashType, transactionMode string) error {
+func (f *Fs) configure(nameFormat, metaFormat, hashType string) error {
 	if err := f.setChunkNameFormat(nameFormat); err != nil {
 		return errors.Wrapf(err, "invalid name format '%s'", nameFormat)
 	}
@@ -386,9 +352,6 @@ func (f *Fs) configure(nameFormat, metaFormat, hashType, transactionMode string)
 	if err := f.setHashType(hashType); err != nil {
 		return err
 	}
-	if err := f.setTransactionMode(transactionMode); err != nil {
-		return err
-	}
 
 	randomSeed := time.Now().UnixNano()
 	f.xactIDRand = rand.New(rand.NewSource(randomSeed))
@@ -448,27 +411,6 @@ func (f *Fs) setHashType(hashType string) error {
 	return nil
 }
 
-func (f *Fs) setTransactionMode(transactionMode string) error {
-	switch transactionMode {
-	case "rename":
-		f.useNoRename = false
-	case "norename":
-		if !f.useMeta {
-			return errors.New("incompatible transaction options")
-		}
-		f.useNoRename = true
-	case "auto":
-		f.useNoRename = !f.CanQuickRename()
-		if f.useNoRename && !f.useMeta {
-			f.useNoRename = false
-			return errors.New("using norename transactions requires metadata")
-		}
-	default:
-		return fmt.Errorf("unsupported transaction mode '%s'", transactionMode)
-	}
-	return nil
-}
-
 // setChunkNameFormat converts pattern based chunk name format
 // into Printf format and Regular expressions for data and
 // control chunks.
@@ -751,7 +693,6 @@ func (f *Fs) processEntries(ctx context.Context, origEntries fs.DirEntries, dirP
 	byRemote := make(map[string]*Object)
 	badEntry := make(map[string]bool)
 	isSubdir := make(map[string]bool)
-	txnByRemote := map[string]string{}
 
 	var tempEntries fs.DirEntries
 	for _, dirOrObject := range sortedEntries {
@@ -764,18 +705,12 @@ func (f *Fs) processEntries(ctx context.Context, origEntries fs.DirEntries, dirP
 				object := f.newObject("", entry, nil)
 				byRemote[remote] = object
 				tempEntries = append(tempEntries, object)
-				if f.useNoRename {
-					txnByRemote[remote], err = object.readXactID(ctx)
-					if err != nil {
-						return nil, err
-					}
-				}
 				break
 			}
 			// this is some kind of chunk
 			// metobject should have been created above if present
+			isSpecial := xactID != "" || ctrlType != ""
 			mainObject := byRemote[mainRemote]
-			isSpecial := xactID != txnByRemote[mainRemote] || ctrlType != ""
 			if mainObject == nil && f.useMeta && !isSpecial {
 				fs.Debugf(f, "skip orphan data chunk %q", remote)
 				break
@@ -874,11 +809,10 @@ func (f *Fs) scanObject(ctx context.Context, remote string, quickScan bool) (fs.
 	}
 
 	var (
-		o             *Object
-		baseObj       fs.Object
-		currentXactID string
-		err           error
-		sameMain      bool
+		o        *Object
+		baseObj  fs.Object
+		err      error
+		sameMain bool
 	)
 
 	if f.useMeta {
@@ -922,14 +856,7 @@ func (f *Fs) scanObject(ctx context.Context, remote string, quickScan bool) (fs.
 		return nil, errors.Wrap(err, "can't detect composite file")
 	}
 
-	if f.useNoRename {
-		currentXactID, err = o.readXactID(ctx)
-		if err != nil {
-			return nil, err
-		}
-	}
 	caseInsensitive := f.features.CaseInsensitive
-
 	for _, dirOrObject := range entries {
 		entry, ok := dirOrObject.(fs.Object)
 		if !ok {
@@ -951,7 +878,7 @@ func (f *Fs) scanObject(ctx context.Context, remote string, quickScan bool) (fs.
 		if !sameMain {
 			continue // skip alien chunks
 		}
-		if ctrlType != "" || xactID != currentXactID {
+		if ctrlType != "" || xactID != "" {
 			if f.useMeta {
 				// temporary/control chunk calls for lazy metadata read
 				o.unsure = true
@@ -1066,57 +993,12 @@ func (o *Object) readMetadata(ctx context.Context) error {
 		}
 		o.md5 = metaInfo.md5
 		o.sha1 = metaInfo.sha1
-		o.xactID = metaInfo.xactID
 	}
 
 	o.isFull = true // cache results
-	o.xIDCached = true
 	return nil
 }
 
-// readXactID returns the transaction ID stored in the passed metadata object
-func (o *Object) readXactID(ctx context.Context) (xactID string, err error) {
-	// if xactID has already been read and cahced return it now
-	if o.xIDCached {
-		return o.xactID, nil
-	}
-	// Avoid reading metadata for backends that don't use xactID to identify permanent chunks
-	if !o.f.useNoRename {
-		return "", errors.New("readXactID requires norename transactions")
-	}
-	if o.main == nil {
-		return "", errors.New("readXactID requires valid metaobject")
-	}
-	if o.main.Size() > maxMetadataSize {
-		return "", nil // this was likely not a metadata object, return empty xactID but don't throw error
-	}
-	reader, err := o.main.Open(ctx)
-	if err != nil {
-		return "", err
-	}
-	data, err := ioutil.ReadAll(reader)
-	_ = reader.Close() // ensure file handle is freed on windows
-	if err != nil {
-		return "", err
-	}
-
-	switch o.f.opt.MetaFormat {
-	case "simplejson":
-		if data != nil && len(data) > maxMetadataSizeWritten {
-			return "", nil // this was likely not a metadata object, return empty xactID but don't throw error
-		}
-		var metadata metaSimpleJSON
-		err = json.Unmarshal(data, &metadata)
-		if err != nil {
-			return "", nil // this was likely not a metadata object, return empty xactID but don't throw error
-		}
-		xactID = metadata.XactID
-	}
-	o.xactID = xactID
-	o.xIDCached = true
-	return xactID, nil
-}
-
 // put implements Put, PutStream, PutUnchecked, Update
 func (f *Fs) put(
 	ctx context.Context, in io.Reader, src fs.ObjectInfo, remote string, options []fs.OpenOption,
@@ -1269,17 +1151,14 @@ func (f *Fs) put(
 	// If previous object was chunked, remove its chunks
 	f.removeOldChunks(ctx, baseRemote)
 
-	if !f.useNoRename {
-		// The transaction suffix will be removed for backends with quick rename operations
-		for chunkNo, chunk := range c.chunks {
-			chunkRemote := f.makeChunkName(baseRemote, chunkNo, "", "")
-			chunkMoved, errMove := f.baseMove(ctx, chunk, chunkRemote, delFailed)
-			if errMove != nil {
-				return nil, errMove
-			}
-			c.chunks[chunkNo] = chunkMoved
+	// Rename data chunks from temporary to final names
+	for chunkNo, chunk := range c.chunks {
+		chunkRemote := f.makeChunkName(baseRemote, chunkNo, "", "")
+		chunkMoved, errMove := f.baseMove(ctx, chunk, chunkRemote, delFailed)
+		if errMove != nil {
+			return nil, errMove
 		}
-		xactID = ""
+		c.chunks[chunkNo] = chunkMoved
 	}
 
 	if !f.useMeta {
@@ -1299,7 +1178,7 @@ func (f *Fs) put(
 	switch f.opt.MetaFormat {
 	case "simplejson":
 		c.updateHashes()
-		metadata, err = marshalSimpleJSON(ctx, sizeTotal, len(c.chunks), c.md5, c.sha1, xactID)
+		metadata, err = marshalSimpleJSON(ctx, sizeTotal, len(c.chunks), c.md5, c.sha1)
 	}
 	if err == nil {
 		metaInfo := f.wrapInfo(src, baseRemote, int64(len(metadata)))
@@ -1311,7 +1190,6 @@ func (f *Fs) put(
 
 	o := f.newObject("", metaObject, c.chunks)
 	o.size = sizeTotal
-	o.xactID = xactID
 	return o, nil
 }
 
@@ -1715,7 +1593,7 @@ func (f *Fs) copyOrMove(ctx context.Context, o *Object, remote string, do copyMo
 	var metadata []byte
 	switch f.opt.MetaFormat {
 	case "simplejson":
-		metadata, err = marshalSimpleJSON(ctx, newObj.size, len(newChunks), md5, sha1, o.xactID)
+		metadata, err = marshalSimpleJSON(ctx, newObj.size, len(newChunks), md5, sha1)
 		if err == nil {
 			metaInfo := f.wrapInfo(metaObject, "", int64(len(metadata)))
 			err = newObj.main.Update(ctx, bytes.NewReader(metadata), metaInfo)
@@ -1931,13 +1809,7 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
 		//fs.Debugf(f, "ChangeNotify: path %q entryType %d", path, entryType)
 		if entryType == fs.EntryObject {
 			mainPath, _, _, xactID := f.parseChunkName(path)
-			metaXactID := ""
-			if f.useNoRename {
-				metaObject, _ := f.base.NewObject(ctx, mainPath)
-				dummyObject := f.newObject("", metaObject, nil)
-				metaXactID, _ = dummyObject.readXactID(ctx)
-			}
-			if mainPath != "" && xactID == metaXactID {
+			if mainPath != "" && xactID == "" {
 				path = mainPath
 			}
 		}
@@ -1958,17 +1830,15 @@ func (f *Fs) Shutdown(ctx context.Context) error {
 
 // Object represents a composite file wrapping one or more data chunks
 type Object struct {
-	remote    string
-	main      fs.Object   // meta object if file is composite, or wrapped non-chunked file, nil if meta format is 'none'
-	chunks    []fs.Object // active data chunks if file is composite, or wrapped file as a single chunk if meta format is 'none'
-	size      int64       // cached total size of chunks in a composite file or -1 for non-chunked files
-	isFull    bool        // true if metadata has been read
-	xIDCached bool        // true if xactID has been read
-	unsure    bool        // true if need to read metadata to detect object type
-	xactID    string      // transaction ID for "norename" or empty string for "renamed" chunks
-	md5       string
-	sha1      string
-	f         *Fs
+	remote string
+	main   fs.Object   // meta object if file is composite, or wrapped non-chunked file, nil if meta format is 'none'
+	chunks []fs.Object // active data chunks if file is composite, or wrapped file as a single chunk if meta format is 'none'
+	size   int64       // cached total size of chunks in a composite file or -1 for non-chunked files
+	isFull bool        // true if metadata has been read
+	unsure bool        // true if need to read metadata to detect object type
+	md5    string
+	sha1   string
+	f      *Fs
 }
 
 func (o *Object) addChunk(chunk fs.Object, chunkNo int) error {
@@ -2296,7 +2166,6 @@ type ObjectInfo struct {
 	src     fs.ObjectInfo
 	fs      *Fs
 	nChunks int    // number of data chunks
-	xactID  string // transaction ID for "norename" or empty string for "renamed" chunks
 	size    int64  // overrides source size by the total size of data chunks
 	remote  string // overrides remote name
 	md5     string // overrides MD5 checksum
@@ -2395,9 +2264,8 @@ type metaSimpleJSON struct {
 	Size     *int64 `json:"size"`    // total size of data chunks
 	ChunkNum *int   `json:"nchunks"` // number of data chunks
 	// optional extra fields
-	MD5    string `json:"md5,omitempty"`
-	SHA1   string `json:"sha1,omitempty"`
-	XactID string `json:"txn,omitempty"` // transaction ID for norename transactions
+	MD5  string `json:"md5,omitempty"`
+	SHA1 string `json:"sha1,omitempty"`
 }
 
 // marshalSimpleJSON
@@ -2407,20 +2275,16 @@ type metaSimpleJSON struct {
 // - if file contents can be mistaken as meta object
 // - if consistent hashing is On but wrapped remote can't provide given hash
 //
-func marshalSimpleJSON(ctx context.Context, size int64, nChunks int, md5, sha1, xactID string) ([]byte, error) {
+func marshalSimpleJSON(ctx context.Context, size int64, nChunks int, md5, sha1 string) ([]byte, error) {
 	version := metadataVersion
-	if xactID == "" && version == 2 {
-		version = 1
-	}
 	metadata := metaSimpleJSON{
 		// required core fields
 		Version:  &version,
 		Size:     &size,
 		ChunkNum: &nChunks,
 		// optional extra fields
-		MD5:    md5,
-		SHA1:   sha1,
-		XactID: xactID,
+		MD5:  md5,
+		SHA1: sha1,
 	}
 	data, err := json.Marshal(&metadata)
 	if err == nil && data != nil && len(data) >= maxMetadataSizeWritten {
@@ -2498,7 +2362,6 @@ func unmarshalSimpleJSON(ctx context.Context, metaObject fs.Object, data []byte)
 	info.nChunks = *metadata.ChunkNum
 	info.md5 = metadata.MD5
 	info.sha1 = metadata.SHA1
-	info.xactID = metadata.XactID
 	return info, true, nil
 }
 
@@ -2531,11 +2394,6 @@ func (f *Fs) Precision() time.Duration {
 	return f.base.Precision()
 }
 
-// CanQuickRename returns true if the Fs supports a quick rename operation
-func (f *Fs) CanQuickRename() bool {
-	return f.base.Features().Move != nil
-}
-
 // Check the interfaces are satisfied
 var (
 	_ fs.Fs              = (*Fs)(nil)

backend/chunker/chunker_internal_test.go

@@ -468,15 +468,9 @@ func testPreventCorruption(t *testing.T, f *Fs) {
 		return obj
 	}
 	billyObj := newFile("billy")
-	billyTxn := billyObj.(*Object).xactID
-	if f.useNoRename {
-		require.True(t, billyTxn != "")
-	} else {
-		require.True(t, billyTxn == "")
-	}
 
 	billyChunkName := func(chunkNo int) string {
-		return f.makeChunkName(billyObj.Remote(), chunkNo, "", billyTxn)
+		return f.makeChunkName(billyObj.Remote(), chunkNo, "", "")
 	}
 
 	err := f.Mkdir(ctx, billyChunkName(1))
@@ -493,13 +487,11 @@ func testPreventCorruption(t *testing.T, f *Fs) {
 	// accessing chunks in strict mode is prohibited
 	f.opt.FailHard = true
 	billyChunk4Name := billyChunkName(4)
-	_, err = f.base.NewObject(ctx, billyChunk4Name)
-	require.NoError(t, err)
-	_, err = f.NewObject(ctx, billyChunk4Name)
+	billyChunk4, err := f.NewObject(ctx, billyChunk4Name)
 	assertOverlapError(err)
 
 	f.opt.FailHard = false
-	billyChunk4, err := f.NewObject(ctx, billyChunk4Name)
+	billyChunk4, err = f.NewObject(ctx, billyChunk4Name)
 	assert.NoError(t, err)
 	require.NotNil(t, billyChunk4)
 
@@ -528,8 +520,7 @@ func testPreventCorruption(t *testing.T, f *Fs) {
 
 	// recreate billy in case it was anyhow corrupted
 	willyObj := newFile("willy")
-	willyTxn := willyObj.(*Object).xactID
-	willyChunkName := f.makeChunkName(willyObj.Remote(), 1, "", willyTxn)
+	willyChunkName := f.makeChunkName(willyObj.Remote(), 1, "", "")
 	f.opt.FailHard = false
 	willyChunk, err := f.NewObject(ctx, willyChunkName)
 	f.opt.FailHard = true
@@ -570,20 +561,17 @@ func testChunkNumberOverflow(t *testing.T, f *Fs) {
 	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
 	contents := random.String(100)
 
-	newFile := func(f fs.Fs, name string) (obj fs.Object, filename string, txnID string) {
-		filename = path.Join(dir, name)
+	newFile := func(f fs.Fs, name string) (fs.Object, string) {
+		filename := path.Join(dir, name)
 		item := fstest.Item{Path: filename, ModTime: modTime}
-		_, obj = fstests.PutTestContents(ctx, t, f, &item, contents, true)
+		_, obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
 		require.NotNil(t, obj)
-		if chunkObj, isChunkObj := obj.(*Object); isChunkObj {
-			txnID = chunkObj.xactID
-		}
-		return
+		return obj, filename
 	}
 
 	f.opt.FailHard = false
-	file, fileName, fileTxn := newFile(f, "wreaker")
-	wreak, _, _ := newFile(f.base, f.makeChunkName("wreaker", wreakNumber, "", fileTxn))
+	file, fileName := newFile(f, "wreaker")
+	wreak, _ := newFile(f.base, f.makeChunkName("wreaker", wreakNumber, "", ""))
 
 	f.opt.FailHard = false
 	fstest.CheckListingWithRoot(t, f, dir, nil, nil, f.Precision())
@@ -662,7 +650,7 @@ func testMetadataInput(t *testing.T, f *Fs) {
 		}
 	}
 
-	metaData, err := marshalSimpleJSON(ctx, 3, 1, "", "", "")
+	metaData, err := marshalSimpleJSON(ctx, 3, 1, "", "")
 	require.NoError(t, err)
 	todaysMeta := string(metaData)
 	runSubtest(todaysMeta, "today")
@@ -676,7 +664,7 @@ func testMetadataInput(t *testing.T, f *Fs) {
 	runSubtest(futureMeta, "future")
 }
 
-// Test that chunker refuses to change on objects with future/unknown metadata
+// test that chunker refuses to change on objects with future/unknowm metadata
 func testFutureProof(t *testing.T, f *Fs) {
 	if f.opt.MetaFormat == "none" {
 		t.Skip("this test requires metadata support")
@@ -750,100 +738,6 @@ func testFutureProof(t *testing.T, f *Fs) {
 	}
 }
 
-// The newer method of doing transactions without renaming should still be able to correctly process chunks that were created with renaming
-// If you attempt to do the inverse, however, the data chunks will be ignored causing commands to perform incorrectly
-func testBackwardsCompatibility(t *testing.T, f *Fs) {
-	if !f.useMeta {
-		t.Skip("Can't do norename transactions without metadata")
-	}
-	const dir = "backcomp"
-	ctx := context.Background()
-	saveOpt := f.opt
-	saveUseNoRename := f.useNoRename
-	defer func() {
-		f.opt.FailHard = false
-		_ = operations.Purge(ctx, f.base, dir)
-		f.opt = saveOpt
-		f.useNoRename = saveUseNoRename
-	}()
-	f.opt.ChunkSize = fs.SizeSuffix(10)
-
-	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
-	contents := random.String(250)
-	newFile := func(f fs.Fs, name string) (fs.Object, string) {
-		filename := path.Join(dir, name)
-		item := fstest.Item{Path: filename, ModTime: modTime}
-		_, obj := fstests.PutTestContents(ctx, t, f, &item, contents, true)
-		require.NotNil(t, obj)
-		return obj, filename
-	}
-
-	f.opt.FailHard = false
-	f.useNoRename = false
-	file, fileName := newFile(f, "renamefile")
-
-	f.opt.FailHard = false
-	item := fstest.NewItem(fileName, contents, modTime)
-
-	var items []fstest.Item
-	items = append(items, item)
-
-	f.useNoRename = true
-	fstest.CheckListingWithRoot(t, f, dir, items, nil, f.Precision())
-	_, err := f.NewObject(ctx, fileName)
-	assert.NoError(t, err)
-
-	f.opt.FailHard = true
-	_, err = f.List(ctx, dir)
-	assert.NoError(t, err)
-
-	f.opt.FailHard = false
-	_ = file.Remove(ctx)
-}
-
-func testChunkerServerSideMove(t *testing.T, f *Fs) {
-	if !f.useMeta {
-		t.Skip("Can't test norename transactions without metadata")
-	}
-
-	ctx := context.Background()
-	const dir = "servermovetest"
-	subRemote := fmt.Sprintf("%s:%s/%s", f.Name(), f.Root(), dir)
-
-	subFs1, err := fs.NewFs(ctx, subRemote+"/subdir1")
-	assert.NoError(t, err)
-	fs1, isChunkerFs := subFs1.(*Fs)
-	assert.True(t, isChunkerFs)
-	fs1.useNoRename = false
-	fs1.opt.ChunkSize = fs.SizeSuffix(3)
-
-	subFs2, err := fs.NewFs(ctx, subRemote+"/subdir2")
-	assert.NoError(t, err)
-	fs2, isChunkerFs := subFs2.(*Fs)
-	assert.True(t, isChunkerFs)
-	fs2.useNoRename = true
-	fs2.opt.ChunkSize = fs.SizeSuffix(3)
-
-	modTime := fstest.Time("2001-02-03T04:05:06.499999999Z")
-	item := fstest.Item{Path: "movefile", ModTime: modTime}
-	contents := "abcdef"
-	_, file := fstests.PutTestContents(ctx, t, fs1, &item, contents, true)
-
-	dstOverwritten, _ := fs2.NewObject(ctx, "movefile")
-	dstFile, err := operations.Move(ctx, fs2, dstOverwritten, "movefile", file)
-	assert.NoError(t, err)
-	assert.Equal(t, int64(len(contents)), dstFile.Size())
-
-	r, err := dstFile.Open(ctx)
-	assert.NoError(t, err)
-	assert.NotNil(t, r)
-	data, err := ioutil.ReadAll(r)
-	assert.NoError(t, err)
-	assert.Equal(t, contents, string(data))
-	_ = r.Close()
-	_ = operations.Purge(ctx, f.base, dir)
-}
-
 // InternalTest dispatches all internal tests
 func (f *Fs) InternalTest(t *testing.T) {
 	t.Run("PutLarge", func(t *testing.T) {
@@ -870,12 +764,6 @@ func (f *Fs) InternalTest(t *testing.T) {
 	t.Run("FutureProof", func(t *testing.T) {
 		testFutureProof(t, f)
 	})
-	t.Run("BackwardsCompatibility", func(t *testing.T) {
-		testBackwardsCompatibility(t, f)
-	})
-	t.Run("ChunkerServerSideMove", func(t *testing.T) {
-		testChunkerServerSideMove(t, f)
-	})
 }
 
 var _ fstests.InternalTester = (*Fs)(nil)

backend/compress/compress.go

@@ -12,8 +12,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"io/ioutil"
-	"os"
 	"regexp"
 	"strings"
 	"time"
@@ -29,7 +27,6 @@ import (
 	"github.com/rclone/rclone/fs/config/configstruct"
 	"github.com/rclone/rclone/fs/fspath"
 	"github.com/rclone/rclone/fs/hash"
-	"github.com/rclone/rclone/fs/object"
 	"github.com/rclone/rclone/fs/operations"
 )
 
@@ -92,26 +89,15 @@ func init() {
 			Level 0 turns off compression.`,
 			Default:  sgzip.DefaultCompression,
 			Advanced: true,
-		}, {
-			Name: "ram_cache_limit",
-			Help: `Some remotes don't allow the upload of files with unknown size.
-				   In this case the compressed file will need to be cached to determine
-				   it's size.
-				   
-				   Files smaller than this limit will be cached in RAM, file larger than 
-				   this limit will be cached on disk`,
-			Default:  fs.SizeSuffix(20 * 1024 * 1024),
-			Advanced: true,
 		}},
 	})
 }
 
 // Options defines the configuration for this backend
 type Options struct {
-	Remote           string        `config:"remote"`
-	CompressionMode  string        `config:"mode"`
-	CompressionLevel int           `config:"level"`
-	RAMCacheLimit    fs.SizeSuffix `config:"ram_cache_limit"`
+	Remote           string `config:"remote"`
+	CompressionMode  string `config:"mode"`
+	CompressionLevel int    `config:"level"`
 }
 
 /*** FILESYSTEM FUNCTIONS ***/
@@ -430,55 +416,8 @@ type compressionResult struct {
 	meta sgzip.GzipMetadata
 }
 
-// replicating some of operations.Rcat functionality because we want to support remotes without streaming
-// support and of course cannot know the size of a compressed file before compressing it.
-func (f *Fs) rcat(ctx context.Context, dstFileName string, in io.ReadCloser, modTime time.Time, options []fs.OpenOption) (o fs.Object, err error) {
-
-	// cache small files in memory and do normal upload
-	buf := make([]byte, f.opt.RAMCacheLimit)
-	if n, err := io.ReadFull(in, buf); err == io.EOF || err == io.ErrUnexpectedEOF {
-		src := object.NewStaticObjectInfo(dstFileName, modTime, int64(len(buf[:n])), false, nil, f.Fs)
-		return f.Fs.Put(ctx, bytes.NewBuffer(buf[:n]), src, options...)
-	}
-
-	// Need to include what we allready read
-	in = &ReadCloserWrapper{
-		Reader: io.MultiReader(bytes.NewReader(buf), in),
-		Closer: in,
-	}
-
-	canStream := f.Fs.Features().PutStream != nil
-	if canStream {
-		src := object.NewStaticObjectInfo(dstFileName, modTime, -1, false, nil, f.Fs)
-		return f.Fs.Features().PutStream(ctx, in, src, options...)
-	}
-
-	fs.Debugf(f, "Target remote doesn't support streaming uploads, creating temporary local file")
-	tempFile, err := ioutil.TempFile("", "rclone-press-")
-	defer func() {
-		// these errors should be relatively uncritical and the upload should've succeeded so it's okay-ish
-		// to ignore them
-		_ = tempFile.Close()
-		_ = os.Remove(tempFile.Name())
-	}()
-	if err != nil {
-		return nil, errors.Wrap(err, "Failed to create temporary local FS to spool file")
-	}
-	if _, err = io.Copy(tempFile, in); err != nil {
-		return nil, errors.Wrap(err, "Failed to write temporary local file")
-	}
-	if _, err = tempFile.Seek(0, 0); err != nil {
-		return nil, err
-	}
-	finfo, err := tempFile.Stat()
-	if err != nil {
-		return nil, err
-	}
-	return f.Fs.Put(ctx, tempFile, object.NewStaticObjectInfo(dstFileName, modTime, finfo.Size(), false, nil, f.Fs))
-}
-
 // Put a compressed version of a file. Returns a wrappable object and metadata.
-func (f *Fs) putCompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption, mimeType string) (fs.Object, *ObjectMetadata, error) {
+func (f *Fs) putCompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption, put putFn, mimeType string) (fs.Object, *ObjectMetadata, error) {
 	// Unwrap reader accounting
 	in, wrap := accounting.UnWrap(in)
 
@@ -532,7 +471,7 @@ func (f *Fs) putCompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, o
 	}
 
 	// Transfer the data
-	o, err := f.rcat(ctx, makeDataName(src.Remote(), src.Size(), f.mode), ioutil.NopCloser(wrappedIn), src.ModTime(ctx), options)
+	o, err := put(ctx, wrappedIn, f.wrapInfo(src, makeDataName(src.Remote(), src.Size(), f.mode), src.Size()), options...)
 	//o, err := operations.Rcat(ctx, f.Fs, makeDataName(src.Remote(), src.Size(), f.mode), ioutil.NopCloser(wrappedIn), src.ModTime(ctx))
 	if err != nil {
 		if o != nil {
@@ -571,7 +510,7 @@ func (f *Fs) putCompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, o
 }
 
 // Put an uncompressed version of a file. Returns a wrappable object and metadata.
-func (f *Fs) putUncompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, put putFn, options []fs.OpenOption, mimeType string) (fs.Object, *ObjectMetadata, error) {
+func (f *Fs) putUncompress(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption, put putFn, mimeType string) (fs.Object, *ObjectMetadata, error) {
 	// Unwrap the accounting, add our metadata hasher, then wrap it back on
 	in, wrap := accounting.UnWrap(in)
 
@@ -638,8 +577,6 @@ func (f *Fs) putMetadata(ctx context.Context, meta *ObjectMetadata, src fs.Objec
 
 // This function will put both the data and metadata for an Object.
 // putData is the function used for data, while putMeta is the function used for metadata.
-// The putData function will only be used when the object is not compressible if the
-// data is compressible this parameter will be ignored.
 func (f *Fs) putWithCustomFunctions(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption,
 	putData putFn, putMeta putFn, compressible bool, mimeType string) (*Object, error) {
 	// Put file then metadata
@@ -647,9 +584,9 @@ func (f *Fs) putWithCustomFunctions(ctx context.Context, in io.Reader, src fs.Ob
 	var meta *ObjectMetadata
 	var err error
 	if compressible {
-		dataObject, meta, err = f.putCompress(ctx, in, src, options, mimeType)
+		dataObject, meta, err = f.putCompress(ctx, in, src, options, putData, mimeType)
 	} else {
-		dataObject, meta, err = f.putUncompress(ctx, in, src, putData, options, mimeType)
+		dataObject, meta, err = f.putUncompress(ctx, in, src, options, putData, mimeType)
 	}
 	if err != nil {
 		return nil, err
@@ -900,7 +837,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
 func (f *Fs) CleanUp(ctx context.Context) error {
 	do := f.Fs.Features().CleanUp
 	if do == nil {
-		return errors.New("can't CleanUp: not supported by underlying remote")
+		return errors.New("can't CleanUp")
 	}
 	return do(ctx)
 }
@@ -909,7 +846,7 @@ func (f *Fs) CleanUp(ctx context.Context) error {
 func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
 	do := f.Fs.Features().About
 	if do == nil {
-		return nil, errors.New("can't About: not supported by underlying remote")
+		return nil, errors.New("About not supported")
 	}
 	return do(ctx)
 }
@@ -985,7 +922,7 @@ func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryT
 func (f *Fs) PublicLink(ctx context.Context, remote string, duration fs.Duration, unlink bool) (string, error) {
 	do := f.Fs.Features().PublicLink
 	if do == nil {
-		return "", errors.New("can't PublicLink: not supported by underlying remote")
+		return "", errors.New("PublicLink not supported")
 	}
 	o, err := f.NewObject(ctx, remote)
 	if err != nil {
@@ -1096,8 +1033,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 			}
 		}
 	} else {
-		// We can only support update when BOTH the old and the new object are uncompressed because only then
-		// the filesize will be known beforehand and name will stay the same
+		// Function that updates object
 		update := func(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (fs.Object, error) {
 			return o.Object, o.Object.Update(ctx, in, src, options...)
 		}
@@ -1185,7 +1121,7 @@ func (o *Object) String() string {
 func (o *Object) Remote() string {
 	origFileName, _, _, err := processFileName(o.Object.Remote())
 	if err != nil {
-		fs.Errorf(o.f, "Could not get remote path for: %s", o.Object.Remote())
+		fs.Errorf(o, "Could not get remote path for: %s", o.Object.Remote())
 		return o.Object.Remote()
 	}
 	return origFileName
@@ -1225,19 +1161,15 @@ func (o *Object) Hash(ctx context.Context, ht hash.Type) (string, error) {
 // multiple storage classes supported
 func (o *Object) SetTier(tier string) error {
 	do, ok := o.Object.(fs.SetTierer)
-	mdo, mok := o.mo.(fs.SetTierer)
-	if !(ok && mok) {
+	if !ok {
 		return errors.New("press: underlying remote does not support SetTier")
 	}
-	if err := mdo.SetTier(tier); err != nil {
-		return err
-	}
 	return do.SetTier(tier)
 }
 
 // GetTier returns storage tier or class of the Object
 func (o *Object) GetTier() string {
-	do, ok := o.mo.(fs.GetTierer)
+	do, ok := o.Object.(fs.GetTierer)
 	if !ok {
 		return ""
 	}
@@ -1340,7 +1272,10 @@ func (o *ObjectInfo) Remote() string {
 
 // Size returns the size of the file
 func (o *ObjectInfo) Size() int64 {
-	return o.size
+	if o.size != -1 {
+		return o.size
+	}
+	return o.src.Size()
 }
 
 // ModTime returns the modification time
@@ -1351,7 +1286,14 @@ func (o *ObjectInfo) ModTime(ctx context.Context) time.Time {
 // Hash returns the selected checksum of the file
 // If no checksum is available it returns ""
 func (o *ObjectInfo) Hash(ctx context.Context, ht hash.Type) (string, error) {
-	return "", nil // cannot know the checksum
+	if ht != hash.MD5 {
+		return "", hash.ErrUnsupported
+	}
+	value, err := o.src.Hash(ctx, ht)
+	if err == hash.ErrUnsupported {
+		return "", hash.ErrUnsupported
+	}
+	return value, err
 }
 
 // ID returns the ID of the Object if known, or "" if not

backend/compress/compress_test.go

@@ -6,17 +6,18 @@ import (
 	"path/filepath"
 	"testing"
 
-	_ "github.com/rclone/rclone/backend/drive"
+	_ "github.com/rclone/rclone/backend/dropbox"
 	_ "github.com/rclone/rclone/backend/local"
-	_ "github.com/rclone/rclone/backend/s3"
-	_ "github.com/rclone/rclone/backend/swift"
 	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/fstest/fstests"
 )
 
 // TestIntegration runs integration tests against the remote
 func TestIntegration(t *testing.T) {
-	opt := fstests.Opt{
+	if *fstest.RemoteName == "" {
+		t.Skip("Skipping as -remote not set")
+	}
+	fstests.Run(t, &fstests.Opt{
 		RemoteName: *fstest.RemoteName,
 		NilObject:  (*Object)(nil),
 		UnimplementableFsMethods: []string{
@@ -28,9 +29,11 @@ func TestIntegration(t *testing.T) {
 			"UserInfo",
 			"Disconnect",
 		},
-		TiersToTest:                  []string{"STANDARD", "STANDARD_IA"},
-		UnimplementableObjectMethods: []string{}}
-	fstests.Run(t, &opt)
+		UnimplementableObjectMethods: []string{
+			"GetTier",
+			"SetTier",
+		},
+	})
 }
 
 // TestRemoteGzip tests GZIP compression

backend/crypt/crypt.go

@@ -101,21 +101,6 @@ names, or for debugging purposes.`,
 			Default:  false,
 			Hide:     fs.OptionHideConfigurator,
 			Advanced: true,
-		}, {
-			Name:     "no_data_encryption",
-			Help:     "Option to either encrypt file data or leave it unencrypted.",
-			Default:  false,
-			Advanced: true,
-			Examples: []fs.OptionExample{
-				{
-					Value: "true",
-					Help:  "Don't encrypt file data, leave it unencrypted.",
-				},
-				{
-					Value: "false",
-					Help:  "Encrypt file data.",
-				},
-			},
 		}},
 	})
 }
@@ -224,7 +209,6 @@ type Options struct {
 	Remote                  string `config:"remote"`
 	FilenameEncryption      string `config:"filename_encryption"`
 	DirectoryNameEncryption bool   `config:"directory_name_encryption"`
-	NoDataEncryption        bool   `config:"no_data_encryption"`
 	Password                string `config:"password"`
 	Password2               string `config:"password2"`
 	ServerSideAcrossConfigs bool   `config:"server_side_across_configs"`
@@ -362,10 +346,6 @@ type putFn func(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ..
 
 // put implements Put or PutStream
 func (f *Fs) put(ctx context.Context, in io.Reader, src fs.ObjectInfo, options []fs.OpenOption, put putFn) (fs.Object, error) {
-	if f.opt.NoDataEncryption {
-		return put(ctx, in, f.newObjectInfo(src, nonce{}), options...)
-	}
-
 	// Encrypt the data into wrappedIn
 	wrappedIn, encrypter, err := f.cipher.encryptData(in)
 	if err != nil {
@@ -637,10 +617,6 @@ func (f *Fs) computeHashWithNonce(ctx context.Context, nonce nonce, src fs.Objec
 //
 // Note that we break lots of encapsulation in this function.
 func (f *Fs) ComputeHash(ctx context.Context, o *Object, src fs.Object, hashType hash.Type) (hashStr string, err error) {
-	if f.opt.NoDataEncryption {
-		return src.Hash(ctx, hashType)
-	}
-
 	// Read the nonce - opening the file is sufficient to read the nonce in
 	// use a limited read so we only read the header
 	in, err := o.Object.Open(ctx, &fs.RangeOption{Start: 0, End: int64(fileHeaderSize) - 1})
@@ -846,13 +822,9 @@ func (o *Object) Remote() string {
 
 // Size returns the size of the file
 func (o *Object) Size() int64 {
-	size := o.Object.Size()
-	if !o.f.opt.NoDataEncryption {
-		var err error
-		size, err = o.f.cipher.DecryptedSize(size)
-		if err != nil {
-			fs.Debugf(o, "Bad size for decrypt: %v", err)
-		}
+	size, err := o.f.cipher.DecryptedSize(o.Object.Size())
+	if err != nil {
+		fs.Debugf(o, "Bad size for decrypt: %v", err)
 	}
 	return size
 }
@@ -870,10 +842,6 @@ func (o *Object) UnWrap() fs.Object {
 
 // Open opens the file for read.  Call Close() on the returned io.ReadCloser
 func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (rc io.ReadCloser, err error) {
-	if o.f.opt.NoDataEncryption {
-		return o.Object.Open(ctx, options...)
-	}
-
 	var openOptions []fs.OpenOption
 	var offset, limit int64 = 0, -1
 	for _, option := range options {

backend/crypt/crypt_test.go

@@ -91,26 +91,3 @@ func TestObfuscate(t *testing.T) {
 		UnimplementableObjectMethods: []string{"MimeType"},
 	})
 }
-
-// TestNoDataObfuscate runs integration tests against the remote
-func TestNoDataObfuscate(t *testing.T) {
-	if *fstest.RemoteName != "" {
-		t.Skip("Skipping as -remote set")
-	}
-	tempdir := filepath.Join(os.TempDir(), "rclone-crypt-test-obfuscate")
-	name := "TestCrypt4"
-	fstests.Run(t, &fstests.Opt{
-		RemoteName: name + ":",
-		NilObject:  (*crypt.Object)(nil),
-		ExtraConfig: []fstests.ExtraConfigItem{
-			{Name: name, Key: "type", Value: "crypt"},
-			{Name: name, Key: "remote", Value: tempdir},
-			{Name: name, Key: "password", Value: obscure.MustObscure("potato2")},
-			{Name: name, Key: "filename_encryption", Value: "obfuscate"},
-			{Name: name, Key: "no_data_encryption", Value: "true"},
-		},
-		SkipBadWindowsCharacters:     true,
-		UnimplementableFsMethods:     []string{"OpenWriterAt"},
-		UnimplementableObjectMethods: []string{"MimeType"},
-	})
-}

backend/drive/drive.go

@@ -73,7 +73,6 @@ const (
 	partialFields    = "id,name,size,md5Checksum,trashed,explicitlyTrashed,modifiedTime,createdTime,mimeType,parents,webViewLink,shortcutDetails,exportLinks"
 	listRGrouping    = 50   // number of IDs to search at once when using ListR
 	listRInputBuffer = 1000 // size of input buffer when using ListR
-	defaultXDGIcon   = "text-html"
 )
 
 // Globals
@@ -129,12 +128,6 @@ var (
 	_mimeTypeCustomTransform = map[string]string{
 		"application/vnd.google-apps.script+json": "application/json",
 	}
-	_mimeTypeToXDGLinkIcons = map[string]string{
-		"application/vnd.google-apps.document":     "x-office-document",
-		"application/vnd.google-apps.drawing":      "x-office-drawing",
-		"application/vnd.google-apps.presentation": "x-office-presentation",
-		"application/vnd.google-apps.spreadsheet":  "x-office-spreadsheet",
-	}
 	fetchFormatsOnce sync.Once                     // make sure we fetch the export/import formats only once
 	_exportFormats   map[string][]string           // allowed export MIME type conversions
 	_importFormats   map[string][]string           // allowed import MIME type conversions
@@ -207,7 +200,7 @@ func init() {
 			}
 			err = configTeamDrive(ctx, opt, m, name)
 			if err != nil {
-				log.Fatalf("Failed to configure Shared Drive: %v", err)
+				log.Fatalf("Failed to configure team drive: %v", err)
 			}
 		},
 		Options: append(driveOAuthOptions(), []fs.Option{{
@@ -247,7 +240,7 @@ a non root folder as its starting point.
 			Advanced: true,
 		}, {
 			Name:     "team_drive",
-			Help:     "ID of the Shared Drive (Team Drive)",
+			Help:     "ID of the Team Drive",
 			Hide:     fs.OptionHideConfigurator,
 			Advanced: true,
 		}, {
@@ -666,7 +659,7 @@ func (f *Fs) shouldRetry(err error) (bool, error) {
 				fs.Errorf(f, "Received download limit error: %v", err)
 				return false, fserrors.FatalError(err)
 			} else if f.opt.StopOnUploadLimit && reason == "teamDriveFileLimitExceeded" {
-				fs.Errorf(f, "Received Shared Drive file limit error: %v", err)
+				fs.Errorf(f, "Received team drive file limit error: %v", err)
 				return false, fserrors.FatalError(err)
 			}
 		}
@@ -955,24 +948,24 @@ func configTeamDrive(ctx context.Context, opt *Options, m configmap.Mapper, name
 		return nil
 	}
 	if opt.TeamDriveID == "" {
-		fmt.Printf("Configure this as a Shared Drive (Team Drive)?\n")
+		fmt.Printf("Configure this as a team drive?\n")
 	} else {
-		fmt.Printf("Change current Shared Drive (Team Drive) ID %q?\n", opt.TeamDriveID)
+		fmt.Printf("Change current team drive ID %q?\n", opt.TeamDriveID)
 	}
 	if !config.Confirm(false) {
 		return nil
 	}
 	f, err := newFs(ctx, name, "", m)
 	if err != nil {
-		return errors.Wrap(err, "failed to make Fs to list Shared Drives")
+		return errors.Wrap(err, "failed to make Fs to list teamdrives")
 	}
-	fmt.Printf("Fetching Shared Drive list...\n")
+	fmt.Printf("Fetching team drive list...\n")
 	teamDrives, err := f.listTeamDrives(ctx)
 	if err != nil {
 		return err
 	}
 	if len(teamDrives) == 0 {
-		fmt.Printf("No Shared Drives found in your account")
+		fmt.Printf("No team drives found in your account")
 		return nil
 	}
 	var driveIDs, driveNames []string
@@ -980,7 +973,7 @@ func configTeamDrive(ctx context.Context, opt *Options, m configmap.Mapper, name
 		driveIDs = append(driveIDs, teamDrive.Id)
 		driveNames = append(driveNames, teamDrive.Name)
 	}
-	driveID := config.Choose("Enter a Shared Drive ID", driveIDs, driveNames, true)
+	driveID := config.Choose("Enter a Team Drive ID", driveIDs, driveNames, true)
 	m.Set("team_drive", driveID)
 	m.Set("root_folder_id", "")
 	opt.TeamDriveID = driveID
@@ -1301,15 +1294,11 @@ func (f *Fs) newLinkObject(remote string, info *drive.File, extension, exportMim
 	if t == nil {
 		return nil, errors.Errorf("unsupported link type %s", exportMimeType)
 	}
-	xdgIcon := _mimeTypeToXDGLinkIcons[info.MimeType]
-	if xdgIcon == "" {
-		xdgIcon = defaultXDGIcon
-	}
 	var buf bytes.Buffer
 	err := t.Execute(&buf, struct {
-		URL, Title, XDGIcon string
+		URL, Title string
 	}{
-		info.WebViewLink, info.Name, xdgIcon,
+		info.WebViewLink, info.Name,
 	})
 	if err != nil {
 		return nil, errors.Wrap(err, "executing template failed")
@@ -2460,7 +2449,6 @@ func (f *Fs) CleanUp(ctx context.Context) error {
 	if err != nil {
 		return err
 	}
-	fs.Logf(f, "Note that emptying the trash happens in the background and can take some time.")
 	return nil
 }
 
@@ -2475,9 +2463,9 @@ func (f *Fs) teamDriveOK(ctx context.Context) (err error) {
 		return f.shouldRetry(err)
 	})
 	if err != nil {
-		return errors.Wrap(err, "failed to get Shared Drive info")
+		return errors.Wrap(err, "failed to get Team/Shared Drive info")
 	}
-	fs.Debugf(f, "read info from Shared Drive %q", td.Name)
+	fs.Debugf(f, "read info from team drive %q", td.Name)
 	return err
 }
 
@@ -2963,7 +2951,7 @@ func (f *Fs) listTeamDrives(ctx context.Context) (drives []*drive.TeamDrive, err
 			return defaultFs.shouldRetry(err)
 		})
 		if err != nil {
-			return drives, errors.Wrap(err, "listing Team Drives failed")
+			return drives, errors.Wrap(err, "listing team drives failed")
 		}
 		drives = append(drives, teamDrives.TeamDrives...)
 		if teamDrives.NextPageToken == "" {
@@ -3064,9 +3052,6 @@ func (f *Fs) copyID(ctx context.Context, id, dest string) (err error) {
 	if destLeaf == "" {
 		destLeaf = info.Name
 	}
-	if destDir == "" {
-		destDir = "."
-	}
 	dstFs, err := cache.Get(ctx, destDir)
 	if err != nil {
 		return err
@@ -3131,8 +3116,8 @@ authenticated with "drive2:" can't read files from "drive:".
 	},
 }, {
 	Name:  "drives",
-	Short: "List the Shared Drives available to this account",
-	Long: `This command lists the Shared Drives (Team Drives) available to this
+	Short: "List the shared drives available to this account",
+	Long: `This command lists the shared drives (teamdrives) available to this
 account.
 
 Usage:
@@ -3427,10 +3412,11 @@ func (o *baseObject) httpResponse(ctx context.Context, url, method string, optio
 	if url == "" {
 		return nil, nil, errors.New("forbidden to download - check sharing permission")
 	}
-	req, err = http.NewRequestWithContext(ctx, method, url, nil)
+	req, err = http.NewRequest(method, url, nil)
 	if err != nil {
 		return req, nil, err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	fs.OpenOptionAddHTTPHeaders(req.Header, options)
 	if o.bytes == 0 {
 		// Don't supply range requests for 0 length objects as they always fail
@@ -3763,7 +3749,7 @@ URL={{ .URL }}{{"\r"}}
 Encoding=UTF-8
 Name={{ .Title }}
 URL={{ .URL }}
-Icon={{ .XDGIcon }}
+Icon=text-html
 Type=Link
 `
 	htmlTemplate = `<html>

backend/drive/upload.go

@@ -77,10 +77,11 @@ func (f *Fs) Upload(ctx context.Context, in io.Reader, size int64, contentType,
 			return false, err
 		}
 		var req *http.Request
-		req, err = http.NewRequestWithContext(ctx, method, urls, body)
+		req, err = http.NewRequest(method, urls, body)
 		if err != nil {
 			return false, err
 		}
+		req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 		googleapi.Expand(req.URL, map[string]string{
 			"fileId": fileID,
 		})
@@ -113,7 +114,8 @@ func (f *Fs) Upload(ctx context.Context, in io.Reader, size int64, contentType,
 
 // Make an http.Request for the range passed in
 func (rx *resumableUpload) makeRequest(ctx context.Context, start int64, body io.ReadSeeker, reqSize int64) *http.Request {
-	req, _ := http.NewRequestWithContext(ctx, "POST", rx.URI, body)
+	req, _ := http.NewRequest("POST", rx.URI, body)
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	req.ContentLength = reqSize
 	totalSize := "*"
 	if rx.ContentLength >= 0 {

backend/dropbox/dropbox.go

@@ -94,14 +94,7 @@ const (
 var (
 	// Description of how to auth for this app
 	dropboxConfig = &oauth2.Config{
-		Scopes: []string{
-			"files.metadata.write",
-			"files.content.write",
-			"files.content.read",
-			"sharing.write",
-			// "file_requests.write",
-			// "members.read", // needed for impersonate - but causes app to need to be approved by Dropbox Team Admin during the flow
-		},
+		Scopes: []string{},
 		// Endpoint: oauth2.Endpoint{
 		// 	AuthURL:  "https://www.dropbox.com/1/oauth2/authorize",
 		// 	TokenURL: "https://api.dropboxapi.com/1/oauth2/token",
@@ -122,19 +115,6 @@ var (
 	errNotSupportedInSharedMode = fserrors.NoRetryError(errors.New("not supported in shared files mode"))
 )
 
-// Gets an oauth config with the right scopes
-func getOauthConfig(m configmap.Mapper) *oauth2.Config {
-	// If not impersonating, use standard scopes
-	if impersonate, _ := m.Get("impersonate"); impersonate == "" {
-		return dropboxConfig
-	}
-	// Make a copy of the config
-	config := *dropboxConfig
-	// Make a copy of the scopes with "members.read" appended
-	config.Scopes = append(config.Scopes, "members.read")
-	return &config
-}
-
 // Register with Fs
 func init() {
 	DbHashType = hash.RegisterHash("DropboxHash", 64, dbhash.New)
@@ -149,7 +129,7 @@ func init() {
 					oauth2.SetAuthURLParam("token_access_type", "offline"),
 				},
 			}
-			err := oauthutil.Config(ctx, "dropbox", name, m, getOauthConfig(m), &opt)
+			err := oauthutil.Config(ctx, "dropbox", name, m, dropboxConfig, &opt)
 			if err != nil {
 				log.Fatalf("Failed to configure token: %v", err)
 			}
@@ -167,23 +147,8 @@ memory.  It can be set smaller if you are tight on memory.`, maxChunkSize),
 			Default:  defaultChunkSize,
 			Advanced: true,
 		}, {
-			Name: "impersonate",
-			Help: `Impersonate this user when using a business account.
-
-Note that if you want to use impersonate, you should make sure this
-flag is set when running "rclone config" as this will cause rclone to
-request the "members.read" scope which it won't normally. This is
-needed to lookup a members email address into the internal ID that
-dropbox uses in the API.
-
-Using the "members.read" scope will require a Dropbox Team Admin
-to approve during the OAuth flow.
-
-You will have to use your own App (setting your own client_id and
-client_secret) to use this option as currently rclone's default set of
-permissions doesn't include "members.read". This can be added once
-v1.55 or later is in use everywhere.
-`,
+			Name:     "impersonate",
+			Help:     "Impersonate this user when using a business account.",
 			Default:  "",
 			Advanced: true,
 		}, {
@@ -219,11 +184,11 @@ shared folder.`,
 			// as invalid characters.
 			// Testing revealed names with trailing spaces and the DEL character don't work.
 			// Also encode invalid UTF-8 bytes as json doesn't handle them properly.
-			Default: encoder.Base |
+			Default: (encoder.Base |
 				encoder.EncodeBackSlash |
 				encoder.EncodeDel |
 				encoder.EncodeRightSpace |
-				encoder.EncodeInvalidUtf8,
+				encoder.EncodeInvalidUtf8),
 		}}...),
 	})
 }
@@ -242,10 +207,8 @@ type Fs struct {
 	name           string         // name of this remote
 	root           string         // the path we are working on
 	opt            Options        // parsed options
-	ci             *fs.ConfigInfo // global config
 	features       *fs.Features   // optional features
 	srv            files.Client   // the connection to the dropbox server
-	svc            files.Client   // the connection to the dropbox server (unauthorized)
 	sharing        sharing.Client // as above, but for generating sharing links
 	users          users.Client   // as above, but for accessing user information
 	team           team.Client    // for the Teams API
@@ -364,34 +327,27 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		}
 	}
 
-	oAuthClient, _, err := oauthutil.NewClient(ctx, name, m, getOauthConfig(m))
+	oAuthClient, _, err := oauthutil.NewClient(ctx, name, m, dropboxConfig)
 	if err != nil {
 		return nil, errors.Wrap(err, "failed to configure dropbox")
 	}
 
-	ci := fs.GetConfig(ctx)
-
 	f := &Fs{
 		name:  name,
 		opt:   *opt,
-		ci:    ci,
 		pacer: fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
 	}
-	cfg := dropbox.Config{
+	config := dropbox.Config{
 		LogLevel:        dropbox.LogOff, // logging in the SDK: LogOff, LogDebug, LogInfo
 		Client:          oAuthClient,    // maybe???
 		HeaderGenerator: f.headerGenerator,
 	}
 
-	// unauthorized config for endpoints that fail with auth
-	ucfg := dropbox.Config{
-		LogLevel: dropbox.LogOff, // logging in the SDK: LogOff, LogDebug, LogInfo
-	}
-
 	// NOTE: needs to be created pre-impersonation so we can look up the impersonated user
-	f.team = team.New(cfg)
+	f.team = team.New(config)
 
 	if opt.Impersonate != "" {
+
 		user := team.UserSelectorArg{
 			Email: opt.Impersonate,
 		}
@@ -406,13 +362,12 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 			return nil, errors.Wrapf(err, "invalid dropbox team member: %q", opt.Impersonate)
 		}
 
-		cfg.AsMemberID = memberIds[0].MemberInfo.Profile.MemberProfile.TeamMemberId
+		config.AsMemberID = memberIds[0].MemberInfo.Profile.MemberProfile.TeamMemberId
 	}
 
-	f.srv = files.New(cfg)
-	f.svc = files.New(ucfg)
-	f.sharing = sharing.New(cfg)
-	f.users = users.New(cfg)
+	f.srv = files.New(config)
+	f.sharing = sharing.New(config)
+	f.users = users.New(config)
 	f.features = (&fs.Features{
 		CaseInsensitive:         true,
 		ReadMimeType:            false,
@@ -671,7 +626,7 @@ func (f *Fs) findSharedFolder(name string) (id string, err error) {
 	return "", fs.ErrorDirNotFound
 }
 
-// mountSharedFolder mount a shared folder to the root namespace
+// mountSharedFolders mount a shared folder to the root namespace
 func (f *Fs) mountSharedFolder(id string) error {
 	arg := sharing.MountFolderArg{
 		SharedFolderId: id,
@@ -683,7 +638,7 @@ func (f *Fs) mountSharedFolder(id string) error {
 	return err
 }
 
-// listReceivedFiles lists shared the user as access to (note this means individual
+// listSharedFolders lists shared the user as access to (note this means individual
 // files not files contained in shared folders)
 func (f *Fs) listReceivedFiles() (entries fs.DirEntries, err error) {
 	started := false
@@ -1201,159 +1156,6 @@ func (f *Fs) About(ctx context.Context) (usage *fs.Usage, err error) {
 	return usage, nil
 }
 
-// ChangeNotify calls the passed function with a path that has had changes.
-// If the implementation uses polling, it should adhere to the given interval.
-//
-// Automatically restarts itself in case of unexpected behavior of the remote.
-//
-// Close the returned channel to stop being notified.
-func (f *Fs) ChangeNotify(ctx context.Context, notifyFunc func(string, fs.EntryType), pollIntervalChan <-chan time.Duration) {
-	go func() {
-		// get the StartCursor early so all changes from now on get processed
-		startCursor, err := f.changeNotifyCursor()
-		if err != nil {
-			fs.Infof(f, "Failed to get StartCursor: %s", err)
-		}
-		var ticker *time.Ticker
-		var tickerC <-chan time.Time
-		for {
-			select {
-			case pollInterval, ok := <-pollIntervalChan:
-				if !ok {
-					if ticker != nil {
-						ticker.Stop()
-					}
-					return
-				}
-				if ticker != nil {
-					ticker.Stop()
-					ticker, tickerC = nil, nil
-				}
-				if pollInterval != 0 {
-					ticker = time.NewTicker(pollInterval)
-					tickerC = ticker.C
-				}
-			case <-tickerC:
-				if startCursor == "" {
-					startCursor, err = f.changeNotifyCursor()
-					if err != nil {
-						fs.Infof(f, "Failed to get StartCursor: %s", err)
-						continue
-					}
-				}
-				fs.Debugf(f, "Checking for changes on remote")
-				startCursor, err = f.changeNotifyRunner(ctx, notifyFunc, startCursor)
-				if err != nil {
-					fs.Infof(f, "Change notify listener failure: %s", err)
-				}
-			}
-		}
-	}()
-}
-
-func (f *Fs) changeNotifyCursor() (cursor string, err error) {
-	var startCursor *files.ListFolderGetLatestCursorResult
-
-	err = f.pacer.Call(func() (bool, error) {
-		arg := files.ListFolderArg{
-			Path:      f.opt.Enc.FromStandardPath(f.slashRoot),
-			Recursive: true,
-		}
-
-		if arg.Path == "/" {
-			arg.Path = ""
-		}
-
-		startCursor, err = f.srv.ListFolderGetLatestCursor(&arg)
-
-		return shouldRetry(err)
-	})
-	if err != nil {
-		return
-	}
-	return startCursor.Cursor, nil
-}
-
-func (f *Fs) changeNotifyRunner(ctx context.Context, notifyFunc func(string, fs.EntryType), startCursor string) (newCursor string, err error) {
-	cursor := startCursor
-	var res *files.ListFolderLongpollResult
-
-	// Dropbox sets a timeout range of 30 - 480
-	timeout := uint64(f.ci.TimeoutOrInfinite() / time.Second)
-
-	if timeout < 30 {
-		timeout = 30
-	}
-
-	if timeout > 480 {
-		timeout = 480
-	}
-
-	err = f.pacer.Call(func() (bool, error) {
-		args := files.ListFolderLongpollArg{
-			Cursor:  cursor,
-			Timeout: timeout,
-		}
-
-		res, err = f.svc.ListFolderLongpoll(&args)
-		return shouldRetry(err)
-	})
-	if err != nil {
-		return
-	}
-
-	if !res.Changes {
-		return cursor, nil
-	}
-
-	if res.Backoff != 0 {
-		fs.Debugf(f, "Waiting to poll for %d seconds", res.Backoff)
-		time.Sleep(time.Duration(res.Backoff) * time.Second)
-	}
-
-	for {
-		var changeList *files.ListFolderResult
-
-		arg := files.ListFolderContinueArg{
-			Cursor: cursor,
-		}
-		err = f.pacer.Call(func() (bool, error) {
-			changeList, err = f.srv.ListFolderContinue(&arg)
-			return shouldRetry(err)
-		})
-		if err != nil {
-			return "", errors.Wrap(err, "list continue")
-		}
-		cursor = changeList.Cursor
-		var entryType fs.EntryType
-		for _, entry := range changeList.Entries {
-			entryPath := ""
-			switch info := entry.(type) {
-			case *files.FolderMetadata:
-				entryType = fs.EntryDirectory
-				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
-			case *files.FileMetadata:
-				entryType = fs.EntryObject
-				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
-			case *files.DeletedMetadata:
-				entryType = fs.EntryObject
-				entryPath = strings.TrimLeft(info.PathDisplay, f.slashRootSlash)
-			default:
-				fs.Errorf(entry, "dropbox ChangeNotify: ignoring unknown EntryType %T", entry)
-				continue
-			}
-
-			if entryPath != "" {
-				notifyFunc(entryPath, entryType)
-			}
-		}
-		if !changeList.HasMore {
-			break
-		}
-	}
-	return cursor, nil
-}
-
 // Hashes returns the supported hash sets.
 func (f *Fs) Hashes() hash.Set {
 	return hash.Set(DbHashType)

backend/fichier/api.go

@@ -48,41 +48,6 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
 
 var isAlphaNumeric = regexp.MustCompile(`^[a-zA-Z0-9]+$`).MatchString
 
-func (f *Fs) createObject(ctx context.Context, remote string) (o *Object, leaf string, directoryID string, err error) {
-	// Create the directory for the object if it doesn't exist
-	leaf, directoryID, err = f.dirCache.FindPath(ctx, remote, true)
-	if err != nil {
-		return
-	}
-	// Temporary Object under construction
-	o = &Object{
-		fs:     f,
-		remote: remote,
-	}
-	return o, leaf, directoryID, nil
-}
-
-func (f *Fs) readFileInfo(ctx context.Context, url string) (*File, error) {
-	request := FileInfoRequest{
-		URL: url,
-	}
-	opts := rest.Opts{
-		Method: "POST",
-		Path:   "/file/info.cgi",
-	}
-
-	var file File
-	err := f.pacer.Call(func() (bool, error) {
-		resp, err := f.rest.CallJSON(ctx, &opts, &request, &file)
-		return shouldRetry(resp, err)
-	})
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't read file info")
-	}
-
-	return &file, err
-}
-
 func (f *Fs) getDownloadToken(ctx context.Context, url string) (*GetTokenResponse, error) {
 	request := DownloadRequest{
 		URL:    url,
@@ -343,56 +308,6 @@ func (f *Fs) deleteFile(ctx context.Context, url string) (response *GenericOKRes
 	return response, nil
 }
 
-func (f *Fs) moveFile(ctx context.Context, url string, folderID int, rename string) (response *MoveFileResponse, err error) {
-	request := &MoveFileRequest{
-		URLs:     []string{url},
-		FolderID: folderID,
-		Rename:   rename,
-	}
-
-	opts := rest.Opts{
-		Method: "POST",
-		Path:   "/file/mv.cgi",
-	}
-
-	response = &MoveFileResponse{}
-	err = f.pacer.Call(func() (bool, error) {
-		resp, err := f.rest.CallJSON(ctx, &opts, request, response)
-		return shouldRetry(resp, err)
-	})
-
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't copy file")
-	}
-
-	return response, nil
-}
-
-func (f *Fs) copyFile(ctx context.Context, url string, folderID int, rename string) (response *CopyFileResponse, err error) {
-	request := &CopyFileRequest{
-		URLs:     []string{url},
-		FolderID: folderID,
-		Rename:   rename,
-	}
-
-	opts := rest.Opts{
-		Method: "POST",
-		Path:   "/file/cp.cgi",
-	}
-
-	response = &CopyFileResponse{}
-	err = f.pacer.Call(func() (bool, error) {
-		resp, err := f.rest.CallJSON(ctx, &opts, request, response)
-		return shouldRetry(resp, err)
-	})
-
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't copy file")
-	}
-
-	return response, nil
-}
-
 func (f *Fs) getUploadNode(ctx context.Context) (response *GetUploadNodeResponse, err error) {
 	// fs.Debugf(f, "Requesting Upload node")
 

backend/fichier/fichier.go

@@ -363,6 +363,7 @@ func (f *Fs) putUnchecked(ctx context.Context, in io.Reader, remote string, size
 		fs:     f,
 		remote: remote,
 		file: File{
+			ACL:         0,
 			CDN:         0,
 			Checksum:    link.Whirlpool,
 			ContentType: "",
@@ -415,79 +416,9 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
 	return nil
 }
 
-// Move src to this remote using server side move operations.
-func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
-	srcObj, ok := src.(*Object)
-	if !ok {
-		fs.Debugf(src, "Can't move - not same remote type")
-		return nil, fs.ErrorCantMove
-	}
-
-	// Create temporary object
-	dstObj, leaf, directoryID, err := f.createObject(ctx, remote)
-	if err != nil {
-		return nil, err
-	}
-
-	folderID, err := strconv.Atoi(directoryID)
-	if err != nil {
-		return nil, err
-	}
-	resp, err := f.moveFile(ctx, srcObj.file.URL, folderID, leaf)
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't move file")
-	}
-	if resp.Status != "OK" {
-		return nil, errors.New("couldn't move file")
-	}
-
-	file, err := f.readFileInfo(ctx, resp.URLs[0])
-	if err != nil {
-		return nil, errors.New("couldn't read file data")
-	}
-	dstObj.setMetaData(*file)
-	return dstObj, nil
-}
-
-// Copy src to this remote using server side move operations.
-func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
-	srcObj, ok := src.(*Object)
-	if !ok {
-		fs.Debugf(src, "Can't move - not same remote type")
-		return nil, fs.ErrorCantMove
-	}
-
-	// Create temporary object
-	dstObj, leaf, directoryID, err := f.createObject(ctx, remote)
-	if err != nil {
-		return nil, err
-	}
-
-	folderID, err := strconv.Atoi(directoryID)
-	if err != nil {
-		return nil, err
-	}
-	resp, err := f.copyFile(ctx, srcObj.file.URL, folderID, leaf)
-	if err != nil {
-		return nil, errors.Wrap(err, "couldn't move file")
-	}
-	if resp.Status != "OK" {
-		return nil, errors.New("couldn't move file")
-	}
-
-	file, err := f.readFileInfo(ctx, resp.URLs[0].ToURL)
-	if err != nil {
-		return nil, errors.New("couldn't read file data")
-	}
-	dstObj.setMetaData(*file)
-	return dstObj, nil
-}
-
 // Check the interfaces are satisfied
 var (
 	_ fs.Fs              = (*Fs)(nil)
-	_ fs.Mover           = (*Fs)(nil)
-	_ fs.Copier          = (*Fs)(nil)
 	_ fs.PutUncheckeder  = (*Fs)(nil)
 	_ dircache.DirCacher = (*Fs)(nil)
 )

backend/fichier/object.go

@@ -72,10 +72,6 @@ func (o *Object) SetModTime(context.Context, time.Time) error {
 	//return errors.New("setting modtime is not supported for 1fichier remotes")
 }
 
-func (o *Object) setMetaData(file File) {
-	o.file = file
-}
-
 // Open opens the file for read.  Call Close() on the returned io.ReadCloser
 func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (io.ReadCloser, error) {
 	fs.FixRangeOption(options, o.file.Size)

backend/fichier/structs.go

@@ -1,10 +1,5 @@
 package fichier
 
-// FileInfoRequest is the request structure of the corresponding request
-type FileInfoRequest struct {
-	URL string `json:"url"`
-}
-
 // ListFolderRequest is the request structure of the corresponding request
 type ListFolderRequest struct {
 	FolderID int `json:"folder_id"`
@@ -54,39 +49,6 @@ type MakeFolderResponse struct {
 	FolderID int    `json:"folder_id"`
 }
 
-// MoveFileRequest is the request structure of the corresponding request
-type MoveFileRequest struct {
-	URLs     []string `json:"urls"`
-	FolderID int      `json:"destination_folder_id"`
-	Rename   string   `json:"rename,omitempty"`
-}
-
-// MoveFileResponse is the response structure of the corresponding request
-type MoveFileResponse struct {
-	Status string   `json:"status"`
-	URLs   []string `json:"urls"`
-}
-
-// CopyFileRequest is the request structure of the corresponding request
-type CopyFileRequest struct {
-	URLs     []string `json:"urls"`
-	FolderID int      `json:"folder_id"`
-	Rename   string   `json:"rename,omitempty"`
-}
-
-// CopyFileResponse is the response structure of the corresponding request
-type CopyFileResponse struct {
-	Status string     `json:"status"`
-	Copied int        `json:"copied"`
-	URLs   []FileCopy `json:"urls"`
-}
-
-// FileCopy is used in the the CopyFileResponse
-type FileCopy struct {
-	FromURL string `json:"from_url"`
-	ToURL   string `json:"to_url"`
-}
-
 // GetUploadNodeResponse is the response structure of the corresponding request
 type GetUploadNodeResponse struct {
 	ID  string `json:"id"`
@@ -124,6 +86,7 @@ type EndFileUploadResponse struct {
 
 // File is the structure how 1Fichier returns a File
 type File struct {
+	ACL         int    `json:"acl"`
 	CDN         int    `json:"cdn"`
 	Checksum    string `json:"checksum"`
 	ContentType string `json:"content-type"`

backend/filefabric/filefabric.go

@@ -536,7 +536,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
 	// Find the leaf in pathID
 	found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
-		if strings.EqualFold(item.Name, leaf) {
+		if item.Name == leaf {
 			pathIDOut = item.ID
 			return true
 		}

backend/ftp/ftp.go

@@ -5,7 +5,6 @@ import (
 	"context"
 	"crypto/tls"
 	"io"
-	"net"
 	"net/textproto"
 	"path"
 	"runtime"
@@ -21,7 +20,6 @@ import (
 	"github.com/rclone/rclone/fs/config/configmap"
 	"github.com/rclone/rclone/fs/config/configstruct"
 	"github.com/rclone/rclone/fs/config/obscure"
-	"github.com/rclone/rclone/fs/fshttp"
 	"github.com/rclone/rclone/fs/hash"
 	"github.com/rclone/rclone/lib/encoder"
 	"github.com/rclone/rclone/lib/env"
@@ -93,17 +91,6 @@ to an encrypted one. Cannot be used in combination with implicit FTP.`,
 			Help:     "Disable using MLSD even if server advertises support",
 			Default:  false,
 			Advanced: true,
-		}, {
-			Name:    "idle_timeout",
-			Default: fs.Duration(60 * time.Second),
-			Help: `Max time before closing idle connections
-
-If no connections have been returned to the connection pool in the time
-given, rclone will empty the connection pool.
-
-Set to 0 to keep connections indefinitely.
-`,
-			Advanced: true,
 		}, {
 			Name:     config.ConfigEncoding,
 			Help:     config.ConfigEncodingHelp,
@@ -131,7 +118,6 @@ type Options struct {
 	SkipVerifyTLSCert bool                 `config:"no_check_certificate"`
 	DisableEPSV       bool                 `config:"disable_epsv"`
 	DisableMLSD       bool                 `config:"disable_mlsd"`
-	IdleTimeout       fs.Duration          `config:"idle_timeout"`
 	Enc               encoder.MultiEncoder `config:"encoding"`
 }
 
@@ -148,9 +134,7 @@ type Fs struct {
 	dialAddr string
 	poolMu   sync.Mutex
 	pool     []*ftp.ServerConn
-	drain    *time.Timer // used to drain the pool when we stop using the connections
 	tokens   *pacer.TokenDispenser
-	tlsConf  *tls.Config
 }
 
 // Object describes an FTP file
@@ -227,36 +211,25 @@ func (dl *debugLog) Write(p []byte) (n int, err error) {
 	return len(p), nil
 }
 
-type dialCtx struct {
-	f   *Fs
-	ctx context.Context
-}
-
-// dial a new connection with fshttp dialer
-func (d *dialCtx) dial(network, address string) (net.Conn, error) {
-	conn, err := fshttp.NewDialer(d.ctx).Dial(network, address)
-	if err != nil {
-		return nil, err
-	}
-	if d.f.tlsConf != nil {
-		conn = tls.Client(conn, d.f.tlsConf)
-	}
-	return conn, err
-}
-
 // Open a new connection to the FTP server.
 func (f *Fs) ftpConnection(ctx context.Context) (*ftp.ServerConn, error) {
 	fs.Debugf(f, "Connecting to FTP server")
-	dCtx := dialCtx{f, ctx}
-	ftpConfig := []ftp.DialOption{ftp.DialWithDialFunc(dCtx.dial)}
-	if f.opt.ExplicitTLS {
-		ftpConfig = append(ftpConfig, ftp.DialWithExplicitTLS(f.tlsConf))
-		// Initial connection needs to be cleartext for explicit TLS
-		conn, err := fshttp.NewDialer(ctx).Dial("tcp", f.dialAddr)
-		if err != nil {
-			return nil, err
+	ftpConfig := []ftp.DialOption{ftp.DialWithTimeout(f.ci.ConnectTimeout)}
+	if f.opt.TLS && f.opt.ExplicitTLS {
+		fs.Errorf(f, "Implicit TLS and explicit TLS are mutually incompatible. Please revise your config")
+		return nil, errors.New("Implicit TLS and explicit TLS are mutually incompatible. Please revise your config")
+	} else if f.opt.TLS {
+		tlsConfig := &tls.Config{
+			ServerName:         f.opt.Host,
+			InsecureSkipVerify: f.opt.SkipVerifyTLSCert,
 		}
-		ftpConfig = append(ftpConfig, ftp.DialWithNetConn(conn))
+		ftpConfig = append(ftpConfig, ftp.DialWithTLS(tlsConfig))
+	} else if f.opt.ExplicitTLS {
+		tlsConfig := &tls.Config{
+			ServerName:         f.opt.Host,
+			InsecureSkipVerify: f.opt.SkipVerifyTLSCert,
+		}
+		ftpConfig = append(ftpConfig, ftp.DialWithExplicitTLS(tlsConfig))
 	}
 	if f.opt.DisableEPSV {
 		ftpConfig = append(ftpConfig, ftp.DialWithDisabledEPSV(true))
@@ -335,32 +308,9 @@ func (f *Fs) putFtpConnection(pc **ftp.ServerConn, err error) {
 	}
 	f.poolMu.Lock()
 	f.pool = append(f.pool, c)
-	if f.opt.IdleTimeout > 0 {
-		f.drain.Reset(time.Duration(f.opt.IdleTimeout)) // nudge on the pool emptying timer
-	}
 	f.poolMu.Unlock()
 }
 
-// Drain the pool of any connections
-func (f *Fs) drainPool(ctx context.Context) (err error) {
-	f.poolMu.Lock()
-	defer f.poolMu.Unlock()
-	if f.opt.IdleTimeout > 0 {
-		f.drain.Stop()
-	}
-	if len(f.pool) != 0 {
-		fs.Debugf(f, "closing %d unused connections", len(f.pool))
-	}
-	for i, c := range f.pool {
-		if cErr := c.Quit(); cErr != nil {
-			err = cErr
-		}
-		f.pool[i] = nil
-	}
-	f.pool = nil
-	return err
-}
-
 // NewFs constructs an Fs from the path, container:path
 func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs, err error) {
 	// defer fs.Trace(nil, "name=%q, root=%q", name, root)("fs=%v, err=%v", &ff, &err)
@@ -388,16 +338,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs
 	if opt.TLS {
 		protocol = "ftps://"
 	}
-	if opt.TLS && opt.ExplicitTLS {
-		return nil, errors.New("Implicit TLS and explicit TLS are mutually incompatible. Please revise your config")
-	}
-	var tlsConfig *tls.Config
-	if opt.TLS || opt.ExplicitTLS {
-		tlsConfig = &tls.Config{
-			ServerName:         opt.Host,
-			InsecureSkipVerify: opt.SkipVerifyTLSCert,
-		}
-	}
 	u := protocol + path.Join(dialAddr+"/", root)
 	ci := fs.GetConfig(ctx)
 	f := &Fs{
@@ -410,15 +350,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs
 		pass:     pass,
 		dialAddr: dialAddr,
 		tokens:   pacer.NewTokenDispenser(opt.Concurrency),
-		tlsConf:  tlsConfig,
 	}
 	f.features = (&fs.Features{
 		CanHaveEmptyDirectories: true,
 	}).Fill(ctx, f)
-	// set the pool drainer timer going
-	if f.opt.IdleTimeout > 0 {
-		f.drain = time.AfterFunc(time.Duration(opt.IdleTimeout), func() { _ = f.drainPool(ctx) })
-	}
 	// Make a connection and pool it to return errors early
 	c, err := f.getFtpConnection(ctx)
 	if err != nil {
@@ -447,12 +382,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (ff fs.Fs
 	return f, err
 }
 
-// Shutdown the backend, closing any background tasks and any
-// cached connections.
-func (f *Fs) Shutdown(ctx context.Context) error {
-	return f.drainPool(ctx)
-}
-
 // translateErrorFile turns FTP errors into rclone errors if possible for a file
 func translateErrorFile(err error) error {
 	switch errX := err.(type) {
@@ -598,7 +527,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
 	}()
 
 	// Wait for List for up to Timeout seconds
-	timer := time.NewTimer(f.ci.TimeoutOrInfinite())
+	timer := time.NewTimer(f.ci.Timeout)
 	select {
 	case listErr = <-errchan:
 		timer.Stop()
@@ -1061,6 +990,5 @@ var (
 	_ fs.Mover       = &Fs{}
 	_ fs.DirMover    = &Fs{}
 	_ fs.PutStreamer = &Fs{}
-	_ fs.Shutdowner  = &Fs{}
 	_ fs.Object      = &Object{}
 )

backend/googlecloudstorage/googlecloudstorage.go

@@ -411,7 +411,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		opt.ServiceAccountCredentials = string(loadedCreds)
 	}
 	if opt.Anonymous {
-		oAuthClient = fshttp.NewClient(ctx)
+		oAuthClient = &http.Client{}
 	} else if opt.ServiceAccountCredentials != "" {
 		oAuthClient, err = getServiceAccountClient(ctx, []byte(opt.ServiceAccountCredentials))
 		if err != nil {
@@ -564,7 +564,7 @@ func (f *Fs) list(ctx context.Context, bucket, directory, prefix string, addBuck
 				remote = path.Join(bucket, remote)
 			}
 			// is this a directory marker?
-			if isDirectory {
+			if isDirectory && object.Size == 0 {
 				continue // skip directory marker
 			}
 			err = fn(remote, object, false)
@@ -1028,10 +1028,11 @@ func (o *Object) Storable() bool {
 
 // Open an object for read
 func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
-	req, err := http.NewRequestWithContext(ctx, "GET", o.url, nil)
+	req, err := http.NewRequest("GET", o.url, nil)
 	if err != nil {
 		return nil, err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	fs.FixRangeOption(options, o.bytes)
 	fs.OpenOptionAddHTTPHeaders(req.Header, options)
 	var res *http.Response

backend/googlephotos/googlephotos.go

@@ -132,33 +132,15 @@ you want to read the media.`,
 			Default:  2000,
 			Help:     `Year limits the photos to be downloaded to those which are uploaded after the given year`,
 			Advanced: true,
-		}, {
-			Name:    "include_archived",
-			Default: false,
-			Help: `Also view and download archived media.
-
-By default rclone does not request archived media. Thus, when syncing,
-archived media is not visible in directory listings or transferred.
-
-Note that media in albums is always visible and synced, no matter
-their archive status.
-
-With this flag, archived media are always visible in directory
-listings and transferred.
-
-Without this flag, archived media will not be visible in directory
-listings and won't be transferred.`,
-			Advanced: true,
 		}}...),
 	})
 }
 
 // Options defines the configuration for this backend
 type Options struct {
-	ReadOnly        bool `config:"read_only"`
-	ReadSize        bool `config:"read_size"`
-	StartYear       int  `config:"start_year"`
-	IncludeArchived bool `config:"include_archived"`
+	ReadOnly  bool `config:"read_only"`
+	ReadSize  bool `config:"read_size"`
+	StartYear int  `config:"start_year"`
 }
 
 // Fs represents a remote storage server
@@ -224,10 +206,6 @@ func (f *Fs) startYear() int {
 	return f.opt.StartYear
 }
 
-func (f *Fs) includeArchived() bool {
-	return f.opt.IncludeArchived
-}
-
 // retryErrorCodes is a slice of error codes that we will retry
 var retryErrorCodes = []int{
 	429, // Too Many Requests.
@@ -519,12 +497,6 @@ func (f *Fs) list(ctx context.Context, filter api.SearchFilter, fn listFn) (err
 	}
 	filter.PageSize = listChunks
 	filter.PageToken = ""
-	if filter.AlbumID == "" { // album ID and filters cannot be set together, else error 400 INVALID_ARGUMENT
-		if filter.Filters == nil {
-			filter.Filters = &api.Filters{}
-		}
-		filter.Filters.IncludeArchivedMedia = &f.opt.IncludeArchived
-	}
 	lastID := ""
 	for {
 		var result api.MediaItems

backend/googlephotos/pattern.go

@@ -24,7 +24,6 @@ type lister interface {
 	listUploads(ctx context.Context, dir string) (entries fs.DirEntries, err error)
 	dirTime() time.Time
 	startYear() int
-	includeArchived() bool
 }
 
 // dirPattern describes a single directory pattern

backend/googlephotos/pattern_test.go

@@ -64,11 +64,6 @@ func (f *testLister) startYear() int {
 	return 2000
 }
 
-// mock includeArchived for testing
-func (f *testLister) includeArchived() bool {
-	return false
-}
-
 func TestPatternMatch(t *testing.T) {
 	for testNumber, test := range []struct {
 		// input

backend/hdfs/fs.go

@@ -7,15 +7,10 @@ import (
 	"fmt"
 	"io"
 	"os"
-	"os/user"
 	"path"
-	"strings"
 	"time"
 
 	"github.com/colinmarc/hdfs/v2"
-	krb "github.com/jcmturner/gokrb5/v8/client"
-	"github.com/jcmturner/gokrb5/v8/config"
-	"github.com/jcmturner/gokrb5/v8/credentials"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config/configmap"
 	"github.com/rclone/rclone/fs/config/configstruct"
@@ -32,49 +27,6 @@ type Fs struct {
 	client   *hdfs.Client
 }
 
-// copy-paste from https://github.com/colinmarc/hdfs/blob/master/cmd/hdfs/kerberos.go
-func getKerberosClient() (*krb.Client, error) {
-	configPath := os.Getenv("KRB5_CONFIG")
-	if configPath == "" {
-		configPath = "/etc/krb5.conf"
-	}
-
-	cfg, err := config.Load(configPath)
-	if err != nil {
-		return nil, err
-	}
-
-	// Determine the ccache location from the environment, falling back to the
-	// default location.
-	ccachePath := os.Getenv("KRB5CCNAME")
-	if strings.Contains(ccachePath, ":") {
-		if strings.HasPrefix(ccachePath, "FILE:") {
-			ccachePath = strings.SplitN(ccachePath, ":", 2)[1]
-		} else {
-			return nil, fmt.Errorf("unusable ccache: %s", ccachePath)
-		}
-	} else if ccachePath == "" {
-		u, err := user.Current()
-		if err != nil {
-			return nil, err
-		}
-
-		ccachePath = fmt.Sprintf("/tmp/krb5cc_%s", u.Uid)
-	}
-
-	ccache, err := credentials.LoadCCache(ccachePath)
-	if err != nil {
-		return nil, err
-	}
-
-	client, err := krb.NewFromCCache(ccache, cfg)
-	if err != nil {
-		return nil, err
-	}
-
-	return client, nil
-}
-
 // NewFs constructs an Fs from the path
 func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, error) {
 	opt := new(Options)
@@ -83,26 +35,11 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		return nil, err
 	}
 
-	options := hdfs.ClientOptions{
+	client, err := hdfs.NewClient(hdfs.ClientOptions{
 		Addresses:           []string{opt.Namenode},
+		User:                opt.Username,
 		UseDatanodeHostname: false,
-	}
-
-	if opt.ServicePrincipalName != "" {
-		options.KerberosClient, err = getKerberosClient()
-		if err != nil {
-			return nil, fmt.Errorf("Problem with kerberos authentication: %s", err)
-		}
-		options.KerberosServicePrincipleName = opt.ServicePrincipalName
-
-		if opt.DataTransferProtection != "" {
-			options.DataTransferProtection = opt.DataTransferProtection
-		}
-	} else {
-		options.User = opt.Username
-	}
-
-	client, err := hdfs.NewClient(options)
+	})
 	if err != nil {
 		return nil, err
 	}

backend/hdfs/hdfs.go

@@ -32,32 +32,6 @@ func init() {
 				Value: "root",
 				Help:  "Connect to hdfs as root",
 			}},
-		}, {
-			Name: "service_principal_name",
-			Help: `Kerberos service principal name for the namenode
-
-Enables KERBEROS authentication. Specifies the Service Principal Name
-(<SERVICE>/<FQDN>) for the namenode.`,
-			Required: false,
-			Examples: []fs.OptionExample{{
-				Value: "hdfs/namenode.hadoop.docker",
-				Help:  "Namenode running as service 'hdfs' with FQDN 'namenode.hadoop.docker'.",
-			}},
-			Advanced: true,
-		}, {
-			Name: "data_transfer_protection",
-			Help: `Kerberos data transfer protection: authentication|integrity|privacy
-
-Specifies whether or not authentication, data signature integrity
-checks, and wire encryption is required when communicating the the
-datanodes. Possible values are 'authentication', 'integrity' and
-'privacy'. Used only with KERBEROS enabled.`,
-			Required: false,
-			Examples: []fs.OptionExample{{
-				Value: "privacy",
-				Help:  "Ensure authentication, integrity and encryption enabled.",
-			}},
-			Advanced: true,
 		}, {
 			Name:     config.ConfigEncoding,
 			Help:     config.ConfigEncodingHelp,
@@ -70,11 +44,9 @@ datanodes. Possible values are 'authentication', 'integrity' and
 
 // Options for this backend
 type Options struct {
-	Namenode               string               `config:"namenode"`
-	Username               string               `config:"username"`
-	ServicePrincipalName   string               `config:"service_principal_name"`
-	DataTransferProtection string               `config:"data_transfer_protection"`
-	Enc                    encoder.MultiEncoder `config:"encoding"`
+	Namenode string               `config:"namenode"`
+	Username string               `config:"username"`
+	Enc      encoder.MultiEncoder `config:"encoding"`
 }
 
 // xPath make correct file path with leading '/'

backend/http/http.go

@@ -183,8 +183,9 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 			return http.ErrUseLastResponse
 		}
 		// check to see if points to a file
-		req, err := http.NewRequestWithContext(ctx, "HEAD", u.String(), nil)
+		req, err := http.NewRequest("HEAD", u.String(), nil)
 		if err == nil {
+			req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 			addHeaders(req, opt)
 			res, err := noRedir.Do(req)
 			err = statusError(res, err)
@@ -390,10 +391,11 @@ func (f *Fs) readDir(ctx context.Context, dir string) (names []string, err error
 		return nil, errors.Errorf("internal error: readDir URL %q didn't end in /", URL)
 	}
 	// Do the request
-	req, err := http.NewRequestWithContext(ctx, "GET", URL, nil)
+	req, err := http.NewRequest("GET", URL, nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "readDir failed")
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	f.addHeaders(req)
 	res, err := f.httpClient.Do(req)
 	if err == nil {
@@ -545,10 +547,11 @@ func (o *Object) stat(ctx context.Context) error {
 		return nil
 	}
 	url := o.url()
-	req, err := http.NewRequestWithContext(ctx, "HEAD", url, nil)
+	req, err := http.NewRequest("HEAD", url, nil)
 	if err != nil {
 		return errors.Wrap(err, "stat failed")
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	o.fs.addHeaders(req)
 	res, err := o.fs.httpClient.Do(req)
 	if err == nil && res.StatusCode == http.StatusNotFound {
@@ -593,10 +596,11 @@ func (o *Object) Storable() bool {
 // Open a remote http file object for reading. Seek is supported
 func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
 	url := o.url()
-	req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
+	req, err := http.NewRequest("GET", url, nil)
 	if err != nil {
 		return nil, errors.Wrap(err, "Open failed")
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 
 	// Add optional headers
 	for k, v := range fs.OpenOptionHeaders(options) {

backend/hubic/auth.go

@@ -5,7 +5,7 @@ import (
 	"net/http"
 	"time"
 
-	"github.com/ncw/swift/v2"
+	"github.com/ncw/swift"
 	"github.com/rclone/rclone/fs"
 )
 
@@ -24,7 +24,7 @@ func newAuth(f *Fs) *auth {
 // Request constructs an http.Request for authentication
 //
 // returns nil for not needed
-func (a *auth) Request(ctx context.Context, c *swift.Connection) (r *http.Request, err error) {
+func (a *auth) Request(*swift.Connection) (r *http.Request, err error) {
 	const retries = 10
 	for try := 1; try <= retries; try++ {
 		err = a.f.getCredentials(context.TODO())
@@ -38,7 +38,7 @@ func (a *auth) Request(ctx context.Context, c *swift.Connection) (r *http.Reques
 }
 
 // Response parses the result of an http request
-func (a *auth) Response(ctx context.Context, resp *http.Response) error {
+func (a *auth) Response(resp *http.Response) error {
 	return nil
 }
 

backend/hubic/hubic.go

@@ -16,7 +16,7 @@ import (
 	"strings"
 	"time"
 
-	swiftLib "github.com/ncw/swift/v2"
+	swiftLib "github.com/ncw/swift"
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/backend/swift"
 	"github.com/rclone/rclone/fs"
@@ -110,10 +110,11 @@ func (f *Fs) String() string {
 //
 // The credentials are read into the Fs
 func (f *Fs) getCredentials(ctx context.Context) (err error) {
-	req, err := http.NewRequestWithContext(ctx, "GET", "https://api.hubic.com/1.0/account/credentials", nil)
+	req, err := http.NewRequest("GET", "https://api.hubic.com/1.0/account/credentials", nil)
 	if err != nil {
 		return err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	resp, err := f.client.Do(req)
 	if err != nil {
 		return err
@@ -163,7 +164,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		Timeout:        10 * ci.Timeout,        // Use the timeouts in the transport
 		Transport:      fshttp.NewTransport(ctx),
 	}
-	err = c.Authenticate(ctx)
+	err = c.Authenticate()
 	if err != nil {
 		return nil, errors.Wrap(err, "error authenticating swift connection")
 	}

backend/jottacloud/jottacloud.go

@@ -63,10 +63,6 @@ const (
 	v1ClientID              = "nibfk8biu12ju7hpqomr8b1e40"
 	v1EncryptedClientSecret = "Vp8eAv7eVElMnQwN-kgU9cbhgApNDaMqWdlDi5qFydlQoji4JBxrGMF2"
 	v1configVersion         = 0
-
-	teliaCloudTokenURL = "https://cloud-auth.telia.se/auth/realms/telia_se/protocol/openid-connect/token"
-	teliaCloudAuthURL  = "https://cloud-auth.telia.se/auth/realms/telia_se/protocol/openid-connect/auth"
-	teliaCloudClientID = "desktop"
 )
 
 var (
@@ -109,18 +105,11 @@ func init() {
 				}
 			}
 
-			fmt.Printf("Choose authentication type:\n" +
-				"1: Standard authentication - use this if you're a normal Jottacloud user.\n" +
-				"2: Legacy authentication - this is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.\n" +
-				"3: Telia Cloud authentication - use this if you are using Telia Cloud.\n")
-
-			switch config.ChooseNumber("Your choice", 1, 3) {
-			case 1:
-				v2config(ctx, name, m)
-			case 2:
+			fmt.Printf("Use legacy authentication?.\nThis is only required for certain whitelabel versions of Jottacloud and not recommended for normal users.\n")
+			if config.Confirm(false) {
 				v1config(ctx, name, m)
-			case 3:
-				teliaCloudConfig(ctx, name, m)
+			} else {
+				v2config(ctx, name, m)
 			}
 		},
 		Options: []fs.Option{{
@@ -239,46 +228,6 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
 	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
 }
 
-func teliaCloudConfig(ctx context.Context, name string, m configmap.Mapper) {
-	teliaCloudOauthConfig := &oauth2.Config{
-		Endpoint: oauth2.Endpoint{
-			AuthURL:  teliaCloudAuthURL,
-			TokenURL: teliaCloudTokenURL,
-		},
-		ClientID:    teliaCloudClientID,
-		Scopes:      []string{"openid", "jotta-default", "offline_access"},
-		RedirectURL: oauthutil.RedirectLocalhostURL,
-	}
-
-	err := oauthutil.Config(ctx, "jottacloud", name, m, teliaCloudOauthConfig, nil)
-	if err != nil {
-		log.Fatalf("Failed to configure token: %v", err)
-		return
-	}
-
-	fmt.Printf("\nDo you want to use a non standard device/mountpoint e.g. for accessing files uploaded using the official Jottacloud client?\n\n")
-	if config.Confirm(false) {
-		oAuthClient, _, err := oauthutil.NewClient(ctx, name, m, teliaCloudOauthConfig)
-		if err != nil {
-			log.Fatalf("Failed to load oAuthClient: %s", err)
-		}
-
-		srv := rest.NewClient(oAuthClient).SetRoot(rootURL)
-		apiSrv := rest.NewClient(oAuthClient).SetRoot(apiURL)
-
-		device, mountpoint, err := setupMountpoint(ctx, srv, apiSrv)
-		if err != nil {
-			log.Fatalf("Failed to setup mountpoint: %s", err)
-		}
-		m.Set(configDevice, device)
-		m.Set(configMountpoint, mountpoint)
-	}
-
-	m.Set("configVersion", strconv.Itoa(configVersion))
-	m.Set(configClientID, teliaCloudClientID)
-	m.Set(configTokenURL, teliaCloudTokenURL)
-}
-
 // v1config configure a jottacloud backend using legacy authentication
 func v1config(ctx context.Context, name string, m configmap.Mapper) {
 	srv := rest.NewClient(fshttp.NewClient(ctx))

backend/local/local.go

@@ -70,20 +70,6 @@ points, as you explicitly acknowledge that they should be skipped.`,
 			Default:  false,
 			NoPrefix: true,
 			Advanced: true,
-		}, {
-			Name: "zero_size_links",
-			Help: `Assume the Stat size of links is zero (and read them instead)
-
-On some virtual filesystems (such ash LucidLink), reading a link size via a Stat call always returns 0.
-However, on unix it reads as the length of the text in the link. This may cause errors like this when
-syncing:
-
-    Failed to copy: corrupted on transfer: sizes differ 0 vs 13
-
-Setting this flag causes rclone to read the link and use that as the size of the link
-instead of 0 which in most cases fixes the problem.`,
-			Default:  false,
-			Advanced: true,
 		}, {
 			Name: "no_unicode_normalization",
 			Help: `Don't apply unicode normalization to paths and filenames (Deprecated)
@@ -148,17 +134,6 @@ Windows/macOS and case sensitive for everything else.  Use this flag
 to override the default choice.`,
 			Default:  false,
 			Advanced: true,
-		}, {
-			Name: "no_preallocate",
-			Help: `Disable preallocation of disk space for transferred files
-
-Preallocation of disk space helps prevent filesystem fragmentation.
-However, some virtual filesystem layers (such as Google Drive File
-Stream) may incorrectly set the actual file size equal to the
-preallocated space, causing checksum and file size checks to fail.
-Use this flag to disable preallocation.`,
-			Default:  false,
-			Advanced: true,
 		}, {
 			Name: "no_sparse",
 			Help: `Disable sparse files for multi-thread downloads
@@ -195,14 +170,12 @@ type Options struct {
 	FollowSymlinks    bool                 `config:"copy_links"`
 	TranslateSymlinks bool                 `config:"links"`
 	SkipSymlinks      bool                 `config:"skip_links"`
-	ZeroSizeLinks     bool                 `config:"zero_size_links"`
 	NoUTFNorm         bool                 `config:"no_unicode_normalization"`
 	NoCheckUpdated    bool                 `config:"no_check_updated"`
 	NoUNC             bool                 `config:"nounc"`
 	OneFileSystem     bool                 `config:"one_file_system"`
 	CaseSensitive     bool                 `config:"case_sensitive"`
 	CaseInsensitive   bool                 `config:"case_insensitive"`
-	NoPreAllocate     bool                 `config:"no_preallocate"`
 	NoSparse          bool                 `config:"no_sparse"`
 	NoSetModTime      bool                 `config:"no_set_modtime"`
 	Enc               encoder.MultiEncoder `config:"encoding"`
@@ -1139,12 +1112,10 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 				return err
 			}
 		}
-		if !o.fs.opt.NoPreAllocate {
-			// Pre-allocate the file for performance reasons
-			err = file.PreAllocate(src.Size(), f)
-			if err != nil {
-				fs.Debugf(o, "Failed to pre-allocate: %v", err)
-			}
+		// Pre-allocate the file for performance reasons
+		err = file.PreAllocate(src.Size(), f)
+		if err != nil {
+			fs.Debugf(o, "Failed to pre-allocate: %v", err)
 		}
 		out = f
 	} else {
@@ -1231,11 +1202,9 @@ func (f *Fs) OpenWriterAt(ctx context.Context, remote string, size int64) (fs.Wr
 		return nil, err
 	}
 	// Pre-allocate the file for performance reasons
-	if !f.opt.NoPreAllocate {
-		err = file.PreAllocate(size, out)
-		if err != nil {
-			fs.Debugf(o, "Failed to pre-allocate: %v", err)
-		}
+	err = file.PreAllocate(size, out)
+	if err != nil {
+		fs.Debugf(o, "Failed to pre-allocate: %v", err)
 	}
 	if !f.opt.NoSparse && file.SetSparseImplemented {
 		sparseWarning.Do(func() {
@@ -1263,8 +1232,7 @@ func (o *Object) setMetadata(info os.FileInfo) {
 	o.mode = info.Mode()
 	o.fs.objectMetaMu.Unlock()
 	// On Windows links read as 0 size so set the correct size here
-	// Optionally, users can turn this feature on with the zero_size_links flag
-	if (runtime.GOOS == "windows" || o.fs.opt.ZeroSizeLinks) && o.translatedLink {
+	if runtime.GOOS == "windows" && o.translatedLink {
 		linkdst, err := os.Readlink(o.path)
 		if err != nil {
 			fs.Errorf(o, "Failed to read link size: %v", err)

backend/onedrive/onedrive.go

@@ -11,7 +11,6 @@ import (
 	"io"
 	"log"
 	"net/http"
-	"net/url"
 	"path"
 	"regexp"
 	"strconv"
@@ -47,6 +46,7 @@ const (
 	minSleep                    = 10 * time.Millisecond
 	maxSleep                    = 2 * time.Second
 	decayConstant               = 2 // bigger for slower decay, exponential
+	graphURL                    = "https://graph.microsoft.com/v1.0"
 	configDriveID               = "drive_id"
 	configDriveType             = "drive_type"
 	driveTypePersonal           = "personal"
@@ -54,40 +54,22 @@ const (
 	driveTypeSharepoint         = "documentLibrary"
 	defaultChunkSize            = 10 * fs.MebiByte
 	chunkSizeMultiple           = 320 * fs.KibiByte
-
-	regionGlobal = "global"
-	regionUS     = "us"
-	regionDE     = "de"
-	regionCN     = "cn"
 )
 
 // Globals
 var (
-	authPath  = "/common/oauth2/v2.0/authorize"
-	tokenPath = "/common/oauth2/v2.0/token"
-
 	// Description of how to auth for this app for a business account
 	oauthConfig = &oauth2.Config{
+		Endpoint: oauth2.Endpoint{
+			AuthURL:  "https://login.microsoftonline.com/common/oauth2/v2.0/authorize",
+			TokenURL: "https://login.microsoftonline.com/common/oauth2/v2.0/token",
+		},
 		Scopes:       []string{"Files.Read", "Files.ReadWrite", "Files.Read.All", "Files.ReadWrite.All", "offline_access", "Sites.Read.All"},
 		ClientID:     rcloneClientID,
 		ClientSecret: obscure.MustReveal(rcloneEncryptedClientSecret),
 		RedirectURL:  oauthutil.RedirectLocalhostURL,
 	}
 
-	graphAPIEndpoint = map[string]string{
-		"global": "https://graph.microsoft.com",
-		"us":     "https://graph.microsoft.us",
-		"de":     "https://graph.microsoft.de",
-		"cn":     "https://microsoftgraph.chinacloudapi.cn",
-	}
-
-	authEndpoint = map[string]string{
-		"global": "https://login.microsoftonline.com",
-		"us":     "https://login.microsoftonline.us",
-		"de":     "https://login.microsoftonline.de",
-		"cn":     "https://login.chinacloudapi.cn",
-	}
-
 	// QuickXorHashType is the hash.Type for OneDrive
 	QuickXorHashType hash.Type
 )
@@ -100,12 +82,6 @@ func init() {
 		Description: "Microsoft OneDrive",
 		NewFs:       NewFs,
 		Config: func(ctx context.Context, name string, m configmap.Mapper) {
-			region, _ := m.Get("region")
-			graphURL := graphAPIEndpoint[region] + "/v1.0"
-			oauthConfig.Endpoint = oauth2.Endpoint{
-				AuthURL:  authEndpoint[region] + authPath,
-				TokenURL: authEndpoint[region] + tokenPath,
-			}
 			ci := fs.GetConfig(ctx)
 			err := oauthutil.Config(ctx, "onedrive", name, m, oauthConfig, nil)
 			if err != nil {
@@ -305,25 +281,6 @@ func init() {
 			config.SaveConfig()
 		},
 		Options: append(oauthutil.SharedOptions, []fs.Option{{
-			Name:    "region",
-			Help:    "Choose national cloud region for OneDrive.",
-			Default: "global",
-			Examples: []fs.OptionExample{
-				{
-					Value: regionGlobal,
-					Help:  "Microsoft Cloud Global",
-				}, {
-					Value: regionUS,
-					Help:  "Microsoft Cloud for US Government",
-				}, {
-					Value: regionDE,
-					Help:  "Microsoft Cloud Germany",
-				}, {
-					Value: regionCN,
-					Help:  "Azure and Office 365 operated by 21Vianet in China",
-				},
-			},
-		}, {
 			Name: "chunk_size",
 			Help: `Chunk size to upload files with - must be multiple of 320k (327,680 bytes).
 
@@ -463,7 +420,6 @@ At the time of writing this only works with OneDrive personal paid accounts.
 
 // Options defines the configuration for this backend
 type Options struct {
-	Region                  string               `config:"region"`
 	ChunkSize               fs.SizeSuffix        `config:"chunk_size"`
 	DriveID                 string               `config:"drive_id"`
 	DriveType               string               `config:"drive_type"`
@@ -593,8 +549,10 @@ func shouldRetry(resp *http.Response, err error) (bool, error) {
 //
 // If `relPath` == '', do not append the slash (See #3664)
 func (f *Fs) readMetaDataForPathRelativeToID(ctx context.Context, normalizedID string, relPath string) (info *api.Item, resp *http.Response, err error) {
-	opts, _ := f.newOptsCallWithIDPath(normalizedID, relPath, true, "GET", "")
-
+	if relPath != "" {
+		relPath = "/" + withTrailingColon(rest.URLPathEscape(f.opt.Enc.FromStandardPath(relPath)))
+	}
+	opts := newOptsCall(normalizedID, "GET", ":"+relPath)
 	err = f.pacer.Call(func() (bool, error) {
 		resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
 		return shouldRetry(resp, err)
@@ -609,8 +567,17 @@ func (f *Fs) readMetaDataForPath(ctx context.Context, path string) (info *api.It
 
 	if f.driveType != driveTypePersonal || firstSlashIndex == -1 {
 		var opts rest.Opts
-		opts = f.newOptsCallWithPath(ctx, path, "GET", "")
-		opts.Path = strings.TrimSuffix(opts.Path, ":")
+		if len(path) == 0 {
+			opts = rest.Opts{
+				Method: "GET",
+				Path:   "/root",
+			}
+		} else {
+			opts = rest.Opts{
+				Method: "GET",
+				Path:   "/root:/" + rest.URLPathEscape(f.opt.Enc.FromStandardPath(path)),
+			}
+		}
 		err = f.pacer.Call(func() (bool, error) {
 			resp, err = f.srv.CallJSON(ctx, &opts, nil, &info)
 			return shouldRetry(resp, err)
@@ -721,12 +688,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		return nil, errors.New("unable to get drive_id and drive_type - if you are upgrading from older versions of rclone, please run `rclone config` and re-configure this backend")
 	}
 
-	rootURL := graphAPIEndpoint[opt.Region] + "/v1.0" + "/drives/" + opt.DriveID
-	oauthConfig.Endpoint = oauth2.Endpoint{
-		AuthURL:  authEndpoint[opt.Region] + authPath,
-		TokenURL: authEndpoint[opt.Region] + tokenPath,
-	}
-
 	root = parsePath(root)
 	oAuthClient, ts, err := oauthutil.NewClient(ctx, name, m, oauthConfig)
 	if err != nil {
@@ -741,7 +702,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		ci:        ci,
 		driveID:   opt.DriveID,
 		driveType: opt.DriveType,
-		srv:       rest.NewClient(oAuthClient).SetRoot(rootURL),
+		srv:       rest.NewClient(oAuthClient).SetRoot(graphURL + "/drives/" + opt.DriveID),
 		pacer:     fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
 	}
 	f.features = (&fs.Features{
@@ -862,7 +823,7 @@ func (f *Fs) CreateDir(ctx context.Context, dirID, leaf string) (newID string, e
 	// fs.Debugf(f, "CreateDir(%q, %q)\n", dirID, leaf)
 	var resp *http.Response
 	var info *api.Item
-	opts := f.newOptsCall(dirID, "POST", "/children")
+	opts := newOptsCall(dirID, "POST", "/children")
 	mkdir := api.CreateItemRequest{
 		Name:             f.opt.Enc.FromStandardName(leaf),
 		ConflictBehavior: "fail",
@@ -894,7 +855,7 @@ type listAllFn func(*api.Item) bool
 func (f *Fs) listAll(ctx context.Context, dirID string, directoriesOnly bool, filesOnly bool, fn listAllFn) (found bool, err error) {
 	// Top parameter asks for bigger pages of data
 	// https://dev.onedrive.com/odata/optional-query-parameters.htm
-	opts := f.newOptsCall(dirID, "GET", "/children?$top=1000")
+	opts := newOptsCall(dirID, "GET", "/children?$top=1000")
 OUTER:
 	for {
 		var result api.ListChildrenResponse
@@ -1033,7 +994,7 @@ func (f *Fs) Mkdir(ctx context.Context, dir string) error {
 
 // deleteObject removes an object by ID
 func (f *Fs) deleteObject(ctx context.Context, id string) error {
-	opts := f.newOptsCall(id, "DELETE", "")
+	opts := newOptsCall(id, "DELETE", "")
 	opts.NoResponse = true
 
 	return f.pacer.Call(func() (bool, error) {
@@ -1088,7 +1049,7 @@ func (f *Fs) Precision() time.Duration {
 
 // waitForJob waits for the job with status in url to complete
 func (f *Fs) waitForJob(ctx context.Context, location string, o *Object) error {
-	deadline := time.Now().Add(f.ci.TimeoutOrInfinite())
+	deadline := time.Now().Add(f.ci.Timeout)
 	for time.Now().Before(deadline) {
 		var resp *http.Response
 		var err error
@@ -1126,7 +1087,7 @@ func (f *Fs) waitForJob(ctx context.Context, location string, o *Object) error {
 
 		time.Sleep(1 * time.Second)
 	}
-	return errors.Errorf("async operation didn't complete after %v", f.ci.TimeoutOrInfinite())
+	return errors.Errorf("async operation didn't complete after %v", f.ci.Timeout)
 }
 
 // Copy src to this remote using server-side copy operations.
@@ -1177,11 +1138,11 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 
 	// Copy the object
 	// The query param is a workaround for OneDrive Business for #4590
-	opts := f.newOptsCall(srcObj.id, "POST", "/copy?@microsoft.graph.conflictBehavior=replace")
+	opts := newOptsCall(srcObj.id, "POST", "/copy?@microsoft.graph.conflictBehavior=replace")
 	opts.ExtraHeaders = map[string]string{"Prefer": "respond-async"}
 	opts.NoResponse = true
 
-	id, dstDriveID, _ := f.parseNormalizedID(directoryID)
+	id, dstDriveID, _ := parseNormalizedID(directoryID)
 
 	replacedLeaf := f.opt.Enc.FromStandardName(leaf)
 	copyReq := api.CopyItemRequest{
@@ -1258,8 +1219,8 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
 		return nil, err
 	}
 
-	id, dstDriveID, _ := f.parseNormalizedID(directoryID)
-	_, srcObjDriveID, _ := f.parseNormalizedID(srcObj.id)
+	id, dstDriveID, _ := parseNormalizedID(directoryID)
+	_, srcObjDriveID, _ := parseNormalizedID(srcObj.id)
 
 	if f.canonicalDriveID(dstDriveID) != srcObj.fs.canonicalDriveID(srcObjDriveID) {
 		// https://docs.microsoft.com/en-us/graph/api/driveitem-move?view=graph-rest-1.0
@@ -1269,7 +1230,7 @@ func (f *Fs) Move(ctx context.Context, src fs.Object, remote string) (fs.Object,
 	}
 
 	// Move the object
-	opts := f.newOptsCall(srcObj.id, "PATCH", "")
+	opts := newOptsCall(srcObj.id, "PATCH", "")
 
 	move := api.MoveItemRequest{
 		Name: f.opt.Enc.FromStandardName(leaf),
@@ -1320,8 +1281,8 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
 		return err
 	}
 
-	parsedDstDirID, dstDriveID, _ := f.parseNormalizedID(dstDirectoryID)
-	_, srcDriveID, _ := f.parseNormalizedID(srcID)
+	parsedDstDirID, dstDriveID, _ := parseNormalizedID(dstDirectoryID)
+	_, srcDriveID, _ := parseNormalizedID(srcID)
 
 	if f.canonicalDriveID(dstDriveID) != srcFs.canonicalDriveID(srcDriveID) {
 		// https://docs.microsoft.com/en-us/graph/api/driveitem-move?view=graph-rest-1.0
@@ -1337,7 +1298,7 @@ func (f *Fs) DirMove(ctx context.Context, src fs.Fs, srcRemote, dstRemote string
 	}
 
 	// Do the move
-	opts := f.newOptsCall(srcID, "PATCH", "")
+	opts := newOptsCall(srcID, "PATCH", "")
 	move := api.MoveItemRequest{
 		Name: f.opt.Enc.FromStandardName(dstLeaf),
 		ParentReference: &api.ItemReference{
@@ -1413,7 +1374,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
 	if err != nil {
 		return "", err
 	}
-	opts := f.newOptsCall(info.GetID(), "POST", "/createLink")
+	opts := newOptsCall(info.GetID(), "POST", "/createLink")
 
 	share := api.CreateShareLinkRequest{
 		Type:     f.opt.LinkType,
@@ -1471,7 +1432,7 @@ func (f *Fs) CleanUp(ctx context.Context) error {
 
 // Finds and removes any old versions for o
 func (o *Object) deleteVersions(ctx context.Context) error {
-	opts := o.fs.newOptsCall(o.id, "GET", "/versions")
+	opts := newOptsCall(o.id, "GET", "/versions")
 	var versions api.VersionsResponse
 	err := o.fs.pacer.Call(func() (bool, error) {
 		resp, err := o.fs.srv.CallJSON(ctx, &opts, nil, &versions)
@@ -1498,7 +1459,7 @@ func (o *Object) deleteVersion(ctx context.Context, ID string) error {
 		return nil
 	}
 	fs.Infof(o, "removing version %q", ID)
-	opts := o.fs.newOptsCall(o.id, "DELETE", "/versions/"+ID)
+	opts := newOptsCall(o.id, "DELETE", "/versions/"+ID)
 	opts.NoResponse = true
 	return o.fs.pacer.Call(func() (bool, error) {
 		resp, err := o.fs.srv.Call(ctx, &opts)
@@ -1643,7 +1604,21 @@ func (o *Object) ModTime(ctx context.Context) time.Time {
 
 // setModTime sets the modification time of the local fs object
 func (o *Object) setModTime(ctx context.Context, modTime time.Time) (*api.Item, error) {
-	opts := o.fs.newOptsCallWithPath(ctx, o.remote, "PATCH", "")
+	var opts rest.Opts
+	leaf, directoryID, _ := o.fs.dirCache.FindPath(ctx, o.remote, false)
+	trueDirID, drive, rootURL := parseNormalizedID(directoryID)
+	if drive != "" {
+		opts = rest.Opts{
+			Method:  "PATCH",
+			RootURL: rootURL,
+			Path:    "/" + drive + "/items/" + trueDirID + ":/" + withTrailingColon(rest.URLPathEscape(o.fs.opt.Enc.FromStandardName(leaf))),
+		}
+	} else {
+		opts = rest.Opts{
+			Method: "PATCH",
+			Path:   "/root:/" + withTrailingColon(rest.URLPathEscape(o.srvPath())),
+		}
+	}
 	update := api.SetFileSystemInfo{
 		FileSystemInfo: api.FileSystemInfoFacet{
 			CreatedDateTime:      api.Timestamp(modTime),
@@ -1690,7 +1665,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 
 	fs.FixRangeOption(options, o.size)
 	var resp *http.Response
-	opts := o.fs.newOptsCall(o.id, "GET", "/content")
+	opts := newOptsCall(o.id, "GET", "/content")
 	opts.Options = options
 
 	err = o.fs.pacer.Call(func() (bool, error) {
@@ -1710,7 +1685,22 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 
 // createUploadSession creates an upload session for the object
 func (o *Object) createUploadSession(ctx context.Context, modTime time.Time) (response *api.CreateUploadResponse, err error) {
-	opts := o.fs.newOptsCallWithPath(ctx, o.remote, "POST", "/createUploadSession")
+	leaf, directoryID, _ := o.fs.dirCache.FindPath(ctx, o.remote, false)
+	id, drive, rootURL := parseNormalizedID(directoryID)
+	var opts rest.Opts
+	if drive != "" {
+		opts = rest.Opts{
+			Method:  "POST",
+			RootURL: rootURL,
+			Path: fmt.Sprintf("/%s/items/%s:/%s:/createUploadSession",
+				drive, id, rest.URLPathEscape(o.fs.opt.Enc.FromStandardName(leaf))),
+		}
+	} else {
+		opts = rest.Opts{
+			Method: "POST",
+			Path:   "/root:/" + rest.URLPathEscape(o.srvPath()) + ":/createUploadSession",
+		}
+	}
 	createRequest := api.CreateUploadRequest{}
 	createRequest.Item.FileSystemInfo.CreatedDateTime = api.Timestamp(modTime)
 	createRequest.Item.FileSystemInfo.LastModifiedDateTime = api.Timestamp(modTime)
@@ -1883,10 +1873,27 @@ func (o *Object) uploadSinglepart(ctx context.Context, in io.Reader, size int64,
 
 	fs.Debugf(o, "Starting singlepart upload")
 	var resp *http.Response
-	opts := o.fs.newOptsCallWithPath(ctx, o.remote, "PUT", "/content")
-	opts.ContentLength = &size
-	opts.Body = in
-	opts.Options = options
+	var opts rest.Opts
+	leaf, directoryID, _ := o.fs.dirCache.FindPath(ctx, o.remote, false)
+	trueDirID, drive, rootURL := parseNormalizedID(directoryID)
+	if drive != "" {
+		opts = rest.Opts{
+			Method:        "PUT",
+			RootURL:       rootURL,
+			Path:          "/" + drive + "/items/" + trueDirID + ":/" + rest.URLPathEscape(o.fs.opt.Enc.FromStandardName(leaf)) + ":/content",
+			ContentLength: &size,
+			Body:          in,
+			Options:       options,
+		}
+	} else {
+		opts = rest.Opts{
+			Method:        "PUT",
+			Path:          "/root:/" + rest.URLPathEscape(o.srvPath()) + ":/content",
+			ContentLength: &size,
+			Body:          in,
+			Options:       options,
+		}
+	}
 
 	err = o.fs.pacer.Call(func() (bool, error) {
 		resp, err = o.fs.srv.CallJSON(ctx, &opts, nil, &info)
@@ -1962,42 +1969,8 @@ func (o *Object) ID() string {
 	return o.id
 }
 
-/*
- *       URL Build routine area start
- *       1. In this area, region-related URL rewrites are applied. As the API is blackbox,
- *          we cannot thoroughly test this part. Please be extremely careful while changing them.
- *       2. If possible, please don't introduce region related code in other region, but patch these helper functions.
- *       3. To avoid region-related issues, please don't manually build rest.Opts from scratch.
- *          Instead, use these helper function, and customize the URL afterwards if needed.
- *
- *       currently, the 21ViaNet's API differs in the following places:
- *       - https://{Endpoint}/drives/{driveID}/items/{leaf}:/{route}
- *           - this API doesn't work (gives invalid request)
- *           - can be replaced with the following API:
- *               - https://{Endpoint}/drives/{driveID}/items/children('{leaf}')/{route}
- *                   - however, this API does NOT support multi-level leaf like a/b/c
- *               - https://{Endpoint}/drives/{driveID}/items/children('@a1')/{route}?@a1=URLEncode("'{leaf}'")
- *                   - this API does support multi-level leaf like a/b/c
- *       - https://{Endpoint}/drives/{driveID}/root/children('@a1')/{route}?@a1=URLEncode({path})
- *	         - Same as above
- */
-
-// parseNormalizedID parses a normalized ID (may be in the form `driveID#itemID` or just `itemID`)
-// and returns itemID, driveID, rootURL.
-// Such a normalized ID can come from (*Item).GetID()
-func (f *Fs) parseNormalizedID(ID string) (string, string, string) {
-	rootURL := graphAPIEndpoint[f.opt.Region] + "/v1.0/drives"
-	if strings.Index(ID, "#") >= 0 {
-		s := strings.Split(ID, "#")
-		return s[1], s[0], rootURL
-	}
-	return ID, "", ""
-}
-
-// newOptsCall build the rest.Opts structure with *a normalizedID(driveID#fileID, or simply fileID)*
-// using url template https://{Endpoint}/drives/{driveID}/items/{itemID}/{route}
-func (f *Fs) newOptsCall(normalizedID string, method string, route string) (opts rest.Opts) {
-	id, drive, rootURL := f.parseNormalizedID(normalizedID)
+func newOptsCall(normalizedID string, method string, route string) (opts rest.Opts) {
+	id, drive, rootURL := parseNormalizedID(normalizedID)
 
 	if drive != "" {
 		return rest.Opts{
@@ -2012,91 +1985,17 @@ func (f *Fs) newOptsCall(normalizedID string, method string, route string) (opts
 	}
 }
 
-func escapeSingleQuote(str string) string {
-	return strings.ReplaceAll(str, "'", "''")
+// parseNormalizedID parses a normalized ID (may be in the form `driveID#itemID` or just `itemID`)
+// and returns itemID, driveID, rootURL.
+// Such a normalized ID can come from (*Item).GetID()
+func parseNormalizedID(ID string) (string, string, string) {
+	if strings.Index(ID, "#") >= 0 {
+		s := strings.Split(ID, "#")
+		return s[1], s[0], graphURL + "/drives"
+	}
+	return ID, "", ""
 }
 
-// newOptsCallWithIDPath build the rest.Opts structure with *a normalizedID (driveID#fileID, or simply fileID) and leaf*
-// using url template https://{Endpoint}/drives/{driveID}/items/{leaf}:/{route} (for international OneDrive)
-// or https://{Endpoint}/drives/{driveID}/items/children('{leaf}')/{route}
-// and https://{Endpoint}/drives/{driveID}/items/children('@a1')/{route}?@a1=URLEncode("'{leaf}'") (for 21ViaNet)
-// if isPath is false, this function will only work when the leaf is "" or a child name (i.e. it doesn't accept multi-level leaf)
-// if isPath is true, multi-level leaf like a/b/c can be passed
-func (f *Fs) newOptsCallWithIDPath(normalizedID string, leaf string, isPath bool, method string, route string) (opts rest.Opts, ok bool) {
-	encoder := f.opt.Enc.FromStandardName
-	if isPath {
-		encoder = f.opt.Enc.FromStandardPath
-	}
-	trueDirID, drive, rootURL := f.parseNormalizedID(normalizedID)
-	if drive == "" {
-		trueDirID = normalizedID
-	}
-	entity := "/items/" + trueDirID + ":/" + withTrailingColon(rest.URLPathEscape(encoder(leaf))) + route
-	if f.opt.Region == regionCN {
-		if isPath {
-			entity = "/items/" + trueDirID + "/children('@a1')" + route + "?@a1=" + url.QueryEscape("'"+encoder(escapeSingleQuote(leaf))+"'")
-		} else {
-			entity = "/items/" + trueDirID + "/children('" + rest.URLPathEscape(encoder(escapeSingleQuote(leaf))) + "')" + route
-		}
-	}
-	if drive == "" {
-		ok = false
-		opts = rest.Opts{
-			Method: method,
-			Path:   entity,
-		}
-		return
-	}
-	ok = true
-	opts = rest.Opts{
-		Method:  method,
-		RootURL: rootURL,
-		Path:    "/" + drive + entity,
-	}
-	return
-}
-
-// newOptsCallWithIDPath build the rest.Opts structure with an *absolute path start from root*
-// using url template https://{Endpoint}/drives/{driveID}/root:/{path}:/{route}
-// or https://{Endpoint}/drives/{driveID}/root/children('@a1')/{route}?@a1=URLEncode({path})
-func (f *Fs) newOptsCallWithRootPath(path string, method string, route string) (opts rest.Opts) {
-	path = strings.TrimSuffix(path, "/")
-	newURL := "/root:/" + withTrailingColon(rest.URLPathEscape(f.opt.Enc.FromStandardPath(path))) + route
-	if f.opt.Region == regionCN {
-		newURL = "/root/children('@a1')" + route + "?@a1=" + url.QueryEscape("'"+escapeSingleQuote(f.opt.Enc.FromStandardPath(path))+"'")
-	}
-	return rest.Opts{
-		Method: method,
-		Path:   newURL,
-	}
-}
-
-// newOptsCallWithPath build the rest.Opt intelligently.
-// It will first try to resolve the path using dircache, which enables support for "Share with me" files.
-// If present in cache, then use ID + Path variant, else fallback into RootPath variant
-func (f *Fs) newOptsCallWithPath(ctx context.Context, path string, method string, route string) (opts rest.Opts) {
-	if path == "" {
-		url := "/root" + route
-		return rest.Opts{
-			Method: method,
-			Path:   url,
-		}
-	}
-
-	// find dircache
-	leaf, directoryID, _ := f.dirCache.FindPath(ctx, path, false)
-	// try to use IDPath variant first
-	if opts, ok := f.newOptsCallWithIDPath(directoryID, leaf, false, method, route); ok {
-		return opts
-	}
-	// fallback to use RootPath variant first
-	return f.newOptsCallWithRootPath(path, method, route)
-}
-
-/*
- *       URL Build routine area end
- */
-
 // Returns the canonical form of the driveID
 func (f *Fs) canonicalDriveID(driveID string) (canonicalDriveID string) {
 	if driveID == "" {

backend/onedrive/onedrive_test.go

@@ -5,7 +5,6 @@ import (
 	"testing"
 
 	"github.com/rclone/rclone/fs"
-	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/fstest/fstests"
 )
 
@@ -20,20 +19,6 @@ func TestIntegration(t *testing.T) {
 	})
 }
 
-// TestIntegrationCn runs integration tests against the remote
-func TestIntegrationCn(t *testing.T) {
-	if *fstest.RemoteName != "" {
-		t.Skip("skipping as -remote is set")
-	}
-	fstests.Run(t, &fstests.Opt{
-		RemoteName: "TestOneDriveCn:",
-		NilObject:  (*Object)(nil),
-		ChunkedUpload: fstests.ChunkedUploadConfig{
-			CeilChunkSize: fstests.NextMultipleOf(chunkSizeMultiple),
-		},
-	})
-}
-
 func (f *Fs) SetUploadChunkSize(cs fs.SizeSuffix) (fs.SizeSuffix, error) {
 	return f.setUploadChunkSize(cs)
 }

backend/opendrive/opendrive.go

@@ -721,7 +721,7 @@ func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut strin
 	for _, folder := range folderList.Folders {
 		// fs.Debugf(nil, "Folder: %s (%s)", folder.Name, folder.FolderID)
 
-		if strings.EqualFold(leaf, folder.Name) {
+		if leaf == folder.Name {
 			// found
 			return folder.FolderID, true, nil
 		}

backend/premiumizeme/premiumizeme.go

@@ -345,7 +345,7 @@ func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
 func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut string, found bool, err error) {
 	// Find the leaf in pathID
 	found, err = f.listAll(ctx, pathID, true, false, func(item *api.Item) bool {
-		if strings.EqualFold(item.Name, leaf) {
+		if item.Name == leaf {
 			pathIDOut = item.ID
 			return true
 		}

backend/putio/fs.go

@@ -282,10 +282,11 @@ func (f *Fs) PutUnchecked(ctx context.Context, in io.Reader, src fs.ObjectInfo,
 func (f *Fs) createUpload(ctx context.Context, name string, size int64, parentID string, modTime time.Time, options []fs.OpenOption) (location string, err error) {
 	// defer log.Trace(f, "name=%v, size=%v, parentID=%v, modTime=%v", name, size, parentID, modTime.String())("location=%v, err=%v", location, &err)
 	err = f.pacer.Call(func() (bool, error) {
-		req, err := http.NewRequestWithContext(ctx, "POST", "https://upload.put.io/files/", nil)
+		req, err := http.NewRequest("POST", "https://upload.put.io/files/", nil)
 		if err != nil {
 			return false, err
 		}
+		req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 		req.Header.Set("tus-resumable", "1.0.0")
 		req.Header.Set("upload-length", strconv.FormatInt(size, 10))
 		b64name := base64.StdEncoding.EncodeToString([]byte(f.opt.Enc.FromStandardName(name)))
@@ -427,19 +428,21 @@ func (f *Fs) transferChunk(ctx context.Context, location string, start int64, ch
 }
 
 func (f *Fs) makeUploadHeadRequest(ctx context.Context, location string) (*http.Request, error) {
-	req, err := http.NewRequestWithContext(ctx, "HEAD", location, nil)
+	req, err := http.NewRequest("HEAD", location, nil)
 	if err != nil {
 		return nil, err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	req.Header.Set("tus-resumable", "1.0.0")
 	return req, nil
 }
 
 func (f *Fs) makeUploadPatchRequest(ctx context.Context, location string, in io.Reader, offset, length int64) (*http.Request, error) {
-	req, err := http.NewRequestWithContext(ctx, "PATCH", location, in)
+	req, err := http.NewRequest("PATCH", location, in)
 	if err != nil {
 		return nil, err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 	req.Header.Set("tus-resumable", "1.0.0")
 	req.Header.Set("upload-offset", strconv.FormatInt(offset, 10))
 	req.Header.Set("content-length", strconv.FormatInt(length, 10))

backend/putio/object.go

@@ -229,10 +229,11 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 	var resp *http.Response
 	headers := fs.OpenOptionHeaders(options)
 	err = o.fs.pacer.Call(func() (bool, error) {
-		req, err := http.NewRequestWithContext(ctx, http.MethodGet, storageURL, nil)
+		req, err := http.NewRequest(http.MethodGet, storageURL, nil)
 		if err != nil {
 			return shouldRetry(err)
 		}
+		req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 		req.Header.Set("User-Agent", o.fs.client.UserAgent)
 
 		// merge headers with extra headers

backend/qingstor/qingstor.go

@@ -872,12 +872,11 @@ func (f *Fs) cleanUpBucket(ctx context.Context, bucket string) (err error) {
 	if err != nil {
 		return err
 	}
-	// maxLimit := int(listLimitSize)
+	maxLimit := int(listLimitSize)
 	var marker *string
 	for {
 		req := qs.ListMultipartUploadsInput{
-			// The default is 200 but this errors if more than 200 is put in so leave at the default
-			// Limit:     &maxLimit,
+			Limit:     &maxLimit,
 			KeyMarker: marker,
 		}
 		var resp *qs.ListMultipartUploadsOutput
@@ -928,7 +927,7 @@ func (f *Fs) CleanUp(ctx context.Context) (err error) {
 	}
 	for _, entry := range entries {
 		cleanErr := f.cleanUpBucket(ctx, f.opt.Enc.FromStandardName(entry.Remote()))
-		if cleanErr != nil {
+		if err != nil {
 			fs.Errorf(f, "Failed to cleanup bucket: %q", cleanErr)
 			err = cleanErr
 		}

backend/s3/s3.go

@@ -33,7 +33,7 @@ import (
 	"github.com/aws/aws-sdk-go/aws/request"
 	"github.com/aws/aws-sdk-go/aws/session"
 	"github.com/aws/aws-sdk-go/service/s3"
-	"github.com/ncw/swift/v2"
+	"github.com/ncw/swift"
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config"
@@ -1185,39 +1185,6 @@ rclone does if you know the bucket exists already.
 It can also be needed if the user you are using does not have bucket
 creation permissions. Before v1.52.0 this would have passed silently
 due to a bug.
-`,
-			Default:  false,
-			Advanced: true,
-		}, {
-			Name: "no_head",
-			Help: `If set, don't HEAD uploaded objects to check integrity
-
-This can be useful when trying to minimise the number of transactions
-rclone does.
-
-Setting it means that if rclone receives a 200 OK message after
-uploading an object with PUT then it will assume that it got uploaded
-properly.
-
-In particular it will assume:
-
-- the metadata, including modtime, storage class and content type was as uploaded
-- the size was as uploaded
-
-It reads the following items from the response for a single part PUT:
-
-- the MD5SUM
-- The uploaded date
-
-For multipart uploads these items aren't read.
-
-If an source object of unknown length is uploaded then rclone **will** do a
-HEAD request.
-
-Setting this flag increases the chance for undetected upload failures,
-in particular an incorrect size, so it isn't recommended for normal
-operation. In practice the chance of an undetected upload failure is
-very small even with this flag.
 `,
 			Default:  false,
 			Advanced: true,
@@ -1318,7 +1285,6 @@ type Options struct {
 	LeavePartsOnError     bool                 `config:"leave_parts_on_error"`
 	ListChunk             int64                `config:"list_chunk"`
 	NoCheckBucket         bool                 `config:"no_check_bucket"`
-	NoHead                bool                 `config:"no_head"`
 	Enc                   encoder.MultiEncoder `config:"encoding"`
 	MemoryPoolFlushTime   fs.Duration          `config:"memory_pool_flush_time"`
 	MemoryPoolUseMmap     bool                 `config:"memory_pool_use_mmap"`
@@ -1462,7 +1428,7 @@ func getClient(ctx context.Context, opt *Options) *http.Client {
 }
 
 // s3Connection makes a connection to s3
-func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S3, *session.Session, error) {
+func s3Connection(ctx context.Context, opt *Options) (*s3.S3, *session.Session, error) {
 	// Make the auth
 	v := credentials.Value{
 		AccessKeyID:     opt.AccessKeyID,
@@ -1540,7 +1506,7 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S
 	awsConfig := aws.NewConfig().
 		WithMaxRetries(0). // Rely on rclone's retry logic
 		WithCredentials(cred).
-		WithHTTPClient(client).
+		WithHTTPClient(getClient(ctx, opt)).
 		WithS3ForcePathStyle(opt.ForcePathStyle).
 		WithS3UseAccelerate(opt.UseAccelerateEndpoint).
 		WithS3UsEast1RegionalEndpoint(endpoints.RegionalS3UsEast1Endpoint)
@@ -1559,6 +1525,9 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (*s3.S
 	if opt.EnvAuth && opt.AccessKeyID == "" && opt.SecretAccessKey == "" {
 		// Enable loading config options from ~/.aws/config (selected by AWS_PROFILE env)
 		awsSessionOpts.SharedConfigState = session.SharedConfigEnable
+		// The session constructor (aws/session/mergeConfigSrcs) will only use the user's preferred credential source
+		// (from the shared config file) if the passed-in Options.Config.Credentials is nil.
+		awsSessionOpts.Config.Credentials = nil
 	}
 	ses, err := session.NewSessionWithOptions(awsSessionOpts)
 	if err != nil {
@@ -1644,8 +1613,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		md5sumBinary := md5.Sum([]byte(opt.SSECustomerKey))
 		opt.SSECustomerKeyMD5 = base64.StdEncoding.EncodeToString(md5sumBinary[:])
 	}
-	srv := getClient(ctx, opt)
-	c, ses, err := s3Connection(ctx, opt, srv)
+	c, ses, err := s3Connection(ctx, opt)
 	if err != nil {
 		return nil, err
 	}
@@ -1660,7 +1628,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		ses:   ses,
 		pacer: fs.NewPacer(ctx, pacer.NewS3(pacer.MinSleep(minSleep))),
 		cache: bucket.NewCache(),
-		srv:   srv,
+		srv:   getClient(ctx, opt),
 		pool: pool.New(
 			time.Duration(opt.MemoryPoolFlushTime),
 			int(opt.ChunkSize),
@@ -1695,9 +1663,12 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		f.setRoot(newRoot)
 		_, err := f.NewObject(ctx, leaf)
 		if err != nil {
-			// File doesn't exist or is a directory so return old f
-			f.setRoot(oldRoot)
-			return f, nil
+			if err == fs.ErrorObjectNotFound || err == fs.ErrorNotAFile {
+				// File doesn't exist or is a directory so return old f
+				f.setRoot(oldRoot)
+				return f, nil
+			}
+			return nil, err
 		}
 		// return an error with an fs which points to the parent
 		return f, fs.ErrorIsFile
@@ -1774,7 +1745,7 @@ func (f *Fs) updateRegionForBucket(bucket string) error {
 	// Make a new session with the new region
 	oldRegion := f.opt.Region
 	f.opt.Region = region
-	c, ses, err := s3Connection(f.ctx, &f.opt, f.srv)
+	c, ses, err := s3Connection(f.ctx, &f.opt)
 	if err != nil {
 		return errors.Wrap(err, "creating new session failed")
 	}
@@ -3171,25 +3142,10 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 
 	multipart := size < 0 || size >= int64(o.fs.opt.UploadCutoff)
 
-	// Metadata for upload
-	metadata := map[string]*string{}
-	fs.Debugf(o, "src = %#v", src)
-
-	// Read metadata from source s3 object if available
-	srcObj, ok := fs.UnWrapObjectInfo(src).(*Object)
-	if ok {
-		fs.Debugf(o, "Reading metadata from %v", srcObj)
-		err := srcObj.readMetaData(ctx) // reads info and meta, returning an error
-		if err != nil {
-			return err
-		}
-		for k, v := range srcObj.meta {
-			metadata[k] = v
-		}
-	}
-
 	// Set the mtime in the meta data
-	metadata[metaMtime] = aws.String(swift.TimeToFloatString(modTime))
+	metadata := map[string]*string{
+		metaMtime: aws.String(swift.TimeToFloatString(modTime)),
+	}
 
 	// read the md5sum if available
 	// - for non multipart
@@ -3278,7 +3234,6 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 		}
 	}
 
-	var resp *http.Response // response from PUT
 	if multipart {
 		err = o.uploadMultipart(ctx, &req, size, in)
 		if err != nil {
@@ -3308,18 +3263,18 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 		}
 
 		// create the vanilla http request
-		httpReq, err := http.NewRequestWithContext(ctx, "PUT", url, in)
+		httpReq, err := http.NewRequest("PUT", url, in)
 		if err != nil {
 			return errors.Wrap(err, "s3 upload: new request")
 		}
+		httpReq = httpReq.WithContext(ctx) // go1.13 can use NewRequestWithContext
 
 		// set the headers we signed and the length
 		httpReq.Header = headers
 		httpReq.ContentLength = size
 
 		err = o.fs.pacer.CallNoRetry(func() (bool, error) {
-			var err error
-			resp, err = o.fs.srv.Do(httpReq)
+			resp, err := o.fs.srv.Do(httpReq)
 			if err != nil {
 				return o.fs.shouldRetry(err)
 			}
@@ -3338,26 +3293,6 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 		}
 	}
 
-	// User requested we don't HEAD the object after uploading it
-	// so make up the object as best we can assuming it got
-	// uploaded properly. If size < 0 then we need to do the HEAD.
-	if o.fs.opt.NoHead && size >= 0 {
-		o.md5 = md5sum
-		o.bytes = size
-		o.lastModified = time.Now()
-		o.meta = req.Metadata
-		o.mimeType = aws.StringValue(req.ContentType)
-		o.storageClass = aws.StringValue(req.StorageClass)
-		// If we have done a single part PUT request then we can read these
-		if resp != nil {
-			if date, err := http.ParseTime(resp.Header.Get("Date")); err == nil {
-				o.lastModified = date
-			}
-			o.setMD5FromEtag(resp.Header.Get("Etag"))
-		}
-		return nil
-	}
-
 	// Read the metadata from the newly created object
 	o.meta = nil // wipe old metadata
 	err = o.readMetaData(ctx)

backend/s3/v2sign.go

@@ -53,7 +53,6 @@ func sign(AccessKey, SecretKey string, req *http.Request) {
 	var md5 string
 	var contentType string
 	var headersToSign []string
-	tmpHeadersToSign := make(map[string][]string)
 	for k, v := range req.Header {
 		k = strings.ToLower(k)
 		switch k {
@@ -63,24 +62,15 @@ func sign(AccessKey, SecretKey string, req *http.Request) {
 			contentType = v[0]
 		default:
 			if strings.HasPrefix(k, "x-amz-") {
-				tmpHeadersToSign[k] = v
+				vall := strings.Join(v, ",")
+				headersToSign = append(headersToSign, k+":"+vall)
 			}
 		}
 	}
-	var keys []string
-	for k := range tmpHeadersToSign {
-		keys = append(keys, k)
-	}
-	// https://docs.aws.amazon.com/AmazonS3/latest/dev/RESTAuthentication.html
-	sort.Strings(keys)
-
-	for _, key := range keys {
-		vall := strings.Join(tmpHeadersToSign[key], ",")
-		headersToSign = append(headersToSign, key+":"+vall)
-	}
 	// Make headers of interest into canonical string
 	var joinedHeadersToSign string
 	if len(headersToSign) > 0 {
+		sort.StringSlice(headersToSign).Sort()
 		joinedHeadersToSign = strings.Join(headersToSign, "\n") + "\n"
 	}
 

backend/sftp/sftp.go

@@ -192,31 +192,6 @@ Home directory can be found in a shared folder called "home"
 
 The subsystem option is ignored when server_command is defined.`,
 			Advanced: true,
-		}, {
-			Name:    "use_fstat",
-			Default: false,
-			Help: `If set use fstat instead of stat
-
-Some servers limit the amount of open files and calling Stat after opening
-the file will throw an error from the server. Setting this flag will call
-Fstat instead of Stat which is called on an already open file handle.
-
-It has been found that this helps with IBM Sterling SFTP servers which have
-"extractability" level set to 1 which means only 1 file can be opened at
-any given time.
-`,
-			Advanced: true,
-		}, {
-			Name:    "idle_timeout",
-			Default: fs.Duration(60 * time.Second),
-			Help: `Max time before closing idle connections
-
-If no connections have been returned to the connection pool in the time
-given, rclone will empty the connection pool.
-
-Set to 0 to keep connections indefinitely.
-`,
-			Advanced: true,
 		}},
 	}
 	fs.Register(fsi)
@@ -224,28 +199,26 @@ Set to 0 to keep connections indefinitely.
 
 // Options defines the configuration for this backend
 type Options struct {
-	Host              string      `config:"host"`
-	User              string      `config:"user"`
-	Port              string      `config:"port"`
-	Pass              string      `config:"pass"`
-	KeyPem            string      `config:"key_pem"`
-	KeyFile           string      `config:"key_file"`
-	KeyFilePass       string      `config:"key_file_pass"`
-	PubKeyFile        string      `config:"pubkey_file"`
-	KnownHostsFile    string      `config:"known_hosts_file"`
-	KeyUseAgent       bool        `config:"key_use_agent"`
-	UseInsecureCipher bool        `config:"use_insecure_cipher"`
-	DisableHashCheck  bool        `config:"disable_hashcheck"`
-	AskPassword       bool        `config:"ask_password"`
-	PathOverride      string      `config:"path_override"`
-	SetModTime        bool        `config:"set_modtime"`
-	Md5sumCommand     string      `config:"md5sum_command"`
-	Sha1sumCommand    string      `config:"sha1sum_command"`
-	SkipLinks         bool        `config:"skip_links"`
-	Subsystem         string      `config:"subsystem"`
-	ServerCommand     string      `config:"server_command"`
-	UseFstat          bool        `config:"use_fstat"`
-	IdleTimeout       fs.Duration `config:"idle_timeout"`
+	Host              string `config:"host"`
+	User              string `config:"user"`
+	Port              string `config:"port"`
+	Pass              string `config:"pass"`
+	KeyPem            string `config:"key_pem"`
+	KeyFile           string `config:"key_file"`
+	KeyFilePass       string `config:"key_file_pass"`
+	PubKeyFile        string `config:"pubkey_file"`
+	KnownHostsFile    string `config:"known_hosts_file"`
+	KeyUseAgent       bool   `config:"key_use_agent"`
+	UseInsecureCipher bool   `config:"use_insecure_cipher"`
+	DisableHashCheck  bool   `config:"disable_hashcheck"`
+	AskPassword       bool   `config:"ask_password"`
+	PathOverride      string `config:"path_override"`
+	SetModTime        bool   `config:"set_modtime"`
+	Md5sumCommand     string `config:"md5sum_command"`
+	Sha1sumCommand    string `config:"sha1sum_command"`
+	SkipLinks         bool   `config:"skip_links"`
+	Subsystem         string `config:"subsystem"`
+	ServerCommand     string `config:"server_command"`
 }
 
 // Fs stores the interface to the remote SFTP files
@@ -263,8 +236,7 @@ type Fs struct {
 	cachedHashes *hash.Set
 	poolMu       sync.Mutex
 	pool         []*conn
-	drain        *time.Timer // used to drain the pool when we stop using the connections
-	pacer        *fs.Pacer   // pacer for operations
+	pacer        *fs.Pacer // pacer for operations
 	savedpswd    string
 }
 
@@ -372,8 +344,6 @@ func (f *Fs) newSftpClient(conn *ssh.Client, opts ...sftp.ClientOption) (*sftp.C
 			return nil, err
 		}
 	}
-	opts = opts[:len(opts):len(opts)] // make sure we don't overwrite the callers opts
-	opts = append(opts, sftp.UseFstat(f.opt.UseFstat))
 
 	return sftp.NewClientPipe(pr, pw, opts...)
 }
@@ -441,9 +411,6 @@ func (f *Fs) putSftpConnection(pc **conn, err error) {
 	}
 	f.poolMu.Lock()
 	f.pool = append(f.pool, c)
-	if f.opt.IdleTimeout > 0 {
-		f.drain.Reset(time.Duration(f.opt.IdleTimeout)) // nudge on the pool emptying timer
-	}
 	f.poolMu.Unlock()
 }
 
@@ -451,12 +418,6 @@ func (f *Fs) putSftpConnection(pc **conn, err error) {
 func (f *Fs) drainPool(ctx context.Context) (err error) {
 	f.poolMu.Lock()
 	defer f.poolMu.Unlock()
-	if f.opt.IdleTimeout > 0 {
-		f.drain.Stop()
-	}
-	if len(f.pool) != 0 {
-		fs.Debugf(f, "closing %d unused connections", len(f.pool))
-	}
 	for i, c := range f.pool {
 		if cErr := c.closed(); cErr == nil {
 			cErr = c.close()
@@ -689,10 +650,6 @@ func NewFsWithConnection(ctx context.Context, f *Fs, name string, root string, m
 	f.mkdirLock = newStringLock()
 	f.pacer = fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant)))
 	f.savedpswd = ""
-	// set the pool drainer timer going
-	if f.opt.IdleTimeout > 0 {
-		f.drain = time.AfterFunc(time.Duration(opt.IdleTimeout), func() { _ = f.drainPool(ctx) })
-	}
 
 	f.features = (&fs.Features{
 		CanHaveEmptyDirectories: true,

backend/sugarsync/sugarsync.go

@@ -531,7 +531,7 @@ func (f *Fs) FindLeaf(ctx context.Context, pathID, leaf string) (pathIDOut strin
 	//fs.Debugf(f, "FindLeaf(%q, %q)", pathID, leaf)
 	// Find the leaf in pathID
 	found, err = f.listAll(ctx, pathID, nil, func(item *api.Collection) bool {
-		if strings.EqualFold(item.Name, leaf) {
+		if item.Name == leaf {
 			pathIDOut = item.Ref
 			return true
 		}

backend/swift/auth.go

@@ -1,11 +1,10 @@
 package swift
 
 import (
-	"context"
 	"net/http"
 	"time"
 
-	"github.com/ncw/swift/v2"
+	"github.com/ncw/swift"
 )
 
 // auth is an authenticator for swift.  It overrides the StorageUrl
@@ -29,19 +28,19 @@ func newAuth(parentAuth swift.Authenticator, storageURL string, authToken string
 }
 
 // Request creates an http.Request for the auth - return nil if not needed
-func (a *auth) Request(ctx context.Context, c *swift.Connection) (*http.Request, error) {
+func (a *auth) Request(c *swift.Connection) (*http.Request, error) {
 	if a.parentAuth == nil {
 		return nil, nil
 	}
-	return a.parentAuth.Request(ctx, c)
+	return a.parentAuth.Request(c)
 }
 
 // Response parses the http.Response
-func (a *auth) Response(ctx context.Context, resp *http.Response) error {
+func (a *auth) Response(resp *http.Response) error {
 	if a.parentAuth == nil {
 		return nil
 	}
-	return a.parentAuth.Response(ctx, resp)
+	return a.parentAuth.Response(resp)
 }
 
 // The public storage URL - set Internal to true to read

backend/swift/swift.go

@@ -13,7 +13,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/ncw/swift/v2"
+	"github.com/ncw/swift"
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config"
@@ -24,7 +24,6 @@ import (
 	"github.com/rclone/rclone/fs/hash"
 	"github.com/rclone/rclone/fs/operations"
 	"github.com/rclone/rclone/fs/walk"
-	"github.com/rclone/rclone/lib/atexit"
 	"github.com/rclone/rclone/lib/bucket"
 	"github.com/rclone/rclone/lib/encoder"
 	"github.com/rclone/rclone/lib/pacer"
@@ -168,11 +167,6 @@ func init() {
 				Help:  "Admin",
 				Value: "admin",
 			}},
-		}, {
-			Name:     "leave_parts_on_error",
-			Help:     `If true avoid calling abort upload on a failure. It should be set to true for resuming uploads across different sessions.`,
-			Default:  false,
-			Advanced: true,
 		}, {
 			Name: "storage_policy",
 			Help: `The storage policy to use when creating a new container
@@ -214,7 +208,6 @@ type Options struct {
 	ApplicationCredentialID     string               `config:"application_credential_id"`
 	ApplicationCredentialName   string               `config:"application_credential_name"`
 	ApplicationCredentialSecret string               `config:"application_credential_secret"`
-	LeavePartsOnError           bool                 `config:"leave_parts_on_error"`
 	StoragePolicy               string               `config:"storage_policy"`
 	EndpointType                string               `config:"endpoint_type"`
 	ChunkSize                   fs.SizeSuffix        `config:"chunk_size"`
@@ -391,7 +384,7 @@ func swiftConnection(ctx context.Context, opt *Options, name string) (*swift.Con
 		if c.AuthUrl == "" {
 			return nil, errors.New("auth not found")
 		}
-		err := c.Authenticate(ctx) // fills in c.StorageUrl and c.AuthToken
+		err := c.Authenticate() // fills in c.StorageUrl and c.AuthToken
 		if err != nil {
 			return nil, err
 		}
@@ -467,7 +460,7 @@ func NewFsWithConnection(ctx context.Context, opt *Options, name, root string, c
 		encodedDirectory := f.opt.Enc.FromStandardPath(f.rootDirectory)
 		err = f.pacer.Call(func() (bool, error) {
 			var rxHeaders swift.Headers
-			info, rxHeaders, err = f.c.Object(ctx, f.rootContainer, encodedDirectory)
+			info, rxHeaders, err = f.c.Object(f.rootContainer, encodedDirectory)
 			return shouldRetryHeaders(rxHeaders, err)
 		})
 		if err == nil && info.ContentType != directoryMarkerContentType {
@@ -506,7 +499,7 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 // Return an Object from a path
 //
 // If it can't be found it returns the error fs.ErrorObjectNotFound.
-func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *swift.Object) (fs.Object, error) {
+func (f *Fs) newObjectWithInfo(remote string, info *swift.Object) (fs.Object, error) {
 	o := &Object{
 		fs:     f,
 		remote: remote,
@@ -516,7 +509,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *swift.O
 	// making sure we read the full metadata for all 0 byte files.
 	// We don't read the metadata for directory marker objects.
 	if info != nil && info.Bytes == 0 && info.ContentType != "application/directory" {
-		err := o.readMetaData(ctx) // reads info and headers, returning an error
+		err := o.readMetaData() // reads info and headers, returning an error
 		if err == fs.ErrorObjectNotFound {
 			// We have a dangling large object here so just return the original metadata
 			fs.Errorf(o, "dangling large object with no contents")
@@ -533,7 +526,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *swift.O
 			return nil, err
 		}
 	} else {
-		err := o.readMetaData(ctx) // reads info and headers, returning an error
+		err := o.readMetaData() // reads info and headers, returning an error
 		if err != nil {
 			return nil, err
 		}
@@ -544,7 +537,7 @@ func (f *Fs) newObjectWithInfo(ctx context.Context, remote string, info *swift.O
 // NewObject finds the Object at remote.  If it can't be found it
 // returns the error fs.ErrorObjectNotFound.
 func (f *Fs) NewObject(ctx context.Context, remote string) (fs.Object, error) {
-	return f.newObjectWithInfo(ctx, remote, nil)
+	return f.newObjectWithInfo(remote, nil)
 }
 
 // listFn is called from list and listContainerRoot to handle an object.
@@ -556,7 +549,7 @@ type listFn func(remote string, object *swift.Object, isDirectory bool) error
 // container to the start.
 //
 // Set recurse to read sub directories
-func (f *Fs) listContainerRoot(ctx context.Context, container, directory, prefix string, addContainer bool, recurse bool, includeDirMarkers bool, fn listFn) error {
+func (f *Fs) listContainerRoot(container, directory, prefix string, addContainer bool, recurse bool, includeDirMarkers bool, fn listFn) error {
 	if prefix != "" && !strings.HasSuffix(prefix, "/") {
 		prefix += "/"
 	}
@@ -571,11 +564,11 @@ func (f *Fs) listContainerRoot(ctx context.Context, container, directory, prefix
 	if !recurse {
 		opts.Delimiter = '/'
 	}
-	return f.c.ObjectsWalk(ctx, container, &opts, func(ctx context.Context, opts *swift.ObjectsOpts) (interface{}, error) {
+	return f.c.ObjectsWalk(container, &opts, func(opts *swift.ObjectsOpts) (interface{}, error) {
 		var objects []swift.Object
 		var err error
 		err = f.pacer.Call(func() (bool, error) {
-			objects, err = f.c.Objects(ctx, container, opts)
+			objects, err = f.c.Objects(container, opts)
 			return shouldRetry(err)
 		})
 		if err == nil {
@@ -613,8 +606,8 @@ func (f *Fs) listContainerRoot(ctx context.Context, container, directory, prefix
 type addEntryFn func(fs.DirEntry) error
 
 // list the objects into the function supplied
-func (f *Fs) list(ctx context.Context, container, directory, prefix string, addContainer bool, recurse bool, includeDirMarkers bool, fn addEntryFn) error {
-	err := f.listContainerRoot(ctx, container, directory, prefix, addContainer, recurse, includeDirMarkers, func(remote string, object *swift.Object, isDirectory bool) (err error) {
+func (f *Fs) list(container, directory, prefix string, addContainer bool, recurse bool, includeDirMarkers bool, fn addEntryFn) error {
+	err := f.listContainerRoot(container, directory, prefix, addContainer, recurse, includeDirMarkers, func(remote string, object *swift.Object, isDirectory bool) (err error) {
 		if isDirectory {
 			remote = strings.TrimRight(remote, "/")
 			d := fs.NewDir(remote, time.Time{}).SetSize(object.Bytes)
@@ -622,7 +615,7 @@ func (f *Fs) list(ctx context.Context, container, directory, prefix string, addC
 		} else {
 			// newObjectWithInfo does a full metadata read on 0 size objects which might be dynamic large objects
 			var o fs.Object
-			o, err = f.newObjectWithInfo(ctx, remote, object)
+			o, err = f.newObjectWithInfo(remote, object)
 			if err != nil {
 				return err
 			}
@@ -639,12 +632,12 @@ func (f *Fs) list(ctx context.Context, container, directory, prefix string, addC
 }
 
 // listDir lists a single directory
-func (f *Fs) listDir(ctx context.Context, container, directory, prefix string, addContainer bool) (entries fs.DirEntries, err error) {
+func (f *Fs) listDir(container, directory, prefix string, addContainer bool) (entries fs.DirEntries, err error) {
 	if container == "" {
 		return nil, fs.ErrorListBucketRequired
 	}
 	// List the objects
-	err = f.list(ctx, container, directory, prefix, addContainer, false, false, func(entry fs.DirEntry) error {
+	err = f.list(container, directory, prefix, addContainer, false, false, func(entry fs.DirEntry) error {
 		entries = append(entries, entry)
 		return nil
 	})
@@ -660,7 +653,7 @@ func (f *Fs) listDir(ctx context.Context, container, directory, prefix string, a
 func (f *Fs) listContainers(ctx context.Context) (entries fs.DirEntries, err error) {
 	var containers []swift.Container
 	err = f.pacer.Call(func() (bool, error) {
-		containers, err = f.c.ContainersAll(ctx, nil)
+		containers, err = f.c.ContainersAll(nil)
 		return shouldRetry(err)
 	})
 	if err != nil {
@@ -691,7 +684,7 @@ func (f *Fs) List(ctx context.Context, dir string) (entries fs.DirEntries, err e
 		}
 		return f.listContainers(ctx)
 	}
-	return f.listDir(ctx, container, directory, f.rootDirectory, f.rootContainer == "")
+	return f.listDir(container, directory, f.rootDirectory, f.rootContainer == "")
 }
 
 // ListR lists the objects and directories of the Fs starting
@@ -714,7 +707,7 @@ func (f *Fs) ListR(ctx context.Context, dir string, callback fs.ListRCallback) (
 	container, directory := f.split(dir)
 	list := walk.NewListRHelper(callback)
 	listR := func(container, directory, prefix string, addContainer bool) error {
-		return f.list(ctx, container, directory, prefix, addContainer, true, false, func(entry fs.DirEntry) error {
+		return f.list(container, directory, prefix, addContainer, true, false, func(entry fs.DirEntry) error {
 			return list.Add(entry)
 		})
 	}
@@ -752,7 +745,7 @@ func (f *Fs) About(ctx context.Context) (*fs.Usage, error) {
 	var containers []swift.Container
 	var err error
 	err = f.pacer.Call(func() (bool, error) {
-		containers, err = f.c.ContainersAll(ctx, nil)
+		containers, err = f.c.ContainersAll(nil)
 		return shouldRetry(err)
 	})
 	if err != nil {
@@ -804,7 +797,7 @@ func (f *Fs) makeContainer(ctx context.Context, container string) error {
 		if !f.noCheckContainer {
 			err = f.pacer.Call(func() (bool, error) {
 				var rxHeaders swift.Headers
-				_, rxHeaders, err = f.c.Container(ctx, container)
+				_, rxHeaders, err = f.c.Container(container)
 				return shouldRetryHeaders(rxHeaders, err)
 			})
 		}
@@ -814,7 +807,7 @@ func (f *Fs) makeContainer(ctx context.Context, container string) error {
 				headers["X-Storage-Policy"] = f.opt.StoragePolicy
 			}
 			err = f.pacer.Call(func() (bool, error) {
-				err = f.c.ContainerCreate(ctx, container, headers)
+				err = f.c.ContainerCreate(container, headers)
 				return shouldRetry(err)
 			})
 			if err == nil {
@@ -835,7 +828,7 @@ func (f *Fs) Rmdir(ctx context.Context, dir string) error {
 	}
 	err := f.cache.Remove(container, func() error {
 		err := f.pacer.Call(func() (bool, error) {
-			err := f.c.ContainerDelete(ctx, container)
+			err := f.c.ContainerDelete(container)
 			return shouldRetry(err)
 		})
 		if err == nil {
@@ -865,7 +858,7 @@ func (f *Fs) Purge(ctx context.Context, dir string) error {
 	go func() {
 		delErr <- operations.DeleteFiles(ctx, toBeDeleted)
 	}()
-	err := f.list(ctx, container, directory, f.rootDirectory, false, true, true, func(entry fs.DirEntry) error {
+	err := f.list(container, directory, f.rootDirectory, false, true, true, func(entry fs.DirEntry) error {
 		if o, ok := entry.(*Object); ok {
 			toBeDeleted <- o
 		}
@@ -905,7 +898,7 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
 	srcContainer, srcPath := srcObj.split()
 	err = f.pacer.Call(func() (bool, error) {
 		var rxHeaders swift.Headers
-		rxHeaders, err = f.c.ObjectCopy(ctx, srcContainer, srcPath, dstContainer, dstPath, nil)
+		rxHeaders, err = f.c.ObjectCopy(srcContainer, srcPath, dstContainer, dstPath, nil)
 		return shouldRetryHeaders(rxHeaders, err)
 	})
 	if err != nil {
@@ -944,11 +937,11 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
 	if t != hash.MD5 {
 		return "", hash.ErrUnsupported
 	}
-	isDynamicLargeObject, err := o.isDynamicLargeObject(ctx)
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
 	if err != nil {
 		return "", err
 	}
-	isStaticLargeObject, err := o.isStaticLargeObject(ctx)
+	isStaticLargeObject, err := o.isStaticLargeObject()
 	if err != nil {
 		return "", err
 	}
@@ -961,8 +954,8 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
 
 // hasHeader checks for the header passed in returning false if the
 // object isn't found.
-func (o *Object) hasHeader(ctx context.Context, header string) (bool, error) {
-	err := o.readMetaData(ctx)
+func (o *Object) hasHeader(header string) (bool, error) {
+	err := o.readMetaData()
 	if err != nil {
 		if err == fs.ErrorObjectNotFound {
 			return false, nil
@@ -974,29 +967,29 @@ func (o *Object) hasHeader(ctx context.Context, header string) (bool, error) {
 }
 
 // isDynamicLargeObject checks for X-Object-Manifest header
-func (o *Object) isDynamicLargeObject(ctx context.Context) (bool, error) {
-	return o.hasHeader(ctx, "X-Object-Manifest")
+func (o *Object) isDynamicLargeObject() (bool, error) {
+	return o.hasHeader("X-Object-Manifest")
 }
 
 // isStaticLargeObjectFile checks for the X-Static-Large-Object header
-func (o *Object) isStaticLargeObject(ctx context.Context) (bool, error) {
-	return o.hasHeader(ctx, "X-Static-Large-Object")
+func (o *Object) isStaticLargeObject() (bool, error) {
+	return o.hasHeader("X-Static-Large-Object")
 }
 
-func (o *Object) isLargeObject(ctx context.Context) (result bool, err error) {
-	result, err = o.hasHeader(ctx, "X-Static-Large-Object")
+func (o *Object) isLargeObject() (result bool, err error) {
+	result, err = o.hasHeader("X-Static-Large-Object")
 	if result {
 		return
 	}
-	result, err = o.hasHeader(ctx, "X-Object-Manifest")
+	result, err = o.hasHeader("X-Object-Manifest")
 	if result {
 		return
 	}
 	return false, nil
 }
 
-func (o *Object) isInContainerVersioning(ctx context.Context, container string) (bool, error) {
-	_, headers, err := o.fs.c.Container(ctx, container)
+func (o *Object) isInContainerVersioning(container string) (bool, error) {
+	_, headers, err := o.fs.c.Container(container)
 	if err != nil {
 		return false, err
 	}
@@ -1032,7 +1025,7 @@ func (o *Object) decodeMetaData(info *swift.Object) (err error) {
 // it also sets the info
 //
 // it returns fs.ErrorObjectNotFound if the object isn't found
-func (o *Object) readMetaData(ctx context.Context) (err error) {
+func (o *Object) readMetaData() (err error) {
 	if o.headers != nil {
 		return nil
 	}
@@ -1040,7 +1033,7 @@ func (o *Object) readMetaData(ctx context.Context) (err error) {
 	var h swift.Headers
 	container, containerPath := o.split()
 	err = o.fs.pacer.Call(func() (bool, error) {
-		info, h, err = o.fs.c.Object(ctx, container, containerPath)
+		info, h, err = o.fs.c.Object(container, containerPath)
 		return shouldRetryHeaders(h, err)
 	})
 	if err != nil {
@@ -1066,7 +1059,7 @@ func (o *Object) ModTime(ctx context.Context) time.Time {
 	if o.fs.ci.UseServerModTime {
 		return o.lastModified
 	}
-	err := o.readMetaData(ctx)
+	err := o.readMetaData()
 	if err != nil {
 		fs.Debugf(o, "Failed to read metadata: %s", err)
 		return o.lastModified
@@ -1081,7 +1074,7 @@ func (o *Object) ModTime(ctx context.Context) time.Time {
 
 // SetModTime sets the modification time of the local fs object
 func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
-	err := o.readMetaData(ctx)
+	err := o.readMetaData()
 	if err != nil {
 		return err
 	}
@@ -1099,7 +1092,7 @@ func (o *Object) SetModTime(ctx context.Context, modTime time.Time) error {
 	}
 	container, containerPath := o.split()
 	return o.fs.pacer.Call(func() (bool, error) {
-		err = o.fs.c.ObjectUpdate(ctx, container, containerPath, newHeaders)
+		err = o.fs.c.ObjectUpdate(container, containerPath, newHeaders)
 		return shouldRetry(err)
 	})
 }
@@ -1120,7 +1113,7 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
 	container, containerPath := o.split()
 	err = o.fs.pacer.Call(func() (bool, error) {
 		var rxHeaders swift.Headers
-		in, rxHeaders, err = o.fs.c.ObjectOpen(ctx, container, containerPath, !isRanging, headers)
+		in, rxHeaders, err = o.fs.c.ObjectOpen(container, containerPath, !isRanging, headers)
 		return shouldRetryHeaders(rxHeaders, err)
 	})
 	return
@@ -1134,41 +1127,50 @@ func min(x, y int64) int64 {
 	return y
 }
 
-func (o *Object) getSegmentsLargeObject(ctx context.Context) (map[string][]string, error) {
-	container, objectName := o.split()
-	segmentContainer, segmentObjects, err := o.fs.c.LargeObjectGetSegments(ctx, container, objectName)
+// removeSegments removes any old segments from o
+//
+// if except is passed in then segments with that prefix won't be deleted
+func (o *Object) removeSegments(except string) error {
+	segmentsContainer, _, err := o.getSegmentsDlo()
 	if err != nil {
-		fs.Debugf(o, "Failed to get list segments of object: %v", err)
-		return nil, err
+		return err
 	}
-	var containerSegments = make(map[string][]string)
-	for _, segment := range segmentObjects {
-		if _, ok := containerSegments[segmentContainer]; !ok {
-			containerSegments[segmentContainer] = make([]string, 0, len(segmentObjects))
+	except = path.Join(o.remote, except)
+	// fs.Debugf(o, "segmentsContainer %q prefix %q", segmentsContainer, prefix)
+	err = o.fs.listContainerRoot(segmentsContainer, o.remote, "", false, true, true, func(remote string, object *swift.Object, isDirectory bool) error {
+		if isDirectory {
+			return nil
 		}
-		segments, _ := containerSegments[segmentContainer]
-		segments = append(segments, segment.Name)
-		containerSegments[segmentContainer] = segments
-	}
-	return containerSegments, nil
-}
-
-func (o *Object) removeSegmentsLargeObject(ctx context.Context, containerSegments map[string][]string) error {
-	if containerSegments == nil || len(containerSegments) <= 0 {
-		return nil
-	}
-	for container, segments := range containerSegments {
-		_, err := o.fs.c.BulkDelete(ctx, container, segments)
-		if err != nil {
-			fs.Debugf(o, "Failed to delete bulk segments %v", err)
-			return err
+		if except != "" && strings.HasPrefix(remote, except) {
+			// fs.Debugf(o, "Ignoring current segment file %q in container %q", remote, segmentsContainer)
+			return nil
 		}
+		fs.Debugf(o, "Removing segment file %q in container %q", remote, segmentsContainer)
+		var err error
+		return o.fs.pacer.Call(func() (bool, error) {
+			err = o.fs.c.ObjectDelete(segmentsContainer, remote)
+			return shouldRetry(err)
+		})
+	})
+	if err != nil {
+		return err
+	}
+	// remove the segments container if empty, ignore errors
+	err = o.fs.pacer.Call(func() (bool, error) {
+		err = o.fs.c.ContainerDelete(segmentsContainer)
+		if err == swift.ContainerNotFound || err == swift.ContainerNotEmpty {
+			return false, err
+		}
+		return shouldRetry(err)
+	})
+	if err == nil {
+		fs.Debugf(o, "Removed empty container %q", segmentsContainer)
 	}
 	return nil
 }
 
-func (o *Object) getSegmentsDlo(ctx context.Context) (segmentsContainer string, prefix string, err error) {
-	if err = o.readMetaData(ctx); err != nil {
+func (o *Object) getSegmentsDlo() (segmentsContainer string, prefix string, err error) {
+	if err = o.readMetaData(); err != nil {
 		return
 	}
 	dirManifest := o.headers["X-Object-Manifest"]
@@ -1192,7 +1194,7 @@ func urlEncode(str string) string {
 	var buf bytes.Buffer
 	for i := 0; i < len(str); i++ {
 		c := str[i]
-		if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '/' || c == '.' || c == '_' || c == '-' {
+		if (c >= '0' && c <= '9') || (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z') || c == '/' || c == '.' {
 			_ = buf.WriteByte(c)
 		} else {
 			_, _ = buf.WriteString(fmt.Sprintf("%%%02X", c))
@@ -1203,14 +1205,14 @@ func urlEncode(str string) string {
 
 // updateChunks updates the existing object using chunks to a separate
 // container.  It returns a string which prefixes current segments.
-func (o *Object) updateChunks(ctx context.Context, in0 io.Reader, headers swift.Headers, size int64, contentType string) (string, error) {
+func (o *Object) updateChunks(in0 io.Reader, headers swift.Headers, size int64, contentType string) (string, error) {
 	container, containerPath := o.split()
 	segmentsContainer := container + "_segments"
 	// Create the segmentsContainer if it doesn't exist
 	var err error
 	err = o.fs.pacer.Call(func() (bool, error) {
 		var rxHeaders swift.Headers
-		_, rxHeaders, err = o.fs.c.Container(ctx, segmentsContainer)
+		_, rxHeaders, err = o.fs.c.Container(segmentsContainer)
 		return shouldRetryHeaders(rxHeaders, err)
 	})
 	if err == swift.ContainerNotFound {
@@ -1219,7 +1221,7 @@ func (o *Object) updateChunks(ctx context.Context, in0 io.Reader, headers swift.
 			headers["X-Storage-Policy"] = o.fs.opt.StoragePolicy
 		}
 		err = o.fs.pacer.Call(func() (bool, error) {
-			err = o.fs.c.ContainerCreate(ctx, segmentsContainer, headers)
+			err = o.fs.c.ContainerCreate(segmentsContainer, headers)
 			return shouldRetry(err)
 		})
 	}
@@ -1232,20 +1234,10 @@ func (o *Object) updateChunks(ctx context.Context, in0 io.Reader, headers swift.
 	uniquePrefix := fmt.Sprintf("%s/%d", swift.TimeToFloatString(time.Now()), size)
 	segmentsPath := path.Join(containerPath, uniquePrefix)
 	in := bufio.NewReader(in0)
-	segmentInfos := make([]string, 0, (size/int64(o.fs.opt.ChunkSize))+1)
-	defer atexit.OnError(&err, func() {
-		if o.fs.opt.LeavePartsOnError {
-			return
-		}
-		fs.Debugf(o, "Delete segments when err raise %v", err)
-		if segmentInfos == nil || len(segmentInfos) == 0 {
-			return
-		}
-		deleteChunks(ctx, o, segmentsContainer, segmentInfos)
-	})()
+	segmentInfos := make([]string, 0, ((size / int64(o.fs.opt.ChunkSize)) + 1))
 	for {
 		// can we read at least one byte?
-		if _, err = in.Peek(1); err != nil {
+		if _, err := in.Peek(1); err != nil {
 			if left > 0 {
 				return "", err // read less than expected
 			}
@@ -1263,13 +1255,15 @@ func (o *Object) updateChunks(ctx context.Context, in0 io.Reader, headers swift.
 		fs.Debugf(o, "Uploading segment file %q into %q", segmentPath, segmentsContainer)
 		err = o.fs.pacer.CallNoRetry(func() (bool, error) {
 			var rxHeaders swift.Headers
-			rxHeaders, err = o.fs.c.ObjectPut(ctx, segmentsContainer, segmentPath, segmentReader, true, "", "", headers)
+			rxHeaders, err = o.fs.c.ObjectPut(segmentsContainer, segmentPath, segmentReader, true, "", "", headers)
 			if err == nil {
 				segmentInfos = append(segmentInfos, segmentPath)
 			}
 			return shouldRetryHeaders(rxHeaders, err)
 		})
 		if err != nil {
+			deleteChunks(o, segmentsContainer, segmentInfos)
+			segmentInfos = nil
 			return "", err
 		}
 		i++
@@ -1280,26 +1274,24 @@ func (o *Object) updateChunks(ctx context.Context, in0 io.Reader, headers swift.
 	emptyReader := bytes.NewReader(nil)
 	err = o.fs.pacer.Call(func() (bool, error) {
 		var rxHeaders swift.Headers
-		rxHeaders, err = o.fs.c.ObjectPut(ctx, container, containerPath, emptyReader, true, "", contentType, headers)
+		rxHeaders, err = o.fs.c.ObjectPut(container, containerPath, emptyReader, true, "", contentType, headers)
 		return shouldRetryHeaders(rxHeaders, err)
 	})
-
-	if err == nil {
-		//reset data
+	if err != nil {
+		deleteChunks(o, segmentsContainer, segmentInfos)
 		segmentInfos = nil
 	}
 	return uniquePrefix + "/", err
 }
 
-func deleteChunks(ctx context.Context, o *Object, segmentsContainer string, segmentInfos []string) {
-	if segmentInfos == nil || len(segmentInfos) == 0 {
-		return
-	}
-	for _, v := range segmentInfos {
-		fs.Debugf(o, "Delete segment file %q on %q", v, segmentsContainer)
-		e := o.fs.c.ObjectDelete(ctx, segmentsContainer, v)
-		if e != nil {
-			fs.Errorf(o, "Error occurred in delete segment file %q on %q, error: %q", v, segmentsContainer, e)
+func deleteChunks(o *Object, segmentsContainer string, segmentInfos []string) {
+	if segmentInfos != nil && len(segmentInfos) > 0 {
+		for _, v := range segmentInfos {
+			fs.Debugf(o, "Delete segment file %q on %q", v, segmentsContainer)
+			e := o.fs.c.ObjectDelete(segmentsContainer, v)
+			if e != nil {
+				fs.Errorf(o, "Error occurred in delete segment file %q on %q, error: %q", v, segmentsContainer, e)
+			}
 		}
 	}
 }
@@ -1320,26 +1312,20 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 	modTime := src.ModTime(ctx)
 
 	// Note whether this is a dynamic large object before starting
-	isLargeObject, err := o.isLargeObject(ctx)
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
 	if err != nil {
 		return err
 	}
 
-	//capture segments before upload
-	var segmentsContainer map[string][]string
-	if isLargeObject {
-		segmentsContainer, _ = o.getSegmentsLargeObject(ctx)
-	}
-
 	// Set the mtime
 	m := swift.Metadata{}
 	m.SetModTime(modTime)
 	contentType := fs.MimeType(ctx, src)
 	headers := m.ObjectHeaders()
 	fs.OpenOptionAddHeaders(options, headers)
-
+	uniquePrefix := ""
 	if size > int64(o.fs.opt.ChunkSize) || (size == -1 && !o.fs.opt.NoChunk) {
-		_, err = o.updateChunks(ctx, in, headers, size, contentType)
+		uniquePrefix, err = o.updateChunks(in, headers, size, contentType)
 		if err != nil {
 			return err
 		}
@@ -1355,7 +1341,7 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 		}
 		var rxHeaders swift.Headers
 		err = o.fs.pacer.CallNoRetry(func() (bool, error) {
-			rxHeaders, err = o.fs.c.ObjectPut(ctx, container, containerPath, in, true, "", contentType, headers)
+			rxHeaders, err = o.fs.c.ObjectPut(container, containerPath, in, true, "", contentType, headers)
 			return shouldRetryHeaders(rxHeaders, err)
 		})
 		if err != nil {
@@ -1373,17 +1359,17 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 			o.size = int64(inCount.BytesRead())
 		}
 	}
-	isInContainerVersioning, _ := o.isInContainerVersioning(ctx, container)
-	// If file was a large object and the container is not enable versioning then remove old/all segments
-	if isLargeObject && len(segmentsContainer) > 0 && !isInContainerVersioning {
-		err := o.removeSegmentsLargeObject(ctx, segmentsContainer)
+
+	// If file was a dynamic large object then remove old/all segments
+	if isDynamicLargeObject {
+		err = o.removeSegments(uniquePrefix)
 		if err != nil {
 			fs.Logf(o, "Failed to remove old segments - carrying on with upload: %v", err)
 		}
 	}
 
 	// Read the metadata from the newly created object if necessary
-	return o.readMetaData(ctx)
+	return o.readMetaData()
 }
 
 // Remove an object
@@ -1391,29 +1377,34 @@ func (o *Object) Remove(ctx context.Context) (err error) {
 	container, containerPath := o.split()
 
 	//check object is large object
-	isLargeObject, err := o.isLargeObject(ctx)
+	isLargeObject, err := o.isLargeObject()
 	if err != nil {
 		return err
 	}
 	//check container has enabled version to reserve segment when delete
 	isInContainerVersioning := false
 	if isLargeObject {
-		isInContainerVersioning, err = o.isInContainerVersioning(ctx, container)
+		isInContainerVersioning, err = o.isInContainerVersioning(container)
 		if err != nil {
 			return err
 		}
 	}
-	//capture segments object if this object is large object
-	var containerSegments map[string][]string
-	if isLargeObject {
-		containerSegments, err = o.getSegmentsLargeObject(ctx)
+	isStaticLargeObject, err := o.isStaticLargeObject()
+	if err != nil {
+		return err
+	}
+	var segmentContainer string
+	var segmentObjects []swift.Object
+
+	if isStaticLargeObject {
+		segmentContainer, segmentObjects, err = o.fs.c.LargeObjectGetSegments(container, containerPath)
 		if err != nil {
 			return err
 		}
 	}
 	// Remove file/manifest first
 	err = o.fs.pacer.Call(func() (bool, error) {
-		err = o.fs.c.ObjectDelete(ctx, container, containerPath)
+		err = o.fs.c.ObjectDelete(container, containerPath)
 		return shouldRetry(err)
 	})
 	if err != nil {
@@ -1424,9 +1415,31 @@ func (o *Object) Remove(ctx context.Context) (err error) {
 		return nil
 	}
 
-	if isLargeObject {
-		return o.removeSegmentsLargeObject(ctx, containerSegments)
+	isDynamicLargeObject, err := o.isDynamicLargeObject()
+	if err != nil {
+		return err
 	}
+	// ...then segments if required
+	//delete segment for dynamic large object
+	if isDynamicLargeObject {
+		return o.removeSegments("")
+	}
+
+	//delete segment for static large object
+	if isStaticLargeObject && len(segmentContainer) > 0 && segmentObjects != nil && len(segmentObjects) > 0 {
+		var segmentNames []string
+		for _, segmentObject := range segmentObjects {
+			if len(segmentObject.Name) == 0 {
+				continue
+			}
+			segmentNames = append(segmentNames, segmentObject.Name)
+		}
+		_, err := o.fs.c.BulkDelete(segmentContainer, segmentNames)
+		if err != nil {
+			return err
+		}
+	}
+
 	return nil
 }
 

backend/swift/swift_internal_test.go

@@ -4,7 +4,7 @@ import (
 	"testing"
 	"time"
 
-	"github.com/ncw/swift/v2"
+	"github.com/ncw/swift"
 	"github.com/rclone/rclone/fs/fserrors"
 	"github.com/stretchr/testify/assert"
 )

backend/swift/swift_test.go

@@ -4,19 +4,15 @@ package swift
 import (
 	"bytes"
 	"context"
-	"errors"
 	"io"
-	"io/ioutil"
 	"testing"
 
-	"github.com/ncw/swift/v2"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/hash"
 	"github.com/rclone/rclone/fs/object"
 	"github.com/rclone/rclone/fstest"
 	"github.com/rclone/rclone/fstest/fstests"
 	"github.com/rclone/rclone/lib/random"
-	"github.com/rclone/rclone/lib/readers"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
@@ -78,80 +74,6 @@ func (f *Fs) testNoChunk(t *testing.T) {
 // Additional tests that aren't in the framework
 func (f *Fs) InternalTest(t *testing.T) {
 	t.Run("NoChunk", f.testNoChunk)
-	t.Run("WithChunk", f.testWithChunk)
-	t.Run("WithChunkFail", f.testWithChunkFail)
-}
-
-func (f *Fs) testWithChunk(t *testing.T) {
-	preConfChunkSize := f.opt.ChunkSize
-	preConfChunk := f.opt.NoChunk
-	f.opt.NoChunk = false
-	f.opt.ChunkSize = 1024 * fs.Byte
-	defer func() {
-		//restore old config after test
-		f.opt.ChunkSize = preConfChunkSize
-		f.opt.NoChunk = preConfChunk
-	}()
-
-	file := fstest.Item{
-		ModTime: fstest.Time("2020-12-31T04:05:06.499999999Z"),
-		Path:    "piped data chunk.txt",
-		Size:    -1, // use unknown size during upload
-	}
-	const contentSize = 2048
-	contents := random.String(contentSize)
-	buf := bytes.NewBufferString(contents)
-	uploadHash := hash.NewMultiHasher()
-	in := io.TeeReader(buf, uploadHash)
-
-	file.Size = -1
-	obji := object.NewStaticObjectInfo(file.Path, file.ModTime, file.Size, true, nil, nil)
-	ctx := context.TODO()
-	obj, err := f.Features().PutStream(ctx, in, obji)
-	require.NoError(t, err)
-	require.NotEmpty(t, obj)
-}
-
-func (f *Fs) testWithChunkFail(t *testing.T) {
-	preConfChunkSize := f.opt.ChunkSize
-	preConfChunk := f.opt.NoChunk
-	f.opt.NoChunk = false
-	f.opt.ChunkSize = 1024 * fs.Byte
-	segmentContainer := f.root + "_segments"
-	defer func() {
-		//restore config
-		f.opt.ChunkSize = preConfChunkSize
-		f.opt.NoChunk = preConfChunk
-	}()
-	path := "piped data chunk with error.txt"
-	file := fstest.Item{
-		ModTime: fstest.Time("2021-01-04T03:46:00.499999999Z"),
-		Path:    path,
-		Size:    -1, // use unknown size during upload
-	}
-	const contentSize = 4096
-	const errPosition = 3072
-	contents := random.String(contentSize)
-	buf := bytes.NewBufferString(contents[:errPosition])
-	errMessage := "potato"
-	er := &readers.ErrorReader{Err: errors.New(errMessage)}
-	in := ioutil.NopCloser(io.MultiReader(buf, er))
-
-	file.Size = contentSize
-	obji := object.NewStaticObjectInfo(file.Path, file.ModTime, file.Size, true, nil, nil)
-	ctx := context.TODO()
-	_, err := f.Features().PutStream(ctx, in, obji)
-	// error is potato
-	require.NotNil(t, err)
-	require.Equal(t, errMessage, err.Error())
-	_, _, err = f.c.Object(ctx, f.rootContainer, path)
-	assert.Equal(t, swift.ObjectNotFound, err)
-	prefix := path
-	objs, err := f.c.Objects(ctx, segmentContainer, &swift.ObjectsOpts{
-		Prefix: prefix,
-	})
-	require.NoError(t, err)
-	require.Empty(t, objs)
 }
 
 var _ fstests.InternalTester = (*Fs)(nil)

backend/tardigrade/fs.go

@@ -1,4 +1,4 @@
-// +build !plan9
+// +build go1.13,!plan9
 
 // Package tardigrade provides an interface to Tardigrade decentralized object storage.
 package tardigrade

backend/tardigrade/object.go

@@ -1,4 +1,4 @@
-// +build !plan9
+// +build go1.13,!plan9
 
 package tardigrade
 

backend/tardigrade/tardigrade_test.go

@@ -1,4 +1,4 @@
-// +build !plan9
+// +build go1.13,!plan9
 
 // Test Tardigrade filesystem interface
 package tardigrade_test

backend/tardigrade/tardigrade_unsupported.go

@@ -1,3 +1,3 @@
-// +build plan9
+// +build !go1.13 plan9
 
 package tardigrade

backend/union/upstream/upstream.go

@@ -67,7 +67,7 @@ func New(ctx context.Context, remote, root string, cacheTime time.Duration) (*Fs
 		return nil, err
 	}
 	f := &Fs{
-		RootPath:    strings.TrimRight(root, "/"),
+		RootPath:    root,
 		writable:    true,
 		creatable:   true,
 		cacheExpiry: time.Now().Unix(),

backend/webdav/odrvcookie/fetch.go

@@ -176,10 +176,11 @@ func (ca *CookieAuth) getSPToken(ctx context.Context) (conf *SharepointSuccessRe
 
 	// Create and execute the first request which returns an auth token for the sharepoint service
 	// With this token we can authenticate on the login page and save the returned cookies
-	req, err := http.NewRequestWithContext(ctx, "POST", "https://login.microsoftonline.com/extSTS.srf", buf)
+	req, err := http.NewRequest("POST", "https://login.microsoftonline.com/extSTS.srf", buf)
 	if err != nil {
 		return nil, err
 	}
+	req = req.WithContext(ctx) // go1.13 can use NewRequestWithContext
 
 	client := fshttp.NewClient(ctx)
 	resp, err := client.Do(req)

backend/webdav/webdav.go

@@ -10,7 +10,6 @@ package webdav
 import (
 	"bytes"
 	"context"
-	"crypto/tls"
 	"encoding/xml"
 	"fmt"
 	"io"
@@ -20,25 +19,20 @@ import (
 	"path"
 	"strconv"
 	"strings"
-	"sync"
 	"time"
 
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/backend/webdav/api"
 	"github.com/rclone/rclone/backend/webdav/odrvcookie"
 	"github.com/rclone/rclone/fs"
-	"github.com/rclone/rclone/fs/config"
 	"github.com/rclone/rclone/fs/config/configmap"
 	"github.com/rclone/rclone/fs/config/configstruct"
 	"github.com/rclone/rclone/fs/config/obscure"
 	"github.com/rclone/rclone/fs/fserrors"
 	"github.com/rclone/rclone/fs/fshttp"
 	"github.com/rclone/rclone/fs/hash"
-	"github.com/rclone/rclone/lib/encoder"
 	"github.com/rclone/rclone/lib/pacer"
 	"github.com/rclone/rclone/lib/rest"
-
-	ntlmssp "github.com/Azure/go-ntlmssp"
 )
 
 const (
@@ -48,22 +42,8 @@ const (
 	defaultDepth  = "1" // depth for PROPFIND
 )
 
-const defaultEncodingSharepointNTLM = (encoder.EncodeWin |
-	encoder.EncodeHashPercent | // required by IIS/8.5 in contrast with onedrive which doesn't need it
-	(encoder.Display &^ encoder.EncodeDot) | // test with IIS/8.5 shows that EncodeDot is not needed
-	encoder.EncodeBackSlash |
-	encoder.EncodeLeftSpace |
-	encoder.EncodeLeftTilde |
-	encoder.EncodeRightPeriod |
-	encoder.EncodeRightSpace |
-	encoder.EncodeInvalidUtf8)
-
 // Register with Fs
 func init() {
-	configEncodingHelp := fmt.Sprintf(
-		"%s\n\nDefault encoding is %s for sharepoint-ntlm or identity otherwise.",
-		config.ConfigEncodingHelp, defaultEncodingSharepointNTLM)
-
 	fs.Register(&fs.RegInfo{
 		Name:        "webdav",
 		Description: "Webdav",
@@ -87,17 +67,14 @@ func init() {
 				Help:  "Owncloud",
 			}, {
 				Value: "sharepoint",
-				Help:  "Sharepoint Online, authenticated by Microsoft account.",
-			}, {
-				Value: "sharepoint-ntlm",
-				Help:  "Sharepoint with NTLM authentication. Usually self-hosted or on-premises.",
+				Help:  "Sharepoint",
 			}, {
 				Value: "other",
 				Help:  "Other site/service or software",
 			}},
 		}, {
 			Name: "user",
-			Help: "User name. In case NTLM authentication is used, the username should be in the format 'Domain\\User'.",
+			Help: "User name",
 		}, {
 			Name:       "pass",
 			Help:       "Password.",
@@ -110,8 +87,13 @@ func init() {
 			Help:     "Command to run to get a bearer token",
 			Advanced: true,
 		}, {
-			Name:     config.ConfigEncoding,
-			Help:     configEncodingHelp,
+			Name:    "delete_on_error",
+			Default: false,
+			Help: `Delete a partially uploaded file on upload failure.
+
+Some webdav backends (eg rclone serve webdav) leave behind half
+written files on error.  This flag causes them to be deleted if the
+upload fails part of the way through.`,
 			Advanced: true,
 		}},
 	})
@@ -119,13 +101,13 @@ func init() {
 
 // Options defines the configuration for this backend
 type Options struct {
-	URL                string               `config:"url"`
-	Vendor             string               `config:"vendor"`
-	User               string               `config:"user"`
-	Pass               string               `config:"pass"`
-	BearerToken        string               `config:"bearer_token"`
-	BearerTokenCommand string               `config:"bearer_token_command"`
-	Enc                encoder.MultiEncoder `config:"encoding"`
+	URL                string `config:"url"`
+	Vendor             string `config:"vendor"`
+	User               string `config:"user"`
+	Pass               string `config:"pass"`
+	BearerToken        string `config:"bearer_token"`
+	BearerTokenCommand string `config:"bearer_token_command"`
+	DeleteOnError bool   `config:"delete_on_error"`
 }
 
 // Fs represents a remote webdav
@@ -142,10 +124,8 @@ type Fs struct {
 	canStream          bool          // set if can stream
 	useOCMtime         bool          // set if can use X-OC-Mtime
 	retryWithZeroDepth bool          // some vendors (sharepoint) won't list files when Depth is 1 (our default)
-	checkBeforePurge   bool          // enables extra check that directory to purge really exists
 	hasMD5             bool          // set if can use owncloud style checksums for MD5
 	hasSHA1            bool          // set if can use owncloud style checksums for SHA1
-	ntlmAuthMu         sync.Mutex    // mutex to serialize NTLM auth roundtrips
 }
 
 // Object describes a webdav object
@@ -209,22 +189,6 @@ func (f *Fs) shouldRetry(resp *http.Response, err error) (bool, error) {
 	return fserrors.ShouldRetry(err) || fserrors.ShouldRetryHTTP(resp, retryErrorCodes), err
 }
 
-// safeRoundTripper is a wrapper for http.RoundTripper that serializes
-// http roundtrips. NTLM authentication sequence can involve up to four
-// rounds of negotiations and might fail due to concurrency.
-// This wrapper allows to use ntlmssp.Negotiator safely with goroutines.
-type safeRoundTripper struct {
-	fs *Fs
-	rt http.RoundTripper
-}
-
-// RoundTrip guards wrapped RoundTripper by a mutex.
-func (srt *safeRoundTripper) RoundTrip(req *http.Request) (*http.Response, error) {
-	srt.fs.ntlmAuthMu.Lock()
-	defer srt.fs.ntlmAuthMu.Unlock()
-	return srt.rt.RoundTrip(req)
-}
-
 // itemIsDir returns true if the item is a directory
 //
 // When a client sees a resourcetype it doesn't recognize it should
@@ -331,11 +295,7 @@ func addSlash(s string) string {
 
 // filePath returns a file path (f.root, file)
 func (f *Fs) filePath(file string) string {
-	subPath := path.Join(f.root, file)
-	if f.opt.Enc != encoder.EncodeZero {
-		subPath = f.opt.Enc.FromStandardPath(subPath)
-	}
-	return rest.URLPathEscape(subPath)
+	return rest.URLPathEscape(path.Join(f.root, file))
 }
 
 // dirPath returns a directory path (f.root, dir)
@@ -374,10 +334,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 	}
 	root = strings.Trim(root, "/")
 
-	if opt.Enc == encoder.EncodeZero && opt.Vendor == "sharepoint-ntlm" {
-		opt.Enc = defaultEncodingSharepointNTLM
-	}
-
 	// Parse the endpoint
 	u, err := url.Parse(opt.URL)
 	if err != nil {
@@ -390,28 +346,10 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
 		opt:         *opt,
 		endpoint:    u,
 		endpointURL: u.String(),
+		srv:         rest.NewClient(fshttp.NewClient(ctx)).SetRoot(u.String()),
 		pacer:       fs.NewPacer(ctx, pacer.NewDefault(pacer.MinSleep(minSleep), pacer.MaxSleep(maxSleep), pacer.DecayConstant(decayConstant))),
 		precision:   fs.ModTimeNotSupported,
 	}
-
-	client := fshttp.NewClient(ctx)
-	if opt.Vendor == "sharepoint-ntlm" {
-		// Disable transparent HTTP/2 support as per https://golang.org/pkg/net/http/ ,
-		// otherwise any connection to IIS 10.0 fails with 'stream error: stream ID 39; HTTP_1_1_REQUIRED'
-		// https://docs.microsoft.com/en-us/iis/get-started/whats-new-in-iis-10/http2-on-iis says:
-		// 'Windows authentication (NTLM/Kerberos/Negotiate) is not supported with HTTP/2.'
-		t := fshttp.NewTransportCustom(ctx, func(t *http.Transport) {
-			t.TLSNextProto = map[string]func(string, *tls.Conn) http.RoundTripper{}
-		})
-
-		// Add NTLM layer
-		client.Transport = &safeRoundTripper{
-			fs: f,
-			rt: ntlmssp.Negotiator{RoundTripper: t},
-		}
-	}
-	f.srv = rest.NewClient(client).SetRoot(u.String())
-
 	f.features = (&fs.Features{
 		CanHaveEmptyDirectories: true,
 	}).Fill(ctx, f)
@@ -537,16 +475,6 @@ func (f *Fs) setQuirks(ctx context.Context, vendor string) error {
 		// to determine if we may have found a file, the request has to be resent
 		// with the depth set to 0
 		f.retryWithZeroDepth = true
-	case "sharepoint-ntlm":
-		// Sharepoint with NTLM authentication
-		// See comment above
-		f.retryWithZeroDepth = true
-
-		// Sharepoint 2016 returns status 204 to the purge request
-		// even if the directory to purge does not really exist
-		// so we must perform an extra check to detect this
-		// condition and return a proper error code.
-		f.checkBeforePurge = true
 	case "other":
 	default:
 		fs.Debugf(f, "Unknown vendor %q", vendor)
@@ -665,11 +593,7 @@ func (f *Fs) listAll(ctx context.Context, dir string, directoriesOnly bool, file
 			fs.Debugf(nil, "Item with unknown path received: %q, %q", u.Path, baseURL.Path)
 			continue
 		}
-		subPath := u.Path[len(baseURL.Path):]
-		if f.opt.Enc != encoder.EncodeZero {
-			subPath = f.opt.Enc.ToStandardPath(subPath)
-		}
-		remote := path.Join(dir, subPath)
+		remote := path.Join(dir, u.Path[len(baseURL.Path):])
 		if strings.HasSuffix(remote, "/") {
 			remote = remote[:len(remote)-1]
 		}
@@ -886,21 +810,6 @@ func (f *Fs) purgeCheck(ctx context.Context, dir string, check bool) error {
 		if notEmpty {
 			return fs.ErrorDirectoryNotEmpty
 		}
-	} else if f.checkBeforePurge {
-		// We are doing purge as the `check` argument is unset.
-		// The quirk says that we are working with Sharepoint 2016.
-		// This provider returns status 204 even if the purged directory
-		// does not really exist so we perform an extra check here.
-		// Only the existence is checked, all other errors must be
-		// ignored here to make the rclone test suite pass.
-		depth := defaultDepth
-		if f.retryWithZeroDepth {
-			depth = "0"
-		}
-		_, err := f.readMetaDataForPath(ctx, dir, depth)
-		if err == fs.ErrorObjectNotFound {
-			return fs.ErrorDirNotFound
-		}
 	}
 	opts := rest.Opts{
 		Method:     "DELETE",
@@ -1294,14 +1203,16 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
 		return o.fs.shouldRetry(resp, err)
 	})
 	if err != nil {
-		// Give the WebDAV server a chance to get its internal state in order after the
-		// error.  The error may have been local in which case we closed the connection.
-		// The server may still be dealing with it for a moment. A sleep isn't ideal but I
-		// haven't been able to think of a better method to find out if the server has
-		// finished - ncw
-		time.Sleep(1 * time.Second)
-		// Remove failed upload
-		_ = o.Remove(ctx)
+		if o.fs.opt.DeleteOnError {
+			// Give the WebDAV server a chance to get its internal state in order after the
+			// error.  The error may have been local in which case we closed the connection.
+			// The server may still be dealing with it for a moment. A sleep isn't ideal but I
+			// haven't been able to think of a better method to find out if the server has
+			// finished - ncw
+			time.Sleep(1 * time.Second)
+			// Remove failed upload
+			_ = o.Remove(ctx)
+		}
 		return err
 	}
 	// read metadata from remote

backend/webdav/webdav_test.go

@@ -38,14 +38,3 @@ func TestIntegration3(t *testing.T) {
 		NilObject:  (*webdav.Object)(nil),
 	})
 }
-
-// TestIntegration runs integration tests against the remote
-func TestIntegration4(t *testing.T) {
-	if *fstest.RemoteName != "" {
-		t.Skip("skipping as -remote is set")
-	}
-	fstests.Run(t, &fstests.Opt{
-		RemoteName: "TestWebdavNTLM:",
-		NilObject:  (*webdav.Object)(nil),
-	})
-}

backend/yandex/yandex.go

@@ -537,7 +537,7 @@ func (f *Fs) waitForJob(ctx context.Context, location string) (err error) {
 		RootURL: location,
 		Method:  "GET",
 	}
-	deadline := time.Now().Add(f.ci.TimeoutOrInfinite())
+	deadline := time.Now().Add(f.ci.Timeout)
 	for time.Now().Before(deadline) {
 		var resp *http.Response
 		var body []byte
@@ -568,7 +568,7 @@ func (f *Fs) waitForJob(ctx context.Context, location string) (err error) {
 
 		time.Sleep(1 * time.Second)
 	}
-	return errors.Errorf("async operation didn't complete after %v", f.ci.TimeoutOrInfinite())
+	return errors.Errorf("async operation didn't complete after %v", f.ci.Timeout)
 }
 
 func (f *Fs) delete(ctx context.Context, path string, hardDelete bool) (err error) {

backend/zoho/zoho.go

@@ -100,7 +100,7 @@ func init() {
 				log.Fatalf("Failed to configure root directory: %v", err)
 			}
 		},
-		Options: append(oauthutil.SharedOptions, []fs.Option{{
+		Options: []fs.Option{{
 			Name: "region",
 			Help: "Zoho region to connect to. You'll have to use the region you organization is registered in.",
 			Examples: []fs.OptionExample{{
@@ -123,7 +123,7 @@ func init() {
 				encoder.EncodeCtl |
 				encoder.EncodeDel |
 				encoder.EncodeInvalidUtf8),
-		}}...),
+		}},
 	})
 }
 

bin/cross-compile.go

@@ -27,22 +27,17 @@ import (
 
 var (
 	// Flags
-	debug           = flag.Bool("d", false, "Print commands instead of running them.")
-	parallel        = flag.Int("parallel", runtime.NumCPU(), "Number of commands to run in parallel.")
-	copyAs          = flag.String("release", "", "Make copies of the releases with this name")
-	gitLog          = flag.String("git-log", "", "git log to include as well")
-	include         = flag.String("include", "^.*$", "os/arch regexp to include")
-	exclude         = flag.String("exclude", "^$", "os/arch regexp to exclude")
-	cgo             = flag.Bool("cgo", false, "Use cgo for the build")
-	noClean         = flag.Bool("no-clean", false, "Don't clean the build directory before running.")
-	tags            = flag.String("tags", "", "Space separated list of build tags")
-	buildmode       = flag.String("buildmode", "", "Passed to go build -buildmode flag")
-	compileOnly     = flag.Bool("compile-only", false, "Just build the binary, not the zip.")
-	extraEnv        = flag.String("env", "", "comma separated list of VAR=VALUE env vars to set")
-	macOSSDK        = flag.String("macos-sdk", "", "macOS SDK to use")
-	macOSArch       = flag.String("macos-arch", "", "macOS arch to use")
-	extraCgoCFlags  = flag.String("cgo-cflags", "", "extra CGO_CFLAGS")
-	extraCgoLdFlags = flag.String("cgo-ldflags", "", "extra CGO_LDFLAGS")
+	debug       = flag.Bool("d", false, "Print commands instead of running them.")
+	parallel    = flag.Int("parallel", runtime.NumCPU(), "Number of commands to run in parallel.")
+	copyAs      = flag.String("release", "", "Make copies of the releases with this name")
+	gitLog      = flag.String("git-log", "", "git log to include as well")
+	include     = flag.String("include", "^.*$", "os/arch regexp to include")
+	exclude     = flag.String("exclude", "^$", "os/arch regexp to exclude")
+	cgo         = flag.Bool("cgo", false, "Use cgo for the build")
+	noClean     = flag.Bool("no-clean", false, "Don't clean the build directory before running.")
+	tags        = flag.String("tags", "", "Space separated list of build tags")
+	buildmode   = flag.String("buildmode", "", "Passed to go build -buildmode flag")
+	compileOnly = flag.Bool("compile-only", false, "Just build the binary, not the zip.")
 )
 
 // GOOS/GOARCH pairs we build for
@@ -52,7 +47,6 @@ var osarches = []string{
 	"windows/386",
 	"windows/amd64",
 	"darwin/amd64",
-	"darwin/arm64",
 	"linux/386",
 	"linux/amd64",
 	"linux/arm",
@@ -78,7 +72,7 @@ var osarches = []string{
 
 // Special environment flags for a given arch
 var archFlags = map[string][]string{
-	"386":    {"GO386=softfloat"},
+	"386":    {"GO386=387"},
 	"mips":   {"GOMIPS=softfloat"},
 	"mipsle": {"GOMIPS=softfloat"},
 	"arm-v7": {"GOARM=7"},
@@ -285,15 +279,6 @@ func stripVersion(goarch string) string {
 	return goarch[:i]
 }
 
-// run the command returning trimmed output
-func runOut(command ...string) string {
-	out, err := exec.Command(command[0], command[1:]...).Output()
-	if err != nil {
-		log.Fatalf("Failed to run %q: %v", command, err)
-	}
-	return strings.TrimSpace(string(out))
-}
-
 // build the binary in dir returning success or failure
 func compileArch(version, goos, goarch, dir string) bool {
 	log.Printf("Compiling %s/%s into %s", goos, goarch, dir)
@@ -329,35 +314,6 @@ func compileArch(version, goos, goarch, dir string) bool {
 		"GOOS=" + goos,
 		"GOARCH=" + stripVersion(goarch),
 	}
-	if *extraEnv != "" {
-		env = append(env, strings.Split(*extraEnv, ",")...)
-	}
-	var (
-		cgoCFlags  []string
-		cgoLdFlags []string
-	)
-	if *macOSSDK != "" {
-		flag := "-isysroot " + runOut("xcrun", "--sdk", *macOSSDK, "--show-sdk-path")
-		cgoCFlags = append(cgoCFlags, flag)
-		cgoLdFlags = append(cgoLdFlags, flag)
-	}
-	if *macOSArch != "" {
-		flag := "-arch " + *macOSArch
-		cgoCFlags = append(cgoCFlags, flag)
-		cgoLdFlags = append(cgoLdFlags, flag)
-	}
-	if *extraCgoCFlags != "" {
-		cgoCFlags = append(cgoCFlags, *extraCgoCFlags)
-	}
-	if *extraCgoLdFlags != "" {
-		cgoLdFlags = append(cgoLdFlags, *extraCgoLdFlags)
-	}
-	if len(cgoCFlags) > 0 {
-		env = append(env, "CGO_CFLAGS="+strings.Join(cgoCFlags, " "))
-	}
-	if len(cgoLdFlags) > 0 {
-		env = append(env, "CGO_LDFLAGS="+strings.Join(cgoLdFlags, " "))
-	}
 	if !*cgo {
 		env = append(env, "CGO_ENABLED=0")
 	} else {

bin/make_test_files.go

@@ -0,0 +1,146 @@
+// +build ignore
+
+// Build a directory structure with the required number of files in
+//
+// Run with go run make_test_files.go [flag] <directory>
+package main
+
+import (
+	cryptrand "crypto/rand"
+	"flag"
+	"io"
+	"log"
+	"math/rand"
+	"os"
+	"path/filepath"
+)
+
+var (
+	// Flags
+	numberOfFiles            = flag.Int("n", 1000, "Number of files to create")
+	averageFilesPerDirectory = flag.Int("files-per-directory", 10, "Average number of files per directory")
+	maxDepth                 = flag.Int("max-depth", 10, "Maximum depth of directory hierarchy")
+	minFileSize              = flag.Int64("min-size", 0, "Minimum size of file to create")
+	maxFileSize              = flag.Int64("max-size", 100, "Maximum size of files to create")
+	minFileNameLength        = flag.Int("min-name-length", 4, "Minimum size of file to create")
+	maxFileNameLength        = flag.Int("max-name-length", 12, "Maximum size of files to create")
+
+	directoriesToCreate int
+	totalDirectories    int
+	fileNames           = map[string]struct{}{} // keep a note of which file name we've used already
+)
+
+// randomString create a random string for test purposes
+func randomString(n int) string {
+	const (
+		vowel     = "aeiou"
+		consonant = "bcdfghjklmnpqrstvwxyz"
+		digit     = "0123456789"
+	)
+	pattern := []string{consonant, vowel, consonant, vowel, consonant, vowel, consonant, digit}
+	out := make([]byte, n)
+	p := 0
+	for i := range out {
+		source := pattern[p]
+		p = (p + 1) % len(pattern)
+		out[i] = source[rand.Intn(len(source))]
+	}
+	return string(out)
+}
+
+// fileName creates a unique random file or directory name
+func fileName() (name string) {
+	for {
+		length := rand.Intn(*maxFileNameLength-*minFileNameLength) + *minFileNameLength
+		name = randomString(length)
+		if _, found := fileNames[name]; !found {
+			break
+		}
+	}
+	fileNames[name] = struct{}{}
+	return name
+}
+
+// dir is a directory in the directory hierarchy being built up
+type dir struct {
+	name     string
+	depth    int
+	children []*dir
+	parent   *dir
+}
+
+// Create a random directory hierarchy under d
+func (d *dir) createDirectories() {
+	for totalDirectories < directoriesToCreate {
+		newDir := &dir{
+			name:   fileName(),
+			depth:  d.depth + 1,
+			parent: d,
+		}
+		d.children = append(d.children, newDir)
+		totalDirectories++
+		switch rand.Intn(4) {
+		case 0:
+			if d.depth < *maxDepth {
+				newDir.createDirectories()
+			}
+		case 1:
+			return
+		}
+	}
+	return
+}
+
+// list the directory hierarchy
+func (d *dir) list(path string, output []string) []string {
+	dirPath := filepath.Join(path, d.name)
+	output = append(output, dirPath)
+	for _, subDir := range d.children {
+		output = subDir.list(dirPath, output)
+	}
+	return output
+}
+
+// writeFile writes a random file at dir/name
+func writeFile(dir, name string) {
+	err := os.MkdirAll(dir, 0777)
+	if err != nil {
+		log.Fatalf("Failed to make directory %q: %v", dir, err)
+	}
+	path := filepath.Join(dir, name)
+	fd, err := os.Create(path)
+	if err != nil {
+		log.Fatalf("Failed to open file %q: %v", path, err)
+	}
+	size := rand.Int63n(*maxFileSize-*minFileSize) + *minFileSize
+	_, err = io.CopyN(fd, cryptrand.Reader, size)
+	if err != nil {
+		log.Fatalf("Failed to write %v bytes to file %q: %v", size, path, err)
+	}
+	err = fd.Close()
+	if err != nil {
+		log.Fatalf("Failed to close file %q: %v", path, err)
+	}
+}
+
+func main() {
+	flag.Parse()
+	args := flag.Args()
+	if len(args) != 1 {
+		log.Fatalf("Require 1 directory argument")
+	}
+	outputDirectory := args[0]
+	log.Printf("Output dir %q", outputDirectory)
+
+	directoriesToCreate = *numberOfFiles / *averageFilesPerDirectory
+	log.Printf("directoriesToCreate %v", directoriesToCreate)
+	root := &dir{name: outputDirectory, depth: 1}
+	for totalDirectories < directoriesToCreate {
+		root.createDirectories()
+	}
+	dirs := root.list("", []string{})
+	for i := 0; i < *numberOfFiles; i++ {
+		dir := dirs[rand.Intn(len(dirs))]
+		writeFile(dir, fileName())
+	}
+}

bin/upload-github

@@ -2,45 +2,49 @@
 #
 # Upload a release
 #
-# Needs the gh tool from https://github.com/cli/cli
+# Needs github-release from https://github.com/aktau/github-release
 
 set -e
 
-REPO="rclone/rclone"
+REPO="rclone"
 
 if [ "$1" == "" ]; then
     echo "Syntax: $0 Version"
     exit 1
 fi
 VERSION="$1"
-ANCHOR=$(grep '^## v' docs/content/changelog.md | head -1 | sed 's/^## //; s/[^A-Za-z0-9-]/-/g; s/--*/-/g')
+if [ "$GITHUB_USER" == "" ]; then
+   echo 1>&2 "Need GITHUB_USER environment variable"
+   exit 1
+fi
+if [ "$GITHUB_TOKEN" == "" ]; then
+   echo 1>&2 "Need GITHUB_TOKEN environment variable"
+   exit 1
+fi
 
-cat > "/tmp/${VERSION}-release-notes" <<EOF
-This is the ${VERSION} release of rclone.
-
-Full details of the changes can be found in [the changelog](https://rclone.org/changelog/#${ANCHOR}).
-EOF
-
-echo "Making release ${VERSION} anchor ${ANCHOR} to repo ${REPO}"
-
-gh release create "${VERSION}" \
+echo "Making release ${VERSION}"
+github-release release \
     --repo ${REPO} \
-    --title "rclone ${VERSION}" \
-    --notes-file "/tmp/${VERSION}-release-notes"
+    --tag ${VERSION} \
+    --name "rclone" \
+    --description "Rclone - rsync for cloud storage. Sync files to and from many cloud storage providers."
 
-for build in build/*; do
-    case $build in
-        *current*) continue ;;
-        *testbuilds*) continue ;;
-    esac
-    echo "Uploading ${build} "
-    gh release upload "${VERSION}" \
-        --clobber \
+for build in `ls build | grep -v current | grep -v testbuilds`; do
+    echo "Uploading ${build}"
+    base="${build%.*}"
+    parts=(${base//-/ })
+    os=${parts[3]}
+    arch=${parts[4]}
+
+    github-release upload \
         --repo ${REPO} \
-        "${build}"
+        --tag ${VERSION} \
+        --name "${build}" \
+        --file build/${build}
 done
 
-gh release view "${VERSION}" \
-    --repo ${REPO}
+github-release info \
+    --repo ${REPO} \
+    --tag ${VERSION}
 
 echo "Done"

cmd/all/all.go

@@ -25,6 +25,7 @@ import (
 	_ "github.com/rclone/rclone/cmd/genautocomplete"
 	_ "github.com/rclone/rclone/cmd/gendocs"
 	_ "github.com/rclone/rclone/cmd/hashsum"
+	_ "github.com/rclone/rclone/cmd/info"
 	_ "github.com/rclone/rclone/cmd/link"
 	_ "github.com/rclone/rclone/cmd/listremotes"
 	_ "github.com/rclone/rclone/cmd/ls"
@@ -33,6 +34,7 @@ import (
 	_ "github.com/rclone/rclone/cmd/lsjson"
 	_ "github.com/rclone/rclone/cmd/lsl"
 	_ "github.com/rclone/rclone/cmd/md5sum"
+	_ "github.com/rclone/rclone/cmd/memtest"
 	_ "github.com/rclone/rclone/cmd/mkdir"
 	_ "github.com/rclone/rclone/cmd/mount"
 	_ "github.com/rclone/rclone/cmd/mount2"
@@ -52,11 +54,6 @@ import (
 	_ "github.com/rclone/rclone/cmd/sha1sum"
 	_ "github.com/rclone/rclone/cmd/size"
 	_ "github.com/rclone/rclone/cmd/sync"
-	_ "github.com/rclone/rclone/cmd/test"
-	_ "github.com/rclone/rclone/cmd/test/histogram"
-	_ "github.com/rclone/rclone/cmd/test/info"
-	_ "github.com/rclone/rclone/cmd/test/makefiles"
-	_ "github.com/rclone/rclone/cmd/test/memory"
 	_ "github.com/rclone/rclone/cmd/touch"
 	_ "github.com/rclone/rclone/cmd/tree"
 	_ "github.com/rclone/rclone/cmd/version"

cmd/cat/cat.go

@@ -6,7 +6,6 @@ import (
 	"io/ioutil"
 	"log"
 	"os"
-	"strings"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/rclone/rclone/fs/config/flags"
@@ -36,8 +35,7 @@ func init() {
 var commandDefinition = &cobra.Command{
 	Use:   "cat remote:path",
 	Short: `Concatenates any files and sends them to stdout.`,
-	// Warning! "|" will be replaced by backticks below
-	Long: strings.ReplaceAll(`
+	Long: `
 rclone cat sends any files to standard output.
 
 You can use it like this to output a single file
@@ -52,11 +50,11 @@ Or like this to output any .txt files in dir or its subdirectories.
 
     rclone --include "*.txt" cat remote:path/to/dir
 
-Use the |--head| flag to print characters only at the start, |--tail| for
-the end and |--offset| and |--count| to print a section in the middle.
+Use the --head flag to print characters only at the start, --tail for
+the end and --offset and --count to print a section in the middle.
 Note that if offset is negative it will count from the end, so
-|--offset -1 --count 1| is equivalent to |--tail 1|.
-`, "|", "`"),
+--offset -1 --count 1 is equivalent to --tail 1.
+`,
 	Run: func(command *cobra.Command, args []string) {
 		usedOffset := offset != 0 || count >= 0
 		usedHead := head > 0

cmd/check/check.go

@@ -45,8 +45,7 @@ func AddFlags(cmdFlags *pflag.FlagSet) {
 }
 
 // FlagsHelp describes the flags for the help
-// Warning! "|" will be replaced by backticks below
-var FlagsHelp = strings.ReplaceAll(`
+var FlagsHelp = strings.Replace(`
 If you supply the |--one-way| flag, it will only check that files in
 the source match the files in the destination, not the other way
 around. This means that extra files in the destination that are not in
@@ -67,7 +66,7 @@ you what happened to it. These are reminiscent of diff files.
 - |+ path| means path was missing on the destination, so only in the source
 - |* path| means path was present in source and destination but different.
 - |! path| means there was an error reading or hashing the source or dest.
-`, "|", "`")
+`, "|", "`", -1)
 
 // GetCheckOpt gets the options corresponding to the check flags
 func GetCheckOpt(fsrc, fdst fs.Fs) (opt *operations.CheckOpt, close func(), err error) {
@@ -131,19 +130,19 @@ func GetCheckOpt(fsrc, fdst fs.Fs) (opt *operations.CheckOpt, close func(), err
 var commandDefinition = &cobra.Command{
 	Use:   "check source:path dest:path",
 	Short: `Checks the files in the source and destination match.`,
-	Long: strings.ReplaceAll(`
+	Long: `
 Checks the files in the source and destination match.  It compares
 sizes and hashes (MD5 or SHA1) and logs a report of files which don't
 match.  It doesn't alter the source or destination.
 
-If you supply the |--size-only| flag, it will only compare the sizes not
+If you supply the --size-only flag, it will only compare the sizes not
 the hashes as well.  Use this for a quick check.
 
-If you supply the |--download| flag, it will download the data from
+If you supply the --download flag, it will download the data from
 both remotes and check them against each other on the fly.  This can
 be useful for remotes that don't support hashes or if you really want
 to check all the data.
-`, "|", "`") + FlagsHelp,
+` + FlagsHelp,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(2, 2, command, args)
 		fsrc, fdst := cmd.NewFsSrcDst(args)

cmd/cmd.go

@@ -382,11 +382,6 @@ func initConfig() {
 	// Finish parsing any command line flags
 	configflags.SetFlags(ci)
 
-	// Hide console window
-	if ci.NoConsole {
-		terminal.HideConsole()
-	}
-
 	// Load filters
 	err := filterflags.Reload(ctx)
 	if err != nil {

cmd/cmount/fs.go

@@ -9,7 +9,6 @@ import (
 	"os"
 	"path"
 	"sync"
-	"sync/atomic"
 	"time"
 
 	"github.com/billziss-gh/cgofuse/fuse"
@@ -24,12 +23,11 @@ const fhUnset = ^uint64(0)
 
 // FS represents the top level filing system
 type FS struct {
-	VFS       *vfs.VFS
-	f         fs.Fs
-	ready     chan (struct{})
-	mu        sync.Mutex // to protect the below
-	handles   []vfs.Handle
-	destroyed int32 // read/write with sync/atomic
+	VFS     *vfs.VFS
+	f       fs.Fs
+	ready   chan (struct{})
+	mu      sync.Mutex // to protect the below
+	handles []vfs.Handle
 }
 
 // NewFS makes a new FS
@@ -189,7 +187,6 @@ func (fsys *FS) Init() {
 // Destroy call).
 func (fsys *FS) Destroy() {
 	defer log.Trace(fsys.f, "")("")
-	atomic.StoreInt32(&fsys.destroyed, 1)
 }
 
 // Getattr reads the attributes for path

cmd/cmount/mount.go

@@ -12,8 +12,6 @@ import (
 	"fmt"
 	"os"
 	"runtime"
-	"strings"
-	"sync/atomic"
 	"time"
 
 	"github.com/billziss-gh/cgofuse/fuse"
@@ -37,19 +35,6 @@ func init() {
 	mountlib.AddRc("cmount", mount)
 }
 
-// Find the option string in the current options
-func findOption(name string, options []string) (found bool) {
-	for _, option := range options {
-		if option == "-o" {
-			continue
-		}
-		if strings.Contains(option, name) {
-			return true
-		}
-	}
-	return false
-}
-
 // mountOptions configures the options from the command line flags
 func mountOptions(VFS *vfs.VFS, device string, mountpoint string, opt *mountlib.Options) (options []string) {
 	// Options
@@ -119,13 +104,6 @@ func mountOptions(VFS *vfs.VFS, device string, mountpoint string, opt *mountlib.
 	for _, option := range opt.ExtraFlags {
 		options = append(options, option)
 	}
-	if runtime.GOOS == "darwin" {
-		if !findOption("modules=iconv", options) {
-			iconv := "modules=iconv,from_code=UTF-8,to_code=UTF-8-MAC"
-			options = append(options, "-o", iconv)
-			fs.Debugf(nil, "Adding \"-o %s\" for macOS", iconv)
-		}
-	}
 	return options
 }
 
@@ -190,10 +168,7 @@ func mount(VFS *vfs.VFS, mountPath string, opt *mountlib.Options) (<-chan error,
 		// Shutdown the VFS
 		fsys.VFS.Shutdown()
 		var umountOK bool
-		if atomic.LoadInt32(&fsys.destroyed) != 0 {
-			fs.Debugf(nil, "Not calling host.Unmount as mount already Destroyed")
-			umountOK = true
-		} else if atexit.Signalled() {
+		if atexit.Signalled() {
 			// If we have received a signal then FUSE will be shutting down already
 			fs.Debugf(nil, "Not calling host.Unmount as signal received")
 			umountOK = true

cmd/cmount/mountpoint_windows.go

@@ -103,9 +103,8 @@ func handleLocalMountpath(mountpath string, opt *mountlib.Options) (string, erro
 	} else if !os.IsNotExist(err) {
 		return "", errors.Wrap(err, "failed to retrieve mountpoint path information")
 	}
-	if isDriveRootPath(mountpath) { // Assume intention with "X:\" was "X:"
-		mountpath = mountpath[:len(mountpath)-1] // WinFsp needs drive mountpoints without trailing path separator
-	}
+	//if isDriveRootPath(mountpath) { // Assume intention with "X:\" was "X:"
+	//	mountpoint = mountpath[:len(mountpath)-1] // WinFsp needs drive mountpoints without trailing path separator
 	if !isDrive(mountpath) {
 		// Assuming directory path, since it is not a pure drive letter string such as "X:".
 		// Drive letter string can be used as is, since we have already checked it does not exist,
@@ -114,12 +113,14 @@ func handleLocalMountpath(mountpath string, opt *mountlib.Options) (string, erro
 			fs.Errorf(nil, "Ignoring --network-mode as it is not supported with directory mountpoint")
 			opt.NetworkMode = false
 		}
-		var err error
-		if mountpath, err = filepath.Abs(mountpath); err != nil { // Ensures parent is found but also more informative log messages
-			return "", errors.Wrap(err, "mountpoint path is not valid: "+mountpath)
-		}
 		parent := filepath.Join(mountpath, "..")
-		if _, err = os.Stat(parent); err != nil {
+		if parent == "" || parent == "." {
+			return "", errors.New("mountpoint directory is not valid: " + parent)
+		}
+		if os.IsPathSeparator(parent[len(parent)-1]) { // Ends in a separator only if it is the root directory
+			return "", errors.New("mountpoint directory is at root: " + parent)
+		}
+		if _, err := os.Stat(parent); err != nil {
 			if os.IsNotExist(err) {
 				return "", errors.New("parent of mountpoint directory does not exist: " + parent)
 			}

cmd/copy/copy.go

@@ -2,7 +2,6 @@ package copy
 
 import (
 	"context"
-	"strings"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/rclone/rclone/fs/config/flags"
@@ -24,8 +23,7 @@ func init() {
 var commandDefinition = &cobra.Command{
 	Use:   "copy source:path dest:path",
 	Short: `Copy files from source to dest, skipping already copied.`,
-	// Note: "|" will be replaced by backticks below
-	Long: strings.ReplaceAll(`
+	Long: `
 Copy the source to the destination.  Doesn't transfer
 unchanged files, testing by size and modification time or
 MD5SUM.  Doesn't delete files from the destination.
@@ -57,8 +55,8 @@ Not to
     destpath/sourcepath/one.txt
     destpath/sourcepath/two.txt
 
-If you are familiar with |rsync|, rclone always works as if you had
-written a trailing |/| - meaning "copy the contents of this directory".
+If you are familiar with ` + "`rsync`" + `, rclone always works as if you had
+written a trailing / - meaning "copy the contents of this directory".
 This applies to all commands and whether you are talking about the
 source or destination.
 
@@ -73,10 +71,10 @@ recently very efficiently like this:
 
     rclone copy --max-age 24h --no-traverse /path/to/src remote:
 
-**Note**: Use the |-P|/|--progress| flag to view real-time transfer statistics.
+**Note**: Use the ` + "`-P`" + `/` + "`--progress`" + ` flag to view real-time transfer statistics.
 
-**Note**: Use the |--dry-run| or the |--interactive|/|-i| flag to test without copying anything.
-`, "|", "`"),
+**Note**: Use the ` + "`--dry-run` or the `--interactive`/`-i`" + ` flag to test without copying anything.
+`,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(2, 2, command, args)
 		fsrc, srcFileName, fdst := cmd.NewFsSrcFileDst(args)

cmd/cryptdecode/cryptdecode.go

@@ -36,9 +36,6 @@ use it like this
 	rclone cryptdecode encryptedremote: encryptedfilename1 encryptedfilename2
 
 	rclone cryptdecode --reverse encryptedremote: filename1 filename2
-
-Another way to accomplish this is by using the ` + "`rclone backend encode` (or `decode`)" + `command.
-See the documentation on the ` + "`crypt`" + ` overlay for more info.
 `,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(2, 11, command, args)

cmd/dedupe/dedupe.go

@@ -5,7 +5,6 @@ import (
 	"log"
 
 	"github.com/rclone/rclone/cmd"
-	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config/flags"
 	"github.com/rclone/rclone/fs/operations"
 	"github.com/spf13/cobra"
@@ -144,9 +143,6 @@ Or
 			args = args[1:]
 		}
 		fdst := cmd.NewFsSrc(args)
-		if !byHash && !fdst.Features().DuplicateFiles {
-			fs.Logf(fdst, "Can't have duplicate names here. Perhaps you wanted --by-hash ? Continuing anyway.")
-		}
 		cmd.Run(false, false, command, func() error {
 			return operations.Deduplicate(context.Background(), fdst, dedupeMode, byHash)
 		})

cmd/delete/delete.go

@@ -2,7 +2,6 @@ package delete
 
 import (
 	"context"
-	"strings"
 
 	"github.com/rclone/rclone/cmd"
 	"github.com/rclone/rclone/fs/config/flags"
@@ -23,17 +22,16 @@ func init() {
 var commandDefinition = &cobra.Command{
 	Use:   "delete remote:path",
 	Short: `Remove the files in path.`,
-	// Warning! "|" will be replaced by backticks below
-	Long: strings.ReplaceAll(`
-Remove the files in path.  Unlike |purge| it obeys include/exclude
+	Long: `
+Remove the files in path.  Unlike ` + "`" + `purge` + "`" + ` it obeys include/exclude
 filters so can be used to selectively delete files.
 
-|rclone delete| only deletes files but leaves the directory structure
+` + "`rclone delete`" + ` only deletes files but leaves the directory structure
 alone. If you want to delete a directory and all of its contents use
-the |purge| command.
+the ` + "`purge`" + ` command.
 
-If you supply the |--rmdirs| flag, it will remove all empty directories along with it.
-You can also use the separate command |rmdir| or |rmdirs| to
+If you supply the --rmdirs flag, it will remove all empty directories along with it.
+You can also use the separate command ` + "`rmdir`" + ` or ` + "`rmdirs`" + ` to
 delete empty directories only.
 
 For example, to delete all files bigger than 100MBytes, you may first want to check what
@@ -50,8 +48,8 @@ That reads "delete everything with a minimum size of 100 MB", hence
 delete all files bigger than 100MBytes.
 
 **Important**: Since this can cause data loss, test first with the
-|--dry-run| or the |--interactive|/|-i| flag.
-`, "|", "`"),
+` + "`--dry-run` or the `--interactive`/`-i`" + ` flag.
+`,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(1, 1, command, args)
 		fsrc := cmd.NewFsSrc(args)

cmd/info/info.go

@@ -1,7 +1,7 @@
 package info
 
 // FIXME once translations are implemented will need a no-escape
-// option for Put so we can make these tests work again
+// option for Put so we can make these tests work agaig
 
 import (
 	"bytes"
@@ -9,7 +9,6 @@ import (
 	"encoding/json"
 	"fmt"
 	"io"
-	"log"
 	"os"
 	"path"
 	"regexp"
@@ -21,8 +20,7 @@ import (
 
 	"github.com/pkg/errors"
 	"github.com/rclone/rclone/cmd"
-	"github.com/rclone/rclone/cmd/test"
-	"github.com/rclone/rclone/cmd/test/info/internal"
+	"github.com/rclone/rclone/cmd/info/internal"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/config/flags"
 	"github.com/rclone/rclone/fs/hash"
@@ -37,7 +35,6 @@ var (
 	checkControl       bool
 	checkLength        bool
 	checkStreaming     bool
-	all                bool
 	uploadWait         time.Duration
 	positionLeftRe     = regexp.MustCompile(`(?s)^(.*)-position-left-([[:xdigit:]]+)$`)
 	positionMiddleRe   = regexp.MustCompile(`(?s)^position-middle-([[:xdigit:]]+)-(.*)-$`)
@@ -45,15 +42,14 @@ var (
 )
 
 func init() {
-	test.Command.AddCommand(commandDefinition)
+	cmd.Root.AddCommand(commandDefinition)
 	cmdFlags := commandDefinition.Flags()
 	flags.StringVarP(cmdFlags, &writeJSON, "write-json", "", "", "Write results to file.")
-	flags.BoolVarP(cmdFlags, &checkNormalization, "check-normalization", "", false, "Check UTF-8 Normalization.")
-	flags.BoolVarP(cmdFlags, &checkControl, "check-control", "", false, "Check control characters.")
+	flags.BoolVarP(cmdFlags, &checkNormalization, "check-normalization", "", true, "Check UTF-8 Normalization.")
+	flags.BoolVarP(cmdFlags, &checkControl, "check-control", "", true, "Check control characters.")
 	flags.DurationVarP(cmdFlags, &uploadWait, "upload-wait", "", 0, "Wait after writing a file.")
-	flags.BoolVarP(cmdFlags, &checkLength, "check-length", "", false, "Check max filename length.")
-	flags.BoolVarP(cmdFlags, &checkStreaming, "check-streaming", "", false, "Check uploads with indeterminate file size.")
-	flags.BoolVarP(cmdFlags, &all, "all", "", false, "Run all tests.")
+	flags.BoolVarP(cmdFlags, &checkLength, "check-length", "", true, "Check max filename length.")
+	flags.BoolVarP(cmdFlags, &checkStreaming, "check-streaming", "", true, "Check uploads with indeterminate file size.")
 }
 
 var commandDefinition = &cobra.Command{
@@ -63,20 +59,10 @@ var commandDefinition = &cobra.Command{
 to write to the paths passed in and how long they can be.  It can take some
 time.  It will write test files into the remote:path passed in.  It outputs
 a bit of go code for each one.
-
-**NB** this can create undeletable files and other hazards - use with care
 `,
+	Hidden: true,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(1, 1e6, command, args)
-		if !checkNormalization && !checkControl && !checkLength && !checkStreaming && !all {
-			log.Fatalf("no tests selected - select a test or use -all")
-		}
-		if all {
-			checkNormalization = true
-			checkControl = true
-			checkLength = true
-			checkStreaming = true
-		}
 		for i := range args {
 			f := cmd.NewFsDir(args[i : i+1])
 			cmd.Run(false, false, command, func() error {

cmd/info/internal/build_csv/main.go

@@ -11,7 +11,7 @@ import (
 	"sort"
 	"strconv"
 
-	"github.com/rclone/rclone/cmd/test/info/internal"
+	"github.com/rclone/rclone/cmd/info/internal"
 )
 
 func main() {

cmd/ls/lshelp/lshelp.go

@@ -1,31 +1,26 @@
 package lshelp
 
-import (
-	"strings"
-)
-
 // Help describes the common help for all the list commands
-// Warning! "|" will be replaced by backticks below
-var Help = strings.ReplaceAll(`
+var Help = `
 Any of the filtering options can be applied to this command.
 
 There are several related list commands
 
-  * |ls| to list size and path of objects only
-  * |lsl| to list modification time, size and path of objects only
-  * |lsd| to list directories only
-  * |lsf| to list objects and directories in easy to parse format
-  * |lsjson| to list objects and directories in JSON format
+  * ` + "`ls`" + ` to list size and path of objects only
+  * ` + "`lsl`" + ` to list modification time, size and path of objects only
+  * ` + "`lsd`" + ` to list directories only
+  * ` + "`lsf`" + ` to list objects and directories in easy to parse format
+  * ` + "`lsjson`" + ` to list objects and directories in JSON format
 
-|ls|,|lsl|,|lsd| are designed to be human readable.
-|lsf| is designed to be human and machine readable.
-|lsjson| is designed to be machine readable.
+` + "`ls`,`lsl`,`lsd`" + ` are designed to be human readable.
+` + "`lsf`" + ` is designed to be human and machine readable.
+` + "`lsjson`" + ` is designed to be machine readable.
 
-Note that |ls| and |lsl| recurse by default - use |--max-depth 1| to stop the recursion.
+Note that ` + "`ls` and `lsl`" + ` recurse by default - use "--max-depth 1" to stop the recursion.
 
-The other list commands |lsd|,|lsf|,|lsjson| do not recurse by default - use |-R| to make them recurse.
+The other list commands ` + "`lsd`,`lsf`,`lsjson`" + ` do not recurse by default - use "-R" to make them recurse.
 
 Listing a non existent directory will produce an error except for
 remotes which can't have empty directories (e.g. s3, swift, or gcs -
 the bucket based remotes).
-`, "|", "`")
+`

cmd/lsjson/lsjson.go

@@ -121,11 +121,14 @@ can be processed line by line as each item is written one to a line.
 				}
 				return nil
 			})
+			if err != nil {
+				return err
+			}
 			if !first {
 				fmt.Println()
 			}
 			fmt.Println("]")
-			return err
+			return nil
 		})
 	},
 }

cmd/memtest/memtest.go

@@ -1,4 +1,4 @@
-package memory
+package memtest
 
 import (
 	"context"
@@ -6,19 +6,19 @@ import (
 	"sync"
 
 	"github.com/rclone/rclone/cmd"
-	"github.com/rclone/rclone/cmd/test"
 	"github.com/rclone/rclone/fs"
 	"github.com/rclone/rclone/fs/operations"
 	"github.com/spf13/cobra"
 )
 
 func init() {
-	test.Command.AddCommand(commandDefinition)
+	cmd.Root.AddCommand(commandDefinition)
 }
 
 var commandDefinition = &cobra.Command{
-	Use:   "memory remote:path",
-	Short: `Load all the objects at remote:path into memory and report memory stats.`,
+	Use:    "memtest remote:path",
+	Short:  `Load all the objects at remote:path and report memory stats.`,
+	Hidden: true,
 	Run: func(command *cobra.Command, args []string) {
 		cmd.CheckArgs(1, 1, command, args)
 		fsrc := cmd.NewFsSrc(args)

cmd/mount/dir.go

@@ -1,4 +1,4 @@
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/file.go

@@ -1,4 +1,4 @@
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/fs.go

@@ -1,6 +1,6 @@
 // FUSE main Fs
 
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/handle.go

@@ -1,4 +1,4 @@
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/mount.go

@@ -1,6 +1,6 @@
 // Package mount implements a FUSE mounting system for rclone remotes.
 
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/mount_test.go

@@ -1,4 +1,4 @@
-// +build linux freebsd
+// +build linux,go1.13 freebsd,go1.13
 
 package mount
 

cmd/mount/mount_unsupported.go

@@ -1,9 +1,13 @@
 // Build for mount for unsupported platforms to stop go complaining
 // about "no buildable Go source files "
 
-// Invert the build constraint: linux freebsd
+// Invert the build constraint: linux,go1.13 freebsd,go1.13
+//
+// !((linux&&go1.13) || (freebsd&&go1.13))
+// == !(linux&&go1.13) && !(freebsd&&go1.13))
+// == (!linux || !go1.13) && (!freebsd || !go1.13))
 
-// +build !linux
-// +build !freebsd
+// +build !linux !go1.13
+// +build !freebsd !go1.13
 
 package mount

cmd/mountlib/mount.go

@@ -71,7 +71,7 @@ const (
 func init() {
 	// DaemonTimeout defaults to non zero for macOS
 	if runtime.GOOS == "darwin" {
-		DefaultOpt.DaemonTimeout = 10 * time.Minute
+		DefaultOpt.DaemonTimeout = 15 * time.Minute
 	}
 }
 
@@ -159,36 +159,34 @@ func NewMountCommand(commandName string, hidden bool, mount MountFn) *cobra.Comm
 		Use:    commandName + " remote:path /path/to/mountpoint",
 		Hidden: hidden,
 		Short:  `Mount the remote as file system on a mountpoint.`,
-		// Warning! "|" will be replaced by backticks below
-		//          "@" will be replaced by the command name
-		Long: strings.ReplaceAll(strings.ReplaceAll(`
-rclone @ allows Linux, FreeBSD, macOS and Windows to
+		Long: `
+rclone ` + commandName + ` allows Linux, FreeBSD, macOS and Windows to
 mount any of Rclone's cloud storage systems as a file system with
 FUSE.
 
-First set up your remote using |rclone config|.  Check it works with |rclone ls| etc.
+First set up your remote using ` + "`rclone config`" + `.  Check it works with ` + "`rclone ls`" + ` etc.
 
 On Linux and OSX, you can either run mount in foreground mode or background (daemon) mode.
-Mount runs in foreground mode by default, use the |--daemon| flag to specify background mode.
+Mount runs in foreground mode by default, use the ` + "`--daemon`" + ` flag to specify background mode.
 You can only run mount in foreground mode on Windows.
 
-On Linux/macOS/FreeBSD start the mount like this, where |/path/to/local/mount|
+On Linux/macOS/FreeBSD start the mount like this, where ` + "`/path/to/local/mount`" + `
 is an **empty** **existing** directory:
 
-    rclone @ remote:path/to/files /path/to/local/mount
+    rclone ` + commandName + ` remote:path/to/files /path/to/local/mount
 
 On Windows you can start a mount in different ways. See [below](#mounting-modes-on-windows)
 for details. The following examples will mount to an automatically assigned drive,
-to specific drive letter |X:|, to path |C:\path\parent\mount|
-(where parent directory or drive must exist, and mount must **not** exist,
+to specific drive letter ` + "`X:`" + `, to path ` + "`C:\\path\\to\\nonexistent\\directory`" + `
+(which must be **non-existent** subdirectory of an **existing** parent directory or drive,
 and is not supported when [mounting as a network drive](#mounting-modes-on-windows)), and
-the last example will mount as network share |\\cloud\remote| and map it to an
+the last example will mount as network share ` + "`\\cloud\remote`" + ` and map it to an
 automatically assigned drive:
 
-    rclone @ remote:path/to/files *
-    rclone @ remote:path/to/files X:
-    rclone @ remote:path/to/files C:\path\parent\mount
-    rclone @ remote:path/to/files \\cloud\remote
+    rclone ` + commandName + ` remote:path/to/files *
+    rclone ` + commandName + ` remote:path/to/files X:
+    rclone ` + commandName + ` remote:path/to/files C:\path\to\nonexistent\directory
+    rclone ` + commandName + ` remote:path/to/files \\cloud\remote
 
 When the program ends while in foreground mode, either via Ctrl+C or receiving
 a SIGINT or SIGTERM signal, the mount should be automatically stopped.
@@ -210,12 +208,12 @@ then an additional 1PB of free space is assumed. If the remote does not
 [support](https://rclone.org/overview/#optional-features) the about feature
 at all, then 1PB is set as both the total and the free size.
 
-**Note**: As of |rclone| 1.52.2, |rclone mount| now requires Go version 1.13
+**Note**: As of ` + "`rclone` 1.52.2, `rclone mount`" + ` now requires Go version 1.13
 or newer on some platforms depending on the underlying FUSE library in use.
 
 ### Installing on Windows
 
-To run rclone @ on Windows, you will need to
+To run rclone ` + commandName + ` on Windows, you will need to
 download and install [WinFsp](http://www.secfs.net/winfsp/).
 
 [WinFsp](https://github.com/billziss-gh/winfsp) is an open source
@@ -223,7 +221,7 @@ Windows File System Proxy which makes it easy to write user space file
 systems for Windows.  It provides a FUSE emulation layer which rclone
 uses combination with [cgofuse](https://github.com/billziss-gh/cgofuse).
 Both of these packages are by Bill Zissimopoulos who was very helpful
-during the implementation of rclone @ for Windows.
+during the implementation of rclone ` + commandName + ` for Windows.
 
 #### Mounting modes on windows
 
@@ -241,55 +239,55 @@ and experience unexpected program errors, freezes or other issues, consider moun
 as a network drive instead.
 
 When mounting as a fixed disk drive you can either mount to an unused drive letter,
-or to a path representing a **non-existent** subdirectory of an **existing** parent
-directory or drive. Using the special value |*| will tell rclone to
+or to a path - which must be **non-existent** subdirectory of an **existing** parent
+directory or drive. Using the special value ` + "`*`" + ` will tell rclone to
 automatically assign the next available drive letter, starting with Z: and moving backward.
 Examples:
 
-    rclone @ remote:path/to/files *
-    rclone @ remote:path/to/files X:
-    rclone @ remote:path/to/files C:\path\parent\mount
-    rclone @ remote:path/to/files X:
+    rclone ` + commandName + ` remote:path/to/files *
+    rclone ` + commandName + ` remote:path/to/files X:
+    rclone ` + commandName + ` remote:path/to/files C:\path\to\nonexistent\directory
+    rclone ` + commandName + ` remote:path/to/files X:
 
-Option |--volname| can be used to set a custom volume name for the mounted
+Option ` + "`--volname`" + ` can be used to set a custom volume name for the mounted
 file system. The default is to use the remote name and path.
 
-To mount as network drive, you can add option |--network-mode|
-to your @ command. Mounting to a directory path is not supported in
+To mount as network drive, you can add option ` + "`--network-mode`" + `
+to your ` + commandName + ` command. Mounting to a directory path is not supported in
 this mode, it is a limitation Windows imposes on junctions, so the remote must always
 be mounted to a drive letter.
 
-    rclone @ remote:path/to/files X: --network-mode
+    rclone ` + commandName + ` remote:path/to/files X: --network-mode
 
-A volume name specified with |--volname| will be used to create the network share path.
-A complete UNC path, such as |\\cloud\remote|, optionally with path
-|\\cloud\remote\madeup\path|, will be used as is. Any other
-string will be used as the share part, after a default prefix |\\server\|.
-If no volume name is specified then |\\server\share| will be used.
+A volume name specified with ` + "`--volname`" + ` will be used to create the network share path.
+A complete UNC path, such as ` + "`\\\\cloud\\remote`" + `, optionally with path
+` + "`\\\\cloud\\remote\\madeup\\path`" + `, will be used as is. Any other
+string will be used as the share part, after a default prefix ` + "`\\\\server\\`" + `.
+If no volume name is specified then ` + "`\\\\server\\share`" + ` will be used.
 You must make sure the volume name is unique when you are mounting more than one drive,
 or else the mount command will fail. The share name will treated as the volume label for
 the mapped drive, shown in Windows Explorer etc, while the complete
-|\\server\share| will be reported as the remote UNC path by
-|net use| etc, just like a normal network drive mapping.
+` + "`\\\\server\\share`" + ` will be reported as the remote UNC path by
+` + "`net use`" + ` etc, just like a normal network drive mapping.
 
-If you specify a full network share UNC path with |--volname|, this will implicitely
-set the |--network-mode| option, so the following two examples have same result:
+If you specify a full network share UNC path with ` + "`--volname`" + `, this will implicitely
+set the ` + "`--network-mode`" + ` option, so the following two examples have same result:
 
-    rclone @ remote:path/to/files X: --network-mode
-    rclone @ remote:path/to/files X: --volname \\server\share
+    rclone ` + commandName + ` remote:path/to/files X: --network-mode
+    rclone ` + commandName + ` remote:path/to/files X: --volname \\server\share
 
 You may also specify the network share UNC path as the mountpoint itself. Then rclone
-will automatically assign a drive letter, same as with |*| and use that as
+will automatically assign a drive letter, same as with ` + "`*`" + ` and use that as
 mountpoint, and instead use the UNC path specified as the volume name, as if it were
-specified with the |--volname| option. This will also implicitely set
-the |--network-mode| option. This means the following two examples have same result:
+specified with the ` + "`--volname`" + ` option. This will also implicitely set
+the ` + "`--network-mode`" + ` option. This means the following two examples have same result:
 
-    rclone @ remote:path/to/files \\cloud\remote
-    rclone @ remote:path/to/files * --volname \\cloud\remote
+    rclone ` + commandName + ` remote:path/to/files \\cloud\remote
+    rclone ` + commandName + ` remote:path/to/files * --volname \\cloud\remote
 
 There is yet another way to enable network mode, and to set the share path,
 and that is to pass the "native" libfuse/WinFsp option directly:
-|--fuse-flag --VolumePrefix=\server\share|. Note that the path
+` + "`--fuse-flag --VolumePrefix=\\server\\share`" + `. Note that the path
 must be with just a single backslash prefix in this case.
 
 
@@ -310,35 +308,21 @@ representing permissions for the POSIX permission scopes: Owner, group and other
 By default, the owner and group will be taken from the current user, and the built-in
 group "Everyone" will be used to represent others. The user/group can be customized
 with FUSE options "UserName" and "GroupName",
-e.g. |-o UserName=user123 -o GroupName="Authenticated Users"|.
+e.g. ` + "`-o UserName=user123 -o GroupName=\"Authenticated Users\"`" + `.
 
 The permissions on each entry will be set according to
-[options](#options) |--dir-perms| and |--file-perms|,
+[options](#options) ` + "`--dir-perms`" + ` and ` + "`--file-perms`" + `,
 which takes a value in traditional [numeric notation](https://en.wikipedia.org/wiki/File-system_permissions#Numeric_notation),
-where the default corresponds to |--file-perms 0666 --dir-perms 0777|.
+where the default corresponds to ` + "`--file-perms 0666 --dir-perms 0777`" + `.
 
 Note that the mapping of permissions is not always trivial, and the result
 you see in Windows Explorer may not be exactly like you expected.
 For example, when setting a value that includes write access, this will be
 mapped to individual permissions "write attributes", "write data" and "append data",
-but not "write extended attributes". Windows will then show this as basic
-permission "Special" instead of "Write", because "Write" includes the
-"write extended attributes" permission.
-
-If you set POSIX permissions for only allowing access to the owner, using
-|--file-perms 0600 --dir-perms 0700|, the user group and the built-in "Everyone"
-group will still be given some special permissions, such as "read attributes"
-and "read permissions", in Windows. This is done for compatibility reasons,
-e.g. to allow users without additional permissions to be able to read basic
-metadata about files like in UNIX. One case that may arise is that other programs
-(incorrectly) interprets this as the file being accessible by everyone. For example
-an SSH client may warn about "unprotected private key file".
-
-WinFsp 2021 (version 1.9, still in beta) introduces a new FUSE option "FileSecurity",
-that allows the complete specification of file security descriptors using
-[SDDL](https://docs.microsoft.com/en-us/windows/win32/secauthz/security-descriptor-string-format).
-With this you can work around issues such as the mentioned "unprotected private key file"
-by specifying |-o FileSecurity="D:P(A;;FA;;;OW)"|, for file all access (FA) to the owner (OW).
+but not "write extended attributes" (WinFsp does not support extended attributes,
+see [this](https://github.com/billziss-gh/winfsp/wiki/NTFS-Compatibility)).
+Windows will then show this as basic permission "Special" instead of "Write",
+because "Write" includes the "write extended attributes" permission.
 
 #### Windows caveats
 
@@ -358,11 +342,11 @@ alternatively using [the nssm service manager](https://nssm.cc/usage).
 
 ### Limitations
 
-Without the use of |--vfs-cache-mode| this can only write files
+Without the use of ` + "`--vfs-cache-mode`" + ` this can only write files
 sequentially, it can only seek when reading.  This means that many
 applications won't work with their files on an rclone mount without
-|--vfs-cache-mode writes| or |--vfs-cache-mode full|.
-See the [VFS File Caching](#vfs-file-caching) section for more info.
+` + "`--vfs-cache-mode writes`" + ` or ` + "`--vfs-cache-mode full`" + `.
+See the [File Caching](#file-caching) section for more info.
 
 The bucket based remotes (e.g. Swift, S3, Google Compute Storage, B2,
 Hubic) do not support the concept of empty directories, so empty
@@ -371,21 +355,21 @@ the directory cache.
 
 Only supported on Linux, FreeBSD, OS X and Windows at the moment.
 
-### rclone @ vs rclone sync/copy
+### rclone ` + commandName + ` vs rclone sync/copy
 
 File systems expect things to be 100% reliable, whereas cloud storage
 systems are a long way from 100% reliable. The rclone sync/copy
-commands cope with this with lots of retries.  However rclone @
+commands cope with this with lots of retries.  However rclone ` + commandName + `
 can't use retries in the same way without making local copies of the
-uploads. Look at the [VFS File Caching](#vfs-file-caching)
-for solutions to make @ more reliable.
+uploads. Look at the [file caching](#file-caching)
+for solutions to make ` + commandName + ` more reliable.
 
 ### Attribute caching
 
-You can use the flag |--attr-timeout| to set the time the kernel caches
+You can use the flag ` + "`--attr-timeout`" + ` to set the time the kernel caches
 the attributes (size, modification time, etc.) for directory entries.
 
-The default is |1s| which caches files just long enough to avoid
+The default is "1s" which caches files just long enough to avoid
 too many callbacks to rclone from the kernel.
 
 In theory 0s should be the correct value for filesystems which can
@@ -396,14 +380,14 @@ few problems such as
 and [excessive time listing directories](https://github.com/rclone/rclone/issues/2095#issuecomment-371141147).
 
 The kernel can cache the info about a file for the time given by
-|--attr-timeout|. You may see corruption if the remote file changes
+` + "`--attr-timeout`" + `. You may see corruption if the remote file changes
 length during this window.  It will show up as either a truncated file
-or a file with garbage on the end.  With |--attr-timeout 1s| this is
-very unlikely but not impossible.  The higher you set |--attr-timeout|
+or a file with garbage on the end.  With ` + "`--attr-timeout 1s`" + ` this is
+very unlikely but not impossible.  The higher you set ` + "`--attr-timeout`" + `
 the more likely it is.  The default setting of "1s" is the lowest
 setting which mitigates the problems above.
 
-If you set it higher (|10s| or |1m| say) then the kernel will call
+If you set it higher ('10s' or '1m' say) then the kernel will call
 back to rclone less often making it more efficient, however there is
 more chance of the corruption issue above.
 
@@ -419,28 +403,28 @@ files to be visible in the mount.
 
 ### systemd
 
-When running rclone @ as a systemd service, it is possible
+When running rclone ` + commandName + ` as a systemd service, it is possible
 to use Type=notify. In this case the service will enter the started state
 after the mountpoint has been successfully set up.
-Units having the rclone @ service specified as a requirement
+Units having the rclone ` + commandName + ` service specified as a requirement
 will see all files and folders immediately in this mode.
 
 ### chunked reading
 
-|--vfs-read-chunk-size| will enable reading the source objects in parts.
+` + "`--vfs-read-chunk-size`" + ` will enable reading the source objects in parts.
 This can reduce the used download quota for some remotes by requesting only chunks
 from the remote that are actually read at the cost of an increased number of requests.
 
-When |--vfs-read-chunk-size-limit| is also specified and greater than
-|--vfs-read-chunk-size|, the chunk size for each open file will get doubled
-for each chunk read, until the specified value is reached. A value of |-1| will disable
+When ` + "`--vfs-read-chunk-size-limit`" + ` is also specified and greater than
+` + "`--vfs-read-chunk-size`" + `, the chunk size for each open file will get doubled
+for each chunk read, until the specified value is reached. A value of -1 will disable
 the limit and the chunk size will grow indefinitely.
 
-With |--vfs-read-chunk-size 100M| and |--vfs-read-chunk-size-limit 0|
+With ` + "`--vfs-read-chunk-size 100M`" + ` and ` + "`--vfs-read-chunk-size-limit 0`" + `
 the following parts will be downloaded: 0-100M, 100M-200M, 200M-300M, 300M-400M and so on.
-When |--vfs-read-chunk-size-limit 500M| is specified, the result would be
+When ` + "`--vfs-read-chunk-size-limit 500M`" + ` is specified, the result would be
 0-100M, 100M-300M, 300M-700M, 700M-1200M, 1200M-1700M and so on.
-`, "|", "`"), "@", commandName) + vfs.Help,
+` + vfs.Help,
 		Run: func(command *cobra.Command, args []string) {
 			cmd.CheckArgs(2, 2, command, args)
 			opt := Opt // make a copy of the options