Version v1.09

This was done by stopping the user configuring single letter remotes
and making sure we don't treat single letter remotes as a remote name,
but as a drive letter.

README.md

@@ -254,16 +254,24 @@ COPYING file included in this package).
 Bugs
 ----
 
-  * Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
-    * quota is 100.0 requests/second/user
   * Empty directories left behind with Local and Drive
     * eg purging a local directory with subdirectories doesn't work
 
 Changelog
 ---------
+  * v1.09 - 2015-02-07
+    * windows: Stop drive letters (eg C:) getting mixed up with remotes (eg drive:)
+    * local: Fix directory separators on Windows
+    * drive: fix rate limit exceeded errors
+  * v1.08 - 2015-02-04
+    * drive: fix subdirectory listing to not list entire drive
+    * drive: Fix SetModTime
+    * dropbox: adapt code to recent library changes
+  * v1.07 - 2014-12-23
+    * google cloud storage: fix memory leak
   * v1.06 - 2014-12-12
     * Fix "Couldn't find home directory" on OSX
-    * Add tenant parameter for swift
+    * swift: Add tenant parameter
     * Use new location of Google API packages
   * v1.05 - 2014-08-09
     * Improved tests and consequently lots of minor fixes

RELEASE.md

@@ -0,0 +1,19 @@
+Required software for making a release
+  * [github-release](https://github.com/aktau/github-release) for uploading packages
+  * [gox](https://github.com/mitchellh/gox) for cross compiling
+    * Run `gox -build-toolchain`
+    * This assumes you have your own source checkout
+  * pandoc for making the html and man pages
+
+Making a release
+  * go get -u -f -v ./...
+  * make test
+  * make tag
+  * edit README.md
+  * git commit fs/version.go README.md docs/content/downloads.md
+  * make retag
+  * # Set the GOPATH for a gox enabled compiler - . ~/bin/go-cross
+  * make cross
+  * make upload
+  * make upload_website
+  * git push --tags origin master

docs/content/docs.md

@@ -132,9 +132,6 @@ COPYING file included in this package).
 Bugs
 ----
 
-  * Doesn't sync individual files yet, only directories.
-  * Drive: Sometimes get: Failed to copy: Upload failed: googleapi: Error 403: Rate Limit Exceeded
-    * quota is 100.0 requests/second/user
   * Empty directories left behind with Local and Drive
     * eg purging a local directory with subdirectories doesn't work
 

docs/content/downloads.md

@@ -2,34 +2,34 @@
 title: "Rclone downloads"
 description: "Download rclone binaries for your OS."
 type: page
-date: "2014-12-12"
+date: "2015-02-07"
 ---
 
-Rclone Download v1.06
+Rclone Download v1.09
 =====================
 
   * Windows
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-windows-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-windows-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-windows-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-windows-amd64.zip)
   * OSX
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-osx-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-osx-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-osx-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-osx-amd64.zip)
   * Linux
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-linux-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-linux-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.06-linux-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-linux-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-linux-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.09-linux-arm.zip)
   * FreeBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-freebsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-freebsd-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.06-freebsd-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-freebsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-freebsd-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.09-freebsd-arm.zip)
   * NetBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-netbsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-netbsd-amd64.zip)
-    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.06-netbsd-arm.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-netbsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-netbsd-amd64.zip)
+    * [ARM - 32 Bit](http://downloads.rclone.org/rclone-v1.09-netbsd-arm.zip)
   * OpenBSD
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-openbsd-386.zip)
-    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.06-openbsd-amd64.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-openbsd-386.zip)
+    * [AMD64 - 64 Bit](http://downloads.rclone.org/rclone-v1.09-openbsd-amd64.zip)
   * Plan 9
-    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.06-plan9-386.zip)
+    * [386 - 32 Bit](http://downloads.rclone.org/rclone-v1.09-plan9-386.zip)
 
 Older downloads can be found [here](http://downloads.rclone.org/)

docs/content/swift.md

@@ -52,12 +52,15 @@ Choose a number from below, or type in your own value
  * Memset Memstore UK v2
  5) https://auth.storage.memset.com/v2.0
 auth> 1
+Tenant name - optional
+tenant>
 Remote config
 --------------------
 [remote]
 user = user_name
 key = password_or_api_key
 auth = https://auth.api.rackspacecloud.com/v1.0
+tenant =
 --------------------
 y) Yes this is OK
 e) Edit this remote

drive/drive.go

@@ -1,14 +1,6 @@
 // Drive interface
 package drive
 
-// Gets this quite often
-// Failed to set mtime: googleapi: Error 403: Rate Limit Exceeded
-
-// FIXME list containers equivalent should list directories?
-
-// FIXME list directory should list to channel for concurrency not
-// append to array
-
 // FIXME need to deal with some corner cases
 // * multiple files with the same name
 // * files can be in multiple directories
@@ -21,13 +13,13 @@ import (
 	"log"
 	"mime"
 	"net/http"
-	"os"
 	"path"
 	"strings"
 	"sync"
 	"time"
 
 	"google.golang.org/api/drive/v2"
+	"google.golang.org/api/googleapi"
 
 	"github.com/ncw/rclone/fs"
 	"github.com/ncw/rclone/googleauth"
@@ -41,6 +33,9 @@ const (
 	driveFolderType    = "application/vnd.google-apps.folder"
 	timeFormatIn       = time.RFC3339
 	timeFormatOut      = "2006-01-02T15:04:05.000000000Z07:00"
+	minSleep           = 10 * time.Millisecond
+	maxSleep           = 2 * time.Second
+	decayConstant      = 2 // bigger for slower decay, exponential
 )
 
 // Globals
@@ -84,6 +79,8 @@ type FsDrive struct {
 	findRootLock sync.Mutex     // Protect findRoot from concurrent use
 	dirCache     dirCache       // Map of directory path to directory id
 	findDirLock  sync.Mutex     // Protect findDir from concurrent use
+	pacer        chan struct{}  // To pace the operations
+	sleepTime    time.Duration  // Time to sleep for each transaction
 }
 
 // FsObjectDrive describes a drive object
@@ -150,6 +147,74 @@ func (f *FsDrive) String() string {
 	return fmt.Sprintf("Google drive root '%s'", f.root)
 }
 
+// Wait for the pace
+func (f *FsDrive) paceWait() {
+	// pacer starts with a token in and whenever we take one out
+	// XXX ms later we put another in.  We could do this with a
+	// Ticker more accurately, but then we'd have to work out how
+	// not to run it when it wasn't needed
+	<-f.pacer
+
+	// Restart the timer
+	go func(t time.Duration) {
+		// fs.Debug(f, "New sleep for %v at %v", t, time.Now())
+		time.Sleep(t)
+		f.pacer <- struct{}{}
+	}(f.sleepTime)
+}
+
+// Refresh the pace given an error that was returned.  It returns a
+// boolean as to whether the operation should be retried.
+//
+// See https://developers.google.com/drive/web/handle-errors
+// http://stackoverflow.com/questions/18529524/403-rate-limit-after-only-1-insert-per-second
+func (f *FsDrive) paceRefresh(err error) bool {
+	again := false
+	oldSleepTime := f.sleepTime
+	if err == nil {
+		f.sleepTime = (f.sleepTime<<decayConstant - f.sleepTime) >> decayConstant
+		if f.sleepTime < minSleep {
+			f.sleepTime = minSleep
+		}
+		if f.sleepTime != oldSleepTime {
+			fs.Debug(f, "Reducing sleep to %v", f.sleepTime)
+		}
+	} else {
+		fs.Debug(f, "Error recived: %v", err)
+		if gerr, ok := err.(*googleapi.Error); ok {
+			if len(gerr.Errors) > 0 {
+				reason := gerr.Errors[0].Reason
+				if reason == "rateLimitExceeded" || reason == "userRateLimitExceeded" {
+					f.sleepTime *= 2
+					if f.sleepTime > maxSleep {
+						f.sleepTime = maxSleep
+					}
+					if f.sleepTime != oldSleepTime {
+						fs.Debug(f, "Rate limited, increasing sleep to %v", f.sleepTime)
+					}
+					again = true
+				}
+			}
+		}
+	}
+
+	return again
+}
+
+// Pace the remote operations to not exceed Google's limits and retry
+// on 403 rate limit exceeded
+//
+// This calls fn, expecting it to place its error in perr
+func (f *FsDrive) pace(perr *error, fn func()) {
+	for {
+		f.paceWait()
+		fn()
+		if !f.paceRefresh(*perr) {
+			break
+		}
+	}
+}
+
 // parseParse parses a drive 'url'
 func parseDrivePath(path string) (root string, err error) {
 	root = strings.Trim(path, "/")
@@ -187,7 +252,10 @@ func (f *FsDrive) listAll(dirId string, title string, directoriesOnly bool, file
 	list := f.svc.Files.List().Q(query).MaxResults(1000)
 OUTER:
 	for {
-		files, err := list.Do()
+		var files *drive.FileList
+		f.pace(&err, func() {
+			files, err = list.Do()
+		})
 		if err != nil {
 			return false, fmt.Errorf("Couldn't list directory: %s", err)
 		}
@@ -218,10 +286,15 @@ func NewFs(name, path string) (fs.Fs, error) {
 	}
 
 	f := &FsDrive{
-		root:     root,
-		dirCache: newDirCache(),
+		root:      root,
+		dirCache:  newDirCache(),
+		pacer:     make(chan struct{}, 1),
+		sleepTime: minSleep,
 	}
 
+	// Put the first pacing token in
+	f.pacer <- struct{}{}
+
 	// Create a new authorized Drive client.
 	f.client = t.Client()
 	f.svc, err = drive.New(f.client)
@@ -230,7 +303,9 @@ func NewFs(name, path string) (fs.Fs, error) {
 	}
 
 	// Read About so we know the root path
-	f.about, err = f.svc.About.Get().Do()
+	f.pace(&err, func() {
+		f.about, err = f.svc.About.Get().Do()
+	})
 	if err != nil {
 		return nil, fmt.Errorf("Couldn't read info about Drive: %s", err)
 	}
@@ -490,13 +565,16 @@ func (f *FsDrive) _findDir(path string, create bool) (pathId string, err error)
 		if create {
 			// fmt.Println("Making", path)
 			// Define the metadata for the directory we are going to create.
-			info := &drive.File{
+			createInfo := &drive.File{
 				Title:       leaf,
 				Description: leaf,
 				MimeType:    driveFolderType,
 				Parents:     []*drive.ParentReference{{Id: pathId}},
 			}
-			info, err := f.svc.Files.Insert(info).Do()
+			var info *drive.File
+			f.pace(&err, func() {
+				info, err = f.svc.Files.Insert(createInfo).Do()
+			})
 			if err != nil {
 				return pathId, fmt.Errorf("Failed to make directory: %v", err)
 			}
@@ -560,7 +638,7 @@ func (f *FsDrive) List() fs.ObjectsChan {
 			fs.Stats.Error()
 			log.Printf("Couldn't find root: %s", err)
 		} else {
-			if *driveFullList {
+			if f.root == "" && *driveFullList {
 				err = f.listDirFull(f.rootId, "", out)
 			} else {
 				err = f.listDirRecursive(f.rootId, "", out)
@@ -603,30 +681,6 @@ func (f *FsDrive) ListDir() fs.DirChan {
 	return out
 }
 
-// seekWrapper wraps an io.Reader with a basic Seek for
-// code.google.com/p/google-api-go-client/googleapi
-// to detect the length (see getReaderSize function)
-type seekWrapper struct {
-	in   io.Reader
-	size int64
-}
-
-// Read bytes from the object - see io.Reader
-func (file *seekWrapper) Read(p []byte) (n int, err error) {
-	return file.in.Read(p)
-}
-
-// Seek - minimal implementation for Google Drive's length detection
-func (file *seekWrapper) Seek(offset int64, whence int) (int64, error) {
-	switch whence {
-	case os.SEEK_CUR:
-		return 0, nil
-	case os.SEEK_END:
-		return file.size, nil
-	}
-	return 0, nil
-}
-
 // Put the object
 //
 // This assumes that the object doesn't not already exists - if you
@@ -654,7 +708,7 @@ func (f *FsDrive) Put(in io.Reader, remote string, modTime time.Time, size int64
 	modifiedDate := modTime.Format(timeFormatOut)
 
 	// Define the metadata for the file we are going to create.
-	info := &drive.File{
+	createInfo := &drive.File{
 		Title:        leaf,
 		Description:  leaf,
 		Parents:      []*drive.ParentReference{{Id: directoryId}},
@@ -663,8 +717,14 @@ func (f *FsDrive) Put(in io.Reader, remote string, modTime time.Time, size int64
 	}
 
 	// Make the API request to upload metadata and file data.
-	in = &seekWrapper{in: in, size: size}
-	info, err = f.svc.Files.Insert(info).Media(in).Do()
+	in = &fs.SeekWrapper{In: in, Size: size}
+	var info *drive.File
+	// Don't retry, return a retry error instead
+	f.paceWait()
+	info, err = f.svc.Files.Insert(createInfo).Media(in).Do()
+	if f.paceRefresh(err) {
+		return o, fs.RetryErrorf("Upload failed - retry: %s", err)
+	}
 	if err != nil {
 		return o, fmt.Errorf("Upload failed: %s", err)
 	}
@@ -685,7 +745,10 @@ func (f *FsDrive) Rmdir() error {
 	if err != nil {
 		return err
 	}
-	children, err := f.svc.Children.List(f.rootId).MaxResults(10).Do()
+	var children *drive.ChildList
+	f.pace(&err, func() {
+		children, err = f.svc.Children.List(f.rootId).MaxResults(10).Do()
+	})
 	if err != nil {
 		return err
 	}
@@ -694,7 +757,9 @@ func (f *FsDrive) Rmdir() error {
 	}
 	// Delete the directory if it isn't the root
 	if f.root != "" {
-		err = f.svc.Files.Delete(f.rootId).Do()
+		f.pace(&err, func() {
+			err = f.svc.Files.Delete(f.rootId).Do()
+		})
 		if err != nil {
 			return err
 		}
@@ -721,7 +786,9 @@ func (f *FsDrive) Purge() error {
 	if err != nil {
 		return err
 	}
-	err = f.svc.Files.Delete(f.rootId).Do()
+	f.pace(&err, func() {
+		err = f.svc.Files.Delete(f.rootId).Do()
+	})
 	f.resetRoot()
 	if err != nil {
 		return err
@@ -826,15 +893,20 @@ func (o *FsObjectDrive) SetModTime(modTime time.Time) {
 		return
 	}
 	// New metadata
-	info := &drive.File{
+	updateInfo := &drive.File{
 		ModifiedDate: modTime.Format(timeFormatOut),
 	}
 	// Set modified date
-	_, err = o.drive.svc.Files.Update(o.id, info).SetModifiedDate(true).Do()
+	var info *drive.File
+	o.drive.pace(&err, func() {
+		info, err = o.drive.svc.Files.Update(o.id, updateInfo).SetModifiedDate(true).Do()
+	})
 	if err != nil {
 		fs.Stats.Error()
 		fs.Log(o, "Failed to update remote mtime: %s", err)
 	}
+	// Update info from read data
+	o.setMetaData(info)
 }
 
 // Is this object storable
@@ -849,7 +921,10 @@ func (o *FsObjectDrive) Open() (in io.ReadCloser, err error) {
 		return nil, err
 	}
 	req.Header.Set("User-Agent", fs.UserAgent)
-	res, err := o.drive.client.Do(req)
+	var res *http.Response
+	o.drive.pace(&err, func() {
+		res, err = o.drive.client.Do(req)
+	})
 	if err != nil {
 		return nil, err
 	}
@@ -866,14 +941,21 @@ func (o *FsObjectDrive) Open() (in io.ReadCloser, err error) {
 //
 // The new object may have been created if an error is returned
 func (o *FsObjectDrive) Update(in io.Reader, modTime time.Time, size int64) error {
-	info := &drive.File{
+	updateInfo := &drive.File{
 		Id:           o.id,
 		ModifiedDate: modTime.Format(timeFormatOut),
 	}
 
 	// Make the API request to upload metadata and file data.
-	in = &seekWrapper{in: in, size: size}
-	info, err := o.drive.svc.Files.Update(info.Id, info).SetModifiedDate(true).Media(in).Do()
+	in = &fs.SeekWrapper{In: in, Size: size}
+	var err error
+	var info *drive.File
+	// Don't retry, return a retry error instead
+	o.drive.paceWait()
+	info, err = o.drive.svc.Files.Update(updateInfo.Id, updateInfo).SetModifiedDate(true).Media(in).Do()
+	if o.drive.paceRefresh(err) {
+		return fs.RetryErrorf("Update failed - retry: %s", err)
+	}
 	if err != nil {
 		return fmt.Errorf("Update failed: %s", err)
 	}
@@ -883,7 +965,11 @@ func (o *FsObjectDrive) Update(in io.Reader, modTime time.Time, size int64) erro
 
 // Remove an object
 func (o *FsObjectDrive) Remove() error {
-	return o.drive.svc.Files.Delete(o.id).Do()
+	var err error
+	o.drive.pace(&err, func() {
+		err = o.drive.svc.Files.Delete(o.id).Do()
+	})
+	return err
 }
 
 // Check the interfaces are satisfied

dropbox/dropbox.go

@@ -290,7 +290,7 @@ func (f *FsDropbox) list(out fs.ObjectsChan) {
 			if !deltaPage.HasMore {
 				break
 			}
-			cursor = deltaPage.Cursor
+			cursor = deltaPage.Cursor.Cursor
 		}
 	}
 }

fs/config.go

@@ -330,9 +330,19 @@ func EditConfig() {
 			name := ChooseRemote()
 			EditRemote(name)
 		case 'n':
-			fmt.Printf("name> ")
-			name := ReadLine()
-			NewRemote(name)
+			for {
+				fmt.Printf("name> ")
+				name := ReadLine()
+				switch {
+				case name == "":
+					fmt.Printf("Can't use empty name\n")
+				case isDriveLetter(name):
+					fmt.Printf("Can't use %q as it can be confused a drive letter\n", name)
+				default:
+					NewRemote(name)
+					break
+				}
+			}
 		case 'd':
 			name := ChooseRemote()
 			DeleteRemote(name)

fs/driveletter.go

@@ -0,0 +1,12 @@
+// +build !windows
+
+package fs
+
+// isDriveLetter returns a bool indicating whether name is a valid
+// Windows drive letter
+//
+// On non windows platforms we don't have drive letters so we always
+// return false
+func isDriveLetter(name string) bool {
+	return false
+}

fs/driveletter_windows.go

@@ -0,0 +1,13 @@
+// +build windows
+
+package fs
+
+// isDriveLetter returns a bool indicating whether name is a valid
+// Windows drive letter
+func isDriveLetter(name string) bool {
+	if len(name) != 1 {
+		return false
+	}
+	c := name[0]
+	return (c >= 'a' && c <= 'z') || (c >= 'A' && c <= 'Z')
+}

fs/fs.go

@@ -142,6 +142,35 @@ type Purger interface {
 	Purge() error
 }
 
+// An optional interface for error as to whether the operation should be retried
+//
+// This should be returned from Update or Put methods as required
+type Retry interface {
+	error
+	Retry() bool
+}
+
+// A type of error
+type retryError string
+
+// Error interface
+func (r retryError) Error() string {
+	return string(r)
+}
+
+// Retry interface
+func (r retryError) Retry() bool {
+	return true
+}
+
+// Check interface
+var _ Retry = retryError("")
+
+// RetryErrorf makes an error which indicates it would like to be retried
+func RetryErrorf(format string, a ...interface{}) error {
+	return retryError(fmt.Sprintf(format, a...))
+}
+
 // A channel of Objects
 type ObjectsChan chan Object
 
@@ -167,9 +196,6 @@ type Dir struct {
 // A channel of Dir objects
 type DirChan chan *Dir
 
-// Pattern to match a url
-var matcher = regexp.MustCompile(`^([\w_-]+):(.*)$`)
-
 // Finds a FsInfo object for the name passed in
 //
 // Services are looked up in the config file
@@ -182,16 +208,22 @@ func Find(name string) (*FsInfo, error) {
 	return nil, fmt.Errorf("Didn't find filing system for %q", name)
 }
 
+// Pattern to match an rclone url
+var matcher = regexp.MustCompile(`^([\w_-]+):(.*)$`)
+
 // NewFs makes a new Fs object from the path
 //
 // The path is of the form remote:path
 //
 // Remotes are looked up in the config file.  If the remote isn't
 // found then NotFoundInConfigFile will be returned.
+//
+// On Windows avoid single character remote names as they can be mixed
+// up with drive letters.
 func NewFs(path string) (Fs, error) {
 	parts := matcher.FindStringSubmatch(path)
 	fsName, configName, fsPath := "local", "local", path
-	if parts != nil {
+	if parts != nil && !isDriveLetter(parts[1]) {
 		configName, fsPath = parts[1], parts[2]
 		var err error
 		fsName, err = ConfigFile.GetValue(configName, "type")

fs/operations.go

@@ -103,8 +103,7 @@ func removeFailedCopy(dst Object) {
 		Debug(dst, "Removing failed copy")
 		removeErr := dst.Remove()
 		if removeErr != nil {
-			Stats.Error()
-			Log(dst, "Failed to remove failed copy: %s", removeErr)
+			Debug(dst, "Failed to remove failed copy: %s", removeErr)
 		}
 	}
 }
@@ -115,6 +114,10 @@ func removeFailedCopy(dst Object) {
 // call Copy() with dst nil on a pre-existing file then some filing
 // systems (eg Drive) may duplicate the file.
 func Copy(f Fs, dst, src Object) {
+	const maxTries = 10
+	tries := 0
+	doUpdate := dst != nil
+tryAgain:
 	in0, err := src.Open()
 	if err != nil {
 		Stats.Error()
@@ -124,7 +127,7 @@ func Copy(f Fs, dst, src Object) {
 	in := NewAccount(in0) // account the transfer
 
 	var actionTaken string
-	if dst != nil {
+	if doUpdate {
 		actionTaken = "Copied (updated existing)"
 		err = dst.Update(in, src.ModTime(), src.Size())
 	} else {
@@ -132,6 +135,13 @@ func Copy(f Fs, dst, src Object) {
 		dst, err = f.Put(in, src.Remote(), src.ModTime(), src.Size())
 	}
 	inErr := in.Close()
+	// Retry if err returned a retry error
+	if r, ok := err.(Retry); ok && r.Retry() && tries < maxTries {
+		tries++
+		Log(src, "Received error: %v - retrying %d/%d", err, tries, maxTries)
+		removeFailedCopy(dst)
+		goto tryAgain
+	}
 	if err == nil {
 		err = inErr
 	}

fs/operations_test.go

@@ -10,6 +10,7 @@ import (
 	"log"
 	"os"
 	"path"
+	"path/filepath"
 	"regexp"
 	"strings"
 	"testing"
@@ -74,6 +75,7 @@ func TestInit(t *testing.T) {
 	if err != nil {
 		t.Fatalf("Failed to create temp dir: %v", err)
 	}
+	localName = filepath.ToSlash(localName)
 	t.Logf("Testing with local %q", localName)
 	flocal, err = fs.NewFs(localName)
 	if err != nil {
@@ -83,6 +85,7 @@ func TestInit(t *testing.T) {
 }
 func TestCalculateModifyWindow(t *testing.T) {
 	fs.CalculateModifyWindow(fremote, flocal)
+	t.Logf("ModifyWindow is %q", fs.Config.ModifyWindow)
 }
 
 func TestMkdir(t *testing.T) {
@@ -104,8 +107,8 @@ func TestCopyWithDryRun(t *testing.T) {
 		{Path: "sub dir/hello world", Size: 11, ModTime: t1, Md5sum: "5eb63bbbe01eeed093cb22bb8f5acdc3"},
 	}
 
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, []fstest.Item{})
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, []fstest.Item{}, fs.Config.ModifyWindow)
 }
 
 // Now without dry run
@@ -119,8 +122,8 @@ func TestCopy(t *testing.T) {
 		{Path: "sub dir/hello world", Size: 11, ModTime: t1, Md5sum: "5eb63bbbe01eeed093cb22bb8f5acdc3"},
 	}
 
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 func TestLsd(t *testing.T) {
@@ -145,8 +148,8 @@ func TestCopyAfterDelete(t *testing.T) {
 	items := []fstest.Item{
 		{Path: "sub dir/hello world", Size: 11, ModTime: t1, Md5sum: "5eb63bbbe01eeed093cb22bb8f5acdc3"},
 	}
-	fstest.CheckListing(t, flocal, []fstest.Item{})
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, []fstest.Item{}, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 func TestCopyRedownload(t *testing.T) {
@@ -158,8 +161,8 @@ func TestCopyRedownload(t *testing.T) {
 	items := []fstest.Item{
 		{Path: "sub dir/hello world", Size: 11, ModTime: t1, Md5sum: "5eb63bbbe01eeed093cb22bb8f5acdc3"},
 	}
-	fstest.CheckListingWithPrecision(t, flocal, items, fremote.Precision())
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 
 	// Clean the directory
 	cleanTempDir(t)
@@ -179,8 +182,8 @@ func TestSyncAfterChangingModtimeOnly(t *testing.T) {
 	items := []fstest.Item{
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 func TestSyncAfterAddingAFile(t *testing.T) {
@@ -193,8 +196,8 @@ func TestSyncAfterAddingAFile(t *testing.T) {
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 		{Path: "potato", Size: 60, ModTime: t3, Md5sum: "d6548b156ea68a4e003e786df99eee76"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 func TestSyncAfterChangingFilesSizeOnly(t *testing.T) {
@@ -207,8 +210,8 @@ func TestSyncAfterChangingFilesSizeOnly(t *testing.T) {
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 		{Path: "potato", Size: 21, ModTime: t3, Md5sum: "100defcf18c42a1e0dc42a789b107cd2"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 // Sync after changing a file's contents, modtime but not length
@@ -222,8 +225,8 @@ func TestSyncAfterChangingContentsOnly(t *testing.T) {
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 		{Path: "potato", Size: 21, ModTime: t2, Md5sum: "e4cb6955d9106df6263c45fcfc10f163"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 // Sync after removing a file and adding a file --dry-run
@@ -248,8 +251,8 @@ func TestSyncAfterRemovingAFileAndAddingAFileDryRun(t *testing.T) {
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 		{Path: "potato2", Size: 60, ModTime: t1, Md5sum: "d6548b156ea68a4e003e786df99eee76"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, before)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, before, fs.Config.ModifyWindow)
 }
 
 // Sync after removing a file and adding a file
@@ -262,8 +265,8 @@ func TestSyncAfterRemovingAFileAndAddingAFile(t *testing.T) {
 		{Path: "empty space", Size: 0, ModTime: t2, Md5sum: "d41d8cd98f00b204e9800998ecf8427e"},
 		{Path: "potato2", Size: 60, ModTime: t1, Md5sum: "d6548b156ea68a4e003e786df99eee76"},
 	}
-	fstest.CheckListing(t, flocal, items)
-	fstest.CheckListing(t, fremote, items)
+	fstest.CheckListingWithPrecision(t, flocal, items, fs.Config.ModifyWindow)
+	fstest.CheckListingWithPrecision(t, fremote, items, fs.Config.ModifyWindow)
 }
 
 func TestLs(t *testing.T) {

fs/seekwrapper.go

@@ -0,0 +1,39 @@
+package fs
+
+import (
+	"io"
+	"os"
+)
+
+// SeekWrapper wraps an io.Reader with a basic Seek method which
+// returns the Size attribute.
+//
+// This is used for google.golang.org/api/googleapi/googleapi.go
+// to detect the length (see getReaderSize function)
+//
+// Without this the getReaderSize function reads the entire file into
+// memory to find its length.
+type SeekWrapper struct {
+	In   io.Reader
+	Size int64
+}
+
+// Read bytes from the object - see io.Reader
+func (file *SeekWrapper) Read(p []byte) (n int, err error) {
+	return file.In.Read(p)
+}
+
+// Seek - minimal implementation for Google API length detection
+func (file *SeekWrapper) Seek(offset int64, whence int) (int64, error) {
+	switch whence {
+	case os.SEEK_CUR:
+		return 0, nil
+	case os.SEEK_END:
+		return file.Size, nil
+	}
+	return 0, nil
+}
+
+// Interfaces that SeekWrapper implements
+var _ io.Reader = (*SeekWrapper)(nil)
+var _ io.Seeker = (*SeekWrapper)(nil)

fs/version.go

@@ -1,3 +1,3 @@
 package fs
 
-const Version = "v1.06"
+const Version = "v1.09"

fstest/fstest.go

@@ -9,6 +9,7 @@ import (
 	"log"
 	"math/rand"
 	"os"
+	"path/filepath"
 	"strings"
 	"testing"
 	"time"
@@ -80,6 +81,7 @@ func (is *Items) Find(t *testing.T, obj fs.Object, precision time.Duration) {
 	i, ok := is.byName[obj.Remote()]
 	if !ok {
 		t.Errorf("Unexpected file %q", obj.Remote())
+		return
 	}
 	delete(is.byName, obj.Remote())
 	i.Check(t, obj, precision)
@@ -99,6 +101,10 @@ func (is *Items) Done(t *testing.T) {
 func CheckListingWithPrecision(t *testing.T, f fs.Fs, items []Item, precision time.Duration) {
 	is := NewItems(items)
 	for obj := range f.List() {
+		if obj == nil {
+			t.Errorf("Unexpected nil in List()")
+			continue
+		}
 		is.Find(t, obj, precision)
 	}
 	is.Done(t)
@@ -136,6 +142,7 @@ func LocalRemote() (path string, err error) {
 		// Now remove the directory
 		err = os.Remove(path)
 	}
+	path = filepath.ToSlash(path)
 	return
 }
 

fstest/fstests/fstests.go

@@ -312,7 +312,7 @@ func TestObjectSetModTime(t *testing.T) {
 	obj := findObject(t, file1.Path)
 	obj.SetModTime(newModTime)
 	file1.ModTime = newModTime
-	file1.CheckModTime(t, obj, newModTime, remote.Precision())
+	file1.CheckModTime(t, obj, obj.ModTime(), remote.Precision())
 	// And make a new object and read it from there too
 	TestObjectModTime(t)
 }

googlecloudstorage/googlecloudstorage.go

@@ -359,8 +359,9 @@ func (f *FsStorage) ListDir() fs.DirChan {
 // The new object may have been created if an error is returned
 func (f *FsStorage) Put(in io.Reader, remote string, modTime time.Time, size int64) (fs.Object, error) {
 	// Temporary FsObject under construction
-	fs := &FsObjectStorage{storage: f, remote: remote}
-	return fs, fs.Update(in, modTime, size)
+	o := &FsObjectStorage{storage: f, remote: remote}
+	in = &fs.SeekWrapper{In: in, Size: size}
+	return o, o.Update(in, modTime, size)
 }
 
 // Mkdir creates the bucket if it doesn't exist
@@ -562,6 +563,7 @@ func (o *FsObjectStorage) Update(in io.Reader, modTime time.Time, size int64) er
 		Updated:     modTime.Format(timeFormatOut), // Doesn't get set
 		Metadata:    metadataFromModTime(modTime),
 	}
+	in = &fs.SeekWrapper{In: in, Size: size}
 	newObject, err := o.storage.svc.Objects.Insert(o.storage.bucket, &object).Media(in).Name(object.Name).PredefinedAcl(o.storage.objectAcl).Do()
 	if err != nil {
 		return err

local/local.go

@@ -1,6 +1,11 @@
 // Local filesystem interface
 package local
 
+// Note that all rclone paths should be / separated.  Anything coming
+// from the filepath module will have \ separators on windows so
+// should be converted using filepath.ToSlash.  Windows is quite happy
+// with / separators so there is no need to convert them back.
+
 import (
 	"crypto/md5"
 	"encoding/hex"
@@ -45,7 +50,7 @@ type FsObjectLocal struct {
 
 // NewFs contstructs an FsLocal from the path
 func NewFs(name, root string) (fs.Fs, error) {
-	root = path.Clean(root)
+	root = filepath.ToSlash(path.Clean(root))
 	f := &FsLocal{root: root}
 	// Check to see if this points to a file
 	fi, err := os.Lstat(f.root)
@@ -69,7 +74,8 @@ func (f *FsLocal) String() string {
 //
 // May return nil if an error occurred
 func (f *FsLocal) newFsObjectWithInfo(remote string, info os.FileInfo) fs.Object {
-	path := filepath.Join(f.root, remote)
+	remote = filepath.ToSlash(remote)
+	path := path.Join(f.root, remote)
 	o := &FsObjectLocal{local: f, remote: remote, path: path}
 	if info != nil {
 		o.info = info
@@ -173,7 +179,7 @@ func (f *FsLocal) ListDir() fs.DirChan {
 
 // Puts the FsObject to the local filesystem
 func (f *FsLocal) Put(in io.Reader, remote string, modTime time.Time, size int64) (fs.Object, error) {
-	dstPath := filepath.Join(f.root, remote)
+	dstPath := path.Join(f.root, remote)
 	// Temporary FsObject under construction - info filled in by Update()
 	o := &FsObjectLocal{local: f, remote: remote, path: dstPath}
 	err := o.Update(in, modTime, size)

notes.txt

@@ -38,6 +38,12 @@ Todo
   * Need to make directory objects otherwise can't upload an empty directory
     * Or could upload empty directories only?
 
+More rsync features
+  * include
+  * exclude
+  * max size
+  * bandwidth limit
+
 Ideas
   * could do encryption - put IV into metadata?
   * optimise remote copy container to another container using remote
@@ -59,14 +65,3 @@ Bugs
   * Non verbose - not sure number transferred got counted up? CHECK
   * When doing copy it recurses the whole of the destination FS which isn't necessary
 
-Making a release
-  * make test
-  * make tag
-  * edit README.md
-  * git commit fs/version.go README.md docs/content/downloads.md
-  * make retag
-  * . ~/bin/go-cross
-  * make cross
-  * make upload
-  * make upload_website
-  * git push --tags origin master