bisync: add to integration tests - fixes #7665

This change officially adds bisync to the nightly integration tests for all
backends.

This will be part of giving us the confidence to take bisync out of beta.

A number of fixes have been added to account for features which can differ on
different backends -- for example, hash types / modtime support, empty
directories, unicode normalization, and unimportant differences in log output.
We will likely find that more of these are needed once we start running these
with the full set of remotes.

Additionally, bisync's extremely sensitive tests revealed a few bugs in other
backends that weren't previously covered by other tests. Fixes for those issues
have been submitted on the following separate PRs (and bisync test failures will
be expected until they are merged):

- #7670 memory: fix deadlock in operations.Purge
- #7688 memory: fix incorrect list entries when rooted at subdirectory
- #7690 memory: fix dst mutating src after server-side copy
- #7692 dropbox: fix chunked uploads when size <= chunkSize

Relatedly, workarounds have been put in place for the following backend
limitations that are unsolvable for the time being:

- #3262 drive is sometimes aware of trashed files/folders when it shouldn't be
- #6199 dropbox can't handle emojis and certain other characters
- #4590 onedrive API has longstanding bug for conflictBehavior=replace in
	server-side copy/move

cmd/bisync/testdata/test_rclone_args/golden/test.log

@@ -1,8 +1,10 @@
 (01)  : test rclone-args
 
 
-(02)  : test initial bisync
-(03)  : bisync resync checksum
+(02)  : touch-glob 2001-01-02 {datadir/} *
+
+(03)  : test initial bisync
+(04)  : bisync resync checksum
 INFO  : Bisyncing with Comparison Settings:
 {
 "Modtime": false,
@@ -20,19 +22,20 @@ INFO  : Resync updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
 
-(04)  : test place newer files on both paths
-
-
-(05)  : touch-glob 2001-01-02 {datadir/} *
-
+(05)  : test place newer files on both paths
 (06)  : copy-file {datadir/}file1.txt {path1/}
 (07)  : copy-file {datadir/}file2.txt {path2/}
-
 (08)  : copy-file {datadir/}file20.txt {path1/}subdir
 (09)  : copy-file {datadir/}file21.txt {path2/}subdir
 
-(10)  : test run bisync with custom options
-(11)  : bisync checksum
+
+(10)  : touch-glob 2001-01-02 {path1/} file1.txt
+(11)  : touch-glob 2001-01-02 {path2/} file2.txt
+(12)  : touch-glob 2001-01-02 {path1/}subdir/ file20.txt
+(13)  : touch-glob 2001-01-02 {path2/}subdir/ file21.txt
+
+(14)  : test run bisync with custom options
+(15)  : bisync checksum
 INFO  : Bisyncing with Comparison Settings:
 {
 "Modtime": false,
@@ -61,15 +64,20 @@ INFO  : Updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
 
-(12)  : touch-glob 2007-07-23 {datadir/} *
+(16)  : touch-glob 2007-07-23 {datadir/} *
 
-(13)  : copy-file {datadir/}file1.txt {path1/}
-(14)  : copy-file {datadir/}file2.txt {path2/}
+(17)  : copy-file {datadir/}file1.txt {path1/}
+(18)  : copy-file {datadir/}file2.txt {path2/}
+(19)  : copy-file {datadir/}file20.txt {path1/}subdir
+(20)  : copy-as {datadir/}file21.txt {path2/} file1.txt
 
-(15)  : copy-file {datadir/}file20.txt {path1/}subdir
-(16)  : copy-as {datadir/}file21.txt {path2/} file1.txt
 
-(17)  : bisync size-only
+(21)  : touch-glob 2007-07-23 {path1/} file1.txt
+(22)  : touch-glob 2007-07-23 {path2/} file2.txt
+(23)  : touch-glob 2007-07-23 {path1/}subdir/ file20.txt
+(24)  : touch-glob 2007-07-23 {path2/}subdir/ file21.txt
+
+(25)  : bisync size-only
 INFO  : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
 INFO  : Bisyncing with Comparison Settings:
 {
@@ -93,7 +101,7 @@ INFO  : - Path2    Do queued copies to                - P
 INFO  : Updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
-(18)  : bisync resync
+(26)  : bisync resync
 INFO  : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
 INFO  : Bisyncing with Comparison Settings:
 {
@@ -112,13 +120,18 @@ INFO  : Resync updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
 
-(19)  : copy-file {datadir/}file1.txt {path1/}
-(20)  : copy-file {datadir/}file2.txt {path2/}
+(27)  : copy-file {datadir/}file1.txt {path1/}
+(28)  : copy-file {datadir/}file2.txt {path2/}
+(29)  : copy-file {datadir/}file20.txt {path1/}subdir
+(30)  : copy-file {datadir/}file21.txt {path2/}subdir
 
-(21)  : copy-file {datadir/}file20.txt {path1/}subdir
-(22)  : copy-file {datadir/}file21.txt {path2/}subdir
 
-(23)  : bisync ignore-size
+(31)  : touch-glob 2007-07-23 {path1/} file1.txt
+(32)  : touch-glob 2007-07-23 {path2/} file2.txt
+(33)  : touch-glob 2007-07-23 {path1/}subdir/ file20.txt
+(34)  : touch-glob 2007-07-23 {path2/}subdir/ file21.txt
+
+(35)  : bisync ignore-size
 INFO  : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
 INFO  : Bisyncing with Comparison Settings:
 {
@@ -145,7 +158,7 @@ INFO  : Updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
 
-(24)  : bisync resync compare-all
+(36)  : bisync resync compare-all
 INFO  : Bisyncing with Comparison Settings:
 {
 "Modtime": true,
@@ -163,16 +176,22 @@ INFO  : Resync updating listings
 INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
 INFO  : Bisync successful
 
-(25)  : copy-as {datadir/}file21.txt {path2/} file2.txt
+(37)  : copy-as {datadir/}file21.txt {path2/} file2.txt
 
-(26)  : touch-glob 2023-08-26 {datadir/} *
+(38)  : touch-glob 2023-08-26 {datadir/} *
 
-(27)  : copy-file {datadir/}file1.txt {path1/}
+(39)  : copy-file {datadir/}file1.txt {path1/}
 
-(28)  : copy-file {datadir/}file20.txt {path1/}subdir
-(29)  : copy-file {datadir/}file21.txt {path2/}subdir
+(40)  : copy-file {datadir/}file20.txt {path1/}subdir
+(41)  : copy-file {datadir/}file21.txt {path2/}subdir
 
-(30)  : bisync compare-all
+
+(42)  : touch-glob 2007-07-23 {path2/} file2.txt
+(43)  : touch-glob 2023-08-26 {path1/} file1.txt
+(44)  : touch-glob 2023-08-26 {path1/}subdir/ file20.txt
+(45)  : touch-glob 2023-08-26 {path2/}subdir/ file21.txt
+
+(46)  : bisync compare-all
 INFO  : Bisyncing with Comparison Settings:
 {
 "Modtime": true,

cmd/bisync/testdata/test_rclone_args/scenario.txt

@@ -2,21 +2,24 @@ test rclone-args
 # Pass generic flags to rclone under test using as an example
 # the --size-only flag, which changes the meaning of operations.
 
+# force specific modification time since file time is lost through git
+touch-glob 2001-01-02 {datadir/} *
 
 test initial bisync
 bisync resync checksum
 
 test place newer files on both paths
-
-# force specific modification time since file time is lost through git
-touch-glob 2001-01-02 {datadir/} *
-
 copy-file {datadir/}file1.txt {path1/}
 copy-file {datadir/}file2.txt {path2/}
-
 copy-file {datadir/}file20.txt {path1/}subdir
 copy-file {datadir/}file21.txt {path2/}subdir
 
+# re-touch dest to avoid race conditions with parallel tests
+touch-glob 2001-01-02 {path1/} file1.txt
+touch-glob 2001-01-02 {path2/} file2.txt
+touch-glob 2001-01-02 {path1/}subdir/ file20.txt
+touch-glob 2001-01-02 {path2/}subdir/ file21.txt
+
 test run bisync with custom options
 bisync checksum
 
@@ -24,19 +27,29 @@ touch-glob 2007-07-23 {datadir/} *
 
 copy-file {datadir/}file1.txt {path1/}
 copy-file {datadir/}file2.txt {path2/}
-
 copy-file {datadir/}file20.txt {path1/}subdir
 copy-as {datadir/}file21.txt {path2/} file1.txt
 
+# re-touch dest to avoid race conditions with parallel tests
+touch-glob 2007-07-23 {path1/} file1.txt
+touch-glob 2007-07-23 {path2/} file2.txt
+touch-glob 2007-07-23 {path1/}subdir/ file20.txt
+touch-glob 2007-07-23 {path2/}subdir/ file21.txt
+
 bisync size-only
 bisync resync
 
 copy-file {datadir/}file1.txt {path1/}
 copy-file {datadir/}file2.txt {path2/}
-
 copy-file {datadir/}file20.txt {path1/}subdir
 copy-file {datadir/}file21.txt {path2/}subdir
 
+# re-touch dest to avoid race conditions with parallel tests
+touch-glob 2007-07-23 {path1/} file1.txt
+touch-glob 2007-07-23 {path2/} file2.txt
+touch-glob 2007-07-23 {path1/}subdir/ file20.txt
+touch-glob 2007-07-23 {path2/}subdir/ file21.txt
+
 bisync ignore-size
 
 bisync resync compare-all
@@ -50,4 +63,10 @@ copy-file {datadir/}file1.txt {path1/}
 copy-file {datadir/}file20.txt {path1/}subdir
 copy-file {datadir/}file21.txt {path2/}subdir
 
+# re-touch dest to avoid race conditions with parallel tests
+touch-glob 2007-07-23 {path2/} file2.txt
+touch-glob 2023-08-26 {path1/} file1.txt
+touch-glob 2023-08-26 {path1/}subdir/ file20.txt
+touch-glob 2023-08-26 {path2/}subdir/ file21.txt
+
 bisync compare-all