jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2026-01-07 11:03:15 +00:00
Code Issues Projects Releases Wiki Activity

bisync: add to integration tests - fixes #7665

Browse Source 
This change officially adds bisync to the nightly integration tests for all
backends.

This will be part of giving us the confidence to take bisync out of beta.

A number of fixes have been added to account for features which can differ on
different backends -- for example, hash types / modtime support, empty
directories, unicode normalization, and unimportant differences in log output.
We will likely find that more of these are needed once we start running these
with the full set of remotes.

Additionally, bisync's extremely sensitive tests revealed a few bugs in other
backends that weren't previously covered by other tests. Fixes for those issues
have been submitted on the following separate PRs (and bisync test failures will
be expected until they are merged):

- #7670 memory: fix deadlock in operations.Purge
- #7688 memory: fix incorrect list entries when rooted at subdirectory
- #7690 memory: fix dst mutating src after server-side copy
- #7692 dropbox: fix chunked uploads when size <= chunkSize

Relatedly, workarounds have been put in place for the following backend
limitations that are unsolvable for the time being:

- #3262 drive is sometimes aware of trashed files/folders when it shouldn't be
- #6199 dropbox can't handle emojis and certain other characters
- #4590 onedrive API has longstanding bug for conflictBehavior=replace in
	server-side copy/move
This commit is contained in:
nielash
2024-03-08 17:53:33 -05:00
parent fecce67ac6
commit 2bebbfaded
56 changed files with 59314 additions and 176 deletions
Download Patch File Download Diff File Expand all files Collapse all files

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.copy1to2.que vendored
View File

@@ -1,3 +1,3 @@
"folder/HeLlO,wOrLd!.txt"
"folder/éééö.txt"
"newfolder/HeLlO,wOrLd!.txt"
"newfolder/éééö.txt"
"測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path1.lst vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path1.lst-new vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path1.lst-old vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/HeLlO,wOrLd!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path2.lst vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/hello,WORLD!.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-05T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path2.lst-new vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-05T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/hello,WORLD!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

4
cmd/bisync/testdata/test_normalization/golden/_testdir_path1.._testdir_path2.path2.lst-old vendored
View File

@@ -2,6 +2,6 @@
- 19 - - 2001-01-03T00:00:00.000000000+0000 "file1.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file2.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "file3.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/hello,WORLD!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "folder/éééö.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/hello,WORLD!.txt"
- 19 - - 2001-01-03T00:00:00.000000000+0000 "newfolder/éééö.txt"
- 19 - - 2001-01-02T00:00:00.000000000+0000 "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"

40
cmd/bisync/testdata/test_normalization/golden/test.log vendored
View File

@@ -103,13 +103,13 @@ INFO : Bisync successful
(19) : copy-as-NFC {datadir/}file1.txt {path1/}測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö 測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö.txt
(20) : copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
(21) : copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
(20) : copy-as-NFC {datadir/}file1.txt {path1/}newfolder éééö.txt
(21) : copy-as-NFC {datadir/}file1.txt {path1/}newfolder HeLlO,wOrLd!.txt
(22) : touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
(23) : copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
(24) : copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
(23) : copy-as-NFD {datadir/}file1.txt {path2/}newfolder éééö.txt
(24) : copy-as-NFD {datadir/}file1.txt {path2/}newfolder hello,WORLD!.txt
(25) : test bisync run with normalization
(26) : bisync norm force
@@ -126,14 +126,14 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File is new - folder/HeLlO,wOrLd!.txt
INFO : - Path1 File is new - folder/éééö.txt
INFO : - Path1 File is new - newfolder/HeLlO,wOrLd!.txt
INFO : - Path1 File is new - newfolder/éééö.txt
INFO : - Path1 File is new - "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"
INFO : Path1: 3 changes:  3 new,  0 modified,  0 deleted
INFO : Path2 checking for diffs
INFO : - Path2 File changed: time (newer) - file1.txt
INFO : - Path2 File is new - folder/éééö.txt
INFO : - Path2 File is new - folder/hello,WORLD!.txt
INFO : - Path2 File is new - newfolder/éééö.txt
INFO : - Path2 File is new - newfolder/hello,WORLD!.txt
INFO : Path2: 3 changes:  2 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older)
INFO : Applying changes
@@ -141,12 +141,12 @@ INFO : Checking potential conflicts...
NOTICE: {path2String}: 0 differences found
NOTICE: {path2String}: 2 matching files
INFO : Finished checking the potential conflicts. %!s(<nil>)
NOTICE: - WARNING New or changed in both paths - folder/HeLlO,wOrLd!.txt
INFO : folder/HeLlO,wOrLd!.txt: Files are equal but will copy anyway to update modtime (will not rename)
INFO : - Path2 Queue copy to Path1 - {path1/}folder/HeLlO,wOrLd!.txt
NOTICE: - WARNING New or changed in both paths - folder/éééö.txt
INFO : folder/éééö.txt: Files are equal but will copy anyway to update modtime (will not rename)
INFO : - Path2 Queue copy to Path1 - {path1/}folder/éééö.txt
NOTICE: - WARNING New or changed in both paths - newfolder/HeLlO,wOrLd!.txt
INFO : newfolder/HeLlO,wOrLd!.txt: Files are equal but will copy anyway to update modtime (will not rename)
INFO : - Path2 Queue copy to Path1 - {path1/}newfolder/HeLlO,wOrLd!.txt
NOTICE: - WARNING New or changed in both paths - newfolder/éééö.txt
INFO : newfolder/éééö.txt: Files are equal but will copy anyway to update modtime (will not rename)
INFO : - Path2 Queue copy to Path1 - {path1/}newfolder/éééö.txt
INFO : - Path1 Queue copy to Path2 - "{path2/}測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Do queued copies to - Path1
@@ -178,8 +178,8 @@ INFO : Bisync successful
(29) : test changed on one path
(30) : touch-copy 2001-01-05 {datadir/}file1.txt {path2/}
(31) : copy-as-NFC {datadir/}file1.txt {path1/}測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö 測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö.txt
(32) : copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
(33) : copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
(32) : copy-as-NFC {datadir/}file1.txt {path1/}newfolder éééö.txt
(33) : copy-as-NFC {datadir/}file1.txt {path1/}newfolder HeLlO,wOrLd!.txt
(34) : bisync norm
INFO : Setting --ignore-listing-checksum as neither --checksum nor --compare checksum are set.
INFO : Bisyncing with Comparison Settings:
@@ -194,8 +194,8 @@ INFO : Bisyncing with Comparison Settings:
INFO : Synching Path1 "{path1/}" with Path2 "{path2/}"
INFO : Building Path1 and Path2 listings
INFO : Path1 checking for diffs
INFO : - Path1 File changed: time (newer) - folder/HeLlO,wOrLd!.txt
INFO : - Path1 File changed: time (newer) - folder/éééö.txt
INFO : - Path1 File changed: time (newer) - newfolder/HeLlO,wOrLd!.txt
INFO : - Path1 File changed: time (newer) - newfolder/éééö.txt
INFO : - Path1 File changed: time (newer) - "測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"
INFO : Path1: 3 changes:  0 new,  3 modified,  0 deleted
INFO : (Modified:  3 newer,  0 older)
@@ -204,8 +204,8 @@ INFO : - Path2 File changed: time (newer)[0
INFO : Path2: 1 changes:  0 new,  1 modified,  0 deleted
INFO : (Modified:  1 newer,  0 older)
INFO : Applying changes
INFO : - Path1 Queue copy to Path2 - {path2/}folder/hello,WORLD!.txt
INFO : - Path1 Queue copy to Path2 - {path2/}folder/éééö.txt
INFO : - Path1 Queue copy to Path2 - {path2/}newfolder/hello,WORLD!.txt
INFO : - Path1 Queue copy to Path2 - {path2/}newfolder/éééö.txt
INFO : - Path1 Queue copy to Path2 - "{path2/}測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö/測試_Руский___ě_áñ👸🏼🧝🏾\u200d♀️💆🏿\u200d♂️🐨🤙🏼🤮🧑🏻\u200d🔧🧑\u200d🔬éö.txt"
INFO : - Path2 Queue copy to Path1 - {path1/}file1.txt
INFO : - Path2 Do queued copies to - Path1

12
cmd/bisync/testdata/test_normalization/scenario.txt vendored
View File

@@ -32,13 +32,13 @@ bisync resync
# copy NFC version to Path1
# note: need to slightly change the name to avoid Drive known issue #3262 which could try to copy the old name from the trash
copy-as-NFC {datadir/}file1.txt {path1/}測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö 測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
copy-as-NFC {datadir/}file1.txt {path1/}newfolder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}newfolder HeLlO,wOrLd!.txt
# place newer NFD version on Path2
touch-copy 2001-01-03 {datadir/}file1.txt {path2/}
copy-as-NFD {datadir/}file1.txt {path2/}folder éééö.txt
copy-as-NFD {datadir/}file1.txt {path2/}folder hello,WORLD!.txt
copy-as-NFD {datadir/}file1.txt {path2/}newfolder éééö.txt
copy-as-NFD {datadir/}file1.txt {path2/}newfolder hello,WORLD!.txt
test bisync run with normalization
bisync norm force
@@ -49,6 +49,6 @@ bisync resync norm
test changed on one path
touch-copy 2001-01-05 {datadir/}file1.txt {path2/}
copy-as-NFC {datadir/}file1.txt {path1/}測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö 測試_Руский___ě_áñ👸🏼🧝🏾‍♀️💆🏿‍♂️🐨🤙🏼🤮🧑🏻‍🔧🧑‍🔬éö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}folder HeLlO,wOrLd!.txt
copy-as-NFC {datadir/}file1.txt {path1/}newfolder éééö.txt
copy-as-NFC {datadir/}file1.txt {path1/}newfolder HeLlO,wOrLd!.txt
bisync norm