bisync: Add support for --create-empty-src-dirs - Fixes #6109

Sync creation and deletion of empty directories.
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=3.%20Bisync%20should%20create/delete%20empty%20directories%20as%20sync%20does%2C%20when%20%2D%2Dcreate%2Dempty%2Dsrc%2Ddirs%20is%20passed

Also fixed an issue causing --resync to erroneously delete empty folders and duplicate files unique to Path2
https://forum.rclone.org/t/bisync-bugs-and-feature-requests/37636#:~:text=2.%20%2D%2Dresync%20deletes%20data%2C%20contrary%20to%20docs

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.copy1to2.que

@@ -0,0 +1 @@
+"subdir"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.delete2.que

@@ -0,0 +1 @@
+"subdir"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.path1.lst

@@ -0,0 +1,7 @@
+# bisync listing v1 from test
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy1.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy2.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy3.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy4.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy5.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.txt"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.path1.lst-new

@@ -0,0 +1,7 @@
+# bisync listing v1 from test
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy1.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy2.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy3.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy4.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy5.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.txt"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.path2.lst

@@ -0,0 +1,7 @@
+# bisync listing v1 from test
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy1.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy2.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy3.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy4.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy5.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.txt"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.path2.lst-new

@@ -0,0 +1,7 @@
+# bisync listing v1 from test
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy1.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy2.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy3.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy4.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.copy5.txt"
+-        0 md5:d41d8cd98f00b204e9800998ecf8427e - 2001-01-02T00:00:00.000000000+0000 "file1.txt"

cmd/bisync/testdata/test_createemptysrcdirs/golden/_testdir_path1.._testdir_path2.resync-copy2to1.que

@@ -0,0 +1 @@
+"subdir"

cmd/bisync/testdata/test_createemptysrcdirs/golden/test.log

@@ -0,0 +1,142 @@
+(01)  : test createemptysrcdirs
+
+
+(02)  : test initial bisync
+(03)  : touch-glob 2001-01-02 {datadir/} placeholder.txt
+(04)  : copy-as {datadir/}placeholder.txt {path1/} file1.txt
+(05)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+(06)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+(07)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+(08)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+(09)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+(10)  : bisync resync
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Copying unique Path2 files to Path1
+INFO  : Resynching Path1 to Path2
+INFO  : Resync updating listings
+INFO  : Bisync successful
+
+(11)  : test 1. Create an empty dir on Path1 by creating subdir/placeholder.txt and then deleting the placeholder
+(12)  : copy-as {datadir/}placeholder.txt {path1/} subdir/placeholder.txt
+(13)  : touch-glob 2001-01-02 {path1/} subdir
+(14)  : delete-file {path1/}subdir/placeholder.txt
+
+(15)  : test 2. Run bisync without --create-empty-src-dirs
+(16)  : bisync
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Path1 checking for diffs
+INFO  : Path2 checking for diffs
+INFO  : No changes found
+INFO  : Updating listings
+INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
+INFO  : Bisync successful
+
+(17)  : test 3. Confirm the subdir exists only on Path1 and not Path2
+(18)  : list-dirs {path1/}
+subdir/
+(19)  : list-dirs {path2/}
+
+(20)  : test 4.Run bisync WITH --create-empty-src-dirs
+(21)  : bisync create-empty-src-dirs
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Path1 checking for diffs
+INFO  : - Path1    File is new                         - subdir
+INFO  : Path1:    1 changes:    1 new,    0 newer,    0 older,    0 deleted
+INFO  : Path2 checking for diffs
+INFO  : Applying changes
+INFO  : - Path1    Queue copy to Path2                 - {path2/}subdir
+INFO  : - Path1    Do queued copies to                 - Path2
+INFO  : Updating listings
+INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
+INFO  : Bisync successful
+
+(22)  : test 5. Confirm the subdir exists on both paths
+(23)  : list-dirs {path1/}
+subdir/
+(24)  : list-dirs {path2/}
+subdir/
+
+(25)  : test 6. Delete the empty dir on Path1 using purge-children (and also add files so the path isn't empty)
+(26)  : purge-children {path1/}
+(27)  : copy-as {datadir/}placeholder.txt {path1/} file1.txt
+(28)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+(29)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+(30)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+(31)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+(32)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+
+(33)  : test 7. Run bisync without --create-empty-src-dirs
+(34)  : bisync
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Path1 checking for diffs
+INFO  : - Path1    File was deleted                    - RCLONE_TEST
+INFO  : - Path1    File was deleted                    - subdir
+INFO  : Path1:    2 changes:    0 new,    0 newer,    0 older,    2 deleted
+INFO  : Path2 checking for diffs
+INFO  : - Path2    File was deleted                    - subdir
+INFO  : Path2:    1 changes:    0 new,    0 newer,    0 older,    1 deleted
+INFO  : Applying changes
+INFO  : - Path2    Queue delete                        - {path2/}RCLONE_TEST
+INFO  : -          Do queued deletes on                - Path2
+INFO  : Updating listings
+INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
+INFO  : Bisync successful
+
+(35)  : test 8. Confirm the subdir exists only on Path2 and not Path1
+(36)  : list-dirs {path1/}
+(37)  : list-dirs {path2/}
+subdir/
+
+(38)  : test 9. Reset, do the delete again, and run bisync WITH --create-empty-src-dirs
+(39)  : bisync resync create-empty-src-dirs
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Copying unique Path2 files to Path1
+INFO  : - Path2    Resync will copy to Path1           - subdir
+INFO  : - Path2    Resync is doing queued copies to    - Path1
+INFO  : Resynching Path1 to Path2
+INFO  : Resynching Path2 to Path1 (for empty dirs)
+INFO  : Resync updating listings
+INFO  : Bisync successful
+(40)  : list-dirs {path1/}
+subdir/
+(41)  : list-dirs {path2/}
+subdir/
+
+(42)  : purge-children {path1/}
+(43)  : copy-as {datadir/}placeholder.txt {path1/} file1.txt
+(44)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+(45)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+(46)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+(47)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+(48)  : copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+(49)  : list-dirs {path1/}
+(50)  : list-dirs {path2/}
+subdir/
+
+(51)  : bisync create-empty-src-dirs
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Path1 checking for diffs
+INFO  : - Path1    File was deleted                    - subdir
+INFO  : Path1:    1 changes:    0 new,    0 newer,    0 older,    1 deleted
+INFO  : Path2 checking for diffs
+INFO  : Applying changes
+INFO  : - Path2    Queue delete                        - {path2/}subdir
+INFO  : -          Do queued deletes on                - Path2
+INFO  : subdir: Removing directory
+INFO  : Updating listings
+INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
+INFO  : Bisync successful
+
+(52)  : test 10. Confirm the subdir has been removed on both paths
+(53)  : list-dirs {path1/}
+(54)  : list-dirs {path2/}
+
+(55)  : test 11. bisync again (because if we leave subdir in listings, test will fail due to mismatched modtime)
+(56)  : bisync create-empty-src-dirs
+INFO  : Synching Path1 "{path1/}" with Path2 "{path2/}"
+INFO  : Path1 checking for diffs
+INFO  : Path2 checking for diffs
+INFO  : No changes found
+INFO  : Updating listings
+INFO  : Validating listings for Path1 "{path1/}" vs Path2 "{path2/}"
+INFO  : Bisync successful

cmd/bisync/testdata/test_createemptysrcdirs/initial/RCLONE_TEST

@@ -0,0 +1 @@
+This file is used for testing the health of rclone accesses to the local/remote file system.  Do not delete.

cmd/bisync/testdata/test_createemptysrcdirs/scenario.txt

@@ -0,0 +1,87 @@
+test createemptysrcdirs
+# Test the --create-empty-src-dirs logic.
+# Should behave the same way as rclone sync.
+# Without this flag, empty directories created/deleted on one side are NOT created/deleted on the other side
+# With this flag, empty directories created/deleted on one side are created/deleted on the other side; the result should be an exact mirror.
+#
+# Placeholders are necessary to ensure that git does not lose our empty folders
+# After the initial setup sync:
+#  1. Create an empty dir on Path1 by creating subdir/placeholder.txt and then deleting the placeholder
+#  2. Run bisync without --create-empty-src-dirs
+#  3. Confirm the subdir exists only on Path1 and not Path2
+#  4. Run bisync WITH --create-empty-src-dirs
+#  5. Confirm the subdir exists on both paths
+#  6. Delete the empty dir on Path1 using purge-children (and also add files so the path isn't empty)
+#  7. Run bisync without --create-empty-src-dirs
+#  8. Confirm the subdir exists only on Path2 and not Path1
+#  9. Reset, do the delete again, and run bisync WITH --create-empty-src-dirs
+#  10. Confirm the subdir has been removed on both paths
+
+test initial bisync
+touch-glob 2001-01-02 {datadir/} placeholder.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+bisync resync
+
+test 1. Create an empty dir on Path1 by creating subdir/placeholder.txt and then deleting the placeholder
+copy-as {datadir/}placeholder.txt {path1/} subdir/placeholder.txt
+touch-glob 2001-01-02 {path1/} subdir
+delete-file {path1/}subdir/placeholder.txt
+
+test 2. Run bisync without --create-empty-src-dirs
+bisync
+
+test 3. Confirm the subdir exists only on Path1 and not Path2
+list-dirs {path1/}
+list-dirs {path2/}
+
+test 4.Run bisync WITH --create-empty-src-dirs
+bisync create-empty-src-dirs
+
+test 5. Confirm the subdir exists on both paths
+list-dirs {path1/}
+list-dirs {path2/}
+
+test 6. Delete the empty dir on Path1 using purge-children (and also add files so the path isn't empty)
+purge-children {path1/}
+copy-as {datadir/}placeholder.txt {path1/} file1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+
+test 7. Run bisync without --create-empty-src-dirs
+bisync
+
+test 8. Confirm the subdir exists only on Path2 and not Path1
+list-dirs {path1/}
+list-dirs {path2/}
+
+test 9. Reset, do the delete again, and run bisync WITH --create-empty-src-dirs
+bisync resync create-empty-src-dirs
+list-dirs {path1/}
+list-dirs {path2/}
+
+purge-children {path1/}
+copy-as {datadir/}placeholder.txt {path1/} file1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy1.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy2.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy3.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy4.txt
+copy-as {datadir/}placeholder.txt {path1/} file1.copy5.txt
+list-dirs {path1/}
+list-dirs {path2/}
+
+bisync create-empty-src-dirs
+
+test 10. Confirm the subdir has been removed on both paths
+list-dirs {path1/}
+list-dirs {path2/}
+
+test 11. bisync again (because if we leave subdir in listings, test will fail due to mismatched modtime)
+bisync create-empty-src-dirs