Bump version to 2.2.0

It can print out the chunk as soon as a chunk is retrieved.  This avoids
reconstructing the file in the memory which can be an issue with large files.

.github/ISSUE_TEMPLATE.md

@@ -0,0 +1,5 @@
+Please submit an issue for bug reports or feature requests.  If you have any questions please post them on https://forum.duplicacy.com.
+
+When you're reporting a bug, please specify the OS, version, command line arguments, or any info that you think is helpful for the diagnosis.  If Duplicacy reports an error, please post the program output here.
+
+Note that this repository hosts the CLI version of Duplicacy only.  If you're reporting anything related to the GUI version, please visit https://forum.duplicacy.com.

.gitignore

@@ -1,3 +0,0 @@
-.idea
-duplicacy_main
-

ACKNOWLEDGEMENTS.md

@@ -8,9 +8,9 @@ Duplicacy is based on the following open source projects:
 |https://github.com/Azure/azure-sdk-for-go  |  Apache-2.0 |
 |https://github.com/tj/go-dropbox           |  MIT |
 |https://github.com/aws/aws-sdk-go          |  Apache-2.0 |
-|https://github.com/goamz/goamz             |  LGPL with static link exception | 
+|https://github.com/goamz/goamz             |  LGPL with static link exception |
 |https://github.com/howeyc/gopass           |  ISC |
 |https://github.com/tmc/keyring             |  ISC |
-|https://github.com/pcwizz/xattr            |  BSD-2-Clause | 
+|https://github.com/pcwizz/xattr            |  BSD-2-Clause |
 |https://github.com/minio/blake2b-simd      |  Apache-2.0 |
 |https://github.com/go-ole/go-ole           |  MIT |

DESIGN.md

@@ -1,215 +1,5 @@
-## Lock-Free Deduplication
-
-The three elements of lock-free deduplication are:
-
-* Use variable-size chunking algorithm to split files into chunks
-* Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
-* Apply a *two-step fossil collection* algorithm to remove chunks that become unreferenced after a backup is deleted
-
-The variable-size chunking algorithm, also called Content-Defined Chunking, is well-known and has been adopted by many
-backup tools. The main advantage of the variable-size chunking algorithm over the fixed-size chunking algorithm (as used
-by rsync) is that in the former the rolling hash is only used to search for boundaries between chunks, after which a far
-more collision-resistant hash function like MD5 or SHA256 is applied on each chunk. In contrast, in the fixed-size
-chunking algorithm, for the purpose of detecting inserts or deletions, a lookup in the known hash table is required every
-time the rolling hash window is shifted by one byte, thus significantly reducing the chunk splitting performance.
-
-What is novel about lock-free deduplication is the absence of a centralized indexing database for tracking all existing
-chunks and for determining which chunks are not needed any more.  Instead, to check if a chunk has already been uploaded
-before, one can just perform a file lookup via the file storage API using the file name derived from the hash of the chunk.
-This effectively turns a cloud storage offering only a very limited
-set of basic file operations into a powerful modern backup backend capable of both block-level and file-level deduplication.  More importantly, the absence of a centralized indexing database means that there is no need to implement a distributed locking mechanism on top of the file storage.
-
-By eliminating the chunk indexing database, lock-free duplication not only reduces the code complexity but also makes the deduplication less error-prone.  Each chunk is saved individually in its own file, and once saved there is no need for modification.  Data corruption is therefore less likely to occur because of the immutability of chunk files.  Another benefit that comes naturally from lock-free duplication is that when one client creates a new chunk, other clients that happen to have the same original file will notice that the chunk  already exist and therefore will not upload the same chunk again.  This pushes the deduplication to its highest level -- clients without knowledge of each other can share identical chunks with no extra effort.
-
-There is one problem, though.
-Deletion of snapshots without an indexing database, when concurrent access is permitted, turns out to be a hard problem.
-If exclusive access to a file storage by a single client can be guaranteed, the deletion procedure can simply search for
-chunks not referenced by any backup and delete them. However, if concurrent access is required, an unreferenced chunk
-can't be trivially removed, because of the possibility that a backup procedure in progress may reference the same chunk.
-The ongoing backup procedure, still unknown to the deletion procedure, may have already encountered that chunk during its
-file scanning phase, but decided not to upload the chunk again since it already exists in the file storage. 
-
-Fortunately, there is a solution to address the deletion problem and make lock-free deduplication practical.  The solution is a *two-step fossil collection* algorithm that deletes unreferenced chunks in two steps: identify and collect them in the first step, and then permanently remove them once certain conditions are met.
-
-## Two-Step Fossil Collection
-
-Interestingly, the two-step fossil collection algorithm hinges on a basic file operation supported almost universally, *file renaming*.
-When the deletion procedure identifies a chunk not referenced by any known snapshots, instead of deleting the chunk file
-immediately, it changes the name of the chunk file (and possibly moves it to a different directory).
-A chunk that has been renamed is called a *fossil*.
-
-The fossil still exists in the file storage.  Two rules are enforced regarding the access of fossils:
-
-* A restore, list, or check procedure that reads existing backups can read the fossil if the original chunk cannot be found.
-* A backup procedure does not check the existence of a fossil. That is, it must upload a chunk if it cannot find the chunk, even if an equivalent fossil exists.
-
-In the first step of the deletion procedure, called the *fossil collection* step, the names of all identified fossils will
-be saved in a fossil collection file. The deletion procedure then exits without performing further actions. This step has not effectively changed any chunk references due to the first fossil access rule.  If a backup procedure references a chunk after it is marked as a fossil, a new chunk will be uploaded because of the second fossil access rule, as shown in Figure 1.
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/fossil_collection_1.png?raw=true" 
-       alt="Reference after Rename"/>
-</p>
-
-The second step, called the *fossil deletion* step, will permanently delete fossils, but only when two conditions are met:
-
-* For each snapshot id, there is a new snapshot that was not seen by the fossil collection step
-* The new snapshot must finish after the fossil collection step
-
-The first condition guarantees that if a backup procedure references a chunk before the deletion procedure turns it into a fossil, the reference will be detected in the fossil deletion step which will then turn the fossil back into a normal chunk.
-
-The second condition guarantees that any backup procedure unknown to the fossil deletion step can start only after the fossil collection step finishes.  Therefore, if it references a chunk that was identified as fossil in the fossil collection step, it should observe the fossil, not the chunk, so it will upload a new chunk, according to the second fossil access rule.
-
-Therefore, if a backup procedure references a chunk before the chunk is marked a fossil, the fossil deletion step will not
-delete the chunk until it sees that backup procedure finishes (as indicated by the appearance of a new snapshot file uploaded to the storage).  This ensures that scenarios depicted in Figure 2 will never happen.
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/fossil_collection_2.png?raw=true" 
-       alt="Reference before Rename"/>
-</p>
-
-
-## Snapshot Format
-
-A snapshot file is a file that the backup procedure uploads to the file storage after it finishes splitting files into
-chunks and uploading all new chunks. It mainly contains metadata for the backup overall, metadata for all the files,
-and chunk references for each file. Here is an example snapshot file for a repository containing 3 files (file1, file2,
-and dir1/file3):
-
-```json
-{
-  "id": "host1",
-  "revision": 1,
-  "tag": "first",
-  "start_time": 1455590487,
-  "end_time": 1455590487,
-  "files": [
-    {
-      "path": "file1",
-      "content": "0:0:2:6108",
-      "hash": "a533c0398194f93b90bd945381ea4f2adb0ad50bd99fd3585b9ec809da395b51",
-      "size": 151901,
-      "time": 1455590487,
-      "mode": 420
-    },
-    {
-      "path": "file2",
-      "content": "2:6108:3:7586",
-      "hash": "f6111c1562fde4df9c0bafe2cf665778c6e25b49bcab5fec63675571293ed644",
-      "size": 172071,
-      "time": 1455590487,
-      "mode": 420
-    },
-    {
-      "path": "dir1/",
-      "size": 102,
-      "time": 1455590487,
-      "mode": 2147484096
-    },
-    {
-      "path": "dir1/file3",
-      "content": "3:7586:4:1734",
-      "hash": "6bf9150424169006388146908d83d07de413de05d1809884c38011b2a74d9d3f",
-      "size": 118457,
-      "time": 1455590487,
-      "mode": 420
-    }
-  ],
-  "chunks": [
-    "9f25db00881a10a8e7bcaa5a12b2659c2358a579118ea45a73c2582681f12919",
-    "6e903aace6cd05e26212fcec1939bb951611c4179c926351f3b20365ef2c212f",
-    "4b0d017bce5491dbb0558c518734429ec19b8a0d7c616f68ddf1b477916621f7",
-    "41841c98800d3b9faa01b1007d1afaf702000da182df89793c327f88a9aba698",
-    "7c11ee13ea32e9bb21a694c5418658b39e8894bbfecd9344927020a9e3129718"
-  ],
-  "lengths": [
-    64638,
-    81155,
-    170593,
-    124309,
-    1734
-  ] 
-}
-```
-
-When Duplicacy splits a file in chunks using the variable-size chunking algorithm, if the end of a file is reached and yet the boundary marker for terminating a chunk
-hasn't been found, the next file, if there is one, will be read in and the chunking algorithm continues. It is as if all 
-files were packed into a big tar file which is then split into chunks.
-
-The *content* field of a file indicates the indexes of starting and ending chunks and the corresponding offsets. For
-instance, *file1* starts at chunk 0 offset 0 while ends at chunk 2 offset 6108, immediately followed by *file2*.
-
-The backup procedure can run in one of two modes. In the default quick mode, only modified or new files are scanned. Chunks only
-referenced by old files that have been modified are removed from the chunk sequence, and then chunks referenced by new 
-files are appended. Indices for unchanged files need to be updated too.
-
-In the safe mode (enabled by the -hash option), all files are scanned and the chunk sequence is regenerated.
-
-The length sequence stores the lengths for all chunks, which are needed when calculating some statistics such as the total
-length of chunks. For a repository containing a large number of files, the size of the snapshot file can be tremendous. 
-To make the situation worse, every time a big snapshot file would have been uploaded even if only a few files have been changed since
-last backup. To save space, the variable-size chunking algorithm is also applied to the three dynamic fields of a snapshot
-file, *files*, *chunks*, and *lengths*.
-
-Chunks produced during this step are deduplicated and uploaded in the same way as regular file chunks. The final snapshot file
-contains sequences of chunk hashes and other fixed size fields:
-
-```json
-{
-  "id": "host1",
-  "revision": 1,
-  "start_time": 1455590487,
-  "tag": "first",
-  "end_time": 1455590487,
-  "file_sequence": [
-    "21e4c69f3832e32349f653f31f13cefc7c52d52f5f3417ae21f2ef5a479c3437",
-  ],
-  "chunk_sequence": [
-    "8a36ffb8f4959394fd39bba4f4a464545ff3dd6eed642ad4ccaa522253f2d5d6"
-  ],
-  "length_sequence": [
-    "fc2758ae60a441c244dae05f035136e6dd33d3f3a0c5eb4b9025a9bed1d0c328"
-  ]
-}
-```
-
-In the extreme case where the repository has not been modified since last backup, a new backup procedure will not create any new chunks,
-as shown by the following output from a real use case:
-
-```
-$ duplicacy backup -stats
-Storage set to sftp://gchen@192.168.1.100/Duplicacy
-Last backup at revision 260 found
-Backup for /Users/gchen/duplicacy at revision 261 completed
-Files: 42367 total, 2,204M bytes; 0 new, 0 bytes
-File chunks: 447 total, 2,238M bytes; 0 new, 0 bytes, 0 bytes uploaded
-Metadata chunks: 6 total, 11,753K bytes; 0 new, 0 bytes, 0 bytes uploaded
-All chunks: 453 total, 2,249M bytes; 0 new, 0 bytes, 0 bytes uploaded
-Total running time: 00:00:05
-```
-
-## Encryption
-
-When encryption is enabled (by the -e option with the *init* or *add* command), Duplicacy will generate 4 random 256 bit keys:
-
-* *Hash Key*: for generating a chunk hash from the content of a chunk
-* *ID Key*: for generating a chunk id from a chunk hash
-* *Chunk Key*: for encrypting chunk files
-* *File Key*: for encrypting non-chunk files such as snapshot files.
-
-Here is a diagram showing how these keys are used:
-
-<p align="center">
-  <img src="https://github.com/gilbertchen/duplicacy-beta/blob/master/images/duplicacy_encryption.png?raw=true" 
-       alt="encryption"/>
-</p>
-
-Chunk hashes are used internally and stored in the snapshot file.  They are never exposed unless the snapshot file is decrypted.  Chunk ids are used as the file names for the chunks and therefore exposed.  When the *cat* command is used to print out a snapshot file, the chunk hashes stored in the snapshot file will be converted into chunk ids first which are then displayed instead.
-
-Chunk content is encrypted by AES-GCM, with an encryption key that is the HMAC-SHA256 of the chunk Hash with the *Chunk Key* as the secret key.
-
-The snapshot is encrypted by AES-GCM too, using an encrypt key that is the HMAC-SHA256 of the file path with the *File Key* as the secret key.
-
-These four random keys are saved in a file named 'config' in the storage, encrypted with a master key derived from the PBKDF2 function on
-the storage password chosen by the user.
+All documentation has been moved to our wiki page:
 
+* [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication)
+* [Snapshot Format](https://github.com/gilbertchen/duplicacy/wiki/Snapshot-Format)
+* [Encryption](https://github.com/gilbertchen/duplicacy/wiki/Encryption)

GUIDE.md

@@ -1,515 +1,20 @@
-# Duplicacy User Guide
-
-## Commands
-
-#### Init
-
-```
-SYNOPSIS:
-   duplicacy init - Initialize the storage if necessary and the current directory as the repository
-
-USAGE:
-   duplicacy init [command options] <snapshot id> <storage url>
-
-OPTIONS:
-   -encrypt, -e                    encrypt the storage with a password
-   -chunk-size, -c 4M              the average size of chunks
-   -max-chunk-size, -max 16M       the maximum size of chunks (defaults to chunk-size * 4)
-   -min-chunk-size, -min 1M        the minimum size of chunks (defaults to chunk-size / 4)
-   -pref-dir <preference directory path> 	Specify alternate location for .duplicacy preferences directory
-```
-
-The *init* command first connects to the storage specified by the storage URL.  If the storage has been already been
-initialized before, it will download the storage configuration (stored in the file named *config*) and ignore the options provided in the command line.  Otherwise, it will create the configuration file from the options and upload the file.
-
-The initialized storage will then become the default storage for other commands if the -storage option is not specified
-for those commands.  This default storage actually has a name, *default*.
-
-After that, it will prepare the the current working directory as the repository to be backed up.  Under the hood, it will create a directory
-named *.duplicacy* in the repository and put a file named *preferences* that stores the snapshot id and encryption and storage options.
-
-The snapshot id is an id used to distinguish different repositories connected to the same storage.  Each repository must have a unique snapshot id.  A snapshot id must contain only characters valid in Linux and Windows paths (alphabet, digits, underscore, dash, etc), but cannot include `/`, `\`, or `@`.
-
-The -e option controls whether or not encryption will be enabled for the storage.  If encryption is enabled, you will be prompted to enter a storage password.
-
-The three chunk size parameters are passed to the variable-size chunking algorithm.  Their values are important to the overall performance, especially for cloud storages.  If the chunk size is too small, a lot of overhead will be in sending requests and receiving responses.  If the chunk size is too large, the effect of deduplication will be less obvious as more data will need to be transferred with each chunk.
-
-The -pref-dir controls the location of the preferences directory. If not specified, a directory named .duplicacy is created in the repository. If specified, it must point to a non-existing directory. The directory is created and a .duplicacy file is created in the repository. The .duplicacy file contains the absolute path name to the preferences directory.
-
-Once a storage has been initialized with these parameters, these parameters cannot be modified any more.
-
-#### Backup
-
-```
-SYNOPSIS:
-   duplicacy backup - Save a snapshot of the repository to the storage
-
-USAGE:
-   duplicacy backup [command options]
-
-OPTIONS:
-   -hash                      detect file differences by hash (rather than size and timestamp)
-   -t <tag>                   assign a tag to the backup
-   -stats                     show statistics during and after backup
-   -threads <n>               number of uploading threads
-   -limit-rate <kB/s>         the maximum upload rate (in kilobytes/sec)   
-   -vss                       enable the Volume Shadow Copy service (Windows only)
-   -storage <storage name>    backup to the specified storage instead of the default one
-```
-
-The *backup* command creates a snapshot of the repository and uploads it to the storage.  If -hash is not provided,
-it will upload new or modified files since last backup by comparing file sizes and timestamps.
-Otherwise, every file is scanned to detect changes.
-
-You can assign a tag to the snapshot so that later you can refer to it by tag in other commands.
-
-If the -stats option is specified, statistical information such as transfer speed, the number of chunks will be displayed
-throughout the backup procedure.
-
-The -threads option can be used to specify more than one thread to upload chunks.
-
-The -limit-rate option sets a cape on the maximum upload rate.
-
-The -vss option works on Windows only to turn on the Volume Shadow Copy service such that files opened by other
-processes with exclusive locks can be read as usual.
-
-When the repository can have multiple storages (added by the *add* command), you can select the storage to back up to
-by giving a storage name.
-
-You can specify patterns to include/exclude files by putting them in a file named *.duplicacy/filters*.  Please refer to the [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#includeexclude-patterns) section for how to specify the patterns.
-
-#### Restore
-```
-SYNOPSIS:
-   duplicacy restore - Restore the repository to a previously saved snapshot
-
-USAGE:
-   duplicacy restore [command options] [--] [pattern] ...
-
-OPTIONS:
-   -r <revision>            the revision number of the snapshot (required)
-   -hash                    detect file differences by hash (rather than size and timestamp)
-   -overwrite               overwrite existing files in the repository
-   -delete                  delete files not in the snapshot
-   -stats                   show statistics during and after restore
-   -threads <n>             number of downloading threads
-   -limit-rate <kB/s>	    the maximum download rate (in kilobytes/sec)
-   -storage <storage name>  restore from the specified storage instead of the default one
-```
-
-The *restore* command restores the repository to a previous revision.  By default the restore procedure will treat
-files that have the same sizes and timestamps as those in the snapshot as unchanged files, but with the -hash option, every file will be fully scanned to make sure they are in fact unchanged.
-
-By default the restore procedure will not overwriting existing files, unless the -overwrite option is specified.
-
-The -delete option indicates that files not in the snapshot will be removed.
-
-If the -stats option is specified, statistical information such as transfer speed, number of chunks will be displayed
-throughout the restore procedure.
-
-The -threads option can be used to specify more than one thread to download chunks.
-
-The -limit-rate option sets a cape on the maximum upload rate.
-
-When the repository can have multiple storages (added by the *add* command), you can select the storage to restore from by specifying the storage name.
-
-Unlike the *backup* procedure that reading the include/exclude patterns from a file, the *restore* procedure reads them
-from the command line.  If the patterns can cause confusion to the command line argument parser, -- should be prepended to
-the patterns.  Please refer to the [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#includeexclude-patterns) section for how to specify patterns.
-
-
-#### List
-```
-SYNOPSIS:
-   duplicacy list - List snapshots
-
-USAGE:
-   duplicacy list [command options]  
-
-OPTIONS:
-   -all, -a                    list snapshots with any id
-   -id <snapshot id>           list snapshots with the specified id rather than the default one
-   -r <revision> [+]           the revision number of the snapshot
-   -t <tag>                    list snapshots with the specified tag
-   -files                      print the file list in each snapshot
-   -chunks                     print chunks in each snapshot or all chunks if no snapshot specified
-   -reset-password             take passwords from input rather than keychain/keyring or env
-   -storage <storage name>     retrieve snapshots from the specified storage
-```
-
-The *list* command lists information about specified snapshots.  By default it will list snapshots created from the
-current repository, but you can list all snapshots stored in the storage by specifying the -all option, or list snapshots
-with a different snapshot id using the -id option, and/or snapshots with a particular tag with the -t option.
-
-The revision number is a number assigned to the snapshot when it is being created.  This number will keep increasing
-every time a new snapshot is created from a repository.  You can refer to snapshots by their revision numbers using 
-the -r option, which either takes a single revision number (-r 123) or a range (-r 123-456).
-There can be multiple -r options.
-
-If -files is specified, for each snapshot to be listed, this command will also print information about every file
-contained in the snapshot.
-
-If -chunks is specified, the command will also print out every chunk the snapshot references.
-
-The -reset-password option is used to reset stored passwords and to allow passwords to be entered again.  Please refer to the [Managing Passwords](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#managing-passwords) section for more information.
-
-When the repository can have multiple storages (added by the *add* command), you can specify the storage to list
-by specifying the storage name.
-
-#### Check
-```
-SYNOPSIS:
-   duplicacy check - Check the integrity of snapshots
-
-USAGE:
-   duplicacy check [command options]
-
-OPTIONS:
-   -all, -a                 check snapshots with any id
-   -id <snapshot id>        check snapshots with the specified id rather than the default one
-   -r <revision> [+]        the revision number of the snapshot
-   -t <tag>                 check snapshots with the specified tag
-   -fossils                 search fossils if a chunk can't be found
-   -resurrect               turn referenced fossils back into chunks
-   -files                   verify the integrity of every file
-   -stats                   show deduplication statistics (imply -all and all revisions)
-   -storage <storage name>  retrieve snapshots from the specified storage```
-```
-The *check* command checks, for each specified snapshot, that all referenced chunks exist in the storage.
-
-By default the *check* command will check snapshots created from the
-current repository, but you can check all snapshots stored in the storage at once by specifying the -all option, or
-snapshots from a different repository using the -id option, and/or snapshots with a particular tag with the -t option.
-
-The revision number is a number assigned to the snapshot when it is being created.  This number will keep increasing
-every time a new snapshot is created from a repository.  You can refer to snapshots by their revision numbers using 
-the -r option, which either takes a single revision number (-r 123) or a range (-r 123-456).
-There can be multiple -r options.
-
-By default the *check* command only verifies the existence of chunks.  To verify the full integrity of a snapshot,
-you should specify the -files option, which will download chunks and compute file hashes in memory, to
-make sure that all hashes match.
-
-By default the *check* command does not find fossils. If the -fossils option is specified, it will find
-the fossil if the referenced chunk does not exist.  if the -resurrect option is specified, it will turn the fossil back into a chunk.
-
-When the repository can have multiple storages (added by the *add* command), you can specify the storage to check
-by specifying the storage name.
-
-
-#### Cat
-```
-SYNOPSIS:
-   duplicacy cat - Print to stdout the specified file, or the snapshot content if no file is specified
-
-USAGE:
-   duplicacy cat [command options] [<file>]
-
-OPTIONS:
-   -id <snapshot id>        retrieve from the snapshot with the specified id
-   -r <revision>            the revision number of the snapshot
-   -storage <storage name>  retrieve the file from the specified storage
-```
-
-The *cat* command prints a file or the entire snapshot content if no file is specified.
-
-The file must be specified with a path relative to the repository.
-
-You can specify a different snapshot id rather than the default id.
-
-The -r option is optional.  If not specified, the latest revision will be selected.
-
-You can use the -storage option to select a different storage other than the default one.
-
-#### Diff
-```
-SYNOPSIS:
-   duplicacy diff - Compare two snapshots or two revisions of a file
-
-USAGE:
-   duplicacy diff [command options] [<file>]
-
-OPTIONS:
-   -id <snapshot id>        diff with the snapshot with the specified id
-   -r <revision> [+]        the revision number of the snapshot
-   -hash                    compute the hashes of on-disk files
-   -storage <storage name>  retrieve files from the specified storage
-```
-The *diff* command compares the same file in two different snapshots if a file is given, otherwise compares the
-two snapshots.
-
-The file must be specified with a path relative to the repository.
-
-You can specify a different snapshot id rather than the default snapshot id.
-
-If only one revision is given by -r, the right hand side of the comparison will be the on-disk file.
-The -hash option can then instruct this command to compute the hash of the file. 
-
-You can use the -storage option to select a different storage other than the default one.
-
-#### History
-```
-SYNOPSIS:
-   duplicacy history - Show the history of a file
-
-USAGE:
-   duplicacy history [command options] <file>
-
-OPTIONS:
-   -id <snapshot id>        find the file in the snapshot with the specified id
-   -r <revision> [+]        show history of the specified revisions
-   -hash                    show the hash of the on-disk file
-   -storage <storage name>  retrieve files from the specified storage
-```
-
-The *history* command shows how the hash, size, and timestamp of a file change over the specified set of revisions.
-
-You can specify a different snapshot id rather than the default snapshot id, and multiple -r options to specify the
-set of revisions.
-
-The -hash option is to compute the hash of the on-disk file.  Otherwise, only the size and timestamp of the on-disk
-file will be included.
-
-You can use the -storage option to select a different storage other than the default one.
-
-#### Prune
-```
-SYNOPSIS:
-   duplicacy prune - Prune snapshots by revision, tag, or retention policy
-
-USAGE:
-   duplicacy prune [command options]
-
-OPTIONS:
-   -id <snapshot id>        delete snapshots with the specified id instead of the default one
-   -all, -a                 match against all snapshot IDs
-   -r <revision> [+]        delete snapshots with the specified revisions
-   -t <tag> [+]             delete snapshots with the specified tags
-   -keep <n:m> [+]          keep 1 snapshot every n days for snapshots older than m days
-   -exhaustive              find all unreferenced chunks by scanning the storage
-   -exclusive               assume exclusive access to the storage (disable two-step fossil collection)
-   -dry-run, -d             show what would have been deleted
-   -delete-only             delete fossils previously collected (if deletable) and don't collect fossils
-   -collect-only            identify and collect fossils, but don't delete fossils previously collected
-   -ignore <id> [+]         ignore the specified snapshot id when deciding if fossils can be deleted
-   -storage <storage name>  prune snapshots from the specified storage
-```
-
-The *prune* command implements the two-step fossil collection algorithm.  It will first find fossil collection files
-from previous runs and check if contained fossils are eligible for permanent deletion (the fossil deletion step).  Then it
-will search for snapshots to be deleted, mark unreferenced chunks as fossils (by renaming) and save them in a new fossil
-collection file stored locally (the fossil collection step).
-
-If a snapshot id is specified, that snapshot id will be used instead of the default one.  The -a option will find
-snapshots with any id.  Snapshots to be deleted can be specified by revision numbers, by a tag, by retention policies,
-or by any combination of them.
-
-The retention policies are specified by the -keep option, which accepts an argument in the form of two numbers *n:m*, where *n* indicates the number of days between two consecutive snapshots to keep, and *m* means that the policy only applies to snapshots at least *m* day old.  If *n* is zero, any snapshots older than *m* days will be removed.
-
-Here are a few sample retention policies:
-
-```sh
-$ duplicacy prune -keep 1:7       # Keep 1 snapshot per day for snapshots older than 7 days
-$ duplicacy prune -keep 7:30      # Keep 1 snapshot every 7 days for snapshots older than 30 days
-$ duplicacy prune -keep 30:180    # Keep 1 snapshot every 30 days for snapshots older than 180 days
-$ duplicacy prune -keep 0:360     # Keep no snapshots older than 360 days
-```
-
-Multiple -keep options must be sorted by their *m* values in decreasing order.  For instance, to combine the above policies into one line, it would become:
-
-```sh
-$ duplicacy prune -keep 0:360 -keep 30:180 -keep 7:30 -keep 1:7
-```
-
-The -exhaustive option will scan the list of all chunks in the storage, therefore it will find not only 
-unreferenced chunks from deleted snapshots, but also chunks that become unreferenced for other reasons, such as
-those from an incomplete backup.  It will also find any file that does not look like a chunk file.
-In contrast, a default *prune* command will only identify 
-chunks referenced by deleted snapshots but not any other snapshots.
-
-The -exclusive option will assume that no other clients are accessing the storage, effectively disabling the 
-*two-step fossil collection* algorithm.  With this option, the *prune* command will immediately remove unreferenced chunks.
-
-The -dryrun option is used to test what changes the *prune* command would have done.  It is guaranteed not to make
-any changes on the storage, not even creating the local fossil collection file.  The following command checks if the
-chunk directory is clean (i.e., if there are any unreferenced chunks, temporary files, or anything else):
-
-```
-$ duplicacy prune -d -exclusive -exhaustive    #  Prints out nothing if the chunk directory is clean
-```
-
-The -delete-only option will skip the fossil collection step, while the -collect-only option will skip the fossil deletion step.
-
-For fossils collected in the fossil collection step to be eligible for safe deletion in the fossil deletion step, at least
-one new snapshot from *each* snapshot id must be created between two runs of the *prune* command.  However, some repository
-may not be set up to back up with a regular schedule, and thus literally blocking other repositories from deleting any fossils.  Duplicacy by default will ignore repositories that have no new backup in the past 7 days.  It also provide an
--ignore option that can be used to skip certain repositories when deciding the deletion criteria.
-
-You can use the -storage option to select a different storage other than the default one.
-
-
-#### Password
-```
-SYNOPSIS:
-   duplicacy password - Change the storage password
-
-USAGE:
-   duplicacy password [command options]
-
-OPTIONS:
-   -storage <storage name>  change the password used to access the specified storage
-```
-
-The *password* command decrypts the storage configuration file *config* using the old password, and re-encrypts the file
-using a new password.  It does not change all the encryption keys used to encrypt and decrypt chunk files,
-snapshot files, etc.
-
-You can specify the storage to change the password for when working with multiple storages.
-
-
-#### Add
-```
-SYNOPSIS:
-   duplicacy add - Add an additional storage to be used for the existing repository
-
-USAGE:
-   duplicacy add [command options] <storage name> <snapshot id> <storage url>
-
-OPTIONS:
-   -encrypt, -e                    Encrypt the storage with a password
-   -chunk-size, -c 4M              the average size of chunks
-   -max-chunk-size, -max 16M       the maximum size of chunks (defaults to chunk-size * 4)
-   -min-chunk-size, -min 1M        the minimum size of chunks (defaults to chunk-size / 4)
-   -compression-level, -l <level>  compression level (defaults to -1)
-   -copy <storage name>            make the new storage copy-compatible with an existing one
-```
-
-The *add* command connects another storage to the current repository.  Like the *init* command, if the storage has not
-been initialized before, a storage configuration file derived from the command line options will be uploaded, but those
-options will be ignored if the configuration file already exists in the storage.
-
-A unique storage name must be given in order to distinguish it from other storages.
-
-The -copy option is required if later you want to copy snapshots between this storage and another storage.
-Two storages are copy-compatible if they have the same average chunk size, the same maximum chunk size,
-the same minimum chunk size, the same chunk seed (used in calculating the rolling hash in the variable-size chunks
-algorithm), and the same hash key.  If the -copy option is specified, these parameters will be copied from
-the existing storage rather than from the command line.
-
-#### Set
-```
-SYNOPSIS:
-   duplicacy set - Change the options for the default or specified storage
-
-USAGE:
-   duplicacy set [command options]
-
-OPTIONS:
-   -encrypt, e[=true]       encrypt the storage with a password
-   -no-backup[=true]        backup to this storage is prohibited
-   -no-restore[=true]       restore from this storage is prohibited
-   -no-save-password[=true] don't save password or access keys to keychain/keyring
-   -key                     add a key/password whose value is supplied by the -value option
-   -value                   the value of the key/password
-   -storage <storage name>  use the specified storage instead of the default one
-```
-
-The *set* command changes the options for the specified storage.
-
-The -e option turns on the storage encryption.  If specified as -e=false, it turns off the storage encryption.
-
-The -no-backup option will not allow backups from this repository to be created.
-
-The -no-restore option will not allow restoring this repository to a different revision.
-
-The -no-save-password option will require every password or token to be entered every time and not saved anywhere.
-
-The -key and -value options are used to store (in plain text) access keys or tokens need by various storages.  Please
-refer to the [Managing Passwords](https://github.com/gilbertchen/duplicacy-beta/blob/master/GUIDE.md#managing-passwords) section for more details.
-
-You can select a storage to change options for by specifying a storage name.
-
-
-#### Copy
-```
-SYNOPSIS:
-   duplicacy copy - Copy snapshots between compatible storages
-
-USAGE:
-   duplicacy copy [command options]
-
-OPTIONS:
-   -id <snapshot id>     copy snapshots with the specified id instead of all snapshot ids
-   -r <revision> [+]     copy snapshots with the specified revisions
-   -from <storage name>  copy snapshots from the specified storage
-   -to <storage name>    copy snapshots to the specified storage
-```
-
-The *copy* command copies snapshots from one storage to another storage.  They must be copy-compatible, i.e., some
-configuration parameters must be the same.  One storage must be initialized with the -copy option provided by the *add* command.
-
-Instead of copying all snapshots, you can specify a set of snapshots to copy by giving the -r options.  The *copy* command
-preserves the revision numbers, so if a revision number already exists on the destination storage the command will fail.
-
-If no -from option is given, the snapshots from the default storage will be copied.  The -to option specified the
-destination storage and is required.
-
-## Include/Exclude Patterns
-
-An include pattern starts with +, and an exclude pattern starts with -.  Patterns may contain wildcard characters * which matches a path string of any length, and ? matches a single character.  Note that both * and ? will match any character including the path separator /.
-
-When matching a path against a list of patterns, the path is compared with the part after + or -, one pattern at a time.  Therefore, the order of the patterns is significant.  If a match with an include pattern is found, the path is said to be included without further comparisons.  If a match with an exclude pattern is found, the path is said to be excluded without further comparison.  If a match is not found, the path will be excluded if all patterns are include patterns, but included otherwise.
-
-Patterns ending with a / apply to directories only, and patterns not ending with a / apply to files only.  When a directory is excluded, all files and subdirectires under it will also be excluded.  Note that the path separator is always /, even on Windows.
-
-The following pattern list includes only files under the directory foo/ but not files under the subdirectory foo/bar:
-
-```
--foo/bar/
-+foo/*
--*
-```
-
-For the *backup* command, the include/exclude patterns are read from a file named *filters* under the *.duplicacy* directory.
-
-For the *restore* command, the include/exclude patterns are specified as the command line arguments.
-
-
-## Managing Passwords
-
-Duplicacy will attempt to retrieve in three ways the storage password and the storage-specific access tokens/keys.
-
-* If a secret vault service is available, Duplicacy will store passwords/keys entered by the user in such a secret vault and later retrieve them when needed.  On Mac OS X it is Keychain, and on Linux it is gnome-keyring.  On Windows the passwords/keys are encrypted and decrypted by the Data Protection API, and encrypted passwords/keys are stored in the file *.duplicacy/keyring*.  However, if the -no-save-password option is specified for the storage, then Duplicacy will not save passwords this way.
-* If an environment variable for a password is provided, Duplicacy will always take it.  The table below shows the name of the environment variable for each kind of password.  Note that if the storage is not the default one, the storage name will be included in the name of the environment variable.
-* If a matching key and its value are saved to the preference file (.duplicacy/preferences) by the *set* command, the value will be used as the password.  The last column in the table below lists the name of the preference key for each type of password.
-
-| password type | environment variable (default storage) | environment variable (non-default storage) | key in preferences  |
-|:----------------:|:----------------:|:----------------:|:----------------:|
-| storage password | DUPLICACY_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_PASSWORD | password |
-| sftp password | DUPLICACY_SSH_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_SSH_PASSWORD | ssh_password |
-| sftp key file | DUPLICACY_SSH_KEY_FILE | DUPLICACY_&lt;STORAGENAME&gt;_SSH_KEY_FILE | ssh_key_file |
-| Dropbox Token | DUPLICACY_DROPBOX_TOKEN | DUPLICACY_&lt;STORAGENAME>&gt;_DROPBOX_TOKEN | dropbox_token |
-| S3 Access ID | DUPLICACY_S3_ID | DUPLICACY_&lt;STORAGENAME&gt;_S3_ID | s3_id |
-| S3 Secret Key | DUPLICACY_S3_SECRET | DUPLICACY_&lt;STORAGENAME&gt;_S3_SECRET | s3_secret |
-| BackBlaze Account ID | DUPLICACY_B2_ID | DUPLICACY_&lt;STORAGENAME&gt;_B2_ID | b2_id |
-| Backblaze Application Key | DUPLICACY_B2_KEY | DUPLICACY_&lt;STORAGENAME&gt;_B2_KEY | b2_key |
-| Azure Access Key | DUPLICACY_AZURE_KEY | DUPLICACY_&lt;STORAGENAME&gt;_AZURE_KEY | azure_key |
-| Google Drive Token File | DUPLICACY_GCD_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_GCD_TOKEN | gcd_token |
-| Microsoft OneDrive Token File | DUPLICACY_ONE_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_ONE_TOKEN | one_token |
-| Hubic Token File | DUPLICACY_HUBIC_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_HUBIC_TOKEN | hubic_token |
-
-Note that the passwords stored in the environment variable and the preference need to be in plaintext and thus are insecure and should be avoided whenever possible.
-
-## Cache
-
-Duplicacy maintains a local cache under the `.duplicacy/cache` folder in the repository.  Only snapshot chunks may be stored in this local cache, and file chunks are never cached.
-
-At the end of a backup operation, Duplicacy will clean up the local cache in such a way that only chunks composing the snapshot file from the last backup will stay in the cache.  All other chunks will be removed from the cache.  However, if the *prune* command has been run before (which will leave a the `.duplicacy/collection` folder in the repository, then the *backup* command won't perform any cache cleanup and instead defer that to the *prune* command.
-
-At the end of a prune operation, Duplicacy will remove all chunks from the local cache except those composing the snapshot file from the last backup (those that would be kept by the *backup* command), as well as chunks that contain information about chunks referenced by *all* backups from *all* repositories connected to the same storage url.  
-
-Other commands, such as *list*, *check*, does not clean up the local cache at all, so the local cache may keep growing if many of these commands run consecutively.  However, once a *backup* or a *prune* command is invoked, the local cache should shrink to its normal size.
-
-## Scripts
-
-You can instruct Duplicacy to run a script before or after executing a command.  For example, if you create a bash script with the name *pre-prune* under the *.duplicacy/scripts* directory, this bash script will be run before the *prune* command starts.  A script named *post-prune* will be run after the *prune* command finishes.  This rule applies to all commands except *init*.
+All documentation has been moved to our wiki page:
+
+* Commands
+    * [init](https://github.com/gilbertchen/duplicacy/wiki/init)
+    * [backup](https://github.com/gilbertchen/duplicacy/wiki/backup)
+    * [restore](https://github.com/gilbertchen/duplicacy/wiki/restore)
+    * [list](https://github.com/gilbertchen/duplicacy/wiki/list)
+    * [check](https://github.com/gilbertchen/duplicacy/wiki/check)
+    * [prune](https://github.com/gilbertchen/duplicacy/wiki/prune)
+    * [cat](https://github.com/gilbertchen/duplicacy/wiki/cat)
+    * [history](https://github.com/gilbertchen/duplicacy/wiki/history)
+    * [diff](https://github.com/gilbertchen/duplicacy/wiki/diff)
+    * [password](https://github.com/gilbertchen/duplicacy/wiki/password)
+    * [add](https://github.com/gilbertchen/duplicacy/wiki/add)
+    * [set](https://github.com/gilbertchen/duplicacy/wiki/set)
+    * [copy](https://github.com/gilbertchen/duplicacy/wiki/copy)
+* [Include/Exclude Patterns](https://github.com/gilbertchen/duplicacy/wiki/Include-Exclude-Patterns)
+* [Managing Passwords](https://github.com/gilbertchen/duplicacy/wiki/Managing-Passwords)
+* [Cache](https://github.com/gilbertchen/duplicacy/wiki/Cache)
+* [Pre-Command and Post-Command Scripts](https://github.com/gilbertchen/duplicacy/wiki/Pre-Command-and-Post-Command-Scripts)

Gopkg.lock

@@ -0,0 +1,230 @@
+# This file is autogenerated, do not edit; changes may be undone by the next 'dep ensure'.
+
+
+[[projects]]
+  name = "cloud.google.com/go"
+  packages = ["compute/metadata","iam","internal","internal/optional","internal/version","storage"]
+  revision = "2d3a6656c17a60b0815b7e06ab0be04eacb6e613"
+  version = "v0.16.0"
+
+[[projects]]
+  name = "github.com/Azure/azure-sdk-for-go"
+  packages = ["version"]
+  revision = "b7fadebe0e7f5c5720986080a01495bd8d27be37"
+  version = "v14.2.0"
+
+[[projects]]
+  name = "github.com/Azure/go-autorest"
+  packages = ["autorest","autorest/adal","autorest/azure","autorest/date"]
+  revision = "0ae36a9e544696de46fdadb7b0d5fb38af48c063"
+  version = "v10.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/aryann/difflib"
+  packages = ["."]
+  revision = "e206f873d14a916d3d26c40ab667bca123f365a3"
+
+[[projects]]
+  name = "github.com/aws/aws-sdk-go"
+  packages = ["aws","aws/awserr","aws/awsutil","aws/client","aws/client/metadata","aws/corehandlers","aws/credentials","aws/credentials/ec2rolecreds","aws/credentials/endpointcreds","aws/credentials/stscreds","aws/defaults","aws/ec2metadata","aws/endpoints","aws/request","aws/session","aws/signer/v4","internal/shareddefaults","private/protocol","private/protocol/query","private/protocol/query/queryutil","private/protocol/rest","private/protocol/restxml","private/protocol/xml/xmlutil","service/s3","service/sts"]
+  revision = "a32b1dcd091264b5dee7b386149b6cc3823395c9"
+  version = "v1.12.31"
+
+[[projects]]
+  name = "github.com/bkaradzic/go-lz4"
+  packages = ["."]
+  revision = "74ddf82598bc4745b965729e9c6a463bedd33049"
+  version = "v1.0.0"
+
+[[projects]]
+  name = "github.com/dgrijalva/jwt-go"
+  packages = ["."]
+  revision = "dbeaa9332f19a944acb5736b4456cfcc02140e29"
+  version = "v3.1.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/azure-sdk-for-go"
+  packages = ["storage"]
+  revision = "bbf89bd4d716c184f158d1e1428c2dbef4a18307"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/cli"
+  packages = ["."]
+  revision = "1de0a1836ce9c3ae1bf737a0869c4f04f28a7f98"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/go-dropbox"
+  packages = ["."]
+  revision = "90711b603312b1f973f3a5da3793ac4f1e5c2f2a"
+
+[[projects]]
+  name = "github.com/gilbertchen/go-ole"
+  packages = ["."]
+  revision = "0e87ea779d9deb219633b828a023b32e1244dd57"
+  version = "v1.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/go.dbus"
+  packages = ["."]
+  revision = "9e442e6378618c083fd3b85b703ffd202721fb17"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/goamz"
+  packages = ["aws","s3"]
+  revision = "eada9f4e8cc2a45db775dee08a2c37597ce4760a"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/gopass"
+  packages = ["."]
+  revision = "bf9dde6d0d2c004a008c27aaee91170c786f6db8"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/keyring"
+  packages = ["."]
+  revision = "8855f5632086e51468cd7ce91056f8da69687ef6"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/gilbertchen/xattr"
+  packages = ["."]
+  revision = "68e7a6806b0137a396d7d05601d7403ae1abac58"
+
+[[projects]]
+  name = "github.com/go-ini/ini"
+  packages = ["."]
+  revision = "32e4c1e6bc4e7d0d8451aa6b75200d19e37a536a"
+  version = "v1.32.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/golang/protobuf"
+  packages = ["proto","protoc-gen-go/descriptor","ptypes","ptypes/any","ptypes/duration","ptypes/timestamp"]
+  revision = "1e59b77b52bf8e4b449a57e6f79f21226d571845"
+
+[[projects]]
+  name = "github.com/googleapis/gax-go"
+  packages = ["."]
+  revision = "317e0006254c44a0ac427cc52a0e083ff0b9622f"
+  version = "v2.0.0"
+
+[[projects]]
+  name = "github.com/jmespath/go-jmespath"
+  packages = ["."]
+  revision = "0b12d6b5"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/kr/fs"
+  packages = ["."]
+  revision = "2788f0dbd16903de03cb8186e5c7d97b69ad387b"
+
+[[projects]]
+  name = "github.com/marstr/guid"
+  packages = ["."]
+  revision = "8bd9a64bf37eb297b492a4101fb28e80ac0b290f"
+  version = "v1.1.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/minio/blake2b-simd"
+  packages = ["."]
+  revision = "3f5f724cb5b182a5c278d6d3d55b40e7f8c2efb4"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/ncw/swift"
+  packages = ["."]
+  revision = "ae9f0ea1605b9aa6434ed5c731ca35d83ba67c55"
+
+[[projects]]
+  name = "github.com/pkg/errors"
+  packages = ["."]
+  revision = "645ef00459ed84a119197bfb8d8205042c6df63d"
+  version = "v0.8.0"
+
+[[projects]]
+  name = "github.com/pkg/sftp"
+  packages = ["."]
+  revision = "98203f5a8333288eb3163b7c667d4260fe1333e9"
+  version = "1.0.0"
+
+[[projects]]
+  name = "github.com/satori/go.uuid"
+  packages = ["."]
+  revision = "f58768cc1a7a7e77a3bd49e98cdd21419399b6a3"
+  version = "v1.2.0"
+
+[[projects]]
+  branch = "master"
+  name = "github.com/vaughan0/go-ini"
+  packages = ["."]
+  revision = "a98ad7ee00ec53921f08832bc06ecf7fd600e6a1"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/crypto"
+  packages = ["curve25519","ed25519","ed25519/internal/edwards25519","pbkdf2","ssh","ssh/agent","ssh/terminal"]
+  revision = "9f005a07e0d31d45e6656d241bb5c0f2efd4bc94"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/net"
+  packages = ["context","context/ctxhttp","http2","http2/hpack","idna","internal/timeseries","lex/httplex","trace"]
+  revision = "9dfe39835686865bff950a07b394c12a98ddc811"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/oauth2"
+  packages = [".","google","internal","jws","jwt"]
+  revision = "f95fa95eaa936d9d87489b15d1d18b97c1ba9c28"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/sys"
+  packages = ["unix","windows"]
+  revision = "82aafbf43bf885069dc71b7e7c2f9d7a614d47da"
+
+[[projects]]
+  branch = "master"
+  name = "golang.org/x/text"
+  packages = ["collate","collate/build","internal/colltab","internal/gen","internal/tag","internal/triegen","internal/ucd","language","secure/bidirule","transform","unicode/bidi","unicode/cldr","unicode/norm","unicode/rangetable"]
+  revision = "88f656faf3f37f690df1a32515b479415e1a6769"
+
+[[projects]]
+  branch = "master"
+  name = "google.golang.org/api"
+  packages = ["drive/v3","gensupport","googleapi","googleapi/internal/uritemplates","googleapi/transport","internal","iterator","option","storage/v1","transport/http"]
+  revision = "17b5f22a248d6d3913171c1a557552ace0d9c806"
+
+[[projects]]
+  name = "google.golang.org/appengine"
+  packages = [".","internal","internal/app_identity","internal/base","internal/datastore","internal/log","internal/modules","internal/remote_api","internal/urlfetch","urlfetch"]
+  revision = "150dc57a1b433e64154302bdc40b6bb8aefa313a"
+  version = "v1.0.0"
+
+[[projects]]
+  branch = "master"
+  name = "google.golang.org/genproto"
+  packages = ["googleapis/api/annotations","googleapis/iam/v1","googleapis/rpc/status"]
+  revision = "891aceb7c239e72692819142dfca057bdcbfcb96"
+
+[[projects]]
+  name = "google.golang.org/grpc"
+  packages = [".","balancer","balancer/roundrobin","codes","connectivity","credentials","encoding","grpclb/grpc_lb_v1/messages","grpclog","internal","keepalive","metadata","naming","peer","resolver","resolver/dns","resolver/passthrough","stats","status","tap","transport"]
+  revision = "5a9f7b402fe85096d2e1d0383435ee1876e863d0"
+  version = "v1.8.0"
+
+[solve-meta]
+  analyzer-name = "dep"
+  analyzer-version = 1
+  inputs-digest = "eff5ae2d9507f0d62cd2e5bdedebb5c59d64f70f476b087c01c35d4a5e1be72d"
+  solver-name = "gps-cdcl"
+  solver-version = 1

Gopkg.toml

@@ -0,0 +1,94 @@
+
+# Gopkg.toml example
+#
+# Refer to https://github.com/golang/dep/blob/master/docs/Gopkg.toml.md
+# for detailed Gopkg.toml documentation.
+#
+# required = ["github.com/user/thing/cmd/thing"]
+# ignored = ["github.com/user/project/pkgX", "bitbucket.org/user/project/pkgA/pkgY"]
+#
+# [[constraint]]
+#   name = "github.com/user/project"
+#   version = "1.0.0"
+#
+# [[constraint]]
+#   name = "github.com/user/project2"
+#   branch = "dev"
+#   source = "github.com/myfork/project2"
+#
+# [[override]]
+#  name = "github.com/x/y"
+#  version = "2.4.0"
+
+
+[[constraint]]
+  name = "cloud.google.com/go"
+  version = "0.16.0"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/aryann/difflib"
+
+[[constraint]]
+  name = "github.com/aws/aws-sdk-go"
+  version = "1.12.31"
+
+[[constraint]]
+  name = "github.com/bkaradzic/go-lz4"
+  version = "1.0.0"
+
+[[constraint]]
+  name = "github.com/gilbertchen/azure-sdk-for-go"
+  branch = "master"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/cli"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/go-dropbox"
+
+[[constraint]]
+  name = "github.com/gilbertchen/go-ole"
+  version = "1.2.0"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/goamz"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/gopass"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/keyring"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/gilbertchen/xattr"
+
+[[constraint]]
+  branch = "master"
+  name = "github.com/minio/blake2b-simd"
+
+[[constraint]]
+  name = "github.com/pkg/sftp"
+  version = "1.0.0"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/crypto"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/net"
+
+[[constraint]]
+  branch = "master"
+  name = "golang.org/x/oauth2"
+
+[[constraint]]
+  branch = "master"
+  name = "google.golang.org/api"

LICENSE.md

@@ -1,5 +1,8 @@
 Copyright © 2017 Acrosync LLC
 
 * Free for personal use or commercial trial
-* Non-trial commercial use requires per-user licenses available from [duplicacy.com](https://duplicacy.com/customer) at a cost of $20 per year
+* Non-trial commercial use requires per-user CLI licenses available from [duplicacy.com](https://duplicacy.com/buy) at a cost of $20 per year
+* A user is defined as the computer account that creates or edits the files to be backed up; if a backup contains files created or edited by multiple users for commercial purposes, one CLI license is required for each user
+* The computer with a valid commercial license for the GUI version may run the CLI version without a CLI license
+* CLI licenses are not required to restore or manage backups; only the backup command requires valid CLI licenses
 * Modification and redistribution are permitted, but commercial use of derivative works is subject to the same requirements of this license

README.md

@@ -1,306 +1,97 @@
 # Duplicacy: A lock-free deduplication cloud backup tool
 
-Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md).  It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services).
+Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy/wiki/Lock-Free-Deduplication).  
 
-The repository hosts source code, design documents, and binary releases of the command line version.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.
+This repository hosts source code, design documents, and binary releases of the command line version of Duplicacy.  There is also a Duplicacy GUI frontend built for Windows and Mac OS X available from https://duplicacy.com.
 
 There is a special edition of Duplicacy developed for VMware vSphere (ESXi) named [Vertical Backup](https://www.verticalbackup.com) that can back up virtual machine files on ESXi to local drives, network or cloud storages.
 
 ## Features
 
-Duplicacy currently supports major cloud storage providers (Amazon S3, Google Cloud Storage, Microsoft Azure, Dropbox, Backblaze, Google Drive, Microsoft OneDrive, and Hubic) and offers all essential features of a modern backup tool:
+There are 3 core advantages of Duplicacy over any other open-source or commercial backup tools:
 
-* Incremental backup: only back up what has been changed
-* Full snapshot : although each backup is incremental, it must behave like a full snapshot for easy restore and deletion
-* Deduplication: identical files must be stored as one copy (file-level deduplication), and identical parts from different files must be stored as one copy (block-level deduplication)
-* Encryption: encrypt not only file contents but also file paths, sizes, times, etc.
-* Deletion: every backup can be deleted independently without affecting others
-* Concurrent access: multiple clients can back up to the same storage at the same time
-* Snapshot migration: all or selected snapshots can be migrated from one storage to another
+* Duplicacy is the *only* cloud backup tool that allows multiple computers to back up to the same cloud storage, taking advantage of cross-computer deduplication whenever possible, without direct communication among them.  This feature literally turns any cloud storage server supporting only a basic set of file operations into a sophisticated deduplication-aware server.  
 
-The key idea of **Lock-Free Deduplication** can be summarized as follows:
+* Unlike other chunk-based backup tools where chunks are grouped into pack files and a chunk database is used to track which chunks are stored inside each pack file, Duplicacy takes a database-less approach where every chunk is saved independently using its hash as the file name to facilitate quick lookups.  The lack of a centralized chunk database not only makes the implementation less error-prone, but also produces a highly maintainable piece of software with plenty of room for development of new features and usability enhancements.
 
-* Use variable-size chunking algorithm to split files into chunks
-* Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
-* Apply a *two-step fossil collection* algorithm to remove chunks that become unreferenced after a backup is deleted
+* Duplicacy is fast.  While the performance wasn't the top-priority design goal, Duplicacy has been shown to outperform other backup tools by a considerable margin, as indicated by the following results obtained from a [benchmarking experiment](https://github.com/gilbertchen/benchmarking) backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other open-source backup tools.
 
-The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md) explains lock-free deduplication in detail.
+[![Comparison of Duplicacy, restic, Attic, duplicity](https://github.com/gilbertchen/duplicacy/blob/master/images/duplicacy_benchmark_speed.png "Comparison of Duplicacy, restic, Attic, duplicity")](https://github.com/gilbertchen/benchmarking)
 
 ## Getting Started
 
-<details>
-<summary>Installation</summary>
-
-Duplicacy is written in Go.  You can run the following command to build the executable (which will be created under `$GOPATH/bin`):
-
-```
-go get -u github.com/gilbertchen/duplicacy/...
-```
-
-You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-cli/releases/latest) to download the pre-built binary suitable for your platform..
-
-</details>
-
-<details>
-<summary>Commands</summary>
-
-Once you have the Duplicacy executable on your path, you can change to the directory that you want to back up (called *repository*) and run the *init* command:
-
-```
-$ cd path/to/your/repository
-$ duplicacy init mywork sftp://user@192.168.1.100/path/to/storage
-```
-
-This *init* command connects the repository with the remote storage at 192.168.1.00 via SFTP.  It will initialize the remote storage if this has not been done before.  It also assigns the snapshot id *mywork* to the repository.  This snapshot id is used to uniquely identify this repository if there are other repositories that also back up to the same storage.
-
-You can now create snapshots of the repository by invoking the *backup* command.  The first snapshot may take a while depending on the size of the repository and the upload bandwidth.  Subsequent snapshots will be much faster, as only new or modified files will be uploaded.  Each snapshot is identified by the snapshot id and an increasing revision number starting from 1.
-
-```sh
-$ duplicacy backup -stats
-```
-
-The *restore* command rolls back the repository to a previous revision:
-```sh
-$ duplicacy restore -r 1
-```
-
-
-
-Duplicacy provides a set of commands, such as list, check, diff, cat history, to manage snapshots:
-
-
-```makefile
-$ duplicacy list            # List all snapshots
-$ duplicacy check           # Check integrity of snapshots
-$ duplicacy diff            # Compare two snapshots, or the same file in two snapshots
-$ duplicacy cat             # Print a file in a snapshot
-$ duplicacy history         # Show how a file changes over time
-```
-
-
-The *prune* command removes snapshots by revisions, or tags, or retention policies:
-
-```sh
-$ duplicacy prune -r 1            # Remove the snapshot with revision number 1
-$ duplicacy prune -t quick        # Remove all snapshots with the tag 'quick'
-$ duplicacy prune -keep 1:7       # Keep 1 snapshot per day for snapshots older than 7 days
-$ duplicacy prune -keep 7:30      # Keep 1 snapshot every 7 days for snapshots older than 30 days
-$ duplicacy prune -keep 0:180     # Remove all snapshots older than 180 days
-```
-
-The first time the *prune* command is called, it removes the specified snapshots but keeps all unreferenced chunks as fossils.
-Since it uses the two-step fossil collection algorithm to clean chunks, you will need to run it again to remove those fossils from the storage:
-
-```sh
-$ duplicacy prune           # Chunks from deleted snapshots will be removed if deletion criteria are met
-```
-
-To back up to multiple storages, use the *add* command to add a new storage.  The *add* command is similar to the *init* command, except that the first argument is a storage name used to distinguish different storages:
-
-```sh
-$ duplicacy add s3 mywork s3://amazon.com/mybucket/path/to/storage
-```
-
-You can back up to any storage by specifying the storage name:
-
-```sh
-$ duplicacy backup -storage s3
-```
-
-However, snapshots created this way will be different on different storages, if the repository has been changed during two backup operations.  A better approach, is to use the *copy* command to copy specified snapshots from one storage to another:
-
-```sh
-$ duplicacy copy -r 1 -to s3   # Copy snapshot at revision 1 to the s3 storage
-$ duplicacy copy -to s3        # Copy every snapshot to the s3 storage
-```
-
-</details>
-
-
-The [User Guide](https://github.com/gilbertchen/duplicacy-cli/blob/master/GUIDE.md) contains a complete reference to
-all commands and other features of Duplicacy.
-
+* [A brief introduction](https://github.com/gilbertchen/duplicacy/wiki/Quick-Start)
+* [Command references](https://github.com/gilbertchen/duplicacy/wiki)
+* [Building from source](https://github.com/gilbertchen/duplicacy/wiki/Installation)
 
 ## Storages
 
-Duplicacy currently supports local file storage, SFTP, and many cloud storage providers.
+Duplicacy currently provides the following storage backends:
 
-<details> <summary>Local disk</summary>
+* Local disk
+* SFTP
+* Dropbox
+* Amazon S3
+* Wasabi
+* DigitalOcean Spaces
+* Google Cloud Storage
+* Microsoft Azure
+* Backblaze B2
+* Google Drive
+* Microsoft OneDrive
+* Hubic
+* OpenStack Swift
+* WebDAV (under beta testing)
+* pcloud (via WebDAV)
+* Box.com (via WebDAV)
 
-```
-Storage URL:  /path/to/storage (on Linux or Mac OS X)
-              C:\path\to\storage (on Windows)
-```
-</details>
+Please consult the [wiki page](https://github.com/gilbertchen/duplicacy/wiki/Storage-Backends) on how to set up Duplicacy to work with each cloud storage.
 
-<details> <summary>SFTP</summary> 
-
-```
-Storage URL:  sftp://username@server/path/to/storage
-```
-
-Login methods include password authentication and public key authentication.  Due to a limitation of the underlying Go SSH library, the key pair for public key authentication must be generated without a passphrase.  To work with a key that has a passphrase, you can set up SSH agent forwarding which is also supported by Duplicacy.
-
-</details>
-
-<details> <summary>Dropbox</summary>
-
-```
-Storage URL:  dropbox://path/to/storage
-```
-
-For Duplicacy to access your Dropbox storage, you must provide an access token that can be obtained in one of two ways:
-
-* Create your own app on the [Dropbox Developer](https://www.dropbox.com/developers) page, and then generate the [access token](https://blogs.dropbox.com/developers/2014/05/generate-an-access-token-for-your-own-account/)
-
-* Or authorize Duplicacy to access its app folder inside your Dropbox (following [this link](https://dl.dropboxusercontent.com/u/95866350/start_dropbox_token.html)), and Dropbox will generate the access token (which is not visible to us, as the redirect page showing the token is merely a static html hosted by Dropbox)
-
-Dropbox has two advantages over other cloud providers.  First, if you are already a paid user then to use the unused space as the backup storage is basically free.  Second, unlike other providers Dropbox does not charge bandwidth or API usage fees.
-
-</details>
-
-<details> <summary>Amazon S3</summary>
-
-```
-Storage URL:  s3://amazon.com/bucket/path/to/storage (default region is us-east-1)
-              s3://region@amazon.com/bucket/path/to/storage (other regions must be specified)
-```
-
-You'll need to input an access key and a secret key to access your Amazon S3 storage.
+For reference, the following chart shows the running times (in seconds) of backing up the [Linux code base](https://github.com/torvalds/linux) to each of those supported storages:
 
 
-</details>
+[![Comparison of Cloud Storages](https://github.com/gilbertchen/duplicacy/blob/master/images/duplicacy_benchmark_cloud.png "Comparison of Cloud Storages")](https://github.com/gilbertchen/cloud-storage-comparison)
 
-<details>  <summary>Google Cloud Storage</summary>
 
-```
-Storage URL:  gcs://bucket/path/to/storage
-```
+For complete benchmark results please visit https://github.com/gilbertchen/cloud-storage-comparison.
 
-Starting from version 2.0.0, a new Google Cloud Storage backend is added which is implemented using the [official Google client library](https://godoc.org/cloud.google.com/go/storage).  You must first obtain a credential file by [authorizing](https://duplicacy.com/gcp_start) Duplicacy to access your Google Cloud Storage account or by [downloading](https://console.cloud.google.com/projectselector/iam-admin/serviceaccounts) a service account credential file.
- 
-You can also use the s3 protocol to access Google Cloud Storage.  To do this, you must enable the [s3 interoperability](https://cloud.google.com/storage/docs/migrating#migration-simple) in your Google Cloud Storage settings and set the storage url as `s3://storage.googleapis.com/bucket/path/to/storage`.
-
-</details>
-
-<details> <summary>Microsoft Azure</summary>
-
-```
-Storage URL:  azure://account/container
-```
-
-You'll need to input the access key once prompted.
-
-</details>
-
-<details> <summary>Backblaze B2</summary>
-
-```
-Storage URL: b2://bucket
-```
-
-You'll need to input the account id and application key.
-
-Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per month), but also the fastest.  We have been working closely with their developers to leverage the full potentials provided by the B2 API in order to maximize the transfer speed.
-
-</details>
-
-<details> <summary>Google Drive</summary>
-
-```
-Storage URL: gcd://path/to/storage
-```
-
-To use Google Drive as the storage,  you first need to download a token file from https://duplicacy.com/gcd_start by
-authorizing Duplicacy to access your Google Drive, and then enter the path to this token file to Duplicacy when prompted.
-
-</details>
-
-<details> <summary>Microsoft OneDrive</summary>
-
-```
-Storage URL: one://path/to/storage
-```
-
-To use Microsoft OneDrive as the storage,  you first need to download a token file from https://duplicacy.com/one_start by
-authorizing Duplicacy to access your OneDrive, and then enter the path to this token file to Duplicacy when prompted.
-
-</details>
-
-<details> <summary>Hubic</summary>
-
-```
-Storage URL: hubic://path/to/storage
-```
-
-To use Hubic as the storage,  you first need to download a token file from https://duplicacy.com/hubic_start by
-authorizing Duplicacy to access your Hubic drive, and then enter the path to this token file to Duplicacy when prompted.
-
-Hubic offers the most free space (25GB) of all major cloud providers and there is no bandwidth charge (same as Google Drive and OneDrive), so it may be worth a try.
-
-</details>
-
-## Feature Comparison with Other Backup Tools
+## Comparison with Other Backup Tools
 
 [duplicity](http://duplicity.nongnu.org) works by applying the rsync algorithm (or more specific, the [librsync](https://github.com/librsync/librsync) library)
 to find the differences from previous backups and only then uploading the differences.  It is the only existing backup tool with extensive cloud support -- the [long list](http://duplicity.nongnu.org/duplicity.1.html#sect7) of storage backends covers almost every cloud provider one can think of.  However, duplicity's biggest flaw lies in its incremental model -- a chain of dependent backups starts with a full backup followed by a number of incremental ones, and ends when another full backup is uploaded.  Deleting one backup will render useless all the subsequent backups on the same chain.  Periodic full backups are required, in order to make previous backups disposable.
 
 [bup](https://github.com/bup/bup) also uses librsync to split files into chunks but save chunks in the git packfile format.  It doesn't support any cloud storage, or deletion of old backups.
 
-[Obnam](http://obnam.org) got the incremental backup model right in the sense that every incremental backup is actually a full snapshot.  Although Obnam also splits files into chunks, it does not adopt either the rsync algorithm or the variable-size chunking algorithm.  As a result, deletions or insertions of a few bytes will foil the
-[deduplication](http://obnam.org/faq/dedup).
-Deletion of old backups is possible, but no cloud storages are supported.
-Multiple clients can back up to the same storage, but only sequential access is granted by the [locking on-disk data structures](http://obnam.org/locking/).
-It is unclear if the lack of cloud backends is due to difficulties in porting the locking data structures to cloud storage APIs.
+[Duplicati](https://duplicati.com) is one of the first backup tools that adopt the chunk-based approach to split files into chunks which are then uploaded to the storage.  The chunk-based approach got the incremental backup model right in the sense that every incremental backup is actually a full snapshot.  As Duplicati splits files into fixed-size chunks,  deletions or insertions of a few bytes will foil the deduplication.  Cloud support is extensive, but multiple clients can't back up to the same storage location.
 
-[Attic](https://attic-backup.org) has been acclaimed by some as the [Holy Grail of backups](https://www.stavros.io/posts/holy-grail-backups).  It follows the same incremental backup model as Obnam, but embraces the variable-size chunk algorithm for better performance and better deduplication.  Deletions of old backup is also supported.  However, no cloud backends are implemented, as in Obnam.  Although concurrent backups from multiple clients to the same storage is in theory possible by the use of locking, it is 
-[not recommended](http://librelist.com/browser//attic/2014/11/11/backing-up-multiple-servers-into-a-single-repository/#e96345aa5a3469a87786675d65da492b) by the developer due to chunk indices being kept in a local cache. 
-Concurrent access is not only a convenience; it is a necessity for better deduplication.  For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines.  Attic still adopts the traditional approach of using a centralized indexing database to manage chunks, and relies heavily on caching to improve performance.  The presence of exclusive locking makes it hard to be adapted for cloud storage APIs and reduces the level of deduplication.
+[Attic](https://attic-backup.org) has been acclaimed by some as the [Holy Grail of backups](https://www.stavros.io/posts/holy-grail-backups).  It follows the same incremental backup model like Duplicati, but embraces the variable-size chunk algorithm for better performance and higher deduplication efficiency (not susceptible to byte insertion and deletion any more).  Deletions of old backup is also supported.  However, no cloud backends are implemented.  Although concurrent backups from multiple clients to the same storage is in theory possible by the use of locking, it is
+[not recommended](http://librelist.com/browser//attic/2014/11/11/backing-up-multiple-servers-into-a-single-repository/#e96345aa5a3469a87786675d65da492b) by the developer due to chunk indices being kept in a local cache.
+Concurrent access is not only a convenience; it is a necessity for better deduplication.  For instance, if multiple machines with the same OS installed can back up their entire drives to the same storage, only one copy of the system files needs to be stored, greatly reducing the storage space regardless of the number of machines.  Attic still adopts the traditional approach of using a centralized indexing database to manage chunks, and relies heavily on caching to improve performance.  The presence of exclusive locking makes it hard to be extended to cloud storages.
 
-[restic](https://restic.github.io) is a more recent addition.  It is worth mentioning here because, like Duplicacy, it is written in Go.  It uses a format similar to the git packfile format.  Multiple clients backing up to the same storage are still guarded by 
-[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks).  A prune operation will therefore completely block all other clients connected to the storage from doing their regular backups.  Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is.  A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss.  This is the exact problem that we avoided by taking the lock-free approach.
+[restic](https://restic.github.io) is a more recent addition. It uses a format similar to the git packfile format.  Multiple clients backing up to the same storage are still guarded by
+[locks](https://github.com/restic/restic/blob/master/doc/Design.md#locks), and because a chunk database is used, deduplication isn't real-time (different clients sharing the same files will upload different copies of the same chunks).  A prune operation will completely block all other clients connected to the storage from doing their regular backups.  Moreover, since most cloud storage services do not provide a locking service, the best effort is to use some basic file operations to simulate a lock, but distributed locking is known to be a hard problem and it is unclear how reliable restic's lock implementation is.  A faulty implementation may cause a prune operation to accidentally delete data still in use, resulting in unrecoverable data loss.  This is the exact problem that we avoided by taking the lock-free approach.
 
 
 The following table compares the feature lists of all these backup tools:
 
 
-| Feature/Tool       | duplicity | bup | Obnam             | Attic           | restic            | **Duplicacy** | 
+| Feature/Tool       | duplicity | bup | Duplicati         | Attic           | restic            | **Duplicacy** |
 |:------------------:|:---------:|:---:|:-----------------:|:---------------:|:-----------------:|:-------------:|
 | Incremental Backup | Yes       | Yes | Yes               | Yes             | Yes               | **Yes**       |
 | Full Snapshot      | No        | Yes | Yes               | Yes             | Yes               | **Yes**       |
+| Compression        | Yes       | Yes | Yes               | Yes             | No                | **Yes**       |
 | Deduplication      | Weak      | Yes | Weak              | Yes             | Yes               | **Yes**       |
 | Encryption         | Yes       | Yes | Yes               | Yes             | Yes               | **Yes**       |
 | Deletion           | No        | No  | Yes               | Yes             | No                | **Yes**       |
-| Concurrent Access  | No        | No  | Exclusive locking | Not recommended | Exclusive locking | **Lock-free** |
-| Cloud Support      | Extensive | No  | No                | No              | S3, B2, OpenStack | **S3, GCS, Azure, Dropbox, Backblaze B2, Google Drive, OneDrive, and Hubic**|
+| Concurrent Access  | No        | No  | No                | Not recommended | Exclusive locking | **Lock-free** |
+| Cloud Support      | Extensive | No  | Extensive         | No              | Limited           | **Extensive** |
 | Snapshot Migration | No        | No  | No                | No              | No                | **Yes**       |
 
-
-## Performance Comparison with Other Backup Tools
-
-Duplicacy is not only more feature-rich but also faster than other backup tools.  The following table lists the running times in seconds of backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other tools.  Clearly Duplicacy is the fastest by a significant margin.
-
-
-|                    |   Duplicacy  |   restic   |   Attic    |  duplicity  | 
-|:------------------:|:----------------:|:----------:|:----------:|:-----------:|
-| Initial backup | 13.7 | 20.7 | 26.9 | 44.2 | 
-| 2nd backup | 4.8  |  8.0 | 15.4 | 19.5 | 
-| 3rd backup | 6.9  | 11.9 | 19.6 | 29.8 | 
-| 4th backup | 3.3  | 7.0  | 13.7 | 18.6 | 
-| 5th backup | 9.9  | 11.4 | 19.9 | 28.0 | 
-| 6th backup | 3.8  | 8.0  | 16.8 | 22.0 | 
-| 7th backup | 5.1  | 7.8  | 14.3 | 21.6 | 
-| 8th backup | 9.5  | 13.5 | 18.3 | 35.0 | 
-| 9th backup | 4.3  | 9.0  | 15.7 | 24.9 | 
-| 10th backup | 7.9 | 20.2 | 32.2 | 35.0 | 
-| 11th backup | 4.6 | 9.1  | 16.8 | 28.1 | 
-| 12th backup | 7.4 | 12.0 | 21.7 | 37.4 | 
-
-
-For more details and other speed comparison results, please visit https://github.com/gilbertchen/benchmarking.  There you can also find test scripts that you can use to run your own experiments.
-
 ## License
 
 * Free for personal use or commercial trial
-* Non-trial commercial use requires per-user licenses available from [duplicacy.com](https://duplicacy.com/customer) at a cost of $20 per year
+* Non-trial commercial use requires per-user CLI licenses available from [duplicacy.com](https://duplicacy.com/buy) at a cost of $20 per year
+* A user is defined as the computer account that creates or edits the files to be backed up; if a backup contains files created or edited by multiple users for commercial purposes, one CLI license is required for each user
+* The computer with a valid commercial license for the GUI version may run the CLI version without a CLI license
+* CLI licenses are not required to restore or manage backups; only the backup command requires valid CLI licenses
 * Modification and redistribution are permitted, but commercial use of derivative works is subject to the same requirements of this license

integration_tests/copy_test.sh

@@ -0,0 +1,31 @@
+#!/bin/bash
+
+
+. ./test_functions.sh
+
+fixture
+
+pushd ${TEST_REPO}
+${DUPLICACY} init integration-tests $TEST_STORAGE -c 1k
+${DUPLICACY} add -copy default secondary integration-tests $SECONDARY_STORAGE
+add_file file1
+add_file file2
+${DUPLICACY} backup 
+${DUPLICACY} copy -from default -to secondary
+add_file file3
+add_file file4
+${DUPLICACY} backup 
+${DUPLICACY} copy -from default -to secondary
+${DUPLICACY} check --files -stats -storage default
+${DUPLICACY} check --files -stats -storage secondary
+# Prune revisions from default storage
+${DUPLICACY} -d -v -log prune -r 1-2 -exclusive -exhaustive -storage default
+# Copy snapshot revisions from secondary back to default
+${DUPLICACY} copy -from secondary -to default
+# Check snapshot revisions again to make sure we're ok!
+${DUPLICACY} check --files -stats -storage default
+${DUPLICACY} check --files -stats -storage secondary
+# Check for orphaned or missing chunks
+${DUPLICACY} prune -exhaustive -exclusive -storage default
+${DUPLICACY} prune -exhaustive -exclusive -storage secondary
+popd

src/duplicacy_acdclient.go

@@ -5,448 +5,450 @@
 package duplicacy
 
 import (
-    "fmt"
-    "time"
-    "bytes"
-    "sync"
-    "io/ioutil"
-    "encoding/json"
-    "io"
-    "net/http"
-    "mime/multipart"
-    "math/rand"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"mime/multipart"
+	"net/http"
+	"sync"
+	"time"
 
-    "golang.org/x/oauth2"
+	"golang.org/x/oauth2"
 )
 
 type ACDError struct {
-    Status     int
-    Message    string   `json:"message"`
+	Status  int
+	Message string `json:"message"`
 }
 
 func (err ACDError) Error() string {
-    return fmt.Sprintf("%d %s", err.Status, err.Message)
+	return fmt.Sprintf("%d %s", err.Status, err.Message)
 }
 
 var ACDRefreshTokenURL = "https://duplicacy.com/acd_refresh"
+
 type ACDClient struct {
-    HTTPClient          *http.Client
+	HTTPClient *http.Client
 
-    TokenFile           string
-    Token               *oauth2.Token
-    TokenLock           *sync.Mutex
+	TokenFile string
+	Token     *oauth2.Token
+	TokenLock *sync.Mutex
 
-    ContentURL          string
-    MetadataURL         string
+	ContentURL  string
+	MetadataURL string
 
-    TestMode            bool
+	TestMode bool
 }
 
-
 func NewACDClient(tokenFile string) (*ACDClient, error) {
 
-    description, err := ioutil.ReadFile(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	description, err := ioutil.ReadFile(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    token := new(oauth2.Token)
-    if err := json.Unmarshal(description, token); err != nil {
-        return nil, err
-    }
+	token := new(oauth2.Token)
+	if err := json.Unmarshal(description, token); err != nil {
+		return nil, err
+	}
 
-    client := &ACDClient{
-        HTTPClient: http.DefaultClient,
-        TokenFile: tokenFile,
-        Token: token,
-        TokenLock: &sync.Mutex{},
-    }
+	client := &ACDClient{
+		HTTPClient: http.DefaultClient,
+		TokenFile:  tokenFile,
+		Token:      token,
+		TokenLock:  &sync.Mutex{},
+	}
 
-    client.GetEndpoint()
+	client.GetEndpoint()
 
-    return client, nil
+	return client, nil
 }
 
 func (client *ACDClient) call(url string, method string, input interface{}, contentType string) (io.ReadCloser, int64, error) {
 
-    LOG_DEBUG("ACD_CALL", "Calling %s", url)
+	//LOG_DEBUG("ACD_CALL", "%s %s", method, url)
 
-    var response *http.Response
+	var response *http.Response
 
-    backoff := 1
-    for i := 0; i < 8; i++ {
-        var inputReader io.Reader
+	backoff := 1
+	for i := 0; i < 8; i++ {
+		var inputReader io.Reader
 
-        switch input.(type) {
-        default:
-            jsonInput, err := json.Marshal(input)
-            if err != nil {
-                return nil, 0, err
-            }
-            inputReader = bytes.NewReader(jsonInput)
-        case []byte:
-            inputReader = bytes.NewReader(input.([]byte))
-        case int:
-            inputReader = bytes.NewReader([]byte(""))
-        case *bytes.Buffer:
-            inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
-        case *RateLimitedReader:
-            input.(*RateLimitedReader).Reset()
-            inputReader = input.(*RateLimitedReader)
-        }
+		switch input.(type) {
+		default:
+			jsonInput, err := json.Marshal(input)
+			if err != nil {
+				return nil, 0, err
+			}
+			inputReader = bytes.NewReader(jsonInput)
+		case []byte:
+			inputReader = bytes.NewReader(input.([]byte))
+		case int:
+			inputReader = bytes.NewReader([]byte(""))
+		case *bytes.Buffer:
+			inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
+		case *RateLimitedReader:
+			input.(*RateLimitedReader).Reset()
+			inputReader = input.(*RateLimitedReader)
+		}
 
-        request, err := http.NewRequest(method, url, inputReader)
-        if err != nil {
-            return nil, 0, err
-        }
+		request, err := http.NewRequest(method, url, inputReader)
+		if err != nil {
+			return nil, 0, err
+		}
 
-        if reader, ok := inputReader.(*RateLimitedReader); ok {
-            request.ContentLength = reader.Length()
-        }
+		if reader, ok := inputReader.(*RateLimitedReader); ok {
+			request.ContentLength = reader.Length()
+		}
 
-        if url != ACDRefreshTokenURL {
-            client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
-            client.TokenLock.Unlock()
-        }
-        if contentType != "" {
-            request.Header.Set("Content-Type", contentType)
-        }
+		if url != ACDRefreshTokenURL {
+			client.TokenLock.Lock()
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
+			client.TokenLock.Unlock()
+		}
+		if contentType != "" {
+			request.Header.Set("Content-Type", contentType)
+		}
 
-        response, err = client.HTTPClient.Do(request)
-        if err != nil {
-            return nil, 0, err
-        }
+		response, err = client.HTTPClient.Do(request)
+		if err != nil {
+			return nil, 0, err
+		}
 
-        if response.StatusCode < 400 {
-            return response.Body, response.ContentLength, nil
-        }
+		if response.StatusCode < 400 {
+			return response.Body, response.ContentLength, nil
+		}
 
-        if response.StatusCode == 404 {
-            buffer := new(bytes.Buffer)
-            buffer.ReadFrom(response.Body)
-            response.Body.Close()
-            return nil, 0, ACDError { Status: response.StatusCode, Message: buffer.String()}
-        }
+		if response.StatusCode == 404 {
+			buffer := new(bytes.Buffer)
+			buffer.ReadFrom(response.Body)
+			response.Body.Close()
+			return nil, 0, ACDError{Status: response.StatusCode, Message: buffer.String()}
+		}
 
-        if response.StatusCode == 400 {
-            defer response.Body.Close()
+		if response.StatusCode == 400 {
+			defer response.Body.Close()
 
-            e := &ACDError {
-                Status: response.StatusCode,
-            }
+			e := &ACDError{
+				Status: response.StatusCode,
+			}
 
-            if err := json.NewDecoder(response.Body).Decode(e); err == nil {
-                return nil, 0, e
-            } else {
-                return nil, 0, ACDError { Status: response.StatusCode, Message: "Bad input parameter"}
-            }
-        }
+			if err := json.NewDecoder(response.Body).Decode(e); err == nil {
+				return nil, 0, e
+			} else {
+				return nil, 0, ACDError{Status: response.StatusCode, Message: "Bad input parameter"}
+			}
+		}
 
-        response.Body.Close()
+		response.Body.Close()
 
-        if response.StatusCode == 401 {
+		if response.StatusCode == 401 {
 
-            if url == ACDRefreshTokenURL {
-                return nil, 0, ACDError { Status: response.StatusCode, Message: "Unauthorized"}
-            }
+			if url == ACDRefreshTokenURL {
+				return nil, 0, ACDError{Status: response.StatusCode, Message: "Unauthorized"}
+			}
 
-            err = client.RefreshToken()
-            if err != nil {
-                return nil, 0, err
-            }
+			err = client.RefreshToken()
+			if err != nil {
+				return nil, 0, err
+			}
 
-            continue
-        } else if response.StatusCode == 403 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Forbidden"}
-        } else if response.StatusCode == 404 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Resource not found"}
-        } else if response.StatusCode == 409 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Conflict"}
-        } else if response.StatusCode == 411 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Length required"}
-        } else if response.StatusCode == 412 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Precondition failed"}
-        } else if response.StatusCode == 429 || response.StatusCode == 500 {
-            reason := "Too many requests"
-            if response.StatusCode == 500 {
-                reason = "Internal server error"
-            }
-            retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
-            LOG_INFO("ACD_RETRY", "%s; retry after %d milliseconds", reason, retryAfter)
-            time.Sleep(retryAfter * time.Millisecond)
-            backoff *= 2
-            continue
-        } else if response.StatusCode == 503 {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Service unavailable"}
-        } else  {
-            return nil, 0, ACDError { Status: response.StatusCode, Message: "Unknown error"}
-        }
-    }
+			continue
+		} else if response.StatusCode == 403 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Forbidden"}
+		} else if response.StatusCode == 404 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Resource not found"}
+		} else if response.StatusCode == 409 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Conflict"}
+		} else if response.StatusCode == 411 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Length required"}
+		} else if response.StatusCode == 412 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Precondition failed"}
+		} else if response.StatusCode == 429 || response.StatusCode == 500 {
+			reason := "Too many requests"
+			if response.StatusCode == 500 {
+				reason = "Internal server error"
+			}
+			retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
+			LOG_INFO("ACD_RETRY", "%s; retry after %d milliseconds", reason, retryAfter)
+			time.Sleep(retryAfter * time.Millisecond)
+			backoff *= 2
+			continue
+		} else if response.StatusCode == 503 {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Service unavailable"}
+		} else {
+			return nil, 0, ACDError{Status: response.StatusCode, Message: "Unknown error"}
+		}
+	}
 
-    return nil, 0, fmt.Errorf("Maximum number of retries reached")
+	return nil, 0, fmt.Errorf("Maximum number of retries reached")
 }
 
 func (client *ACDClient) RefreshToken() (err error) {
 
-    client.TokenLock.Lock()
-    defer client.TokenLock.Unlock()
+	client.TokenLock.Lock()
+	defer client.TokenLock.Unlock()
 
-    readCloser, _, err := client.call(ACDRefreshTokenURL, "POST", client.Token, "")
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := client.call(ACDRefreshTokenURL, "POST", client.Token, "")
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    if err = json.NewDecoder(readCloser).Decode(client.Token); err != nil {
-        return err
-    }
+	if err = json.NewDecoder(readCloser).Decode(client.Token); err != nil {
+		return err
+	}
 
-    description, err := json.Marshal(client.Token)
-    if err != nil {
-        return err
-    }
+	description, err := json.Marshal(client.Token)
+	if err != nil {
+		return err
+	}
 
-    err = ioutil.WriteFile(client.TokenFile, description, 0644)
-    if err != nil {
-        return err
-    }
+	err = ioutil.WriteFile(client.TokenFile, description, 0644)
+	if err != nil {
+		return err
+	}
 
-    return nil
+	return nil
 }
 
 type ACDGetEndpointOutput struct {
-    CustomerExists bool      `json:"customerExists"`
-    ContentURL     string    `json:"contentUrl"`
-    MetadataURL    string    `json:"metadataUrl"`
+	CustomerExists bool   `json:"customerExists"`
+	ContentURL     string `json:"contentUrl"`
+	MetadataURL    string `json:"metadataUrl"`
 }
 
 func (client *ACDClient) GetEndpoint() (err error) {
 
-    readCloser, _, err := client.call("https://drive.amazonaws.com/drive/v1/account/endpoint", "GET", 0, "")
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := client.call("https://drive.amazonaws.com/drive/v1/account/endpoint", "GET", 0, "")
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    output := &ACDGetEndpointOutput {}
+	output := &ACDGetEndpointOutput{}
 
-    if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-        return err
-    }
+	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+		return err
+	}
 
-    client.ContentURL = output.ContentURL
-    client.MetadataURL = output.MetadataURL
+	client.ContentURL = output.ContentURL
+	client.MetadataURL = output.MetadataURL
 
-    return nil
+	return nil
 }
 
 type ACDEntry struct {
-    Name     string    `json:"name"`
-    ID       string    `json:"id"`
-    Size     int64     `json:"size"`
-    Kind     string    `json:"kind"`
+	Name string `json:"name"`
+	ID   string `json:"id"`
+	Size int64  `json:"size"`
+	Kind string `json:"kind"`
 }
 
 type ACDListEntriesOutput struct {
-    Count     int          `json:"count"`
-    NextToken string       `json:"nextToken"`
-    Entries   []ACDEntry   `json:"data"`
+	Count     int        `json:"count"`
+	NextToken string     `json:"nextToken"`
+	Entries   []ACDEntry `json:"data"`
 }
 
-func (client *ACDClient) ListEntries(parentID string, listFiles bool) ([]ACDEntry, error) {
+func (client *ACDClient) ListEntries(parentID string, listFiles bool, listDirectories bool) ([]ACDEntry, error) {
 
-    startToken := ""
+	startToken := ""
 
-    entries := []ACDEntry{}
+	entries := []ACDEntry{}
 
-    for {
+	for {
 
-        url := client.MetadataURL + "nodes/" + parentID + "/children?filters="
+		url := client.MetadataURL + "nodes/" + parentID + "/children?"
 
-        if listFiles {
-            url += "kind:FILE"
-        } else {
-            url += "kind:FOLDER"
-        }
+		if listFiles && !listDirectories {
+			url += "filters=kind:FILE&"
+		} else if !listFiles && listDirectories {
+			url += "filters=kind:FOLDER&"
+		}
 
-        if startToken != "" {
-            url += "&startToken=" + startToken
-        }
+		if startToken != "" {
+			url += "startToken=" + startToken + "&"
+		}
 
-        if client.TestMode {
-            url += "&limit=8"
-        }
+		if client.TestMode {
+			url += "limit=8"
+		} else {
+			url += "limit=200"
+		}
 
-        readCloser, _, err := client.call(url, "GET", 0, "")
-        if err != nil {
-            return nil, err
-        }
+		readCloser, _, err := client.call(url, "GET", 0, "")
+		if err != nil {
+			return nil, err
+		}
 
-        defer readCloser.Close()
+		defer readCloser.Close()
 
-        output := &ACDListEntriesOutput {}
+		output := &ACDListEntriesOutput{}
 
-        if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-            return nil, err
-        }
+		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+			return nil, err
+		}
 
-        entries = append(entries, output.Entries...)
+		entries = append(entries, output.Entries...)
 
-        startToken = output.NextToken
-        if startToken == "" {
-            break
-        }
-    }
+		startToken = output.NextToken
+		if startToken == "" {
+			break
+		}
+	}
 
-    return entries, nil
+	return entries, nil
 }
 
 func (client *ACDClient) ListByName(parentID string, name string) (string, bool, int64, error) {
 
-    url := client.MetadataURL + "nodes"
+	url := client.MetadataURL + "nodes"
 
-    if parentID == "" {
-        url += "?filters=Kind:FOLDER+AND+isRoot:true"
-    } else {
-        url += "/" + parentID + "/children?filters=name:" + name
-    }
+	if parentID == "" {
+		url += "?filters=Kind:FOLDER+AND+isRoot:true"
+	} else {
+		url += "/" + parentID + "/children?filters=name:" + name
+	}
 
-    readCloser, _, err := client.call(url, "GET", 0, "")
-    if err != nil {
-        return "", false, 0, err
-    }
+	readCloser, _, err := client.call(url, "GET", 0, "")
+	if err != nil {
+		return "", false, 0, err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    output := &ACDListEntriesOutput {}
+	output := &ACDListEntriesOutput{}
 
-    if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-        return "", false, 0, err
-    }
+	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+		return "", false, 0, err
+	}
 
-    if len(output.Entries) == 0 {
-        return "", false, 0, nil
-    }
+	if len(output.Entries) == 0 {
+		return "", false, 0, nil
+	}
 
-    return output.Entries[0].ID, output.Entries[0].Kind == "FOLDER", output.Entries[0].Size, nil
+	return output.Entries[0].ID, output.Entries[0].Kind == "FOLDER", output.Entries[0].Size, nil
 }
 
 func (client *ACDClient) DownloadFile(fileID string) (io.ReadCloser, int64, error) {
 
-    url := client.ContentURL + "nodes/" + fileID + "/content"
+	url := client.ContentURL + "nodes/" + fileID + "/content"
 
-    return client.call(url, "GET", 0, "")
+	return client.call(url, "GET", 0, "")
 }
 
 func (client *ACDClient) UploadFile(parentID string, name string, content []byte, rateLimit int) (fileID string, err error) {
 
-    url := client.ContentURL + "nodes?suppress=deduplication"
+	url := client.ContentURL + "nodes?suppress=deduplication"
 
-    body := &bytes.Buffer{}
-    writer := multipart.NewWriter(body)
+	body := &bytes.Buffer{}
+	writer := multipart.NewWriter(body)
 
-    metadata := make(map[string]interface{})
-    metadata["name"] = name
-    metadata["kind"] = "FILE"
-    metadata["parents"] = []string{ parentID }
+	metadata := make(map[string]interface{})
+	metadata["name"] = name
+	metadata["kind"] = "FILE"
+	metadata["parents"] = []string{parentID}
 
-    metadataJSON, err := json.Marshal(metadata)
-    if err != nil {
-        return "", err
-    }
+	metadataJSON, err := json.Marshal(metadata)
+	if err != nil {
+		return "", err
+	}
 
-    err = writer.WriteField("metadata", string(metadataJSON))
-    if err != nil {
-        return "", err
-    }
+	err = writer.WriteField("metadata", string(metadataJSON))
+	if err != nil {
+		return "", err
+	}
 
-    part, err := writer.CreateFormFile("content", name)
-    if err != nil {
-        return "", err
-    }
+	part, err := writer.CreateFormFile("content", name)
+	if err != nil {
+		return "", err
+	}
 
-    _, err = part.Write(content)
-    if err != nil {
-        return "", err
-    }
+	_, err = part.Write(content)
+	if err != nil {
+		return "", err
+	}
 
-    writer.Close()
+	writer.Close()
 
-    var input interface{}
-    input = body
-    if rateLimit > 0 {
-        input = CreateRateLimitedReader(body.Bytes(), rateLimit)
-    }
+	var input interface{}
+	input = body
+	if rateLimit > 0 {
+		input = CreateRateLimitedReader(body.Bytes(), rateLimit)
+	}
 
-    readCloser, _, err := client.call(url, "POST", input, writer.FormDataContentType())
+	readCloser, _, err := client.call(url, "POST", input, writer.FormDataContentType())
 
-    if err != nil {
-        return "", err
-    }
+	if err != nil {
+		return "", err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    entry := ACDEntry {}
-    if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
-        return "", err
-    }
+	entry := ACDEntry{}
+	if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
+		return "", err
+	}
 
-    return entry.ID, nil
+	return entry.ID, nil
 }
 
 func (client *ACDClient) DeleteFile(fileID string) error {
 
-    url := client.MetadataURL + "trash/" + fileID
+	url := client.MetadataURL + "trash/" + fileID
 
-    readCloser, _, err := client.call(url, "PUT", 0, "")
-    if err != nil {
-        return  err
-    }
+	readCloser, _, err := client.call(url, "PUT", 0, "")
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *ACDClient) MoveFile(fileID string, fromParentID string, toParentID string) error {
 
-    url := client.MetadataURL + "nodes/" + toParentID + "/children"
+	url := client.MetadataURL + "nodes/" + toParentID + "/children"
 
-    parameters := make(map[string]string)
-    parameters["fromParent"] = fromParentID
-    parameters["childId"] = fileID
+	parameters := make(map[string]string)
+	parameters["fromParent"] = fromParentID
+	parameters["childId"] = fileID
 
-    readCloser, _, err := client.call(url, "POST", parameters, "")
-    if err != nil {
-        return  err
-    }
+	readCloser, _, err := client.call(url, "POST", parameters, "")
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *ACDClient) CreateDirectory(parentID string, name string) (string, error) {
 
-    url := client.MetadataURL + "nodes"
+	url := client.MetadataURL + "nodes"
 
-    parameters := make(map[string]interface{})
-    parameters["name"] = name
-    parameters["kind"] = "FOLDER"
-    parameters["parents"] = []string {parentID}
+	parameters := make(map[string]interface{})
+	parameters["name"] = name
+	parameters["kind"] = "FOLDER"
+	parameters["parents"] = []string{parentID}
 
-    readCloser, _, err := client.call(url, "POST", parameters, "")
-    if err != nil {
-        return "", err
-    }
+	readCloser, _, err := client.call(url, "POST", parameters, "")
+	if err != nil {
+		return "", err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    entry := ACDEntry {}
-    if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
-        return "", err
-    }
+	entry := ACDEntry{}
+	if err = json.NewDecoder(readCloser).Decode(&entry); err != nil {
+		return "", err
+	}
 
-    return entry.ID, nil
+	return entry.ID, nil
 }

src/duplicacy_acdclient_test.go

@@ -5,149 +5,149 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
-    "crypto/sha256"
-    "encoding/hex"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
+	crypto_rand "crypto/rand"
+	"math/rand"
 )
 
 func TestACDClient(t *testing.T) {
 
-    acdClient, err := NewACDClient("acd-token.json")
-    if err != nil {
-        t.Errorf("Failed to create the ACD client: %v", err)
-        return
-    }
+	acdClient, err := NewACDClient("acd-token.json")
+	if err != nil {
+		t.Errorf("Failed to create the ACD client: %v", err)
+		return
+	}
 
-    acdClient.TestMode = true
+	acdClient.TestMode = true
 
-    rootID, _, _, err := acdClient.ListByName("", "")
-    if err != nil {
-        t.Errorf("Failed to get the root node: %v", err)
-        return
-    }
+	rootID, _, _, err := acdClient.ListByName("", "")
+	if err != nil {
+		t.Errorf("Failed to get the root node: %v", err)
+		return
+	}
 
-    if rootID == "" {
-        t.Errorf("No root node")
-        return
-    }
+	if rootID == "" {
+		t.Errorf("No root node")
+		return
+	}
 
-    testID, _, _, err := acdClient.ListByName(rootID, "test")
-    if err != nil {
-        t.Errorf("Failed to list the test directory: %v", err)
-        return
-    }
-    if testID == "" {
-        testID, err = acdClient.CreateDirectory(rootID, "test")
-        if err != nil {
-            t.Errorf("Failed to create the test directory: %v", err)
-            return
-        }
-    }
+	testID, _, _, err := acdClient.ListByName(rootID, "test")
+	if err != nil {
+		t.Errorf("Failed to list the test directory: %v", err)
+		return
+	}
+	if testID == "" {
+		testID, err = acdClient.CreateDirectory(rootID, "test")
+		if err != nil {
+			t.Errorf("Failed to create the test directory: %v", err)
+			return
+		}
+	}
 
-    test1ID, _, _, err := acdClient.ListByName(testID, "test1")
-    if err != nil {
-        t.Errorf("Failed to list the test1 directory: %v", err)
-        return
-    }
-    if test1ID == "" {
-        test1ID, err = acdClient.CreateDirectory(testID, "test1")
-        if err != nil {
-            t.Errorf("Failed to create the test1 directory: %v", err)
-            return
-        }
-    }
+	test1ID, _, _, err := acdClient.ListByName(testID, "test1")
+	if err != nil {
+		t.Errorf("Failed to list the test1 directory: %v", err)
+		return
+	}
+	if test1ID == "" {
+		test1ID, err = acdClient.CreateDirectory(testID, "test1")
+		if err != nil {
+			t.Errorf("Failed to create the test1 directory: %v", err)
+			return
+		}
+	}
 
-    test2ID, _, _, err := acdClient.ListByName(testID, "test2")
-    if err != nil {
-        t.Errorf("Failed to list the test2 directory: %v", err)
-        return
-    }
-    if test2ID == "" {
-        test2ID, err = acdClient.CreateDirectory(testID, "test2")
-        if err != nil {
-            t.Errorf("Failed to create the test2 directory: %v", err)
-            return
-        }
-    }
+	test2ID, _, _, err := acdClient.ListByName(testID, "test2")
+	if err != nil {
+		t.Errorf("Failed to list the test2 directory: %v", err)
+		return
+	}
+	if test2ID == "" {
+		test2ID, err = acdClient.CreateDirectory(testID, "test2")
+		if err != nil {
+			t.Errorf("Failed to create the test2 directory: %v", err)
+			return
+		}
+	}
 
-    fmt.Printf("test1: %s, test2: %s\n", test1ID, test2ID)
+	fmt.Printf("test1: %s, test2: %s\n", test1ID, test2ID)
 
-    numberOfFiles := 20
-    maxFileSize := 64 * 1024
+	numberOfFiles := 20
+	maxFileSize := 64 * 1024
 
-    for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+	for i := 0; i < numberOfFiles; i++ {
+		content := make([]byte, rand.Int()%maxFileSize+1)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-        hasher := sha256.New()
-        hasher.Write(content)
-        filename := hex.EncodeToString(hasher.Sum(nil))
+		hasher := sha256.New()
+		hasher.Write(content)
+		filename := hex.EncodeToString(hasher.Sum(nil))
 
-        fmt.Printf("file: %s\n", filename)
+		fmt.Printf("file: %s\n", filename)
 
-        _, err = acdClient.UploadFile(test1ID, filename, content, 100)
-        if err != nil {
-            /*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
-                t.Errorf("Failed to upload the file %s: %v", filename, err)
-                return
-            }
-        }
-    }
+		_, err = acdClient.UploadFile(test1ID, filename, content, 100)
+		if err != nil {
+			/*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
+				t.Errorf("Failed to upload the file %s: %v", filename, err)
+				return
+			}
+		}
+	}
 
-    entries, err := acdClient.ListEntries(test1ID, true)
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	entries, err := acdClient.ListEntries(test1ID, true, false)
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    for _, entry := range entries {
-        err = acdClient.MoveFile(entry.ID, test1ID, test2ID)
-        if err != nil {
-            t.Errorf("Failed to move %s: %v", entry.Name, err)
-            return
-        }
-    }
+	for _, entry := range entries {
+		err = acdClient.MoveFile(entry.ID, test1ID, test2ID)
+		if err != nil {
+			t.Errorf("Failed to move %s: %v", entry.Name, err)
+			return
+		}
+	}
 
-    entries, err = acdClient.ListEntries(test2ID, true)
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	entries, err = acdClient.ListEntries(test2ID, true, false)
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    for _, entry := range entries {
-        readCloser, _, err := acdClient.DownloadFile(entry.ID)
-        if err != nil {
-            t.Errorf("Error downloading file %s: %v", entry.Name, err)
-            return
-        }
+	for _, entry := range entries {
+		readCloser, _, err := acdClient.DownloadFile(entry.ID)
+		if err != nil {
+			t.Errorf("Error downloading file %s: %v", entry.Name, err)
+			return
+		}
 
-        hasher := sha256.New()
-        io.Copy(hasher, readCloser)
-        hash := hex.EncodeToString(hasher.Sum(nil))
+		hasher := sha256.New()
+		io.Copy(hasher, readCloser)
+		hash := hex.EncodeToString(hasher.Sum(nil))
 
-        if hash != entry.Name {
-            t.Errorf("File %s, hash %s", entry.Name, hash)
-        }
+		if hash != entry.Name {
+			t.Errorf("File %s, hash %s", entry.Name, hash)
+		}
 
-        readCloser.Close()
-    }
+		readCloser.Close()
+	}
 
-    for _, entry := range entries {
+	for _, entry := range entries {
 
-        err = acdClient.DeleteFile(entry.ID)
-        if err != nil {
-            t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
-            return
-        }
-    }
+		err = acdClient.DeleteFile(entry.ID)
+		if err != nil {
+			t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
+			return
+		}
+	}
 
 }

src/duplicacy_acdstorage.go

@@ -5,400 +5,449 @@
 package duplicacy
 
 import (
-    "fmt"
-    "path"
-    "strings"
-    "sync"
+	"fmt"
+	"path"
+	"strings"
+	"sync"
+	"time"
 )
 
 type ACDStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    client *ACDClient
-    idCache map[string]string
-    idCacheLock    *sync.Mutex
-    numberOfThreads int
+	client          *ACDClient
+	idCache         map[string]string
+	idCacheLock     *sync.Mutex
+	numberOfThreads int
 }
 
 // CreateACDStorage creates an ACD storage object.
 func CreateACDStorage(tokenFile string, storagePath string, threads int) (storage *ACDStorage, err error) {
 
-    client, err := NewACDClient(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	client, err := NewACDClient(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    storage = &ACDStorage {
-        client: client,
-        idCache: make(map[string]string),
-        idCacheLock: &sync.Mutex{},
-        numberOfThreads: threads,
-    }
+	storage = &ACDStorage{
+		client:          client,
+		idCache:         make(map[string]string),
+		idCacheLock:     &sync.Mutex{},
+		numberOfThreads: threads,
+	}
 
-    storagePathID, _, _, err := storage.getIDFromPath(0, storagePath)
-    if err != nil {
-        return nil, err
-    }
+	storagePathID, err := storage.getIDFromPath(0, storagePath, false)
+	if err != nil {
+		return nil, err
+	}
 
-    storage.idCache[""] = storagePathID
+	// Set 'storagePath' as the root of the storage and clean up the id cache accordingly
+	storage.idCache = make(map[string]string)
+	storage.idCache[""] = storagePathID
 
-    for _, dir := range []string { "chunks", "fossils", "snapshots" } {
-        dirID, isDir, _, err := client.ListByName(storagePathID, dir)
-        if err != nil {
-            return nil, err
-        }
-        if dirID == "" {
-            dirID, err = client.CreateDirectory(storagePathID, dir)
-            if err != nil {
-                return nil, err
-            }
-        } else if !isDir {
-            return nil, fmt.Errorf("%s/%s is not a directory", storagePath + "/" + dir)
-        }
-        storage.idCache[dir] = dirID
-    }
-
-    return storage, nil
+	for _, dir := range []string{"chunks", "fossils", "snapshots"} {
+		dirID, isDir, _, err := client.ListByName(storagePathID, dir)
+		if err != nil {
+			return nil, err
+		}
+		if dirID == "" {
+			if err != nil {
+				return nil, err
+			}
+		} else if !isDir {
+			return nil, fmt.Errorf("%s is not a directory", storagePath+"/"+dir)
+		}
+		storage.idCache[dir] = dirID
+	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 func (storage *ACDStorage) getPathID(path string) string {
-    storage.idCacheLock.Lock()
-    pathID := storage.idCache[path]
-    storage.idCacheLock.Unlock()
-    return pathID
+	storage.idCacheLock.Lock()
+	pathID := storage.idCache[path]
+	storage.idCacheLock.Unlock()
+	return pathID
 }
 
 func (storage *ACDStorage) findPathID(path string) (string, bool) {
-    storage.idCacheLock.Lock()
-    pathID, ok := storage.idCache[path]
-    storage.idCacheLock.Unlock()
-    return pathID, ok
+	storage.idCacheLock.Lock()
+	pathID, ok := storage.idCache[path]
+	storage.idCacheLock.Unlock()
+	return pathID, ok
 }
 
 func (storage *ACDStorage) savePathID(path string, pathID string) {
-    storage.idCacheLock.Lock()
-    storage.idCache[path] = pathID
-    storage.idCacheLock.Unlock()
+	storage.idCacheLock.Lock()
+	storage.idCache[path] = pathID
+	storage.idCacheLock.Unlock()
 }
 
 func (storage *ACDStorage) deletePathID(path string) {
-    storage.idCacheLock.Lock()
-    delete(storage.idCache, path)
-    storage.idCacheLock.Unlock()
+	storage.idCacheLock.Lock()
+	delete(storage.idCache, path)
+	storage.idCacheLock.Unlock()
 }
 
-
-func (storage *ACDStorage) convertFilePath(filePath string) (string) {
-    if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
-        return "fossils/" + filePath[len("chunks/"):len(filePath) - len(".fsl")]
-    }
-    return filePath
+// convertFilePath converts the path for a fossil in the form of 'chunks/id.fsl' to 'fossils/id'.  This is because
+// ACD doesn't support file renaming. Instead, it only allows one file to be moved from one directory to another.
+// By adding a layer of path conversion we're pretending that we can rename between 'chunks/id' and 'chunks/id.fsl'
+func (storage *ACDStorage) convertFilePath(filePath string) string {
+	if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
+		return "fossils/" + filePath[len("chunks/"):len(filePath)-len(".fsl")]
+	}
+	return filePath
 }
 
-func (storage *ACDStorage) getIDFromPath(threadIndex int, path string) (fileID string, isDir bool, size int64, err error) {
+// getIDFromPath returns the id of the given path.  If 'createDirectories' is true, create the given path and all its
+// parent directories if they don't exist.  Note that if 'createDirectories' is false, it may return an empty 'fileID'
+// if the file doesn't exist.
+func (storage *ACDStorage) getIDFromPath(threadIndex int, filePath string, createDirectories bool) (fileID string, err error) {
 
-    parentID, ok := storage.findPathID("")
-    if !ok {
-        parentID, isDir, size, err = storage.client.ListByName("", "")
-        if err != nil {
-            return "", false, 0, err
-        }
-    }
+	if fileID, ok := storage.findPathID(filePath); ok {
+		return fileID, nil
+	}
 
-    names := strings.Split(path, "/")
-    for i, name := range names {
-        parentID, isDir, _, err = storage.client.ListByName(parentID, name)
-        if err != nil {
-            return "", false, 0, err
-        }
-        if parentID == "" {
-            if i == len(names) - 1 {
-                return "", false, 0, nil
-            } else {
-                return "", false, 0, fmt.Errorf("File path '%s' does not exist", path)
-            }
-        }
-        if i != len(names) - 1 && !isDir {
-            return "", false, 0, fmt.Errorf("Invalid path %s", path)
-        }
-    }
+	parentID, ok := storage.findPathID("")
+	if !ok {
+		parentID, _, _, err = storage.client.ListByName("", "")
+		if err != nil {
+			return "", err
+		}
+		storage.savePathID("", parentID)
+	}
 
-    return parentID, isDir, size, err
+	names := strings.Split(filePath, "/")
+	current := ""
+	for i, name := range names {
+
+		current = path.Join(current, name)
+		fileID, ok := storage.findPathID(current)
+		if ok {
+			parentID = fileID
+			continue
+		}
+		isDir := false
+		fileID, isDir, _, err = storage.client.ListByName(parentID, name)
+		if err != nil {
+			return "", err
+		}
+		if fileID == "" {
+			if !createDirectories {
+				return "", nil
+			}
+			// Create the current directory
+			fileID, err = storage.client.CreateDirectory(parentID, name)
+			if err != nil {
+				// Check if the directory has been created by another thread
+				if e, ok := err.(ACDError); !ok || e.Status != 409 {
+					return "", fmt.Errorf("Failed to create directory '%s': %v", current, err)
+				}
+				// A 409 means the directory may have already created by another thread.  Wait 10 seconds
+				// until we seed the directory.
+				for i := 0; i < 10; i++ {
+					var createErr error
+					fileID, isDir, _, createErr = storage.client.ListByName(parentID, name)
+					if createErr != nil {
+						return "", createErr
+					}
+					if fileID == "" {
+						time.Sleep(time.Second)
+					} else {
+						break
+					}
+				}
+				if fileID == "" {
+					return "", fmt.Errorf("All attempts to create directory '%s' failed: %v", current, err)
+				}
+			} else {
+				isDir = true
+			}
+		} else {
+			storage.savePathID(current, fileID)
+		}
+		if i != len(names)-1 && !isDir {
+			return "", fmt.Errorf("Path '%s' is not a directory", current)
+		}
+		parentID = fileID
+	}
+
+	return parentID, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *ACDStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    var err error
+	var err error
 
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    if dir == "snapshots" {
+	if dir == "snapshots" {
 
-        entries, err := storage.client.ListEntries(storage.getPathID(dir), false)
-        if err != nil {
-            return nil, nil, err
-        }
+		entries, err := storage.client.ListEntries(storage.getPathID(dir), false, true)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        subDirs := []string{}
+		subDirs := []string{}
 
-        for _, entry := range entries {
-            storage.savePathID(entry.Name, entry.ID)
-            subDirs = append(subDirs, entry.Name + "/")
-        }
-        return subDirs, nil, nil
-    } else if strings.HasPrefix(dir, "snapshots/") {
-        name := dir[len("snapshots/"):]
-        pathID, ok := storage.findPathID(dir)
-        if !ok {
-            pathID, _, _, err = storage.client.ListByName(storage.getPathID("snapshots"), name)
-            if err != nil {
-                return nil, nil, err
-            }
-            if pathID == "" {
-                return nil, nil, nil
-            }
-        }
+		for _, entry := range entries {
+			storage.savePathID(entry.Name, entry.ID)
+			subDirs = append(subDirs, entry.Name+"/")
+		}
+		return subDirs, nil, nil
+	} else if strings.HasPrefix(dir, "snapshots/") {
+		name := dir[len("snapshots/"):]
+		pathID, ok := storage.findPathID(dir)
+		if !ok {
+			pathID, _, _, err = storage.client.ListByName(storage.getPathID("snapshots"), name)
+			if err != nil {
+				return nil, nil, err
+			}
+			if pathID == "" {
+				return nil, nil, nil
+			}
+			storage.savePathID(dir, pathID)
+		}
 
-        entries, err := storage.client.ListEntries(pathID, true)
-        if err != nil {
-            return nil, nil, err
-        }
+		entries, err := storage.client.ListEntries(pathID, true, false)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        files := []string{}
+		files := []string{}
 
-        for _, entry := range entries {
-            storage.savePathID(dir + "/" + entry.Name, entry.ID)
-            files = append(files, entry.Name)
-        }
-        return files, nil, nil
-    } else {
-        files := []string{}
-        sizes := []int64{}
-        for _, parent := range []string {"chunks", "fossils" } {
-            entries, err := storage.client.ListEntries(storage.getPathID(parent), true)
-            if err != nil {
-                return nil, nil, err
-            }
-
-
-            for _, entry := range entries {
-                name := entry.Name
-                if parent == "fossils" {
-                    name += ".fsl"
-                }
-
-                storage.savePathID(parent + "/" + entry.Name, entry.ID)
-                files = append(files, name)
-                sizes = append(sizes, entry.Size)
-            }
-        }
-        return files, sizes, nil
-    }
+		for _, entry := range entries {
+			storage.savePathID(dir+"/"+entry.Name, entry.ID)
+			files = append(files, entry.Name)
+		}
+		return files, nil, nil
+	} else {
+		files := []string{}
+		sizes := []int64{}
+		parents := []string{"chunks", "fossils"}
+		for i := 0; i < len(parents); i++ {
+			parent := parents[i]
+			pathID, ok := storage.findPathID(parent)
+			if !ok {
+				continue
+			}
+			entries, err := storage.client.ListEntries(pathID, true, true)
+			if err != nil {
+				return nil, nil, err
+			}
+			for _, entry := range entries {
+				if entry.Kind != "FOLDER" {
+					name := entry.Name
+					if strings.HasPrefix(parent, "fossils") {
+						name = parent + "/" + name + ".fsl"
+						name = name[len("fossils/"):]
+					} else {
+						name = parent + "/" + name
+						name = name[len("chunks/"):]
+					}
+					files = append(files, name)
+					sizes = append(sizes, entry.Size)
+				} else {
+					parents = append(parents, parent+"/"+entry.Name)
+				}
+				storage.savePathID(parent+"/"+entry.Name, entry.ID)
+			}
+		}
+		return files, sizes, nil
+	}
 
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *ACDStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    filePath = storage.convertFilePath(filePath)
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, _, _, err = storage.getIDFromPath(threadIndex, filePath)
-        if err != nil {
-            return err
-        }
-        if fileID == "" {
-            LOG_TRACE("ACD_STORAGE", "File %s has disappeared before deletion", filePath)
-            return nil
-        }
-        storage.savePathID(filePath, fileID)
-    }
+	filePath = storage.convertFilePath(filePath)
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
+	if err != nil {
+		return err
+	}
+	if fileID == "" {
+		LOG_TRACE("ACD_STORAGE", "File '%s' to be deleted does not exist", filePath)
+		return nil
+	}
 
-    err = storage.client.DeleteFile(fileID)
-    if e, ok := err.(ACDError); ok && e.Status == 409 {
-        LOG_DEBUG("ACD_DELETE", "Ignore 409 conflict error")
-        return nil
-    }
-    return err
+	err = storage.client.DeleteFile(fileID)
+	if e, ok := err.(ACDError); ok && e.Status == 409 {
+		LOG_DEBUG("ACD_DELETE", "Ignore 409 conflict error")
+		return nil
+	}
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *ACDStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    from = storage.convertFilePath(from)
-    to = storage.convertFilePath(to)
+	from = storage.convertFilePath(from)
+	to = storage.convertFilePath(to)
 
-    fileID, ok := storage.findPathID(from)
-    if !ok {
-        return fmt.Errorf("Attempting to rename file %s with unknown id", from)
-    }
+	fileID, ok := storage.findPathID(from)
+	if !ok {
+		return fmt.Errorf("Attempting to rename file %s with unknown id", from)
+	}
 
-    fromParentID := storage.getPathID("chunks")
-    toParentID := storage.getPathID("fossils")
+	fromParent := path.Dir(from)
+	fromParentID, err := storage.getIDFromPath(threadIndex, fromParent, false)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", fromParent, err)
+	}
+	if fromParentID == "" {
+		return fmt.Errorf("The parent directory '%s' does not exist", fromParent)
+	}
 
-    if strings.HasPrefix(from, "fossils") {
-        fromParentID, toParentID = toParentID, fromParentID
-    }
+	toParent := path.Dir(to)
+	toParentID, err := storage.getIDFromPath(threadIndex, toParent, true)
+	if err != nil {
+		return fmt.Errorf("Failed to retrieve the id of the parent directory '%s': %v", toParent, err)
+	}
 
-    err = storage.client.MoveFile(fileID, fromParentID, toParentID)
-    if err != nil {
-        if e, ok := err.(ACDError); ok && e.Status == 409 {
-            LOG_DEBUG("ACD_MOVE", "Ignore 409 conflict error")
-        } else {
-            return err
-        }
-    }
+	err = storage.client.MoveFile(fileID, fromParentID, toParentID)
+	if err != nil {
+		if e, ok := err.(ACDError); ok && e.Status == 409 {
+			LOG_DEBUG("ACD_MOVE", "Ignore 409 conflict error")
+		} else {
+			return err
+		}
+	}
 
-    storage.savePathID(to, storage.getPathID(from))
-    storage.deletePathID(from)
+	storage.savePathID(to, storage.getPathID(from))
+	storage.deletePathID(from)
 
-    return nil
+	return nil
 }
 
 // CreateDirectory creates a new directory.
 func (storage *ACDStorage) CreateDirectory(threadIndex int, dir string) (err error) {
 
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    if dir == "chunks" || dir == "snapshots" {
-        return nil
-    }
+	parentPath := path.Dir(dir)
+	if parentPath == "." {
+		parentPath = ""
+	}
+	parentID, ok := storage.findPathID(parentPath)
+	if !ok {
+		return fmt.Errorf("Path directory '%s' has unknown id", parentPath)
+	}
 
-    if strings.HasPrefix(dir, "snapshots/") {
-        name := dir[len("snapshots/"):]
-        dirID, err := storage.client.CreateDirectory(storage.getPathID("snapshots"), name)
-        if err != nil {
-            if e, ok := err.(ACDError); ok && e.Status == 409 {
-                return nil
-            } else {
-                return err
-            }
-        }
-        storage.savePathID(dir, dirID)
-        return nil
+	name := path.Base(dir)
+	dirID, err := storage.client.CreateDirectory(parentID, name)
+	if err != nil {
+		if e, ok := err.(ACDError); ok && e.Status == 409 {
+			return nil
+		} else {
+			return err
+		}
+	}
+	storage.savePathID(dir, dirID)
 
-    }
-
-    return nil
+	return nil
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *ACDStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
-    }
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
+	}
 
-    filePath = storage.convertFilePath(filePath)
-    fileID := ""
-    fileID, isDir, size, err = storage.getIDFromPath(threadIndex, filePath)
-    if err != nil {
-        return false, false, 0, err
-    }
-    if fileID == "" {
-        return false, false, 0, nil
-    }
+	filePath = storage.convertFilePath(filePath)
 
-    return true, isDir, size, nil
-}
+	parentPath := path.Dir(filePath)
+	if parentPath == "." {
+		parentPath = ""
+	}
+	parentID, err := storage.getIDFromPath(threadIndex, parentPath, false)
+	if err != nil {
+		return false, false, 0, err
+	}
+	if parentID == "" {
+		return false, false, 0, nil
+	}
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *ACDStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    parentID := ""
-    filePath = "chunks/" + chunkID
-    realPath := filePath
-    if isFossil {
-        parentID = storage.getPathID("fossils")
-        filePath += ".fsl"
-        realPath = "fossils/" + chunkID + ".fsl"
-    } else {
-        parentID = storage.getPathID("chunks")
-    }
+	name := path.Base(filePath)
+	fileID, isDir, size, err := storage.client.ListByName(parentID, name)
+	if err != nil {
+		return false, false, 0, err
+	}
+	if fileID == "" {
+		return false, false, 0, nil
+	}
 
-    fileID := ""
-    fileID, _, size, err = storage.client.ListByName(parentID, chunkID)
-    if fileID != "" {
-        storage.savePathID(realPath, fileID)
-    }
-    return filePath, fileID != "", size, err
+	storage.savePathID(filePath, fileID)
+	return true, isDir, size, nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *ACDStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    fileID, ok := storage.findPathID(filePath)
-    if !ok {
-        fileID, _, _, err = storage.getIDFromPath(threadIndex, filePath)
-        if err != nil {
-            return err
-        }
-        if fileID == "" {
-            return fmt.Errorf("File path '%s' does not exist", filePath)
-        }
-        storage.savePathID(filePath, fileID)
-    }
+	fileID, err := storage.getIDFromPath(threadIndex, filePath, false)
+	if err != nil {
+		return err
+	}
+	if fileID == "" {
+		return fmt.Errorf("File path '%s' does not exist", filePath)
+	}
 
-    readCloser, _, err := storage.client.DownloadFile(fileID)
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := storage.client.DownloadFile(fileID)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *ACDStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    parent := path.Dir(filePath)
+	parent := path.Dir(filePath)
+	if parent == "." {
+		parent = ""
+	}
 
-    if parent == "." {
-        parent = ""
-    }
+	parentID, err := storage.getIDFromPath(threadIndex, parent, true)
+	if err != nil {
+		return err
+	}
+	if parentID == "" {
+		return fmt.Errorf("File path '%s' does not exist", parent)
+	}
 
-    parentID, ok := storage.findPathID(parent)
+	fileID, err := storage.client.UploadFile(parentID, path.Base(filePath), content, storage.UploadRateLimit/storage.numberOfThreads)
+	if err == nil {
+		storage.savePathID(filePath, fileID)
+		return nil
+	}
 
-    if !ok {
-        parentID, _, _, err = storage.getIDFromPath(threadIndex, parent)
-        if err != nil {
-            return err
-        }
-        if parentID == "" {
-            return fmt.Errorf("File path '%s' does not exist", parent)
-        }
-        storage.savePathID(parent, parentID)
-    }
-
-    fileID, err := storage.client.UploadFile(parentID, path.Base(filePath), content, storage.UploadRateLimit / storage.numberOfThreads)
-    if err == nil {
-        storage.savePathID(filePath, fileID)
-        return nil
-    }
-
-    if e, ok := err.(ACDError); ok && e.Status == 409 {
-        LOG_TRACE("ACD_UPLOAD", "File %s already exists", filePath)
-        return nil
-    } else {
-        return err
-    }
+	if e, ok := err.(ACDError); ok && e.Status == 409 {
+		LOG_TRACE("ACD_UPLOAD", "File %s already exists", filePath)
+		return nil
+	} else {
+		return err
+	}
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *ACDStorage) IsCacheNeeded() (bool) { return true }
+func (storage *ACDStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *ACDStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *ACDStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *ACDStorage) IsStrongConsistent() (bool) { return true }
+func (storage *ACDStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *ACDStorage) IsFastListing() (bool) { return true }
+func (storage *ACDStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *ACDStorage) EnableTestMode() {}

src/duplicacy_azurestorage.go

@@ -5,198 +5,185 @@
 package duplicacy
 
 import (
-    "fmt"
-    "strings"
+	"fmt"
+	"strings"
 
-    "github.com/gilbertchen/azure-sdk-for-go/storage"
+	"github.com/gilbertchen/azure-sdk-for-go/storage"
 )
 
 type AzureStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    clients []*storage.BlobStorageClient
-    container string
+	containers []*storage.Container
 }
 
 func CreateAzureStorage(accountName string, accountKey string,
-                        container string, threads int) (azureStorage *AzureStorage, err error) {
+	containerName string, threads int) (azureStorage *AzureStorage, err error) {
 
-    var clients []*storage.BlobStorageClient
-    for i := 0; i < threads; i++ {
+	var containers []*storage.Container
+	for i := 0; i < threads; i++ {
 
-        client, err := storage.NewBasicClient(accountName, accountKey)
+		client, err := storage.NewBasicClient(accountName, accountKey)
 
-        if err != nil {
-            return nil, err
-        }
+		if err != nil {
+			return nil, err
+		}
 
-        blobService := client.GetBlobService()
-        clients = append(clients, &blobService)
-    }
+		blobService := client.GetBlobService()
+		container := blobService.GetContainerReference(containerName)
+		containers = append(containers, container)
+	}
 
-    exist, err := clients[0].ContainerExists(container)
-    if err != nil {
-        return nil, err
-    }
+	exist, err := containers[0].Exists()
+	if err != nil {
+		return nil, err
+	}
 
-    if !exist {
-        return nil, fmt.Errorf("container %s does not exist", container)
-    }
+	if !exist {
+		return nil, fmt.Errorf("container %s does not exist", containerName)
+	}
 
-    azureStorage = &AzureStorage {
-        clients: clients,
-        container: container,
-    }
+	azureStorage = &AzureStorage{
+		containers: containers,
+	}
 
-    return
+	azureStorage.DerivedStorage = azureStorage
+	azureStorage.SetDefaultNestingLevels([]int{0}, 0)
+	return
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (azureStorage *AzureStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
 
-    type ListBlobsParameters struct {
-        Prefix     string
-        Delimiter  string
-        Marker     string
-        Include    string
-        MaxResults uint
-        Timeout    uint
-    }
+	type ListBlobsParameters struct {
+		Prefix     string
+		Delimiter  string
+		Marker     string
+		Include    string
+		MaxResults uint
+		Timeout    uint
+	}
 
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
-        dir += "/"
-    }
-    dirLength := len(dir)
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
+	dirLength := len(dir)
 
-    parameters := storage.ListBlobsParameters {
-        Prefix: dir,
-        Delimiter: "",
-    }
+	parameters := storage.ListBlobsParameters{
+		Prefix:    dir,
+		Delimiter: "",
+	}
 
-    subDirs := make(map[string]bool)
+	subDirs := make(map[string]bool)
 
-    for {
+	for {
 
-        results, err := azureStorage.clients[threadIndex].ListBlobs(azureStorage.container, parameters)
-        if err != nil {
-            return nil, nil, err
-        }
+		results, err := azureStorage.containers[threadIndex].ListBlobs(parameters)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        if dir == "snapshots/" {
-            for _, blob := range results.Blobs {
-                 name := strings.Split(blob.Name[dirLength:], "/")[0]
-                 subDirs[name + "/"] = true
-            }
-        } else {
-            for _, blob := range results.Blobs {
-                files = append(files, blob.Name[dirLength:])
-                sizes = append(sizes, blob.Properties.ContentLength)
-            }
-        }
+		if dir == "snapshots/" {
+			for _, blob := range results.Blobs {
+				name := strings.Split(blob.Name[dirLength:], "/")[0]
+				subDirs[name+"/"] = true
+			}
+		} else {
+			for _, blob := range results.Blobs {
+				files = append(files, blob.Name[dirLength:])
+				sizes = append(sizes, blob.Properties.ContentLength)
+			}
+		}
 
-        if results.NextMarker == "" {
-            break
-        }
+		if results.NextMarker == "" {
+			break
+		}
 
-        parameters.Marker = results.NextMarker
-    }
+		parameters.Marker = results.NextMarker
+	}
 
-    if dir == "snapshots/" {
+	if dir == "snapshots/" {
 
-        for subDir, _ := range subDirs {
-            files = append(files, subDir)
-        }
+		for subDir := range subDirs {
+			files = append(files, subDir)
+		}
 
-    }
+	}
 
-    return files, sizes, nil
+	return files, sizes, nil
 
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *AzureStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    _, err = storage.clients[threadIndex].DeleteBlobIfExists(storage.container, filePath)
-    return err
+	_, err = storage.containers[threadIndex].GetBlobReference(filePath).DeleteIfExists(nil)
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *AzureStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    source := storage.clients[threadIndex].GetBlobURL(storage.container, from)
-    err = storage.clients[threadIndex].CopyBlob(storage.container, to, source)
-    if err != nil {
-        return err
-    }
-    return storage.DeleteFile(threadIndex, from)
+	source := storage.containers[threadIndex].GetBlobReference(from)
+	destination := storage.containers[threadIndex].GetBlobReference(to)
+	err = destination.Copy(source.GetURL(), nil)
+	if err != nil {
+		return err
+	}
+	return storage.DeleteFile(threadIndex, from)
 }
 
 // CreateDirectory creates a new directory.
 func (storage *AzureStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    return nil
+	return nil
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *AzureStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
-    properties, err := storage.clients[threadIndex].GetBlobProperties(storage.container, filePath)
-    if err != nil {
-        if strings.Contains(err.Error(), "404") {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
+	blob := storage.containers[threadIndex].GetBlobReference(filePath)
+	err = blob.GetProperties(nil)
+	if err != nil {
+		if strings.Contains(err.Error(), "404") {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
-    return true, false, properties.ContentLength, nil
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *AzureStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
+	return true, false, blob.Properties.ContentLength, nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *AzureStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    readCloser, err := storage.clients[threadIndex].GetBlob(storage.container, filePath)
-    if err != nil {
-        return err
-    }
+	readCloser, err := storage.containers[threadIndex].GetBlobReference(filePath).Get(nil)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.clients))
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/len(storage.containers))
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *AzureStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-   reader := CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.clients))
-   return storage.clients[threadIndex].CreateBlockBlobFromReader(storage.container, filePath, uint64(len(content)), reader, nil)
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.containers))
+	blob := storage.containers[threadIndex].GetBlobReference(filePath)
+	return blob.CreateBlockBlobFromReader(reader, nil)
 
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *AzureStorage) IsCacheNeeded() (bool) { return true }
+func (storage *AzureStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *AzureStorage)  IsMoveFileImplemented() (bool) { return true }
+func (storage *AzureStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *AzureStorage) IsStrongConsistent() (bool) { return true }
+func (storage *AzureStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *AzureStorage) IsFastListing() (bool) { return true }
+func (storage *AzureStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *AzureStorage) EnableTestMode() {}

src/duplicacy_b2client_test.go

@@ -5,129 +5,129 @@
 package duplicacy
 
 import (
-    "testing"
-    "crypto/sha256"
-    "encoding/hex"
-    "encoding/json"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"testing"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
-    "io"
-    "io/ioutil"
+	crypto_rand "crypto/rand"
+	"io"
+	"io/ioutil"
+	"math/rand"
 )
 
 func createB2ClientForTest(t *testing.T) (*B2Client, string) {
-    config, err := ioutil.ReadFile("test_storage.conf")
-    if err != nil {
-        t.Errorf("Failed to read config file: %v", err)
-        return nil, ""
-    }
+	config, err := ioutil.ReadFile("test_storage.conf")
+	if err != nil {
+		t.Errorf("Failed to read config file: %v", err)
+		return nil, ""
+	}
 
-    storages := make(map[string]map[string]string)
+	storages := make(map[string]map[string]string)
 
-    err = json.Unmarshal(config, &storages)
-    if err != nil {
-        t.Errorf("Failed to parse config file: %v", err)
-        return nil, ""
-    }
+	err = json.Unmarshal(config, &storages)
+	if err != nil {
+		t.Errorf("Failed to parse config file: %v", err)
+		return nil, ""
+	}
 
-    b2, found := storages["b2"]
-    if !found {
-        t.Errorf("Failed to find b2 config")
-        return nil, ""
-    }
+	b2, found := storages["b2"]
+	if !found {
+		t.Errorf("Failed to find b2 config")
+		return nil, ""
+	}
 
-    return NewB2Client(b2["account"], b2["key"]), b2["bucket"]
+	return NewB2Client(b2["account"], b2["key"], b2["directory"], 1), b2["bucket"]
 
 }
 
 func TestB2Client(t *testing.T) {
 
-    b2Client, bucket := createB2ClientForTest(t)
-    if b2Client == nil {
-        return
-    }
+	b2Client, bucket := createB2ClientForTest(t)
+	if b2Client == nil {
+		return
+	}
 
-    b2Client.TestMode = true
+	b2Client.TestMode = true
 
-    err := b2Client.AuthorizeAccount()
-    if err != nil {
-        t.Errorf("Failed to authorize the b2 account: %v", err)
-        return
-    }
+	err := b2Client.AuthorizeAccount(0)
+	if err != nil {
+		t.Errorf("Failed to authorize the b2 account: %v", err)
+		return
+	}
 
-    err = b2Client.FindBucket(bucket)
-    if err != nil {
-        t.Errorf("Failed to find bucket '%s': %v", bucket, err)
-        return
-    }
+	err = b2Client.FindBucket(bucket)
+	if err != nil {
+		t.Errorf("Failed to find bucket '%s': %v", bucket, err)
+		return
+	}
 
-    testDirectory := "b2client_test/"
+	testDirectory := "b2client_test/"
 
-    files, err := b2Client.ListFileNames(testDirectory, false, false)
-    if err != nil {
-        t.Errorf("Failed to list files: %v", err)
-        return
-    }
+	files, err := b2Client.ListFileNames(0, testDirectory, false, false)
+	if err != nil {
+		t.Errorf("Failed to list files: %v", err)
+		return
+	}
 
-    for _, file := range files {
-        err = b2Client.DeleteFile(file.FileName, file.FileID)
-        if err != nil {
-            t.Errorf("Failed to delete file '%s': %v", file.FileName, err)
-        }
-    }
+	for _, file := range files {
+		err = b2Client.DeleteFile(0, file.FileName, file.FileID)
+		if err != nil {
+			t.Errorf("Failed to delete file '%s': %v", file.FileName, err)
+		}
+	}
 
-    maxSize := 10000
-    for i := 0; i < 20; i++ {
-        size := rand.Int() % maxSize + 1
-        content := make([]byte, size)
-        _, err := crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+	maxSize := 10000
+	for i := 0; i < 20; i++ {
+		size := rand.Int()%maxSize + 1
+		content := make([]byte, size)
+		_, err := crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-        hash := sha256.Sum256(content)
-        name := hex.EncodeToString(hash[:])
+		hash := sha256.Sum256(content)
+		name := hex.EncodeToString(hash[:])
 
-        err = b2Client.UploadFile(testDirectory + name, content, 100)
-        if err != nil {
-            t.Errorf("Error uploading file '%s': %v", name, err)
-            return
-        }
-    }
+		err = b2Client.UploadFile(0, testDirectory+name, content, 100)
+		if err != nil {
+			t.Errorf("Error uploading file '%s': %v", name, err)
+			return
+		}
+	}
 
-    files, err = b2Client.ListFileNames(testDirectory, false, false)
-    if err != nil {
-        t.Errorf("Failed to list files: %v", err)
-        return
-    }
+	files, err = b2Client.ListFileNames(0, testDirectory, false, false)
+	if err != nil {
+		t.Errorf("Failed to list files: %v", err)
+		return
+	}
 
-    for _, file := range files {
+	for _, file := range files {
 
-        readCloser, _, err := b2Client.DownloadFile(file.FileName)
-        if err != nil {
-            t.Errorf("Error downloading file '%s': %v", file.FileName, err)
-            return
-        }
+		readCloser, _, err := b2Client.DownloadFile(0, file.FileName)
+		if err != nil {
+			t.Errorf("Error downloading file '%s': %v", file.FileName, err)
+			return
+		}
 
-        defer readCloser.Close()
+		defer readCloser.Close()
 
-        hasher := sha256.New()
-        _, err = io.Copy(hasher, readCloser)
+		hasher := sha256.New()
+		_, err = io.Copy(hasher, readCloser)
 
-        hash := hex.EncodeToString(hasher.Sum(nil))
+		hash := hex.EncodeToString(hasher.Sum(nil))
 
-        if testDirectory + hash != file.FileName {
-            t.Errorf("File %s has hash %s", file.FileName, hash)
-        }
+		if testDirectory+hash != file.FileName {
+			t.Errorf("File %s has hash %s", file.FileName, hash)
+		}
 
-    }
+	}
 
-    for _, file := range files {
-        err = b2Client.DeleteFile(file.FileName, file.FileID)
-        if err != nil {
-            t.Errorf("Failed to delete file '%s': %v", file.FileName, err)
-        }
-    }
+	for _, file := range files {
+		err = b2Client.DeleteFile(0, file.FileName, file.FileID)
+		if err != nil {
+			t.Errorf("Failed to delete file '%s': %v", file.FileName, err)
+		}
+	}
 }

src/duplicacy_b2storage.go

@@ -5,251 +5,237 @@
 package duplicacy
 
 import (
-    "strings"
+	"strings"
 )
 
 type B2Storage struct {
-    RateLimitedStorage
+	StorageBase
 
-    clients []*B2Client
+	client *B2Client
 }
 
 // CreateB2Storage creates a B2 storage object.
-func CreateB2Storage(accountID string, applicationKey string, bucket string, threads int) (storage *B2Storage, err error) {
+func CreateB2Storage(accountID string, applicationKey string, bucket string, storageDir string, threads int) (storage *B2Storage, err error) {
 
-    var clients []*B2Client
+	client := NewB2Client(accountID, applicationKey, storageDir, threads)
 
-    for i := 0; i < threads; i++ {
-        client := NewB2Client(accountID, applicationKey)
+	err = client.AuthorizeAccount(0)
+	if err != nil {
+		return nil, err
+	}
 
-        err = client.AuthorizeAccount()
-        if err != nil {
-            return nil, err
-        }
+	err = client.FindBucket(bucket)
+	if err != nil {
+		return nil, err
+	}
 
-        err = client.FindBucket(bucket)
-        if err != nil {
-            return nil, err
-        }
+	storage = &B2Storage{
+		client: client,
+	}
 
-        clients = append(clients, client)
-    }
-
-    storage = &B2Storage {
-        clients: clients,
-    }
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *B2Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
-    length := len(dir) + 1
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
+	length := len(dir) + 1
 
-    includeVersions := false
-    if dir == "chunks" {
-        includeVersions = true
-    }
+	includeVersions := false
+	if dir == "chunks" {
+		includeVersions = true
+	}
 
-    entries, err := storage.clients[threadIndex].ListFileNames(dir, false, includeVersions)
-    if err != nil {
-        return nil, nil, err
-    }
+	entries, err := storage.client.ListFileNames(threadIndex, dir, false, includeVersions)
+	if err != nil {
+		return nil, nil, err
+	}
 
-    if dir == "snapshots" {
+	if dir == "snapshots" {
 
-        subDirs := make(map[string]bool)
+		subDirs := make(map[string]bool)
 
-        for _, entry := range entries {
-            name := entry.FileName[length:]
-            subDir := strings.Split(name, "/")[0]
-            subDirs[subDir + "/"] = true
-        }
+		for _, entry := range entries {
+			name := entry.FileName[length:]
+			subDir := strings.Split(name, "/")[0]
+			subDirs[subDir+"/"] = true
+		}
 
-        for subDir, _ := range subDirs {
-            files = append(files, subDir)
-        }
-    } else if dir == "chunks" {
-        lastFile := ""
-        for _, entry := range entries {
-            if entry.FileName == lastFile {
-                continue
-            }
-            lastFile = entry.FileName
-            if entry.Action == "hide" {
-                files = append(files, entry.FileName[length:] + ".fsl")
-            } else {
-                files = append(files, entry.FileName[length:])
-            }
-            sizes = append(sizes, entry.Size)
-        }
-    } else {
-        for _, entry := range entries {
-            files = append(files, entry.FileName[length:])
-        }
-    }
+		for subDir := range subDirs {
+			files = append(files, subDir)
+		}
+	} else if dir == "chunks" {
+		lastFile := ""
+		for _, entry := range entries {
+			if entry.FileName == lastFile {
+				continue
+			}
+			lastFile = entry.FileName
+			if entry.Action == "hide" {
+				files = append(files, entry.FileName[length:]+".fsl")
+			} else {
+				files = append(files, entry.FileName[length:])
+			}
+			sizes = append(sizes, entry.Size)
+		}
+	} else {
+		for _, entry := range entries {
+			files = append(files, entry.FileName[length:])
+		}
+	}
 
-    return files, sizes, nil
+	return files, sizes, nil
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *B2Storage) DeleteFile(threadIndex int, filePath string) (err error) {
 
-    if strings.HasSuffix(filePath, ".fsl") {
-        filePath = filePath[:len(filePath) - len(".fsl")]
-        entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, true)
-        if err != nil {
-            return err
-        }
+	if strings.HasSuffix(filePath, ".fsl") {
+		filePath = filePath[:len(filePath)-len(".fsl")]
+		entries, err := storage.client.ListFileNames(threadIndex, filePath, true, true)
+		if err != nil {
+			return err
+		}
 
-        toBeDeleted := false
+		toBeDeleted := false
 
-        for _, entry := range entries {
-            if entry.FileName != filePath || (!toBeDeleted && entry.Action != "hide" ) {
-                continue
-            }
+		for _, entry := range entries {
+			if entry.FileName != filePath || (!toBeDeleted && entry.Action != "hide") {
+				continue
+			}
 
-            toBeDeleted = true
+			toBeDeleted = true
 
-            err = storage.clients[threadIndex].DeleteFile(filePath, entry.FileID)
-            if err != nil {
-                return err
-            }
-        }
+			err = storage.client.DeleteFile(threadIndex, filePath, entry.FileID)
+			if err != nil {
+				return err
+			}
+		}
 
-        return nil
+		return nil
 
-    } else {
-        entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, false)
-        if err != nil {
-            return err
-        }
+	} else {
+		entries, err := storage.client.ListFileNames(threadIndex, filePath, true, false)
+		if err != nil {
+			return err
+		}
 
-        if len(entries) == 0 {
-            return nil
-        }
-        return storage.clients[threadIndex].DeleteFile(filePath, entries[0].FileID)
-    }
+		if len(entries) == 0 {
+			return nil
+		}
+		return storage.client.DeleteFile(threadIndex, filePath, entries[0].FileID)
+	}
 }
 
 // MoveFile renames the file.
 func (storage *B2Storage) MoveFile(threadIndex int, from string, to string) (err error) {
 
-    filePath := ""
+	filePath := ""
 
-    if strings.HasSuffix(from, ".fsl") {
-        filePath = to
-        if from != to + ".fsl" {
-            filePath = ""
-        }
-    } else if strings.HasSuffix(to, ".fsl") {
-        filePath = from
-        if to != from + ".fsl" {
-            filePath = ""
-        }
-    }
+	if strings.HasSuffix(from, ".fsl") {
+		filePath = to
+		if from != to+".fsl" {
+			filePath = ""
+		}
+	} else if strings.HasSuffix(to, ".fsl") {
+		filePath = from
+		if to != from+".fsl" {
+			filePath = ""
+		}
+	}
 
-    if filePath == "" {
-        LOG_FATAL("STORAGE_MOVE", "Moving file '%s' to '%s' is not supported", from, to)
-        return nil
-    }
+	if filePath == "" {
+		LOG_FATAL("STORAGE_MOVE", "Moving file '%s' to '%s' is not supported", from, to)
+		return nil
+	}
 
-    if filePath == from {
-        _, err = storage.clients[threadIndex].HideFile(from)
-        return err
-    } else {
-        entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, true)
-        if err != nil {
-            return err
-        }
-        if len(entries) == 0 || entries[0].FileName != filePath || entries[0].Action != "hide" {
-            return nil
-        }
+	if filePath == from {
+		_, err = storage.client.HideFile(threadIndex, from)
+		return err
+	} else {
+		entries, err := storage.client.ListFileNames(threadIndex, filePath, true, true)
+		if err != nil {
+			return err
+		}
+		if len(entries) == 0 || entries[0].FileName != filePath || entries[0].Action != "hide" {
+			return nil
+		}
 
-        return storage.clients[threadIndex].DeleteFile(filePath, entries[0].FileID)
-    }
+		return storage.client.DeleteFile(threadIndex, filePath, entries[0].FileID)
+	}
 }
 
 // CreateDirectory creates a new directory.
 func (storage *B2Storage) CreateDirectory(threadIndex int, dir string) (err error) {
-    return nil
+	return nil
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *B2Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
-    isFossil := false
-    if strings.HasSuffix(filePath, ".fsl") {
-        isFossil = true
-        filePath = filePath[:len(filePath) - len(".fsl")]
-    }
+	isFossil := false
+	if strings.HasSuffix(filePath, ".fsl") {
+		isFossil = true
+		filePath = filePath[:len(filePath)-len(".fsl")]
+	}
 
-    entries, err := storage.clients[threadIndex].ListFileNames(filePath, true, isFossil)
-    if err != nil {
-        return false, false, 0, err
-    }
+	entries, err := storage.client.ListFileNames(threadIndex, filePath, true, isFossil)
+	if err != nil {
+		return false, false, 0, err
+	}
 
-    if len(entries) == 0 || entries[0].FileName != filePath {
-        return false, false, 0, nil
-    }
+	if len(entries) == 0 || entries[0].FileName != filePath {
+		return false, false, 0, nil
+	}
 
-    if isFossil {
-        if entries[0].Action == "hide" {
-            return true, false, entries[0].Size, nil
-        } else {
-            return false, false, 0, nil
-        }
-    }
-    return true, false, entries[0].Size, nil
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *B2Storage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-    return filePath, exist, size, err
+	if isFossil {
+		if entries[0].Action == "hide" {
+			return true, false, entries[0].Size, nil
+		} else {
+			return false, false, 0, nil
+		}
+	}
+	return true, false, entries[0].Size, nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *B2Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-    readCloser, _, err := storage.clients[threadIndex].DownloadFile(filePath)
-    if err != nil {
-        return err
-    }
+	filePath = strings.Replace(filePath, " ", "%20", -1)
+	readCloser, _, err := storage.client.DownloadFile(threadIndex, filePath)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.clients))
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.client.Threads)
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *B2Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    return storage.clients[threadIndex].UploadFile(filePath, content, storage.UploadRateLimit / len(storage.clients))
+	filePath = strings.Replace(filePath, " ", "%20", -1)
+	return storage.client.UploadFile(threadIndex, filePath, content, storage.UploadRateLimit/storage.client.Threads)
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *B2Storage) IsCacheNeeded() (bool) { return true }
+func (storage *B2Storage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *B2Storage) IsMoveFileImplemented() (bool) { return true }
+func (storage *B2Storage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *B2Storage) IsStrongConsistent() (bool) { return true }
+func (storage *B2Storage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *B2Storage) IsFastListing() (bool) { return true }
+func (storage *B2Storage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *B2Storage) EnableTestMode() {
-    for _, client := range storage.clients {
-        client.TestMode = true
-    }
+	storage.client.TestMode = true
 }

src/duplicacy_backupmanager_test.go

@@ -5,290 +5,362 @@
 package duplicacy
 
 import (
-    "os"
-    "io"
-    "path"
-    "testing"
-    "math/rand"
-    "encoding/hex"
-    "time"
-    "crypto/sha256"
-    crypto_rand "crypto/rand"
+	crypto_rand "crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"io"
+	"math/rand"
+	"os"
+	"path"
+	"testing"
+	"time"
 
-    "runtime/debug"
+	"runtime/debug"
 )
 
 func createRandomFile(path string, maxSize int) {
-    file, err := os.OpenFile(path, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0644)
-    if err != nil {
-        LOG_ERROR("RANDOM_FILE", "Can't open %s for writing: %v", path, err)
-        return
-    }
+	file, err := os.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+	if err != nil {
+		LOG_ERROR("RANDOM_FILE", "Can't open %s for writing: %v", path, err)
+		return
+	}
 
-    defer file.Close()
+	defer file.Close()
 
-    size := maxSize / 2 + rand.Int() % (maxSize / 2)
+	size := maxSize/2 + rand.Int()%(maxSize/2)
 
-    buffer := make([]byte, 32 * 1024)
-    for size > 0 {
-        bytes := size
-        if bytes > cap(buffer) {
-            bytes = cap(buffer)
-        }
-        crypto_rand.Read(buffer[:bytes])
-        bytes, err = file.Write(buffer[:bytes])
-        if err != nil {
-            LOG_ERROR("RANDOM_FILE", "Failed to write to %s: %v", path, err)
-            return
-        }
-        size -= bytes
-    }
+	buffer := make([]byte, 32*1024)
+	for size > 0 {
+		bytes := size
+		if bytes > cap(buffer) {
+			bytes = cap(buffer)
+		}
+		crypto_rand.Read(buffer[:bytes])
+		bytes, err = file.Write(buffer[:bytes])
+		if err != nil {
+			LOG_ERROR("RANDOM_FILE", "Failed to write to %s: %v", path, err)
+			return
+		}
+		size -= bytes
+	}
 }
 
 func modifyFile(path string, portion float32) {
 
-    stat, err := os.Stat(path)
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Can't stat the file %s: %v", path, err)
-        return
-    }
+	stat, err := os.Stat(path)
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Can't stat the file %s: %v", path, err)
+		return
+	}
 
-    modifiedTime := stat.ModTime()
+	modifiedTime := stat.ModTime()
 
-    file, err := os.OpenFile(path, os.O_WRONLY, 0644)
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Can't open %s for writing: %v", path, err)
-        return
-    }
+	file, err := os.OpenFile(path, os.O_WRONLY, 0644)
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Can't open %s for writing: %v", path, err)
+		return
+	}
 
-    defer func() {
-        if file != nil {
-            file.Close()
-        }
-    } ()
+	defer func() {
+		if file != nil {
+			file.Close()
+		}
+	}()
 
-    size, err := file.Seek(0, 2)
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Can't seek to the end of the file %s: %v", path, err)
-        return
-    }
+	size, err := file.Seek(0, 2)
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Can't seek to the end of the file %s: %v", path, err)
+		return
+	}
 
-    length := int (float32(size) * portion)
-    start := rand.Int() % (int(size) - length)
+	length := int(float32(size) * portion)
+	start := rand.Int() % (int(size) - length)
 
-    _, err = file.Seek(int64(start), 0)
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Can't seek to the offset %d: %v", start, err)
-        return
-    }
+	_, err = file.Seek(int64(start), 0)
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Can't seek to the offset %d: %v", start, err)
+		return
+	}
 
-    buffer := make([]byte, length)
-    crypto_rand.Read(buffer)
+	buffer := make([]byte, length)
+	crypto_rand.Read(buffer)
 
-    _, err = file.Write(buffer)
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Failed to write to %s: %v", path, err)
-        return
-    }
+	_, err = file.Write(buffer)
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Failed to write to %s: %v", path, err)
+		return
+	}
 
-    file.Close()
-    file = nil
+	file.Close()
+	file = nil
 
-    // Add 2 seconds to the modified time for the changes to be detectable in quick mode.
-    modifiedTime = modifiedTime.Add(time.Second * 2)
-    err = os.Chtimes(path, modifiedTime, modifiedTime)
+	// Add 2 seconds to the modified time for the changes to be detectable in quick mode.
+	modifiedTime = modifiedTime.Add(time.Second * 2)
+	err = os.Chtimes(path, modifiedTime, modifiedTime)
 
-    if err != nil {
-        LOG_ERROR("MODIFY_FILE", "Failed to change the modification time of %s: %v", path, err)
-        return
-    }
+	if err != nil {
+		LOG_ERROR("MODIFY_FILE", "Failed to change the modification time of %s: %v", path, err)
+		return
+	}
+}
+
+func checkExistence(t *testing.T, path string, exists bool, isDir bool) {
+	stat, err := os.Stat(path)
+	if exists {
+		if err != nil {
+			t.Errorf("%s does not exist: %v", path, err)
+		} else if isDir {
+			if !stat.Mode().IsDir() {
+				t.Errorf("%s is not a directory", path)
+			}
+		} else {
+			if stat.Mode().IsDir() {
+				t.Errorf("%s is not a file", path)
+			}
+		}
+	} else {
+		if err == nil || !os.IsNotExist(err) {
+			t.Errorf("%s may exist: %v", path, err)
+		}
+	}
 }
 
 func truncateFile(path string) {
-    file, err := os.OpenFile(path, os.O_WRONLY, 0644)
-    if err != nil {
-        LOG_ERROR("TRUNCATE_FILE", "Can't open %s for writing: %v", path, err)
-        return
-    }
+	file, err := os.OpenFile(path, os.O_WRONLY, 0644)
+	if err != nil {
+		LOG_ERROR("TRUNCATE_FILE", "Can't open %s for writing: %v", path, err)
+		return
+	}
 
-    defer file.Close()
+	defer file.Close()
 
-    oldSize, err := file.Seek(0, 2)
-    if err != nil {
-        LOG_ERROR("TRUNCATE_FILE", "Can't seek to the end of the file %s: %v", path, err)
-        return
-    }
+	oldSize, err := file.Seek(0, 2)
+	if err != nil {
+		LOG_ERROR("TRUNCATE_FILE", "Can't seek to the end of the file %s: %v", path, err)
+		return
+	}
 
-    newSize := rand.Int63() % oldSize
+	newSize := rand.Int63() % oldSize
 
-    err = file.Truncate(newSize)
-    if err != nil {
-        LOG_ERROR("TRUNCATE_FILE", "Can't truncate the file %s to size %d: %v", path, newSize, err)
-        return
-    }
+	err = file.Truncate(newSize)
+	if err != nil {
+		LOG_ERROR("TRUNCATE_FILE", "Can't truncate the file %s to size %d: %v", path, newSize, err)
+		return
+	}
 }
 
 func getFileHash(path string) (hash string) {
 
-    file, err := os.Open(path)
-      if err != nil {
-        LOG_ERROR("FILE_HASH", "Can't open %s for reading: %v", path, err)
-        return ""
-      }
+	file, err := os.Open(path)
+	if err != nil {
+		LOG_ERROR("FILE_HASH", "Can't open %s for reading: %v", path, err)
+		return ""
+	}
 
-    defer file.Close()
+	defer file.Close()
 
-    hasher := sha256.New()
-    _, err = io.Copy(hasher, file)
-      if err != nil {
-        LOG_ERROR("FILE_HASH", "Can't read file %s: %v", path, err)
-        return ""
-      }
+	hasher := sha256.New()
+	_, err = io.Copy(hasher, file)
+	if err != nil {
+		LOG_ERROR("FILE_HASH", "Can't read file %s: %v", path, err)
+		return ""
+	}
 
-      return hex.EncodeToString(hasher.Sum(nil))
+	return hex.EncodeToString(hasher.Sum(nil))
 }
 
 func TestBackupManager(t *testing.T) {
 
-    rand.Seed(time.Now().UnixNano())
-    setTestingT(t)
-    SetLoggingLevel(INFO)
+	rand.Seed(time.Now().UnixNano())
+	setTestingT(t)
+	SetLoggingLevel(INFO)
 
-    defer func() {
-        if r := recover(); r != nil {
-            switch e := r.(type) {
-            case Exception:
-                t.Errorf("%s %s", e.LogID, e.Message)
-                debug.PrintStack()
-            default:
-                t.Errorf("%v", e)
-                debug.PrintStack()
-            }
-        }
-    } ()
+	defer func() {
+		if r := recover(); r != nil {
+			switch e := r.(type) {
+			case Exception:
+				t.Errorf("%s %s", e.LogID, e.Message)
+				debug.PrintStack()
+			default:
+				t.Errorf("%v", e)
+				debug.PrintStack()
+			}
+		}
+	}()
 
-    testDir := path.Join(os.TempDir(), "duplicacy_test")
-    os.RemoveAll(testDir)
-    os.MkdirAll(testDir, 0700)
+	testDir := path.Join(os.TempDir(), "duplicacy_test")
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)
 
-    os.Mkdir(testDir + "/repository1", 0700)
-    os.Mkdir(testDir + "/repository1/dir1", 0700)
+	os.Mkdir(testDir+"/repository1", 0700)
+	os.Mkdir(testDir+"/repository1/dir1", 0700)
+	os.Mkdir(testDir+"/repository1/.duplicacy", 0700)
+	os.Mkdir(testDir+"/repository2", 0700)
+	os.Mkdir(testDir+"/repository2/.duplicacy", 0700)
 
-    maxFileSize := 1000000
-    //maxFileSize := 200000
+	maxFileSize := 1000000
+	//maxFileSize := 200000
 
-    createRandomFile(testDir + "/repository1/file1", maxFileSize)
-    createRandomFile(testDir + "/repository1/file2", maxFileSize)
-    createRandomFile(testDir + "/repository1/dir1/file3", maxFileSize)
+	createRandomFile(testDir+"/repository1/file1", maxFileSize)
+	createRandomFile(testDir+"/repository1/file2", maxFileSize)
+	createRandomFile(testDir+"/repository1/dir1/file3", maxFileSize)
 
-    threads := 1
+	threads := 1
 
-    storage, err := loadStorage(testDir + "/storage", threads)
-    if err != nil {
-        t.Errorf("Failed to create storage: %v", err)
-        return
-    }
+	storage, err := loadStorage(testDir+"/storage", threads)
+	if err != nil {
+		t.Errorf("Failed to create storage: %v", err)
+		return
+	}
 
-    delay := 0
-    if _, ok := storage.(*ACDStorage); ok {
-        delay = 1
-    }
-    if _, ok := storage.(*OneDriveStorage); ok {
-        delay = 5 
-    }
+	delay := 0
+	if _, ok := storage.(*ACDStorage); ok {
+		delay = 1
+	}
+	if _, ok := storage.(*OneDriveStorage); ok {
+		delay = 5
+	}
 
-    password := "duplicacy"
+	password := "duplicacy"
 
-    cleanStorage(storage)
+	cleanStorage(storage)
 
-    time.Sleep(time.Duration(delay) * time.Second)
-    if testFixedChunkSize {
-        if !ConfigStorage(storage, 100, 64 * 1024, 64 * 1024, 64 * 1024, password, nil) {
-            t.Errorf("Failed to initialize the storage")
-        }
-    } else {
-        if !ConfigStorage(storage, 100, 64 * 1024, 256 * 1024, 16 * 1024, password, nil) {
-            t.Errorf("Failed to initialize the storage")
-        }
-    }
+	time.Sleep(time.Duration(delay) * time.Second)
+	if testFixedChunkSize {
+		if !ConfigStorage(storage, 16384, 100, 64*1024, 64*1024, 64*1024, password, nil, false) {
+			t.Errorf("Failed to initialize the storage")
+		}
+	} else {
+		if !ConfigStorage(storage, 16384, 100, 64*1024, 256*1024, 16*1024, password, nil, false) {
+			t.Errorf("Failed to initialize the storage")
+		}
+	}
 
+	time.Sleep(time.Duration(delay) * time.Second)
 
-    time.Sleep(time.Duration(delay) * time.Second)
+	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
+	backupManager := CreateBackupManager("host1", storage, testDir, password, "")
+	backupManager.SetupSnapshotCache("default")
 
-    SetDuplicacyPreferencePath(testDir + "/repository1")
-    backupManager := CreateBackupManager("host1", storage, testDir, password)
-    backupManager.SetupSnapshotCache("default")
+	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "first", false, false, 0, false)
+	time.Sleep(time.Duration(delay) * time.Second)
+	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
+	backupManager.Restore(testDir+"/repository2", threads /*inPlace=*/, false /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil)
 
-    SetDuplicacyPreferencePath(testDir + "/repository1")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/true, threads, "first", false, false)
-    time.Sleep(time.Duration(delay) * time.Second)
-    SetDuplicacyPreferencePath(testDir + "/repository2")
-    backupManager.Restore(testDir + "/repository2", threads, /*inPlace=*/false, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/nil)
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
+		if _, err := os.Stat(testDir + "/repository2/" + f); os.IsNotExist(err) {
+			t.Errorf("File %s does not exist", f)
+			continue
+		}
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
-        if _, err := os.Stat(testDir + "/repository2/" + f); os.IsNotExist(err) {
-            t.Errorf("File %s does not exist", f)
-            continue
-        }
+		hash1 := getFileHash(testDir + "/repository1/" + f)
+		hash2 := getFileHash(testDir + "/repository2/" + f)
+		if hash1 != hash2 {
+			t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
+		}
+	}
 
-        hash1 := getFileHash(testDir + "/repository1/" + f)
-        hash2 := getFileHash(testDir + "/repository2/" + f)
-        if hash1 != hash2 {
-            t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
-        }
-    }
+	modifyFile(testDir+"/repository1/file1", 0.1)
+	modifyFile(testDir+"/repository1/file2", 0.2)
+	modifyFile(testDir+"/repository1/dir1/file3", 0.3)
 
-    modifyFile(testDir + "/repository1/file1", 0.1)
-    modifyFile(testDir + "/repository1/file2", 0.2)
-    modifyFile(testDir + "/repository1/dir1/file3", 0.3)
+	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, true, threads, "second", false, false, 0, false)
+	time.Sleep(time.Duration(delay) * time.Second)
+	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
+	backupManager.Restore(testDir+"/repository2", 2 /*inPlace=*/, true /*quickMode=*/, true, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil)
 
-    SetDuplicacyPreferencePath(testDir + "/repository1")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/true, threads, "second", false, false)
-    time.Sleep(time.Duration(delay) * time.Second)
-    SetDuplicacyPreferencePath(testDir + "/repository2")
-    backupManager.Restore(testDir + "/repository2", 2, /*inPlace=*/true, /*quickMode=*/true, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/nil)
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
+		hash1 := getFileHash(testDir + "/repository1/" + f)
+		hash2 := getFileHash(testDir + "/repository2/" + f)
+		if hash1 != hash2 {
+			t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
+		}
+	}
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
-        hash1 := getFileHash(testDir + "/repository1/" + f)
-        hash2 := getFileHash(testDir + "/repository2/" + f)
-        if hash1 != hash2 {
-            t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
-        }
-    }
+	// Truncate file2 and add a few empty directories
+	truncateFile(testDir + "/repository1/file2")
+	os.Mkdir(testDir+"/repository1/dir2", 0700)
+	os.Mkdir(testDir+"/repository1/dir2/dir3", 0700)
+	os.Mkdir(testDir+"/repository1/dir4", 0700)
+	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "third", false, false, 0, false)
+	time.Sleep(time.Duration(delay) * time.Second)
 
-    truncateFile(testDir + "/repository1/file2")
-    SetDuplicacyPreferencePath(testDir + "/repository1")
-    backupManager.Backup(testDir + "/repository1", /*quickMode=*/false, threads, "third", false, false)
-    time.Sleep(time.Duration(delay) * time.Second)
-    SetDuplicacyPreferencePath(testDir + "/repository2")
-    backupManager.Restore(testDir + "/repository2", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/nil)
+	// Create some directories and files under repository2 that will be deleted during restore
+	os.Mkdir(testDir+"/repository2/dir5", 0700)
+	os.Mkdir(testDir+"/repository2/dir5/dir6", 0700)
+	os.Mkdir(testDir+"/repository2/dir7", 0700)
+	createRandomFile(testDir+"/repository2/file4", 100)
+	createRandomFile(testDir+"/repository2/dir5/file5", 100)
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
-        hash1 := getFileHash(testDir + "/repository1/" + f)
-        hash2 := getFileHash(testDir + "/repository2/" + f)
-        if hash1 != hash2 {
-            t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
-        }
-    }
+	SetDuplicacyPreferencePath(testDir + "/repository2/.duplicacy")
+	backupManager.Restore(testDir+"/repository2", 3 /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ true /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, nil)
 
-    // Remove file2 and dir1/file3 and restore them from revision 3
-    os.Remove(testDir + "/repository1/file2")
-    os.Remove(testDir + "/repository1/dir1/file3")
-    backupManager.Restore(testDir + "/repository1", 3, /*inPlace=*/true, /*quickMode=*/false, threads, /*overwrite=*/true,
-                          /*deleteMode=*/false, /*showStatistics=*/false, /*patterns=*/[]string{"+file2", "+dir1/file3", "-*"})
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
+		hash1 := getFileHash(testDir + "/repository1/" + f)
+		hash2 := getFileHash(testDir + "/repository2/" + f)
+		if hash1 != hash2 {
+			t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
+		}
+	}
 
-    for _, f := range []string{ "file1", "file2", "dir1/file3" } {
-        hash1 := getFileHash(testDir + "/repository1/" + f)
-        hash2 := getFileHash(testDir + "/repository2/" + f)
-        if hash1 != hash2 {
-            t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
-        }
-    }
+	// These files/dirs should not exist because deleteMode == true
+	checkExistence(t, testDir+"/repository2/dir5", false, false)
+	checkExistence(t, testDir+"/repository2/dir5/dir6", false, false)
+	checkExistence(t, testDir+"/repository2/dir7", false, false)
+	checkExistence(t, testDir+"/repository2/file4", false, false)
+	checkExistence(t, testDir+"/repository2/dir5/file5", false, false)
 
-    /*buf := make([]byte, 1<<16)
-    runtime.Stack(buf, true)
-    fmt.Printf("%s", buf)*/
+	// These empty dirs should exist
+	checkExistence(t, testDir+"/repository2/dir2", true, true)
+	checkExistence(t, testDir+"/repository2/dir2/dir3", true, true)
+	checkExistence(t, testDir+"/repository2/dir4", true, true)
+
+	// Remove file2 and dir1/file3 and restore them from revision 3
+	os.Remove(testDir + "/repository1/file2")
+	os.Remove(testDir + "/repository1/dir1/file3")
+	SetDuplicacyPreferencePath(testDir + "/repository1/.duplicacy")
+	backupManager.Restore(testDir+"/repository1", 3 /*inPlace=*/, true /*quickMode=*/, false, threads /*overwrite=*/, true,
+		/*deleteMode=*/ false /*setowner=*/, false /*showStatistics=*/, false /*patterns=*/, []string{"+file2", "+dir1/file3", "-*"})
+
+	for _, f := range []string{"file1", "file2", "dir1/file3"} {
+		hash1 := getFileHash(testDir + "/repository1/" + f)
+		hash2 := getFileHash(testDir + "/repository2/" + f)
+		if hash1 != hash2 {
+			t.Errorf("File %s has different hashes: %s vs %s", f, hash1, hash2)
+		}
+	}
+
+	numberOfSnapshots := backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 3 {
+		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{1, 2, 3} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, []int{1} /*tags*/, nil /*retentions*/, nil,
+		/*exhaustive*/ false /*exclusive=*/, false /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
+	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 2 {
+		t.Errorf("Expected 2 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+	backupManager.Backup(testDir+"/repository1" /*quickMode=*/, false, threads, "fourth", false, false, 0, false)
+	backupManager.SnapshotManager.PruneSnapshots("host1", "host1" /*revisions*/, nil /*tags*/, nil /*retentions*/, nil,
+		/*exhaustive*/ false /*exclusive=*/, true /*ignoredIDs*/, nil /*dryRun*/, false /*deleteOnly*/, false /*collectOnly*/, false, 1)
+	numberOfSnapshots = backupManager.SnapshotManager.ListSnapshots( /*snapshotID*/ "host1" /*revisionsToList*/, nil /*tag*/, "" /*showFiles*/, false /*showChunks*/, false)
+	if numberOfSnapshots != 3 {
+		t.Errorf("Expected 3 snapshots but got %d", numberOfSnapshots)
+	}
+	backupManager.SnapshotManager.CheckSnapshots( /*snapshotID*/ "host1" /*revisions*/, []int{2, 3, 4} /*tag*/, "",
+		/*showStatistics*/ false /*showTabular*/, false /*checkFiles*/, false /*searchFossils*/, false /*resurrect*/, false)
+
+	/*buf := make([]byte, 1<<16)
+	  runtime.Stack(buf, true)
+	  fmt.Printf("%s", buf)*/
 }

src/duplicacy_benchmark.go

@@ -0,0 +1,235 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"bytes"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"os"
+	"path/filepath"
+	"time"
+)
+
+func benchmarkSplit(reader *bytes.Reader, fileSize int64, chunkSize int, compression bool, encryption bool, annotation string) {
+
+	config := CreateConfig()
+	config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
+	config.AverageChunkSize = chunkSize
+	config.MaximumChunkSize = chunkSize * 4
+	config.MinimumChunkSize = chunkSize / 4
+	config.ChunkSeed = []byte("duplicacy")
+
+	config.HashKey = DEFAULT_KEY
+	config.IDKey = DEFAULT_KEY
+
+	maker := CreateChunkMaker(config, false)
+
+	startTime := float64(time.Now().UnixNano()) / 1e9
+	numberOfChunks := 0
+	reader.Seek(0, os.SEEK_SET)
+	maker.ForEachChunk(reader,
+		func(chunk *Chunk, final bool) {
+			if compression {
+				key := ""
+				if encryption {
+					key = "0123456789abcdef0123456789abcdef"
+				}
+				err := chunk.Encrypt([]byte(key), "")
+				if err != nil {
+					LOG_ERROR("BENCHMARK_ENCRYPT", "Failed to encrypt the chunk: %v", err)
+				}
+			}
+			config.PutChunk(chunk)
+			numberOfChunks++
+		},
+		func(size int64, hash string) (io.Reader, bool) {
+			return nil, false
+		})
+
+	runningTime := float64(time.Now().UnixNano())/1e9 - startTime
+	speed := int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_SPLIT", "Split %s bytes into %d chunks %s in %.2fs: %s/s", PrettySize(fileSize), numberOfChunks, annotation,
+		runningTime, PrettySize(speed))
+
+	return
+}
+
+func benchmarkRun(threads int, chunkCount int, job func(threadIndex int, chunkIndex int)) {
+	indexChannel := make(chan int, chunkCount)
+	stopChannel := make(chan int, threads)
+	finishChannel := make(chan int, threads)
+
+	// Start the uploading goroutines
+	for i := 0; i < threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case chunkIndex := <-indexChannel:
+					job(threadIndex, chunkIndex)
+					finishChannel <- 0
+				case <-stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
+
+	for i := 0; i < chunkCount; i++ {
+		indexChannel <- i
+	}
+
+	for i := 0; i < chunkCount; i++ {
+		<-finishChannel
+	}
+
+	for i := 0; i < threads; i++ {
+		stopChannel <- 0
+	}
+}
+
+func Benchmark(localDirectory string, storage Storage, fileSize int64, chunkSize int, chunkCount int, uploadThreads int, downloadThreads int) bool {
+
+	filename := filepath.Join(localDirectory, "benchmark.dat")
+
+	defer func() {
+		os.Remove(filename)
+	}()
+
+	LOG_INFO("BENCHMARK_GENERATE", "Generating %s byte random data in memory", PrettySize(fileSize))
+	data := make([]byte, fileSize)
+	_, err := rand.Read(data)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_RAND", "Failed to generate random data: %v", err)
+		return false
+	}
+
+	startTime := float64(time.Now().UnixNano()) / 1e9
+	LOG_INFO("BENCHMARK_WRITE", "Writing random data to local disk")
+	err = ioutil.WriteFile(filename, data, 0600)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_WRITE", "Failed to write the random data: %v", err)
+		return false
+	}
+
+	runningTime := float64(time.Now().UnixNano())/1e9 - startTime
+	speed := int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_WRITE", "Wrote %s bytes in %.2fs: %s/s", PrettySize(fileSize), runningTime, PrettySize(speed))
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	LOG_INFO("BENCHMARK_READ", "Reading the random data from local disk")
+	file, err := os.Open(filename)
+	if err != nil {
+		LOG_ERROR("BENCHMARK_OPEN", "Failed to open the random data file: %v", err)
+		return false
+	}
+	segment := make([]byte, 1024*1024)
+	for err == nil {
+		_, err = file.Read(segment)
+	}
+	if err != io.EOF {
+		LOG_ERROR("BENCHMARK_OPEN", "Failed to read the random data file: %v", err)
+		return false
+	}
+	file.Close()
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(fileSize) / runningTime)
+	LOG_INFO("BENCHMARK_READ", "Read %s bytes in %.2fs: %s/s", PrettySize(fileSize), runningTime, PrettySize(speed))
+
+	buffer := bytes.NewReader(data)
+	benchmarkSplit(buffer, fileSize, chunkSize, false, false, "without compression/encryption")
+	benchmarkSplit(buffer, fileSize, chunkSize, true, false, "with compression but without encryption")
+	benchmarkSplit(buffer, fileSize, chunkSize, true, true, "with compression and encryption")
+
+	storage.CreateDirectory(0, "benchmark")
+	existingFiles, _, err := storage.ListFiles(0, "benchmark/")
+	if err != nil {
+		LOG_ERROR("BENCHMARK_LIST", "Failed to list the benchmark directory: %v", err)
+		return false
+	}
+
+	var existingChunks []string
+	for _, f := range existingFiles {
+		if len(f) > 0 && f[len(f)-1] != '/' {
+			existingChunks = append(existingChunks, "benchmark/"+f)
+		}
+	}
+
+	if len(existingChunks) > 0 {
+		LOG_INFO("BENCHMARK_DELETE", "Deleting %d temporary files from previous benchmark runs", len(existingChunks))
+		benchmarkRun(uploadThreads, len(existingChunks), func(threadIndex int, chunkIndex int) {
+			storage.DeleteFile(threadIndex, existingChunks[chunkIndex])
+		})
+	}
+
+	chunks := make([][]byte, chunkCount)
+	chunkHashes := make([]string, chunkCount)
+	LOG_INFO("BENCHMARK_GENERATE", "Generating %d chunks", chunkCount)
+	for i := 0; i < chunkCount; i++ {
+		chunks[i] = make([]byte, chunkSize)
+		_, err = rand.Read(chunks[i])
+		if err != nil {
+			LOG_ERROR("BENCHMARK_RAND", "Failed to generate random data: %v", err)
+			return false
+		}
+		hashInBytes := sha256.Sum256(chunks[i])
+		chunkHashes[i] = hex.EncodeToString(hashInBytes[:])
+
+	}
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	benchmarkRun(uploadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+		err := storage.UploadFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex), chunks[chunkIndex])
+		if err != nil {
+			LOG_ERROR("BENCHMARK_UPLOAD", "Failed to upload the chunk: %v", err)
+			return
+		}
+	})
+
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(chunkSize*chunkCount) / runningTime)
+	LOG_INFO("BENCHMARK_UPLOAD", "Uploaded %s bytes in %.2fs: %s/s", PrettySize(int64(chunkSize*chunkCount)), runningTime, PrettySize(speed))
+
+	config := CreateConfig()
+
+	startTime = float64(time.Now().UnixNano()) / 1e9
+	hashError := false
+	benchmarkRun(downloadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+		chunk := config.GetChunk()
+		chunk.Reset(false)
+		err := storage.DownloadFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex), chunk)
+		if err != nil {
+			LOG_ERROR("BENCHMARK_DOWNLOAD", "Failed to download the chunk: %v", err)
+			return
+		}
+
+		hashInBytes := sha256.Sum256(chunk.GetBytes())
+		hash := hex.EncodeToString(hashInBytes[:])
+		if hash != chunkHashes[chunkIndex] {
+			LOG_WARN("BENCHMARK_HASH", "Chunk %d has mismatched hashes: %s != %s", chunkIndex, chunkHashes[chunkIndex], hash)
+			hashError = true
+		}
+
+		config.PutChunk(chunk)
+	})
+
+	runningTime = float64(time.Now().UnixNano())/1e9 - startTime
+	speed = int64(float64(chunkSize*chunkCount) / runningTime)
+	LOG_INFO("BENCHMARK_DOWNLOAD", "Downloaded %s bytes in %.2fs: %s/s", PrettySize(int64(chunkSize*chunkCount)), runningTime, PrettySize(speed))
+
+	if !hashError {
+		benchmarkRun(uploadThreads, chunkCount, func(threadIndex int, chunkIndex int) {
+			storage.DeleteFile(threadIndex, fmt.Sprintf("benchmark/chunk%d", chunkIndex))
+		})
+		LOG_INFO("BENCHMARK_DELETE", "Deleted %d temporary files from the storage", chunkCount)
+	}
+
+	return true
+}

src/duplicacy_chunk.go

@@ -5,59 +5,61 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "hash"
-    "bytes"
-    "runtime"
-    "crypto/cipher"
-    "crypto/aes"
-    "crypto/rand"
-    "encoding/hex"
-    "compress/zlib"
-
-    "github.com/bkaradzic/go-lz4"
+	"bytes"
+	"compress/zlib"
+	"crypto/aes"
+	"crypto/cipher"
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"hash"
+	"io"
+	"os"
+	"runtime"
 
+	"github.com/bkaradzic/go-lz4"
 )
 
 // A chunk needs to acquire a new buffer and return the old one for every encrypt/decrypt operation, therefore
 // we maintain a pool of previously used buffers.
-var chunkBufferPool chan *bytes.Buffer = make(chan *bytes.Buffer, runtime.NumCPU() * 16)
+var chunkBufferPool chan *bytes.Buffer = make(chan *bytes.Buffer, runtime.NumCPU()*16)
 
 func AllocateChunkBuffer() (buffer *bytes.Buffer) {
-    select {
-    case buffer = <- chunkBufferPool:
-    default:
-        buffer = new(bytes.Buffer)
-    }
-    return buffer
+	select {
+	case buffer = <-chunkBufferPool:
+	default:
+		buffer = new(bytes.Buffer)
+	}
+	return buffer
 }
 
 func ReleaseChunkBuffer(buffer *bytes.Buffer) {
-    select {
-    case chunkBufferPool <- buffer:
-    default:
-        LOG_INFO("CHUNK_BUFFER", "Discarding a free chunk buffer due to a full pool")
-    }
+	select {
+	case chunkBufferPool <- buffer:
+	default:
+		LOG_INFO("CHUNK_BUFFER", "Discarding a free chunk buffer due to a full pool")
+	}
 }
 
 // Chunk is the object being passed between the chunk maker, the chunk uploader, and chunk downloader.  It can be
 // read and written like a bytes.Buffer, and provides convenient functions to calculate the hash and id of the chunk.
 type Chunk struct {
-    buffer  *bytes.Buffer  // Where the actual data is stored.  It may be nil for hash-only chunks, where chunks
-                           // are only used to compute the hashes
+	buffer *bytes.Buffer // Where the actual data is stored.  It may be nil for hash-only chunks, where chunks
+	// are only used to compute the hashes
 
-    size    int            // The size of data stored.  This field is needed if buffer is nil
+	size int // The size of data stored.  This field is needed if buffer is nil
 
-    hasher hash.Hash       // Keeps track of the hash of data stored in the buffer.  It may be nil, since sometimes
-                           // it isn't necessary to compute the hash, for instance, when the encrypted data is being
-                           // read into the primary buffer
+	hasher hash.Hash // Keeps track of the hash of data stored in the buffer.  It may be nil, since sometimes
+	// it isn't necessary to compute the hash, for instance, when the encrypted data is being
+	// read into the primary buffer
 
-    hash   []byte          // The hash of the chunk data.  It is always in the binary format
-    id     string          // The id of the chunk data (used as the file name for saving the chunk); always in hex format
+	hash []byte // The hash of the chunk data.  It is always in the binary format
+	id   string // The id of the chunk data (used as the file name for saving the chunk); always in hex format
 
-    config *Config         // Every chunk is associated with a Config object.  Which hashing algorithm to use is determined
-                           // by the config
+	config *Config // Every chunk is associated with a Config object.  Which hashing algorithm to use is determined
+	// by the config
 }
 
 // Magic word to identify a duplicacy format encrypted file, plus a version number.
@@ -66,317 +68,330 @@ var ENCRYPTION_HEADER = "duplicacy\000"
 // CreateChunk creates a new chunk.
 func CreateChunk(config *Config, bufferNeeded bool) *Chunk {
 
-    var buffer *bytes.Buffer
+	var buffer *bytes.Buffer
 
-    if bufferNeeded {
-        buffer = AllocateChunkBuffer()
-        buffer.Reset()
-        if buffer.Cap() < config.MaximumChunkSize {
-            buffer.Grow(config.MaximumChunkSize - buffer.Cap())
-        }
-    }
+	if bufferNeeded {
+		buffer = AllocateChunkBuffer()
+		buffer.Reset()
+		if buffer.Cap() < config.MaximumChunkSize {
+			buffer.Grow(config.MaximumChunkSize - buffer.Cap())
+		}
+	}
 
-    return &Chunk {
-        buffer     : buffer,
-        config     : config,
-    }
+	return &Chunk{
+		buffer: buffer,
+		config: config,
+	}
 }
 
 // GetLength returns the length of available data
 func (chunk *Chunk) GetLength() int {
-    if chunk.buffer != nil {
-        return len(chunk.buffer.Bytes())
-    } else {
-        return chunk.size
-    }
+	if chunk.buffer != nil {
+		return len(chunk.buffer.Bytes())
+	} else {
+		return chunk.size
+	}
 }
 
 // GetBytes returns data available in this chunk
-func (chunk *Chunk) GetBytes() [] byte {
-    return chunk.buffer.Bytes()
+func (chunk *Chunk) GetBytes() []byte {
+	return chunk.buffer.Bytes()
 }
 
 // Reset makes the chunk reusable by clearing the existing data in the buffers.  'hashNeeded' indicates whether the
 // hash of the new data to be read is needed.  If the data to be read in is encrypted, there is no need to
 // calculate the hash so hashNeeded should be 'false'.
 func (chunk *Chunk) Reset(hashNeeded bool) {
-    if chunk.buffer != nil {
-        chunk.buffer.Reset()
-    }
-    if hashNeeded {
-        chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
-    } else {
-        chunk.hasher = nil
-    }
-    chunk.hash = nil
-    chunk.id = ""
-    chunk.size = 0
+	if chunk.buffer != nil {
+		chunk.buffer.Reset()
+	}
+	if hashNeeded {
+		chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
+	} else {
+		chunk.hasher = nil
+	}
+	chunk.hash = nil
+	chunk.id = ""
+	chunk.size = 0
 }
 
 // Write implements the Writer interface.
-func (chunk *Chunk) Write(p []byte) (int, error){
+func (chunk *Chunk) Write(p []byte) (int, error) {
 
-    // buffer may be nil, when the chunk is used for computing the hash only.
-    if chunk.buffer == nil {
-        chunk.size += len(p)
-    } else {
-        chunk.buffer.Write(p)
-    }
+	// buffer may be nil, when the chunk is used for computing the hash only.
+	if chunk.buffer == nil {
+		chunk.size += len(p)
+	} else {
+		chunk.buffer.Write(p)
+	}
 
-    // hasher may be nil, when the chunk is used to stored encrypted content
-    if chunk.hasher != nil {
-        chunk.hasher.Write(p)
-    }
-    return len(p), nil
+	// hasher may be nil, when the chunk is used to stored encrypted content
+	if chunk.hasher != nil {
+		chunk.hasher.Write(p)
+	}
+	return len(p), nil
 }
 
 // GetHash returns the chunk hash.
 func (chunk *Chunk) GetHash() string {
-    if (len(chunk.hash) == 0) {
-        chunk.hash = chunk.hasher.Sum(nil)
-    }
+	if len(chunk.hash) == 0 {
+		chunk.hash = chunk.hasher.Sum(nil)
+	}
 
-    return string(chunk.hash)
+	return string(chunk.hash)
 }
 
 // GetID returns the chunk id.
 func (chunk *Chunk) GetID() string {
-    if len(chunk.id) == 0 {
-        if len(chunk.hash) == 0 {
-            chunk.hash = chunk.hasher.Sum(nil)
-        }
+	if len(chunk.id) == 0 {
+		if len(chunk.hash) == 0 {
+			chunk.hash = chunk.hasher.Sum(nil)
+		}
 
-        hasher := chunk.config.NewKeyedHasher(chunk.config.IDKey)
-        hasher.Write([]byte(chunk.hash))
-        chunk.id = hex.EncodeToString(hasher.Sum(nil))
-    }
+		hasher := chunk.config.NewKeyedHasher(chunk.config.IDKey)
+		hasher.Write([]byte(chunk.hash))
+		chunk.id = hex.EncodeToString(hasher.Sum(nil))
+	}
 
-    return chunk.id
+	return chunk.id
 }
 
 func (chunk *Chunk) VerifyID() {
-    hasher := chunk.config.NewKeyedHasher(chunk.config.HashKey)
-    hasher.Write(chunk.buffer.Bytes())
-    hash := hasher.Sum(nil)
-    hasher = chunk.config.NewKeyedHasher(chunk.config.IDKey)
-    hasher.Write([]byte(hash))
-    chunkID := hex.EncodeToString(hasher.Sum(nil))
-    if chunkID != chunk.GetID() {
-        LOG_ERROR("CHUNK_ID", "The chunk id should be %s instead of %s, length: %d", chunkID, chunk.GetID(), len(chunk.buffer.Bytes()))
-    }
+	hasher := chunk.config.NewKeyedHasher(chunk.config.HashKey)
+	hasher.Write(chunk.buffer.Bytes())
+	hash := hasher.Sum(nil)
+	hasher = chunk.config.NewKeyedHasher(chunk.config.IDKey)
+	hasher.Write([]byte(hash))
+	chunkID := hex.EncodeToString(hasher.Sum(nil))
+	if chunkID != chunk.GetID() {
+		LOG_ERROR("CHUNK_ID", "The chunk id should be %s instead of %s, length: %d", chunkID, chunk.GetID(), len(chunk.buffer.Bytes()))
+	}
 }
 
 // Encrypt encrypts the plain data stored in the chunk buffer.  If derivationKey is not nil, the actual
 // encryption key will be HMAC-SHA256(encryptionKey, derivationKey).
 func (chunk *Chunk) Encrypt(encryptionKey []byte, derivationKey string) (err error) {
 
-    var aesBlock cipher.Block
-    var gcm cipher.AEAD
-    var nonce []byte
-    var offset int
+	var aesBlock cipher.Block
+	var gcm cipher.AEAD
+	var nonce []byte
+	var offset int
 
-    encryptedBuffer := AllocateChunkBuffer()
-    encryptedBuffer.Reset()
-    defer func() {
-        ReleaseChunkBuffer(encryptedBuffer)
-    } ()
+	encryptedBuffer := AllocateChunkBuffer()
+	encryptedBuffer.Reset()
+	defer func() {
+		ReleaseChunkBuffer(encryptedBuffer)
+	}()
 
-    if len(encryptionKey) > 0 {
+	if len(encryptionKey) > 0 {
 
-        key := encryptionKey
+		key := encryptionKey
 
-        if len(derivationKey) > 0 {
-            hasher := chunk.config.NewKeyedHasher([]byte(derivationKey))
-            hasher.Write(encryptionKey)
-            key = hasher.Sum(nil)
-        }
+		if len(derivationKey) > 0 {
+			hasher := chunk.config.NewKeyedHasher([]byte(derivationKey))
+			hasher.Write(encryptionKey)
+			key = hasher.Sum(nil)
+		}
 
-        aesBlock, err = aes.NewCipher(key)
-        if err != nil {
-            return err
-        }
+		aesBlock, err = aes.NewCipher(key)
+		if err != nil {
+			return err
+		}
 
-        gcm, err = cipher.NewGCM(aesBlock)
-        if err != nil {
-            return err
-        }
+		gcm, err = cipher.NewGCM(aesBlock)
+		if err != nil {
+			return err
+		}
 
-        // Start with the magic number and the version number.
-        encryptedBuffer.Write([]byte(ENCRYPTION_HEADER))
+		// Start with the magic number and the version number.
+		encryptedBuffer.Write([]byte(ENCRYPTION_HEADER))
 
-        // Followed by the nonce
-        nonce = make([]byte, gcm.NonceSize())
-        _, err := rand.Read(nonce)
-        if err != nil {
-            return err
-        }
-        encryptedBuffer.Write(nonce)
-        offset = encryptedBuffer.Len()
+		// Followed by the nonce
+		nonce = make([]byte, gcm.NonceSize())
+		_, err := rand.Read(nonce)
+		if err != nil {
+			return err
+		}
+		encryptedBuffer.Write(nonce)
+		offset = encryptedBuffer.Len()
 
-    }
+	}
 
-    // offset is either 0 or the length of header + nonce
+	// offset is either 0 or the length of header + nonce
 
-    if chunk.config.CompressionLevel >= -1 && chunk.config.CompressionLevel <= 9 {
-        deflater, _ := zlib.NewWriterLevel(encryptedBuffer, chunk.config.CompressionLevel)
-        deflater.Write(chunk.buffer.Bytes())
-        deflater.Close()
-    } else if chunk.config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
-        encryptedBuffer.Write([]byte("LZ4 "))
-        // Make sure we have enough space in encryptedBuffer
-        availableLength := encryptedBuffer.Cap() - len(encryptedBuffer.Bytes())
-        maximumLength := lz4.CompressBound(len(chunk.buffer.Bytes()))
-        if availableLength < maximumLength {
-            encryptedBuffer.Grow(maximumLength - availableLength)
-        }
-        written, err := lz4.Encode(encryptedBuffer.Bytes()[offset + 4:], chunk.buffer.Bytes())
-        if err != nil {
-            return fmt.Errorf("LZ4 compression error: %v", err)
-        }
-        // written is actually encryptedBuffer[offset + 4:], but we need to move the write pointer
-        // and this seems to be the only way
-        encryptedBuffer.Write(written)
-    } else {
-        return fmt.Errorf("Invalid compression level: %d", chunk.config.CompressionLevel)
-    }
+	if chunk.config.CompressionLevel >= -1 && chunk.config.CompressionLevel <= 9 {
+		deflater, _ := zlib.NewWriterLevel(encryptedBuffer, chunk.config.CompressionLevel)
+		deflater.Write(chunk.buffer.Bytes())
+		deflater.Close()
+	} else if chunk.config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
+		encryptedBuffer.Write([]byte("LZ4 "))
+		// Make sure we have enough space in encryptedBuffer
+		availableLength := encryptedBuffer.Cap() - len(encryptedBuffer.Bytes())
+		maximumLength := lz4.CompressBound(len(chunk.buffer.Bytes()))
+		if availableLength < maximumLength {
+			encryptedBuffer.Grow(maximumLength - availableLength)
+		}
+		written, err := lz4.Encode(encryptedBuffer.Bytes()[offset+4:], chunk.buffer.Bytes())
+		if err != nil {
+			return fmt.Errorf("LZ4 compression error: %v", err)
+		}
+		// written is actually encryptedBuffer[offset + 4:], but we need to move the write pointer
+		// and this seems to be the only way
+		encryptedBuffer.Write(written)
+	} else {
+		return fmt.Errorf("Invalid compression level: %d", chunk.config.CompressionLevel)
+	}
 
-    if len(encryptionKey) == 0 {
-        chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
-        return nil
-    }
+	if len(encryptionKey) == 0 {
+		chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
+		return nil
+	}
 
-    // PKCS7 is used.  Compressed chunk sizes leaks information about the original chunks so we want the padding sizes
-    // to be the maximum allowed by PKCS7
-    dataLength := encryptedBuffer.Len() - offset
-    paddingLength := dataLength % 256
-    if paddingLength == 0 {
-        paddingLength = 256
-    }
+	// PKCS7 is used.  Compressed chunk sizes leaks information about the original chunks so we want the padding sizes
+	// to be the maximum allowed by PKCS7
+	dataLength := encryptedBuffer.Len() - offset
+	paddingLength := 256 - dataLength%256
 
-    encryptedBuffer.Write(bytes.Repeat([]byte{byte(paddingLength)}, paddingLength))
-    encryptedBuffer.Write(bytes.Repeat([]byte{0}, gcm.Overhead()))
+	encryptedBuffer.Write(bytes.Repeat([]byte{byte(paddingLength)}, paddingLength))
+	encryptedBuffer.Write(bytes.Repeat([]byte{0}, gcm.Overhead()))
 
-    // The encrypted data will be appended to the duplicacy header and the once.
-    encryptedBytes := gcm.Seal(encryptedBuffer.Bytes()[:offset], nonce,
-                               encryptedBuffer.Bytes()[offset: offset + dataLength + paddingLength], nil)
+	// The encrypted data will be appended to the duplicacy header and the once.
+	encryptedBytes := gcm.Seal(encryptedBuffer.Bytes()[:offset], nonce,
+		encryptedBuffer.Bytes()[offset:offset+dataLength+paddingLength], nil)
 
-    encryptedBuffer.Truncate(len(encryptedBytes))
+	encryptedBuffer.Truncate(len(encryptedBytes))
 
-    chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
+	chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
 
-    return nil
+	return nil
 
 }
 
+// This is to ensure compatibility with Vertical Backup, which still uses HMAC-SHA256 (instead of HMAC-BLAKE2) to
+// derive the key used to encrypt/decrypt files and chunks.
+
+var DecryptWithHMACSHA256 = false
+
+func init() {
+	if value, found := os.LookupEnv("DUPLICACY_DECRYPT_WITH_HMACSHA256"); found && value != "0" {
+		DecryptWithHMACSHA256 = true
+	}
+}
+
 // Decrypt decrypts the encrypted data stored in the chunk buffer.  If derivationKey is not nil, the actual
 // encryption key will be HMAC-SHA256(encryptionKey, derivationKey).
 func (chunk *Chunk) Decrypt(encryptionKey []byte, derivationKey string) (err error) {
 
-    var offset int
+	var offset int
 
-    encryptedBuffer := AllocateChunkBuffer()
-    encryptedBuffer.Reset()
-    defer func() {
-        ReleaseChunkBuffer(encryptedBuffer)
-    } ()
+	encryptedBuffer := AllocateChunkBuffer()
+	encryptedBuffer.Reset()
+	defer func() {
+		ReleaseChunkBuffer(encryptedBuffer)
+	}()
 
-    chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
+	chunk.buffer, encryptedBuffer = encryptedBuffer, chunk.buffer
 
-    if len(encryptionKey) > 0 {
+	if len(encryptionKey) > 0 {
 
-        key := encryptionKey
+		key := encryptionKey
 
-        if len(derivationKey) > 0 {
-            hasher := chunk.config.NewKeyedHasher([]byte(derivationKey))
-            hasher.Write(encryptionKey)
-            key = hasher.Sum(nil)
-        }
+		if len(derivationKey) > 0 {
+			var hasher hash.Hash
+			if DecryptWithHMACSHA256 {
+				hasher = hmac.New(sha256.New, []byte(derivationKey))
+			} else {
+				hasher = chunk.config.NewKeyedHasher([]byte(derivationKey))
+			}
 
-        aesBlock, err := aes.NewCipher(key)
-        if err != nil {
-            return err
-        }
+			hasher.Write(encryptionKey)
+			key = hasher.Sum(nil)
+		}
 
-        gcm, err := cipher.NewGCM(aesBlock)
-        if err != nil {
-            return err
-        }
+		aesBlock, err := aes.NewCipher(key)
+		if err != nil {
+			return err
+		}
 
-        headerLength := len(ENCRYPTION_HEADER)
-        offset = headerLength + gcm.NonceSize()
+		gcm, err := cipher.NewGCM(aesBlock)
+		if err != nil {
+			return err
+		}
 
-        if len(encryptedBuffer.Bytes()) < offset {
-            return fmt.Errorf("No enough encrypted data (%d bytes) provided", len(encryptedBuffer.Bytes()))
-        }
+		headerLength := len(ENCRYPTION_HEADER)
+		offset = headerLength + gcm.NonceSize()
 
-        if string(encryptedBuffer.Bytes()[:headerLength - 1]) != ENCRYPTION_HEADER[:headerLength - 1] {
-            return fmt.Errorf("The storage doesn't seem to be encrypted")
-        }
+		if len(encryptedBuffer.Bytes()) < offset {
+			return fmt.Errorf("No enough encrypted data (%d bytes) provided", len(encryptedBuffer.Bytes()))
+		}
 
-        if encryptedBuffer.Bytes()[headerLength - 1] != 0 {
-            return fmt.Errorf("Unsupported encryption version %d", encryptedBuffer.Bytes()[headerLength - 1])
-        }
+		if string(encryptedBuffer.Bytes()[:headerLength-1]) != ENCRYPTION_HEADER[:headerLength-1] {
+			return fmt.Errorf("The storage doesn't seem to be encrypted")
+		}
 
-        nonce := encryptedBuffer.Bytes()[headerLength: offset]
+		if encryptedBuffer.Bytes()[headerLength-1] != 0 {
+			return fmt.Errorf("Unsupported encryption version %d", encryptedBuffer.Bytes()[headerLength-1])
+		}
 
-        decryptedBytes, err := gcm.Open(encryptedBuffer.Bytes()[:offset], nonce,
-                                        encryptedBuffer.Bytes()[offset:], nil)
+		nonce := encryptedBuffer.Bytes()[headerLength:offset]
 
-        if err != nil {
-            return err
-        }
+		decryptedBytes, err := gcm.Open(encryptedBuffer.Bytes()[:offset], nonce,
+			encryptedBuffer.Bytes()[offset:], nil)
 
-        paddingLength := int(decryptedBytes[len(decryptedBytes) - 1])
-        if paddingLength == 0 {
-            paddingLength = 256
-        }
-        if len(decryptedBytes) <= paddingLength {
-            return fmt.Errorf("Incorrect padding length %d out of %d bytes", paddingLength, len(decryptedBytes))
-        }
+		if err != nil {
+			return err
+		}
 
-        for i := 0; i < paddingLength; i++ {
-            padding := decryptedBytes[len(decryptedBytes) - 1 - i]
-            if padding != byte(paddingLength) {
-                return fmt.Errorf("Incorrect padding of length %d: %x", paddingLength,
-                                  decryptedBytes[len(decryptedBytes) - paddingLength:])
-            }
-        }
+		paddingLength := int(decryptedBytes[len(decryptedBytes)-1])
+		if paddingLength == 0 {
+			paddingLength = 256
+		}
+		if len(decryptedBytes) <= paddingLength {
+			return fmt.Errorf("Incorrect padding length %d out of %d bytes", paddingLength, len(decryptedBytes))
+		}
 
-        encryptedBuffer.Truncate(len(decryptedBytes) - paddingLength)
-    }
+		for i := 0; i < paddingLength; i++ {
+			padding := decryptedBytes[len(decryptedBytes)-1-i]
+			if padding != byte(paddingLength) {
+				return fmt.Errorf("Incorrect padding of length %d: %x", paddingLength,
+					decryptedBytes[len(decryptedBytes)-paddingLength:])
+			}
+		}
 
-    encryptedBuffer.Read(encryptedBuffer.Bytes()[:offset])
+		encryptedBuffer.Truncate(len(decryptedBytes) - paddingLength)
+	}
 
-    compressed := encryptedBuffer.Bytes()
-    if len(compressed) > 4 && string(compressed[:4]) == "LZ4 " {
-        chunk.buffer.Reset()
-        decompressed, err := lz4.Decode(chunk.buffer.Bytes(), encryptedBuffer.Bytes()[4:])
-        if err != nil {
-            return err
-        }
+	encryptedBuffer.Read(encryptedBuffer.Bytes()[:offset])
 
-        chunk.buffer.Write(decompressed)
-        chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
-        chunk.hasher.Write(decompressed)
-        chunk.hash = nil
-        return nil
-    }
-    inflater, err := zlib.NewReader(encryptedBuffer)
-    if err != nil {
-        return err
-    }
+	compressed := encryptedBuffer.Bytes()
+	if len(compressed) > 4 && string(compressed[:4]) == "LZ4 " {
+		chunk.buffer.Reset()
+		decompressed, err := lz4.Decode(chunk.buffer.Bytes(), encryptedBuffer.Bytes()[4:])
+		if err != nil {
+			return err
+		}
 
-    defer inflater.Close()
+		chunk.buffer.Write(decompressed)
+		chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
+		chunk.hasher.Write(decompressed)
+		chunk.hash = nil
+		return nil
+	}
+	inflater, err := zlib.NewReader(encryptedBuffer)
+	if err != nil {
+		return err
+	}
 
-    chunk.buffer.Reset()
-    chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
-    chunk.hash = nil
+	defer inflater.Close()
 
-    if _, err = io.Copy(chunk, inflater); err != nil {
-        return err
-    }
+	chunk.buffer.Reset()
+	chunk.hasher = chunk.config.NewKeyedHasher(chunk.config.HashKey)
+	chunk.hash = nil
 
-    return nil
+	if _, err = io.Copy(chunk, inflater); err != nil {
+		return err
+	}
+
+	return nil
 
 }
-

src/duplicacy_chunk_test.go

@@ -5,69 +5,76 @@
 package duplicacy
 
 import (
-    "testing"
-    "bytes"
-    crypto_rand "crypto/rand"
-    "math/rand"
+	"bytes"
+	crypto_rand "crypto/rand"
+	"math/rand"
+	"testing"
 )
 
 func TestChunk(t *testing.T) {
 
-    key := []byte("duplicacydefault")
+	key := []byte("duplicacydefault")
 
-    config := CreateConfig()
-    config.HashKey = key
-    config.IDKey = key
-    config.MinimumChunkSize = 100
-    config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
-    maxSize := 1000000
+	config := CreateConfig()
+	config.HashKey = key
+	config.IDKey = key
+	config.MinimumChunkSize = 100
+	config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
+	maxSize := 1000000
 
-    for i := 0; i < 500; i++ {
+	remainderLength := -1
 
-        size := rand.Int() % maxSize
+	for i := 0; i < 500; i++ {
 
-        plainData := make([]byte, size)
-        crypto_rand.Read(plainData)
-        chunk := CreateChunk(config, true)
-        chunk.Reset(true)
-        chunk.Write(plainData)
+		size := rand.Int() % maxSize
 
-        hash := chunk.GetHash()
-        id := chunk.GetID()
+		plainData := make([]byte, size)
+		crypto_rand.Read(plainData)
+		chunk := CreateChunk(config, true)
+		chunk.Reset(true)
+		chunk.Write(plainData)
 
-        err := chunk.Encrypt(key, "")
-        if err != nil {
-            t.Errorf("Failed to encrypt the data: %v", err)
-            continue
-        }
+		hash := chunk.GetHash()
+		id := chunk.GetID()
 
-        encryptedData := make([]byte, chunk.GetLength())
-        copy(encryptedData, chunk.GetBytes())
+		err := chunk.Encrypt(key, "")
+		if err != nil {
+			t.Errorf("Failed to encrypt the data: %v", err)
+			continue
+		}
 
-        chunk.Reset(false)
-        chunk.Write(encryptedData)
-        err = chunk.Decrypt(key, "")
-        if err != nil {
-            t.Errorf("Failed to decrypt the data: %v", err)
-            continue
-        }
+		encryptedData := make([]byte, chunk.GetLength())
+		copy(encryptedData, chunk.GetBytes())
 
-        decryptedData := chunk.GetBytes()
+		if remainderLength == -1 {
+			remainderLength = len(encryptedData) % 256
+		} else if len(encryptedData)%256 != remainderLength {
+			t.Errorf("Incorrect padding size")
+		}
 
-        if hash != chunk.GetHash() {
-            t.Errorf("Original hash: %x, decrypted hash: %x", hash, chunk.GetHash())
-        }
+		chunk.Reset(false)
+		chunk.Write(encryptedData)
+		err = chunk.Decrypt(key, "")
+		if err != nil {
+			t.Errorf("Failed to decrypt the data: %v", err)
+			continue
+		}
 
-        if id != chunk.GetID() {
-            t.Errorf("Original id: %s, decrypted hash: %s", id, chunk.GetID())
-        }
+		decryptedData := chunk.GetBytes()
 
-        if bytes.Compare(plainData, decryptedData) != 0 {
-            t.Logf("orginal length: %d, decrypted length: %d", len(plainData), len(decryptedData))
-            t.Errorf("Original data:\n%x\nDecrypted data:\n%x\n", plainData, decryptedData)
-        }
+		if hash != chunk.GetHash() {
+			t.Errorf("Original hash: %x, decrypted hash: %x", hash, chunk.GetHash())
+		}
 
+		if id != chunk.GetID() {
+			t.Errorf("Original id: %s, decrypted hash: %s", id, chunk.GetID())
+		}
 
-    }
+		if bytes.Compare(plainData, decryptedData) != 0 {
+			t.Logf("Original length: %d, decrypted length: %d", len(plainData), len(decryptedData))
+			t.Errorf("Original data:\n%x\nDecrypted data:\n%x\n", plainData, decryptedData)
+		}
+
+	}
 
 }

src/duplicacy_chunkdownloader.go

@@ -5,24 +5,25 @@
 package duplicacy
 
 import (
-    "sync/atomic"
-    "time"
+	"io"
+	"sync/atomic"
+	"time"
 )
 
 // ChunkDownloadTask encapsulates information need to download a chunk.
 type ChunkDownloadTask struct {
-    chunk *Chunk                // The chunk that will be downloaded; initially nil
-    chunkIndex int              // The index of this chunk in the chunk list
-    chunkHash string            // The chunk hash
-    chunkLength int             // The length of the chunk; may be zero
-    needed bool                 // Whether this chunk can be skipped if a local copy exists
-    isDownloading bool          // 'true' means the chunk has been downloaded or is being downloaded
+	chunk         *Chunk // The chunk that will be downloaded; initially nil
+	chunkIndex    int    // The index of this chunk in the chunk list
+	chunkHash     string // The chunk hash
+	chunkLength   int    // The length of the chunk; may be zero
+	needed        bool   // Whether this chunk can be skipped if a local copy exists
+	isDownloading bool   // 'true' means the chunk has been downloaded or is being downloaded
 }
 
 // ChunkDownloadCompletion represents the nofication when a chunk has been downloaded.
 type ChunkDownloadCompletion struct {
-    chunkIndex int              // The index of this chunk in the chunk list
-    chunk *Chunk                // The chunk that has been downloaded
+	chunkIndex int    // The index of this chunk in the chunk list
+	chunk      *Chunk // The chunk that has been downloaded
 }
 
 // ChunkDownloader is capable of performing multi-threaded downloading.  Chunks to be downloaded are first organized
@@ -30,344 +31,407 @@ type ChunkDownloadCompletion struct {
 // corresponding ChunkDownloadTask is sent to the dowloading goroutine.  Once a chunk is downloaded, it will be
 // inserted in the completed task list.
 type ChunkDownloader struct {
-    config *Config                                     // Associated config
-    storage Storage                                    // Download from this storage
-    snapshotCache *FileStorage                         // Used as cache if not nil; usually for downloading snapshot chunks
-    showStatistics bool                                // Show a stats log for each chunk if true
-    threads int                                        // Number of threads
+	config         *Config      // Associated config
+	storage        Storage      // Download from this storage
+	snapshotCache  *FileStorage // Used as cache if not nil; usually for downloading snapshot chunks
+	showStatistics bool         // Show a stats log for each chunk if true
+	threads        int          // Number of threads
 
-    taskList [] ChunkDownloadTask                      // The list of chunks to be downloaded
-    completedTasks map[int]bool                        // Store downloaded chunks
-    lastChunkIndex int                                 // a monotonically increasing number indicating the last chunk to be downloaded
+	taskList       []ChunkDownloadTask // The list of chunks to be downloaded
+	completedTasks map[int]bool        // Store downloaded chunks
+	lastChunkIndex int                 // a monotonically increasing number indicating the last chunk to be downloaded
 
-    taskQueue chan ChunkDownloadTask                   // Downloading goroutines are waiting on this channel for input
-    stopChannel chan bool                              // Used to stop the dowloading goroutines
-    completionChannel chan ChunkDownloadCompletion     // A downloading goroutine sends back the chunk via this channel after downloading
+	taskQueue         chan ChunkDownloadTask       // Downloading goroutines are waiting on this channel for input
+	stopChannel       chan bool                    // Used to stop the dowloading goroutines
+	completionChannel chan ChunkDownloadCompletion // A downloading goroutine sends back the chunk via this channel after downloading
 
-    startTime int64                                    // The time it starts downloading
-    totalChunkSize int64                               // Total chunk size
-    downloadedChunkSize int64                          // Downloaded chunk size
-    numberOfDownloadedChunks int                       // The number of chunks that have been downloaded 
-    numberOfDownloadingChunks int                      // The number of chunks still being downloaded
-    numberOfActiveChunks int                           // The number of chunks that is being downloaded or has been downloaded but not reclaimed
+	startTime                 int64 // The time it starts downloading
+	totalChunkSize            int64 // Total chunk size
+	downloadedChunkSize       int64 // Downloaded chunk size
+	numberOfDownloadedChunks  int   // The number of chunks that have been downloaded
+	numberOfDownloadingChunks int   // The number of chunks still being downloaded
+	numberOfActiveChunks      int   // The number of chunks that is being downloaded or has been downloaded but not reclaimed
 }
 
 func CreateChunkDownloader(config *Config, storage Storage, snapshotCache *FileStorage, showStatistics bool, threads int) *ChunkDownloader {
-    downloader := &ChunkDownloader {
-        config: config,
-        storage: storage,
-        snapshotCache: snapshotCache,
-        showStatistics: showStatistics,
-        threads: threads,
+	downloader := &ChunkDownloader{
+		config:         config,
+		storage:        storage,
+		snapshotCache:  snapshotCache,
+		showStatistics: showStatistics,
+		threads:        threads,
 
-        taskList: nil,
-        completedTasks: make(map[int]bool),
-        lastChunkIndex: 0,
+		taskList:       nil,
+		completedTasks: make(map[int]bool),
+		lastChunkIndex: 0,
 
-        taskQueue: make(chan ChunkDownloadTask, threads),
-        stopChannel: make(chan bool),
-        completionChannel: make(chan ChunkDownloadCompletion),
+		taskQueue:         make(chan ChunkDownloadTask, threads),
+		stopChannel:       make(chan bool),
+		completionChannel: make(chan ChunkDownloadCompletion),
 
-        startTime: time.Now().Unix(),
-    }
+		startTime: time.Now().Unix(),
+	}
 
-    // Start the downloading goroutines
-    for i := 0; i < downloader.threads; i++ {
-        go func(threadIndex int) {
-            defer CatchLogException()
-            for {
-                select {
-                case task := <- downloader.taskQueue:
-                    downloader.Download(threadIndex, task)
-                case <- downloader.stopChannel:
-                    return
-                }
-            }
-        } (i)
-    }
+	// Start the downloading goroutines
+	for i := 0; i < downloader.threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case task := <-downloader.taskQueue:
+					downloader.Download(threadIndex, task)
+				case <-downloader.stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
 
-    return downloader
+	return downloader
 }
 
 // AddFiles adds chunks needed by the specified files to the download list.
-func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files [] *Entry) {
+func (downloader *ChunkDownloader) AddFiles(snapshot *Snapshot, files []*Entry) {
 
-    downloader.taskList = nil
-    lastChunkIndex := -1
-    maximumChunks := 0
-    downloader.totalChunkSize = 0
-    for _, file := range files {
-        if file.Size == 0 {
-            continue
-        }
-        for i := file.StartChunk; i <= file.EndChunk; i++ {
-            if lastChunkIndex != i {
-                task := ChunkDownloadTask {
-                    chunkIndex: len(downloader.taskList),
-                    chunkHash: snapshot.ChunkHashes[i],
-                    chunkLength: snapshot.ChunkLengths[i],
-                    needed: false,
-                }
-                downloader.taskList = append(downloader.taskList, task)
-                downloader.totalChunkSize += int64(snapshot.ChunkLengths[i])
-            } else {
-                downloader.taskList[len(downloader.taskList) - 1].needed = true
-            }
-            lastChunkIndex = i
-        }
-        file.StartChunk = len(downloader.taskList) - (file.EndChunk - file.StartChunk) - 1
-        file.EndChunk = len(downloader.taskList) - 1
-        if file.EndChunk - file.StartChunk > maximumChunks {
-            maximumChunks = file.EndChunk - file.StartChunk
-        }
-    }
+	downloader.taskList = nil
+	lastChunkIndex := -1
+	maximumChunks := 0
+	downloader.totalChunkSize = 0
+	for _, file := range files {
+		if file.Size == 0 {
+			continue
+		}
+		for i := file.StartChunk; i <= file.EndChunk; i++ {
+			if lastChunkIndex != i {
+				task := ChunkDownloadTask{
+					chunkIndex:  len(downloader.taskList),
+					chunkHash:   snapshot.ChunkHashes[i],
+					chunkLength: snapshot.ChunkLengths[i],
+					needed:      false,
+				}
+				downloader.taskList = append(downloader.taskList, task)
+				downloader.totalChunkSize += int64(snapshot.ChunkLengths[i])
+			} else {
+				downloader.taskList[len(downloader.taskList)-1].needed = true
+			}
+			lastChunkIndex = i
+		}
+		file.StartChunk = len(downloader.taskList) - (file.EndChunk - file.StartChunk) - 1
+		file.EndChunk = len(downloader.taskList) - 1
+		if file.EndChunk-file.StartChunk > maximumChunks {
+			maximumChunks = file.EndChunk - file.StartChunk
+		}
+	}
 }
 
 // AddChunk adds a single chunk the download list.
 func (downloader *ChunkDownloader) AddChunk(chunkHash string) int {
-    task := ChunkDownloadTask {
-        chunkIndex: len(downloader.taskList),
-        chunkHash: chunkHash,
-        chunkLength: 0,
-        needed: true,
-        isDownloading: false,
-    }
-    downloader.taskList = append(downloader.taskList, task)
-    if downloader.numberOfActiveChunks < downloader.threads {
-        downloader.taskQueue <- task
-        downloader.numberOfDownloadingChunks++
-        downloader.numberOfActiveChunks++
-        downloader.taskList[len(downloader.taskList) - 1].isDownloading = true
-    }
-    return len(downloader.taskList) - 1
+	task := ChunkDownloadTask{
+		chunkIndex:    len(downloader.taskList),
+		chunkHash:     chunkHash,
+		chunkLength:   0,
+		needed:        true,
+		isDownloading: false,
+	}
+	downloader.taskList = append(downloader.taskList, task)
+	if downloader.numberOfActiveChunks < downloader.threads {
+		downloader.taskQueue <- task
+		downloader.numberOfDownloadingChunks++
+		downloader.numberOfActiveChunks++
+		downloader.taskList[len(downloader.taskList)-1].isDownloading = true
+	}
+	return len(downloader.taskList) - 1
 }
 
 // Prefetch adds up to 'threads' chunks needed by a file to the download list
 func (downloader *ChunkDownloader) Prefetch(file *Entry) {
 
-    // Any chunks before the first chunk of this filea are not needed any more, so they can be reclaimed.
-    downloader.Reclaim(file.StartChunk)
+	// Any chunks before the first chunk of this filea are not needed any more, so they can be reclaimed.
+	downloader.Reclaim(file.StartChunk)
 
-    for i := file.StartChunk; i <= file.EndChunk; i++ {
-        task := &downloader.taskList[i]
-        if task.needed {
-            if !task.isDownloading {
-                if downloader.numberOfActiveChunks >= downloader.threads {
-                    return
-                }
+	for i := file.StartChunk; i <= file.EndChunk; i++ {
+		task := &downloader.taskList[i]
+		if task.needed {
+			if !task.isDownloading {
+				if downloader.numberOfActiveChunks >= downloader.threads {
+					return
+				}
 
-                LOG_DEBUG("DOWNLOAD_PREFETCH", "Prefetching %s chunk %s", file.Path,
-                          downloader.config.GetChunkIDFromHash(task.chunkHash))
-                downloader.taskQueue <- *task
-                task.isDownloading = true
-                downloader.numberOfDownloadingChunks++
-                downloader.numberOfActiveChunks++
-            }
-        } else{
-            LOG_DEBUG("DOWNLOAD_PREFETCH", "%s chunk %s is not needed", file.Path,
-                      downloader.config.GetChunkIDFromHash(task.chunkHash))
-        }
-    }
+				LOG_DEBUG("DOWNLOAD_PREFETCH", "Prefetching %s chunk %s", file.Path,
+					downloader.config.GetChunkIDFromHash(task.chunkHash))
+				downloader.taskQueue <- *task
+				task.isDownloading = true
+				downloader.numberOfDownloadingChunks++
+				downloader.numberOfActiveChunks++
+			}
+		} else {
+			LOG_DEBUG("DOWNLOAD_PREFETCH", "%s chunk %s is not needed", file.Path,
+				downloader.config.GetChunkIDFromHash(task.chunkHash))
+		}
+	}
 }
 
 // Reclaim releases the downloaded chunk to the chunk pool
 func (downloader *ChunkDownloader) Reclaim(chunkIndex int) {
 
-    if downloader.lastChunkIndex == chunkIndex {
-        return
-    }
+	if downloader.lastChunkIndex >= chunkIndex {
+		return
+	}
 
-    for i, _ := range downloader.completedTasks {
-        if i < chunkIndex && downloader.taskList[i].chunk != nil {
-            downloader.config.PutChunk(downloader.taskList[i].chunk)
-            downloader.taskList[i].chunk = nil
-            delete(downloader.completedTasks, i)
-            downloader.numberOfActiveChunks--
-        }
-    }
+	for i := range downloader.completedTasks {
+		if i < chunkIndex && downloader.taskList[i].chunk != nil {
+			downloader.config.PutChunk(downloader.taskList[i].chunk)
+			downloader.taskList[i].chunk = nil
+			delete(downloader.completedTasks, i)
+			downloader.numberOfActiveChunks--
+		}
+	}
 
-    downloader.lastChunkIndex = chunkIndex
+	for i := downloader.lastChunkIndex; i < chunkIndex; i++ {
+		// These chunks are never downloaded if 'isDownloading' is false; note that 'isDownloading' isn't reset to
+		// false after a chunk has been downloaded
+		if !downloader.taskList[i].isDownloading {
+			atomic.AddInt64(&downloader.totalChunkSize, -int64(downloader.taskList[i].chunkLength))
+		}
+	}
+	downloader.lastChunkIndex = chunkIndex
 }
 
 // WaitForChunk waits until the specified chunk is ready
 func (downloader *ChunkDownloader) WaitForChunk(chunkIndex int) (chunk *Chunk) {
 
-    // Reclain any chunk not needed
-    downloader.Reclaim(chunkIndex)
+	// Reclaim any chunk not needed
+	downloader.Reclaim(chunkIndex)
 
-    // If we haven't started download the specified chunk, download it now
-    if !downloader.taskList[chunkIndex].isDownloading {
-        LOG_DEBUG("DOWNLOAD_FETCH", "Fetching chunk %s",
-                  downloader.config.GetChunkIDFromHash(downloader.taskList[chunkIndex].chunkHash))
-        downloader.taskQueue <- downloader.taskList[chunkIndex]
-        downloader.taskList[chunkIndex].isDownloading = true
-        downloader.numberOfDownloadingChunks++
-        downloader.numberOfActiveChunks++
-    }
+	// If we haven't started download the specified chunk, download it now
+	if !downloader.taskList[chunkIndex].isDownloading {
+		LOG_DEBUG("DOWNLOAD_FETCH", "Fetching chunk %s",
+			downloader.config.GetChunkIDFromHash(downloader.taskList[chunkIndex].chunkHash))
+		downloader.taskQueue <- downloader.taskList[chunkIndex]
+		downloader.taskList[chunkIndex].isDownloading = true
+		downloader.numberOfDownloadingChunks++
+		downloader.numberOfActiveChunks++
+	}
 
-    // We also need to look ahead and prefetch other chunks as many as permitted by the number of threads
-    for i := chunkIndex + 1; i < len(downloader.taskList); i++ {
-        if downloader.numberOfActiveChunks >= downloader.threads {
-            break
-        }
-        task := &downloader.taskList[i]
-        if !task.needed {
-            break
-        }
+	// We also need to look ahead and prefetch other chunks as many as permitted by the number of threads
+	for i := chunkIndex + 1; i < len(downloader.taskList); i++ {
+		if downloader.numberOfActiveChunks >= downloader.threads {
+			break
+		}
+		task := &downloader.taskList[i]
+		if !task.needed {
+			break
+		}
 
-        if !task.isDownloading {
-            LOG_DEBUG("DOWNLOAD_PREFETCH", "Prefetching chunk %s", downloader.config.GetChunkIDFromHash(task.chunkHash))
-            downloader.taskQueue <- *task
-            task.isDownloading = true
-            downloader.numberOfDownloadingChunks++
-            downloader.numberOfActiveChunks++
-        }
-    }
+		if !task.isDownloading {
+			LOG_DEBUG("DOWNLOAD_PREFETCH", "Prefetching chunk %s", downloader.config.GetChunkIDFromHash(task.chunkHash))
+			downloader.taskQueue <- *task
+			task.isDownloading = true
+			downloader.numberOfDownloadingChunks++
+			downloader.numberOfActiveChunks++
+		}
+	}
 
-    // Now wait until the chunk to be downloaded appears in the completed tasks
-    for _, found := downloader.completedTasks[chunkIndex]; !found; _, found = downloader.completedTasks[chunkIndex] {
-        completion := <- downloader.completionChannel
-        downloader.completedTasks[completion.chunkIndex] = true
-        downloader.taskList[completion.chunkIndex].chunk = completion.chunk
-        downloader.numberOfDownloadedChunks++
-        downloader.numberOfDownloadingChunks--
-    }
-    return downloader.taskList[chunkIndex].chunk
+	// Now wait until the chunk to be downloaded appears in the completed tasks
+	for _, found := downloader.completedTasks[chunkIndex]; !found; _, found = downloader.completedTasks[chunkIndex] {
+		completion := <-downloader.completionChannel
+		downloader.completedTasks[completion.chunkIndex] = true
+		downloader.taskList[completion.chunkIndex].chunk = completion.chunk
+		downloader.numberOfDownloadedChunks++
+		downloader.numberOfDownloadingChunks--
+	}
+	return downloader.taskList[chunkIndex].chunk
 }
 
 // Stop terminates all downloading goroutines
 func (downloader *ChunkDownloader) Stop() {
-    for downloader.numberOfDownloadingChunks > 0 {
-        completion := <- downloader.completionChannel
-        downloader.completedTasks[completion.chunkIndex] = true
-        downloader.taskList[completion.chunkIndex].chunk = completion.chunk
-        downloader.numberOfDownloadedChunks++
-        downloader.numberOfDownloadingChunks--
-    }
+	for downloader.numberOfDownloadingChunks > 0 {
+		completion := <-downloader.completionChannel
+		downloader.completedTasks[completion.chunkIndex] = true
+		downloader.taskList[completion.chunkIndex].chunk = completion.chunk
+		downloader.numberOfDownloadedChunks++
+		downloader.numberOfDownloadingChunks--
+	}
 
-    for i, _ := range downloader.completedTasks {
-        downloader.config.PutChunk(downloader.taskList[i].chunk)
-        downloader.taskList[i].chunk = nil
-        downloader.numberOfActiveChunks--
-    }
+	for i := range downloader.completedTasks {
+		downloader.config.PutChunk(downloader.taskList[i].chunk)
+		downloader.taskList[i].chunk = nil
+		downloader.numberOfActiveChunks--
+	}
 
-    for i := 0; i < downloader.threads; i++ {
-        downloader.stopChannel <- true
-    }
+	for i := 0; i < downloader.threads; i++ {
+		downloader.stopChannel <- true
+	}
 }
 
 // Download downloads a chunk from the storage.
 func (downloader *ChunkDownloader) Download(threadIndex int, task ChunkDownloadTask) bool {
 
-    cachedPath := ""
-    chunk := downloader.config.GetChunk()
-    chunkID := downloader.config.GetChunkIDFromHash(task.chunkHash)
+	cachedPath := ""
+	chunk := downloader.config.GetChunk()
+	chunkID := downloader.config.GetChunkIDFromHash(task.chunkHash)
 
-    if downloader.snapshotCache != nil && downloader.storage.IsCacheNeeded() {
+	if downloader.snapshotCache != nil && downloader.storage.IsCacheNeeded() {
 
-        var exist bool
-        var err error
+		var exist bool
+		var err error
 
-        // Reset the chunk with a hasher -- we're reading from the cache where chunk are not encrypted or compressed
-        chunk.Reset(true)
+		// Reset the chunk with a hasher -- we're reading from the cache where chunk are not encrypted or compressed
+		chunk.Reset(true)
 
-        cachedPath, exist, _, err = downloader.snapshotCache.FindChunk(threadIndex, chunkID, false)
-        if err != nil {
-            LOG_WARN("DOWNLOAD_CACHE", "Failed to find the cache path for the chunk %s: %v", chunkID, err)
-        } else if exist {
-            err = downloader.snapshotCache.DownloadFile(0, cachedPath, chunk)
-            if err != nil {
-                LOG_WARN("DOWNLOAD_CACHE", "Failed to load the chunk %s from the snapshot cache: %v", chunkID, err)
-            } else {
-                actualChunkID := chunk.GetID()
-                if actualChunkID != chunkID {
-                    LOG_WARN("DOWNLOAD_CACHE_CORRUPTED",
-                             "The chunk %s load from the snapshot cache has a hash id of %s", chunkID, actualChunkID)
-                } else {
-                    LOG_DEBUG("CHUNK_CACHE", "Chunk %s has been loaded from the snapshot cache", chunkID)
+		cachedPath, exist, _, err = downloader.snapshotCache.FindChunk(threadIndex, chunkID, false)
+		if err != nil {
+			LOG_WARN("DOWNLOAD_CACHE", "Failed to find the cache path for the chunk %s: %v", chunkID, err)
+		} else if exist {
+			err = downloader.snapshotCache.DownloadFile(0, cachedPath, chunk)
+			if err != nil {
+				LOG_WARN("DOWNLOAD_CACHE", "Failed to load the chunk %s from the snapshot cache: %v", chunkID, err)
+			} else {
+				actualChunkID := chunk.GetID()
+				if actualChunkID != chunkID {
+					LOG_WARN("DOWNLOAD_CACHE_CORRUPTED",
+						"The chunk %s load from the snapshot cache has a hash id of %s", chunkID, actualChunkID)
+				} else {
+					LOG_DEBUG("CHUNK_CACHE", "Chunk %s has been loaded from the snapshot cache", chunkID)
 
-                    downloader.completionChannel <- ChunkDownloadCompletion{ chunk: chunk, chunkIndex:task.chunkIndex }
-                    return false
-                }
-            }
-        }
-    }
+					downloader.completionChannel <- ChunkDownloadCompletion{chunk: chunk, chunkIndex: task.chunkIndex}
+					return false
+				}
+			}
+		}
+	}
 
-    // Reset the chunk without a hasher -- the downloaded content will be encrypted and/or compressed and the hasher
-    // will be set up before the encryption
-    chunk.Reset(false)
+	// Reset the chunk without a hasher -- the downloaded content will be encrypted and/or compressed and the hasher
+	// will be set up before the encryption
+	chunk.Reset(false)
 
-    // Find the chunk by ID first.
-    chunkPath, exist, _, err := downloader.storage.FindChunk(threadIndex, chunkID, false)
-    if err != nil {
-        LOG_ERROR("DOWNLOAD_CHUNK", "Failed to find the chunk %s: %v", chunkID, err)
-        return false
-    }
+	const MaxDownloadAttempts = 3
+	for downloadAttempt := 0; ; downloadAttempt++ {
 
-    if !exist {
-        // No chunk is found.  Have to find it in the fossil pool again.
-        chunkPath, exist, _, err = downloader.storage.FindChunk(threadIndex, chunkID, true)
-        if err != nil {
-            LOG_ERROR("DOWNLOAD_CHUNK", "Failed to find the chunk %s: %v", chunkID, err)
-            return false
-        }
+		// Find the chunk by ID first.
+		chunkPath, exist, _, err := downloader.storage.FindChunk(threadIndex, chunkID, false)
+		if err != nil {
+			LOG_ERROR("DOWNLOAD_CHUNK", "Failed to find the chunk %s: %v", chunkID, err)
+			return false
+		}
 
-        if !exist {
-            // A chunk is not found.  This is a serious error and hopefully it will never happen.
-            LOG_FATAL("DOWNLOAD_CHUNK", "Chunk %s can't be found", chunkID)
-            return false
-        }
-        LOG_DEBUG("CHUNK_FOSSIL", "Chunk %s has been marked as a fossil", chunkID)
-    }
+		if !exist {
+			// No chunk is found.  Have to find it in the fossil pool again.
+			fossilPath, exist, _, err := downloader.storage.FindChunk(threadIndex, chunkID, true)
+			if err != nil {
+				LOG_ERROR("DOWNLOAD_CHUNK", "Failed to find the chunk %s: %v", chunkID, err)
+				return false
+			}
 
-    err = downloader.storage.DownloadFile(threadIndex, chunkPath, chunk)
-    if err != nil {
-        LOG_ERROR("UPLOAD_FATAL", "Failed to download the chunk %s: %v", chunkID, err)
-        return false
-    }
+			if !exist {
 
+				retry := false
 
-    err = chunk.Decrypt(downloader.config.ChunkKey, task.chunkHash)
-    if err != nil {
-        LOG_ERROR("UPLOAD_CHUNK", "Failed to decrypt the chunk %s: %v", chunkID, err)
-        return false
-    }
+				// Retry for Hubic or WebDAV as it may return 404 even when the chunk exists
+				if _, ok := downloader.storage.(*HubicStorage); ok {
+					retry = true
+				}
 
-    actualChunkID := chunk.GetID()
-    if actualChunkID != chunkID {
-        LOG_FATAL("UPLOAD_CORRUPTED", "The chunk %s has a hash id of %s", chunkID, actualChunkID)
-        return false
-    }
+				if _, ok := downloader.storage.(*WebDAVStorage); ok {
+					retry = true
+				}
 
-    if len(cachedPath) > 0 {
-        // Save a copy to the local snapshot cache
-        err = downloader.snapshotCache.UploadFile(threadIndex, cachedPath, chunk.GetBytes())
-        if err != nil {
-            LOG_WARN("DOWNLOAD_CACHE", "Failed to add the chunk %s to the snapshot cache: %v", chunkID, err)
-        }
-    }
+				if retry && downloadAttempt < MaxDownloadAttempts {
+					LOG_WARN("DOWNLOAD_RETRY", "Failed to find the chunk %s; retrying", chunkID)
+					continue
+				}
 
-    downloadedChunkSize := atomic.AddInt64(&downloader.downloadedChunkSize, int64(chunk.GetLength()))
+				// A chunk is not found.  This is a serious error and hopefully it will never happen.
+				if err != nil {
+					LOG_FATAL("DOWNLOAD_CHUNK", "Chunk %s can't be found: %v", chunkID, err)
+				} else {
+					LOG_FATAL("DOWNLOAD_CHUNK", "Chunk %s can't be found", chunkID)
+				}
+				return false
+			}
 
-    if (downloader.showStatistics || IsTracing()) && downloader.totalChunkSize > 0 {
+			// We can't download the fossil directly.  We have to turn it back into a regular chunk and try
+			// downloading again.
+			err = downloader.storage.MoveFile(threadIndex, fossilPath, chunkPath)
+			if err != nil {
+				LOG_FATAL("DOWNLOAD_CHUNK", "Failed to resurrect chunk %s: %v", chunkID, err)
+				return false
+			}
 
-        now := time.Now().Unix()
-        if now <= downloader.startTime {
-            now = downloader.startTime + 1
-        }
-        speed := downloadedChunkSize / (now - downloader.startTime)
-        remainingTime := int64(0)
-        if speed > 0 {
-            remainingTime = (downloader.totalChunkSize - downloadedChunkSize) / speed + 1
-        }
-        percentage := float32(downloadedChunkSize * 1000 / downloader.totalChunkSize)
-        LOG_INFO("DOWNLOAD_PROGRESS", "Downloaded chunk %d size %d, %sB/s %s %.1f%%",
-                    task.chunkIndex + 1, chunk.GetLength(),
-                    PrettySize(speed), PrettyTime(remainingTime), percentage / 10)
-    } else {
-        LOG_DEBUG("CHUNK_DOWNLOAD", "Chunk %s has been downloaded", chunkID)
-    }
+			LOG_WARN("DOWNLOAD_RESURRECT", "Fossil %s has been resurrected", chunkID)
+			continue
+		}
 
-    downloader.completionChannel <- ChunkDownloadCompletion{ chunk: chunk, chunkIndex:task.chunkIndex }
-    return true
+		err = downloader.storage.DownloadFile(threadIndex, chunkPath, chunk)
+		if err != nil {
+			_, isHubic := downloader.storage.(*HubicStorage)
+			// Retry on EOF or if it is a Hubic backend as it may return 404 even when the chunk exists
+			if (err == io.ErrUnexpectedEOF || isHubic) && downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "Failed to download the chunk %s: %v; retrying", chunkID, err)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_ERROR("DOWNLOAD_CHUNK", "Failed to download the chunk %s: %v", chunkID, err)
+				return false
+			}
+		}
+
+		err = chunk.Decrypt(downloader.config.ChunkKey, task.chunkHash)
+		if err != nil {
+			if downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "Failed to decrypt the chunk %s: %v; retrying", chunkID, err)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_ERROR("DOWNLOAD_DECRYPT", "Failed to decrypt the chunk %s: %v", chunkID, err)
+				return false
+			}
+		}
+
+		actualChunkID := chunk.GetID()
+		if actualChunkID != chunkID {
+			if downloadAttempt < MaxDownloadAttempts {
+				LOG_WARN("DOWNLOAD_RETRY", "The chunk %s has a hash id of %s; retrying", chunkID, actualChunkID)
+				chunk.Reset(false)
+				continue
+			} else {
+				LOG_FATAL("DOWNLOAD_CORRUPTED", "The chunk %s has a hash id of %s", chunkID, actualChunkID)
+				return false
+			}
+		}
+
+		break
+	}
+
+	if len(cachedPath) > 0 {
+		// Save a copy to the local snapshot cache
+		err := downloader.snapshotCache.UploadFile(threadIndex, cachedPath, chunk.GetBytes())
+		if err != nil {
+			LOG_WARN("DOWNLOAD_CACHE", "Failed to add the chunk %s to the snapshot cache: %v", chunkID, err)
+		}
+	}
+
+	downloadedChunkSize := atomic.AddInt64(&downloader.downloadedChunkSize, int64(chunk.GetLength()))
+
+	if (downloader.showStatistics || IsTracing()) && downloader.totalChunkSize > 0 {
+
+		now := time.Now().Unix()
+		if now <= downloader.startTime {
+			now = downloader.startTime + 1
+		}
+		speed := downloadedChunkSize / (now - downloader.startTime)
+		remainingTime := int64(0)
+		if speed > 0 {
+			remainingTime = (downloader.totalChunkSize-downloadedChunkSize)/speed + 1
+		}
+		percentage := float32(downloadedChunkSize * 1000 / downloader.totalChunkSize)
+		LOG_INFO("DOWNLOAD_PROGRESS", "Downloaded chunk %d size %d, %sB/s %s %.1f%%",
+			task.chunkIndex+1, chunk.GetLength(),
+			PrettySize(speed), PrettyTime(remainingTime), percentage/10)
+	} else {
+		LOG_DEBUG("CHUNK_DOWNLOAD", "Chunk %s has been downloaded", chunkID)
+	}
+
+	downloader.completionChannel <- ChunkDownloadCompletion{chunk: chunk, chunkIndex: task.chunkIndex}
+	return true
 }

src/duplicacy_chunkmaker.go

@@ -5,294 +5,293 @@
 package duplicacy
 
 import (
-    "io"
-    "crypto/sha256"
-    "encoding/hex"
-    "encoding/binary"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"io"
 )
 
 // ChunkMaker breaks data into chunks using buzhash.  To save memory, the chunk maker only use a circular buffer
 // whose size is double the minimum chunk size.
 type ChunkMaker struct {
-    maximumChunkSize int
-    minimumChunkSize int
-    bufferCapacity   int
+	maximumChunkSize int
+	minimumChunkSize int
+	bufferCapacity   int
 
-    hashMask      uint64
-    randomTable  [256]uint64
+	hashMask    uint64
+	randomTable [256]uint64
 
-    buffer []byte
-    bufferSize int
-    bufferStart int
+	buffer      []byte
+	bufferSize  int
+	bufferStart int
 
-    config *Config
+	config *Config
 
-    hashOnly  bool
-    hashOnlyChunk *Chunk
+	hashOnly      bool
+	hashOnlyChunk *Chunk
 }
 
 // CreateChunkMaker creates a chunk maker.  'randomSeed' is used to generate the character-to-integer table needed by
 // buzhash.
 func CreateChunkMaker(config *Config, hashOnly bool) *ChunkMaker {
-    size := 1
-    for size * 2 <= config.AverageChunkSize {
-        size *= 2
-    }
+	size := 1
+	for size*2 <= config.AverageChunkSize {
+		size *= 2
+	}
 
-    if size != config.AverageChunkSize {
-        LOG_FATAL("CHUNK_SIZE", "Invalid average chunk size: %d is not a power of 2", config.AverageChunkSize)
-        return nil
-    }
+	if size != config.AverageChunkSize {
+		LOG_FATAL("CHUNK_SIZE", "Invalid average chunk size: %d is not a power of 2", config.AverageChunkSize)
+		return nil
+	}
 
-    maker := &ChunkMaker {
-        hashMask: uint64(config.AverageChunkSize - 1),
-        maximumChunkSize: config.MaximumChunkSize,
-        minimumChunkSize: config.MinimumChunkSize,
-        bufferCapacity: 2 * config.MinimumChunkSize,
-        config: config,
-        hashOnly: hashOnly,
-    }
+	maker := &ChunkMaker{
+		hashMask:         uint64(config.AverageChunkSize - 1),
+		maximumChunkSize: config.MaximumChunkSize,
+		minimumChunkSize: config.MinimumChunkSize,
+		bufferCapacity:   2 * config.MinimumChunkSize,
+		config:           config,
+		hashOnly:         hashOnly,
+	}
 
-    if hashOnly {
-        maker.hashOnlyChunk = CreateChunk(config, false)
-    }
+	if hashOnly {
+		maker.hashOnlyChunk = CreateChunk(config, false)
+	}
 
-    randomData := sha256.Sum256(config.ChunkSeed)
+	randomData := sha256.Sum256(config.ChunkSeed)
 
-    for i := 0; i < 64; i++ {
-        for j := 0; j < 4; j++ {
-            maker.randomTable[4 * i + j] = binary.LittleEndian.Uint64(randomData[8 * j : 8 * j + 8])
-        }
-        randomData = sha256.Sum256(randomData[:])
-    }
+	for i := 0; i < 64; i++ {
+		for j := 0; j < 4; j++ {
+			maker.randomTable[4*i+j] = binary.LittleEndian.Uint64(randomData[8*j : 8*j+8])
+		}
+		randomData = sha256.Sum256(randomData[:])
+	}
 
-    maker.buffer = make([]byte, 2 * config.MinimumChunkSize)
+	maker.buffer = make([]byte, 2*config.MinimumChunkSize)
 
-    return maker
+	return maker
 }
 
 func rotateLeft(value uint64, bits uint) uint64 {
-    return (value << (bits & 0x3f)) | (value >> (64 - (bits & 0x3f)))
+	return (value << (bits & 0x3f)) | (value >> (64 - (bits & 0x3f)))
 }
 
 func rotateLeftByOne(value uint64) uint64 {
-    return (value << 1) | (value >> 63)
+	return (value << 1) | (value >> 63)
 }
 
-func (maker *ChunkMaker) buzhashSum(sum uint64, data [] byte) uint64 {
-    for i := 0; i < len(data); i++ {
-        sum = rotateLeftByOne(sum) ^ maker.randomTable[data[i]]
-    }
-    return sum
+func (maker *ChunkMaker) buzhashSum(sum uint64, data []byte) uint64 {
+	for i := 0; i < len(data); i++ {
+		sum = rotateLeftByOne(sum) ^ maker.randomTable[data[i]]
+	}
+	return sum
 }
 
 func (maker *ChunkMaker) buzhashUpdate(sum uint64, out byte, in byte, length int) uint64 {
-    return rotateLeftByOne(sum) ^ rotateLeft(maker.randomTable[out], uint(length)) ^ maker.randomTable[in]
+	return rotateLeftByOne(sum) ^ rotateLeft(maker.randomTable[out], uint(length)) ^ maker.randomTable[in]
 }
 
 // ForEachChunk reads data from 'reader'.  If EOF is encountered, it will call 'nextReader' to ask for next file.  If
 // 'nextReader' returns false, it will process remaining data in the buffer and then quit.  When a chunk is identified,
 // it will call 'endOfChunk' to return the chunk size and a boolean flag indicating if it is the last chunk.
 func (maker *ChunkMaker) ForEachChunk(reader io.Reader, endOfChunk func(chunk *Chunk, final bool),
-                                      nextReader func(size int64, hash string)(io.Reader, bool)) {
+	nextReader func(size int64, hash string) (io.Reader, bool)) {
 
-    maker.bufferStart = 0
-    maker.bufferSize = 0
+	maker.bufferStart = 0
+	maker.bufferSize = 0
 
-    var minimumReached bool
-    var hashSum uint64
-    var chunk *Chunk
+	var minimumReached bool
+	var hashSum uint64
+	var chunk *Chunk
 
-    fileSize := int64(0)
-    fileHasher := maker.config.NewFileHasher()
+	fileSize := int64(0)
+	fileHasher := maker.config.NewFileHasher()
 
-    // Start a new chunk.
-    startNewChunk := func() {
-        hashSum = 0
-        minimumReached = false
-        if maker.hashOnly {
-            chunk = maker.hashOnlyChunk
-            chunk.Reset(true)
-        } else {
-            chunk = maker.config.GetChunk()
-            chunk.Reset(true)
-        }
-    }
+	// Start a new chunk.
+	startNewChunk := func() {
+		hashSum = 0
+		minimumReached = false
+		if maker.hashOnly {
+			chunk = maker.hashOnlyChunk
+			chunk.Reset(true)
+		} else {
+			chunk = maker.config.GetChunk()
+			chunk.Reset(true)
+		}
+	}
 
-    // Move data from the buffer to the chunk.
-    fill := func(count int) {
-        if maker.bufferStart + count < maker.bufferCapacity {
-            chunk.Write(maker.buffer[maker.bufferStart : maker.bufferStart + count])
-            maker.bufferStart += count
-            maker.bufferSize -= count
-        } else {
-            chunk.Write(maker.buffer[maker.bufferStart :])
-            chunk.Write(maker.buffer[: count - (maker.bufferCapacity - maker.bufferStart)])
-            maker.bufferStart = count - (maker.bufferCapacity - maker.bufferStart)
-            maker.bufferSize -= count
-        }
-    }
+	// Move data from the buffer to the chunk.
+	fill := func(count int) {
+		if maker.bufferStart+count < maker.bufferCapacity {
+			chunk.Write(maker.buffer[maker.bufferStart : maker.bufferStart+count])
+			maker.bufferStart += count
+			maker.bufferSize -= count
+		} else {
+			chunk.Write(maker.buffer[maker.bufferStart:])
+			chunk.Write(maker.buffer[:count-(maker.bufferCapacity-maker.bufferStart)])
+			maker.bufferStart = count - (maker.bufferCapacity - maker.bufferStart)
+			maker.bufferSize -= count
+		}
+	}
 
-    startNewChunk()
+	startNewChunk()
 
-    var err error
+	var err error
 
-    isEOF := false
+	isEOF := false
 
-    if maker.minimumChunkSize == maker.maximumChunkSize {
+	if maker.minimumChunkSize == maker.maximumChunkSize {
 
-        if maker.bufferCapacity < maker.minimumChunkSize {
-            maker.buffer = make([]byte, maker.minimumChunkSize)
-        }
+		if maker.bufferCapacity < maker.minimumChunkSize {
+			maker.buffer = make([]byte, maker.minimumChunkSize)
+		}
 
-        for {
-            maker.bufferStart = 0
-            for maker.bufferStart < maker.minimumChunkSize && !isEOF {
-                count, err := reader.Read(maker.buffer[maker.bufferStart : maker.minimumChunkSize])
+		for {
+			maker.bufferStart = 0
+			for maker.bufferStart < maker.minimumChunkSize && !isEOF {
+				count, err := reader.Read(maker.buffer[maker.bufferStart:maker.minimumChunkSize])
 
-                if err != nil {
-                    if err != io.EOF {
-                        LOG_ERROR("CHUNK_MAKER", "Failed to read %d bytes: %s", count, err.Error())
-                        return
-                    } else {
-                        isEOF = true
-                    }
-                }
-                maker.bufferStart += count
-            }
+				if err != nil {
+					if err != io.EOF {
+						LOG_ERROR("CHUNK_MAKER", "Failed to read %d bytes: %s", count, err.Error())
+						return
+					} else {
+						isEOF = true
+					}
+				}
+				maker.bufferStart += count
+			}
 
-            fileHasher.Write(maker.buffer[:maker.bufferStart])
-            fileSize += int64(maker.bufferStart)
-            chunk.Write(maker.buffer[:maker.bufferStart])
+			fileHasher.Write(maker.buffer[:maker.bufferStart])
+			fileSize += int64(maker.bufferStart)
+			chunk.Write(maker.buffer[:maker.bufferStart])
 
-            if isEOF {
-                var ok bool
-                reader, ok = nextReader(fileSize, hex.EncodeToString(fileHasher.Sum(nil)))
-                if !ok {
-                    endOfChunk(chunk, true)
-                    return
-                } else {
-                    endOfChunk(chunk, false)
-                    startNewChunk()
-                    fileSize = 0
-                    fileHasher = maker.config.NewFileHasher()
-                    isEOF = false
-                }
-            } else {
-                endOfChunk(chunk, false)
-                startNewChunk()
-            }
-        }
+			if isEOF {
+				var ok bool
+				reader, ok = nextReader(fileSize, hex.EncodeToString(fileHasher.Sum(nil)))
+				if !ok {
+					endOfChunk(chunk, true)
+					return
+				} else {
+					endOfChunk(chunk, false)
+					startNewChunk()
+					fileSize = 0
+					fileHasher = maker.config.NewFileHasher()
+					isEOF = false
+				}
+			} else {
+				endOfChunk(chunk, false)
+				startNewChunk()
+			}
+		}
 
-    }
+	}
 
-    for {
+	for {
 
-        // If the buffer still has some space left and EOF is not seen, read more data.
-        for maker.bufferSize < maker.bufferCapacity && !isEOF {
-            start := maker.bufferStart + maker.bufferSize
-            count := maker.bufferCapacity - start
-            if start >= maker.bufferCapacity {
-                start -= maker.bufferCapacity
-                count = maker.bufferStart - start
-            }
+		// If the buffer still has some space left and EOF is not seen, read more data.
+		for maker.bufferSize < maker.bufferCapacity && !isEOF {
+			start := maker.bufferStart + maker.bufferSize
+			count := maker.bufferCapacity - start
+			if start >= maker.bufferCapacity {
+				start -= maker.bufferCapacity
+				count = maker.bufferStart - start
+			}
 
-            count, err = reader.Read(maker.buffer[start : start + count])
+			count, err = reader.Read(maker.buffer[start : start+count])
 
-            if err != nil && err != io.EOF {
-                LOG_ERROR("CHUNK_MAKER", "Failed to read %d bytes: %s", count, err.Error())
-                return
-            }
+			if err != nil && err != io.EOF {
+				LOG_ERROR("CHUNK_MAKER", "Failed to read %d bytes: %s", count, err.Error())
+				return
+			}
 
-            maker.bufferSize += count
-            fileHasher.Write(maker.buffer[start : start + count])
-            fileSize += int64(count)
+			maker.bufferSize += count
+			fileHasher.Write(maker.buffer[start : start+count])
+			fileSize += int64(count)
 
-            // if EOF is seen, try to switch to next file and continue
-            if err == io.EOF {
-                var ok bool
-                reader, ok = nextReader(fileSize, hex.EncodeToString(fileHasher.Sum(nil)))
-                if !ok {
-                    isEOF = true
-                } else {
-                    fileSize = 0
-                    fileHasher = maker.config.NewFileHasher()
-                    isEOF = false
-                }
-            }
-        }
+			// if EOF is seen, try to switch to next file and continue
+			if err == io.EOF {
+				var ok bool
+				reader, ok = nextReader(fileSize, hex.EncodeToString(fileHasher.Sum(nil)))
+				if !ok {
+					isEOF = true
+				} else {
+					fileSize = 0
+					fileHasher = maker.config.NewFileHasher()
+					isEOF = false
+				}
+			}
+		}
 
-        // No eough data to meet the minimum chunk size requirement, so just return as a chunk.
-        if maker.bufferSize < maker.minimumChunkSize {
-            fill(maker.bufferSize)
-            endOfChunk(chunk, true)
-            return
-        }
+		// No eough data to meet the minimum chunk size requirement, so just return as a chunk.
+		if maker.bufferSize < maker.minimumChunkSize {
+			fill(maker.bufferSize)
+			endOfChunk(chunk, true)
+			return
+		}
 
+		// Minimum chunk size has been reached.  Calculate the buzhash for the minimum size chunk.
+		if !minimumReached {
 
-        // Minimum chunk size has been reached.  Calculate the buzhash for the minimum size chunk.
-        if (!minimumReached) {
+			bytes := maker.minimumChunkSize
 
-            bytes := maker.minimumChunkSize
+			if maker.bufferStart+bytes < maker.bufferCapacity {
+				hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart:maker.bufferStart+bytes])
+			} else {
+				hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart:])
+				hashSum = maker.buzhashSum(hashSum,
+					maker.buffer[:bytes-(maker.bufferCapacity-maker.bufferStart)])
+			}
 
-            if maker.bufferStart + bytes < maker.bufferCapacity {
-                hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart : maker.bufferStart + bytes])
-            } else {
-                hashSum = maker.buzhashSum(0, maker.buffer[maker.bufferStart :])
-                hashSum = maker.buzhashSum(hashSum,
-                                           maker.buffer[: bytes - (maker.bufferCapacity - maker.bufferStart)])
-            }
+			if (hashSum & maker.hashMask) == 0 {
+				// This is a minimum size chunk
+				fill(bytes)
+				endOfChunk(chunk, false)
+				startNewChunk()
+				continue
+			}
 
-            if (hashSum & maker.hashMask) == 0 {
-                // This is a minimum size chunk
-                fill(bytes)
-                endOfChunk(chunk, false)
-                startNewChunk()
-                continue
-            }
+			minimumReached = true
+		}
 
-            minimumReached = true
-        }
+		// Now check the buzhash of the data in the buffer, shifting one byte at a time.
+		bytes := maker.bufferSize - maker.minimumChunkSize
+		isEOC := false
+		maxSize := maker.maximumChunkSize - chunk.GetLength()
+		for i := 0; i < maker.bufferSize-maker.minimumChunkSize; i++ {
+			out := maker.bufferStart + i
+			if out >= maker.bufferCapacity {
+				out -= maker.bufferCapacity
+			}
+			in := maker.bufferStart + i + maker.minimumChunkSize
+			if in >= maker.bufferCapacity {
+				in -= maker.bufferCapacity
+			}
 
-        // Now check the buzhash of the data in the buffer, shifting one byte at a time.
-        bytes := maker.bufferSize - maker.minimumChunkSize
-        isEOC := false
-        maxSize := maker.maximumChunkSize - chunk.GetLength()
-        for i := 0; i < maker.bufferSize - maker.minimumChunkSize; i++ {
-            out := maker.bufferStart + i
-            if out >= maker.bufferCapacity {
-                out -= maker.bufferCapacity
-            }
-            in := maker.bufferStart + i + maker.minimumChunkSize
-            if in >= maker.bufferCapacity {
-                in -= maker.bufferCapacity
-            }
+			hashSum = maker.buzhashUpdate(hashSum, maker.buffer[out], maker.buffer[in], maker.minimumChunkSize)
+			if (hashSum&maker.hashMask) == 0 || i == maxSize-maker.minimumChunkSize-1 {
+				// A chunk is completed.
+				bytes = i + 1 + maker.minimumChunkSize
+				isEOC = true
+				break
+			}
+		}
 
-            hashSum = maker.buzhashUpdate(hashSum, maker.buffer[out], maker.buffer[in], maker.minimumChunkSize)
-            if (hashSum & maker.hashMask) == 0 || i == maxSize - maker.minimumChunkSize - 1 {
-                // A chunk is completed.
-                bytes = i + 1 + maker.minimumChunkSize
-                isEOC = true
-                break
-            }
-        }
+		fill(bytes)
 
-        fill(bytes)
+		if isEOC {
+			if isEOF && maker.bufferSize == 0 {
+				endOfChunk(chunk, true)
+				return
+			}
+			endOfChunk(chunk, false)
+			startNewChunk()
+			continue
+		}
 
-        if isEOC {
-            if isEOF && maker.bufferSize == 0 {
-                endOfChunk(chunk, true)
-                return
-            }
-            endOfChunk(chunk, false)
-            startNewChunk()
-            continue
-        }
-
-        if isEOF {
-            fill(maker.bufferSize)
-            endOfChunk(chunk, true)
-            return
-        }
-    }
+		if isEOF {
+			fill(maker.bufferSize)
+			endOfChunk(chunk, true)
+			return
+		}
+	}
 }

src/duplicacy_chunkmaker_test.go

@@ -5,128 +5,127 @@
 package duplicacy
 
 import (
-    "testing"
-    "bytes"
-    crypto_rand "crypto/rand"
-    "math/rand"
-    "io"
-    "sort"
+	"bytes"
+	crypto_rand "crypto/rand"
+	"io"
+	"math/rand"
+	"sort"
+	"testing"
 )
 
 func splitIntoChunks(content []byte, n, averageChunkSize, maxChunkSize, minChunkSize,
-                     bufferCapacity int) ([]string, int) {
+	bufferCapacity int) ([]string, int) {
 
-    config := CreateConfig()
+	config := CreateConfig()
 
-    config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
-    config.AverageChunkSize = averageChunkSize
-    config.MaximumChunkSize = maxChunkSize
-    config.MinimumChunkSize = minChunkSize
-    config.ChunkSeed = []byte("duplicacy")
+	config.CompressionLevel = DEFAULT_COMPRESSION_LEVEL
+	config.AverageChunkSize = averageChunkSize
+	config.MaximumChunkSize = maxChunkSize
+	config.MinimumChunkSize = minChunkSize
+	config.ChunkSeed = []byte("duplicacy")
 
-    config.HashKey = DEFAULT_KEY
-    config.IDKey = DEFAULT_KEY
+	config.HashKey = DEFAULT_KEY
+	config.IDKey = DEFAULT_KEY
 
-    maker := CreateChunkMaker(config, false)
+	maker := CreateChunkMaker(config, false)
 
-    var chunks [] string
-    totalChunkSize := 0
-    totalFileSize := int64(0)
+	var chunks []string
+	totalChunkSize := 0
+	totalFileSize := int64(0)
 
-    //LOG_INFO("CHUNK_SPLIT", "bufferCapacity: %d", bufferCapacity)
+	//LOG_INFO("CHUNK_SPLIT", "bufferCapacity: %d", bufferCapacity)
 
-    buffers := make([] *bytes.Buffer, n)
-    sizes := make([] int, n)
-    sizes[0] = 0
-    for i := 1; i < n; i++ {
-        same := true
-        for same {
-            same = false
-            sizes[i] = rand.Int() % n
-            for j := 0; j < i; j++ {
-                if sizes[i] == sizes[j] {
-                    same = true
-                    break
-                }
-            }
-        }
-    }
+	buffers := make([]*bytes.Buffer, n)
+	sizes := make([]int, n)
+	sizes[0] = 0
+	for i := 1; i < n; i++ {
+		same := true
+		for same {
+			same = false
+			sizes[i] = rand.Int() % n
+			for j := 0; j < i; j++ {
+				if sizes[i] == sizes[j] {
+					same = true
+					break
+				}
+			}
+		}
+	}
 
-    sort.Sort(sort.IntSlice(sizes))
+	sort.Sort(sort.IntSlice(sizes))
 
-    for i := 0; i < n - 1; i++ {
-        buffers[i] = bytes.NewBuffer(content[sizes[i] : sizes[i + 1]])
-    }
-    buffers[n - 1] = bytes.NewBuffer(content[sizes[n - 1]:])
+	for i := 0; i < n-1; i++ {
+		buffers[i] = bytes.NewBuffer(content[sizes[i]:sizes[i+1]])
+	}
+	buffers[n-1] = bytes.NewBuffer(content[sizes[n-1]:])
 
-    i := 0
+	i := 0
 
-    maker.ForEachChunk(buffers[0],
-        func (chunk *Chunk, final bool) {
-            //LOG_INFO("CHUNK_SPLIT", "i: %d, chunk: %s, size: %d", i, chunk.GetHash(), size)
-            chunks = append(chunks, chunk.GetHash())
-            totalChunkSize += chunk.GetLength()
-        },
-        func (size int64, hash string) (io.Reader, bool) {
-            totalFileSize += size
-            i++
-            if i >= len(buffers) {
-                return nil, false
-            }
-            return buffers[i], true
-        })
+	maker.ForEachChunk(buffers[0],
+		func(chunk *Chunk, final bool) {
+			//LOG_INFO("CHUNK_SPLIT", "i: %d, chunk: %s, size: %d", i, chunk.GetHash(), size)
+			chunks = append(chunks, chunk.GetHash())
+			totalChunkSize += chunk.GetLength()
+		},
+		func(size int64, hash string) (io.Reader, bool) {
+			totalFileSize += size
+			i++
+			if i >= len(buffers) {
+				return nil, false
+			}
+			return buffers[i], true
+		})
 
-    if (totalFileSize != int64(totalChunkSize)) {
-        LOG_ERROR("CHUNK_SPLIT", "total chunk size: %d, total file size: %d", totalChunkSize, totalFileSize)
-    }
-    return chunks, totalChunkSize
+	if totalFileSize != int64(totalChunkSize) {
+		LOG_ERROR("CHUNK_SPLIT", "total chunk size: %d, total file size: %d", totalChunkSize, totalFileSize)
+	}
+	return chunks, totalChunkSize
 }
 
 func TestChunkMaker(t *testing.T) {
 
+	//sizes := [...] int { 64 }
+	sizes := [...]int{64, 256, 1024, 1024 * 10}
 
-    //sizes := [...] int { 64 }
-    sizes := [...] int { 64, 256, 1024, 1024 * 10 }
+	for _, size := range sizes {
 
-    for _, size := range sizes {
+		content := make([]byte, size)
+		_, err := crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			continue
+		}
 
-        content := make([]byte, size)
-        _, err := crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            continue
-        }
+		chunkArray1, totalSize1 := splitIntoChunks(content, 10, 32, 64, 16, 32)
 
-        chunkArray1, totalSize1 := splitIntoChunks(content, 10, 32, 64, 16, 32)
+		capacities := [...]int{32, 33, 34, 61, 62, 63, 64, 65, 66, 126, 127, 128, 129, 130,
+			255, 256, 257, 511, 512, 513, 1023, 1024, 1025,
+			32, 48, 64, 128, 256, 512, 1024, 2048}
 
-        capacities := [...]int { 32, 33, 34, 61, 62, 63, 64, 65, 66, 126, 127, 128, 129, 130,
-                                 255, 256, 257, 511, 512, 513, 1023, 1024, 1025,
-                                 32, 48, 64, 128, 256, 512, 1024, 2048, }
+		//capacities := [...]int { 32 }
 
-        //capacities := [...]int { 32 }
+		for _, capacity := range capacities {
 
-        for _, capacity := range capacities {
+			for _, n := range [...]int{6, 7, 8, 9, 10} {
+				chunkArray2, totalSize2 := splitIntoChunks(content, n, 32, 64, 16, capacity)
 
-            for _, n := range [...]int { 6, 7, 8, 9, 10 } {
-                chunkArray2, totalSize2 := splitIntoChunks(content, n, 32, 64, 16, capacity)
+				if totalSize1 != totalSize2 {
+					t.Errorf("[size %d, capacity %d] total size is %d instead of %d",
+						size, capacity, totalSize2, totalSize1)
+				}
 
-                if totalSize1 != totalSize2 {
-                    t.Errorf("[size %d, capacity %d] total size is %d instead of %d",
-                             size, capacity, totalSize2, totalSize1)
-                }
-
-                if len(chunkArray1) != len(chunkArray2) {
-                    t.Errorf("[size %d, capacity %d] number of chunks is %d instead of %d",
-                             size, capacity, len(chunkArray2), len(chunkArray1))
-                } else {
-                    for i := 0; i < len(chunkArray1); i++ {
-                        if chunkArray1[i] != chunkArray2[i] {
-                            t.Errorf("[size %d, capacity %d, chunk %d] chunk is different", size, capacity, i)
-                        }
-                    }
-                }
-            }
-        }
-    }
+				if len(chunkArray1) != len(chunkArray2) {
+					t.Errorf("[size %d, capacity %d] number of chunks is %d instead of %d",
+						size, capacity, len(chunkArray2), len(chunkArray1))
+				} else {
+					for i := 0; i < len(chunkArray1); i++ {
+						if chunkArray1[i] != chunkArray2[i] {
+							t.Errorf("[size %d, capacity %d, chunk %d] chunk is different", size, capacity, i)
+						}
+					}
+				}
+			}
+		}
+	}
 
 }

src/duplicacy_chunkoperator.go

@@ -0,0 +1,209 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"sync"
+	"sync/atomic"
+	"time"
+)
+
+// These are operations that ChunkOperator will perform.
+const (
+	ChunkOperationFind      = 0
+	ChunkOperationDelete    = 1
+	ChunkOperationFossilize = 2
+	ChunkOperationResurrect = 3
+)
+
+// ChunkOperatorTask is used to pass parameters for different kinds of chunk operations.
+type ChunkOperatorTask struct {
+	operation int    // The type of operation
+	chunkID   string // The chunk id
+	filePath  string // The path of the chunk file; it may be empty
+}
+
+// ChunkOperator is capable of performing multi-threaded operations on chunks.
+type ChunkOperator struct {
+	numberOfActiveTasks int64                  // The number of chunks that are being operated on
+	storage             Storage                // This storage
+	threads             int                    // Number of threads
+	taskQueue           chan ChunkOperatorTask // Operating goroutines are waiting on this channel for input
+	stopChannel         chan bool              // Used to stop all the goroutines
+
+	fossils     []string    // For fossilize operation, the paths of the fossils are stored in this slice
+	fossilsLock *sync.Mutex // The lock for 'fossils'
+}
+
+// CreateChunkOperator creates a new ChunkOperator.
+func CreateChunkOperator(storage Storage, threads int) *ChunkOperator {
+	operator := &ChunkOperator{
+		storage: storage,
+		threads: threads,
+
+		taskQueue:   make(chan ChunkOperatorTask, threads*4),
+		stopChannel: make(chan bool),
+
+		fossils:     make([]string, 0),
+		fossilsLock: &sync.Mutex{},
+	}
+
+	// Start the operator goroutines
+	for i := 0; i < operator.threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case task := <-operator.taskQueue:
+					operator.Run(threadIndex, task)
+				case <-operator.stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
+
+	return operator
+}
+
+func (operator *ChunkOperator) Stop() {
+	if atomic.LoadInt64(&operator.numberOfActiveTasks) < 0 {
+		return
+	}
+
+	for atomic.LoadInt64(&operator.numberOfActiveTasks) > 0 {
+		time.Sleep(100 * time.Millisecond)
+	}
+	for i := 0; i < operator.threads; i++ {
+		operator.stopChannel <- false
+	}
+
+	// Assign -1 to numberOfActiveTasks so Stop() can be called multiple times
+	atomic.AddInt64(&operator.numberOfActiveTasks, int64(-1))
+}
+
+func (operator *ChunkOperator) AddTask(operation int, chunkID string, filePath string) {
+
+	task := ChunkOperatorTask{
+		operation: operation,
+		chunkID:   chunkID,
+		filePath:  filePath,
+	}
+	operator.taskQueue <- task
+	atomic.AddInt64(&operator.numberOfActiveTasks, int64(1))
+}
+
+func (operator *ChunkOperator) Find(chunkID string) {
+	operator.AddTask(ChunkOperationFind, chunkID, "")
+}
+
+func (operator *ChunkOperator) Delete(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationDelete, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Fossilize(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationFossilize, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Resurrect(chunkID string, filePath string) {
+	operator.AddTask(ChunkOperationResurrect, chunkID, filePath)
+}
+
+func (operator *ChunkOperator) Run(threadIndex int, task ChunkOperatorTask) {
+	defer func() {
+		atomic.AddInt64(&operator.numberOfActiveTasks, int64(-1))
+	}()
+
+	// task.filePath may be empty.  If so, find the chunk first.
+	if task.operation == ChunkOperationDelete || task.operation == ChunkOperationFossilize {
+		if task.filePath == "" {
+			filePath, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+			if err != nil {
+				LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+				return
+			} else if !exist {
+				if task.operation == ChunkOperationDelete {
+					LOG_WARN("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+					return
+				}
+
+				fossilPath, exist, _, _ := operator.storage.FindChunk(threadIndex, task.chunkID, true)
+				if exist {
+					LOG_WARN("CHUNK_FOSSILIZE", "Chunk %s is already a fossil", task.chunkID)
+					operator.fossilsLock.Lock()
+					operator.fossils = append(operator.fossils, fossilPath)
+					operator.fossilsLock.Unlock()
+				} else {
+					LOG_ERROR("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+				}
+				return
+			}
+			task.filePath = filePath
+		}
+	}
+
+	if task.operation == ChunkOperationFind {
+		_, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+		if err != nil {
+			LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+		} else if !exist {
+			LOG_ERROR("CHUNK_FIND", "Chunk %s does not exist in the storage", task.chunkID)
+		} else {
+			LOG_DEBUG("CHUNK_FIND", "Chunk %s exists in the storage", task.chunkID)
+		}
+	} else if task.operation == ChunkOperationDelete {
+		err := operator.storage.DeleteFile(threadIndex, task.filePath)
+		if err != nil {
+			LOG_WARN("CHUNK_DELETE", "Failed to remove the file %s: %v", task.filePath, err)
+		} else {
+			if task.chunkID != "" {
+				LOG_INFO("CHUNK_DELETE", "The chunk %s has been permanently removed", task.chunkID)
+			} else {
+				LOG_INFO("CHUNK_DELETE", "Deleted file %s from the storage", task.filePath)
+			}
+		}
+	} else if task.operation == ChunkOperationFossilize {
+
+		fossilPath := task.filePath + ".fsl"
+
+		err := operator.storage.MoveFile(threadIndex, task.filePath, fossilPath)
+		if err != nil {
+			if _, exist, _, _ := operator.storage.FindChunk(threadIndex, task.chunkID, true); exist {
+				err := operator.storage.DeleteFile(threadIndex, task.filePath)
+				if err == nil {
+					LOG_TRACE("CHUNK_DELETE", "Deleted chunk file %s as the fossil already exists", task.chunkID)
+				}
+				operator.fossilsLock.Lock()
+				operator.fossils = append(operator.fossils, fossilPath)
+				operator.fossilsLock.Unlock()
+			} else {
+				LOG_ERROR("CHUNK_DELETE", "Failed to fossilize the chunk %s: %v", task.chunkID, err)
+			}
+		} else {
+			LOG_TRACE("CHUNK_FOSSILIZE", "The chunk %s has been marked as a fossil", task.chunkID)
+			operator.fossilsLock.Lock()
+			operator.fossils = append(operator.fossils, fossilPath)
+			operator.fossilsLock.Unlock()
+		}
+	} else if task.operation == ChunkOperationResurrect {
+		chunkPath, exist, _, err := operator.storage.FindChunk(threadIndex, task.chunkID, false)
+		if err != nil {
+			LOG_ERROR("CHUNK_FIND", "Failed to locate the path for the chunk %s: %v", task.chunkID, err)
+		}
+
+		if exist {
+			operator.storage.DeleteFile(threadIndex, task.filePath)
+			LOG_INFO("FOSSIL_RESURRECT", "The chunk %s already exists", task.chunkID)
+		} else {
+			err := operator.storage.MoveFile(threadIndex, task.filePath, chunkPath)
+			if err != nil {
+				LOG_ERROR("FOSSIL_RESURRECT", "Failed to resurrect the chunk %s from the fossil %s: %v",
+					task.chunkID, task.filePath, err)
+			} else {
+				LOG_INFO("FOSSIL_RESURRECT", "The chunk %s has been resurrected", task.filePath)
+			}
+		}
+	}
+}

src/duplicacy_chunkuploader.go

@@ -5,14 +5,14 @@
 package duplicacy
 
 import (
-    "sync/atomic"
-    "time"
+	"sync/atomic"
+	"time"
 )
 
 // ChunkUploadTask represents a chunk to be uploaded.
 type ChunkUploadTask struct {
-    chunk * Chunk
-    chunkIndex int
+	chunk      *Chunk
+	chunkIndex int
 }
 
 // ChunkUploader uploads chunks to the storage using one or more uploading goroutines.  Chunks are added
@@ -20,128 +20,132 @@ type ChunkUploadTask struct {
 // called when the downloading is completed.  Note that ChunkUploader does not release chunks to the
 // chunk pool; instead
 type ChunkUploader struct {
-    config *Config                    // Associated config
-    storage Storage                   // Download from this storage
-    snapshotCache *FileStorage        // Used as cache if not nil; usually for uploading snapshot chunks
-    threads int                       // Number of uploading goroutines
-    taskQueue chan ChunkUploadTask    // Uploading goroutines are listening on this channel for upload jobs
-    stopChannel chan bool             // Used to terminate uploading goroutines
+	config        *Config              // Associated config
+	storage       Storage              // Download from this storage
+	snapshotCache *FileStorage         // Used as cache if not nil; usually for uploading snapshot chunks
+	threads       int                  // Number of uploading goroutines
+	taskQueue     chan ChunkUploadTask // Uploading goroutines are listening on this channel for upload jobs
+	stopChannel   chan bool            // Used to terminate uploading goroutines
 
-    numberOfUploadingTasks int32      // The number of uploading tasks
+	numberOfUploadingTasks int32 // The number of uploading tasks
 
-    // Uploading goroutines call this function after having downloaded chunks
-    completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)
+	// Uploading goroutines call this function after having downloaded chunks
+	completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)
 }
 
 // CreateChunkUploader creates a chunk uploader.
 func CreateChunkUploader(config *Config, storage Storage, snapshotCache *FileStorage, threads int,
-                         completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)) *ChunkUploader {
-    uploader := &ChunkUploader {
-        config: config,
-        storage: storage,
-        snapshotCache: snapshotCache,
-        threads: threads,
-        taskQueue: make(chan ChunkUploadTask, 1),
-        stopChannel: make(chan bool),
-        completionFunc: completionFunc,
-    }
+	completionFunc func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int)) *ChunkUploader {
+	uploader := &ChunkUploader{
+		config:         config,
+		storage:        storage,
+		snapshotCache:  snapshotCache,
+		threads:        threads,
+		taskQueue:      make(chan ChunkUploadTask, 1),
+		stopChannel:    make(chan bool),
+		completionFunc: completionFunc,
+	}
 
-    return uploader
+	return uploader
 }
 
 // Starts starts uploading goroutines.
 func (uploader *ChunkUploader) Start() {
-    for i := 0; i < uploader.threads; i++ {
-        go func(threadIndex int) {
-            defer CatchLogException()
-            for {
-                select {
-                case task := <- uploader.taskQueue:
-                    uploader.Upload(threadIndex, task)
-                case <- uploader.stopChannel:
-                    return
-                }
-            }
-        } (i)
-    }
+	for i := 0; i < uploader.threads; i++ {
+		go func(threadIndex int) {
+			defer CatchLogException()
+			for {
+				select {
+				case task := <-uploader.taskQueue:
+					uploader.Upload(threadIndex, task)
+				case <-uploader.stopChannel:
+					return
+				}
+			}
+		}(i)
+	}
 }
 
 // StartChunk sends a chunk to be uploaded to  a waiting uploading goroutine.  It may block if all uploading goroutines are busy.
 func (uploader *ChunkUploader) StartChunk(chunk *Chunk, chunkIndex int) {
-    atomic.AddInt32(&uploader.numberOfUploadingTasks, 1)
-    uploader.taskQueue <- ChunkUploadTask {
-        chunk: chunk,
-        chunkIndex: chunkIndex,
-    }
+	atomic.AddInt32(&uploader.numberOfUploadingTasks, 1)
+	uploader.taskQueue <- ChunkUploadTask{
+		chunk:      chunk,
+		chunkIndex: chunkIndex,
+	}
 }
 
 // Stop stops all uploading goroutines.
 func (uploader *ChunkUploader) Stop() {
-    for atomic.LoadInt32(&uploader.numberOfUploadingTasks) > 0 {
-        time.Sleep(100 * time.Millisecond)
-    }
-    for i := 0; i < uploader.threads; i++ {
-        uploader.stopChannel <- false
-    }
+	for atomic.LoadInt32(&uploader.numberOfUploadingTasks) > 0 {
+		time.Sleep(100 * time.Millisecond)
+	}
+	for i := 0; i < uploader.threads; i++ {
+		uploader.stopChannel <- false
+	}
 }
 
 // Upload is called by the uploading goroutines to perform the actual uploading
 func (uploader *ChunkUploader) Upload(threadIndex int, task ChunkUploadTask) bool {
 
-    chunk := task.chunk
-    chunkSize := chunk.GetLength()
-    chunkID := chunk.GetID()
+	chunk := task.chunk
+	chunkSize := chunk.GetLength()
+	chunkID := chunk.GetID()
 
-    // For a snapshot chunk, verify that its chunk id is correct
-    if uploader.snapshotCache != nil {
-        chunk.VerifyID()
-    }
+	// For a snapshot chunk, verify that its chunk id is correct
+	if uploader.snapshotCache != nil {
+		chunk.VerifyID()
+	}
 
-    if uploader.snapshotCache != nil && uploader.storage.IsCacheNeeded() {
-        // Save a copy to the local snapshot.
-        chunkPath, exist, _, err := uploader.snapshotCache.FindChunk(threadIndex, chunkID, false)
-        if err != nil {
-            LOG_WARN("UPLOAD_CACHE", "Failed to find the cache path for the chunk %s: %v", chunkID, err)
-        } else if exist {
-            LOG_DEBUG("CHUNK_CACHE", "Chunk %s already exists in the snapshot cache", chunkID)
-        } else if err = uploader.snapshotCache.UploadFile(threadIndex, chunkPath, chunk.GetBytes()); err != nil {
-            LOG_WARN("UPLOAD_CACHE", "Failed to save the chunk %s to the snapshot cache: %v", chunkID, err)
-        } else {
-            LOG_DEBUG("CHUNK_CACHE", "Chunk %s has been saved to the snapshot cache", chunkID)
-        }
-    }
+	if uploader.snapshotCache != nil && uploader.storage.IsCacheNeeded() {
+		// Save a copy to the local snapshot.
+		chunkPath, exist, _, err := uploader.snapshotCache.FindChunk(threadIndex, chunkID, false)
+		if err != nil {
+			LOG_WARN("UPLOAD_CACHE", "Failed to find the cache path for the chunk %s: %v", chunkID, err)
+		} else if exist {
+			LOG_DEBUG("CHUNK_CACHE", "Chunk %s already exists in the snapshot cache", chunkID)
+		} else if err = uploader.snapshotCache.UploadFile(threadIndex, chunkPath, chunk.GetBytes()); err != nil {
+			LOG_WARN("UPLOAD_CACHE", "Failed to save the chunk %s to the snapshot cache: %v", chunkID, err)
+		} else {
+			LOG_DEBUG("CHUNK_CACHE", "Chunk %s has been saved to the snapshot cache", chunkID)
+		}
+	}
 
-    // This returns the path the chunk file should be at.
-    chunkPath, exist, _, err := uploader.storage.FindChunk(threadIndex, chunkID, false)
-    if err != nil {
-        LOG_ERROR("UPLOAD_CHUNK", "Failed to find the path for the chunk %s: %v", chunkID, err)
-        return false
-    }
+	// This returns the path the chunk file should be at.
+	chunkPath, exist, _, err := uploader.storage.FindChunk(threadIndex, chunkID, false)
+	if err != nil {
+		LOG_ERROR("UPLOAD_CHUNK", "Failed to find the path for the chunk %s: %v", chunkID, err)
+		return false
+	}
 
-    if exist {
-        // Chunk deduplication by name in effect here.
-        LOG_DEBUG("CHUNK_DUPLICATE", "Chunk %s already exists", chunkID)
+	if exist {
+		// Chunk deduplication by name in effect here.
+		LOG_DEBUG("CHUNK_DUPLICATE", "Chunk %s already exists", chunkID)
 
-        uploader.completionFunc(chunk, task.chunkIndex, false, chunkSize, 0)
-        atomic.AddInt32(&uploader.numberOfUploadingTasks, -1)
-        return false
-    }
+		uploader.completionFunc(chunk, task.chunkIndex, true, chunkSize, 0)
+		atomic.AddInt32(&uploader.numberOfUploadingTasks, -1)
+		return false
+	}
 
-    // Encrypt the chunk only after we know that it must be uploaded.
-    err = chunk.Encrypt(uploader.config.ChunkKey, chunk.GetHash())
-    if err != nil {
-        LOG_ERROR("UPLOAD_CHUNK", "Failed to encrypt the chunk %s: %v", chunkID, err)
-        return false
-    }
+	// Encrypt the chunk only after we know that it must be uploaded.
+	err = chunk.Encrypt(uploader.config.ChunkKey, chunk.GetHash())
+	if err != nil {
+		LOG_ERROR("UPLOAD_CHUNK", "Failed to encrypt the chunk %s: %v", chunkID, err)
+		return false
+	}
 
-    err = uploader.storage.UploadFile(threadIndex, chunkPath, chunk.GetBytes())
-    if err != nil {
-        LOG_ERROR("UPLOAD_CHUNK", "Failed to upload the chunk %s: %v", chunkID, err)
-        return false
-    }
+	if !uploader.config.dryRun {
+		err = uploader.storage.UploadFile(threadIndex, chunkPath, chunk.GetBytes())
+		if err != nil {
+			LOG_ERROR("UPLOAD_CHUNK", "Failed to upload the chunk %s: %v", chunkID, err)
+			return false
+		}
+		LOG_DEBUG("CHUNK_UPLOAD", "Chunk %s has been uploaded", chunkID)
+	} else {
+		LOG_DEBUG("CHUNK_UPLOAD", "Uploading was skipped for chunk %s", chunkID)
+	}
 
-    LOG_DEBUG("CHUNK_UPLOAD", "Chunk %s has been uploaded", chunkID)
-    uploader.completionFunc(chunk, task.chunkIndex, false, chunkSize, chunk.GetLength())
-    atomic.AddInt32(&uploader.numberOfUploadingTasks, -1)
-    return true
+	uploader.completionFunc(chunk, task.chunkIndex, false, chunkSize, chunk.GetLength())
+	atomic.AddInt32(&uploader.numberOfUploadingTasks, -1)
+	return true
 }

src/duplicacy_chunkuploader_test.go

@@ -5,126 +5,124 @@
 package duplicacy
 
 import (
-    "os"
-    "time"
-    "path"
-    "testing"
-    "runtime/debug"
+	"os"
+	"path"
+	"runtime/debug"
+	"testing"
+	"time"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
+	crypto_rand "crypto/rand"
+	"math/rand"
 )
 
 func TestUploaderAndDownloader(t *testing.T) {
 
-    rand.Seed(time.Now().UnixNano())
-    setTestingT(t)
-    SetLoggingLevel(INFO)
+	rand.Seed(time.Now().UnixNano())
+	setTestingT(t)
+	SetLoggingLevel(INFO)
 
-    defer func() {
-        if r := recover(); r != nil {
-            switch e := r.(type) {
-            case Exception:
-                t.Errorf("%s %s", e.LogID, e.Message)
-                debug.PrintStack()
-            default:
-                t.Errorf("%v", e)
-                debug.PrintStack()
-            }
-        }
-    } ()
+	defer func() {
+		if r := recover(); r != nil {
+			switch e := r.(type) {
+			case Exception:
+				t.Errorf("%s %s", e.LogID, e.Message)
+				debug.PrintStack()
+			default:
+				t.Errorf("%v", e)
+				debug.PrintStack()
+			}
+		}
+	}()
 
-    testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
-    os.RemoveAll(testDir)
-    os.MkdirAll(testDir, 0700)
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)
 
-    t.Logf("storage: %s", testStorageName)
+	t.Logf("storage: %s", testStorageName)
 
-    storage, err := loadStorage(testDir, 1)
-    if err != nil {
-        t.Errorf("Failed to create storage: %v", err)
-        return
-    }
-    storage.EnableTestMode()
-    storage.SetRateLimits(testRateLimit, testRateLimit)
+	storage, err := loadStorage(testDir, 1)
+	if err != nil {
+		t.Errorf("Failed to create storage: %v", err)
+		return
+	}
+	storage.EnableTestMode()
+	storage.SetRateLimits(testRateLimit, testRateLimit)
 
-    for _, dir := range []string { "chunks", "snapshots" }  {
-        err = storage.CreateDirectory(0, dir)
-        if err != nil {
-            t.Errorf("Failed to create directory %s: %v", dir, err)
-            return
-        }
-    }
+	for _, dir := range []string{"chunks", "snapshots"} {
+		err = storage.CreateDirectory(0, dir)
+		if err != nil {
+			t.Errorf("Failed to create directory %s: %v", dir, err)
+			return
+		}
+	}
 
+	numberOfChunks := 100
+	maxChunkSize := 64 * 1024
 
-    numberOfChunks := 100
-    maxChunkSize := 64 * 1024
+	if testQuickMode {
+		numberOfChunks = 10
+	}
 
-    if testQuickMode {
-        numberOfChunks = 10
-    }
+	var chunks []*Chunk
 
-    var chunks []*Chunk
+	config := CreateConfig()
+	config.MinimumChunkSize = 100
+	config.chunkPool = make(chan *Chunk, numberOfChunks*2)
+	totalFileSize := 0
 
-    config := CreateConfig()
-    config.MinimumChunkSize = 100
-    config.chunkPool = make(chan *Chunk, numberOfChunks * 2)
-    totalFileSize := 0
+	for i := 0; i < numberOfChunks; i++ {
+		content := make([]byte, rand.Int()%maxChunkSize+1)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-    for i := 0; i < numberOfChunks; i++ {
-        content := make([]byte, rand.Int() % maxChunkSize + 1)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+		chunk := CreateChunk(config, true)
+		chunk.Reset(true)
+		chunk.Write(content)
+		chunks = append(chunks, chunk)
 
-        chunk := CreateChunk(config, true)
-        chunk.Reset(true)
-        chunk.Write(content)
-        chunks = append(chunks, chunk)
+		t.Logf("Chunk: %s, size: %d", chunk.GetID(), chunk.GetLength())
+		totalFileSize += chunk.GetLength()
+	}
 
-        t.Logf("Chunk: %s, size: %d", chunk.GetID(), chunk.GetLength())
-        totalFileSize += chunk.GetLength()
-    }
+	completionFunc := func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int) {
+		t.Logf("Chunk %s size %d (%d/%d) uploaded", chunk.GetID(), chunkSize, chunkIndex, len(chunks))
+	}
 
-    completionFunc := func(chunk *Chunk, chunkIndex int, skipped bool, chunkSize int, uploadSize int) {
-        t.Logf("Chunk %s size %d (%d/%d) uploaded", chunk.GetID(), chunkSize, chunkIndex, len(chunks))
-    }
+	chunkUploader := CreateChunkUploader(config, storage, nil, testThreads, nil)
+	chunkUploader.completionFunc = completionFunc
+	chunkUploader.Start()
 
-    chunkUploader := CreateChunkUploader(config, storage, nil, testThreads, nil)
-    chunkUploader.completionFunc = completionFunc
-    chunkUploader.Start()
+	for i, chunk := range chunks {
+		chunkUploader.StartChunk(chunk, i)
+	}
 
-    for i, chunk := range chunks {
-        chunkUploader.StartChunk(chunk, i)
-    }
+	chunkUploader.Stop()
 
-    chunkUploader.Stop()
+	chunkDownloader := CreateChunkDownloader(config, storage, nil, true, testThreads)
+	chunkDownloader.totalChunkSize = int64(totalFileSize)
 
+	for _, chunk := range chunks {
+		chunkDownloader.AddChunk(chunk.GetHash())
+	}
 
-    chunkDownloader := CreateChunkDownloader(config, storage, nil, true, testThreads)
-    chunkDownloader.totalChunkSize = int64(totalFileSize)
+	for i, chunk := range chunks {
+		downloaded := chunkDownloader.WaitForChunk(i)
+		if downloaded.GetID() != chunk.GetID() {
+			t.Errorf("Uploaded: %s, downloaded: %s", chunk.GetID(), downloaded.GetID())
+		}
+	}
 
-    for _, chunk := range chunks {
-        chunkDownloader.AddChunk(chunk.GetHash())
-    }
+	chunkDownloader.Stop()
 
-    for i, chunk := range chunks {
-        downloaded := chunkDownloader.WaitForChunk(i)
-        if downloaded.GetID() != chunk.GetID() {
-            t.Error("Uploaded: %s, downloaded: %s", chunk.GetID(), downloaded.GetID())
-        }
-    }
-
-    chunkDownloader.Stop()
-
-    for _, file := range listChunks(storage) {
-        err = storage.DeleteFile(0, "chunks/" + file)
-        if err != nil {
-            t.Errorf("Failed to delete the file %s: %v", file, err)
-            return
-        }
-    }
+	for _, file := range listChunks(storage) {
+		err = storage.DeleteFile(0, "chunks/"+file)
+		if err != nil {
+			t.Errorf("Failed to delete the file %s: %v", file, err)
+			return
+		}
+	}
 
 }

src/duplicacy_config.go

@@ -5,20 +5,21 @@
 package duplicacy
 
 import (
-    "encoding/json"
-    "bytes"
-    "os"
-    "fmt"
-    "hash"
-    "runtime"
-    "runtime/debug"
-    "sync/atomic"
-    "crypto/rand"
-    "crypto/hmac"
-    "crypto/sha256"
-    "encoding/hex"
+	"bytes"
+	"crypto/hmac"
+	"crypto/rand"
+	"crypto/sha256"
+	"encoding/binary"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"hash"
+	"os"
+	"runtime"
+	"runtime/debug"
+	"sync/atomic"
 
-    blake2 "github.com/minio/blake2b-simd"
+	blake2 "github.com/minio/blake2b-simd"
 )
 
 // If encryption is turned off, use this key for HMAC-SHA256 or chunk ID generation etc.
@@ -28,209 +29,228 @@ var DEFAULT_KEY = []byte("duplicacy")
 // standard zlib levels of -1 to 9.
 var DEFAULT_COMPRESSION_LEVEL = 100
 
+// The new header of the config file (to differentiate from the old format where the salt and iterations are fixed)
+var CONFIG_HEADER = "duplicacy\001"
+
+// The length of the salt used in the new format
+var CONFIG_SALT_LENGTH = 32
+
+// The default iterations for key derivation
+var CONFIG_DEFAULT_ITERATIONS = 16384
+
 type Config struct {
-    CompressionLevel int                   `json:"compression-level"`
-    AverageChunkSize int                   `json:"average-chunk-size"`
-    MaximumChunkSize int                   `json:"max-chunk-size"`
-    MinimumChunkSize int                   `json:"min-chunk-size"`
+	CompressionLevel int `json:"compression-level"`
+	AverageChunkSize int `json:"average-chunk-size"`
+	MaximumChunkSize int `json:"max-chunk-size"`
+	MinimumChunkSize int `json:"min-chunk-size"`
 
-    ChunkSeed []byte                       `json:"chunk-seed"`
+	ChunkSeed []byte `json:"chunk-seed"`
 
-    // Use HMAC-SHA256(hashKey, plaintext) as the chunk hash.
-    // Use HMAC-SHA256(idKey, chunk hash) as the file name of the chunk
-    // For chunks, use HMAC-SHA256(chunkKey, chunk hash) as the encryption key
-    // For files, use HMAC-SHA256(fileKey, file path) as the encryption key
+	FixedNesting bool `json:"fixed-nesting"`
 
-    // the HMAC-SHA256 key of the chunk data
-    HashKey  []byte                        `json:"-"`
+	// Use HMAC-SHA256(hashKey, plaintext) as the chunk hash.
+	// Use HMAC-SHA256(idKey, chunk hash) as the file name of the chunk
+	// For chunks, use HMAC-SHA256(chunkKey, chunk hash) as the encryption key
+	// For files, use HMAC-SHA256(fileKey, file path) as the encryption key
 
-    // used to generate an id from the chunk hash
-    IDKey    []byte                        `json:"-"`
+	// the HMAC-SHA256 key of the chunk data
+	HashKey []byte `json:"-"`
 
-    // for encrypting a chunk
-    ChunkKey []byte                        `json:"-"`
+	// used to generate an id from the chunk hash
+	IDKey []byte `json:"-"`
 
-    // for encrypting a non-chunk file
-    FileKey  []byte                        `json:"-"`
+	// for encrypting a chunk
+	ChunkKey []byte `json:"-"`
 
-    chunkPool chan *Chunk                  `json:"-"`
-    numberOfChunks int32
+	// for encrypting a non-chunk file
+	FileKey []byte `json:"-"`
+
+	chunkPool      chan *Chunk
+	numberOfChunks int32
+	dryRun         bool
 }
 
-    // Create an alias to avoid recursive calls on Config.MarshalJSON
+// Create an alias to avoid recursive calls on Config.MarshalJSON
 type aliasedConfig Config
 
 type jsonableConfig struct {
-    *aliasedConfig
-    ChunkSeed string `json:"chunk-seed"`
-    HashKey   string `json:"hash-key"`
-    IDKey     string `json:"id-key"`
-    ChunkKey  string `json:"chunk-key"`
-    FileKey   string `json:"file-key"`
+	*aliasedConfig
+	ChunkSeed string `json:"chunk-seed"`
+	HashKey   string `json:"hash-key"`
+	IDKey     string `json:"id-key"`
+	ChunkKey  string `json:"chunk-key"`
+	FileKey   string `json:"file-key"`
 }
 
-func (config *Config) MarshalJSON() ([] byte, error) {
+func (config *Config) MarshalJSON() ([]byte, error) {
 
-    return json.Marshal(&jsonableConfig {
-        aliasedConfig: (*aliasedConfig)(config),
-        ChunkSeed:     hex.EncodeToString(config.ChunkSeed),
-        HashKey:       hex.EncodeToString(config.HashKey),
-        IDKey:         hex.EncodeToString(config.IDKey),
-        ChunkKey:      hex.EncodeToString(config.ChunkKey),
-        FileKey:       hex.EncodeToString(config.FileKey),
-    })
+	return json.Marshal(&jsonableConfig{
+		aliasedConfig: (*aliasedConfig)(config),
+		ChunkSeed:     hex.EncodeToString(config.ChunkSeed),
+		HashKey:       hex.EncodeToString(config.HashKey),
+		IDKey:         hex.EncodeToString(config.IDKey),
+		ChunkKey:      hex.EncodeToString(config.ChunkKey),
+		FileKey:       hex.EncodeToString(config.FileKey),
+	})
 }
 
 func (config *Config) UnmarshalJSON(description []byte) (err error) {
 
-    aliased := &jsonableConfig {
-        aliasedConfig: (*aliasedConfig)(config),
-    }
+	aliased := &jsonableConfig{
+		aliasedConfig: (*aliasedConfig)(config),
+	}
 
-    if err = json.Unmarshal(description, &aliased); err != nil {
-        return err
-    }
+	if err = json.Unmarshal(description, &aliased); err != nil {
+		return err
+	}
 
-    if config.ChunkSeed, err = hex.DecodeString(aliased.ChunkSeed); err != nil {
-        return fmt.Errorf("Invalid representation of the chunk seed in the config")
-    }
-    if config.HashKey, err = hex.DecodeString(aliased.HashKey); err != nil {
-        return fmt.Errorf("Invalid representation of the hash key in the config")
-    }
-    if config.IDKey, err = hex.DecodeString(aliased.IDKey); err != nil {
-        return fmt.Errorf("Invalid representation of the id key in the config")
-    }
-    if config.ChunkKey, err = hex.DecodeString(aliased.ChunkKey); err != nil {
-        return fmt.Errorf("Invalid representation of the chunk key in the config")
-    }
-    if config.FileKey, err = hex.DecodeString(aliased.FileKey); err != nil {
-        return fmt.Errorf("Invalid representation of the file key in the config")
-    }
+	if config.ChunkSeed, err = hex.DecodeString(aliased.ChunkSeed); err != nil {
+		return fmt.Errorf("Invalid representation of the chunk seed in the config")
+	}
+	if config.HashKey, err = hex.DecodeString(aliased.HashKey); err != nil {
+		return fmt.Errorf("Invalid representation of the hash key in the config")
+	}
+	if config.IDKey, err = hex.DecodeString(aliased.IDKey); err != nil {
+		return fmt.Errorf("Invalid representation of the id key in the config")
+	}
+	if config.ChunkKey, err = hex.DecodeString(aliased.ChunkKey); err != nil {
+		return fmt.Errorf("Invalid representation of the chunk key in the config")
+	}
+	if config.FileKey, err = hex.DecodeString(aliased.FileKey); err != nil {
+		return fmt.Errorf("Invalid representation of the file key in the config")
+	}
 
-    return nil
+	return nil
 }
 
 func (config *Config) IsCompatiableWith(otherConfig *Config) bool {
 
-    return config.CompressionLevel == otherConfig.CompressionLevel &&
-           config.AverageChunkSize == otherConfig.AverageChunkSize &&
-           config.MaximumChunkSize == otherConfig.MaximumChunkSize &&
-           config.MinimumChunkSize == otherConfig.MinimumChunkSize &&
-           bytes.Equal(config.ChunkSeed, otherConfig.ChunkSeed)    &&
-           bytes.Equal(config.HashKey, otherConfig.HashKey)
+	return config.CompressionLevel == otherConfig.CompressionLevel &&
+		config.AverageChunkSize == otherConfig.AverageChunkSize &&
+		config.MaximumChunkSize == otherConfig.MaximumChunkSize &&
+		config.MinimumChunkSize == otherConfig.MinimumChunkSize &&
+		bytes.Equal(config.ChunkSeed, otherConfig.ChunkSeed) &&
+		bytes.Equal(config.HashKey, otherConfig.HashKey)
 }
 
 func (config *Config) Print() {
 
-    LOG_INFO("CONFIG_INFO", "Compression level: %d", config.CompressionLevel)
-    LOG_INFO("CONFIG_INFO", "Average chunk size: %d", config.AverageChunkSize)
-    LOG_INFO("CONFIG_INFO", "Maximum chunk size: %d", config.MaximumChunkSize)
-    LOG_INFO("CONFIG_INFO", "Minimum chunk size: %d", config.MinimumChunkSize)
-    LOG_INFO("CONFIG_INFO", "Chunk seed: %x", config.ChunkSeed)
+	LOG_INFO("CONFIG_INFO", "Compression level: %d", config.CompressionLevel)
+	LOG_INFO("CONFIG_INFO", "Average chunk size: %d", config.AverageChunkSize)
+	LOG_INFO("CONFIG_INFO", "Maximum chunk size: %d", config.MaximumChunkSize)
+	LOG_INFO("CONFIG_INFO", "Minimum chunk size: %d", config.MinimumChunkSize)
+	LOG_INFO("CONFIG_INFO", "Chunk seed: %x", config.ChunkSeed)
 }
 
 func CreateConfigFromParameters(compressionLevel int, averageChunkSize int, maximumChunkSize int, mininumChunkSize int,
-                  isEncrypted bool, copyFrom *Config) (config *Config) {
+	isEncrypted bool, copyFrom *Config, bitCopy bool) (config *Config) {
 
-    config = &Config {
-        CompressionLevel: compressionLevel,
-        AverageChunkSize: averageChunkSize,
-        MaximumChunkSize: maximumChunkSize,
-        MinimumChunkSize: mininumChunkSize,
-    }
+	config = &Config{
+		CompressionLevel: compressionLevel,
+		AverageChunkSize: averageChunkSize,
+		MaximumChunkSize: maximumChunkSize,
+		MinimumChunkSize: mininumChunkSize,
+		FixedNesting:     true,
+	}
 
-    if isEncrypted {
-        // Randomly generate keys
-        keys := make([]byte, 32 * 5)
-        _, err := rand.Read(keys)
-        if err != nil {
-            LOG_ERROR("CONFIG_KEY", "Failed to generate random keys: %v", err)
-            return nil
-        }
+	if isEncrypted {
+		// Randomly generate keys
+		keys := make([]byte, 32*5)
+		_, err := rand.Read(keys)
+		if err != nil {
+			LOG_ERROR("CONFIG_KEY", "Failed to generate random keys: %v", err)
+			return nil
+		}
 
-        config.ChunkSeed = keys[:32]
-        config.HashKey   = keys[32:64]
-        config.IDKey     = keys[64:96]
-        config.ChunkKey  = keys[96:128]
-        config.FileKey   = keys[128:]
-    } else {
-        config.ChunkSeed = DEFAULT_KEY
-        config.HashKey   = DEFAULT_KEY
-        config.IDKey     = DEFAULT_KEY
-    }
+		config.ChunkSeed = keys[:32]
+		config.HashKey = keys[32:64]
+		config.IDKey = keys[64:96]
+		config.ChunkKey = keys[96:128]
+		config.FileKey = keys[128:]
+	} else {
+		config.ChunkSeed = DEFAULT_KEY
+		config.HashKey = DEFAULT_KEY
+		config.IDKey = DEFAULT_KEY
+	}
 
-    if copyFrom != nil {
-        config.CompressionLevel = copyFrom.CompressionLevel
+	if copyFrom != nil {
+		config.CompressionLevel = copyFrom.CompressionLevel
 
-        config.AverageChunkSize = copyFrom.AverageChunkSize
-        config.MaximumChunkSize = copyFrom.MaximumChunkSize
-        config.MinimumChunkSize = copyFrom.MinimumChunkSize
+		config.AverageChunkSize = copyFrom.AverageChunkSize
+		config.MaximumChunkSize = copyFrom.MaximumChunkSize
+		config.MinimumChunkSize = copyFrom.MinimumChunkSize
 
-        config.ChunkSeed        = copyFrom.ChunkSeed
-        config.HashKey          = copyFrom.HashKey
-    }
+		config.ChunkSeed = copyFrom.ChunkSeed
+		config.HashKey = copyFrom.HashKey
 
-    config.chunkPool = make(chan *Chunk, runtime.NumCPU() * 16)
+		if bitCopy {
+			config.IDKey = copyFrom.IDKey
+			config.ChunkKey = copyFrom.ChunkKey
+			config.FileKey = copyFrom.FileKey
+		}
+	}
 
-    return config
+	config.chunkPool = make(chan *Chunk, runtime.NumCPU()*16)
+
+	return config
 }
 
 func CreateConfig() (config *Config) {
-    return &Config {
-        HashKey: DEFAULT_KEY,
-        IDKey: DEFAULT_KEY,
-        CompressionLevel: DEFAULT_COMPRESSION_LEVEL,
-        chunkPool: make(chan *Chunk, runtime.NumCPU() * 16),
-    }
+	return &Config{
+		HashKey:          DEFAULT_KEY,
+		IDKey:            DEFAULT_KEY,
+		CompressionLevel: DEFAULT_COMPRESSION_LEVEL,
+		chunkPool:        make(chan *Chunk, runtime.NumCPU()*16),
+	}
 }
 
 func (config *Config) GetChunk() (chunk *Chunk) {
-    select {
-    case chunk = <- config.chunkPool :
-    default:
-        numberOfChunks := atomic.AddInt32(&config.numberOfChunks, 1)
-        if numberOfChunks >= int32(runtime.NumCPU() * 16) {
-            LOG_WARN("CONFIG_CHUNK", "%d chunks have been allocated", numberOfChunks)
-            if _, found := os.LookupEnv("DUPLICACY_CHUNK_DEBUG"); found {
-                debug.PrintStack()
-            }
-        }
-        chunk = CreateChunk(config, true)
-    }
-    return chunk
+	select {
+	case chunk = <-config.chunkPool:
+	default:
+		numberOfChunks := atomic.AddInt32(&config.numberOfChunks, 1)
+		if numberOfChunks >= int32(runtime.NumCPU()*16) {
+			LOG_WARN("CONFIG_CHUNK", "%d chunks have been allocated", numberOfChunks)
+			if _, found := os.LookupEnv("DUPLICACY_CHUNK_DEBUG"); found {
+				debug.PrintStack()
+			}
+		}
+		chunk = CreateChunk(config, true)
+	}
+	return chunk
 }
 
-func (config *Config) PutChunk(chunk *Chunk){
+func (config *Config) PutChunk(chunk *Chunk) {
 
-    if chunk == nil {
-        return
-    }
+	if chunk == nil {
+		return
+	}
 
-    select {
-    case config.chunkPool <- chunk:
-    default:
-        LOG_INFO("CHUNK_BUFFER", "Discarding a free chunk due to a full pool")
-    }
+	select {
+	case config.chunkPool <- chunk:
+	default:
+		LOG_INFO("CHUNK_BUFFER", "Discarding a free chunk due to a full pool")
+	}
 }
 
 func (config *Config) NewKeyedHasher(key []byte) hash.Hash {
-    if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
-        hasher, err := blake2.New(&blake2.Config{ Size: 32, Key:key })
-        if err != nil {
-            LOG_ERROR("HASH_KEY", "Invalid hash key: %x", key)
-        }
-        return hasher
-    } else {
-        return hmac.New(sha256.New, key)
-    }
+	if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
+		hasher, err := blake2.New(&blake2.Config{Size: 32, Key: key})
+		if err != nil {
+			LOG_ERROR("HASH_KEY", "Invalid hash key: %x", key)
+		}
+		return hasher
+	} else {
+		return hmac.New(sha256.New, key)
+	}
 }
 
 var SkipFileHash = false
 
 func init() {
-    if value, found := os.LookupEnv("DUPLICACY_SKIP_FILE_HASH"); found && value != "" && value != "0" {
-        SkipFileHash = true
-    }
+	if value, found := os.LookupEnv("DUPLICACY_SKIP_FILE_HASH"); found && value != "" && value != "0" {
+		SkipFileHash = true
+	}
 }
 
 // Implement a dummy hasher to be used when SkipFileHash is true.
@@ -238,190 +258,243 @@ type DummyHasher struct {
 }
 
 func (hasher *DummyHasher) Write(p []byte) (int, error) {
-    return len(p), nil
+	return len(p), nil
 }
 
 func (hasher *DummyHasher) Sum(b []byte) []byte {
-    return []byte("")
+	return []byte("")
 }
 
 func (hasher *DummyHasher) Reset() {
 }
 
 func (hasher *DummyHasher) Size() int {
-    return 0
+	return 0
 }
 
 func (hasher *DummyHasher) BlockSize() int {
-    return 0
+	return 0
 }
 
 func (config *Config) NewFileHasher() hash.Hash {
-    if SkipFileHash {
-        return &DummyHasher {}
-    } else if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
-        hasher, _ := blake2.New(&blake2.Config{ Size: 32 })
-        return hasher
-    } else {
-        return sha256.New()
-    }
+	if SkipFileHash {
+		return &DummyHasher{}
+	} else if config.CompressionLevel == DEFAULT_COMPRESSION_LEVEL {
+		hasher, _ := blake2.New(&blake2.Config{Size: 32})
+		return hasher
+	} else {
+		return sha256.New()
+	}
 }
 
 // Calculate the file hash using the corresponding hasher
 func (config *Config) ComputeFileHash(path string, buffer []byte) string {
 
-    file, err := os.Open(path)
-      if err != nil {
-        return ""
-      }
+	file, err := os.Open(path)
+	if err != nil {
+		return ""
+	}
 
-    hasher := config.NewFileHasher()
-    defer file.Close()
+	hasher := config.NewFileHasher()
+	defer file.Close()
 
-    count := 1
-    for count > 0 {
-        count, err = file.Read(buffer)
-        hasher.Write(buffer[:count])
-    }
+	count := 1
+	for count > 0 {
+		count, err = file.Read(buffer)
+		hasher.Write(buffer[:count])
+	}
 
-      return hex.EncodeToString(hasher.Sum(nil))
+	return hex.EncodeToString(hasher.Sum(nil))
 }
 
 // GetChunkIDFromHash creates a chunk id from the chunk hash.  The chunk id will be used as the name of the chunk
 // file, so it is publicly exposed.  The chunk hash is the HMAC-SHA256 of what is contained in the chunk and should
 // never be exposed.
 func (config *Config) GetChunkIDFromHash(hash string) string {
-    hasher := config.NewKeyedHasher(config.IDKey)
-    hasher.Write([]byte(hash))
-    return hex.EncodeToString(hasher.Sum(nil))
+	hasher := config.NewKeyedHasher(config.IDKey)
+	hasher.Write([]byte(hash))
+	return hex.EncodeToString(hasher.Sum(nil))
 }
 
 func DownloadConfig(storage Storage, password string) (config *Config, isEncrypted bool, err error) {
-    // Although the default key is passed to the function call the key is not actually used since there is no need to
-    // calculate the hash or id of the config file.
-    configFile := CreateChunk(CreateConfig(), true)
+	// Although the default key is passed to the function call the key is not actually used since there is no need to
+	// calculate the hash or id of the config file.
+	configFile := CreateChunk(CreateConfig(), true)
 
-    exist, _, _, err := storage.GetFileInfo(0, "config")
-    if err != nil {
-        return nil, false, err
-    }
+	exist, _, _, err := storage.GetFileInfo(0, "config")
+	if err != nil {
+		return nil, false, err
+	}
 
-    if !exist {
-        return nil, false, nil
-    }
+	if !exist {
+		return nil, false, nil
+	}
 
-    err = storage.DownloadFile(0, "config", configFile)
-    if err != nil {
-        return nil, false, err
-    }
+	err = storage.DownloadFile(0, "config", configFile)
+	if err != nil {
+		return nil, false, err
+	}
 
-    var masterKey []byte
+	if len(configFile.GetBytes()) < len(ENCRYPTION_HEADER) {
+		return nil, false, fmt.Errorf("The storage has an invalid config file")
+	}
 
-    if len(password) > 0 {
-        masterKey = GenerateKeyFromPassword(password)
+	if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)-1]) == ENCRYPTION_HEADER[:len(ENCRYPTION_HEADER)-1] && len(password) == 0 {
+		return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
+	}
 
-        // Decrypt the config file.  masterKey == nil means no encryption.
-        err = configFile.Decrypt(masterKey, "")
-        if err != nil {
-            return nil, false, fmt.Errorf("Failed to retrieve the config file: %v", err)
-        }
-    }
+	var masterKey []byte
 
-    config = CreateConfig()
+	if len(password) > 0 {
 
-    err = json.Unmarshal(configFile.GetBytes(), config)
+		if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)]) == ENCRYPTION_HEADER {
+			// This is the old config format with a static salt and a fixed number of iterations
+			masterKey = GenerateKeyFromPassword(password, DEFAULT_KEY, CONFIG_DEFAULT_ITERATIONS)
+			LOG_TRACE("CONFIG_FORMAT", "Using a static salt and %d iterations for key derivation", CONFIG_DEFAULT_ITERATIONS)
+		} else if string(configFile.GetBytes()[:len(CONFIG_HEADER)]) == CONFIG_HEADER {
+			// This is the new config format with a random salt and a configurable number of iterations
+			encryptedLength := len(configFile.GetBytes()) - CONFIG_SALT_LENGTH - 4
 
-    if err != nil {
-        if bytes.Equal(configFile.GetBytes()[:9], []byte("duplicacy")) {
-            return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
-        } else {
-            return nil, false, fmt.Errorf("Failed to parse the config file: %v", err)
-        }
-    }
+			// Extract the salt and the number of iterations
+			saltStart := configFile.GetBytes()[len(CONFIG_HEADER):]
+			iterations := binary.LittleEndian.Uint32(saltStart[CONFIG_SALT_LENGTH : CONFIG_SALT_LENGTH+4])
+			LOG_TRACE("CONFIG_ITERATIONS", "Using %d iterations for key derivation", iterations)
+			masterKey = GenerateKeyFromPassword(password, saltStart[:CONFIG_SALT_LENGTH], int(iterations))
 
-    return config, false, nil
+			// Copy to a temporary buffer to replace the header and remove the salt and the number of riterations
+			var encrypted bytes.Buffer
+			encrypted.Write([]byte(ENCRYPTION_HEADER))
+			encrypted.Write(saltStart[CONFIG_SALT_LENGTH+4:])
+
+			configFile.Reset(false)
+			configFile.Write(encrypted.Bytes())
+			if len(configFile.GetBytes()) != encryptedLength {
+				LOG_ERROR("CONFIG_DOWNLOAD", "Encrypted config has %d bytes instead of expected %d bytes", len(configFile.GetBytes()), encryptedLength)
+			}
+		} else {
+			return nil, true, fmt.Errorf("The config file has an invalid header")
+		}
+
+		// Decrypt the config file.  masterKey == nil means no encryption.
+		err = configFile.Decrypt(masterKey, "")
+		if err != nil {
+			return nil, false, fmt.Errorf("Failed to retrieve the config file: %v", err)
+		}
+	}
+
+	config = CreateConfig()
+
+	err = json.Unmarshal(configFile.GetBytes(), config)
+	if err != nil {
+		return nil, false, fmt.Errorf("Failed to parse the config file: %v", err)
+	}
+
+	storage.SetNestingLevels(config)
+
+	return config, false, nil
 
 }
 
-func UploadConfig(storage Storage, config *Config, password string) (bool) {
+func UploadConfig(storage Storage, config *Config, password string, iterations int) bool {
 
-    // This is the key to encrypt the config file.
-    var masterKey []byte
+	// This is the key to encrypt the config file.
+	var masterKey []byte
+	salt := make([]byte, CONFIG_SALT_LENGTH)
 
-    if len(password) > 0 {
+	if len(password) > 0 {
 
-        if len(password) < 8 {
-            LOG_ERROR("CONFIG_PASSWORD", "The password must be at least 8 characters")
-            return false
-        }
+		if len(password) < 8 {
+			LOG_ERROR("CONFIG_PASSWORD", "The password must be at least 8 characters")
+			return false
+		}
 
-        masterKey = GenerateKeyFromPassword(password)
-    }
+		_, err := rand.Read(salt)
+		if err != nil {
+			LOG_ERROR("CONFIG_KEY", "Failed to generate random salt: %v", err)
+			return false
+		}
 
-    description, err := json.MarshalIndent(config, "", "    ")
-    if err != nil {
-        LOG_ERROR("CONFIG_MARSHAL", "Failed to marshal the config: %v", err)
-        return false
-    }
+		masterKey = GenerateKeyFromPassword(password, salt, iterations)
+	}
 
-    // Although the default key is passed to the function call the key is not actually used since there is no need to
-    // calculate the hash or id of the config file.
-    chunk := CreateChunk(CreateConfig(), true)
-    chunk.Write(description)
+	description, err := json.MarshalIndent(config, "", "    ")
+	if err != nil {
+		LOG_ERROR("CONFIG_MARSHAL", "Failed to marshal the config: %v", err)
+		return false
+	}
 
-    if len(password) > 0 {
-        // Encrypt the config file with masterKey.  If masterKey is nil then no encryption is performed.
-        err = chunk.Encrypt(masterKey, "")
+	// Although the default key is passed to the function call the key is not actually used since there is no need to
+	// calculate the hash or id of the config file.
+	chunk := CreateChunk(CreateConfig(), true)
+	chunk.Write(description)
 
-        if err != nil {
-            LOG_ERROR("CONFIG_CREATE", "Failed to create the config file: %v", err)
-            return false
-        }
-    }
+	if len(password) > 0 {
+		// Encrypt the config file with masterKey.  If masterKey is nil then no encryption is performed.
+		err = chunk.Encrypt(masterKey, "")
+		if err != nil {
+			LOG_ERROR("CONFIG_CREATE", "Failed to create the config file: %v", err)
+			return false
+		}
 
-    err = storage.UploadFile(0, "config", chunk.GetBytes())
-    if err != nil {
-        LOG_ERROR("CONFIG_INIT", "Failed to configure the storage: %v", err)
-        return false
-    }
+		// The new encrypted format for config is CONFIG_HEADER + salt + #iterations + encrypted content
+		encryptedLength := len(chunk.GetBytes()) + CONFIG_SALT_LENGTH + 4
 
-    if IsTracing() {
-        config.Print()
-    }
+		// Copy to a temporary buffer to replace the header and add the salt and the number of iterations
+		var encrypted bytes.Buffer
+		encrypted.Write([]byte(CONFIG_HEADER))
+		encrypted.Write(salt)
+		binary.Write(&encrypted, binary.LittleEndian, uint32(iterations))
+		encrypted.Write(chunk.GetBytes()[len(ENCRYPTION_HEADER):])
 
-    for _, subDir := range []string {"chunks", "snapshots"} {
-        err = storage.CreateDirectory(0, subDir)
-        if err != nil {
-            LOG_ERROR("CONFIG_MKDIR", "Failed to create storage subdirectory: %v", err)
-        }
-    }
+		chunk.Reset(false)
+		chunk.Write(encrypted.Bytes())
+		if len(chunk.GetBytes()) != encryptedLength {
+			LOG_ERROR("CONFIG_CREATE", "Encrypted config has %d bytes instead of expected %d bytes", len(chunk.GetBytes()), encryptedLength)
+		}
+	}
 
-    return true
+	err = storage.UploadFile(0, "config", chunk.GetBytes())
+	if err != nil {
+		LOG_ERROR("CONFIG_INIT", "Failed to configure the storage: %v", err)
+		return false
+	}
+
+	if IsTracing() {
+		config.Print()
+	}
+
+	for _, subDir := range []string{"chunks", "snapshots"} {
+		err = storage.CreateDirectory(0, subDir)
+		if err != nil {
+			LOG_ERROR("CONFIG_MKDIR", "Failed to create storage subdirectory: %v", err)
+		}
+	}
+
+	return true
 }
 
 // ConfigStorage makes the general storage space available for storing duplicacy format snapshots.  In essence,
 // it simply creates a file named 'config' that stores various parameters as well as a set of keys if encryption
 // is enabled.
-func ConfigStorage(storage Storage, compressionLevel int, averageChunkSize int, maximumChunkSize int,
-                   minimumChunkSize int, password string, copyFrom *Config) bool {
+func ConfigStorage(storage Storage, iterations int, compressionLevel int, averageChunkSize int, maximumChunkSize int,
+	minimumChunkSize int, password string, copyFrom *Config, bitCopy bool) bool {
 
-    exist, _, _, err := storage.GetFileInfo(0, "config")
-    if err != nil {
-        LOG_ERROR("CONFIG_INIT", "Failed to check if there is an existing config file: %v", err)
-        return false
-    }
+	exist, _, _, err := storage.GetFileInfo(0, "config")
+	if err != nil {
+		LOG_ERROR("CONFIG_INIT", "Failed to check if there is an existing config file: %v", err)
+		return false
+	}
 
-    if exist {
-        LOG_INFO("CONFIG_EXIST", "The storage has already been configured")
-        return false
-    }
+	if exist {
+		LOG_INFO("CONFIG_EXIST", "The storage has already been configured")
+		return false
+	}
 
+	config := CreateConfigFromParameters(compressionLevel, averageChunkSize, maximumChunkSize, minimumChunkSize, len(password) > 0,
+		copyFrom, bitCopy)
+	if config == nil {
+		return false
+	}
 
-    config := CreateConfigFromParameters(compressionLevel, averageChunkSize, maximumChunkSize, minimumChunkSize, len(password) > 0,
-                           copyFrom)
-    if config == nil {
-        return false
-    }
-
-    return UploadConfig(storage, config, password)
+	return UploadConfig(storage, config, password, iterations)
 }

src/duplicacy_dropboxstorage.go

@@ -5,292 +5,236 @@
 package duplicacy
 
 import (
-    "fmt"
-    "path"
-    "strings"
-    "github.com/gilbertchen/go-dropbox"
+	"fmt"
+	"strings"
+
+	"github.com/gilbertchen/go-dropbox"
 )
 
 type DropboxStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    clients []*dropbox.Files
-    storageDir string
+	clients        []*dropbox.Files
+	minimumNesting int // The minimum level of directories to dive into before searching for the chunk file.
+	storageDir     string
 }
 
 // CreateDropboxStorage creates a dropbox storage object.
-func CreateDropboxStorage(accessToken string, storageDir string, threads int) (storage *DropboxStorage, err error) {
+func CreateDropboxStorage(accessToken string, storageDir string, minimumNesting int, threads int) (storage *DropboxStorage, err error) {
 
-    var clients []*dropbox.Files
-    for i := 0; i < threads; i++ {
-        client := dropbox.NewFiles(dropbox.NewConfig(accessToken))
-        clients = append(clients, client)
-    }
+	var clients []*dropbox.Files
+	for i := 0; i < threads; i++ {
+		client := dropbox.NewFiles(dropbox.NewConfig(accessToken))
+		clients = append(clients, client)
+	}
 
-    if storageDir == "" || storageDir[0] != '/' {
-        storageDir = "/" + storageDir
-    }
+	if storageDir == "" || storageDir[0] != '/' {
+		storageDir = "/" + storageDir
+	}
 
-    if len(storageDir) > 1 && storageDir[len(storageDir) - 1] == '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
-    }
+	if len(storageDir) > 1 && storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
+	}
 
-    storage = &DropboxStorage {
-        clients: clients,
-        storageDir: storageDir,
-    }
+	storage = &DropboxStorage{
+		clients:        clients,
+		storageDir:     storageDir,
+		minimumNesting: minimumNesting,
+	}
 
-    err = storage.CreateDirectory(0, "")
-    if err != nil {
-        return nil, fmt.Errorf("Can't create storage directory: %v", err)
-    }
+	err = storage.CreateDirectory(0, "")
+	if err != nil {
+		return nil, fmt.Errorf("Can't create storage directory: %v", err)
+	}
 
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{1}, 1)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *DropboxStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
 
-    if dir != "" && dir[0] != '/' {
-        dir = "/" + dir
-    }
+	if dir != "" && dir[0] != '/' {
+		dir = "/" + dir
+	}
 
-    if len(dir) > 1 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	if len(dir) > 1 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    input := &dropbox.ListFolderInput {
-        Path : storage.storageDir + dir,
-        Recursive : false,
-        IncludeMediaInfo: false,
-        IncludeDeleted: false,
-    }
+	input := &dropbox.ListFolderInput{
+		Path:             storage.storageDir + dir,
+		Recursive:        false,
+		IncludeMediaInfo: false,
+		IncludeDeleted:   false,
+	}
 
-    output, err := storage.clients[threadIndex].ListFolder(input)
+	output, err := storage.clients[threadIndex].ListFolder(input)
 
-    for {
+	for {
 
-        if err != nil {
-            return nil, nil, err
-        }
+		if err != nil {
+			return nil, nil, err
+		}
 
-        for _, entry := range output.Entries {
-            name := entry.Name
-            if entry.Tag == "folder" {
-                name += "/"
-            }
-            files = append(files, name)
-            sizes = append(sizes, int64(entry.Size))
-        }
+		for _, entry := range output.Entries {
+			name := entry.Name
+			if entry.Tag == "folder" {
+				name += "/"
+			}
+			files = append(files, name)
+			sizes = append(sizes, int64(entry.Size))
+		}
 
-        if output.HasMore {
-            output, err = storage.clients[threadIndex].ListFolderContinue(
-                              &dropbox.ListFolderContinueInput { Cursor: output.Cursor, })
+		if output.HasMore {
+			output, err = storage.clients[threadIndex].ListFolderContinue(
+				&dropbox.ListFolderContinueInput{Cursor: output.Cursor})
 
-        } else {
-            break
-        }
+		} else {
+			break
+		}
 
-    }
+	}
 
-    return files, sizes, nil
+	return files, sizes, nil
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *DropboxStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    if filePath != "" && filePath[0] != '/' {
-        filePath = "/" + filePath
-    }
+	if filePath != "" && filePath[0] != '/' {
+		filePath = "/" + filePath
+	}
 
-    input := &dropbox.DeleteInput {
-        Path: storage.storageDir + filePath,
-    }
-    _, err = storage.clients[threadIndex].Delete(input)
-    if err != nil {
-        if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path_lookup/not_found/") {
-            return nil
-        }
-    }
+	input := &dropbox.DeleteInput{
+		Path: storage.storageDir + filePath,
+	}
+	_, err = storage.clients[threadIndex].Delete(input)
+	if err != nil {
+		if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path_lookup/not_found/") {
+			return nil
+		}
+	}
 
-    return err
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *DropboxStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    if from != "" && from[0] != '/' {
-        from = "/" + from
-    }
-    if to != "" && to[0] != '/' {
-        to = "/" + to
-    }
-    input := &dropbox.MoveInput {
-        FromPath: storage.storageDir + from,
-        ToPath: storage.storageDir + to,
-    }
-    _, err = storage.clients[threadIndex].Move(input)
-    return err
+	if from != "" && from[0] != '/' {
+		from = "/" + from
+	}
+	if to != "" && to[0] != '/' {
+		to = "/" + to
+	}
+	input := &dropbox.MoveInput{
+		FromPath: storage.storageDir + from,
+		ToPath:   storage.storageDir + to,
+	}
+	_, err = storage.clients[threadIndex].Move(input)
+	return err
 }
 
 // CreateDirectory creates a new directory.
 func (storage *DropboxStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    if dir != "" && dir[0] != '/' {
-        dir = "/" + dir
-    }
+	if dir != "" && dir[0] != '/' {
+		dir = "/" + dir
+	}
 
-    if len(dir) > 1 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	if len(dir) > 1 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    input := &dropbox.CreateFolderInput {
-        Path : storage.storageDir + dir,
-    }
+	input := &dropbox.CreateFolderInput{
+		Path: storage.storageDir + dir,
+	}
 
-    _, err = storage.clients[threadIndex].CreateFolder(input)
-    if err != nil {
-        if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path/conflict/") {
-            return nil
-        }
-    }
-    return err
+	_, err = storage.clients[threadIndex].CreateFolder(input)
+	if err != nil {
+		if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path/conflict/") {
+			return nil
+		}
+	}
+	return err
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *DropboxStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    if filePath != "" && filePath[0] != '/' {
-        filePath = "/" + filePath
-    }
+	if filePath != "" && filePath[0] != '/' {
+		filePath = "/" + filePath
+	}
 
-    input := &dropbox.GetMetadataInput {
-        Path: storage.storageDir + filePath,
-        IncludeMediaInfo: false,
-    }
+	input := &dropbox.GetMetadataInput{
+		Path:             storage.storageDir + filePath,
+		IncludeMediaInfo: false,
+	}
 
-    output, err := storage.clients[threadIndex].GetMetadata(input)
-    if err != nil {
-        if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path/not_found/") {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
-
-    return true, output.Tag == "folder", int64(output.Size), nil
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *DropboxStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := "/chunks"
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    // The minimum level of directories to dive into before searching for the chunk file.
-    minimumLevel := 1
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            var size int64
-            exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-            if err != nil {
-                return "", false, 0, err
-            }
-            if exist {
-                return filePath, exist, size, nil
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        exist, _, _, err = storage.GetFileInfo(threadIndex, subDir)
-        if err != nil {
-            return "", false, 0, err
-        }
-
-        if exist {
-            dir = subDir
-            continue
-        }
-
-        if level < minimumLevel {
-            // Create the subdirectory if it doesn't exist.
-            err = storage.CreateDirectory(threadIndex, subDir)
-            if err != nil {
-                return "", false, 0, err
-            }
-
-            dir = subDir
-            continue
-        }
-
-        // Teh chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[1:] + suffix, false, 0, nil
-
-    }
-
-    LOG_FATAL("CHUNK_FIND", "Chunk %s is still not found after having searched a maximum level of directories",
-              chunkID)
-    return "", false, 0, nil
+	output, err := storage.clients[threadIndex].GetMetadata(input)
+	if err != nil {
+		if e, ok := err.(*dropbox.Error); ok && strings.HasPrefix(e.Summary, "path/not_found/") {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
+	return true, output.Tag == "folder", int64(output.Size), nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *DropboxStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-    if filePath != "" && filePath[0] != '/' {
-        filePath = "/" + filePath
-    }
+	if filePath != "" && filePath[0] != '/' {
+		filePath = "/" + filePath
+	}
 
-    input := &dropbox.DownloadInput {
-        Path: storage.storageDir + filePath,
-    }
+	input := &dropbox.DownloadInput{
+		Path: storage.storageDir + filePath,
+	}
 
-    output, err := storage.clients[threadIndex].Download(input)
-    if err != nil {
-        return err
-    }
+	output, err := storage.clients[threadIndex].Download(input)
+	if err != nil {
+		return err
+	}
 
-    defer output.Body.Close()
+	defer output.Body.Close()
 
-    _, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit / len(storage.clients))
-    return err
+	_, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit/len(storage.clients))
+	return err
 
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *DropboxStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    if filePath != "" && filePath[0] != '/' {
-        filePath = "/" + filePath
-    }
+	if filePath != "" && filePath[0] != '/' {
+		filePath = "/" + filePath
+	}
 
-    input := &dropbox.UploadInput {
-        Path: storage.storageDir + filePath,
-        Mode: dropbox.WriteModeOverwrite,
-        AutoRename: false,
-        Mute: true,
-        Reader: CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.clients)),
-    }
+	input := &dropbox.UploadInput{
+		Path:       storage.storageDir + filePath,
+		Mode:       dropbox.WriteModeOverwrite,
+		AutoRename: false,
+		Mute:       true,
+		Reader:     CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.clients)),
+	}
 
-    _, err = storage.clients[threadIndex].Upload(input)
-    return err
+	_, err = storage.clients[threadIndex].Upload(input)
+	return err
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *DropboxStorage) IsCacheNeeded() (bool) { return true }
+func (storage *DropboxStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *DropboxStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *DropboxStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *DropboxStorage) IsStrongConsistent() (bool) { return false }
+func (storage *DropboxStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *DropboxStorage) IsFastListing() (bool) { return false }
+func (storage *DropboxStorage) IsFastListing() bool { return false }
 
 // Enable the test mode.
 func (storage *DropboxStorage) EnableTestMode() {}

src/duplicacy_entry.go

@@ -4,563 +4,588 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "path/filepath"
-    "io/ioutil"
-    "sort"
-    "regexp"
-    "strconv"
-    "time"
-    "encoding/json"
-    "encoding/base64"
-    "strings"
-
+	"encoding/base64"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path/filepath"
+	"regexp"
+	"runtime"
+	"sort"
+	"strconv"
+	"strings"
+	"time"
 )
 
-
 // This is the hidden directory in the repository for storing various files.
 var DUPLICACY_DIRECTORY = ".duplicacy"
 var DUPLICACY_FILE = ".duplicacy"
 
+// Mask for file permission bits
+var fileModeMask = os.ModePerm | os.ModeSetuid | os.ModeSetgid | os.ModeSticky
+
 // Regex for matching 'StartChunk:StartOffset:EndChunk:EndOffset'
 var contentRegex = regexp.MustCompile(`^([0-9]+):([0-9]+):([0-9]+):([0-9]+)`)
 
 // Entry encapsulates information about a file or directory.
 type Entry struct {
-    Path string
-    Size int64
-    Time int64
-    Mode uint32
-    Link string
-    Hash string
+	Path string
+	Size int64
+	Time int64
+	Mode uint32
+	Link string
+	Hash string
 
-    UID  int
-    GID  int
+	UID int
+	GID int
 
-    StartChunk int
-    StartOffset int
-    EndChunk int
-    EndOffset int
+	StartChunk  int
+	StartOffset int
+	EndChunk    int
+	EndOffset   int
 
-    Attributes map[string][]byte
+	Attributes map[string][]byte
 }
 
 // CreateEntry creates an entry from file properties.
 func CreateEntry(path string, size int64, time int64, mode uint32) *Entry {
 
-    if len(path) > 0 && path[len(path) - 1] != '/' && (mode & uint32(os.ModeDir)) != 0 {
-        path += "/"
-    }
+	if len(path) > 0 && path[len(path)-1] != '/' && (mode&uint32(os.ModeDir)) != 0 {
+		path += "/"
+	}
 
-    return &Entry {
-        Path : path,
-        Size : size,
-        Time : time,
-        Mode : mode,
+	return &Entry{
+		Path: path,
+		Size: size,
+		Time: time,
+		Mode: mode,
 
-        UID  : -1,
-        GID  : -1,
-    }
+		UID: -1,
+		GID: -1,
+	}
 
 }
 
 // CreateEntryFromFileInfo creates an entry from a 'FileInfo' object.
 func CreateEntryFromFileInfo(fileInfo os.FileInfo, directory string) *Entry {
-    path := directory + fileInfo.Name()
+	path := directory + fileInfo.Name()
 
-    mode := fileInfo.Mode()
+	mode := fileInfo.Mode()
 
-    if mode & os.ModeDir != 0 && mode & os.ModeSymlink != 0 {
-        mode ^= os.ModeDir
-    }
+	if mode&os.ModeDir != 0 && mode&os.ModeSymlink != 0 {
+		mode ^= os.ModeDir
+	}
 
-    if path[len(path) - 1] != '/' && mode & os.ModeDir != 0 {
-        path += "/"
-    }
+	if path[len(path)-1] != '/' && mode&os.ModeDir != 0 {
+		path += "/"
+	}
 
-    entry := &Entry {
-        Path: path,
-        Size: fileInfo.Size(),
-        Time: fileInfo.ModTime().Unix(),
-        Mode: uint32(mode),
-    }
+	entry := &Entry{
+		Path: path,
+		Size: fileInfo.Size(),
+		Time: fileInfo.ModTime().Unix(),
+		Mode: uint32(mode),
+	}
 
-    GetOwner(entry, &fileInfo)
+	GetOwner(entry, &fileInfo)
 
-    return entry
+	return entry
 }
 
 // CreateEntryFromJSON creates an entry from a json description.
 func (entry *Entry) UnmarshalJSON(description []byte) (err error) {
 
-    var object map[string]interface {}
+	var object map[string]interface{}
 
-    err = json.Unmarshal(description, &object)
-    if err != nil {
-        return err
-    }
+	err = json.Unmarshal(description, &object)
+	if err != nil {
+		return err
+	}
 
-    var value interface {}
-    var ok bool
+	var value interface{}
+	var ok bool
 
-    if value, ok = object["name"]; ok {
-        pathInBase64, ok := value.(string)
-        if !ok {
-            return fmt.Errorf("Name is not a string for a file in the snapshot")
-        }
-        path, err := base64.StdEncoding.DecodeString(pathInBase64)
-        if err != nil {
-            return fmt.Errorf("Invalid name '%s' in the snapshot", pathInBase64)
-        }
-        entry.Path = string(path)
-    } else if value, ok = object["path"]; !ok {
-        return fmt.Errorf("Path is not specified for a file in the snapshot")
-    } else if entry.Path, ok = value.(string); !ok {
-        return fmt.Errorf("Path is not a string for a file in the snapshot")
-    }
+	if value, ok = object["name"]; ok {
+		pathInBase64, ok := value.(string)
+		if !ok {
+			return fmt.Errorf("Name is not a string for a file in the snapshot")
+		}
+		path, err := base64.StdEncoding.DecodeString(pathInBase64)
+		if err != nil {
+			return fmt.Errorf("Invalid name '%s' in the snapshot", pathInBase64)
+		}
+		entry.Path = string(path)
+	} else if value, ok = object["path"]; !ok {
+		return fmt.Errorf("Path is not specified for a file in the snapshot")
+	} else if entry.Path, ok = value.(string); !ok {
+		return fmt.Errorf("Path is not a string for a file in the snapshot")
+	}
 
-    if value, ok = object["size"]; !ok {
-        return fmt.Errorf("Size is not specified for file '%s' in the snapshot", entry.Path)
-    } else if _, ok = value.(float64); !ok {
-        return fmt.Errorf("Size is not a valid integer for file '%s' in the snapshot", entry.Path)
-    }
-    entry.Size = int64(value.(float64))
+	if value, ok = object["size"]; !ok {
+		return fmt.Errorf("Size is not specified for file '%s' in the snapshot", entry.Path)
+	} else if _, ok = value.(float64); !ok {
+		return fmt.Errorf("Size is not a valid integer for file '%s' in the snapshot", entry.Path)
+	}
+	entry.Size = int64(value.(float64))
 
-    if value, ok = object["time"]; !ok {
-        return fmt.Errorf("Time is not specified for file '%s' in the snapshot", entry.Path)
-    } else if _, ok = value.(float64); !ok {
-        return fmt.Errorf("Time is not a valid integer for file '%s' in the snapshot", entry.Path)
-    }
-    entry.Time = int64(value.(float64))
+	if value, ok = object["time"]; !ok {
+		return fmt.Errorf("Time is not specified for file '%s' in the snapshot", entry.Path)
+	} else if _, ok = value.(float64); !ok {
+		return fmt.Errorf("Time is not a valid integer for file '%s' in the snapshot", entry.Path)
+	}
+	entry.Time = int64(value.(float64))
 
-    if value, ok = object["mode"]; !ok {
-        return fmt.Errorf("float64 is not specified for file '%s' in the snapshot", entry.Path)
-    } else if _, ok = value.(float64); !ok {
-        return fmt.Errorf("Mode is not a valid integer for file '%s' in the snapshot", entry.Path)
-    }
-    entry.Mode = uint32(value.(float64))
+	if value, ok = object["mode"]; !ok {
+		return fmt.Errorf("float64 is not specified for file '%s' in the snapshot", entry.Path)
+	} else if _, ok = value.(float64); !ok {
+		return fmt.Errorf("Mode is not a valid integer for file '%s' in the snapshot", entry.Path)
+	}
+	entry.Mode = uint32(value.(float64))
 
-    if value, ok = object["hash"]; !ok {
-        return fmt.Errorf("Hash is not specified for file '%s' in the snapshot", entry.Path)
-    } else if entry.Hash, ok = value.(string); !ok {
-        return fmt.Errorf("Hash is not a string for file '%s' in the snapshot", entry.Path)
-    }
+	if value, ok = object["hash"]; !ok {
+		return fmt.Errorf("Hash is not specified for file '%s' in the snapshot", entry.Path)
+	} else if entry.Hash, ok = value.(string); !ok {
+		return fmt.Errorf("Hash is not a string for file '%s' in the snapshot", entry.Path)
+	}
 
-    if value, ok = object["link"]; ok {
-        var link string
-        if link, ok = value.(string); !ok {
-            return fmt.Errorf("Symlink is not a valid string for file '%s' in the snapshot", entry.Path)
-        }
-        entry.Link = link
-    }
+	if value, ok = object["link"]; ok {
+		var link string
+		if link, ok = value.(string); !ok {
+			return fmt.Errorf("Symlink is not a valid string for file '%s' in the snapshot", entry.Path)
+		}
+		entry.Link = link
+	}
 
-    entry.UID = -1
-    if value, ok = object["uid"]; ok {
-        if _, ok = value.(float64); ok {
-            entry.UID = int(value.(float64))
-        }
-    }
+	entry.UID = -1
+	if value, ok = object["uid"]; ok {
+		if _, ok = value.(float64); ok {
+			entry.UID = int(value.(float64))
+		}
+	}
 
-    entry.GID = -1
-    if value, ok = object["gid"]; ok {
-        if _, ok = value.(float64); ok {
-            entry.GID = int(value.(float64))
-        }
-    }
+	entry.GID = -1
+	if value, ok = object["gid"]; ok {
+		if _, ok = value.(float64); ok {
+			entry.GID = int(value.(float64))
+		}
+	}
 
-    if value, ok = object["attributes"]; ok {
-        if attributes, ok := value.(map[string]interface {}); !ok {
-            return fmt.Errorf("Attributes are invalid for file '%s' in the snapshot", entry.Path)
-        } else {
-            entry.Attributes = make(map[string][]byte)
-            for name, object := range attributes {
-                if object == nil {
-                    entry.Attributes[name] = []byte("")
-                } else if attributeInBase64, ok := object.(string); !ok {
-                    return fmt.Errorf("Attribute '%s' is invalid for file '%s' in the snapshot", name, entry.Path)
-                } else if attribute, err := base64.StdEncoding.DecodeString(attributeInBase64); err != nil {
-                    return fmt.Errorf("Failed to decode attribute '%s' for file '%s' in the snapshot: %v",
-                                      name, entry.Path, err)
-                } else {
-                    entry.Attributes[name] = attribute
-                }
-            }
-        }
-    }
+	if value, ok = object["attributes"]; ok {
+		if attributes, ok := value.(map[string]interface{}); !ok {
+			return fmt.Errorf("Attributes are invalid for file '%s' in the snapshot", entry.Path)
+		} else {
+			entry.Attributes = make(map[string][]byte)
+			for name, object := range attributes {
+				if object == nil {
+					entry.Attributes[name] = []byte("")
+				} else if attributeInBase64, ok := object.(string); !ok {
+					return fmt.Errorf("Attribute '%s' is invalid for file '%s' in the snapshot", name, entry.Path)
+				} else if attribute, err := base64.StdEncoding.DecodeString(attributeInBase64); err != nil {
+					return fmt.Errorf("Failed to decode attribute '%s' for file '%s' in the snapshot: %v",
+						name, entry.Path, err)
+				} else {
+					entry.Attributes[name] = attribute
+				}
+			}
+		}
+	}
 
-    if entry.IsFile() && entry.Size > 0 {
-        if value, ok = object["content"]; !ok {
-            return fmt.Errorf("Content is not specified for file '%s' in the snapshot", entry.Path)
-        }
+	if entry.IsFile() && entry.Size > 0 {
+		if value, ok = object["content"]; !ok {
+			return fmt.Errorf("Content is not specified for file '%s' in the snapshot", entry.Path)
+		}
 
-        if content, ok := value.(string); !ok {
-            return fmt.Errorf("Content is invalid for file '%s' in the snapshot", entry.Path)
-        } else {
+		if content, ok := value.(string); !ok {
+			return fmt.Errorf("Content is invalid for file '%s' in the snapshot", entry.Path)
+		} else {
 
-            matched := contentRegex.FindStringSubmatch(content)
-            if matched == nil {
-                return fmt.Errorf("Content is specified in a wrong format for file '%s' in the snapshot", entry.Path)
-            }
+			matched := contentRegex.FindStringSubmatch(content)
+			if matched == nil {
+				return fmt.Errorf("Content is specified in a wrong format for file '%s' in the snapshot", entry.Path)
+			}
 
-            entry.StartChunk, _ = strconv.Atoi(matched[1])
-            entry.StartOffset, _ = strconv.Atoi(matched[2])
-            entry.EndChunk, _ = strconv.Atoi(matched[3])
-            entry.EndOffset, _ = strconv.Atoi(matched[4])
-        }
-    }
+			entry.StartChunk, _ = strconv.Atoi(matched[1])
+			entry.StartOffset, _ = strconv.Atoi(matched[2])
+			entry.EndChunk, _ = strconv.Atoi(matched[3])
+			entry.EndOffset, _ = strconv.Atoi(matched[4])
+		}
+	}
 
-    return nil
+	return nil
 
 }
 
 func (entry *Entry) convertToObject(encodeName bool) map[string]interface{} {
 
-    object := make(map[string]interface{})
+	object := make(map[string]interface{})
 
-    if encodeName {
-        object["name"] = base64.StdEncoding.EncodeToString([]byte(entry.Path))
-    } else {
-        object["path"] = entry.Path
-    }
-    object["size"] = entry.Size
-    object["time"] = entry.Time
-    object["mode"] = entry.Mode
-    object["hash"] = entry.Hash
+	if encodeName {
+		object["name"] = base64.StdEncoding.EncodeToString([]byte(entry.Path))
+	} else {
+		object["path"] = entry.Path
+	}
+	object["size"] = entry.Size
+	object["time"] = entry.Time
+	object["mode"] = entry.Mode
+	object["hash"] = entry.Hash
 
-    if entry.IsLink() {
-        object["link"] = entry.Link
-    }
+	if entry.IsLink() {
+		object["link"] = entry.Link
+	}
 
-    if entry.IsFile() && entry.Size > 0 {
-        object["content"] = fmt.Sprintf("%d:%d:%d:%d",
-                                        entry.StartChunk, entry.StartOffset, entry.EndChunk, entry.EndOffset)
-    }
+	if entry.IsFile() && entry.Size > 0 {
+		object["content"] = fmt.Sprintf("%d:%d:%d:%d",
+			entry.StartChunk, entry.StartOffset, entry.EndChunk, entry.EndOffset)
+	}
 
-    if entry.UID != -1 && entry.GID != -1 {
-        object["uid"] = entry.UID
-        object["gid"] = entry.GID
-    }
+	if entry.UID != -1 && entry.GID != -1 {
+		object["uid"] = entry.UID
+		object["gid"] = entry.GID
+	}
 
-    if len(entry.Attributes) > 0 {
-        object["attributes"] = entry.Attributes
-    }
+	if len(entry.Attributes) > 0 {
+		object["attributes"] = entry.Attributes
+	}
 
-    return object
+	return object
 }
 
 // MarshalJSON returns the json description of an entry.
-func (entry *Entry) MarshalJSON() ([] byte, error) {
+func (entry *Entry) MarshalJSON() ([]byte, error) {
 
-    object := entry.convertToObject(true)
-    description, err := json.Marshal(object)
-    return description, err
+	object := entry.convertToObject(true)
+	description, err := json.Marshal(object)
+	return description, err
 }
 
 func (entry *Entry) IsFile() bool {
-    return entry.Mode & uint32(os.ModeType) == 0
+	return entry.Mode&uint32(os.ModeType) == 0
 }
 
 func (entry *Entry) IsDir() bool {
-    return entry.Mode & uint32(os.ModeDir) != 0
+	return entry.Mode&uint32(os.ModeDir) != 0
 }
 
 func (entry *Entry) IsLink() bool {
-    return entry.Mode & uint32(os.ModeSymlink) != 0
+	return entry.Mode&uint32(os.ModeSymlink) != 0
 }
 
 func (entry *Entry) GetPermissions() os.FileMode {
-    return os.FileMode(entry.Mode) & os.ModePerm
+	return os.FileMode(entry.Mode) & fileModeMask
 }
 
 func (entry *Entry) IsSameAs(other *Entry) bool {
-    return entry.Size == other.Size && entry.Time <= other.Time + 1 && entry.Time >= other.Time - 1
+	return entry.Size == other.Size && entry.Time <= other.Time+1 && entry.Time >= other.Time-1
 }
 
 func (entry *Entry) IsSameAsFileInfo(other os.FileInfo) bool {
-    time := other.ModTime().Unix()
-    return entry.Size == other.Size() && entry.Time <= time + 1 && entry.Time >= time - 1
+	time := other.ModTime().Unix()
+	return entry.Size == other.Size() && entry.Time <= time+1 && entry.Time >= time-1
 }
 
 func (entry *Entry) String(maxSizeDigits int) string {
-    modifiedTime := time.Unix(entry.Time, 0).Format("2006-01-02 15:04:05")
-    return fmt.Sprintf("%*d %s %64s %s", maxSizeDigits, entry.Size, modifiedTime, entry.Hash, entry.Path)
+	modifiedTime := time.Unix(entry.Time, 0).Format("2006-01-02 15:04:05")
+	return fmt.Sprintf("%*d %s %64s %s", maxSizeDigits, entry.Size, modifiedTime, entry.Hash, entry.Path)
 }
 
-func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo) bool {
+func (entry *Entry) RestoreMetadata(fullPath string, fileInfo *os.FileInfo, setOwner bool) bool {
 
-    if fileInfo == nil {
-        stat, err := os.Stat(fullPath)
-        fileInfo = &stat
-        if err != nil {
-            LOG_ERROR("RESTORE_STAT", "Failed to retrieve the file info: %v", err)
-            return false
-        }
-    }
+	if fileInfo == nil {
+		stat, err := os.Lstat(fullPath)
+		fileInfo = &stat
+		if err != nil {
+			LOG_ERROR("RESTORE_STAT", "Failed to retrieve the file info: %v", err)
+			return false
+		}
+	}
 
-    if (*fileInfo).Mode() & os.ModePerm != entry.GetPermissions() {
-        err := os.Chmod(fullPath, entry.GetPermissions())
-        if err != nil {
-            LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions: %v", err)
-            return false
-        }
-    }
+	// Note that chown can remove setuid/setgid bits so should be called before chmod
+	if setOwner {
+		if !SetOwner(fullPath, entry, fileInfo) {
+			return false
+		}
+	}
 
-    if (*fileInfo).ModTime().Unix() != entry.Time {
-        modifiedTime := time.Unix(entry.Time, 0)
-        err := os.Chtimes(fullPath, modifiedTime, modifiedTime)
-        if err != nil {
-            LOG_ERROR("RESTORE_CHTIME", "Failed to set the modification time: %v", err)
-            return false
-        }
-    }
+	// Only set the permission if the file is not a symlink
+	if !entry.IsLink() && (*fileInfo).Mode()&fileModeMask != entry.GetPermissions() {
+		err := os.Chmod(fullPath, entry.GetPermissions())
+		if err != nil {
+			LOG_ERROR("RESTORE_CHMOD", "Failed to set the file permissions: %v", err)
+			return false
+		}
+	}
 
-    if len(entry.Attributes) > 0 {
-        entry.SetAttributesToFile(fullPath)
-    }
+	// Only set the time if the file is not a symlink
+	if !entry.IsLink() && (*fileInfo).ModTime().Unix() != entry.Time {
+		modifiedTime := time.Unix(entry.Time, 0)
+		err := os.Chtimes(fullPath, modifiedTime, modifiedTime)
+		if err != nil {
+			LOG_ERROR("RESTORE_CHTIME", "Failed to set the modification time: %v", err)
+			return false
+		}
+	}
 
-    return SetOwner(fullPath, entry, fileInfo)
+	if len(entry.Attributes) > 0 {
+		entry.SetAttributesToFile(fullPath)
+	}
+
+	return true
 }
 
-
 // Return -1 if 'left' should appear before 'right', 1 if opposite, and 0 if they are the same.
 // Files are always arranged before subdirectories under the same parent directory.
 func (left *Entry) Compare(right *Entry) int {
 
-    path1 := left.Path
-    path2 := right.Path
+	path1 := left.Path
+	path2 := right.Path
 
-    p := 0
-    for ; p < len(path1) && p < len(path2); p++ {
-        if path1[p] != path2[p] {
-            break
-        }
-    }
+	p := 0
+	for ; p < len(path1) && p < len(path2); p++ {
+		if path1[p] != path2[p] {
+			break
+		}
+	}
 
-    // c1, c2 is the first byte that differs
-    var c1, c2 byte
-    if p < len(path1) {
-        c1 = path1[p]
-    }
-    if p < len(path2) {
-        c2 = path2[p]
-    }
+	// c1, c2 is the first byte that differs
+	var c1, c2 byte
+	if p < len(path1) {
+		c1 = path1[p]
+	}
+	if p < len(path2) {
+		c2 = path2[p]
+	}
 
-    // c3, c4 indicates how the current component ends
-    // c3 == '/':  the current component is a directory
-    // c3 != '/':  the current component is the last one
-    c3 := c1
-    for i := p; c3 != '/' && i < len(path1); i++ {
-        c3 = path1[i]
-    }
+	// c3, c4 indicates how the current component ends
+	// c3 == '/':  the current component is a directory
+	// c3 != '/':  the current component is the last one
+	c3 := c1
+	for i := p; c3 != '/' && i < len(path1); i++ {
+		c3 = path1[i]
+	}
 
-    c4 := c2
-    for i := p; c4 != '/' && i < len(path2); i++ {
-        c4 = path2[i]
-    }
+	c4 := c2
+	for i := p; c4 != '/' && i < len(path2); i++ {
+		c4 = path2[i]
+	}
 
-    if c3 == '/' {
-        if c4 == '/' {
-            // We are comparing two directory components
-            if c1 == '/' {
-                // left is shorter
-                // Note that c2 maybe smaller than c1 but c1 is '/' which is counted
-                // as 0
-                return -1
-            } else if c2 == '/' {
-                // right is shorter
-                return 1
-            } else {
-                return int(c1) - int(c2)
-            }
-        } else {
-            return 1
-        }
-    } else {
-        // We're at the last component of left and left is a file
-        if c4 == '/' {
-            // the current component of right is a directory
-            return -1
-        } else {
-            return int(c1) - int(c2)
-        }
-    }
+	if c3 == '/' {
+		if c4 == '/' {
+			// We are comparing two directory components
+			if c1 == '/' {
+				// left is shorter
+				// Note that c2 maybe smaller than c1 but c1 is '/' which is counted
+				// as 0
+				return -1
+			} else if c2 == '/' {
+				// right is shorter
+				return 1
+			} else {
+				return int(c1) - int(c2)
+			}
+		} else {
+			return 1
+		}
+	} else {
+		// We're at the last component of left and left is a file
+		if c4 == '/' {
+			// the current component of right is a directory
+			return -1
+		} else {
+			return int(c1) - int(c2)
+		}
+	}
 }
 
 // This is used to sort entries by their names.
 type ByName []*Entry
 
-func (entries ByName) Len() int { return len(entries) }
+func (entries ByName) Len() int      { return len(entries) }
 func (entries ByName) Swap(i, j int) { entries[i], entries[j] = entries[j], entries[i] }
 func (entries ByName) Less(i, j int) bool {
-    return entries[i].Compare(entries[j]) < 0
+	return entries[i].Compare(entries[j]) < 0
 }
 
 // This is used to sort entries by their starting chunks (and starting offsets if the starting chunks are the same).
 type ByChunk []*Entry
 
-func (entries ByChunk) Len() int { return len(entries) }
+func (entries ByChunk) Len() int      { return len(entries) }
 func (entries ByChunk) Swap(i, j int) { entries[i], entries[j] = entries[j], entries[i] }
 func (entries ByChunk) Less(i, j int) bool {
-    return entries[i].StartChunk < entries[j].StartChunk ||
-           (entries[i].StartChunk == entries[j].StartChunk && entries[i].StartOffset < entries[j].StartOffset)
+	return entries[i].StartChunk < entries[j].StartChunk ||
+		(entries[i].StartChunk == entries[j].StartChunk && entries[i].StartOffset < entries[j].StartOffset)
 }
 
 // This is used to sort FileInfo objects.
 type FileInfoCompare []os.FileInfo
 
-func (files FileInfoCompare) Len() int { return len(files) }
+func (files FileInfoCompare) Len() int      { return len(files) }
 func (files FileInfoCompare) Swap(i, j int) { files[i], files[j] = files[j], files[i] }
 func (files FileInfoCompare) Less(i, j int) bool {
 
-    left := files[i]
-    right := files[j]
+	left := files[i]
+	right := files[j]
 
-    if left.IsDir() && left.Mode() & os.ModeSymlink == 0 {
-        if right.IsDir() && right.Mode() & os.ModeSymlink == 0 {
-            return left.Name() < right.Name()
-        } else {
-            return false
-        }
-    } else {
-        if right.IsDir() && right.Mode() & os.ModeSymlink == 0 {
-            return true
-        } else {
-            return left.Name() < right.Name()
-        }
-    }
+	if left.IsDir() && left.Mode()&os.ModeSymlink == 0 {
+		if right.IsDir() && right.Mode()&os.ModeSymlink == 0 {
+			return left.Name() < right.Name()
+		} else {
+			return false
+		}
+	} else {
+		if right.IsDir() && right.Mode()&os.ModeSymlink == 0 {
+			return true
+		} else {
+			return left.Name() < right.Name()
+		}
+	}
 }
 
 // ListEntries returns a list of entries representing file and subdirectories under the directory 'path'.  Entry paths
 // are normalized as relative to 'top'.  'patterns' are used to exclude or include certain files.
-func ListEntries(top string, path string, fileList *[]*Entry, patterns [] string, discardAttributes bool) (directoryList []*Entry,
-                                                                                   skippedFiles [] string, err error) {
+func ListEntries(top string, path string, fileList *[]*Entry, patterns []string, nobackupFile string, discardAttributes bool) (directoryList []*Entry,
+	skippedFiles []string, err error) {
 
-    LOG_DEBUG("LIST_ENTRIES", "Listing %s", path)
+	LOG_DEBUG("LIST_ENTRIES", "Listing %s", path)
 
-    fullPath := joinPath(top, path)
+	fullPath := joinPath(top, path)
 
-    files := make([]os.FileInfo, 0, 1024)
+	files := make([]os.FileInfo, 0, 1024)
 
-    files, err = ioutil.ReadDir(fullPath)
-    if err != nil {
-        return directoryList, nil, err
-    }
+	files, err = ioutil.ReadDir(fullPath)
+	if err != nil {
+		return directoryList, nil, err
+	}
 
-    normalizedPath := path
-    if len(normalizedPath) > 0 && normalizedPath[len(normalizedPath) - 1] != '/' {
-        normalizedPath += "/"
-    }
+	// This binary search works because ioutil.ReadDir returns files sorted by Name() by default
+	if nobackupFile != "" {
+		ii := sort.Search(len(files), func(ii int) bool { return strings.Compare(files[ii].Name(), nobackupFile) >= 0 })
+		if ii < len(files) && files[ii].Name() == nobackupFile {
+			LOG_DEBUG("LIST_NOBACKUP", "%s is excluded due to nobackup file", path)
+			return directoryList, skippedFiles, nil
+		}
+	}
 
-    normalizedTop := top
-    if normalizedTop != "" && normalizedTop[len(normalizedTop) - 1] != '/' {
-        normalizedTop += "/"
-    }
+	normalizedPath := path
+	if len(normalizedPath) > 0 && normalizedPath[len(normalizedPath)-1] != '/' {
+		normalizedPath += "/"
+	}
 
-    sort.Sort(FileInfoCompare(files))
+	normalizedTop := top
+	if normalizedTop != "" && normalizedTop[len(normalizedTop)-1] != '/' {
+		normalizedTop += "/"
+	}
 
-    entries := make([]*Entry, 0, 4)
+	sort.Sort(FileInfoCompare(files))
 
-    for _, f := range files {
-        if f.Name() == DUPLICACY_DIRECTORY {
-            continue
-        }
-        entry := CreateEntryFromFileInfo(f, normalizedPath)
-        if len(patterns) > 0 && !MatchPath(entry.Path, patterns) {
-            LOG_DEBUG("LIST_EXCLUDE", "%s is excluded", entry.Path)
-            continue
-        }
-        if entry.IsLink() {
-            isRegular := false
-            isRegular, entry.Link, err = Readlink(filepath.Join(top, entry.Path))
-            if err != nil {
-                LOG_WARN("LIST_LINK", "Failed to read the symlink %s: %v", entry.Path, err )
-                skippedFiles = append(skippedFiles, entry.Path)
-                continue
-            }
+	entries := make([]*Entry, 0, 4)
 
-            if isRegular {
-                entry.Mode ^= uint32(os.ModeSymlink)
-            } else if path == "" && filepath.IsAbs(entry.Link) && !strings.HasPrefix(entry.Link, normalizedTop) {
-                stat, err := os.Stat(filepath.Join(top, entry.Path))
-                if err != nil {
-                    LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err )
-                    skippedFiles = append(skippedFiles, entry.Path)
-                    continue
-                }
-                entry = CreateEntryFromFileInfo(stat, "")
-            }
-        }
+	for _, f := range files {
+		if f.Name() == DUPLICACY_DIRECTORY {
+			continue
+		}
+		entry := CreateEntryFromFileInfo(f, normalizedPath)
+		if len(patterns) > 0 && !MatchPath(entry.Path, patterns) {
+			continue
+		}
+		if entry.IsLink() {
+			isRegular := false
+			isRegular, entry.Link, err = Readlink(filepath.Join(top, entry.Path))
+			if err != nil {
+				LOG_WARN("LIST_LINK", "Failed to read the symlink %s: %v", entry.Path, err)
+				skippedFiles = append(skippedFiles, entry.Path)
+				continue
+			}
 
-        if !discardAttributes {
-            entry.ReadAttributes(top)
-        }
+			if isRegular {
+				entry.Mode ^= uint32(os.ModeSymlink)
+			} else if path == "" && (filepath.IsAbs(entry.Link) || filepath.HasPrefix(entry.Link, `\\`)) && !strings.HasPrefix(entry.Link, normalizedTop) {
+				stat, err := os.Stat(filepath.Join(top, entry.Path))
+				if err != nil {
+					LOG_WARN("LIST_LINK", "Failed to read the symlink: %v", err)
+					skippedFiles = append(skippedFiles, entry.Path)
+					continue
+				}
 
-        if f.Mode() & (os.ModeNamedPipe | os.ModeSocket | os.ModeDevice) != 0 {
-            LOG_WARN("LIST_SKIP", "Skipped non-regular file %s", entry.Path)
-            skippedFiles = append(skippedFiles, entry.Path)
-            continue
-        }
+				newEntry := CreateEntryFromFileInfo(stat, "")
+				if runtime.GOOS == "windows" {
+					// On Windows, stat.Name() is the last component of the target, so we need to construct the correct
+					// path from f.Name(); note that a "/" is append assuming a symbolic link is always a directory
+					newEntry.Path = filepath.Join(normalizedPath, f.Name()) + "/"
+				}
+				entry = newEntry
+			}
+		}
 
-        entries = append(entries, entry)
-    }
+		if !discardAttributes {
+			entry.ReadAttributes(top)
+		}
 
-    // For top level directory we need to sort again because symlinks may have been changed
-    if path == "" {
-        sort.Sort(ByName(entries))
-    }
+		if f.Mode()&(os.ModeNamedPipe|os.ModeSocket|os.ModeDevice) != 0 {
+			LOG_WARN("LIST_SKIP", "Skipped non-regular file %s", entry.Path)
+			skippedFiles = append(skippedFiles, entry.Path)
+			continue
+		}
 
-    for _, entry := range entries {
-        if entry.IsDir() {
-            directoryList = append(directoryList, entry)
-        } else {
-            *fileList = append(*fileList, entry)
-        }
-    }
+		entries = append(entries, entry)
+	}
 
-    for i, j := 0, len(directoryList) - 1; i < j; i, j = i + 1, j - 1 {
-        directoryList[i], directoryList[j] = directoryList[j], directoryList[i]
-    }
+	// For top level directory we need to sort again because symlinks may have been changed
+	if path == "" {
+		sort.Sort(ByName(entries))
+	}
 
-    return directoryList, skippedFiles, nil
+	for _, entry := range entries {
+		if entry.IsDir() {
+			directoryList = append(directoryList, entry)
+		} else {
+			*fileList = append(*fileList, entry)
+		}
+	}
+
+	for i, j := 0, len(directoryList)-1; i < j; i, j = i+1, j-1 {
+		directoryList[i], directoryList[j] = directoryList[j], directoryList[i]
+	}
+
+	return directoryList, skippedFiles, nil
 }
 
 // Diff returns how many bytes remain unmodifiled between two files.
-func (entry *Entry) Diff(chunkHashes[]string, chunkLengths[]int,
-                         otherHashes[]string, otherLengths [] int) (modifiedLength int64) {
+func (entry *Entry) Diff(chunkHashes []string, chunkLengths []int,
+	otherHashes []string, otherLengths []int) (modifiedLength int64) {
 
-    var offset1, offset2 int64
-    i1 := entry.StartChunk
-    i2 := 0
-    for i1 <= entry.EndChunk && i2 < len(otherHashes) {
+	var offset1, offset2 int64
+	i1 := entry.StartChunk
+	i2 := 0
+	for i1 <= entry.EndChunk && i2 < len(otherHashes) {
 
-        start := 0
-        if i1 == entry.StartChunk {
-            start = entry.StartOffset
-        }
-        end := chunkLengths[i1]
-        if i1 == entry.EndChunk {
-            end = entry.EndOffset
-        }
+		start := 0
+		if i1 == entry.StartChunk {
+			start = entry.StartOffset
+		}
+		end := chunkLengths[i1]
+		if i1 == entry.EndChunk {
+			end = entry.EndOffset
+		}
 
-        if offset1 < offset2 {
-            modifiedLength += int64(end - start)
-            offset1 += int64(end - start)
-            i1++
-        } else if offset1 > offset2 {
-            offset2 += int64(otherLengths[i2])
-            i2++
-        } else {
-            if chunkHashes[i1] == otherHashes[i2] && end - start == otherLengths[i2] {
-            } else {
-                modifiedLength += int64(chunkLengths[i1])
-            }
-            offset1 += int64(end - start)
-            offset2 += int64(otherLengths[i2])
-            i1++
-            i2++
-        }
-    }
+		if offset1 < offset2 {
+			modifiedLength += int64(end - start)
+			offset1 += int64(end - start)
+			i1++
+		} else if offset1 > offset2 {
+			offset2 += int64(otherLengths[i2])
+			i2++
+		} else {
+			if chunkHashes[i1] == otherHashes[i2] && end-start == otherLengths[i2] {
+			} else {
+				modifiedLength += int64(chunkLengths[i1])
+			}
+			offset1 += int64(end - start)
+			offset2 += int64(otherLengths[i2])
+			i1++
+			i2++
+		}
+	}
 
-    return modifiedLength
+	return modifiedLength
 }

src/duplicacy_entry_test.go

@@ -5,216 +5,214 @@
 package duplicacy
 
 import (
-    "testing"
-    "io/ioutil"
-    "os"
-    "path/filepath"
-    "math/rand"
-    "sort"
+	"io/ioutil"
+	"math/rand"
+	"os"
+	"path/filepath"
+	"sort"
+	"testing"
 )
 
 func TestEntrySort(t *testing.T) {
 
-    DATA := [...]string {
-        "ab",
-        "ab-",
-        "ab0",
-        "ab1",
-        "\xBB\xDDfile",
-        "\xFF\xDDfile",
-        "ab/",
-        "ab/c",
-        "ab+/c-",
-        "ab+/c0",
-        "ab+/c/",
-        "ab+/c/d",
-        "ab+/c+/",
-        "ab+/c+/d",
-        "ab+/c0/",
-        "ab+/c0/d",
-        "ab-/",
-        "ab-/c",
-        "ab0/",
-        "ab1/",
-        "ab1/c",
-        "ab1/\xBB\xDDfile",
-        "ab1/\xFF\xDDfile",
-    }
+	DATA := [...]string{
+		"ab",
+		"ab-",
+		"ab0",
+		"ab1",
+		"\xBB\xDDfile",
+		"\xFF\xDDfile",
+		"ab/",
+		"ab/c",
+		"ab+/c-",
+		"ab+/c0",
+		"ab+/c/",
+		"ab+/c/d",
+		"ab+/c+/",
+		"ab+/c+/d",
+		"ab+/c0/",
+		"ab+/c0/d",
+		"ab-/",
+		"ab-/c",
+		"ab0/",
+		"ab1/",
+		"ab1/c",
+		"ab1/\xBB\xDDfile",
+		"ab1/\xFF\xDDfile",
+	}
 
-    var entry1, entry2 *Entry
+	var entry1, entry2 *Entry
 
-    for i, p1 := range DATA {
-        if p1[len(p1) - 1] == '/' {
-            entry1 = CreateEntry(p1, 0, 0, 0700 | uint32(os.ModeDir))
-        } else {
-            entry1 = CreateEntry(p1, 0, 0, 0700)
-        }
-        for j, p2 := range DATA {
+	for i, p1 := range DATA {
+		if p1[len(p1)-1] == '/' {
+			entry1 = CreateEntry(p1, 0, 0, 0700|uint32(os.ModeDir))
+		} else {
+			entry1 = CreateEntry(p1, 0, 0, 0700)
+		}
+		for j, p2 := range DATA {
 
-            if p2[len(p2) - 1] == '/' {
-                entry2 = CreateEntry(p2, 0, 0, 0700 | uint32(os.ModeDir))
-            } else {
-                entry2 = CreateEntry(p2, 0, 0, 0700)
-            }
+			if p2[len(p2)-1] == '/' {
+				entry2 = CreateEntry(p2, 0, 0, 0700|uint32(os.ModeDir))
+			} else {
+				entry2 = CreateEntry(p2, 0, 0, 0700)
+			}
 
-            compared := entry1.Compare(entry2)
+			compared := entry1.Compare(entry2)
 
-            if compared < 0 {
-                compared = -1
-            } else if compared > 0 {
-                compared = 1
-            }
+			if compared < 0 {
+				compared = -1
+			} else if compared > 0 {
+				compared = 1
+			}
 
-            var expected int
-            if i < j {
-                expected = -1
-            } else if i > j {
-                expected = 1
-            } else {
-                expected = 0
-            }
+			var expected int
+			if i < j {
+				expected = -1
+			} else if i > j {
+				expected = 1
+			} else {
+				expected = 0
+			}
 
-            if compared != expected {
-                t.Errorf("%s vs %s: %d, expected: %d", p1, p2, compared, expected)
-            }
+			if compared != expected {
+				t.Errorf("%s vs %s: %d, expected: %d", p1, p2, compared, expected)
+			}
 
-        }
-    }
+		}
+	}
 }
 
 func TestEntryList(t *testing.T) {
 
-    testDir := filepath.Join(os.TempDir(), "duplicacy_test")
-    os.RemoveAll(testDir)
-    os.MkdirAll(testDir, 0700)
+	testDir := filepath.Join(os.TempDir(), "duplicacy_test")
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)
 
-    DATA := [...]string {
-        "ab",
-        "ab-",
-        "ab0",
-        "ab1",
-        "ab+/",
-        "ab+/c",
-        "ab+/c+",
-        "ab+/c1",
-        "ab+/c-/",
-        "ab+/c-/d",
-        "ab+/c0/",
-        "ab+/c0/d",
-        "ab2/",
-        "ab2/c",
-        "ab3/",
-        "ab3/c",
-    }
+	DATA := [...]string{
+		"ab",
+		"ab-",
+		"ab0",
+		"ab1",
+		"ab+/",
+		"ab+/c",
+		"ab+/c+",
+		"ab+/c1",
+		"ab+/c-/",
+		"ab+/c-/d",
+		"ab+/c0/",
+		"ab+/c0/d",
+		"ab2/",
+		"ab2/c",
+		"ab3/",
+		"ab3/c",
+	}
 
+	var entry1, entry2 *Entry
 
-    var entry1, entry2 *Entry
+	for i, p1 := range DATA {
+		if p1[len(p1)-1] == '/' {
+			entry1 = CreateEntry(p1, 0, 0, 0700|uint32(os.ModeDir))
+		} else {
+			entry1 = CreateEntry(p1, 0, 0, 0700)
+		}
+		for j, p2 := range DATA {
 
-    for i, p1 := range DATA {
-        if p1[len(p1) - 1] == '/' {
-            entry1 = CreateEntry(p1, 0, 0, 0700 | uint32(os.ModeDir))
-        } else {
-            entry1 = CreateEntry(p1, 0, 0, 0700)
-        }
-        for j, p2 := range DATA {
+			if p2[len(p2)-1] == '/' {
+				entry2 = CreateEntry(p2, 0, 0, 0700|uint32(os.ModeDir))
+			} else {
+				entry2 = CreateEntry(p2, 0, 0, 0700)
+			}
 
-            if p2[len(p2) - 1] == '/' {
-                entry2 = CreateEntry(p2, 0, 0, 0700 | uint32(os.ModeDir))
-            } else {
-                entry2 = CreateEntry(p2, 0, 0, 0700)
-            }
+			compared := entry1.Compare(entry2)
 
-            compared := entry1.Compare(entry2)
+			if compared < 0 {
+				compared = -1
+			} else if compared > 0 {
+				compared = 1
+			}
 
-            if compared < 0 {
-                compared = -1
-            } else if compared > 0 {
-                compared = 1
-            }
+			var expected int
+			if i < j {
+				expected = -1
+			} else if i > j {
+				expected = 1
+			} else {
+				expected = 0
+			}
 
-            var expected int
-            if i < j {
-                expected = -1
-            } else if i > j {
-                expected = 1
-            } else {
-                expected = 0
-            }
+			if compared != expected {
+				t.Errorf("%s vs %s: %d, expected: %d", p1, p2, compared, expected)
+			}
 
-            if compared != expected {
-                t.Errorf("%s vs %s: %d, expected: %d", p1, p2, compared, expected)
-            }
+		}
+	}
 
-        }
-    }
+	for _, file := range DATA {
 
-    for _, file := range DATA {
+		fullPath := filepath.Join(testDir, file)
+		if file[len(file)-1] == '/' {
+			err := os.Mkdir(fullPath, 0700)
+			if err != nil {
+				t.Errorf("Mkdir(%s) returned an error: %s", fullPath, err)
+			}
+			continue
+		}
 
-        fullPath := filepath.Join(testDir, file)
-        if file[len(file) - 1] == '/' {
-            err := os.Mkdir(fullPath, 0700)
-            if err != nil {
-                t.Errorf("Mkdir(%s) returned an error: %s", fullPath, err)
-            }
-            continue
-        }
+		err := ioutil.WriteFile(fullPath, []byte(file), 0700)
+		if err != nil {
+			t.Errorf("WriteFile(%s) returned an error: %s", fullPath, err)
+		}
+	}
 
-        err := ioutil.WriteFile(fullPath, []byte(file), 0700)
-        if err != nil {
-            t.Errorf("WriteFile(%s) returned an error: %s", fullPath, err)
-        }
-    }
+	directories := make([]*Entry, 0, 4)
+	directories = append(directories, CreateEntry("", 0, 0, 0))
 
-    directories := make([]*Entry, 0, 4)
-    directories = append(directories, CreateEntry("", 0, 0, 0))
+	entries := make([]*Entry, 0, 4)
 
-    entries := make([]*Entry, 0, 4)
+	for len(directories) > 0 {
+		directory := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
+		entries = append(entries, directory)
+		subdirectories, _, err := ListEntries(testDir, directory.Path, &entries, nil, "", false)
+		if err != nil {
+			t.Errorf("ListEntries(%s, %s) returned an error: %s", testDir, directory.Path, err)
+		}
+		directories = append(directories, subdirectories...)
+	}
 
-    for len(directories) > 0 {
-        directory := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
-        entries = append(entries, directory)
-        subdirectories, _, err := ListEntries(testDir, directory.Path, &entries, nil, false)
-        if err != nil {
-            t.Errorf("ListEntries(%s, %s) returned an error: %s", testDir, directory.Path, err)
-        }
-        directories = append(directories, subdirectories...)
-    }
+	entries = entries[1:]
 
-    entries = entries[1:]
+	for _, entry := range entries {
+		t.Logf("entry: %s", entry.Path)
+	}
 
-    for _, entry := range entries {
-        t.Logf("entry: %s", entry.Path)
-    }
+	if len(entries) != len(DATA) {
+		t.Errorf("Got %d entries instead of %d", len(entries), len(DATA))
+		return
+	}
 
-    if len(entries) != len(DATA) {
-        t.Errorf("Got %d entries instead of %d", len(entries), len(DATA))
-        return
-    }
+	for i := 0; i < len(entries); i++ {
+		if entries[i].Path != DATA[i] {
+			t.Errorf("entry: %s, expected: %s", entries[i].Path, DATA[i])
+		}
+	}
 
-    for i := 0; i < len(entries); i++ {
-        if entries[i].Path != DATA[i] {
-            t.Errorf("entry: %s, expected: %s", entries[i].Path, DATA[i])
-        }
-    }
+	t.Logf("shuffling %d entries", len(entries))
+	for i := range entries {
+		j := rand.Intn(i + 1)
+		entries[i], entries[j] = entries[j], entries[i]
+	}
 
-    t.Logf("shuffling %d entries", len(entries))
-    for i := range entries {
-        j := rand.Intn(i + 1)
-        entries[i], entries[j] = entries[j], entries[i]
-    }
+	sort.Sort(ByName(entries))
 
-    sort.Sort(ByName(entries))
+	for i := 0; i < len(entries); i++ {
+		if entries[i].Path != DATA[i] {
+			t.Errorf("entry: %s, expected: %s", entries[i].Path, DATA[i])
+		}
+	}
 
-    for i := 0; i < len(entries); i++ {
-        if entries[i].Path != DATA[i] {
-            t.Errorf("entry: %s, expected: %s", entries[i].Path, DATA[i])
-        }
-    }
-
-    if !t.Failed() {
-        os.RemoveAll(testDir)
-    }
+	if !t.Failed() {
+		os.RemoveAll(testDir)
+	}
 
 }
-

src/duplicacy_filereader.go

@@ -5,70 +5,66 @@
 package duplicacy
 
 import (
-    "os"
+	"os"
 )
 
 // FileReader wraps a number of files and turns them into a series of readers.
 type FileReader struct {
-    top string
-    files [] *Entry
+	top   string
+	files []*Entry
 
-    CurrentFile *os.File
-    CurrentIndex int
-    CurrentEntry *Entry
+	CurrentFile  *os.File
+	CurrentIndex int
+	CurrentEntry *Entry
 
-    SkippedFiles [] string
+	SkippedFiles []string
 }
 
 // CreateFileReader creates a file reader.
-func CreateFileReader(top string, files[] *Entry) (*FileReader) {
+func CreateFileReader(top string, files []*Entry) *FileReader {
 
-    reader := &FileReader {
-        top: top,
-        files: files,
-        CurrentIndex: -1,
-    }
+	reader := &FileReader{
+		top:          top,
+		files:        files,
+		CurrentIndex: -1,
+	}
 
-    reader.NextFile()
+	reader.NextFile()
 
-    return reader
+	return reader
 }
 
-// NextFile switchs to the next file in the file reader.
-func (reader *FileReader) NextFile() bool{
+// NextFile switches to the next file in the file reader.
+func (reader *FileReader) NextFile() bool {
 
-    if reader.CurrentFile != nil {
-        reader.CurrentFile.Close()
-    }
+	if reader.CurrentFile != nil {
+		reader.CurrentFile.Close()
+	}
 
-    reader.CurrentIndex++
-    for reader.CurrentIndex < len(reader.files) {
+	reader.CurrentIndex++
+	for reader.CurrentIndex < len(reader.files) {
 
-        reader.CurrentEntry = reader.files[reader.CurrentIndex]
-        if !reader.CurrentEntry.IsFile() || reader.CurrentEntry.Size == 0 {
-            reader.CurrentIndex++
-            continue
-        }
+		reader.CurrentEntry = reader.files[reader.CurrentIndex]
+		if !reader.CurrentEntry.IsFile() || reader.CurrentEntry.Size == 0 {
+			reader.CurrentIndex++
+			continue
+		}
 
-        var err error
+		var err error
 
-        fullPath := joinPath(reader.top, reader.CurrentEntry.Path)
-        reader.CurrentFile, err = os.OpenFile(fullPath, os.O_RDONLY, 0)
-        if err != nil {
-            LOG_WARN("OPEN_FAILURE", "Failed to open file for reading: %v", err)
-            reader.CurrentEntry.Size = 0
-            reader.SkippedFiles = append(reader.SkippedFiles, reader.CurrentEntry.Path)
-            reader.CurrentIndex++
-            continue
-        }
+		fullPath := joinPath(reader.top, reader.CurrentEntry.Path)
+		reader.CurrentFile, err = os.OpenFile(fullPath, os.O_RDONLY, 0)
+		if err != nil {
+			LOG_WARN("OPEN_FAILURE", "Failed to open file for reading: %v", err)
+			reader.CurrentEntry.Size = 0
+			reader.SkippedFiles = append(reader.SkippedFiles, reader.CurrentEntry.Path)
+			reader.CurrentIndex++
+			continue
+		}
 
-        return true
-    }
+		return true
+	}
 
-    reader.CurrentFile = nil
-    return false
+	reader.CurrentFile = nil
+	return false
 }
-
-
-
-

src/duplicacy_filestorage.go

@@ -5,252 +5,228 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "path"
-    "io"
-    "io/ioutil"
-    "time"
-    "math/rand"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"os"
+	"path"
+	"strings"
+	"time"
 )
 
 // FileStorage is a local on-disk file storage implementing the Storage interface.
 type FileStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    storageDir string
-    numberOfThreads int
+	isCacheNeeded   bool // Network storages require caching
+	storageDir      string
+	numberOfThreads int
 }
 
 // CreateFileStorage creates a file storage.
-func CreateFileStorage(storageDir string, threads int) (storage *FileStorage, err error) {
+func CreateFileStorage(storageDir string, isCacheNeeded bool, threads int) (storage *FileStorage, err error) {
 
-    var stat os.FileInfo
+	var stat os.FileInfo
 
-    stat, err = os.Stat(storageDir)
-    if os.IsNotExist(err) {
-        err = os.MkdirAll(storageDir, 0744)
-        if err != nil {
-            return nil, err
-        }
-    } else {
-        if !stat.IsDir() {
-            return nil, fmt.Errorf("The storage path %s is a file", storageDir)
-        }
-    }
+	stat, err = os.Stat(storageDir)
+	if err != nil {
+		if os.IsNotExist(err) {
+			err = os.MkdirAll(storageDir, 0744)
+			if err != nil {
+				return nil, err
+			}
+		} else {
+			return nil, err
+		}
+	} else {
+		if !stat.IsDir() {
+			return nil, fmt.Errorf("The storage path %s is a file", storageDir)
+		}
+	}
 
-    for storageDir[len(storageDir) - 1] ==  '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
-    }
+	for storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
+	}
 
-    storage = &FileStorage {
-        storageDir : storageDir,
-        numberOfThreads: threads,
-    }
+	storage = &FileStorage{
+		storageDir:      storageDir,
+		isCacheNeeded:   isCacheNeeded,
+		numberOfThreads: threads,
+	}
 
-    // Random number fo generating the temporary chunk file suffix.
-    rand.Seed(time.Now().UnixNano())
+	// Random number fo generating the temporary chunk file suffix.
+	rand.Seed(time.Now().UnixNano())
 
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively).
 func (storage *FileStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
 
-    fullPath := path.Join(storage.storageDir, dir)
+	fullPath := path.Join(storage.storageDir, dir)
 
-    list, err := ioutil.ReadDir(fullPath)
-    if err != nil {
-        if os.IsNotExist(err) {
-            return nil, nil, nil
-        }
-        return nil, nil, err
-    }
+	list, err := ioutil.ReadDir(fullPath)
+	if err != nil {
+		if os.IsNotExist(err) {
+			return nil, nil, nil
+		}
+		return nil, nil, err
+	}
 
-    for _, f := range list {
-        name := f.Name()
-        if f.IsDir() && name[len(name) - 1] != '/' {
-            name += "/"
-        }
-        files = append(files, name)
-        sizes = append(sizes, f.Size())
-    }
+	for _, f := range list {
+		name := f.Name()
+		if f.IsDir() && name[len(name)-1] != '/' {
+			name += "/"
+		}
+		files = append(files, name)
+		sizes = append(sizes, f.Size())
+	}
 
-    return files, sizes, nil
+	return files, sizes, nil
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *FileStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    err = os.Remove(path.Join(storage.storageDir, filePath))
-    if err == nil || os.IsNotExist(err) {
-        return nil
-    } else {
-        return err
-    }
+	err = os.Remove(path.Join(storage.storageDir, filePath))
+	if err == nil || os.IsNotExist(err) {
+		return nil
+	} else {
+		return err
+	}
 }
 
 // MoveFile renames the file.
 func (storage *FileStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    return os.Rename(path.Join(storage.storageDir, from), path.Join(storage.storageDir, to))
+	return os.Rename(path.Join(storage.storageDir, from), path.Join(storage.storageDir, to))
 }
 
 // CreateDirectory creates a new directory.
 func (storage *FileStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    err = os.Mkdir(path.Join(storage.storageDir, dir), 0744)
-    if err != nil && os.IsExist(err) {
-        return nil
-    } else {
-        return err
-    }
+	err = os.Mkdir(path.Join(storage.storageDir, dir), 0744)
+	if err != nil && os.IsExist(err) {
+		return nil
+	} else {
+		return err
+	}
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *FileStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
-    stat, err := os.Stat(path.Join(storage.storageDir, filePath))
-    if err != nil {
-        if os.IsNotExist(err) {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
-
-    return true, stat.IsDir(), stat.Size(), nil
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with the
-// suffix '.fsl'.
-func (storage *FileStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := path.Join(storage.storageDir, "chunks")
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    // The minimum level of directories to dive into before searching for the chunk file.
-    minimumLevel := 2
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            if stat, err := os.Stat(filePath); err == nil && !stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, stat.Size(), nil
-            } else if err == nil && stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, 0, fmt.Errorf("The path %s is a directory", filePath)
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        stat, err := os.Stat(subDir)
-        if err == nil && stat.IsDir() {
-            dir = subDir
-            continue
-        }
-
-        if level < minimumLevel {
-            // Create the subdirectory if it doesn't exist.
-
-            if err == nil && !stat.IsDir() {
-                return "", false, 0, fmt.Errorf("The path %s is not a directory", subDir)
-            }
-
-            err = os.Mkdir(subDir, 0744)
-            if err != nil {
-                // The directory may have been created by other threads so check it again.
-                stat, _ := os.Stat(subDir)
-                if stat == nil || !stat.IsDir() {
-                    return "", false, 0, err
-                }
-            }
-
-            dir = subDir
-            continue
-        }
-
-        // The chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[len(storage.storageDir) + 1:] + suffix, false, 0, nil
-
-    }
-
-    LOG_FATAL("CHUNK_FIND", "Chunk %s is still not found after having searched a maximum level of directories",
-              chunkID)
-    return "", false, 0, nil
+	stat, err := os.Stat(path.Join(storage.storageDir, filePath))
+	if err != nil {
+		if os.IsNotExist(err) {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
+	return true, stat.IsDir(), stat.Size(), nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *FileStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-    file, err := os.Open(path.Join(storage.storageDir, filePath))
+	file, err := os.Open(path.Join(storage.storageDir, filePath))
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    defer file.Close()
-    if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit / storage.numberOfThreads); err != nil {
-        return err
-    }
+	defer file.Close()
+	if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.numberOfThreads); err != nil {
+		return err
+	}
 
-    return nil
+	return nil
 
 }
 
 // UploadFile writes 'content' to the file at 'filePath'
 func (storage *FileStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 
-    fullPath := path.Join(storage.storageDir, filePath)
+	fullPath := path.Join(storage.storageDir, filePath)
 
-    letters := "abcdefghijklmnopqrstuvwxyz"
-    suffix := make([]byte, 8)
-    for i := range suffix {
-        suffix[i] = letters[rand.Intn(len(letters))]
-    }
+	if len(strings.Split(filePath, "/")) > 2 {
+		dir := path.Dir(fullPath)
+		// Use Lstat() instead of Stat() since 1) Stat() doesn't work for deduplicated disks on Windows and 2) there isn't
+		// really a need to follow the link if filePath is a link.
+		stat, err := os.Lstat(dir)
+		if err != nil {
+			if !os.IsNotExist(err) {
+				return err
+			}
+			err = os.MkdirAll(dir, 0744)
+			if err != nil {
+				return err
+			}
+		} else {
+			if !stat.IsDir() {
+				return fmt.Errorf("The path %s is not a directory", dir)
+			}
+		}
+	}
 
-    temporaryFile := fullPath + "." + string(suffix) + ".tmp"
+	letters := "abcdefghijklmnopqrstuvwxyz"
+	suffix := make([]byte, 8)
+	for i := range suffix {
+		suffix[i] = letters[rand.Intn(len(letters))]
+	}
 
-    file, err := os.OpenFile(temporaryFile, os.O_WRONLY | os.O_CREATE | os.O_TRUNC, 0644)
-    if err != nil {
-        return err
-    }
+	temporaryFile := fullPath + "." + string(suffix) + ".tmp"
 
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
-    _, err = io.Copy(file, reader)
-    if err != nil {
-        file.Close()
-        return err
-    }
+	file, err := os.OpenFile(temporaryFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0644)
+	if err != nil {
+		return err
+	}
 
-    file.Close()
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
+	_, err = io.Copy(file, reader)
+	if err != nil {
+		file.Close()
+		return err
+	}
 
-    err = os.Rename(temporaryFile, fullPath)
-    if err != nil {
+	err = file.Sync()
+	if err != nil {
+		file.Close()
+		return err
+	}
 
-        if _, e := os.Stat(fullPath); e == nil {
-            os.Remove(temporaryFile)
-            return nil
-        } else {
-            return err
-        }
-    }
+	err = file.Close()
+	if err != nil {
+		return err
+	}
 
-    return nil
+	err = os.Rename(temporaryFile, fullPath)
+	if err != nil {
+
+		if _, e := os.Stat(fullPath); e == nil {
+			os.Remove(temporaryFile)
+			return nil
+		} else {
+			return err
+		}
+	}
+
+	return nil
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *FileStorage) IsCacheNeeded () (bool) { return false }
+func (storage *FileStorage) IsCacheNeeded() bool { return storage.isCacheNeeded }
 
 // If the 'MoveFile' method is implemented.
-func (storage *FileStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *FileStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *FileStorage) IsStrongConsistent() (bool) { return true }
+func (storage *FileStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *FileStorage) IsFastListing() (bool) { return false }
+func (storage *FileStorage) IsFastListing() bool { return false }
 
 // Enable the test mode.
 func (storage *FileStorage) EnableTestMode() {}

src/duplicacy_gcsstorage.go

@@ -5,65 +5,64 @@
 package duplicacy
 
 import (
-    "io"
+	"encoding/json"
 	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
 	"net"
+	"net/url"
 	"time"
-    "net/url"
-    "math/rand"
-    "io/ioutil"
-    "encoding/json"
 
-    "golang.org/x/net/context"
-    "golang.org/x/oauth2"
-	"golang.org/x/oauth2/google"
 	gcs "cloud.google.com/go/storage"
+	"golang.org/x/net/context"
+	"golang.org/x/oauth2"
+	"golang.org/x/oauth2/google"
+	"google.golang.org/api/googleapi"
 	"google.golang.org/api/iterator"
 	"google.golang.org/api/option"
-    "google.golang.org/api/googleapi"
 )
 
 type GCSStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    bucket *gcs.BucketHandle
+	bucket     *gcs.BucketHandle
 	storageDir string
 
-    numberOfThreads int
-    TestMode bool
-
+	numberOfThreads int
+	TestMode        bool
 }
 
 type GCSConfig struct {
-    ClientID     string          `json:"client_id"`
-    ClientSecret string          `json:"client_secret"`
-    Endpoint     oauth2.Endpoint `json:"end_point"`
-    Token        oauth2.Token    `json:"token"`
+	ClientID     string          `json:"client_id"`
+	ClientSecret string          `json:"client_secret"`
+	Endpoint     oauth2.Endpoint `json:"end_point"`
+	Token        oauth2.Token    `json:"token"`
 }
 
 // CreateGCSStorage creates a GCD storage object.
 func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, threads int) (storage *GCSStorage, err error) {
 
-    ctx := context.Background()
+	ctx := context.Background()
 
-    description, err := ioutil.ReadFile(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	description, err := ioutil.ReadFile(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    var object map[string]interface {}
+	var object map[string]interface{}
 
-    err = json.Unmarshal(description, &object)
-    if err != nil {
-        return nil, err
-    }
+	err = json.Unmarshal(description, &object)
+	if err != nil {
+		return nil, err
+	}
 
 	isServiceAccount := false
-    if value, ok := object["type"]; ok {
+	if value, ok := object["type"]; ok {
 		if authType, ok := value.(string); ok && authType == "service_account" {
 			isServiceAccount = true
 		}
-	}		
+	}
 
 	var tokenSource oauth2.TokenSource
 
@@ -74,7 +73,7 @@ func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, th
 		}
 		tokenSource = config.TokenSource(ctx)
 	} else {
-		gcsConfig := &GCSConfig {}
+		gcsConfig := &GCSConfig{}
 		if err := json.Unmarshal(description, gcsConfig); err != nil {
 			return nil, err
 		}
@@ -87,82 +86,82 @@ func CreateGCSStorage(tokenFile string, bucketName string, storageDir string, th
 		tokenSource = config.TokenSource(ctx, &gcsConfig.Token)
 	}
 
-    options := option.WithTokenSource(tokenSource)
-    client, err := gcs.NewClient(ctx, options)
+	options := option.WithTokenSource(tokenSource)
+	client, err := gcs.NewClient(ctx, options)
 
 	bucket := client.Bucket(bucketName)
 
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
-        storageDir += "/"
-    }
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+		storageDir += "/"
+	}
 
-    storage = &GCSStorage {
-        bucket: bucket,
-		storageDir: storageDir,
-        numberOfThreads: threads,
-    }
-
-    return storage, nil
+	storage = &GCSStorage{
+		bucket:          bucket,
+		storageDir:      storageDir,
+		numberOfThreads: threads,
+	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 func (storage *GCSStorage) shouldRetry(backoff *int, err error) (bool, error) {
 
-    retry := false
-    message := ""
-    if err == nil {
-        return false, nil
-    } else if e, ok := err.(*googleapi.Error); ok {
-        if 500 <= e.Code && e.Code < 600 {
-            // Retry for 5xx response codes.
-            message = fmt.Sprintf("HTTP status code %d", e.Code)
-            retry = true
-        } else if e.Code == 429 {
-            // Too many requests{
-            message = "HTTP status code 429"
-            retry = true
-        } else if e.Code == 403 {
-            // User Rate Limit Exceeded
-            message = "User Rate Limit Exceeded"
-            retry = true
-        }
-    } else if e, ok := err.(*url.Error); ok {
-        message = e.Error()
-        retry = true
-    } else if err == io.ErrUnexpectedEOF {
-        // Retry on unexpected EOFs and temporary network errors.
-        message = "Unexpected EOF"
-        retry = true
-    } else if err, ok := err.(net.Error); ok {
-        message = "Temporary network error"
-        retry = err.Temporary()
-    }
+	retry := false
+	message := ""
+	if err == nil {
+		return false, nil
+	} else if e, ok := err.(*googleapi.Error); ok {
+		if 500 <= e.Code && e.Code < 600 {
+			// Retry for 5xx response codes.
+			message = fmt.Sprintf("HTTP status code %d", e.Code)
+			retry = true
+		} else if e.Code == 429 {
+			// Too many requests{
+			message = "HTTP status code 429"
+			retry = true
+		} else if e.Code == 403 {
+			// User Rate Limit Exceeded
+			message = "User Rate Limit Exceeded"
+			retry = true
+		}
+	} else if e, ok := err.(*url.Error); ok {
+		message = e.Error()
+		retry = true
+	} else if err == io.ErrUnexpectedEOF {
+		// Retry on unexpected EOFs and temporary network errors.
+		message = "Unexpected EOF"
+		retry = true
+	} else if err, ok := err.(net.Error); ok {
+		message = "Temporary network error"
+		retry = err.Temporary()
+	}
 
-    if !retry || *backoff >= 256 {
-        return false, err
-    }
+	if !retry || *backoff >= 256 {
+		return false, err
+	}
 
-    delay := float32(*backoff) * rand.Float32()
-    LOG_INFO("GCS_RETRY", "%s; retrying after %.2f seconds", message, delay)
-    time.Sleep(time.Duration(float32(*backoff) * float32(time.Second)))
-    *backoff *= 2
-    return true, nil
+	delay := float32(*backoff) * rand.Float32()
+	LOG_INFO("GCS_RETRY", "%s; retrying after %.2f seconds", message, delay)
+	time.Sleep(time.Duration(float32(*backoff) * float32(time.Second)))
+	*backoff *= 2
+	return true, nil
 }
 
-
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *GCSStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-	query := gcs.Query {
+	query := gcs.Query{
 		Prefix: storage.storageDir + dir + "/",
 	}
 	dirOnly := false
 	prefixLength := len(query.Prefix)
 
-    if dir == "snapshots" {
+	if dir == "snapshots" {
 		query.Delimiter = "/"
 		dirOnly = true
 	}
@@ -174,7 +173,7 @@ func (storage *GCSStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 		attributes, err := iter.Next()
 		if err == iterator.Done {
 			break
-		}	
+		}
 		if err != nil {
 			return nil, nil, err
 		}
@@ -198,7 +197,7 @@ func (storage *GCSStorage) ListFiles(threadIndex int, dir string) ([]string, []i
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *GCSStorage) DeleteFile(threadIndex int, filePath string) (err error) {
 	err = storage.bucket.Object(storage.storageDir + filePath).Delete(context.Background())
-    if err == gcs.ErrObjectNotExist {
+	if err == gcs.ErrObjectNotExist {
 		return nil
 	}
 	return err
@@ -240,28 +239,15 @@ func (storage *GCSStorage) GetFileInfo(threadIndex int, filePath string) (exist
 	return true, false, attributes.Size, nil
 }
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *GCSStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-	exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-	return filePath, exist, size, err
-}
-
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *GCSStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 	readCloser, err := storage.bucket.Object(storage.storageDir + filePath).NewReader(context.Background())
 	if err != nil {
 		return err
 	}
-    defer readCloser.Close()
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
-    return err
+	defer readCloser.Close()
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
@@ -271,7 +257,7 @@ func (storage *GCSStorage) UploadFile(threadIndex int, filePath string, content
 	for {
 		writeCloser := storage.bucket.Object(storage.storageDir + filePath).NewWriter(context.Background())
 		defer writeCloser.Close()
-		reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
+		reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
 		_, err = io.Copy(writeCloser, reader)
 
 		if retry, e := storage.shouldRetry(&backoff, err); e == nil && !retry {
@@ -282,22 +268,22 @@ func (storage *GCSStorage) UploadFile(threadIndex int, filePath string, content
 			return err
 		}
 	}
-	
+
 	return err
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *GCSStorage) IsCacheNeeded() (bool) { return true }
+func (storage *GCSStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *GCSStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *GCSStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *GCSStorage) IsStrongConsistent() (bool) { return true }
+func (storage *GCSStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *GCSStorage) IsFastListing() (bool) { return true }
+func (storage *GCSStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *GCSStorage) EnableTestMode() { storage.TestMode = true }

src/duplicacy_hubicclient.go

@@ -5,456 +5,469 @@
 package duplicacy
 
 import (
-    "fmt"
-    "net"
-    "time"
-    "sync"
-    "bytes"
-    "strings"
-    "io/ioutil"
-    "encoding/json"
-    "io"
-    "net/http"
-    net_url "net/url"
-    "math/rand"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net"
+	"net/http"
+	net_url "net/url"
+	"strings"
+	"sync"
+	"time"
 
-    "golang.org/x/oauth2"
+	"golang.org/x/oauth2"
 )
 
 type HubicError struct {
-    Status     int
-    Message    string
+	Status  int
+	Message string
 }
 
 func (err HubicError) Error() string {
-    return fmt.Sprintf("%d %s", err.Status, err.Message)
+	return fmt.Sprintf("%d %s", err.Status, err.Message)
 }
 
 var HubicRefreshTokenURL = "https://duplicacy.com/hubic_refresh"
 var HubicCredentialURL = "https://api.hubic.com/1.0/account/credentials"
 
 type HubicCredential struct {
-    Token string
-    Endpoint string
-    Expires time.Time
+	Token    string
+	Endpoint string
+	Expires  time.Time
 }
 
 type HubicClient struct {
-    HTTPClient          *http.Client
+	HTTPClient *http.Client
 
-    TokenFile           string
-    Token               *oauth2.Token
-    TokenLock           *sync.Mutex
+	TokenFile string
+	Token     *oauth2.Token
+	TokenLock *sync.Mutex
 
-    Credential          HubicCredential
-    CredentialLock      *sync.Mutex
+	Credential     HubicCredential
+	CredentialLock *sync.Mutex
 
-    TestMode            bool
+	TestMode bool
 }
 
 func NewHubicClient(tokenFile string) (*HubicClient, error) {
 
-    description, err := ioutil.ReadFile(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	description, err := ioutil.ReadFile(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    token := new(oauth2.Token)
-    if err := json.Unmarshal(description, token); err != nil {
-        return nil, fmt.Errorf("%v: %s", err, description)
-    }
+	token := new(oauth2.Token)
+	if err := json.Unmarshal(description, token); err != nil {
+		return nil, fmt.Errorf("%v: %s", err, description)
+	}
 
-    client := &HubicClient{
-        HTTPClient: &http.Client {
-            Transport: &http.Transport {
-                Dial: (&net.Dialer{
-                        Timeout:   30 * time.Second,
-                        KeepAlive: 30 * time.Second,
-                }).Dial,
-                TLSHandshakeTimeout:   60 * time.Second,
-                ResponseHeaderTimeout: 30 * time.Second,
-                ExpectContinueTimeout: 10 * time.Second,
-            },
-        },
-        TokenFile: tokenFile,
-        Token: token,
-        TokenLock: &sync.Mutex{},
-        CredentialLock: &sync.Mutex{},
-    }
+	client := &HubicClient{
+		HTTPClient: &http.Client{
+			Transport: &http.Transport{
+				Dial: (&net.Dialer{
+					Timeout:   30 * time.Second,
+					KeepAlive: 30 * time.Second,
+				}).Dial,
+				TLSHandshakeTimeout:   60 * time.Second,
+				ResponseHeaderTimeout: 300 * time.Second,
+				ExpectContinueTimeout: 10 * time.Second,
+			},
+		},
+		TokenFile:      tokenFile,
+		Token:          token,
+		TokenLock:      &sync.Mutex{},
+		CredentialLock: &sync.Mutex{},
+	}
 
-    err = client.RefreshToken()
-    if err != nil {
-        return nil, err
-    }
+	err = client.RefreshToken(false)
+	if err != nil {
+		return nil, err
+	}
 
-    err = client.GetCredential()
-    if err != nil {
-        return nil, err
-    }
+	err = client.GetCredential()
+	if err != nil {
+		return nil, err
+	}
 
-    return client, nil
+	return client, nil
 }
 
 func (client *HubicClient) call(url string, method string, input interface{}, extraHeader map[string]string) (io.ReadCloser, int64, string, error) {
 
-    var response *http.Response
+	var response *http.Response
 
-    backoff := 1
-    for i := 0; i < 8; i++ {
+	backoff := 1
+	for i := 0; i < 11; i++ {
 
-        LOG_DEBUG("HUBIC_CALL", "%s %s", method, url)
+		LOG_DEBUG("HUBIC_CALL", "%s %s", method, url)
 
-        //fmt.Printf("%s %s\n", method, url)
+		//fmt.Printf("%s %s\n", method, url)
 
-        var inputReader io.Reader
+		var inputReader io.Reader
 
-        switch input.(type) {
-        default:
-            jsonInput, err := json.Marshal(input)
-            if err != nil {
-                return nil, 0, "", err
-            }
-            inputReader = bytes.NewReader(jsonInput)
-        case []byte:
-            inputReader = bytes.NewReader(input.([]byte))
-        case int:
-            inputReader = bytes.NewReader([]byte(""))
-        case *bytes.Buffer:
-            inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
-        case *RateLimitedReader:
-            input.(*RateLimitedReader).Reset()
-            inputReader = input.(*RateLimitedReader)
-        }
+		switch input.(type) {
+		default:
+			jsonInput, err := json.Marshal(input)
+			if err != nil {
+				return nil, 0, "", err
+			}
+			inputReader = bytes.NewReader(jsonInput)
+		case []byte:
+			inputReader = bytes.NewReader(input.([]byte))
+		case int:
+			inputReader = bytes.NewReader([]byte(""))
+		case *bytes.Buffer:
+			inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
+		case *RateLimitedReader:
+			input.(*RateLimitedReader).Reset()
+			inputReader = input.(*RateLimitedReader)
+		}
 
-        request, err := http.NewRequest(method, url, inputReader)
-        if err != nil {
-            return nil, 0, "", err
-        }
+		request, err := http.NewRequest(method, url, inputReader)
+		if err != nil {
+			return nil, 0, "", err
+		}
 
-        if reader, ok := inputReader.(*RateLimitedReader); ok {
-            request.ContentLength = reader.Length()
-        }
+		if reader, ok := inputReader.(*RateLimitedReader); ok {
+			request.ContentLength = reader.Length()
+		}
 
-        if url == HubicCredentialURL {
-            client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
-            client.TokenLock.Unlock()
-        } else if url != HubicRefreshTokenURL {
-            client.CredentialLock.Lock()
-            request.Header.Set("X-Auth-Token", client.Credential.Token)
-            client.CredentialLock.Unlock()
-        }
+		if url == HubicCredentialURL {
+			client.TokenLock.Lock()
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
+			client.TokenLock.Unlock()
+		} else if url != HubicRefreshTokenURL {
+			client.CredentialLock.Lock()
+			request.Header.Set("X-Auth-Token", client.Credential.Token)
+			client.CredentialLock.Unlock()
+		}
 
-        for key, value := range extraHeader {
-            request.Header.Set(key, value)
-        }
+		for key, value := range extraHeader {
+			request.Header.Set(key, value)
+		}
 
-        response, err = client.HTTPClient.Do(request)
-        if err != nil {
-            return nil, 0, "", err
-        }
+		response, err = client.HTTPClient.Do(request)
+		if err != nil {
+			if url != HubicCredentialURL {
+				retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
+				LOG_INFO("HUBIC_CALL", "%s %s returned an error: %v; retry after %d milliseconds", method, url, err, retryAfter)
+				time.Sleep(retryAfter * time.Millisecond)
+				backoff *= 2
+				continue
+			}
+			return nil, 0, "", err
+		}
 
-        contentType := ""
-        if len(response.Header["Content-Type"]) > 0 {
-            contentType = response.Header["Content-Type"][0]
-        }
+		contentType := ""
+		if len(response.Header["Content-Type"]) > 0 {
+			contentType = response.Header["Content-Type"][0]
+		}
 
-        if response.StatusCode < 400 {
-            return response.Body, response.ContentLength, contentType, nil
-        }
+		if response.StatusCode < 400 {
+			return response.Body, response.ContentLength, contentType, nil
+		}
 
-        /*buffer := bytes.NewBufferString("")
-        io.Copy(buffer, response.Body)
-        fmt.Printf("%s\n", buffer.String())*/
+		/*buffer := bytes.NewBufferString("")
+		  io.Copy(buffer, response.Body)
+		  fmt.Printf("%s\n", buffer.String())*/
 
-        response.Body.Close()
+		response.Body.Close()
 
-        if response.StatusCode == 401 {
+		if response.StatusCode == 401 {
 
-            if url == HubicRefreshTokenURL {
-                return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Authorization error when refreshing token"}
-            }
+			if url == HubicRefreshTokenURL {
+				return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Authorization error when refreshing token"}
+			}
 
-            if url == HubicCredentialURL {
-                return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Authorization error when retrieving credentials"}
-            }
+			if url == HubicCredentialURL {
+				return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Authorization error when retrieving credentials"}
+			}
 
-            err = client.RefreshToken()
-            if err != nil {
-                return nil, 0, "", err
-            }
+			err = client.RefreshToken(true)
+			if err != nil {
+				return nil, 0, "", err
+			}
 
-            err = client.GetCredential()
-            if err != nil {
-                return nil, 0, "", err
-            }
-            continue
-        } else if response.StatusCode >= 500 && response.StatusCode < 600 {
-            retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
-            LOG_INFO("HUBIC_RETRY", "Response status: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
-            time.Sleep(retryAfter * time.Millisecond)
-            backoff *= 2
-            continue
-        } else {
-            return nil, 0, "", HubicError { Status: response.StatusCode, Message: "Hubic API error"}
-        }
-    }
+			err = client.GetCredential()
+			if err != nil {
+				return nil, 0, "", err
+			}
+			continue
+		} else if response.StatusCode >= 500 && response.StatusCode < 600 {
+			retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
+			LOG_INFO("HUBIC_RETRY", "Response status: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
+			time.Sleep(retryAfter * time.Millisecond)
+			backoff *= 2
+			continue
+		} else if response.StatusCode == 408 {
+			retryAfter := time.Duration((0.5 + rand.Float32()) * 1000.0 * float32(backoff))
+			LOG_INFO("HUBIC_RETRY", "Response status: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
+			time.Sleep(retryAfter * time.Millisecond)
+			backoff *= 2
+			continue
+		} else {
+			return nil, 0, "", HubicError{Status: response.StatusCode, Message: "Hubic API error"}
+		}
+	}
 
-    return nil, 0, "", fmt.Errorf("Maximum number of retries reached")
+	return nil, 0, "", fmt.Errorf("Maximum number of retries reached")
 }
 
-func (client *HubicClient) RefreshToken() (err error) {
-    client.TokenLock.Lock()
-    defer client.TokenLock.Unlock()
+func (client *HubicClient) RefreshToken(force bool) (err error) {
+	client.TokenLock.Lock()
+	defer client.TokenLock.Unlock()
 
-    if client.Token.Valid() {
-        return nil
-    }
+	if !force && client.Token.Valid() {
+		return nil
+	}
 
-    readCloser, _, _, err := client.call(HubicRefreshTokenURL, "POST", client.Token, nil)
-    if err != nil {
-        return err
-    }
+	readCloser, _, _, err := client.call(HubicRefreshTokenURL, "POST", client.Token, nil)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    if err = json.NewDecoder(readCloser).Decode(&client.Token); err != nil {
-        return err
-    }
+	if err = json.NewDecoder(readCloser).Decode(&client.Token); err != nil {
+		return err
+	}
 
-    description, err := json.Marshal(client.Token)
-    if err != nil {
-        return err
-    }
+	description, err := json.Marshal(client.Token)
+	if err != nil {
+		return err
+	}
 
-    err = ioutil.WriteFile(client.TokenFile, description, 0644)
-    if err != nil {
-        return err
-    }
+	err = ioutil.WriteFile(client.TokenFile, description, 0644)
+	if err != nil {
+		return err
+	}
 
-    return nil
+	return nil
 }
 
 func (client *HubicClient) GetCredential() (err error) {
-    client.CredentialLock.Lock()
-    defer client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	defer client.CredentialLock.Unlock()
 
-    readCloser, _, _, err := client.call(HubicCredentialURL, "GET", 0, nil)
-    if err != nil {
-        return err
-    }
+	readCloser, _, _, err := client.call(HubicCredentialURL, "GET", 0, nil)
+	if err != nil {
+		return err
+	}
 
-    buffer := bytes.NewBufferString("")
-    io.Copy(buffer, readCloser)
-    readCloser.Close()
+	buffer := bytes.NewBufferString("")
+	io.Copy(buffer, readCloser)
+	readCloser.Close()
 
-    if err = json.NewDecoder(buffer).Decode(&client.Credential); err != nil {
-        return fmt.Errorf("%v (response: %s)", err, buffer)
-    }
+	if err = json.NewDecoder(buffer).Decode(&client.Credential); err != nil {
+		return fmt.Errorf("%v (response: %s)", err, buffer)
+	}
 
-    return nil
+	return nil
 }
 
 type HubicEntry struct {
-    Name     string          `json:"name"`
-    Size     int64           `json:"bytes"`
-    Type     string          `json:"content_type"`
-    Subdir   string          `json:"subdir"`
+	Name   string `json:"name"`
+	Size   int64  `json:"bytes"`
+	Type   string `json:"content_type"`
+	Subdir string `json:"subdir"`
 }
 
 func (client *HubicClient) ListEntries(path string) ([]HubicEntry, error) {
 
-    if len(path) > 0 && path[len(path) - 1] != '/' {
-        path += "/"
-    }
+	if len(path) > 0 && path[len(path)-1] != '/' {
+		path += "/"
+	}
 
-    count := 1000
-    if client.TestMode {
-        count = 8
-    }
+	count := 1000
+	if client.TestMode {
+		count = 8
+	}
 
-    marker := ""
+	marker := ""
 
-    var entries []HubicEntry
+	var entries []HubicEntry
 
-    for {
+	for {
 
-        client.CredentialLock.Lock()
-        url := client.Credential.Endpoint + "/default"
-        client.CredentialLock.Unlock()
-        url += fmt.Sprintf("?format=json&limit=%d&delimiter=%%2f", count)
-        if path != "" {
-            url += "&prefix=" + net_url.QueryEscape(path)
-        }
-        if marker != "" {
-            url += "&marker=" + net_url.QueryEscape(marker)
-        }
+		client.CredentialLock.Lock()
+		url := client.Credential.Endpoint + "/default"
+		client.CredentialLock.Unlock()
+		url += fmt.Sprintf("?format=json&limit=%d&delimiter=%%2f", count)
+		if path != "" {
+			url += "&prefix=" + net_url.QueryEscape(path)
+		}
+		if marker != "" {
+			url += "&marker=" + net_url.QueryEscape(marker)
+		}
 
-        readCloser, _, _, err := client.call(url, "GET", 0, nil)
-        if err != nil {
-            return nil, err
-        }
+		readCloser, _, _, err := client.call(url, "GET", 0, nil)
+		if err != nil {
+			return nil, err
+		}
 
-        defer readCloser.Close()
+		defer readCloser.Close()
 
-        var output []HubicEntry
+		var output []HubicEntry
 
-        if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-            return nil, err
-        }
+		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+			return nil, err
+		}
 
-        for _, entry := range output {
-            if entry.Subdir == "" {
-                marker = entry.Name
-            } else {
-                marker = entry.Subdir
-                for len(entry.Subdir) > 0 && entry.Subdir[len(entry.Subdir) - 1] == '/' {
-                    entry.Subdir = entry.Subdir[:len(entry.Subdir) - 1]
-                }
-                entry.Name = entry.Subdir
-                entry.Type = "application/directory"
-            }
-            if path != "" && strings.HasPrefix(entry.Name, path) {
-                entry.Name = entry.Name[len(path):]
-            }
-            entries = append(entries, entry)
-        }
-        if len(output) < count {
-            break
-        }
-    }
+		for _, entry := range output {
+			if entry.Subdir == "" {
+				marker = entry.Name
+			} else {
+				marker = entry.Subdir
+				for len(entry.Subdir) > 0 && entry.Subdir[len(entry.Subdir)-1] == '/' {
+					entry.Subdir = entry.Subdir[:len(entry.Subdir)-1]
+				}
+				entry.Name = entry.Subdir
+				entry.Type = "application/directory"
+			}
+			if path != "" && strings.HasPrefix(entry.Name, path) {
+				entry.Name = entry.Name[len(path):]
+			}
+			entries = append(entries, entry)
+		}
+		if len(output) < count {
+			break
+		}
+	}
 
-    return entries, nil
+	return entries, nil
 }
 
 func (client *HubicClient) GetFileInfo(path string) (bool, bool, int64, error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
-    }
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + path
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + path
+	client.CredentialLock.Unlock()
 
-    readCloser, size, contentType, err := client.call(url, "HEAD", 0, nil)
-    if err != nil {
-        if e, ok := err.(HubicError); ok && e.Status == 404 {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
+	readCloser, size, contentType, err := client.call(url, "HEAD", 0, nil)
+	if err != nil {
+		if e, ok := err.(HubicError); ok && e.Status == 404 {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
-    readCloser.Close()
+	readCloser.Close()
 
-    return true, contentType == "application/directory", size, nil
+	return true, contentType == "application/directory", size, nil
 }
 
 func (client *HubicClient) DownloadFile(path string) (io.ReadCloser, int64, error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
-    }
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + path
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + path
+	client.CredentialLock.Unlock()
 
-    readCloser, size, _, err := client.call(url, "GET", 0, nil)
-    return readCloser, size, err
+	readCloser, size, _, err := client.call(url, "GET", 0, nil)
+	return readCloser, size, err
 }
 
 func (client *HubicClient) UploadFile(path string, content []byte, rateLimit int) (err error) {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
-    }
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + path
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + path
+	client.CredentialLock.Unlock()
 
-    header := make(map[string]string)
-    header["Content-Type"] = "application/octet-stream"
+	header := make(map[string]string)
+	header["Content-Type"] = "application/octet-stream"
 
-    readCloser, _, _, err := client.call(url, "PUT", CreateRateLimitedReader(content, rateLimit), header)
+	readCloser, _, _, err := client.call(url, "PUT", CreateRateLimitedReader(content, rateLimit), header)
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *HubicClient) DeleteFile(path string) error {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
-    }
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + path
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + path
+	client.CredentialLock.Unlock()
 
-    readCloser, _, _, err := client.call(url, "DELETE", 0, nil)
+	readCloser, _, _, err := client.call(url, "DELETE", 0, nil)
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *HubicClient) MoveFile(from string, to string) error {
 
-    for len(from) > 0 && from[len(from) - 1] == '/' {
-        from = from[:len(from) - 1]
-    }
+	for len(from) > 0 && from[len(from)-1] == '/' {
+		from = from[:len(from)-1]
+	}
 
-    for len(to) > 0 && to[len(to) - 1] == '/' {
-        to = to[:len(to) - 1]
-    }
+	for len(to) > 0 && to[len(to)-1] == '/' {
+		to = to[:len(to)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + from
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + from
+	client.CredentialLock.Unlock()
 
-    header := make(map[string]string)
-    header["Destination"] = "default/" + to
+	header := make(map[string]string)
+	header["Destination"] = "default/" + to
 
-    readCloser, _, _, err := client.call(url, "COPY", 0, header)
+	readCloser, _, _, err := client.call(url, "COPY", 0, header)
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
+	readCloser.Close()
 
-    return client.DeleteFile(from)
+	return client.DeleteFile(from)
 }
 
-func (client *HubicClient) CreateDirectory(path string) (error) {
+func (client *HubicClient) CreateDirectory(path string) error {
 
-    for len(path) > 0 && path[len(path) - 1] == '/' {
-        path = path[:len(path) - 1]
-    }
+	for len(path) > 0 && path[len(path)-1] == '/' {
+		path = path[:len(path)-1]
+	}
 
-    client.CredentialLock.Lock()
-    url := client.Credential.Endpoint + "/default/" + path
-    client.CredentialLock.Unlock()
+	client.CredentialLock.Lock()
+	url := client.Credential.Endpoint + "/default/" + path
+	client.CredentialLock.Unlock()
 
-    header := make(map[string]string)
-    header["Content-Type"] = "application/directory"
+	header := make(map[string]string)
+	header["Content-Type"] = "application/directory"
 
-    readCloser, _, _, err := client.call(url, "PUT", "", header)
+	readCloser, _, _, err := client.call(url, "PUT", "", header)
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }

src/duplicacy_hubicclient_test.go

@@ -5,145 +5,145 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
-    "crypto/sha256"
-    "encoding/hex"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
+	crypto_rand "crypto/rand"
+	"math/rand"
 )
 
 func TestHubicClient(t *testing.T) {
 
-    hubicClient, err := NewHubicClient("hubic-token.json")
-    if err != nil {
-        t.Errorf("Failed to create the Hubic client: %v", err)
-        return
-    }
+	hubicClient, err := NewHubicClient("hubic-token.json")
+	if err != nil {
+		t.Errorf("Failed to create the Hubic client: %v", err)
+		return
+	}
 
-    hubicClient.TestMode = true
+	hubicClient.TestMode = true
 
-    existingFiles, err := hubicClient.ListEntries("")
-    for _, file := range existingFiles {
-        fmt.Printf("name: %s, isDir: %t\n", file.Name, file.Type == "application/directory")
-    }
+	existingFiles, err := hubicClient.ListEntries("")
+	for _, file := range existingFiles {
+		fmt.Printf("name: %s, isDir: %t\n", file.Name, file.Type == "application/directory")
+	}
 
-    testExists, _, _, err := hubicClient.GetFileInfo("test")
-    if err != nil {
-        t.Errorf("Failed to list the test directory: %v", err)
-        return
-    }
-    if !testExists {
-        err = hubicClient.CreateDirectory("test")
-        if err != nil {
-            t.Errorf("Failed to create the test directory: %v", err)
-            return
-        }
-    }
+	testExists, _, _, err := hubicClient.GetFileInfo("test")
+	if err != nil {
+		t.Errorf("Failed to list the test directory: %v", err)
+		return
+	}
+	if !testExists {
+		err = hubicClient.CreateDirectory("test")
+		if err != nil {
+			t.Errorf("Failed to create the test directory: %v", err)
+			return
+		}
+	}
 
-    test1Exists, _, _, err := hubicClient.GetFileInfo("test/test1")
-    if err != nil {
-        t.Errorf("Failed to list the test1 directory: %v", err)
-        return
-    }
-    if !test1Exists {
-        err = hubicClient.CreateDirectory("test/test1")
-        if err != nil {
-            t.Errorf("Failed to create the test1 directory: %v", err)
-            return
-        }
-    }
+	test1Exists, _, _, err := hubicClient.GetFileInfo("test/test1")
+	if err != nil {
+		t.Errorf("Failed to list the test1 directory: %v", err)
+		return
+	}
+	if !test1Exists {
+		err = hubicClient.CreateDirectory("test/test1")
+		if err != nil {
+			t.Errorf("Failed to create the test1 directory: %v", err)
+			return
+		}
+	}
 
-    test2Exists, _, _, err := hubicClient.GetFileInfo("test/test2")
-    if err != nil {
-        t.Errorf("Failed to list the test2 directory: %v", err)
-        return
-    }
-    if !test2Exists {
-        err = hubicClient.CreateDirectory("test/test2")
-        if err != nil {
-            t.Errorf("Failed to create the test2 directory: %v", err)
-            return
-        }
-    }
+	test2Exists, _, _, err := hubicClient.GetFileInfo("test/test2")
+	if err != nil {
+		t.Errorf("Failed to list the test2 directory: %v", err)
+		return
+	}
+	if !test2Exists {
+		err = hubicClient.CreateDirectory("test/test2")
+		if err != nil {
+			t.Errorf("Failed to create the test2 directory: %v", err)
+			return
+		}
+	}
 
-    numberOfFiles := 20
-    maxFileSize := 64 * 1024
+	numberOfFiles := 20
+	maxFileSize := 64 * 1024
 
-    for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+	for i := 0; i < numberOfFiles; i++ {
+		content := make([]byte, rand.Int()%maxFileSize+1)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-        hasher := sha256.New()
-        hasher.Write(content)
-        filename := hex.EncodeToString(hasher.Sum(nil))
+		hasher := sha256.New()
+		hasher.Write(content)
+		filename := hex.EncodeToString(hasher.Sum(nil))
 
-        fmt.Printf("file: %s\n", filename)
+		fmt.Printf("file: %s\n", filename)
 
-        err = hubicClient.UploadFile("test/test1/" + filename, content, 100)
-        if err != nil {
-            /*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
-                t.Errorf("Failed to upload the file %s: %v", filename, err)
-                return
-            }
-        }
-    }
+		err = hubicClient.UploadFile("test/test1/"+filename, content, 100)
+		if err != nil {
+			/*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
+				t.Errorf("Failed to upload the file %s: %v", filename, err)
+				return
+			}
+		}
+	}
 
-    entries, err := hubicClient.ListEntries("test/test1")
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	entries, err := hubicClient.ListEntries("test/test1")
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    for _, entry := range entries {
+	for _, entry := range entries {
 
-        exists, isDir, size, err := hubicClient.GetFileInfo("test/test1/" + entry.Name)
-        fmt.Printf("%s exists: %t, isDir: %t, size: %d, err: %v\n", "test/test1/" + entry.Name, exists, isDir, size, err)
+		exists, isDir, size, err := hubicClient.GetFileInfo("test/test1/" + entry.Name)
+		fmt.Printf("%s exists: %t, isDir: %t, size: %d, err: %v\n", "test/test1/"+entry.Name, exists, isDir, size, err)
 
-        err = hubicClient.MoveFile("test/test1/" + entry.Name, "test/test2/" + entry.Name)
-        if err != nil {
-            t.Errorf("Failed to move %s: %v", entry.Name, err)
-            return
-        }
-    }
+		err = hubicClient.MoveFile("test/test1/"+entry.Name, "test/test2/"+entry.Name)
+		if err != nil {
+			t.Errorf("Failed to move %s: %v", entry.Name, err)
+			return
+		}
+	}
 
-    entries, err = hubicClient.ListEntries("test/test2")
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	entries, err = hubicClient.ListEntries("test/test2")
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    for _, entry := range entries {
-        readCloser, _, err := hubicClient.DownloadFile("test/test2/" + entry.Name)
-        if err != nil {
-            t.Errorf("Error downloading file %s: %v", entry.Name, err)
-            return
-        }
+	for _, entry := range entries {
+		readCloser, _, err := hubicClient.DownloadFile("test/test2/" + entry.Name)
+		if err != nil {
+			t.Errorf("Error downloading file %s: %v", entry.Name, err)
+			return
+		}
 
-        hasher := sha256.New()
-        io.Copy(hasher, readCloser)
-        hash := hex.EncodeToString(hasher.Sum(nil))
+		hasher := sha256.New()
+		io.Copy(hasher, readCloser)
+		hash := hex.EncodeToString(hasher.Sum(nil))
 
-        if hash != entry.Name {
-            t.Errorf("File %s, hash %s", entry.Name, hash)
-        }
+		if hash != entry.Name {
+			t.Errorf("File %s, hash %s", entry.Name, hash)
+		}
 
-        readCloser.Close()
-    }
+		readCloser.Close()
+	}
 
-    for _, entry := range entries {
+	for _, entry := range entries {
 
-        err = hubicClient.DeleteFile("test/test2/" + entry.Name)
-        if err != nil {
-            t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
-            return
-        }
-    }
+		err = hubicClient.DeleteFile("test/test2/" + entry.Name)
+		if err != nil {
+			t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
+			return
+		}
+	}
 
 }

src/duplicacy_hubicstorage.go

@@ -5,203 +5,194 @@
 package duplicacy
 
 import (
-    "fmt"
-    "strings"
+	"fmt"
+	"strings"
 )
 
 type HubicStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    client *HubicClient
-    storageDir string
-    numberOfThreads int
+	client          *HubicClient
+	storageDir      string
+	numberOfThreads int
 }
 
 // CreateHubicStorage creates an Hubic storage object.
 func CreateHubicStorage(tokenFile string, storagePath string, threads int) (storage *HubicStorage, err error) {
 
-    for len(storagePath) > 0 && storagePath[len(storagePath) - 1] == '/' {
-        storagePath = storagePath[:len(storagePath) - 1]
-    }
+	for len(storagePath) > 0 && storagePath[len(storagePath)-1] == '/' {
+		storagePath = storagePath[:len(storagePath)-1]
+	}
 
-    client, err := NewHubicClient(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	client, err := NewHubicClient(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    exists, isDir, _, err := client.GetFileInfo(storagePath)
-    if err != nil {
-        return nil, err
-    }
+	exists, isDir, _, err := client.GetFileInfo(storagePath)
+	if err != nil {
+		return nil, err
+	}
 
-    if !exists {
-        return nil, fmt.Errorf("Path '%s' doesn't exist", storagePath)
-    }
+	if !exists {
+		return nil, fmt.Errorf("Path '%s' doesn't exist", storagePath)
+	}
 
-    if !isDir {
-        return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
-    }
+	if !isDir {
+		return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
+	}
 
-    storage = &HubicStorage {
-        client: client,
-        storageDir: storagePath,
-        numberOfThreads: threads,
-    }
+	storage = &HubicStorage{
+		client:          client,
+		storageDir:      storagePath,
+		numberOfThreads: threads,
+	}
 
-    for _, path := range []string { "chunks", "snapshots" } {
-        dir := storagePath + "/" + path
-        exists, isDir, _, err := client.GetFileInfo(dir)
-        if err != nil {
-            return nil, err
-        }
-        if !exists {
-            err = client.CreateDirectory(storagePath + "/" + path)
-            if err != nil {
-                return nil, err
-            }
-        } else if !isDir {
-            return nil, fmt.Errorf("%s is not a directory", dir)
-        }
-    }
-
-    return storage, nil
+	for _, path := range []string{"chunks", "snapshots"} {
+		dir := storagePath + "/" + path
+		exists, isDir, _, err := client.GetFileInfo(dir)
+		if err != nil {
+			return nil, err
+		}
+		if !exists {
+			err = client.CreateDirectory(storagePath + "/" + path)
+			if err != nil {
+				return nil, err
+			}
+		} else if !isDir {
+			return nil, fmt.Errorf("%s is not a directory", dir)
+		}
+	}
 
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *HubicStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    if dir == "snapshots" {
-        entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
-        if err != nil {
-            return nil, nil, err
-        }
+	if dir == "snapshots" {
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        subDirs := []string{}
-        for _, entry := range entries {
-            if entry.Type == "application/directory" {
-                subDirs = append(subDirs, entry.Name + "/")
-            }
-        }
-        return subDirs, nil, nil
-    } else if strings.HasPrefix(dir, "snapshots/") {
-        entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
-        if err != nil {
-            return nil, nil, err
-        }
+		subDirs := []string{}
+		for _, entry := range entries {
+			if entry.Type == "application/directory" {
+				subDirs = append(subDirs, entry.Name+"/")
+			}
+		}
+		return subDirs, nil, nil
+	} else if strings.HasPrefix(dir, "snapshots/") {
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        files := []string{}
+		files := []string{}
 
-        for _, entry := range entries {
-            if entry.Type == "application/directory" {
-                continue
-            }
-            files = append(files, entry.Name)
-        }
-        return files, nil, nil
-    } else {
-        files := []string{}
-        sizes := []int64{}
-        entries, err := storage.client.ListEntries(storage.storageDir + "/chunks")
-        if err != nil {
-            return nil, nil, err
-        }
+		for _, entry := range entries {
+			if entry.Type == "application/directory" {
+				continue
+			}
+			files = append(files, entry.Name)
+		}
+		return files, nil, nil
+	} else {
+		files := []string{}
+		sizes := []int64{}
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        for _, entry := range entries {
-            if entry.Type == "application/directory" {
-                continue
-            }
-            files = append(files, entry.Name)
-            sizes = append(sizes, entry.Size)
-        }
-        return files, sizes, nil
-    }
+		for _, entry := range entries {
+			if entry.Type == "application/directory" {
+				files = append(files, entry.Name+"/")
+				sizes = append(sizes, 0)
+			} else {
+				files = append(files, entry.Name)
+				sizes = append(sizes, entry.Size)
+			}
+		}
+		return files, sizes, nil
+	}
 
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *HubicStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    err = storage.client.DeleteFile(storage.storageDir + "/" + filePath)
-    if e, ok := err.(HubicError); ok && e.Status == 404 {
-        LOG_DEBUG("HUBIC_DELETE", "Ignore 404 error")
-        return nil
-    }
-    return err
+	err = storage.client.DeleteFile(storage.storageDir + "/" + filePath)
+	if e, ok := err.(HubicError); ok && e.Status == 404 {
+		LOG_DEBUG("HUBIC_DELETE", "Ignore 404 error")
+		return nil
+	}
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *HubicStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    fromPath := storage.storageDir + "/" + from
-    toPath := storage.storageDir + "/" + to
+	fromPath := storage.storageDir + "/" + from
+	toPath := storage.storageDir + "/" + to
 
-    return storage.client.MoveFile(fromPath, toPath)
+	return storage.client.MoveFile(fromPath, toPath)
 }
 
 // CreateDirectory creates a new directory.
 func (storage *HubicStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    return storage.client.CreateDirectory(storage.storageDir + "/" + dir)
+	return storage.client.CreateDirectory(storage.storageDir + "/" + dir)
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *HubicStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
-    }
-    return storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *HubicStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
-    return filePath, exist, size, err
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
+	}
+	return storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *HubicStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThreads)
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThreads)
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *HubicStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    return storage.client.UploadFile(storage.storageDir + "/" + filePath, content, storage.UploadRateLimit / storage.numberOfThreads)
+	return storage.client.UploadFile(storage.storageDir+"/"+filePath, content, storage.UploadRateLimit/storage.numberOfThreads)
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *HubicStorage) IsCacheNeeded() (bool) { return true }
+func (storage *HubicStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *HubicStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *HubicStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *HubicStorage) IsStrongConsistent() (bool) { return false }
+func (storage *HubicStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *HubicStorage) IsFastListing() (bool) { return true }
+func (storage *HubicStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *HubicStorage) EnableTestMode() {
-    storage.client.TestMode = true
+	storage.client.TestMode = true
 }

src/duplicacy_keyring.go

@@ -7,24 +7,24 @@
 package duplicacy
 
 import (
-    "github.com/gilbertchen/keyring"
+	"github.com/gilbertchen/keyring"
 )
 
 func SetKeyringFile(path string) {
-    // We only use keyring file on Windows
+	// We only use keyring file on Windows
 }
 
 func keyringGet(key string) (value string) {
-    value, err := keyring.Get("duplicacy", key)
-    if err != nil {
-        LOG_DEBUG("KEYRING_GET", "Failed to get the value from the keyring: %v", err)
-    }
-    return value
+	value, err := keyring.Get("duplicacy", key)
+	if err != nil {
+		LOG_DEBUG("KEYRING_GET", "Failed to get the value from the keyring: %v", err)
+	}
+	return value
 }
 
 func keyringSet(key string, value string) {
-    err := keyring.Set("duplicacy", key, value)
-    if err != nil {
-        LOG_DEBUG("KEYRING_GET", "Failed to store the value to the keyring: %v", err)
-    }
+	err := keyring.Set("duplicacy", key, value)
+	if err != nil {
+		LOG_DEBUG("KEYRING_GET", "Failed to store the value to the keyring: %v", err)
+	}
 }

src/duplicacy_keyring_windows.go

@@ -5,156 +5,166 @@
 package duplicacy
 
 import (
-    "syscall"
-    "unsafe"
-    "io/ioutil"
-    "encoding/json"
+	"encoding/json"
+	"io/ioutil"
+	"syscall"
+	"unsafe"
 )
 
 var keyringFile string
 
 var (
-    dllcrypt32  = syscall.NewLazyDLL("Crypt32.dll")
-    dllkernel32 = syscall.NewLazyDLL("Kernel32.dll")
+	dllcrypt32  = syscall.NewLazyDLL("Crypt32.dll")
+	dllkernel32 = syscall.NewLazyDLL("Kernel32.dll")
 
-    procEncryptData = dllcrypt32.NewProc("CryptProtectData")
-    procDecryptData = dllcrypt32.NewProc("CryptUnprotectData")
-    procLocalFree   = dllkernel32.NewProc("LocalFree")
+	procEncryptData = dllcrypt32.NewProc("CryptProtectData")
+	procDecryptData = dllcrypt32.NewProc("CryptUnprotectData")
+	procLocalFree   = dllkernel32.NewProc("LocalFree")
 )
 
 type DATA_BLOB struct {
-    cbData uint32
-    pbData *byte
+	cbData uint32
+	pbData *byte
 }
 
 func SetKeyringFile(path string) {
-    keyringFile = path
+	keyringFile = path
 }
 
 func keyringEncrypt(value []byte) ([]byte, error) {
 
-    dataIn := DATA_BLOB {
-        pbData: &value[0],
-        cbData: uint32(len(value)),
-    }
-    dataOut := DATA_BLOB {}
+	dataIn := DATA_BLOB{
+		pbData: &value[0],
+		cbData: uint32(len(value)),
+	}
+	dataOut := DATA_BLOB{}
 
-    r, _, err := procEncryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
-                                      0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))
-    if r == 0 {
-        return nil, err
-    }
+	r, _, err := procEncryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
+		0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))
+	if r == 0 {
+		return nil, err
+	}
 
-    address := uintptr(unsafe.Pointer(dataOut.pbData))
-    defer procLocalFree.Call(address)
+	address := uintptr(unsafe.Pointer(dataOut.pbData))
+	defer procLocalFree.Call(address)
 
-    encryptedData := make([]byte, dataOut.cbData)
-    for i := 0; i < len(encryptedData); i++ {
-        encryptedData[i] = *(*byte)(unsafe.Pointer(uintptr(int(address) + i)))
-    }
-    return encryptedData, nil
+	encryptedData := make([]byte, dataOut.cbData)
+	for i := 0; i < len(encryptedData); i++ {
+		encryptedData[i] = *(*byte)(unsafe.Pointer(uintptr(int(address) + i)))
+	}
+	return encryptedData, nil
 }
 
 func keyringDecrypt(value []byte) ([]byte, error) {
 
-    dataIn := DATA_BLOB {
-        pbData: &value[0],
-        cbData: uint32(len(value)),
-    }
-    dataOut := DATA_BLOB {}
+	dataIn := DATA_BLOB{
+		pbData: &value[0],
+		cbData: uint32(len(value)),
+	}
+	dataOut := DATA_BLOB{}
 
-    r, _, err := procDecryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
-                                      0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))
-    if r == 0 {
-        return nil, err
-    }
+	r, _, err := procDecryptData.Call(uintptr(unsafe.Pointer(&dataIn)),
+		0, 0, 0, 0, 0, uintptr(unsafe.Pointer(&dataOut)))
+	if r == 0 {
+		return nil, err
+	}
 
-    address := uintptr(unsafe.Pointer(dataOut.pbData))
-    defer procLocalFree.Call(address)
+	address := uintptr(unsafe.Pointer(dataOut.pbData))
+	defer procLocalFree.Call(address)
 
-    decryptedData := make([]byte, dataOut.cbData)
-    for i := 0; i < len(decryptedData); i++ {
-        address := int(uintptr(unsafe.Pointer(dataOut.pbData)))
-        decryptedData[i] = *(*byte)(unsafe.Pointer(uintptr(int(address) + i)))
-    }
-    return decryptedData, nil
+	decryptedData := make([]byte, dataOut.cbData)
+	for i := 0; i < len(decryptedData); i++ {
+		address := int(uintptr(unsafe.Pointer(dataOut.pbData)))
+		decryptedData[i] = *(*byte)(unsafe.Pointer(uintptr(int(address) + i)))
+	}
+	return decryptedData, nil
 }
 
 func keyringGet(key string) (value string) {
-    if keyringFile == "" {
-        LOG_DEBUG("KEYRING_NOT_INITIALIZED", "Keyring file not set")
-        return ""
-    }
+	if keyringFile == "" {
+		LOG_DEBUG("KEYRING_NOT_INITIALIZED", "Keyring file not set")
+		return ""
+	}
 
-    description, err := ioutil.ReadFile(keyringFile)
-    if err != nil {
-        LOG_DEBUG("KEYRING_READ", "Keyring file not read: %v", err)
-        return ""
-    }
+	description, err := ioutil.ReadFile(keyringFile)
+	if err != nil {
+		LOG_DEBUG("KEYRING_READ", "Keyring file not read: %v", err)
+		return ""
+	}
 
-    var keyring map[string][]byte
-    err = json.Unmarshal(description, &keyring)
-    if err != nil {
-        LOG_DEBUG("KEYRING_PARSE", "Failed to parse the keyring storage file %s: %v", keyringFile, err)
-        return ""
-    }
+	var keyring map[string][]byte
+	err = json.Unmarshal(description, &keyring)
+	if err != nil {
+		LOG_DEBUG("KEYRING_PARSE", "Failed to parse the keyring storage file %s: %v", keyringFile, err)
+		return ""
+	}
 
-    encryptedValue := keyring[key]
+	encryptedValue := keyring[key]
 
-    if len(encryptedValue) == 0 {
-        return ""
-    }
+	if len(encryptedValue) == 0 {
+		return ""
+	}
 
-    valueInBytes, err := keyringDecrypt(encryptedValue)
-    if err != nil {
-        LOG_DEBUG("KEYRING_DECRYPT", "Failed to decrypt the value: %v", err)
-        return ""
-    }
+	valueInBytes, err := keyringDecrypt(encryptedValue)
+	if err != nil {
+		LOG_DEBUG("KEYRING_DECRYPT", "Failed to decrypt the value: %v", err)
+		return ""
+	}
 
-    return string(valueInBytes)
+	return string(valueInBytes)
 }
 
 func keyringSet(key string, value string) bool {
-    if value == "" {
-        return false
-    }
-    if keyringFile == "" {
-        LOG_DEBUG("KEYRING_NOT_INITIALIZED", "Keyring file not set")
-        return false
-    }
+	if value == "" {
+		return false
+	}
+	if keyringFile == "" {
+		LOG_DEBUG("KEYRING_NOT_INITIALIZED", "Keyring file not set")
+		return false
+	}
 
-    keyring := make(map[string][]byte)
+	keyring := make(map[string][]byte)
 
-    description, err := ioutil.ReadFile(keyringFile)
-    if err == nil {
-        err = json.Unmarshal(description, &keyring)
-        if err != nil {
-            LOG_DEBUG("KEYRING_PARSE", "Failed to parse the keyring storage file %s: %v", keyringFile, err)
-        }
-    }
+	description, err := ioutil.ReadFile(keyringFile)
+	if err == nil {
+		err = json.Unmarshal(description, &keyring)
+		if err != nil {
+			LOG_DEBUG("KEYRING_PARSE", "Failed to parse the keyring storage file %s: %v", keyringFile, err)
+		}
+	}
 
-    if value == "" {
-        keyring[key] = nil
-    } else {
-        encryptedValue, err := keyringEncrypt([]byte(value))
-        if err != nil {
-            LOG_DEBUG("KEYRING_ENCRYPT", "Failed to encrypt the value: %v", err)
-            return false
-        }
-        keyring[key] = encryptedValue
-    }
+	if value == "" {
+		keyring[key] = nil
+	} else {
 
-    description, err = json.MarshalIndent(keyring, "", "    ")
-    if err != nil {
-        LOG_DEBUG("KEYRING_MARSHAL", "Failed to marshal the keyring storage: %v", err)
-        return false
-    }
+		// Check if the value to be set is the same as the existing one
+		existingEncryptedValue := keyring[key]
+		if len(existingEncryptedValue) > 0 {
+			existingValue, err := keyringDecrypt(existingEncryptedValue)
+			if err == nil && string(existingValue) == value {
+				return true
+			}
+		}
 
-    err = ioutil.WriteFile(keyringFile, description, 0600)
-    if err != nil {
-        LOG_DEBUG("KEYRING_WRITE", "Failed to save the keyring storage to file %s: %v", keyringFile, err)
-        return false
-    }
+		encryptedValue, err := keyringEncrypt([]byte(value))
+		if err != nil {
+			LOG_DEBUG("KEYRING_ENCRYPT", "Failed to encrypt the value: %v", err)
+			return false
+		}
+		keyring[key] = encryptedValue
+	}
 
-    return true
+	description, err = json.MarshalIndent(keyring, "", "    ")
+	if err != nil {
+		LOG_DEBUG("KEYRING_MARSHAL", "Failed to marshal the keyring storage: %v", err)
+		return false
+	}
+
+	err = ioutil.WriteFile(keyringFile, description, 0600)
+	if err != nil {
+		LOG_DEBUG("KEYRING_WRITE", "Failed to save the keyring storage to file %s: %v", keyringFile, err)
+		return false
+	}
+
+	return true
 }

src/duplicacy_log.go

@@ -5,178 +5,184 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "time"
-    "sync"
-    "testing"
-    "runtime/debug"
+	"fmt"
+	"os"
+	"runtime/debug"
+	"sync"
+	"testing"
+	"time"
 )
 
 const (
-    DEBUG  = -2
-    TRACE  = -1
-    INFO   = 0
-    WARN = 1
-    ERROR = 2
-    FATAL = 3
-    ASSERT = 4
+	DEBUG  = -2
+	TRACE  = -1
+	INFO   = 0
+	WARN   = 1
+	ERROR  = 2
+	FATAL  = 3
+	ASSERT = 4
 )
 
+var LogFunction func(level int, logID string, message string)
 
 var printLogHeader = false
 
 func EnableLogHeader() {
-    printLogHeader = true
+	printLogHeader = true
 }
 
 var printStackTrace = false
 
 func EnableStackTrace() {
-    printStackTrace = true
+	printStackTrace = true
 }
 
 var testingT *testing.T
 
 func setTestingT(t *testing.T) {
-    testingT = t
+	testingT = t
 }
 
 func getLevelName(level int) string {
-    switch level {
-    case DEBUG:
-        return "DEBUG"
-    case TRACE:
-        return "TRACE"
-    case INFO:
-        return "INFO"
-    case WARN:
-        return "WARN"
-    case ERROR:
-        return "ERROR"
-    case FATAL:
-        return "FATAL"
-    case ASSERT:
-        return "ASSERT"
-    default:
-        return fmt.Sprintf("[%d]", level)
-    }
+	switch level {
+	case DEBUG:
+		return "DEBUG"
+	case TRACE:
+		return "TRACE"
+	case INFO:
+		return "INFO"
+	case WARN:
+		return "WARN"
+	case ERROR:
+		return "ERROR"
+	case FATAL:
+		return "FATAL"
+	case ASSERT:
+		return "ASSERT"
+	default:
+		return fmt.Sprintf("[%d]", level)
+	}
 }
 
 var loggingLevel int
 
 func IsDebugging() bool {
-    return loggingLevel <= DEBUG
+	return loggingLevel <= DEBUG
 }
 
 func IsTracing() bool {
-    return loggingLevel <= TRACE
+	return loggingLevel <= TRACE
 }
 
 func SetLoggingLevel(level int) {
-    loggingLevel = level
+	loggingLevel = level
 }
 
 func LOG_DEBUG(logID string, format string, v ...interface{}) {
-    logf(DEBUG, logID, format, v...)
+	logf(DEBUG, logID, format, v...)
 }
 
 func LOG_TRACE(logID string, format string, v ...interface{}) {
-    logf(TRACE, logID, format, v...)
+	logf(TRACE, logID, format, v...)
 }
 
 func LOG_INFO(logID string, format string, v ...interface{}) {
-    logf(INFO, logID, format, v...)
+	logf(INFO, logID, format, v...)
 }
 
 func LOG_WARN(logID string, format string, v ...interface{}) {
-    logf(WARN, logID, format, v...)
+	logf(WARN, logID, format, v...)
 }
 
 func LOG_ERROR(logID string, format string, v ...interface{}) {
-    logf(ERROR, logID, format, v...)
+	logf(ERROR, logID, format, v...)
 }
 
 func LOG_FATAL(logID string, format string, v ...interface{}) {
-    logf(FATAL, logID, format, v...)
+	logf(FATAL, logID, format, v...)
 }
 
 func LOG_ASSERT(logID string, format string, v ...interface{}) {
-    logf(ASSERT, logID, format, v...)
+	logf(ASSERT, logID, format, v...)
 }
 
 type Exception struct {
-    Level int
-    LogID string
-    Message string
+	Level   int
+	LogID   string
+	Message string
 }
 
 var logMutex sync.Mutex
 
 func logf(level int, logID string, format string, v ...interface{}) {
 
-    message := fmt.Sprintf(format, v...)
+	message := fmt.Sprintf(format, v...)
 
-    now := time.Now()
+	if LogFunction != nil {
+		LogFunction(level, logID, message)
+		return
+	}
 
-    // Uncomment this line to enable unbufferred logging for tests
-    // fmt.Printf("%s %s %s %s\n", now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
+	now := time.Now()
 
-    if testingT != nil {
-        if level < WARN {
-            if level >= loggingLevel {
-                testingT.Logf("%s %s %s %s\n",
-                              now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
-            }
-        } else {
-            testingT.Errorf("%s %s %s %s\n",
-                            now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
-        }
-    } else {
-        logMutex.Lock()
-        defer logMutex.Unlock()
+	// Uncomment this line to enable unbufferred logging for tests
+	// fmt.Printf("%s %s %s %s\n", now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
 
-        if level >= loggingLevel {
-            if printLogHeader {
-                fmt.Printf("%s %s %s %s\n",
-                           now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
-            } else {
-                fmt.Printf("%s\n", message)
-            }
-        }
-    }
+	if testingT != nil {
+		if level <= WARN {
+			if level >= loggingLevel {
+				testingT.Logf("%s %s %s %s\n",
+					now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
+			}
+		} else {
+			testingT.Errorf("%s %s %s %s\n",
+				now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
+		}
+	} else {
+		logMutex.Lock()
+		defer logMutex.Unlock()
 
-    if level > WARN {
-        panic(Exception{
-            Level: level,
-            LogID: logID,
-            Message: message,
-        })
-    }
+		if level >= loggingLevel {
+			if printLogHeader {
+				fmt.Printf("%s %s %s %s\n",
+					now.Format("2006-01-02 15:04:05.000"), getLevelName(level), logID, message)
+			} else {
+				fmt.Printf("%s\n", message)
+			}
+		}
+	}
+
+	if level > WARN {
+		panic(Exception{
+			Level:   level,
+			LogID:   logID,
+			Message: message,
+		})
+	}
 }
 
 const (
-    duplicacyExitCode = 100
-    otherExitCode     = 101
+	duplicacyExitCode = 100
+	otherExitCode     = 101
 )
 
 // This is the function to be called before exiting when an error occurs.
 var RunAtError func() = func() {}
 
 func CatchLogException() {
-    if r := recover(); r != nil {
-        switch e := r.(type) {
-        case Exception:
-            if printStackTrace {
-                debug.PrintStack()
-            }
-            RunAtError()
-            os.Exit(duplicacyExitCode)
-        default:
-            fmt.Fprintf(os.Stderr, "%v\n", e)
-            debug.PrintStack()
-            RunAtError()
-            os.Exit(otherExitCode)
-        }
-    }
+	if r := recover(); r != nil {
+		switch e := r.(type) {
+		case Exception:
+			if printStackTrace {
+				debug.PrintStack()
+			}
+			RunAtError()
+			os.Exit(duplicacyExitCode)
+		default:
+			fmt.Fprintf(os.Stderr, "%v\n", e)
+			debug.PrintStack()
+			RunAtError()
+			os.Exit(otherExitCode)
+		}
+	}
 }

src/duplicacy_oneclient.go

@@ -5,372 +5,374 @@
 package duplicacy
 
 import (
-    "fmt"
-    "time"
-    "sync"
-    "bytes"
-    "strings"
-    "io/ioutil"
-    "encoding/json"
-    "io"
-    "net/http"
-    "math/rand"
+	"bytes"
+	"encoding/json"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net/http"
+	"strings"
+	"sync"
+	"time"
 
-    "golang.org/x/oauth2"
+	"golang.org/x/oauth2"
 )
 
 type OneDriveError struct {
-    Status     int
-    Message    string
+	Status  int
+	Message string
 }
 
 func (err OneDriveError) Error() string {
-    return fmt.Sprintf("%d %s", err.Status, err.Message)
+	return fmt.Sprintf("%d %s", err.Status, err.Message)
 }
 
 type OneDriveErrorResponse struct {
-    Error OneDriveError   `json:"error"`
+	Error OneDriveError `json:"error"`
 }
 
 var OneDriveRefreshTokenURL = "https://duplicacy.com/one_refresh"
 var OneDriveAPIURL = "https://api.onedrive.com/v1.0"
 
 type OneDriveClient struct {
-    HTTPClient          *http.Client
+	HTTPClient *http.Client
 
-    TokenFile           string
-    Token               *oauth2.Token
-    TokenLock           *sync.Mutex
+	TokenFile string
+	Token     *oauth2.Token
+	TokenLock *sync.Mutex
 
-    IsConnected         bool
-    TestMode            bool
+	IsConnected bool
+	TestMode    bool
 }
 
 func NewOneDriveClient(tokenFile string) (*OneDriveClient, error) {
 
-    description, err := ioutil.ReadFile(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	description, err := ioutil.ReadFile(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    token := new(oauth2.Token)
-    if err := json.Unmarshal(description, token); err != nil {
-        return nil, err
-    }
+	token := new(oauth2.Token)
+	if err := json.Unmarshal(description, token); err != nil {
+		return nil, err
+	}
 
-    client := &OneDriveClient{
-        HTTPClient: http.DefaultClient,
-        TokenFile: tokenFile,
-        Token: token,
-        TokenLock: &sync.Mutex{},
-    }
+	client := &OneDriveClient{
+		HTTPClient: http.DefaultClient,
+		TokenFile:  tokenFile,
+		Token:      token,
+		TokenLock:  &sync.Mutex{},
+	}
 
-    return client, nil
+	client.RefreshToken(false)
+
+	return client, nil
 }
 
 func (client *OneDriveClient) call(url string, method string, input interface{}, contentType string) (io.ReadCloser, int64, error) {
 
-    var response *http.Response
+	var response *http.Response
 
-    backoff := 1
-    for i := 0; i < 8; i++ {
+	backoff := 1
+	for i := 0; i < 8; i++ {
 
-        LOG_DEBUG("ONEDRIVE_CALL", "%s %s", method, url)
+		LOG_DEBUG("ONEDRIVE_CALL", "%s %s", method, url)
 
-        var inputReader io.Reader
+		var inputReader io.Reader
 
-        switch input.(type) {
-        default:
-            jsonInput, err := json.Marshal(input)
-            if err != nil {
-                return nil, 0, err
-            }
-            inputReader = bytes.NewReader(jsonInput)
-        case []byte:
-            inputReader = bytes.NewReader(input.([]byte))
-        case int:
-            inputReader = bytes.NewReader([]byte(""))
-        case *bytes.Buffer:
-            inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
-        case *RateLimitedReader:
-            input.(*RateLimitedReader).Reset()
-            inputReader = input.(*RateLimitedReader)
-        }
+		switch input.(type) {
+		default:
+			jsonInput, err := json.Marshal(input)
+			if err != nil {
+				return nil, 0, err
+			}
+			inputReader = bytes.NewReader(jsonInput)
+		case []byte:
+			inputReader = bytes.NewReader(input.([]byte))
+		case int:
+			inputReader = nil
+		case *bytes.Buffer:
+			inputReader = bytes.NewReader(input.(*bytes.Buffer).Bytes())
+		case *RateLimitedReader:
+			input.(*RateLimitedReader).Reset()
+			inputReader = input.(*RateLimitedReader)
+		}
 
-        request, err := http.NewRequest(method, url, inputReader)
-        if err != nil {
-            return nil, 0, err
-        }
+		request, err := http.NewRequest(method, url, inputReader)
+		if err != nil {
+			return nil, 0, err
+		}
 
-        if reader, ok := inputReader.(*RateLimitedReader); ok {
-            request.ContentLength = reader.Length()
-        }
+		if reader, ok := inputReader.(*RateLimitedReader); ok {
+			request.ContentLength = reader.Length()
+		}
 
-        if url != OneDriveRefreshTokenURL {
-            client.TokenLock.Lock()
-            request.Header.Set("Authorization", "Bearer " + client.Token.AccessToken)
-            client.TokenLock.Unlock()
-        }
-        if contentType != "" {
-            request.Header.Set("Content-Type", contentType)
-        }
+		if url != OneDriveRefreshTokenURL {
+			client.TokenLock.Lock()
+			request.Header.Set("Authorization", "Bearer "+client.Token.AccessToken)
+			client.TokenLock.Unlock()
+		}
+		if contentType != "" {
+			request.Header.Set("Content-Type", contentType)
+		}
 
-        response, err = client.HTTPClient.Do(request)
-        if err != nil {
-            if client.IsConnected {
-                if strings.Contains(err.Error(), "TLS handshake timeout") {
-                    // Give a long timeout regardless of backoff when a TLS timeout happens, hoping that
-                    // idle connections are not to be reused on reconnect.
-                    retryAfter := time.Duration(rand.Float32() * 60000 + 180000)
-                    LOG_INFO("ONEDRIVE_RETRY", "TLS handshake timeout; retry after %d milliseconds", retryAfter)
-                    time.Sleep(retryAfter * time.Millisecond)
-                } else {
-                    // For all other errors just blindly retry until the maximum is reached
-                    retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
-                    LOG_INFO("ONEDRIVE_RETRY", "%v; retry after %d milliseconds", err, retryAfter)
-                    time.Sleep(retryAfter * time.Millisecond)
-                }
-                backoff *= 2
-                continue
-            }
-            return nil, 0, err
-        }
+		response, err = client.HTTPClient.Do(request)
+		if err != nil {
+			if client.IsConnected {
+				if strings.Contains(err.Error(), "TLS handshake timeout") {
+					// Give a long timeout regardless of backoff when a TLS timeout happens, hoping that
+					// idle connections are not to be reused on reconnect.
+					retryAfter := time.Duration(rand.Float32()*60000 + 180000)
+					LOG_INFO("ONEDRIVE_RETRY", "TLS handshake timeout; retry after %d milliseconds", retryAfter)
+					time.Sleep(retryAfter * time.Millisecond)
+				} else {
+					// For all other errors just blindly retry until the maximum is reached
+					retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
+					LOG_INFO("ONEDRIVE_RETRY", "%v; retry after %d milliseconds", err, retryAfter)
+					time.Sleep(retryAfter * time.Millisecond)
+				}
+				backoff *= 2
+				continue
+			}
+			return nil, 0, err
+		}
 
-        client.IsConnected = true
+		client.IsConnected = true
 
-        if response.StatusCode < 400 {
-            return response.Body, response.ContentLength, nil
-        }
+		if response.StatusCode < 400 {
+			return response.Body, response.ContentLength, nil
+		}
 
-        defer response.Body.Close()
+		defer response.Body.Close()
 
-        errorResponse := &OneDriveErrorResponse {
-            Error:  OneDriveError { Status: response.StatusCode },
-        }
+		errorResponse := &OneDriveErrorResponse{
+			Error: OneDriveError{Status: response.StatusCode},
+		}
 
-        if response.StatusCode == 401 {
+		if response.StatusCode == 401 {
 
-            if url == OneDriveRefreshTokenURL {
-                return nil, 0, OneDriveError { Status: response.StatusCode, Message: "Authorization error when refreshing token"}
-            }
+			if url == OneDriveRefreshTokenURL {
+				return nil, 0, OneDriveError{Status: response.StatusCode, Message: "Authorization error when refreshing token"}
+			}
 
-            err = client.RefreshToken()
-            if err != nil {
-                return nil, 0, err
-            }
-            continue
-        } else if response.StatusCode > 401 && response.StatusCode != 404 {
-            retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
-            LOG_INFO("ONEDRIVE_RETRY", "Response code: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
-            time.Sleep(retryAfter * time.Millisecond)
-            backoff *= 2
-            continue
-        } else {
-            if err := json.NewDecoder(response.Body).Decode(errorResponse); err != nil {
-                return nil, 0, OneDriveError { Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response"), }
-            }
+			err = client.RefreshToken(true)
+			if err != nil {
+				return nil, 0, err
+			}
+			continue
+		} else if response.StatusCode > 401 && response.StatusCode != 404 {
+			retryAfter := time.Duration(rand.Float32() * 1000.0 * float32(backoff))
+			LOG_INFO("ONEDRIVE_RETRY", "Response code: %d; retry after %d milliseconds", response.StatusCode, retryAfter)
+			time.Sleep(retryAfter * time.Millisecond)
+			backoff *= 2
+			continue
+		} else {
+			if err := json.NewDecoder(response.Body).Decode(errorResponse); err != nil {
+				return nil, 0, OneDriveError{Status: response.StatusCode, Message: fmt.Sprintf("Unexpected response")}
+			}
 
-            errorResponse.Error.Status = response.StatusCode
-            return nil, 0, errorResponse.Error
-        }
-    }
+			errorResponse.Error.Status = response.StatusCode
+			return nil, 0, errorResponse.Error
+		}
+	}
 
-    return nil, 0, fmt.Errorf("Maximum number of retries reached")
+	return nil, 0, fmt.Errorf("Maximum number of retries reached")
 }
 
-func (client *OneDriveClient) RefreshToken() (err error) {
-    client.TokenLock.Lock()
-    defer client.TokenLock.Unlock()
+func (client *OneDriveClient) RefreshToken(force bool) (err error) {
+	client.TokenLock.Lock()
+	defer client.TokenLock.Unlock()
 
-    if client.Token.Valid() {
-        return nil
-    }
+	if !force && client.Token.Valid() {
+		return nil
+	}
 
-    readCloser, _, err := client.call(OneDriveRefreshTokenURL, "POST", client.Token, "")
-    if err != nil {
-        return fmt.Errorf("failed to refresh the access token: %v", err) 
-    }
+	readCloser, _, err := client.call(OneDriveRefreshTokenURL, "POST", client.Token, "")
+	if err != nil {
+		return fmt.Errorf("failed to refresh the access token: %v", err)
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    if err = json.NewDecoder(readCloser).Decode(client.Token); err != nil {
-        return err
-    }
+	if err = json.NewDecoder(readCloser).Decode(client.Token); err != nil {
+		return err
+	}
 
-    description, err := json.Marshal(client.Token)
-    if err != nil {
-        return err
-    }
+	description, err := json.Marshal(client.Token)
+	if err != nil {
+		return err
+	}
 
-    err = ioutil.WriteFile(client.TokenFile, description, 0644)
-    if err != nil {
-        return err
-    }
+	err = ioutil.WriteFile(client.TokenFile, description, 0644)
+	if err != nil {
+		return err
+	}
 
-    return nil
+	return nil
 }
 
 type OneDriveEntry struct {
-    ID       string
-    Name     string
-    Folder   map[string] interface {}
-    Size     int64
+	ID     string
+	Name   string
+	Folder map[string]interface{}
+	Size   int64
 }
 
 type OneDriveListEntriesOutput struct {
-    Entries   []OneDriveEntry   `json:"value"`
-    NextLink string             `json:"@odata.nextLink"`
+	Entries  []OneDriveEntry `json:"value"`
+	NextLink string          `json:"@odata.nextLink"`
 }
 
 func (client *OneDriveClient) ListEntries(path string) ([]OneDriveEntry, error) {
 
-    entries := []OneDriveEntry{}
+	entries := []OneDriveEntry{}
 
-    url := OneDriveAPIURL + "/drive/root:/" + path + ":/children"
-    if path == "" {
-        url = OneDriveAPIURL + "/drive/root/children"
-    }
-    if client.TestMode {
-        url += "?top=8"
-    } else {
-        url += "?top=1000"
-    }
-    url += "&select=name,size,folder"
+	url := OneDriveAPIURL + "/drive/root:/" + path + ":/children"
+	if path == "" {
+		url = OneDriveAPIURL + "/drive/root/children"
+	}
+	if client.TestMode {
+		url += "?top=8"
+	} else {
+		url += "?top=1000"
+	}
+	url += "&select=name,size,folder"
 
-    for {
-        readCloser, _, err := client.call(url, "GET", 0, "")
-        if err != nil {
-            return nil, err
-        }
+	for {
+		readCloser, _, err := client.call(url, "GET", 0, "")
+		if err != nil {
+			return nil, err
+		}
 
-        defer readCloser.Close()
+		defer readCloser.Close()
 
-        output := &OneDriveListEntriesOutput {}
+		output := &OneDriveListEntriesOutput{}
 
-        if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-            return nil, err
-        }
+		if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+			return nil, err
+		}
 
-        entries = append(entries, output.Entries...)
+		entries = append(entries, output.Entries...)
 
-        url = output.NextLink
-        if url == "" {
-            break
-        }
-    }
+		url = output.NextLink
+		if url == "" {
+			break
+		}
+	}
 
-    return entries, nil
+	return entries, nil
 }
 
 func (client *OneDriveClient) GetFileInfo(path string) (string, bool, int64, error) {
 
-    url := OneDriveAPIURL + "/drive/root:/" + path
-    url += "?select=id,name,size,folder"
+	url := OneDriveAPIURL + "/drive/root:/" + path
+	url += "?select=id,name,size,folder"
 
-    readCloser, _, err := client.call(url, "GET", 0, "")
-    if err != nil {
-        if e, ok := err.(OneDriveError); ok && e.Status == 404 {
-            return "", false, 0, nil
-        } else {
-            return "", false, 0, err
-        }
-    }
+	readCloser, _, err := client.call(url, "GET", 0, "")
+	if err != nil {
+		if e, ok := err.(OneDriveError); ok && e.Status == 404 {
+			return "", false, 0, nil
+		} else {
+			return "", false, 0, err
+		}
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    output := &OneDriveEntry{}
+	output := &OneDriveEntry{}
 
-    if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
-        return "", false, 0, err
-    }
+	if err = json.NewDecoder(readCloser).Decode(&output); err != nil {
+		return "", false, 0, err
+	}
 
-    return output.ID, len(output.Folder) != 0, output.Size, nil
+	return output.ID, len(output.Folder) != 0, output.Size, nil
 }
 
 func (client *OneDriveClient) DownloadFile(path string) (io.ReadCloser, int64, error) {
 
-    url := OneDriveAPIURL + "/drive/items/root:/" + path + ":/content"
+	url := OneDriveAPIURL + "/drive/items/root:/" + path + ":/content"
 
-    return client.call(url, "GET", 0, "")
+	return client.call(url, "GET", 0, "")
 }
 
 func (client *OneDriveClient) UploadFile(path string, content []byte, rateLimit int) (err error) {
 
-    url := OneDriveAPIURL + "/drive/root:/" + path + ":/content"
+	url := OneDriveAPIURL + "/drive/root:/" + path + ":/content"
 
-    readCloser, _, err := client.call(url, "PUT", CreateRateLimitedReader(content, rateLimit), "application/octet-stream")
+	readCloser, _, err := client.call(url, "PUT", CreateRateLimitedReader(content, rateLimit), "application/octet-stream")
 
-    if err != nil {
-        return err
-    }
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *OneDriveClient) DeleteFile(path string) error {
 
-    url := OneDriveAPIURL + "/drive/root:/" + path
+	url := OneDriveAPIURL + "/drive/root:/" + path
 
-    readCloser, _, err := client.call(url, "DELETE", 0, "")
-    if err != nil {
-        return  err
-    }
+	readCloser, _, err := client.call(url, "DELETE", 0, "")
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
 func (client *OneDriveClient) MoveFile(path string, parent string) error {
 
-    url := OneDriveAPIURL + "/drive/root:/" + path
+	url := OneDriveAPIURL + "/drive/root:/" + path
 
-    parentReference := make(map[string]string)
-    parentReference["path"] = "/drive/root:/" + parent
+	parentReference := make(map[string]string)
+	parentReference["path"] = "/drive/root:/" + parent
 
-    parameters := make(map[string]interface{})
-    parameters["parentReference"] = parentReference
+	parameters := make(map[string]interface{})
+	parameters["parentReference"] = parentReference
 
-    readCloser, _, err := client.call(url, "PATCH", parameters, "application/json")
-    if err != nil {
-        return  err
-    }
+	readCloser, _, err := client.call(url, "PATCH", parameters, "application/json")
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }
 
-func (client *OneDriveClient) CreateDirectory(path string, name string) (error) {
+func (client *OneDriveClient) CreateDirectory(path string, name string) error {
 
-    url := OneDriveAPIURL + "/root/children"
+	url := OneDriveAPIURL + "/root/children"
 
-    if path != "" {
+	if path != "" {
 
-        parentID, isDir, _, err := client.GetFileInfo(path)
-        if err != nil {
-            return err
-        }
+		parentID, isDir, _, err := client.GetFileInfo(path)
+		if err != nil {
+			return err
+		}
 
-        if parentID == "" {
-            return fmt.Errorf("The path '%s' does not exist", path)
-        }
+		if parentID == "" {
+			return fmt.Errorf("The path '%s' does not exist", path)
+		}
 
-        if !isDir {
-            return fmt.Errorf("The path '%s' is not a directory", path)
-        }
+		if !isDir {
+			return fmt.Errorf("The path '%s' is not a directory", path)
+		}
 
-        url = OneDriveAPIURL + "/drive/items/" + parentID + "/children"
-    }
+		url = OneDriveAPIURL + "/drive/items/" + parentID + "/children"
+	}
 
-    parameters := make(map[string]interface{})
-    parameters["name"] = name
-    parameters["folder"] = make(map[string]int)
+	parameters := make(map[string]interface{})
+	parameters["name"] = name
+	parameters["folder"] = make(map[string]int)
 
-    readCloser, _, err := client.call(url, "POST", parameters, "application/json")
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := client.call(url, "POST", parameters, "application/json")
+	if err != nil {
+		return err
+	}
 
-    readCloser.Close()
-    return nil
+	readCloser.Close()
+	return nil
 }

src/duplicacy_oneclient_test.go

@@ -5,142 +5,141 @@
 package duplicacy
 
 import (
-    "io"
-    "fmt"
-    "testing"
-    "crypto/sha256"
-    "encoding/hex"
+	"crypto/sha256"
+	"encoding/hex"
+	"fmt"
+	"io"
+	"testing"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
+	crypto_rand "crypto/rand"
+	"math/rand"
 )
 
 func TestOneDriveClient(t *testing.T) {
 
-    oneDriveClient, err := NewOneDriveClient("one-token.json")
-    if err != nil {
-        t.Errorf("Failed to create the OneDrive client: %v", err)
-        return
-    }
+	oneDriveClient, err := NewOneDriveClient("one-token.json")
+	if err != nil {
+		t.Errorf("Failed to create the OneDrive client: %v", err)
+		return
+	}
 
-    oneDriveClient.TestMode = true
+	oneDriveClient.TestMode = true
 
-    existingFiles, err := oneDriveClient.ListEntries("")
-    for _, file := range existingFiles {
-        fmt.Printf("name: %s, isDir: %t\n", file.Name, len(file.Folder) != 0)
-    }
+	existingFiles, err := oneDriveClient.ListEntries("")
+	for _, file := range existingFiles {
+		fmt.Printf("name: %s, isDir: %t\n", file.Name, len(file.Folder) != 0)
+	}
 
+	testID, _, _, err := oneDriveClient.GetFileInfo("test")
+	if err != nil {
+		t.Errorf("Failed to list the test directory: %v", err)
+		return
+	}
+	if testID == "" {
+		err = oneDriveClient.CreateDirectory("", "test")
+		if err != nil {
+			t.Errorf("Failed to create the test directory: %v", err)
+			return
+		}
+	}
 
-    testID, _, _, err := oneDriveClient.GetFileInfo("test")
-    if err != nil {
-        t.Errorf("Failed to list the test directory: %v", err)
-        return
-    }
-    if testID == "" {
-        err = oneDriveClient.CreateDirectory("", "test")
-        if err != nil {
-            t.Errorf("Failed to create the test directory: %v", err)
-            return
-        }
-    }
+	test1ID, _, _, err := oneDriveClient.GetFileInfo("test/test1")
+	if err != nil {
+		t.Errorf("Failed to list the test1 directory: %v", err)
+		return
+	}
+	if test1ID == "" {
+		err = oneDriveClient.CreateDirectory("test", "test1")
+		if err != nil {
+			t.Errorf("Failed to create the test1 directory: %v", err)
+			return
+		}
+	}
 
-    test1ID, _, _, err := oneDriveClient.GetFileInfo("test/test1")
-    if err != nil {
-        t.Errorf("Failed to list the test1 directory: %v", err)
-        return
-    }
-    if test1ID == "" {
-        err = oneDriveClient.CreateDirectory("test", "test1")
-        if err != nil {
-            t.Errorf("Failed to create the test1 directory: %v", err)
-            return
-        }
-    }
+	test2ID, _, _, err := oneDriveClient.GetFileInfo("test/test2")
+	if err != nil {
+		t.Errorf("Failed to list the test2 directory: %v", err)
+		return
+	}
+	if test2ID == "" {
+		err = oneDriveClient.CreateDirectory("test", "test2")
+		if err != nil {
+			t.Errorf("Failed to create the test2 directory: %v", err)
+			return
+		}
+	}
 
-    test2ID, _, _, err := oneDriveClient.GetFileInfo("test/test2")
-    if err != nil {
-        t.Errorf("Failed to list the test2 directory: %v", err)
-        return
-    }
-    if test2ID == "" {
-        err = oneDriveClient.CreateDirectory("test", "test2")
-        if err != nil {
-            t.Errorf("Failed to create the test2 directory: %v", err)
-            return
-        }
-    }
+	numberOfFiles := 20
+	maxFileSize := 64 * 1024
 
-    numberOfFiles := 20
-    maxFileSize := 64 * 1024
+	for i := 0; i < numberOfFiles; i++ {
+		content := make([]byte, rand.Int()%maxFileSize+1)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-    for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+		hasher := sha256.New()
+		hasher.Write(content)
+		filename := hex.EncodeToString(hasher.Sum(nil))
 
-        hasher := sha256.New()
-        hasher.Write(content)
-        filename := hex.EncodeToString(hasher.Sum(nil))
+		fmt.Printf("file: %s\n", filename)
 
-        fmt.Printf("file: %s\n", filename)
+		err = oneDriveClient.UploadFile("test/test1/"+filename, content, 100)
+		if err != nil {
+			/*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
+				t.Errorf("Failed to upload the file %s: %v", filename, err)
+				return
+			}
+		}
+	}
 
-        err = oneDriveClient.UploadFile("test/test1/" + filename, content, 100)
-        if err != nil {
-            /*if e, ok := err.(ACDError); !ok || e.Status != 409 */ {
-                t.Errorf("Failed to upload the file %s: %v", filename, err)
-                return
-            }
-        }
-    }
+	entries, err := oneDriveClient.ListEntries("test/test1")
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    entries, err := oneDriveClient.ListEntries("test/test1")
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	for _, entry := range entries {
+		err = oneDriveClient.MoveFile("test/test1/"+entry.Name, "test/test2")
+		if err != nil {
+			t.Errorf("Failed to move %s: %v", entry.Name, err)
+			return
+		}
+	}
 
-    for _, entry := range entries {
-        err = oneDriveClient.MoveFile("test/test1/" + entry.Name, "test/test2")
-        if err != nil {
-            t.Errorf("Failed to move %s: %v", entry.Name, err)
-            return
-        }
-    }
+	entries, err = oneDriveClient.ListEntries("test/test2")
+	if err != nil {
+		t.Errorf("Error list randomly generated files: %v", err)
+		return
+	}
 
-    entries, err = oneDriveClient.ListEntries("test/test2")
-    if err != nil {
-        t.Errorf("Error list randomly generated files: %v", err)
-        return
-    }
+	for _, entry := range entries {
+		readCloser, _, err := oneDriveClient.DownloadFile("test/test2/" + entry.Name)
+		if err != nil {
+			t.Errorf("Error downloading file %s: %v", entry.Name, err)
+			return
+		}
 
-    for _, entry := range entries {
-        readCloser, _, err := oneDriveClient.DownloadFile("test/test2/" + entry.Name)
-        if err != nil {
-            t.Errorf("Error downloading file %s: %v", entry.Name, err)
-            return
-        }
+		hasher := sha256.New()
+		io.Copy(hasher, readCloser)
+		hash := hex.EncodeToString(hasher.Sum(nil))
 
-        hasher := sha256.New()
-        io.Copy(hasher, readCloser)
-        hash := hex.EncodeToString(hasher.Sum(nil))
+		if hash != entry.Name {
+			t.Errorf("File %s, hash %s", entry.Name, hash)
+		}
 
-        if hash != entry.Name {
-            t.Errorf("File %s, hash %s", entry.Name, hash)
-        }
+		readCloser.Close()
+	}
 
-        readCloser.Close()
-    }
+	for _, entry := range entries {
 
-    for _, entry := range entries {
-
-        err = oneDriveClient.DeleteFile("test/test2/" + entry.Name)
-        if err != nil {
-            t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
-            return
-        }
-    }
+		err = oneDriveClient.DeleteFile("test/test2/" + entry.Name)
+		if err != nil {
+			t.Errorf("Failed to delete the file %s: %v", entry.Name, err)
+			return
+		}
+	}
 
 }

src/duplicacy_onestorage.go

@@ -5,238 +5,241 @@
 package duplicacy
 
 import (
-    "fmt"
-    "path"
-    "strings"
+	"fmt"
+	"path"
+	"strings"
 )
 
 type OneDriveStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    client *OneDriveClient
-    storageDir string
-    numberOfThread int
+	client         *OneDriveClient
+	storageDir     string
+	numberOfThread int
 }
 
 // CreateOneDriveStorage creates an OneDrive storage object.
 func CreateOneDriveStorage(tokenFile string, storagePath string, threads int) (storage *OneDriveStorage, err error) {
 
-    for len(storagePath) > 0 && storagePath[len(storagePath) - 1] == '/' {
-        storagePath = storagePath[:len(storagePath) - 1]
-    }
+	for len(storagePath) > 0 && storagePath[len(storagePath)-1] == '/' {
+		storagePath = storagePath[:len(storagePath)-1]
+	}
 
-    client, err := NewOneDriveClient(tokenFile)
-    if err != nil {
-        return nil, err
-    }
+	client, err := NewOneDriveClient(tokenFile)
+	if err != nil {
+		return nil, err
+	}
 
-    fileID, isDir, _, err := client.GetFileInfo(storagePath)
-    if err != nil {
-        return nil, err
-    }
+	fileID, isDir, _, err := client.GetFileInfo(storagePath)
+	if err != nil {
+		return nil, err
+	}
 
-    if fileID == "" {
-        return nil, fmt.Errorf("Path '%s' doesn't exist", storagePath)
-    }
+	if fileID == "" {
+		return nil, fmt.Errorf("Path '%s' doesn't exist", storagePath)
+	}
 
-    if !isDir {
-        return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
-    }
+	if !isDir {
+		return nil, fmt.Errorf("Path '%s' is not a directory", storagePath)
+	}
 
-    storage = &OneDriveStorage {
-        client: client,
-        storageDir: storagePath,
-        numberOfThread: threads,
-    }
+	storage = &OneDriveStorage{
+		client:         client,
+		storageDir:     storagePath,
+		numberOfThread: threads,
+	}
 
-    for _, path := range []string { "chunks", "fossils", "snapshots" } {
-        dir := storagePath + "/" + path
-        dirID, isDir, _, err := client.GetFileInfo(dir)
-        if err != nil {
-            return nil, err
-        }
-        if dirID == "" {
-            err = client.CreateDirectory(storagePath, path)
-            if err != nil {
-                return nil, err
-            }
-        } else if !isDir {
-            return nil, fmt.Errorf("%s is not a directory", dir)
-        }
-    }
+	for _, path := range []string{"chunks", "fossils", "snapshots"} {
+		dir := storagePath + "/" + path
+		dirID, isDir, _, err := client.GetFileInfo(dir)
+		if err != nil {
+			return nil, err
+		}
+		if dirID == "" {
+			err = client.CreateDirectory(storagePath, path)
+			if err != nil {
+				return nil, err
+			}
+		} else if !isDir {
+			return nil, fmt.Errorf("%s is not a directory", dir)
+		}
+	}
 
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 
 }
 
+func (storage *OneDriveStorage) convertFilePath(filePath string) string {
+	if strings.HasPrefix(filePath, "chunks/") && strings.HasSuffix(filePath, ".fsl") {
+		return "fossils/" + filePath[len("chunks/"):len(filePath)-len(".fsl")]
+	}
+	return filePath
+}
+
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *OneDriveStorage) ListFiles(threadIndex int, dir string) ([]string, []int64, error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    if dir == "snapshots" {
-        entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
-        if err != nil {
-            return nil, nil, err
-        }
+	if dir == "snapshots" {
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        subDirs := []string{}
-        for _, entry := range entries {
-            if len(entry.Folder) > 0 {
-                subDirs = append(subDirs, entry.Name + "/")
-            }
-        }
-        return subDirs, nil, nil
-    } else if strings.HasPrefix(dir, "snapshots/") {
-        entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
-        if err != nil {
-            return nil, nil, err
-        }
+		subDirs := []string{}
+		for _, entry := range entries {
+			if len(entry.Folder) > 0 {
+				subDirs = append(subDirs, entry.Name+"/")
+			}
+		}
+		return subDirs, nil, nil
+	} else if strings.HasPrefix(dir, "snapshots/") || strings.HasPrefix(dir, "benchmark") {
+		entries, err := storage.client.ListEntries(storage.storageDir + "/" + dir)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        files := []string{}
+		files := []string{}
 
-        for _, entry := range entries {
-            if len(entry.Folder) == 0 {
-                files = append(files, entry.Name)
-            }
-        }
-        return files, nil, nil
-    } else {
-        files := []string{}
-        sizes := []int64{}
-        for _, parent := range []string {"chunks", "fossils" } {
-            entries, err := storage.client.ListEntries(storage.storageDir + "/" + parent)
-            if err != nil {
-                return nil, nil, err
-            }
+		for _, entry := range entries {
+			if len(entry.Folder) == 0 {
+				files = append(files, entry.Name)
+			}
+		}
+		return files, nil, nil
+	} else {
+		files := []string{}
+		sizes := []int64{}
+		parents := []string{"chunks", "fossils"}
+		for i := 0; i < len(parents); i++ {
+			parent := parents[i]
+			entries, err := storage.client.ListEntries(storage.storageDir + "/" + parent)
+			if err != nil {
+				return nil, nil, err
+			}
 
-            for _, entry := range entries {
-                name := entry.Name
-                if parent == "fossils" {
-                    name += ".fsl"
-                }
-                files = append(files, name)
-                sizes = append(sizes, entry.Size)
-            }
-        }
-        return files, sizes, nil
-    }
+			for _, entry := range entries {
+				if len(entry.Folder) == 0 {
+					name := entry.Name
+					if strings.HasPrefix(parent, "fossils") {
+						name = parent + "/" + name + ".fsl"
+						name = name[len("fossils/"):]
+					} else {
+						name = parent + "/" + name
+						name = name[len("chunks/"):]
+					}
+					files = append(files, name)
+					sizes = append(sizes, entry.Size)
+				} else {
+					parents = append(parents, parent+"/"+entry.Name)
+				}
+			}
+		}
+		return files, sizes, nil
+	}
 
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *OneDriveStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    if strings.HasSuffix(filePath, ".fsl") && strings.HasPrefix(filePath, "chunks/") {
-        filePath = "fossils/" + filePath[len("chunks/"):len(filePath) - len(".fsl")]
-    }
+	filePath = storage.convertFilePath(filePath)
 
-    err = storage.client.DeleteFile(storage.storageDir + "/" + filePath)
-    if e, ok := err.(OneDriveError); ok && e.Status == 404 {
-        LOG_DEBUG("ONEDRIVE_DELETE", "Ignore 404 error")
-        return nil
-    }
-    return err
+	err = storage.client.DeleteFile(storage.storageDir + "/" + filePath)
+	if e, ok := err.(OneDriveError); ok && e.Status == 404 {
+		LOG_DEBUG("ONEDRIVE_DELETE", "Ignore 404 error")
+		return nil
+	}
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *OneDriveStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    fromPath := storage.storageDir + "/" + from
-    toParent := storage.storageDir + "/fossils"
-    if strings.HasSuffix(from, ".fsl") {
-        fromPath = storage.storageDir + "/fossils/" + from[len("chunks/"):len(from) - len(".fsl")]
-        toParent = storage.storageDir + "/chunks"
-    }
 
-    err = storage.client.MoveFile(fromPath, toParent)
-    if err != nil {
-        if e, ok := err.(OneDriveError); ok && e.Status == 409 {
-            LOG_DEBUG("ONEDRIVE_MOVE", "Ignore 409 conflict error")
-        } else {
-            return err
-        }
-    }
-    return nil
+	fromPath := storage.storageDir + "/" + storage.convertFilePath(from)
+	toPath := storage.storageDir + "/" + storage.convertFilePath(to)
+
+	err = storage.client.MoveFile(fromPath, path.Dir(toPath))
+	if err != nil {
+		if e, ok := err.(OneDriveError); ok && e.Status == 409 {
+			LOG_DEBUG("ONEDRIVE_MOVE", "Ignore 409 conflict error")
+		} else {
+			return err
+		}
+	}
+	return nil
 }
 
 // CreateDirectory creates a new directory.
 func (storage *OneDriveStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    for len(dir) > 0 && dir[len(dir) - 1] == '/' {
-        dir = dir[:len(dir) - 1]
-    }
+	for len(dir) > 0 && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
 
-    parent := path.Dir(dir)
+	parent := path.Dir(dir)
 
-    if parent == "." {
-        return storage.client.CreateDirectory(storage.storageDir, dir)
-    } else {
-        return storage.client.CreateDirectory(storage.storageDir + "/" + parent, path.Base(dir))
-    }
+	if parent == "." {
+		return storage.client.CreateDirectory(storage.storageDir, dir)
+	} else {
+		return storage.client.CreateDirectory(storage.storageDir+"/"+parent, path.Base(dir))
+	}
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *OneDriveStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    for len(filePath) > 0 && filePath[len(filePath) - 1] == '/' {
-        filePath = filePath[:len(filePath) - 1]
-    }
-    fileID, isDir, size, err := storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
-    return fileID != "", isDir, size, err
-}
+	for len(filePath) > 0 && filePath[len(filePath)-1] == '/' {
+		filePath = filePath[:len(filePath)-1]
+	}
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *OneDriveStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-    filePath = "chunks/" + chunkID
-    realPath := storage.storageDir + "/" + filePath
-    if isFossil {
-        filePath += ".fsl"
-        realPath = storage.storageDir + "/fossils/" + chunkID
-    }
+	filePath = storage.convertFilePath(filePath)
 
-    fileID, _, size, err := storage.client.GetFileInfo(realPath)
-    return filePath, fileID != "", size, err
+	fileID, isDir, size, err := storage.client.GetFileInfo(storage.storageDir + "/" + filePath)
+	return fileID != "", isDir, size, err
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *OneDriveStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
-    if err != nil {
-        return err
-    }
+	readCloser, _, err := storage.client.DownloadFile(storage.storageDir + "/" + filePath)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / storage.numberOfThread)
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.numberOfThread)
+	return err
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *OneDriveStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
-    err = storage.client.UploadFile(storage.storageDir + "/" + filePath, content, storage.UploadRateLimit / storage.numberOfThread)
+	err = storage.client.UploadFile(storage.storageDir+"/"+filePath, content, storage.UploadRateLimit/storage.numberOfThread)
 
-    if e, ok := err.(OneDriveError); ok && e.Status == 409 {
-        LOG_TRACE("ONEDRIVE_UPLOAD", "File %s already exists", filePath)
-        return nil
-    } else {
-        return err
-    }
+	if e, ok := err.(OneDriveError); ok && e.Status == 409 {
+		LOG_TRACE("ONEDRIVE_UPLOAD", "File %s already exists", filePath)
+		return nil
+	} else {
+		return err
+	}
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *OneDriveStorage) IsCacheNeeded() (bool) { return true }
+func (storage *OneDriveStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *OneDriveStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *OneDriveStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *OneDriveStorage) IsStrongConsistent() (bool) { return false }
+func (storage *OneDriveStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *OneDriveStorage) IsFastListing() (bool) { return true }
+func (storage *OneDriveStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *OneDriveStorage) EnableTestMode() {
-    storage.client.TestMode = true
+	storage.client.TestMode = true
 }

src/duplicacy_preference.go

@@ -5,118 +5,128 @@
 package duplicacy
 
 import (
-    "encoding/json"
-    "path"
-    "io/ioutil"
-    "reflect"
-    "os"
+	"encoding/json"
+	"io/ioutil"
+	"os"
+	"path"
+	"reflect"
+	"strings"
 )
 
 // Preference stores options for each storage.
 type Preference struct {
-    Name string               `json:"name"`
-    SnapshotID string         `json:"id"`
-    StorageURL string         `json:"storage"`
-    Encrypted bool            `json:"encrypted"`
-    BackupProhibited bool     `json:"no_backup"`
-    RestoreProhibited bool    `json:"no_restore"`
-    DoNotSavePassword bool    `json:"no_save_password"`
-    Keys map[string]string    `json:"keys"`
+	Name              string            `json:"name"`
+	SnapshotID        string            `json:"id"`
+	RepositoryPath    string            `json:"repository"`
+	StorageURL        string            `json:"storage"`
+	Encrypted         bool              `json:"encrypted"`
+	BackupProhibited  bool              `json:"no_backup"`
+	RestoreProhibited bool              `json:"no_restore"`
+	DoNotSavePassword bool              `json:"no_save_password"`
+	NobackupFile      string            `json:"nobackup_file"`
+	Keys              map[string]string `json:"keys"`
 }
 
 var preferencePath string
-var Preferences [] Preference
+var Preferences []Preference
 
 func LoadPreferences(repository string) bool {
-    
-    preferencePath = path.Join(repository, DUPLICACY_DIRECTORY)
-    
-    stat, err := os.Stat(preferencePath)
-    if err != nil {
-        LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", repository, err)
-        return false
-    }
-    
-    if !stat.IsDir() {
-        content, err := ioutil.ReadFile(preferencePath)
-        if err != nil {
-            LOG_ERROR("DOT_DUPLICACY_PATH", "Failed to locate the preference path: %v", err)
-            return false
-        }
-        realPreferencePath := string(content)
-        stat, err := os.Stat(realPreferencePath)
-        if err != nil {
-            LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", content, err)
-            return false
-        }
-        if !stat.IsDir() {
-            LOG_ERROR("PREFERENCE_PATH", "The preference path %s is not a directory",  realPreferencePath)
-        }
 
-        preferencePath = realPreferencePath
-    }
-    
-    description, err := ioutil.ReadFile(path.Join(preferencePath, "preferences"))
-    if err != nil {
-        LOG_ERROR("PREFERENCE_OPEN", "Failed to read the preference file from repository %s: %v", repository, err)
-        return false
-    }
+	preferencePath = path.Join(repository, DUPLICACY_DIRECTORY)
 
-    err = json.Unmarshal(description, &Preferences)
-    if err != nil {
-        LOG_ERROR("PREFERENCE_PARSE", "Failed to parse the preference file for repository %s: %v", repository, err)
-        return false
-    }
+	stat, err := os.Stat(preferencePath)
+	if err != nil {
+		LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", repository, err)
+		return false
+	}
 
-    if len(Preferences) == 0 {
-        LOG_ERROR("PREFERENCE_NONE", "No preference found in the preference file")
-        return false
-    }
+	if !stat.IsDir() {
+		content, err := ioutil.ReadFile(preferencePath)
+		if err != nil {
+			LOG_ERROR("DOT_DUPLICACY_PATH", "Failed to locate the preference path: %v", err)
+			return false
+		}
+		realPreferencePath := strings.TrimSpace(string(content))
+		stat, err := os.Stat(realPreferencePath)
+		if err != nil {
+			LOG_ERROR("PREFERENCE_PATH", "Failed to retrieve the information about the directory %s: %v", content, err)
+			return false
+		}
+		if !stat.IsDir() {
+			LOG_ERROR("PREFERENCE_PATH", "The preference path %s is not a directory", realPreferencePath)
+		}
 
-    return true
+		preferencePath = realPreferencePath
+	}
+
+	description, err := ioutil.ReadFile(path.Join(preferencePath, "preferences"))
+	if err != nil {
+		LOG_ERROR("PREFERENCE_OPEN", "Failed to read the preference file from repository %s: %v", repository, err)
+		return false
+	}
+
+	err = json.Unmarshal(description, &Preferences)
+	if err != nil {
+		LOG_ERROR("PREFERENCE_PARSE", "Failed to parse the preference file for repository %s: %v", repository, err)
+		return false
+	}
+
+	if len(Preferences) == 0 {
+		LOG_ERROR("PREFERENCE_NONE", "No preference found in the preference file")
+		return false
+	}
+
+	for _, preference := range Preferences {
+		if strings.ToLower(preference.Name) == "ssh" {
+			LOG_ERROR("PREFERENCE_INVALID", "'%s' is an invalid storage name", preference.Name)
+			return false
+		}
+	}
+
+	return true
 }
 
 func GetDuplicacyPreferencePath() string {
-    if preferencePath == "" {
-        LOG_ERROR("PREFERENCE_PATH", "The preference path has not been set")
-        return ""
-    }
-    return preferencePath
+	if preferencePath == "" {
+		LOG_ERROR("PREFERENCE_PATH", "The preference path has not been set")
+		return ""
+	}
+	return preferencePath
 }
 
 // Normally 'preferencePath' is set in LoadPreferences; however, if LoadPreferences is not called, this function
 // provide another change to set 'preferencePath'
 func SetDuplicacyPreferencePath(p string) {
-    preferencePath = p
+	preferencePath = p
 }
 
-func SavePreferences() (bool) {
-    description, err := json.MarshalIndent(Preferences, "", "    ")
-    if err != nil {
-        LOG_ERROR("PREFERENCE_MARSHAL", "Failed to marshal the repository preferences: %v", err)
-        return false
-    }
-    preferenceFile := path.Join(GetDuplicacyPreferencePath(), "preferences")
-    
-    err = ioutil.WriteFile(preferenceFile, description, 0644)
-    if err != nil {
-        LOG_ERROR("PREFERENCE_WRITE", "Failed to save the preference file %s: %v", preferenceFile, err)
-        return false
-    }
+func SavePreferences() bool {
+	description, err := json.MarshalIndent(Preferences, "", "    ")
+	if err != nil {
+		LOG_ERROR("PREFERENCE_MARSHAL", "Failed to marshal the repository preferences: %v", err)
+		return false
+	}
+	preferenceFile := path.Join(GetDuplicacyPreferencePath(), "preferences")
 
-    return true
+	err = ioutil.WriteFile(preferenceFile, description, 0600)
+	if err != nil {
+		LOG_ERROR("PREFERENCE_WRITE", "Failed to save the preference file %s: %v", preferenceFile, err)
+		return false
+	}
+
+	return true
 }
 
-func FindPreference(name string) (*Preference) {
-    for _, preference := range Preferences {
-        if preference.Name == name || preference.StorageURL == name {
-            return &preference
-        }
-    }
+func FindPreference(name string) *Preference {
+	for i, preference := range Preferences {
+		if preference.Name == name || preference.StorageURL == name {
+			return &Preferences[i]
+		}
+	}
 
-    return nil
+	return nil
 }
 
 func (preference *Preference) Equal(other *Preference) bool {
-    return reflect.DeepEqual(preference, other)
+	return reflect.DeepEqual(preference, other)
 }

src/duplicacy_s3cstorage.go

@@ -5,208 +5,192 @@
 package duplicacy
 
 import (
-    "time"
-    "github.com/gilbertchen/goamz/aws"
-    "github.com/gilbertchen/goamz/s3"
+	"time"
+
+	"github.com/gilbertchen/goamz/aws"
+	"github.com/gilbertchen/goamz/s3"
 )
 
 // S3CStorage is a storage backend for s3 compatible storages that require V2 Signing.
 type S3CStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    buckets []*s3.Bucket
-    storageDir string
+	buckets    []*s3.Bucket
+	storageDir string
 }
 
 // CreateS3CStorage creates a amazon s3 storage object.
 func CreateS3CStorage(regionName string, endpoint string, bucketName string, storageDir string,
-                     accessKey string, secretKey string, threads int) (storage *S3CStorage, err error) {
+	accessKey string, secretKey string, threads int) (storage *S3CStorage, err error) {
 
-    var region aws.Region
+	var region aws.Region
 
-    if endpoint == "" {
-        if regionName == "" {
-            regionName = "us-east-1"
-        }
-        region = aws.Regions[regionName]
-    } else {
-        region = aws.Region { Name: regionName, S3Endpoint:"https://" + endpoint }
-    }
+	if endpoint == "" {
+		if regionName == "" {
+			regionName = "us-east-1"
+		}
+		region = aws.Regions[regionName]
+	} else {
+		region = aws.Region{Name: regionName, S3Endpoint: "https://" + endpoint}
+	}
 
-    auth := aws.Auth{ AccessKey: accessKey, SecretKey: secretKey }
+	auth := aws.Auth{AccessKey: accessKey, SecretKey: secretKey}
 
-    var buckets []*s3.Bucket
-    for i := 0; i < threads; i++ {
-        s3Client := s3.New(auth, region)
-        s3Client.AttemptStrategy = aws.AttemptStrategy{
-            Min:   8,
-            Total: 300 * time.Second,
-            Delay: 1000 * time.Millisecond,
-        }
+	var buckets []*s3.Bucket
+	for i := 0; i < threads; i++ {
+		s3Client := s3.New(auth, region)
+		s3Client.AttemptStrategy = aws.AttemptStrategy{
+			Min:   8,
+			Total: 300 * time.Second,
+			Delay: 1000 * time.Millisecond,
+		}
 
-        bucket := s3Client.Bucket(bucketName)
-        buckets = append(buckets, bucket)
-    }
+		bucket := s3Client.Bucket(bucketName)
+		buckets = append(buckets, bucket)
+	}
 
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
-        storageDir += "/"
-    }
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+		storageDir += "/"
+	}
 
-    storage = &S3CStorage {
-        buckets: buckets,
-        storageDir: storageDir,
-    }
+	storage = &S3CStorage{
+		buckets:    buckets,
+		storageDir: storageDir,
+	}
 
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *S3CStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
-        dir += "/"
-    }
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
 
-    dirLength := len(storage.storageDir + dir)
-    if dir == "snapshots/" {
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "/", "", 100)
-        if err != nil {
-            return nil, nil, err
-        }
+	dirLength := len(storage.storageDir + dir)
+	if dir == "snapshots/" {
+		results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "/", "", 100)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        for _, subDir := range results.CommonPrefixes {
-            files = append(files, subDir[dirLength:])
-        }
-        return files, nil, nil
-    } else if dir == "chunks/" {
-        marker := ""
-        for {
-            results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", marker, 1000)
-            if err != nil {
-                return nil, nil, err
-            }
+		for _, subDir := range results.CommonPrefixes {
+			files = append(files, subDir[dirLength:])
+		}
+		return files, nil, nil
+	} else if dir == "chunks/" {
+		marker := ""
+		for {
+			results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "", marker, 1000)
+			if err != nil {
+				return nil, nil, err
+			}
 
-            for _, object := range results.Contents {
-                files = append(files, object.Key[dirLength:])
-                sizes = append(sizes, object.Size)
-            }
+			for _, object := range results.Contents {
+				files = append(files, object.Key[dirLength:])
+				sizes = append(sizes, object.Size)
+			}
 
-            if !results.IsTruncated {
-                break
-            }
+			if !results.IsTruncated {
+				break
+			}
 
-            marker = results.Contents[len(results.Contents) - 1].Key
-        }
-        return files, sizes, nil
+			marker = results.Contents[len(results.Contents)-1].Key
+		}
+		return files, sizes, nil
 
-    } else {
+	} else {
 
-        results, err := storage.buckets[threadIndex].List(storage.storageDir + dir, "", "", 1000)
-        if err != nil {
-            return nil, nil, err
-        }
+		results, err := storage.buckets[threadIndex].List(storage.storageDir+dir, "", "", 1000)
+		if err != nil {
+			return nil, nil, err
+		}
 
-        for _, object := range results.Contents {
-            files = append(files, object.Key[dirLength:])
-        }
-        return files, nil, nil
-    }
+		for _, object := range results.Contents {
+			files = append(files, object.Key[dirLength:])
+		}
+		return files, nil, nil
+	}
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *S3CStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    return storage.buckets[threadIndex].Del(storage.storageDir + filePath)
+	return storage.buckets[threadIndex].Del(storage.storageDir + filePath)
 }
 
 // MoveFile renames the file.
 func (storage *S3CStorage) MoveFile(threadIndex int, from string, to string) (err error) {
 
-    options := s3.CopyOptions { ContentType: "application/duplicacy" }
-    _, err = storage.buckets[threadIndex].PutCopy(storage.storageDir + to, s3.Private, options, storage.buckets[threadIndex].Name + "/" + storage.storageDir + from)
-    if err != nil {
-        return nil
-    }
+	options := s3.CopyOptions{ContentType: "application/duplicacy"}
+	_, err = storage.buckets[threadIndex].PutCopy(storage.storageDir+to, s3.Private, options, storage.buckets[threadIndex].Name+"/"+storage.storageDir+from)
+	if err != nil {
+		return nil
+	}
 
-    return storage.DeleteFile(threadIndex, from)
+	return storage.DeleteFile(threadIndex, from)
 }
 
 // CreateDirectory creates a new directory.
 func (storage *S3CStorage) CreateDirectory(threadIndex int, dir string) (err error) {
-    return nil
+	return nil
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *S3CStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    response, err := storage.buckets[threadIndex].Head(storage.storageDir + filePath, nil)
-    if err != nil {
-        if e, ok := err.(*s3.Error); ok && (e.StatusCode == 403 || e.StatusCode == 404) {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
-
-    if response.StatusCode == 403 || response.StatusCode == 404 {
-        return false, false, 0, nil
-    } else {
-        return true, false, response.ContentLength, nil
-    }
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *S3CStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
+	response, err := storage.buckets[threadIndex].Head(storage.storageDir+filePath, nil)
+	if err != nil {
+		if e, ok := err.(*s3.Error); ok && (e.StatusCode == 403 || e.StatusCode == 404) {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
+	if response.StatusCode == 403 || response.StatusCode == 404 {
+		return false, false, 0, nil
+	} else {
+		return true, false, response.ContentLength, nil
+	}
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *S3CStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-    readCloser, err := storage.buckets[threadIndex].GetReader(storage.storageDir + filePath)
-    if err != nil {
-        return err
-    }
+	readCloser, err := storage.buckets[threadIndex].GetReader(storage.storageDir + filePath)
+	if err != nil {
+		return err
+	}
 
-    defer readCloser.Close()
+	defer readCloser.Close()
 
-    _, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit / len(storage.buckets))
-    return err
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/len(storage.buckets))
+	return err
 
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *S3CStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 
-    options := s3.Options { }
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.buckets))
-    return storage.buckets[threadIndex].PutReader(storage.storageDir + filePath, reader, int64(len(content)), "application/duplicacy", s3.Private, options)
+	options := s3.Options{}
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/len(storage.buckets))
+	return storage.buckets[threadIndex].PutReader(storage.storageDir+filePath, reader, int64(len(content)), "application/duplicacy", s3.Private, options)
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *S3CStorage) IsCacheNeeded () (bool) { return true }
+func (storage *S3CStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *S3CStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *S3CStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *S3CStorage) IsStrongConsistent() (bool) { return false }
+func (storage *S3CStorage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *S3CStorage) IsFastListing() (bool) { return true }
+func (storage *S3CStorage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *S3CStorage) EnableTestMode() {}

src/duplicacy_s3storage.go

@@ -2,255 +2,255 @@
 // Free for personal use and commercial trial
 // Commercial use requires per-user licenses available from https://duplicacy.com
 
+// NOTE: The code in the Wasabi storage module relies on all functions
+// in this one except MoveFile(), IsMoveFileImplemented() and
+// IsStrongConsistent().  Changes to the API here will need to be
+// reflected there.
+
 package duplicacy
 
 import (
-    "github.com/aws/aws-sdk-go/aws"
-    "github.com/aws/aws-sdk-go/aws/awserr"
-    "github.com/aws/aws-sdk-go/aws/credentials"
-    "github.com/aws/aws-sdk-go/aws/session"
-    "github.com/aws/aws-sdk-go/service/s3"
+	"reflect"
+	"strings"
+
+	"github.com/aws/aws-sdk-go/aws"
+	"github.com/aws/aws-sdk-go/aws/awserr"
+	"github.com/aws/aws-sdk-go/aws/credentials"
+	"github.com/aws/aws-sdk-go/aws/session"
+	"github.com/aws/aws-sdk-go/service/s3"
 )
 
 type S3Storage struct {
-    RateLimitedStorage
+	StorageBase
 
-    client *s3.S3
-    bucket string
-    storageDir string
-    numberOfThreads int
+	client          *s3.S3
+	bucket          string
+	storageDir      string
+	numberOfThreads int
 }
 
 // CreateS3Storage creates a amazon s3 storage object.
 func CreateS3Storage(regionName string, endpoint string, bucketName string, storageDir string,
-                     accessKey string, secretKey string, threads int,
-                     isSSLSupported bool, isMinioCompatible bool) (storage *S3Storage, err error) {
+	accessKey string, secretKey string, threads int,
+	isSSLSupported bool, isMinioCompatible bool) (storage *S3Storage, err error) {
 
-    token := ""
-    
-    auth := credentials.NewStaticCredentials(accessKey, secretKey, token)
+	token := ""
 
-    if regionName == "" && endpoint == "" {
-        defaultRegionConfig := &aws.Config {
-            Region: aws.String("us-east-1"),
-            Credentials: auth,
-        }
-        
-        s3Client := s3.New(session.New(defaultRegionConfig))
+	auth := credentials.NewStaticCredentials(accessKey, secretKey, token)
 
-        response, err := s3Client.GetBucketLocation(&s3.GetBucketLocationInput{Bucket: aws.String(bucketName)})
+	if regionName == "" && endpoint == "" {
+		defaultRegionConfig := &aws.Config{
+			Region:      aws.String("us-east-1"),
+			Credentials: auth,
+		}
 
-        if err != nil {
-            return nil, err
-        }
-    
-        regionName = "us-east-1"
-        if response.LocationConstraint != nil {
-            regionName = *response.LocationConstraint
-        }
-    }
-   
-    config := &aws.Config {
-        Region: aws.String(regionName),
-        Credentials: auth,
-        Endpoint: aws.String(endpoint),
-        S3ForcePathStyle: aws.Bool(isMinioCompatible),
-        DisableSSL: aws.Bool(!isSSLSupported),
-    }    
-    
-    if len(storageDir) > 0 && storageDir[len(storageDir) - 1] != '/' {
-        storageDir += "/"
-    }
+		s3Client := s3.New(session.New(defaultRegionConfig))
 
-    storage = &S3Storage {
-        client: s3.New(session.New(config)),
-        bucket: bucketName,
-        storageDir: storageDir,
-        numberOfThreads: threads,
-    }
-    
-    return storage, nil
+		response, err := s3Client.GetBucketLocation(&s3.GetBucketLocationInput{Bucket: aws.String(bucketName)})
+
+		if err != nil {
+			return nil, err
+		}
+
+		regionName = "us-east-1"
+		if response.LocationConstraint != nil {
+			regionName = *response.LocationConstraint
+		}
+	}
+
+	s3Config := &aws.Config{
+		Region:           aws.String(regionName),
+		Credentials:      auth,
+		Endpoint:         aws.String(endpoint),
+		S3ForcePathStyle: aws.Bool(isMinioCompatible),
+		DisableSSL:       aws.Bool(!isSSLSupported),
+	}
+
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+		storageDir += "/"
+	}
+
+	storage = &S3Storage{
+		client:          s3.New(session.New(s3Config)),
+		bucket:          bucketName,
+		storageDir:      storageDir,
+		numberOfThreads: threads,
+	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
 }
 
 // ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
 func (storage *S3Storage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
-    if len(dir) > 0 && dir[len(dir) - 1] != '/' {
-        dir += "/"
-    }
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
 
-    if dir == "snapshots/" {
-        dir = storage.storageDir + dir
-        input := s3.ListObjectsInput {
-            Bucket: aws.String(storage.bucket),
-            Prefix: aws.String(dir),
-            Delimiter: aws.String("/"),
-            MaxKeys: aws.Int64(1000),
-        }
-        
-        output, err := storage.client.ListObjects(&input)
-        if err != nil {
-            return nil, nil, err
-        }
-        
-        for _, subDir := range output.CommonPrefixes {
-            files = append(files, (*subDir.Prefix)[len(dir):])
-        }
-        return files, nil, nil
-    } else {
-        dir = storage.storageDir + dir
-        marker := ""
-        for {
-            input := s3.ListObjectsInput {
-                Bucket: aws.String(storage.bucket),
-                Prefix: aws.String(dir),
-                MaxKeys: aws.Int64(1000),
-                Marker: aws.String(marker),
-            }
+	if dir == "snapshots/" {
+		dir = storage.storageDir + dir
+		input := s3.ListObjectsInput{
+			Bucket:    aws.String(storage.bucket),
+			Prefix:    aws.String(dir),
+			Delimiter: aws.String("/"),
+			MaxKeys:   aws.Int64(1000),
+		}
 
-            output, err := storage.client.ListObjects(&input)
-            if err != nil {
-                return nil, nil, err
-            }
+		output, err := storage.client.ListObjects(&input)
+		if err != nil {
+			return nil, nil, err
+		}
 
-            for _, object := range output.Contents {
-                files = append(files, (*object.Key)[len(dir):])
-                sizes = append(sizes, *object.Size)
-            }
+		for _, subDir := range output.CommonPrefixes {
+			files = append(files, (*subDir.Prefix)[len(dir):])
+		}
+		return files, nil, nil
+	} else {
+		dir = storage.storageDir + dir
+		marker := ""
+		for {
+			input := s3.ListObjectsInput{
+				Bucket:  aws.String(storage.bucket),
+				Prefix:  aws.String(dir),
+				MaxKeys: aws.Int64(1000),
+				Marker:  aws.String(marker),
+			}
 
-            if !*output.IsTruncated {
-                break
-            }
+			output, err := storage.client.ListObjects(&input)
+			if err != nil {
+				return nil, nil, err
+			}
 
-            marker = *output.Contents[len(output.Contents) - 1].Key            
-        }
-        return files, sizes, nil
-    }    
+			for _, object := range output.Contents {
+				files = append(files, (*object.Key)[len(dir):])
+				sizes = append(sizes, *object.Size)
+			}
+
+			if !*output.IsTruncated {
+				break
+			}
+
+			marker = *output.Contents[len(output.Contents)-1].Key
+		}
+		return files, sizes, nil
+	}
 
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *S3Storage) DeleteFile(threadIndex int, filePath string) (err error) {
-    input := &s3.DeleteObjectInput {
-        Bucket: aws.String(storage.bucket),
-        Key: aws.String(storage.storageDir + filePath),
-    }
-    _, err = storage.client.DeleteObject(input)
-    return err
+	input := &s3.DeleteObjectInput{
+		Bucket: aws.String(storage.bucket),
+		Key:    aws.String(storage.storageDir + filePath),
+	}
+	_, err = storage.client.DeleteObject(input)
+	return err
 }
 
 // MoveFile renames the file.
 func (storage *S3Storage) MoveFile(threadIndex int, from string, to string) (err error) {
 
-    input := &s3.CopyObjectInput {
-        Bucket: aws.String(storage.bucket),
-        CopySource: aws.String(storage.bucket + "/" + storage.storageDir + from),
-        Key: aws.String(storage.storageDir + to),
-    }
+	input := &s3.CopyObjectInput{
+		Bucket:     aws.String(storage.bucket),
+		CopySource: aws.String(storage.bucket + "/" + storage.storageDir + from),
+		Key:        aws.String(storage.storageDir + to),
+	}
 
-    _, err = storage.client.CopyObject(input)
-    if err != nil {
-        return err
-    }
-  
-    return storage.DeleteFile(threadIndex, from)
+	_, err = storage.client.CopyObject(input)
+	if err != nil {
+		return err
+	}
+
+	return storage.DeleteFile(threadIndex, from)
 
 }
 
 // CreateDirectory creates a new directory.
 func (storage *S3Storage) CreateDirectory(threadIndex int, dir string) (err error) {
-    return nil
+	return nil
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *S3Storage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
 
-    input := &s3.HeadObjectInput {
-        Bucket: aws.String(storage.bucket),
-        Key: aws.String(storage.storageDir + filePath),
-    }
+	input := &s3.HeadObjectInput{
+		Bucket: aws.String(storage.bucket),
+		Key:    aws.String(storage.storageDir + filePath),
+	}
 
-    output, err := storage.client.HeadObject(input)
-    if err != nil {
-        if e, ok := err.(awserr.RequestFailure); ok && (e.StatusCode() == 403 || e.StatusCode() == 404) {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }    
-    }
-    
-    if output == nil || output.ContentLength == nil {
-        return false, false, 0, nil
-    } else {
-        return true, false, *output.ContentLength, nil
-    }
-}
-
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *S3Storage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-
-    filePath = "chunks/" + chunkID
-    if isFossil {
-        filePath += ".fsl"
-    }
-
-    exist, _, size, err = storage.GetFileInfo(threadIndex, filePath)
-
-    if err != nil {
-        return "", false, 0, err
-    } else {
-        return filePath, exist, size, err
-    }
+	output, err := storage.client.HeadObject(input)
+	if err != nil {
+		if e, ok := err.(awserr.RequestFailure); ok && (e.StatusCode() == 403 || e.StatusCode() == 404) {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
+	if output == nil || output.ContentLength == nil {
+		return false, false, 0, nil
+	} else {
+		return true, false, *output.ContentLength, nil
+	}
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *S3Storage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
 
-   input := &s3.GetObjectInput {
-        Bucket: aws.String(storage.bucket),
-        Key: aws.String(storage.storageDir + filePath),
-    }
-    
-    output, err := storage.client.GetObject(input)
-    if err != nil {
-        return err
-    }
-    
-    defer output.Body.Close()
-    
-    _, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit / len(storage.bucket))
-    return err
+	input := &s3.GetObjectInput{
+		Bucket: aws.String(storage.bucket),
+		Key:    aws.String(storage.storageDir + filePath),
+	}
+
+	output, err := storage.client.GetObject(input)
+	if err != nil {
+		return err
+	}
+
+	defer output.Body.Close()
+
+	_, err = RateLimitedCopy(chunk, output.Body, storage.DownloadRateLimit/storage.numberOfThreads)
+	return err
 
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *S3Storage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 
-    input := &s3.PutObjectInput {
-        Bucket: aws.String(storage.bucket),
-        Key: aws.String(storage.storageDir + filePath),
-        ACL: aws.String(s3.ObjectCannedACLPrivate),
-        Body: CreateRateLimitedReader(content, storage.UploadRateLimit / len(storage.bucket)),
-        ContentType: aws.String("application/duplicacy"),
-    }
-    
-    _, err = storage.client.PutObject(input)
-    return err
+	attempts := 0
+
+	for {
+		input := &s3.PutObjectInput{
+			Bucket:      aws.String(storage.bucket),
+			Key:         aws.String(storage.storageDir + filePath),
+			ACL:         aws.String(s3.ObjectCannedACLPrivate),
+			Body:        CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads),
+			ContentType: aws.String("application/duplicacy"),
+		}
+
+		_, err = storage.client.PutObject(input)
+		if err == nil || attempts >= 3 || !strings.Contains(err.Error(), "XAmzContentSHA256Mismatch") {
+			return err
+		}
+
+		LOG_INFO("S3_RETRY", "Retrying on %s: %v", reflect.TypeOf(err), err)
+		attempts += 1
+	}
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *S3Storage) IsCacheNeeded () (bool) { return true }
+func (storage *S3Storage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *S3Storage) IsMoveFileImplemented() (bool) { return true }
+func (storage *S3Storage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *S3Storage) IsStrongConsistent() (bool) { return false }
+func (storage *S3Storage) IsStrongConsistent() bool { return false }
 
 // If the storage supports fast listing of files names.
-func (storage *S3Storage) IsFastListing() (bool) { return true }
+func (storage *S3Storage) IsFastListing() bool { return true }
 
 // Enable the test mode.
 func (storage *S3Storage) EnableTestMode() {}

src/duplicacy_sftpstorage.go

@@ -5,305 +5,361 @@
 package duplicacy
 
 import (
-    "fmt"
-    "io"
-    "os"
-    "net"
-    "path"
-    "time"
-    "runtime"
-      "math/rand"
+	"fmt"
+	"io"
+	"math/rand"
+	"net"
+	"os"
+	"path"
+	"runtime"
+	"strings"
+	"time"
+	"sync"
 
-    "golang.org/x/crypto/ssh"
-    "github.com/pkg/sftp"
+	"github.com/pkg/sftp"
+	"golang.org/x/crypto/ssh"
 )
 
 type SFTPStorage struct {
-    RateLimitedStorage
+	StorageBase
 
-    client *sftp.Client
-    storageDir string
-    numberOfThreads int
+	client          *sftp.Client
+	clientLock      sync.Mutex
+	minimumNesting  int // The minimum level of directories to dive into before searching for the chunk file.
+	storageDir      string
+	numberOfThreads int
+	numberOfTries   int
+	serverAddress   string
+	sftpConfig      *ssh.ClientConfig
 }
 
 func CreateSFTPStorageWithPassword(server string, port int, username string, storageDir string,
-                                   password string, threads int) (storage *SFTPStorage, err error) {
+	minimumNesting int, password string, threads int) (storage *SFTPStorage, err error) {
 
-    authMethods := [] ssh.AuthMethod { ssh.Password(password) }
+	authMethods := []ssh.AuthMethod{ssh.Password(password)}
 
+	hostKeyCallback := func(hostname string, remote net.Addr,
+		key ssh.PublicKey) error {
+		return nil
+	}
 
-    hostKeyCallback := func(hostname string, remote net.Addr,
-                                            key ssh.PublicKey) error {
-        return nil
-    }
-
-    return CreateSFTPStorage(server, port, username, storageDir, authMethods, hostKeyCallback, threads)
+	return CreateSFTPStorage(server, port, username, storageDir, minimumNesting, authMethods, hostKeyCallback, threads)
 }
 
-func CreateSFTPStorage(server string, port int, username string, storageDir string,
-                       authMethods [] ssh.AuthMethod,
-                       hostKeyCallback func(hostname string, remote net.Addr,
-                                            key ssh.PublicKey) error, threads int) (storage *SFTPStorage, err error) {
+func CreateSFTPStorage(server string, port int, username string, storageDir string, minimumNesting int,
+	authMethods []ssh.AuthMethod,
+	hostKeyCallback func(hostname string, remote net.Addr,
+		key ssh.PublicKey) error, threads int) (storage *SFTPStorage, err error) {
 
-    config := &ssh.ClientConfig{
-        User: username,
-        Auth: authMethods,
-        HostKeyCallback: hostKeyCallback,
-    }
+	sftpConfig := &ssh.ClientConfig{
+		User:            username,
+		Auth:            authMethods,
+		HostKeyCallback: hostKeyCallback,
+	}
 
-    if server == "sftp.hidrive.strato.com" {
-        config.Ciphers = []string {"aes128-cbc", "aes128-ctr", "aes256-ctr"}
-    }
+	if server == "sftp.hidrive.strato.com" {
+		sftpConfig.Ciphers = []string{"aes128-ctr", "aes256-ctr"}
+	}
 
-    serverAddress := fmt.Sprintf("%s:%d", server, port)
-    connection, err := ssh.Dial("tcp", serverAddress, config)
-    if err != nil {
-        return nil, err
-    }
+	serverAddress := fmt.Sprintf("%s:%d", server, port)
+	connection, err := ssh.Dial("tcp", serverAddress, sftpConfig)
+	if err != nil {
+		return nil, err
+	}
 
-    client, err := sftp.NewClient(connection)
-    if err != nil {
-        connection.Close()
-        return nil, err
-    }
+	client, err := sftp.NewClient(connection)
+	if err != nil {
+		connection.Close()
+		return nil, err
+	}
 
-    for storageDir[len(storageDir) - 1] ==  '/' {
-        storageDir = storageDir[:len(storageDir) - 1]
-    }
+	for storageDir[len(storageDir)-1] == '/' {
+		storageDir = storageDir[:len(storageDir)-1]
+	}
 
-    fileInfo, err := client.Stat(storageDir)
-    if err != nil {
-        return nil, fmt.Errorf("Can't access the storage path %s: %v", storageDir, err)
-    }
+	fileInfo, err := client.Stat(storageDir)
+	if err != nil {
+		return nil, fmt.Errorf("Can't access the storage path %s: %v", storageDir, err)
+	}
 
-    if !fileInfo.IsDir() {
-        return nil, fmt.Errorf("The storage path %s is not a directory", storageDir)
-    }
+	if !fileInfo.IsDir() {
+		return nil, fmt.Errorf("The storage path %s is not a directory", storageDir)
+	}
 
-    storage = &SFTPStorage {
-        client: client,
-        storageDir: storageDir,
-        numberOfThreads: threads,
-    }
+	storage = &SFTPStorage{
+		client:          client,
+		storageDir:      storageDir,
+		minimumNesting:  minimumNesting,
+		numberOfThreads: threads,
+		numberOfTries:   6,
+		serverAddress:   serverAddress,
+		sftpConfig:      sftpConfig,
+	}
 
-    // Random number fo generating the temporary chunk file suffix.
-    rand.Seed(time.Now().UnixNano())
+	// Random number fo generating the temporary chunk file suffix.
+	rand.Seed(time.Now().UnixNano())
 
-    runtime.SetFinalizer(storage, CloseSFTPStorage)
+	runtime.SetFinalizer(storage, CloseSFTPStorage)
 
-    return storage, nil
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+	return storage, nil
 }
 
 func CloseSFTPStorage(storage *SFTPStorage) {
-    storage.client.Close()
+	if storage.client != nil {
+		storage.client.Close()
+		storage.client = nil
+	}
 }
 
+func (storage *SFTPStorage) getSFTPClient() *sftp.Client {
+	storage.clientLock.Lock()
+	defer storage.clientLock.Unlock()
+	return storage.client
+}
+
+func (storage *SFTPStorage) retry(f func () error) error {
+	delay := time.Second
+	for i := 0;; i++ {
+		err := f()
+		if err != nil && strings.Contains(err.Error(), "EOF") && i < storage.numberOfTries {
+			LOG_WARN("SFTP_RETRY", "Encountered an error (%v); retry after %d second(s)", err, delay/time.Second)
+			time.Sleep(delay)
+			delay *= 2
+
+			storage.clientLock.Lock()
+			if storage.client != nil {
+				storage.client.Close()
+				storage.client = nil
+			}
+
+			connection, err := ssh.Dial("tcp", storage.serverAddress, storage.sftpConfig)
+			if err != nil {
+				storage.clientLock.Unlock()
+				return err
+			}
+
+			client, err := sftp.NewClient(connection)
+			if err != nil {
+				connection.Close()
+				storage.clientLock.Unlock()
+				return err
+			}
+			storage.client = client
+			storage.clientLock.Unlock()
+			continue
+		}
+		return err
+	}
+}
 // ListFiles return the list of files and subdirectories under 'file' (non-recursively)
 func (storage *SFTPStorage) ListFiles(threadIndex int, dirPath string) (files []string, sizes []int64, err error) {
 
-    entries, err := storage.client.ReadDir(path.Join(storage.storageDir, dirPath))
-    if err != nil {
-        return nil, nil, err
-    }
+	var entries []os.FileInfo
+	err = storage.retry(func() error {
+		entries, err = storage.getSFTPClient().ReadDir(path.Join(storage.storageDir, dirPath))
+		return err
+	})
+	if err != nil {
+		return nil, nil, err
+	}
 
-    for _, entry := range entries {
-        name := entry.Name()
-        if entry.IsDir() && name[len(name) - 1] != '/' {
-            name += "/"
-        }
+	for _, entry := range entries {
+		name := entry.Name()
+		if entry.IsDir() && name[len(name)-1] != '/' {
+			name += "/"
+		}
 
-        files = append(files, name)
-        sizes = append(sizes, entry.Size())
-    }
+		files = append(files, name)
+		sizes = append(sizes, entry.Size())
+	}
 
-    return files, sizes, nil
+	return files, sizes, nil
 }
 
 // DeleteFile deletes the file or directory at 'filePath'.
 func (storage *SFTPStorage) DeleteFile(threadIndex int, filePath string) (err error) {
-    fullPath := path.Join(storage.storageDir, filePath)
-    fileInfo, err := storage.client.Stat(fullPath)
-    if err != nil {
-        if os.IsNotExist(err) {
-            LOG_TRACE("SFTP_STORAGE", "File %s has disappeared before deletion", filePath)
-            return nil
-        }
-        return err
-    }
-    if fileInfo == nil {
-        return nil
-    }
-    return storage.client.Remove(path.Join(storage.storageDir, filePath))
+	fullPath := path.Join(storage.storageDir, filePath)
+	var fileInfo os.FileInfo
+	err = storage.retry(func() error {
+		fileInfo, err = storage.getSFTPClient().Stat(fullPath)
+		return err
+	})
+	if err != nil {
+		if os.IsNotExist(err) {
+			LOG_TRACE("SFTP_STORAGE", "File %s has disappeared before deletion", filePath)
+			return nil
+		}
+		return err
+	}
+	if fileInfo == nil {
+		return nil
+	}
+	return storage.retry(func() error { return storage.getSFTPClient().Remove(path.Join(storage.storageDir, filePath)) })
 }
 
 // MoveFile renames the file.
 func (storage *SFTPStorage) MoveFile(threadIndex int, from string, to string) (err error) {
-    toPath := path.Join(storage.storageDir, to)
-    fileInfo, err := storage.client.Stat(toPath)
-    if fileInfo != nil {
-        return fmt.Errorf("The destination file %s already exists", toPath)
-    }
-    return storage.client.Rename(path.Join(storage.storageDir, from),
-                                 path.Join(storage.storageDir, to))
+	toPath := path.Join(storage.storageDir, to)
+	var fileInfo os.FileInfo
+	err = storage.retry(func() error {
+		fileInfo, err = storage.getSFTPClient().Stat(toPath)
+		return err
+	})
+	if fileInfo != nil {
+		return fmt.Errorf("The destination file %s already exists", toPath)
+	}
+	err = storage.retry(func() error { return storage.getSFTPClient().Rename(path.Join(storage.storageDir, from),
+		path.Join(storage.storageDir, to)) })
+	return err
 }
 
 // CreateDirectory creates a new directory.
 func (storage *SFTPStorage) CreateDirectory(threadIndex int, dirPath string) (err error) {
-    fullPath := path.Join(storage.storageDir, dirPath)
-    fileInfo, err := storage.client.Stat(fullPath)
-    if fileInfo != nil && fileInfo.IsDir() {
-        return nil
-    }
-    return storage.client.Mkdir(path.Join(storage.storageDir, dirPath))
+	fullPath := path.Join(storage.storageDir, dirPath)
+	var fileInfo os.FileInfo
+	err = storage.retry(func() error {
+		fileInfo, err = storage.getSFTPClient().Stat(fullPath)
+		return err
+	})
+	if fileInfo != nil && fileInfo.IsDir() {
+		return nil
+	}
+	return storage.retry(func() error { return storage.getSFTPClient().Mkdir(path.Join(storage.storageDir, dirPath)) })
 }
 
 // GetFileInfo returns the information about the file or directory at 'filePath'.
 func (storage *SFTPStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
-    fileInfo, err := storage.client.Stat(path.Join(storage.storageDir, filePath))
-    if err != nil {
-        if os.IsNotExist(err) {
-            return false, false, 0, nil
-        } else {
-            return false, false, 0, err
-        }
-    }
+	var fileInfo os.FileInfo
+	err = storage.retry(func() error {
+		fileInfo, err = storage.getSFTPClient().Stat(path.Join(storage.storageDir, filePath))
+		return err
+	})
 
-    if fileInfo == nil {
-        return false, false, 0, nil
-    }
+	if err != nil {
+		if os.IsNotExist(err) {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
 
-    return true, fileInfo.IsDir(), fileInfo.Size(), nil
-}
+	if fileInfo == nil {
+		return false, false, 0, nil
+	}
 
-// FindChunk finds the chunk with the specified id.  If 'isFossil' is true, it will search for chunk files with
-// the suffix '.fsl'.
-func (storage *SFTPStorage) FindChunk(threadIndex int, chunkID string, isFossil bool) (filePath string, exist bool, size int64, err error) {
-     dir := path.Join(storage.storageDir, "chunks")
-
-    suffix := ""
-    if isFossil {
-        suffix = ".fsl"
-    }
-
-    // The minimum level of directories to dive into before searching for the chunk file.
-    minimumLevel := 2
-
-    for level := 0; level * 2 < len(chunkID); level ++ {
-        if level >= minimumLevel {
-            filePath = path.Join(dir, chunkID[2 * level:]) + suffix
-            if stat, err := storage.client.Stat(filePath); err == nil && !stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, stat.Size(), nil
-            } else if err == nil && stat.IsDir() {
-                return filePath[len(storage.storageDir) + 1:], true, 0, fmt.Errorf("The path %s is a directory", filePath)
-            }
-        }
-
-        // Find the subdirectory the chunk file may reside.
-        subDir := path.Join(dir, chunkID[2 * level: 2 * level + 2])
-        stat, err := storage.client.Stat(subDir)
-        if err == nil && stat.IsDir() {
-            dir = subDir
-            continue
-        }
-
-        if level < minimumLevel {
-            // Create the subdirectory if is doesn't exist.
-
-            if err == nil && !stat.IsDir() {
-                return "", false, 0, fmt.Errorf("The path %s is not a directory", subDir)
-            }
-
-            err = storage.client.Mkdir(subDir)
-            if err != nil {
-                // The directory may have been created by other threads so check it again.
-                stat, _ := storage.client.Stat(subDir)
-                if stat == nil || !stat.IsDir() {
-                    return "", false, 0, fmt.Errorf("Failed to create the directory %s: %v", subDir, err)
-                }
-            }
-
-            dir = subDir
-            continue
-        }
-
-        // Teh chunk must be under this subdirectory but it doesn't exist.
-        return path.Join(dir, chunkID[2 * level:])[len(storage.storageDir) + 1:] + suffix, false, 0, nil
-
-    }
-
-    LOG_FATAL("CHUNK_FIND", "Chunk %s is still not found after having searched a maximum level of directories",
-              chunkID)
-    return "", false, 0, nil
+	return true, fileInfo.IsDir(), fileInfo.Size(), nil
 }
 
 // DownloadFile reads the file at 'filePath' into the chunk.
 func (storage *SFTPStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
-    file, err := storage.client.Open(path.Join(storage.storageDir, filePath))
+	return storage.retry(func() error {
+		file, err := storage.getSFTPClient().Open(path.Join(storage.storageDir, filePath))
 
-    if err != nil {
-        return err
-    }
+		if err != nil {
+			return err
+		}
 
-    defer file.Close()
-    if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit / storage.numberOfThreads); err != nil {
-        return err
-    }
-
-    return nil
+		defer file.Close()
+		if _, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.numberOfThreads); err != nil {
+			return err
+		}
+		return nil
+	})
 }
 
 // UploadFile writes 'content' to the file at 'filePath'.
 func (storage *SFTPStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
 
-    fullPath := path.Join(storage.storageDir, filePath)
+	fullPath := path.Join(storage.storageDir, filePath)
 
-    letters := "abcdefghijklmnopqrstuvwxyz"
-    suffix := make([]byte, 8)
-    for i := range suffix {
-        suffix[i] = letters[rand.Intn(len(letters))]
-    }
+	dirs := strings.Split(filePath, "/")
+	if len(dirs) > 1 {
+		fullDir := path.Dir(fullPath)
+		err = storage.retry(func() error {
+			_, err := storage.getSFTPClient().Stat(fullDir)
+			return err
+		})
+		if err != nil {
+			// The error may be caused by a non-existent fullDir, or a broken connection.  In either case,
+			// we just assume it is the former because there isn't a way to tell which is the case.
+			for i := range dirs[1 : len(dirs)-1] {
+				subDir := path.Join(storage.storageDir, path.Join(dirs[0:i+2]...))
+				// We don't check the error; just keep going blindly but always store the last err
+				err = storage.getSFTPClient().Mkdir(subDir)
+			}
 
-    temporaryFile := fullPath + "." + string(suffix) + ".tmp"
+			// If there is an error creating the dirs, we check fullDir one more time, because another thread
+			// may happen to create the same fullDir ahead of this thread
+			if err != nil {
+				err = storage.retry(func() error {
+					_, err := storage.getSFTPClient().Stat(fullDir)
+					return err
+				})
+				if err != nil {
+					return err
+				}
+			}
+		}
+	}
 
-    file, err := storage.client.OpenFile(temporaryFile, os.O_WRONLY | os.O_CREATE | os.O_TRUNC)
-    if err != nil {
-        return err
-    }
+	return storage.retry(func() error {
 
-    reader := CreateRateLimitedReader(content, storage.UploadRateLimit / storage.numberOfThreads)
-    _, err = io.Copy(file, reader)
-    if err != nil {
-        file.Close()
-        return err
-    }
-    file.Close()
+		letters := "abcdefghijklmnopqrstuvwxyz"
+		suffix := make([]byte, 8)
+		for i := range suffix {
+			suffix[i] = letters[rand.Intn(len(letters))]
+		}
 
-    err = storage.client.Rename(temporaryFile, fullPath)
-    if err != nil {
+		temporaryFile := fullPath + "." + string(suffix) + ".tmp"
 
-        if _, err = storage.client.Stat(fullPath); err == nil {
-            storage.client.Remove(temporaryFile)
-            return nil
-        } else {
-            return fmt.Errorf("Uploaded file but failed to store it at %s: %v", fullPath, err)
-        }
-    }
+		file, err := storage.getSFTPClient().OpenFile(temporaryFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC)
+		if err != nil {
+			return err
+		}
 
-    return nil
+		reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.numberOfThreads)
+		_, err = io.Copy(file, reader)
+		if err != nil {
+			file.Close()
+			return err
+		}
+		file.Close()
+
+		err = storage.getSFTPClient().Rename(temporaryFile, fullPath)
+		if err != nil {
+			if _, err = storage.getSFTPClient().Stat(fullPath); err == nil {
+				storage.getSFTPClient().Remove(temporaryFile)
+				return nil
+			} else {
+				return fmt.Errorf("Uploaded file but failed to store it at %s: %v", fullPath, err)
+			}
+		}
+
+		return nil
+	})
 }
 
 // If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
 // managing snapshots.
-func (storage *SFTPStorage) IsCacheNeeded () (bool) { return true }
+func (storage *SFTPStorage) IsCacheNeeded() bool { return true }
 
 // If the 'MoveFile' method is implemented.
-func (storage *SFTPStorage) IsMoveFileImplemented() (bool) { return true }
+func (storage *SFTPStorage) IsMoveFileImplemented() bool { return true }
 
 // If the storage can guarantee strong consistency.
-func (storage *SFTPStorage) IsStrongConsistent() (bool) { return true }
+func (storage *SFTPStorage) IsStrongConsistent() bool { return true }
 
 // If the storage supports fast listing of files names.
-func (storage *SFTPStorage) IsFastListing() (bool) { return false }
+func (storage *SFTPStorage) IsFastListing() bool {
+	for _, level := range storage.readLevels {
+		if level > 1 {
+			return false
+		}
+	}
+	return true
+}
 
 // Enable the test mode.
 func (storage *SFTPStorage) EnableTestMode() {}

src/duplicacy_shadowcopy.go

@@ -3,11 +3,12 @@
 // Commercial use requires per-user licenses available from https://duplicacy.com
 
 // +build !windows
+// +build !darwin
 
 package duplicacy
 
-func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
-    return top
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {
+	return top
 }
 
 func DeleteShadowCopy() {}

src/duplicacy_shadowcopy_darwin.go

@@ -0,0 +1,170 @@
+//
+// Shadow copy module for Mac OSX using APFS snapshot
+//
+//
+// This module copyright 2018 Adam Marcus (https://github.com/amarcu5)
+// and may be distributed under the same terms as Duplicacy.
+
+package duplicacy
+
+import (
+	"context"
+	"errors"
+	"io/ioutil"
+	"os"
+	"os/exec"
+	"strings"
+	"syscall"
+	"time"
+)
+
+var snapshotPath string
+var snapshotDate string
+
+// Converts char array to string
+func CharsToString(ca []int8) string {
+
+	len := len(ca)
+	ba := make([]byte, len)
+
+	for i, v := range ca {
+		ba[i] = byte(v)
+		if ba[i] == 0 {
+			len = i
+			break
+		}
+	}
+
+	return string(ba[:len])
+}
+
+// Get ID of device containing path
+func GetPathDeviceId(path string) (deviceId int32, err error) {
+
+	stat := syscall.Stat_t{}
+
+	err = syscall.Stat(path, &stat)
+	if err != nil {
+		return 0, err
+	}
+
+	return stat.Dev, nil
+}
+
+// Executes shell command with timeout and returns stdout
+func CommandWithTimeout(timeoutInSeconds int, name string, arg ...string) (output string, err error) {
+
+	ctx, cancel := context.WithTimeout(context.Background(), time.Duration(timeoutInSeconds)*time.Second)
+	defer cancel()
+
+	cmd := exec.CommandContext(ctx, name, arg...)
+	out, err := cmd.Output()
+
+	if ctx.Err() == context.DeadlineExceeded {
+		err = errors.New("Command '" + name + "' timed out")
+	}
+
+	output = string(out)
+	return output, err
+}
+
+func DeleteShadowCopy() {
+
+	if snapshotPath == "" {
+		return
+	}
+
+	err := exec.Command("/sbin/umount", "-f", snapshotPath).Run()
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while unmounting snapshot")
+		return
+	}
+
+	err = exec.Command("tmutil", "deletelocalsnapshots", snapshotDate).Run()
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while deleting local snapshot")
+		return
+	}
+
+	err = os.RemoveAll(snapshotPath)
+	if err != nil {
+		LOG_ERROR("VSS_DELETE", "Error while deleting temporary mount directory")
+		return
+	}
+
+	LOG_INFO("VSS_DELETE", "Shadow copy unmounted and deleted at %s", snapshotPath)
+
+	snapshotPath = ""
+}
+
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {
+
+	if !shadowCopy {
+		return top
+	}
+
+	// Check repository filesystem is APFS
+	stat := syscall.Statfs_t{}
+	err := syscall.Statfs(top, &stat)
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to determine filesystem of repository path")
+		return top
+	}
+	if CharsToString(stat.Fstypename[:]) != "apfs" {
+		LOG_WARN("VSS_INIT", "VSS requires APFS filesystem")
+		return top
+	}
+
+	// Check path is local as tmutil snapshots will not support APFS formatted external drives
+	deviceIdLocal, err := GetPathDeviceId("/")
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to get device ID of path: /")
+		return top
+	}
+	deviceIdRepository, err := GetPathDeviceId(top)
+	if err != nil {
+		LOG_ERROR("VSS_INIT", "Unable to get device ID of path: ", top)
+		return top
+	}
+	if deviceIdLocal != deviceIdRepository {
+		LOG_WARN("VSS_PATH", "VSS not supported for non-local repository path: ", top)
+		return top
+	}
+
+	if timeoutInSeconds <= 60 {
+		timeoutInSeconds = 60
+	}
+
+	// Create mount point
+	snapshotPath, err = ioutil.TempDir("/tmp/", "snp_")
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Failed to create temporary mount directory")
+		return top
+	}
+
+	// Use tmutil to create snapshot
+	tmutilOutput, err := CommandWithTimeout(timeoutInSeconds, "tmutil", "snapshot")
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Error while calling tmutil: ", err)
+		return top
+	}
+
+	colonPos := strings.IndexByte(tmutilOutput, ':')
+	if colonPos < 0 {
+		LOG_ERROR("VSS_CREATE", "Snapshot creation failed: ", tmutilOutput)
+		return top
+	}
+	snapshotDate = strings.TrimSpace(tmutilOutput[colonPos+1:])
+
+	// Mount snapshot as readonly and hide from GUI i.e. Finder
+	_, err = CommandWithTimeout(timeoutInSeconds,
+		"/sbin/mount", "-t", "apfs", "-o", "nobrowse,-r,-s=com.apple.TimeMachine."+snapshotDate, "/", snapshotPath)
+	if err != nil {
+		LOG_ERROR("VSS_CREATE", "Error while mounting snapshot: ", err)
+		return top
+	}
+
+	LOG_INFO("VSS_DONE", "Shadow copy created and mounted at %s", snapshotPath)
+
+	return snapshotPath + top
+}

src/duplicacy_shadowcopy_windows.go

@@ -5,327 +5,325 @@
 package duplicacy
 
 import (
-    "syscall"
-    "unsafe"
-    "time"
-    "os"
-    "runtime"
+	"os"
+	"runtime"
+	"syscall"
+	"time"
+	"unsafe"
 
-    ole "github.com/gilbertchen/go-ole"
+	ole "github.com/gilbertchen/go-ole"
 )
 
 //507C37B4-CF5B-4e95-B0AF-14EB9767467E
 var IID_IVSS_ASYNC = &ole.GUID{0x507C37B4, 0xCF5B, 0x4e95, [8]byte{0xb0, 0xaf, 0x14, 0xeb, 0x97, 0x67, 0x46, 0x7e}}
 
 type IVSSAsync struct {
-    ole.IUnknown
+	ole.IUnknown
 }
 
 type IVSSAsyncVtbl struct {
-    ole.IUnknownVtbl
-    cancel                       uintptr
-    wait                         uintptr
-    queryStatus                  uintptr
+	ole.IUnknownVtbl
+	cancel      uintptr
+	wait        uintptr
+	queryStatus uintptr
 }
 
-func (async *IVSSAsync) VTable() * IVSSAsyncVtbl {
-    return (*IVSSAsyncVtbl)(unsafe.Pointer(async.RawVTable))
+func (async *IVSSAsync) VTable() *IVSSAsyncVtbl {
+	return (*IVSSAsyncVtbl)(unsafe.Pointer(async.RawVTable))
 }
 
-var VSS_S_ASYNC_PENDING   int32 = 0x00042309
-var VSS_S_ASYNC_FINISHED  int32 = 0x0004230A
+var VSS_S_ASYNC_PENDING int32 = 0x00042309
+var VSS_S_ASYNC_FINISHED int32 = 0x0004230A
 var VSS_S_ASYNC_CANCELLED int32 = 0x0004230B
 
 func (async *IVSSAsync) Wait(seconds int) bool {
 
-    startTime := time.Now().Unix()
-    for {
-        ret, _, _ := syscall.Syscall(async.VTable().wait, 2, uintptr(unsafe.Pointer(async)), uintptr(1000), 0)
-        if ret != 0 {
-            LOG_WARN("IVSSASYNC_WAIT", "IVssAsync::Wait returned %d\n", ret)
-        }
+	startTime := time.Now().Unix()
+	for {
+		ret, _, _ := syscall.Syscall(async.VTable().wait, 2, uintptr(unsafe.Pointer(async)), uintptr(1000), 0)
+		if ret != 0 {
+			LOG_WARN("IVSSASYNC_WAIT", "IVssAsync::Wait returned %d\n", ret)
+		}
 
-        var status int32
-        ret, _, _ = syscall.Syscall(async.VTable().queryStatus, 3, uintptr(unsafe.Pointer(async)),
-                                    uintptr(unsafe.Pointer(&status)), 0)
-        if ret != 0 {
-            LOG_WARN("IVSSASYNC_QUERY", "IVssAsync::QueryStatus returned %d\n", ret)
-        }
+		var status int32
+		ret, _, _ = syscall.Syscall(async.VTable().queryStatus, 3, uintptr(unsafe.Pointer(async)),
+			uintptr(unsafe.Pointer(&status)), 0)
+		if ret != 0 {
+			LOG_WARN("IVSSASYNC_QUERY", "IVssAsync::QueryStatus returned %d\n", ret)
+		}
 
-        if status == VSS_S_ASYNC_FINISHED {
-            return true
-        }
-        if time.Now().Unix() - startTime > int64(seconds) {
-            LOG_WARN("IVSSASYNC_TIMEOUT", "IVssAsync is pending for more than %d seconds\n", seconds)
-            return false
-        }
-    }
+		if status == VSS_S_ASYNC_FINISHED {
+			return true
+		}
+		if time.Now().Unix()-startTime > int64(seconds) {
+			LOG_WARN("IVSSASYNC_TIMEOUT", "IVssAsync is pending for more than %d seconds\n", seconds)
+			return false
+		}
+	}
 }
 
 func getIVSSAsync(unknown *ole.IUnknown, iid *ole.GUID) (async *IVSSAsync) {
-    r, _, _ := syscall.Syscall(
-        unknown.VTable().QueryInterface,
-        3,
-        uintptr(unsafe.Pointer(unknown)),
-        uintptr(unsafe.Pointer(iid)),
-        uintptr(unsafe.Pointer(&async)))
+	r, _, _ := syscall.Syscall(
+		unknown.VTable().QueryInterface,
+		3,
+		uintptr(unsafe.Pointer(unknown)),
+		uintptr(unsafe.Pointer(iid)),
+		uintptr(unsafe.Pointer(&async)))
 
-    if r != 0 {
-        LOG_WARN("IVSSASYNC_QUERY", "IVSSAsync::QueryInterface returned %d\n", r)
-        return nil
-    }
-    return
+	if r != 0 {
+		LOG_WARN("IVSSASYNC_QUERY", "IVSSAsync::QueryInterface returned %d\n", r)
+		return nil
+	}
+	return
 }
 
-
 //665c1d5f-c218-414d-a05d-7fef5f9d5c86
 var IID_IVSS = &ole.GUID{0x665c1d5f, 0xc218, 0x414d, [8]byte{0xa0, 0x5d, 0x7f, 0xef, 0x5f, 0x9d, 0x5c, 0x86}}
 
 type IVSS struct {
-    ole.IUnknown
+	ole.IUnknown
 }
 
 type IVSSVtbl struct {
-    ole.IUnknownVtbl
-    getWriterComponentsCount       uintptr
-    getWriterComponents            uintptr
-    initializeForBackup            uintptr
-    setBackupState                 uintptr
-    initializeForRestore           uintptr
-    setRestoreState                uintptr
-    gatherWriterMetadata           uintptr
-    getWriterMetadataCount         uintptr
-    getWriterMetadata              uintptr
-    freeWriterMetadata             uintptr
-    addComponent                   uintptr
-    prepareForBackup               uintptr
-    abortBackup                    uintptr
-    gatherWriterStatus             uintptr
-    getWriterStatusCount           uintptr
-    freeWriterStatus               uintptr
-    getWriterStatus                uintptr
-    setBackupSucceeded             uintptr
-    setBackupOptions               uintptr
-    setSelectedForRestore          uintptr
-    setRestoreOptions              uintptr
-    setAdditionalRestores          uintptr
-    setPreviousBackupStamp         uintptr
-    saveAsXML                      uintptr
-    backupComplete                 uintptr
-    addAlternativeLocationMapping  uintptr
-    addRestoreSubcomponent         uintptr
-    setFileRestoreStatus           uintptr
-    addNewTarget                   uintptr
-    setRangesFilePath              uintptr
-    preRestore                     uintptr
-    postRestore                    uintptr
-    setContext                     uintptr
-    startSnapshotSet               uintptr
-    addToSnapshotSet               uintptr
-    doSnapshotSet                  uintptr
-    deleteSnapshots                uintptr
-    importSnapshots                uintptr
-    breakSnapshotSet               uintptr
-    getSnapshotProperties          uintptr
-    query                          uintptr
-    isVolumeSupported              uintptr
-    disableWriterClasses           uintptr
-    enableWriterClasses            uintptr
-    disableWriterInstances         uintptr
-    exposeSnapshot                 uintptr
-    revertToSnapshot               uintptr
-    queryRevertStatus              uintptr
+	ole.IUnknownVtbl
+	getWriterComponentsCount      uintptr
+	getWriterComponents           uintptr
+	initializeForBackup           uintptr
+	setBackupState                uintptr
+	initializeForRestore          uintptr
+	setRestoreState               uintptr
+	gatherWriterMetadata          uintptr
+	getWriterMetadataCount        uintptr
+	getWriterMetadata             uintptr
+	freeWriterMetadata            uintptr
+	addComponent                  uintptr
+	prepareForBackup              uintptr
+	abortBackup                   uintptr
+	gatherWriterStatus            uintptr
+	getWriterStatusCount          uintptr
+	freeWriterStatus              uintptr
+	getWriterStatus               uintptr
+	setBackupSucceeded            uintptr
+	setBackupOptions              uintptr
+	setSelectedForRestore         uintptr
+	setRestoreOptions             uintptr
+	setAdditionalRestores         uintptr
+	setPreviousBackupStamp        uintptr
+	saveAsXML                     uintptr
+	backupComplete                uintptr
+	addAlternativeLocationMapping uintptr
+	addRestoreSubcomponent        uintptr
+	setFileRestoreStatus          uintptr
+	addNewTarget                  uintptr
+	setRangesFilePath             uintptr
+	preRestore                    uintptr
+	postRestore                   uintptr
+	setContext                    uintptr
+	startSnapshotSet              uintptr
+	addToSnapshotSet              uintptr
+	doSnapshotSet                 uintptr
+	deleteSnapshots               uintptr
+	importSnapshots               uintptr
+	breakSnapshotSet              uintptr
+	getSnapshotProperties         uintptr
+	query                         uintptr
+	isVolumeSupported             uintptr
+	disableWriterClasses          uintptr
+	enableWriterClasses           uintptr
+	disableWriterInstances        uintptr
+	exposeSnapshot                uintptr
+	revertToSnapshot              uintptr
+	queryRevertStatus             uintptr
 }
 
-func (vss *IVSS) VTable() * IVSSVtbl {
-    return (*IVSSVtbl)(unsafe.Pointer(vss.RawVTable))
+func (vss *IVSS) VTable() *IVSSVtbl {
+	return (*IVSSVtbl)(unsafe.Pointer(vss.RawVTable))
 }
 
 func (vss *IVSS) InitializeForBackup() int {
-    ret, _, _ := syscall.Syscall(vss.VTable().initializeForBackup, 2, uintptr(unsafe.Pointer(vss)), 0, 0)
-    return int(ret)
+	ret, _, _ := syscall.Syscall(vss.VTable().initializeForBackup, 2, uintptr(unsafe.Pointer(vss)), 0, 0)
+	return int(ret)
 }
 
 func (vss *IVSS) GatherWriterMetadata() (int, *IVSSAsync) {
-    var unknown *ole.IUnknown
-    ret, _, _ := syscall.Syscall(vss.VTable().gatherWriterMetadata, 2,
-                                 uintptr(unsafe.Pointer(vss)),
-                                 uintptr(unsafe.Pointer(&unknown)), 0)
+	var unknown *ole.IUnknown
+	ret, _, _ := syscall.Syscall(vss.VTable().gatherWriterMetadata, 2,
+		uintptr(unsafe.Pointer(vss)),
+		uintptr(unsafe.Pointer(&unknown)), 0)
 
-    if ret != 0 {
-        return int(ret), nil
-    } else {
-        return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
-    }
+	if ret != 0 {
+		return int(ret), nil
+	} else {
+		return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
+	}
 }
 
 func (vss *IVSS) StartSnapshotSet(snapshotID *ole.GUID) int {
-    ret, _, _ := syscall.Syscall(vss.VTable().startSnapshotSet, 2,
-                                 uintptr(unsafe.Pointer(vss)),
-                                 uintptr(unsafe.Pointer(snapshotID)), 0)
-    return int(ret)
+	ret, _, _ := syscall.Syscall(vss.VTable().startSnapshotSet, 2,
+		uintptr(unsafe.Pointer(vss)),
+		uintptr(unsafe.Pointer(snapshotID)), 0)
+	return int(ret)
 }
 
 func (vss *IVSS) AddToSnapshotSet(drive string, snapshotID *ole.GUID) int {
 
-    volumeName := syscall.StringToUTF16Ptr(drive)
+	volumeName := syscall.StringToUTF16Ptr(drive)
 
-    var ret uintptr
-    if runtime.GOARCH == "386" {
-        // On 32-bit Windows, GUID is passed by value
-        ret, _, _ = syscall.Syscall9(vss.VTable().addToSnapshotSet, 7,
-                                     uintptr(unsafe.Pointer(vss)),
-                                     uintptr(unsafe.Pointer(volumeName)),
-                                     0, 0, 0, 0,
-                                     uintptr(unsafe.Pointer(snapshotID)), 0, 0)
-    } else {
-        ret, _, _ = syscall.Syscall6(vss.VTable().addToSnapshotSet, 4,
-                                     uintptr(unsafe.Pointer(vss)),
-                                     uintptr(unsafe.Pointer(volumeName)),
-                                     uintptr(unsafe.Pointer(ole.IID_NULL)),
-                                     uintptr(unsafe.Pointer(snapshotID)), 0, 0)
-    }
-    return int(ret)
+	var ret uintptr
+	if runtime.GOARCH == "386" {
+		// On 32-bit Windows, GUID is passed by value
+		ret, _, _ = syscall.Syscall9(vss.VTable().addToSnapshotSet, 7,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(unsafe.Pointer(volumeName)),
+			0, 0, 0, 0,
+			uintptr(unsafe.Pointer(snapshotID)), 0, 0)
+	} else {
+		ret, _, _ = syscall.Syscall6(vss.VTable().addToSnapshotSet, 4,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(unsafe.Pointer(volumeName)),
+			uintptr(unsafe.Pointer(ole.IID_NULL)),
+			uintptr(unsafe.Pointer(snapshotID)), 0, 0)
+	}
+	return int(ret)
 }
 
 func (vss *IVSS) SetBackupState() int {
-    VSS_BT_COPY := 5
-    ret, _, _ := syscall.Syscall6(vss.VTable().setBackupState, 4,
-                                 uintptr(unsafe.Pointer(vss)),
-                                 0, 0, uintptr(VSS_BT_COPY), 0, 0)
-    return int(ret)
+	VSS_BT_COPY := 5
+	ret, _, _ := syscall.Syscall6(vss.VTable().setBackupState, 4,
+		uintptr(unsafe.Pointer(vss)),
+		0, 0, uintptr(VSS_BT_COPY), 0, 0)
+	return int(ret)
 }
 
 func (vss *IVSS) PrepareForBackup() (int, *IVSSAsync) {
-    var unknown *ole.IUnknown
-    ret, _, _ := syscall.Syscall(vss.VTable().prepareForBackup, 2,
-                                 uintptr(unsafe.Pointer(vss)),
-                                 uintptr(unsafe.Pointer(&unknown)), 0)
+	var unknown *ole.IUnknown
+	ret, _, _ := syscall.Syscall(vss.VTable().prepareForBackup, 2,
+		uintptr(unsafe.Pointer(vss)),
+		uintptr(unsafe.Pointer(&unknown)), 0)
 
-    if ret != 0 {
-        return int(ret), nil
-    } else {
-        return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
-    }
+	if ret != 0 {
+		return int(ret), nil
+	} else {
+		return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
+	}
 }
 
 func (vss *IVSS) DoSnapshotSet() (int, *IVSSAsync) {
-    var unknown *ole.IUnknown
-    ret, _, _ := syscall.Syscall(vss.VTable().doSnapshotSet, 2,
-                                 uintptr(unsafe.Pointer(vss)),
-                                 uintptr(unsafe.Pointer(&unknown)), 0)
+	var unknown *ole.IUnknown
+	ret, _, _ := syscall.Syscall(vss.VTable().doSnapshotSet, 2,
+		uintptr(unsafe.Pointer(vss)),
+		uintptr(unsafe.Pointer(&unknown)), 0)
 
-    if ret != 0 {
-        return int(ret), nil
-    } else {
-        return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
-    }
+	if ret != 0 {
+		return int(ret), nil
+	} else {
+		return int(ret), getIVSSAsync(unknown, IID_IVSS_ASYNC)
+	}
 }
 
 type SnapshotProperties struct {
-    SnapshotID ole.GUID
-    SnapshotSetID ole.GUID
-    SnapshotsCount uint32
-    SnapshotDeviceObject *uint16
-    OriginalVolumeName *uint16
-    OriginatingMachine *uint16
-    ServiceMachine *uint16
-    ExposedName *uint16
-    ExposedPath *uint16
-    ProviderId ole.GUID
-    SnapshotAttributes uint32
-    CreationTimestamp int64
-    Status int
+	SnapshotID           ole.GUID
+	SnapshotSetID        ole.GUID
+	SnapshotsCount       uint32
+	SnapshotDeviceObject *uint16
+	OriginalVolumeName   *uint16
+	OriginatingMachine   *uint16
+	ServiceMachine       *uint16
+	ExposedName          *uint16
+	ExposedPath          *uint16
+	ProviderId           ole.GUID
+	SnapshotAttributes   uint32
+	CreationTimestamp    int64
+	Status               int
 }
 
-func (vss *IVSS) GetSnapshotProperties(snapshotSetID ole.GUID, properties *SnapshotProperties) (int) {
-    var ret uintptr
-    if runtime.GOARCH == "386" {
-        address := uint(uintptr(unsafe.Pointer(&snapshotSetID)))
-        ret, _, _ = syscall.Syscall6(vss.VTable().getSnapshotProperties, 6,
-                                     uintptr(unsafe.Pointer(vss)),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 4)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 8)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 12)))),
-                                     uintptr(unsafe.Pointer(properties)))
-    } else {
-        ret, _, _ = syscall.Syscall(vss.VTable().getSnapshotProperties, 3,
-                                    uintptr(unsafe.Pointer(vss)),
-                                    uintptr(unsafe.Pointer(&snapshotSetID)),
-                                    uintptr(unsafe.Pointer(properties)))
-    }
-    return int(ret)
+func (vss *IVSS) GetSnapshotProperties(snapshotSetID ole.GUID, properties *SnapshotProperties) int {
+	var ret uintptr
+	if runtime.GOARCH == "386" {
+		address := uint(uintptr(unsafe.Pointer(&snapshotSetID)))
+		ret, _, _ = syscall.Syscall6(vss.VTable().getSnapshotProperties, 6,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 4)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 8)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 12)))),
+			uintptr(unsafe.Pointer(properties)))
+	} else {
+		ret, _, _ = syscall.Syscall(vss.VTable().getSnapshotProperties, 3,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(unsafe.Pointer(&snapshotSetID)),
+			uintptr(unsafe.Pointer(properties)))
+	}
+	return int(ret)
 }
 
 func (vss *IVSS) DeleteSnapshots(snapshotID ole.GUID) (int, int, ole.GUID) {
 
-    VSS_OBJECT_SNAPSHOT := 3
+	VSS_OBJECT_SNAPSHOT := 3
 
-    deleted := int32(0)
+	deleted := int32(0)
 
-    var deletedGUID ole.GUID
+	var deletedGUID ole.GUID
 
-    var ret uintptr
-    if runtime.GOARCH == "386" {
-        address := uint(uintptr(unsafe.Pointer(&snapshotID)))
-        ret, _, _ = syscall.Syscall9(vss.VTable().deleteSnapshots, 9,
-                                     uintptr(unsafe.Pointer(vss)),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 4)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 8)))),
-                                     uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 12)))),
-                                     uintptr(VSS_OBJECT_SNAPSHOT),
-                                     uintptr(1),
-                                     uintptr(unsafe.Pointer(&deleted)),
-                                     uintptr(unsafe.Pointer(&deletedGUID)))
-    } else {
-        ret, _, _ = syscall.Syscall6(vss.VTable().deleteSnapshots, 6,
-                                     uintptr(unsafe.Pointer(vss)),
-                                     uintptr(unsafe.Pointer(&snapshotID)),
-                                     uintptr(VSS_OBJECT_SNAPSHOT),
-                                     uintptr(1),
-                                     uintptr(unsafe.Pointer(&deleted)),
-                                     uintptr(unsafe.Pointer(&deletedGUID)))
-    }
+	var ret uintptr
+	if runtime.GOARCH == "386" {
+		address := uint(uintptr(unsafe.Pointer(&snapshotID)))
+		ret, _, _ = syscall.Syscall9(vss.VTable().deleteSnapshots, 9,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 4)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 8)))),
+			uintptr(*(*uint32)(unsafe.Pointer(uintptr(address + 12)))),
+			uintptr(VSS_OBJECT_SNAPSHOT),
+			uintptr(1),
+			uintptr(unsafe.Pointer(&deleted)),
+			uintptr(unsafe.Pointer(&deletedGUID)))
+	} else {
+		ret, _, _ = syscall.Syscall6(vss.VTable().deleteSnapshots, 6,
+			uintptr(unsafe.Pointer(vss)),
+			uintptr(unsafe.Pointer(&snapshotID)),
+			uintptr(VSS_OBJECT_SNAPSHOT),
+			uintptr(1),
+			uintptr(unsafe.Pointer(&deleted)),
+			uintptr(unsafe.Pointer(&deletedGUID)))
+	}
 
-    return int(ret), int(deleted), deletedGUID
+	return int(ret), int(deleted), deletedGUID
 }
 
+func uint16ArrayToString(p *uint16) string {
+	if p == nil {
+		return ""
+	}
+	s := make([]uint16, 0)
+	address := uintptr(unsafe.Pointer(p))
+	for {
+		c := *(*uint16)(unsafe.Pointer(address))
+		if c == 0 {
+			break
+		}
 
-func uint16ArrayToString(p *uint16) (string) {
-    if p == nil {
-        return ""
-    }
-    s := make([]uint16, 0)
-    address := uintptr(unsafe.Pointer(p))
-    for {
-        c := *(*uint16)(unsafe.Pointer(address))
-        if c == 0 {
-            break
-        }
+		s = append(s, c)
+		address = uintptr(int(address) + 2)
+	}
 
-        s = append(s, c)
-        address = uintptr(int(address) + 2)
-    }
-
-    return syscall.UTF16ToString(s)
+	return syscall.UTF16ToString(s)
 }
 
 func getIVSS(unknown *ole.IUnknown, iid *ole.GUID) (ivss *IVSS) {
-    r, _, _ := syscall.Syscall(
-        unknown.VTable().QueryInterface,
-        3,
-        uintptr(unsafe.Pointer(unknown)),
-        uintptr(unsafe.Pointer(iid)),
-        uintptr(unsafe.Pointer(&ivss)))
+	r, _, _ := syscall.Syscall(
+		unknown.VTable().QueryInterface,
+		3,
+		uintptr(unsafe.Pointer(unknown)),
+		uintptr(unsafe.Pointer(iid)),
+		uintptr(unsafe.Pointer(&ivss)))
 
-    if r != 0 {
-        LOG_WARN("IVSS_QUERY", "IVSS::QueryInterface returned %d\n", r)
-        return nil
-    }
+	if r != 0 {
+		LOG_WARN("IVSS_QUERY", "IVSS::QueryInterface returned %d\n", r)
+		return nil
+	}
 
-    return ivss
+	return ivss
 }
 
 var vssBackupComponent *IVSS
@@ -333,193 +331,192 @@ var snapshotID ole.GUID
 var shadowLink string
 
 func DeleteShadowCopy() {
-    if vssBackupComponent != nil {
-        defer vssBackupComponent.Release()
+	if vssBackupComponent != nil {
+		defer vssBackupComponent.Release()
 
-        LOG_TRACE("VSS_DELETE", "Deleting the shadow copy used for this backup")
-        ret, _, _ := vssBackupComponent.DeleteSnapshots(snapshotID)
-        if ret != 0 {
-            LOG_WARN("VSS_DELETE", "Failed to delete the shadow copy: %x\n", uint(ret))
-        } else {
-            LOG_INFO("VSS_DELETE", "The shadow copy has been successfully deleted")
-        }
-    }
+		LOG_TRACE("VSS_DELETE", "Deleting the shadow copy used for this backup")
+		ret, _, _ := vssBackupComponent.DeleteSnapshots(snapshotID)
+		if ret != 0 {
+			LOG_WARN("VSS_DELETE", "Failed to delete the shadow copy: %x\n", uint(ret))
+		} else {
+			LOG_INFO("VSS_DELETE", "The shadow copy has been successfully deleted")
+		}
+	}
 
-    if shadowLink != "" {
-        err := os.Remove(shadowLink)
-        if err != nil {
-            LOG_WARN("VSS_SYMLINK", "Failed to remove the symbolic link for the shadow copy: %v", err)
-        }
-    }
+	if shadowLink != "" {
+		err := os.Remove(shadowLink)
+		if err != nil {
+			LOG_WARN("VSS_SYMLINK", "Failed to remove the symbolic link for the shadow copy: %v", err)
+		}
+	}
 
-    ole.CoUninitialize()
+	ole.CoUninitialize()
 }
 
-func CreateShadowCopy(top string, shadowCopy bool) (shadowTop string) {
+func CreateShadowCopy(top string, shadowCopy bool, timeoutInSeconds int) (shadowTop string) {
 
-    if !shadowCopy {
-        return top
-    }
+	if !shadowCopy {
+		return top
+	}
 
-    ole.CoInitialize(0)
-    defer ole.CoUninitialize()
+	if timeoutInSeconds <= 60 {
+		timeoutInSeconds = 60
+	}
+	ole.CoInitialize(0)
+	defer ole.CoUninitialize()
 
-    dllVssApi := syscall.NewLazyDLL("VssApi.dll")
-    procCreateVssBackupComponents :=
-                    dllVssApi.NewProc("?CreateVssBackupComponents@@YAJPEAPEAVIVssBackupComponents@@@Z")
-    if runtime.GOARCH == "386" {
-        procCreateVssBackupComponents =
-                    dllVssApi.NewProc("?CreateVssBackupComponents@@YGJPAPAVIVssBackupComponents@@@Z")
-    }
+	dllVssApi := syscall.NewLazyDLL("VssApi.dll")
+	procCreateVssBackupComponents :=
+		dllVssApi.NewProc("?CreateVssBackupComponents@@YAJPEAPEAVIVssBackupComponents@@@Z")
+	if runtime.GOARCH == "386" {
+		procCreateVssBackupComponents =
+			dllVssApi.NewProc("?CreateVssBackupComponents@@YGJPAPAVIVssBackupComponents@@@Z")
+	}
 
-    if len(top) < 3 || top[1] != ':' || (top[2] != '/' && top[2] != '\\') {
-        LOG_ERROR("VSS_PATH", "Invalid repository path: %s", top)
-        return top
-    }
-    volume := top[:1] + ":\\"
+	if len(top) < 3 || top[1] != ':' || (top[2] != '/' && top[2] != '\\') {
+		LOG_ERROR("VSS_PATH", "Invalid repository path: %s", top)
+		return top
+	}
+	volume := top[:1] + ":\\"
 
-    LOG_INFO("VSS_CREATE", "Creating a shadow copy for %s", volume)
+	LOG_INFO("VSS_CREATE", "Creating a shadow copy for %s", volume)
 
-    var unknown *ole.IUnknown
-    r, _, err := procCreateVssBackupComponents.Call(uintptr(unsafe.Pointer(&unknown)))
+	var unknown *ole.IUnknown
+	r, _, err := procCreateVssBackupComponents.Call(uintptr(unsafe.Pointer(&unknown)))
 
-    if r == 0x80070005 {
-        LOG_ERROR("VSS_CREATE", "Only administrators can create shadow copies")
-        return top
-    }
+	if r == 0x80070005 {
+		LOG_ERROR("VSS_CREATE", "Only administrators can create shadow copies")
+		return top
+	}
 
-    if r != 0 {
-        LOG_ERROR("VSS_CREATE", "Failed to create the VSS backup component: %d", r)
-        return top
-    }
+	if r != 0 {
+		LOG_ERROR("VSS_CREATE", "Failed to create the VSS backup component: %d", r)
+		return top
+	}
 
-    vssBackupComponent = getIVSS(unknown, IID_IVSS)
-    if vssBackupComponent == nil {
-        LOG_ERROR("VSS_CREATE", "Failed to create the VSS backup component")
-        return top
-    }
+	vssBackupComponent = getIVSS(unknown, IID_IVSS)
+	if vssBackupComponent == nil {
+		LOG_ERROR("VSS_CREATE", "Failed to create the VSS backup component")
+		return top
+	}
 
-    ret := vssBackupComponent.InitializeForBackup()
-    if ret != 0 {
-        LOG_ERROR("VSS_INIT", "Shadow copy creation failed: InitializeForBackup returned %x", uint(ret))
-        return top
-    }
+	ret := vssBackupComponent.InitializeForBackup()
+	if ret != 0 {
+		LOG_ERROR("VSS_INIT", "Shadow copy creation failed: InitializeForBackup returned %x", uint(ret))
+		return top
+	}
 
-    var async *IVSSAsync
-    ret, async = vssBackupComponent.GatherWriterMetadata()
-    if ret != 0 {
-        LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata returned %x", uint(ret))
-        return top
-    }
+	var async *IVSSAsync
+	ret, async = vssBackupComponent.GatherWriterMetadata()
+	if ret != 0 {
+		LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata returned %x", uint(ret))
+		return top
+	}
 
-    if async == nil {
-        LOG_ERROR("VSS_GATHER",
-                  "Shadow copy creation failed: GatherWriterMetadata failed to return a valid IVssAsync object")
-        return top
-    }
+	if async == nil {
+		LOG_ERROR("VSS_GATHER",
+			"Shadow copy creation failed: GatherWriterMetadata failed to return a valid IVssAsync object")
+		return top
+	}
 
-    if !async.Wait(20) {
-        LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata didn't finish properly")
-        return top
-    }
-    async.Release()
+	if !async.Wait(timeoutInSeconds) {
+		LOG_ERROR("VSS_GATHER", "Shadow copy creation failed: GatherWriterMetadata didn't finish properly")
+		return top
+	}
+	async.Release()
 
-    var snapshotSetID ole.GUID
+	var snapshotSetID ole.GUID
 
-    ret = vssBackupComponent.StartSnapshotSet(&snapshotSetID)
-    if ret != 0 {
-        LOG_ERROR("VSS_START", "Shadow copy creation failed: StartSnapshotSet returned %x", uint(ret))
-        return top
-    }
+	ret = vssBackupComponent.StartSnapshotSet(&snapshotSetID)
+	if ret != 0 {
+		LOG_ERROR("VSS_START", "Shadow copy creation failed: StartSnapshotSet returned %x", uint(ret))
+		return top
+	}
 
-    ret = vssBackupComponent.AddToSnapshotSet(volume, &snapshotID)
-    if ret != 0 {
-        LOG_ERROR("VSS_ADD", "Shadow copy creation failed: AddToSnapshotSet returned %x", uint(ret))
-        return top
-    }
+	ret = vssBackupComponent.AddToSnapshotSet(volume, &snapshotID)
+	if ret != 0 {
+		LOG_ERROR("VSS_ADD", "Shadow copy creation failed: AddToSnapshotSet returned %x", uint(ret))
+		return top
+	}
 
-    s, _ := ole.StringFromIID(&snapshotID)
-    LOG_DEBUG("VSS_ID", "Creating shadow copy %s", s)
+	s, _ := ole.StringFromIID(&snapshotID)
+	LOG_DEBUG("VSS_ID", "Creating shadow copy %s", s)
 
-    ret = vssBackupComponent.SetBackupState()
-    if ret != 0 {
-        LOG_ERROR("VSS_SET", "Shadow copy creation failed: SetBackupState returned %x", uint(ret))
-        return top
-    }
+	ret = vssBackupComponent.SetBackupState()
+	if ret != 0 {
+		LOG_ERROR("VSS_SET", "Shadow copy creation failed: SetBackupState returned %x", uint(ret))
+		return top
+	}
 
-    ret, async = vssBackupComponent.PrepareForBackup()
-    if ret != 0 {
-        LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup returned %x", uint(ret))
-        return top
-    }
-    if async == nil {
-        LOG_ERROR("VSS_PREPARE",
-                  "Shadow copy creation failed: PrepareForBackup failed to return a valid IVssAsync object")
-        return top
-    }
+	ret, async = vssBackupComponent.PrepareForBackup()
+	if ret != 0 {
+		LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup returned %x", uint(ret))
+		return top
+	}
+	if async == nil {
+		LOG_ERROR("VSS_PREPARE",
+			"Shadow copy creation failed: PrepareForBackup failed to return a valid IVssAsync object")
+		return top
+	}
 
-    if !async.Wait(20) {
-        LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup didn't finish properly")
-        return top
-    }
-    async.Release()
+	if !async.Wait(timeoutInSeconds) {
+		LOG_ERROR("VSS_PREPARE", "Shadow copy creation failed: PrepareForBackup didn't finish properly")
+		return top
+	}
+	async.Release()
 
-    ret, async = vssBackupComponent.DoSnapshotSet()
-    if ret != 0 {
-        LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet returned %x", uint(ret))
-        return top
-    }
-    if async == nil {
-        LOG_ERROR("VSS_SNAPSHOT",
-                  "Shadow copy creation failed: DoSnapshotSet failed to return a valid IVssAsync object")
-        return top
-    }
+	ret, async = vssBackupComponent.DoSnapshotSet()
+	if ret != 0 {
+		LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet returned %x", uint(ret))
+		return top
+	}
+	if async == nil {
+		LOG_ERROR("VSS_SNAPSHOT",
+			"Shadow copy creation failed: DoSnapshotSet failed to return a valid IVssAsync object")
+		return top
+	}
 
-    if !async.Wait(60) {
-        LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet didn't finish properly")
-        return top
-    }
-    async.Release()
+	if !async.Wait(timeoutInSeconds) {
+		LOG_ERROR("VSS_SNAPSHOT", "Shadow copy creation failed: DoSnapshotSet didn't finish properly")
+		return top
+	}
+	async.Release()
 
+	properties := SnapshotProperties{}
 
-    properties := SnapshotProperties {
-    }
+	ret = vssBackupComponent.GetSnapshotProperties(snapshotID, &properties)
+	if ret != 0 {
+		LOG_ERROR("VSS_PROPERTIES", "GetSnapshotProperties returned %x", ret)
+		return top
+	}
 
-    ret = vssBackupComponent.GetSnapshotProperties(snapshotID, &properties)
-    if ret != 0 {
-        LOG_ERROR("VSS_PROPERTIES", "GetSnapshotProperties returned %x", ret)
-        return top
-    }
+	SnapshotIDString, _ := ole.StringFromIID(&properties.SnapshotID)
+	SnapshotSetIDString, _ := ole.StringFromIID(&properties.SnapshotSetID)
 
-    SnapshotIDString, _ := ole.StringFromIID(&properties.SnapshotID)
-    SnapshotSetIDString, _ := ole.StringFromIID(&properties.SnapshotSetID)
+	LOG_DEBUG("VSS_PROPERTY", "SnapshotID: %s", SnapshotIDString)
+	LOG_DEBUG("VSS_PROPERTY", "SnapshotSetID: %s", SnapshotSetIDString)
 
-    LOG_DEBUG("VSS_PROPERTY", "SnapshotID: %s", SnapshotIDString)
-    LOG_DEBUG("VSS_PROPERTY", "SnapshotSetID: %s", SnapshotSetIDString)
+	LOG_DEBUG("VSS_PROPERTY", "SnapshotDeviceObject: %s", uint16ArrayToString(properties.SnapshotDeviceObject))
+	LOG_DEBUG("VSS_PROPERTY", "OriginalVolumeName: %s", uint16ArrayToString(properties.OriginalVolumeName))
+	LOG_DEBUG("VSS_PROPERTY", "OriginatingMachine: %s", uint16ArrayToString(properties.OriginatingMachine))
+	LOG_DEBUG("VSS_PROPERTY", "OriginatingMachine: %s", uint16ArrayToString(properties.OriginatingMachine))
+	LOG_DEBUG("VSS_PROPERTY", "ServiceMachine: %s", uint16ArrayToString(properties.ServiceMachine))
+	LOG_DEBUG("VSS_PROPERTY", "ExposedName: %s", uint16ArrayToString(properties.ExposedName))
+	LOG_DEBUG("VSS_PROPERTY", "ExposedPath: %s", uint16ArrayToString(properties.ExposedPath))
 
-    LOG_DEBUG("VSS_PROPERTY", "SnapshotDeviceObject: %s", uint16ArrayToString(properties.SnapshotDeviceObject))
-    LOG_DEBUG("VSS_PROPERTY", "OriginalVolumeName: %s", uint16ArrayToString(properties.OriginalVolumeName))
-    LOG_DEBUG("VSS_PROPERTY", "OriginatingMachine: %s", uint16ArrayToString(properties.OriginatingMachine))
-    LOG_DEBUG("VSS_PROPERTY", "OriginatingMachine: %s", uint16ArrayToString(properties.OriginatingMachine))
-    LOG_DEBUG("VSS_PROPERTY", "ServiceMachine: %s", uint16ArrayToString(properties.ServiceMachine))
-    LOG_DEBUG("VSS_PROPERTY", "ExposedName: %s", uint16ArrayToString(properties.ExposedName))
-    LOG_DEBUG("VSS_PROPERTY", "ExposedPath: %s", uint16ArrayToString(properties.ExposedPath))
+	LOG_INFO("VSS_DONE", "Shadow copy %s created", SnapshotIDString)
 
-    LOG_INFO("VSS_DONE", "Shadow copy %s created", SnapshotIDString)
+	snapshotPath := uint16ArrayToString(properties.SnapshotDeviceObject)
 
-    snapshotPath := uint16ArrayToString(properties.SnapshotDeviceObject)
-    
-    preferencePath := GetDuplicacyPreferencePath()
-    shadowLink =  preferencePath + "\\shadow"
-    os.Remove(shadowLink)
-    err = os.Symlink(snapshotPath + "\\", shadowLink)
-    if err != nil {
-        LOG_ERROR("VSS_SYMLINK", "Failed to create a symbolic link to the shadow copy just created: %v", err)
-        return top
-    }
+	preferencePath := GetDuplicacyPreferencePath()
+	shadowLink = preferencePath + "\\shadow"
+	os.Remove(shadowLink)
+	err = os.Symlink(snapshotPath+"\\", shadowLink)
+	if err != nil {
+		LOG_ERROR("VSS_SYMLINK", "Failed to create a symbolic link to the shadow copy just created: %v", err)
+		return top
+	}
 
-    return shadowLink + "\\" + top[2:]
+	return shadowLink + "\\" + top[2:]
 
 }
-
-

src/duplicacy_snapshot.go

@@ -5,406 +5,480 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "time"
-    "path"
-    "strings"
-    "strconv"
-    "io/ioutil"
-    "encoding/json"
-    "encoding/hex"
+	"encoding/hex"
+	"encoding/json"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"path/filepath"
+	"strconv"
+	"strings"
+	"time"
 )
 
 // Snapshot represents a backup of the repository.
 type Snapshot struct {
-    ID             string      // the snapshot id; must be different for different repositories
-    Revision       int         // the revision number
-    Options        string      // options used to create this snapshot (some not included)
-    Tag            string      // user-assigned tag
-    StartTime      int64       // at what time the snapshot was created
-    EndTime        int64       // at what time the snapshot was done
-    FileSize       int64       // total file size
-    NumberOfFiles  int64       // number of files
+	ID            string // the snapshot id; must be different for different repositories
+	Revision      int    // the revision number
+	Options       string // options used to create this snapshot (some not included)
+	Tag           string // user-assigned tag
+	StartTime     int64  // at what time the snapshot was created
+	EndTime       int64  // at what time the snapshot was done
+	FileSize      int64  // total file size
+	NumberOfFiles int64  // number of files
 
-    // A sequence of chunks whose aggregated content is the json representation of 'Files'.
-    FileSequence   []string
+	// A sequence of chunks whose aggregated content is the json representation of 'Files'.
+	FileSequence []string
 
-    // A sequence of chunks whose aggregated content is the json representation of 'ChunkHashes'.
-    ChunkSequence  []string
+	// A sequence of chunks whose aggregated content is the json representation of 'ChunkHashes'.
+	ChunkSequence []string
 
-    // A sequence of chunks whose aggregated content is the json representation of 'ChunkLengths'.
-    LengthSequence []string
+	// A sequence of chunks whose aggregated content is the json representation of 'ChunkLengths'.
+	LengthSequence []string
 
-    Files          []*Entry    // list of files and subdirectories
+	Files []*Entry // list of files and subdirectories
 
-    ChunkHashes    []string    // a sequence of chunks representing the file content
-    ChunkLengths   []int       // the length of each chunk
+	ChunkHashes  []string // a sequence of chunks representing the file content
+	ChunkLengths []int    // the length of each chunk
 
-    Flag           bool        // used to mark certain snapshots for deletion or copy
+	Flag bool // used to mark certain snapshots for deletion or copy
 
-    discardAttributes bool
+	discardAttributes bool
 }
 
 // CreateEmptySnapshot creates an empty snapshot.
-func CreateEmptySnapshot (id string) (snapshto *Snapshot) {
-    return &Snapshot{
-        ID : id,
-        Revision : 0,
-        StartTime: time.Now().Unix(),
-    }
+func CreateEmptySnapshot(id string) (snapshto *Snapshot) {
+	return &Snapshot{
+		ID:        id,
+		Revision:  0,
+		StartTime: time.Now().Unix(),
+	}
 }
 
 // CreateSnapshotFromDirectory creates a snapshot from the local directory 'top'.  Only 'Files'
 // will be constructed, while 'ChunkHashes' and 'ChunkLengths' can only be populated after uploading.
-func CreateSnapshotFromDirectory(id string, top string) (snapshot *Snapshot, skippedDirectories []string,
-                                                         skippedFiles []string, err error) {
+func CreateSnapshotFromDirectory(id string, top string, nobackupFile string) (snapshot *Snapshot, skippedDirectories []string,
+	skippedFiles []string, err error) {
 
-    snapshot = &Snapshot {
-        ID : id,
-        Revision: 0,
-        StartTime: time.Now().Unix(),
-    }
+	snapshot = &Snapshot{
+		ID:        id,
+		Revision:  0,
+		StartTime: time.Now().Unix(),
+	}
 
-    var patterns []string
-    
-    patternFile, err := ioutil.ReadFile(path.Join(GetDuplicacyPreferencePath(), "filters"))
-    if err == nil {
-        for _, pattern := range strings.Split(string(patternFile), "\n") {
-            pattern = strings.TrimSpace(pattern)
-            if len(pattern) == 0 {
-                continue
-            }
+	var patterns []string
 
-            if pattern[0] != '+' && pattern[0] != '-' {
-                pattern = "+" + pattern
-            }
+	patterns = ProcessFilters()
 
-            if pattern == "+" || pattern == "-" {
-                continue
-            }
+	directories := make([]*Entry, 0, 256)
+	directories = append(directories, CreateEntry("", 0, 0, 0))
 
-            patterns = append(patterns, pattern)
-        }
+	snapshot.Files = make([]*Entry, 0, 256)
 
-        LOG_INFO("SNAPSHOT_FILTER", "Loaded %d include/exclude pattern(s)", len(patterns))
+	attributeThreshold := 1024 * 1024
+	if attributeThresholdValue, found := os.LookupEnv("DUPLICACY_ATTRIBUTE_THRESHOLD"); found && attributeThresholdValue != "" {
+		attributeThreshold, _ = strconv.Atoi(attributeThresholdValue)
+	}
 
-        if IsTracing() {
-            for _, pattern := range patterns {
-                LOG_TRACE("SNAPSHOT_PATTERN", "Pattern: %s", pattern)
-            }
-        }
+	for len(directories) > 0 {
 
-    }
+		directory := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
+		snapshot.Files = append(snapshot.Files, directory)
+		subdirectories, skipped, err := ListEntries(top, directory.Path, &snapshot.Files, patterns, nobackupFile, snapshot.discardAttributes)
+		if err != nil {
+			LOG_WARN("LIST_FAILURE", "Failed to list subdirectory: %v", err)
+			skippedDirectories = append(skippedDirectories, directory.Path)
+			continue
+		}
 
-    directories := make([]*Entry, 0, 256)
-    directories = append(directories, CreateEntry("", 0, 0, 0))
+		directories = append(directories, subdirectories...)
+		skippedFiles = append(skippedFiles, skipped...)
 
-    snapshot.Files = make([]*Entry, 0, 256)
+		if !snapshot.discardAttributes && len(snapshot.Files) > attributeThreshold {
+			LOG_INFO("LIST_ATTRIBUTES", "Discarding file attributes")
+			snapshot.discardAttributes = true
+			for _, file := range snapshot.Files {
+				file.Attributes = nil
+			}
+		}
+	}
 
-    attributeThreshold := 1024 * 1024
-    if attributeThresholdValue, found := os.LookupEnv("DUPLICACY_ATTRIBUTE_THRESHOLD"); found && attributeThresholdValue != "" {
-        attributeThreshold, _ = strconv.Atoi(attributeThresholdValue)
-    }
+	// Remove the root entry
+	snapshot.Files = snapshot.Files[1:]
 
-    for len(directories) > 0 {
+	return snapshot, skippedDirectories, skippedFiles, nil
+}
 
-        directory := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
-        snapshot.Files = append(snapshot.Files, directory)
-        subdirectories, skipped, err := ListEntries(top, directory.Path, &snapshot.Files, patterns, snapshot.discardAttributes)
-        if err != nil {
-            LOG_WARN("LIST_FAILURE", "Failed to list subdirectory: %v", err)
-            skippedDirectories = append(skippedDirectories, directory.Path)
-            continue
-        }
+func AppendPattern(patterns []string, new_pattern string) (new_patterns []string) {
+	for _, pattern := range patterns {
+		if pattern == new_pattern {
+			LOG_INFO("SNAPSHOT_FILTER", "Ignoring duplicate pattern: %s ...", new_pattern)
+			return patterns
+		}
+	}
+	new_patterns = append(patterns, new_pattern)
+	return new_patterns
+}
+func ProcessFilters() (patterns []string) {
+	patterns = ProcessFilterFile(joinPath(GetDuplicacyPreferencePath(), "filters"), make([]string, 0))
 
-        directories = append(directories, subdirectories...)
-        skippedFiles = append(skippedFiles, skipped...)
+	LOG_DEBUG("REGEX_DEBUG", "There are %d compiled regular expressions stored", len(RegexMap))
 
-        if !snapshot.discardAttributes && len(snapshot.Files) > attributeThreshold {
-            LOG_INFO("LIST_ATTRIBUTES", "Discarding file attributes")
-            snapshot.discardAttributes = true
-            for _, file := range snapshot.Files {
-                file.Attributes = nil
-            }
-        }
-    }
+	LOG_INFO("SNAPSHOT_FILTER", "Loaded %d include/exclude pattern(s)", len(patterns))
 
-    // Remove the root entry
-    snapshot.Files = snapshot.Files[1:]
+	if IsTracing() {
+		for _, pattern := range patterns {
+			LOG_TRACE("SNAPSHOT_PATTERN", "Pattern: %s", pattern)
+		}
 
-    return snapshot, skippedDirectories, skippedFiles, nil
+	}
+
+	return patterns
+}
+
+func ProcessFilterFile(patternFile string, includedFiles []string) (patterns []string) {
+	for _, file := range includedFiles {
+		if file == patternFile {
+			// cycle in include mechanism discovered.
+			LOG_ERROR("SNAPSHOT_FILTER", "The filter file %s has already been included", patternFile)
+			return patterns
+		}
+	}
+	includedFiles = append(includedFiles, patternFile)
+	LOG_INFO("SNAPSHOT_FILTER", "Parsing filter file %s", patternFile)
+	patternFileContent, err := ioutil.ReadFile(patternFile)
+	if err == nil {
+		patternFileLines := strings.Split(string(patternFileContent), "\n")
+		patterns = ProcessFilterLines(patternFileLines, includedFiles)
+	}
+	return patterns
+}
+
+func ProcessFilterLines(patternFileLines []string, includedFiles []string) (patterns []string) {
+	for _, pattern := range patternFileLines {
+		pattern = strings.TrimSpace(pattern)
+		if len(pattern) == 0 {
+			continue
+		}
+
+		if strings.HasPrefix(pattern, "@") {
+			patternIncludeFile := strings.TrimSpace(pattern[1:])
+			if patternIncludeFile == "" {
+				continue
+			}
+			if ! filepath.IsAbs(patternIncludeFile) {
+				basePath := ""
+				if len(includedFiles) == 0 {
+					basePath, _ = os.Getwd()
+				} else {
+					basePath = filepath.Dir(includedFiles[len(includedFiles)-1])
+				}
+				patternIncludeFile = joinPath(basePath, patternIncludeFile)
+			}
+			for _, pattern := range ProcessFilterFile(patternIncludeFile, includedFiles) {
+				patterns = AppendPattern(patterns, pattern)
+			}
+			continue
+		}
+
+		if pattern[0] == '#' {
+			continue
+		}
+
+		if IsUnspecifiedFilter(pattern) {
+			pattern = "+" + pattern
+		}
+
+		if IsEmptyFilter(pattern) {
+			continue
+		}
+
+		if strings.HasPrefix(pattern, "i:") || strings.HasPrefix(pattern, "e:") {
+			valid, err := IsValidRegex(pattern[2:])
+			if !valid || err != nil {
+				LOG_ERROR("SNAPSHOT_FILTER", "Invalid regular expression encountered for filter: \"%s\", error: %v", pattern, err)
+			}
+		}
+
+		patterns = AppendPattern(patterns, pattern)
+	}
+
+	return patterns
 }
 
 // This is the struct used to save/load incomplete snapshots
 type IncompleteSnapshot struct {
-    Files [] *Entry
-    ChunkHashes []string
-    ChunkLengths [] int
+	Files        []*Entry
+	ChunkHashes  []string
+	ChunkLengths []int
 }
 
 // LoadIncompleteSnapshot loads the incomplete snapshot if it exists
 func LoadIncompleteSnapshot() (snapshot *Snapshot) {
-    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")
-    description, err := ioutil.ReadFile(snapshotFile)    
-    if err != nil {
-        LOG_DEBUG("INCOMPLETE_LOCATE", "Failed to locate incomplete snapshot: %v", err)
-        return nil
-    }
+	snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")
+	description, err := ioutil.ReadFile(snapshotFile)
+	if err != nil {
+		LOG_DEBUG("INCOMPLETE_LOCATE", "Failed to locate incomplete snapshot: %v", err)
+		return nil
+	}
 
-    var incompleteSnapshot IncompleteSnapshot
+	var incompleteSnapshot IncompleteSnapshot
 
-    err = json.Unmarshal(description, &incompleteSnapshot)
-    if err != nil {
-        LOG_DEBUG("INCOMPLETE_PARSE", "Failed to parse incomplete snapshot: %v", err)
-        return nil
-    }
+	err = json.Unmarshal(description, &incompleteSnapshot)
+	if err != nil {
+		LOG_DEBUG("INCOMPLETE_PARSE", "Failed to parse incomplete snapshot: %v", err)
+		return nil
+	}
 
-    var chunkHashes []string
-    for _, chunkHash := range incompleteSnapshot.ChunkHashes {
-        hash, err := hex.DecodeString(chunkHash)
-        if err != nil {
-            LOG_DEBUG("INCOMPLETE_DECODE", "Failed to decode incomplete snapshot: %v", err)
-            return nil
-        }
-        chunkHashes = append(chunkHashes, string(hash))
-    }
+	var chunkHashes []string
+	for _, chunkHash := range incompleteSnapshot.ChunkHashes {
+		hash, err := hex.DecodeString(chunkHash)
+		if err != nil {
+			LOG_DEBUG("INCOMPLETE_DECODE", "Failed to decode incomplete snapshot: %v", err)
+			return nil
+		}
+		chunkHashes = append(chunkHashes, string(hash))
+	}
 
-    snapshot = &Snapshot {
-        Files: incompleteSnapshot.Files,
-        ChunkHashes: chunkHashes,
-        ChunkLengths: incompleteSnapshot.ChunkLengths,
-    }      
-    LOG_INFO("INCOMPLETE_LOAD", "Incomplete snpashot loaded from %s", snapshotFile)
-    return snapshot
+	snapshot = &Snapshot{
+		Files:        incompleteSnapshot.Files,
+		ChunkHashes:  chunkHashes,
+		ChunkLengths: incompleteSnapshot.ChunkLengths,
+	}
+	LOG_INFO("INCOMPLETE_LOAD", "Incomplete snapshot loaded from %s", snapshotFile)
+	return snapshot
 }
 
 // SaveIncompleteSnapshot saves the incomplete snapshot under the preference directory
 func SaveIncompleteSnapshot(snapshot *Snapshot) {
-    var files []*Entry
-    for _, file := range snapshot.Files {
-        // All unprocessed files will have a size of -1
-        if file.Size >= 0 {
-            file.Attributes = nil
-            files = append(files, file)
-        } else {
-            break
-        }
-    }
-    var chunkHashes []string
-    for _, chunkHash := range snapshot.ChunkHashes {
-        chunkHashes = append(chunkHashes, hex.EncodeToString([]byte(chunkHash)))
-    }
+	var files []*Entry
+	for _, file := range snapshot.Files {
+		// All unprocessed files will have a size of -1
+		if file.Size >= 0 {
+			file.Attributes = nil
+			files = append(files, file)
+		} else {
+			break
+		}
+	}
+	var chunkHashes []string
+	for _, chunkHash := range snapshot.ChunkHashes {
+		chunkHashes = append(chunkHashes, hex.EncodeToString([]byte(chunkHash)))
+	}
 
-    incompleteSnapshot := IncompleteSnapshot {
-        Files: files,
-        ChunkHashes: chunkHashes,
-        ChunkLengths: snapshot.ChunkLengths,
-    }
+	incompleteSnapshot := IncompleteSnapshot{
+		Files:        files,
+		ChunkHashes:  chunkHashes,
+		ChunkLengths: snapshot.ChunkLengths,
+	}
 
-    description, err := json.MarshalIndent(incompleteSnapshot, "", "  ")
-    if err != nil {
-        LOG_WARN("INCOMPLETE_ENCODE", "Failed to encode the incomplete snapshot: %v", err)
-        return
-    }
+	description, err := json.MarshalIndent(incompleteSnapshot, "", "  ")
+	if err != nil {
+		LOG_WARN("INCOMPLETE_ENCODE", "Failed to encode the incomplete snapshot: %v", err)
+		return
+	}
 
-    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")    
-    err = ioutil.WriteFile(snapshotFile, description, 0644)
-    if err != nil {
-        LOG_WARN("INCOMPLETE_WRITE", "Failed to save the incomplete snapshot: %v", err)        
-        return
-    }
+	snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")
+	err = ioutil.WriteFile(snapshotFile, description, 0644)
+	if err != nil {
+		LOG_WARN("INCOMPLETE_WRITE", "Failed to save the incomplete snapshot: %v", err)
+		return
+	}
 
-    LOG_INFO("INCOMPLETE_SAVE", "Incomplete snapshot saved to %s", snapshotFile)
+	LOG_INFO("INCOMPLETE_SAVE", "Incomplete snapshot saved to %s", snapshotFile)
 }
 
 func RemoveIncompleteSnapshot() {
-    snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")    
-    if stat, err := os.Stat(snapshotFile); err == nil && !stat.IsDir() {
-        err = os.Remove(snapshotFile)
-        if err != nil {
-            LOG_INFO("INCOMPLETE_SAVE", "Failed to remove ncomplete snapshot: %v", err)
-        } else {
-            LOG_INFO("INCOMPLETE_SAVE", "Removed incomplete snapshot %s", snapshotFile)    
-        }
-    }
+	snapshotFile := path.Join(GetDuplicacyPreferencePath(), "incomplete")
+	if stat, err := os.Stat(snapshotFile); err == nil && !stat.IsDir() {
+		err = os.Remove(snapshotFile)
+		if err != nil {
+			LOG_INFO("INCOMPLETE_SAVE", "Failed to remove ncomplete snapshot: %v", err)
+		} else {
+			LOG_INFO("INCOMPLETE_SAVE", "Removed incomplete snapshot %s", snapshotFile)
+		}
+	}
 }
 
 // CreateSnapshotFromDescription creates a snapshot from json decription.
 func CreateSnapshotFromDescription(description []byte) (snapshot *Snapshot, err error) {
 
-    var root map[string] interface{}
+	var root map[string]interface{}
 
-    err = json.Unmarshal(description, &root)
-    if err != nil {
-        return nil, err
-    }
+	err = json.Unmarshal(description, &root)
+	if err != nil {
+		return nil, err
+	}
 
-    snapshot = &Snapshot {}
+	snapshot = &Snapshot{}
 
-    if value, ok := root["id"]; !ok {
-        return nil, fmt.Errorf("No id is specified in the snapshot")
-    } else if snapshot.ID, ok = value.(string); !ok {
-        return nil, fmt.Errorf("Invalid id is specified in the snapshot")
-    }
+	if value, ok := root["id"]; !ok {
+		return nil, fmt.Errorf("No id is specified in the snapshot")
+	} else if snapshot.ID, ok = value.(string); !ok {
+		return nil, fmt.Errorf("Invalid id is specified in the snapshot")
+	}
 
-    if value, ok := root["revision"]; !ok {
-        return nil, fmt.Errorf("No revision is specified in the snapshot")
-    } else if _, ok = value.(float64); !ok {
-        return nil, fmt.Errorf("Invalid revision is specified in the snapshot")
-    } else {
-        snapshot.Revision = int(value.(float64))
-    }
+	if value, ok := root["revision"]; !ok {
+		return nil, fmt.Errorf("No revision is specified in the snapshot")
+	} else if _, ok = value.(float64); !ok {
+		return nil, fmt.Errorf("Invalid revision is specified in the snapshot")
+	} else {
+		snapshot.Revision = int(value.(float64))
+	}
 
-    if value, ok := root["tag"]; !ok {
-    } else if snapshot.Tag, ok = value.(string); !ok {
-        return nil, fmt.Errorf("Invalid tag is specified in the snapshot")
-    }
+	if value, ok := root["tag"]; !ok {
+	} else if snapshot.Tag, ok = value.(string); !ok {
+		return nil, fmt.Errorf("Invalid tag is specified in the snapshot")
+	}
 
-    if value, ok := root["options"]; !ok {
-    } else if snapshot.Options, ok = value.(string); !ok {
-        return nil, fmt.Errorf("Invalid options is specified in the snapshot")
-    }
+	if value, ok := root["options"]; !ok {
+	} else if snapshot.Options, ok = value.(string); !ok {
+		return nil, fmt.Errorf("Invalid options is specified in the snapshot")
+	}
 
-    if value, ok := root["start_time"]; !ok {
-        return nil, fmt.Errorf("No creation time is specified in the snapshot")
-    } else if _, ok = value.(float64); !ok {
-        return nil, fmt.Errorf("Invalid creation time is specified in the snapshot")
-    } else {
-        snapshot.StartTime = int64(value.(float64))
-    }
+	if value, ok := root["start_time"]; !ok {
+		return nil, fmt.Errorf("No creation time is specified in the snapshot")
+	} else if _, ok = value.(float64); !ok {
+		return nil, fmt.Errorf("Invalid creation time is specified in the snapshot")
+	} else {
+		snapshot.StartTime = int64(value.(float64))
+	}
 
-    if value, ok := root["end_time"]; !ok {
-        return nil, fmt.Errorf("No creation time is specified in the snapshot")
-    } else if _, ok = value.(float64); !ok {
-        return nil, fmt.Errorf("Invalid creation time is specified in the snapshot")
-    } else {
-        snapshot.EndTime = int64(value.(float64))
-    }
+	if value, ok := root["end_time"]; !ok {
+		return nil, fmt.Errorf("No creation time is specified in the snapshot")
+	} else if _, ok = value.(float64); !ok {
+		return nil, fmt.Errorf("Invalid creation time is specified in the snapshot")
+	} else {
+		snapshot.EndTime = int64(value.(float64))
+	}
 
-    if value, ok := root["file_size"]; ok {
-        if _, ok = value.(float64); ok {
-            snapshot.FileSize = int64(value.(float64))
-        }
-    }
+	if value, ok := root["file_size"]; ok {
+		if _, ok = value.(float64); ok {
+			snapshot.FileSize = int64(value.(float64))
+		}
+	}
 
-    if value, ok := root["number_of_files"]; ok {
-        if _, ok = value.(float64); ok {
-            snapshot.NumberOfFiles = int64(value.(float64))
-        }
-    }
+	if value, ok := root["number_of_files"]; ok {
+		if _, ok = value.(float64); ok {
+			snapshot.NumberOfFiles = int64(value.(float64))
+		}
+	}
 
-    for _, sequenceType := range []string { "files", "chunks", "lengths" } {
-        if value, ok := root[sequenceType]; !ok {
-            return nil, fmt.Errorf("No %s are specified in the snapshot", sequenceType)
-        } else if _, ok = value.([]interface{}); !ok {
-            return nil, fmt.Errorf("Invalid %s are specified in the snapshot", sequenceType)
-        } else {
-            array := value.([]interface{})
-            sequence := make([]string, len(array))
-            for i := 0; i < len(array); i++ {
-                if hashInHex, ok := array[i].(string); !ok {
-                    return nil, fmt.Errorf("Invalid file sequence is specified in the snapshot")
-                } else if hash, err := hex.DecodeString(hashInHex); err != nil {
-                    return nil, fmt.Errorf("Hash %s is not a valid hex string in the snapshot", hashInHex)
-                } else {
-                    sequence[i] = string(hash)
-                }
-            }
+	for _, sequenceType := range []string{"files", "chunks", "lengths"} {
+		if value, ok := root[sequenceType]; !ok {
+			return nil, fmt.Errorf("No %s are specified in the snapshot", sequenceType)
+		} else if _, ok = value.([]interface{}); !ok {
+			return nil, fmt.Errorf("Invalid %s are specified in the snapshot", sequenceType)
+		} else {
+			array := value.([]interface{})
+			sequence := make([]string, len(array))
+			for i := 0; i < len(array); i++ {
+				if hashInHex, ok := array[i].(string); !ok {
+					return nil, fmt.Errorf("Invalid file sequence is specified in the snapshot")
+				} else if hash, err := hex.DecodeString(hashInHex); err != nil {
+					return nil, fmt.Errorf("Hash %s is not a valid hex string in the snapshot", hashInHex)
+				} else {
+					sequence[i] = string(hash)
+				}
+			}
 
-            snapshot.SetSequence(sequenceType, sequence)
-        }
-    }
+			snapshot.SetSequence(sequenceType, sequence)
+		}
+	}
 
-    return snapshot, nil
+	return snapshot, nil
 }
 
 // LoadChunks construct 'ChunkHashes' from the json description.
 func (snapshot *Snapshot) LoadChunks(description []byte) (err error) {
 
-    var root [] interface {}
-    err = json.Unmarshal(description, &root)
-    if err != nil {
-        return err
-    }
+	var root []interface{}
+	err = json.Unmarshal(description, &root)
+	if err != nil {
+		return err
+	}
 
-    snapshot.ChunkHashes = make([]string, len(root))
+	snapshot.ChunkHashes = make([]string, len(root))
 
-    for i, object := range root {
-        if hashInHex, ok := object.(string); !ok {
-            return fmt.Errorf("Invalid chunk hash is specified in the snapshot")
-        } else if hash, err := hex.DecodeString(hashInHex); err != nil {
-            return fmt.Errorf("The chunk hash %s is not a valid hex string", hashInHex)
-        } else {
-            snapshot.ChunkHashes[i] = string(hash)
-        }
-    }
+	for i, object := range root {
+		if hashInHex, ok := object.(string); !ok {
+			return fmt.Errorf("Invalid chunk hash is specified in the snapshot")
+		} else if hash, err := hex.DecodeString(hashInHex); err != nil {
+			return fmt.Errorf("The chunk hash %s is not a valid hex string", hashInHex)
+		} else {
+			snapshot.ChunkHashes[i] = string(hash)
+		}
+	}
 
-    return err
+	return err
+}
+
+// ClearChunks removes loaded chunks from memory
+func (snapshot *Snapshot) ClearChunks() {
+	snapshot.ChunkHashes = nil
 }
 
 // LoadLengths construct 'ChunkLengths' from the json description.
 func (snapshot *Snapshot) LoadLengths(description []byte) (err error) {
-    return json.Unmarshal(description, &snapshot.ChunkLengths)
+	return json.Unmarshal(description, &snapshot.ChunkLengths)
 }
 
 // MarshalJSON creates a json representation of the snapshot.
-func (snapshot *Snapshot) MarshalJSON() ([] byte, error) {
+func (snapshot *Snapshot) MarshalJSON() ([]byte, error) {
 
-    object := make(map[string]interface{})
+	object := make(map[string]interface{})
 
-    object["id"] = snapshot.ID
-    object["revision"] = snapshot.Revision
-    object["options"] = snapshot.Options
-    object["tag"] = snapshot.Tag
-    object["start_time"] = snapshot.StartTime
-    object["end_time"] = snapshot.EndTime
+	object["id"] = snapshot.ID
+	object["revision"] = snapshot.Revision
+	object["options"] = snapshot.Options
+	object["tag"] = snapshot.Tag
+	object["start_time"] = snapshot.StartTime
+	object["end_time"] = snapshot.EndTime
 
-    if snapshot.FileSize != 0 && snapshot.NumberOfFiles != 0 {
-        object["file_size"] = snapshot.FileSize
-        object["number_of_files"] = snapshot.NumberOfFiles
-    }
-    object["files"] = encodeSequence(snapshot.FileSequence)
-    object["chunks"] = encodeSequence(snapshot.ChunkSequence)
-    object["lengths"] = encodeSequence(snapshot.LengthSequence)
+	if snapshot.FileSize != 0 && snapshot.NumberOfFiles != 0 {
+		object["file_size"] = snapshot.FileSize
+		object["number_of_files"] = snapshot.NumberOfFiles
+	}
+	object["files"] = encodeSequence(snapshot.FileSequence)
+	object["chunks"] = encodeSequence(snapshot.ChunkSequence)
+	object["lengths"] = encodeSequence(snapshot.LengthSequence)
 
-    return json.Marshal(object)
+	return json.Marshal(object)
 }
 
 // MarshalSequence creates a json represetion for the specified chunk sequence.
-func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([] byte, error) {
+func (snapshot *Snapshot) MarshalSequence(sequenceType string) ([]byte, error) {
 
-    if sequenceType == "files" {
-        return json.Marshal(snapshot.Files)
-    } else if sequenceType == "chunks" {
-        return json.Marshal(encodeSequence(snapshot.ChunkHashes))
-    } else {
-        return json.Marshal(snapshot.ChunkLengths)
-    }
+	if sequenceType == "files" {
+		return json.Marshal(snapshot.Files)
+	} else if sequenceType == "chunks" {
+		return json.Marshal(encodeSequence(snapshot.ChunkHashes))
+	} else {
+		return json.Marshal(snapshot.ChunkLengths)
+	}
 }
 
 // SetSequence assign a chunk sequence to the specified field.
-func (snapshot *Snapshot) SetSequence(sequenceType string, sequence [] string) {
-    if sequenceType == "files" {
-        snapshot.FileSequence = sequence
-    } else if sequenceType == "chunks" {
-        snapshot.ChunkSequence = sequence
-    } else {
-        snapshot.LengthSequence = sequence
-    }
+func (snapshot *Snapshot) SetSequence(sequenceType string, sequence []string) {
+	if sequenceType == "files" {
+		snapshot.FileSequence = sequence
+	} else if sequenceType == "chunks" {
+		snapshot.ChunkSequence = sequence
+	} else {
+		snapshot.LengthSequence = sequence
+	}
 }
 
 // encodeSequence turns a sequence of binary hashes into a sequence of hex hashes.
-func encodeSequence(sequence[] string) ([] string) {
+func encodeSequence(sequence []string) []string {
 
-    sequenceInHex := make([]string, len(sequence))
+	sequenceInHex := make([]string, len(sequence))
 
-    for i, hash := range sequence {
-        sequenceInHex[i] = hex.EncodeToString([]byte(hash))
-    }
+	for i, hash := range sequence {
+		sequenceInHex[i] = hex.EncodeToString([]byte(hash))
+	}
 
-    return sequenceInHex
+	return sequenceInHex
 }
-
-

src/duplicacy_storage_test.go

@@ -5,22 +5,21 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "time"
-    "flag"
-    "path"
-    "testing"
-    "strings"
-    "strconv"
-    "io/ioutil"
-    "crypto/sha256"
-    "encoding/hex"
-    "encoding/json"
-    "runtime/debug"
+	"crypto/sha256"
+	"encoding/hex"
+	"encoding/json"
+	"flag"
+	"fmt"
+	"io/ioutil"
+	"os"
+	"path"
+	"runtime/debug"
+	"strconv"
+	"testing"
+	"time"
 
-    crypto_rand "crypto/rand"
-    "math/rand"
+	crypto_rand "crypto/rand"
+	"math/rand"
 )
 
 var testStorageName string
@@ -30,427 +29,580 @@ var testThreads int
 var testFixedChunkSize bool
 
 func init() {
-    flag.StringVar(&testStorageName, "storage", "", "the test storage to use")
-    flag.IntVar(&testRateLimit, "limit-rate", 0, "maximum transfer speed in kbytes/sec")
-    flag.BoolVar(&testQuickMode, "quick", false, "quick test")
-    flag.IntVar(&testThreads, "threads", 1, "number of downloading/uploading threads")
-    flag.BoolVar(&testFixedChunkSize, "fixed-chunk-size", false, "fixed chunk size")
-    flag.Parse()
+	flag.StringVar(&testStorageName, "storage", "", "the test storage to use")
+	flag.IntVar(&testRateLimit, "limit-rate", 0, "maximum transfer speed in kbytes/sec")
+	flag.BoolVar(&testQuickMode, "quick", false, "quick test")
+	flag.IntVar(&testThreads, "threads", 1, "number of downloading/uploading threads")
+	flag.BoolVar(&testFixedChunkSize, "fixed-chunk-size", false, "fixed chunk size")
+	flag.Parse()
 }
 
 func loadStorage(localStoragePath string, threads int) (Storage, error) {
 
-    if testStorageName == "" || testStorageName == "file" {
-        return CreateFileStorage(localStoragePath, threads)
-    }
+	if testStorageName == "" || testStorageName == "file" {
+		storage, err := CreateFileStorage(localStoragePath, false, threads)
+		if storage != nil {
+			// Use a read level of at least 2 because this will catch more errors than a read level of 1.
+			storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		}
+		return storage, err
+	}
 
-    config, err := ioutil.ReadFile("test_storage.conf")
-    if err != nil {
-        return nil, err
-    }
+	description, err := ioutil.ReadFile("test_storage.conf")
+	if err != nil {
+		return nil, err
+	}
 
-    storages := make(map[string]map[string]string)
+	configs := make(map[string]map[string]string)
 
-    err = json.Unmarshal(config, &storages)
-    if err != nil {
-        return nil, err
-    }
+	err = json.Unmarshal(description, &configs)
+	if err != nil {
+		return nil, err
+	}
 
-    storage, found := storages[testStorageName]
-    if !found {
-        return nil, fmt.Errorf("No storage named '%s' found", testStorageName)
-    }
+	config, found := configs[testStorageName]
+	if !found {
+		return nil, fmt.Errorf("No storage named '%s' found", testStorageName)
+	}
 
-    if testStorageName == "sftp" {
-        port, _ := strconv.Atoi(storage["port"])
-        return CreateSFTPStorageWithPassword(storage["server"], port, storage["username"], storage["directory"], storage["password"], threads)
-    } else if testStorageName == "s3" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, false)
-    } else if testStorageName == "s3c" {
-        return CreateS3CStorage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads)
-    } else if testStorageName == "minio" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, false, true)
-    } else if testStorageName == "minios" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, true)
-    } else if testStorageName == "dropbox" {
-        return CreateDropboxStorage(storage["token"], storage["directory"], threads)
-    } else if testStorageName == "b2" {
-        return CreateB2Storage(storage["account"], storage["key"], storage["bucket"], threads)
-    } else if testStorageName == "gcs-s3" {
-        return CreateS3Storage(storage["region"], storage["endpoint"], storage["bucket"], storage["directory"], storage["access_key"], storage["secret_key"], threads, true, false)
-    } else if testStorageName == "gcs" {
-        return CreateGCSStorage(storage["token_file"], storage["bucket"], storage["directory"], threads)
-    } else if testStorageName == "gcs-sa" {
-        return CreateGCSStorage(storage["token_file"], storage["bucket"], storage["directory"], threads)
-    } else if testStorageName == "azure" {
-        return CreateAzureStorage(storage["account"], storage["key"], storage["container"], threads)
-    } else if testStorageName == "acd" {
-        return CreateACDStorage(storage["token_file"], storage["storage_path"], threads)
-    } else if testStorageName == "gcd" {
-        return CreateGCDStorage(storage["token_file"], storage["storage_path"], threads)
-    } else if testStorageName == "one" {
-        return CreateOneDriveStorage(storage["token_file"], storage["storage_path"], threads)
-    } else if testStorageName == "hubic" {
-        return CreateHubicStorage(storage["token_file"], storage["storage_path"], threads)
-    } else {
-        return nil, fmt.Errorf("Invalid storage named: %s", testStorageName)
-    }
+	if testStorageName == "flat" {
+		storage, err := CreateFileStorage(localStoragePath, false, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "samba" {
+		storage, err := CreateFileStorage(localStoragePath, true, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "sftp" {
+		port, _ := strconv.Atoi(config["port"])
+		storage, err := CreateSFTPStorageWithPassword(config["server"], port, config["username"], config["directory"], 2, config["password"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "s3" {
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, false)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "wasabi" {
+		storage, err := CreateWasabiStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "s3c" {
+		storage, err := CreateS3CStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "digitalocean" {
+		storage, err := CreateS3CStorage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "minio" {
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, false, true)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "minios" {
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, true)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "dropbox" {
+		storage, err := CreateDropboxStorage(config["token"], config["directory"], 1, threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "b2" {
+		storage, err := CreateB2Storage(config["account"], config["key"], config["bucket"], config["directory"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "gcs-s3" {
+		storage, err := CreateS3Storage(config["region"], config["endpoint"], config["bucket"], config["directory"], config["access_key"], config["secret_key"], threads, true, false)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "gcs" {
+		storage, err := CreateGCSStorage(config["token_file"], config["bucket"], config["directory"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "gcs-sa" {
+		storage, err := CreateGCSStorage(config["token_file"], config["bucket"], config["directory"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "azure" {
+		storage, err := CreateAzureStorage(config["account"], config["key"], config["container"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "acd" {
+		storage, err := CreateACDStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "gcd" {
+		storage, err := CreateGCDStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "one" {
+		storage, err := CreateOneDriveStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "hubic" {
+		storage, err := CreateHubicStorage(config["token_file"], config["storage_path"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "memset" {
+		storage, err := CreateSwiftStorage(config["storage_url"], config["key"], threads)
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	} else if testStorageName == "pcloud" || testStorageName == "box" {
+		storage, err := CreateWebDAVStorage(config["host"], 0, config["username"], config["password"], config["storage_path"], false, threads)
+		if err != nil {
+			return nil, err
+		}
+		storage.SetDefaultNestingLevels([]int{2, 3}, 2)
+		return storage, err
+	}
+	return nil, fmt.Errorf("Invalid storage named: %s", testStorageName)
 }
 
 func cleanStorage(storage Storage) {
 
-    directories := make([]string, 0, 1024)
-    snapshots := make([]string, 0, 1024)
+	directories := make([]string, 0, 1024)
+	snapshots := make([]string, 0, 1024)
 
-    directories = append(directories, "snapshots/")
+	directories = append(directories, "snapshots/")
 
-    LOG_INFO("STORAGE_LIST", "Listing snapshots in the storage") 
-    for len(directories) > 0 {
+	LOG_INFO("STORAGE_LIST", "Listing snapshots in the storage")
+	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
-        files, _, err := storage.ListFiles(0, dir)
-        if err != nil {
-            LOG_ERROR("STORAGE_LIST", "Failed to list the directory %s: %v", dir, err)
-            return
-        }
+		files, _, err := storage.ListFiles(0, dir)
+		if err != nil {
+			LOG_ERROR("STORAGE_LIST", "Failed to list the directory %s: %v", dir, err)
+			return
+		}
 
-        for _, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
-            } else {
-                snapshots = append(snapshots, dir + file)
-            }
-        }
-    }
+		for _, file := range files {
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
+			} else {
+				snapshots = append(snapshots, dir+file)
+			}
+		}
+	}
 
-    LOG_INFO("STORAGE_DELETE", "Deleting %d snapshots in the storage", len(snapshots))
-    for _, snapshot := range snapshots {
-        storage.DeleteFile(0, snapshot)
-    } 
+	LOG_INFO("STORAGE_DELETE", "Deleting %d snapshots in the storage", len(snapshots))
+	for _, snapshot := range snapshots {
+		storage.DeleteFile(0, snapshot)
+	}
 
-    for _, chunk := range listChunks(storage) {
-        storage.DeleteFile(0, "chunks/" + chunk)
-    }
+	for _, chunk := range listChunks(storage) {
+		storage.DeleteFile(0, "chunks/"+chunk)
+	}
 
-    storage.DeleteFile(0, "config")
+	storage.DeleteFile(0, "config")
 
-    return
+	return
 }
 
 func listChunks(storage Storage) (chunks []string) {
 
-    directories := make([]string, 0, 1024)
+	directories := make([]string, 0, 1024)
 
-    directories = append(directories, "chunks/")
+	directories = append(directories, "chunks/")
 
-    for len(directories) > 0 {
+	for len(directories) > 0 {
 
-        dir := directories[len(directories) - 1]
-        directories = directories[:len(directories) - 1]
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
 
-        files, _, err := storage.ListFiles(0, dir)
-        if err != nil {
-            LOG_ERROR("CHUNK_LIST", "Failed to list the directory %s: %v", dir, err)
-            return nil
-        }
+		files, _, err := storage.ListFiles(0, dir)
+		if err != nil {
+			LOG_ERROR("CHUNK_LIST", "Failed to list the directory %s: %v", dir, err)
+			return nil
+		}
 
-        for _, file := range files {
-            if len(file) > 0 && file[len(file) - 1] == '/' {
-                directories = append(directories, dir + file)
-            } else {
-                chunk := dir + file
-                chunk = chunk[len("chunks/"):]
-                chunks = append(chunks, chunk)
-            }
-        }
-    }
+		for _, file := range files {
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
+			} else {
+				chunk := dir + file
+				chunk = chunk[len("chunks/"):]
+				chunks = append(chunks, chunk)
+			}
+		}
+	}
 
-    return
+	return
 }
 
 func moveChunk(t *testing.T, storage Storage, chunkID string, isFossil bool, delay int) {
 
-    filePath, exist, _, err := storage.FindChunk(0, chunkID, isFossil)
+	filePath, exist, _, err := storage.FindChunk(0, chunkID, isFossil)
 
-    if err != nil {
-        t.Errorf("Error find chunk %s: %v", chunkID, err)
-        return
-    }
+	if err != nil {
+		t.Errorf("Error find chunk %s: %v", chunkID, err)
+		return
+	}
 
-    to := filePath + ".fsl"
-    if isFossil {
-        to = filePath[:len(filePath) - len(".fsl")]
-    }
+	to := filePath + ".fsl"
+	if isFossil {
+		to = filePath[:len(filePath)-len(".fsl")]
+	}
 
-    err = storage.MoveFile(0, filePath, to)
-    if err != nil {
-        t.Errorf("Error renaming file %s to %s: %v", filePath, to, err)
-    }
+	err = storage.MoveFile(0, filePath, to)
+	if err != nil {
+		t.Errorf("Error renaming file %s to %s: %v", filePath, to, err)
+	}
 
-    time.Sleep(time.Duration(delay) * time.Second)
+	time.Sleep(time.Duration(delay) * time.Second)
 
-    _, exist, _, err = storage.FindChunk(0, chunkID, isFossil)
-    if err != nil {
-        t.Errorf("Error get file info for chunk %s: %v", chunkID, err)
-    }
+	_, exist, _, err = storage.FindChunk(0, chunkID, isFossil)
+	if err != nil {
+		t.Errorf("Error get file info for chunk %s: %v", chunkID, err)
+	}
 
-    if exist {
-        t.Errorf("File %s still exists after renaming", filePath)
-    }
+	if exist {
+		t.Errorf("File %s still exists after renaming", filePath)
+	}
 
-    _, exist, _, err = storage.FindChunk(0, chunkID, !isFossil)
-    if err != nil {
-        t.Errorf("Error get file info for %s: %v", to, err)
-    }
+	_, exist, _, err = storage.FindChunk(0, chunkID, !isFossil)
+	if err != nil {
+		t.Errorf("Error get file info for %s: %v", to, err)
+	}
 
-    if !exist {
-        t.Errorf("File %s doesn't exist", to)
-    }
+	if !exist {
+		t.Errorf("File %s doesn't exist", to)
+	}
 
 }
 
 func TestStorage(t *testing.T) {
 
-    rand.Seed(time.Now().UnixNano())
-    setTestingT(t)
-    SetLoggingLevel(INFO)
+	rand.Seed(time.Now().UnixNano())
+	setTestingT(t)
+	SetLoggingLevel(INFO)
 
-    defer func() {
-        if r := recover(); r != nil {
-            switch e := r.(type) {
-            case Exception:
-                t.Errorf("%s %s", e.LogID, e.Message)
-                debug.PrintStack()
-            default:
-                t.Errorf("%v", e)
-                debug.PrintStack()
-            }
-        }
-    } ()
+	defer func() {
+		if r := recover(); r != nil {
+			switch e := r.(type) {
+			case Exception:
+				t.Errorf("%s %s", e.LogID, e.Message)
+				debug.PrintStack()
+			default:
+				t.Errorf("%v", e)
+				debug.PrintStack()
+			}
+		}
+	}()
 
-    testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
-    os.RemoveAll(testDir)
-    os.MkdirAll(testDir, 0700)
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)
 
-    LOG_INFO("STORAGE_TEST", "storage: %s", testStorageName)
+	LOG_INFO("STORAGE_TEST", "storage: %s", testStorageName)
 
-    storage, err := loadStorage(testDir, 1)
-    if err != nil {
-        t.Errorf("Failed to create storage: %v", err)
-        return
-    }
-    storage.EnableTestMode()
-    storage.SetRateLimits(testRateLimit, testRateLimit)
+	threads := 8
+	storage, err := loadStorage(testDir, threads)
+	if err != nil {
+		t.Errorf("Failed to create storage: %v", err)
+		return
+	}
+	storage.EnableTestMode()
+	storage.SetRateLimits(testRateLimit, testRateLimit)
 
-    delay := 0
-    if _, ok := storage.(*ACDStorage); ok {
-        delay = 5
-    }
-    if _, ok := storage.(*HubicStorage); ok {
-        delay = 2
-    }
+	delay := 0
+	if _, ok := storage.(*ACDStorage); ok {
+		delay = 5
+	}
+	if _, ok := storage.(*HubicStorage); ok {
+		delay = 2
+	}
 
-    for _, dir := range []string { "chunks", "snapshots" }  {
-        err = storage.CreateDirectory(0, dir)
-        if err != nil {
-            t.Errorf("Failed to create directory %s: %v", dir, err)
-            return
-        }
-    }
+	for _, dir := range []string{"chunks", "snapshots"} {
+		err = storage.CreateDirectory(0, dir)
+		if err != nil {
+			t.Errorf("Failed to create directory %s: %v", dir, err)
+			return
+		}
+	}
 
-    storage.CreateDirectory(0, "snapshots/repository1")
-    storage.CreateDirectory(0, "snapshots/repository2")
-    time.Sleep(time.Duration(delay) * time.Second)
-    {
+	storage.CreateDirectory(0, "snapshots/repository1")
+	storage.CreateDirectory(0, "snapshots/repository2")
 
-        // Upload fake snapshot files so that for storages having no concept of directories,
-        // ListFiles("snapshots") still returns correct snapshot IDs.
+	storage.CreateDirectory(0, "shared")
 
-        // Create a random file not a text file to make ACD Storage happy.
-        content := make([]byte, 100)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+	// Upload to the same directory by multiple goroutines
+	count := threads
+	finished := make(chan int, count)
+	for i := 0; i < count; i++ {
+		go func(threadIndex int, name string) {
+			err := storage.UploadFile(threadIndex, name, []byte("this is a test file"))
+			if err != nil {
+				t.Errorf("Error to upload '%s': %v", name, err)
+			}
+			finished <- 0
+		}(i, fmt.Sprintf("shared/a/b/c/%d", i))
+	}
 
-        err = storage.UploadFile(0, "snapshots/repository1/1", content)
-        if err != nil {
-            t.Errorf("Error to upload snapshots/repository1/1: %v", err)
-        }
+	for i := 0; i < count; i++ {
+		<-finished
+	}
 
-        err = storage.UploadFile(0, "snapshots/repository2/1", content)
-        if err != nil {
-            t.Errorf("Error to upload snapshots/repository2/1: %v", err)
-        }
-    }
+	for i := 0; i < count; i++ {
+		storage.DeleteFile(0, fmt.Sprintf("shared/a/b/c/%d", i))
+	}
+	storage.DeleteFile(0, "shared/a/b/c")
+	storage.DeleteFile(0, "shared/a/b")
+	storage.DeleteFile(0, "shared/a")
 
-    time.Sleep(time.Duration(delay) * time.Second)
+	time.Sleep(time.Duration(delay) * time.Second)
+	{
 
-    snapshotDirs, _, err := storage.ListFiles(0, "snapshots/")
-    if err != nil {
-        t.Errorf("Failed to list snapshot ids: %v", err)
-        return
-    }
+		// Upload fake snapshot files so that for storages having no concept of directories,
+		// ListFiles("snapshots") still returns correct snapshot IDs.
 
-    snapshotIDs := []string {}
-    for _, snapshotDir := range snapshotDirs {
-        if len(snapshotDir) > 0 && snapshotDir[len(snapshotDir) - 1] == '/' {
-            snapshotIDs = append(snapshotIDs, snapshotDir[:len(snapshotDir) - 1])
-        }
-    }
+		// Create a random file not a text file to make ACD Storage happy.
+		content := make([]byte, 100)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-    if len(snapshotIDs) < 2 {
-        t.Errorf("Snapshot directories not created")
-        return
-    }
+		err = storage.UploadFile(0, "snapshots/repository1/1", content)
+		if err != nil {
+			t.Errorf("Error to upload snapshots/repository1/1: %v", err)
+		}
 
-    for _, snapshotID := range snapshotIDs {
-        snapshots, _, err := storage.ListFiles(0, "snapshots/" + snapshotID)
-        if err != nil {
-            t.Errorf("Failed to list snapshots for %s: %v", snapshotID, err)
-            return
-        }
-        for _, snapshot := range snapshots {
-            storage.DeleteFile(0, "snapshots/" + snapshotID + "/" + snapshot)
-        }
-    }
+		err = storage.UploadFile(0, "snapshots/repository2/1", content)
+		if err != nil {
+			t.Errorf("Error to upload snapshots/repository2/1: %v", err)
+		}
+	}
 
-    time.Sleep(time.Duration(delay) * time.Second)
+	time.Sleep(time.Duration(delay) * time.Second)
 
-    storage.DeleteFile(0, "config")
+	snapshotDirs, _, err := storage.ListFiles(0, "snapshots/")
+	if err != nil {
+		t.Errorf("Failed to list snapshot ids: %v", err)
+		return
+	}
 
-    for _, file := range []string { "snapshots/repository1/1", "snapshots/repository2/1"} {
-        exist, _, _, err := storage.GetFileInfo(0, file)
-        if err != nil {
-            t.Errorf("Failed to get file info for %s: %v", file, err)
-            return
-        }
-        if exist {
-            t.Errorf("File %s still exists after deletion", file)
-            return
-        }
-    }
+	snapshotIDs := []string{}
+	for _, snapshotDir := range snapshotDirs {
+		if len(snapshotDir) > 0 && snapshotDir[len(snapshotDir)-1] == '/' {
+			snapshotIDs = append(snapshotIDs, snapshotDir[:len(snapshotDir)-1])
+		}
+	}
 
-    numberOfFiles := 20
-    maxFileSize := 64 * 1024
+	if len(snapshotIDs) < 2 {
+		t.Errorf("Snapshot directories not created")
+		return
+	}
 
-    if testQuickMode {
-        numberOfFiles = 2
-    }
+	for _, snapshotID := range snapshotIDs {
+		snapshots, _, err := storage.ListFiles(0, "snapshots/"+snapshotID)
+		if err != nil {
+			t.Errorf("Failed to list snapshots for %s: %v", snapshotID, err)
+			return
+		}
+		for _, snapshot := range snapshots {
+			storage.DeleteFile(0, "snapshots/"+snapshotID+"/"+snapshot)
+		}
+	}
 
-    chunks := []string{}
+	time.Sleep(time.Duration(delay) * time.Second)
 
-    for i := 0; i < numberOfFiles; i++ {
-        content := make([]byte, rand.Int() % maxFileSize + 1)
-        _, err = crypto_rand.Read(content)
-        if err != nil {
-            t.Errorf("Error generating random content: %v", err)
-            return
-        }
+	storage.DeleteFile(0, "config")
 
-        hasher := sha256.New()
-        hasher.Write(content)
-        chunkID := hex.EncodeToString(hasher.Sum(nil))
-        chunks = append(chunks, chunkID)
+	for _, file := range []string{"snapshots/repository1/1", "snapshots/repository2/1"} {
+		exist, _, _, err := storage.GetFileInfo(0, file)
+		if err != nil {
+			t.Errorf("Failed to get file info for %s: %v", file, err)
+			return
+		}
+		if exist {
+			t.Errorf("File %s still exists after deletion", file)
+			return
+		}
+	}
 
-        filePath, exist, _, err := storage.FindChunk(0, chunkID, false)
-        if err != nil {
-            t.Errorf("Failed to list the chunk %s: %v", chunkID, err)
-            return
-        }
-        if exist {
-            t.Errorf("Chunk %s already exists", chunkID)
-        }
+	numberOfFiles := 10
+	maxFileSize := 64 * 1024
 
-        err = storage.UploadFile(0, filePath, content)
-        if err != nil {
-            t.Errorf("Failed to upload the file %s: %v", filePath, err)
-            return
-        }
-        LOG_INFO("STORAGE_CHUNK", "Uploaded chunk: %s, size: %d", chunkID, len(content))
-    }
+	if testQuickMode {
+		numberOfFiles = 2
+	}
 
-    allChunks := [] string {}
-    for _, file := range listChunks(storage) {
-        file = strings.Replace(file, "/", "", -1)
-        if len(file) == 64 {
-            allChunks = append(allChunks, file)
-        }
-    }
+	chunks := []string{}
 
-    LOG_INFO("STORAGE_FOSSIL", "Making %s a fossil", chunks[0])
-    moveChunk(t, storage, chunks[0], false, delay)
-    LOG_INFO("STORAGE_FOSSIL", "Making %s a chunk", chunks[0])
-    moveChunk(t, storage, chunks[0], true, delay)
+	for i := 0; i < numberOfFiles; i++ {
+		content := make([]byte, rand.Int()%maxFileSize+1)
+		_, err = crypto_rand.Read(content)
+		if err != nil {
+			t.Errorf("Error generating random content: %v", err)
+			return
+		}
 
-    config := CreateConfig()
-    config.MinimumChunkSize = 100
-    config.chunkPool = make(chan *Chunk, numberOfFiles * 2)
+		hasher := sha256.New()
+		hasher.Write(content)
+		chunkID := hex.EncodeToString(hasher.Sum(nil))
+		chunks = append(chunks, chunkID)
 
-    chunk := CreateChunk(config, true)
+		filePath, exist, _, err := storage.FindChunk(0, chunkID, false)
+		if err != nil {
+			t.Errorf("Failed to list the chunk %s: %v", chunkID, err)
+			return
+		}
+		if exist {
+			t.Errorf("Chunk %s already exists", chunkID)
+		}
 
+		err = storage.UploadFile(0, filePath, content)
+		if err != nil {
+			t.Errorf("Failed to upload the file %s: %v", filePath, err)
+			return
+		}
+		LOG_INFO("STORAGE_CHUNK", "Uploaded chunk: %s, size: %d", filePath, len(content))
+	}
 
-    for _, chunkID := range chunks {
+	LOG_INFO("STORAGE_FOSSIL", "Making %s a fossil", chunks[0])
+	moveChunk(t, storage, chunks[0], false, delay)
+	LOG_INFO("STORAGE_FOSSIL", "Making %s a chunk", chunks[0])
+	moveChunk(t, storage, chunks[0], true, delay)
 
-        chunk.Reset(false)
-        filePath, exist, _, err := storage.FindChunk(0, chunkID, false)
-        if err != nil {
-            t.Errorf("Error getting file info for chunk %s: %v", chunkID, err)
-            continue
-        } else if !exist {
-            t.Errorf("Chunk %s does not exist", chunkID)
-            continue
-        } else {
-            err = storage.DownloadFile(0, filePath, chunk)
-            if err != nil {
-                t.Errorf("Error downloading file %s: %v", filePath, err)
-                continue
-            }
-            LOG_INFO("STORAGE_CHUNK", "Downloaded chunk: %s, size: %d", chunkID, chunk.GetLength())
-        }
+	config := CreateConfig()
+	config.MinimumChunkSize = 100
+	config.chunkPool = make(chan *Chunk, numberOfFiles*2)
 
-        hasher := sha256.New()
-        hasher.Write(chunk.GetBytes())
-        hash := hex.EncodeToString(hasher.Sum(nil))
+	chunk := CreateChunk(config, true)
 
-        if hash != chunkID {
-            t.Errorf("File %s, hash %s, size %d", chunkID, hash, chunk.GetBytes())
-        }
-    }
+	for _, chunkID := range chunks {
 
-    LOG_INFO("STORAGE_FOSSIL", "Making %s a fossil", chunks[1])
-    moveChunk(t, storage, chunks[1], false, delay)
+		chunk.Reset(false)
+		filePath, exist, _, err := storage.FindChunk(0, chunkID, false)
+		if err != nil {
+			t.Errorf("Error getting file info for chunk %s: %v", chunkID, err)
+			continue
+		} else if !exist {
+			t.Errorf("Chunk %s does not exist", chunkID)
+			continue
+		} else {
+			err = storage.DownloadFile(0, filePath, chunk)
+			if err != nil {
+				t.Errorf("Error downloading file %s: %v", filePath, err)
+				continue
+			}
+			LOG_INFO("STORAGE_CHUNK", "Downloaded chunk: %s, size: %d", filePath, chunk.GetLength())
+		}
 
-    filePath, exist, _, err := storage.FindChunk(0, chunks[1], true)
-    if err != nil {
-        t.Errorf("Error getting file info for fossil %s: %v", chunks[1], err)
-    } else if !exist {
-        t.Errorf("Fossil %s does not exist", chunks[1])
-    } else {
-        err = storage.DeleteFile(0, filePath)
-        if err != nil {
-            t.Errorf("Failed to delete file %s: %v", filePath)
-        } else {
-            time.Sleep(time.Duration(delay) * time.Second)
-            filePath, exist, _, err = storage.FindChunk(0, chunks[1], true)
-            if err != nil {
-                t.Errorf("Error get file info for deleted fossil %s: %v", chunks[1], err)
-            } else if exist {
-                t.Errorf("Fossil %s still exists after deletion", chunks[1])
-            }
-        }
-    }
+		hasher := sha256.New()
+		hasher.Write(chunk.GetBytes())
+		hash := hex.EncodeToString(hasher.Sum(nil))
 
-    for _, file := range allChunks {
+		if hash != chunkID {
+			t.Errorf("File %s, hash %s, size %d", chunkID, hash, chunk.GetBytes())
+		}
+	}
 
-        err = storage.DeleteFile(0, "chunks/" + file)
-        if err != nil {
-            t.Errorf("Failed to delete the file %s: %v", file, err)
-            return
-        }
-    }
+	LOG_INFO("STORAGE_FOSSIL", "Making %s a fossil", chunks[1])
+	moveChunk(t, storage, chunks[1], false, delay)
+
+	filePath, exist, _, err := storage.FindChunk(0, chunks[1], true)
+	if err != nil {
+		t.Errorf("Error getting file info for fossil %s: %v", chunks[1], err)
+	} else if !exist {
+		t.Errorf("Fossil %s does not exist", chunks[1])
+	} else {
+		err = storage.DeleteFile(0, filePath)
+		if err != nil {
+			t.Errorf("Failed to delete file %s: %v", filePath, err)
+		} else {
+			time.Sleep(time.Duration(delay) * time.Second)
+			filePath, exist, _, err = storage.FindChunk(0, chunks[1], true)
+			if err != nil {
+				t.Errorf("Error get file info for deleted fossil %s: %v", chunks[1], err)
+			} else if exist {
+				t.Errorf("Fossil %s still exists after deletion", chunks[1])
+			}
+		}
+	}
+
+	allChunks := []string{}
+	for _, file := range listChunks(storage) {
+		allChunks = append(allChunks, file)
+	}
+
+	for _, file := range allChunks {
+
+		err = storage.DeleteFile(0, "chunks/"+file)
+		if err != nil {
+			t.Errorf("Failed to delete the file %s: %v", file, err)
+			return
+		}
+	}
+
+}
+
+func TestCleanStorage(t *testing.T) {
+	setTestingT(t)
+	SetLoggingLevel(INFO)
+
+	defer func() {
+		if r := recover(); r != nil {
+			switch e := r.(type) {
+			case Exception:
+				t.Errorf("%s %s", e.LogID, e.Message)
+				debug.PrintStack()
+			default:
+				t.Errorf("%v", e)
+				debug.PrintStack()
+			}
+		}
+	}()
+
+	testDir := path.Join(os.TempDir(), "duplicacy_test", "storage_test")
+	os.RemoveAll(testDir)
+	os.MkdirAll(testDir, 0700)
+
+	LOG_INFO("STORAGE_TEST", "storage: %s", testStorageName)
+
+	storage, err := loadStorage(testDir, 1)
+	if err != nil {
+		t.Errorf("Failed to create storage: %v", err)
+		return
+	}
+
+	directories := make([]string, 0, 1024)
+	directories = append(directories, "snapshots/")
+	directories = append(directories, "chunks/")
+
+	for len(directories) > 0 {
+
+		dir := directories[len(directories)-1]
+		directories = directories[:len(directories)-1]
+
+		LOG_INFO("LIST_FILES", "Listing %s", dir)
+
+		files, _, err := storage.ListFiles(0, dir)
+		if err != nil {
+			LOG_ERROR("LIST_FILES", "Failed to list the directory %s: %v", dir, err)
+			return
+		}
+
+		for _, file := range files {
+			if len(file) > 0 && file[len(file)-1] == '/' {
+				directories = append(directories, dir+file)
+			} else {
+				storage.DeleteFile(0, dir+file)
+				LOG_INFO("DELETE_FILE", "Deleted file %s", file)
+			}
+		}
+	}
+
+	storage.DeleteFile(0, "config")
+	LOG_INFO("DELETE_FILE", "Deleted config")
+
+	files, _, err := storage.ListFiles(0, "chunks/")
+	for _, file := range files {
+		if len(file) > 0 && file[len(file)-1] != '/' {
+			LOG_DEBUG("FILE_EXIST", "File %s exists after deletion", file)
+		}
+	}
 
 }

src/duplicacy_swiftstorage.go

@@ -0,0 +1,251 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+
+package duplicacy
+
+import (
+	"strconv"
+	"strings"
+	"time"
+
+	"github.com/ncw/swift"
+)
+
+type SwiftStorage struct {
+	StorageBase
+
+	connection *swift.Connection
+	container  string
+	storageDir string
+	threads    int
+}
+
+// CreateSwiftStorage creates an OpenStack Swift storage object.  storageURL is in the form of
+// `user@authURL/container/path?arg1=value1&arg2=value2``
+func CreateSwiftStorage(storageURL string, key string, threads int) (storage *SwiftStorage, err error) {
+
+	// This is the map to store all arguments
+	arguments := make(map[string]string)
+
+	// Check if there are arguments provided as a query string
+	if strings.Contains(storageURL, "?") {
+		urlAndArguments := strings.SplitN(storageURL, "?", 2)
+		storageURL = urlAndArguments[0]
+		for _, pair := range strings.Split(urlAndArguments[1], "&") {
+			if strings.Contains(pair, "=") {
+				keyAndValue := strings.Split(pair, "=")
+				arguments[keyAndValue[0]] = keyAndValue[1]
+			}
+		}
+	}
+
+	// Take out the user name if there is one
+	if strings.Contains(storageURL, "@") {
+		userAndURL := strings.Split(storageURL, "@")
+		arguments["user"] = userAndURL[0]
+		storageURL = userAndURL[1]
+	}
+
+	// The version is used to split authURL and container/path
+	versions := []string{"/v1/", "/v1.0/", "/v2/", "/v2.0/", "/v3/", "/v3.0/", "/v4/", "/v4.0/"}
+	storageDir := ""
+	for _, version := range versions {
+		if strings.Contains(storageURL, version) {
+			urlAndStorageDir := strings.SplitN(storageURL, version, 2)
+			storageURL = urlAndStorageDir[0] + version[0:len(version)-1]
+			storageDir = urlAndStorageDir[1]
+		}
+	}
+
+	// If no container/path is specified, find them from the arguments
+	if storageDir == "" {
+		storageDir = arguments["storage_dir"]
+	}
+
+	// Now separate the container name from the storage path
+	container := ""
+	if strings.Contains(storageDir, "/") {
+		containerAndStorageDir := strings.SplitN(storageDir, "/", 2)
+		container = containerAndStorageDir[0]
+		storageDir = containerAndStorageDir[1]
+		if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+			storageDir += "/"
+		}
+	} else {
+		container = storageDir
+		storageDir = ""
+	}
+
+	// Number of retries on err
+	retries := 4
+	if value, ok := arguments["retries"]; ok {
+		retries, _ = strconv.Atoi(value)
+	}
+
+	// Connect channel timeout
+	connectionTimeout := 10
+	if value, ok := arguments["connection_timeout"]; ok {
+		connectionTimeout, _ = strconv.Atoi(value)
+	}
+
+	// Data channel timeout
+	timeout := 60
+	if value, ok := arguments["timeout"]; ok {
+		timeout, _ = strconv.Atoi(value)
+	}
+
+	// Auth version; default to auto-detect
+	authVersion := 0
+	if value, ok := arguments["auth_version"]; ok {
+		authVersion, _ = strconv.Atoi(value)
+	}
+
+	// Allow http to be used by setting "protocol=http" in arguments
+	if _, ok := arguments["protocol"]; !ok {
+		arguments["protocol"] = "https"
+	}
+
+	// Please refer to https://godoc.org/github.com/ncw/swift#Connection
+	connection := swift.Connection{
+		Domain:         arguments["domain"],
+		DomainId:       arguments["domain_id"],
+		UserName:       arguments["user"],
+		UserId:         arguments["user_id"],
+		ApiKey:         key,
+		AuthUrl:        arguments["protocol"] + "://" + storageURL,
+		Retries:        retries,
+		UserAgent:      arguments["user_agent"],
+		ConnectTimeout: time.Duration(connectionTimeout) * time.Second,
+		Timeout:        time.Duration(timeout) * time.Second,
+		Region:         arguments["region"],
+		AuthVersion:    authVersion,
+		Internal:       false,
+		Tenant:         arguments["tenant"],
+		TenantId:       arguments["tenant_id"],
+		EndpointType:   swift.EndpointType(arguments["endpiont_type"]),
+		TenantDomain:   arguments["tenant_domain"],
+		TenantDomainId: arguments["tenant_domain_id"],
+		TrustId:        arguments["trust_id"],
+	}
+
+	_, _, err = connection.Container(container)
+	if err != nil {
+		return nil, err
+	}
+
+	storage = &SwiftStorage{
+		connection: &connection,
+		container:  container,
+		storageDir: storageDir,
+		threads:    threads,
+	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{1}, 1)
+	return storage, nil
+}
+
+// ListFiles return the list of files and subdirectories under 'dir' (non-recursively)
+func (storage *SwiftStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
+	if len(dir) > 0 && dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
+	isSnapshotDir := dir == "snapshots/"
+	dir = storage.storageDir + dir
+
+	options := swift.ObjectsOpts{
+		Prefix: dir,
+		Limit:  1000,
+	}
+
+	if isSnapshotDir {
+		options.Delimiter = '/'
+	}
+
+	objects, err := storage.connection.ObjectsAll(storage.container, &options)
+	if err != nil {
+		return nil, nil, err
+	}
+
+	for _, obj := range objects {
+		if isSnapshotDir {
+			if obj.SubDir != "" {
+				files = append(files, obj.SubDir[len(dir):])
+				sizes = append(sizes, 0)
+			}
+		} else {
+			files = append(files, obj.Name[len(dir):])
+			sizes = append(sizes, obj.Bytes)
+		}
+	}
+
+	return files, sizes, nil
+}
+
+// DeleteFile deletes the file or directory at 'filePath'.
+func (storage *SwiftStorage) DeleteFile(threadIndex int, filePath string) (err error) {
+	return storage.connection.ObjectDelete(storage.container, storage.storageDir+filePath)
+}
+
+// MoveFile renames the file.
+func (storage *SwiftStorage) MoveFile(threadIndex int, from string, to string) (err error) {
+	return storage.connection.ObjectMove(storage.container, storage.storageDir+from,
+		storage.container, storage.storageDir+to)
+}
+
+// CreateDirectory creates a new directory.
+func (storage *SwiftStorage) CreateDirectory(threadIndex int, dir string) (err error) {
+	// Does nothing as directories do not exist in OpenStack Swift
+	return nil
+}
+
+// GetFileInfo returns the information about the file or directory at 'filePath'.
+func (storage *SwiftStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
+	object, _, err := storage.connection.Object(storage.container, storage.storageDir+filePath)
+
+	if err != nil {
+		if err == swift.ObjectNotFound {
+			return false, false, 0, nil
+		} else {
+			return false, false, 0, err
+		}
+	}
+
+	return true, false, object.Bytes, nil
+}
+
+// DownloadFile reads the file at 'filePath' into the chunk.
+func (storage *SwiftStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
+
+	file, _, err := storage.connection.ObjectOpen(storage.container, storage.storageDir+filePath, false, nil)
+	if err != nil {
+		return err
+	}
+	_, err = RateLimitedCopy(chunk, file, storage.DownloadRateLimit/storage.threads)
+	return err
+}
+
+// UploadFile writes 'content' to the file at 'filePath'.
+func (storage *SwiftStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
+	reader := CreateRateLimitedReader(content, storage.UploadRateLimit/storage.threads)
+	_, err = storage.connection.ObjectPut(storage.container, storage.storageDir+filePath, reader, true, "", "application/duplicacy", nil)
+	return err
+}
+
+// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
+// managing snapshots.
+func (storage *SwiftStorage) IsCacheNeeded() bool { return true }
+
+// If the 'MoveFile' method is implemented.
+func (storage *SwiftStorage) IsMoveFileImplemented() bool { return true }
+
+// If the storage can guarantee strong consistency.
+func (storage *SwiftStorage) IsStrongConsistent() bool { return false }
+
+// If the storage supports fast listing of files names.
+func (storage *SwiftStorage) IsFastListing() bool { return true }
+
+// Enable the test mode.
+func (storage *SwiftStorage) EnableTestMode() {
+}

src/duplicacy_utils.go

@@ -5,300 +5,333 @@
 package duplicacy
 
 import (
-    "fmt"
-    "os"
-    "bufio"
-    "io"
-    "io/ioutil"
-    "time"
-    "path"
-    "path/filepath"
-    "regexp"
-    "strings"
-    "strconv"
-    "runtime"
-    "crypto/sha256"
+	"bufio"
+	"crypto/sha256"
+	"fmt"
+	"io"
+	"os"
+	"regexp"
+	"strconv"
+	"strings"
+	"time"
 
-    "golang.org/x/crypto/pbkdf2"
-    "github.com/gilbertchen/gopass"
+	"github.com/gilbertchen/gopass"
+	"golang.org/x/crypto/pbkdf2"
 )
 
 var RunInBackground bool = false
 
 type RateLimitedReader struct {
-    Content []byte
-    Rate float64
-    Next int
-    StartTime time.Time
+	Content   []byte
+	Rate      float64
+	Next      int
+	StartTime time.Time
 }
 
-func CreateRateLimitedReader(content []byte, rate int) (*RateLimitedReader) {
-    return &RateLimitedReader {
-        Content: content,
-        Rate: float64(rate * 1024),
-        Next: 0,
-    }
+var RegexMap map[string]*regexp.Regexp
+
+func init() {
+
+	if RegexMap == nil {
+		RegexMap = make(map[string]*regexp.Regexp)
+	}
+
 }
 
-func (reader *RateLimitedReader) Length() (int64) {
-    return int64(len(reader.Content))
+func CreateRateLimitedReader(content []byte, rate int) *RateLimitedReader {
+	return &RateLimitedReader{
+		Content: content,
+		Rate:    float64(rate * 1024),
+		Next:    0,
+	}
+}
+
+func IsEmptyFilter(pattern string) bool {
+	if pattern == "+" || pattern == "-" || pattern == "i:" || pattern == "e:" {
+		return true
+	} else {
+		return false
+	}
+}
+
+func IsUnspecifiedFilter(pattern string) bool {
+	if pattern[0] != '+' && pattern[0] != '-' && !strings.HasPrefix(pattern, "i:") && !strings.HasPrefix(pattern, "e:")  {
+		return true
+	} else {
+		return false
+	}
+}
+
+func IsValidRegex(pattern string) (valid bool, err error) {
+
+	var re *regexp.Regexp = nil
+
+	if re, valid = RegexMap[pattern]; valid && re != nil {
+		return true, nil
+	}
+
+	re, err = regexp.Compile(pattern)
+
+	if err != nil {
+		return false, err
+	} else {
+		RegexMap[pattern] = re
+		LOG_DEBUG("REGEX_STORED", "Saved compiled regex for pattern \"%s\", regex=%#v", pattern, re)
+		return true, err
+	}
+}
+
+func (reader *RateLimitedReader) Length() int64 {
+	return int64(len(reader.Content))
 }
 
 func (reader *RateLimitedReader) Reset() {
-    reader.Next = 0
+	reader.Next = 0
 }
 
-func (reader *RateLimitedReader)  Seek(offset int64, whence int) (int64, error) {
-    if whence == io.SeekStart {
-        reader.Next = int(offset)
-    } else if whence == io.SeekCurrent {
-        reader.Next += int(offset)
-    } else {
-        reader.Next = len(reader.Content) - int(offset)
-    }
-    return int64(reader.Next), nil
+func (reader *RateLimitedReader) Seek(offset int64, whence int) (int64, error) {
+	if whence == io.SeekStart {
+		reader.Next = int(offset)
+	} else if whence == io.SeekCurrent {
+		reader.Next += int(offset)
+	} else {
+		reader.Next = len(reader.Content) - int(offset)
+	}
+	return int64(reader.Next), nil
 }
 
 func (reader *RateLimitedReader) Read(p []byte) (n int, err error) {
 
-    if reader.Next >= len(reader.Content) {
-        return 0, io.EOF
-    }
+	if reader.Next >= len(reader.Content) {
+		return 0, io.EOF
+	}
 
-    if reader.Rate <= 0 {
-        n := copy(p, reader.Content[reader.Next:])
-        reader.Next += n
-        if reader.Next >= len(reader.Content) {
-            return n, io.EOF
-        }
-        return n, nil
-    }
+	if reader.Rate <= 0 {
+		n := copy(p, reader.Content[reader.Next:])
+		reader.Next += n
+		if reader.Next >= len(reader.Content) {
+			return n, io.EOF
+		}
+		return n, nil
+	}
 
-    if reader.StartTime.IsZero() {
-        reader.StartTime = time.Now()
-    }
+	if reader.StartTime.IsZero() {
+		reader.StartTime = time.Now()
+	}
 
-    elapsed := time.Since(reader.StartTime).Seconds()
-    delay := float64(reader.Next) / reader.Rate - elapsed
-    end := reader.Next + int(reader.Rate / 5)
-    if delay > 0 {
-        time.Sleep(time.Duration(delay *  float64(time.Second)))
-    } else {
-        end += - int(delay * reader.Rate)
-    }
+	elapsed := time.Since(reader.StartTime).Seconds()
+	delay := float64(reader.Next)/reader.Rate - elapsed
+	end := reader.Next + int(reader.Rate/5)
+	if delay > 0 {
+		time.Sleep(time.Duration(delay * float64(time.Second)))
+	} else {
+		end += -int(delay * reader.Rate)
+	}
 
-    if end > len(reader.Content) {
-        end = len(reader.Content)
-    }
+	if end > len(reader.Content) {
+		end = len(reader.Content)
+	}
 
-    n = copy(p, reader.Content[reader.Next : end])
-    reader.Next += n
-    return n, nil
+	n = copy(p, reader.Content[reader.Next:end])
+	reader.Next += n
+	return n, nil
 }
 
 func RateLimitedCopy(writer io.Writer, reader io.Reader, rate int) (written int64, err error) {
-    if rate <= 0 {
-        return io.Copy(writer, reader)
-    }
-    for range time.Tick(time.Second / 5) {
-        n, err := io.CopyN(writer, reader, int64(rate * 1024 / 5))
-        written += n
-        if err != nil {
-            if err == io.EOF {
-                return written, nil
-            } else {
-                return written, err
-            }
-        }
-    }
-    return written, nil
+	if rate <= 0 {
+		return io.Copy(writer, reader)
+	}
+	for range time.Tick(time.Second / 5) {
+		n, err := io.CopyN(writer, reader, int64(rate*1024/5))
+		written += n
+		if err != nil {
+			if err == io.EOF {
+				return written, nil
+			} else {
+				return written, err
+			}
+		}
+	}
+	return written, nil
 }
 
 // GenerateKeyFromPassword generates a key from the password.
-func GenerateKeyFromPassword(password string) []byte {
-    return pbkdf2.Key([]byte(password), DEFAULT_KEY, 16384, 32, sha256.New)
+func GenerateKeyFromPassword(password string, salt []byte, iterations int) []byte {
+	return pbkdf2.Key([]byte(password), salt, iterations, 32, sha256.New)
+}
+
+// Get password from preference, env, but don't start any keyring request
+func GetPasswordFromPreference(preference Preference, passwordType string) string {
+	passwordID := passwordType
+	if preference.Name != "default" {
+		passwordID = preference.Name + "_" + passwordID
+	}
+
+	{
+		name := strings.ToUpper("duplicacy_" + passwordID)
+		LOG_DEBUG("PASSWORD_ENV_VAR", "Reading the environment variable %s", name)
+		if password, found := os.LookupEnv(name); found && password != "" {
+			return password
+		}
+
+		re := regexp.MustCompile(`[^a-zA-Z0-9_]`)
+		namePlain := re.ReplaceAllString(name, "_")
+		if namePlain != name {
+			LOG_DEBUG("PASSWORD_ENV_VAR", "Reading the environment variable %s", namePlain)
+			if password, found := os.LookupEnv(namePlain); found && password != "" {
+				return password
+			}
+		}
+	}
+
+	// If the password is stored in the preference, there is no need to include the storage name
+	// (i.e., preference.Name) in the key, so the key name should really be passwordType rather
+	// than passwordID; we're using passwordID here only for backward compatibility
+	if len(preference.Keys) > 0 && len(preference.Keys[passwordID]) > 0 {
+		LOG_DEBUG("PASSWORD_PREFERENCE", "Reading %s from preferences", passwordID)
+		return preference.Keys[passwordID]
+	}
+
+	if len(preference.Keys) > 0 && len(preference.Keys[passwordType]) > 0 {
+		LOG_DEBUG("PASSWORD_PREFERENCE", "Reading %s from preferences", passwordType)
+		return preference.Keys[passwordType]
+	}
+
+	return ""
 }
 
 // GetPassword attempts to get the password from KeyChain/KeyRing, environment variables, or keyboard input.
 func GetPassword(preference Preference, passwordType string, prompt string,
-                 showPassword bool, resetPassword bool) (string) {
+	showPassword bool, resetPassword bool) string {
+	passwordID := passwordType
 
-    passwordID := passwordType
-    if preference.Name != "default" {
-        passwordID = preference.Name + "_" + passwordID
-    }
+	preferencePassword := GetPasswordFromPreference(preference, passwordType)
+	if preferencePassword != "" {
+		return preferencePassword
+	}
 
-    {
-        name := strings.ToUpper("duplicacy_" + passwordID)
-        LOG_DEBUG("PASSWORD_ENV_VAR", "Reading the environment variable %s", name)
-        if password, found := os.LookupEnv(name); found && password != "" {
-            return password
-        }
-    }
+	if preference.Name != "default" {
+		passwordID = preference.Name + "_" + passwordID
+	}
 
-    if len(preference.Keys) > 0 && len(preference.Keys[passwordID]) > 0 {
-        LOG_DEBUG("PASSWORD_KEYCHAIN", "Reading %s from preferences", passwordID)
-        return preference.Keys[passwordID]
-    }
+	if resetPassword && !RunInBackground {
+		keyringSet(passwordID, "")
+	} else {
+		password := keyringGet(passwordID)
+		if password != "" {
+			LOG_DEBUG("PASSWORD_KEYCHAIN", "Reading %s from keychain/keyring", passwordType)
+			return password
+		}
 
-    if resetPassword && !RunInBackground {
-        keyringSet(passwordID, "")
-    } else {
-        password := keyringGet(passwordID)
-        if password != "" {
-            return password
-        }
+		if RunInBackground {
+			LOG_INFO("PASSWORD_MISSING", "%s is not found in Keychain/Keyring", passwordID)
+			return ""
+		}
 
-        if RunInBackground {
-            LOG_INFO("PASSWORD_MISSING", "%s is not found in Keychain/Keyring", passwordID)
-            return ""
-        }
+	}
 
-    }
+	password := ""
+	fmt.Printf("%s", prompt)
+	if showPassword {
+		scanner := bufio.NewScanner(os.Stdin)
+		scanner.Scan()
+		password = scanner.Text()
+	} else {
+		passwordInBytes, err := gopass.GetPasswdMasked()
+		if err != nil {
+			LOG_ERROR("PASSWORD_READ", "Failed to read the password: %v", err)
+			return ""
+		}
+		password = string(passwordInBytes)
+	}
 
-    password := ""
-    fmt.Printf("%s", prompt)
-    if showPassword {
-        scanner := bufio.NewScanner(os.Stdin)
-        scanner.Scan()
-        password = scanner.Text()
-    } else {
-        passwordInBytes, err := gopass.GetPasswdMasked()
-        if err != nil {
-            LOG_ERROR("PASSWORD_READ", "Failed to read the password: %v", err)
-            return ""
-        }
-        password = string(passwordInBytes)
-    }
-
-    return password
+	return password
 }
 
 // SavePassword saves the specified password in the keyring/keychain.
 func SavePassword(preference Preference, passwordType string, password string) {
-    if password == "" || RunInBackground {
-        return
-    }
 
-    if preference.DoNotSavePassword {
-        return
-    }
-    passwordID := passwordType
-    if preference.Name != "default" {
-        passwordID = preference.Name + "_" + passwordID
-    }
-    keyringSet(passwordID, password)
+	if password == "" || RunInBackground {
+		return
+	}
+
+	if preference.DoNotSavePassword {
+		return
+	}
+
+	// If the password is retrieved from env or preference, don't save it to keyring
+	if GetPasswordFromPreference(preference, passwordType) == password {
+		return
+	}
+
+	passwordID := passwordType
+	if preference.Name != "default" {
+		passwordID = preference.Name + "_" + passwordID
+	}
+	keyringSet(passwordID, password)
 }
 
-// RemoveEmptyDirectories remove all empty subdirectoreies under top.
-func RemoveEmptyDirectories(top string) {
-
-    stack := make([]string, 0, 256)
-
-    stack = append(stack, top)
-
-    for len(stack) > 0 {
-
-        dir := stack[len(stack) - 1]
-        stack = stack[:len(stack) - 1]
-
-        files, err := ioutil.ReadDir(dir)
-
-        if err != nil {
-            continue
-        }
-
-        for _, file := range files {
-            if file.IsDir() && file.Name()[0] != '.' {
-                stack = append(stack, path.Join(dir, file.Name()))
-            }
-        }
-
-        if len(files) == 0 {
-            if os.Remove(dir) != nil {
-                continue
-            }
-
-            dir = path.Dir(dir)
-            for (len(dir) > len(top)) {
-                files, err := ioutil.ReadDir(dir)
-                if err != nil {
-                    break
-                }
-
-                if len(files) == 0 {
-                    if os.Remove(dir) != nil {
-                        break;
-                    }
-                }
-                dir = path.Dir(dir)
-            }
-        }
-    }
-}
-
-
 // The following code was modified from the online article  'Matching Wildcards: An Algorithm', by Kirk J. Krauss,
 // Dr. Dobb's, August 26, 2008. However, the version in the article doesn't handle cases like matching 'abcccd'
 // against '*ccd', and the version here fixed that issue.
 //
 func matchPattern(text string, pattern string) bool {
 
-    textLength := len(text)
-    patternLength := len(pattern)
-    afterLastWildcard := 0
-    afterLastMatched := 0
+	textLength := len(text)
+	patternLength := len(pattern)
+	afterLastWildcard := 0
+	afterLastMatched := 0
 
-    t := 0
-    p := 0
+	t := 0
+	p := 0
 
-    for {
-        if t >= textLength {
-            if p >= patternLength {
-                return true           // "x" matches "x"
-            } else if pattern[p] == '*' {
-                p++
-                continue              // "x*" matches "x" or "xy"
-            }
-            return false              // "x" doesn't match "xy"
-        }
+	for {
+		if t >= textLength {
+			if p >= patternLength {
+				return true // "x" matches "x"
+			} else if pattern[p] == '*' {
+				p++
+				continue // "x*" matches "x" or "xy"
+			}
+			return false // "x" doesn't match "xy"
+		}
 
-        w := byte(0)
-        if p < patternLength {
-            w = pattern[p]
-        }
+		w := byte(0)
+		if p < patternLength {
+			w = pattern[p]
+		}
 
-        if text[t] != w  {
-            if w == '?' {
-                t++
-                p++
-                continue
-            } else if w == '*' {
-                p++
-                afterLastWildcard = p
-                if p >= patternLength {
-                    return true
-                }
-            } else if afterLastWildcard > 0 {
-                p = afterLastWildcard
-                t = afterLastMatched
-                t++
-            } else {
-                return false
-            }
+		if text[t] != w {
+			if w == '?' {
+				t++
+				p++
+				continue
+			} else if w == '*' {
+				p++
+				afterLastWildcard = p
+				if p >= patternLength {
+					return true
+				}
+			} else if afterLastWildcard > 0 {
+				p = afterLastWildcard
+				t = afterLastMatched
+				t++
+			} else {
+				return false
+			}
 
-            for t < textLength && text[t] != pattern[p] && pattern[p] != '?' {
-                t++
-            }
+			for t < textLength && text[t] != pattern[p] && pattern[p] != '?' {
+				t++
+			}
 
-            if t >= textLength {
-                return false
-            }
-            afterLastMatched = t
-        }
-        t++
-        p++
-   }
+			if t >= textLength {
+				return false
+			}
+			afterLastMatched = t
+		}
+		t++
+		p++
+	}
 
 }
 
@@ -306,96 +339,124 @@ func matchPattern(text string, pattern string) bool {
 // either '+' or '-', whereas '-' indicates exclusion and '+' indicates inclusion.  Wildcards like '*' and '?' may
 // appear in the patterns.  In case no matching pattern is found, the file will be excluded if all patterns are
 // include patterns, and included otherwise.
-func MatchPath(filePath string, patterns [] string) (included bool) {
+func MatchPath(filePath string, patterns []string) (included bool) {
 
-    allIncludes := true
-    for _, pattern := range patterns {
+	var re *regexp.Regexp = nil
+	var found bool
+	var matched bool
 
-        if pattern[0] == '+' {
-            if matchPattern(filePath, pattern[1:]) {
-                return true
-            }
-        } else if pattern[0] == '-' {
-            allIncludes = false
-             if matchPattern(filePath, pattern[1:]) {
-                return false
-            }
-        }
-    }
+	allIncludes := true
 
-    return !allIncludes
+	for _, pattern := range patterns {
+		if pattern[0] == '+' {
+			if matchPattern(filePath, pattern[1:]) {
+				LOG_DEBUG("PATTERN_INCLUDE", "%s is included by pattern %s", filePath, pattern)
+				return true
+			}
+		} else if pattern[0] == '-' {
+			allIncludes = false
+			if matchPattern(filePath, pattern[1:]) {
+				LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded by pattern %s", filePath, pattern)
+				return false
+			}
+		} else if strings.HasPrefix(pattern, "i:") || strings.HasPrefix(pattern, "e:") {
+			if re, found = RegexMap[pattern[2:]]; found {
+				matched = re.MatchString(filePath)
+			} else {
+				re, err := regexp.Compile(pattern)
+				if err != nil {
+					LOG_ERROR("REGEX_ERROR", "Invalid regex encountered for pattern \"%s\" - %v", pattern[2:], err)
+				}
+				RegexMap[pattern] = re
+				matched = re.MatchString(filePath)
+			}
+			if matched {
+				if strings.HasPrefix(pattern, "i:") {
+					LOG_DEBUG("PATTERN_INCLUDE", "%s is included by pattern %s", filePath, pattern)
+					return true
+				} else {
+					LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded by pattern %s", filePath, pattern)
+					return false
+
+				}
+			} else {
+				if strings.HasPrefix(pattern, "e:") {
+					allIncludes = false
+				}
+			}
+		}
+	}
+
+	if allIncludes {
+		LOG_DEBUG("PATTERN_EXCLUDE", "%s is excluded", filePath)
+		return false
+	} else {
+		LOG_DEBUG("PATTERN_INCLUDE", "%s is included", filePath)
+		return true
+	}
 }
 
-func joinPath(components ...string) string {
+func PrettyNumber(number int64) string {
 
-    combinedPath := path.Join(components...)
-    if len(combinedPath) > 257 && runtime.GOOS == "windows" {
-        combinedPath = `\\?\` + filepath.Join(components...)
-    }
-    return combinedPath
+	G := int64(1024 * 1024 * 1024)
+	M := int64(1024 * 1024)
+	K := int64(1024)
+
+	if number > 1000*G {
+		return fmt.Sprintf("%dG", number/G)
+	} else if number > G {
+		return fmt.Sprintf("%d,%03dM", number/(1000*M), (number/M)%1000)
+	} else if number > M {
+		return fmt.Sprintf("%d,%03dK", number/(1000*K), (number/K)%1000)
+	} else if number > K {
+		return fmt.Sprintf("%dK", number/K)
+	} else {
+		return fmt.Sprintf("%d", number)
+	}
 }
 
-func PrettyNumber(number int64) (string) {
-
-    G := int64(1024 * 1024 * 1024)
-    M := int64(1024 * 1024)
-    K := int64(1024)
-
-    if number > 1000 * G {
-        return fmt.Sprintf("%dG", number / G)
-    } else if number > G {
-        return fmt.Sprintf("%d,%03dM", number / (1000 * M), (number / M) % 1000)
-    } else if number > M {
-        return fmt.Sprintf("%d,%03dK", number / (1000 * K), (number / K) % 1000)
-    } else if number > K {
-        return fmt.Sprintf("%dK", number / K)
-    } else {
-        return fmt.Sprintf("%d", number)
-    }
+func PrettySize(size int64) string {
+	if size > 1024*1024 {
+		return fmt.Sprintf("%.2fM", float64(size)/(1024.0*1024.0))
+	} else if size > 1024 {
+		return fmt.Sprintf("%.0fK", float64(size)/1024.0)
+	} else {
+		return fmt.Sprintf("%d", size)
+	}
 }
 
-func PrettySize(size int64) (string) {
-    if size > 1024 * 1024 {
-        return fmt.Sprintf("%.2fM", float64(size) / (1024.0 * 1024.0))
-    } else if size > 1024 {
-        return fmt.Sprintf("%.0fK", float64(size) / 1024.0)
-    } else {
-        return fmt.Sprintf("%d", size)
-    }
+func PrettyTime(seconds int64) string {
+
+	day := int64(3600 * 24)
+
+	if seconds > day*2 {
+		return fmt.Sprintf("%d days %02d:%02d:%02d",
+			seconds/day, (seconds%day)/3600, (seconds%3600)/60, seconds%60)
+	} else if seconds > day {
+		return fmt.Sprintf("1 day %02d:%02d:%02d", (seconds%day)/3600, (seconds%3600)/60, seconds%60)
+	} else if seconds > 0 {
+		return fmt.Sprintf("%02d:%02d:%02d", seconds/3600, (seconds%3600)/60, seconds%60)
+	} else {
+		return "n/a"
+	}
 }
 
-func PrettyTime(seconds int64) (string) {
+func AtoSize(sizeString string) int {
+	sizeString = strings.ToLower(sizeString)
 
-    day := int64(3600 * 24)
+	sizeRegex := regexp.MustCompile(`^([0-9]+)([mk])?$`)
+	matched := sizeRegex.FindStringSubmatch(sizeString)
+	if matched == nil {
+		return 0
+	}
 
-    if seconds > day * 2 {
-        return fmt.Sprintf("%d days %02d:%02d:%02d",
-                           seconds / day, (seconds % day) / 3600, (seconds % 3600) / 60, seconds % 60)
-    } else if seconds > day {
-        return fmt.Sprintf("1 day %02d:%02d:%02d", (seconds % day) / 3600, (seconds % 3600) / 60, seconds % 60)
-    } else if seconds > 0 {
-        return fmt.Sprintf("%02d:%02d:%02d", seconds / 3600, (seconds % 3600) / 60, seconds % 60)
-    } else {
-        return "n/a"
-    }
-}
-
-func AtoSize(sizeString string) (int) {
-    sizeString = strings.ToLower(sizeString)
-
-    sizeRegex := regexp.MustCompile(`^([0-9]+)([mk])?$`)
-    matched := sizeRegex.FindStringSubmatch(sizeString)
-    if matched == nil {
-        return 0
-    }
-
-    size, _ := strconv.Atoi(matched[1])
-
-    if matched[2] == "m" {
-        size *= 1024 * 1024
-    } else if matched[2] == "k" {
-        size *= 1024
-    }
-
-    return size
+	size, _ := strconv.Atoi(matched[1])
+
+	if matched[2] == "m" {
+		size *= 1024 * 1024
+	} else if matched[2] == "k" {
+		size *= 1024
+	}
+
+	return size
 }

src/duplicacy_utils_others.go

@@ -7,79 +7,84 @@
 package duplicacy
 
 import (
-    "os"
-    "bytes"
-    "syscall"
-    "path/filepath"
+	"bytes"
+	"os"
+	"path"
+	"path/filepath"
+	"syscall"
 
-    "github.com/gilbertchen/xattr"
+	"github.com/gilbertchen/xattr"
 )
 
 func Readlink(path string) (isRegular bool, s string, err error) {
-    s, err = os.Readlink(path)
-    return false, s, err
+	s, err = os.Readlink(path)
+	return false, s, err
 }
 
 func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
-    stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
-    if ok && stat != nil {
-        entry.UID = int(stat.Uid)
-        entry.GID = int(stat.Gid)
-    } else {
-        entry.UID = -1
-        entry.GID = -1
-    }
+	stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
+	if ok && stat != nil {
+		entry.UID = int(stat.Uid)
+		entry.GID = int(stat.Gid)
+	} else {
+		entry.UID = -1
+		entry.GID = -1
+	}
 }
 
-func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) (bool) {
-    stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
-    if ok && stat != nil && (int(stat.Uid) != entry.UID || int(stat.Gid) != entry.GID) {
-        if entry.UID != -1 && entry.GID != -1 {
-            err := os.Chown(fullPath, entry.UID, entry.GID)
-            if err != nil {
-                LOG_ERROR("RESTORE_CHOWN", "Failed to change uid or gid: %v", err)
-                return false
-            }
-        }
-    }
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
+	stat, ok := (*fileInfo).Sys().(*syscall.Stat_t)
+	if ok && stat != nil && (int(stat.Uid) != entry.UID || int(stat.Gid) != entry.GID) {
+		if entry.UID != -1 && entry.GID != -1 {
+			err := os.Lchown(fullPath, entry.UID, entry.GID)
+			if err != nil {
+				LOG_ERROR("RESTORE_CHOWN", "Failed to change uid or gid: %v", err)
+				return false
+			}
+		}
+	}
 
-    return true
+	return true
 }
 
 func (entry *Entry) ReadAttributes(top string) {
 
-    fullPath := filepath.Join(top, entry.Path)
-    attributes, _ := xattr.Listxattr(fullPath)
-    if len(attributes) > 0 {
-        entry.Attributes = make(map[string][]byte)
-        for _, name := range attributes {
-            attribute, err := xattr.Getxattr(fullPath, name)
-            if err == nil {
-                entry.Attributes[name] = attribute
-            }
-        }
-    }
+	fullPath := filepath.Join(top, entry.Path)
+	attributes, _ := xattr.Listxattr(fullPath)
+	if len(attributes) > 0 {
+		entry.Attributes = make(map[string][]byte)
+		for _, name := range attributes {
+			attribute, err := xattr.Getxattr(fullPath, name)
+			if err == nil {
+				entry.Attributes[name] = attribute
+			}
+		}
+	}
 }
 
 func (entry *Entry) SetAttributesToFile(fullPath string) {
-    names, _ := xattr.Listxattr(fullPath)
+	names, _ := xattr.Listxattr(fullPath)
 
-    for _, name := range names {
+	for _, name := range names {
 
-        newAttribute, found := entry.Attributes[name]
-        if found {
-            oldAttribute, _ := xattr.Getxattr(fullPath, name)
-            if bytes.Equal(oldAttribute, newAttribute) {
-                xattr.Setxattr(fullPath, name, newAttribute)
-            }
-            delete(entry.Attributes, name)
-        } else {
-            xattr.Removexattr(fullPath, name)
-        }
-    }
+		newAttribute, found := entry.Attributes[name]
+		if found {
+			oldAttribute, _ := xattr.Getxattr(fullPath, name)
+			if !bytes.Equal(oldAttribute, newAttribute) {
+				xattr.Setxattr(fullPath, name, newAttribute)
+			}
+			delete(entry.Attributes, name)
+		} else {
+			xattr.Removexattr(fullPath, name)
+		}
+	}
 
-    for name, attribute := range entry.Attributes {
-        xattr.Setxattr(fullPath, name, attribute)
-    }
+	for name, attribute := range entry.Attributes {
+		xattr.Setxattr(fullPath, name, attribute)
+	}
 
 }
+
+func joinPath(components ...string) string {
+	return path.Join(components...)
+}

src/duplicacy_utils_test.go

@@ -5,135 +5,145 @@
 package duplicacy
 
 import (
-    "io"
-    "io/ioutil"
-    "time"
-    "bytes"
+	"bytes"
+	"io"
+	"io/ioutil"
+	"time"
 
-    crypto_rand "crypto/rand"
-
-    "testing"
+	crypto_rand "crypto/rand"
 
+	"testing"
 )
 
 func TestMatchPattern(t *testing.T) {
 
-    // Test cases were copied from Matching Wildcards: An Empirical Way to Tame an Algorithm
-    // By Kirk J. Krauss, October 07, 2014
+	// Test cases were copied from Matching Wildcards: An Empirical Way to Tame an Algorithm
+	// By Kirk J. Krauss, October 07, 2014
 
-    DATA := [] struct {
-        text string
-        pattern string
-        matched bool
-    } {
-        // Cases with repeating character sequences.
-        { "abcccd", "*ccd", true },
-        { "mississipissippi", "*issip*ss*", true },
-        { "xxxx*zzzzzzzzy*f", "xxxx*zzy*fffff", false },
-        { "xxxx*zzzzzzzzy*f", "xxx*zzy*f", true },
-        { "xxxxzzzzzzzzyf", "xxxx*zzy*fffff", false },
-        { "xxxxzzzzzzzzyf", "xxxx*zzy*f", true },
-        { "xyxyxyzyxyz", "xy*z*xyz", true },
-        { "mississippi", "*sip*", true },
-        { "xyxyxyxyz", "xy*xyz", true },
-        { "mississippi", "mi*sip*", true },
-        { "ababac", "*abac*", true },
-        { "ababac", "*abac*", true },
-        { "aaazz", "a*zz*", true },
-        { "a12b12", "*12*23", false },
-        { "a12b12", "a12b", false },
-        { "a12b12", "*12*12*", true },
+	DATA := []struct {
+		text    string
+		pattern string
+		matched bool
+	}{
+		// Cases with repeating character sequences.
+		{"abcccd", "*ccd", true},
+		{"mississipissippi", "*issip*ss*", true},
+		{"xxxx*zzzzzzzzy*f", "xxxx*zzy*fffff", false},
+		{"xxxx*zzzzzzzzy*f", "xxx*zzy*f", true},
+		{"xxxxzzzzzzzzyf", "xxxx*zzy*fffff", false},
+		{"xxxxzzzzzzzzyf", "xxxx*zzy*f", true},
+		{"xyxyxyzyxyz", "xy*z*xyz", true},
+		{"mississippi", "*sip*", true},
+		{"xyxyxyxyz", "xy*xyz", true},
+		{"mississippi", "mi*sip*", true},
+		{"ababac", "*abac*", true},
+		{"ababac", "*abac*", true},
+		{"aaazz", "a*zz*", true},
+		{"a12b12", "*12*23", false},
+		{"a12b12", "a12b", false},
+		{"a12b12", "*12*12*", true},
 
-        // More double wildcard scenarios.
-        { "XYXYXYZYXYz", "XY*Z*XYz", true },
-        { "missisSIPpi", "*SIP*", true },
-        { "mississipPI", "*issip*PI", true },
-        { "xyxyxyxyz", "xy*xyz", true },
-        { "miSsissippi", "mi*sip*", true },
-        { "miSsissippi", "mi*Sip*", false },
-        { "abAbac", "*Abac*", true },
-        { "abAbac", "*Abac*", true },
-        { "aAazz", "a*zz*", true },
-        { "A12b12", "*12*23", false },
-        { "a12B12", "*12*12*", true },
-        { "oWn", "*oWn*", true },
+		// More double wildcard scenarios.
+		{"XYXYXYZYXYz", "XY*Z*XYz", true},
+		{"missisSIPpi", "*SIP*", true},
+		{"mississipPI", "*issip*PI", true},
+		{"xyxyxyxyz", "xy*xyz", true},
+		{"miSsissippi", "mi*sip*", true},
+		{"miSsissippi", "mi*Sip*", false},
+		{"abAbac", "*Abac*", true},
+		{"abAbac", "*Abac*", true},
+		{"aAazz", "a*zz*", true},
+		{"A12b12", "*12*23", false},
+		{"a12B12", "*12*12*", true},
+		{"oWn", "*oWn*", true},
 
-        // Completely tame (no wildcards) cases.
-        { "bLah", "bLah", true },
-        { "bLah", "bLaH", false },
+		// Completely tame (no wildcards) cases.
+		{"bLah", "bLah", true},
+		{"bLah", "bLaH", false},
 
-        // Simple mixed wildcard tests suggested by IBMer Marlin Deckert.
-        { "a", "*?", true },
-        { "ab", "*?", true },
-        { "abc", "*?", true },
+		// Simple mixed wildcard tests suggested by IBMer Marlin Deckert.
+		{"a", "*?", true},
+		{"ab", "*?", true},
+		{"abc", "*?", true},
 
-        // More mixed wildcard tests including coverage for false positives.
-        { "a", "??", false },
-        { "ab", "?*?", true },
-        { "ab", "*?*?*", true },
-        { "abc", "?*?*?", true },
-        { "abc", "?*?*&?", false },
-        { "abcd", "?b*??", true },
-        { "abcd", "?a*??", false },
-        { "abcd", "?*?c?", true },
-        { "abcd", "?*?d?", false },
-        { "abcde", "?*b*?*d*?", true },
+		// More mixed wildcard tests including coverage for false positives.
+		{"a", "??", false},
+		{"ab", "?*?", true},
+		{"ab", "*?*?*", true},
+		{"abc", "?*?*?", true},
+		{"abc", "?*?*&?", false},
+		{"abcd", "?b*??", true},
+		{"abcd", "?a*??", false},
+		{"abcd", "?*?c?", true},
+		{"abcd", "?*?d?", false},
+		{"abcde", "?*b*?*d*?", true},
 
-        // Single-character-match cases.
-        { "bLah", "bL?h", true },
-        { "bLaaa", "bLa?", false },
-        { "bLah", "bLa?", true },
-        { "bLaH", "?Lah", false },
-        { "bLaH", "?LaH", true },
-    }
+		// Single-character-match cases.
+		{"bLah", "bL?h", true},
+		{"bLaaa", "bLa?", false},
+		{"bLah", "bLa?", true},
+		{"bLaH", "?Lah", false},
+		{"bLaH", "?LaH", true},
+	}
 
-    for _, data := range DATA {
-        if matchPattern(data.text, data.pattern) != data.matched {
-            t.Errorf("text: %s, pattern %s, expected: %t", data.text, data.pattern, data.matched)
-        }
-    }
+	for _, data := range DATA {
+		if matchPattern(data.text, data.pattern) != data.matched {
+			t.Errorf("text: %s, pattern %s, expected: %t", data.text, data.pattern, data.matched)
+		}
+	}
 
+	for _, pattern := range []string{ "+", "-", "i:", "e:", "+a", "-a", "i:a", "e:a"} {
+		if IsUnspecifiedFilter(pattern) {
+			t.Errorf("pattern %s has a specified filter", pattern)
+		}
+	}
+
+	for _, pattern := range []string{ "i", "e", "ia", "ib", "a", "b"} {
+		if !IsUnspecifiedFilter(pattern) {
+			t.Errorf("pattern %s does not have a specified filter", pattern)
+		}
+	}
 }
 
 func TestRateLimit(t *testing.T) {
-    content := make([]byte, 100 * 1024)
-    _, err := crypto_rand.Read(content)
-    if err != nil {
-        t.Errorf("Error generating random content: %v", err)
-        return
-    }
+	content := make([]byte, 100*1024)
+	_, err := crypto_rand.Read(content)
+	if err != nil {
+		t.Errorf("Error generating random content: %v", err)
+		return
+	}
 
-    expectedRate := 10
-    rateLimiter := CreateRateLimitedReader(content, expectedRate)
+	expectedRate := 10
+	rateLimiter := CreateRateLimitedReader(content, expectedRate)
 
-    startTime := time.Now()
-    n, err := io.Copy(ioutil.Discard, rateLimiter)
-    if err != nil {
-        t.Errorf("Error reading from the rate limited reader: %v", err)
-        return
-    }
-    if int(n) != len(content) {
-        t.Errorf("Wrote %s bytes instead of %s", n, len(content))
-        return
-    }
+	startTime := time.Now()
+	n, err := io.Copy(ioutil.Discard, rateLimiter)
+	if err != nil {
+		t.Errorf("Error reading from the rate limited reader: %v", err)
+		return
+	}
+	if int(n) != len(content) {
+		t.Errorf("Wrote %d bytes instead of %d", n, len(content))
+		return
+	}
 
-    elapsed := time.Since(startTime)
-    actualRate := float64(len(content)) / elapsed.Seconds() / 1024
-    t.Logf("Elapsed time: %s, actual rate: %.3f kB/s, expected rate: %d kB/s", elapsed, actualRate, expectedRate)
+	elapsed := time.Since(startTime)
+	actualRate := float64(len(content)) / elapsed.Seconds() / 1024
+	t.Logf("Elapsed time: %s, actual rate: %.3f kB/s, expected rate: %d kB/s", elapsed, actualRate, expectedRate)
 
-    startTime = time.Now()
-    n, err = RateLimitedCopy(ioutil.Discard, bytes.NewBuffer(content), expectedRate)
-    if err != nil {
-        t.Errorf("Error writing with rate limit: %v", err)
-        return
-    }
-    if int(n) != len(content) {
-        t.Errorf("Copied %s bytes instead of %s", n, len(content))
-        return
-    }
+	startTime = time.Now()
+	n, err = RateLimitedCopy(ioutil.Discard, bytes.NewBuffer(content), expectedRate)
+	if err != nil {
+		t.Errorf("Error writing with rate limit: %v", err)
+		return
+	}
+	if int(n) != len(content) {
+		t.Errorf("Copied %d bytes instead of %d", n, len(content))
+		return
+	}
 
-    elapsed = time.Since(startTime)
-    actualRate = float64(len(content)) / elapsed.Seconds() / 1024
-    t.Logf("Elapsed time: %s, actual rate: %.3f kB/s, expected rate: %d kB/s", elapsed, actualRate, expectedRate)
+	elapsed = time.Since(startTime)
+	actualRate = float64(len(content)) / elapsed.Seconds() / 1024
+	t.Logf("Elapsed time: %s, actual rate: %.3f kB/s, expected rate: %d kB/s", elapsed, actualRate, expectedRate)
 
 }

src/duplicacy_utils_windows.go

@@ -5,46 +5,49 @@
 package duplicacy
 
 import (
-    "os"
-    "fmt"
-    "syscall"
-    "unsafe"
+	"fmt"
+	"os"
+	"path/filepath"
+	"strings"
+	"syscall"
+	"unsafe"
 )
 
 type symbolicLinkReparseBuffer struct {
-    SubstituteNameOffset uint16
-    SubstituteNameLength uint16
-    PrintNameOffset      uint16
-    PrintNameLength      uint16
-    Flags                uint32
-    PathBuffer           [1]uint16
+	SubstituteNameOffset uint16
+	SubstituteNameLength uint16
+	PrintNameOffset      uint16
+	PrintNameLength      uint16
+	Flags                uint32
+	PathBuffer           [1]uint16
 }
 
 type mountPointReparseBuffer struct {
-    SubstituteNameOffset uint16
-    SubstituteNameLength uint16
-    PrintNameOffset      uint16
-    PrintNameLength      uint16
-    PathBuffer           [1]uint16
+	SubstituteNameOffset uint16
+	SubstituteNameLength uint16
+	PrintNameOffset      uint16
+	PrintNameLength      uint16
+	PathBuffer           [1]uint16
 }
 
 type reparseDataBuffer struct {
-    ReparseTag        uint32
-    ReparseDataLength uint16
-    Reserved          uint16
+	ReparseTag        uint32
+	ReparseDataLength uint16
+	Reserved          uint16
 
-    // GenericReparseBuffer
-    reparseBuffer byte
+	// GenericReparseBuffer
+	reparseBuffer byte
 }
-const (
-    FSCTL_GET_REPARSE_POINT          = 0x900A8
-    MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 16 * 1024
-    IO_REPARSE_TAG_MOUNT_POINT       = 0xA0000003
-    IO_REPARSE_TAG_SYMLINK           = 0xA000000C
-    IO_REPARSE_TAG_DEDUP             = 0x80000013
-    SYMBOLIC_LINK_FLAG_DIRECTORY     = 0x1
 
-    FILE_READ_ATTRIBUTES             = 0x0080
+const (
+	FSCTL_GET_REPARSE_POINT          = 0x900A8
+	MAXIMUM_REPARSE_DATA_BUFFER_SIZE = 16 * 1024
+	IO_REPARSE_TAG_MOUNT_POINT       = 0xA0000003
+	IO_REPARSE_TAG_SYMLINK           = 0xA000000C
+	IO_REPARSE_TAG_DEDUP             = 0x80000013
+	SYMBOLIC_LINK_FLAG_DIRECTORY     = 0x1
+
+	FILE_READ_ATTRIBUTES = 0x0080
 )
 
 // We copied golang source code for Readlink but made a simple modification here:  use FILE_READ_ATTRIBUTES instead of
@@ -53,58 +56,58 @@ const (
 
 // Readlink returns the destination of the named symbolic link.
 func Readlink(path string) (isRegular bool, s string, err error) {
-    fd, err := syscall.CreateFile(syscall.StringToUTF16Ptr(path), FILE_READ_ATTRIBUTES,
-                                  syscall.FILE_SHARE_READ, nil, syscall.OPEN_EXISTING,
-                                  syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS, 0)
-    if err != nil {
-        return false, "", err
-    }
-    defer syscall.CloseHandle(fd)
+	fd, err := syscall.CreateFile(syscall.StringToUTF16Ptr(path), FILE_READ_ATTRIBUTES,
+		syscall.FILE_SHARE_READ, nil, syscall.OPEN_EXISTING,
+		syscall.FILE_FLAG_OPEN_REPARSE_POINT|syscall.FILE_FLAG_BACKUP_SEMANTICS, 0)
+	if err != nil {
+		return false, "", err
+	}
+	defer syscall.CloseHandle(fd)
 
-    rdbbuf := make([]byte, syscall.MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
-    var bytesReturned uint32
-    err = syscall.DeviceIoControl(fd, syscall.FSCTL_GET_REPARSE_POINT, nil, 0, &rdbbuf[0],
-                                  uint32(len(rdbbuf)), &bytesReturned, nil)
-    if err != nil {
-        return false, "", err
-    }
+	rdbbuf := make([]byte, syscall.MAXIMUM_REPARSE_DATA_BUFFER_SIZE)
+	var bytesReturned uint32
+	err = syscall.DeviceIoControl(fd, syscall.FSCTL_GET_REPARSE_POINT, nil, 0, &rdbbuf[0],
+		uint32(len(rdbbuf)), &bytesReturned, nil)
+	if err != nil {
+		return false, "", err
+	}
 
-    rdb := (*reparseDataBuffer)(unsafe.Pointer(&rdbbuf[0]))
-    switch rdb.ReparseTag {
-    case IO_REPARSE_TAG_SYMLINK:
-        data := (*symbolicLinkReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
-        p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
-        if data.PrintNameLength > 0 {
-            s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength + data.PrintNameOffset)/2])
-        } else {
-            s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength + data.SubstituteNameOffset)/2])
-        }
-    case IO_REPARSE_TAG_MOUNT_POINT:
-        data := (*mountPointReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
-        p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
-        if data.PrintNameLength > 0 {
-            s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength + data.PrintNameOffset)/2])
-        } else {
-            s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength + data.SubstituteNameOffset)/2])
-        }
-    case IO_REPARSE_TAG_DEDUP:
-        return true, "", nil
-    default:
-        // the path is not a symlink or junction but another type of reparse
-        // point
-        return false, "", fmt.Errorf("Unhandled reparse point type %x", rdb.ReparseTag)
-    }
+	rdb := (*reparseDataBuffer)(unsafe.Pointer(&rdbbuf[0]))
+	switch rdb.ReparseTag {
+	case IO_REPARSE_TAG_SYMLINK:
+		data := (*symbolicLinkReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
+		p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
+		if data.PrintNameLength > 0 {
+			s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength+data.PrintNameOffset)/2])
+		} else {
+			s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength+data.SubstituteNameOffset)/2])
+		}
+	case IO_REPARSE_TAG_MOUNT_POINT:
+		data := (*mountPointReparseBuffer)(unsafe.Pointer(&rdb.reparseBuffer))
+		p := (*[0xffff]uint16)(unsafe.Pointer(&data.PathBuffer[0]))
+		if data.PrintNameLength > 0 {
+			s = syscall.UTF16ToString(p[data.PrintNameOffset/2 : (data.PrintNameLength+data.PrintNameOffset)/2])
+		} else {
+			s = syscall.UTF16ToString(p[data.SubstituteNameOffset/2 : (data.SubstituteNameLength+data.SubstituteNameOffset)/2])
+		}
+	case IO_REPARSE_TAG_DEDUP:
+		return true, "", nil
+	default:
+		// the path is not a symlink or junction but another type of reparse
+		// point
+		return false, "", fmt.Errorf("Unhandled reparse point type %x", rdb.ReparseTag)
+	}
 
-    return false, s, nil
+	return false, s, nil
 }
 
 func GetOwner(entry *Entry, fileInfo *os.FileInfo) {
-    entry.UID = -1
-    entry.GID = -1
+	entry.UID = -1
+	entry.GID = -1
 }
 
-func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) (bool) {
-    return true
+func SetOwner(fullPath string, entry *Entry, fileInfo *os.FileInfo) bool {
+	return true
 }
 
 func (entry *Entry) ReadAttributes(top string) {
@@ -113,3 +116,13 @@ func (entry *Entry) ReadAttributes(top string) {
 func (entry *Entry) SetAttributesToFile(fullPath string) {
 
 }
+
+func joinPath(components ...string) string {
+
+	combinedPath := `\\?\` + filepath.Join(components...)
+	// If the path is on a samba drive we must use the UNC format
+	if strings.HasPrefix(combinedPath, `\\?\\\`) {
+		combinedPath = `\\?\UNC\` + combinedPath[6:]
+	}
+	return combinedPath
+}

src/duplicacy_wasabistorage.go

@@ -0,0 +1,192 @@
+//
+// Storage module for Wasabi (https://www.wasabi.com)
+//
+
+// Wasabi is nominally compatible with AWS S3, but the copy-and-delete
+// method used for renaming objects creates additional expense under
+// Wasabi's billing system.  This module is a pass-through to the
+// existing S3 module for everything other than that one operation.
+//
+// This module copyright 2017 Mark Feit (https://github.com/markfeit)
+// and may be distributed under the same terms as Duplicacy.
+
+package duplicacy
+
+import (
+	"crypto/hmac"
+	"crypto/sha1"
+	"encoding/base64"
+	"errors"
+	"fmt"
+	"net/http"
+	"time"
+)
+
+type WasabiStorage struct {
+	StorageBase
+
+	s3         *S3Storage
+	region     string
+	endpoint   string
+	bucket     string
+	storageDir string
+	key        string
+	secret     string
+	client     *http.Client
+}
+
+// See the Storage interface in duplicacy_storage.go for function
+// descriptions.
+
+func CreateWasabiStorage(
+	regionName string, endpoint string,
+	bucketName string, storageDir string,
+	accessKey string, secretKey string,
+	threads int,
+) (storage *WasabiStorage, err error) {
+
+	s3storage, error := CreateS3Storage(regionName, endpoint, bucketName,
+		storageDir, accessKey, secretKey, threads,
+		true,  // isSSLSupported
+		false, // isMinioCompatible
+	)
+
+	if err != nil {
+		return nil, error
+	}
+
+	wasabi := &WasabiStorage{
+
+		// Pass-through to existing S3 module
+		s3: s3storage,
+
+		// Local copies required for renaming
+		region:     regionName,
+		endpoint:   endpoint,
+		bucket:     bucketName,
+		storageDir: storageDir,
+		key:        accessKey,
+		secret:     secretKey,
+		client:     &http.Client{},
+	}
+
+	wasabi.DerivedStorage = wasabi
+	wasabi.SetDefaultNestingLevels([]int{0}, 0)
+
+	return wasabi, nil
+}
+
+func (storage *WasabiStorage) ListFiles(
+	threadIndex int, dir string,
+) (files []string, sizes []int64, err error) {
+	return storage.s3.ListFiles(threadIndex, dir)
+}
+
+func (storage *WasabiStorage) DeleteFile(
+	threadIndex int, filePath string,
+) (err error) {
+	return storage.s3.DeleteFile(threadIndex, filePath)
+
+}
+
+// This is a lightweight implementation of a call to Wasabi for a
+// rename.  It's designed to get the job done with as few dependencies
+// on other packages as possible rather than being somethng
+// general-purpose and reusable.
+func (storage *WasabiStorage) MoveFile(threadIndex int, from string, to string) (err error) {
+
+	var fromPath string
+	// The from path includes the bucket.  Take care not to include an empty storageDir
+	// string as Wasabi's backend will return 404 on URLs with double slashes.
+	if storage.storageDir == "" {
+		fromPath = fmt.Sprintf("/%s/%s", storage.bucket, from)
+	} else {
+		fromPath = fmt.Sprintf("/%s/%s/%s", storage.bucket, storage.storageDir, from)
+	}
+
+	object := fmt.Sprintf("https://%s@%s%s", storage.region, storage.endpoint, fromPath)
+
+	toPath := to
+	// The object's new name is relative to the top of the bucket.
+	if storage.storageDir != "" {
+		toPath = fmt.Sprintf("%s/%s", storage.storageDir, to)
+	}
+
+	timestamp := time.Now().Format(time.RFC1123Z)
+
+	signingString := fmt.Sprintf("MOVE\n\n\n%s\n%s", timestamp, fromPath)
+
+	signer := hmac.New(sha1.New, []byte(storage.secret))
+	signer.Write([]byte(signingString))
+
+	signature := base64.StdEncoding.EncodeToString(signer.Sum(nil))
+
+	authorization := fmt.Sprintf("AWS %s:%s", storage.key, signature)
+
+	request, err := http.NewRequest("MOVE", object, nil)
+	if err != nil {
+		return err
+	}
+	request.Header.Add("Authorization", authorization)
+	request.Header.Add("Date", timestamp)
+	request.Header.Add("Destination", toPath)
+	request.Header.Add("Host", storage.endpoint)
+	request.Header.Add("Overwrite", "true")
+
+	response, err := storage.client.Do(request)
+	if err != nil {
+		return err
+	}
+	defer response.Body.Close()
+
+	if response.StatusCode != 200 {
+		return errors.New(response.Status)
+	}
+
+	return nil
+}
+
+func (storage *WasabiStorage) CreateDirectory(
+	threadIndex int, dir string,
+) (err error) {
+	return storage.s3.CreateDirectory(threadIndex, dir)
+}
+
+func (storage *WasabiStorage) GetFileInfo(
+	threadIndex int, filePath string,
+) (exist bool, isDir bool, size int64, err error) {
+	return storage.s3.GetFileInfo(threadIndex, filePath)
+}
+
+func (storage *WasabiStorage) DownloadFile(
+	threadIndex int, filePath string, chunk *Chunk,
+) (err error) {
+	return storage.s3.DownloadFile(threadIndex, filePath, chunk)
+}
+
+func (storage *WasabiStorage) UploadFile(
+	threadIndex int, filePath string, content []byte,
+) (err error) {
+	return storage.s3.UploadFile(threadIndex, filePath, content)
+}
+
+func (storage *WasabiStorage) IsCacheNeeded() bool {
+	return storage.s3.IsCacheNeeded()
+}
+
+func (storage *WasabiStorage) IsMoveFileImplemented() bool {
+	// This is implemented locally since S3 does a copy and delete
+	return true
+}
+
+func (storage *WasabiStorage) IsStrongConsistent() bool {
+	// Wasabi has it, S3 doesn't.
+	return true
+}
+
+func (storage *WasabiStorage) IsFastListing() bool {
+	return storage.s3.IsFastListing()
+}
+
+func (storage *WasabiStorage) EnableTestMode() {
+}

src/duplicacy_webdavstorage.go

@@ -0,0 +1,462 @@
+// Copyright (c) Acrosync LLC. All rights reserved.
+// Free for personal use and commercial trial
+// Commercial use requires per-user licenses available from https://duplicacy.com
+//
+//
+// This storage backend is based on the work by Yuri Karamani from https://github.com/karamani/webdavclnt,
+// released under the MIT license.
+//
+package duplicacy
+
+import (
+	"bytes"
+	"encoding/xml"
+	"errors"
+	"fmt"
+	"io"
+	"io/ioutil"
+	"math/rand"
+	"net/http"
+	//"net/http/httputil"
+	"strconv"
+	"strings"
+	"sync"
+	"time"
+)
+
+type WebDAVStorage struct {
+	StorageBase
+
+	host       string
+	port       int
+	username   string
+	password   string
+	storageDir string
+	useHTTP    bool
+
+	client             *http.Client
+	threads            int
+	directoryCache     map[string]int // stores directories known to exist by this backend
+	directoryCacheLock sync.Mutex     // lock for accessing directoryCache
+}
+
+var (
+	errWebDAVAuthorizationFailure = errors.New("Authentication failed")
+	errWebDAVMovedPermanently     = errors.New("Moved permanently")
+	errWebDAVNotExist             = errors.New("Path does not exist")
+	errWebDAVMaximumBackoff       = errors.New("Maximum backoff reached")
+	errWebDAVMethodNotAllowed     = errors.New("Method not allowed")
+)
+
+func CreateWebDAVStorage(host string, port int, username string, password string, storageDir string, useHTTP bool, threads int) (storage *WebDAVStorage, err error) {
+	if len(storageDir) > 0 && storageDir[len(storageDir)-1] != '/' {
+		storageDir += "/"
+	}
+
+	storage = &WebDAVStorage{
+		host:       host,
+		port:       port,
+		username:   username,
+		password:   password,
+		storageDir: "",
+		useHTTP:    useHTTP,
+
+		client:         http.DefaultClient,
+		threads:        threads,
+		directoryCache: make(map[string]int),
+	}
+
+	// Make sure it doesn't follow redirect
+	storage.client.CheckRedirect = func(req *http.Request, via []*http.Request) error {
+		return http.ErrUseLastResponse
+	}
+
+	exist, isDir, _, err := storage.GetFileInfo(0, storageDir)
+	if err != nil {
+		return nil, err
+	}
+	if !exist {
+		return nil, fmt.Errorf("Storage path %s does not exist", storageDir)
+	}
+	if !isDir {
+		return nil, fmt.Errorf("Storage path %s is not a directory", storageDir)
+	}
+	storage.storageDir = storageDir
+
+	for _, dir := range []string{"snapshots", "chunks"} {
+		storage.CreateDirectory(0, dir)
+	}
+
+	storage.DerivedStorage = storage
+	storage.SetDefaultNestingLevels([]int{0}, 0)
+	return storage, nil
+}
+
+func (storage *WebDAVStorage) createConnectionString(uri string) string {
+
+	url := storage.host
+
+	if storage.useHTTP {
+		url = "http://" + url
+	} else {
+		url = "https://" + url
+	}
+
+	if storage.port > 0 {
+		url += fmt.Sprintf(":%d", storage.port)
+	}
+	return url + "/" + storage.storageDir + uri
+}
+
+func (storage *WebDAVStorage) retry(backoff int) int {
+	delay := rand.Intn(backoff*500) + backoff*500
+	time.Sleep(time.Duration(delay) * time.Millisecond)
+	backoff *= 2
+	return backoff
+}
+
+func (storage *WebDAVStorage) sendRequest(method string, uri string, depth int, data []byte) (io.ReadCloser, http.Header, error) {
+
+	backoff := 1
+	for i := 0; i < 8; i++ {
+
+		var dataReader io.Reader
+		headers := make(map[string]string)
+		if method == "PROPFIND" {
+			headers["Content-Type"] = "application/xml"
+			headers["Depth"] = fmt.Sprintf("%d", depth)
+			dataReader = bytes.NewReader(data)
+		} else if method == "PUT" {
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = CreateRateLimitedReader(data, storage.UploadRateLimit/storage.threads)
+		} else if method == "MOVE" {
+			headers["Destination"] = storage.createConnectionString(string(data))
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = bytes.NewReader([]byte(""))
+		} else {
+			headers["Content-Type"] = "application/octet-stream"
+			dataReader = bytes.NewReader(data)
+		}
+
+		request, err := http.NewRequest(method, storage.createConnectionString(uri), dataReader)
+		if err != nil {
+			return nil, nil, err
+		}
+
+		if len(storage.username) > 0 {
+			request.SetBasicAuth(storage.username, storage.password)
+		}
+
+		for key, value := range headers {
+			request.Header.Set(key, value)
+		}
+
+		if method == "PUT" {
+			request.ContentLength = int64(len(data))
+		}
+
+		//requestDump, err := httputil.DumpRequest(request, true)
+		//LOG_INFO("debug", "Request: %s", requestDump)
+
+		response, err := storage.client.Do(request)
+		if err != nil {
+			LOG_TRACE("WEBDAV_RETRY", "URL request '%s %s' returned an error (%v)", method, uri, err)
+			backoff = storage.retry(backoff)
+			continue
+		}
+
+		if response.StatusCode < 300 {
+			return response.Body, response.Header, nil
+		}
+
+		if response.StatusCode == 301 {
+			return nil, nil, errWebDAVMovedPermanently
+		}
+
+		response.Body.Close()
+		if response.StatusCode == 404 {
+			// Retry if it is UPLOAD, otherwise return immediately
+			if method != "PUT" {
+				return nil, nil, errWebDAVNotExist
+			}
+		} else if response.StatusCode == 405 {
+			return nil, nil, errWebDAVMethodNotAllowed
+		}
+		LOG_INFO("WEBDAV_RETRY", "URL request '%s %s' returned status code %d", method, uri, response.StatusCode)
+		backoff = storage.retry(backoff)
+	}
+	return nil, nil, errWebDAVMaximumBackoff
+}
+
+type WebDAVProperties map[string]string
+
+type WebDAVPropValue struct {
+	XMLName xml.Name `xml:""`
+	Value   string   `xml:",innerxml"`
+}
+
+type WebDAVProp struct {
+	PropList []WebDAVPropValue `xml:",any"`
+}
+
+type WebDAVPropStat struct {
+	Prop *WebDAVProp `xml:"prop"`
+}
+
+type WebDAVResponse struct {
+	Href     string          `xml:"href"`
+	PropStat *WebDAVPropStat `xml:"propstat"`
+}
+
+type WebDAVMultiStatus struct {
+	Responses []WebDAVResponse `xml:"response"`
+}
+
+func (storage *WebDAVStorage) getProperties(uri string, depth int, properties ...string) (map[string]WebDAVProperties, error) {
+
+	propfind := "<prop>"
+	for _, p := range properties {
+		propfind += fmt.Sprintf("<%s/>", p)
+	}
+	propfind += "</prop>"
+
+	body := fmt.Sprintf(`<?xml version="1.0" encoding="utf-8" ?><propfind xmlns="DAV:">%s</propfind>`, propfind)
+
+	readCloser, _, err := storage.sendRequest("PROPFIND", uri, depth, []byte(body))
+	if err != nil {
+		return nil, err
+	}
+	defer readCloser.Close()
+	content, err := ioutil.ReadAll(readCloser)
+	if err != nil {
+		return nil, err
+	}
+
+	object := WebDAVMultiStatus{}
+	err = xml.Unmarshal(content, &object)
+	if err != nil {
+		return nil, err
+	}
+
+	if object.Responses == nil || len(object.Responses) == 0 {
+		return nil, errors.New("no WebDAV responses")
+	}
+
+	responses := make(map[string]WebDAVProperties)
+
+	for _, responseTag := range object.Responses {
+
+		if responseTag.PropStat == nil || responseTag.PropStat.Prop == nil || responseTag.PropStat.Prop.PropList == nil {
+			return nil, errors.New("no WebDAV properties")
+		}
+
+		properties := make(WebDAVProperties)
+		for _, prop := range responseTag.PropStat.Prop.PropList {
+			properties[prop.XMLName.Local] = prop.Value
+		}
+
+		responseKey := responseTag.Href
+		responses[responseKey] = properties
+
+	}
+
+	return responses, nil
+}
+
+// ListFiles return the list of files and subdirectories under 'dir'.  A subdirectories returned must have a trailing '/', with
+// a size of 0.  If 'dir' is 'snapshots', only subdirectories will be returned.  If 'dir' is 'snapshots/repository_id', then only
+// files will be returned.  If 'dir' is 'chunks', the implementation can return the list either recusively or non-recusively.
+func (storage *WebDAVStorage) ListFiles(threadIndex int, dir string) (files []string, sizes []int64, err error) {
+	if dir[len(dir)-1] != '/' {
+		dir += "/"
+	}
+	properties, err := storage.getProperties(dir, 1, "getcontentlength", "resourcetype")
+	if err != nil {
+		return nil, nil, err
+	}
+
+	prefixLength := len(storage.storageDir) + len(dir) + 1
+
+	for file, m := range properties {
+		if len(file) <= prefixLength {
+			continue
+		}
+
+		isDir := false
+		size := 0
+		if resourceType, exist := m["resourcetype"]; exist && strings.Contains(resourceType, "collection") {
+			isDir = true
+		} else if length, exist := m["getcontentlength"]; exist {
+			if length == "" {
+				isDir = true
+			} else {
+				size, _ = strconv.Atoi(length)
+			}
+		} else {
+			continue
+		}
+
+		if !isDir {
+			if dir != "snapshots/" {
+				files = append(files, file[prefixLength:])
+				sizes = append(sizes, int64(size))
+			}
+		} else {
+			// This is a dir
+			file := file[prefixLength:]
+			if file[len(file)-1] != '/' {
+				file += "/"
+			}
+			files = append(files, file)
+			sizes = append(sizes, int64(0))
+		}
+	}
+
+	return files, sizes, nil
+}
+
+// GetFileInfo returns the information about the file or directory at 'filePath'.
+func (storage *WebDAVStorage) GetFileInfo(threadIndex int, filePath string) (exist bool, isDir bool, size int64, err error) {
+
+	properties, err := storage.getProperties(filePath, 0, "getcontentlength", "resourcetype")
+	if err != nil {
+		if err == errWebDAVNotExist {
+			return false, false, 0, nil
+		}
+		if err == errWebDAVMovedPermanently {
+			// This must be a directory
+			return true, true, 0, nil
+		}
+		return false, false, 0, err
+	}
+
+	m, exist := properties["/"+storage.storageDir+filePath]
+
+	// If no properties exist for the given filePath, remove the trailing / from filePath and search again
+	if !exist && filePath != "" && filePath[len(filePath) - 1] == '/' {
+		m, exist = properties["/"+storage.storageDir+filePath[:len(filePath) - 1]]
+	}
+
+	if !exist {
+		return false, false, 0, nil
+	} else if resourceType, exist := m["resourcetype"]; exist && strings.Contains(resourceType, "collection") {
+		return true, true, 0, nil
+	} else if length, exist := m["getcontentlength"]; exist && length != "" {
+		value, _ := strconv.Atoi(length)
+		return true, false, int64(value), nil
+	} else {
+		return true, true, 0, nil
+	}
+}
+
+// DeleteFile deletes the file or directory at 'filePath'.
+func (storage *WebDAVStorage) DeleteFile(threadIndex int, filePath string) (err error) {
+	readCloser, _, err := storage.sendRequest("DELETE", filePath, 0, []byte(""))
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// MoveFile renames the file.
+func (storage *WebDAVStorage) MoveFile(threadIndex int, from string, to string) (err error) {
+	readCloser, _, err := storage.sendRequest("MOVE", from, 0, []byte(to))
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// createParentDirectory creates the parent directory if it doesn't exist in the cache
+func (storage *WebDAVStorage) createParentDirectory(threadIndex int, dir string) (err error) {
+
+	found := strings.LastIndex(dir, "/")
+	if found == -1 {
+		return nil
+	}
+	parent := dir[:found]
+
+	storage.directoryCacheLock.Lock()
+	_, exist := storage.directoryCache[parent]
+	storage.directoryCacheLock.Unlock()
+
+	if exist {
+		return nil
+	}
+
+	err = storage.CreateDirectory(threadIndex, parent)
+	if err == nil {
+		storage.directoryCacheLock.Lock()
+		storage.directoryCache[parent] = 1
+		storage.directoryCacheLock.Unlock()
+	}
+	return err
+}
+
+// CreateDirectory creates a new directory.
+func (storage *WebDAVStorage) CreateDirectory(threadIndex int, dir string) (err error) {
+	for dir != "" && dir[len(dir)-1] == '/' {
+		dir = dir[:len(dir)-1]
+	}
+
+	if dir == "" {
+		return nil
+	}
+
+	// If there is an error in creating the parent directory, proceed anyway
+	storage.createParentDirectory(threadIndex, dir)
+
+	readCloser, _, err := storage.sendRequest("MKCOL", dir, 0, []byte(""))
+	if err != nil {
+		if err == errWebDAVMethodNotAllowed || err == errWebDAVMovedPermanently {
+			// We simply ignore these errors and assume that the directory already exists
+			return nil
+		}
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// DownloadFile reads the file at 'filePath' into the chunk.
+func (storage *WebDAVStorage) DownloadFile(threadIndex int, filePath string, chunk *Chunk) (err error) {
+	readCloser, _, err := storage.sendRequest("GET", filePath, 0, nil)
+	if err != nil {
+		return err
+	}
+
+	_, err = RateLimitedCopy(chunk, readCloser, storage.DownloadRateLimit/storage.threads)
+	return err
+}
+
+// UploadFile writes 'content' to the file at 'filePath'.
+func (storage *WebDAVStorage) UploadFile(threadIndex int, filePath string, content []byte) (err error) {
+
+	// If there is an error in creating the parent directory, proceed anyway
+	storage.createParentDirectory(threadIndex, filePath)
+
+	readCloser, _, err := storage.sendRequest("PUT", filePath, 0, content)
+	if err != nil {
+		return err
+	}
+	readCloser.Close()
+	return nil
+}
+
+// If a local snapshot cache is needed for the storage to avoid downloading/uploading chunks too often when
+// managing snapshots.
+func (storage *WebDAVStorage) IsCacheNeeded() bool { return true }
+
+// If the 'MoveFile' method is implemented.
+func (storage *WebDAVStorage) IsMoveFileImplemented() bool { return true }
+
+// If the storage can guarantee strong consistency.
+func (storage *WebDAVStorage) IsStrongConsistent() bool { return false }
+
+// If the storage supports fast listing of files names.
+func (storage *WebDAVStorage) IsFastListing() bool { return false }
+
+// Enable the test mode.
+func (storage *WebDAVStorage) EnableTestMode() {}