From 74ab8d8c2314f01bd8c7884b0240667bb5315480 Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:38:37 -0400 Subject: [PATCH 01/13] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 79577b0..4a14671 100644 --- a/README.md +++ b/README.md @@ -109,12 +109,16 @@ all commands and other features of Duplicacy. Duplicacy currently supports local file storage, SFTP, and 5 cloud storage providers. +
+ Local disk + #### Local disk ``` Storage URL: /path/to/storage (on Linux or Mac OS X) C:\path\to\storage (on Windows) ``` +
#### SFTP From b56baa80c38a003c9267b1a2c3c49b9fef19bf3d Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:42:07 -0400 Subject: [PATCH 02/13] Update README.md --- README.md | 40 ++++++++++++++++++++++++++-------------- 1 file changed, 26 insertions(+), 14 deletions(-) diff --git a/README.md b/README.md index 4a14671..5d98c45 100644 --- a/README.md +++ b/README.md @@ -109,18 +109,14 @@ all commands and other features of Duplicacy. Duplicacy currently supports local file storage, SFTP, and 5 cloud storage providers. -
- Local disk - -#### Local disk - +
Local disk ``` Storage URL: /path/to/storage (on Linux or Mac OS X) C:\path\to\storage (on Windows) ```
-#### SFTP +
SFTP + + Dropbox ``` Storage URL: dropbox://path/to/storage @@ -142,7 +140,9 @@ For Duplicacy to access your Dropbox storage, you must provide an access token t Dropbox has two advantages over other cloud providers. First, if you are already a paid user then to use the unused space as the backup storage is basically free. Second, unlike other providers Dropbox does not charge bandwidth or API usage fees. -#### Amazon S3 + + + Amazon S3 ``` Storage URL: s3://amazon.com/bucket/path/to/storage (default region is us-east-1) @@ -152,7 +152,9 @@ Storage URL: s3://amazon.com/bucket/path/to/storage (default region is us-east- You'll need to input an access key and a secret key to access your Amazon S3 storage. -#### Google Cloud Storage + + + Google Cloud Storage ``` Storage URL: gcs://bucket/path/to/storage @@ -162,7 +164,9 @@ Starting from version 2.0.0, a new Google Cloud Storage backend is added which i You can also use the s3 protocol to access Google Cloud Storage. To do this, you must enable the [s3 interoperability](https://cloud.google.com/storage/docs/migrating#migration-simple) in your Google Cloud Storage settings and set the storage url as `s3://storage.googleapis.com/bucket/path/to/storage`. -#### Microsoft Azure + + + Microsoft Azure ``` Storage URL: azure://account/container @@ -170,7 +174,9 @@ Storage URL: azure://account/container You'll need to input the access key once prompted. -#### Backblaze + + + Backblaze B2 ``` Storage URL: b2://bucket @@ -180,7 +186,9 @@ You'll need to input the account id and application key. Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per month), but also the fastest. We have been working closely with their developers to leverage the full potentials provided by the B2 API in order to maximize the transfer speed. -#### Google Drive + + + Google Drive ``` Storage URL: gcd://path/to/storage @@ -189,7 +197,9 @@ Storage URL: gcd://path/to/storage To use Google Drive as the storage, you first need to download a token file from https://duplicacy.com/gcd_start by authorizing Duplicacy to access your Google Drive, and then enter the path to this token file to Duplicacy when prompted. -#### Microsoft OneDrive + + + Microsoft OneDrive ``` Storage URL: one://path/to/storage @@ -198,7 +208,9 @@ Storage URL: one://path/to/storage To use Microsoft OneDrive as the storage, you first need to download a token file from https://duplicacy.com/one_start by authorizing Duplicacy to access your OneDrive, and then enter the path to this token file to Duplicacy when prompted. -#### Hubic + + + Hubic ``` Storage URL: hubic://path/to/storage From 5334f459987db8a9ebf8de492e336d163298675b Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:43:07 -0400 Subject: [PATCH 03/13] Update README.md --- README.md | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/README.md b/README.md index 5d98c45..e35a0ed 100644 --- a/README.md +++ b/README.md @@ -116,7 +116,7 @@ Storage URL: /path/to/storage (on Linux or Mac OS X) ``` - SFTP SFTP - Dropbox +
Dropbox ``` Storage URL: dropbox://path/to/storage @@ -142,7 +142,7 @@ Dropbox has two advantages over other cloud providers. First, if you are alread
- Amazon S3 +
Amazon S3 ``` Storage URL: s3://amazon.com/bucket/path/to/storage (default region is us-east-1) @@ -154,7 +154,7 @@ You'll need to input an access key and a secret key to access your Amazon S3 sto
- Google Cloud Storage +
Google Cloud Storage ``` Storage URL: gcs://bucket/path/to/storage @@ -166,7 +166,7 @@ You can also use the s3 protocol to access Google Cloud Storage. To do this, yo
- Microsoft Azure +
Microsoft Azure ``` Storage URL: azure://account/container @@ -176,7 +176,7 @@ You'll need to input the access key once prompted.
- Backblaze B2 +
Backblaze B2 ``` Storage URL: b2://bucket @@ -188,7 +188,7 @@ Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per m
- Google Drive +
Google Drive ``` Storage URL: gcd://path/to/storage @@ -199,7 +199,7 @@ authorizing Duplicacy to access your Google Drive, and then enter the path to th
- Microsoft OneDrive +
Microsoft OneDrive ``` Storage URL: one://path/to/storage @@ -210,7 +210,7 @@ authorizing Duplicacy to access your OneDrive, and then enter the path to this t
- Hubic +
Hubic ``` Storage URL: hubic://path/to/storage From 593b409329d75d2358260c55b6896accadd480b9 Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:44:20 -0400 Subject: [PATCH 04/13] Update README.md --- README.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index e35a0ed..c9893a7 100644 --- a/README.md +++ b/README.md @@ -110,13 +110,14 @@ all commands and other features of Duplicacy. Duplicacy currently supports local file storage, SFTP, and 5 cloud storage providers.
Local disk + ``` Storage URL: /path/to/storage (on Linux or Mac OS X) C:\path\to\storage (on Windows) ```
-
SFTP SFTP ``` Storage URL: sftp://username@server/path/to/storage @@ -221,6 +222,7 @@ authorizing Duplicacy to access your Hubic drive, and then enter the path to thi Hubic offers the most free space (25GB) of all major cloud providers and there is no bandwidth charge (same as Google Drive and OneDrive), so it may be worth a try. +
## Feature Comparison with Other Backup Tools From 96e2f7809682d6454f034133564a7fafd1409afd Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:45:59 -0400 Subject: [PATCH 05/13] Update README.md --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index c9893a7..890fbcb 100644 --- a/README.md +++ b/README.md @@ -26,7 +26,8 @@ The key idea of **Lock-Free Deduplication** can be summarized as follows: The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md) explains lock-free deduplication in detail. -## Getting Started +
## Getting Started + Duplicacy is written in Go. You can run the following command to build the executable (which will be created under `$GOPATH/bin`): @@ -105,6 +106,8 @@ $ duplicacy copy -to s3 # Copy every snapshot to the s3 storage The [User Guide](https://github.com/gilbertchen/duplicacy-cli/blob/master/GUIDE.md) contains a complete reference to all commands and other features of Duplicacy. +
+ ## Storages Duplicacy currently supports local file storage, SFTP, and 5 cloud storage providers. From 83da36cae045c5a33ec081c1eb19f0b8f8f0fe6f Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:54:03 -0400 Subject: [PATCH 06/13] Update README.md --- README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 890fbcb..a5903af 100644 --- a/README.md +++ b/README.md @@ -26,8 +26,7 @@ The key idea of **Lock-Free Deduplication** can be summarized as follows: The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/DESIGN.md) explains lock-free deduplication in detail. -
## Getting Started - +## Getting Started Duplicacy is written in Go. You can run the following command to build the executable (which will be created under `$GOPATH/bin`): @@ -52,7 +51,13 @@ You can now create snapshots of the repository by invoking the *backup* command. $ duplicacy backup -stats ``` -Duplicacy provides a set of commands, such as list, check, diff, cat history, to manage snapshots: +The *restore* command rolls back the repository to a previous revision: +```sh +$ duplicacy restore -r 1 +``` + + +
```makefile $ duplicacy list # List all snapshots @@ -62,10 +67,6 @@ $ duplicacy cat # Print a file in a snapshot $ duplicacy history # Show how a file changes over time ``` -The *restore* command rolls back the repository to a previous revision: -```sh -$ duplicacy restore -r 1 -``` The *prune* command removes snapshots by revisions, or tags, or retention policies: @@ -103,10 +104,11 @@ $ duplicacy copy -r 1 -to s3 # Copy snapshot at revision 1 to the s3 storage $ duplicacy copy -to s3 # Copy every snapshot to the s3 storage ``` +
+ The [User Guide](https://github.com/gilbertchen/duplicacy-cli/blob/master/GUIDE.md) contains a complete reference to all commands and other features of Duplicacy. -
## Storages From c31d2a30d9615e0cc9ae912c45cd9a9453a57d42 Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:54:36 -0400 Subject: [PATCH 07/13] Update README.md --- README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index a5903af..8365b6f 100644 --- a/README.md +++ b/README.md @@ -57,7 +57,10 @@ $ duplicacy restore -r 1 ``` -
+
+ +Duplicacy provides a set of commands, such as list, check, diff, cat history, to manage snapshots: + ```makefile $ duplicacy list # List all snapshots From ac655c8780673635889fd0872222b32f1785ada6 Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:57:40 -0400 Subject: [PATCH 08/13] Update README.md --- README.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/README.md b/README.md index 8365b6f..23046a3 100644 --- a/README.md +++ b/README.md @@ -28,6 +28,7 @@ The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/D ## Getting Started +
Installation Duplicacy is written in Go. You can run the following command to build the executable (which will be created under `$GOPATH/bin`): ``` @@ -36,6 +37,10 @@ go get -u github.com/gilbertchen/duplicacy/... You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-cli/releases/latest) to download the pre-built binary suitable for your platform.. +
+ +
Commands + Once you have the Duplicacy executable on your path, you can change to the directory that you want to back up (called *repository*) and run the *init* command: ``` @@ -57,10 +62,9 @@ $ duplicacy restore -r 1 ``` -
- + Duplicacy provides a set of commands, such as list, check, diff, cat history, to manage snapshots: - + ```makefile $ duplicacy list # List all snapshots @@ -109,6 +113,7 @@ $ duplicacy copy -to s3 # Copy every snapshot to the s3 storage
+ The [User Guide](https://github.com/gilbertchen/duplicacy-cli/blob/master/GUIDE.md) contains a complete reference to all commands and other features of Duplicacy. From f2451911f2a667b194d793be88a15edf5ec0af8f Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:58:10 -0400 Subject: [PATCH 09/13] Update README.md --- README.md | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 23046a3..d016d78 100644 --- a/README.md +++ b/README.md @@ -28,7 +28,9 @@ The [design document](https://github.com/gilbertchen/duplicacy-cli/blob/master/D ## Getting Started -
Installation +
+Installation + Duplicacy is written in Go. You can run the following command to build the executable (which will be created under `$GOPATH/bin`): ``` @@ -39,7 +41,8 @@ You can also visit the [releases page](https://github.com/gilbertchen/duplicacy-
-
Commands +
+Commands Once you have the Duplicacy executable on your path, you can change to the directory that you want to back up (called *repository*) and run the *init* command: From 7d3119963128d66cbf40964f1bb6c0da5a58070f Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 14:58:43 -0400 Subject: [PATCH 10/13] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index d016d78..781b386 100644 --- a/README.md +++ b/README.md @@ -123,7 +123,7 @@ all commands and other features of Duplicacy. ## Storages -Duplicacy currently supports local file storage, SFTP, and 5 cloud storage providers. +Duplicacy currently supports local file storage, SFTP, and many cloud storage providers.
Local disk From 93a61a6e4990a085cb818a222c859812cf8e254b Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 15:01:10 -0400 Subject: [PATCH 11/13] Update README.md --- README.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/README.md b/README.md index 781b386..448815e 100644 --- a/README.md +++ b/README.md @@ -280,6 +280,8 @@ The following table compares the feature lists of all these backup tools: Duplicacy is not only more feature-rich but also faster than other backup tools. The following table lists the running times in seconds of backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other tools. Clearly Duplicacy is the faster by a comfortable margin. +
+ | | Duplicacy | restic | Attic | duplicity | |:------------------:|:----------------:|:----------:|:----------:|:-----------:| | Initial backup | 13.7 | 20.7 | 26.9 | 44.2 | @@ -295,6 +297,8 @@ Duplicacy is not only more feature-rich but also faster than other backup tools. | 11th backup | 4.6 | 9.1 | 16.8 | 28.1 | | 12th backup | 7.4 | 12.0 | 21.7 | 37.4 | +
+ For more details and other speed comparison results, please visit https://github.com/gilbertchen/backup-tools-comparison. There you can also find test scripts that allow you to run your own experiments. ## License From f9ede565ff1780bbee5f88478bc614f6b5abf12e Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 15:02:08 -0400 Subject: [PATCH 12/13] Update README.md --- README.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/README.md b/README.md index 448815e..ae145c3 100644 --- a/README.md +++ b/README.md @@ -280,7 +280,6 @@ The following table compares the feature lists of all these backup tools: Duplicacy is not only more feature-rich but also faster than other backup tools. The following table lists the running times in seconds of backing up the [Linux code base](https://github.com/torvalds/linux) using Duplicacy and 3 other tools. Clearly Duplicacy is the faster by a comfortable margin. -
| | Duplicacy | restic | Attic | duplicity | |:------------------:|:----------------:|:----------:|:----------:|:-----------:| @@ -297,7 +296,6 @@ Duplicacy is not only more feature-rich but also faster than other backup tools. | 11th backup | 4.6 | 9.1 | 16.8 | 28.1 | | 12th backup | 7.4 | 12.0 | 21.7 | 37.4 | -
For more details and other speed comparison results, please visit https://github.com/gilbertchen/backup-tools-comparison. There you can also find test scripts that allow you to run your own experiments. From 88141216e970da0272df687246a7525e46ca598e Mon Sep 17 00:00:00 2001 From: gilbertchen Date: Fri, 7 Jul 2017 15:32:03 -0400 Subject: [PATCH 13/13] Update GUIDE.md --- GUIDE.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/GUIDE.md b/GUIDE.md index 471270a..640c5c2 100644 --- a/GUIDE.md +++ b/GUIDE.md @@ -500,6 +500,16 @@ Duplicacy will attempt to retrieve in three ways the storage password and the st Note that the passwords stored in the environment variable and the preference need to be in plaintext and thus are insecure and should be avoided whenever possible. +## Cache + +Duplicacy maintains a local cache under the `.duplicacy/cache` folder in the repository. Only snapshot chunks may be stored in this local cache, and file chunks are never cached. + +At the end of a backup operation, Duplicacy will clean up the local cache in such a way that only chunks composing the snapshot file from the last backup will stay in the cache. All other chunks will be removed from the cache. However, if the *prune* command has been run before (which will leave a the `.duplicacy/collection` folder in the repository, then the *backup* command won't perform any cache cleanup and instead defer that to the *prune* command. + +At the end of a prune operation, Duplicacy will remove all chunks from the local cache except those composing the snapshot file from the last backup (those that would be kept by the *backup* command), as well as chunks that contain information about chunks referenced by *all* backups from *all* repositories connected to the same storage url. + +Other commands, such as *list*, *check*, does not clean up the local cache at all, so the local cache may keep growing if many of these commands run consectively. However, once a *backup* or a *prune* command is invoked, the local cache should shrink to its normal size. + ## Scripts You can instruct Duplicacy to run a script before or after executing a command. For example, if you create a bash script with the name *pre-prune* under the *.duplicacy/scripts* directory, this bash script will be run before the *prune* command starts. A script named *post-prune* will be run after the *prune* command finishes. This rule applies to all commands except *init*.