1
0
mirror of https://github.com/gilbertchen/duplicacy synced 2025-12-06 00:03:38 +00:00

Compare commits

...

24 Commits

Author SHA1 Message Date
gilbertchen
1d12fa3dd8 Update GUIDE.md 2017-01-29 10:42:42 -05:00
gilbertchen
6c793f25ee Update README.md 2016-11-14 21:48:06 -05:00
gilbertchen
901605ca68 Update README.md 2016-11-14 21:47:41 -05:00
gilbertchen
1f83a6e793 Update DESIGN.md 2016-11-12 23:14:28 -05:00
gilbertchen
117317af3f Update GUIDE.md 2016-10-04 19:25:47 -04:00
gilbertchen
77dbabf5d3 Update README.md 2016-10-04 14:33:51 -04:00
gilbertchen
2fde44c2ec Update README.md 2016-10-04 14:25:57 -04:00
gilbertchen
6828843dfc Update README.md 2016-10-02 12:40:01 -04:00
gilbertchen
895a785765 Update README.md 2016-10-02 11:01:10 -04:00
gilbertchen
f71570728a Update GUIDE.md 2016-09-26 22:56:57 -04:00
gilbertchen
405cad1d7b Update README.md 2016-09-26 22:54:32 -04:00
gilbertchen
1be6a78cc0 Update README.md 2016-09-26 08:24:51 -04:00
gilbertchen
6ebc2394e5 Update README.md 2016-09-20 20:05:40 -04:00
gilbertchen
79c440f9fe Update README.md 2016-09-20 19:47:41 -04:00
gilbertchen
b54558e6fe Update README.md 2016-09-20 19:26:16 -04:00
gilbertchen
0d8d691664 Update DESIGN.md 2016-08-03 21:19:47 -04:00
gilbertchen
7abcd5b45e Update DESIGN.md 2016-08-02 20:20:07 -04:00
gilbertchen
1d4979cde4 Update README.md 2016-07-14 14:31:18 -04:00
gilbertchen
79ccd78a3b Update README.md 2016-07-14 13:47:39 -04:00
gilbertchen
51a99d9a2c Update README.md 2016-07-14 13:47:26 -04:00
gilbertchen
2b8e9a1f11 Update README.md 2016-07-14 11:31:33 -04:00
gilbertchen
8c76154a21 Update README.md 2016-07-14 11:30:39 -04:00
gilbertchen
3a46779a58 Update GUIDE.md 2016-07-14 11:28:49 -04:00
gilbertchen
f0ff4a3ec1 Update README.md 2016-05-18 15:11:30 -04:00
3 changed files with 53 additions and 10 deletions

View File

@@ -137,7 +137,7 @@ hasn't been found, the next file, if there is one, will be read in and the chunk
files were packed into a big tar file which is then split into chunks.
The *content* field of a file indicates the indexes of starting and ending chunks and the corresponding offsets. For
instance, *fiel1* starts at chunk 0 offset 0 while ends at chunk 2 offset 6108, immediately followed by *file2*.
instance, *file1* starts at chunk 0 offset 0 while ends at chunk 2 offset 6108, immediately followed by *file2*.
The backup procedure can run in one of two modes. In the default quick mode, only modified or new files are scanned. Chunks only
referenced by old files that have been modified are removed from the chunk sequence, and then chunks referenced by new
@@ -210,6 +210,6 @@ Chunk content is encrypted by AES-GCM, with an encryption key that is the HMAC-S
The snapshot is encrypted by AES-GCM too, using an encrypt key that is the HMAC-SHA256 of the file path with the *File Key* as the secret key.
There four random keys are saved in a file named 'config' in the file storage, encrypted with a master key derived from the PBKDF2 function on
These four random keys are saved in a file named 'config' in the storage, encrypted with a master key derived from the PBKDF2 function on
the storage password chosen by the user.

View File

@@ -160,6 +160,7 @@ OPTIONS:
-fossils search fossils if a chunk can't be found
-resurrect turn referenced fossils back into chunks
-files verify the integrity of every file
-stats show deduplication statistics (imply -all and all revisions)
-storage <storage name> retrieve snapshots from the specified storage```
```
The *check* command checks, for each specified snapshot, that all referenced chunks exist in the storage.
@@ -465,12 +466,16 @@ Duplicacy will attempt to retrieve in three ways the storage password and the st
|:----------------:|:----------------:|:----------------:|:----------------:|
| storage password | DUPLICACY_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_PASSWORD | password |
| sftp password | DUPLICACY_SSH_PASSWORD | DUPLICACY_&lt;STORAGENAME&gt;_SSH_PASSWORD | ssh_password |
| sftp key file | DUPLICACY_SSH_KEY_FILE | DUPLICACY_&lt;STORAGENAME&gt;_SSH_KEY_FILE | ssh_keyfile |
| Dropbox Token | DUPLICACY_DROPBOX_TOKEN | DUPLICACY_&lt;STORAGENAME>&gt;_DROPBOX_TOKEN | dropbox_token |
| S3 Access ID | DUPLICACY_S3_ID | DUPLICACY_&lt;STORAGENAME&gt;_S3_ID | s3_id |
| S3 Secret Key | DUPLICACY_S3_SECRET | DUPLICACY_&lt;STORAGENAME&gt;_S3_SECRET | s3_secret |
| BackBlaze Account ID | DUPLICACY_B2_ID | DUPLICACY_&lt;STORAGENAME&gt;_B2_ID | b2_id |
| Backblaze Application Key | DUPLICACY_B2_KEY | DUPLICACY_&lt;STORAGENAME&gt;_B2_KEY | b2_key |
| Azure Access Key | DUPLICACY_AZURE_KEY | DUPLICACY_&lt;STORAGENAME&gt;_AZURE_KEY | azure_key |
| Google Drive Token File | DUPLICACY_GCD_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_GCD_TOKEN | gcd_token |
| Microsoft OneDrive Token File | DUPLICACY_ONE_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_ONE_TOKEN | one_token |
| Hubic Token File | DUPLICACY_HUBIC_TOKEN | DUPLICACY_&lt;STORAGENAME&gt;_HUBIC_TOKEN | hubic_token |
Note that the passwords stored in the environment variable and the preference need to be in plaintext and thus are insecure and should be avoided whenever possible.

View File

@@ -1,10 +1,12 @@
# Duplicacy: A new generation cloud backup tool
# Duplicacy: A lock-free deduplication cloud backup tool
This repository contains only binary releases and documentation for Duplicacy. It also serves as an issue tracker for user-developer communication during the beta testing phase.
Duplicacy is a new generation cross-platform cloud backup tool based on the idea of [Lock-Free Deduplication](https://github.com/gilbertchen/duplicacy-beta/blob/master/DESIGN.md). It is the only cloud backup tool that allows multiple computers to back up to the same storage simultaneously without using any locks (thus readily amenable to various cloud storage services).
## Overview
The repository hosts design documents as well as binary releases of the command line version. There is also a Duplicacy GUI frontend built for Windows and Mac OS X downloadable from https://duplicacy.com. The source code of the command line version is available to the commercial users of the Duplicacy GUI version upon request.
Duplicacy supports major cloud storage providers (Amazon S3, Google Cloud Storage, Microsoft Azure, Dropbox, and BackBlaze) and offers all essential features of a modern backup tool:
## Features
Duplicacy currently supports major cloud storage providers (Amazon S3, Google Cloud Storage, Microsoft Azure, Dropbox, Backblaze, Google Drive, Microsoft OneDrive, and Hubic) and offers all essential features of a modern backup tool:
* Incremental backup: only back up what has been changed
* Full snapshot : although each backup is incremental, it must behave like a full snapshot for easy restore and deletion
@@ -14,7 +16,7 @@ Duplicacy supports major cloud storage providers (Amazon S3, Google Cloud Storag
* Concurrent access: multiple clients can back up to the same storage at the same time
* Snapshot migration: all or selected snapshots can be migrated from one storage to another
The key idea behind Duplicacy is a concept called **Lock-Free Deduplication**, which can be summarized as follows:
The key idea of **Lock-Free Deduplication** can be summarized as follows:
* Use variable-size chunking algorithm to split files into chunks
* Store each chunk in the storage using a file name derived from its hash, and rely on the file system API to manage chunks without using a centralized indexing database
@@ -153,7 +155,7 @@ Storage URL: azure://account/container
You'll need to input the access key once prompted.
#### BackBlaze
#### Backblaze
```
Storage URL: b2://bucket
@@ -161,7 +163,37 @@ Storage URL: b2://bucket
You'll need to input the account id and application key.
BackBlaze offers perhaps the least expensive cloud storage at 0.5 cent per GB per month. Unfortunately their API does not support file renaming, so the -exclusive option is required when pruning old backups. This means concurrent access and deletion can't be permitted at the same time.
Backblaze's B2 storage is not only the least expensive (at 0.5 cent per GB per month), but also the fastest. We have been working closely with their developers to leverage the full potentials provided by the B2 API in order to maximumize the transfer speed. As a result, the B2 storage is the only one to support the multi-threading option which can easily max out your upload link.
#### Google Drive
```
Storage URL: gcd://path/to/storage
```
To use Google Drive as the storage, you first need to download a token file from https://duplicacy.com/gcd_start by
authorizing Duplicacy to access your Google Drive, and then enter the path to this token file to Duplicacy when prompted.
#### Microsoft OneDrive
```
Storage URL: one://path/to/storage
```
To use Microsoft OneDrive as the storage, you first need to download a token file from https://duplicacy.com/one_start by
authorizing Duplicacy to access your OneDrive, and then enter the path to this token file to Duplicacy when prompted.
#### Hubic
```
Storage URL: hubic://path/to/storage
```
To use Hubic as the storage, you first need to download a token file from https://duplicacy.com/hubic_start by
authorizing Duplicacy to access your Hubic drive, and then enter the path to this token file to Duplicacy when prompted.
Hubic offers the most free space (25GB) of all major cloud providers and there is no bandwidth charge (same as Google Drive and OneDrive), so it may be worth a try.
## Comparison with Other Backup Tools
@@ -195,6 +227,12 @@ The following table compares the feature lists of all these backup tools:
| Encryption | Yes | Yes | Yes | Yes | Yes | **Yes** |
| Deletion | No | No | Yes | Yes | No | **Yes** |
| Concurrent Access | No | No | Exclusive locking | Not recommended | Exclusive locking | **Lock-free** |
| Cloud Support | Extensive | No | No | No | S3 only | **S3, GCS, Azure, Dropbox, BackBlaze**|
| Cloud Support | Extensive | No | No | No | S3 only | **S3, GCS, Azure, Dropbox, Backblaze, Google Drive, OneDrive, and Hubic**|
| Snapshot Migration | No | No | No | No | No | **Yes** |
##License
Duplicacy CLI is free for personal use without restrictions.
For commercial use, a valid [commercial license](https://duplicacy.com/buy.html) is required for each computer on which backups will be created. There are no restrictions if Duplicacy CLI is used to restore files from backups or check the integrity of backups.