jgaunt/duplicacy
1
0
Fork 0
mirror of https://github.com/gilbertchen/duplicacy synced 2025-12-06 00:03:38 +00:00
Code Issues Releases Wiki Activity

Merge pull request #244 from gilbertchen/pbkdf2_random_salt

Browse Source 
Fix security weakness in storage key derivation
This commit is contained in:
gilbertchen
2017-10-26 13:47:59 -04:00
committed by GitHub
parent b0b08cec4c 787c421a0c
commit 8ad981b64d
5 changed files with 137 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files

28
GUIDE.md
View File

@@ -13,10 +13,11 @@ USAGE:
OPTIONS:
-encrypt, -e encrypt the storage with a password
-chunk-size, -c 4M the average size of chunks
-max-chunk-size, -max 16M the maximum size of chunks (defaults to chunk-size * 4)
-min-chunk-size, -min 1M the minimum size of chunks (defaults to chunk-size / 4)
-pref-dir <preference directory path> Specify alternate location for .duplicacy preferences directory
-chunk-size, -c <size> the average size of chunks (default is 4M)
-max-chunk-size, -max <size> the maximum size of chunks (default is chunk-size*4)
-min-chunk-size, -min <size> the minimum size of chunks (default is chunk-size/4)
-iterations <i> the number of iterations used in storage key deriviation (default is 16384)
-pref-dir <path> alternate location for the .duplicacy directory (absolute or relative to current directory)
```
The *init* command first connects to the storage specified by the storage URL. If the storage has been already been initialized before, it will download the storage configuration (stored in the file named *config*) and ignore the options provided in the command line. Otherwise, it will create the configuration file from the options and upload the file.
@@ -31,7 +32,9 @@ The `-e` option controls whether or not encryption will be enabled for the stora
The three chunk size parameters are passed to the variable-size chunking algorithm. Their values are important to the overall performance, especially for cloud storages. If the chunk size is too small, a lot of overhead will be in sending requests and receiving responses. If the chunk size is too large, the effect of de-duplication will be less obvious as more data will need to be transferred with each chunk.
The `-pref-dir` controls the location of the preferences directory. If not specified, a directory named .duplicacy is created in the repository. If specified, it must point to a non-existing directory. The directory is created and a .duplicacy file is created in the repository. The .duplicacy file contains the absolute path name to the preferences directory.
The `-iterations` option speicifies how many iterations are used to generate the key that encrypts the `config` file from the storage password.
The `-pref-dir` option controls the location of the preferences directory. If not specified, a directory named .duplicacy is created in the repository. If specified, it must point to a non-existing directory. The directory is created and a .duplicacy file is created in the repository. The .duplicacy file contains the absolute path name to the preferences directory.
Once a storage has been initialized with these parameters, these parameters cannot be modified any more.
@@ -314,12 +317,15 @@ USAGE:
OPTIONS:
-storage <storage name> change the password used to access the specified storage
-iterations <i> the number of iterations used in storage key deriviation (default is 16384)
```
The *password* command decrypts the storage configuration file *config* using the old password, and re-encrypts the file
using a new password. It does not change all the encryption keys used to encrypt and decrypt chunk files,
snapshot files, etc.
The `-iterations` option speicifies how many iterations are used to generate the key that encrypts the `config` file from the storage password.
You can specify the storage to change the password for when working with multiple storages.
@@ -332,11 +338,11 @@ USAGE:
duplicacy add [command options] <storage name> <snapshot id> <storage url>
OPTIONS:
-encrypt, -e Encrypt the storage with a password
-chunk-size, -c 4M the average size of chunks
-max-chunk-size, -max 16M the maximum size of chunks (defaults to chunk-size * 4)
-min-chunk-size, -min 1M the minimum size of chunks (defaults to chunk-size / 4)
-compression-level, -l <level> compression level (defaults to -1)
-encrypt, -e encrypt the storage with a password
-chunk-size, -c <size> the average size of chunks (defaults to 4M)
-max-chunk-size, -max <size> the maximum size of chunks (defaults to chunk-size*4)
-min-chunk-size, -min <size> the minimum size of chunks (defaults to chunk-size/4)
-iterations <i> the number of iterations used in storage key deriviation (default is 16384)
-copy <storage name> make the new storage copy-compatible with an existing one
```
@@ -344,6 +350,8 @@ The *add* command connects another storage to the current repository. Like the
A unique storage name must be given in order to distinguish it from other storages.
The `-iterations` option speicifies how many iterations are used to generate the key that encrypts the `config` file from the storage password.
The `-copy` option is required if later you want to copy snapshots between this storage and another storage. Two storages are copy-compatible if they have the same average chunk size, the same maximum chunk size, the same minimum chunk size, the same chunk seed (used in calculating the rolling hash in the variable-size chunks algorithm), and the same hash key. If the `-copy` option is specified, these parameters will be copied from the existing storage rather than from the command line.
#### Set

58
duplicacy/duplicacy_main.go
View File

@@ -396,7 +396,11 @@ func configRepository(context *cli.Context, init bool) {
}
}
duplicacy.ConfigStorage(storage, compressionLevel, averageChunkSize, maximumChunkSize,
iterations := context.Int("iterations")
if iterations == 0 {
iterations = duplicacy.CONFIG_DEFAULT_ITERATIONS
}
duplicacy.ConfigStorage(storage, iterations, compressionLevel, averageChunkSize, maximumChunkSize,
minimumChunkSize, storagePassword, otherConfig)
}
@@ -576,7 +580,12 @@ func changePassword(context *cli.Context) {
return
}
duplicacy.UploadConfig(storage, config, newPassword)
iterations := context.Int("iterations")
if iterations == 0 {
iterations = duplicacy.CONFIG_DEFAULT_ITERATIONS
}
duplicacy.UploadConfig(storage, config, newPassword, iterations)
duplicacy.SavePassword(*preference, "password", newPassword)
@@ -1174,23 +1183,28 @@ func main() {
cli.StringFlag{
Name: "chunk-size, c",
Value: "4M",
Usage: "the average size of chunks",
Argument: "4M",
Usage: "the average size of chunks (defaults to 4M)",
Argument: "<size>",
},
cli.StringFlag{
Name: "max-chunk-size, max",
Usage: "the maximum size of chunks (defaults to chunk-size * 4)",
Argument: "16M",
Usage: "the maximum size of chunks (defaults to chunk-size*4)",
Argument: "<size>",
},
cli.StringFlag{
Name: "min-chunk-size, min",
Usage: "the minimum size of chunks (defaults to chunk-size / 4)",
Argument: "1M",
Usage: "the minimum size of chunks (defaults to chunk-size/4)",
Argument: "<size>",
},
cli.IntFlag{
Name: "iterations",
Usage: "the number of iterations used in storage key deriviation (default is 16384)",
Argument: "<i>",
},
cli.StringFlag{
Name: "pref-dir",
Usage: "Specify alternate location for .duplicacy preferences directory (absolute or relative to current directory)",
Argument: "<preferences directory path>",
Usage: "alternate location for the .duplicacy directory (absolute or relative to current directory)",
Argument: "<path>",
},
},
Usage: "Initialize the storage if necessary and the current directory as the repository",
@@ -1538,6 +1552,11 @@ func main() {
Usage: "change the password used to access the specified storage",
Argument: "<storage name>",
},
cli.IntFlag{
Name: "iterations",
Usage: "the number of iterations used in storage key deriviation (default is 16384)",
Argument: "<i>",
},
},
Usage: "Change the storage password",
ArgsUsage: " ",
@@ -1549,23 +1568,28 @@ func main() {
Flags: []cli.Flag{
cli.BoolFlag{
Name: "encrypt, e",
Usage: "Encrypt the storage with a password",
Usage: "encrypt the storage with a password",
},
cli.StringFlag{
Name: "chunk-size, c",
Value: "4M",
Usage: "the average size of chunks",
Argument: "4M",
Usage: "the average size of chunks (default is 4M)",
Argument: "<size>",
},
cli.StringFlag{
Name: "max-chunk-size, max",
Usage: "the maximum size of chunks (defaults to chunk-size * 4)",
Argument: "16M",
Usage: "the maximum size of chunks (default is chunk-size*4)",
Argument: "<size>",
},
cli.StringFlag{
Name: "min-chunk-size, min",
Usage: "the minimum size of chunks (defaults to chunk-size / 4)",
Argument: "1M",
Usage: "the minimum size of chunks (default is chunk-size/4)",
Argument: "<size>",
},
cli.IntFlag{
Name: "iterations",
Usage: "the number of iterations used in storage key deriviation (default is 16384)",
Argument: "<i>",
},
cli.StringFlag{
Name: "copy",

4
src/duplicacy_backupmanager_test.go
View File

@@ -227,11 +227,11 @@ func TestBackupManager(t *testing.T) {
time.Sleep(time.Duration(delay) * time.Second)
if testFixedChunkSize {
if !ConfigStorage(storage, 100, 64*1024, 64*1024, 64*1024, password, nil) {
if !ConfigStorage(storage, 16384, 100, 64*1024, 64*1024, 64*1024, password, nil) {
t.Errorf("Failed to initialize the storage")
}
} else {
if !ConfigStorage(storage, 100, 64*1024, 256*1024, 16*1024, password, nil) {
if !ConfigStorage(storage, 16384, 100, 64*1024, 256*1024, 16*1024, password, nil) {
t.Errorf("Failed to initialize the storage")
}
}

86
src/duplicacy_config.go
View File

@@ -9,6 +9,7 @@ import (
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/binary"
"encoding/hex"
"encoding/json"
"fmt"
@@ -28,6 +29,15 @@ var DEFAULT_KEY = []byte("duplicacy")
// standard zlib levels of -1 to 9.
var DEFAULT_COMPRESSION_LEVEL = 100
// The new header of the config file (to differentiate from the old format where the salt and iterations are fixed)
var CONFIG_HEADER = "duplicacy\001"
// The length of the salt used in the new format
var CONFIG_SALT_LENGTH = 32
// The default iterations for key derivation
var CONFIG_DEFAULT_ITERATIONS = 16384
type Config struct {
CompressionLevel int `json:"compression-level"`
AverageChunkSize int `json:"average-chunk-size"`
@@ -316,10 +326,45 @@ func DownloadConfig(storage Storage, password string) (config *Config, isEncrypt
return nil, false, err
}
if len(configFile.GetBytes()) < len(ENCRYPTION_HEADER) {
return nil, false, fmt.Errorf("The storage has an invalid config file")
}
if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)-1]) == ENCRYPTION_HEADER[:len(ENCRYPTION_HEADER)-1] && len(password) == 0 {
return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
}
var masterKey []byte
if len(password) > 0 {
masterKey = GenerateKeyFromPassword(password)
if string(configFile.GetBytes()[:len(ENCRYPTION_HEADER)]) == ENCRYPTION_HEADER {
// This is the old config format with a static salt and a fixed number of iterations
masterKey = GenerateKeyFromPassword(password, DEFAULT_KEY, CONFIG_DEFAULT_ITERATIONS)
LOG_TRACE("CONFIG_FORMAT", "Using a static salt and %d iterations for key derivation", CONFIG_DEFAULT_ITERATIONS)
} else if string(configFile.GetBytes()[:len(CONFIG_HEADER)]) == CONFIG_HEADER {
// This is the new config format with a random salt and a configurable number of iterations
encryptedLength := len(configFile.GetBytes()) - CONFIG_SALT_LENGTH - 4
// Extract the salt and the number of iterations
saltStart := configFile.GetBytes()[len(CONFIG_HEADER):]
iterations := binary.LittleEndian.Uint32(saltStart[CONFIG_SALT_LENGTH : CONFIG_SALT_LENGTH+4])
LOG_TRACE("CONFIG_ITERATIONS", "Using %d iterations for key derivation", iterations)
masterKey = GenerateKeyFromPassword(password, saltStart[:CONFIG_SALT_LENGTH], int(iterations))
// Copy to a temporary buffer to replace the header and remove the salt and the number of riterations
var encrypted bytes.Buffer
encrypted.Write([]byte(ENCRYPTION_HEADER))
encrypted.Write(saltStart[CONFIG_SALT_LENGTH+4:])
configFile.Reset(false)
configFile.Write(encrypted.Bytes())
if len(configFile.GetBytes()) != encryptedLength {
LOG_ERROR("CONFIG_DOWNLOAD", "Encrypted config has %d bytes instead of expected %d bytes", len(configFile.GetBytes()), encryptedLength)
}
} else {
return nil, true, fmt.Errorf("The config file has an invalid header")
}
// Decrypt the config file. masterKey == nil means no encryption.
err = configFile.Decrypt(masterKey, "")
@@ -331,23 +376,19 @@ func DownloadConfig(storage Storage, password string) (config *Config, isEncrypt
config = CreateConfig()
err = json.Unmarshal(configFile.GetBytes(), config)
if err != nil {
if bytes.Equal(configFile.GetBytes()[:9], []byte("duplicacy")) {
return nil, true, fmt.Errorf("The storage is likely to have been initialized with a password before")
} else {
return nil, false, fmt.Errorf("Failed to parse the config file: %v", err)
}
return nil, false, fmt.Errorf("Failed to parse the config file: %v", err)
}
return config, false, nil
}
func UploadConfig(storage Storage, config *Config, password string) bool {
func UploadConfig(storage Storage, config *Config, password string, iterations int) bool {
// This is the key to encrypt the config file.
var masterKey []byte
salt := make([]byte, CONFIG_SALT_LENGTH)
if len(password) > 0 {
@@ -356,7 +397,13 @@ func UploadConfig(storage Storage, config *Config, password string) bool {
return false
}
masterKey = GenerateKeyFromPassword(password)
_, err := rand.Read(salt)
if err != nil {
LOG_ERROR("CONFIG_KEY", "Failed to generate random salt: %v", err)
return false
}
masterKey = GenerateKeyFromPassword(password, salt, iterations)
}
description, err := json.MarshalIndent(config, "", " ")
@@ -373,11 +420,26 @@ func UploadConfig(storage Storage, config *Config, password string) bool {
if len(password) > 0 {
// Encrypt the config file with masterKey. If masterKey is nil then no encryption is performed.
err = chunk.Encrypt(masterKey, "")
if err != nil {
LOG_ERROR("CONFIG_CREATE", "Failed to create the config file: %v", err)
return false
}
// The new encrypted format for config is CONFIG_HEADER + salt + #iterations + encrypted content
encryptedLength := len(chunk.GetBytes()) + CONFIG_SALT_LENGTH + 4
// Copy to a temporary buffer to replace the header and add the salt and the number of iterations
var encrypted bytes.Buffer
encrypted.Write([]byte(CONFIG_HEADER))
encrypted.Write(salt)
binary.Write(&encrypted, binary.LittleEndian, uint32(iterations))
encrypted.Write(chunk.GetBytes()[len(ENCRYPTION_HEADER):])
chunk.Reset(false)
chunk.Write(encrypted.Bytes())
if len(chunk.GetBytes()) != encryptedLength {
LOG_ERROR("CONFIG_CREATE", "Encrypted config has %d bytes instead of expected %d bytes", len(chunk.GetBytes()), encryptedLength)
}
}
err = storage.UploadFile(0, "config", chunk.GetBytes())
@@ -403,7 +465,7 @@ func UploadConfig(storage Storage, config *Config, password string) bool {
// ConfigStorage makes the general storage space available for storing duplicacy format snapshots. In essence,
// it simply creates a file named 'config' that stores various parameters as well as a set of keys if encryption
// is enabled.
func ConfigStorage(storage Storage, compressionLevel int, averageChunkSize int, maximumChunkSize int,
func ConfigStorage(storage Storage, iterations int, compressionLevel int, averageChunkSize int, maximumChunkSize int,
minimumChunkSize int, password string, copyFrom *Config) bool {
exist, _, _, err := storage.GetFileInfo(0, "config")
@@ -423,5 +485,5 @@ func ConfigStorage(storage Storage, compressionLevel int, averageChunkSize int,
return false
}
return UploadConfig(storage, config, password)
return UploadConfig(storage, config, password, iterations)
}

4
src/duplicacy_utils.go
View File

@@ -159,8 +159,8 @@ func RateLimitedCopy(writer io.Writer, reader io.Reader, rate int) (written int6
}
// GenerateKeyFromPassword generates a key from the password.
func GenerateKeyFromPassword(password string) []byte {
return pbkdf2.Key([]byte(password), DEFAULT_KEY, 16384, 32, sha256.New)
func GenerateKeyFromPassword(password string, salt []byte, iterations int) []byte {
return pbkdf2.Key([]byte(password), salt, iterations, 32, sha256.New)
}
// Get password from preference, env, but don't start any keyring request