1
0
mirror of https://github.com/gilbertchen/duplicacy synced 2025-12-15 15:53:26 +00:00
Commit Graph

34 Commits

Author SHA1 Message Date
Gilbert Chen
fede9c74b5 Merge branch 'master' into memory_optimization 2022-04-07 23:26:14 -04:00
Gilbert Chen
a953c4ec28 Don't parse test parameters in init()
This is to make test parameter parsing work with newer versions
of Go
2022-04-07 22:31:18 -04:00
Gilbert Chen
d9f6545d63 Rewrite the backup procedure to reduce memory usage
Main changes:

* Change the listing order of files/directories so that the local and remote
  snapshots can be compared on-the-fly.

* Introduce a new struct called EntryList that maintains a list of
  files/directories, which are kept in memory when the number is lower, and
  serialized into a file when there are too many.

* EntryList can also be turned into an on-disk incomplete snapshot quickly,
  to support fast-resume on next run.

* ChunkOperator can now download and upload chunks, thus replacing original
  ChunkDownloader and ChunkUploader.  The new ChunkDownloader is only used
  to prefetch chunks during the restore operation.
2021-10-24 23:34:49 -04:00
Gilbert Chen
6ad698328f Fixed test build errors caused by previous merges 2020-09-26 12:01:38 -04:00
gilbertchen
1fedfd1b1a Merge branch 'master' into mac-exclude 2020-09-25 20:13:43 -04:00
gilbertchen
3fd3f6b267 Merge pull request #606 from gilbertchen/erasure_coding
Implement Erasure Coding
2020-09-25 14:30:39 -04:00
Gilbert Chen
73ae3f809e Revert "Add a -max-list-rate option to backup to slow down the listing"
This reverts commit 67a3103467.
2020-09-22 22:08:43 -04:00
Gilbert Chen
67a3103467 Add a -max-list-rate option to backup to slow down the listing
This option sets the maximum number of files that can be listed in one
second.
2020-09-22 08:27:09 -04:00
Gilbert Chen
16d2c14c5a Follow-up changes for the -persist PR
* Restore/check should report an error instead of a success at the end if there
  were any errors and -persist is specified
* Don't compute the file hash before passing the file to the chunk maker; this is
  redundant as the chunk maker will produce the file hash
* Add a LOG_WERROR function to switch between LOG_WARN and LOG_ERROR dynamically
2020-09-18 11:23:35 -04:00
Gilbert Chen
923a6fbc5b Implement Erasure Coding 2020-09-03 12:54:48 -04:00
Tet Woo Lee
4ae16dec7f add -persist in check and restore mode (for PR) 2020-05-06 18:39:52 +12:00
Gilbert Chen
0e970da222 Fixed build errors in tests caused by snapshotManager.CheckSnapshots 2020-04-06 12:19:42 -04:00
Gilbert Chen
a99f059b52 Allow a custom location for the filters file
You can now add a key 'filters' in the preferences file that points to the
path of the filters file.  If this key is not found in the preferences,
the default location '.duplicacy/filters' is used.

There is a new option '-filters' for the set command that set this key in
the preferences, but you can also edit the file directly.
2019-11-23 15:23:26 -05:00
Gilbert Chen
f022a6f684 Fixed build errors in tests 2019-11-22 21:17:17 -05:00
Michael Cook
0762c448c4 gofmt -s 2018-12-29 13:20:10 +01:00
Patrick Seal
a1efbe3b73 Add exclude_by_attribute preference 2018-09-21 21:35:40 -07:00
Gilbert Chen
dd53b4797e Implement multi-threaded pruning 2018-06-04 21:52:07 -04:00
Gilbert Chen
aadd2aa390 Add an -enum-only option to the backup command to enumerate the repository only 2018-05-24 11:34:46 -04:00
Frank
4dd5c43307 Add nobackup-file preference.
Directories containing a file with this name will not be backed up. I find it easier to drop a .nobackup file in directories I don't want backed up instead of maintaining a file of exclusions. This is also useful for scripts that create data in the repository but don't want it to be backed up.
2018-04-01 12:50:00 -07:00
Gilbert Chen
be2856ebbd Add a -vss-timeout option to set VSS creation timeout 2018-03-21 22:34:14 -04:00
Gilbert Chen
4aa2edb164 Fixed a test build error caused by the new bit-identical argument 2017-11-21 22:22:17 -05:00
Gilbert Chen
34e49d4589 Fixed test build errors due to a function prototype change 2017-11-21 12:24:55 -05:00
Gilbert Chen
86767b3df6 Implement new chunk directory structure 2017-11-07 12:05:39 -05:00
Gilbert Chen
9608a7f6b6 Use random salt and make the number of iterations configurable for storage key derivation 2017-10-20 23:21:26 -04:00
Gilbert Chen
923cd0aa63 Run goimports on all source files 2017-09-20 23:07:43 -04:00
Gilbert Chen
22ddc04698 Restore empty directories 2017-08-05 10:56:15 -04:00
Gilbert Chen
823b82060c Add a storage prefix flat:// that can handle a flat chunk directory 2017-07-27 22:42:48 -04:00
Gilbert Chen
9be4927c87 Set preference path before backup and restore 2017-07-19 15:15:19 -04:00
Gilbert Chen
7f04a79111 Replace Fair Source 5 with our own free-for-personal-use license 2017-07-13 23:33:14 -04:00
Gilbert Chen
d0c376f593 Implement fast resume; refactor GetDuplicacyPreferencePath() 2017-06-22 22:53:33 -04:00
Gilbert Chen
dd11641611 Fixed a bug that caused restoration of two adjacent files to crash 2017-06-09 21:05:44 -04:00
Gilbert Chen
42337d84c3 Move source files into src for a better looking front page 2017-06-07 14:04:11 -04:00
Gilbert Chen
4a1dc01ff4 Reorganize src directory 2017-06-02 16:33:36 -04:00
Gilbert Chen
bdf017e552 Move source files to src/ 2017-05-23 10:25:00 -04:00