1
0
mirror of https://github.com/rclone/rclone.git synced 2026-01-09 20:13:21 +00:00

chunker: prevent chunk corruption, survive meta-like input

This commit is contained in:
Ivan Andreev
2019-10-09 12:21:45 +03:00
committed by Nick Craig-Wood
parent 7aa2b4191c
commit 9049bb62ca
4 changed files with 506 additions and 79 deletions

View File

@@ -115,11 +115,16 @@ original content.
When the `list` rclone command scans a directory on wrapped remote,
the potential chunk files are accounted for, grouped and assembled into
composite directory entries. Any temporary chunks are hidden.
`list` can sometimes come across composite files with missing or invalid
chunks, eg if wrapped file system has been tampered with or damaged.
If chunker detects a missing chunk it will by default silently ignore
the whole group. You can use the `--chunker-fail-on-bad-chunks`
rclone option to make `list` fail with a loud error message.
List and other commands can sometimes come across composite files with
missing or invalid chunks, eg. shadowed by like-named directory or
another file. This usually means that wrapped file system has been directly
tampered with or damaged. If chunker detects a missing chunk it will
by default print warning, skip the whole incomplete group of chunks but
proceed with current command.
You can set the `--chunker-fail-hard` flag to have commands abort with
error message in such cases.
#### Chunk names
@@ -368,19 +373,18 @@ Metadata is a small JSON file named after the composite file.
- Simple JSON supports hash sums and chunk validation.
- It has the following fields: ver, size, nchunks, md5, sha1.
#### --chunker-fail-on-bad-chunks
#### --chunker-fail-hard
The list command might encounter files with missinng or invalid chunks.
This boolean flag tells what rclone should do in such cases.
Choose how chunker should handle files with missing or invalid chunks.
- Config: fail_on_bad_chunks
- Env Var: RCLONE_CHUNKER_FAIL_ON_BAD_CHUNKS
- Config: fail_hard
- Env Var: RCLONE_CHUNKER_FAIL_HARD
- Type: bool
- Default: false
- Examples:
- "true"
- Fail with error.
- Report errors and abort current command.
- "false"
- Silently ignore invalid object.
- Warn user, skip incomplete file and proceed.
<!--- autogenerated options stop -->