jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2026-01-07 11:03:15 +00:00
Code Issues Projects Releases Wiki Activity

union: add :writback to act as a simple cache

Browse Source 
This adds a :writeback tag to upstreams. If set on a single upstream
then it writes back objects not found into that upstream.

Fixes #6934
This commit is contained in:
Nick Craig-Wood
2023-07-22 08:23:57 +01:00
parent b1c0ae5e7d
commit b33140ddeb
4 changed files with 128 additions and 13 deletions
Download Patch File Download Diff File Expand all files Collapse all files

53
docs/content/union.md
View File

@@ -6,24 +6,27 @@ versionIntroduced: "v1.44"
# {{< icon "fa fa-link" >}} Union
The `union` remote provides a unification similar to UnionFS using other remotes.
Paths may be as deep as required or a local path,
e.g. `remote:directory/subdirectory` or `/directory/subdirectory`.
The `union` backend joins several remotes together to make a single unified view of them.
During the initial setup with `rclone config` you will specify the upstream
remotes as a space separated list. The upstream remotes can either be a local paths or other remotes.
remotes as a space separated list. The upstream remotes can either be a local
paths or other remotes.
Attribute `:ro` and `:nc` can be attach to the end of path to tag the remote as **read only** or **no create**,
e.g. `remote:directory/subdirectory:ro` or `remote:directory/subdirectory:nc`.
The attributes `:ro`, `:nc` and `:nc` can be attached to the end of the remote
to tag the remote as **read only**, **no create** or **writeback**, e.g.
`remote:directory/subdirectory:ro` or `remote:directory/subdirectory:nc`.
- `:ro` means files will only be read from here and never written
- `:nc` means new files or directories won't be created here
- `:writeback` means files found in different remotes will be written back here. See the [writeback section](#writeback) for more info.
Subfolders can be used in upstream remotes. Assume a union remote named `backup`
with the remotes `mydrive:private/backup`. Invoking `rclone mkdir backup:desktop`
is exactly the same as invoking `rclone mkdir mydrive:private/backup/desktop`.
There will be no special handling of paths containing `..` segments.
Invoking `rclone mkdir backup:../desktop` is exactly the same as invoking
`rclone mkdir mydrive:private/backup/../desktop`.
There is no special handling of paths containing `..` segments. Invoking `rclone
mkdir backup:../desktop` is exactly the same as invoking `rclone mkdir
mydrive:private/backup/../desktop`.
## Configuration
@@ -172,6 +175,36 @@ The policies definition are inspired by [trapexit/mergerfs](https://github.com/t
| newest | Pick the file / directory with the largest mtime. |
| rand (random) | Calls **all** and then randomizes. Returns only one upstream. |
### Writeback {#writeback}
The tag `:writeback` on an upstream remote can be used to make a simple cache
system like this:
```
[union]
type = union
action_policy = all
create_policy = all
search_policy = ff
upstreams = /local:writeback remote:dir
```
When files are opened for read, if the file is in `remote:dir` but not `/local`
then rclone will copy the file entirely into `/local` before returning a
reference to the file in `/local`. The copy will be done with the equivalent of
`rclone copy` so will use `--multi-thread-streams` if configured. Any copies
will be logged with an INFO log.
When files are written, they will be written to both `remote:dir` and `/local`.
As many remotes as desired can be added to `upstreams` but there should only be
one `:writeback` tag.
Rclone does not manage the `:writeback` remote in any way other than writing
files back to it. So if you need to expire old files or manage the size then you
will have to do this yourself.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/union/union.go then run make backenddocs" >}}
### Standard options