1
0
mirror of https://github.com/rclone/rclone.git synced 2025-12-27 21:53:27 +00:00

Compare commits

..

21 Commits

Author SHA1 Message Date
Nick Craig-Wood
f2d16ab4c5 Version v1.68.2 2024-11-15 12:20:50 +00:00
Nick Craig-Wood
c0fc4fe0ca s3: fix multitenant multipart uploads with CEPH
CEPH uses a special bucket form `tenant:bucket` for multitentant
access using S3 as documented here:

https://docs.ceph.com/en/reef/radosgw/multitenancy/#s3

However when doing multipart uploads, in the reply from
`CreateMultipart` the `tenant:` was missing from the `Bucket` response
rclone was using to build the `UploadPart` request. This caused a 404
failure return. This may be a CEPH bug, but it is easy to work around.

This changes the code to use the `Bucket` and `Key` that we used in
`CreateMultipart` in `UploadPart` rather than the one returned from
`CreateMultipart` which fixes the problem.

See: https://forum.rclone.org/t/rclone-zcat-does-not-work-with-a-multitenant-ceph-backend/48618
2024-11-14 16:50:19 +00:00
Nick Craig-Wood
669b2f2669 local: fix permission and ownership on symlinks with --links and --metadata
Before this change, if writing to a local backend with --metadata and
--links, if the incoming metadata contained mode or ownership
information then rclone would apply the mode/ownership to the
destination of the link not the link itself.

This fixes the problem by using the link safe sycall variants
lchown/fchmodat when --links and --metadata is in use. Note that Linux
does not support setting permissions on symlinks, so rclone emits a
debug message in this case.

This also fixes setting times on symlinks on Windows which wasn't
implemented for atime, mtime and was incorrectly setting the target of
the symlink for btime.

See: https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
2024-11-14 16:36:22 +00:00
Dimitrios Slamaris
e1ba10a86e bisync: fix output capture restoring the wrong output for logrus
Before this change, if rclone is used as a library and logrus is used
after a call to rc `sync/bisync`, logging does not work anymore and
leads to writing to a closed pipe.

This change restores the output correctly.

Fixes #8158
2024-11-14 16:36:22 +00:00
Nick Craig-Wood
022442cf58 build: fix comments after golangci-lint upgrade 2024-11-14 16:36:22 +00:00
dependabot[bot]
5cc4488294 build(deps): bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1
Bumps [github.com/golang-jwt/jwt/v4](https://github.com/golang-jwt/jwt) from 4.5.0 to 4.5.1.
- [Release notes](https://github.com/golang-jwt/jwt/releases)
- [Changelog](https://github.com/golang-jwt/jwt/blob/main/VERSION_HISTORY.md)
- [Commits](https://github.com/golang-jwt/jwt/compare/v4.5.0...v4.5.1)

---
updated-dependencies:
- dependency-name: github.com/golang-jwt/jwt/v4
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>
2024-11-14 16:36:22 +00:00
Nick Craig-Wood
ec9566c5c3 pikpak: fix fatal crash on startup with token that can't be refreshed 2024-11-14 16:36:22 +00:00
Nick Craig-Wood
f6976eb4c4 serve s3: fix excess locking which was making serve s3 single threaded
The fix for this was in the upstream library to narrow the locking
window.

See: https://forum.rclone.org/t/can-rclone-serve-s3-handle-more-than-one-client/48329/
2024-11-14 16:36:22 +00:00
Nick Craig-Wood
c242c00799 onedrive: fix Retry-After handling to look at 503 errors also
According to the Microsoft docs a Retry-After header can be returned
on 429 errors and 503 errors, but before this change we were only
checking for it on 429 errors.

See: https://forum.rclone.org/t/onedrive-503-response-retry-after-not-used/48045
2024-11-14 16:36:22 +00:00
Kaloyan Raev
bf954b74ff s3: Storj provider: fix server-side copy of files bigger than 5GB
Like some other S3-compatible providers, Storj does not currently
implements UploadPartCopy and returns NotImplemented errors for
multi-part server side copies.

This patch works around the problem by raising --s3-copy-cutoff for
Storj to the maximum. This means that rclone will never use
multi-part copies for files in Storj. This includes files larger than
5GB which (according to AWS documentation) must be copied with
multi-part copy. This works fine for Storj.

See https://github.com/storj/roadmap/issues/40
2024-11-14 16:36:22 +00:00
tgfisher
88f0770d0a docs: mention that inline comments are not supported in a filter-file 2024-11-14 16:36:22 +00:00
Randy Bush
41d905c9b0 docs: fix forward refs in step 9 of using your own client id 2024-11-14 16:36:22 +00:00
Alexandre Hamez
300a063b5e docs: fix Scaleway Glacier website URL 2024-11-14 16:36:21 +00:00
Simon Bos
61bf29ed5e dlna: fix loggingResponseWriter disregarding log level 2024-11-14 16:36:21 +00:00
Nick Craig-Wood
3191717572 s3: fix crash when using --s3-download-url after migration to SDKv2
Before this change rclone was crashing when the download URL did not
supply an X-Amz-Storage-Class header.

This change allows the header to be missing.

See: https://forum.rclone.org/t/sigsegv-on-ubuntu-24-04/48047
2024-11-14 16:36:21 +00:00
Nick Craig-Wood
961dfe97b5 docs: update overview to show pcloud can set modtime
See 258092f9c6 and #7896
2024-11-14 16:36:21 +00:00
Nick Craig-Wood
22612b4b38 Add RcloneView as a sponsor 2024-11-14 16:36:21 +00:00
Nick Craig-Wood
b9927461c3 accounting: fix wrong message on SIGUSR2 to enable/disable bwlimit
This was caused by the message code only looking at one of the
bandwidth filters, not all of them.

Fixes #8104
2024-11-14 16:36:21 +00:00
wiserain
6d04be99f2 pikpak: fix cid/gcid calculations for fs.OverrideRemote
Previously, cid/gcid (custom hash for pikpak) calculations failed when 
attempting to unwrap object info from `fs.OverrideRemote`. 

This commit introduces a new function that can correctly unwrap 
object info from both regular objects and `fs.OverrideRemote` types, 
ensuring uploads with accurate cid/gcid calculations in all scenarios.
2024-11-14 16:36:21 +00:00
nielash
06ae0dfa54 local: fix --copy-links on macOS when cloning
Before this change, --copy-links erroneously behaved like --links when using cloning
on macOS, and cloning was not supported at all when using --links.

After this change, --copy-links does what it's supposed to, and takes advantage of
cloning when possible, by copying the file being linked to instead of the link
itself.

Cloning is now also supported in --links mode for regular files (which benefit
most from cloning). symlinks in --links mode continue to be tossed back to be
handled by rclone's special translation logic.

See https://forum.rclone.org/t/macos-local-to-local-copy-with-copy-links-causes-error/47671/5?u=nielash
2024-11-14 16:36:21 +00:00
Nick Craig-Wood
912f29b5b8 Start v1.68.2-DEV development 2024-09-24 17:25:53 +01:00
36 changed files with 604 additions and 97 deletions

57
MANUAL.html generated
View File

@@ -81,7 +81,7 @@
<header id="title-block-header">
<h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p>
<p class="date">Sep 24, 2024</p>
<p class="date">Nov 15, 2024</p>
</header>
<h1 id="rclone-syncs-your-files-to-cloud-storage">Rclone syncs your files to cloud storage</h1>
<p><img width="50%" src="https://rclone.org/img/logo_on_light__horizontal_color.svg" alt="rclone logo" style="float:right; padding: 5px;" ></p>
@@ -8323,12 +8323,14 @@ file2.avi</code></pre>
<p>Adds path/file names to an rclone command based on rules in a named file. The file contains a list of remarks and pattern rules. Include rules start with <code>+</code> and exclude rules with <code>-</code>. <code>!</code> clears existing rules. Rules are processed in the order they are defined.</p>
<p>This flag can be repeated. See above for the order filter flags are processed in.</p>
<p>Arrange the order of filter rules with the most restrictive first and work down.</p>
<p>Lines starting with # or ; are ignored, and can be used to write comments. Inline comments are not supported. <em>Use <code>-vv --dump filters</code> to see how they appear in the final regexp.</em></p>
<p>E.g. for <code>filter-file.txt</code>:</p>
<pre><code># a sample filter rule file
- secret*.jpg
+ *.jpg
+ *.png
+ file2.avi
- /dir/tmp/** # WARNING! This text will be treated as part of the path.
- /dir/Trash/**
+ /dir/**
# exclude everything else
@@ -10418,7 +10420,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
<tr class="odd">
<td>pCloud</td>
<td style="text-align: center;">MD5, SHA1 ⁷</td>
<td style="text-align: center;">R</td>
<td style="text-align: center;">R/W</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">No</td>
<td style="text-align: center;">W</td>
@@ -11968,7 +11970,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.68.1&quot;)</code></pre>
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.68.2&quot;)</code></pre>
<h2 id="performance">Performance</h2>
<p>Flags helpful for increasing performance.</p>
<pre><code> --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -17168,7 +17170,7 @@ acl = private
upload_cutoff = 5M
chunk_size = 5M
copy_cutoff = 5M</code></pre>
<p><a href="https://www.online.net/en/storage/c14-cold-storage">C14 Cold Storage</a> is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" <code>storage_class</code>. So you can configure your remote with the <code>storage_class = GLACIER</code> option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)</p>
<p><a href="https://www.scaleway.com/en/glacier-cold-storage/">Scaleway Glacier</a> is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" <code>storage_class</code>. So you can configure your remote with the <code>storage_class = GLACIER</code> option to upload directly to Scaleway Glacier. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)</p>
<h3 id="lyve">Seagate Lyve Cloud</h3>
<p><a href="https://www.seagate.com/gb/en/services/cloud/storage/">Seagate Lyve Cloud</a> is an S3 compatible object storage platform from <a href="https://seagate.com/">Seagate</a> intended for enterprise use.</p>
<p>Here is a config run through for a remote called <code>remote</code> - you may choose a different name of course. Note that to create an access key and secret key you will need to create a service account first.</p>
@@ -24613,7 +24615,7 @@ rclone backend copyid drive: ID1 path1 ID2 path2</code></pre>
<li><p>Click on the "+ CREATE CREDENTIALS" button at the top of the screen, then select "OAuth client ID".</p></li>
<li><p>Choose an application type of "Desktop app" and click "Create". (the default name is fine)</p></li>
<li><p>It will show you a client ID and client secret. Make a note of these.</p>
<p>(If you selected "External" at Step 5 continue to Step 9. If you chose "Internal" you don't need to publish and can skip straight to Step 10 but your destination drive must be part of the same Google Workspace.)</p></li>
<p>(If you selected "External" at Step 5 continue to Step 10. If you chose "Internal" you don't need to publish and can skip straight to Step 11 but your destination drive must be part of the same Google Workspace.)</p></li>
<li><p>Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm. You will also want to add yourself as a test user.</p></li>
<li><p>Provide the noted client ID and client secret to rclone.</p></li>
</ol>
@@ -36964,6 +36966,51 @@ $ tree /tmp/c
<li>"error": return an error based on option value</li>
</ul>
<h1 id="changelog-1">Changelog</h1>
<h2 id="v1.68.2---2024-11-15">v1.68.2 - 2024-11-15</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.68.1...v1.68.2">See commits</a></p>
<ul>
<li>Security fixes
<ul>
<li>local backend: CVE-2024-52522: fix permission and ownership on symlinks with <code>--links</code> and <code>--metadata</code> (Nick Craig-Wood)
<ul>
<li>Only affects users using <code>--metadata</code> and <code>--links</code> and copying files to the local backend</li>
<li>See https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv</li>
</ul></li>
<li>build: bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (dependabot)
<ul>
<li>This is an issue in a dependency which is used for JWT certificates</li>
<li>See https://github.com/golang-jwt/jwt/security/advisories/GHSA-29wx-vh33-7x7r</li>
</ul></li>
</ul></li>
<li>Bug Fixes
<ul>
<li>accounting: Fix wrong message on SIGUSR2 to enable/disable bwlimit (Nick Craig-Wood)</li>
<li>bisync: Fix output capture restoring the wrong output for logrus (Dimitrios Slamaris)</li>
<li>dlna: Fix loggingResponseWriter disregarding log level (Simon Bos)</li>
<li>serve s3: Fix excess locking which was making serve s3 single threaded (Nick Craig-Wood)</li>
<li>doc fixes (Nick Craig-Wood, tgfisher, Alexandre Hamez, Randy Bush)</li>
</ul></li>
<li>Local
<ul>
<li>Fix permission and ownership on symlinks with <code>--links</code> and <code>--metadata</code> (Nick Craig-Wood)</li>
<li>Fix <code>--copy-links</code> on macOS when cloning (nielash)</li>
</ul></li>
<li>Onedrive
<ul>
<li>Fix Retry-After handling to look at 503 errors also (Nick Craig-Wood)</li>
</ul></li>
<li>Pikpak
<ul>
<li>Fix cid/gcid calculations for fs.OverrideRemote (wiserain)</li>
<li>Fix fatal crash on startup with token that can't be refreshed (Nick Craig-Wood)</li>
</ul></li>
<li>S3
<ul>
<li>Fix crash when using <code>--s3-download-url</code> after migration to SDKv2 (Nick Craig-Wood)</li>
<li>Storj provider: fix server-side copy of files bigger than 5GB (Kaloyan Raev)</li>
<li>Fix multitenant multipart uploads with CEPH (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.68.1---2024-09-24">v1.68.1 - 2024-09-24</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.68.0...v1.68.1">See commits</a></p>
<ul>

47
MANUAL.md generated
View File

@@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Sep 24, 2024
% Nov 15, 2024
# Rclone syncs your files to cloud storage
@@ -16970,6 +16970,8 @@ processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
Lines starting with # or ; are ignored, and can be used to write comments. Inline comments are not supported. _Use `-vv --dump filters` to see how they appear in the final regexp._
E.g. for `filter-file.txt`:
# a sample filter rule file
@@ -16977,6 +16979,7 @@ E.g. for `filter-file.txt`:
+ *.jpg
+ *.png
+ file2.avi
- /dir/tmp/** # WARNING! This text will be treated as part of the path.
- /dir/Trash/**
+ /dir/**
# exclude everything else
@@ -19787,7 +19790,7 @@ Here is an overview of the major features of each cloud storage system.
| OpenDrive | MD5 | R/W | Yes | Partial ⁸ | - | - |
| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |
| pCloud | MD5, SHA1 ⁷ | R | No | No | W | - |
| pCloud | MD5, SHA1 ⁷ | R/W | No | No | W | - |
| PikPak | MD5 | R | No | No | R | - |
| Pixeldrain | SHA256 | R/W | No | No | R | RW |
| premiumize.me | - | - | Yes | No | R | - |
@@ -20494,7 +20497,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
```
@@ -27833,8 +27836,8 @@ chunk_size = 5M
copy_cutoff = 5M
```
[C14 Cold Storage](https://www.online.net/en/storage/c14-cold-storage) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
So you can configure your remote with the `storage_class = GLACIER` option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
[Scaleway Glacier](https://www.scaleway.com/en/glacier-cold-storage/) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
So you can configure your remote with the `storage_class = GLACIER` option to upload directly to Scaleway Glacier. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
### Seagate Lyve Cloud {#lyve}
@@ -37898,9 +37901,9 @@ then select "OAuth client ID".
9. It will show you a client ID and client secret. Make a note of these.
(If you selected "External" at Step 5 continue to Step 9.
(If you selected "External" at Step 5 continue to Step 10.
If you chose "Internal" you don't need to publish and can skip straight to
Step 10 but your destination drive must be part of the same Google Workspace.)
Step 11 but your destination drive must be part of the same Google Workspace.)
10. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
You will also want to add yourself as a test user.
@@ -54612,6 +54615,36 @@ Options:
# Changelog
## v1.68.2 - 2024-11-15
[See commits](https://github.com/rclone/rclone/compare/v1.68.1...v1.68.2)
* Security fixes
* local backend: CVE-2024-52522: fix permission and ownership on symlinks with `--links` and `--metadata` (Nick Craig-Wood)
* Only affects users using `--metadata` and `--links` and copying files to the local backend
* See https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
* build: bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (dependabot)
* This is an issue in a dependency which is used for JWT certificates
* See https://github.com/golang-jwt/jwt/security/advisories/GHSA-29wx-vh33-7x7r
* Bug Fixes
* accounting: Fix wrong message on SIGUSR2 to enable/disable bwlimit (Nick Craig-Wood)
* bisync: Fix output capture restoring the wrong output for logrus (Dimitrios Slamaris)
* dlna: Fix loggingResponseWriter disregarding log level (Simon Bos)
* serve s3: Fix excess locking which was making serve s3 single threaded (Nick Craig-Wood)
* doc fixes (Nick Craig-Wood, tgfisher, Alexandre Hamez, Randy Bush)
* Local
* Fix permission and ownership on symlinks with `--links` and `--metadata` (Nick Craig-Wood)
* Fix `--copy-links` on macOS when cloning (nielash)
* Onedrive
* Fix Retry-After handling to look at 503 errors also (Nick Craig-Wood)
* Pikpak
* Fix cid/gcid calculations for fs.OverrideRemote (wiserain)
* Fix fatal crash on startup with token that can't be refreshed (Nick Craig-Wood)
* S3
* Fix crash when using `--s3-download-url` after migration to SDKv2 (Nick Craig-Wood)
* Storj provider: fix server-side copy of files bigger than 5GB (Kaloyan Raev)
* Fix multitenant multipart uploads with CEPH (Nick Craig-Wood)
## v1.68.1 - 2024-09-24
[See commits](https://github.com/rclone/rclone/compare/v1.68.0...v1.68.1)

71
MANUAL.txt generated
View File

@@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Sep 24, 2024
Nov 15, 2024
Rclone syncs your files to cloud storage
@@ -16428,6 +16428,10 @@ processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
Lines starting with # or ; are ignored, and can be used to write
comments. Inline comments are not supported. Use -vv --dump filters to
see how they appear in the final regexp.
E.g. for filter-file.txt:
# a sample filter rule file
@@ -16435,6 +16439,7 @@ E.g. for filter-file.txt:
+ *.jpg
+ *.png
+ file2.avi
- /dir/tmp/** # WARNING! This text will be treated as part of the path.
- /dir/Trash/**
+ /dir/**
# exclude everything else
@@ -19269,7 +19274,7 @@ Here is an overview of the major features of each cloud storage system.
OpenDrive MD5 R/W Yes Partial ⁸ - -
OpenStack Swift MD5 R/W No No R/W -
Oracle Object Storage MD5 R/W No No R/W -
pCloud MD5, SHA1 ⁷ R No No W -
pCloud MD5, SHA1 ⁷ R/W No No W -
PikPak MD5 R No No R -
Pixeldrain SHA256 R/W No No R RW
premiumize.me - - Yes No R -
@@ -20076,7 +20081,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
Performance
@@ -27364,13 +27369,13 @@ rclone like this:
chunk_size = 5M
copy_cutoff = 5M
C14 Cold Storage is the low-cost S3 Glacier alternative from Scaleway
Scaleway Glacier is the low-cost S3 Glacier alternative from Scaleway
and it works the same way as on S3 by accepting the "GLACIER"
storage_class. So you can configure your remote with the
storage_class = GLACIER option to upload directly to C14. Don't forget
that in this state you can't read files back after, you will need to
restore them to "STANDARD" storage_class first before being able to read
them (see "restore" section above)
storage_class = GLACIER option to upload directly to Scaleway Glacier.
Don't forget that in this state you can't read files back after, you
will need to restore them to "STANDARD" storage_class first before being
able to read them (see "restore" section above)
Seagate Lyve Cloud
@@ -37324,9 +37329,9 @@ Here is how to create your own Google Drive client ID for rclone:
9. It will show you a client ID and client secret. Make a note of
these.
(If you selected "External" at Step 5 continue to Step 9. If you
(If you selected "External" at Step 5 continue to Step 10. If you
chose "Internal" you don't need to publish and can skip straight to
Step 10 but your destination drive must be part of the same Google
Step 11 but your destination drive must be part of the same Google
Workspace.)
10. Go to "Oauth consent screen" and then click "PUBLISH APP" button and
@@ -54290,6 +54295,52 @@ Options:
Changelog
v1.68.2 - 2024-11-15
See commits
- Security fixes
- local backend: CVE-2024-52522: fix permission and ownership on
symlinks with --links and --metadata (Nick Craig-Wood)
- Only affects users using --metadata and --links and copying
files to the local backend
- See
https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
- build: bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1
(dependabot)
- This is an issue in a dependency which is used for JWT
certificates
- See
https://github.com/golang-jwt/jwt/security/advisories/GHSA-29wx-vh33-7x7r
- Bug Fixes
- accounting: Fix wrong message on SIGUSR2 to enable/disable
bwlimit (Nick Craig-Wood)
- bisync: Fix output capture restoring the wrong output for logrus
(Dimitrios Slamaris)
- dlna: Fix loggingResponseWriter disregarding log level (Simon
Bos)
- serve s3: Fix excess locking which was making serve s3 single
threaded (Nick Craig-Wood)
- doc fixes (Nick Craig-Wood, tgfisher, Alexandre Hamez, Randy
Bush)
- Local
- Fix permission and ownership on symlinks with --links and
--metadata (Nick Craig-Wood)
- Fix --copy-links on macOS when cloning (nielash)
- Onedrive
- Fix Retry-After handling to look at 503 errors also (Nick
Craig-Wood)
- Pikpak
- Fix cid/gcid calculations for fs.OverrideRemote (wiserain)
- Fix fatal crash on startup with token that can't be refreshed
(Nick Craig-Wood)
- S3
- Fix crash when using --s3-download-url after migration to SDKv2
(Nick Craig-Wood)
- Storj provider: fix server-side copy of files bigger than 5GB
(Kaloyan Raev)
- Fix multitenant multipart uploads with CEPH (Nick Craig-Wood)
v1.68.1 - 2024-09-24
See commits

View File

@@ -1 +1 @@
v1.68.1
v1.68.2

View File

@@ -6,6 +6,7 @@ package local
import (
"context"
"fmt"
"path/filepath"
"runtime"
"github.com/go-darwin/apfs"
@@ -22,7 +23,7 @@ import (
//
// If it isn't possible then return fs.ErrorCantCopy
func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object, error) {
if runtime.GOOS != "darwin" || f.opt.TranslateSymlinks || f.opt.NoClone {
if runtime.GOOS != "darwin" || f.opt.NoClone {
return nil, fs.ErrorCantCopy
}
srcObj, ok := src.(*Object)
@@ -30,6 +31,9 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
fs.Debugf(src, "Can't clone - not same remote type")
return nil, fs.ErrorCantCopy
}
if f.opt.TranslateSymlinks && srcObj.translatedLink { // in --links mode, use cloning only for regular files
return nil, fs.ErrorCantCopy
}
// Fetch metadata if --metadata is in use
meta, err := fs.GetMetadataOptions(ctx, f, src, fs.MetadataAsOpenOptions(ctx))
@@ -44,11 +48,18 @@ func (f *Fs) Copy(ctx context.Context, src fs.Object, remote string) (fs.Object,
return nil, err
}
err = Clone(srcObj.path, f.localPath(remote))
srcPath := srcObj.path
if f.opt.FollowSymlinks { // in --copy-links mode, find the real file being pointed to and pass that in instead
srcPath, err = filepath.EvalSymlinks(srcPath)
if err != nil {
return nil, err
}
}
err = Clone(srcPath, f.localPath(remote))
if err != nil {
return nil, err
}
fs.Debugf(remote, "server-side cloned!")
// Set metadata if --metadata is in use
if meta != nil {

16
backend/local/lchmod.go Normal file
View File

@@ -0,0 +1,16 @@
//go:build windows || plan9 || js || linux
package local
import "os"
const haveLChmod = false
// lChmod changes the mode of the named file to mode. If the file is a symbolic
// link, it changes the link, not the target. If there is an error,
// it will be of type *PathError.
func lChmod(name string, mode os.FileMode) error {
// Can't do this safely on this OS - chmoding a symlink always
// changes the destination.
return nil
}

View File

@@ -0,0 +1,41 @@
//go:build !windows && !plan9 && !js && !linux
package local
import (
"os"
"syscall"
"golang.org/x/sys/unix"
)
const haveLChmod = true
// syscallMode returns the syscall-specific mode bits from Go's portable mode bits.
//
// Borrowed from the syscall source since it isn't public.
func syscallMode(i os.FileMode) (o uint32) {
o |= uint32(i.Perm())
if i&os.ModeSetuid != 0 {
o |= syscall.S_ISUID
}
if i&os.ModeSetgid != 0 {
o |= syscall.S_ISGID
}
if i&os.ModeSticky != 0 {
o |= syscall.S_ISVTX
}
return o
}
// lChmod changes the mode of the named file to mode. If the file is a symbolic
// link, it changes the link, not the target. If there is an error,
// it will be of type *PathError.
func lChmod(name string, mode os.FileMode) error {
// NB linux does not support AT_SYMLINK_NOFOLLOW as a parameter to fchmodat
// and returns ENOTSUP if you try, so we don't support this on linux
if e := unix.Fchmodat(unix.AT_FDCWD, name, syscallMode(mode), unix.AT_SYMLINK_NOFOLLOW); e != nil {
return &os.PathError{Op: "lChmod", Path: name, Err: e}
}
return nil
}

View File

@@ -1,4 +1,4 @@
//go:build windows || plan9 || js
//go:build plan9 || js
package local

View File

@@ -0,0 +1,19 @@
//go:build windows
package local
import (
"time"
)
const haveLChtimes = true
// lChtimes changes the access and modification times of the named
// link, similar to the Unix utime() or utimes() functions.
//
// The underlying filesystem may truncate or round the values to a
// less precise time unit.
// If there is an error, it will be of type *PathError.
func lChtimes(name string, atime time.Time, mtime time.Time) error {
return setTimes(name, atime, mtime, time.Time{}, true)
}

View File

@@ -73,7 +73,6 @@ func TestUpdatingCheck(t *testing.T) {
r.WriteFile(filePath, "content updated", time.Now())
_, err = in.Read(buf)
require.NoError(t, err)
}
// Test corrupted on transfer
@@ -224,7 +223,7 @@ func TestHashOnUpdate(t *testing.T) {
assert.Equal(t, "9a0364b9e99bb480dd25e1f0284c8555", md5)
// Reupload it with different contents but same size and timestamp
var b = bytes.NewBufferString("CONTENT")
b := bytes.NewBufferString("CONTENT")
src := object.NewStaticObjectInfo(filePath, when, int64(b.Len()), true, nil, f)
err = o.Update(ctx, b, src)
require.NoError(t, err)
@@ -269,22 +268,66 @@ func TestMetadata(t *testing.T) {
r := fstest.NewRun(t)
const filePath = "metafile.txt"
when := time.Now()
const dayLength = len("2001-01-01")
whenRFC := when.Format(time.RFC3339Nano)
r.WriteFile(filePath, "metadata file contents", when)
f := r.Flocal.(*Fs)
// Set fs into "-l" / "--links" mode
f.opt.TranslateSymlinks = true
// Write a symlink to the file
symlinkPath := "metafile-link.txt"
osSymlinkPath := filepath.Join(f.root, symlinkPath)
symlinkPath += linkSuffix
require.NoError(t, os.Symlink(filePath, osSymlinkPath))
symlinkModTime := fstest.Time("2002-02-03T04:05:10.123123123Z")
require.NoError(t, lChtimes(osSymlinkPath, symlinkModTime, symlinkModTime))
// Get the object
obj, err := f.NewObject(ctx, filePath)
require.NoError(t, err)
o := obj.(*Object)
// Get the symlink object
symlinkObj, err := f.NewObject(ctx, symlinkPath)
require.NoError(t, err)
symlinkO := symlinkObj.(*Object)
// Record metadata for o
oMeta, err := o.Metadata(ctx)
require.NoError(t, err)
// Test symlink first to check it doesn't mess up file
t.Run("Symlink", func(t *testing.T) {
testMetadata(t, r, symlinkO, symlinkModTime)
})
// Read it again
oMetaNew, err := o.Metadata(ctx)
require.NoError(t, err)
// Check that operating on the symlink didn't change the file it was pointing to
// See: https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
assert.Equal(t, oMeta, oMetaNew, "metadata setting on symlink messed up file")
// Now run the same tests on the file
t.Run("File", func(t *testing.T) {
testMetadata(t, r, o, when)
})
}
func testMetadata(t *testing.T, r *fstest.Run, o *Object, when time.Time) {
ctx := context.Background()
whenRFC := when.Format(time.RFC3339Nano)
const dayLength = len("2001-01-01")
f := r.Flocal.(*Fs)
features := f.Features()
var hasXID, hasAtime, hasBtime bool
var hasXID, hasAtime, hasBtime, canSetXattrOnLinks bool
switch runtime.GOOS {
case "darwin", "freebsd", "netbsd", "linux":
hasXID, hasAtime, hasBtime = true, true, true
canSetXattrOnLinks = runtime.GOOS != "linux"
case "openbsd", "solaris":
hasXID, hasAtime = true, true
case "windows":
@@ -307,6 +350,10 @@ func TestMetadata(t *testing.T) {
require.NoError(t, err)
assert.Nil(t, m)
if !canSetXattrOnLinks && o.translatedLink {
t.Skip("Skip remainder of test as can't set xattr on symlinks on this OS")
}
inM := fs.Metadata{
"potato": "chips",
"cabbage": "soup",
@@ -321,18 +368,21 @@ func TestMetadata(t *testing.T) {
})
checkTime := func(m fs.Metadata, key string, when time.Time) {
t.Helper()
mt, ok := o.parseMetadataTime(m, key)
assert.True(t, ok)
dt := mt.Sub(when)
precision := time.Second
assert.True(t, dt >= -precision && dt <= precision, fmt.Sprintf("%s: dt %v outside +/- precision %v", key, dt, precision))
assert.True(t, dt >= -precision && dt <= precision, fmt.Sprintf("%s: dt %v outside +/- precision %v want %v got %v", key, dt, precision, mt, when))
}
checkInt := func(m fs.Metadata, key string, base int) int {
t.Helper()
value, ok := o.parseMetadataInt(m, key, base)
assert.True(t, ok)
return value
}
t.Run("Read", func(t *testing.T) {
m, err := o.Metadata(ctx)
require.NoError(t, err)
@@ -342,13 +392,12 @@ func TestMetadata(t *testing.T) {
checkInt(m, "mode", 8)
checkTime(m, "mtime", when)
assert.Equal(t, len(whenRFC), len(m["mtime"]))
assert.Equal(t, whenRFC[:dayLength], m["mtime"][:dayLength])
if hasAtime {
if hasAtime && !o.translatedLink { // symlinks generally don't record atime
checkTime(m, "atime", when)
}
if hasBtime {
if hasBtime && !o.translatedLink { // symlinks generally don't record btime
checkTime(m, "btime", when)
}
if hasXID {
@@ -372,6 +421,10 @@ func TestMetadata(t *testing.T) {
"mode": "0767",
"potato": "wedges",
}
if !canSetXattrOnLinks && o.translatedLink {
// Don't change xattr if not supported on symlinks
delete(newM, "potato")
}
err := o.writeMetadata(newM)
require.NoError(t, err)
@@ -381,7 +434,11 @@ func TestMetadata(t *testing.T) {
mode := checkInt(m, "mode", 8)
if runtime.GOOS != "windows" {
assert.Equal(t, 0767, mode&0777, fmt.Sprintf("mode wrong - expecting 0767 got 0%o", mode&0777))
expectedMode := 0767
if o.translatedLink && runtime.GOOS == "linux" {
expectedMode = 0777 // perms of symlinks always read as 0777 on linux
}
assert.Equal(t, expectedMode, mode&0777, fmt.Sprintf("mode wrong - expecting 0%o got 0%o", expectedMode, mode&0777))
}
checkTime(m, "mtime", newMtime)
@@ -391,11 +448,10 @@ func TestMetadata(t *testing.T) {
if haveSetBTime {
checkTime(m, "btime", newBtime)
}
if xattrSupported {
if xattrSupported && (canSetXattrOnLinks || !o.translatedLink) {
assert.Equal(t, "wedges", m["potato"])
}
})
}
func TestFilter(t *testing.T) {
@@ -572,4 +628,35 @@ func TestCopySymlink(t *testing.T) {
linkContents, err := os.Readlink(dstPath)
require.NoError(t, err)
assert.Equal(t, "file.txt", linkContents)
// Set fs into "-L/--copy-links" mode
f.opt.FollowSymlinks = true
f.opt.TranslateSymlinks = false
f.lstat = os.Stat
// Create dst
require.NoError(t, f.Mkdir(ctx, "dst2"))
// Do copy from src into dst
src, err = f.NewObject(ctx, "src/link.txt")
require.NoError(t, err)
require.NotNil(t, src)
dst, err = operations.Copy(ctx, f, nil, "dst2/link.txt", src)
require.NoError(t, err)
require.NotNil(t, dst)
// Test that we made a NON-symlink and it has the right contents
dstPath = filepath.Join(r.LocalName, "dst2", "link.txt")
fi, err := os.Lstat(dstPath)
require.NoError(t, err)
assert.True(t, fi.Mode()&os.ModeSymlink == 0)
want := fstest.NewItem("dst2/link.txt", "hello world", when)
fstest.CompareItems(t, []fs.DirEntry{dst}, []fstest.Item{want}, nil, f.precision, "")
// Test that copying a normal file also works
dst, err = operations.Copy(ctx, f, nil, "dst2/file.txt", dst)
require.NoError(t, err)
require.NotNil(t, dst)
want = fstest.NewItem("dst2/file.txt", "hello world", when)
fstest.CompareItems(t, []fs.DirEntry{dst}, []fstest.Item{want}, nil, f.precision, "")
}

View File

@@ -105,7 +105,11 @@ func (o *Object) writeMetadataToFile(m fs.Metadata) (outErr error) {
}
if haveSetBTime {
if btimeOK {
err = setBTime(o.path, btime)
if o.translatedLink {
err = lsetBTime(o.path, btime)
} else {
err = setBTime(o.path, btime)
}
if err != nil {
outErr = fmt.Errorf("failed to set birth (creation) time: %w", err)
}
@@ -121,7 +125,11 @@ func (o *Object) writeMetadataToFile(m fs.Metadata) (outErr error) {
if runtime.GOOS == "windows" || runtime.GOOS == "plan9" {
fs.Debugf(o, "Ignoring request to set ownership %o.%o on this OS", gid, uid)
} else {
err = os.Chown(o.path, uid, gid)
if o.translatedLink {
err = os.Lchown(o.path, uid, gid)
} else {
err = os.Chown(o.path, uid, gid)
}
if err != nil {
outErr = fmt.Errorf("failed to change ownership: %w", err)
}
@@ -132,7 +140,16 @@ func (o *Object) writeMetadataToFile(m fs.Metadata) (outErr error) {
if mode >= 0 {
umode := uint(mode)
if umode <= math.MaxUint32 {
err = os.Chmod(o.path, os.FileMode(umode))
if o.translatedLink {
if haveLChmod {
err = lChmod(o.path, os.FileMode(umode))
} else {
fs.Debugf(o, "Unable to set mode %v on a symlink on this OS", os.FileMode(umode))
err = nil
}
} else {
err = os.Chmod(o.path, os.FileMode(umode))
}
if err != nil {
outErr = fmt.Errorf("failed to change permissions: %w", err)
}

View File

@@ -13,3 +13,9 @@ func setBTime(name string, btime time.Time) error {
// Does nothing
return nil
}
// lsetBTime changes the birth time of the link passed in
func lsetBTime(name string, btime time.Time) error {
// Does nothing
return nil
}

View File

@@ -9,15 +9,20 @@ import (
const haveSetBTime = true
// setBTime sets the birth time of the file passed in
func setBTime(name string, btime time.Time) (err error) {
// setTimes sets any of atime, mtime or btime
// if link is set it sets a link rather than the target
func setTimes(name string, atime, mtime, btime time.Time, link bool) (err error) {
pathp, err := syscall.UTF16PtrFromString(name)
if err != nil {
return err
}
fileFlag := uint32(syscall.FILE_FLAG_BACKUP_SEMANTICS)
if link {
fileFlag |= syscall.FILE_FLAG_OPEN_REPARSE_POINT
}
h, err := syscall.CreateFile(pathp,
syscall.FILE_WRITE_ATTRIBUTES, syscall.FILE_SHARE_WRITE, nil,
syscall.OPEN_EXISTING, syscall.FILE_FLAG_BACKUP_SEMANTICS, 0)
syscall.OPEN_EXISTING, fileFlag, 0)
if err != nil {
return err
}
@@ -27,6 +32,28 @@ func setBTime(name string, btime time.Time) (err error) {
err = closeErr
}
}()
bFileTime := syscall.NsecToFiletime(btime.UnixNano())
return syscall.SetFileTime(h, &bFileTime, nil, nil)
var patime, pmtime, pbtime *syscall.Filetime
if !atime.IsZero() {
t := syscall.NsecToFiletime(atime.UnixNano())
patime = &t
}
if !mtime.IsZero() {
t := syscall.NsecToFiletime(mtime.UnixNano())
pmtime = &t
}
if !btime.IsZero() {
t := syscall.NsecToFiletime(btime.UnixNano())
pbtime = &t
}
return syscall.SetFileTime(h, pbtime, patime, pmtime)
}
// setBTime sets the birth time of the file passed in
func setBTime(name string, btime time.Time) (err error) {
return setTimes(name, time.Time{}, time.Time{}, btime, false)
}
// lsetBTime changes the birth time of the link passed in
func lsetBTime(name string, btime time.Time) error {
return setTimes(name, time.Time{}, time.Time{}, btime, true)
}

View File

@@ -202,9 +202,14 @@ type SharingLinkType struct {
type LinkType string
const (
ViewLinkType LinkType = "view" // ViewLinkType (role: read) A view-only sharing link, allowing read-only access.
EditLinkType LinkType = "edit" // EditLinkType (role: write) An edit sharing link, allowing read-write access.
EmbedLinkType LinkType = "embed" // EmbedLinkType (role: read) A view-only sharing link that can be used to embed content into a host webpage. Embed links are not available for OneDrive for Business or SharePoint.
// ViewLinkType (role: read) A view-only sharing link, allowing read-only access.
ViewLinkType LinkType = "view"
// EditLinkType (role: write) An edit sharing link, allowing read-write access.
EditLinkType LinkType = "edit"
// EmbedLinkType (role: read) A view-only sharing link that can be used to embed
// content into a host webpage. Embed links are not available for OneDrive for
// Business or SharePoint.
EmbedLinkType LinkType = "embed"
)
// LinkScope represents the scope of the link represented by this permission.
@@ -212,9 +217,12 @@ const (
type LinkScope string
const (
AnonymousScope LinkScope = "anonymous" // AnonymousScope = Anyone with the link has access, without needing to sign in. This may include people outside of your organization.
OrganizationScope LinkScope = "organization" // OrganizationScope = Anyone signed into your organization (tenant) can use the link to get access. Only available in OneDrive for Business and SharePoint.
// AnonymousScope = Anyone with the link has access, without needing to sign in.
// This may include people outside of your organization.
AnonymousScope LinkScope = "anonymous"
// OrganizationScope = Anyone signed into your organization (tenant) can use the
// link to get access. Only available in OneDrive for Business and SharePoint.
OrganizationScope LinkScope = "organization"
)
// PermissionsType provides information about a sharing permission granted for a DriveItem resource.
@@ -236,10 +244,14 @@ type PermissionsType struct {
type Role string
const (
ReadRole Role = "read" // ReadRole provides the ability to read the metadata and contents of the item.
WriteRole Role = "write" // WriteRole provides the ability to read and modify the metadata and contents of the item.
OwnerRole Role = "owner" // OwnerRole represents the owner role for SharePoint and OneDrive for Business.
MemberRole Role = "member" // MemberRole represents the member role for SharePoint and OneDrive for Business.
// ReadRole provides the ability to read the metadata and contents of the item.
ReadRole Role = "read"
// WriteRole provides the ability to read and modify the metadata and contents of the item.
WriteRole Role = "write"
// OwnerRole represents the owner role for SharePoint and OneDrive for Business.
OwnerRole Role = "owner"
// MemberRole represents the member role for SharePoint and OneDrive for Business.
MemberRole Role = "member"
)
// PermissionsResponse is the response to the list permissions method

View File

@@ -827,7 +827,7 @@ func shouldRetry(ctx context.Context, resp *http.Response, err error) (bool, err
retry = true
fs.Debugf(nil, "HTTP 401: Unable to initialize RPS. Trying again.")
}
case 429: // Too Many Requests.
case 429, 503: // Too Many Requests, Server Too Busy
// see https://docs.microsoft.com/en-us/sharepoint/dev/general-development/how-to-avoid-getting-throttled-or-blocked-in-sharepoint-online
if values := resp.Header["Retry-After"]; len(values) == 1 && values[0] != "" {
retryAfter, parseErr := strconv.Atoi(values[0])

View File

@@ -378,11 +378,23 @@ func calcGcid(r io.Reader, size int64) (string, error) {
return hex.EncodeToString(totalHash.Sum(nil)), nil
}
// unWrapObjectInfo returns the underlying Object unwrapped as much as
// possible or nil even if it is an OverrideRemote
func unWrapObjectInfo(oi fs.ObjectInfo) fs.Object {
if o, ok := oi.(fs.Object); ok {
return fs.UnWrapObject(o)
} else if do, ok := oi.(*fs.OverrideRemote); ok {
// Unwrap if it is an operations.OverrideRemote
return do.UnWrap()
}
return nil
}
// calcCid calculates Cid from source
//
// Cid is a simplified version of Gcid
func calcCid(ctx context.Context, src fs.ObjectInfo) (cid string, err error) {
srcObj := fs.UnWrapObjectInfo(src)
srcObj := unWrapObjectInfo(src)
if srcObj == nil {
return "", fmt.Errorf("failed to unwrap object from src: %s", src)
}

View File

@@ -561,6 +561,7 @@ func newFs(ctx context.Context, name, path string, m configmap.Mapper) (*Fs, err
if strings.Contains(err.Error(), "invalid_grant") {
return f, f.reAuthorize(ctx)
}
return nil, err
}
return f, nil
@@ -1773,7 +1774,7 @@ func (o *Object) upload(ctx context.Context, in io.Reader, src fs.ObjectInfo, wi
gcid, err := o.fs.getGcid(ctx, src)
if err != nil || gcid == "" {
fs.Debugf(o, "calculating gcid: %v", err)
if srcObj := fs.UnWrapObjectInfo(src); srcObj != nil && srcObj.Fs().Features().IsLocal {
if srcObj := unWrapObjectInfo(src); srcObj != nil && srcObj.Fs().Features().IsLocal {
// No buffering; directly calculate gcid from source
rc, err := srcObj.Open(ctx)
if err != nil {

View File

@@ -3368,6 +3368,10 @@ func setQuirks(opt *Options) {
opt.ChunkSize = 64 * fs.Mebi
}
useAlreadyExists = false // returns BucketAlreadyExists
// Storj doesn't support multi-part server side copy:
// https://github.com/storj/roadmap/issues/40
// So make cutoff very large which it does support
opt.CopyCutoff = math.MaxInt64
case "Synology":
useMultipartEtag = false
useAlreadyExists = false // untested
@@ -5746,7 +5750,7 @@ func (o *Object) downloadFromURL(ctx context.Context, bucketPath string, options
ContentEncoding: header("Content-Encoding"),
ContentLanguage: header("Content-Language"),
ContentType: header("Content-Type"),
StorageClass: types.StorageClass(*header("X-Amz-Storage-Class")),
StorageClass: types.StorageClass(deref(header("X-Amz-Storage-Class"))),
}
o.setMetaData(&head)
return resp.Body, err
@@ -5940,8 +5944,8 @@ func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectIn
chunkSize: int64(chunkSize),
size: size,
f: f,
bucket: mOut.Bucket,
key: mOut.Key,
bucket: ui.req.Bucket,
key: ui.req.Key,
uploadID: mOut.UploadId,
multiPartUploadInput: &mReq,
completedParts: make([]types.CompletedPart, 0),

View File

@@ -5,20 +5,13 @@ import (
"bytes"
"log"
"github.com/rclone/rclone/fs"
"github.com/sirupsen/logrus"
)
// CaptureOutput runs a function capturing its output.
func CaptureOutput(fun func()) []byte {
logSave := log.Writer()
logrusSave := logrus.StandardLogger().Writer()
defer func() {
err := logrusSave.Close()
if err != nil {
fs.Errorf(nil, "error closing logrusSave: %v", err)
}
}()
logrusSave := logrus.StandardLogger().Out
buf := &bytes.Buffer{}
log.SetOutput(buf)
logrus.SetOutput(buf)

View File

@@ -66,7 +66,8 @@ func quotePath(path string) string {
return escapePath(path, true)
}
var Colors bool // Colors controls whether terminal colors are enabled
// Colors controls whether terminal colors are enabled
var Colors bool
// Color handles terminal colors for bisync
func Color(style string, s string) string {

View File

@@ -107,7 +107,7 @@ func (lrw *loggingResponseWriter) logRequest(code int, err interface{}) {
err = ""
}
fs.LogPrintf(level, lrw.request.URL, "%s %s %d %s %s",
fs.LogLevelPrintf(level, lrw.request.URL, "%s %s %d %s %s",
lrw.request.RemoteAddr, lrw.request.Method, code,
lrw.request.Header.Get("SOAPACTION"), err)
}

View File

@@ -5,6 +5,36 @@ description: "Rclone Changelog"
# Changelog
## v1.68.2 - 2024-11-15
[See commits](https://github.com/rclone/rclone/compare/v1.68.1...v1.68.2)
* Security fixes
* local backend: CVE-2024-52522: fix permission and ownership on symlinks with `--links` and `--metadata` (Nick Craig-Wood)
* Only affects users using `--metadata` and `--links` and copying files to the local backend
* See https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
* build: bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1 (dependabot)
* This is an issue in a dependency which is used for JWT certificates
* See https://github.com/golang-jwt/jwt/security/advisories/GHSA-29wx-vh33-7x7r
* Bug Fixes
* accounting: Fix wrong message on SIGUSR2 to enable/disable bwlimit (Nick Craig-Wood)
* bisync: Fix output capture restoring the wrong output for logrus (Dimitrios Slamaris)
* dlna: Fix loggingResponseWriter disregarding log level (Simon Bos)
* serve s3: Fix excess locking which was making serve s3 single threaded (Nick Craig-Wood)
* doc fixes (Nick Craig-Wood, tgfisher, Alexandre Hamez, Randy Bush)
* Local
* Fix permission and ownership on symlinks with `--links` and `--metadata` (Nick Craig-Wood)
* Fix `--copy-links` on macOS when cloning (nielash)
* Onedrive
* Fix Retry-After handling to look at 503 errors also (Nick Craig-Wood)
* Pikpak
* Fix cid/gcid calculations for fs.OverrideRemote (wiserain)
* Fix fatal crash on startup with token that can't be refreshed (Nick Craig-Wood)
* S3
* Fix crash when using `--s3-download-url` after migration to SDKv2 (Nick Craig-Wood)
* Storj provider: fix server-side copy of files bigger than 5GB (Kaloyan Raev)
* Fix multitenant multipart uploads with CEPH (Nick Craig-Wood)
## v1.68.1 - 2024-09-24
[See commits](https://github.com/rclone/rclone/compare/v1.68.0...v1.68.1)

View File

@@ -929,7 +929,7 @@ rclone [flags]
--use-json-log Use json log format
--use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-bearer-token string Bearer token instead of user/pass (e.g. a Macaroon)

View File

@@ -1809,9 +1809,9 @@ then select "OAuth client ID".
9. It will show you a client ID and client secret. Make a note of these.
(If you selected "External" at Step 5 continue to Step 9.
(If you selected "External" at Step 5 continue to Step 10.
If you chose "Internal" you don't need to publish and can skip straight to
Step 10 but your destination drive must be part of the same Google Workspace.)
Step 11 but your destination drive must be part of the same Google Workspace.)
10. Go to "Oauth consent screen" and then click "PUBLISH APP" button and confirm.
You will also want to add yourself as a test user.

View File

@@ -505,6 +505,8 @@ processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
Lines starting with # or ; are ignored, and can be used to write comments. Inline comments are not supported. _Use `-vv --dump filters` to see how they appear in the final regexp._
E.g. for `filter-file.txt`:
# a sample filter rule file
@@ -512,6 +514,7 @@ E.g. for `filter-file.txt`:
+ *.jpg
+ *.png
+ file2.avi
- /dir/tmp/** # WARNING! This text will be treated as part of the path.
- /dir/Trash/**
+ /dir/**
# exclude everything else

View File

@@ -115,7 +115,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.68.2")
```

View File

@@ -46,7 +46,7 @@ Here is an overview of the major features of each cloud storage system.
| OpenDrive | MD5 | R/W | Yes | Partial ⁸ | - | - |
| OpenStack Swift | MD5 | R/W | No | No | R/W | - |
| Oracle Object Storage | MD5 | R/W | No | No | R/W | - |
| pCloud | MD5, SHA1 ⁷ | R | No | No | W | - |
| pCloud | MD5, SHA1 ⁷ | R/W | No | No | W | - |
| PikPak | MD5 | R | No | No | R | - |
| Pixeldrain | SHA256 | R/W | No | No | R | RW |
| premiumize.me | - | - | Yes | No | R | - |

View File

@@ -3476,8 +3476,8 @@ chunk_size = 5M
copy_cutoff = 5M
```
[C14 Cold Storage](https://www.online.net/en/storage/c14-cold-storage) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
So you can configure your remote with the `storage_class = GLACIER` option to upload directly to C14. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
[Scaleway Glacier](https://www.scaleway.com/en/glacier-cold-storage/) is the low-cost S3 Glacier alternative from Scaleway and it works the same way as on S3 by accepting the "GLACIER" `storage_class`.
So you can configure your remote with the `storage_class = GLACIER` option to upload directly to Scaleway Glacier. Don't forget that in this state you can't read files back after, you will need to restore them to "STANDARD" storage_class first before being able to read them (see "restore" section above)
### Seagate Lyve Cloud {#lyve}

View File

@@ -61,3 +61,4 @@ Thank you very much to our sponsors:
{{< sponsor src="/img/logos/warp.svg" width="300" height="200" title="Visit our sponsor warp.dev" link="https://www.warp.dev/?utm_source=rclone&utm_medium=referral&utm_campaign=rclone_20231103">}}
{{< sponsor src="/img/logos/sia.svg" width="200" height="200" title="Visit our sponsor sia" link="https://sia.tech">}}
{{< sponsor src="/img/logos/route4me.svg" width="400" height="200" title="Visit our sponsor Route4Me" link="https://route4me.com/">}}
{{< sponsor src="/img/logos/rcloneview.svg" width="300" height="200" title="Visit our sponsor RcloneView" link="https://rcloneview.com/">}}

View File

@@ -1 +1 @@
v1.68.1
v1.68.2

View File

@@ -41,7 +41,12 @@ type tokenBucket struct {
//
// Call with lock held
func (bs *buckets) _isOff() bool { //nolint:unused // Don't include unused when running golangci-lint in case its on windows where this is not called
return bs[0] == nil
for i := range bs {
if bs[i] != nil {
return false
}
}
return true
}
// Disable the limits

View File

@@ -1,4 +1,4 @@
package fs
// VersionTag of rclone
var VersionTag = "v1.68.1"
var VersionTag = "v1.68.2"

4
go.mod
View File

@@ -59,7 +59,7 @@ require (
github.com/prometheus/client_golang v1.19.1
github.com/putdotio/go-putio/putio v0.0.0-20200123120452-16d982cac2b8
github.com/quasilyte/go-ruleguard/dsl v0.3.22
github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87
github.com/rclone/gofakes3 v0.0.3
github.com/rfjakob/eme v1.1.2
github.com/rivo/uniseg v0.4.7
github.com/rogpeppe/go-internal v1.12.0
@@ -223,7 +223,7 @@ require (
require (
github.com/Microsoft/go-winio v0.6.1 // indirect
github.com/ProtonMail/go-crypto v1.0.0
github.com/golang-jwt/jwt/v4 v4.5.0
github.com/golang-jwt/jwt/v4 v4.5.1
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
github.com/pkg/xattr v0.4.9
golang.org/x/mobile v0.0.0-20240716161057-1ad2df20a8b6

8
go.sum
View File

@@ -275,8 +275,8 @@ github.com/gofrs/flock v0.8.1 h1:+gYjHKf32LDeiEEFhQaotPbLuUXjY5ZqxKgXy7n59aw=
github.com/gofrs/flock v0.8.1/go.mod h1:F1TvTiK9OcQqauNUHlbJvyl9Qa1QvF/gOUDKA14jxHU=
github.com/gogo/protobuf v1.3.2 h1:Ov1cvc58UF3b5XjBnZv7+opcTcQFZebYjWzi34vdm4Q=
github.com/gogo/protobuf v1.3.2/go.mod h1:P1XiOD3dCwIKUDQYPy72D8LYyHL2YPYrpS2s69NZV8Q=
github.com/golang-jwt/jwt/v4 v4.5.0 h1:7cYmW1XlMY7h7ii7UhUyChSgS5wUJEnm9uZVTGqOWzg=
github.com/golang-jwt/jwt/v4 v4.5.0/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v4 v4.5.1 h1:JdqV9zKUdtaa9gdPlywC3aeoEsR681PlKC+4F5gQgeo=
github.com/golang-jwt/jwt/v4 v4.5.1/go.mod h1:m21LjoU+eqJr34lmDMbreY2eSTRJ1cv77w39/MY0Ch0=
github.com/golang-jwt/jwt/v5 v5.2.1 h1:OuVbFODueb089Lh128TAcimifWaLhJwVflnrgM17wHk=
github.com/golang-jwt/jwt/v5 v5.2.1/go.mod h1:pqrtFR0X4osieyHYxtmOUWsAWrfe1Q5UVIyoH402zdk=
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
@@ -519,8 +519,8 @@ github.com/quic-go/quic-go v0.40.1 h1:X3AGzUNFs0jVuO3esAGnTfvdgvL4fq655WaOi1snv1
github.com/quic-go/quic-go v0.40.1/go.mod h1:PeN7kuVJ4xZbxSv/4OX6S1USOX8MJvydwpTx31vx60c=
github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93 h1:UVArwN/wkKjMVhh2EQGC0tEc1+FqiLlvYXY5mQ2f8Wg=
github.com/rasky/go-xdr v0.0.0-20170124162913-1a41d1a06c93/go.mod h1:Nfe4efndBz4TibWycNE+lqyJZiMX4ycx+QKV8Ta0f/o=
github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87 h1:0YRo2aYhE+SCZsjWYMFe8zLD18xieXy7wQ8M9Ywcr/g=
github.com/rclone/gofakes3 v0.0.3-0.20240807151802-e80146f8de87/go.mod h1:z7+o2VUwitO0WuVHReQlOW9jZ03LpeJ0PUFSULyTIds=
github.com/rclone/gofakes3 v0.0.3 h1:0sKCxJ8TUUAG5KXGuc/fcDKGnzB/j6IjNQui9ntIZPo=
github.com/rclone/gofakes3 v0.0.3/go.mod h1:z7+o2VUwitO0WuVHReQlOW9jZ03LpeJ0PUFSULyTIds=
github.com/relvacode/iso8601 v1.3.0 h1:HguUjsGpIMh/zsTczGN3DVJFxTU/GX+MMmzcKoMO7ko=
github.com/relvacode/iso8601 v1.3.0/go.mod h1:FlNp+jz+TXpyRqgmM7tnzHHzBnz776kmAH2h3sZCn0I=
github.com/rfjakob/eme v1.1.2 h1:SxziR8msSOElPayZNFfQw4Tjx/Sbaeeh3eRvrHVMUs4=

View File

@@ -17,7 +17,8 @@ import (
)
const (
BufferSize = 1024 * 1024 // BufferSize is the default size of the pages used in the reader
// BufferSize is the default size of the pages used in the reader
BufferSize = 1024 * 1024
bufferCacheSize = 64 // max number of buffers to keep in cache
bufferCacheFlushTime = 5 * time.Second // flush the cached buffers after this long
)

103
rclone.1 generated
View File

@@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "rclone" "1" "Sep 24, 2024" "User Manual" ""
.TH "rclone" "1" "Nov 15, 2024" "User Manual" ""
.hy
.SH Rclone syncs your files to cloud storage
.PP
@@ -20955,6 +20955,12 @@ See above for the order filter flags are processed in.
Arrange the order of filter rules with the most restrictive first and
work down.
.PP
Lines starting with # or ; are ignored, and can be used to write
comments.
Inline comments are not supported.
\f[I]Use \f[CI]-vv --dump filters\f[I] to see how they appear in the
final regexp.\f[R]
.PP
E.g.
for \f[C]filter-file.txt\f[R]:
.IP
@@ -20965,6 +20971,7 @@ for \f[C]filter-file.txt\f[R]:
+ *.jpg
+ *.png
+ file2.avi
- /dir/tmp/** # WARNING! This text will be treated as part of the path.
- /dir/Trash/**
+ /dir/**
# exclude everything else
@@ -24965,7 +24972,7 @@ pCloud
T}@T{
MD5, SHA1 \[u2077]
T}@T{
R
R/W
T}@T{
No
T}@T{
@@ -27719,7 +27726,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.68.1\[dq])
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.68.2\[dq])
\f[R]
.fi
.SS Performance
@@ -37393,11 +37400,12 @@ copy_cutoff = 5M
\f[R]
.fi
.PP
C14 Cold Storage (https://www.online.net/en/storage/c14-cold-storage) is
Scaleway Glacier (https://www.scaleway.com/en/glacier-cold-storage/) is
the low-cost S3 Glacier alternative from Scaleway and it works the same
way as on S3 by accepting the \[dq]GLACIER\[dq] \f[C]storage_class\f[R].
So you can configure your remote with the
\f[C]storage_class = GLACIER\f[R] option to upload directly to C14.
\f[C]storage_class = GLACIER\f[R] option to upload directly to Scaleway
Glacier.
Don\[aq]t forget that in this state you can\[aq]t read files back after,
you will need to restore them to \[dq]STANDARD\[dq] storage_class first
before being able to read them (see \[dq]restore\[dq] section above)
@@ -50216,9 +50224,9 @@ It will show you a client ID and client secret.
Make a note of these.
.RS 4
.PP
(If you selected \[dq]External\[dq] at Step 5 continue to Step 9.
(If you selected \[dq]External\[dq] at Step 5 continue to Step 10.
If you chose \[dq]Internal\[dq] you don\[aq]t need to publish and can
skip straight to Step 10 but your destination drive must be part of the
skip straight to Step 11 but your destination drive must be part of the
same Google Workspace.)
.RE
.IP "10." 4
@@ -72522,6 +72530,87 @@ Options:
.IP \[bu] 2
\[dq]error\[dq]: return an error based on option value
.SH Changelog
.SS v1.68.2 - 2024-11-15
.PP
See commits (https://github.com/rclone/rclone/compare/v1.68.1...v1.68.2)
.IP \[bu] 2
Security fixes
.RS 2
.IP \[bu] 2
local backend: CVE-2024-52522: fix permission and ownership on symlinks
with \f[C]--links\f[R] and \f[C]--metadata\f[R] (Nick Craig-Wood)
.RS 2
.IP \[bu] 2
Only affects users using \f[C]--metadata\f[R] and \f[C]--links\f[R] and
copying files to the local backend
.IP \[bu] 2
See
https://github.com/rclone/rclone/security/advisories/GHSA-hrxh-9w67-g4cv
.RE
.IP \[bu] 2
build: bump github.com/golang-jwt/jwt/v4 from 4.5.0 to 4.5.1
(dependabot)
.RS 2
.IP \[bu] 2
This is an issue in a dependency which is used for JWT certificates
.IP \[bu] 2
See
https://github.com/golang-jwt/jwt/security/advisories/GHSA-29wx-vh33-7x7r
.RE
.RE
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
accounting: Fix wrong message on SIGUSR2 to enable/disable bwlimit (Nick
Craig-Wood)
.IP \[bu] 2
bisync: Fix output capture restoring the wrong output for logrus
(Dimitrios Slamaris)
.IP \[bu] 2
dlna: Fix loggingResponseWriter disregarding log level (Simon Bos)
.IP \[bu] 2
serve s3: Fix excess locking which was making serve s3 single threaded
(Nick Craig-Wood)
.IP \[bu] 2
doc fixes (Nick Craig-Wood, tgfisher, Alexandre Hamez, Randy Bush)
.RE
.IP \[bu] 2
Local
.RS 2
.IP \[bu] 2
Fix permission and ownership on symlinks with \f[C]--links\f[R] and
\f[C]--metadata\f[R] (Nick Craig-Wood)
.IP \[bu] 2
Fix \f[C]--copy-links\f[R] on macOS when cloning (nielash)
.RE
.IP \[bu] 2
Onedrive
.RS 2
.IP \[bu] 2
Fix Retry-After handling to look at 503 errors also (Nick Craig-Wood)
.RE
.IP \[bu] 2
Pikpak
.RS 2
.IP \[bu] 2
Fix cid/gcid calculations for fs.OverrideRemote (wiserain)
.IP \[bu] 2
Fix fatal crash on startup with token that can\[aq]t be refreshed (Nick
Craig-Wood)
.RE
.IP \[bu] 2
S3
.RS 2
.IP \[bu] 2
Fix crash when using \f[C]--s3-download-url\f[R] after migration to
SDKv2 (Nick Craig-Wood)
.IP \[bu] 2
Storj provider: fix server-side copy of files bigger than 5GB (Kaloyan
Raev)
.IP \[bu] 2
Fix multitenant multipart uploads with CEPH (Nick Craig-Wood)
.RE
.SS v1.68.1 - 2024-09-24
.PP
See commits (https://github.com/rclone/rclone/compare/v1.68.0...v1.68.1)