1
0
mirror of https://github.com/rclone/rclone.git synced 2025-12-26 21:23:15 +00:00

Compare commits

..

29 Commits

Author SHA1 Message Date
Nick Craig-Wood
4133a197bc Version v1.70.3 2025-07-09 10:51:25 +01:00
Nick Craig-Wood
a30a4909fe azureblob: fix server side copy error "requires exactly one scope"
Before this change, if not using shared key or SAS URL authentication
for the source, rclone gave this error

    ManagedIdentityCredential.GetToken() requires exactly one scope

when doing server side copies.

This was introduced in:

3a5ddfcd3c azureblob: implement multipart server side copy

This fixes the problem by creating a temporary SAS URL using user
delegation to read the source blob when copying.

Fixes #8662
2025-07-09 10:32:12 +01:00
albertony
cdc6d22929 docs: explain the json log format in more detail 2025-07-09 10:32:12 +01:00
albertony
e319406f52 check: fix difference report (was reporting error counts) 2025-07-09 10:32:12 +01:00
Nick Craig-Wood
ac54cccced linkbox: fix upload error "user upload file not exist"
Linkbox have started issuing 302 redirects on some of their PUT
requests when rclone uploads a file.

This is problematic for several reasons:

1. This is the wrong redirect code - it should be 307 to preserve the method
2. Since Expect/100-Continue isn't supported the whole body gets uploaded

This fixes the problem by first doing a HEAD request on the URL. This
will allow us to read the redirect Location and not upload the body to
the wrong place.

It should still work (albeit a little more inefficiently) if Linkbox
stop redirecting the PUT requests.

See: https://forum.rclone.org/t/linkbox-upload-error/51795
Fixes: #8606
2025-07-09 10:32:12 +01:00
Nick Craig-Wood
4c4d366e29 march: fix deadlock when using --no-traverse - fixes #8656
This ocurred whenever there were more than 100 files in the source due
to the output channel filling up.

The fix is not to use list.NewSorter but take more care to output the
dst objects in the same order the src objects are delivered. As the
src objects are delivered sorted, no sorting is needed.

In order not to cause another deadlock, we need to send nil dst
objects which is safe since this adjusts the termination conditions
for the channels.

Thanks to @jeremy for the test script the Go tests are based on.
2025-07-09 10:32:12 +01:00
wiserain
64fc3d05ae pikpak: improve error handling for missing links and unrecoverable 500s
This commit improves error handling in two specific scenarios:

* Missing Download Links: A 5-second delay is introduced when a download
  link is missing, as low-level retries aren't enough. Empirically, it
  takes about 30s-1m for the link to become available. This resolves
  failed integration tests: backend: TestIntegration/FsMkdir/FsPutFiles/
  ObjectUpdate, vfs: TestFileReadAtNonZeroLength

* Unrecoverable 500 Errors: The shouldRetry method is updated to skip
  retries for 500 errors from "idx.shub.mypikpak.com" indicating "no
  record for gcid." These errors are non-recoverable, so retrying is futile.
2025-07-09 10:32:12 +01:00
WeidiDeng
90386efeb1 webdav: fix setting modtime to that of local object instead of remote
In this commit the source of the modtime got changed to the wrong object by accident

0b9671313b webdav: add an ownCloud Infinite Scale vendor that enables tus chunked upload support

This reverts that change and fixes the integration tests.
2025-07-09 10:32:12 +01:00
Davide Bizzarri
5f78b47295 fix: b2 versionAt read metadata 2025-07-09 10:32:12 +01:00
Nick Craig-Wood
775ee90fa5 Start v1.70.3-DEV development 2025-07-02 15:36:43 +01:00
Nick Craig-Wood
444392bf9c docs: fix filescom/filelu link mixup
See: https://forum.rclone.org/t/a-small-bug-in-rclone-documentation/51774
2025-07-02 15:35:18 +01:00
Nick Craig-Wood
d36259749f docs: update link for filescom 2025-06-30 11:10:31 +01:00
Nick Craig-Wood
4010380ea8 Version v1.70.2 2025-06-27 12:30:18 +01:00
Ali Zein Yousuf
c138e52a57 docs: update client ID instructions to current Azure AD portal - fixes #8027 2025-06-27 12:23:00 +01:00
necaran
e22ce597ad mega: fix tls handshake failure - fixes #8565
The cipher suites used by Mega's storage endpoints: https://github.com/meganz/webclient/issues/103
are no longer supported by default since Go 1.22: https://tip.golang.org/doc/go1.22#minor_library_changes
This therefore assigns the cipher suites explicitly to include the one Mega needs.
2025-06-26 17:06:03 +01:00
Nick Craig-Wood
79bd9e7913 pacer: fix nil pointer deref in RetryError - fixes #8077
Before this change, if RetryAfterError was called with a nil err, then
it's Error method would return this when wrapped in a fmt.Errorf
statement

    error %!v(PANIC=Error method: runtime error: invalid memory address or nil pointer dereference))

Looking at the code, it looks like RetryAfterError will usually be
called with a nil pointer, so this patch makes sure it has a sensible
error.
2025-06-26 17:05:37 +01:00
nielash
32f9393ac8 convmv: fix moving to unicode-equivalent name - fixes #8634
Before this change, using convmv to convert filenames between NFD and NFC could
fail on certain backends (such as onedrive) that were insensitive to the
difference. This change fixes the issue by extending the existing
needsMoveCaseInsensitive logic for use in this scenario.
2025-06-26 17:05:37 +01:00
nielash
f97c876eb1 convmv: make --dry-run logs less noisy
Before this change, convmv dry runs would log a SkipDestructive message for
every single object, even objects that would not really be moved during a real
run. This made it quite difficult to tell what would actually happen during the
real run. This change fixes that by returning silently in such cases (as would
happen during a real run.)
2025-06-26 17:05:37 +01:00
nielash
9b43836e19 sync: avoid copying dir metadata to itself
In convmv, src and dst can point to the same directory. Unless a dir's name is
changing, we should leave it alone and not attempt to copy its metadata to
itself.
2025-06-26 17:05:37 +01:00
Nick Craig-Wood
ff817e8764 combine: fix directory not found errors with ListP interface - Fixes #8627
In

b1d774c2e3 combine: implement ListP interface

We introduced the ListP interface to the combine backend. This was
passing the wrong remote to the upstreams. This was picked up by the
integration tests but was ignored by accident.
2025-06-26 17:05:37 +01:00
Nick Craig-Wood
3c63dec849 local: fix --skip-links on Windows when skipping Junction points
Due to a change in Go which was enabled by the `go 1.22` in `go.mod`
rclone has stopped skipping junction points ("My Documents" in
particular) if `--skip-links` is set on Windows.

This is because the output from os.Lstat has changed and junction
points are no longer marked with os.ModeSymlink but with
os.ModeIrregular instead.

This fix now skips os.ModeIrregular objects if --skip-links is set on
Windows only.

Fixes #8561
See: https://github.com/golang/go/issues/73827
2025-06-26 17:05:37 +01:00
dependabot[bot]
33876c5806 build: bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix GHSA-vrw8-fxc6-2r93
See: https://github.com/go-chi/chi/security/advisories/GHSA-vrw8-fxc6-2r93
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2025-06-26 17:05:37 +01:00
Nick Craig-Wood
fa3b444341 log: fix deadlock when using systemd logging - fixes #8621
In this commit the logging system was re-worked

dfa4d94827 fs: Remove github.com/sirupsen/logrus and replace with log/slog

Unfortunately the systemd logging was still using the plain log
package and this caused a deadlock as it was recursively calling the
logging package.

The fix was to use the dedicated systemd journal logging routines in
the process removing a TODO!
2025-06-26 17:05:37 +01:00
Nick Craig-Wood
e5fc424955 docs: googlephotos: detail how to make your own client_id - fixes #8622 2025-06-26 17:05:37 +01:00
Nick Craig-Wood
06badeffa3 pikpak: fix uploads fail with "aws-chunked encoding is not supported" error
This downgrades the AWS SDK slightly (this is still an upgrade from
rclone v1.69.3) to work around a breakage in the upstream SDK when
used with pikpak. This isn't a long term solution - either they will
fix it upstream or we will implement a workaround.

See: https://github.com/aws/aws-sdk-go-v2/issues/3007
See: #8629
2025-06-26 16:58:02 +01:00
Nick Craig-Wood
eb71d1be18 Start v1.70.2-DEV development 2025-06-25 16:40:16 +01:00
Nick Craig-Wood
7506a3c84c docs: Remove Warp as a sponsor 2025-06-25 16:38:25 +01:00
Nick Craig-Wood
831abd3406 docs: add files.com as a Gold sponsor 2025-06-25 16:38:25 +01:00
Nick Craig-Wood
9c08cd80c7 docs: add links to SecureBuild docker image 2025-06-25 16:38:25 +01:00
41 changed files with 942 additions and 193 deletions

210
MANUAL.html generated
View File

@@ -81,7 +81,7 @@
<header id="title-block-header">
<h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p>
<p class="date">Jun 19, 2025</p>
<p class="date">Jul 09, 2025</p>
</header>
<h1 id="name">NAME</h1>
<p>rclone - manage files on cloud storage</p>
@@ -222,8 +222,8 @@ Use &quot;rclone help backends&quot; for a list of supported services.
<li>Dropbox</li>
<li>Enterprise File Fabric</li>
<li>Fastmail Files</li>
<li>Files.com</li>
<li>FileLu Cloud Storage</li>
<li>Files.com</li>
<li>FlashBlade</li>
<li>FTP</li>
<li>Gofile</li>
@@ -426,6 +426,7 @@ choco install rclone</code></pre>
<p><a href="https://repology.org/project/rclone/versions"><img src="https://repology.org/badge/vertical-allrepos/rclone.svg?columns=3" alt="Packaging status" /></a></p>
<h2 id="docker">Docker installation</h2>
<p>The rclone developers maintain a <a href="https://hub.docker.com/r/rclone/rclone">docker image for rclone</a>.</p>
<p><strong>Note:</strong> We also now offer a paid version of rclone with enterprise-grade security and zero CVEs through our partner <a href="https://securebuild.com/blog/introducing-securebuild">SecureBuild</a>. If you are interested, check out their website and the <a href="https://securebuild.com/images/rclone">Rclone SecureBuild Image</a>.</p>
<p>These images are built as part of the release process based on a minimal Alpine Linux.</p>
<p>The <code>:latest</code> tag will always point to the latest stable release. You can use the <code>:beta</code> tag to get the latest build from master. You can also use version tags, e.g. <code>:1.49.1</code>, <code>:1.49</code> or <code>:1</code>.</p>
<pre><code>$ docker pull rclone/rclone:latest
@@ -8257,7 +8258,8 @@ y/n/s/!/q&gt; n</code></pre>
<pre><code>--log-file rclone.log --log-level DEBUG --windows-event-log ERROR</code></pre>
<p>This option is only supported Windows platforms.</p>
<h3 id="use-json-log">--use-json-log</h3>
<p>This switches the log format to JSON for rclone. The fields of JSON log are <code>level</code>, <code>msg</code>, <code>source</code>, <code>time</code>. The JSON logs will be printed on a single line, but are shown expanded here for clarity.</p>
<p>This switches the log format to JSON. The log messages are then streamed as individual JSON objects, with fields: <code>level</code>, <code>msg</code>, <code>source</code>, and <code>time</code>. The resulting format is what is sometimes referred to as <a href="https://en.wikipedia.org/wiki/JSON_streaming#Newline-delimited_JSON">newline-delimited JSON</a> (NDJSON), or JSON Lines (JSONL). This is well suited for processing by traditional line-oriented tools and shell pipelines, but a complete log file is not strictly valid JSON and needs a parser that can handle it.</p>
<p>The JSON logs will be printed on a single line, but are shown expanded here for clarity.</p>
<div class="sourceCode" id="cb654"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb654-1"><a href="#cb654-1" aria-hidden="true"></a><span class="fu">{</span></span>
<span id="cb654-2"><a href="#cb654-2" aria-hidden="true"></a> <span class="dt">&quot;time&quot;</span><span class="fu">:</span> <span class="st">&quot;2025-05-13T17:30:51.036237518+01:00&quot;</span><span class="fu">,</span></span>
<span id="cb654-3"><a href="#cb654-3" aria-hidden="true"></a> <span class="dt">&quot;level&quot;</span><span class="fu">:</span> <span class="st">&quot;debug&quot;</span><span class="fu">,</span></span>
@@ -13230,7 +13232,7 @@ Showing nodes accounting for 1537.03kB, 100% of 1537.03kB total
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.70.1&quot;)</code></pre>
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.70.3&quot;)</code></pre>
<h2 id="performance">Performance</h2>
<p>Flags helpful for increasing performance.</p>
<pre><code> --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -27604,6 +27606,13 @@ y/e/d&gt; y</code></pre>
<p>Rclone cannot delete files anywhere except under <code>album</code>.</p>
<h3 id="deleting-albums">Deleting albums</h3>
<p>The Google Photos API does not support deleting albums - see <a href="https://issuetracker.google.com/issues/135714733">bug #135714733</a>.</p>
<h2 id="making-your-own-client_id-1">Making your own client_id</h2>
<p>When you use rclone with Google photos in its default configuration you are using rclone's client_id. This is shared between all the rclone users. There is a global rate limit on the number of queries per second that each client_id can do set by Google.</p>
<p>If there is a problem with this client_id (eg quota too low or the client_id stops working) then you can make your own.</p>
<p>Please follow the steps in <a href="https://rclone.org/drive/#making-your-own-client-id">the google drive docs</a>. You will need these scopes instead of the drive ones detailed:</p>
<pre><code>https://www.googleapis.com/auth/photoslibrary.appendonly
https://www.googleapis.com/auth/photoslibrary.readonly.appcreateddata
https://www.googleapis.com/auth/photoslibrary.edit.appcreateddata</code></pre>
<h1 id="hasher">Hasher</h1>
<p>Hasher is a special overlay backend to create remotes which handle checksums for other remotes. It's main functions include: - Emulate hash types unimplemented by backends - Cache checksums to help with slow hashing of large local or (S)FTP files - Warm up checksum cache from external SUM files</p>
<h2 id="getting-started-1">Getting started</h2>
@@ -32075,7 +32084,10 @@ y/e/d&gt; y</code></pre>
<h4 id="creating-client-id-for-onedrive-personal">Creating Client ID for OneDrive Personal</h4>
<p>To create your own Client ID, please follow these steps:</p>
<ol type="1">
<li>Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade and then click <code>New registration</code>.</li>
<li>Open https://portal.azure.com/?quickstart=true#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview and then under the <code>Add</code> menu click <code>App registration</code>.
<ul>
<li>If you have not created an Azure account, you will be prompted to. This is free, but you need to provide a phone number, address, and credit card for identity verification.</li>
</ul></li>
<li>Enter a name for your app, choose account type <code>Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)</code>, select <code>Web</code> in <code>Redirect URI</code>, then type (do not copy and paste) <code>http://localhost:53682/</code> and click Register. Copy and keep the <code>Application (client) ID</code> under the app name for later use.</li>
<li>Under <code>manage</code> select <code>Certificates &amp; secrets</code>, click <code>New client secret</code>. Enter a description (can be anything) and set <code>Expires</code> to 24 months. Copy and keep that secret <em>Value</em> for later use (you <em>won't</em> be able to see this value afterwards).</li>
<li>Under <code>manage</code> select <code>API permissions</code>, click <code>Add a permission</code> and select <code>Microsoft Graph</code> then select <code>delegated permissions</code>.</li>
@@ -32647,75 +32659,75 @@ rclone rc vfs/refresh recursive=true</code></pre>
<p>Permissions are also supported, if <code>--onedrive-metadata-permissions</code> is set. The accepted values for <code>--onedrive-metadata-permissions</code> are "<code>read</code>", "<code>write</code>", "<code>read,write</code>", and "<code>off</code>" (the default). "<code>write</code>" supports adding new permissions, updating the "role" of existing permissions, and removing permissions. Updating and removing require the Permission ID to be known, so it is recommended to use "<code>read,write</code>" instead of "<code>write</code>" if you wish to update/remove permissions.</p>
<p>Permissions are read/written in JSON format using the same schema as the <a href="https://learn.microsoft.com/en-us/onedrive/developer/rest-api/resources/permission?view=odsp-graph-online">OneDrive API</a>, which differs slightly between OneDrive Personal and Business.</p>
<p>Example for OneDrive Personal:</p>
<div class="sourceCode" id="cb1405"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb1405-1"><a href="#cb1405-1" aria-hidden="true"></a><span class="ot">[</span></span>
<span id="cb1405-2"><a href="#cb1405-2" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1405-3"><a href="#cb1405-3" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;1234567890ABC!123&quot;</span><span class="fu">,</span></span>
<span id="cb1405-4"><a href="#cb1405-4" aria-hidden="true"></a> <span class="dt">&quot;grantedTo&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1405-5"><a href="#cb1405-5" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1405-6"><a href="#cb1405-6" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1405-7"><a href="#cb1405-7" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1405-8"><a href="#cb1405-8" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1405-9"><a href="#cb1405-9" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1405-10"><a href="#cb1405-10" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1405-11"><a href="#cb1405-11" aria-hidden="true"></a> <span class="dt">&quot;invitation&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1405-12"><a href="#cb1405-12" aria-hidden="true"></a> <span class="dt">&quot;email&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1405-13"><a href="#cb1405-13" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1405-14"><a href="#cb1405-14" aria-hidden="true"></a> <span class="dt">&quot;link&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1405-15"><a href="#cb1405-15" aria-hidden="true"></a> <span class="dt">&quot;webUrl&quot;</span><span class="fu">:</span> <span class="st">&quot;https://1drv.ms/t/s!1234567890ABC&quot;</span></span>
<span id="cb1405-16"><a href="#cb1405-16" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1405-17"><a href="#cb1405-17" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1405-18"><a href="#cb1405-18" aria-hidden="true"></a> <span class="st">&quot;read&quot;</span></span>
<span id="cb1405-19"><a href="#cb1405-19" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1405-20"><a href="#cb1405-20" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;s!1234567890ABC&quot;</span></span>
<span id="cb1405-21"><a href="#cb1405-21" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1405-22"><a href="#cb1405-22" aria-hidden="true"></a><span class="ot">]</span></span></code></pre></div>
<p>Example for OneDrive Business:</p>
<div class="sourceCode" id="cb1406"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb1406-1"><a href="#cb1406-1" aria-hidden="true"></a><span class="ot">[</span></span>
<span id="cb1406-2"><a href="#cb1406-2" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1406-3"><a href="#cb1406-3" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;48d31887-5fad-4d73-a9f5-3c356e68a038&quot;</span><span class="fu">,</span></span>
<span id="cb1406-4"><a href="#cb1406-4" aria-hidden="true"></a> <span class="dt">&quot;grantedToIdentities&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1406-5"><a href="#cb1406-5" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1406-6"><a href="#cb1406-6" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-7"><a href="#cb1406-7" aria-hidden="true"></a> <span class="dt">&quot;displayName&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1406-8"><a href="#cb1406-8" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-9"><a href="#cb1406-9" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1406-10"><a href="#cb1406-10" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1406-11"><a href="#cb1406-11" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1406-12"><a href="#cb1406-12" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1406-13"><a href="#cb1406-13" aria-hidden="true"></a> <span class="dt">&quot;link&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-14"><a href="#cb1406-14" aria-hidden="true"></a> <span class="dt">&quot;type&quot;</span><span class="fu">:</span> <span class="st">&quot;view&quot;</span><span class="fu">,</span></span>
<span id="cb1406-15"><a href="#cb1406-15" aria-hidden="true"></a> <span class="dt">&quot;scope&quot;</span><span class="fu">:</span> <span class="st">&quot;users&quot;</span><span class="fu">,</span></span>
<span id="cb1406-16"><a href="#cb1406-16" aria-hidden="true"></a> <span class="dt">&quot;webUrl&quot;</span><span class="fu">:</span> <span class="st">&quot;https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s&quot;</span></span>
<span id="cb1406-17"><a href="#cb1406-17" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-18"><a href="#cb1406-18" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1406-19"><a href="#cb1406-19" aria-hidden="true"></a> <span class="st">&quot;read&quot;</span></span>
<span id="cb1406-20"><a href="#cb1406-20" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1406-21"><a href="#cb1406-21" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;u!LKj1lkdlals90j1nlkascl&quot;</span></span>
<span id="cb1406-22"><a href="#cb1406-22" aria-hidden="true"></a> <span class="fu">}</span><span class="ot">,</span></span>
<span id="cb1406-23"><a href="#cb1406-23" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1406-24"><a href="#cb1406-24" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;5D33DD65C6932946&quot;</span><span class="fu">,</span></span>
<span id="cb1406-25"><a href="#cb1406-25" aria-hidden="true"></a> <span class="dt">&quot;grantedTo&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-26"><a href="#cb1406-26" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-27"><a href="#cb1406-27" aria-hidden="true"></a> <span class="dt">&quot;displayName&quot;</span><span class="fu">:</span> <span class="st">&quot;John Doe&quot;</span><span class="fu">,</span></span>
<span id="cb1406-28"><a href="#cb1406-28" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;efee1b77-fb3b-4f65-99d6-274c11914d12&quot;</span></span>
<span id="cb1406-29"><a href="#cb1406-29" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-30"><a href="#cb1406-30" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1406-31"><a href="#cb1406-31" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1406-32"><a href="#cb1406-32" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-33"><a href="#cb1406-33" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1406-34"><a href="#cb1406-34" aria-hidden="true"></a> <span class="st">&quot;owner&quot;</span></span>
<span id="cb1406-35"><a href="#cb1406-35" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1406-36"><a href="#cb1406-36" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U&quot;</span></span>
<span id="cb1406-37"><a href="#cb1406-37" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1406-38"><a href="#cb1406-38" aria-hidden="true"></a><span class="ot">]</span></span></code></pre></div>
<span id="cb1406-3"><a href="#cb1406-3" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;1234567890ABC!123&quot;</span><span class="fu">,</span></span>
<span id="cb1406-4"><a href="#cb1406-4" aria-hidden="true"></a> <span class="dt">&quot;grantedTo&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-5"><a href="#cb1406-5" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-6"><a href="#cb1406-6" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1406-7"><a href="#cb1406-7" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-8"><a href="#cb1406-8" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1406-9"><a href="#cb1406-9" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1406-10"><a href="#cb1406-10" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-11"><a href="#cb1406-11" aria-hidden="true"></a> <span class="dt">&quot;invitation&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-12"><a href="#cb1406-12" aria-hidden="true"></a> <span class="dt">&quot;email&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1406-13"><a href="#cb1406-13" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-14"><a href="#cb1406-14" aria-hidden="true"></a> <span class="dt">&quot;link&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1406-15"><a href="#cb1406-15" aria-hidden="true"></a> <span class="dt">&quot;webUrl&quot;</span><span class="fu">:</span> <span class="st">&quot;https://1drv.ms/t/s!1234567890ABC&quot;</span></span>
<span id="cb1406-16"><a href="#cb1406-16" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1406-17"><a href="#cb1406-17" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1406-18"><a href="#cb1406-18" aria-hidden="true"></a> <span class="st">&quot;read&quot;</span></span>
<span id="cb1406-19"><a href="#cb1406-19" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1406-20"><a href="#cb1406-20" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;s!1234567890ABC&quot;</span></span>
<span id="cb1406-21"><a href="#cb1406-21" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1406-22"><a href="#cb1406-22" aria-hidden="true"></a><span class="ot">]</span></span></code></pre></div>
<p>Example for OneDrive Business:</p>
<div class="sourceCode" id="cb1407"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb1407-1"><a href="#cb1407-1" aria-hidden="true"></a><span class="ot">[</span></span>
<span id="cb1407-2"><a href="#cb1407-2" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1407-3"><a href="#cb1407-3" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;48d31887-5fad-4d73-a9f5-3c356e68a038&quot;</span><span class="fu">,</span></span>
<span id="cb1407-4"><a href="#cb1407-4" aria-hidden="true"></a> <span class="dt">&quot;grantedToIdentities&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1407-5"><a href="#cb1407-5" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1407-6"><a href="#cb1407-6" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1407-7"><a href="#cb1407-7" aria-hidden="true"></a> <span class="dt">&quot;displayName&quot;</span><span class="fu">:</span> <span class="st">&quot;ryan@contoso.com&quot;</span></span>
<span id="cb1407-8"><a href="#cb1407-8" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1407-9"><a href="#cb1407-9" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1407-10"><a href="#cb1407-10" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1407-11"><a href="#cb1407-11" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1407-12"><a href="#cb1407-12" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1407-13"><a href="#cb1407-13" aria-hidden="true"></a> <span class="dt">&quot;link&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1407-14"><a href="#cb1407-14" aria-hidden="true"></a> <span class="dt">&quot;type&quot;</span><span class="fu">:</span> <span class="st">&quot;view&quot;</span><span class="fu">,</span></span>
<span id="cb1407-15"><a href="#cb1407-15" aria-hidden="true"></a> <span class="dt">&quot;scope&quot;</span><span class="fu">:</span> <span class="st">&quot;users&quot;</span><span class="fu">,</span></span>
<span id="cb1407-16"><a href="#cb1407-16" aria-hidden="true"></a> <span class="dt">&quot;webUrl&quot;</span><span class="fu">:</span> <span class="st">&quot;https://contoso.sharepoint.com/:w:/t/design/a577ghg9hgh737613bmbjf839026561fmzhsr85ng9f3hjck2t5s&quot;</span></span>
<span id="cb1407-17"><a href="#cb1407-17" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1407-18"><a href="#cb1407-18" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1407-19"><a href="#cb1407-19" aria-hidden="true"></a> <span class="st">&quot;read&quot;</span></span>
<span id="cb1407-20"><a href="#cb1407-20" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1407-21"><a href="#cb1407-21" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;u!LKj1lkdlals90j1nlkascl&quot;</span></span>
<span id="cb1407-22"><a href="#cb1407-22" aria-hidden="true"></a> <span class="fu">}</span><span class="ot">,</span></span>
<span id="cb1407-23"><a href="#cb1407-23" aria-hidden="true"></a> <span class="fu">{</span></span>
<span id="cb1407-24"><a href="#cb1407-24" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;5D33DD65C6932946&quot;</span><span class="fu">,</span></span>
<span id="cb1407-25"><a href="#cb1407-25" aria-hidden="true"></a> <span class="dt">&quot;grantedTo&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1407-26"><a href="#cb1407-26" aria-hidden="true"></a> <span class="dt">&quot;user&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1407-27"><a href="#cb1407-27" aria-hidden="true"></a> <span class="dt">&quot;displayName&quot;</span><span class="fu">:</span> <span class="st">&quot;John Doe&quot;</span><span class="fu">,</span></span>
<span id="cb1407-28"><a href="#cb1407-28" aria-hidden="true"></a> <span class="dt">&quot;id&quot;</span><span class="fu">:</span> <span class="st">&quot;efee1b77-fb3b-4f65-99d6-274c11914d12&quot;</span></span>
<span id="cb1407-29"><a href="#cb1407-29" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1407-30"><a href="#cb1407-30" aria-hidden="true"></a> <span class="dt">&quot;application&quot;</span><span class="fu">:</span> <span class="fu">{},</span></span>
<span id="cb1407-31"><a href="#cb1407-31" aria-hidden="true"></a> <span class="dt">&quot;device&quot;</span><span class="fu">:</span> <span class="fu">{}</span></span>
<span id="cb1407-32"><a href="#cb1407-32" aria-hidden="true"></a> <span class="fu">},</span></span>
<span id="cb1407-33"><a href="#cb1407-33" aria-hidden="true"></a> <span class="dt">&quot;roles&quot;</span><span class="fu">:</span> <span class="ot">[</span></span>
<span id="cb1407-34"><a href="#cb1407-34" aria-hidden="true"></a> <span class="st">&quot;owner&quot;</span></span>
<span id="cb1407-35"><a href="#cb1407-35" aria-hidden="true"></a> <span class="ot">]</span><span class="fu">,</span></span>
<span id="cb1407-36"><a href="#cb1407-36" aria-hidden="true"></a> <span class="dt">&quot;shareId&quot;</span><span class="fu">:</span> <span class="st">&quot;FWxc1lasfdbEAGM5fI7B67aB5ZMPDMmQ11U&quot;</span></span>
<span id="cb1407-37"><a href="#cb1407-37" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1407-38"><a href="#cb1407-38" aria-hidden="true"></a><span class="ot">]</span></span></code></pre></div>
<p>To write permissions, pass in a "permissions" metadata key using this same format. The <a href="https://rclone.org/docs/#metadata-mapper"><code>--metadata-mapper</code></a> tool can be very helpful for this.</p>
<p>When adding permissions, an email address can be provided in the <code>User.ID</code> or <code>DisplayName</code> properties of <code>grantedTo</code> or <code>grantedToIdentities</code>. Alternatively, an ObjectID can be provided in <code>User.ID</code>. At least one valid recipient must be provided in order to add a permission for a user. Creating a Public Link is also supported, if <code>Link.Scope</code> is set to <code>"anonymous"</code>.</p>
<p>Example request to add a "read" permission with <code>--metadata-mapper</code>:</p>
<div class="sourceCode" id="cb1407"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb1407-1"><a href="#cb1407-1" aria-hidden="true"></a><span class="fu">{</span></span>
<span id="cb1407-2"><a href="#cb1407-2" aria-hidden="true"></a> <span class="dt">&quot;Metadata&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1407-3"><a href="#cb1407-3" aria-hidden="true"></a> <span class="dt">&quot;permissions&quot;</span><span class="fu">:</span> <span class="st">&quot;[{</span><span class="ch">\&quot;</span><span class="st">grantedToIdentities</span><span class="ch">\&quot;</span><span class="st">:[{</span><span class="ch">\&quot;</span><span class="st">user</span><span class="ch">\&quot;</span><span class="st">:{</span><span class="ch">\&quot;</span><span class="st">id</span><span class="ch">\&quot;</span><span class="st">:</span><span class="ch">\&quot;</span><span class="st">ryan@contoso.com</span><span class="ch">\&quot;</span><span class="st">}}],</span><span class="ch">\&quot;</span><span class="st">roles</span><span class="ch">\&quot;</span><span class="st">:[</span><span class="ch">\&quot;</span><span class="st">read</span><span class="ch">\&quot;</span><span class="st">]}]&quot;</span></span>
<span id="cb1407-4"><a href="#cb1407-4" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1407-5"><a href="#cb1407-5" aria-hidden="true"></a><span class="fu">}</span></span></code></pre></div>
<div class="sourceCode" id="cb1408"><pre class="sourceCode json"><code class="sourceCode json"><span id="cb1408-1"><a href="#cb1408-1" aria-hidden="true"></a><span class="fu">{</span></span>
<span id="cb1408-2"><a href="#cb1408-2" aria-hidden="true"></a> <span class="dt">&quot;Metadata&quot;</span><span class="fu">:</span> <span class="fu">{</span></span>
<span id="cb1408-3"><a href="#cb1408-3" aria-hidden="true"></a> <span class="dt">&quot;permissions&quot;</span><span class="fu">:</span> <span class="st">&quot;[{</span><span class="ch">\&quot;</span><span class="st">grantedToIdentities</span><span class="ch">\&quot;</span><span class="st">:[{</span><span class="ch">\&quot;</span><span class="st">user</span><span class="ch">\&quot;</span><span class="st">:{</span><span class="ch">\&quot;</span><span class="st">id</span><span class="ch">\&quot;</span><span class="st">:</span><span class="ch">\&quot;</span><span class="st">ryan@contoso.com</span><span class="ch">\&quot;</span><span class="st">}}],</span><span class="ch">\&quot;</span><span class="st">roles</span><span class="ch">\&quot;</span><span class="st">:[</span><span class="ch">\&quot;</span><span class="st">read</span><span class="ch">\&quot;</span><span class="st">]}]&quot;</span></span>
<span id="cb1408-4"><a href="#cb1408-4" aria-hidden="true"></a> <span class="fu">}</span></span>
<span id="cb1408-5"><a href="#cb1408-5" aria-hidden="true"></a><span class="fu">}</span></span></code></pre></div>
<p>Note that adding a permission can fail if a conflicting permission already exists for the file/folder.</p>
<p>To update an existing permission, include both the Permission ID and the new <code>roles</code> to be assigned. <code>roles</code> is the only property that can be changed.</p>
<p>To remove permissions, pass in a blob containing only the permissions you wish to keep (which can be empty, to remove all.) Note that the <code>owner</code> role will be ignored, as it cannot be removed.</p>
@@ -40079,6 +40091,66 @@ $ tree /tmp/c
<li>"error": return an error based on option value</li>
</ul>
<h1 id="changelog-1">Changelog</h1>
<h2 id="v1.70.3---2025-07-09">v1.70.3 - 2025-07-09</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.70.2...v1.70.3">See commits</a></p>
<ul>
<li>Bug Fixes
<ul>
<li>check: Fix difference report (was reporting error counts) (albertony)</li>
<li>march: Fix deadlock when using <code>--no-traverse</code> (Nick Craig-Wood)</li>
<li>doc fixes (albertony, Nick Craig-Wood)</li>
</ul></li>
<li>Azure Blob
<ul>
<li>Fix server side copy error "requires exactly one scope" (Nick Craig-Wood)</li>
</ul></li>
<li>B2
<ul>
<li>Fix finding objects when using <code>--b2-version-at</code> (Davide Bizzarri)</li>
</ul></li>
<li>Linkbox
<ul>
<li>Fix upload error "user upload file not exist" (Nick Craig-Wood)</li>
</ul></li>
<li>Pikpak
<ul>
<li>Improve error handling for missing links and unrecoverable 500s (wiserain)</li>
</ul></li>
<li>WebDAV
<ul>
<li>Fix setting modtime to that of local object instead of remote (WeidiDeng)</li>
</ul></li>
</ul>
<h2 id="v1.70.2---2025-06-27">v1.70.2 - 2025-06-27</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.70.1...v1.70.2">See commits</a></p>
<ul>
<li>Bug Fixes
<ul>
<li>convmv: Make --dry-run logs less noisy (nielash)</li>
<li>sync: Avoid copying dir metadata to itself (nielash)</li>
<li>build: Bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix GHSA-vrw8-fxc6-2r93 (dependabot[bot])</li>
<li>convmv: Fix moving to unicode-equivalent name (nielash)</li>
<li>log: Fix deadlock when using systemd logging (Nick Craig-Wood)</li>
<li>pacer: Fix nil pointer deref in RetryError (Nick Craig-Wood)</li>
<li>doc fixes (Ali Zein Yousuf, Nick Craig-Wood)</li>
</ul></li>
<li>Local
<ul>
<li>Fix --skip-links on Windows when skipping Junction points (Nick Craig-Wood)</li>
</ul></li>
<li>Combine
<ul>
<li>Fix directory not found errors with ListP interface (Nick Craig-Wood)</li>
</ul></li>
<li>Mega
<ul>
<li>Fix tls handshake failure (necaran)</li>
</ul></li>
<li>Pikpak
<ul>
<li>Fix uploads fail with "aws-chunked encoding is not supported" error (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.70.1---2025-06-19">v1.70.1 - 2025-06-19</h2>
<p><a href="https://github.com/rclone/rclone/compare/v1.70.0...v1.70.1">See commits</a></p>
<ul>

87
MANUAL.md generated
View File

@@ -1,6 +1,6 @@
% rclone(1) User Manual
% Nick Craig-Wood
% Jun 19, 2025
% Jul 09, 2025
# NAME
@@ -192,8 +192,8 @@ WebDAV or S3, that work out of the box.)
- Dropbox
- Enterprise File Fabric
- Fastmail Files
- Files.com
- FileLu Cloud Storage
- Files.com
- FlashBlade
- FTP
- Gofile
@@ -512,6 +512,12 @@ package is here.
The rclone developers maintain a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
**Note:** We also now offer a paid version of rclone with
enterprise-grade security and zero CVEs through our partner
[SecureBuild](https://securebuild.com/blog/introducing-securebuild).
If you are interested, check out their website and the [Rclone
SecureBuild Image](https://securebuild.com/images/rclone).
These images are built as part of the release process based on a
minimal Alpine Linux.
@@ -16326,9 +16332,16 @@ This option is only supported Windows platforms.
### --use-json-log ###
This switches the log format to JSON for rclone. The fields of JSON
log are `level`, `msg`, `source`, `time`. The JSON logs will be
printed on a single line, but are shown expanded here for clarity.
This switches the log format to JSON. The log messages are then
streamed as individual JSON objects, with fields: `level`, `msg`, `source`,
and `time`. The resulting format is what is sometimes referred to as
[newline-delimited JSON](https://en.wikipedia.org/wiki/JSON_streaming#Newline-delimited_JSON)
(NDJSON), or JSON Lines (JSONL). This is well suited for processing by
traditional line-oriented tools and shell pipelines, but a complete log
file is not strictly valid JSON and needs a parser that can handle it.
The JSON logs will be printed on a single line, but are shown expanded
here for clarity.
```json
{
@@ -22398,7 +22411,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.3")
```
@@ -42271,6 +42284,25 @@ Rclone cannot delete files anywhere except under `album`.
The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733).
## Making your own client_id
When you use rclone with Google photos in its default configuration you
are using rclone's client_id. This is shared between all the rclone
users. There is a global rate limit on the number of queries per
second that each client_id can do set by Google.
If there is a problem with this client_id (eg quota too low or the
client_id stops working) then you can make your own.
Please follow the steps in [the google drive docs](https://rclone.org/drive/#making-your-own-client-id).
You will need these scopes instead of the drive ones detailed:
```
https://www.googleapis.com/auth/photoslibrary.appendonly
https://www.googleapis.com/auth/photoslibrary.readonly.appcreateddata
https://www.googleapis.com/auth/photoslibrary.edit.appcreateddata
```
# Hasher
Hasher is a special overlay backend to create remotes which handle
@@ -48337,7 +48369,8 @@ For example, you might see throttling.
To create your own Client ID, please follow these steps:
1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade and then click `New registration`.
1. Open https://portal.azure.com/?quickstart=true#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview and then under the `Add` menu click `App registration`.
* If you have not created an Azure account, you will be prompted to. This is free, but you need to provide a phone number, address, and credit card for identity verification.
2. Enter a name for your app, choose account type `Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)`, select `Web` in `Redirect URI`, then type (do not copy and paste) `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use.
3. Under `manage` select `Certificates & secrets`, click `New client secret`. Enter a description (can be anything) and set `Expires` to 24 months. Copy and keep that secret _Value_ for later use (you _won't_ be able to see this value afterwards).
4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`.
@@ -59114,6 +59147,46 @@ Options:
# Changelog
## v1.70.3 - 2025-07-09
[See commits](https://github.com/rclone/rclone/compare/v1.70.2...v1.70.3)
* Bug Fixes
* check: Fix difference report (was reporting error counts) (albertony)
* march: Fix deadlock when using `--no-traverse` (Nick Craig-Wood)
* doc fixes (albertony, Nick Craig-Wood)
* Azure Blob
* Fix server side copy error "requires exactly one scope" (Nick Craig-Wood)
* B2
* Fix finding objects when using `--b2-version-at` (Davide Bizzarri)
* Linkbox
* Fix upload error "user upload file not exist" (Nick Craig-Wood)
* Pikpak
* Improve error handling for missing links and unrecoverable 500s (wiserain)
* WebDAV
* Fix setting modtime to that of local object instead of remote (WeidiDeng)
## v1.70.2 - 2025-06-27
[See commits](https://github.com/rclone/rclone/compare/v1.70.1...v1.70.2)
* Bug Fixes
* convmv: Make --dry-run logs less noisy (nielash)
* sync: Avoid copying dir metadata to itself (nielash)
* build: Bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix GHSA-vrw8-fxc6-2r93 (dependabot[bot])
* convmv: Fix moving to unicode-equivalent name (nielash)
* log: Fix deadlock when using systemd logging (Nick Craig-Wood)
* pacer: Fix nil pointer deref in RetryError (Nick Craig-Wood)
* doc fixes (Ali Zein Yousuf, Nick Craig-Wood)
* Local
* Fix --skip-links on Windows when skipping Junction points (Nick Craig-Wood)
* Combine
* Fix directory not found errors with ListP interface (Nick Craig-Wood)
* Mega
* Fix tls handshake failure (necaran)
* Pikpak
* Fix uploads fail with "aws-chunked encoding is not supported" error (Nick Craig-Wood)
## v1.70.1 - 2025-06-19
[See commits](https://github.com/rclone/rclone/compare/v1.70.0...v1.70.1)

95
MANUAL.txt generated
View File

@@ -1,6 +1,6 @@
rclone(1) User Manual
Nick Craig-Wood
Jun 19, 2025
Jul 09, 2025
NAME
@@ -179,8 +179,8 @@ S3, that work out of the box.)
- Dropbox
- Enterprise File Fabric
- Fastmail Files
- Files.com
- FileLu Cloud Storage
- Files.com
- FlashBlade
- FTP
- Gofile
@@ -491,6 +491,10 @@ Docker installation
The rclone developers maintain a docker image for rclone.
Note: We also now offer a paid version of rclone with enterprise-grade
security and zero CVEs through our partner SecureBuild. If you are
interested, check out their website and the Rclone SecureBuild Image.
These images are built as part of the release process based on a minimal
Alpine Linux.
@@ -15793,9 +15797,16 @@ This option is only supported Windows platforms.
--use-json-log
This switches the log format to JSON for rclone. The fields of JSON log
are level, msg, source, time. The JSON logs will be printed on a single
line, but are shown expanded here for clarity.
This switches the log format to JSON. The log messages are then streamed
as individual JSON objects, with fields: level, msg, source, and time.
The resulting format is what is sometimes referred to as
newline-delimited JSON (NDJSON), or JSON Lines (JSONL). This is well
suited for processing by traditional line-oriented tools and shell
pipelines, but a complete log file is not strictly valid JSON and needs
a parser that can handle it.
The JSON logs will be printed on a single line, but are shown expanded
here for clarity.
{
"time": "2025-05-13T17:30:51.036237518+01:00",
@@ -21959,7 +21970,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.3")
Performance
@@ -41642,6 +41653,23 @@ Deleting albums
The Google Photos API does not support deleting albums - see bug
#135714733.
Making your own client_id
When you use rclone with Google photos in its default configuration you
are using rclone's client_id. This is shared between all the rclone
users. There is a global rate limit on the number of queries per second
that each client_id can do set by Google.
If there is a problem with this client_id (eg quota too low or the
client_id stops working) then you can make your own.
Please follow the steps in the google drive docs. You will need these
scopes instead of the drive ones detailed:
https://www.googleapis.com/auth/photoslibrary.appendonly
https://www.googleapis.com/auth/photoslibrary.readonly.appcreateddata
https://www.googleapis.com/auth/photoslibrary.edit.appcreateddata
Hasher
Hasher is a special overlay backend to create remotes which handle
@@ -47861,8 +47889,11 @@ Creating Client ID for OneDrive Personal
To create your own Client ID, please follow these steps:
1. Open
https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade
and then click New registration.
https://portal.azure.com/?quickstart=true#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview
and then under the Add menu click App registration.
- If you have not created an Azure account, you will be prompted
to. This is free, but you need to provide a phone number,
address, and credit card for identity verification.
2. Enter a name for your app, choose account type
Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox),
select Web in Redirect URI, then type (do not copy and paste)
@@ -58777,6 +58808,54 @@ Options:
Changelog
v1.70.3 - 2025-07-09
See commits
- Bug Fixes
- check: Fix difference report (was reporting error counts)
(albertony)
- march: Fix deadlock when using --no-traverse (Nick Craig-Wood)
- doc fixes (albertony, Nick Craig-Wood)
- Azure Blob
- Fix server side copy error "requires exactly one scope" (Nick
Craig-Wood)
- B2
- Fix finding objects when using --b2-version-at (Davide Bizzarri)
- Linkbox
- Fix upload error "user upload file not exist" (Nick Craig-Wood)
- Pikpak
- Improve error handling for missing links and unrecoverable 500s
(wiserain)
- WebDAV
- Fix setting modtime to that of local object instead of remote
(WeidiDeng)
v1.70.2 - 2025-06-27
See commits
- Bug Fixes
- convmv: Make --dry-run logs less noisy (nielash)
- sync: Avoid copying dir metadata to itself (nielash)
- build: Bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix
GHSA-vrw8-fxc6-2r93 (dependabot[bot])
- convmv: Fix moving to unicode-equivalent name (nielash)
- log: Fix deadlock when using systemd logging (Nick Craig-Wood)
- pacer: Fix nil pointer deref in RetryError (Nick Craig-Wood)
- doc fixes (Ali Zein Yousuf, Nick Craig-Wood)
- Local
- Fix --skip-links on Windows when skipping Junction points (Nick
Craig-Wood)
- Combine
- Fix directory not found errors with ListP interface (Nick
Craig-Wood)
- Mega
- Fix tls handshake failure (necaran)
- Pikpak
- Fix uploads fail with "aws-chunked encoding is not supported"
error (Nick Craig-Wood)
v1.70.1 - 2025-06-19
See commits

View File

@@ -39,6 +39,7 @@ Rclone *("rsync for cloud storage")* is a command-line program to sync files and
* Dropbox [:page_facing_up:](https://rclone.org/dropbox/)
* Enterprise File Fabric [:page_facing_up:](https://rclone.org/filefabric/)
* Fastmail Files [:page_facing_up:](https://rclone.org/webdav/#fastmail-files)
* FileLu [:page_facing_up:](https://rclone.org/filelu/)
* Files.com [:page_facing_up:](https://rclone.org/filescom/)
* FlashBlade [:page_facing_up:](https://rclone.org/s3/#pure-storage-flashblade)
* FTP [:page_facing_up:](https://rclone.org/ftp/)

View File

@@ -1 +1 @@
v1.70.1
v1.70.3

View File

@@ -72,6 +72,7 @@ const (
emulatorAccount = "devstoreaccount1"
emulatorAccountKey = "Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw=="
emulatorBlobEndpoint = "http://127.0.0.1:10000/devstoreaccount1"
sasCopyValidity = time.Hour // how long SAS should last when doing server side copy
)
var (
@@ -559,6 +560,11 @@ type Fs struct {
pacer *fs.Pacer // To pace and retry the API calls
uploadToken *pacer.TokenDispenser // control concurrency
publicAccess container.PublicAccessType // Container Public Access Level
// user delegation cache
userDelegationMu sync.Mutex
userDelegation *service.UserDelegationCredential
userDelegationExpiry time.Time
}
// Object describes an azure object
@@ -1688,6 +1694,38 @@ func (f *Fs) Purge(ctx context.Context, dir string) error {
return f.deleteContainer(ctx, container)
}
// Get a user delegation which is valid for at least sasCopyValidity
//
// This value is cached in f
func (f *Fs) getUserDelegation(ctx context.Context) (*service.UserDelegationCredential, error) {
f.userDelegationMu.Lock()
defer f.userDelegationMu.Unlock()
if f.userDelegation != nil && time.Until(f.userDelegationExpiry) > sasCopyValidity {
return f.userDelegation, nil
}
// Validity window
start := time.Now().UTC()
expiry := start.Add(2 * sasCopyValidity)
startStr := start.Format(time.RFC3339)
expiryStr := expiry.Format(time.RFC3339)
// Acquire user delegation key from the service client
info := service.KeyInfo{
Start: &startStr,
Expiry: &expiryStr,
}
userDelegationKey, err := f.svc.GetUserDelegationCredential(ctx, info, nil)
if err != nil {
return nil, fmt.Errorf("failed to get user delegation key: %w", err)
}
f.userDelegation = userDelegationKey
f.userDelegationExpiry = expiry
return f.userDelegation, nil
}
// getAuth gets auth to copy o.
//
// tokenOK is used to signal that token based auth (Microsoft Entra
@@ -1699,7 +1737,7 @@ func (f *Fs) Purge(ctx context.Context, dir string) error {
// URL (not a SAS) and token will be empty.
//
// If tokenOK is true it may also return a token for the auth.
func (o *Object) getAuth(ctx context.Context, tokenOK bool, noAuth bool) (srcURL string, token *string, err error) {
func (o *Object) getAuth(ctx context.Context, noAuth bool) (srcURL string, err error) {
f := o.fs
srcBlobSVC := o.getBlobSVC()
srcURL = srcBlobSVC.URL()
@@ -1708,29 +1746,47 @@ func (o *Object) getAuth(ctx context.Context, tokenOK bool, noAuth bool) (srcURL
case noAuth:
// If same storage account then no auth needed
case f.cred != nil:
if !tokenOK {
return srcURL, token, errors.New("not supported: Microsoft Entra ID")
}
options := policy.TokenRequestOptions{}
accessToken, err := f.cred.GetToken(ctx, options)
// Generate a User Delegation SAS URL using Azure AD credentials
userDelegationKey, err := f.getUserDelegation(ctx)
if err != nil {
return srcURL, token, fmt.Errorf("failed to create access token: %w", err)
return "", fmt.Errorf("sas creation: %w", err)
}
token = &accessToken.Token
// Build the SAS values
perms := sas.BlobPermissions{Read: true}
container, containerPath := o.split()
start := time.Now().UTC()
expiry := start.Add(sasCopyValidity)
vals := sas.BlobSignatureValues{
StartTime: start,
ExpiryTime: expiry,
Permissions: perms.String(),
ContainerName: container,
BlobName: containerPath,
}
// Sign with the delegation key
queryParameters, err := vals.SignWithUserDelegation(userDelegationKey)
if err != nil {
return "", fmt.Errorf("signing SAS with user delegation failed: %w", err)
}
// Append the SAS to the URL
srcURL = srcBlobSVC.URL() + "?" + queryParameters.Encode()
case f.sharedKeyCred != nil:
// Generate a short lived SAS URL if using shared key credentials
expiry := time.Now().Add(time.Hour)
expiry := time.Now().Add(sasCopyValidity)
sasOptions := blob.GetSASURLOptions{}
srcURL, err = srcBlobSVC.GetSASURL(sas.BlobPermissions{Read: true}, expiry, &sasOptions)
if err != nil {
return srcURL, token, fmt.Errorf("failed to create SAS URL: %w", err)
return srcURL, fmt.Errorf("failed to create SAS URL: %w", err)
}
case f.anonymous || f.opt.SASURL != "":
// If using a SASURL or anonymous, no need for any extra auth
default:
return srcURL, token, errors.New("unknown authentication type")
return srcURL, errors.New("unknown authentication type")
}
return srcURL, token, nil
return srcURL, nil
}
// Do multipart parallel copy.
@@ -1751,7 +1807,7 @@ func (f *Fs) copyMultipart(ctx context.Context, remote, dstContainer, dstPath st
o.fs = f
o.remote = remote
srcURL, token, err := src.getAuth(ctx, true, false)
srcURL, err := src.getAuth(ctx, false)
if err != nil {
return nil, fmt.Errorf("multipart copy: %w", err)
}
@@ -1795,7 +1851,8 @@ func (f *Fs) copyMultipart(ctx context.Context, remote, dstContainer, dstPath st
Count: partSize,
},
// Specifies the authorization scheme and signature for the copy source.
CopySourceAuthorization: token,
// We use SAS URLs as this doesn't seem to work always
// CopySourceAuthorization: token,
// CPKInfo *blob.CPKInfo
// CPKScopeInfo *blob.CPKScopeInfo
}
@@ -1865,7 +1922,7 @@ func (f *Fs) copySinglepart(ctx context.Context, remote, dstContainer, dstPath s
dstBlobSVC := f.getBlobSVC(dstContainer, dstPath)
// Get the source auth - none needed for same storage account
srcURL, _, err := src.getAuth(ctx, false, f == src.fs)
srcURL, err := src.getAuth(ctx, f == src.fs)
if err != nil {
return nil, fmt.Errorf("single part copy: source auth: %w", err)
}

View File

@@ -1673,6 +1673,21 @@ func (o *Object) getMetaData(ctx context.Context) (info *api.File, err error) {
return o.getMetaDataListing(ctx)
}
}
// If using versionAt we need to list the find the correct version.
if o.fs.opt.VersionAt.IsSet() {
info, err := o.getMetaDataListing(ctx)
if err != nil {
return nil, err
}
if info.Action == "hide" {
// Rerturn object not found error if the current version is deleted.
return nil, fs.ErrorObjectNotFound
}
return info, nil
}
_, info, err = o.getOrHead(ctx, "HEAD", nil)
return info, err
}

View File

@@ -446,14 +446,14 @@ func (f *Fs) InternalTestVersions(t *testing.T) {
t.Run("List", func(t *testing.T) {
fstest.CheckListing(t, f, test.want)
})
// b2 NewObject doesn't work with VersionAt
//t.Run("NewObject", func(t *testing.T) {
// gotObj, gotErr := f.NewObject(ctx, fileName)
// assert.Equal(t, test.wantErr, gotErr)
// if gotErr == nil {
// assert.Equal(t, test.wantSize, gotObj.Size())
// }
//})
t.Run("NewObject", func(t *testing.T) {
gotObj, gotErr := f.NewObject(ctx, fileName)
assert.Equal(t, test.wantErr, gotErr)
if gotErr == nil {
assert.Equal(t, test.wantSize, gotObj.Size())
}
})
})
}
})

View File

@@ -858,7 +858,7 @@ func (f *Fs) ListP(ctx context.Context, dir string, callback fs.ListRCallback) e
}
return wrappedCallback(entries)
}
return listP(ctx, dir, wrappedCallback)
return listP(ctx, uRemote, wrappedCallback)
}
// ListR lists the objects and directories of the Fs starting

View File

@@ -617,16 +617,36 @@ func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, op
case 1:
// upload file using link from first step
var res *http.Response
var location string
// Check to see if we are being redirected
opts := &rest.Opts{
Method: "HEAD",
RootURL: getFirstStepResult.Data.SignURL,
Options: options,
NoRedirect: true,
}
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
res, err = o.fs.srv.Call(ctx, opts)
return o.fs.shouldRetry(ctx, res, err)
})
if res != nil {
location = res.Header.Get("Location")
if location != "" {
// set the URL to the new Location
opts.RootURL = location
err = nil
}
}
if err != nil {
return fmt.Errorf("head upload URL: %w", err)
}
file := io.MultiReader(bytes.NewReader(first10mBytes), in)
opts := &rest.Opts{
Method: "PUT",
RootURL: getFirstStepResult.Data.SignURL,
Options: options,
Body: file,
ContentLength: &size,
}
opts.Method = "PUT"
opts.Body = file
opts.ContentLength = &size
err = o.fs.pacer.CallNoRetry(func() (bool, error) {
res, err = o.fs.srv.Call(ctx, opts)

View File

@@ -1201,7 +1201,15 @@ func (o *Object) Storable() bool {
o.fs.objectMetaMu.RLock()
mode := o.mode
o.fs.objectMetaMu.RUnlock()
if mode&os.ModeSymlink != 0 && !o.fs.opt.TranslateSymlinks {
// On Windows items with os.ModeIrregular are likely Junction
// points so we treat them as symlinks for the purpose of ignoring them.
// https://github.com/golang/go/issues/73827
symlinkFlag := os.ModeSymlink
if runtime.GOOS == "windows" {
symlinkFlag |= os.ModeIrregular
}
if mode&symlinkFlag != 0 && !o.fs.opt.TranslateSymlinks {
if !o.fs.opt.SkipSymlinks {
fs.Logf(o, "Can't follow symlink without -L/--copy-links")
}

View File

@@ -17,9 +17,11 @@ Improvements:
import (
"context"
"crypto/tls"
"errors"
"fmt"
"io"
"net/http"
"path"
"slices"
"strings"
@@ -216,7 +218,25 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
defer megaCacheMu.Unlock()
srv := megaCache[opt.User]
if srv == nil {
srv = mega.New().SetClient(fshttp.NewClient(ctx))
// srv = mega.New().SetClient(fshttp.NewClient(ctx))
// Workaround for Mega's use of insecure cipher suites which are no longer supported by default since Go 1.22.
// Relevant issues:
// https://github.com/rclone/rclone/issues/8565
// https://github.com/meganz/webclient/issues/103
clt := fshttp.NewClient(ctx)
clt.Transport = fshttp.NewTransportCustom(ctx, func(t *http.Transport) {
var ids []uint16
// Read default ciphers
for _, cs := range tls.CipherSuites() {
ids = append(ids, cs.ID)
}
// Insecure but Mega uses TLS_RSA_WITH_AES_128_GCM_SHA256 for storage endpoints
// (e.g. https://gfs302n114.userstorage.mega.co.nz) as of June 18, 2025.
t.TLSClientConfig.CipherSuites = append(ids, tls.TLS_RSA_WITH_AES_128_GCM_SHA256)
})
srv = mega.New().SetClient(clt)
srv.SetRetries(ci.LowLevelRetries) // let mega do the low level retries
srv.SetHTTPS(opt.UseHTTPS)
srv.SetLogger(func(format string, v ...any) {

View File

@@ -155,6 +155,7 @@ func (f *Fs) getFile(ctx context.Context, ID string) (info *api.File, err error)
err = f.pacer.Call(func() (bool, error) {
resp, err = f.rst.CallJSON(ctx, &opts, nil, &info)
if err == nil && !info.Links.ApplicationOctetStream.Valid() {
time.Sleep(5 * time.Second)
return true, errors.New("no link")
}
return f.shouldRetry(ctx, resp, err)

View File

@@ -467,6 +467,11 @@ func (f *Fs) shouldRetry(ctx context.Context, resp *http.Response, err error) (b
// when a zero-byte file was uploaded with an invalid captcha token
f.rst.captcha.Invalidate()
return true, err
} else if strings.Contains(apiErr.Reason, "idx.shub.mypikpak.com") && apiErr.Code == 500 {
// internal server error: Post "http://idx.shub.mypikpak.com": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
// This typically happens when trying to retrieve a gcid for which no record exists.
// No retry is needed in this case.
return false, err
}
}

View File

@@ -1550,7 +1550,7 @@ func (o *Object) extraHeaders(ctx context.Context, src fs.ObjectInfo) map[string
extraHeaders := map[string]string{}
if o.fs.useOCMtime || o.fs.hasOCMD5 || o.fs.hasOCSHA1 {
if o.fs.useOCMtime {
extraHeaders["X-OC-Mtime"] = fmt.Sprintf("%d", o.modTime.Unix())
extraHeaders["X-OC-Mtime"] = fmt.Sprintf("%d", src.ModTime(ctx).Unix())
}
// Set one upload checksum
// Owncloud uses one checksum only to check the upload and stores its own SHA1 and MD5

View File

@@ -141,7 +141,7 @@ func TestTransform(t *testing.T) {
}
// const alphabet = "ƀɀɠʀҠԀڀڠݠހ߀ကႠᄀᄠᅀᆀᇠሠበዠጠᐠᑀᑠᒀᒠᓀᓠᔀᔠᕀᕠᖀᖠᗀᗠᘀᘠᙀᚠᛀកᠠᡀᣀᦀ᧠ᨠᯀᰀᴀ⇠⋀⍀⍠⎀⎠⏀␀─┠╀╠▀■◀◠☀☠♀♠⚀⚠⛀⛠✀✠❀➀➠⠠⡀⡠⢀⢠⣀⣠⤀⤠⥀⥠⦠⨠⩀⪀⪠⫠⬀⬠⭀ⰀⲀⲠⳀⴀⵀ⺠⻀㇀㐀㐠㑀㑠㒀㒠㓀㓠㔀㔠㕀㕠㖀㖠㗀㗠㘀㘠㙀㙠㚀㚠㛀㛠㜀㜠㝀㝠㞀㞠㟀㟠㠀㠠㡀㡠㢀㢠㣀㣠㤀㤠㥀㥠㦀㦠㧀㧠㨀㨠㩀㩠㪀㪠㫀㫠㬀㬠㭀㭠㮀㮠㯀㯠㰀㰠㱀㱠㲀㲠㳀㳠㴀㴠㵀㵠㶀㶠㷀㷠㸀㸠㹀㹠㺀㺠㻀㻠㼀㼠㽀㽠㾀㾠㿀㿠䀀䀠䁀䁠䂀䂠䃀䃠䄀䄠䅀䅠䆀䆠䇀䇠䈀䈠䉀䉠䊀䊠䋀䋠䌀䌠䍀䍠䎀䎠䏀䏠䐀䐠䑀䑠䒀䒠䓀䓠䔀䔠䕀䕠䖀䖠䗀䗠䘀䘠䙀䙠䚀䚠䛀䛠䜀䜠䝀䝠䞀䞠䟀䟠䠀䠠䡀䡠䢀䢠䣀䣠䤀䤠䥀䥠䦀䦠䧀䧠䨀䨠䩀䩠䪀䪠䫀䫠䬀䬠䭀䭠䮀䮠䯀䯠䰀䰠䱀䱠䲀䲠䳀䳠䴀䴠䵀䵠䶀䷀䷠一丠乀习亀亠什仠伀传佀你侀侠俀俠倀倠偀偠傀傠僀僠儀儠兀兠冀冠净几刀删剀剠劀加勀勠匀匠區占厀厠叀叠吀吠呀呠咀咠哀哠唀唠啀啠喀喠嗀嗠嘀嘠噀噠嚀嚠囀因圀圠址坠垀垠埀埠堀堠塀塠墀墠壀壠夀夠奀奠妀妠姀姠娀娠婀婠媀媠嫀嫠嬀嬠孀孠宀宠寀寠尀尠局屠岀岠峀峠崀崠嵀嵠嶀嶠巀巠帀帠幀幠庀庠廀廠开张彀彠往徠忀忠怀怠恀恠悀悠惀惠愀愠慀慠憀憠懀懠戀戠所扠技抠拀拠挀挠捀捠掀掠揀揠搀搠摀摠撀撠擀擠攀攠敀敠斀斠旀无昀映晀晠暀暠曀曠最朠杀杠枀枠柀柠栀栠桀桠梀梠检棠椀椠楀楠榀榠槀槠樀樠橀橠檀檠櫀櫠欀欠歀歠殀殠毀毠氀氠汀池沀沠泀泠洀洠浀浠涀涠淀淠渀渠湀湠満溠滀滠漀漠潀潠澀澠激濠瀀瀠灀灠炀炠烀烠焀焠煀煠熀熠燀燠爀爠牀牠犀犠狀狠猀猠獀獠玀玠珀珠琀琠瑀瑠璀璠瓀瓠甀甠畀畠疀疠痀痠瘀瘠癀癠皀皠盀盠眀眠着睠瞀瞠矀矠砀砠础硠碀碠磀磠礀礠祀祠禀禠秀秠稀稠穀穠窀窠竀章笀笠筀筠简箠節篠簀簠籀籠粀粠糀糠紀素絀絠綀綠緀締縀縠繀繠纀纠绀绠缀缠罀罠羀羠翀翠耀耠聀聠肀肠胀胠脀脠腀腠膀膠臀臠舀舠艀艠芀芠苀苠茀茠荀荠莀莠菀菠萀萠葀葠蒀蒠蓀蓠蔀蔠蕀蕠薀薠藀藠蘀蘠虀虠蚀蚠蛀蛠蜀蜠蝀蝠螀螠蟀蟠蠀蠠血衠袀袠裀裠褀褠襀襠覀覠觀觠言訠詀詠誀誠諀諠謀謠譀譠讀讠诀诠谀谠豀豠貀負賀賠贀贠赀赠趀趠跀跠踀踠蹀蹠躀躠軀軠輀輠轀轠辀辠迀迠退造遀遠邀邠郀郠鄀鄠酀酠醀醠釀釠鈀鈠鉀鉠銀銠鋀鋠錀錠鍀鍠鎀鎠鏀鏠鐀鐠鑀鑠钀钠铀铠销锠镀镠門閠闀闠阀阠陀陠隀隠雀雠需霠靀靠鞀鞠韀韠頀頠顀顠颀颠飀飠餀餠饀饠馀馠駀駠騀騠驀驠骀骠髀髠鬀鬠魀魠鮀鮠鯀鯠鰀鰠鱀鱠鲀鲠鳀鳠鴀鴠鵀鵠鶀鶠鷀鷠鸀鸠鹀鹠麀麠黀黠鼀鼠齀齠龀龠ꀀꀠꁀꁠꂀꂠꃀꃠꄀꄠꅀꅠꆀꆠꇀꇠꈀꈠꉀꉠꊀꊠꋀꋠꌀꌠꍀꍠꎀꎠꏀꏠꐀꐠꑀꑠ꒠ꔀꔠꕀꕠꖀꖠꗀꗠꙀꚠꛀ꜀꜠ꝀꞀꡀ測試_Русский___ě_áñ"
const alphabet = "abcdefg123456789"
const alphabet = "abcdefg123456789"
var extras = []string{"apple", "banana", "appleappleapplebanana", "splitbananasplit"}
@@ -251,3 +251,25 @@ func detectEncoding(s string) string {
}
return "OTHER"
}
func TestUnicodeEquivalence(t *testing.T) {
r := fstest.NewRun(t)
defer r.Finalise()
ctx := context.Background()
r.Mkdir(ctx, r.Fremote)
const remote = "Über"
item := r.WriteObject(ctx, remote, "", t1)
obj, err := r.Fremote.NewObject(ctx, remote) // can't use r.CheckRemoteListing here as it forces NFC
require.NoError(t, err)
require.NotEmpty(t, obj)
err = transform.SetOptions(ctx, "all,nfc")
require.NoError(t, err)
err = sync.Transform(ctx, r.Fremote, true, true)
assert.NoError(t, err)
item.Path = norm.NFC.String(item.Path)
r.CheckRemoteListing(t, []fstest.Item{item}, nil)
}

View File

@@ -123,8 +123,8 @@ WebDAV or S3, that work out of the box.)
{{< provider name="Dropbox" home="https://www.dropbox.com/" config="/dropbox/" >}}
{{< provider name="Enterprise File Fabric" home="https://storagemadeeasy.com/about/" config="/filefabric/" >}}
{{< provider name="Fastmail Files" home="https://www.fastmail.com/" config="/webdav/#fastmail-files" >}}
{{< provider name="Files.com" home="https://www.files.com/" config="/filescom/" >}}
{{< provider name="FileLu Cloud Storage" home="https://filelu.com/" config="/filelu/" >}}
{{< provider name="Files.com" home="https://www.files.com/" config="/filescom/" >}}
{{< provider name="FlashBlade" home="https://www.purestorage.com/products/unstructured-data-storage.html" config="/s3/#pure-storage-flashblade" >}}
{{< provider name="FTP" home="https://en.wikipedia.org/wiki/File_Transfer_Protocol" config="/ftp/" >}}
{{< provider name="Gofile" home="https://gofile.io/" config="/gofile/" >}}

View File

@@ -5,6 +5,46 @@ description: "Rclone Changelog"
# Changelog
## v1.70.3 - 2025-07-09
[See commits](https://github.com/rclone/rclone/compare/v1.70.2...v1.70.3)
* Bug Fixes
* check: Fix difference report (was reporting error counts) (albertony)
* march: Fix deadlock when using `--no-traverse` (Nick Craig-Wood)
* doc fixes (albertony, Nick Craig-Wood)
* Azure Blob
* Fix server side copy error "requires exactly one scope" (Nick Craig-Wood)
* B2
* Fix finding objects when using `--b2-version-at` (Davide Bizzarri)
* Linkbox
* Fix upload error "user upload file not exist" (Nick Craig-Wood)
* Pikpak
* Improve error handling for missing links and unrecoverable 500s (wiserain)
* WebDAV
* Fix setting modtime to that of local object instead of remote (WeidiDeng)
## v1.70.2 - 2025-06-27
[See commits](https://github.com/rclone/rclone/compare/v1.70.1...v1.70.2)
* Bug Fixes
* convmv: Make --dry-run logs less noisy (nielash)
* sync: Avoid copying dir metadata to itself (nielash)
* build: Bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix GHSA-vrw8-fxc6-2r93 (dependabot[bot])
* convmv: Fix moving to unicode-equivalent name (nielash)
* log: Fix deadlock when using systemd logging (Nick Craig-Wood)
* pacer: Fix nil pointer deref in RetryError (Nick Craig-Wood)
* doc fixes (Ali Zein Yousuf, Nick Craig-Wood)
* Local
* Fix --skip-links on Windows when skipping Junction points (Nick Craig-Wood)
* Combine
* Fix directory not found errors with ListP interface (Nick Craig-Wood)
* Mega
* Fix tls handshake failure (necaran)
* Pikpak
* Fix uploads fail with "aws-chunked encoding is not supported" error (Nick Craig-Wood)
## v1.70.1 - 2025-06-19
[See commits](https://github.com/rclone/rclone/compare/v1.70.0...v1.70.1)

View File

@@ -998,7 +998,7 @@ rclone [flags]
--use-json-log Use json log format
--use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.3")
-v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number
--webdav-auth-redirect Preserve authentication on redirect

View File

@@ -1545,9 +1545,16 @@ This option is only supported Windows platforms.
### --use-json-log ###
This switches the log format to JSON for rclone. The fields of JSON
log are `level`, `msg`, `source`, `time`. The JSON logs will be
printed on a single line, but are shown expanded here for clarity.
This switches the log format to JSON. The log messages are then
streamed as individual JSON objects, with fields: `level`, `msg`, `source`,
and `time`. The resulting format is what is sometimes referred to as
[newline-delimited JSON](https://en.wikipedia.org/wiki/JSON_streaming#Newline-delimited_JSON)
(NDJSON), or JSON Lines (JSONL). This is well suited for processing by
traditional line-oriented tools and shell pipelines, but a complete log
file is not strictly valid JSON and needs a parser that can handle it.
The JSON logs will be printed on a single line, but are shown expanded
here for clarity.
```json
{

View File

@@ -9,6 +9,9 @@ type: page
Rclone is single executable (`rclone`, or `rclone.exe` on Windows) that you can
simply download as a zip archive and extract into a location of your choosing.
See the [install](https://rclone.org/install/) documentation for more details.
We also offer a secure [enterprise-grade, zero CVE docker
image](https://securebuild.com/images/rclone) through our partner
[SecureBuild](https://securebuild.com/blog/introducing-securebuild).
## Release {{% version %}} OS requirements {#osrequirements}

View File

@@ -119,7 +119,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.1")
--user-agent string Set the user-agent to a specified string (default "rclone/v1.70.3")
```

View File

@@ -632,3 +632,22 @@ Rclone cannot delete files anywhere except under `album`.
### Deleting albums
The Google Photos API does not support deleting albums - see [bug #135714733](https://issuetracker.google.com/issues/135714733).
## Making your own client_id
When you use rclone with Google photos in its default configuration you
are using rclone's client_id. This is shared between all the rclone
users. There is a global rate limit on the number of queries per
second that each client_id can do set by Google.
If there is a problem with this client_id (eg quota too low or the
client_id stops working) then you can make your own.
Please follow the steps in [the google drive docs](https://rclone.org/drive/#making-your-own-client-id).
You will need these scopes instead of the drive ones detailed:
```
https://www.googleapis.com/auth/photoslibrary.appendonly
https://www.googleapis.com/auth/photoslibrary.readonly.appcreateddata
https://www.googleapis.com/auth/photoslibrary.edit.appcreateddata
```

View File

@@ -225,6 +225,12 @@ package is here.
The rclone developers maintain a [docker image for rclone](https://hub.docker.com/r/rclone/rclone).
**Note:** We also now offer a paid version of rclone with
enterprise-grade security and zero CVEs through our partner
[SecureBuild](https://securebuild.com/blog/introducing-securebuild).
If you are interested, check out their website and the [Rclone
SecureBuild Image](https://securebuild.com/images/rclone).
These images are built as part of the release process based on a
minimal Alpine Linux.

View File

@@ -132,7 +132,8 @@ For example, you might see throttling.
To create your own Client ID, please follow these steps:
1. Open https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade and then click `New registration`.
1. Open https://portal.azure.com/?quickstart=true#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/~/Overview and then under the `Add` menu click `App registration`.
* If you have not created an Azure account, you will be prompted to. This is free, but you need to provide a phone number, address, and credit card for identity verification.
2. Enter a name for your app, choose account type `Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)`, select `Web` in `Redirect URI`, then type (do not copy and paste) `http://localhost:53682/` and click Register. Copy and keep the `Application (client) ID` under the app name for later use.
3. Under `manage` select `Certificates & secrets`, click `New client secret`. Enter a description (can be anything) and set `Expires` to 24 months. Copy and keep that secret _Value_ for later use (you _won't_ be able to see this value afterwards).
4. Under `manage` select `API permissions`, click `Add a permission` and select `Microsoft Graph` then select `delegated permissions`.

View File

@@ -57,7 +57,7 @@ off donation.
Thank you very much to our sponsors:
{{< sponsor src="/img/logos/idrive_e2.svg" width="300" height="200" title="Visit our sponsor IDrive e2" link="https://www.idrive.com/e2/?refer=rclone">}}
{{< sponsor src="/img/logos/warp.svg" width="285" height="200" title="Visit our sponsor warp.dev" link="https://www.warp.dev/?utm_source=rclone&utm_medium=referral&utm_campaign=rclone_20231103">}}
{{< sponsor src="/img/logos/filescom-enterprise-grade-workflows.png" width="300" height="200" title="Start Your Free Trial Today" link="https://files.com/?utm_source=rclone&utm_medium=referral&utm_campaign=banner&utm_term=rclone">}}
{{< sponsor src="/img/logos/sia.svg" width="200" height="200" title="Visit our sponsor sia" link="https://sia.tech">}}
{{< sponsor src="/img/logos/route4me.svg" width="400" height="200" title="Visit our sponsor Route4Me" link="https://route4me.com/">}}
{{< sponsor src="/img/logos/rcloneview.svg" width="300" height="200" title="Visit our sponsor RcloneView" link="https://rcloneview.com/">}}

View File

@@ -19,6 +19,15 @@
</div>
</div>
<div class="card">
<div class="card-header" style="padding: 5px 15px;">
Gold Sponsor
</div>
<div class="card-body">
<a href="https://files.com/?utm_source=rclone&utm_medium=referral&utm_campaign=banner&utm_term=rclone" target="_blank" rel="noopener" title="Start Your Free Trial Today"><img style="max-width: 100%; height: auto;" src="/img/logos/filescom-enterprise-grade-workflows.png"></a><br />
</div>
</div>
{{if .IsHome}}
<div class="card">
<div class="card-header" style="padding: 5px 15px;">

View File

@@ -66,8 +66,8 @@
<a class="dropdown-item" href="/koofr/#digi-storage"><i class="fa fa-cloud fa-fw"></i> Digi Storage</a>
<a class="dropdown-item" href="/dropbox/"><i class="fab fa-dropbox fa-fw"></i> Dropbox</a>
<a class="dropdown-item" href="/filefabric/"><i class="fa fa-cloud fa-fw"></i> Enterprise File Fabric</a>
<a class="dropdown-item" href="/filelu/"><i class="fa fa-brands fa-files-pinwheel fa-fw"></i> Files.com</a>
<a class="dropdown-item" href="/filescom/"><i class="fa fa-cloud fa-fw"></i> FileLu Cloud Storage</a>
<a class="dropdown-item" href="/filelu/"><i class="fa fa-folder"></i> FileLu Cloud Storage</a>
<a class="dropdown-item" href="/filescom/"><i class="fa fa-brands fa-files-pinwheel fa-fw"></i> Files.com</a>
<a class="dropdown-item" href="/ftp/"><i class="fa fa-file fa-fw"></i> FTP</a>
<a class="dropdown-item" href="/gofile/"><i class="fa fa-folder fa-fw"></i> Gofile</a>
<a class="dropdown-item" href="/googlecloudstorage/"><i class="fab fa-google fa-fw"></i> Google Cloud Storage</a>

View File

@@ -1 +1 @@
v1.70.1
v1.70.3

View File

@@ -5,10 +5,7 @@
package log
import (
"fmt"
"log"
"log/slog"
"strconv"
"github.com/coreos/go-systemd/v22/journal"
"github.com/rclone/rclone/fs"
@@ -18,10 +15,8 @@ import (
func startSystemdLog(handler *OutputHandler) bool {
handler.clearFormatFlags(logFormatDate | logFormatTime | logFormatMicroseconds | logFormatUTC | logFormatLongFile | logFormatShortFile | logFormatPid)
handler.setFormatFlags(logFormatNoLevel)
// TODO: Use the native journal.Print approach rather than a custom implementation
handler.SetOutput(func(level slog.Level, text string) {
text = fmt.Sprintf("<%s>%-6s: %s", systemdLogPrefix(level), level, text)
_ = log.Output(4, text)
_ = journal.Print(slogLevelToSystemdPriority(level), "%-6s: %s\n", level, text)
})
return true
}
@@ -37,12 +32,12 @@ var slogLevelToSystemdPrefix = map[slog.Level]journal.Priority{
slog.LevelDebug: journal.PriDebug,
}
func systemdLogPrefix(l slog.Level) string {
func slogLevelToSystemdPriority(l slog.Level) journal.Priority {
prio, ok := slogLevelToSystemdPrefix[l]
if !ok {
return ""
return journal.PriInfo
}
return strconv.Itoa(int(prio))
return prio
}
func isJournalStream() bool {

View File

@@ -16,7 +16,6 @@ import (
"github.com/rclone/rclone/fs/list"
"github.com/rclone/rclone/fs/walk"
"github.com/rclone/rclone/lib/transform"
"golang.org/x/sync/errgroup"
"golang.org/x/text/unicode/norm"
)
@@ -291,6 +290,7 @@ func (m *March) matchListings(srcChan, dstChan <-chan fs.DirEntry, srcOnly, dstO
srcPrev, dstPrev fs.DirEntry
srcPrevName, dstPrevName string
src, dst fs.DirEntry
srcHasMore, dstHasMore = true, true
srcName, dstName string
)
srcDone := func() {
@@ -311,14 +311,14 @@ func (m *March) matchListings(srcChan, dstChan <-chan fs.DirEntry, srcOnly, dstO
}
// Reload src and dst if needed - we set them to nil if used
if src == nil {
src = <-srcChan
src, srcHasMore = <-srcChan
srcName = m.srcKey(src)
}
if dst == nil {
dst = <-dstChan
dst, dstHasMore = <-dstChan
dstName = m.dstKey(dst)
}
if src == nil && dst == nil {
if !srcHasMore && !dstHasMore {
break
}
if src != nil && srcPrev != nil {
@@ -419,38 +419,65 @@ func (m *March) processJob(job listDirJob) ([]listDirJob, error) {
// If NoTraverse is set, then try to find a matching object
// for each item in the srcList to head dst object
if m.NoTraverse && !m.NoCheckDest {
startedDst = true
workers := ci.Checkers
originalSrcChan := srcChan
srcChan = make(chan fs.DirEntry, 100)
ls, err := list.NewSorter(m.Ctx, m.Fdst, list.SortToChan(dstChan), m.dstKey)
if err != nil {
return nil, err
type matchTask struct {
src fs.DirEntry // src object to find in destination
dstMatch chan<- fs.DirEntry // channel to receive matching dst object or nil
}
matchTasks := make(chan matchTask, workers)
dstMatches := make(chan (<-chan fs.DirEntry), workers)
// Create the tasks from the originalSrcChan. These are put into matchTasks for
// processing and dstMatches so they can be retrieved in order.
go func() {
for src := range originalSrcChan {
srcChan <- src
dstMatch := make(chan fs.DirEntry, 1)
matchTasks <- matchTask{
src: src,
dstMatch: dstMatch,
}
dstMatches <- dstMatch
}
close(matchTasks)
}()
// Get the tasks from the queue and find a matching object.
var workerWg sync.WaitGroup
for range workers {
workerWg.Add(1)
go func() {
defer workerWg.Done()
for t := range matchTasks {
leaf := path.Base(t.src.Remote())
dst, err := m.Fdst.NewObject(m.Ctx, path.Join(job.dstRemote, leaf))
if err != nil {
dst = nil
}
t.dstMatch <- dst
}
}()
}
startedDst = true
// Close dstResults when all the workers have finished
go func() {
workerWg.Wait()
close(dstMatches)
}()
// Read the matches in order and send them to dstChan if found.
wg.Add(1)
go func() {
defer wg.Done()
defer ls.CleanUp()
g, gCtx := errgroup.WithContext(m.Ctx)
g.SetLimit(ci.Checkers)
for src := range originalSrcChan {
srcChan <- src
if srcObj, ok := src.(fs.Object); ok {
g.Go(func() error {
leaf := path.Base(srcObj.Remote())
dstObj, err := m.Fdst.NewObject(gCtx, path.Join(job.dstRemote, leaf))
if err == nil {
_ = ls.Add(fs.DirEntries{dstObj}) // ignore errors
}
return nil // ignore errors
})
}
}
dstListErr = g.Wait()
sendErr := ls.Send()
if dstListErr == nil {
dstListErr = sendErr
for dstMatch := range dstMatches {
dst := <-dstMatch
// Note that dst may be nil here
// We send these on so we don't deadlock the reader
dstChan <- dst
}
close(srcChan)
close(dstChan)

View File

@@ -249,7 +249,7 @@ func (c *checkMarch) reportResults(ctx context.Context, err error) error {
fs.Logf(c.opt.Fsrc, "%d %s missing", c.srcFilesMissing.Load(), entity)
}
fs.Logf(c.opt.Fdst, "%d differences found", accounting.Stats(ctx).GetErrors())
fs.Logf(c.opt.Fdst, "%d differences found", c.differences.Load())
if errs := accounting.Stats(ctx).GetErrors(); errs > 0 {
fs.Logf(c.opt.Fdst, "%d errors while checking", errs)
}

View File

@@ -428,6 +428,10 @@ func move(ctx context.Context, fdst fs.Fs, dst fs.Object, remote string, src fs.
origRemote := remote // avoid double-transform on fallback to copy
remote = transform.Path(ctx, remote, false)
ci := fs.GetConfig(ctx)
newDst = dst
if ci.DryRun && dst != nil && SameObject(src, dst) && src.Remote() == transform.Path(ctx, dst.Remote(), false) {
return // avoid SkipDestructive log for objects that won't really be moved
}
var tr *accounting.Transfer
if isTransfer {
tr = accounting.Stats(ctx).NewTransfer(src, fdst)
@@ -440,8 +444,11 @@ func move(ctx context.Context, fdst fs.Fs, dst fs.Object, remote string, src fs.
}
tr.Done(ctx, err)
}()
newDst = dst
if SkipDestructive(ctx, src, "move") {
action := "move"
if remote != src.Remote() {
action += " to " + remote
}
if SkipDestructive(ctx, src, action) {
in := tr.Account(ctx, nil)
in.DryRun(src.Size())
return newDst, nil
@@ -1939,6 +1946,9 @@ func MoveBackupDir(ctx context.Context, backupDir fs.Fs, dst fs.Object) (err err
func needsMoveCaseInsensitive(fdst fs.Fs, fsrc fs.Fs, dstFileName string, srcFileName string, cp bool) bool {
dstFilePath := path.Join(fdst.Root(), dstFileName)
srcFilePath := path.Join(fsrc.Root(), srcFileName)
if !cp && fdst.Name() == fsrc.Name() && dstFileName != srcFileName && norm.NFC.String(dstFilePath) == norm.NFC.String(srcFilePath) {
return true
}
return !cp && fdst.Name() == fsrc.Name() && fdst.Features().CaseInsensitive && dstFileName != srcFileName && strings.EqualFold(dstFilePath, srcFilePath)
}

View File

@@ -1109,6 +1109,9 @@ func (s *syncCopyMove) markDirModifiedObject(o fs.Object) {
// be nil.
func (s *syncCopyMove) copyDirMetadata(ctx context.Context, f fs.Fs, dst fs.Directory, dir string, src fs.Directory) (newDst fs.Directory) {
var err error
if dst != nil && src.Remote() == dst.Remote() && operations.OverlappingFilterCheck(ctx, s.fdst, s.fsrc) {
return nil // src and dst can be the same in convmv
}
equal := operations.DirsEqual(ctx, src, dst, operations.DirsEqualOpt{ModifyWindow: s.modifyWindow, SetDirModtime: s.setDirModTime, SetDirMetadata: s.setDirMetadata})
if !s.setDirModTimeAfter && equal {
return nil

View File

@@ -216,6 +216,35 @@ func TestCopyNoTraverse(t *testing.T) {
r.CheckRemoteItems(t, file1)
}
func TestCopyNoTraverseDeadlock(t *testing.T) {
r := fstest.NewRun(t)
if !r.Fremote.Features().IsLocal {
t.Skip("Only runs on local")
}
const nFiles = 200
t1 := fstest.Time("2001-02-03T04:05:06.499999999Z")
// Create lots of source files.
items := make([]fstest.Item, nFiles)
for i := range items {
name := fmt.Sprintf("file%d.txt", i)
items[i] = r.WriteFile(name, fmt.Sprintf("content%d", i), t1)
}
r.CheckLocalItems(t, items...)
// Set --no-traverse
ctx, ci := fs.AddConfig(context.Background())
ci.NoTraverse = true
// Initial copy to establish destination.
require.NoError(t, CopyDir(ctx, r.Fremote, r.Flocal, false))
r.CheckRemoteItems(t, items...)
// Second copy which shouldn't deadlock
require.NoError(t, CopyDir(ctx, r.Flocal, r.Fremote, false))
r.CheckRemoteItems(t, items...)
}
// Now with --check-first
func TestCopyCheckFirst(t *testing.T) {
ctx := context.Background()

View File

@@ -1,4 +1,4 @@
package fs
// VersionTag of rclone
var VersionTag = "v1.70.1"
var VersionTag = "v1.70.3"

7
go.mod
View File

@@ -20,8 +20,8 @@ require (
github.com/aws/aws-sdk-go-v2 v1.36.3
github.com/aws/aws-sdk-go-v2/config v1.29.14
github.com/aws/aws-sdk-go-v2/credentials v1.17.67
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.77
github.com/aws/aws-sdk-go-v2/service/s3 v1.80.0
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.49
github.com/aws/aws-sdk-go-v2/service/s3 v1.72.3
github.com/aws/smithy-go v1.22.3
github.com/buengese/sgzip v0.1.1
github.com/cloudinary/cloudinary-go/v2 v2.10.0
@@ -33,7 +33,7 @@ require (
github.com/dropbox/dropbox-sdk-go-unofficial/v6 v6.0.5
github.com/gabriel-vasile/mimetype v1.4.9
github.com/gdamore/tcell/v2 v2.8.1
github.com/go-chi/chi/v5 v5.2.1
github.com/go-chi/chi/v5 v5.2.2
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348
github.com/go-git/go-billy/v5 v5.6.2
github.com/google/uuid v1.6.0
@@ -91,7 +91,6 @@ require (
gopkg.in/validator.v2 v2.0.1
gopkg.in/yaml.v3 v3.0.1
storj.io/uplink v1.13.1
)
require (

12
go.sum
View File

@@ -120,8 +120,8 @@ github.com/aws/aws-sdk-go-v2/credentials v1.17.67 h1:9KxtdcIA/5xPNQyZRgUSpYOE6j9
github.com/aws/aws-sdk-go-v2/credentials v1.17.67/go.mod h1:p3C44m+cfnbv763s52gCqrjaqyPikj9Sg47kUVaNZQQ=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30 h1:x793wxmUWVDhshP8WW2mlnXuFrO4cOd3HLBroh1paFw=
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.30/go.mod h1:Jpne2tDnYiFascUEs2AWHJL9Yp7A5ZVy3TNyxaAjD6M=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.77 h1:xaRN9fags7iJznsMEjtcEuON1hGfCZ0y5MVfEMKtrx8=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.77/go.mod h1:lolsiGkT47AZ3DWqtxgEQM/wVMpayi7YWNjl3wHSRx8=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.49 h1:7gss+6H2mrrFtBrkokJRR2TzQD9qkpGA4N6BvIP/pCM=
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.17.49/go.mod h1:30PBx0ENoUCJm2AxzgCue8j7KEjb9ci4enxy6CCOjbE=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 h1:ZK5jHhnrioRkUNOc+hOgQKlUL5JeC3S6JgLxtQ+Rm0Q=
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34/go.mod h1:p4VfIceZokChbA9FzMbRGz5OV+lekcVtHlPKEO0gSZY=
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 h1:SZwFm17ZUNNg5Np0ioo/gq8Mn6u9w19Mri8DnJ15Jf0=
@@ -138,8 +138,8 @@ github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15 h1:dM9/92u2
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15/go.mod h1:SwFBy2vjtA0vZbjjaFtfN045boopadnoVPhu4Fv66vY=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.15 h1:moLQUoVq91LiqT1nbvzDukyqAlCv89ZmwaHw/ZFlFZg=
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.15/go.mod h1:ZH34PJUc8ApjBIfgQCFvkWcUDBtl/WTD+uiYHjd8igA=
github.com/aws/aws-sdk-go-v2/service/s3 v1.80.0 h1:fV4XIU5sn/x8gjRouoJpDVHj+ExJaUk4prYF+eb6qTs=
github.com/aws/aws-sdk-go-v2/service/s3 v1.80.0/go.mod h1:qbn305Je/IofWBJ4bJz/Q7pDEtnnoInw/dGt71v6rHE=
github.com/aws/aws-sdk-go-v2/service/s3 v1.72.3 h1:WZOmJfCDV+4tYacLxpiojoAdT5sxTfB3nTqQNtZu+J4=
github.com/aws/aws-sdk-go-v2/service/s3 v1.72.3/go.mod h1:xMekrnhmJ5aqmyxtmALs7mlvXw5xRh+eYjOjvrIIFJ4=
github.com/aws/aws-sdk-go-v2/service/sso v1.25.3 h1:1Gw+9ajCV1jogloEv1RRnvfRFia2cL6c9cuKV2Ps+G8=
github.com/aws/aws-sdk-go-v2/service/sso v1.25.3/go.mod h1:qs4a9T5EMLl/Cajiw2TcbNt2UNo/Hqlyp+GiuG4CFDI=
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.30.1 h1:hXmVKytPfTy5axZ+fYbR5d0cFmC3JvwLm5kM83luako=
@@ -246,8 +246,8 @@ github.com/gin-contrib/sse v1.0.0 h1:y3bT1mUWUxDpW4JLQg/HnTqV4rozuW4tC9eFKTxYI9E
github.com/gin-contrib/sse v1.0.0/go.mod h1:zNuFdwarAygJBht0NTKiSi3jRf6RbqeILZ9Sp6Slhe0=
github.com/gin-gonic/gin v1.10.0 h1:nTuyha1TYqgedzytsKYqna+DfLos46nTv2ygFy86HFU=
github.com/gin-gonic/gin v1.10.0/go.mod h1:4PMNQiOhvDRa013RKVbsiNwoyezlm2rm0uX/T7kzp5Y=
github.com/go-chi/chi/v5 v5.2.1 h1:KOIHODQj58PmL80G2Eak4WdvUzjSJSm0vG72crDCqb8=
github.com/go-chi/chi/v5 v5.2.1/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-chi/chi/v5 v5.2.2 h1:CMwsvRVTbXVytCk1Wd72Zy1LAsAh9GxMmSNWLHCG618=
github.com/go-chi/chi/v5 v5.2.2/go.mod h1:L2yAIGWB3H+phAw1NxKwWM+7eUH/lU8pOMm5hHcoops=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348 h1:JnrjqG5iR07/8k7NqrLNilRsl3s1EPRQEGvbPyOce68=
github.com/go-darwin/apfs v0.0.0-20211011131704-f84b94dbf348/go.mod h1:Czxo/d1g948LtrALAZdL04TL/HnkopquAjxYUuI02bo=
github.com/go-git/go-billy/v5 v5.6.2 h1:6Q86EsPXMa7c3YZ3aLAQsMA0VlWmy43r6FHqa/UNbRM=

View File

@@ -2,6 +2,8 @@
package pacer
import (
"errors"
"fmt"
"sync"
"time"
@@ -235,15 +237,22 @@ type retryAfterError struct {
}
func (r *retryAfterError) Error() string {
return r.error.Error()
return fmt.Sprintf("%v: trying again in %v", r.error, r.retryAfter)
}
func (r *retryAfterError) Cause() error {
return r.error
}
func (r *retryAfterError) Unwrap() error {
return r.error
}
// RetryAfterError returns a wrapped error that can be used by Calculator implementations
func RetryAfterError(err error, retryAfter time.Duration) error {
if err == nil {
err = errors.New("too many requests")
}
return &retryAfterError{
error: err,
retryAfter: retryAfter,

View File

@@ -2,6 +2,8 @@ package pacer
import (
"errors"
"fmt"
"strings"
"sync"
"testing"
"time"
@@ -350,3 +352,82 @@ func TestCallParallel(t *testing.T) {
assert.Equal(t, 5, called)
wait.Broadcast()
}
func TestRetryAfterError_NonNilErr(t *testing.T) {
orig := errors.New("test failure")
dur := 2 * time.Second
err := RetryAfterError(orig, dur)
rErr, ok := err.(*retryAfterError)
if !ok {
t.Fatalf("expected *retryAfterError, got %T", err)
}
if !strings.Contains(err.Error(), "test failure") {
t.Errorf("Error() = %q, want it to contain original message", err.Error())
}
if !strings.Contains(err.Error(), dur.String()) {
t.Errorf("Error() = %q, want it to contain retryAfter %v", err.Error(), dur)
}
if rErr.retryAfter != dur {
t.Errorf("retryAfter = %v, want %v", rErr.retryAfter, dur)
}
if !errors.Is(err, orig) {
t.Error("errors.Is(err, orig) = false, want true")
}
}
func TestRetryAfterError_NilErr(t *testing.T) {
dur := 5 * time.Second
err := RetryAfterError(nil, dur)
if !strings.Contains(err.Error(), "too many requests") {
t.Errorf("Error() = %q, want it to mention default message", err.Error())
}
if !strings.Contains(err.Error(), dur.String()) {
t.Errorf("Error() = %q, want it to contain retryAfter %v", err.Error(), dur)
}
}
func TestCauseMethod(t *testing.T) {
orig := errors.New("underlying")
dur := time.Second
rErr := RetryAfterError(orig, dur).(*retryAfterError)
cause := rErr.Cause()
if !errors.Is(cause, orig) {
t.Errorf("Cause() does not wrap original: got %v", cause)
}
}
func TestIsRetryAfter_True(t *testing.T) {
orig := errors.New("oops")
dur := 3 * time.Second
err := RetryAfterError(orig, dur)
gotDur, ok := IsRetryAfter(err)
if !ok {
t.Error("IsRetryAfter returned false, want true")
}
if gotDur != dur {
t.Errorf("got %v, want %v", gotDur, dur)
}
}
func TestIsRetryAfter_Nested(t *testing.T) {
orig := errors.New("fail")
dur := 4 * time.Second
retryErr := RetryAfterError(orig, dur)
nested := fmt.Errorf("wrapped: %w", retryErr)
gotDur, ok := IsRetryAfter(nested)
if !ok {
t.Error("IsRetryAfter on nested error returned false, want true")
}
if gotDur != dur {
t.Errorf("got %v, want %v", gotDur, dur)
}
}
func TestIsRetryAfter_False(t *testing.T) {
if _, ok := IsRetryAfter(errors.New("other")); ok {
t.Error("IsRetryAfter = true for non-retry error, want false")
}
}

156
rclone.1 generated
View File

@@ -1,7 +1,7 @@
.\"t
.\" Automatically generated by Pandoc 2.9.2.1
.\"
.TH "rclone" "1" "Jun 19, 2025" "User Manual" ""
.TH "rclone" "1" "Jul 09, 2025" "User Manual" ""
.hy
.SH NAME
.PP
@@ -252,10 +252,10 @@ Enterprise File Fabric
.IP \[bu] 2
Fastmail Files
.IP \[bu] 2
Files.com
.IP \[bu] 2
FileLu Cloud Storage
.IP \[bu] 2
Files.com
.IP \[bu] 2
FlashBlade
.IP \[bu] 2
FTP
@@ -747,6 +747,12 @@ status (https://repology.org/badge/vertical-allrepos/rclone.svg?columns=3)] (htt
The rclone developers maintain a docker image for
rclone (https://hub.docker.com/r/rclone/rclone).
.PP
\f[B]Note:\f[R] We also now offer a paid version of rclone with
enterprise-grade security and zero CVEs through our partner
SecureBuild (https://securebuild.com/blog/introducing-securebuild).
If you are interested, check out their website and the Rclone
SecureBuild Image (https://securebuild.com/images/rclone).
.PP
These images are built as part of the release process based on a minimal
Alpine Linux.
.PP
@@ -20145,9 +20151,18 @@ would use
This option is only supported Windows platforms.
.SS --use-json-log
.PP
This switches the log format to JSON for rclone.
The fields of JSON log are \f[C]level\f[R], \f[C]msg\f[R],
\f[C]source\f[R], \f[C]time\f[R].
This switches the log format to JSON.
The log messages are then streamed as individual JSON objects, with
fields: \f[C]level\f[R], \f[C]msg\f[R], \f[C]source\f[R], and
\f[C]time\f[R].
The resulting format is what is sometimes referred to as
newline-delimited
JSON (https://en.wikipedia.org/wiki/JSON_streaming#Newline-delimited_JSON)
(NDJSON), or JSON Lines (JSONL).
This is well suited for processing by traditional line-oriented tools
and shell pipelines, but a complete log file is not strictly valid JSON
and needs a parser that can handle it.
.PP
The JSON logs will be printed on a single line, but are shown expanded
here for clarity.
.IP
@@ -30210,7 +30225,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.70.1\[dq])
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.70.3\[dq])
\f[R]
.fi
.SS Performance
@@ -55710,6 +55725,28 @@ Rclone cannot delete files anywhere except under \f[C]album\f[R].
.PP
The Google Photos API does not support deleting albums - see bug
#135714733 (https://issuetracker.google.com/issues/135714733).
.SS Making your own client_id
.PP
When you use rclone with Google photos in its default configuration you
are using rclone\[aq]s client_id.
This is shared between all the rclone users.
There is a global rate limit on the number of queries per second that
each client_id can do set by Google.
.PP
If there is a problem with this client_id (eg quota too low or the
client_id stops working) then you can make your own.
.PP
Please follow the steps in the google drive
docs (https://rclone.org/drive/#making-your-own-client-id).
You will need these scopes instead of the drive ones detailed:
.IP
.nf
\f[C]
https://www.googleapis.com/auth/photoslibrary.appendonly
https://www.googleapis.com/auth/photoslibrary.readonly.appcreateddata
https://www.googleapis.com/auth/photoslibrary.edit.appcreateddata
\f[R]
.fi
.SH Hasher
.PP
Hasher is a special overlay backend to create remotes which handle
@@ -63750,8 +63787,14 @@ For example, you might see throttling.
To create your own Client ID, please follow these steps:
.IP "1." 3
Open
https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationsListBlade
and then click \f[C]New registration\f[R].
https://portal.azure.com/?quickstart=true#view/Microsoft_AAD_IAM/ActiveDirectoryMenuBlade/\[ti]/Overview
and then under the \f[C]Add\f[R] menu click \f[C]App registration\f[R].
.RS 4
.IP \[bu] 2
If you have not created an Azure account, you will be prompted to.
This is free, but you need to provide a phone number, address, and
credit card for identity verification.
.RE
.IP "2." 3
Enter a name for your app, choose account type
\f[C]Accounts in any organizational directory (Any Azure AD directory - Multitenant) and personal Microsoft accounts (e.g. Skype, Xbox)\f[R],
@@ -78205,6 +78248,101 @@ Options:
.IP \[bu] 2
\[dq]error\[dq]: return an error based on option value
.SH Changelog
.SS v1.70.3 - 2025-07-09
.PP
See commits (https://github.com/rclone/rclone/compare/v1.70.2...v1.70.3)
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
check: Fix difference report (was reporting error counts) (albertony)
.IP \[bu] 2
march: Fix deadlock when using \f[C]--no-traverse\f[R] (Nick Craig-Wood)
.IP \[bu] 2
doc fixes (albertony, Nick Craig-Wood)
.RE
.IP \[bu] 2
Azure Blob
.RS 2
.IP \[bu] 2
Fix server side copy error \[dq]requires exactly one scope\[dq] (Nick
Craig-Wood)
.RE
.IP \[bu] 2
B2
.RS 2
.IP \[bu] 2
Fix finding objects when using \f[C]--b2-version-at\f[R] (Davide
Bizzarri)
.RE
.IP \[bu] 2
Linkbox
.RS 2
.IP \[bu] 2
Fix upload error \[dq]user upload file not exist\[dq] (Nick Craig-Wood)
.RE
.IP \[bu] 2
Pikpak
.RS 2
.IP \[bu] 2
Improve error handling for missing links and unrecoverable 500s
(wiserain)
.RE
.IP \[bu] 2
WebDAV
.RS 2
.IP \[bu] 2
Fix setting modtime to that of local object instead of remote
(WeidiDeng)
.RE
.SS v1.70.2 - 2025-06-27
.PP
See commits (https://github.com/rclone/rclone/compare/v1.70.1...v1.70.2)
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
convmv: Make --dry-run logs less noisy (nielash)
.IP \[bu] 2
sync: Avoid copying dir metadata to itself (nielash)
.IP \[bu] 2
build: Bump github.com/go-chi/chi/v5 from 5.2.1 to 5.2.2 to fix
GHSA-vrw8-fxc6-2r93 (dependabot[bot])
.IP \[bu] 2
convmv: Fix moving to unicode-equivalent name (nielash)
.IP \[bu] 2
log: Fix deadlock when using systemd logging (Nick Craig-Wood)
.IP \[bu] 2
pacer: Fix nil pointer deref in RetryError (Nick Craig-Wood)
.IP \[bu] 2
doc fixes (Ali Zein Yousuf, Nick Craig-Wood)
.RE
.IP \[bu] 2
Local
.RS 2
.IP \[bu] 2
Fix --skip-links on Windows when skipping Junction points (Nick
Craig-Wood)
.RE
.IP \[bu] 2
Combine
.RS 2
.IP \[bu] 2
Fix directory not found errors with ListP interface (Nick Craig-Wood)
.RE
.IP \[bu] 2
Mega
.RS 2
.IP \[bu] 2
Fix tls handshake failure (necaran)
.RE
.IP \[bu] 2
Pikpak
.RS 2
.IP \[bu] 2
Fix uploads fail with \[dq]aws-chunked encoding is not supported\[dq]
error (Nick Craig-Wood)
.RE
.SS v1.70.1 - 2025-06-19
.PP
See commits (https://github.com/rclone/rclone/compare/v1.70.0...v1.70.1)