1
0
mirror of https://github.com/rclone/rclone.git synced 2026-01-05 10:03:17 +00:00

Compare commits

..

15 Commits

Author SHA1 Message Date
Nick Craig-Wood
73bcae2245 Version v1.72.1 2025-12-10 12:54:18 +00:00
Nick Craig-Wood
e8d6de09c6 s3: add more regions for Selectel 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
bc3305efd1 log: fix backtrace not going to the --log-file #9014
Before the log re-organisation in:

8d353039a6 log: add log rotation to --log-file

rclone would write any backtraces to the --log-file which was very
convenient for users.

This got accidentally disabled due to a typo which meant backtraces
started going to stderr even if --log-file was supplied.

This fixes the problem.
2025-12-10 12:44:23 +00:00
Nick Craig-Wood
3dd0232a26 build: fix lint warning after linter upgrade 2025-12-10 12:44:23 +00:00
Jonas Tingeborn
e6ce9d4a98 configfile: add piped config support - fixes #9012 2025-12-10 12:44:23 +00:00
Tingsong Xu
c37c6248fc fs/log: fix PID not included in JSON log output
When using `--log-format pid,json`, the PID was not being added to the JSON log output. This fix adds PID support to JSON logging.
2025-12-10 12:44:23 +00:00
Nick Craig-Wood
a8bece6585 build: adjust lint rules to exclude new errors from linter update 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
9bc877ad44 proxy: fix error handling in tests spotted by the linter 2025-12-10 12:44:23 +00:00
Johannes Rothe
2631392740 googlecloudstorage: improve endpoint parameter docs
When specifying a custom endpoint with a subpath, there is a limitation
in the Google cloud storage integration that the subpath is ignored
during upload operations. For example with the custom endpoint
"example.org/custom/endpoint" on upload the /custom/endpoint is not
reflected.

As this is most likely an issue with the underlying API client, there is
no way to fix this in rclone. By extending the documentation at least
rclone users are made aware of this limitation.

Related forum thread: https://forum.rclone.org/t/googlecloudstorage-custom-endpoint-subpath-removed-for-upload/53059
2025-12-10 12:44:23 +00:00
Nick Craig-Wood
1b3a5fe1fa docs: note where a provider has an S3 compatible alternative 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
615d5876a7 Add Shade as sponsor 2025-12-10 12:44:23 +00:00
Duncan Smart
ed36d2330f docs: Clarify OAuth scopes for readonly Google Drive access 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
532c16c1e9 docs: update sponsor logos 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
a08d66793b docs: fix lint error in changelog 2025-12-10 12:44:23 +00:00
Nick Craig-Wood
c4a2440492 Start v1.72.1-DEV development 2025-12-10 12:30:26 +00:00
58 changed files with 643 additions and 3221 deletions

View File

@@ -229,7 +229,7 @@ jobs:
cache: false cache: false
- name: Cache - name: Cache
uses: actions/cache@v5 uses: actions/cache@v4
with: with:
path: | path: |
~/go/pkg/mod ~/go/pkg/mod

View File

@@ -129,7 +129,7 @@ jobs:
- name: Load Go Build Cache for Docker - name: Load Go Build Cache for Docker
id: go-cache id: go-cache
uses: actions/cache@v5 uses: actions/cache@v4
with: with:
key: ${{ runner.os }}-${{ steps.imageos.outputs.result }}-go-${{ env.CACHE_NAME }}-${{ env.PLATFORM }}-${{ hashFiles('**/go.mod') }}-${{ hashFiles('**/go.sum') }} key: ${{ runner.os }}-${{ steps.imageos.outputs.result }}-go-${{ env.CACHE_NAME }}-${{ env.PLATFORM }}-${{ hashFiles('**/go.mod') }}-${{ hashFiles('**/go.sum') }}
restore-keys: | restore-keys: |
@@ -183,7 +183,7 @@ jobs:
touch "/tmp/digests/${digest#sha256:}" touch "/tmp/digests/${digest#sha256:}"
- name: Upload Image Digest - name: Upload Image Digest
uses: actions/upload-artifact@v6 uses: actions/upload-artifact@v5
with: with:
name: digests-${{ env.PLATFORM }} name: digests-${{ env.PLATFORM }}
path: /tmp/digests/* path: /tmp/digests/*
@@ -198,7 +198,7 @@ jobs:
steps: steps:
- name: Download Image Digests - name: Download Image Digests
uses: actions/download-artifact@v7 uses: actions/download-artifact@v6
with: with:
path: /tmp/digests path: /tmp/digests
pattern: digests-* pattern: digests-*

142
MANUAL.html generated
View File

@@ -233,7 +233,7 @@
<header id="title-block-header"> <header id="title-block-header">
<h1 class="title">rclone(1) User Manual</h1> <h1 class="title">rclone(1) User Manual</h1>
<p class="author">Nick Craig-Wood</p> <p class="author">Nick Craig-Wood</p>
<p class="date">Nov 21, 2025</p> <p class="date">Dec 10, 2025</p>
</header> </header>
<h1 id="name">NAME</h1> <h1 id="name">NAME</h1>
<p>rclone - manage files on cloud storage</p> <p>rclone - manage files on cloud storage</p>
@@ -4531,9 +4531,9 @@ SquareBracket</code></pre>
<pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!.txt&quot; --name-transform &quot;all,command=echo&quot; <pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!.txt&quot; --name-transform &quot;all,command=echo&quot;
// Output: stories/The Quick Brown Fox!.txt</code></pre> // Output: stories/The Quick Brown Fox!.txt</code></pre>
<pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!&quot; --name-transform &quot;date=-{YYYYMMDD}&quot; <pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!&quot; --name-transform &quot;date=-{YYYYMMDD}&quot;
// Output: stories/The Quick Brown Fox!-20251121</code></pre> // Output: stories/The Quick Brown Fox!-20251210</code></pre>
<pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!&quot; --name-transform &quot;date=-{macfriendlytime}&quot; <pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!&quot; --name-transform &quot;date=-{macfriendlytime}&quot;
// Output: stories/The Quick Brown Fox!-2025-11-21 0505PM</code></pre> // Output: stories/The Quick Brown Fox!-2025-12-10 1247PM</code></pre>
<pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!.txt&quot; --name-transform &quot;all,regex=[\\.\\w]/ab&quot; <pre class="console"><code>rclone convmv &quot;stories/The Quick Brown Fox!.txt&quot; --name-transform &quot;all,regex=[\\.\\w]/ab&quot;
// Output: ababababababab/ababab ababababab ababababab ababab!abababab</code></pre> // Output: ababababababab/ababab ababababab ababababab ababab!abababab</code></pre>
<p>The regex command generally accepts Perl-style regular expressions, <p>The regex command generally accepts Perl-style regular expressions,
@@ -22567,7 +22567,7 @@ split into groups.</p>
--tpslimit float Limit HTTP transactions per second to this --tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar --use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.72.0&quot;)</code></pre> --user-agent string Set the user-agent to a specified string (default &quot;rclone/v1.72.1&quot;)</code></pre>
<h2 id="performance">Performance</h2> <h2 id="performance">Performance</h2>
<p>Flags helpful for increasing performance.</p> <p>Flags helpful for increasing performance.</p>
<pre><code> --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi) <pre><code> --buffer-size SizeSuffix In memory buffer size when reading files for each --transfer (default 16Mi)
@@ -23024,7 +23024,7 @@ split into groups.</p>
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don&#39;t attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don&#39;t attempt to check the bucket exists or create it
@@ -25234,7 +25234,29 @@ investigation:</p>
<li><a <li><a
href="https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt"><code>TestBisyncRemoteRemote/normalization</code></a></li> href="https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt"><code>TestBisyncRemoteRemote/normalization</code></a></li>
</ul></li> </ul></li>
<li>Updated: 2025-11-21-010037 <li><code>TestGoFile</code> (<code>gofile</code>)
<ul>
<li><a
href="https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt"><code>TestBisyncRemoteLocal/all_changed</code></a></li>
<li><a
href="https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt"><code>TestBisyncRemoteLocal/backupdir</code></a></li>
<li><a
href="https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt"><code>TestBisyncRemoteLocal/basic</code></a></li>
<li><a
href="https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt"><code>TestBisyncRemoteLocal/changes</code></a></li>
<li><a
href="https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt"><code>TestBisyncRemoteLocal/check_access</code></a></li>
<li><a href="https://pub.rclone.org/integration-tests/current/">78
more</a></li>
</ul></li>
<li><code>TestPcloud</code> (<code>pcloud</code>)
<ul>
<li><a
href="https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt"><code>TestBisyncRemoteRemote/check_access</code></a></li>
<li><a
href="https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt"><code>TestBisyncRemoteRemote/check_access_filters</code></a></li>
</ul></li>
<li>Updated: 2025-12-10-010012
<!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---></li> <!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---></li>
</ul> </ul>
<p>The following backends either have not been tested recently or have <p>The following backends either have not been tested recently or have
@@ -28374,15 +28396,30 @@ centers for low latency.</li>
<li>St. Petersburg</li> <li>St. Petersburg</li>
<li>Provider: Selectel,Servercore</li> <li>Provider: Selectel,Servercore</li>
</ul></li> </ul></li>
<li>"gis-1" <li>"ru-3"
<ul> <ul>
<li>Moscow</li> <li>St. Petersburg</li>
<li>Provider: Servercore</li> <li>Provider: Selectel</li>
</ul></li> </ul></li>
<li>"ru-7" <li>"ru-7"
<ul> <ul>
<li>Moscow</li> <li>Moscow</li>
<li>Provider: Servercore</li> <li>Provider: Selectel,Servercore</li>
</ul></li>
<li>"gis-1"
<ul>
<li>Moscow</li>
<li>Provider: Selectel,Servercore</li>
</ul></li>
<li>"kz-1"
<ul>
<li>Kazakhstan</li>
<li>Provider: Selectel</li>
</ul></li>
<li>"uz-2"
<ul>
<li>Uzbekistan</li>
<li>Provider: Selectel</li>
</ul></li> </ul></li>
<li>"uz-2" <li>"uz-2"
<ul> <ul>
@@ -29690,17 +29727,37 @@ AWS,Alibaba,ArvanCloud,Ceph,ChinaMobile,Cloudflare,Cubbit,DigitalOcean,Dreamhost
</ul></li> </ul></li>
<li>"s3.ru-1.storage.selcloud.ru" <li>"s3.ru-1.storage.selcloud.ru"
<ul> <ul>
<li>Saint Petersburg</li> <li>St. Petersburg</li>
<li>Provider: Selectel</li>
</ul></li>
<li>"s3.ru-3.storage.selcloud.ru"
<ul>
<li>St. Petersburg</li>
<li>Provider: Selectel</li>
</ul></li>
<li>"s3.ru-7.storage.selcloud.ru"
<ul>
<li>Moscow</li>
<li>Provider: Selectel,Servercore</li> <li>Provider: Selectel,Servercore</li>
</ul></li> </ul></li>
<li>"s3.gis-1.storage.selcloud.ru" <li>"s3.gis-1.storage.selcloud.ru"
<ul> <ul>
<li>Moscow</li> <li>Moscow</li>
<li>Provider: Servercore</li> <li>Provider: Selectel,Servercore</li>
</ul></li> </ul></li>
<li>"s3.ru-7.storage.selcloud.ru" <li>"s3.kz-1.storage.selcloud.ru"
<ul> <ul>
<li>Moscow</li> <li>Kazakhstan</li>
<li>Provider: Selectel</li>
</ul></li>
<li>"s3.uz-2.storage.selcloud.ru"
<ul>
<li>Uzbekistan</li>
<li>Provider: Selectel</li>
</ul></li>
<li>"s3.ru-1.storage.selcloud.ru"
<ul>
<li>Saint Petersburg</li>
<li>Provider: Servercore</li> <li>Provider: Servercore</li>
</ul></li> </ul></li>
<li>"s3.uz-2.srvstorage.uz" <li>"s3.uz-2.srvstorage.uz"
@@ -41496,6 +41553,9 @@ storage options, and sharing capabilities. With support for high storage
limits and seamless integration with rclone, FileLu makes managing files limits and seamless integration with rclone, FileLu makes managing files
in the cloud easy. Its cross-platform file backup services let you in the cloud easy. Its cross-platform file backup services let you
upload and back up files from any internet-connected device.</p> upload and back up files from any internet-connected device.</p>
<p><strong>Note</strong> FileLu now has a fully featured S3 backend <a
href="/s3#filelu-s5">FileLu S5</a>, an industry standard S3 compatible
object store.</p>
<h2 id="configuration-16">Configuration</h2> <h2 id="configuration-16">Configuration</h2>
<p>Here is an example of how to make a remote called <p>Here is an example of how to make a remote called
<code>filelu</code>. First, run:</p> <code>filelu</code>. First, run:</p>
@@ -43418,14 +43478,36 @@ decompressed.</p>
<li>Default: false</li> <li>Default: false</li>
</ul> </ul>
<h4 id="gcs-endpoint">--gcs-endpoint</h4> <h4 id="gcs-endpoint">--gcs-endpoint</h4>
<p>Endpoint for the service.</p> <p>Custom endpoint for the storage API. Leave blank to use the provider
<p>Leave blank normally.</p> default.</p>
<p>When using a custom endpoint that includes a subpath (e.g.
example.org/custom/endpoint), the subpath will be ignored during upload
operations due to a limitation in the underlying Google API Go client
library. Download and listing operations will work correctly with the
full endpoint path. If you require subpath support for uploads, avoid
using subpaths in your custom endpoint configuration.</p>
<p>Properties:</p> <p>Properties:</p>
<ul> <ul>
<li>Config: endpoint</li> <li>Config: endpoint</li>
<li>Env Var: RCLONE_GCS_ENDPOINT</li> <li>Env Var: RCLONE_GCS_ENDPOINT</li>
<li>Type: string</li> <li>Type: string</li>
<li>Required: false</li> <li>Required: false</li>
<li>Examples:
<ul>
<li>"storage.example.org"
<ul>
<li>Specify a custom endpoint</li>
</ul></li>
<li>"storage.example.org:4443"
<ul>
<li>Specifying a custom endpoint with port</li>
</ul></li>
<li>"storage.example.org:4443/gcs/api"
<ul>
<li>Specifying a subpath, see the note, uploads won't use the custom
path!</li>
</ul></li>
</ul></li>
</ul> </ul>
<h4 id="gcs-encoding">--gcs-encoding</h4> <h4 id="gcs-encoding">--gcs-encoding</h4>
<p>The encoding for the backend.</p> <p>The encoding for the backend.</p>
@@ -43670,7 +43752,7 @@ account. It is a ~21 character numerical string.</li>
<code>https://www.googleapis.com/auth/drive</code> to grant read/write <code>https://www.googleapis.com/auth/drive</code> to grant read/write
access to Google Drive specifically. You can also use access to Google Drive specifically. You can also use
<code>https://www.googleapis.com/auth/drive.readonly</code> for read <code>https://www.googleapis.com/auth/drive.readonly</code> for read
only access.</li> only access with <code>--drive-scope=drive.readonly</code>.</li>
<li>Click "Authorise"</li> <li>Click "Authorise"</li>
</ul> </ul>
<h5 id="configure-rclone-assuming-a-new-install">3. Configure rclone, <h5 id="configure-rclone-assuming-a-new-install">3. Configure rclone,
@@ -62593,6 +62675,32 @@ the output.</p>
<!-- autogenerated options stop --> <!-- autogenerated options stop -->
<!-- markdownlint-disable line-length --> <!-- markdownlint-disable line-length -->
<h1 id="changelog-1">Changelog</h1> <h1 id="changelog-1">Changelog</h1>
<h2 id="v1.72.1---2025-12-10">v1.72.1 - 2025-12-10</h2>
<p><a
href="https://github.com/rclone/rclone/compare/v1.72.0...v1.72.1">See
commits</a></p>
<ul>
<li>Bug Fixes
<ul>
<li>build: update to go1.25.5 to fix <a
href="https://pkg.go.dev/vuln/GO-2025-4155">CVE-2025-61729</a></li>
<li>doc fixes (Duncan Smart, Nick Craig-Wood)</li>
<li>configfile: Fix piped config support (Jonas Tingeborn)</li>
<li>log
<ul>
<li>Fix PID not included in JSON log output (Tingsong Xu)</li>
<li>Fix backtrace not going to the --log-file (Nick Craig-Wood)</li>
</ul></li>
</ul></li>
<li>Google Cloud Storage
<ul>
<li>Improve endpoint parameter docs (Johannes Rothe)</li>
</ul></li>
<li>S3
<ul>
<li>Add missing regions for Selectel provider (Nick Craig-Wood)</li>
</ul></li>
</ul>
<h2 id="v1.72.0---2025-11-21">v1.72.0 - 2025-11-21</h2> <h2 id="v1.72.0---2025-11-21">v1.72.0 - 2025-11-21</h2>
<p><a <p><a
href="https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0">See href="https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0">See

98
MANUAL.md generated
View File

@@ -1,6 +1,6 @@
% rclone(1) User Manual % rclone(1) User Manual
% Nick Craig-Wood % Nick Craig-Wood
% Nov 21, 2025 % Dec 10, 2025
# NAME # NAME
@@ -5369,12 +5369,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20251121 // Output: stories/The Quick Brown Fox!-20251210
``` ```
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2025-11-21 0505PM // Output: stories/The Quick Brown Fox!-2025-12-10 1247PM
``` ```
```console ```console
@@ -24802,7 +24802,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this --tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar --use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.72.0") --user-agent string Set the user-agent to a specified string (default "rclone/v1.72.1")
``` ```
@@ -25319,7 +25319,7 @@ Backend-only flags (these can be set in the config file also).
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it
@@ -27514,7 +27514,17 @@ The following backends have known issues that need more investigation:
<!--- start list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---> <!--- start list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
- `TestDropbox` (`dropbox`) - `TestDropbox` (`dropbox`)
- [`TestBisyncRemoteRemote/normalization`](https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt) - [`TestBisyncRemoteRemote/normalization`](https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt)
- Updated: 2025-11-21-010037 - `TestGoFile` (`gofile`)
- [`TestBisyncRemoteLocal/all_changed`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/backupdir`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/basic`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/changes`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/check_access`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [78 more](https://pub.rclone.org/integration-tests/current/)
- `TestPcloud` (`pcloud`)
- [`TestBisyncRemoteRemote/check_access`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- [`TestBisyncRemoteRemote/check_access_filters`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- Updated: 2025-12-10-010012
<!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---> <!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
The following backends either have not been tested recently or have known issues The following backends either have not been tested recently or have known issues
@@ -30343,12 +30353,21 @@ Properties:
- "ru-1" - "ru-1"
- St. Petersburg - St. Petersburg
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "gis-1" - "ru-3"
- Moscow - St. Petersburg
- Provider: Servercore - Provider: Selectel
- "ru-7" - "ru-7"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "gis-1"
- Moscow
- Provider: Selectel,Servercore
- "kz-1"
- Kazakhstan
- Provider: Selectel
- "uz-2"
- Uzbekistan
- Provider: Selectel
- "uz-2" - "uz-2"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan
- Provider: Servercore - Provider: Servercore
@@ -31140,13 +31159,25 @@ Properties:
- SeaweedFS S3 localhost - SeaweedFS S3 localhost
- Provider: SeaweedFS - Provider: SeaweedFS
- "s3.ru-1.storage.selcloud.ru" - "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg - St. Petersburg
- Provider: Selectel
- "s3.ru-3.storage.selcloud.ru"
- St. Petersburg
- Provider: Selectel
- "s3.ru-7.storage.selcloud.ru"
- Moscow
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "s3.gis-1.storage.selcloud.ru" - "s3.gis-1.storage.selcloud.ru"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "s3.ru-7.storage.selcloud.ru" - "s3.kz-1.storage.selcloud.ru"
- Moscow - Kazakhstan
- Provider: Selectel
- "s3.uz-2.storage.selcloud.ru"
- Uzbekistan
- Provider: Selectel
- "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg
- Provider: Servercore - Provider: Servercore
- "s3.uz-2.srvstorage.uz" - "s3.uz-2.srvstorage.uz"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan
@@ -43973,6 +44004,9 @@ managing files in the cloud easy. Its cross-platform file backup
services let you upload and back up files from any internet-connected services let you upload and back up files from any internet-connected
device. device.
**Note** FileLu now has a fully featured S3 backend [FileLu S5](/s3#filelu-s5),
an industry standard S3 compatible object store.
## Configuration ## Configuration
Here is an example of how to make a remote called `filelu`. First, run: Here is an example of how to make a remote called `filelu`. First, run:
@@ -46071,9 +46105,14 @@ Properties:
#### --gcs-endpoint #### --gcs-endpoint
Endpoint for the service. Custom endpoint for the storage API. Leave blank to use the provider default.
Leave blank normally. When using a custom endpoint that includes a subpath (e.g. example.org/custom/endpoint),
the subpath will be ignored during upload operations due to a limitation in the
underlying Google API Go client library.
Download and listing operations will work correctly with the full endpoint path.
If you require subpath support for uploads, avoid using subpaths in your custom
endpoint configuration.
Properties: Properties:
@@ -46081,6 +46120,13 @@ Properties:
- Env Var: RCLONE_GCS_ENDPOINT - Env Var: RCLONE_GCS_ENDPOINT
- Type: string - Type: string
- Required: false - Required: false
- Examples:
- "storage.example.org"
- Specify a custom endpoint
- "storage.example.org:4443"
- Specifying a custom endpoint with port
- "storage.example.org:4443/gcs/api"
- Specifying a subpath, see the note, uploads won't use the custom path!
#### --gcs-encoding #### --gcs-encoding
@@ -46379,7 +46425,7 @@ account key" button.
`https://www.googleapis.com/auth/drive` `https://www.googleapis.com/auth/drive`
to grant read/write access to Google Drive specifically. to grant read/write access to Google Drive specifically.
You can also use `https://www.googleapis.com/auth/drive.readonly` for read You can also use `https://www.googleapis.com/auth/drive.readonly` for read
only access. only access with `--drive-scope=drive.readonly`.
- Click "Authorise" - Click "Authorise"
##### 3. Configure rclone, assuming a new install ##### 3. Configure rclone, assuming a new install
@@ -66867,6 +66913,22 @@ Options:
# Changelog # Changelog
## v1.72.1 - 2025-12-10
[See commits](https://github.com/rclone/rclone/compare/v1.72.0...v1.72.1)
- Bug Fixes
- build: update to go1.25.5 to fix [CVE-2025-61729](https://pkg.go.dev/vuln/GO-2025-4155)
- doc fixes (Duncan Smart, Nick Craig-Wood)
- configfile: Fix piped config support (Jonas Tingeborn)
- log
- Fix PID not included in JSON log output (Tingsong Xu)
- Fix backtrace not going to the --log-file (Nick Craig-Wood)
- Google Cloud Storage
- Improve endpoint parameter docs (Johannes Rothe)
- S3
- Add missing regions for Selectel provider (Nick Craig-Wood)
## v1.72.0 - 2025-11-21 ## v1.72.0 - 2025-11-21
[See commits](https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0) [See commits](https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0)
@@ -66887,7 +66949,7 @@ Options:
- [rclone test speed](https://rclone.org/commands/rclone_test_speed/): Add command to test a specified remotes speed (dougal) - [rclone test speed](https://rclone.org/commands/rclone_test_speed/): Add command to test a specified remotes speed (dougal)
- New Features - New Features
- backends: many backends have has a paged listing (`ListP`) interface added - backends: many backends have has a paged listing (`ListP`) interface added
- this enables progress when listing large directories and reduced memory usage - this enables progress when listing large directories and reduced memory usage
- build - build
- Bump golang.org/x/crypto from 0.43.0 to 0.45.0 to fix CVE-2025-58181 (dependabot[bot]) - Bump golang.org/x/crypto from 0.43.0 to 0.45.0 to fix CVE-2025-58181 (dependabot[bot])
- Modernize code and tests (Nick Craig-Wood, russcoss, juejinyuxitu, reddaisyy, dulanting, Oleksandr Redko) - Modernize code and tests (Nick Craig-Wood, russcoss, juejinyuxitu, reddaisyy, dulanting, Oleksandr Redko)

99
MANUAL.txt generated
View File

@@ -1,6 +1,6 @@
rclone(1) User Manual rclone(1) User Manual
Nick Craig-Wood Nick Craig-Wood
Nov 21, 2025 Dec 10, 2025
NAME NAME
@@ -4588,10 +4588,10 @@ Examples:
// Output: stories/The Quick Brown Fox!.txt // Output: stories/The Quick Brown Fox!.txt
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20251121 // Output: stories/The Quick Brown Fox!-20251210
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2025-11-21 0505PM // Output: stories/The Quick Brown Fox!-2025-12-10 1247PM
rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,regex=[\\.\\w]/ab" rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,regex=[\\.\\w]/ab"
// Output: ababababababab/ababab ababababab ababababab ababab!abababab // Output: ababababababab/ababab ababababab ababababab ababab!abababab
@@ -23110,7 +23110,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this --tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar --use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.72.0") --user-agent string Set the user-agent to a specified string (default "rclone/v1.72.1")
Performance Performance
@@ -23597,7 +23597,7 @@ Backend-only flags (these can be set in the config file also).
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it
@@ -25734,7 +25734,17 @@ The following backends have known issues that need more investigation:
- TestDropbox (dropbox) - TestDropbox (dropbox)
- TestBisyncRemoteRemote/normalization - TestBisyncRemoteRemote/normalization
- Updated: 2025-11-21-010037 - TestGoFile (gofile)
- TestBisyncRemoteLocal/all_changed
- TestBisyncRemoteLocal/backupdir
- TestBisyncRemoteLocal/basic
- TestBisyncRemoteLocal/changes
- TestBisyncRemoteLocal/check_access
- 78 more
- TestPcloud (pcloud)
- TestBisyncRemoteRemote/check_access
- TestBisyncRemoteRemote/check_access_filters
- Updated: 2025-12-10-010012
The following backends either have not been tested recently or have The following backends either have not been tested recently or have
known issues that are deemed unfixable for the time being: known issues that are deemed unfixable for the time being:
@@ -28517,12 +28527,21 @@ Properties:
- "ru-1" - "ru-1"
- St. Petersburg - St. Petersburg
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "gis-1" - "ru-3"
- Moscow - St. Petersburg
- Provider: Servercore - Provider: Selectel
- "ru-7" - "ru-7"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "gis-1"
- Moscow
- Provider: Selectel,Servercore
- "kz-1"
- Kazakhstan
- Provider: Selectel
- "uz-2"
- Uzbekistan
- Provider: Selectel
- "uz-2" - "uz-2"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan
- Provider: Servercore - Provider: Servercore
@@ -29315,13 +29334,25 @@ Properties:
- SeaweedFS S3 localhost - SeaweedFS S3 localhost
- Provider: SeaweedFS - Provider: SeaweedFS
- "s3.ru-1.storage.selcloud.ru" - "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg - St. Petersburg
- Provider: Selectel
- "s3.ru-3.storage.selcloud.ru"
- St. Petersburg
- Provider: Selectel
- "s3.ru-7.storage.selcloud.ru"
- Moscow
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "s3.gis-1.storage.selcloud.ru" - "s3.gis-1.storage.selcloud.ru"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "s3.ru-7.storage.selcloud.ru" - "s3.kz-1.storage.selcloud.ru"
- Moscow - Kazakhstan
- Provider: Selectel
- "s3.uz-2.storage.selcloud.ru"
- Uzbekistan
- Provider: Selectel
- "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg
- Provider: Servercore - Provider: Servercore
- "s3.uz-2.srvstorage.uz" - "s3.uz-2.srvstorage.uz"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan
@@ -41720,6 +41751,9 @@ integration with rclone, FileLu makes managing files in the cloud easy.
Its cross-platform file backup services let you upload and back up files Its cross-platform file backup services let you upload and back up files
from any internet-connected device. from any internet-connected device.
Note FileLu now has a fully featured S3 backend FileLu S5, an industry
standard S3 compatible object store.
Configuration Configuration
Here is an example of how to make a remote called filelu. First, run: Here is an example of how to make a remote called filelu. First, run:
@@ -43696,9 +43730,15 @@ Properties:
--gcs-endpoint --gcs-endpoint
Endpoint for the service. Custom endpoint for the storage API. Leave blank to use the provider
default.
Leave blank normally. When using a custom endpoint that includes a subpath (e.g.
example.org/custom/endpoint), the subpath will be ignored during upload
operations due to a limitation in the underlying Google API Go client
library. Download and listing operations will work correctly with the
full endpoint path. If you require subpath support for uploads, avoid
using subpaths in your custom endpoint configuration.
Properties: Properties:
@@ -43706,6 +43746,14 @@ Properties:
- Env Var: RCLONE_GCS_ENDPOINT - Env Var: RCLONE_GCS_ENDPOINT
- Type: string - Type: string
- Required: false - Required: false
- Examples:
- "storage.example.org"
- Specify a custom endpoint
- "storage.example.org:4443"
- Specifying a custom endpoint with port
- "storage.example.org:4443/gcs/api"
- Specifying a subpath, see the note, uploads won't use the
custom path!
--gcs-encoding --gcs-encoding
@@ -43989,7 +44037,8 @@ key" button.
- In the next field, "OAuth Scopes", enter - In the next field, "OAuth Scopes", enter
https://www.googleapis.com/auth/drive to grant read/write access to https://www.googleapis.com/auth/drive to grant read/write access to
Google Drive specifically. You can also use Google Drive specifically. You can also use
https://www.googleapis.com/auth/drive.readonly for read only access. https://www.googleapis.com/auth/drive.readonly for read only access
with --drive-scope=drive.readonly.
- Click "Authorise" - Click "Authorise"
3. Configure rclone, assuming a new install 3. Configure rclone, assuming a new install
@@ -64010,6 +64059,22 @@ Options:
Changelog Changelog
v1.72.1 - 2025-12-10
See commits
- Bug Fixes
- build: update to go1.25.5 to fix CVE-2025-61729
- doc fixes (Duncan Smart, Nick Craig-Wood)
- configfile: Fix piped config support (Jonas Tingeborn)
- log
- Fix PID not included in JSON log output (Tingsong Xu)
- Fix backtrace not going to the --log-file (Nick Craig-Wood)
- Google Cloud Storage
- Improve endpoint parameter docs (Johannes Rothe)
- S3
- Add missing regions for Selectel provider (Nick Craig-Wood)
v1.72.0 - 2025-11-21 v1.72.0 - 2025-11-21
See commits See commits

View File

@@ -109,7 +109,6 @@ directories to and from different cloud storage providers.
- Selectel Object Storage [:page_facing_up:](https://rclone.org/s3/#selectel) - Selectel Object Storage [:page_facing_up:](https://rclone.org/s3/#selectel)
- Servercore Object Storage [:page_facing_up:](https://rclone.org/s3/#servercore) - Servercore Object Storage [:page_facing_up:](https://rclone.org/s3/#servercore)
- SFTP [:page_facing_up:](https://rclone.org/sftp/) - SFTP [:page_facing_up:](https://rclone.org/sftp/)
- Shade [:page_facing_up:](https://rclone.org/shade/)
- SMB / CIFS [:page_facing_up:](https://rclone.org/smb/) - SMB / CIFS [:page_facing_up:](https://rclone.org/smb/)
- Spectra Logic [:page_facing_up:](https://rclone.org/s3/#spectralogic) - Spectra Logic [:page_facing_up:](https://rclone.org/s3/#spectralogic)
- StackPath [:page_facing_up:](https://rclone.org/s3/#stackpath) - StackPath [:page_facing_up:](https://rclone.org/s3/#stackpath)

View File

@@ -1 +1 @@
v1.73.0 v1.72.1

View File

@@ -55,7 +55,6 @@ import (
_ "github.com/rclone/rclone/backend/s3" _ "github.com/rclone/rclone/backend/s3"
_ "github.com/rclone/rclone/backend/seafile" _ "github.com/rclone/rclone/backend/seafile"
_ "github.com/rclone/rclone/backend/sftp" _ "github.com/rclone/rclone/backend/sftp"
_ "github.com/rclone/rclone/backend/shade"
_ "github.com/rclone/rclone/backend/sharefile" _ "github.com/rclone/rclone/backend/sharefile"
_ "github.com/rclone/rclone/backend/sia" _ "github.com/rclone/rclone/backend/sia"
_ "github.com/rclone/rclone/backend/smb" _ "github.com/rclone/rclone/backend/smb"

View File

@@ -86,56 +86,12 @@ var (
metadataMu sync.Mutex metadataMu sync.Mutex
) )
// system metadata keys which this backend owns
var systemMetadataInfo = map[string]fs.MetadataHelp{
"cache-control": {
Help: "Cache-Control header",
Type: "string",
Example: "no-cache",
},
"content-disposition": {
Help: "Content-Disposition header",
Type: "string",
Example: "inline",
},
"content-encoding": {
Help: "Content-Encoding header",
Type: "string",
Example: "gzip",
},
"content-language": {
Help: "Content-Language header",
Type: "string",
Example: "en-US",
},
"content-type": {
Help: "Content-Type header",
Type: "string",
Example: "text/plain",
},
"tier": {
Help: "Tier of the object",
Type: "string",
Example: "Hot",
ReadOnly: true,
},
"mtime": {
Help: "Time of last modification, read from rclone metadata",
Type: "RFC 3339",
Example: "2006-01-02T15:04:05.999999999Z07:00",
},
}
// Register with Fs // Register with Fs
func init() { func init() {
fs.Register(&fs.RegInfo{ fs.Register(&fs.RegInfo{
Name: "azureblob", Name: "azureblob",
Description: "Microsoft Azure Blob Storage", Description: "Microsoft Azure Blob Storage",
NewFs: NewFs, NewFs: NewFs,
MetadataInfo: &fs.MetadataInfo{
System: systemMetadataInfo,
Help: `User metadata is stored as x-ms-meta- keys. Azure metadata keys are case insensitive and are always returned in lower case.`,
},
Options: []fs.Option{{ Options: []fs.Option{{
Name: "account", Name: "account",
Help: `Azure Storage Account Name. Help: `Azure Storage Account Name.
@@ -854,9 +810,6 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
f.features = (&fs.Features{ f.features = (&fs.Features{
ReadMimeType: true, ReadMimeType: true,
WriteMimeType: true, WriteMimeType: true,
ReadMetadata: true,
WriteMetadata: true,
UserMetadata: true,
BucketBased: true, BucketBased: true,
BucketBasedRootOK: true, BucketBasedRootOK: true,
SetTier: true, SetTier: true,
@@ -1204,289 +1157,6 @@ func (o *Object) updateMetadataWithModTime(modTime time.Time) {
o.meta[modTimeKey] = modTime.Format(timeFormatOut) o.meta[modTimeKey] = modTime.Format(timeFormatOut)
} }
// parseXMsTags parses the value of the x-ms-tags header into a map.
// It expects comma-separated key=value pairs. Whitespace around keys and
// values is trimmed. Empty pairs and empty keys are rejected.
func parseXMsTags(s string) (map[string]string, error) {
if strings.TrimSpace(s) == "" {
return map[string]string{}, nil
}
out := make(map[string]string)
parts := strings.Split(s, ",")
for _, p := range parts {
p = strings.TrimSpace(p)
if p == "" {
continue
}
kv := strings.SplitN(p, "=", 2)
if len(kv) != 2 {
return nil, fmt.Errorf("invalid tag %q", p)
}
k := strings.TrimSpace(kv[0])
v := strings.TrimSpace(kv[1])
if k == "" {
return nil, fmt.Errorf("invalid tag key in %q", p)
}
out[k] = v
}
return out, nil
}
// mapMetadataToAzure maps a generic metadata map to Azure HTTP headers,
// user metadata, tags and optional modTime override.
// Reserved x-ms-* keys (except x-ms-tags) are ignored for user metadata.
//
// Pass a logger to surface non-fatal parsing issues (e.g. bad mtime).
func mapMetadataToAzure(meta map[string]string, logf func(string, ...any)) (headers blob.HTTPHeaders, userMeta map[string]*string, tags map[string]string, modTime *time.Time, err error) {
if meta == nil {
return headers, nil, nil, nil, nil
}
tmp := make(map[string]string)
for k, v := range meta {
lowerKey := strings.ToLower(k)
switch lowerKey {
case "cache-control":
headers.BlobCacheControl = pString(v)
case "content-disposition":
headers.BlobContentDisposition = pString(v)
case "content-encoding":
headers.BlobContentEncoding = pString(v)
case "content-language":
headers.BlobContentLanguage = pString(v)
case "content-type":
headers.BlobContentType = pString(v)
case "x-ms-tags":
parsed, perr := parseXMsTags(v)
if perr != nil {
return headers, nil, nil, nil, perr
}
// allocate only if there are tags
if len(parsed) > 0 {
tags = parsed
}
case "mtime":
// Accept multiple layouts for tolerance
var parsed time.Time
var pErr error
for _, layout := range []string{time.RFC3339Nano, time.RFC3339, timeFormatOut} {
parsed, pErr = time.Parse(layout, v)
if pErr == nil {
modTime = &parsed
break
}
}
// Log and ignore if unparseable
if modTime == nil && logf != nil {
logf("metadata: couldn't parse mtime %q: %v", v, pErr)
}
case "tier":
// ignore - handled elsewhere
default:
// Filter out other reserved headers so they don't end up as user metadata
if strings.HasPrefix(lowerKey, "x-ms-") {
continue
}
tmp[lowerKey] = v
}
}
userMeta = toAzureMetaPtr(tmp)
return headers, userMeta, tags, modTime, nil
}
// toAzureMetaPtr converts a map[string]string to map[string]*string as used by Azure SDK
func toAzureMetaPtr(in map[string]string) map[string]*string {
if len(in) == 0 {
return nil
}
out := make(map[string]*string, len(in))
for k, v := range in {
vv := v
out[k] = &vv
}
return out
}
// assembleCopyParams prepares headers, metadata and tags for copy operations.
//
// It starts from the source properties, optionally overlays mapped metadata
// from rclone's metadata options, ensures mtime presence when mapping is
// enabled, and returns whether mapping was actually requested (hadMapping).
// assembleCopyParams prepares headers, metadata and tags for copy operations.
//
// If includeBaseMeta is true, start user metadata from the source's metadata
// and overlay mapped values. This matches multipart copy commit behavior.
// If false, only include mapped user metadata (no source baseline) which
// matches previous singlepart StartCopyFromURL semantics.
func assembleCopyParams(ctx context.Context, f *Fs, src fs.Object, srcProps *blob.GetPropertiesResponse, includeBaseMeta bool) (headers blob.HTTPHeaders, meta map[string]*string, tags map[string]string, hadMapping bool, err error) {
// Start from source properties
headers = blob.HTTPHeaders{
BlobCacheControl: srcProps.CacheControl,
BlobContentDisposition: srcProps.ContentDisposition,
BlobContentEncoding: srcProps.ContentEncoding,
BlobContentLanguage: srcProps.ContentLanguage,
BlobContentMD5: srcProps.ContentMD5,
BlobContentType: srcProps.ContentType,
}
// Optionally deep copy user metadata pointers from source. Normalise keys to
// lower-case to avoid duplicate x-ms-meta headers when we later inject/overlay
// metadata (Azure treats keys case-insensitively but Go's http.Header will
// join duplicate keys into a comma separated list, which breaks shared-key
// signing).
if includeBaseMeta && len(srcProps.Metadata) > 0 {
meta = make(map[string]*string, len(srcProps.Metadata))
for k, v := range srcProps.Metadata {
if v != nil {
vv := *v
meta[strings.ToLower(k)] = &vv
}
}
}
// Only consider mapping if metadata pipeline is enabled
if fs.GetConfig(ctx).Metadata {
mapped, mapErr := fs.GetMetadataOptions(ctx, f, src, fs.MetadataAsOpenOptions(ctx))
if mapErr != nil {
return headers, meta, nil, false, fmt.Errorf("failed to map metadata: %w", mapErr)
}
if mapped != nil {
// Map rclone metadata to Azure shapes
mappedHeaders, userMeta, mappedTags, mappedModTime, herr := mapMetadataToAzure(mapped, func(format string, args ...any) { fs.Debugf(f, format, args...) })
if herr != nil {
return headers, meta, nil, false, fmt.Errorf("metadata mapping: %w", herr)
}
hadMapping = true
// Overlay headers (only non-nil)
if mappedHeaders.BlobCacheControl != nil {
headers.BlobCacheControl = mappedHeaders.BlobCacheControl
}
if mappedHeaders.BlobContentDisposition != nil {
headers.BlobContentDisposition = mappedHeaders.BlobContentDisposition
}
if mappedHeaders.BlobContentEncoding != nil {
headers.BlobContentEncoding = mappedHeaders.BlobContentEncoding
}
if mappedHeaders.BlobContentLanguage != nil {
headers.BlobContentLanguage = mappedHeaders.BlobContentLanguage
}
if mappedHeaders.BlobContentType != nil {
headers.BlobContentType = mappedHeaders.BlobContentType
}
// Overlay user metadata
if len(userMeta) > 0 {
if meta == nil {
meta = make(map[string]*string, len(userMeta))
}
for k, v := range userMeta {
meta[k] = v
}
}
// Apply tags if any
if len(mappedTags) > 0 {
tags = mappedTags
}
// Ensure mtime present using mapped or source time
if _, ok := meta[modTimeKey]; !ok {
when := src.ModTime(ctx)
if mappedModTime != nil {
when = *mappedModTime
}
val := when.Format(time.RFC3339Nano)
if meta == nil {
meta = make(map[string]*string, 1)
}
meta[modTimeKey] = &val
}
// Ensure content-type fallback to source if not set by mapper
if headers.BlobContentType == nil {
headers.BlobContentType = srcProps.ContentType
}
} else {
// Mapping enabled but not provided: ensure mtime present based on source ModTime
if _, ok := meta[modTimeKey]; !ok {
when := src.ModTime(ctx)
val := when.Format(time.RFC3339Nano)
if meta == nil {
meta = make(map[string]*string, 1)
}
meta[modTimeKey] = &val
}
}
}
return headers, meta, tags, hadMapping, nil
}
// applyMappedMetadata applies mapped metadata and headers to the object state for uploads.
//
// It reads `--metadata`, `--metadata-set`, and `--metadata-mapper` outputs via fs.GetMetadataOptions
// and updates o.meta, o.tags and ui.httpHeaders accordingly.
func (o *Object) applyMappedMetadata(ctx context.Context, src fs.ObjectInfo, ui *uploadInfo, options []fs.OpenOption) (modTime time.Time, err error) {
// Start from the source modtime; may be overridden by metadata
modTime = src.ModTime(ctx)
// Fetch mapped metadata if --metadata is enabled
meta, err := fs.GetMetadataOptions(ctx, o.fs, src, options)
if err != nil {
return modTime, err
}
if meta == nil {
// No metadata processing requested
return modTime, nil
}
// Map metadata using common helper
headers, userMeta, tags, mappedModTime, err := mapMetadataToAzure(meta, func(format string, args ...any) { fs.Debugf(o, format, args...) })
if err != nil {
return modTime, err
}
// Merge headers into ui
if headers.BlobCacheControl != nil {
ui.httpHeaders.BlobCacheControl = headers.BlobCacheControl
}
if headers.BlobContentDisposition != nil {
ui.httpHeaders.BlobContentDisposition = headers.BlobContentDisposition
}
if headers.BlobContentEncoding != nil {
ui.httpHeaders.BlobContentEncoding = headers.BlobContentEncoding
}
if headers.BlobContentLanguage != nil {
ui.httpHeaders.BlobContentLanguage = headers.BlobContentLanguage
}
if headers.BlobContentType != nil {
ui.httpHeaders.BlobContentType = headers.BlobContentType
}
// Apply user metadata to o.meta with a single critical section
if len(userMeta) > 0 {
metadataMu.Lock()
if o.meta == nil {
o.meta = make(map[string]string, len(userMeta))
}
for k, v := range userMeta {
if v != nil {
o.meta[k] = *v
}
}
metadataMu.Unlock()
}
// Apply tags
if len(tags) > 0 {
if o.tags == nil {
o.tags = make(map[string]string, len(tags))
}
for k, v := range tags {
o.tags[k] = v
}
}
if mappedModTime != nil {
modTime = *mappedModTime
}
return modTime, nil
}
// Returns whether file is a directory marker or not // Returns whether file is a directory marker or not
func isDirectoryMarker(size int64, metadata map[string]*string, remote string) bool { func isDirectoryMarker(size int64, metadata map[string]*string, remote string) bool {
// Directory markers are 0 length // Directory markers are 0 length
@@ -2281,19 +1951,18 @@ func (f *Fs) copyMultipart(ctx context.Context, remote, dstContainer, dstPath st
return nil, err return nil, err
} }
// Prepare metadata/headers/tags for destination // Convert metadata from source object
// For multipart commit, include base metadata from source then overlay mapped
commitHeaders, commitMeta, commitTags, _, err := assembleCopyParams(ctx, f, src, srcProperties, true)
if err != nil {
return nil, fmt.Errorf("multipart copy: %w", err)
}
// Convert metadata from source or mapper
options := blockblob.CommitBlockListOptions{ options := blockblob.CommitBlockListOptions{
Metadata: commitMeta, Metadata: srcProperties.Metadata,
Tags: commitTags, Tier: parseTier(f.opt.AccessTier),
Tier: parseTier(f.opt.AccessTier), HTTPHeaders: &blob.HTTPHeaders{
HTTPHeaders: &commitHeaders, BlobCacheControl: srcProperties.CacheControl,
BlobContentDisposition: srcProperties.ContentDisposition,
BlobContentEncoding: srcProperties.ContentEncoding,
BlobContentLanguage: srcProperties.ContentLanguage,
BlobContentMD5: srcProperties.ContentMD5,
BlobContentType: srcProperties.ContentType,
},
} }
// Finalise the upload session // Finalise the upload session
@@ -2324,36 +1993,10 @@ func (f *Fs) copySinglepart(ctx context.Context, remote, dstContainer, dstPath s
return nil, fmt.Errorf("single part copy: source auth: %w", err) return nil, fmt.Errorf("single part copy: source auth: %w", err)
} }
// Prepare mapped metadata/tags/headers if requested // Start the copy
options := blob.StartCopyFromURLOptions{ options := blob.StartCopyFromURLOptions{
Tier: parseTier(f.opt.AccessTier), Tier: parseTier(f.opt.AccessTier),
} }
var postHeaders *blob.HTTPHeaders
// Read source properties and assemble params; this also handles the case when mapping is disabled
srcProps, err := src.readMetaDataAlways(ctx)
if err != nil {
return nil, fmt.Errorf("single part copy: read source properties: %w", err)
}
// For singlepart copy, do not include base metadata from source in StartCopyFromURL
headers, meta, tags, hadMapping, aerr := assembleCopyParams(ctx, f, src, srcProps, false)
if aerr != nil {
return nil, fmt.Errorf("single part copy: %w", aerr)
}
// Apply tags and post-copy headers only when mapping requested changes
if len(tags) > 0 {
options.BlobTags = make(map[string]string, len(tags))
for k, v := range tags {
options.BlobTags[k] = v
}
}
if hadMapping {
// Only set metadata explicitly when mapping was requested; otherwise
// let the service copy source metadata (including mtime) automatically.
if len(meta) > 0 {
options.Metadata = meta
}
postHeaders = &headers
}
var startCopy blob.StartCopyFromURLResponse var startCopy blob.StartCopyFromURLResponse
err = f.pacer.Call(func() (bool, error) { err = f.pacer.Call(func() (bool, error) {
startCopy, err = dstBlobSVC.StartCopyFromURL(ctx, srcURL, &options) startCopy, err = dstBlobSVC.StartCopyFromURL(ctx, srcURL, &options)
@@ -2383,16 +2026,6 @@ func (f *Fs) copySinglepart(ctx context.Context, remote, dstContainer, dstPath s
pollTime = min(2*pollTime, time.Second) pollTime = min(2*pollTime, time.Second)
} }
// If mapper requested header changes, set them post-copy
if postHeaders != nil {
blb := f.getBlobSVC(dstContainer, dstPath)
_, setErr := blb.SetHTTPHeaders(ctx, *postHeaders, nil)
if setErr != nil {
return nil, fmt.Errorf("single part copy: failed to set headers: %w", setErr)
}
}
// Metadata (when requested) is set via StartCopyFromURL options.Metadata
return f.NewObject(ctx, remote) return f.NewObject(ctx, remote)
} }
@@ -2524,35 +2157,6 @@ func (o *Object) getMetadata() (metadata map[string]*string) {
return metadata return metadata
} }
// Metadata returns metadata for an object
//
// It returns a combined view of system and user metadata.
func (o *Object) Metadata(ctx context.Context) (fs.Metadata, error) {
// Ensure metadata is loaded
if err := o.readMetaData(ctx); err != nil {
return nil, err
}
m := fs.Metadata{}
// System metadata we expose
if !o.modTime.IsZero() {
m["mtime"] = o.modTime.Format(time.RFC3339Nano)
}
if o.accessTier != "" {
m["tier"] = string(o.accessTier)
}
// Merge user metadata (already lower-cased keys)
metadataMu.Lock()
for k, v := range o.meta {
m[k] = v
}
metadataMu.Unlock()
return m, nil
}
// decodeMetaDataFromPropertiesResponse sets the metadata from the data passed in // decodeMetaDataFromPropertiesResponse sets the metadata from the data passed in
// //
// Sets // Sets
@@ -3391,18 +2995,16 @@ func (o *Object) prepareUpload(ctx context.Context, src fs.ObjectInfo, options [
// containerPath = containerPath[:len(containerPath)-1] // containerPath = containerPath[:len(containerPath)-1]
// } // }
// Start with default content-type based on source // Update Mod time
ui.httpHeaders = blob.HTTPHeaders{ o.updateMetadataWithModTime(src.ModTime(ctx))
BlobContentType: pString(fs.MimeType(ctx, src)),
}
// Apply mapped metadata/headers/tags if requested
modTime, err := o.applyMappedMetadata(ctx, src, &ui, options)
if err != nil { if err != nil {
return ui, err return ui, err
} }
// Ensure mtime is set in metadata based on possibly overridden modTime
o.updateMetadataWithModTime(modTime) // Create the HTTP headers for the upload
ui.httpHeaders = blob.HTTPHeaders{
BlobContentType: pString(fs.MimeType(ctx, src)),
}
// Compute the Content-MD5 of the file. As we stream all uploads it // Compute the Content-MD5 of the file. As we stream all uploads it
// will be set in PutBlockList API call using the 'x-ms-blob-content-md5' header // will be set in PutBlockList API call using the 'x-ms-blob-content-md5' header

View File

@@ -5,16 +5,11 @@ package azureblob
import ( import (
"context" "context"
"encoding/base64" "encoding/base64"
"fmt"
"net/http"
"strings" "strings"
"testing" "testing"
"time"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blob"
"github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob" "github.com/Azure/azure-sdk-for-go/sdk/storage/azblob/blockblob"
"github.com/rclone/rclone/fs" "github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/object"
"github.com/rclone/rclone/fstest" "github.com/rclone/rclone/fstest"
"github.com/rclone/rclone/fstest/fstests" "github.com/rclone/rclone/fstest/fstests"
"github.com/rclone/rclone/lib/random" "github.com/rclone/rclone/lib/random"
@@ -153,417 +148,4 @@ func (f *Fs) testWriteUncommittedBlocks(t *testing.T) {
func (f *Fs) InternalTest(t *testing.T) { func (f *Fs) InternalTest(t *testing.T) {
t.Run("Features", f.testFeatures) t.Run("Features", f.testFeatures)
t.Run("WriteUncommittedBlocks", f.testWriteUncommittedBlocks) t.Run("WriteUncommittedBlocks", f.testWriteUncommittedBlocks)
t.Run("Metadata", f.testMetadataPaths)
}
// helper to read blob properties for an object
func getProps(ctx context.Context, t *testing.T, o fs.Object) *blob.GetPropertiesResponse {
ao := o.(*Object)
props, err := ao.readMetaDataAlways(ctx)
require.NoError(t, err)
return props
}
// helper to assert select headers and user metadata
func assertHeadersAndMetadata(t *testing.T, props *blob.GetPropertiesResponse, want map[string]string, wantUserMeta map[string]string) {
// Headers
get := func(p *string) string {
if p == nil {
return ""
}
return *p
}
if v, ok := want["content-type"]; ok {
assert.Equal(t, v, get(props.ContentType), "content-type")
}
if v, ok := want["cache-control"]; ok {
assert.Equal(t, v, get(props.CacheControl), "cache-control")
}
if v, ok := want["content-disposition"]; ok {
assert.Equal(t, v, get(props.ContentDisposition), "content-disposition")
}
if v, ok := want["content-encoding"]; ok {
assert.Equal(t, v, get(props.ContentEncoding), "content-encoding")
}
if v, ok := want["content-language"]; ok {
assert.Equal(t, v, get(props.ContentLanguage), "content-language")
}
// User metadata (case-insensitive keys from service)
norm := make(map[string]*string, len(props.Metadata))
for kk, vv := range props.Metadata {
norm[strings.ToLower(kk)] = vv
}
for k, v := range wantUserMeta {
pv, ok := norm[strings.ToLower(k)]
if assert.True(t, ok, fmt.Sprintf("missing user metadata key %q", k)) {
if pv == nil {
assert.Equal(t, v, "", k)
} else {
assert.Equal(t, v, *pv, k)
}
} else {
// Log available keys for diagnostics
keys := make([]string, 0, len(props.Metadata))
for kk := range props.Metadata {
keys = append(keys, kk)
}
t.Logf("available user metadata keys: %v", keys)
}
}
}
// helper to read blob tags for an object
func getTagsMap(ctx context.Context, t *testing.T, o fs.Object) map[string]string {
ao := o.(*Object)
blb := ao.getBlobSVC()
resp, err := blb.GetTags(ctx, nil)
require.NoError(t, err)
out := make(map[string]string)
for _, tag := range resp.BlobTagSet {
if tag.Key != nil {
k := *tag.Key
v := ""
if tag.Value != nil {
v = *tag.Value
}
out[k] = v
}
}
return out
}
// Test metadata across different write paths
func (f *Fs) testMetadataPaths(t *testing.T) {
ctx := context.Background()
if testing.Short() {
t.Skip("skipping in short mode")
}
// Common expected metadata and headers
baseMeta := fs.Metadata{
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
// Note: Don't set content-encoding here to avoid download decoding differences
// We will set a custom user metadata key
"potato": "royal",
// and modtime
"mtime": fstest.Time("2009-05-06T04:05:06.499999999Z").Format(time.RFC3339Nano),
}
// Singlepart upload
t.Run("PutSinglepart", func(t *testing.T) {
// size less than chunk size
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("meta-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
// override content-type via metadata mapping
meta := fs.Metadata{}
meta.Merge(baseMeta)
meta["content-type"] = "text/plain"
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", meta)
defer func() { _ = obj.Remove(ctx) }()
props := getProps(ctx, t, obj)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
}, map[string]string{
"potato": "royal",
})
_ = http.StatusOK // keep import for parity but don't inspect RawResponse
})
// Multipart upload
t.Run("PutMultipart", func(t *testing.T) {
// size greater than chunk size to force multipart
contents := random.String(int(f.opt.ChunkSize + 1024))
item := fstest.NewItem("meta-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{}
meta.Merge(baseMeta)
meta["content-type"] = "application/json"
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/html", meta)
defer func() { _ = obj.Remove(ctx) }()
props := getProps(ctx, t, obj)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/json",
"cache-control": "no-cache",
"content-disposition": "inline",
"content-language": "en-US",
}, map[string]string{
"potato": "royal",
})
// Tags: Singlepart upload
t.Run("PutSinglepartTags", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize / 2))
item := fstest.NewItem("tags-single.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
"x-ms-tags": "env=dev,team=sync",
}
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "text/plain", meta)
defer func() { _ = obj.Remove(ctx) }()
tags := getTagsMap(ctx, t, obj)
assert.Equal(t, "dev", tags["env"])
assert.Equal(t, "sync", tags["team"])
})
// Tags: Multipart upload
t.Run("PutMultipartTags", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize + 2048))
item := fstest.NewItem("tags-multipart.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
meta := fs.Metadata{
"x-ms-tags": "project=alpha,release=2025-08",
}
obj := fstests.PutTestContentsMetadata(ctx, t, f, &item, true, contents, true, "application/octet-stream", meta)
defer func() { _ = obj.Remove(ctx) }()
tags := getTagsMap(ctx, t, obj)
assert.Equal(t, "alpha", tags["project"])
assert.Equal(t, "2025-08", tags["release"])
})
})
// Singlepart copy with metadata-set mapping; omit content-type to exercise fallback
t.Run("CopySinglepart", func(t *testing.T) {
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("meta-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"cache-control": "private, max-age=60",
"content-disposition": "attachment; filename=foo.txt",
"content-language": "fr",
// no content-type: should fallback to source
"potato": "maris",
}
// do copy
dstName := "meta-copy-single-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
// content-type should fallback to source (text/plain)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
"cache-control": "private, max-age=60",
"content-disposition": "attachment; filename=foo.txt",
"content-language": "fr",
}, map[string]string{
"potato": "maris",
})
// mtime should be populated on copy when --metadata is used
// and should equal the source ModTime (RFC3339Nano)
// Read user metadata (case-insensitive)
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
// parse and compare times ignoring formatting differences
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// CopySinglepart with only --metadata (no MetadataSet) must inject mtime and preserve src content-type
t.Run("CopySinglepart_MetadataOnly", func(t *testing.T) {
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("meta-copy-single-only-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
dstName := "meta-copy-single-only-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "text/plain",
}, map[string]string{})
// Assert mtime injected
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// Multipart copy with metadata-set mapping; omit content-type to exercise fallback
t.Run("CopyMultipart", func(t *testing.T) {
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 1024))
srcItem := fstest.NewItem("meta-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"cache-control": "max-age=0, no-cache",
// omit content-type to trigger fallback
"content-language": "de",
"potato": "desiree",
}
dstName := "meta-copy-multi-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
// content-type should fallback to source (application/octet-stream)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/octet-stream",
"cache-control": "max-age=0, no-cache",
"content-language": "de",
}, map[string]string{
"potato": "desiree",
})
// mtime should be populated on copy when --metadata is used
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// CopyMultipart with only --metadata must inject mtime and preserve src content-type
t.Run("CopyMultipart_MetadataOnly", func(t *testing.T) {
contents := random.String(int(f.opt.CopyCutoff + 2048))
srcItem := fstest.NewItem("meta-copy-multi-only-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
dstName := "meta-copy-multi-only-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
props := getProps(ctx2, t, dst)
assertHeadersAndMetadata(t, props, map[string]string{
"content-type": "application/octet-stream",
}, map[string]string{})
m := props.Metadata
var gotMtime string
for k, v := range m {
if strings.EqualFold(k, "mtime") && v != nil {
gotMtime = *v
break
}
}
if assert.NotEmpty(t, gotMtime, "mtime not set on destination metadata") {
parsed, err := time.Parse(time.RFC3339Nano, gotMtime)
require.NoError(t, err)
assert.True(t, srcObj.ModTime(ctx2).Equal(parsed), "dst mtime should equal src ModTime")
}
})
// Tags: Singlepart copy
t.Run("CopySinglepartTags", func(t *testing.T) {
// create small source
contents := random.String(int(f.opt.ChunkSize / 2))
srcItem := fstest.NewItem("tags-copy-single-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "text/plain", nil)
defer func() { _ = srcObj.Remove(ctx) }()
// set mapping via MetadataSet including tags
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"x-ms-tags": "copy=single,mode=test",
}
dstName := "tags-copy-single-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
tags := getTagsMap(ctx2, t, dst)
assert.Equal(t, "single", tags["copy"])
assert.Equal(t, "test", tags["mode"])
})
// Tags: Multipart copy
t.Run("CopyMultipartTags", func(t *testing.T) {
// create large source to force multipart
contents := random.String(int(f.opt.CopyCutoff + 4096))
srcItem := fstest.NewItem("tags-copy-multi-src.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
srcObj := fstests.PutTestContentsMetadata(ctx, t, f, &srcItem, true, contents, true, "application/octet-stream", nil)
defer func() { _ = srcObj.Remove(ctx) }()
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
ci.MetadataSet = fs.Metadata{
"x-ms-tags": "copy=multi,mode=test",
}
dstName := "tags-copy-multi-dst.txt"
dst, err := f.Copy(ctx2, srcObj, dstName)
require.NoError(t, err)
defer func() { _ = dst.Remove(ctx2) }()
tags := getTagsMap(ctx2, t, dst)
assert.Equal(t, "multi", tags["copy"])
assert.Equal(t, "test", tags["mode"])
})
// Negative: invalid x-ms-tags must error
t.Run("InvalidXMsTags", func(t *testing.T) {
contents := random.String(32)
item := fstest.NewItem("tags-invalid.txt", contents, fstest.Time("2001-05-06T04:05:06.499999999Z"))
// construct ObjectInfo with invalid x-ms-tags
buf := strings.NewReader(contents)
// Build obj info with metadata
meta := fs.Metadata{
"x-ms-tags": "badpair-without-equals",
}
// force metadata on
ctx2, ci := fs.AddConfig(ctx)
ci.Metadata = true
obji := object.NewStaticObjectInfo(item.Path, item.ModTime, int64(len(contents)), true, nil, nil)
obji = obji.WithMetadata(meta).WithMimeType("text/plain")
_, err := f.Put(ctx2, buf, obji)
require.Error(t, err)
assert.Contains(t, err.Error(), "invalid tag")
})
} }

View File

@@ -133,32 +133,23 @@ type File struct {
Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file. Info map[string]string `json:"fileInfo"` // The custom information that was uploaded with the file. This is a JSON object, holding the name/value pairs that were uploaded with the file.
} }
// StorageAPI is as returned from the b2_authorize_account call // AuthorizeAccountResponse is as returned from the b2_authorize_account call
type StorageAPI struct { type AuthorizeAccountResponse struct {
AbsoluteMinimumPartSize int `json:"absoluteMinimumPartSize"` // The smallest possible size of a part of a large file. AbsoluteMinimumPartSize int `json:"absoluteMinimumPartSize"` // The smallest possible size of a part of a large file.
AccountID string `json:"accountId"` // The identifier for the account.
Allowed struct { // An object (see below) containing the capabilities of this auth token, and any restrictions on using it. Allowed struct { // An object (see below) containing the capabilities of this auth token, and any restrictions on using it.
Buckets []struct { // When present, access is restricted to one or more buckets. BucketID string `json:"bucketId"` // When present, access is restricted to one bucket.
ID string `json:"id"` // ID of bucket BucketName string `json:"bucketName"` // When present, name of bucket - may be empty
Name string `json:"name"` // When present, name of bucket - may be empty Capabilities []string `json:"capabilities"` // A list of strings, each one naming a capability the key has.
} `json:"buckets"`
Capabilities []string `json:"capabilities"` // A list of strings, each one naming a capability the key has for every bucket.
NamePrefix any `json:"namePrefix"` // When present, access is restricted to files whose names start with the prefix NamePrefix any `json:"namePrefix"` // When present, access is restricted to files whose names start with the prefix
} `json:"allowed"` } `json:"allowed"`
APIURL string `json:"apiUrl"` // The base URL to use for all API calls except for uploading and downloading files. APIURL string `json:"apiUrl"` // The base URL to use for all API calls except for uploading and downloading files.
AuthorizationToken string `json:"authorizationToken"` // An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
DownloadURL string `json:"downloadUrl"` // The base URL to use for downloading files. DownloadURL string `json:"downloadUrl"` // The base URL to use for downloading files.
MinimumPartSize int `json:"minimumPartSize"` // DEPRECATED: This field will always have the same value as recommendedPartSize. Use recommendedPartSize instead. MinimumPartSize int `json:"minimumPartSize"` // DEPRECATED: This field will always have the same value as recommendedPartSize. Use recommendedPartSize instead.
RecommendedPartSize int `json:"recommendedPartSize"` // The recommended size for each part of a large file. We recommend using this part size for optimal upload performance. RecommendedPartSize int `json:"recommendedPartSize"` // The recommended size for each part of a large file. We recommend using this part size for optimal upload performance.
} }
// AuthorizeAccountResponse is as returned from the b2_authorize_account call
type AuthorizeAccountResponse struct {
AccountID string `json:"accountId"` // The identifier for the account.
AuthorizationToken string `json:"authorizationToken"` // An authorization token to use with all calls, other than b2_authorize_account, that need an Authorization header.
APIs struct { // Supported APIs for this account / key. These are API-dependent JSON objects.
Storage StorageAPI `json:"storageApi"`
} `json:"apiInfo"`
}
// ListBucketsRequest is parameters for b2_list_buckets call // ListBucketsRequest is parameters for b2_list_buckets call
type ListBucketsRequest struct { type ListBucketsRequest struct {
AccountID string `json:"accountId"` // The identifier for the account. AccountID string `json:"accountId"` // The identifier for the account.

View File

@@ -607,29 +607,17 @@ func NewFs(ctx context.Context, name, root string, m configmap.Mapper) (fs.Fs, e
if err != nil { if err != nil {
return nil, fmt.Errorf("failed to authorize account: %w", err) return nil, fmt.Errorf("failed to authorize account: %w", err)
} }
// If this is a key limited to one or more buckets, one of them must exist // If this is a key limited to a single bucket, it must exist already
// and be ours. if f.rootBucket != "" && f.info.Allowed.BucketID != "" {
if f.rootBucket != "" && len(f.info.APIs.Storage.Allowed.Buckets) != 0 { allowedBucket := f.opt.Enc.ToStandardName(f.info.Allowed.BucketName)
buckets := f.info.APIs.Storage.Allowed.Buckets if allowedBucket == "" {
var rootFound = false return nil, errors.New("bucket that application key is restricted to no longer exists")
var rootID string
for _, b := range buckets {
allowedBucket := f.opt.Enc.ToStandardName(b.Name)
if allowedBucket == "" {
fs.Debugf(f, "bucket %q that application key is restricted to no longer exists", b.ID)
continue
}
if allowedBucket == f.rootBucket {
rootFound = true
rootID = b.ID
}
} }
if !rootFound { if allowedBucket != f.rootBucket {
return nil, fmt.Errorf("you must use bucket(s) %q with this application key", buckets) return nil, fmt.Errorf("you must use bucket %q with this application key", allowedBucket)
} }
f.cache.MarkOK(f.rootBucket) f.cache.MarkOK(f.rootBucket)
f.setBucketID(f.rootBucket, rootID) f.setBucketID(f.rootBucket, f.info.Allowed.BucketID)
} }
if f.rootBucket != "" && f.rootDirectory != "" { if f.rootBucket != "" && f.rootDirectory != "" {
// Check to see if the (bucket,directory) is actually an existing file // Check to see if the (bucket,directory) is actually an existing file
@@ -655,7 +643,7 @@ func (f *Fs) authorizeAccount(ctx context.Context) error {
defer f.authMu.Unlock() defer f.authMu.Unlock()
opts := rest.Opts{ opts := rest.Opts{
Method: "GET", Method: "GET",
Path: "/b2api/v4/b2_authorize_account", Path: "/b2api/v1/b2_authorize_account",
RootURL: f.opt.Endpoint, RootURL: f.opt.Endpoint,
UserName: f.opt.Account, UserName: f.opt.Account,
Password: f.opt.Key, Password: f.opt.Key,
@@ -668,13 +656,13 @@ func (f *Fs) authorizeAccount(ctx context.Context) error {
if err != nil { if err != nil {
return fmt.Errorf("failed to authenticate: %w", err) return fmt.Errorf("failed to authenticate: %w", err)
} }
f.srv.SetRoot(f.info.APIs.Storage.APIURL+"/b2api/v1").SetHeader("Authorization", f.info.AuthorizationToken) f.srv.SetRoot(f.info.APIURL+"/b2api/v1").SetHeader("Authorization", f.info.AuthorizationToken)
return nil return nil
} }
// hasPermission returns if the current AuthorizationToken has the selected permission // hasPermission returns if the current AuthorizationToken has the selected permission
func (f *Fs) hasPermission(permission string) bool { func (f *Fs) hasPermission(permission string) bool {
return slices.Contains(f.info.APIs.Storage.Allowed.Capabilities, permission) return slices.Contains(f.info.Allowed.Capabilities, permission)
} }
// getUploadURL returns the upload info with the UploadURL and the AuthorizationToken // getUploadURL returns the upload info with the UploadURL and the AuthorizationToken
@@ -1079,83 +1067,44 @@ type listBucketFn func(*api.Bucket) error
// listBucketsToFn lists the buckets to the function supplied // listBucketsToFn lists the buckets to the function supplied
func (f *Fs) listBucketsToFn(ctx context.Context, bucketName string, fn listBucketFn) error { func (f *Fs) listBucketsToFn(ctx context.Context, bucketName string, fn listBucketFn) error {
responses := make([]api.ListBucketsResponse, len(f.info.APIs.Storage.Allowed.Buckets))[:0] var account = api.ListBucketsRequest{
AccountID: f.info.AccountID,
call := func(id string) error { BucketID: f.info.Allowed.BucketID,
var account = api.ListBucketsRequest{ }
AccountID: f.info.AccountID, if bucketName != "" && account.BucketID == "" {
BucketID: id, account.BucketName = f.opt.Enc.FromStandardName(bucketName)
}
if bucketName != "" && account.BucketID == "" {
account.BucketName = f.opt.Enc.FromStandardName(bucketName)
}
var response api.ListBucketsResponse
opts := rest.Opts{
Method: "POST",
Path: "/b2_list_buckets",
}
err := f.pacer.Call(func() (bool, error) {
resp, err := f.srv.CallJSON(ctx, &opts, &account, &response)
return f.shouldRetry(ctx, resp, err)
})
if err != nil {
return err
}
responses = append(responses, response)
return nil
} }
for i := range f.info.APIs.Storage.Allowed.Buckets { var response api.ListBucketsResponse
b := &f.info.APIs.Storage.Allowed.Buckets[i] opts := rest.Opts{
// Empty names indicate a bucket that no longer exists, this is non-fatal Method: "POST",
// for multi-bucket API keys. Path: "/b2_list_buckets",
if b.Name == "" {
continue
}
// When requesting a specific bucket skip over non-matching names
if bucketName != "" && b.Name != bucketName {
continue
}
err := call(b.ID)
if err != nil {
return err
}
} }
err := f.pacer.Call(func() (bool, error) {
if len(f.info.APIs.Storage.Allowed.Buckets) == 0 { resp, err := f.srv.CallJSON(ctx, &opts, &account, &response)
err := call("") return f.shouldRetry(ctx, resp, err)
if err != nil { })
return err if err != nil {
} return err
} }
f.bucketIDMutex.Lock() f.bucketIDMutex.Lock()
f.bucketTypeMutex.Lock() f.bucketTypeMutex.Lock()
f._bucketID = make(map[string]string, 1) f._bucketID = make(map[string]string, 1)
f._bucketType = make(map[string]string, 1) f._bucketType = make(map[string]string, 1)
for i := range response.Buckets {
for ri := range responses { bucket := &response.Buckets[i]
response := &responses[ri] bucket.Name = f.opt.Enc.ToStandardName(bucket.Name)
for i := range response.Buckets { f.cache.MarkOK(bucket.Name)
bucket := &response.Buckets[i] f._bucketID[bucket.Name] = bucket.ID
bucket.Name = f.opt.Enc.ToStandardName(bucket.Name) f._bucketType[bucket.Name] = bucket.Type
f.cache.MarkOK(bucket.Name)
f._bucketID[bucket.Name] = bucket.ID
f._bucketType[bucket.Name] = bucket.Type
}
} }
f.bucketTypeMutex.Unlock() f.bucketTypeMutex.Unlock()
f.bucketIDMutex.Unlock() f.bucketIDMutex.Unlock()
for ri := range responses { for i := range response.Buckets {
response := &responses[ri] bucket := &response.Buckets[i]
for i := range response.Buckets { err = fn(bucket)
bucket := &response.Buckets[i] if err != nil {
err := fn(bucket) return err
if err != nil {
return err
}
} }
} }
return nil return nil
@@ -1657,7 +1606,7 @@ func (f *Fs) PublicLink(ctx context.Context, remote string, expire fs.Duration,
bucket, bucketPath := f.split(remote) bucket, bucketPath := f.split(remote)
var RootURL string var RootURL string
if f.opt.DownloadURL == "" { if f.opt.DownloadURL == "" {
RootURL = f.info.APIs.Storage.DownloadURL RootURL = f.info.DownloadURL
} else { } else {
RootURL = f.opt.DownloadURL RootURL = f.opt.DownloadURL
} }
@@ -2008,7 +1957,7 @@ func (o *Object) getOrHead(ctx context.Context, method string, options []fs.Open
// Use downloadUrl from backblaze if downloadUrl is not set // Use downloadUrl from backblaze if downloadUrl is not set
// otherwise use the custom downloadUrl // otherwise use the custom downloadUrl
if o.fs.opt.DownloadURL == "" { if o.fs.opt.DownloadURL == "" {
opts.RootURL = o.fs.info.APIs.Storage.DownloadURL opts.RootURL = o.fs.info.DownloadURL
} else { } else {
opts.RootURL = o.fs.opt.DownloadURL opts.RootURL = o.fs.opt.DownloadURL
} }

View File

@@ -403,14 +403,14 @@ func (c *Cipher) deobfuscateSegment(ciphertext string) (string, error) {
if ciphertext == "" { if ciphertext == "" {
return "", nil return "", nil
} }
before, after, ok := strings.Cut(ciphertext, ".") pos := strings.Index(ciphertext, ".")
if !ok { if pos == -1 {
return "", ErrorNotAnEncryptedFile return "", ErrorNotAnEncryptedFile
} // No . } // No .
num := before num := ciphertext[:pos]
if num == "!" { if num == "!" {
// No rotation; probably original was not valid unicode // No rotation; probably original was not valid unicode
return after, nil return ciphertext[pos+1:], nil
} }
dir, err := strconv.Atoi(num) dir, err := strconv.Atoi(num)
if err != nil { if err != nil {
@@ -425,7 +425,7 @@ func (c *Cipher) deobfuscateSegment(ciphertext string) (string, error) {
var result bytes.Buffer var result bytes.Buffer
inQuote := false inQuote := false
for _, runeValue := range after { for _, runeValue := range ciphertext[pos+1:] {
switch { switch {
case inQuote: case inQuote:
_, _ = result.WriteRune(runeValue) _, _ = result.WriteRune(runeValue)

View File

@@ -6,7 +6,6 @@ import (
"context" "context"
"crypto/md5" "crypto/md5"
"encoding/hex" "encoding/hex"
"errors"
"fmt" "fmt"
"io" "io"
"path" "path"
@@ -25,8 +24,7 @@ import (
var ( var (
hashType = hash.MD5 hashType = hash.MD5
// the object storage is persistent // the object storage is persistent
buckets = newBucketsInfo() buckets = newBucketsInfo()
errWriteOnly = errors.New("can't read when using --memory-discard")
) )
// Register with Fs // Register with Fs
@@ -35,32 +33,12 @@ func init() {
Name: "memory", Name: "memory",
Description: "In memory object storage system.", Description: "In memory object storage system.",
NewFs: NewFs, NewFs: NewFs,
Options: []fs.Option{{ Options: []fs.Option{},
Name: "discard",
Default: false,
Advanced: true,
Help: `If set all writes will be discarded and reads will return an error
If set then when files are uploaded the contents not be saved. The
files will appear to have been uploaded but will give an error on
read. Files will have their MD5 sum calculated on upload which takes
very little CPU time and allows the transfers to be checked.
This can be useful for testing performance.
Probably most easily used by using the connection string syntax:
:memory,discard:bucket
`,
}},
}) })
} }
// Options defines the configuration for this backend // Options defines the configuration for this backend
type Options struct { type Options struct{}
Discard bool `config:"discard"`
}
// Fs represents a remote memory server // Fs represents a remote memory server
type Fs struct { type Fs struct {
@@ -186,7 +164,6 @@ type objectData struct {
hash string hash string
mimeType string mimeType string
data []byte data []byte
size int64
} }
// Object describes a memory object // Object describes a memory object
@@ -581,7 +558,7 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
if t != hashType { if t != hashType {
return "", hash.ErrUnsupported return "", hash.ErrUnsupported
} }
if o.od.hash == "" && !o.fs.opt.Discard { if o.od.hash == "" {
sum := md5.Sum(o.od.data) sum := md5.Sum(o.od.data)
o.od.hash = hex.EncodeToString(sum[:]) o.od.hash = hex.EncodeToString(sum[:])
} }
@@ -590,7 +567,7 @@ func (o *Object) Hash(ctx context.Context, t hash.Type) (string, error) {
// Size returns the size of an object in bytes // Size returns the size of an object in bytes
func (o *Object) Size() int64 { func (o *Object) Size() int64 {
return o.od.size return int64(len(o.od.data))
} }
// ModTime returns the modification time of the object // ModTime returns the modification time of the object
@@ -616,9 +593,6 @@ func (o *Object) Storable() bool {
// Open an object for read // Open an object for read
func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) { func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.ReadCloser, err error) {
if o.fs.opt.Discard {
return nil, errWriteOnly
}
var offset, limit int64 = 0, -1 var offset, limit int64 = 0, -1
for _, option := range options { for _, option := range options {
switch x := option.(type) { switch x := option.(type) {
@@ -650,24 +624,13 @@ func (o *Object) Open(ctx context.Context, options ...fs.OpenOption) (in io.Read
// The new object may have been created if an error is returned // The new object may have been created if an error is returned
func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (err error) { func (o *Object) Update(ctx context.Context, in io.Reader, src fs.ObjectInfo, options ...fs.OpenOption) (err error) {
bucket, bucketPath := o.split() bucket, bucketPath := o.split()
var data []byte data, err := io.ReadAll(in)
var size int64
var hash string
if o.fs.opt.Discard {
h := md5.New()
size, err = io.Copy(h, in)
hash = hex.EncodeToString(h.Sum(nil))
} else {
data, err = io.ReadAll(in)
size = int64(len(data))
}
if err != nil { if err != nil {
return fmt.Errorf("failed to update memory object: %w", err) return fmt.Errorf("failed to update memory object: %w", err)
} }
o.od = &objectData{ o.od = &objectData{
data: data, data: data,
size: size, hash: "",
hash: hash,
modTime: src.ModTime(ctx), modTime: src.ModTime(ctx),
mimeType: fs.MimeType(ctx, src), mimeType: fs.MimeType(ctx, src),
} }

View File

@@ -222,11 +222,3 @@ type UserInfo struct {
} `json:"steps"` } `json:"steps"`
} `json:"journey"` } `json:"journey"`
} }
// DiffResult is the response from /diff
type DiffResult struct {
Result int `json:"result"`
DiffID int64 `json:"diffid"`
Entries []map[string]any `json:"entries"`
Error string `json:"error"`
}

View File

@@ -171,7 +171,6 @@ type Fs struct {
dirCache *dircache.DirCache // Map of directory path to directory id dirCache *dircache.DirCache // Map of directory path to directory id
pacer *fs.Pacer // pacer for API calls pacer *fs.Pacer // pacer for API calls
tokenRenewer *oauthutil.Renew // renew the token on expiry tokenRenewer *oauthutil.Renew // renew the token on expiry
lastDiffID int64 // change tracking state for diff long-polling
} }
// Object describes a pcloud object // Object describes a pcloud object
@@ -1034,137 +1033,6 @@ func (f *Fs) Shutdown(ctx context.Context) error {
return nil return nil
} }
// ChangeNotify implements fs.Features.ChangeNotify
func (f *Fs) ChangeNotify(ctx context.Context, notify func(string, fs.EntryType), ch <-chan time.Duration) {
// Start long-poll loop in background
go f.changeNotifyLoop(ctx, notify, ch)
}
// changeNotifyLoop contains the blocking long-poll logic.
func (f *Fs) changeNotifyLoop(ctx context.Context, notify func(string, fs.EntryType), ch <-chan time.Duration) {
// Standard polling interval
interval := 30 * time.Second
// Start with diffID = 0 to get the current state
var diffID int64
// Helper to process changes from the diff API
handleChanges := func(entries []map[string]any) {
notifiedPaths := make(map[string]bool)
for _, entry := range entries {
meta, ok := entry["metadata"].(map[string]any)
if !ok {
continue
}
// Robust extraction of ParentFolderID
var pid int64
if val, ok := meta["parentfolderid"]; ok {
switch v := val.(type) {
case float64:
pid = int64(v)
case int64:
pid = v
case int:
pid = int64(v)
}
}
// Resolve the path using dirCache.GetInv
// pCloud uses "d" prefix for directory IDs in cache, but API returns numbers
dirID := fmt.Sprintf("d%d", pid)
parentPath, ok := f.dirCache.GetInv(dirID)
if !ok {
// Parent not in cache, so we can ignore this change as it is outside
// of what the mount has seen or cares about.
continue
}
name, _ := meta["name"].(string)
fullPath := path.Join(parentPath, name)
// Determine EntryType (File or Directory)
entryType := fs.EntryObject
if isFolder, ok := meta["isfolder"].(bool); ok && isFolder {
entryType = fs.EntryDirectory
}
// Deduplicate notifications for this batch
if !notifiedPaths[fullPath] {
fs.Debugf(f, "ChangeNotify: detected change in %q (type: %v)", fullPath, entryType)
notify(fullPath, entryType)
notifiedPaths[fullPath] = true
}
}
}
for {
// Check context and channel
select {
case <-ctx.Done():
return
case newInterval, ok := <-ch:
if !ok {
return
}
interval = newInterval
default:
}
// Setup /diff Request
opts := rest.Opts{
Method: "GET",
Path: "/diff",
Parameters: url.Values{},
}
if diffID != 0 {
opts.Parameters.Set("diffid", strconv.FormatInt(diffID, 10))
opts.Parameters.Set("block", "1")
} else {
opts.Parameters.Set("last", "0")
}
// Perform Long-Poll
// Timeout set to 90s (server usually blocks for 60s max)
reqCtx, cancel := context.WithTimeout(ctx, 90*time.Second)
var result api.DiffResult
_, err := f.srv.CallJSON(reqCtx, &opts, nil, &result)
cancel()
if err != nil {
if errors.Is(err, context.Canceled) {
return
}
// Ignore timeout errors as they are normal for long-polling
if !errors.Is(err, context.DeadlineExceeded) {
fs.Infof(f, "ChangeNotify: polling error: %v. Waiting %v.", err, interval)
time.Sleep(interval)
}
continue
}
// If result is not 0, reset DiffID to resync
if result.Result != 0 {
diffID = 0
time.Sleep(2 * time.Second)
continue
}
if result.DiffID != 0 {
diffID = result.DiffID
f.lastDiffID = diffID
}
if len(result.Entries) > 0 {
handleChanges(result.Entries)
}
}
}
// Hashes returns the supported hash sets. // Hashes returns the supported hash sets.
func (f *Fs) Hashes() hash.Set { func (f *Fs) Hashes() hash.Set {
// EU region supports SHA1 and SHA256 (but rclone doesn't // EU region supports SHA1 and SHA256 (but rclone doesn't
@@ -1533,7 +1401,6 @@ var (
_ fs.ListPer = (*Fs)(nil) _ fs.ListPer = (*Fs)(nil)
_ fs.Abouter = (*Fs)(nil) _ fs.Abouter = (*Fs)(nil)
_ fs.Shutdowner = (*Fs)(nil) _ fs.Shutdowner = (*Fs)(nil)
_ fs.ChangeNotifier = (*Fs)(nil)
_ fs.Object = (*Object)(nil) _ fs.Object = (*Object)(nil)
_ fs.IDer = (*Object)(nil) _ fs.IDer = (*Object)(nil)
) )

View File

@@ -1,26 +1,26 @@
name: Linode name: Linode
description: Linode Object Storage description: Linode Object Storage
endpoint: endpoint:
nl-ams-1.linodeobjects.com: Amsterdam, NL (nl-ams-1) nl-ams-1.linodeobjects.com: Amsterdam (Netherlands), nl-ams-1
us-southeast-1.linodeobjects.com: Atlanta, GA, US (us-southeast-1) us-southeast-1.linodeobjects.com: Atlanta, GA (USA), us-southeast-1
in-maa-1.linodeobjects.com: Chennai, IN (in-maa-1) in-maa-1.linodeobjects.com: Chennai (India), in-maa-1
us-ord-1.linodeobjects.com: Chicago, IL, US (us-ord-1) us-ord-1.linodeobjects.com: Chicago, IL (USA), us-ord-1
eu-central-1.linodeobjects.com: Frankfurt, DE (eu-central-1) eu-central-1.linodeobjects.com: Frankfurt (Germany), eu-central-1
id-cgk-1.linodeobjects.com: Jakarta, ID (id-cgk-1) id-cgk-1.linodeobjects.com: Jakarta (Indonesia), id-cgk-1
gb-lon-1.linodeobjects.com: London 2, UK (gb-lon-1) gb-lon-1.linodeobjects.com: London 2 (Great Britain), gb-lon-1
us-lax-1.linodeobjects.com: Los Angeles, CA, US (us-lax-1) us-lax-1.linodeobjects.com: Los Angeles, CA (USA), us-lax-1
es-mad-1.linodeobjects.com: Madrid, ES (es-mad-1) es-mad-1.linodeobjects.com: Madrid (Spain), es-mad-1
us-mia-1.linodeobjects.com: Miami, FL, US (us-mia-1) au-mel-1.linodeobjects.com: Melbourne (Australia), au-mel-1
it-mil-1.linodeobjects.com: Milan, IT (it-mil-1) us-mia-1.linodeobjects.com: Miami, FL (USA), us-mia-1
us-east-1.linodeobjects.com: Newark, NJ, US (us-east-1) it-mil-1.linodeobjects.com: Milan (Italy), it-mil-1
jp-osa-1.linodeobjects.com: Osaka, JP (jp-osa-1) us-east-1.linodeobjects.com: Newark, NJ (USA), us-east-1
fr-par-1.linodeobjects.com: Paris, FR (fr-par-1) jp-osa-1.linodeobjects.com: Osaka (Japan), jp-osa-1
br-gru-1.linodeobjects.com: Sao Paulo, BR (br-gru-1) fr-par-1.linodeobjects.com: Paris (France), fr-par-1
us-sea-1.linodeobjects.com: Seattle, WA, US (us-sea-1) br-gru-1.linodeobjects.com: São Paulo (Brazil), br-gru-1
ap-south-1.linodeobjects.com: Singapore, SG (ap-south-1) us-sea-1.linodeobjects.com: Seattle, WA (USA), us-sea-1
sg-sin-1.linodeobjects.com: Singapore 2, SG (sg-sin-1) ap-south-1.linodeobjects.com: Singapore, ap-south-1
se-sto-1.linodeobjects.com: Stockholm, SE (se-sto-1) sg-sin-1.linodeobjects.com: Singapore 2, sg-sin-1
jp-tyo-1.linodeobjects.com: Tokyo 3, JP (jp-tyo-1) se-sto-1.linodeobjects.com: Stockholm (Sweden), se-sto-1
us-iad-10.linodeobjects.com: Washington, DC, US (us-iad-10) us-iad-1.linodeobjects.com: Washington, DC, (USA), us-iad-1
acl: {} acl: {}
bucket_acl: true bucket_acl: true

View File

@@ -30,11 +30,9 @@ import (
v4signer "github.com/aws/aws-sdk-go-v2/aws/signer/v4" v4signer "github.com/aws/aws-sdk-go-v2/aws/signer/v4"
awsconfig "github.com/aws/aws-sdk-go-v2/config" awsconfig "github.com/aws/aws-sdk-go-v2/config"
"github.com/aws/aws-sdk-go-v2/credentials" "github.com/aws/aws-sdk-go-v2/credentials"
"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
"github.com/aws/aws-sdk-go-v2/feature/s3/manager" "github.com/aws/aws-sdk-go-v2/feature/s3/manager"
"github.com/aws/aws-sdk-go-v2/service/s3" "github.com/aws/aws-sdk-go-v2/service/s3"
"github.com/aws/aws-sdk-go-v2/service/s3/types" "github.com/aws/aws-sdk-go-v2/service/s3/types"
"github.com/aws/aws-sdk-go-v2/service/sts"
"github.com/aws/smithy-go" "github.com/aws/smithy-go"
"github.com/aws/smithy-go/logging" "github.com/aws/smithy-go/logging"
"github.com/aws/smithy-go/middleware" "github.com/aws/smithy-go/middleware"
@@ -327,30 +325,6 @@ If empty it will default to the environment variable "AWS_PROFILE" or
Help: "An AWS session token.", Help: "An AWS session token.",
Advanced: true, Advanced: true,
Sensitive: true, Sensitive: true,
}, {
Name: "role_arn",
Help: `ARN of the IAM role to assume.
Leave blank if not using assume role.`,
Advanced: true,
}, {
Name: "role_session_name",
Help: `Session name for assumed role.
If empty, a session name will be generated automatically.`,
Advanced: true,
}, {
Name: "role_session_duration",
Help: `Session duration for assumed role.
If empty, the default session duration will be used.`,
Advanced: true,
}, {
Name: "role_external_id",
Help: `External ID for assumed role.
Leave blank if not using an external ID.`,
Advanced: true,
}, { }, {
Name: "upload_concurrency", Name: "upload_concurrency",
Help: `Concurrency for multipart uploads and copies. Help: `Concurrency for multipart uploads and copies.
@@ -953,10 +927,6 @@ type Options struct {
SharedCredentialsFile string `config:"shared_credentials_file"` SharedCredentialsFile string `config:"shared_credentials_file"`
Profile string `config:"profile"` Profile string `config:"profile"`
SessionToken string `config:"session_token"` SessionToken string `config:"session_token"`
RoleARN string `config:"role_arn"`
RoleSessionName string `config:"role_session_name"`
RoleSessionDuration fs.Duration `config:"role_session_duration"`
RoleExternalID string `config:"role_external_id"`
UploadConcurrency int `config:"upload_concurrency"` UploadConcurrency int `config:"upload_concurrency"`
ForcePathStyle bool `config:"force_path_style"` ForcePathStyle bool `config:"force_path_style"`
V2Auth bool `config:"v2_auth"` V2Auth bool `config:"v2_auth"`
@@ -1320,34 +1290,6 @@ func s3Connection(ctx context.Context, opt *Options, client *http.Client) (s3Cli
opt.Region = "us-east-1" opt.Region = "us-east-1"
} }
// Handle assume role if RoleARN is specified
if opt.RoleARN != "" {
fs.Debugf(nil, "Using assume role with ARN: %s", opt.RoleARN)
// Set region for the config before creating STS client
awsConfig.Region = opt.Region
// Create STS client using the base credentials
stsClient := sts.NewFromConfig(awsConfig)
// Configure AssumeRole options
assumeRoleOptions := func(aro *stscreds.AssumeRoleOptions) {
// Set session name if provided, otherwise use a default
if opt.RoleSessionName != "" {
aro.RoleSessionName = opt.RoleSessionName
}
if opt.RoleSessionDuration != 0 {
aro.Duration = time.Duration(opt.RoleSessionDuration)
}
if opt.RoleExternalID != "" {
aro.ExternalID = &opt.RoleExternalID
}
}
// Create AssumeRole credentials provider
awsConfig.Credentials = stscreds.NewAssumeRoleProvider(stsClient, opt.RoleARN, assumeRoleOptions)
}
provider = loadProvider(opt.Provider) provider = loadProvider(opt.Provider)
if provider == nil { if provider == nil {
fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider) fs.Logf("s3", "s3 provider %q not known - please set correctly", opt.Provider)

View File

@@ -1,27 +0,0 @@
// Package api has type definitions for shade
package api
// ListDirResponse -------------------------------------------------
// Format from shade api
type ListDirResponse struct {
Type string `json:"type"` // "file" or "tree"
Path string `json:"path"` // Full path including root
Ino int `json:"ino"` // inode number
Mtime int64 `json:"mtime"` // Modified time in milliseconds
Ctime int64 `json:"ctime"` // Created time in milliseconds
Size int64 `json:"size"` // Size in bytes
Hash string `json:"hash"` // MD5 hash
Draft bool `json:"draft"` // Whether this is a draft file
}
// PartURL Type for multipart upload/download
type PartURL struct {
URL string `json:"url"`
Headers map[string]string `json:"headers,omitempty"`
}
// CompletedPart Type for completed parts when making a multipart upload.
type CompletedPart struct {
ETag string
PartNumber int32
}

File diff suppressed because it is too large Load Diff

View File

@@ -1,21 +0,0 @@
package shade_test
import (
"testing"
"github.com/rclone/rclone/backend/shade"
"github.com/rclone/rclone/fstest/fstests"
)
// TestIntegration runs integration tests against the remote
func TestIntegration(t *testing.T) {
name := "TestShade"
fstests.Run(t, &fstests.Opt{
RemoteName: name + ":",
NilObject: (*shade.Object)(nil),
SkipInvalidUTF8: true,
ExtraConfig: []fstests.ExtraConfigItem{
{Name: name, Key: "eventually_consistent_delay", Value: "7"},
},
})
}

View File

@@ -1,336 +0,0 @@
//multipart upload for shade
package shade
import (
"bytes"
"context"
"fmt"
"io"
"net/http"
"net/url"
"path"
"sort"
"sync"
"github.com/rclone/rclone/backend/shade/api"
"github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/chunksize"
"github.com/rclone/rclone/lib/multipart"
"github.com/rclone/rclone/lib/rest"
)
var warnStreamUpload sync.Once
type shadeChunkWriter struct {
initToken string
chunkSize int64
size int64
f *Fs
o *Object
completedParts []api.CompletedPart
completedPartsMu sync.Mutex
}
// uploadMultipart handles multipart upload for larger files
func (o *Object) uploadMultipart(ctx context.Context, src fs.ObjectInfo, in io.Reader, options ...fs.OpenOption) error {
chunkWriter, err := multipart.UploadMultipart(ctx, src, in, multipart.UploadMultipartOptions{
Open: o.fs,
OpenOptions: options,
})
if err != nil {
return err
}
var shadeWriter = chunkWriter.(*shadeChunkWriter)
o.size = shadeWriter.size
return nil
}
// OpenChunkWriter returns the chunk size and a ChunkWriter
//
// Pass in the remote and the src object
// You can also use options to hint at the desired chunk size
func (f *Fs) OpenChunkWriter(ctx context.Context, remote string, src fs.ObjectInfo, options ...fs.OpenOption) (info fs.ChunkWriterInfo, writer fs.ChunkWriter, err error) {
// Temporary Object under construction
o := &Object{
fs: f,
remote: remote,
}
uploadParts := f.opt.MaxUploadParts
if uploadParts < 1 {
uploadParts = 1
} else if uploadParts > maxUploadParts {
uploadParts = maxUploadParts
}
size := src.Size()
fs.FixRangeOption(options, size)
// calculate size of parts
chunkSize := f.opt.ChunkSize
// size can be -1 here meaning we don't know the size of the incoming file. We use ChunkSize
// buffers here (default 64 MB). With a maximum number of parts (10,000) this will be a file of
// 640 GB.
if size == -1 {
warnStreamUpload.Do(func() {
fs.Logf(f, "Streaming uploads using chunk size %v will have maximum file size of %v",
chunkSize, fs.SizeSuffix(int64(chunkSize)*int64(uploadParts)))
})
} else {
chunkSize = chunksize.Calculator(src, size, uploadParts, chunkSize)
}
token, err := o.fs.refreshJWTToken(ctx)
if err != nil {
return info, nil, fmt.Errorf("failed to get token: %w", err)
}
err = f.ensureParentDirectories(ctx, remote)
if err != nil {
return info, nil, fmt.Errorf("failed to ensure parent directories: %w", err)
}
fullPath := remote
if f.root != "" {
fullPath = path.Join(f.root, remote)
}
// Initiate multipart upload
type initRequest struct {
Path string `json:"path"`
PartSize int64 `json:"partSize"`
}
reqBody := initRequest{
Path: fullPath,
PartSize: int64(chunkSize),
}
var initResp struct {
Token string `json:"token"`
}
opts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/%s/upload/multipart", o.fs.drive),
RootURL: o.fs.endpoint,
ExtraHeaders: map[string]string{
"Authorization": "Bearer " + token,
},
Options: options,
}
err = o.fs.pacer.Call(func() (bool, error) {
res, err := o.fs.srv.CallJSON(ctx, &opts, reqBody, &initResp)
if err != nil {
return res != nil && res.StatusCode == http.StatusTooManyRequests, err
}
return false, nil
})
if err != nil {
return info, nil, fmt.Errorf("failed to initiate multipart upload: %w", err)
}
chunkWriter := &shadeChunkWriter{
initToken: initResp.Token,
chunkSize: int64(chunkSize),
size: size,
f: f,
o: o,
}
info = fs.ChunkWriterInfo{
ChunkSize: int64(chunkSize),
Concurrency: f.opt.Concurrency,
LeavePartsOnError: false,
}
return info, chunkWriter, err
}
// WriteChunk will write chunk number with reader bytes, where chunk number >= 0
func (s *shadeChunkWriter) WriteChunk(ctx context.Context, chunkNumber int, reader io.ReadSeeker) (bytesWritten int64, err error) {
token, err := s.f.refreshJWTToken(ctx)
if err != nil {
return 0, err
}
// Read chunk
var chunk bytes.Buffer
n, err := io.Copy(&chunk, reader)
if n == 0 {
return 0, nil
}
if err != nil {
return 0, fmt.Errorf("failed to read chunk: %w", err)
}
// Get presigned URL for this part
var partURL api.PartURL
partOpts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/%s/upload/multipart/part/%d?token=%s", s.f.drive, chunkNumber+1, url.QueryEscape(s.initToken)),
RootURL: s.f.endpoint,
ExtraHeaders: map[string]string{
"Authorization": "Bearer " + token,
},
}
err = s.f.pacer.Call(func() (bool, error) {
res, err := s.f.srv.CallJSON(ctx, &partOpts, nil, &partURL)
if err != nil {
return res != nil && res.StatusCode == http.StatusTooManyRequests, err
}
return false, nil
})
if err != nil {
return 0, fmt.Errorf("failed to get part URL: %w", err)
}
opts := rest.Opts{
Method: "PUT",
RootURL: partURL.URL,
Body: &chunk,
ContentType: "",
ContentLength: &n,
}
// Add headers
var uploadRes *http.Response
if len(partURL.Headers) > 0 {
opts.ExtraHeaders = make(map[string]string)
for k, v := range partURL.Headers {
opts.ExtraHeaders[k] = v
}
}
err = s.f.pacer.Call(func() (bool, error) {
uploadRes, err = s.f.srv.Call(ctx, &opts)
if err != nil {
return uploadRes != nil && uploadRes.StatusCode == http.StatusTooManyRequests, err
}
return false, nil
})
if err != nil {
return 0, fmt.Errorf("failed to upload part %d: %w", chunk, err)
}
if uploadRes.StatusCode != http.StatusOK && uploadRes.StatusCode != http.StatusCreated {
body, _ := io.ReadAll(uploadRes.Body)
fs.CheckClose(uploadRes.Body, &err)
return 0, fmt.Errorf("part upload failed with status %d: %s", uploadRes.StatusCode, string(body))
}
// Get ETag from response
etag := uploadRes.Header.Get("ETag")
fs.CheckClose(uploadRes.Body, &err)
s.completedPartsMu.Lock()
defer s.completedPartsMu.Unlock()
s.completedParts = append(s.completedParts, api.CompletedPart{
PartNumber: int32(chunkNumber + 1),
ETag: etag,
})
return n, nil
}
// Close complete chunked writer finalising the file.
func (s *shadeChunkWriter) Close(ctx context.Context) error {
// Complete multipart upload
sort.Slice(s.completedParts, func(i, j int) bool {
return s.completedParts[i].PartNumber < s.completedParts[j].PartNumber
})
type completeRequest struct {
Parts []api.CompletedPart `json:"parts"`
}
var completeBody completeRequest
if s.completedParts == nil {
completeBody = completeRequest{Parts: []api.CompletedPart{}}
} else {
completeBody = completeRequest{Parts: s.completedParts}
}
token, err := s.f.refreshJWTToken(ctx)
if err != nil {
return err
}
completeOpts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/%s/upload/multipart/complete?token=%s", s.f.drive, url.QueryEscape(s.initToken)),
RootURL: s.f.endpoint,
ExtraHeaders: map[string]string{
"Authorization": "Bearer " + token,
},
}
var response http.Response
err = s.f.pacer.Call(func() (bool, error) {
res, err := s.f.srv.CallJSON(ctx, &completeOpts, completeBody, &response)
if err != nil && res == nil {
return false, err
}
if res.StatusCode == http.StatusTooManyRequests {
return true, err // Retry on 429
}
if res.StatusCode != http.StatusOK && res.StatusCode != http.StatusCreated {
body, _ := io.ReadAll(res.Body)
return false, fmt.Errorf("complete multipart failed with status %d: %s", res.StatusCode, string(body))
}
return false, nil
})
if err != nil {
return fmt.Errorf("failed to complete multipart upload: %w", err)
}
return nil
}
// Abort chunk write
//
// You can and should call Abort without calling Close.
func (s *shadeChunkWriter) Abort(ctx context.Context) error {
token, err := s.f.refreshJWTToken(ctx)
if err != nil {
return err
}
opts := rest.Opts{
Method: "POST",
Path: fmt.Sprintf("/%s/upload/abort/multipart?token=%s", s.f.drive, url.QueryEscape(s.initToken)),
RootURL: s.f.endpoint,
ExtraHeaders: map[string]string{
"Authorization": "Bearer " + token,
},
}
err = s.f.pacer.Call(func() (bool, error) {
res, err := s.f.srv.Call(ctx, &opts)
if err != nil {
fs.Debugf(s.f, "Failed to abort multipart upload: %v", err)
return false, nil // Don't retry abort
}
if res.StatusCode != http.StatusOK && res.StatusCode != http.StatusCreated {
fs.Debugf(s.f, "Abort returned status %d", res.StatusCode)
}
return false, nil
})
if err != nil {
return fmt.Errorf("failed to abort multipart upload: %w", err)
}
return nil
}

View File

@@ -84,7 +84,6 @@ docs = [
"protondrive.md", "protondrive.md",
"seafile.md", "seafile.md",
"sftp.md", "sftp.md",
"shade.md",
"smb.md", "smb.md",
"storj.md", "storj.md",
"sugarsync.md", "sugarsync.md",

View File

@@ -389,8 +389,8 @@ func parseHash(str string) (string, string, error) {
if str == "-" { if str == "-" {
return "", "", nil return "", "", nil
} }
if before, after, ok := strings.Cut(str, ":"); ok { if pos := strings.Index(str, ":"); pos > 0 {
name, val := before, after name, val := str[:pos], str[pos+1:]
if name != "" && val != "" { if name != "" && val != "" {
return name, val, nil return name, val, nil
} }

View File

@@ -26,10 +26,6 @@ Note that |ls| and |lsl| recurse by default - use |--max-depth 1| to stop the re
The other list commands |lsd|,|lsf|,|lsjson| do not recurse by default - The other list commands |lsd|,|lsf|,|lsjson| do not recurse by default -
use |-R| to make them recurse. use |-R| to make them recurse.
List commands prefer a recursive method that uses more memory but fewer
transactions by default. Use |--disable ListR| to suppress the behavior.
See [|--fast-list|](/docs/#fast-list) for more details.
Listing a nonexistent directory will produce an error except for Listing a nonexistent directory will produce an error except for
remotes which can't have empty directories (e.g. s3, swift, or gcs - remotes which can't have empty directories (e.g. s3, swift, or gcs -
the bucket-based remotes).`, "|", "`") the bucket-based remotes).`, "|", "`")

View File

@@ -13,26 +13,6 @@ docs](https://docs.aws.amazon.com/general/latest/gr/signature-version-4.html)).
`--auth-key` is not provided then `serve s3` will allow anonymous `--auth-key` is not provided then `serve s3` will allow anonymous
access. access.
Like all rclone flags `--auth-key` can be set via environment
variables, in this case `RCLONE_AUTH_KEY`. Since this flag can be
repeated, the input to `RCLONE_AUTH_KEY` is CSV encoded. Because the
`accessKey,secretKey` has a comma in, this means it needs to be in
quotes.
```console
export RCLONE_AUTH_KEY='"user,pass"'
rclone serve s3 ...
```
Or to supply multiple identities:
```console
export RCLONE_AUTH_KEY='"user1,pass1","user2,pass2"'
rclone serve s3 ...
```
Setting this variable without quotes will produce an error.
Please note that some clients may require HTTPS endpoints. See [the Please note that some clients may require HTTPS endpoints. See [the
SSL docs](#tls-ssl) for more information. SSL docs](#tls-ssl) for more information.

View File

@@ -70,11 +70,6 @@ func newServer(ctx context.Context, f fs.Fs, opt *Options, vfsOpt *vfscommon.Opt
w.s3Secret = getAuthSecret(opt.AuthKey) w.s3Secret = getAuthSecret(opt.AuthKey)
} }
authList, err := authlistResolver(opt.AuthKey)
if err != nil {
return nil, fmt.Errorf("parsing auth list failed: %q", err)
}
var newLogger logger var newLogger logger
w.faker = gofakes3.New( w.faker = gofakes3.New(
newBackend(w), newBackend(w),
@@ -82,7 +77,7 @@ func newServer(ctx context.Context, f fs.Fs, opt *Options, vfsOpt *vfscommon.Opt
gofakes3.WithLogger(newLogger), gofakes3.WithLogger(newLogger),
gofakes3.WithRequestID(rand.Uint64()), gofakes3.WithRequestID(rand.Uint64()),
gofakes3.WithoutVersioning(), gofakes3.WithoutVersioning(),
gofakes3.WithV4Auth(authList), gofakes3.WithV4Auth(authlistResolver(opt.AuthKey)),
gofakes3.WithIntegrityCheck(true), // Check Content-MD5 if supplied gofakes3.WithIntegrityCheck(true), // Check Content-MD5 if supplied
) )
@@ -97,7 +92,7 @@ func newServer(ctx context.Context, f fs.Fs, opt *Options, vfsOpt *vfscommon.Opt
w._vfs = vfs.New(f, vfsOpt) w._vfs = vfs.New(f, vfsOpt)
if len(opt.AuthKey) > 0 { if len(opt.AuthKey) > 0 {
w.faker.AddAuthKeys(authList) w.faker.AddAuthKeys(authlistResolver(opt.AuthKey))
} }
} }

View File

@@ -3,7 +3,6 @@ package s3
import ( import (
"context" "context"
"encoding/hex" "encoding/hex"
"errors"
"io" "io"
"os" "os"
"path" "path"
@@ -126,14 +125,15 @@ func rmdirRecursive(p string, VFS *vfs.VFS) {
} }
} }
func authlistResolver(list []string) (map[string]string, error) { func authlistResolver(list []string) map[string]string {
authList := make(map[string]string) authList := make(map[string]string)
for _, v := range list { for _, v := range list {
parts := strings.Split(v, ",") parts := strings.Split(v, ",")
if len(parts) != 2 { if len(parts) != 2 {
return nil, errors.New("invalid auth pair: expecting a single comma") fs.Infof(nil, "Ignored: invalid auth pair %s", v)
continue
} }
authList[parts[0]] = parts[1] authList[parts[0]] = parts[1]
} }
return authList, nil return authList
} }

View File

@@ -58,10 +58,10 @@ type conn struct {
// interoperate with the rclone sftp backend // interoperate with the rclone sftp backend
func (c *conn) execCommand(ctx context.Context, out io.Writer, command string) (err error) { func (c *conn) execCommand(ctx context.Context, out io.Writer, command string) (err error) {
binary, args := command, "" binary, args := command, ""
before, after, ok := strings.Cut(command, " ") space := strings.Index(command, " ")
if ok { if space >= 0 {
binary = before binary = command[:space]
args = strings.TrimLeft(after, " ") args = strings.TrimLeft(command[space+1:], " ")
} }
args = shellUnEscape(args) args = shellUnEscape(args)
fs.Debugf(c.what, "exec command: binary = %q, args = %q", binary, args) fs.Debugf(c.what, "exec command: binary = %q, args = %q", binary, args)

View File

@@ -45,10 +45,6 @@ var OptionsInfo = fs.Options{{
Name: "disable_dir_list", Name: "disable_dir_list",
Default: false, Default: false,
Help: "Disable HTML directory list on GET request for a directory", Help: "Disable HTML directory list on GET request for a directory",
}, {
Name: "disable_zip",
Default: false,
Help: "Disable zip download of directories",
}}. }}.
Add(libhttp.ConfigInfo). Add(libhttp.ConfigInfo).
Add(libhttp.AuthConfigInfo). Add(libhttp.AuthConfigInfo).
@@ -61,7 +57,6 @@ type Options struct {
Template libhttp.TemplateConfig Template libhttp.TemplateConfig
EtagHash string `config:"etag_hash"` EtagHash string `config:"etag_hash"`
DisableDirList bool `config:"disable_dir_list"` DisableDirList bool `config:"disable_dir_list"`
DisableZip bool `config:"disable_zip"`
} }
// Opt is options set by command line flags // Opt is options set by command line flags
@@ -413,24 +408,6 @@ func (w *WebDAV) serveDir(rw http.ResponseWriter, r *http.Request, dirRemote str
return return
} }
dir := node.(*vfs.Dir) dir := node.(*vfs.Dir)
if r.URL.Query().Get("download") == "zip" && !w.opt.DisableZip {
fs.Infof(dirRemote, "%s: Zipping directory", r.RemoteAddr)
zipName := path.Base(dirRemote)
if dirRemote == "" {
zipName = "root"
}
rw.Header().Set("Content-Disposition", "attachment; filename=\""+zipName+".zip\"")
rw.Header().Set("Content-Type", "application/zip")
rw.Header().Set("Last-Modified", time.Now().UTC().Format(http.TimeFormat))
err := vfs.CreateZip(ctx, dir, rw)
if err != nil {
serve.Error(ctx, dirRemote, rw, "Failed to create zip", err)
return
}
return
}
dirEntries, err := dir.ReadDirAll() dirEntries, err := dir.ReadDirAll()
if err != nil { if err != nil {
@@ -440,7 +417,6 @@ func (w *WebDAV) serveDir(rw http.ResponseWriter, r *http.Request, dirRemote str
// Make the entries for display // Make the entries for display
directory := serve.NewDirectory(dirRemote, w.server.HTMLTemplate()) directory := serve.NewDirectory(dirRemote, w.server.HTMLTemplate())
directory.DisableZip = w.opt.DisableZip
for _, node := range dirEntries { for _, node := range dirEntries {
if vfscommon.Opt.NoModTime { if vfscommon.Opt.NoModTime {
directory.AddHTMLEntry(node.Path(), node.IsDir(), node.Size(), time.Time{}) directory.AddHTMLEntry(node.Path(), node.IsDir(), node.Size(), time.Time{})

View File

@@ -202,7 +202,6 @@ WebDAV or S3, that work out of the box.)
{{< provider name="Selectel" home="https://selectel.ru/services/cloud/storage/" config="/s3/#selectel" >}} {{< provider name="Selectel" home="https://selectel.ru/services/cloud/storage/" config="/s3/#selectel" >}}
{{< provider name="Servercore Object Storage" home="https://servercore.com/services/object-storage/" config="/s3/#servercore" >}} {{< provider name="Servercore Object Storage" home="https://servercore.com/services/object-storage/" config="/s3/#servercore" >}}
{{< provider name="SFTP" home="https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol" config="/sftp/" >}} {{< provider name="SFTP" home="https://en.wikipedia.org/wiki/SSH_File_Transfer_Protocol" config="/sftp/" >}}
{{< provider name="Shade" home="https://shade.inc" config="/shade/" >}}
{{< provider name="Sia" home="https://sia.tech/" config="/sia/" >}} {{< provider name="Sia" home="https://sia.tech/" config="/sia/" >}}
{{< provider name="SMB / CIFS" home="https://en.wikipedia.org/wiki/Server_Message_Block" config="/smb/" >}} {{< provider name="SMB / CIFS" home="https://en.wikipedia.org/wiki/Server_Message_Block" config="/smb/" >}}
{{< provider name="Spectra Logic" home="https://spectralogic.com/blackpearl-nearline-object-gateway/" config="/s3/#spectralogic" >}} {{< provider name="Spectra Logic" home="https://spectralogic.com/blackpearl-nearline-object-gateway/" config="/s3/#spectralogic" >}}

View File

@@ -1048,15 +1048,3 @@ put them back in again. -->
- jijamik <30904953+jijamik@users.noreply.github.com> - jijamik <30904953+jijamik@users.noreply.github.com>
- Dominik Sander <git@dsander.de> - Dominik Sander <git@dsander.de>
- Nikolay Kiryanov <nikolay@kiryanov.ru> - Nikolay Kiryanov <nikolay@kiryanov.ru>
- Diana <5275194+DianaNites@users.noreply.github.com>
- Duncan Smart <duncan.smart@gmail.com>
- vicerace <vicerace@sohu.com>
- Cliff Frey <cliff@openai.com>
- Vladislav Tropnikov <vtr.name@gmail.com>
- Leo <i@hardrain980.com>
- Johannes Rothe <mail@johannes-rothe.de>
- Tingsong Xu <tingsong.xu@rightcapital.com>
- Jonas Tingeborn <134889+jojje@users.noreply.github.com>
- jhasse-shade <jacob@shade.inc>
- vyv03354 <VYV03354@nifty.ne.jp>
- masrlinu <masrlinu@users.noreply.github.com> <5259918+masrlinu@users.noreply.github.com>

View File

@@ -103,26 +103,6 @@ MD5 hashes are stored with blobs. However blobs that were uploaded in
chunks only have an MD5 if the source remote was capable of MD5 chunks only have an MD5 if the source remote was capable of MD5
hashes, e.g. the local disk. hashes, e.g. the local disk.
### Metadata and tags
Rclone can map arbitrary metadata to Azure Blob headers, user metadata, and tags
when `--metadata` is enabled (or when using `--metadata-set` / `--metadata-mapper`).
- Headers: Set these keys in metadata to map to the corresponding blob headers:
- `cache-control`, `content-disposition`, `content-encoding`, `content-language`, `content-type`.
- User metadata: Any other non-reserved keys are written as user metadata
(keys are normalized to lowercase). Keys starting with `x-ms-` are reserved and
are not stored as user metadata.
- Tags: Provide `x-ms-tags` as a comma-separated list of `key=value` pairs, e.g.
`x-ms-tags=env=dev,team=sync`. These are applied as blob tags on upload and on
server-side copies. Whitespace around keys/values is ignored.
- Modtime override: Provide `mtime` in RFC3339/RFC3339Nano format to override the
stored modtime persisted in user metadata. If `mtime` cannot be parsed, rclone
logs a debug message and ignores the override.
Notes:
- Rclone ignores reserved `x-ms-*` keys (except `x-ms-tags`) for user metadata.
### Performance ### Performance
When uploading large files, increasing the value of When uploading large files, increasing the value of

View File

@@ -283,7 +283,7 @@ It is useful to know how many requests are sent to the server in different scena
All copy commands send the following 4 requests: All copy commands send the following 4 requests:
```text ```text
/b2api/v4/b2_authorize_account /b2api/v1/b2_authorize_account
/b2api/v1/b2_create_bucket /b2api/v1/b2_create_bucket
/b2api/v1/b2_list_buckets /b2api/v1/b2_list_buckets
/b2api/v1/b2_list_file_names /b2api/v1/b2_list_file_names

View File

@@ -1049,7 +1049,17 @@ The following backends have known issues that need more investigation:
<!--- start list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---> <!--- start list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
- `TestDropbox` (`dropbox`) - `TestDropbox` (`dropbox`)
- [`TestBisyncRemoteRemote/normalization`](https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt) - [`TestBisyncRemoteRemote/normalization`](https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt)
- Updated: 2025-11-21-010037 - `TestGoFile` (`gofile`)
- [`TestBisyncRemoteLocal/all_changed`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/backupdir`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/basic`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/changes`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [`TestBisyncRemoteLocal/check_access`](https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
- [78 more](https://pub.rclone.org/integration-tests/current/)
- `TestPcloud` (`pcloud`)
- [`TestBisyncRemoteRemote/check_access`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- [`TestBisyncRemoteRemote/check_access_filters`](https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
- Updated: 2025-12-10-010012
<!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs ---> <!--- end list_failures - DO NOT EDIT THIS SECTION - use make commanddocs --->
The following backends either have not been tested recently or have known issues The following backends either have not been tested recently or have known issues

View File

@@ -369,7 +369,7 @@ rclone [flags]
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it
@@ -1023,7 +1023,7 @@ rclone [flags]
--use-json-log Use json log format --use-json-log Use json log format
--use-mmap Use mmap allocator (see docs) --use-mmap Use mmap allocator (see docs)
--use-server-modtime Use server modified time instead of object metadata --use-server-modtime Use server modified time instead of object metadata
--user-agent string Set the user-agent to a specified string (default "rclone/v1.72.0") --user-agent string Set the user-agent to a specified string (default "rclone/v1.72.1")
-v, --verbose count Print lots more stuff (repeat for more) -v, --verbose count Print lots more stuff (repeat for more)
-V, --version Print the version number -V, --version Print the version number
--webdav-auth-redirect Preserve authentication on redirect --webdav-auth-redirect Preserve authentication on redirect

View File

@@ -231,12 +231,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20251121 // Output: stories/The Quick Brown Fox!-20251210
``` ```
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2025-11-21 0505PM // Output: stories/The Quick Brown Fox!-2025-12-10 1247PM
``` ```
```console ```console

View File

@@ -82,7 +82,6 @@ See the following for detailed instructions for
- [rsync.net](/sftp/#rsync-net) - [rsync.net](/sftp/#rsync-net)
- [Seafile](/seafile/) - [Seafile](/seafile/)
- [SFTP](/sftp/) - [SFTP](/sftp/)
- [Shade](/shade/)
- [Sia](/sia/) - [Sia](/sia/)
- [SMB](/smb/) - [SMB](/smb/)
- [Storj](/storj/) - [Storj](/storj/)
@@ -3278,10 +3277,6 @@ The available flags are:
- `mapper` dumps the JSON blobs being sent to the program supplied with - `mapper` dumps the JSON blobs being sent to the program supplied with
`--metadata-mapper` and received from it. It can be useful for debugging `--metadata-mapper` and received from it. It can be useful for debugging
the metadata mapper interface. the metadata mapper interface.
- `curl` dumps the HTTP request as a `curl` command. Can be used with
the other HTTP debugging flags (e.g. `requests`, `bodies`). By
default the auth will be masked - use with `auth` to have the curl
commands with authentication too.
## Filtering ## Filtering

View File

@@ -121,7 +121,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this --tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar --use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default "rclone/v1.72.0") --user-agent string Set the user-agent to a specified string (default "rclone/v1.72.1")
``` ```
@@ -638,7 +638,7 @@ Backend-only flags (these can be set in the config file also).
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don't attempt to check the bucket exists or create it

View File

@@ -785,9 +785,14 @@ Properties:
#### --gcs-endpoint #### --gcs-endpoint
Endpoint for the service. Custom endpoint for the storage API. Leave blank to use the provider default.
Leave blank normally. When using a custom endpoint that includes a subpath (e.g. example.org/custom/endpoint),
the subpath will be ignored during upload operations due to a limitation in the
underlying Google API Go client library.
Download and listing operations will work correctly with the full endpoint path.
If you require subpath support for uploads, avoid using subpaths in your custom
endpoint configuration.
Properties: Properties:
@@ -795,6 +800,13 @@ Properties:
- Env Var: RCLONE_GCS_ENDPOINT - Env Var: RCLONE_GCS_ENDPOINT
- Type: string - Type: string
- Required: false - Required: false
- Examples:
- "storage.example.org"
- Specify a custom endpoint
- "storage.example.org:4443"
- Specifying a custom endpoint with port
- "storage.example.org:4443/gcs/api"
- Specifying a subpath, see the note, uploads won't use the custom path!
#### --gcs-encoding #### --gcs-encoding

View File

@@ -59,7 +59,6 @@ Here is an overview of the major features of each cloud storage system.
| Quatrix by Maytech | - | R/W | No | No | - | - | | Quatrix by Maytech | - | R/W | No | No | - | - |
| Seafile | - | - | No | No | - | - | | Seafile | - | - | No | No | - | - |
| SFTP | MD5, SHA1 ² | DR/W | Depends | No | - | - | | SFTP | MD5, SHA1 ² | DR/W | Depends | No | - | - |
| Shade | - | - | Yes | No | - | - |
| Sia | - | - | No | No | - | - | | Sia | - | - | No | No | - | - |
| SMB | - | R/W | Yes | No | - | - | | SMB | - | R/W | Yes | No | - | - |
| SugarSync | - | - | No | No | - | - | | SugarSync | - | - | No | No | - | - |
@@ -541,7 +540,7 @@ upon backend-specific capabilities.
| OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes | | OpenDrive | Yes | Yes | Yes | Yes | No | No | No | No | No | Yes | Yes |
| OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No | | OpenStack Swift | Yes ¹ | Yes | No | No | No | Yes | Yes | No | No | Yes | No |
| Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No | | Oracle Object Storage | No | Yes | No | No | Yes | Yes | Yes | Yes | No | No | No |
| pCloud | Yes | Yes | Yes | Yes | Yes | Yes | No | No | Yes | Yes | Yes | | pCloud | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
| PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes | | PikPak | Yes | Yes | Yes | Yes | Yes | No | No | No | Yes | Yes | Yes |
| Pixeldrain | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes | | Pixeldrain | Yes | No | Yes | Yes | No | No | Yes | No | Yes | Yes | Yes |
| premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes | | premiumize.me | Yes | No | Yes | Yes | No | No | No | No | Yes | Yes | Yes |

View File

@@ -173,31 +173,6 @@ So if the folder you want rclone to use your is "My Music/", then use the return
id from ```rclone lsf``` command (ex. `dxxxxxxxx2`) as the `root_folder_id` variable id from ```rclone lsf``` command (ex. `dxxxxxxxx2`) as the `root_folder_id` variable
value in the config file. value in the config file.
### Change notifications and mounts
The pCloud backend supports realtime updates for rclone mounts via change
notifications. rclone uses pClouds diff longpolling API to detect changes and
will automatically refresh directory listings in the mounted filesystem when
changes occur.
Notes and behavior:
- Works automatically when using `rclone mount` and requires no additional
configuration.
- Notifications are directoryscoped: when rclone detects a change, it refreshes
the affected directory so new/removed/renamed files become visible promptly.
- Updates are near realtime. The backend uses a longpoll with short fallback
polling intervals, so you should see changes appear quickly without manual
refreshes.
If you want to debug or verify notifications, you can use the helper command:
```bash
rclone test changenotify remote:
```
This will log incoming change notifications for the given remote.
<!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/pcloud/pcloud.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length --> <!-- autogenerated options start - DO NOT EDIT - instead edit fs.RegInfo in backend/pcloud/pcloud.go and run make backenddocs to verify --> <!-- markdownlint-disable-line line-length -->
### Standard options ### Standard options

View File

@@ -745,68 +745,6 @@ If none of these option actually end up providing `rclone` with AWS
credentials then S3 interaction will be non-authenticated (see the credentials then S3 interaction will be non-authenticated (see the
[anonymous access](#anonymous-access) section for more info). [anonymous access](#anonymous-access) section for more info).
#### Assume Role (Cross-Account Access)
If you need to access S3 resources in a different AWS account, you can use IAM role assumption.
This is useful for cross-account access scenarios where you have credentials in one account
but need to access resources in another account.
To use assume role, configure the following parameters:
- `role_arn` - The ARN (Amazon Resource Name) of the IAM role to assume in the target account.
Format: `arn:aws:iam::ACCOUNT-ID:role/ROLE-NAME`
- `role_session_name` (optional) - A name for the assumed role session. If not specified,
rclone will generate one automatically.
- `role_session_duration` (optional) - Duration for which the assumed role credentials are valid.
If not specified, AWS default duration will be used (typically 1 hour).
- `role_external_id` (optional) - An external ID required by the role's trust policy for additional security.
This is typically used when the role is accessed by a third party.
The assume role feature works with both direct credentials (`env_auth = false`) and environment-based
authentication (`env_auth = true`). Rclone will first authenticate using the base credentials, then
use those credentials to assume the specified role.
Example configuration for cross-account access:
```
[s3-cross-account]
type = s3
provider = AWS
env_auth = true
region = us-east-1
role_arn = arn:aws:iam::123456789012:role/CrossAccountS3Role
role_session_name = rclone-session
role_external_id = unique-role-external-id-12345
```
In this example:
- Base credentials are obtained from the environment (IAM role, credentials file, or environment variables)
- These credentials are then used to assume the role `CrossAccountS3Role` in account `123456789012`
- An external ID is provided for additional security as required by the role's trust policy
The target role's trust policy in the destination account must allow the source account or user to assume it.
Example trust policy:
```json
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::SOURCE-ACCOUNT-ID:root"
},
"Action": "sts:AssumeRole",
"Condition": {
"StringEquals": {
"sts:ExternalID": "unique-role-external-id-12345"
}
}
}
]
}
```
### S3 Permissions ### S3 Permissions
When using the `sync` subcommand of `rclone` the following minimum When using the `sync` subcommand of `rclone` the following minimum
@@ -1445,12 +1383,21 @@ Properties:
- "ru-1" - "ru-1"
- St. Petersburg - St. Petersburg
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "gis-1" - "ru-3"
- Moscow - St. Petersburg
- Provider: Servercore - Provider: Selectel
- "ru-7" - "ru-7"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "gis-1"
- Moscow
- Provider: Selectel,Servercore
- "kz-1"
- Kazakhstan
- Provider: Selectel
- "uz-2"
- Uzbekistan
- Provider: Selectel
- "uz-2" - "uz-2"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan
- Provider: Servercore - Provider: Servercore
@@ -2242,13 +2189,25 @@ Properties:
- SeaweedFS S3 localhost - SeaweedFS S3 localhost
- Provider: SeaweedFS - Provider: SeaweedFS
- "s3.ru-1.storage.selcloud.ru" - "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg - St. Petersburg
- Provider: Selectel
- "s3.ru-3.storage.selcloud.ru"
- St. Petersburg
- Provider: Selectel
- "s3.ru-7.storage.selcloud.ru"
- Moscow
- Provider: Selectel,Servercore - Provider: Selectel,Servercore
- "s3.gis-1.storage.selcloud.ru" - "s3.gis-1.storage.selcloud.ru"
- Moscow - Moscow
- Provider: Servercore - Provider: Selectel,Servercore
- "s3.ru-7.storage.selcloud.ru" - "s3.kz-1.storage.selcloud.ru"
- Moscow - Kazakhstan
- Provider: Selectel
- "s3.uz-2.storage.selcloud.ru"
- Uzbekistan
- Provider: Selectel
- "s3.ru-1.storage.selcloud.ru"
- Saint Petersburg
- Provider: Servercore - Provider: Servercore
- "s3.uz-2.srvstorage.uz" - "s3.uz-2.srvstorage.uz"
- Tashkent, Uzbekistan - Tashkent, Uzbekistan

View File

@@ -1,218 +0,0 @@
# {{< icon "fa fa-moon" >}} Shade
This is a backend for the [Shade](https://shade.inc/) platform
## About Shade
[Shade](https://shade.inc/) is an AI-powered cloud NAS that makes your cloud files behave like a local drive, optimized for media and creative workflows. It provides fast, secure access with natural-language search, easy sharing, and scalable cloud storage.
## Accounts & Pricing
To use this backend, you need to [create a free account](https://app.shade.inc/) on Shade. You can start with a free account and get 20GB of storage for free.
## Usage
Paths are specified as `remote:path`
Paths may be as deep as required, e.g. `remote:directory/subdirectory`.
## Configuration
Here is an example of making a Shade configuration.
First, create a [create a free account](https://app.shade.inc/) account and choose a plan.
You will need to log in and get the `API Key` and `Drive ID` for your account from the settings section of your account and created drive respectively.
Now run
`rclone config`
Follow this interactive process:
```sh
$ rclone config
e) Edit existing remote
n) New remote
d) Delete remote
r) Rename remote
c) Copy remote
s) Set configuration password
q) Quit config
e/n/d/r/c/s/q> n
Enter name for new remote.
name> Shade
Option Storage.
Type of storage to configure.
Choose a number from below, or type in your own value.
[OTHER OPTIONS]
xx / Shade FS
\ (shade)
[OTHER OPTIONS]
Storage> xx
Option drive_id.
The ID of your drive, see this in the drive settings. Individual rclone configs must be made per drive.
Enter a value.
drive_id> [YOUR_ID]
Option api_key.
An API key for your account.
Enter a value.
api_key> [YOUR_API_KEY]
Edit advanced config?
y) Yes
n) No (default)
y/n> n
Configuration complete.
Options:
- type: shade
- drive_id: [YOUR_ID]
- api_key: [YOUR_API_KEY]
Keep this "Shade" remote?
y) Yes this is OK (default)
e) Edit this remote
d) Delete this remote
y/e/d> y
```
### Modification times and hashes
Shade does not support hashes and writing mod times.
### Transfers
Shade uses multipart uploads by default. This means that files will be chunked and sent up to Shade concurrently. In order to configure how many simultaneous uploads you want to use, upload the 'concurrency' option in the advanced config section. Note that this uses more memory and initiates more http requests.
### Deleting files
Please note that when deleting files in Shade via rclone it will delete the file instantly, instead of sending it to the trash. This means that it will not be recoverable.
{{< rem autogenerated options start" - DO NOT EDIT - instead edit fs.RegInfo in backend/box/box.go then run make backenddocs" >}}
### Standard options
Here are the Standard options specific to shade (Shade FS).
#### --shade-drive-id
The ID of your drive, see this in the drive settings. Individual rclone configs must be made per drive.
Properties:
- Config: drive_id
- Env Var: RCLONE_SHADE_DRIVE_ID
- Type: string
- Required: true
#### --shade-api-key
An API key for your account. You can find this under Settings > API Keys
Properties:
- Config: api_key
- Env Var: RCLONE_SHADE_API_KEY
- Type: string
- Required: true
### Advanced options
Here are the Advanced options specific to shade (Shade FS).
#### --shade-endpoint
Endpoint for the service.
Leave blank normally.
Properties:
- Config: endpoint
- Env Var: RCLONE_SHADE_ENDPOINT
- Type: string
- Required: false
#### --shade-chunk-size
Chunk size to use for uploading.
Any files larger than this will be uploaded in chunks of this size.
Note that this is stored in memory per transfer, so increasing it will
increase memory usage.
Minimum is 5MB, maximum is 5GB.
Properties:
- Config: chunk_size
- Env Var: RCLONE_SHADE_CHUNK_SIZE
- Type: SizeSuffix
- Default: 64Mi
#### --shade-encoding
The encoding for the backend.
See the [encoding section in the overview](/overview/#encoding) for more info.
Properties:
- Config: encoding
- Env Var: RCLONE_SHADE_ENCODING
- Type: Encoding
- Default: Slash,BackSlash,Del,Ctl,InvalidUtf8,Dot
#### --shade-description
Description of the remote.
Properties:
- Config: description
- Env Var: RCLONE_SHADE_DESCRIPTION
- Type: string
- Required: false
{{< rem autogenerated options stop >}}
## Limitations
Note that Shade is case insensitive so you can't have a file called
"Hello.doc" and one called "hello.doc".
Shade only supports filenames up to 255 characters in length.
`rclone about` is not supported by the Shade backend. Backends without
this capability cannot determine free space for an rclone mount or
use policy `mfs` (most free space) as a member of an rclone union
remote.
See [List of backends that do not support rclone about](https://rclone.org/overview/#optional-features) and [rclone about](https://rclone.org/commands/rclone_about/)
## Backend commands
Here are the commands specific to the shade backend.
Run them with
rclone backend COMMAND remote:
The help below will explain what arguments each command takes.
See the [backend](/commands/rclone_backend/) command for more
info on how to pass options and arguments.
These can be run on a running backend using the rc command
[backend/command](/rc/#backend-command).

View File

@@ -107,7 +107,6 @@
<a class="dropdown-item" href="/seafile/"><i class="fa fa-server fa-fw"></i> Seafile</a> <a class="dropdown-item" href="/seafile/"><i class="fa fa-server fa-fw"></i> Seafile</a>
<a class="dropdown-item" href="/sftp/"><i class="fa fa-server fa-fw"></i> SFTP</a> <a class="dropdown-item" href="/sftp/"><i class="fa fa-server fa-fw"></i> SFTP</a>
<a class="dropdown-item" href="/sia/"><i class="fa fa-globe fa-fw"></i> Sia</a> <a class="dropdown-item" href="/sia/"><i class="fa fa-globe fa-fw"></i> Sia</a>
<a class="dropdown-item" href="/shade/"><i class="fa fa-moon fa-fw"></i> Shade</a>
<a class="dropdown-item" href="/smb/"><i class="fa fa-server fa-fw"></i> SMB / CIFS</a> <a class="dropdown-item" href="/smb/"><i class="fa fa-server fa-fw"></i> SMB / CIFS</a>
<a class="dropdown-item" href="/storj/"><i class="fas fa-dove fa-fw"></i> Storj</a> <a class="dropdown-item" href="/storj/"><i class="fas fa-dove fa-fw"></i> Storj</a>
<a class="dropdown-item" href="/sugarsync/"><i class="fas fa-dove fa-fw"></i> SugarSync</a> <a class="dropdown-item" href="/sugarsync/"><i class="fas fa-dove fa-fw"></i> SugarSync</a>

View File

@@ -1 +1 @@
v1.73.0 v1.72.1

View File

@@ -29,16 +29,16 @@ func (bp *BwPair) String() string {
// Set the bandwidth from a string which is either // Set the bandwidth from a string which is either
// SizeSuffix or SizeSuffix:SizeSuffix (for tx:rx bandwidth) // SizeSuffix or SizeSuffix:SizeSuffix (for tx:rx bandwidth)
func (bp *BwPair) Set(s string) (err error) { func (bp *BwPair) Set(s string) (err error) {
before, after, ok := strings.Cut(s, ":") colon := strings.Index(s, ":")
stx, srx := s, "" stx, srx := s, ""
if ok { if colon >= 0 {
stx, srx = before, after stx, srx = s[:colon], s[colon+1:]
} }
err = bp.Tx.Set(stx) err = bp.Tx.Set(stx)
if err != nil { if err != nil {
return err return err
} }
if !ok { if colon < 0 {
bp.Rx = bp.Tx bp.Rx = bp.Tx
} else { } else {
err = bp.Rx.Set(srx) err = bp.Rx.Set(srx)

View File

@@ -14,7 +14,6 @@ const (
DumpGoRoutines DumpGoRoutines
DumpOpenFiles DumpOpenFiles
DumpMapper DumpMapper
DumpCurl
) )
type dumpChoices struct{} type dumpChoices struct{}
@@ -30,7 +29,6 @@ func (dumpChoices) Choices() []BitsChoicesInfo {
{uint64(DumpGoRoutines), "goroutines"}, {uint64(DumpGoRoutines), "goroutines"},
{uint64(DumpOpenFiles), "openfiles"}, {uint64(DumpOpenFiles), "openfiles"},
{uint64(DumpMapper), "mapper"}, {uint64(DumpMapper), "mapper"},
{uint64(DumpCurl), "curl"},
} }
} }

View File

@@ -15,8 +15,6 @@ import (
"net/http/httputil" "net/http/httputil"
"net/url" "net/url"
"os" "os"
"slices"
"strings"
"sync" "sync"
"time" "time"
@@ -26,7 +24,6 @@ import (
"github.com/rclone/rclone/lib/structs" "github.com/rclone/rclone/lib/structs"
"github.com/youmark/pkcs8" "github.com/youmark/pkcs8"
"golang.org/x/net/publicsuffix" "golang.org/x/net/publicsuffix"
"moul.io/http2curl/v2"
) )
const ( const (
@@ -442,18 +439,6 @@ func cleanAuths(buf []byte) []byte {
return buf return buf
} }
// cleanCurl gets rid of Auth headers in a curl command
func cleanCurl(cmd *http2curl.CurlCommand) {
for _, authBuf := range authBufs {
auth := "'" + string(authBuf)
for i, arg := range *cmd {
if strings.HasPrefix(arg, auth) {
(*cmd)[i] = auth + "XXXX'"
}
}
}
}
var expireWindow = 30 * time.Second var expireWindow = 30 * time.Second
func isCertificateExpired(cc *tls.Config) bool { func isCertificateExpired(cc *tls.Config) bool {
@@ -507,26 +492,6 @@ func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error
fs.Debugf(nil, "%s", separatorReq) fs.Debugf(nil, "%s", separatorReq)
logMutex.Unlock() logMutex.Unlock()
} }
// Dump curl request
if t.dump&(fs.DumpCurl) != 0 {
cmd, err := http2curl.GetCurlCommand(req)
if err != nil {
fs.Debugf(nil, "Failed to create curl command: %v", err)
} else {
// Patch -X HEAD into --head
for i := range len(*cmd) - 1 {
if (*cmd)[i] == "-X" && (*cmd)[i+1] == "'HEAD'" {
(*cmd)[i] = "--head"
*cmd = slices.Delete(*cmd, i+1, i+2)
break
}
}
if t.dump&fs.DumpAuth == 0 {
cleanCurl(cmd)
}
fs.Debugf(nil, "HTTP REQUEST: %v", cmd)
}
}
// Do round trip // Do round trip
resp, err = t.Transport.RoundTrip(req) resp, err = t.Transport.RoundTrip(req)
// Logf response // Logf response

View File

@@ -19,7 +19,6 @@ import (
"github.com/rclone/rclone/fs" "github.com/rclone/rclone/fs"
"github.com/stretchr/testify/assert" "github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require" "github.com/stretchr/testify/require"
"moul.io/http2curl/v2"
) )
func TestCleanAuth(t *testing.T) { func TestCleanAuth(t *testing.T) {
@@ -62,32 +61,6 @@ func TestCleanAuths(t *testing.T) {
} }
} }
func TestCleanCurl(t *testing.T) {
for _, test := range []struct {
in []string
want []string
}{{
[]string{""},
[]string{""},
}, {
[]string{"floo"},
[]string{"floo"},
}, {
[]string{"'Authorization: AAAAAAAAA'", "'Potato: Help'", ""},
[]string{"'Authorization: XXXX'", "'Potato: Help'", ""},
}, {
[]string{"'X-Auth-Token: AAAAAAAAA'", "'Potato: Help'", ""},
[]string{"'X-Auth-Token: XXXX'", "'Potato: Help'", ""},
}, {
[]string{"'X-Auth-Token: AAAAAAAAA'", "'Authorization: AAAAAAAAA'", "'Potato: Help'", ""},
[]string{"'X-Auth-Token: XXXX'", "'Authorization: XXXX'", "'Potato: Help'", ""},
}} {
in := http2curl.CurlCommand(test.in)
cleanCurl(&in)
assert.Equal(t, test.want, test.in, test.in)
}
}
var certSerial = int64(0) var certSerial = int64(0)
// Create a test certificate and key pair that is valid for a specific // Create a test certificate and key pair that is valid for a specific

View File

@@ -1301,7 +1301,6 @@ func TestSyncAfterRemovingAFileAndAddingAFileSubDirWithErrors(t *testing.T) {
err := Sync(ctx, r.Fremote, r.Flocal, false) err := Sync(ctx, r.Fremote, r.Flocal, false)
assert.Equal(t, fs.ErrorNotDeleting, err) assert.Equal(t, fs.ErrorNotDeleting, err)
testLoggerVsLsf(ctx, r.Fremote, r.Flocal, operations.GetLoggerOpt(ctx).JSON, t) testLoggerVsLsf(ctx, r.Fremote, r.Flocal, operations.GetLoggerOpt(ctx).JSON, t)
accounting.GlobalStats().ResetCounters()
r.CheckLocalListing( r.CheckLocalListing(
t, t,

View File

@@ -13,7 +13,6 @@ import (
_ "github.com/rclone/rclone/backend/all" _ "github.com/rclone/rclone/backend/all"
"github.com/rclone/rclone/fs" "github.com/rclone/rclone/fs"
"github.com/rclone/rclone/fs/accounting"
"github.com/rclone/rclone/fs/filter" "github.com/rclone/rclone/fs/filter"
"github.com/rclone/rclone/fs/operations" "github.com/rclone/rclone/fs/operations"
"github.com/rclone/rclone/fs/walk" "github.com/rclone/rclone/fs/walk"
@@ -508,7 +507,6 @@ func TestError(t *testing.T) {
err = Sync(ctx, r.Fremote, r.Flocal, true) err = Sync(ctx, r.Fremote, r.Flocal, true)
// testLoggerVsLsf(ctx, r.Fremote, r.Flocal, operations.GetLoggerOpt(ctx).JSON, t) // testLoggerVsLsf(ctx, r.Fremote, r.Flocal, operations.GetLoggerOpt(ctx).JSON, t)
assert.Error(t, err) assert.Error(t, err)
accounting.GlobalStats().ResetCounters()
r.CheckLocalListing(t, []fstest.Item{file1}, []string{"toe", "toe/toe"}) r.CheckLocalListing(t, []fstest.Item{file1}, []string{"toe", "toe/toe"})
r.CheckRemoteListing(t, []fstest.Item{file1}, []string{"toe", "toe/toe"}) r.CheckRemoteListing(t, []fstest.Item{file1}, []string{"toe", "toe/toe"})

View File

@@ -1,4 +1,4 @@
package fs package fs
// VersionTag of rclone // VersionTag of rclone
var VersionTag = "v1.73.0" var VersionTag = "v1.72.1"

View File

@@ -368,7 +368,7 @@ func Run(t *testing.T, opt *Opt) {
} }
file1Contents string file1Contents string
file1MimeType = "text/csv" file1MimeType = "text/csv"
file1Metadata = fs.Metadata{"rclonetest": "potato"} file1Metadata = fs.Metadata{"rclone-test": "potato"}
file2 = fstest.Item{ file2 = fstest.Item{
ModTime: fstest.Time("2001-02-03T04:05:10.123123123Z"), ModTime: fstest.Time("2001-02-03T04:05:10.123123123Z"),
Path: `hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt`, Path: `hello? sausage/êé/Hello, 世界/ " ' @ < > & ? + ≠/z.txt`,

View File

@@ -662,10 +662,6 @@ backends:
ignoretests: ignoretests:
- cmd/bisync - cmd/bisync
- cmd/gitannex - cmd/gitannex
- backend: "shade"
remote: "TestShade:"
fastlist: false
- backend: "archive" - backend: "archive"
remote: "TestArchive:" remote: "TestArchive:"
fastlist: false fastlist: false

2
go.mod
View File

@@ -25,7 +25,6 @@ require (
github.com/aws/aws-sdk-go-v2/credentials v1.18.21 github.com/aws/aws-sdk-go-v2/credentials v1.18.21
github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4 github.com/aws/aws-sdk-go-v2/feature/s3/manager v1.20.4
github.com/aws/aws-sdk-go-v2/service/s3 v1.90.0 github.com/aws/aws-sdk-go-v2/service/s3 v1.90.0
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1
github.com/aws/smithy-go v1.23.2 github.com/aws/smithy-go v1.23.2
github.com/buengese/sgzip v0.1.1 github.com/buengese/sgzip v0.1.1
github.com/cloudinary/cloudinary-go/v2 v2.13.0 github.com/cloudinary/cloudinary-go/v2 v2.13.0
@@ -134,6 +133,7 @@ require (
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.13 // indirect github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.19.13 // indirect
github.com/aws/aws-sdk-go-v2/service/sso v1.30.1 // indirect github.com/aws/aws-sdk-go-v2/service/sso v1.30.1 // indirect
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5 // indirect github.com/aws/aws-sdk-go-v2/service/ssooidc v1.35.5 // indirect
github.com/aws/aws-sdk-go-v2/service/sts v1.39.1 // indirect
github.com/bahlo/generic-list-go v0.2.0 // indirect github.com/bahlo/generic-list-go v0.2.0 // indirect
github.com/beorn7/perks v1.0.1 // indirect github.com/beorn7/perks v1.0.1 // indirect
github.com/bodgit/plumbing v1.3.0 // indirect github.com/bodgit/plumbing v1.3.0 // indirect

View File

@@ -218,12 +218,12 @@ rclone convmv "stories/The Quick Brown Fox!.txt" --name-transform "all,command=e
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{YYYYMMDD}"
// Output: stories/The Quick Brown Fox!-20251121 // Output: stories/The Quick Brown Fox!-20251210
``` ```
```console ```console
rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}" rclone convmv "stories/The Quick Brown Fox!" --name-transform "date=-{macfriendlytime}"
// Output: stories/The Quick Brown Fox!-2025-11-21 0508PM // Output: stories/The Quick Brown Fox!-2025-12-10 1253PM
``` ```
```console ```console

185
rclone.1 generated
View File

@@ -15,7 +15,7 @@
. ftr VB CB . ftr VB CB
. ftr VBI CBI . ftr VBI CBI
.\} .\}
.TH "rclone" "1" "Nov 21, 2025" "User Manual" "" .TH "rclone" "1" "Dec 10, 2025" "User Manual" ""
.hy .hy
.SH NAME .SH NAME
.PP .PP
@@ -6260,14 +6260,14 @@ rclone convmv \[dq]stories/The Quick Brown Fox!.txt\[dq] --name-transform \[dq]a
.nf .nf
\f[C] \f[C]
rclone convmv \[dq]stories/The Quick Brown Fox!\[dq] --name-transform \[dq]date=-{YYYYMMDD}\[dq] rclone convmv \[dq]stories/The Quick Brown Fox!\[dq] --name-transform \[dq]date=-{YYYYMMDD}\[dq]
// Output: stories/The Quick Brown Fox!-20251121 // Output: stories/The Quick Brown Fox!-20251210
\f[R] \f[R]
.fi .fi
.IP .IP
.nf .nf
\f[C] \f[C]
rclone convmv \[dq]stories/The Quick Brown Fox!\[dq] --name-transform \[dq]date=-{macfriendlytime}\[dq] rclone convmv \[dq]stories/The Quick Brown Fox!\[dq] --name-transform \[dq]date=-{macfriendlytime}\[dq]
// Output: stories/The Quick Brown Fox!-2025-11-21 0505PM // Output: stories/The Quick Brown Fox!-2025-12-10 1247PM
\f[R] \f[R]
.fi .fi
.IP .IP
@@ -31741,7 +31741,7 @@ Flags for general networking and HTTP stuff.
--tpslimit float Limit HTTP transactions per second to this --tpslimit float Limit HTTP transactions per second to this
--tpslimit-burst int Max burst of transactions for --tpslimit (default 1) --tpslimit-burst int Max burst of transactions for --tpslimit (default 1)
--use-cookies Enable session cookiejar --use-cookies Enable session cookiejar
--user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.72.0\[dq]) --user-agent string Set the user-agent to a specified string (default \[dq]rclone/v1.72.1\[dq])
\f[R] \f[R]
.fi .fi
.SS Performance .SS Performance
@@ -32258,7 +32258,7 @@ Backend-only flags (these can be set in the config file also).
--gcs-description string Description of the remote --gcs-description string Description of the remote
--gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created --gcs-directory-markers Upload an empty object with a trailing slash when a new directory is created
--gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot) --gcs-encoding Encoding The encoding for the backend (default Slash,CrLf,InvalidUtf8,Dot)
--gcs-endpoint string Endpoint for the service --gcs-endpoint string Custom endpoint for the storage API. Leave blank to use the provider default
--gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars) --gcs-env-auth Get GCP IAM credentials from runtime (environment variables or instance meta data if no env vars)
--gcs-location string Location for the newly created buckets --gcs-location string Location for the newly created buckets
--gcs-no-check-bucket If set, don\[aq]t attempt to check the bucket exists or create it --gcs-no-check-bucket If set, don\[aq]t attempt to check the bucket exists or create it
@@ -34968,7 +34968,31 @@ The following backends have known issues that need more investigation:
\f[V]TestBisyncRemoteRemote/normalization\f[R] (https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt) \f[V]TestBisyncRemoteRemote/normalization\f[R] (https://pub.rclone.org/integration-tests/current/dropbox-cmd.bisync-TestDropbox-1.txt)
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
Updated: 2025-11-21-010037 \f[V]TestGoFile\f[R] (\f[V]gofile\f[R])
.RS 2
.IP \[bu] 2
\f[V]TestBisyncRemoteLocal/all_changed\f[R] (https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
.IP \[bu] 2
\f[V]TestBisyncRemoteLocal/backupdir\f[R] (https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
.IP \[bu] 2
\f[V]TestBisyncRemoteLocal/basic\f[R] (https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
.IP \[bu] 2
\f[V]TestBisyncRemoteLocal/changes\f[R] (https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
.IP \[bu] 2
\f[V]TestBisyncRemoteLocal/check_access\f[R] (https://pub.rclone.org/integration-tests/current/gofile-cmd.bisync-TestGoFile-1.txt)
.IP \[bu] 2
78 more (https://pub.rclone.org/integration-tests/current/)
.RE
.IP \[bu] 2
\f[V]TestPcloud\f[R] (\f[V]pcloud\f[R])
.RS 2
.IP \[bu] 2
\f[V]TestBisyncRemoteRemote/check_access\f[R] (https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
.IP \[bu] 2
\f[V]TestBisyncRemoteRemote/check_access_filters\f[R] (https://pub.rclone.org/integration-tests/current/pcloud-cmd.bisync-TestPcloud-1.txt)
.RE
.IP \[bu] 2
Updated: 2025-12-10-010012
.PP .PP
The following backends either have not been tested recently or have The following backends either have not been tested recently or have
known issues that are deemed unfixable for the time being: known issues that are deemed unfixable for the time being:
@@ -39263,12 +39287,13 @@ Petersburg
Provider: Selectel,Servercore Provider: Selectel,Servercore
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]gis-1\[dq] \[dq]ru-3\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Moscow St.
Petersburg
.IP \[bu] 2 .IP \[bu] 2
Provider: Servercore Provider: Selectel
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]ru-7\[dq] \[dq]ru-7\[dq]
@@ -39276,7 +39301,31 @@ Provider: Servercore
.IP \[bu] 2 .IP \[bu] 2
Moscow Moscow
.IP \[bu] 2 .IP \[bu] 2
Provider: Servercore Provider: Selectel,Servercore
.RE
.IP \[bu] 2
\[dq]gis-1\[dq]
.RS 2
.IP \[bu] 2
Moscow
.IP \[bu] 2
Provider: Selectel,Servercore
.RE
.IP \[bu] 2
\[dq]kz-1\[dq]
.RS 2
.IP \[bu] 2
Kazakhstan
.IP \[bu] 2
Provider: Selectel
.RE
.IP \[bu] 2
\[dq]uz-2\[dq]
.RS 2
.IP \[bu] 2
Uzbekistan
.IP \[bu] 2
Provider: Selectel
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]uz-2\[dq] \[dq]uz-2\[dq]
@@ -41371,7 +41420,25 @@ Provider: SeaweedFS
\[dq]s3.ru-1.storage.selcloud.ru\[dq] \[dq]s3.ru-1.storage.selcloud.ru\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Saint Petersburg St.
Petersburg
.IP \[bu] 2
Provider: Selectel
.RE
.IP \[bu] 2
\[dq]s3.ru-3.storage.selcloud.ru\[dq]
.RS 2
.IP \[bu] 2
St.
Petersburg
.IP \[bu] 2
Provider: Selectel
.RE
.IP \[bu] 2
\[dq]s3.ru-7.storage.selcloud.ru\[dq]
.RS 2
.IP \[bu] 2
Moscow
.IP \[bu] 2 .IP \[bu] 2
Provider: Selectel,Servercore Provider: Selectel,Servercore
.RE .RE
@@ -41381,13 +41448,29 @@ Provider: Selectel,Servercore
.IP \[bu] 2 .IP \[bu] 2
Moscow Moscow
.IP \[bu] 2 .IP \[bu] 2
Provider: Servercore Provider: Selectel,Servercore
.RE .RE
.IP \[bu] 2 .IP \[bu] 2
\[dq]s3.ru-7.storage.selcloud.ru\[dq] \[dq]s3.kz-1.storage.selcloud.ru\[dq]
.RS 2 .RS 2
.IP \[bu] 2 .IP \[bu] 2
Moscow Kazakhstan
.IP \[bu] 2
Provider: Selectel
.RE
.IP \[bu] 2
\[dq]s3.uz-2.storage.selcloud.ru\[dq]
.RS 2
.IP \[bu] 2
Uzbekistan
.IP \[bu] 2
Provider: Selectel
.RE
.IP \[bu] 2
\[dq]s3.ru-1.storage.selcloud.ru\[dq]
.RS 2
.IP \[bu] 2
Saint Petersburg
.IP \[bu] 2 .IP \[bu] 2
Provider: Servercore Provider: Servercore
.RE .RE
@@ -57445,6 +57528,9 @@ With support for high storage limits and seamless integration with
rclone, FileLu makes managing files in the cloud easy. rclone, FileLu makes managing files in the cloud easy.
Its cross-platform file backup services let you upload and back up files Its cross-platform file backup services let you upload and back up files
from any internet-connected device. from any internet-connected device.
.PP
\f[B]Note\f[R] FileLu now has a fully featured S3 backend FileLu S5, an
industry standard S3 compatible object store.
.SS Configuration .SS Configuration
.PP .PP
Here is an example of how to make a remote called \f[V]filelu\f[R]. Here is an example of how to make a remote called \f[V]filelu\f[R].
@@ -60216,9 +60302,17 @@ Type: bool
Default: false Default: false
.SS --gcs-endpoint .SS --gcs-endpoint
.PP .PP
Endpoint for the service. Custom endpoint for the storage API.
Leave blank to use the provider default.
.PP .PP
Leave blank normally. When using a custom endpoint that includes a subpath (e.g.
example.org/custom/endpoint), the subpath will be ignored during upload
operations due to a limitation in the underlying Google API Go client
library.
Download and listing operations will work correctly with the full
endpoint path.
If you require subpath support for uploads, avoid using subpaths in your
custom endpoint configuration.
.PP .PP
Properties: Properties:
.IP \[bu] 2 .IP \[bu] 2
@@ -60229,6 +60323,29 @@ Env Var: RCLONE_GCS_ENDPOINT
Type: string Type: string
.IP \[bu] 2 .IP \[bu] 2
Required: false Required: false
.IP \[bu] 2
Examples:
.RS 2
.IP \[bu] 2
\[dq]storage.example.org\[dq]
.RS 2
.IP \[bu] 2
Specify a custom endpoint
.RE
.IP \[bu] 2
\[dq]storage.example.org:4443\[dq]
.RS 2
.IP \[bu] 2
Specifying a custom endpoint with port
.RE
.IP \[bu] 2
\[dq]storage.example.org:4443/gcs/api\[dq]
.RS 2
.IP \[bu] 2
Specifying a subpath, see the note, uploads won\[aq]t use the custom
path!
.RE
.RE
.SS --gcs-encoding .SS --gcs-encoding
.PP .PP
The encoding for the backend. The encoding for the backend.
@@ -60557,7 +60674,7 @@ In the next field, \[dq]OAuth Scopes\[dq], enter
access to Google Drive specifically. access to Google Drive specifically.
You can also use You can also use
\f[V]https://www.googleapis.com/auth/drive.readonly\f[R] for read only \f[V]https://www.googleapis.com/auth/drive.readonly\f[R] for read only
access. access with \f[V]--drive-scope=drive.readonly\f[R].
.IP \[bu] 2 .IP \[bu] 2
Click \[dq]Authorise\[dq] Click \[dq]Authorise\[dq]
.SS 3. Configure rclone, assuming a new install .SS 3. Configure rclone, assuming a new install
@@ -87115,6 +87232,40 @@ Options:
.IP \[bu] 2 .IP \[bu] 2
\[dq]error\[dq]: Return an error based on option value. \[dq]error\[dq]: Return an error based on option value.
.SH Changelog .SH Changelog
.SS v1.72.1 - 2025-12-10
.PP
See commits (https://github.com/rclone/rclone/compare/v1.72.0...v1.72.1)
.IP \[bu] 2
Bug Fixes
.RS 2
.IP \[bu] 2
build: update to go1.25.5 to fix
CVE-2025-61729 (https://pkg.go.dev/vuln/GO-2025-4155)
.IP \[bu] 2
doc fixes (Duncan Smart, Nick Craig-Wood)
.IP \[bu] 2
configfile: Fix piped config support (Jonas Tingeborn)
.IP \[bu] 2
log
.RS 2
.IP \[bu] 2
Fix PID not included in JSON log output (Tingsong Xu)
.IP \[bu] 2
Fix backtrace not going to the --log-file (Nick Craig-Wood)
.RE
.RE
.IP \[bu] 2
Google Cloud Storage
.RS 2
.IP \[bu] 2
Improve endpoint parameter docs (Johannes Rothe)
.RE
.IP \[bu] 2
S3
.RS 2
.IP \[bu] 2
Add missing regions for Selectel provider (Nick Craig-Wood)
.RE
.SS v1.72.0 - 2025-11-21 .SS v1.72.0 - 2025-11-21
.PP .PP
See commits (https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0) See commits (https://github.com/rclone/rclone/compare/v1.71.0...v1.72.0)