jgaunt/rclone
1
0
Fork 0
mirror of https://github.com/rclone/rclone.git synced 2025-12-06 00:03:32 +00:00
Code Issues Projects Releases Wiki Activity

sync: implement --list-cutoff to allow on disk sorting for reduced memory use

Browse Source 
Before this change, rclone had to load an entire directory into RAM in
order to sort it so it could be synced.

With directories with millions of entries, this used too much memory.

This fixes the probem by using an on disk sort when there are more
than --list-cutoff entries in a directory.

Fixes #7974
This commit is contained in:
Nick Craig-Wood
2024-12-09 11:30:34 +00:00
parent 0148bd4668
commit 385465bfa9
9 changed files with 493 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files

1
go.mod
View File

@@ -47,6 +47,7 @@ require (
github.com/klauspost/compress v1.18.0
github.com/koofr/go-httpclient v0.0.0-20240520111329-e20f8f203988
github.com/koofr/go-koofrclient v0.0.0-20221207135200-cbd7fc9ad6a6
github.com/lanrat/extsort v1.0.2
github.com/mattn/go-colorable v0.1.14
github.com/mattn/go-runewidth v0.0.16
github.com/minio/minio-go/v7 v7.0.87