Synchronize directories - async compare with google drive
Posted: 2020-07-26, 06:52 UTC
Hello,
Currently, the "synchronize directories" functionality performs a file-by-file comparison. If I'm comparing the content of a local drive with a "cloud" drive such a google drive, it means that files are accessed one by one remotely, which causes the files to be downloaded one by one, which slows down by a 10x factor of more if there are plenty of small files.
It would be better to compare multiple files at the same time in that case. I'm aware that it may be slower when synchronizing two local directories, but this could be detected if there is a difference of throughput between the two directories.
Another solution would be to read the first 1KB of each file in parallel to trigger any download, then proceed sequentially.
Currently, the "synchronize directories" functionality performs a file-by-file comparison. If I'm comparing the content of a local drive with a "cloud" drive such a google drive, it means that files are accessed one by one remotely, which causes the files to be downloaded one by one, which slows down by a 10x factor of more if there are plenty of small files.
It would be better to compare multiple files at the same time in that case. I'm aware that it may be slower when synchronizing two local directories, but this could be detected if there is a difference of throughput between the two directories.
Another solution would be to read the first 1KB of each file in parallel to trigger any download, then proceed sequentially.