Copy and verify millions of small files

[Sh1ndy]

Hello all,

First of all: I'm new to Total Commander and find it a very very useful tool! Thanks for this great software

Currently I try to copy a few million of small files over a 10G network from one server to another. The size per each file varies between 20 and 60KB.
Overall size of all files is ca. 1.8TB.

I'm copying with the "validate" flag on to make sure the files are copied as they should be. But: The whole copy process is quite slow. After running for 3 hours it only copied 350.000 files/10GB.

Is there anything I can do to increase the speed? The hardware on both sides is not running on its limits. CPU, RAM, network is a only a bit more used compared to the daily workload without copying.

Regards
Sh1ndy

[Horst.Epp]

Remove the validate flag.
You can later create checksum files on one server and run the check on the other server.

[ghisler(Author)]

I would pack them into a ZIP file and send this as a whole, then unpack it locally on the remote machine.

This is much faster and the files are protected by checksums.