MultiTheaded File Copy
Moderators: white, Hacker, petermad, Stefan2
MultiTheaded File Copy
Lately i've seen that Robocopy allows to sepecify the number of parallel transfered files while copying/moving. ( /MT:N, where N is the number of threads)
This Feature speeds up filecopy, especially if lots of small files are involved.
Technicaly i can't explain the reason, but it does!
Instead of copying each file from the copy queue, it takes n-items from the queue and transfers them (almost) at once.
This Feature would be a real gain and makes tc even better for big copy Tasks than it allready is .
Regards,
Metin
This Feature speeds up filecopy, especially if lots of small files are involved.
Technicaly i can't explain the reason, but it does!
Instead of copying each file from the copy queue, it takes n-items from the queue and transfers them (almost) at once.
This Feature would be a real gain and makes tc even better for big copy Tasks than it allready is .
Regards,
Metin
- ghisler(Author)
- Site Admin
- Posts: 48199
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
That doesn't make much sense to me, it would cause a lot of jumping of the harddisk heads between the different files. Or maybe you use an SSD (solid state disk)?
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: MultiTheaded File Copy
Hi
It's a very old topic.
But multi-threaded copying is VERY useful as you're copying something from a corporate NAS over a VPN (Mapped drive). Such a NAS has a lot of performance, but due to the delays caused by the Internet and VPN when copying small files, the transfer is very very low.
The second thing SMB 3.0 has support for multithreading and it also helps when the NAS is connected via 2 x 1Gbps, and our computer 2.5Gbps. Then it is better to have more than one copying thread. But that's probably a separate topic. There are just so many uses when it will make things better (for example QoS on router).
This could work similarly to F2 Queue when copying e.g. F3 Multithread.
It's a very old topic.
But multi-threaded copying is VERY useful as you're copying something from a corporate NAS over a VPN (Mapped drive). Such a NAS has a lot of performance, but due to the delays caused by the Internet and VPN when copying small files, the transfer is very very low.
The second thing SMB 3.0 has support for multithreading and it also helps when the NAS is connected via 2 x 1Gbps, and our computer 2.5Gbps. Then it is better to have more than one copying thread. But that's probably a separate topic. There are just so many uses when it will make things better (for example QoS on router).
This could work similarly to F2 Queue when copying e.g. F3 Multithread.
- ghisler(Author)
- Site Admin
- Posts: 48199
- Joined: 2003-02-04, 09:46 UTC
- Location: Switzerland
- Contact:
Re: MultiTheaded File Copy
I'm already getting complaints from time to time that copying with TC saturates the connection and slows down the server. That would be even worse with parallel copying.But multi-threaded copying is VERY useful as you're copying something from a corporate NAS over a VPN (Mapped drive).
For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.
Author of Total Commander
https://www.ghisler.com
https://www.ghisler.com
Re: MultiTheaded File Copy
Perhaps consider a setting, similar to SamePartitions, and introduce AllowMultiThreadedDisks or something like that - when set, allow for an option to be chosen in the copy/move dialog e.g. Use MultiThreading (by default it remains off, but you can choose it for certain actions)ghisler(Author) wrote: ↑2023-04-24, 14:23 UTCFor SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.
F4MiniMenu (Forum) - Open selected file(s) from TC in defined editor(s) - A (minimalistic) clone of F4Menu
Source at GitHub (AutoHotkey). TCSyncComments (copy file comments)
Source at GitHub (AutoHotkey). TCSyncComments (copy file comments)
Re: MultiTheaded File Copy
Interesting complains Anyway, it could just be an option, disabled by default. It is important that it can work on network resources (without drive leather) and mapped drives. Now at work I got a proposal to use some scripts (Robocopy) for this, and I'd rather just use Total Commander which we paid for.I'm already getting complaints from time to time that copying with TC saturates the connection and slows down the server. That would be even worse with parallel copying.
I guess for an SSD it's not a big deal. But if you're mainly working on NAS, with a lag between you and the NAS, it's very annoying to work with small files. Even deleting is slow (this could also be done multi-threaded).For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.
Re: MultiTheaded File Copy
I support this idea. Make it optional and let users decide.
Re: MultiTheaded File Copy
Any news? Is there a chance for this option?
Re: MultiTheaded File Copy
As a workaround, you can create TC buttons to use FastCopy.
It can run jobs in parallel and also has some settings to control how it's done.
It can run jobs in parallel and also has some settings to control how it's done.
Windows 11 Home x64 Version 23H2 (OS Build 22631.3672)
TC 11.03 x64 / x86
Everything 1.5.0.1378a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
TC 11.03 x64 / x86
Everything 1.5.0.1378a (x64), Everything Toolbar 1.3.3, Listary Pro 6.3.0.78
QAP 11.6.3.3 x64
Re: MultiTheaded File Copy
Source: Windows VPN and enterprise cached NAS
Target Local Intel M.2 SSD
Max download speed ~300Mbps
6088 files - 19,1MB (small files)
Total Commander: 39:27 (almost 40 minutes)
FastCopy (configured parallel, buffer 32MB): 07:11
Robocopy /E /MT:32: 00:29
Robocopy /E /MT:64: 00:24
Yes, Robocopy with 64 threads is almost 100 times faster than standard Total Commander copying. That's why I think it's a very useful thing.
Target Local Intel M.2 SSD
Max download speed ~300Mbps
6088 files - 19,1MB (small files)
Total Commander: 39:27 (almost 40 minutes)
FastCopy (configured parallel, buffer 32MB): 07:11
Robocopy /E /MT:32: 00:29
Robocopy /E /MT:64: 00:24
Yes, Robocopy with 64 threads is almost 100 times faster than standard Total Commander copying. That's why I think it's a very useful thing.
Re: MultiTheaded File Copy
Now that SSD are widespread, I think it's a good option to have.
Also, it would be very useful on cloud drives such as Google Drive to use the whole bandwidth.
Also, it would be very useful on cloud drives such as Google Drive to use the whole bandwidth.
There are ways to figure it out, but it could simply be an option you can trigger on and off during copy (and it remembers the last choice based based on the source and destination drives for instance).ghisler(Author) wrote: ↑2023-04-24, 14:23 UTC For SSDs it would be nice, but there doesn't seem to be an easy way to detect whether a drive is an SSD or HDD.