Page 1 of 1

Specify number of threads when copying (and moving?) files

Posted: 2012-02-07, 14:07 UTC
by alexjak
Hi all,

I often have to copy files to/from network share. I don't know how the share is implemented in my case, but from my experience, when copying a file, after initial big delay the copying speeds are good. It means that copying a couple of large files is OK, but copying a large number of small files becomes too long.

Archiving them all and then copying an archive is not an option to me. So what I do now is splitting them into groups and copying these groups in parallel. This helps, but it is a lot of manual work - very time consuming and error prone.

What I would like is an option in copying dialog to specify how many threads to use for copying (default 1). TC would then start copying all selected files in parallel using that number of threads. I guess that similar option would be useful for moving files too.

As having multiple threads for copying would be useful in some cases and not in others, I think that this parameter should be configurable on per copying basis, not an option in settings.

Thanks a lot,

Alex

Posted: 2012-02-07, 14:49 UTC
by umbra
Have you tried the Background Transfer Manager feature? You might use several instances - one instance as one thread.

Posted: 2012-02-07, 18:23 UTC
by alexjak
umbra wrote:Have you tried the Background Transfer Manager feature? You might use several instances - one instance as one thread.
That's what I am doing at the moment - but it is very inconvenient.

Posted: 2012-02-07, 21:04 UTC
by umbra
Yes, you have to split operations into several BTMs manually but on the other hand it gives you great control and a chance to optimize those operations according to your preferences.

What you are asking for is good only for some network shares. But all other devices (hdd, usb, memory cards, ...) are very inefficient in parallel access. And having such option in copy dialog would be usually useless and what's worse some users would activate it despite the fact, it would slow everything down.

However if the option would be visible only if source/destination was a network share ...

Posted: 2012-02-07, 21:18 UTC
by sqa_wizard
2alexjak: Using several threads for copying small files does not speed up anything.
All threads are usually accessing the same harddisk!
This means each thread has to wait until the read/write job of other thread has been finished => Those threads are processed in sequence, not in parallel.

File copy means:
1. info about the file location on the harddisk.
2. move head to this position
3. read the data

In case of a small file copy, the major time is used for step 1. an 2.
These are all harddisk related tasks.
There is no advantage by using a faster CPU or multiple threads.

Posted: 2012-02-07, 23:09 UTC
by alexjak
sqa_wizard wrote:2alexjak: Using several threads for copying small files does not speed up anything.
All threads are usually accessing the same harddisk!
....
There is no advantage by using a faster CPU or multiple threads.
I don't want to argue with you, maybe for writing directly to HDD this is the case. However, I am not writing directly to HDD, I am writing files to a network share in a big organization, I have no idea where this share is physically located and what machine is behind it. In my case, splitting file copying into several threads speeds things up significantly - this is a fact tested by experiment.