Multicore sfv/sha creation

hlloyge · Post by *hlloyge » 2013-08-05, 13:44 UTC

Right now, when I select files and check "create sfv", the files are read one after another, checksum is calculated, and written in file.
It would be very nice if we could use our multi-core cpus and fast HDDs/SSDs so we could create checksums in parallel.
No harm done - it works with some other software, and is considerably faster if there is lots of files in folder.

MVV · Post by *MVV » 2013-08-05, 15:52 UTC

Currently CPUs are much faster than HDDs so hash calculation loads more HDD than CPU so using separate cores won't speed hashing up because of busy HDD.

Sob · Post by *Sob » 2013-08-05, 20:11 UTC

I wouldn't be so sure about it. Many people have SSDs and those are fast. And even without SSD, TC's hashing speeds are not very fast (see my test from last year; I did a quick check now and it seems it did not change even with last TC version). Some of those speeds are well under 100MB/s, it's not big problem even for HDD. But parallel reading would not help much (with HDD), because seeking would most likely kill the performance (possibly solvable by some buffering, but relatively hard). The best first step would be improving hashing speeds. Either by some better implementaion in TC or by using system functions where available. MD5 improved 2.5 times or SHA-1 improved 4 times would make quite a difference.

hlloyge · Post by *hlloyge » 2013-08-06, 19:10 UTC

Well, to see what I am talking about, check out this program, http://www.exactfile.com/, and try to make sfv of, I don't know, 100 mp3 files. Or FLAC files.