Total Commander

Naive CRC32 routine used in TC is slow. At 200 MB/s it is a bottleneck with modern drives. Much better implementations are know (check [url=https://create dothere stephan hyphenhere brumme dothere com/crc32/]this summary (URL deactivated) [/url] for example). I tested "slicing-by-16" routine -- it works ~5x times faster. Please equip Total Commander with something like this, it really makes difference with SSDs and files in OS cache.





User came back to modify his post for to add spam links > URLs removed >> User banned

Please help us and report such posts (see the [!] button) if you find them on browsing the forum, thanks.




[MTAG]

Security and Performance: Blake2b (or just Blake2):

- BLAKE2b: optimized for 64-bit platforms—including NEON-enabled ARMs—and produces digests of any size between 1 and 64 bytes.
- BLAKE2 includes the 4-way parallel BLAKE2bp designed for increased performance on multicore or SIMD CPUs.
- BLAKE2 shines on 64-bit CPUs: on an Intel Core i5-6600 (Skylake microarchitecture, 3310MHz), BLAKE2b can process 1 gibibyte per second, or a speed rate of 3.08 cycles per byte.

I also noticed that CRC32 (SFV) is a bit slow.
The TC Help says it's the fastest, but based on my little test it's slower than most.
Ranked fastest to slowest:
SHA > MD5 > SHA512 = SHA384 > CRC32 (SFV) > SHA256 > SHA3_512
I didn't test the others.

Could it be improved?
It's not really a big deal, we can always use a different checksum, but maybe if it's an easy fix...

I hope it's OK to re-post the non-spam link from OP, it seems legit:
https://create.stephan-brumme.com/crc32/

SHA*/MD5 use hardware accelerated functions of the operating system, so they are indeed fast on systems supporting this. To my knowledge, there is no hardware acceleration available for CRC32.

Well, that's not completely true... there's the PCLMULQDQ instruction and the Intel paper:
(It was years ago when I tried to implement it based on that, and I think I concluded that there are quite a few mistakes in the examples there... but it's been a long time ago and may have been a misunderstanding on my side, of course)

Anyway, I would expect there are some working implementations e.g. here (such as crc32_by8_sse.asm or crc32_refl_by8_sse.asm maybe?)

But even the "simple" (pure x86/x64) implementation of CRC by Igor Pavlov in 7-Zip sources (public domain) is pretty fast.