Faster CRC32 computation

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
Eugene717
New Member
New Member
Posts: 1
Joined: 2018-12-19, 18:11 UTC

Faster CRC32 computation

Post by *Eugene717 »

Naive CRC32 routine used in TC is slow. At 200 MB/s it is a bottleneck with modern drives. Much better implementations are know (check [url=https://create dothere stephan hyphenhere brumme dothere com/crc32/]this summary (URL deactivated) [/url] for example). I tested "slicing-by-16" routine -- it works ~5x times faster. Please equip Total Commander with something like this, it really makes difference with SSDs and files in OS cache.





User came back to modify his post for to add spam links > URLs removed >> User banned

Please help us and report such posts (see the [!] button) if you find them on browsing the forum, thanks.




[MTAG]
Last edited by Eugene717 on 2019-07-22, 20:10 UTC, edited 1 time in total.
Axis
Junior Member
Junior Member
Posts: 88
Joined: 2015-07-14, 18:53 UTC

Re: Faster CRC32 computation

Post by *Axis »

Security and Performance: Blake2b (or just Blake2):

- BLAKE2b: optimized for 64-bit platforms—including NEON-enabled ARMs—and produces digests of any size between 1 and 64 bytes.
- BLAKE2 includes the 4-way parallel BLAKE2bp designed for increased performance on multicore or SIMD CPUs.
- BLAKE2 shines on 64-bit CPUs: on an Intel Core i5-6600 (Skylake microarchitecture, 3310MHz), BLAKE2b can process 1 gibibyte per second, or a speed rate of 3.08 cycles per byte.
chea
Junior Member
Junior Member
Posts: 43
Joined: 2005-07-23, 11:42 UTC

Re: Faster CRC32 computation

Post by *chea »

I also noticed that CRC32 (SFV) is a bit slow.
The TC Help says it's the fastest, but based on my little test it's slower than most.
Ranked fastest to slowest:
SHA > MD5 > SHA512 = SHA384 > CRC32 (SFV) > SHA256 > SHA3_512
I didn't test the others.

Could it be improved?
It's not really a big deal, we can always use a different checksum, but maybe if it's an easy fix...

I hope it's OK to re-post the non-spam link from OP, it seems legit:
https://create.stephan-brumme.com/crc32/
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Faster CRC32 computation

Post by *ghisler(Author) »

SHA*/MD5 use hardware accelerated functions of the operating system, so they are indeed fast on systems supporting this. To my knowledge, there is no hardware acceleration available for CRC32.
Author of Total Commander
https://www.ghisler.com
gigaman
Member
Member
Posts: 131
Joined: 2003-02-14, 11:28 UTC

Re: Faster CRC32 computation

Post by *gigaman »

Well, that's not completely true... there's the PCLMULQDQ instruction and the Intel paper:
(It was years ago when I tried to implement it based on that, and I think I concluded that there are quite a few mistakes in the examples there... but it's been a long time ago and may have been a misunderstanding on my side, of course)

Anyway, I would expect there are some working implementations e.g. here (such as crc32_by8_sse.asm or crc32_refl_by8_sse.asm maybe?)

But even the "simple" (pure x86/x64) implementation of CRC by Igor Pavlov in 7-Zip sources (public domain) is pretty fast.
Post Reply