Ensuring file integrity ( CRC MD5 ECSV ... )

[PL]

Ensuring file integrity has become more of a problem since more and more files are transmitted by Internet. Third party programs like the firewall Zonealarm interfered with transfers, resulting in many corrupted files. This problems are cause for more rigorous and time efficient checking and possible self correcting programs/implementations.

I applaud the feature of creating and checking sfv's from within TC.

For large ~700MB files I would like to have a format that allows me to store crc32's of at most 5MB split chunks. This would be more secure and efficient than a single md5 for the whole file, because you would be able to repair the file rather than to re download it.

Be able to create a extended SFV or ECSV for dir and their sub dirs would also save time rather than to use sync. compare.


I hope someone understood me ; )

[JackFoo]

Hmm, you can already split the file into chunks of whatever size, then you can md5 them... Besides most files you download either come in parts with an sfv file containing CRC of every part or as a big file(without sfv), so I can't really understand, what's you request??

Cheers

[PL]

Hmm, you can already split the file into chunks of whatever size, then you can md5 them... Besides most files you download either come in parts with an sfv file containing CRC of every part or as a big file(without sfv), so I can't really understand, what's you request??

Cheers

Yes, this is were I got the idea from, but splitting would take to long for any "normal" user to except or be able to handle.

Here you only need to read, not read and write the sequential 5MB of the file and calculate the crc32 for each chunk independently.

For manual repair I would only need to know were the file is corrupt, in chunks of 5MB of course. For the repair to work the person with the org file only need to send me the "multiple crc text file" for this particular file, easily calculated from TC or from a bundled std alone program with TC.


With such a format you could synchronize big files over the Internet or over a p2p lan such as DC and be sure that its 100% ok, this is targeted for the avi's that people don't store in rar format for obvious reasons.


Of course this might not be the right forum for a new format that I haven't found on the internet, yet ... but I give it try, anyway Total Commander should get more useful feature's I think.

[bago]

With such a format you could synchronize big files over the Internet or over a p2p lan such as DC and be sure that its 100% ok, this is targeted for the avi's that people don't store in rar format for obvious reasons.

This is already handled by p2p client of the eDonkey network (eMule for example)

[PL]

With such a format you could synchronize big files over the Internet or over a p2p lan such as DC and be sure that its 100% ok, this is targeted for the avi's that people don't store in rar format for obvious reasons.

This is already handled by p2p client of the eDonkey network (eMule for example)

Could you tell me of a stand alone program that does this, the p2p network would be a later implementation for those networks that don't have this feature already. This feature are for big file transfers in general. I don't think you like to use eDonkey to transfer files among friends. And before you replay that, well you can split the files make a sfv blah blah join them ... I tried to explain how its done but Its way to much hazel even for the advanced lazy user.

[bago]

I don't think you like to use eDonkey to transfer files among friends.

I do that. Why not?
Why should I loose time doing split/checksums and more when there is an utility to transfer file in this way? (emule)

You can have "non-public p2p networks" too.

[PL]

I don't think you like to use eDonkey to transfer files among friends.

I do that. Why not?
Why should I loose time doing split/checksums and more when there is an utility to transfer file in this way? (emule)

You can have "non-public p2p networks" too.

Im not sure you understood this feature, its way more then a transfer check, its a orginal file verification repair format, how ? by the ability to create public "multiple crc sfv textfiles" like the official sfv releases.

[bago]

Im not sure you understood this feature, its way more then a transfer check, its a orginal file verification repair format, how ? by the ability to create public "multiple crc sfv textfiles" like the official sfv releases.

Probably I'm missing something....

Why do you need crc for each chunk? I need it only for transfer purpose. To check "original file verification" I only need a single CRC, no?

[Hacker]

bago,
PL wants to know where the file is corrupt, so he only has to send one chunk.

PL,
How about WinRAR's ability to create recovery volumes? Each and any one corrupt volume can be repaired using one recovery volume.

HTH
Roman

[bago]

bago,
PL wants to know where the file is corrupt, so he only has to send one chunk.

I understood this. I said that I also have his problem and I started using software that does chuck checksums and automatically resend wrong chunks (eMule for example).

Mine is a suggestion: don't do manually what free software already does for you!

[Hacker]

Well yeah, but both sides have to have eMule installed and running during the file transfer, right? If yes, then that's no solution for email or such.

Roman

[bago]

Well yeah, but both sides have to have eMule installed and running during the file transfer, right? If yes, then that's no solution for email or such.

Roman

Yes, that's right.

Is not so common to transfer big files via email because of the 7bit encoding: they're talking that they don't want to split/rar the file in smaller ones (in order to have crc for each chunk) because of overhead of compression/decompression of the chunks that on big files would take time.

That's why I thought it was a synchronous transfer and not email.

[MxxCon]

somewhat similar to winrar's recovery volumes there are .PAR files that are very common on newsgroups.
check out http://www.fluidstudios.com/fsraid.html
this program can make .PAR files to recover corrupted ones..

[Hacker]

The RAR recovery volumes are based on the PAR thingies, AFAIK.

Roman

[Spaghettificus]

Ensuring file integrity has become more of a problem since more and more files are transmitted by Internet. Third party programs like the firewall Zonealarm interfered with transfers, resulting in many corrupted files. This problems are cause for more rigorous and time efficient checking and possible self correcting programs/implementations.

I applaud the feature of creating and checking sfv's from within TC.

THIS IS A TREMEMDOUSLY IMPORTANT FEATURE. Let's not forget Overclocking, beta radiation and bad IDE cables!

I would love any new code for TC which would help SAMBA and FTP transfers.

Right now, an optimal solutiion is to use the RAR format with the parity and recovery options. The great thing about RAR, is you can specify as much parity and recovery info as you need! Just enter a percent! It's easy to use, but it's not nearly as transparent as I want it to be.