Archive crc checksum: from device or RAM?

[Umpal]

Could not find similar request/topic so here is my suggestion (unless there already is a solution for it then I'll be grateful for telling me about it).

Situation:
You create an archive and a crc checksum for it. The archive is not big, let's say 70MB. Next you copy that file and crc sfv along with it on a USB stick. Then you want to check if the file is copied properly because your memory USB stick sometimes writes files incorrectly*.

Observation:
In the situation described above Total Commander reads the file NOT from the USB stick but from RAM BUFFER. Is there a way to make the TC check the crc ALWAYS from the device or it would have to be implemented? Once again in a situation when you just copied the file, because when the file is erased from the buffer then TC is forced to read it from the device, I know that perfectly.

Cheers,
Peter

*I have one 16GB Transcend fantastic USB stick but sometimes I experience crc error. And no, I don't wanna get rid of it, I love it

[MVV]

AFAIK TC opens files with special flag causing unbuffered reading from file.

[Umpal]

AFAIK TC opens files with special flag causing unbuffered reading from file.

Is it accessible to user in settings or you're talking about a built-in feature?

[milo1012]

Total Commander reads the file NOT from the USB stick but from RAM BUFFER

Not TC, but Windows, because TC uses WinAPI functions for reading files.

But I know the problem.
Especially since Vista Windows file caching can sometimes be a pain in the a** (Superfetch), exactly due to situations like yours.
You can force to clear the Windows file cache manually.
The easiest solution I found so far is to use RAMMap,
and use
Empty -> Empty Standby List

The only command line solution I found so far can be found here.
After you cleared the cache all files need to be read from disk/stick again.

Besides that I would also endorse a native TC solution to bypass caching.

[MVV]

Umpal,
Well, I checked with Process Monitor, TC opens with buffering in case of regular CRC check. It only opens with no buffering in case of copy with verification (so you can try it, it is available just from copy dialog).

[milo1012]

It only opens with no buffering in case of copy with verification

Sure, it checks how the file compares immediately after copy operation.
But what if I want to do this minutes/days/weeks later with CBC or sync tool?
Especially for "unstable" media, which can sometimes read correctly and sometimes not at all, it is a nuisance.
Recently I wanted to verify some DVD images, but Windows already cached a lot of files due to autorun or previous TC operations on the disc,
and so I ended up needing to clear the cache manually.

So I highly support a bypass-cache feature for all other compare operations in TC.

[MVV]

You won't need a cache after days or weeks. But in a minutes data may be read correctly even if you re-plug your stick. I notice damages on my stick after some time, not in a minutes after writing.
Anyway, it would be better to fix corruption and not only detect it, so I would prefer some recovery data. I know some tools, e.g. ICEECC or RSC32, first one can't be integrated into TC, second one may be integrated via Multiarc/buttonbars but I don't know how reliable it is.

[milo1012]

I notice damages on my stick after some time, not in a minutes after writing

And your experience excels mine, because...?
I had quite a share of experience with "unstable" CD/DVD media, and yes,
also some SD cards and USB sticks which were completely unreliable for different reasons.
A minute later you can read them, another minute later you can't, and so on.

I know some tools, e.g. ICEECC or RSC32

I don't think these count as common programs, they are either quite new or not really widespread.
There is Parchive/PAR2 since years, and for optical drives you can use Dvdisaster quite comfortably.
I prefer RAR's Recovery Record, since it's basically the same deal (Reed-Solomon code) and it works the same way.
You can also spare extra files that way, because everything can reside in one file and we don't need any TC integration (besides maybe a command switch).
Here is a (Russian) site with a list of similar software.

But to get back to topic: it wouldn't hurt to use FILE_FLAG_NO_BUFFERING optionally for CBC and similar.

[MVV]

I agree that TC should use FILE_FLAG_NO_BUFFERING for all checksum checking procedures.

Problem of RAR is that you have to keep your data in an archive, and it is not a freeware. So any program that keeps recovery records in separate files is much better. Also I don't really want to open some GUIs and select files/folders from it, I want to work with selected files in TC. So integration with TC and reliability are the most important aspects for me as for TC user:) Probably I'll try MultiPar because it is not too outdated, also it is tiny and supports command line.

[Hacker]

Probably I'll try MultiPar because [...]

... and because RSC32 is nowhere to be found.

Roman

[Ovg]

... and because RSC32 is nowhere to be found.

Hm... https://www.livebusinesschat.com/smf/index.php?topic=5511.0

[Hacker]

Ovg,
Ah, OK, thank you.

Roman

[Umpal]

I see I'm not the only one who'd like to see this feature changed (even as an option) in the future version. I hope Christian reads it.

@milo1012, thanks for the tip, seems some kind of a solution but... I'd prefer having it from TC itself.

@MVV, I feel a bit stupid I completely missed such an easy and obvious option. I'll try to remember next time . Thanks!