Verify after copy

[sqgl]

TC already has the synchronise function. Why do you suppose that is?

Because an edited file can have same size but different content and timestamps you can't trust on?

Ok, true, I chose a bad example. I do however often use it for copying important files to USB drives. I do get errors sometimes. Rarely, but enough to bother. In fact I more often get (false) errors reported in the verification process. Should only happen roughly half the time!?

Sometimes it seems to not compare to the original but to what is in RAM/Cache (which actually makes it not very reliable)... but that is another thread.

[knnknn]

After my last post I said to myself that I will make a screencap the next time an undetected CRC error happens.

So here it is: An example of a completely CORRUPTED COPY that happened to me recently.

As you can see this is not about 1 single byte that has been copied wrongly (yes, such copy errors exist, too).

And it's not about a single block (say 1k of bytes) that has been copied wrongly (yes, such copy errors exist, too).

It's about complete corruption = unusable file.

And no, there was no copy error. Only the later CRC verification showed that something went wrong.

Image: http://img375.imageshack.us/img375/1921/crcerrors.gif

And by the way: I found out the reason. And it had nothing to do with broken hard drives or broken cables.

[sqgl]

Thanks knnknn, now maybe the others will believe us.

[Hacker]

knnknn,

nd it had nothing to do with broken hard drives or broken cables.

Well, will you share?

Roman

[Henrie]

He made me curious too, for what the cause was.

But i guess he did not share because it is not relevant for his request. Which would than become on a sidetrack again because everyone would talk about the cause of the problems and not about the possibility that no matter what the cause, it could be usefull to have files checked after copy.

[knnknn]

He made me curious too, for what the cause was.

But i guess he did not share because it is not relevant for his request. Which would than become on a sidetrack again because everyone would talk about the cause of the problems and not about the possibility that no matter what the cause, it could be usefull to have files checked after copy.

Good point.

But since you got curious I tell you the cause:

The cause was an aging Power Supply. I had several other hard drives connected to it and this hard drive was simply 1 too much:
The hard drive constantly went on and off. On and off. But it was too quiet to hear it from further away.

Obviously it wrote wrong data but didn't deliver any CRC errors since it went off and on.

When you have aging cables or shaky connectors or broken network HUBs then single bytes can get messed up or single blocks (e.g. 1024 bytes or so). And usually there are no CRC errors when you copy from (S)ATA-harddisk to (S)ATA-harddisk WITHIN THE SAME PC.
But this was the first time that a whole file became COMPLETELY unusable. It was ESPECIALLY strange since it was within the same PC.

Not only that there were reports of copy errors, but FileDate and FileSize also were correct.

As soon as I replaced the power supply everything worked fine.

[SQUIRE]

Firstly, if you have a flaky power supply then all bets are off, irrespective of whether or not there is a verify-after-copy facility. All subsystems in the PC cannot now be relied upon to function correctly. That fault could quite easily (and more likely in my opinion) have affected the PC's RAM causing bad data to be transferred to the HDD. You would not know unless you were using ECC RAM and even then multiple bit errors can cancel out. Doing a read-after-write back to the same intermittently faulty RAM might show no errors. So how would a verify function protect you?

Secondly, was your screen capture taken on the same PC before you discovered the failing PSU? In that event even the snapshot you see could as easily be invalid for the reason given above. In my experience, normal day to day error detection does a perfectly adequate job of flagging faulty cables and peripherals without going into extreme paranoia mode. If you need near 100% guaranteed error-free results then you really ought to be using a hardware solution with redundant subsystems. A software verify function can only provide a false sense of security.

[knnknn]

Firstly, if you have a flaky power supply then all bets are off, irrespective of whether or not there is a verify-after-copy facility.
All subsystems in the PC cannot now be relied upon to function correctly.

All subsystems WORKED CORRECTLY except the power to this 1 hard drive.

That fault could quite easily (and more likely in my opinion) have affected the PC's RAM causing bad data to be transferred to the HDD.

Whatever the reason is: CRC verification caught it.

You would not know unless you were using ECC RAM and even then multiple bit errors can cancel out. Doing a read-after-write back to the same intermittently faulty RAM might show no errors. So how would a verify function protect you?

I DON'T CARE WHAT CAUSED IT.

That is not the point.

The point is that you MOVE GIGABYTES OF DATA which arrive CORRUPTED.

CRC checks catch it. They ALARM YOU. They make it possible to REPLACE malfunctioning hardware. NOT LOSING YOUR DATA is the target.

Secondly, was your screen capture taken on the same PC before you discovered the failing PSU? In that event even the snapshot you see could as easily be invalid for the reason given above.

You make a logical fallacy here:

Whether the

* COPIED file was CORRUPTLY WRITTEN
* or the copied file was not corrupt (merely CORRUPLTY READ)
* or the ORIGINAL file was CORRUPTLY READ
* or the ORIGINAL file was CORRUPTLY TRANSFERRED

is IRRELEVANT for catching the hardware culprit. As soon as CRC errors occur you know that something is wrong with the hardware: Cables, sockets, hubs etc.

Without CRC checks you wouldn't even notice. Maybe even for years, if you use a faulty LAN hub.

And no, I checked the file several times after I replaced the faulty hardware: With the same results: Complete corruption.

I can't believe that after 60+ posts I still have to point out OBVIOUS FACTS: Copy errors _DO_ HAPPEN. Whether you like it or not. Due to hardware or software or whatever.

Get over it.

And I can't believe that I paid for Total Commander (whose MOST SIGNIFICANT PURPOSE is to copy files) and yet I have to rely on third party solutions to make sure that my files are copied correctly.

TC has functions which hardly anyone uses like "Overwrite if larger" but lacks a real safety function like a simple checkbox "[x] CRC check copied files".

In my experience, normal day to day error detection does a perfectly adequate job of flagging faulty cables and peripherals without going into extreme paranoia mode.

So you admit that you don't do CRC checks. In other words: You have no clue how many of the files you have ever copied arrived 100% intact, yet you make bold statements about error detections.

Look, you can choose to not verify your files as much as you like. I don't care about your files. But speaking for myself: I want a CRC option.

If you need near 100% guaranteed error-free results then you really ought to be using a hardware solution with redundant subsystems. A software verify function can only provide a false sense of security.

Unbelievable. I gave already examples how CRC checks CAUGHT SIGNIFICANT data corruption and I have to read such excuses that "it only provides a false sense of security".

You are basically claiming "Inspections of airplanes provide a false sense of security". Yeah, but what you conveniently failed to mention is that inspections indeed can prevent crashes.

[SQUIRE]

There, there, calm down and blow your nose. There's no need for a temper tantrum and a screaming fit over a function that you have a personal grudge against but which most everyone else seems perfectly content with, is there? Why, anyone would think that you're some kind of obsessive, compulsive nerd with a short fuse and an intolerant attitude to comment which isn't agreeable (which I'm sure you're not, of course).

All subsystems WORKED CORRECTLY except the power to this 1 hard drive.

Just out of curiosity how could you be so certain of this when your faulty PSU is the power source for every single component in that machine?
Since your PSU has been slowly dying over a long period - would you bet your life that NO errors have slipped past your 3rd party validation precaution in spite of your best efforts?

Whatever the reason is: CRC verification caught it.

You're not paying attention. If your RAM is unstable then so too is any software CRC verification routine you relied upon.
The only way you can say this with confidence is if you removed both source and target drives to a known good system and tested them there separately and rigorously. Otherwise, if you used a failing system to run your verification software on, its conclusion cannot be relied upon as I hope you'll agree. The result could have just as easily gone the other way leaving you in blissful ignorance, right?

CRC checks catch it. They ALARM YOU. They make it possible to REPLACE malfunctioning hardware. NOT LOSING YOUR DATA is the target.

Um, you appear to have a touching faith in the power of CRCs to save you from harm. The fact is CRCs are only good at detecting error conditions when the number of bit errors is relatively low. They are certainly not a 100% foolproof guarantee of data integrity. That being so, a line has to be drawn somewhere between reasonable confidence that data has been transferred uncorrupted and military grade levels of zero tolerance. Not slowing down prosumer systems with paranoid software verification for every little move that's made is a reasonable compromise position to take I suspect.

You need only disable your state-of-the-art anti-virus program that pokes into every event it deems 'suspicious' to notice the dramatic slowdown that busybody software can have on a computer. Add that to an always-on verification tool and you have a recipe for a system that runs like molasses on a cold day. You might be fine with that but I'm sure an awful lot of people would beg to disagree.

You are basically claiming "Inspections of airplanes provide a false sense of security". Yeah, but what you conveniently failed to mention is that inspections indeed can prevent crashes.

That's an interesting stretch to make from CRC validation to aircraft inspections and rather absurd if I may say so. Since I work with aircraft systems that's hardly likely. Do take note that modern commercial airplanes run triply-redundant HARDWARE control systems to protect against failure within common-sense cost-benefit limits.

And I can't believe that I paid for Total Commander (whose MOST SIGNIFICANT PURPOSE is to copy files) and yet I have to rely on third party solutions to make sure that my files are copied correctly.

If you check the forum you might notice data corruption to be the least of TC's problems. Indeed, do you think it would have survived and gone from strength to strength for this long if users found their precious data being habitually trashed? Does that tell you something about the importance of your particular fetish? Sure it's entirely possible you might get the odd rare mishap but as you so eloquently put it: Get over it. It simply doesn't happen to the overwhelming majority and if it causes you a lot of grief then I would suggest the problem lies at your end.

Maybe you ought to ask for a refund from Ghisler since you're so obviously a dissatisfied customer, eh...?

[mod]SQUIRE,
This is crossing the line. Arguing about a function's sense is OK, calling users temper tantrum throwing obsessive compulsive nerds with a short fuse is not.

Hacker (Moderator)[/mod]

[knnknn]

All subsystems WORKED CORRECTLY except the power to this 1 hard drive.

Just out of curiosity how could you be so certain of this when your faulty PSU is the power source for every single component in that machine?
Since your PSU has been slowly dying over a long period - would you bet your life that NO errors have slipped past your 3rd party validation precaution in spite of your best efforts?

The only way you can say this with confidence is if you removed both source and target drives to a known good system and tested them there separately and rigorously. Otherwise, if you used a failing system to run your verification software on, its conclusion cannot be relied upon as I hope you'll agree. The result could have just as easily gone the other way leaving you in blissful ignorance, right?

CRC checks catch it. They ALARM YOU. They make it possible to REPLACE malfunctioning hardware. NOT LOSING YOUR DATA is the target.

Um, you appear to have a touching faith in the power of CRCs to save you from harm. The fact is CRCs are only good at detecting error conditions when the number of bit errors is relatively low. They are certainly not a 100% foolproof guarantee of data integrity. That being so, a line has to be drawn somewhere between reasonable confidence that data has been transferred uncorrupted and military grade levels of zero tolerance. Not slowing down prosumer systems with paranoid software verification for every little move that's made is a reasonable compromise position to take I suspect.

You need only disable your state-of-the-art anti-virus program that pokes into every event it deems 'suspicious' to notice the dramatic slowdown that busybody software can have on a computer. Add that to an always-on verification tool and you have a recipe for a system that runs like molasses on a cold day. You might be fine with that but I'm sure an awful lot of people would beg to disagree.

And I can't believe that I paid for Total Commander (whose MOST SIGNIFICANT PURPOSE is to copy files) and yet I have to rely on third party solutions to make sure that my files are copied correctly.

If you check the forum you might notice data corruption to be the least of TC's problems. Indeed, do you think it would have survived and gone from strength to strength for this long if users found their precious data being habitually trashed? Does that tell you something about the importance of your particular fetish? Sure it's entirely possible you might get the odd rare mishap but as you so eloquently put it: Get over it. It simply doesn't happen to the overwhelming majority and if it causes you a lot of grief then I would suggest the problem lies at your end.

Maybe you ought to ask for a refund from Ghisler since you're so obviously a dissatisfied customer, eh...?

Your hole post repeats the same old excuses: "We can never be sure", "I don't want CRC checking", "Nobody ever notices any CRC errors anyway".

The only difference is that you spice it up with personal insults, e.g. calling the need for CRC "my fetish".

You still try to argue against real world experience with faulty data transfers that _DO_ happen and _CAN_ be caught via CRC checking.

You chose to ignore reality. Lame.

Whatever the reason is: CRC verification caught it.

You're not paying attention. If your RAM is unstable then so too is any software CRC verification routine you relied upon.

Oh, wow. Now you added a twist that will surely convince me: "Faulty RAM can make corrupted files calculate as 100% CRC correct. Therefore we don't need CRC checks".

Everybody should read what SQUIRE posted.

You are basically claiming "Inspections of airplanes provide a false sense of security". Yeah, but what you conveniently failed to mention is that inspections indeed can prevent crashes.

That's an interesting stretch to make from CRC validation to aircraft inspections and rather absurd if I may say so. Since I work with aircraft systems that's hardly likely. Do take note that modern commercial airplanes run triply-redundant HARDWARE control systems to protect against failure within common-sense cost-benefit limits.

And again: Everybody should read what he just posted. Tell that the next time when an airplane inspector appears "We don't need you. We have triply-redundant hardware".

Unbelievable ignorance.

[mod]knnknn,
You please do not reciprocate. For moderation we have moderators.
When in doubt, let it be, and that goes for everyone.

Hacker (Moderator)[/mod]

[Hacker]

SQUIRE,

if you used a failing system to run your verification software on, its conclusion cannot be relied upon as I hope you'll agree

I have to disagree here. Assuming I had a system that would not work correctly and I would copy data which would copy OK but the CRC function would (wrongly) give an error, I would at least become suspicious and I think that's the whole point.

Add that to an always-on verification tool and you have a recipe for a system that runs like molasses on a cold day. You might be fine with that but I'm sure an awful lot of people would beg to disagree.

Well, that's why it would be optional. We also have VerifyZip=1 by default by the way.
I mean, I do not think I would use such a function (I even set VerifyZip=0) but I do not see a compelling reason why to oppose its implementation.
- programming effort - the code is there, so low
- translation effort - one string for checkbox, one string for failed verify
- default off
- the only real problem I personally see is caching, as data might be verified from the cache instead of the written data, not sure if this could be reliably solved for all types of media / filesystems / WFX plugins, etc.

Just my two cents.
Roman

[knnknn]

SQUIRE,

if you used a failing system to run your verification software on, its conclusion cannot be relied upon as I hope you'll agree

I have to disagree here. Assuimng I had a system that would not work correctly and I would copy data whch would copy OK but the CRC function would (wrongly) give an error, I would at least become suspicious and I think that's the whole point.

Hacker says it exactly like it is!

Add that to an always-on verification tool and you have a recipe for a system that runs like molasses on a cold day. You might be fine with that but I'm sure an awful lot of people would beg to disagree.

Well, that's why it would be optional. We also have VerifyZip=1 by default by the way.

Again: Bingo!

I mean, I do not think I would use such a function (I even set VerifyZip=0) but I do not see a compelling reason why to oppose its implementation.
- programming effort - the code is there, so low
- translation effort - one string for checkbox, one string for failed verify
- default off

Exactly!

- the only real problem I personally see is caching

True, but third party software claims to have solved the cache problem, too.

[SQUIRE]

Don't take so personally junior, it clouds your judgement. You have been banging on about your CRC problem from day one - if that's not a fetish then what is? It's not about you, it's about trying to understand why you have the temerity to insist that everyone should have to suffer a massive performance hit just because ONE person had a few problems with his dodgy hardware.

Apart from being in a minority of one you haven't come up with any convincing information to shed light on the queries raised thus far, have you? That's simply not good enough, you know. The fact remains that you have been brandishing evidence derived from a faulty system. That fact alone would make anyone's eyebrows shoot up in any situation and fail your case immediately. It has accordingly been drawn to your attention. Regrettably, you have yet to answer calmly and convincingly. You're not going to get too many converts to your cause that way I'm afraid.

[Hacker]

SQUIRE,

It's not about you, it's about trying to understand why you have the temerity to insist that everyone should have to suffer a massive performance hit just because ONE person had a few problems with his dodgy hardware.

I still do not understand where the performance hit should come from.

Roman

[SQUIRE]

@ Hacker:

I have to disagree here. Assuming I had a system that would not work correctly and I would copy data whch would copy OK but the CRC function would (wrongly) give an error, I would at least become suspicious and I think that's the whole point.

(1) You already have a malfunctioning system.
(2) What makes you think it would be kind enough to hand you a CRC go/no go indication whether or not there was a fault? It could just as easily have kept silent and allowed bad data to be written for the next two months, no? It would be sheer chance to catch it in the act given the intermittent nature of the problem.
(3) As I said before, in such a situation all bets are off and you simply cannot rely on anything you may or may not see. I don't know if you've ever had to diagnose an intermittently failing PSU but its symptoms can range all the way from mysterious memory parity check errors to spontaneous reboots to HDDs and fans briefly failing to spin up - you name it, it can and will happen. Even your Bios can go haywire. A software verify function running on that kind of flaky system cannot sensibly be relied on for protection and would only give a false sense of security. You could never be sure if it was lying to you or or telling you the truth.

Since you participate in Ghisler's beta tests, you're doubtless all too familiar with how even the most innocuous changes to the program can often have completely unforeseen side effects. So irrespective of the coding effort and the default off condition, does it make sense to add even more cruft to the pot when the number of likely users can be numbered on the fingers of one hand, figuratively speaking? There is of course no limit to the number of functions that can be added to TC (the notorious CD burner being one). The question is what overwhelming benefit would that bring apart from software bloat?

I still do not understand where the performance hit should come from.

If enabled, the CRC verification function will obviously be doing its sums with every byte that passes across the I/O interface. Depending on how paranoid a polynomial you wish to employ, that must have an impact on your computer's performance to a greater or lesser degree.

Secondly, each file being accessed falls into the embrace of your all-singing, all-dancing modern antivirus suite. To see the possible impact of AV on even an ordinary copy function, look at the benchmark here. Especially interesting is the hit taken by the system when reading and writing 1000s of small files. Add the two functions together and I'm sure you can predict the likely outcome even if you're not Paul the psychic octopus.

I can already hear the weeping and gnashing of teeth as users who unwittingly enable verification wonder why their PC is crawling on its knees when they use their networks.