?Copy with Verification false alarm?

The behaviour described in the bug report is either by design, or would be far too complex/time-consuming to be changed

Moderators: sheep, Hacker, Stefan2, white

Herbert
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-05, 16:55 UTC

?Copy with Verification false alarm?

Post by *Herbert » 2013-10-05, 17:09 UTC

Hi,

I was copying 50Gb of small files between two internal Sata NTFS HDD´s with verify enabled, when TC warned me that a file and its copy didn´t pass the test.
I chose ignore and later used the "Vergleich nach Inhalt" comparison, which told me they were identical.

Could this be a bug in the program or faulty hardware?
Is it possible that the two methods used for comparing lead to different results?
Or are some false alarms to be expected with the comparison used while copying?


edit: windows 7 prof, SP1 64
Last edited by Herbert on 2013-10-06, 19:12 UTC, edited 1 time in total.

Herbert
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-05, 16:55 UTC

Post by *Herbert » 2013-10-05, 18:18 UTC

Happens frequently here.
The affected files vary.

edit:

Not at home right now, so I can't check 100%, but come to think of it all those files that threw an error so far have been in numbered sequences with quite long filenames.
e.g.
country_city_street_house_room_table_leg_bolt_1_4.dat
country_city_street_house_room_table_leg_bolt_2_4.dat
And
country_city_street_house_room_table_leg_bolt_3_4.dat
would throw a false crc error.

might be coincidence, but maybe something gets mixed up.

Beta 5.

Fastcopy works fine doing the same.

I will run extensive memory-tests, just to be sure.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-10-07, 10:06 UTC

Sorry, cannot reproduce - I just copied 50 GB of files, no problem at all.
Some questions:
1. What are these .dat files? Are they database files? Maybe they are open and changed while TC was copying?
2. Which Antivirus do you use? It's possible that it was interfering
3. Which copy method do you use? Configuration-Options-Copy/Delete
4. How big were the files for which you got the error? The verify function is different when the file is >10MB than when it's <=10MB.
Author of Total Commander
http://www.ghisler.com

Herbert
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-05, 16:55 UTC

Post by *Herbert » 2013-10-07, 11:52 UTC

Thanks for your reply.

1. Sorry, the .dat was supposed to be just an example.
The files vary. I wrote some down at the beginning and those were a .zip and a two .flv
Each time I did a copy operation, other files would throw an error, though.

2. I use avast! free. I´ll run a test without Virussoftware these days as well as checking my RAM and report back.

3.
Standard Kopiermethode
Kopieren von Datum/Zeit von Verzeichnissen
Verifizieren nach dem Kopieren
F8/Entf löscht in den Papierkorb

4. All the errors I wrote down were with files bigger than 10 Mb.

User avatar
Dalai
Power Member
Power Member
Posts: 6782
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Post by *Dalai » 2013-10-07, 12:02 UTC

There's a good chance that avast interferes here with its Network Scanner and/or other modules that are known to cause trouble (incomplete loading of websites, massive slowdowns on instant messengers etc). Try disabling all modules but FileSystem, Mail and Behaviour (Verhaltensscanner in German, don't know the correct English term).

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups

Herbert
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-05, 16:55 UTC

Post by *Herbert » 2013-10-09, 10:59 UTC

Thanks for your reply!

I completely disabled avast and the firewall but the problems stay. Did´t change Windows 7 UAC so far, though. Would that be worth a try?

My conclusions so far:

Copying the same data between two disks will throw an error faster than copying data on the same disk.
It doesn´t seem to be the files themselves or at least not the files alone.
If the very same copy operation ist repeated some of the same files MIGHT throw an error again.
If I copy just the subfolder with the files which caused the error, the pattern of offending files will change.
(Might be the reason why I saw a difference when copying within the same hdd )
If I change something else, like copying user permissions the pattern changes again.
In copies of 10 to 150 GB the error might happen 0-10 times. No relation between the overall size of the data copied and how often the error is thrown.
Seems to happen mainly with files in numbered sequences.
I did some shorter RAM tests without error, but yet have to do an extensive one.

edit: I have 32Gb of RAM installed, Windows Pagefile set to 200Mb-2Gb

User avatar
Dalai
Power Member
Power Member
Posts: 6782
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Post by *Dalai » 2013-10-09, 12:40 UTC

Please try the copy method for large files.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-10-09, 14:58 UTC

I will add some more logging details to the next beta to find out what's wrong.
Author of Total Commander
http://www.ghisler.com

Herbert
Junior Member
Junior Member
Posts: 6
Joined: 2013-10-05, 16:55 UTC

Post by *Herbert » 2013-10-09, 19:06 UTC

Switching to advanced properties and selecting to use also the method for large files did actually work! (I kept the predefined buffer sizes. )
Thanks for the suggestion and sorry I didn´t think of it myself.
Copied 25 and 150 Gb without the verification failing, which wouldn´t work beforehand.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-10-11, 15:39 UTC

Yes, this other copy method uses ReadFile/WriteFile, so it has direct access to the copied data (which it doesn't when using CopyFileEx).
Author of Total Commander
http://www.ghisler.com

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-10-29, 16:57 UTC

To those who had this problem: Can you please retry with beta 7 now?
Please turn on logging in Configuration - Options - Log file. Please log file operations and log successful and failed operations.

When you get the error, you should get more details about the checked file size, and the MD5 of the source and target. Please post them here, and compare them with the real size and MD5 of the copied file.

Thanks!
Author of Total Commander
http://www.ghisler.com

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-12-09, 11:12 UTC

Since no one is willing to test this, and probably only a few people are using verify anyway, I will unfortunately have to remove the verify function from the next beta.
Author of Total Commander
http://www.ghisler.com

Sob
Power Member
Power Member
Posts: 908
Joined: 2005-01-19, 17:33 UTC

Post by *Sob » 2013-12-10, 01:31 UTC

I just copied a little over 1TB of data from one disk to another using standard copy method with verify enabled, total over 700 000 files with various sizes (from zero to hundereds of megabytes). Windows 7, TC 8.50 beta 12 x64, clean ini except enabled logging and verification. The result - zero errors. So as I see it, it works fine.

User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 38451
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) » 2013-12-10, 16:52 UTC

Thanks for your tests. The problem is that this error doesn't happen everywhere. It must be caused by some third party program interfering with TC, e.g. a virus scanner. If I can't get feedback from users with this problem, I have to remove the function because it's just too unreliable. :(
Author of Total Commander
http://www.ghisler.com

User avatar
white
Power Member
Power Member
Posts: 2020
Joined: 2003-11-19, 08:16 UTC
Location: Netherlands

Post by *white » 2013-12-16, 11:23 UTC

ghisler(Author) wrote:If I can't get feedback from users with this problem, I have to remove the function because it's just too unreliable. :(
To me it still looks like defective memory. Is user Herbert the only user that reported this problem? What exactly can people do to test this?

Post Reply