Generate MD5 log for all copied files

English support forum

Moderators: white, Hacker, petermad, Stefan2

Post Reply
isidro
Junior Member
Junior Member
Posts: 96
Joined: 2006-03-21, 04:39 UTC
Location: argentina

Generate MD5 log for all copied files

Post by *isidro »

It would be VERY useful to have MD5 computed while copying and have results on one MD5 file on destination (name should be source upper folder/drive). That would provide an integrity verification on destination media to ensure backup was done ok. Current Verify after Copy isn't useful because most files will probably be read from Cache instead and future integrity check can't be done.
Now my backups consists of four steps: 1) copy all. 2) Generate MD5 in source. 3) Generate MD5 on dest. 4) compare generated MD5 files to make sure nothing got corrupted.
With MD5 checkbox on copy it will be reduced to 2 steps: Copy+MD5 and use that MD5 file to verify dest. Faster an simpler and having MD5 will ensure integrity in the future.
User avatar
Dalai
Power Member
Power Member
Posts: 9364
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Generate MD5 log for all copied files

Post by *Dalai »

This would indeed be useful. Support++ for the request itself.

One point of discussion is: Where exactly integrate this feature in the copy dialog? Where to put the type of checksum choice? Sometimes I don't want MD5 but SHA1 or SHA256 instead. And what about the different style of checksums, all checksums in one file vs. one checksum file per file?

@Mods: This thread should be moved to TC Suggestions subforum.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
isidro
Junior Member
Junior Member
Posts: 96
Joined: 2006-03-21, 04:39 UTC
Location: argentina

Re: Generate MD5 log for all copied files

Post by *isidro »

Use last type of checksum used by user. Checksums should be per root folders or per full set and not per file (ie: bakups of 10k mp3, flacs or photos will stupily duplicate number of files). It should work exactly as Pack works now with: "Create separate archives, one per selected file/dir"
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Generate MD5 log for all copied files

Post by *MVV »

You have to add a step into your algorithm: 2) flush target device cache. Otherwise your OS may simply get data from cache on your step 3).
l0o
Junior Member
Junior Member
Posts: 4
Joined: 2019-12-03, 12:45 UTC

Re: Generate MD5 log for all copied files

Post by *l0o »

Hello!
Any update to this idea? :)
I'd like to have such option (generate md5 is just by the way - during moving, or copying) for future check. It would be very useful!

Where exactly integrate this feature in the copy dialog?
Maybe where "Verify after copy" is? "Save md5 of copied/moved files" or smthng like that.

Where to put the type of checksum choice? Sometimes I don't want MD5 but SHA1 or SHA256 instead.

Well anyway better to have MD5 than nothing and those precious checksums wouldn't vanish after compute.

And what about the different style of checksums, all checksums in one file vs. one checksum file per file?

Let's keep it simple: All in one file saved to destination folder. Assume it would be useful for moving big data (to have verify possibility in future).
isidro
Junior Member
Junior Member
Posts: 96
Joined: 2006-03-21, 04:39 UTC
Location: argentina

Re: Generate MD5 log for all copied files

Post by *isidro »

MVV wrote: 2019-07-08, 06:11 UTC You have to add a step into your algorithm: 2) flush target device cache. Otherwise your OS may simply get data from cache on your step 3).
That would be true on a small copy, in my case usually I copy between 1Tb and 6Tb, there's no way for cache holding that amount of data, so I don't bother flushing. I had detected bad memory chips due to one MD5 differences every 300Gb or so. My data would have been gradually corrupted if MD5 checks weren't done.
Anyway, MD5 of source copied files on destination would save me 3 hours of reading again the same 6Tb.
User avatar
Goblin_NN
Junior Member
Junior Member
Posts: 7
Joined: 2020-10-11, 04:32 UTC
Location: USSR

Re: Generate MD5 log for all copied files

Post by *Goblin_NN »

Last edited by white on 2024-03-13, 10:51 UTC, edited 1 time in total.
Reason: Fixed links
isidro
Junior Member
Junior Member
Posts: 96
Joined: 2006-03-21, 04:39 UTC
Location: argentina

Re: Generate MD5 log for all copied files

Post by *isidro »

Goblin_NN wrote: 2020-10-11, 04:48 UTC be sure to md5? try blake3
Irrelevant to this topic. And my bottleneck is drive speed and not checksum compute time.
isidro
Junior Member
Junior Member
Posts: 96
Joined: 2006-03-21, 04:39 UTC
Location: argentina

Re: Generate MD5 log for all copied files

Post by *isidro »

Blake is better, but still same problem: unnecessary reading the whole HD twice: one for Hash gereration, and another when copying.
[Having the verification results written to a text file would be very useful as well.] >>> Just discovered as a good workaround that all items can be marked and then copy pasted.
Last edited by white on 2024-03-13, 10:52 UTC, edited 1 time in total.
Reason: Fixed links
nemoW
Junior Member
Junior Member
Posts: 3
Joined: 2021-05-21, 06:49 UTC

Re: Generate MD5 log for all copied files

Post by *nemoW »

I strongly support this feature. Didn’t find any Windows utilities that could create a file checksum while copying
User avatar
white
Power Member
Power Member
Posts: 4594
Joined: 2003-11-19, 08:16 UTC
Location: Netherlands

Re: Generate MD5 log for all copied files

Post by *white »

Moderator message from: white » 2024-03-13, 10:54 UTC

Fixed the links in these posts:
Goblin_NN wrote: 2020-10-11, 04:48 UTC
isidro wrote: 2022-09-27, 03:38 UTC
JOUBE
Power Member
Power Member
Posts: 1448
Joined: 2004-07-08, 08:58 UTC

Re: Generate MD5 log for all copied files

Post by *JOUBE »

nemoW wrote: 2024-03-13, 10:15 UTC I strongly support this feature. Didn’t find any Windows utilities that could create a file checksum while copying
This is not a good idea.
JOUBE wrote: 2023-07-20, 05:42 UTC Checksuming the target via the source is always a "self-fulfilling prophecy".
Post Reply