Synchronize Directories: Compare dir by CRC-MD5 file

[ikke_nl]

Hi,

If you have two directories with CRC of MD5 checksum files of them, I would improve the synchronization if only these files were used to compare.

Making the checksum files can be done before the synchronization to minimize the comparing itself.

Perhaps if this feature is wanted by more people, perhaps this feature would be nice to have in TC.

[ghisler(Author)]

Can you explain more in detail what you mean? TC would still have to calculate the checksums of all the files to check whether they have been changed. Or maybe I misunderstand you?

[ikke_nl]

HI,

I often synchronize between computers large files and synchronizing (by content) 5-15G over a 100MBps network is taking very long.

The computers themselves can make a CRC/MD5 checksum quickly. SATA2 still beats the network. I always make a CRC file of the directory (on both computers) to check if the files are not altered/corrupt.

If TC could check if there is a CRC/MD5 file present in both directories and synchronizes based on the checksums, comparing (in mine case) should go extremely fast.

[Sob]

It's an interesting idea. But what about outdated checksum files? It's probably no problem for you or me if we willingly decide to use this functionality. But there are many other users who might not even know that it exists. And if they have some long forgotten checksum files and the files changed in the mean time, TC must handle it somehow. It can't just blindly trust them, because it could produce wrong results. Maybe checking if checksum file is newer than all the files it references could be enough. But then again, what to do if it's not? User must be notified, but only if he really wants to use this functionality. But how can TC know that? Requiring user to explicitly input name of checksum file to use for this purpose could be the solution. But are there any other possible problems?

[ikke_nl]

I have to agree that there are several pitfalls when you look into this.

To use this functionality would require a proper understanding of its result and a good description what to expect when you use it.

On the other hand a compare (only on date and size only) at this moment can also cause a problem, when an application only changes the content of a file and not the date or size.

I saw that an other checksum file generators also add the date of the file. If this were also the case with TC, then that could be checked as 'safeguard'?

[Sob]

Sfv files often contain some additional info about size and date. But it's just free-text comments, there's no standard for it. Not that it would be too hard to invent some completely new format containing all needed information. I'm not aware of any universal standard for checksum files, so TC might as well establish one, it has enough users to get noticed. :)

But the main question remains, are there enough users who would appreciate this "remote checksums" functionality?

[ikke_nl]

The format of such a file is indeed 'free to make', however in the hope of some standardization I have sent a email to one of the 'inventors' Prof. Rivest if he is willing to put something on 'paper' about a file format...

And of source to try to put the date in there aswell....

There is a dutch saying:
You already have 'No', but perhaps you'll get a "Yes".

[ikke_nl]

I found that there is a similar English saying also: You'll never know unless you try.

I have also found an other reason for this feature:

When scanning photos with two scanners, if you combine the scanned files you will need to rename the files. E.g. both scanners use the "image001.tiff" format (where 001 gets incremented every time).

How will you know that the renamed file is not altered in the proces? Syncronizing is not an option, since the filename is changed.
If Synchronize directory only compares on date,size and CRC (and ignores the filename) you can be (99,9%) sure that the two files are the same.

I have renamed all the photos in order of the photobook, and I was wondering if I was missing some so I need to compare the files on mine 'main' computer with the computer which is attached to scanner #2.

[ikke_nl]

Can someone give me a hint how to make my own addon?

I will try to programm it for myself, but I donnot know how to use the synchronize directories tool.

Thanks in advance.

[CopyCats]

Well, there's a guide for each of the four plug-in types on this page: http://www.ghisler.com/plugins.htm (search for 'guide').

This non-official site also has information for developers:

http://www.totalcmd.net/directory/developer.html

(However, I do not know if any of the information there will be relevant to your question, or if it's even possible to control the sync function in plug-ins).

[ikke_nl]

Thanks.

But do you suggest that I use a 'Lister' based addon, or a 'content' addon?

[ghisler(Author)]

You will have to write a content plugin which implements the ContentCompareFiles function. Have a look at the help file, it describes how to handle this function. You get the names of the two files, and return either "equal" or "unequal" or some custom result with an icon you can provide.