Compare by content file size limit

[likeasomebooody]

I use compare by content a lot, it's a great feature. Recently I tried to compare close to 4TB worth of content. There were several massive binary files ranging from 80GB to 1.1TB in size. When copying that much data I typically take the time to verify no data corruption due to subtle hardware/RAM failure took place. I used version 9.0a later updating to 9.22a. Under "synchronize directories" dialog asymetric/subdirs/by content/ignore date check boxes were ticked. Both versions failed to do a by content comparison simply skipping the content compare. There was no error message or warning of any kind. I tried several times on two different machines and in all cases comparison done as if "by content" checkbox was not checked. Both source and destination were USB drives connected to USB 3.0 hub copying at USB 3.0 speeds. I used windows 8.1 and windows 10 LTSB.

What is the maximum file size that TC compare by content feature can handle? And if the limitation exists is it possible to remove it?

[Usher]

TC must buffer comparing results before displaying them. Do you think that TC can use the whole available space in RAM and on disks (maybe another 4 TB) for that job? Do you realize how long it will take?

Do you really need to know location and value of any different byte? What is wrong with using "synchronize directories" or "find duplicate" options to eliminate duplicates and limit number of files for comparison?

I can't see any bug here, it's rather missing feature - TC might warn that it was mission impossible.

[ghisler(Author)]

TC doesn't buffer them by itself, it just maps the two files as a whole into memory.

[Usher]

2ghisler(Author)
Do you mean virtual memory?
What about comparing two BluRay disk images?

[ghisler(Author)]

Yes, TC just calls MapViewOfFile:
https://docs.microsoft.com/en-us/windows/desktop/api/memoryapi/nf-memoryapi-mapviewoffile

[MVV]

Perhaps it would greatly reduce memory requirement if TC would limit mapping window size (e.g. by an INI option) and move it during comparison. It should be easy to do for just comparison but it be a bit harder for visualization, since it will require more window movements. Text mode may be disabled for files that don't fit into configured window.

[ghisler(Author)]

It would make the comparison and display much more complex. TC would constandly need to check whether the position goes beyond the current buffer, and also subtract the buffer offset from the address.

[Usher]

2ghisler(Author)
So just add warning if file is too big to compare, please…

[DrShark]

It would make the comparison and display much more complex. TC would constandly need to check whether the position goes beyond the current buffer, and also subtract the buffer offset from the address.

If it will make it possible to compare larger files on 32-bit Windows, I think it's worth to consider for implementing.

[Usher]

If it will make it possible to compare larger files on 32-bit Windows, I think it's worth to consider for implementing.

Yes, but still with warning that it will take a long time and with setting for max file size.

[MVV]

ghisler(Author),
Yes, it is harder to implement, but it is the right way.

At least adding a question before loading large files into compare dialog would be nice.

Currently we have:
1. Compare two large files in TC,
2. TC sees a difference, so compare required.
3. TC opens compare dialog.
4. TC shows an error:

Code: Select all

Not enough memory, search aborted.

The two files are DIFFERENT!

Would be nice to get the following:
1. Compare two large files in TC,
2. TC sees a difference, so compare required.
3. TC checks file sizes and, if one of the two files is large, asks user:

Code: Select all

The two files are DIFFERENT!
Do you want to see differences between two large files?

C:\file1.bin
C:\file2.bin

4. TC opens compare dialog.
5. TC shows an error: "Not enough memory, search aborted. The two files are DIFFERENT!"