Synchronisation over FTP misses changed files in text mode

Please report only one bug per message!

Moderators: white, Hacker, petermad, Stefan2

Post Reply
mk
New Member
New Member
Posts: 1
Joined: 2011-02-23, 08:18 UTC

Synchronisation over FTP misses changed files in text mode

Post by *mk »

I have two html files which differ (only) in the following line:

local file:
<frameset cols="30%,70%"border="0" frameborder="0" framespacing="0">

remote file:
<frameset cols="230px,*"border="0" frameborder="0" framespacing="0">

If I synchronise these directories using SyncDirs with
asymetric unchecked
recursive checked
ignore date checked
it lists the file as compared in text mode and equal. If I compare them individually (e.g. by double clicking on them in the comparison list of SyncDirs) the comparison tool correctly finds the differences of the files, regardless of whether I compare them binary or in text mode.

If I copy both files to local directories and synchronise these, TC correctly identifies the files as unsynched. In FTP mode however, it fails to do so.

I have screenshots of this behaviour and could provide the files in question, though I would rather not post them publicly here. If anyone would like to look into this behaviour just PM me.

One last bit of information: I am the administrator of the used FTP and can hence detail it's configuration if needed. In general it is just a vsftpd on a Ubuntu 8.04 system.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

During FTP sync, TC checks the number of line breaks - if the local file size minus the number of line breaks equals the remote file size, TC assumes that the files are identical (differing only in the line breaks).
Author of Total Commander
https://www.ghisler.com
Sob
Power Member
Power Member
Posts: 941
Joined: 2005-01-19, 17:33 UTC

Post by *Sob »

It's not always reliable then. Lets say I have some file with X CRLF line breaks. When I delete X characters from file, without changing the number of line breaks, TC won't spot the difference and will think that the line breaks were just converted to LFs. But ok, it should not happen very often (user mk being just unfortunate exception ;) and the "=txt" symbol may be enough as a warning.

But wouldn't it be better to not use this assumption at all, when the selected transfer mode is set to binary?

And what's worse, when by content is checked (server supports XCRC or similar), the files are marked as same (using plain "=" symbol, not "=txt") even though there's no way how they could have the same checksum when they have different size.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

when by content is checked (server supports XCRC or similar), the files are marked as same (using plain "=" symbol, not "=txt") even though there's no way how they could have the same checksum when they have different size.
Actually when "by content" is checked, and the file sizes differ, TC calculates the local CRC as if the file didn't have CR characters in it (just LF, not CRLF). An equal sign is shown only when the CRC matches the CRC reported by the server. So I wonder why it doesn't work in your case.

However, in your case there seems to be no comparison by content - the two files just happen to have the same size. In this case, "by content" should be grayed out (it is on my server)...
Author of Total Commander
https://www.ghisler.com
Sob
Power Member
Power Member
Posts: 941
Joined: 2005-01-19, 17:33 UTC

Post by *Sob »

I didn't know that TC skips CRs when calculating CRC. Now it makes sense. But...

If I have for example:

1) local file1.txt: 123<cr><lf>456<cr><lf>789<cr><lf>
server file1.txt: 123<lf>456<lf>789<lf>

Then they'll be correctly(*1) shown as same in both cases, because with by content CRs are skipped while calculating CRC and without by content the number of local CRs is substracted from local file size and then the sizes are compared and they match.

(*1) - but it can be argued that if I chose to always use only binary transfer mode, then the files should not be shown as same, because as binary, they are not

2) local file2.txt: 123<cr><lf>456<cr><lf>789<cr><lf>
server file2.txt: 123456789abc

Then with by content they will be correctly shown as different, but without by content they will be incorrectly shown as same. But it's necessary trade-off. Either there can be possibility for text-mode comparison without downloading whole files with slight possibility of errors, or it can't be possible at all.
In this case, "by content" should be grayed out (it is on my server)...
Isn't it by content grayed when server does not support any of CRC commands and not grayed if it does?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Isn't it by content grayed when server does not support any of CRC commands and not grayed if it does?
Yes, that's how it should be (and is according to my tests)...
Author of Total Commander
https://www.ghisler.com
MaXyM
Junior Member
Junior Member
Posts: 7
Joined: 2013-05-26, 10:12 UTC

Post by *MaXyM »

ghisler(Author) wrote: However, in your case there seems to be no comparison by content - the two files just happen to have the same size. In this case, "by content" should be grayed out (it is on my server)...
I would like to bring the subject up again.
I use Tcmd sync for years, and I'm sad I cannot rely on it for 100%. Actually I used to but I would like to see it improved.

I'm working mostly with SFTP connection (sftp plugin by Christian) but sometimes with FTP. I will describe both cases

SFTP
If two files has the same size, sync reports them as equal until I check 'by content'. But in that case all files are reported with question mark.
We all agree that this is quite useless especially while working with a lot of files.

FTP
If two files has the same size, sync reports them as equal.
Unfortunately 'by content' checkbox is greyed out (probably server doesn't provide CRC)
Why there is no question mark? Why reported as equal even if we are not sure?

If I missed something, then I'm sorry. Please show me a way I would sync local directory against remote one.

Suggestion:
Add 'true sync' option for remote directories, which will just download files and compare by content locally. Nowadays internet is fast enough. And there are cases where quality of comparison is more important than time spent in the process.

Please consider providing straight info (pupup or message) while remote server doesn't support CRC

with regards

PS. Meaning of question mark is not described on wiki pages:
http://www.ghisler.ch/wiki/index.php/Synchronize_directories
kristalharri
New Member
New Member
Posts: 1
Joined: 2014-02-03, 05:34 UTC
Location: Laguna Hills

Post by *kristalharri »

My vote also goes to Tcmd sync....I have never seen such a flabbergast tool..It is functional in terms of storage and utility too...
How to access Google shared contacts ?
Post Reply