SFTP plugin - issues with keep alive and disconnect/timeout

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: white, Hacker, petermad, Stefan2

mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

SFTP plugin - issues with keep alive and disconnect/timeout

Post by *mastabog »

Hi,

I'm an avid user of the SFTP plugin and I should have written this post a lot sooner. I see there are no options for client initiated keep-alive messages.

It also appears that the client does not respond to server initiated keep-alive messages.

This means that some servers timeout the connection, some quite quickly ... it's quite frustrating, especially when I'm editing a remote file and the connection dies before I get a chance to save it.

Is there anything I can do to fix this on the client side? Any hidden options? The openssh libraries do support client keep-alive messages but none are exposed in the TC plugin GUI.

Anyone else ran into this problem?

Thanks.
Last edited by mastabog on 2012-11-18, 03:08 UTC, edited 2 times in total.
mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

Post by *mastabog »

I forgot to mention that both the openssh linux client (sftp) and the putty psftp client properly respond to server keep-alives and do not experience this problem.

Here are some details. The server is sshd under Linux.

1. If I enable ssh keep-alives from the server by setting ClientAliveInterval 10 and ClientAliveCountMax 3 in sshd_config then the TC client does not respond to them and the connection is severed by the server in about 40 seconds, as it perceived the client to be dead:

Code: Select all

Nov 18 01:54:50 svr-perf sshd[4668]: subsystem request for sftp by user root
Nov 18 01:55:31 svr-perf sshd[4668]: Timeout, client not responding.
2. If sshd_config is configured with TCPKeepAlive=yes but with ClientAliveInterval=0 (i.e. only TCP level keep-alive packats from the server) then the connection dies after about 15 minutes because oddly the TC client is closing it:

Code: Select all

Nov 18 02:00:41 svr-perf sshd[4967]: subsystem request for sftp by user root
Nov 18 02:15:45 svr-perf sshd[4967]: Read error from remote host 172.26.233.118: Connection reset by peer
3. If I disable all keep-alives from the server, nothing is different from scenario 2. ... the TC client is terminating the connection after 15 minutes:

Code: Select all

Nov 18 02:28:07 svr-perf sshd[6032]: subsystem request for sftp by user root
Nov 18 02:43:11 svr-perf sshd[6032]: Read error from remote host 172.26.233.118: Connection reset by peer
Clearly, all server keep-alive messages are ignored by TC, both tcp and ssh level ones.

Any clues on how to make the TC sftp client either respond to server keep-alive messages, or send client keep-alive messages?

If it cannot be done now, then I'm asking Christian to implement a client keep-alive feature for this plugin ... you can just request a path, e.g. RequestAbsolutePath('.')

Regards
mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

Post by *mastabog »

As a side issue: using the dlls from libcurl v7.27 or v7.28 causes a fatal exception upon connection (I'm suing the 32-bit version of TC and public key auth).

Dlls from libcurl v7.26 work (it's not available on the curl page that the TC docs link to, you have to dig for it manually).

Error for v7.27 and v7.28:

Code: Select all

---------------------------
Total Commander 8.01
---------------------------
Exception in sftpplug.wfx:FsFindFirstW!
Access violation at address 6305E326. Read of address 00000000
Windows 7 SP1 6.1 (Build 7601)

Please report this error to the Author, with a description
of what you were doing when this error occurred!

Windows exception: C0000005
Stack trace:
6305E326
444AD1  4460A5  446A22  423F38  429556  42968C
652B87  652F1A  6D682B  4AB749  4E5EA8  656449
6D6E13  4460A5  446A22  423F38  445FE3  653052
>656270  4DC7BB  638BD5  444C24  444AD1  4460A5
444A0F  446DD6  444AD1  446A22  423F38  445FE3
423F38  445F51  423F38  4460A5  446A22  423F38
445FE3  423F38  429604  42969C  6D9F34  
Raw:
4460A5  423F38  445FE3  423F38  4267A5  447AA1
444AD1  446C59  445F51  423F38  446D9B  447158
4360F3  444AD1  446CF1  435E92  446033  4460A5
446A22  446A45  423F38  429556  42968C  669CD2
40650F  6D764F  652B87  445F51  423F38  446D9B
444AD1  446CF1  4460A5  446A22  445FE3  423F38
6D5128  5DB277  5C0000  445F51  423F38  446D9B
444AD1  446CF1  4460A5  6D6E13  6CFB90  6D6781
652F1A  445FE3  423F38  696DFE  5E3EE0  5E3F04
660071  64E77E  6D682B  6547C0  4AB749  4E5EA8

Press Ctrl+C to copy this report!
Continue execution?
---------------------------
Yes   No   
---------------------------
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Yes, it seems that the OpenSSH dll isn't compatible with the new OpenSSL 1.x included in these Libcurl versions. The error seems to occur when the OpenSSH dll tries to call the OpenSSL dll, so there is nothing I can do to fix it. :(
Author of Total Commander
https://www.ghisler.com
mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

Post by *mastabog »

ghisler(Author) wrote:Yes, it seems that the OpenSSH dll isn't compatible with the new OpenSSL 1.x included in these Libcurl versions. The error seems to occur when the OpenSSH dll tries to call the OpenSSL dll, so there is nothing I can do to fix it. :(
What about the actual issue of this topic? (the keep-alive and disconnection problem)
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

I'm not aware of keep alive commands on SFTP, so I wonder how this is implemeneted. Maybe they open a terminal and send bogus Unix shell commands?
Author of Total Commander
https://www.ghisler.com
mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

Post by *mastabog »

ghisler(Author) wrote:I'm not aware of keep alive commands on SFTP, so I wonder how this is implemeneted. Maybe they open a terminal and send bogus Unix shell commands?
I gave a short example in my post above:
I'm asking Christian to implement a client keep-alive feature for this plugin ... you can just request a path, e.g. RequestAbsolutePath('.')
Basically, sending any dummy ssh/sftp commands over the secure channel will keep it alive. I guess you can implement a timer to monitor activity and when it expires, send an SFTP command that is very light in terms of traffic, like RequestAbsolutePath(.). That would keep it alive.

other sftp clients do have keep-alive features, e.g. WinSCP, see its "Connection" panel in the configuration. It can send null ssh packets or execute dummy protocol commands.
Sob
Power Member
Power Member
Posts: 941
Joined: 2005-01-19, 17:33 UTC

Post by *Sob »

No need to handle sending dummy commands yourself, library has function for it:

http://trac.libssh2.org/wiki/KeepAlive

It's there since version 1.2.5.
mastabog
Junior Member
Junior Member
Posts: 72
Joined: 2004-03-07, 21:36 UTC

Post by *mastabog »

That's even better. The page Sob linked to clearly says "Sending keepalive's is off by default. ". It's actually on the front page of http://trac.libssh2.org/ as well.

Christian: could you please implement it for your plugin? The inability to keep the session alive is driving me crazy ...

Would it be possible to recompile a libssh2.dll with that setting enabled by default?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

I will consider it as an option.
Author of Total Commander
https://www.ghisler.com
tungwaiyip
Junior Member
Junior Member
Posts: 23
Joined: 2003-03-07, 23:27 UTC
Location: San Francisco, US
Contact:

Post by *tungwaiyip »

Any update on this? Total Commander can be very effective for sftp editing. But frequent disconnection make it a mess to recover from.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Post by *ghisler(Author) »

Sorry, I haven't had the time to do this yet. It's not something you can just turn on, it has to be called regularly. And the interval isn't known - some servers may drop the connection after 15 minutes, some after 30 seconds...
Author of Total Commander
https://www.ghisler.com
tungwaiyip
Junior Member
Junior Member
Posts: 23
Joined: 2003-03-07, 23:27 UTC
Location: San Francisco, US
Contact:

Post by *tungwaiyip »

Many software, like PuTTY, has a configurable keepalive interval. It will be a huge improvement compares to disconnection and lost edit.
Bozzy
Junior Member
Junior Member
Posts: 15
Joined: 2013-09-10, 15:06 UTC

Post by *Bozzy »

Still no news on that? I've had the same problem for years and I've lived with it so far, but a solution would be much appreciated.

If the library supports sending keep alives by itself, it shouldn't be too difficult to implement in SFTP plugin. I suppose it's something to "turn on" and a "packet interval" to set, maybe nothing more.

Christian?
Bozzy
Junior Member
Junior Member
Posts: 15
Joined: 2013-09-10, 15:06 UTC

Post by *Bozzy »

I've taken a look at libssh2. I'm not a C/C++ coder, but what I understood is that there are two functions:

- libssh2_keepalive_config: with this one you set the keepalive packets interval and if an answer from the server is expected (which could be ignored IMHO).

- libssh2_keepalive_send: this one sends the keepalive packet (if the interval is passed) and returns the number of seconds to wait until a new call is necessary.

If I understand correctly, the library doesn't send keepalives automatically, but provides useful ways to help the user send them at the right pace. It's user responsibility to call libssh2_keepalive_send every some interval (not necessary the real configured interval - it can be much more frequent, since packets are sent only when the configured interval is passed).

So, IMHO, implementing it should be very simple for a C/C++ dev. Just call configure when connecting and have a timer call send every 1s or 5s.

I simply can't imagine where to put this in the plugin sources, but maybe some C/C++ developer can... or maybe Christian itself ;)

EDIT: the interval could then be configured per-connection, as any other parameter, and the keepalive logic could be put "fixed" in the code, since if interval is set to zero, the library itself will never send keepalive packets, even if libssh2_keepalive_send is called.
Post Reply