[Plugins] FTP, SFTP and WebDAV : Performance

[SithLord]

Hi,

I've been doing some heavy tests to determine how TC performs with network related services and here are my results :

Test environment :

• Wireless AP : Linksys WRT320N running DD-WRT firmware
• Desktop : Linux on Core i7
• Android : Samsung GSII running CyanogenMOD9 4.0.4
• TotalCommander 2.02beta2
• FTPCafe and AndFTP were used to compare behaviours as ALL have FTP and SFTP support. Only TC was used for WebDAV performance.
• Bash shell to "scp" from the device to SSH server

The Android device features a SSH server (along with the client) manually installed (ie: Not installed via any APK nor "Click"-type procedure) which is Dropbear.

I own a server on the same network providing FTPS (FTP over explicit TLS) service (PureFTPd), SSH service (OpenSSH) and WebDAV (ownCloud) which is only accessible through HTTPS (ie: WebDAVS)

Test conditions :

• Transfering a dummy file of about 400 MB to let the network flow stabilize
• The network flow is ALWAYS in the "Android TO service" direction. FTP/SFTP/WebDAV clients UPLOAD the data and in the case of Android SSH server, the test was both initiated from the device and from the desktop but always respecting the aforementioned direction
• Measuring the transfert rate reported by the app and verification through 2 probing tools on the server : Nethogs to get the immediate speed and BMon to get graph-like history data and get average speed.

Test results :

• TotalCommander
  • SFTP : "jumpy" fluctuations between ~720KB/s and ~960KB/s
  • FTP : stable transfert ~3.6MB/s
  • DAV : stable transfert ~3.3MB/s
• Command line SSH
  • Desktop : scp Android:/path/to/file . : stable transfert ~3.1MB/s
  • Android : scp file Desktop:/path/to/file : stable transfert ~2.8MB/s
• AndFTP
  • SFTP : stable transfert ~1.25MB/s
  • SCP : stable transfert ~1.45MB/s
  • FTP : stable transfert ~1.9MB/s
• FTPCafe
  • SFTP : stable transfert ~1.43MB/s
  • FTP : stable transfert ~2.87MB/s

Conclusions

• FTPCafe isn't bad at all
• AndFTP is average everywhere! Not sucking but not shining too...
• Command line SSH rockets to the sky beating everyone!
• TotalCommander ROCKS on network operations but sucks bigtime at SFTP. There's definitely a LOT OF ROOM for improvement here !

Bugs encountered
Only one ! WebDAV plugin on TC for a very simple thing. On FTP and SFTP plugins, I fill the profile with all data EXCEPT the Password so when I connect, TC prompts me to enter it.

This is NOT the case with WebDAV ! If the password entry is empty, it just FAILS to connect and does not popup a dialog to enter it. Need a fix on this Christian

[ghisler(Author)]

1. SFTP: This seems to be generally a slow protocol. Furthermore, I'm using a 3rd party library, so I cannot say whether I can improve it or not. Using the SCP transfer method would certainly be better (this is already an option in the Windows version).

2. WebDAV: You are right, I should ask for the password if the user doesn't enter one.

[SithLord]

Hi,

Indeed, SFTP using SFTP subsystem is a slow protocol and SFTP using SCP is way faster on the same setup.

Eventhough you use a third party library, there's room for improvement.

For example, on FTP side in my LAN, I achieve gigabit saturation speed in transfers (~100MB/s) even with TLS encryption. Of course, to achieve that, some tweaks were needed on server side. The biggest cost in high speed transferts is encryption which is very CPU intensive.

FTP/SFTP/WebDAV are "short lived" sessions ( connect, ul/dl, disconnect) which means very strong encryption is not necessarily needed. Lowering a bit, but not too much, the encryption strength gives a huge boost in performance. For instance, my FTP TLS settings use a ciphersuite of "MEDIUM:!aNULL:!MD5" which is far enough to strongly protect everything while being very CPU friendly. FTP clients connect using the "RC4-SHA" algorithm which is good enough.

I use the same trick on SSH. The server is configured by default to use very strong encryption of course for interactive sessions (most of the time AES256) but I've aliased the "scp" command in my Shell profile to "scp -c arcfour128" (AKA RC4-SHA) which also gives gigabit speed on transfers in my LAN.

Perhaps you could add an option to the SFTP plugin in this way such as "Use MEDIUM encryption (enhances speed)" that would be DISABLED by default and would be checked only by people understanding the consequences. This would pass a "MEDIUM" type ciphersuite to the SFTP library or, if it supports it directly, the "arcfour128" keyword as OpenSSH does.

What do you think ?

PS: Are you using JSCH library ? If it's the case, it supports everything needed

[ghisler(Author)]

No, I'm not using JSCH. I'm using sshj:
https://github.com/shikhar/sshj