New SFTP plugin available now

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

I find out the cause a very slow speed when using SFTP mode.

After the fix, I got the following speed results:

Code: Select all

1) Orig SFTPplug 2.80 b4  x64 (SFTP)   6 MiB/s
2) Orig SFTPplug 2.80 b4  x64 (SCP)   10 MiB/s
3) New  SFTPplug AllInOne x64 (SFTP)  50 MiB/s
4) New  SFTPplug AllInOne x64 (SCP)   51 MiB/s
5) curl.exe -v -k sftp://...  (SFTP)  56 MiB/s
The source code is here: github.com

Old benchmarks: https://www.ghisler.ch/board/viewtopic.php?p=360224#p360224
Last edited by remittor on 2020-11-05, 14:23 UTC, edited 3 times in total.
CoolWater
Power Member
Power Member
Posts: 734
Joined: 2003-03-27, 16:33 UTC

Re: New SFTP plugin available now

Post by *CoolWater »

Please also provide a 32 bit version. Thanks :-)

I only adapted the changes in SftpDownloadFileW() to b4 and compiled it like before. Here, SFTP mode now performs even better than SCP mode :shock: this shouldn't be possible at all :D SFTP mode reaches about 33 MB/s while SCP mode only makes 22 MB/s.
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: New SFTP plugin available now

Post by *ghisler(Author) »

I checked your changes - apparently you start with a read block size of 60000 bytes, and increase it up to 1.9 MBytes until the function fails, then you reduce it again. My code uses a fixed block size of 32kBytes.

I'm not surprised that it's even faster than SCP, because you use a fixed block size for SCP.
Author of Total Commander
https://www.ghisler.com
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

Also increased the speed for upload transmission.

After the fix, I got the following speed results:

Code: Select all

1) Orig SFTPplug 2.80 b4  x64 (SFTP)   4 MiB/s
2) Orig SFTPplug 2.80 b4  x64 (SCP)   13 MiB/s
3) New  SFTPplug AllInOne x64 (SFTP)  42 MiB/s
4) New  SFTPplug AllInOne x64 (SCP)   45 MiB/s
5) curl.exe -v -k -T sftp://..(SFTP)  25 MiB/s
The source code is here: github.com

Download SFTPplug ALLinONE: https://files.catbox.moe/ieft3s.zip
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

Christian, why was text mode support made?
After all, text mode is only needed for console FTP/SFTP clients.

I can assume that it is only useful for people who use standard Notepad utility. But the standard Notepad already has support for LF EOL.
gdpr deleted 6
Power Member
Power Member
Posts: 872
Joined: 2013-09-04, 14:07 UTC

Re: New SFTP plugin available now

Post by *gdpr deleted 6 »

remittor wrote: 2020-11-05, 14:27 UTC I can assume that it is only useful for people who use standard Notepad utility. But the standard Notepad already has support for LF EOL.
I don't know whether your assumption has legs, but lets roll with it. The CR/LF conversion was added to the plug-in around December 2008. And, well, Notepad didn't have LF support then. It didn't have it until the beginning of 2018. That's a butt-load of time right there between the end of 2008 and beginning of 2018. Also, this only holds true for Notepads on Win10. On any other Win version supported by TC, i guess the butt-load of time between the plug-in having introduced CR/LF conversions and Notepad supporting Unix-style LF approaches infinity. :D
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

elgonzo, The plugin uses non-standard behavior.

wiki Filezilla:
According to the FTP specification, ASCII files are always transferred using a CR+LF pair as line ending.
So in case the file is transferred from the client to the server, the client has to make sure CR+LF is used. Therefore it has to add nothing (on Microsoft Windows),...
RFC 959 (FTP): [3.1.1.1. ASCII TYPE]
The sender converts the data from an internal character representation to the standard 8-bit NVT-ASCII representation (see the Telnet specification). The receiver will convert the data from the standard form to his own internal form.
In accordance with the NVT standard, the <CRLF> sequence should be used where necessary to denote the end of a line of text.
The SFTP-plugin should not perform any conversions when using text mode.
It turns out that the plugin is trying to emulate text mode :?
User avatar
Dalai
Power Member
Power Member
Posts: 9364
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: New SFTP plugin available now

Post by *Dalai »

2remittor
SFTP has nothing to do with FTP or FTPS, despite the similarity in name. So neither of your quotes apply here. Besides, you can just disable the conversion if you don't like it.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups, CertificateInfo, SignatureInfo, LineBreakInfo - Download-Mirror
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

Dalai wrote: 2020-11-05, 19:47 UTC 2remittor
SFTP has nothing to do with FTP or FTPS....
Text mode (line break conversion) is the same in all three cases.

Proof: SFTP (latest draft) 5.3. Determining Server Newline Convention
gdpr deleted 6
Power Member
Power Member
Posts: 872
Joined: 2013-09-04, 14:07 UTC

Re: New SFTP plugin available now

Post by *gdpr deleted 6 »

2remittor
No, it's not the same between FTP and SFTP. You will have to read the specs more carefully.

The FTP spec encourages line endings to be CR/LF, where the sender should make sure all line endings are CR/LF; that includes the client should convert where necessary when the client is the sender (It's in the quote you provided above.)

While the default canonical line break in SFTP is also CR/LF, the SFTP protocol allows for this canonical form to be changed by the server (a facility the FTP protocol does not provide). And unlike FTP, the SFTP spec discourages the client from doing line ending conversions when encountering a non-canonical line ending, regardless of the client being sender or receiver. (It's in the section of the SFTP spec you linked to.)


I guess it would be of benefit for discussions about the SFTP plug-in if we stop making comparisons with or interpretations of unrelated specs and protocols (regardless of them being inaccurate or not), as doing so will inevitably muddle the discussion.


(P.S: Also, my previous comment was about your Notepad assumption and about Notepad support for Unix-style line endings. Didn't say anything about SFTP or the SFTP plug-in there...)
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

2ghisler(Author)
Fix bug: incorrect copy resume for files larger than 2 GB
https://github.com/remittor/SFTPplug/commit/dac9be69b733a9e7b1a683a5bd4c13901978709a
CoolWater
Power Member
Power Member
Posts: 734
Joined: 2003-03-27, 16:33 UTC

Re: New SFTP plugin available now

Post by *CoolWater »

ghisler(Author) wrote: 2020-11-03, 16:50 UTC I checked your changes - apparently you start with a read block size of 60000 bytes, and increase it up to 1.9 MBytes until the function fails, then you reduce it again. My code uses a fixed block size of 32kBytes.

I'm not surprised that it's even faster than SCP, because you use a fixed block size for SCP.
Is there any disadvantage using dynamic block sizes?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: New SFTP plugin available now

Post by *ghisler(Author) »

It doesn't seem to check how long it takes to transfer one block. So when transferring files over a slow connection (e.g. via smartphone access point), it could take several seconds per block during which the operation couldn't be aborted.
Author of Total Commander
https://www.ghisler.com
CoolWater
Power Member
Power Member
Posts: 734
Joined: 2003-03-27, 16:33 UTC

Re: New SFTP plugin available now

Post by *CoolWater »

Ok, that sounds reasonable. Nevertheless, it should be at least possible to set an INI switch to enable dynamic block size. Or maybe the logic could be improved, i.e. start with a smaller block size, check block transfer time and increase/decrease block size accordingly unless no error occurs.

Just leaving as is seems not to be the right way when much more speed is possible in theory...

Regards,
CoolWater
remittor
Junior Member
Junior Member
Posts: 49
Joined: 2019-10-02, 07:18 UTC

Re: New SFTP plugin available now

Post by *remittor »

CoolWater wrote: 2020-11-17, 16:30 UTC Or maybe the logic could be improved...
This is quite feasible. Always make the Buffer size maximum (1.9 MB), and retrieve information about the received data from the internal structure of libssh2.
To do this, you need to include in the project the internal headers of libssh2.
Post Reply