New SFTP plugin available now

[JOUBE]

Thank you for the information. Please work on it calm, but please work on it But calm...

Because of the SFTP-Plugin is the best tool. For example for file manager operations: searching, run small commands/scripts on the server and so on... Only the upload/download speed is a bit to slow...

Thanks

Joube

[CoolWater]

- strange choice of buffer size, seems to converge to 32 seconds(!) per buffer with no upper buffer size limit

I just sent you my final adaptions. But I think you misunderstood the buffer size changes. Please see my email for some clarification.

[CoolWater]

There is a third party Putty based plugin:
https://totalcmd.net/plugring/sftp4tc.html
Unfortunately it hasn't been updated since 2017.

I found a lib called TGPuttyLib, which uses the latest Putty sources. It is quite easily to handle but doesn't expose all needed functions like ssh agent calls.

Whereas file transfers are really fast! There is a demo GUI to test it:
https://www.syncovery.com/release/TGPuttyLibVCLDemo64-Build19.zip (64 bit)
https://www.syncovery.com/release/TGPuttyLibVCLDemo32-Build19.zip (32 bit)

[ghisler(Author)]

Here is the new beta version of the SFTP plugin with dynamic upload and download buffer sizes provided by CoolWater. It also contains an updated libssh2 library based on the latest final release 1.11:
https://www.totalcommander.ch/beta/sftpplug310b4.zip
The source code is available here:
https://www.totalcommander.ch/beta/sftpplug_src310b4.zip

I have tested this extensively with several GBytes of data. Please let me know if you encounter any problems with it.

[JOUBE]

Here is the new beta version of the SFTP plugin with dynamic upload and download buffer sizes provided by CoolWater. It also contains an updated libssh2 library based on the latest final release 1.11:

Thanks a lot. I'll test it, but unfortunately it will take some days.

Joube

[JOUBE]

A first simple download test with a big tar.gz file (because of: already compressed) via vpn via internet in comparison to filezilla: Looks good (very good).

The SFTP plugin is now just as fast as Filezilla (in fact, in the test it was even slightly faster than filezilla, but that could be due to many circumstances). But a little more uneven in speed than filezilla.

Completely unimportant in the beta at the moment, but essential for a release: the remaining time is completely wrong (for example it shows 2 minutes when it still takes 15 minutes).

So very good in the first step.

I will continue testing, downloads and uploads and among other things: I have the impression that other things (directory changes and file lists) are now a little slower. Can this be? But like I said, that's just an impression, I'll keep testing.

Thanks in advance.

[ghisler(Author)]

The remaining time is calculated by Total Commander, not by the plugin. It's based on the speed during the last few seconds (averaged), so it can take a moment to get accurate values, or not get good values when the speed varies too much.

[JOUBE]

The remaining time is calculated by Total Commander, not by the plugin. It's based on the speed during the last few seconds (averaged), so it can take a moment to get accurate values, or not get good values when the speed varies too much.

"a moment" = 20 minutes ? ... If it always says "2 minutes" in the first 10 minutes and after 10 minutes "1 minute" and after 14 minutes "45 seconds", then TC should leave the number or determine it more precisely, that only leads to trouble: for me too.

Maybe it is possible to overgive the time from Plugin to TC? In any case, it's not a good thing in the moment and it's pointless.

Edit: Because the transfer bar correctly shows the transfer amount at the current position at every moment, can't the Tc refer to this when calculating the time range and use that as a basis for time calculation?

[ghisler(Author)]

I have checked it in the debugger now: The problem is that the plugin now sends progress callback messages to Total Commander even when the amount of uploaded data hasn't changed. That's because the block size is so big that the ssh library returns ERROR_EAGAIN multiple times until the block has finally been sent. This results in incorrect speed readings. I will add a fix to Total Commander 11.03 RC5. I can't fix it in the plugin because the callbacks are also important to not freeze the program during the transfer.

[JOUBE]

I have checked it in the debugger now: The problem is that the plugin now sends progress callback messages to Total Commander even when the amount of uploaded data hasn't changed. That's because the block size is so big that the ssh library returns ERROR_EAGAIN multiple times until the block has finally been sent. This results in incorrect speed readings. I will add a fix to Total Commander 11.03 RC5. I can't fix it in the plugin because the callbacks are also important to not freeze the program during the transfer.

At first: the most important thing is that the speed has been significantly increased. Thanks for that.
The remaining time display is basically just an accessory.
But if the remaining time is shown, then it should be in the right order of magnitude.

- But if that's not possible, then it's better to simply leave it out and not display the remaining time.
- The green bar shows the correct data, so the Tc could perhaps refer to it and calculate remaining time itself based on this.

These would be my suggestions so that there are no complaints at the release time of the new version of the plugin.

It is important that the new routines are only used for uploading and downloading files and never used in other parts of the plugin, nor when loading the directory list, changing directories, connecting, etc. (which now seems a bit slower and more awkward to me? But I can fool me)

Joube

[KozakMak]

https://ibb.co/3YJJqFF

It's probably oftop, sorry... i'm not very good at this
What could it be if I can copy files, but cannot delete or modify them?

Do I need some special rights on the server?

[JOUBE]

2KozakMak
https://www.google.com/search?q=sftp+download+delete+rights

[Flint]

A user reported on the Russian forum, that with OpenWrt the plugin sometimes does not show a file/dir starting with an exclamation mark. I installed OpenWrt in a virtual machine (by default it has Dropbear server) and could reproduce the issue both with SFTP plugin verison 3.05, and 3.10b4 (in SCP mode). If there are several files/subdirs starting with an explamation mark, TC does not show the first of them, and shows the rest.

When I run the "ls -la" command, all the exclamation mark entries go first, even before the . and .. entries. Could that be the reason? For example, here is the output from the KiTTY connection:

Code: Select all

root@OpenWrt:/tmp/test# ls -la
-rw-r--r--    1 root     root             0 Feb 22 15:16 !file
drwxr-xr-x    2 root     root            40 Feb 22 15:16 !subdir
drwxr-xr-x    3 root     root           100 Feb 22 15:16 .
drwxrwxrwt   17 root     root           440 Feb 22 15:16 ..
-rw-r--r--    1 root     root             0 Feb 22 15:16 asdf

And here is how the same dir looks in TC:
https://i.imgur.com/NEo7qv1.png

[CoolWater]

Please check the setting "Show entire directory listings in log (for debugging purposes)" in SFTP settings dialog and post the output from the log window.

Somehow the output format of the first line seems to be unknown to the parser...

Thanks!

[Flint]

With this option, the log shows two lines missing: the . one, and the first exclamation mark. But if I run "ls -la" within TC, it shows the full output, just like KiTTY:

Code: Select all

Get directory: \tmp\test
export LC_ALL=C
ls -la /tmp/test/
drwxr-xr-x    2 root     root            40 Feb 22 17:37 !subdir
drwxrwxrwt   16 root     root           420 Feb 22 17:37 ..
-rw-r--r--    1 root     root             0 Feb 22 17:37 asdf

Code: Select all

Quote: ls -la /tmp/test/
-rw-r--r--    1 root     root             0 Feb 22 17:37 !file
drwxr-xr-x    2 root     root            40 Feb 22 17:37 !subdir
drwxr-xr-x    3 root     root           100 Feb 22 17:37 .
drwxrwxrwt   16 root     root           420 Feb 22 17:37 ..
-rw-r--r--    1 root     root             0 Feb 22 17:37 asdf

(Disregard the difference in timestamps between this and my previous output, I rebooted the VM, and it reset the memory-stored /tmp contents, so I recreated it.)