New SFTP plugin available now

[CoolWater]

Thats is really strange! Could you try the following command: export LC_ALL=C && ls -la /tmp/test/
Is the output the same as in your second sample?

[Flint]

Exactly the same as normal ls -la. After all, locale should not cut off some files from the listing, that would be even werider.

Code: Select all

Quote: export LC_ALL=C && ls -la /tmp/test/
-rw-r--r--    1 root     root             0 Feb 22 17:37 !file
drwxr-xr-x    2 root     root            40 Feb 22 17:37 !subdir
drwxr-xr-x    3 root     root           100 Feb 22 17:37 .
drwxrwxrwt   16 root     root           420 Feb 22 17:37 ..
-rw-r--r--    1 root     root             0 Feb 22 17:37 asdf

[CoolWater]

Yes, it should be the same output of course. But for the file listing, the output gets parsed whereas the command line output is returned exactly as the server sends the data.

I could check it in the debugger. But therefore it would be helpful if you could send a link to the openwrt image you used. What VM environment do you use?

[Flint]

2CoolWater
Actually, it seems this happens with all the servers if you use SCP mode. I've just checked connecting to a CentOS 7 machine. If I use normal mode, the output is as expected. But when I set the "SCP for everything" mode and reconnected, the "ls -la" output is missing the first line - the single dot line. The first visible line in the log is the .. entry. With most servers it simply does not matter, because these entries are ignored. And since the . and .. are always the very first, losing one of them is not a problem. But since OpenWrt/Dropbear puts a significant entry at the first place, it causes the issue.

If you still want to test OpenWrt specifically, I'm using VirtualBox, and here is the instruction I followed:
https://openwrt.org/docs/guide-user/virtualization/virtualbox-vm
and used the image openwrt-23.05.2-x86-64-generic-ext4-combined.img.gz.

[ghisler(Author)]

It could be because the listing doesn't start with entries "." or ".." as usual, probably because the exclamation mark is sorted before the dot.

[Flint]

2ghisler(Author)
As I said, it happens with any first entry, it gets completely removed. It's just that usually this entry is a "." and gets ignored anyway, that's why nobody noticed it. But when the exclamation mark entry suddenly went into the first place and got removed, it became apparent that something is wrong.

I don't think the first entry should be removed at all, whatever it is. It's for the parser to decide what's important and what is not.

[JOUBE]

I have now tested this problem (under "scp only") with an OpenSSH server under debian bookworm (12). There are no missed entries for the exact same files/directories with the same names and rights. Even with the usage of the sticky bit (t), which I first suspected was the culprit.

I have a "real" OpenWrt system here with Dropbear. This is not a virtual machine. If I create the same files/directories as on Debian, the problem occurs as described. Even without the sticky bit being set. Edit: Now I have additional tested it with WinSCP. WinSCP has not this problem and shows all entrys.

[Spine]

I confirm the bug. Log under the spoiler.
dropbear - 2022.82-1

Spoiler

Key exchange method: curve25519-sha256
Host key method: ssh-ed25519
Encryption method (both directions): aes128-ctr
MAC (message authentication code) method (both directions): hmac-sha2-256
Supported authentication methods: publickey,password
Authentication via password for user: root
CONNECT \A95x
Get directory: \
ls -la --time-style="+>>%Y%m%d_%H%M%S" /
ls: unrecognized option: time-style=+>>%Y%m%d_%H%M%S
Get directory: \tmp\tmp
export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x 2 root root 40 Feb 26 10:35 !2
drwxrwxrwt 27 root root 740 Feb 26 10:40 ..
Get directory: \tmp\tmp\!2
export LC_ALL=C
ls -la /tmp/tmp/!2/
drwxr-xr-x 4 root root 80 Feb 26 10:41 ..
Get directory: \tmp\tmp
export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x 2 root root 40 Feb 26 10:35 !2
drwxrwxrwt 27 root root 740 Feb 26 10:40 ..
Get directory: \tmp\tmp\!2
export LC_ALL=C
ls -la /tmp/tmp/!2/
drwxr-xr-x 4 root root 80 Feb 26 10:41 ..
KEEP-ALIVE \A95x
Download file: (SCP)/tmp/tmp/!2/test2
SCP download error: SCP_PROTOCOL (28)
Upload file: (SCP)/tmp/tmp/!2/test2
Set date/time for: /tmp/tmp/!2/test2
Get directory: \tmp\tmp\!2
export LC_ALL=C
ls -la /tmp/tmp/!2/
drwxr-xr-x 4 root root 80 Feb 26 10:41 ..
-rw-r--r-- 1 root root 1 Feb 26 10:41 test2
Get directory: \tmp\tmp
export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x 2 root root 60 Feb 26 10:41 !2
drwxrwxrwt 27 root root 740 Feb 26 10:40 ..
Create directory: \tmp\tmp\!0
Get directory: \tmp\tmp
export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x 2 root root 40 Feb 26 10:41 !1
drwxr-xr-x 2 root root 60 Feb 26 10:41 !2
drwxrwxrwt 27 root root 740 Feb 26 10:40 ..
Get directory: \tmp\tmp\!1
export LC_ALL=C
ls -la /tmp/tmp/!1/
drwxr-xr-x 5 root root 100 Feb 26 10:42 ..
Get directory: \tmp\tmp
export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x 2 root root 40 Feb 26 10:41 !1
drwxr-xr-x 2 root root 60 Feb 26 10:41 !2
drwxrwxrwt 27 root root 740 Feb 26 10:40 ..
KEEP-ALIVE \A95x
DISCONNECT \A95x

[ghisler(Author)]

2JOUBE
Does your server return names starting with ! before the dot entries when you use the command line, e.g.
ls -l
?

Because on my server ghisler.com an entry like
!test
is returned between letters 's' and 'u', e.g.
sys
!test
usr

So the error only happens due to the strange sorting of the server. I will try to find a workaround.

[JOUBE]

2JOUBE

Connection to a dropbear (scp for all) on Openwrt:

Code: Select all

Verzeichnis einlesen: \tmp\test
export LC_ALL=C
ls -la /tmp/test/
drwxr-xr-x    2 root     root            40 Feb 26 09:34 !subdir
drwxrwxrwt   19 root     root           480 Feb 26 11:13 ..
-rw-r--r--    1 root     root             0 Feb 26 09:33 asdf
Quote: ls -la
-rw-r--r--    1 root     root             0 Feb 26 09:33 !file
drwxr-xr-x    2 root     root            40 Feb 26 09:34 !subdir
drwxr-xr-x    3 root     root           100 Feb 26 09:34 .
drwxrwxrwt   19 root     root           480 Feb 26 11:13 ..
-rw-r--r--    1 root     root             0 Feb 26 09:33 asdf
Quote: ls -l
-rw-r--r--    1 root     root             0 Feb 26 09:33 !file
drwxr-xr-x    2 root     root            40 Feb 26 09:34 !subdir
-rw-r--r--    1 root     root             0 Feb 26 09:33 asdf

Is this enough of the log file? First the plugin reads the directory itself, then I via Tc commandline: ls -la and ls -l

It's a miracle... Or a nearly ununderstandable bug within the Plugin... Edit: Or a bug in the lib...

[Spine]

2JOUBE
putty

Code: Select all

root@A95X:~# ls -l /tmp/tmp/
drwxr-xr-x    2 root     root            40 Feb 26 10:42 !0
drwxr-xr-x    2 root     root            40 Feb 26 10:41 !1
drwxr-xr-x    2 root     root            60 Feb 26 10:41 !2
root@A95X:~# ls -la /tmp/tmp/
drwxr-xr-x    2 root     root            40 Feb 26 10:42 !0
drwxr-xr-x    2 root     root            40 Feb 26 10:41 !1
drwxr-xr-x    2 root     root            60 Feb 26 10:41 !2
drwxr-xr-x    5 root     root           100 Feb 26 10:42 .
drwxrwxrwt   27 root     root           740 Feb 26 13:40 ..
root@A95X:~#

TC

Code: Select all

========================
Connecting to: x.x.x.x
IP address: x.x.x.x:22
Key exchange method: curve25519-sha256
Host key method: ssh-ed25519
Encryption method (both directions): aes128-ctr
MAC (message authentication code) method (both directions): hmac-sha2-256
Server fingerprint: del
Supported authentication methods: publickey,password
Authentication via password for user: root
CONNECT \A95x
Get directory: \tmp\tmp
ls -la --time-style="+>>%Y%m%d_%H%M%S" /tmp/tmp/
ls: unrecognized option: time-style=+>>%Y%m%d_%H%M%S

export LC_ALL=C
ls -la /tmp/tmp/
drwxr-xr-x    2 root     root            40 Feb 26 10:41 !1
drwxr-xr-x    2 root     root            60 Feb 26 10:41 !2
drwxrwxrwt   27 root     root           740 Feb 26 13:45 ..
Quote: ls -l
drwxr-xr-x    2 root     root            40 Feb 26 10:42 !0
drwxr-xr-x    2 root     root            40 Feb 26 10:41 !1
drwxr-xr-x    2 root     root            60 Feb 26 10:41 !2
Quote: ls -la
drwxr-xr-x    2 root     root            40 Feb 26 10:42 !0
drwxr-xr-x    2 root     root            40 Feb 26 10:41 !1
drwxr-xr-x    2 root     root            60 Feb 26 10:41 !2
drwxr-xr-x    5 root     root           100 Feb 26 10:42 .
drwxrwxrwt   27 root     root           740 Feb 26 13:45 ..
DISCONNECT \A95x

[JOUBE]

2Ghisler(author)
This is the log of the connection (scp for all) with the OpenSSH on Debian bookworm:

Code: Select all

Verzeichnis einlesen: \home\user\test
ls -la --time-style="+>>%Y%m%d_%H%M%S" /home/user/test/
drwxrwxrwt 24 pi pi 4096 >>20240226_103010 ..
-rw-r--r--  1 pi pi    0 >>20240226_031256 asdf
-rw-r--r--  1 pi pi    0 >>20240226_031233 !file
drwxr-xr-x  2 pi pi 4096 >>20240226_031405 !subdir
Quote: ls -la
insgesamt 12
drwxrwxrwt  3 pi pi 4096 26. Feb 03:14 .
drwxrwxrwt 24 pi pi 4096 26. Feb 10:30 ..
-rw-r--r--  1 pi pi    0 26. Feb 03:12 asdf
-rw-r--r--  1 pi pi    0 26. Feb 03:12 !file
drwxr-xr-x  2 pi pi 4096 26. Feb 03:14 !subdir
Quote: ls -l
insgesamt 4
-rw-r--r-- 1 pi pi    0 26. Feb 03:12 asdf
-rw-r--r-- 1 pi pi    0 26. Feb 03:12 !file
drwxr-xr-x 2 pi pi 4096 26. Feb 03:14 !subdir

Maybe you are right with that:

So the error only happens due to the strange sorting of the server. I will try to find a workaround.

Please note, if using plugins sftp routines with openssh than the . and the .. are unsorted at different places somewhere in the file list of the log. So sorting is not a problem with SFTP. Maybe it is usefull to use the same routines also for scp...

[Dalai]

The culprit is the default sorting used by the locale. Example from a (standard) Rasbian 10 machine:

Code: Select all

pi@raspi:/ $ LANG=de_DE.UTF-8 ls -la /tmp/somedir/
insgesamt 0
drwxr-xr-x 10 pi   root 200 Feb 26 12:31  .
drwxrwxrwt 14 root root 320 Feb 26 12:28  ..
drwxr-xr-x  2 pi   root  40 Feb 26 12:25 '!1'
drwxr-xr-x  2 pi   root  40 Feb 26 12:27  1
drwxr-xr-x  2 pi   root  40 Feb 26 12:25 '!2'
drwxr-xr-x  2 pi   root  40 Feb 26 12:27  2
drwxr-xr-x  2 pi   root  40 Feb 26 12:31 '!a'
drwxr-xr-x  2 pi   root  40 Feb 26 12:31  a
drwxr-xr-x  2 pi   root  40 Feb 26 12:31 '!b'
drwxr-xr-x  2 pi   root  40 Feb 26 12:31  b

Code: Select all

pi@raspi:/ $ LANG=C ls -la /tmp/somedir/
total 0
drwxr-xr-x  2 pi   root  40 Feb 26 12:25 '!1'
drwxr-xr-x  2 pi   root  40 Feb 26 12:25 '!2'
drwxr-xr-x  2 pi   root  40 Feb 26 12:31 '!a'
drwxr-xr-x  2 pi   root  40 Feb 26 12:31 '!b'
drwxr-xr-x 10 pi   root 200 Feb 26 12:31  .
drwxrwxrwt 14 root root 320 Feb 26 12:28  ..
drwxr-xr-x  2 pi   root  40 Feb 26 12:27  1
drwxr-xr-x  2 pi   root  40 Feb 26 12:27  2
drwxr-xr-x  2 pi   root  40 Feb 26 12:31  a
drwxr-xr-x  2 pi   root  40 Feb 26 12:31  b

LANG=de_DE.UTF-8 is the default on this system, I just specified it so you can immediately see which locale is used.

NB: I'll never get used to the crappy sorting of the de_DE locale. There's a reason why I set an exclamation mark as the first character so that it gets sorted BEFORE everything else. Instead the following character is also considered for sorting which is really annoying . But I digress.

Regards
Dalai

[Flint]

Again, the different sorting is not an source of the problem. I mean, yes, that's unusual, but I don't remember any standard requiring that the dot-entries must always go first. The real issue is that in SCP mode the first entry gets deleted. Normally, the first two entries are "." and "..", and you can see that in SCP mode you'll only get the ".." entry, the "." one is completely lost. That should not happen! By a happy coincidence, this "." entry is not used by TC, so its disappearance does not cause issues, but as we can see in the OpenWrt example, the order of items is not a given. And if any other entry goes to the first place, it gets deleted instead of the "." entry.

Even if by some miraculous act of God all the SSH servers on Earth start always returning "." and ".." entries first, we still cannot say "the first item is always unused, so we'll completely ignore the bug that destroys it". It's just wrong. Who knows why it happens, and what other side-effects it may have.

[JOUBE]

I mean, yes, that's unusual

No, not at all and the plugin can already handle this very well: I have a Debian system here with OpenSSH that has the entries . and .. (even separately) somewhere between the list of files and directories (according to log *): a completely unsorted list is returned, which is then sorted by the plugin/or the Tc). And the plugin/TC can handle it without any problems. However, the sftp part of the plugin is used.

Therefore my guess is that either the plugin treats scp differently, or even the lib (libssh2.dll) already makes this error.

Joube

*) folder has 118 entries, . at line 61, .. at line 43 (yes: .. before . ). The list is completely unsorted