Unrealistischer Upload führt zu Upload-Abbruch

[HSchoenb]

Hi

Ich habe hier ein komisches Problem.
Getestet auf 2 Rechnern im selben Netz (Win11 mit TC11.03 und Win10 mit TC9.51)

Wenn ich eine große Datei via FTP hochlade, dann ist sie laut UI in 1 Sekunde hochgeladen.
Laut Logfile mit einer Geschwindigkeit von 1.9 GB/s !!
Daraufhin versucht Total Commander vom Server eine Bestätigung zu bekommen.
Da die Verbindung zum FTP-Server aber irgendwo immernoch ausgelastet ist, trennt TC die Verbindung nach einem Timeout.

Hinweis: Aktiv/Passiv-Modus ändern bringt nichts.


Router Info:
--------------------------------------------
Telekom Glasfaser
Geschwindigkeit des Internetzugangs (verfügbare Bitrate): ↓ 113,3 Mbit/s ↑ 56,7 Mbit/s
=> d.h. maximal mögliche Upload-Geschwindigkeit: 56,7/8 = 7,0875 Mb/s


Logfile (IP und Name geändert):
--------------------------------------------------------------------------------
Connect to: (28/09/2024 08:36:23)
hostname=ftp.webserver.de
username=myUser
startdir=
ftp.webserver.de=12.345.67.890
220 ::ffff:12.345.67.890 FTP server ready
USER myUser
331 Password required for myUser
PASS ***********
230 User myUser logged in.
SYST
215 UNIX Type: L8
FEAT
211-Features:
211-CLNT
211-EPRT
211-EPSV
211-HOST
211-LANG en-US
211-MDTM
211-MFF modify;UNIX.group;UNIX.mode;
211-MFMT
211-MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.groupname*;UNIX.mode*;UNIX.owner*;UNIX.ownername*;
211-REST STREAM
211-SIZE
211-TVFS
211-UTF8
211 End
CLNT Total Commander (UTF-8)
200 OK
OPTS UTF8 ON
200 UTF8 set to on
Connect ok!
PWD
257 "/" is the current directory
Get directory
TYPE A
200 Type set to A
PASV
227 Entering Passive Mode (12,345,67,890,154,53).
Connected to PASV port
MLSD
150 Opening ASCII mode data connection for MLSD
Download
Waiting for server...
226 Transfer complete
CWD www.webserver.de
250 CWD command successful
PWD
257 "/www.webserver.de" is the current directory
Get directory
PASV
227 Entering Passive Mode (12,345,67,890,143,129).
Connected to PASV port
MLSD
150 Opening ASCII mode data connection for MLSD
Download
Waiting for server...
226 Transfer complete
CWD download
250 CWD command successful
PWD
257 "/www.webserver.de/download" is the current directory
Get directory
PASV
227 Entering Passive Mode (12,345,67,890,154,115).
Connected to PASV port
MLSD
150 Opening ASCII mode data connection for MLSD
Download
Waiting for server...
226 Transfer complete
TYPE I
200 Type set to I
PASV
227 Entering Passive Mode (12,345,67,890,155,135).
Connected to PASV port
STOR MyFile.zip
150 Opening BINARY mode data connection for MyFile.zip
Upload: 403.012.890 bytes, 1.969.697 kbytes/s
OFFLINE4, no reply
----------
Connect to: (28/09/2024 08:38:43)
hostname=ftp.webserver.de
username=myUser
startdir=/www.webserver.de/download
ftp.webserver.de=12.345.67.890
220 ::ffff:12.345.67.890 FTP server ready
USER myUser
331 Password required for myUser
PASS ***********
230 User myUser logged in.
SYST
215 UNIX Type: L8
FEAT
211-Features:
211-CLNT
211-EPRT
211-EPSV
211-HOST
211-LANG en-US
211-MDTM
211-MFF modify;UNIX.group;UNIX.mode;
211-MFMT
211-MLST modify*;perm*;size*;type*;unique*;UNIX.group*;UNIX.groupname*;UNIX.mode*;UNIX.owner*;UNIX.ownername*;
211-REST STREAM
211-SIZE
211-TVFS
211-UTF8
211 End
CLNT Total Commander (UTF-8)
200 OK
OPTS UTF8 ON
200 UTF8 set to on
Connect ok!
CWD /www.webserver.de/download
250 CWD command successful
PWD
257 "/www.webserver.de/download" is the current directory
TYPE I
200 Type set to I
SIZE MyFile.zip
213 137321495
Upload size mismatch!
Copied (28/09/2024 08:38:44): e:\web_upload\MyFile.zip -> ftp://ftp.webserver.de/www.webserver.de/download/MyFile.zip 403.012.890 bytes, 18.768 kbytes/s
SIZE MyFile.zip
213 137321495
PASV
227 Entering Passive Mode (12,345,67,890,164,239).
Connected to PASV port
APPE MyFile.zip
150 Opening BINARY mode data connection for MyFile.zip
Upload: 403.012.890 bytes, 6.500 kbytes/s
226 Transfer complete
SIZE MyFile.zip
213 403012890
Copied (28/09/2024 08:39:27): e:\web_upload\MyFile.zip -> ftp://ftp.webserver.de/www.webserver.de/download/MyFile.zip 403.012.890 bytes, 9.858 kbytes/s
Get directory
TYPE A
200 Type set to A
PASV
227 Entering Passive Mode (12,345,67,890,166,65).
Connected to PASV port
MLSD
150 Opening ASCII mode data connection for MLSD
Download
Waiting for server...
226 Transfer complete

[petermad]

2HSchoenb
Topics in this part of the forum are supposed to be in English.

I have taken the liberty of making a Google-translation for you:

Hi

I have a strange problem here.
Tested on 2 computers in the same network (Win11 with TC11.03 and Win10 with TC9.51)

If I upload a large file via FTP, according to the UI it is uploaded in 1 second.
According to the log file with a speed of 1.9 GB/s!!
Total Commander then tries to get a confirmation from the server.
Since the connection to the FTP server is still busy somewhere, TC disconnects the connection after a timeout.

Note: Changing active/passive mode does nothing.


Router info:
------------------------------------------------
Telecom fiber optic
Internet access speed (available bitrate): ↓ 113.3 Mbit/s ↑ 56.7 Mbit/s
=> i.e. maximum possible upload speed: 56.7/8 = 7.0875 Mb/s

[ghisler(Author)]

Looks like a server bug, apparently it kills the connection when trying to upload a big file. I don't think there is anything I can do here.
Maybe it's a limit of the user account (quota settings).

[HSchoenb]

It cannot be an issue with the FTP Server itself.
My internet connection has a maximum upload speed of 7 Mb/s.
And the file is uploaded with a speed of 1969 Mb/s!

And I can still upload multiple even larger files to that server from an other site.

I can only imagine is that my Internet provider receives the file with maximum speed.
And then slowly transfers the file to the FTP server.
And because the data is still transferred to the FTP server, the FTP server can not send any reply.

As you can see the issue happens with the command STOR MyFile.zip only.
APPE MyFile.zip works fine with 6.500 kbytes/s.

[Horst.Epp]

...
I can only imagine is that my Internet provider receives the file with maximum speed.
And then slowly transfers the file to the FTP server.
And because the data is still transferred to the FTP server, the FTP server can not send any reply.

No internet provider uses such a construct.
There may be proxy hosts or other elements in the path to your FTP server
but the FTP protocol runs to the target.

[HSchoenb]

Ok, if you do not see a solution to it from the Total Commader site, lets change this whole bug report into feature requests:
Option A: Increase the timeout after an upload
Option A: Please add a speed limit to the FTP options.


Back to the issue:
It works with with WinSCP.
I will do more tests with their settings to see if I can reproduce it with other options.

It is not the FTP server.
I tested other servers as well. (https://www.sftp.net/public-online-ftp-servers) I used ftp.scene.org
Created a test file in Windows command "fsutil file createnew testFile 500000000".
Upload progress bar is done in less than a second, then stalls.
Upload speed in the UI shows again 2GB/s.
Please test that server yourself if you think it is the FTP server.


>No internet provider uses such a construct.
But why did it work before I changed my internet connection to glas fiber?
And why does it work with Total Commander and the same FTP server from a different network with some other internet connection?
How do I get this 1969 Mb/s upload speed if my inernet connection is 7 Mb/s?

[Dalai]

How much RAM does your system have and how large are the files you upload? I suspect that the system reads as much data from the file as it can as fast as it can into its file cache and then slowly writes it to the target (=uploads the file). The more (free) RAM the system has the more likely this occurs. This behavior is known from local networks but it might also occur with (fast) internet connections.

[petermad]

2Dalai

how large are the files you upload

I guess the file is 500Mb

Created a test file in Windows command "fsutil file createnew testFile 500000000".

[ghisler(Author)]

The timeout is currently 10 seconds. Does it really take longer to start the upload? Then try increasing the timeout in wcx_ftp.ini:
Below section header [General], please add the following line:
DataConnectTimeout=30
This would increase the timeout to 30 seconds.

You can edit the wcx_ftp.ini via Configuration - Change settings files directly. Make sure not to edit the wincmd.ini instead!

[HSchoenb]

How much RAM does your system have and how large are the files you upload?
I suspect that the system reads as much data from the file as it can as fast as it can into its file cache and then slowly writes it to the target (=uploads the file).

64GB.

I know Windows does file caching onto USB drives and other devices. And TC shows the same issue if you copy files onto thse USB drives (copy speed of x GB/s and then stalls). But in such cases TC waits until the files is completely transferred.
Back to FTP:
Since when does Windows cache TCP connections?
And TC measure the write speed into the TCP socket, not the speed at which it can read files.
Otherwise everyone copying a file from an SSD to any FTP server would have the same issue.

Does it really take longer to start the upload?

Start?
The upload starts and is at 100% in half a second.
And then stalls and aborts after 20 seconds.

DataConnectTimeout=30

The copy of 500MB should take 500000/6500 = 79sec.
I set the timeout to 100s.

It works:

Code: Select all

227 Entering Passive Mode ().
Connected to PASV port
STOR testFile
150 Opening BINARY mode data connection for testFile
Upload: 500.000.000 bytes, 1.899.109 kbytes/s
226 Transfer complete
SIZE testFile
213 500000000
Copied (13/10/2024 10:10:44): e:\programmierung\testFile -> ftp://ftp.myServer.de/temp/testFile 500.000.000 bytes, 6.500 kbytes/s

Another test.
If I abort the transfer that is shown as 100% and stalled, then TC reports:

Code: Select all

450 PORT: data transfer in progress

Can this be used to detect if the file transfer is complete?


About WinSCP Difference:
According to the log it uses "Extended Passive Mode", while TC uses "Passive Mode".
But even if I disable the passive mode in TC, I have the same issue.
WinSCP has a speed limitation feature (disabled in my case) and the log file states that the source file is read in chunks.
Perhaps because it is required if you change the upload speed during the transfer.
And then it has to transfers the data in chunks as well.

[ghisler(Author)]

The copy of 500MB should take 500000/6500 = 79sec.
I set the timeout to 100s.

The timeout doesn't need to cover the entire upload, just the time between when Total Commander sends the command
STOR filename
and the time when the upload actually starts.

Moderator message from: ghisler(Author) » 2024-10-15, 10:35 UTC

Moved to will not be changed

[HSchoenb]

The timeout doesn't need to cover the entire upload, just the time between when Total Commander sends the command
STOR filename
and the time when the upload actually starts.

No, that does not work.
As I said the upload starts and is at 100% within half a second.
The timeout is required between the time the upload is at 100% and Total Commander aborting the upload because it does not get some "upload done" message from the FTP server.
Total Commander does not detect that the upload is still in progress.
Total Commander does not show the right upload percentage %.


As stated, if I abort the upload manually after it is shown as 100%, the FTP server returns to Total Commander:

Code: Select all

450 PORT: data transfer in progress

>Moved to will not be changed

Isn't it a bug if TC does not check for "data transfer in progress"?

[ghisler(Author)]

Unfortunately I have never seen any FTP server with such odd behaviour. I just doesn't make any sense that the percentage would jump to 100% immediately. Do you perhaps know what server software is used?