[Implemented in other way] Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Here you can propose new features, make suggestions etc.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

[Implemented in other way] Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

When testing media player, users sometimes test switching lister modes and press all numbers. If you press 5 for some huge video (for example, 16 GB), you must wait for parsing all the file. It may take a minute or even more.
Parsing cannot be cancelled. You should not try to press other numbers or Esc - in this case TC may crash.
That's why it should be default size limit for HTML parser (for example 16 MB, changeable in wincmd.ini).
Last edited by Usher on 2021-04-30, 02:50 UTC, edited 1 time in total.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *ghisler(Author) »

I will check whether I can make it cancellable.
Author of Total Commander
https://www.ghisler.com
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

For me the best solution is warning for large files (with user defined size limit), which gives possibility to cancel parsing.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *ghisler(Author) »

Wouldn't it be better to show a progress dialog after e.g. 2 seconds of parsing time which would allow the user to abort parsing of any file?
Author of Total Commander
https://www.ghisler.com
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Hacker »

Christian,
Wouldn't it be better to show a progress dialog after e.g. 2 seconds of parsing time which would allow the user to abort parsing of any file?
I think it would be better.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

2ghisler(Author)
I don't think so. Have you seen any html/xhtml file greater than 100 MB? Have you ever seen email message greater than 100 MiB? Even mht or XML files with embedded Base64-encoded binaries shouldn't be so great. So for valid files progress indication is futile, and 2 seconds may be enough to switch to all Lister modes.

If the warning contains filename, the reason of the warning will be obvious.

I did check. I have found 33282 files:
1. 1 of them is 256 MiB file recovered from broken partition, only first 64 KiB are HTML content.
2. 2 copies of some XML migration log, the newer one is 158 MiB, the older one is 85 MiB. Maybe some MS Office install logs are greater?
3. All the rest is less than 16 MiB each.

There are files (compressed with brotli or gz) in the browsers' caches which have no extension, but the greatest ones (about 5 MiB compressed, about 25 MiB unpacked) contain javascript code.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
DrShark
Power Member
Power Member
Posts: 1872
Joined: 2006-11-03, 22:26 UTC
Location: Kyiv, 68/262
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *DrShark »

Usher wrote: 2021-03-26, 13:16 UTCHave you seen any html/xhtml file greater than 100 MB? Have you ever seen email message greater than 100 MiB?Even mht or XML files with embedded Base64-encoded binaries shouldn't be so great.
One of Windows report generating functions can create quite large *.mht files, as mentioned here: https://ghisler.ch/board/viewtopic.php?p=329533#p329533.
Such a file in .html format has a comparable size.

BTW, in the past I suggested an option to control the files of which size should be passed to F4 editor:
https://ghisler.ch/board/viewtopic.php?f=14&t=72724
Maybe some solution with common/similar design could be implemented for both Lister/F3 viewer and Editor?
Donate for Ukraine to help stop Russian invasion!
Ukraine's National Bank special bank account:
UA843000010000000047330992708
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

DrShark wrote: 2021-03-26, 14:38 UTC One of Windows report generating functions can create quite large *.mht files, as mentioned here: https://ghisler.ch/board/viewtopic.php?p=329533#p329533.
Such a file in .html format has a comparable size.
They are comparable with my logs from p.2. Anyone found anything greater?
DrShark wrote: 2021-03-26, 14:38 UTC BTW, in the past I suggested an option to control the files of which size should be passed to F4 editor:
https://ghisler.ch/board/viewtopic.php?f=14&t=72724
Maybe some solution with common/similar design could be implemented for both Lister/F3 viewer and Editor?
And there was my similar solution there…

Here it could be as follows:
wincmd.ini, section [Lister]
HTMLParsingLimit=16
where
-1 - don't parse at all
0- no limit
number > 0 - max size in MiB
Default value may be OS dependent (the older OS, the slower hardware), for example:
16 - Windows 9x/2000
32 -Windows XP
64 - Windows Vista/7
128 - Windows8+

Now the Warning proposition:

Code: Select all

Warning! It may take a long time to parse a large file as HTML (164 MB):
D:\a very very very\long long long\really full path\to a very very very large file\filename.ext
           Do you want to continue?
             [Yes]  [[NO]]  [Esc]
"No" is selected, Esc=NO, file extension should be always visible…
Andrzej P. Wozniak
Polish subforum moderator
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

ghisler(Author) wrote: 2021-03-26, 07:44 UTC Wouldn't it be better to show a progress dialog after e.g. 2 seconds of parsing time which would allow the user to abort parsing of any file?
It doesn't work as expected. When I press 5 and TC displays progress bar, I can click"Cancel" - and it's OK. But then TC switches view to mode 5 and continues to display progress bar so I have to click "Cancel" again - and it's wrong. The same occurs when I press Esc to cancel operation.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *ghisler(Author) »

It seems to happen only when you press '5' while a lister plugin is shown. Then I'm getting the key down event twice. When I choose '5' from the menu or when in text mode, it doesn't happen. I will try to find a workaround.
Author of Total Commander
https://www.ghisler.com
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

History.txt wrote: 21.04.21 Fixed: Lister: Press '5' while a plugin was shown -> conversion to html was done twice because the key event was received twice (64)
Still not fixed for 32-bit TC.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *ghisler(Author) »

It never happened here in 32-bit, so I only changed the 64-bit verison. I have just made some more tests with pressing '5' in 32-bit lister, then aborting the conversion with a mouse click or ESC, but I never got a second extraction. Lister was in mode 1, 2, 3, 5, 6, 7 and in various plugins (ulister, logtail, etc) when I pressed '5'.

Can you give me more details of when this happens?
1. Type of file you viewed
2. Mode in which you were when you pressed '5'
3. How you aborted the conversion

Thanks!
Author of Total Commander
https://www.ghisler.com
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

TC 32-bit in Windows XP, tested with new Media Player as in this topic: viewtopic.php?f=15&t=74262
I have described all needed steps, just see my original description:
Usher wrote:If you press 5 for some huge video (for example, 16 GB), you must wait for parsing all the file.
Use at least 1 GB video for tests (avi, mkv, mp4, webm, etc., format is irrelevant), with 32 MB file in my case it just blinks too fast to click anything.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *ghisler(Author) »

I can confirm the problem when playing a video (or large audio track) with the internal media player. This even happens in 64-bit too.
It has a completely different reason than the nug I already fixed...
Author of Total Commander
https://www.ghisler.com
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: [Implemented in other way] Limit file size for HTML parser in Lister 5 mode (bug and suggestion)

Post by *Usher »

2ghisler(Author)
History.ini wrote:23.04.21 Fixed: Lister: Press '5' while the new media player was shown -> conversion to html was done twice even when not aborted (32/64)
Confirm fixed in TC 10b8.
Andrzej P. Wozniak
Polish subforum moderator
Post Reply