Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Slavic · Post by *Slavic » 2025-05-30, 06:27 UTC

Option to see HTML or XMl without tags is quite useful, Lister shows HTML files this way by default. However, for the files in UTF-16 encoding (rare for HTML but typical for XML) Lister shows them correctly, but in the Options menu they are marked as UTF-8, which is wrong. Apparently the menu items "HTML text (strip tags)" and "Unicode" cannot be marked at the same time. This problem happens with both HTML and XML files.

Save this example in Notepad as test16.htm with encoding UTF-16 LE:

Code: Select all

<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-16" />
</head>

<body>
<p>This is a test example in UTF-16 encoding.</p>
</body>
</html>

1. Lister opens this file in HTML mode (no tags), but shows UTF-8 in the Options menu.
2. Selection of Unicode, which means UTF-16, in the menu shows full text with all tags.
3. Selection of HTML shows the example without tags, but again as UTF-8 in the menu.

Suggestion: UTF-16 files should be shown as Unicode in the menu, with or without the HTML/XML tags.

Post by *ghisler(Author) » 2025-05-30, 08:07 UTC

I can confirm this, but it's not actually a bug: What happens is that Lister first converts the file from html to plain text utf-8, and then uses the utf-8 viewer to display it. You can verify this by using some accented characters in your sample, e.g. in french:

Code: Select all

<!DOCTYPE html>
<html lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-16" />
</head>

<body>
<p>Ceci est un exemple de test codé en UTF-16.</p>
</body>
</html>

When you first view it in Lister, it shows up as

Ceci est un exemple de test codé en UTF-16.

with html and utf-8 checked. When you then click on "html" in the menu, it shows up as

Ceci est un exemple de test codÃ© en UTF-16.

with utf-8 unchecked in the menu. As you can see, the é is now split up into the 2 codes making up the utf-8 character.

That's why I would prefer to keep utf-8 checked, and not "Unicode".

Slavic · Post by *Slavic » 2025-05-30, 16:57 UTC

I understand this logic: in HTML mode Lister shows in the menu the encoding of internal conversion instead of original encoding. But this is not intuitively obvious and should be documented in some way. Currently this conversion is not explained and both HTML encodings are mentioned as equally possible:

5. HTML-Text (strip tags)
...
Also supports UTF-8- and UTF-16-encoded HTML, if the file contains the appropriate META HTTP-EQUIV header.

With some explanation, user will not be misguided which exactly encoding has a viewed document.

browny · Post by *browny » 2025-05-30, 20:43 UTC

Tags are mostly expected to be in UTF-8 or ANSI, while UTF-16LE makes the file an odd creation.
Hard to find good logic in a broken file.

Post by *ghisler(Author) » 2025-06-02, 09:38 UTC

Fir the above reasons I will not change this, sorry.

Moderator message from: ghisler(Author) » 2025-06-02, 09:38 UTC

Moved to will not be changed

AntonyD · Post by *AntonyD » 2025-06-02, 13:06 UTC

2ghisler(Author)
Why not at least add a note to the menu item title that this is the encoding from an INTERNAL tool?
Do you really think a person can remember forever a fact that has just been revealed to the public?
And in a couple of months - a similar question won't come up again?

Total Commander

Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Re: Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Re: Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Re: Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Re: Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu

Re: Lister may incorrectly show UTF-16 HTML/XML as UTF-8 in menu