[TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

English support forum

Moderators: white, Hacker, petermad, Stefan2

Post Reply
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

[TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *andry81 »

OS: Windows x64

Steps to repro:

1. create utf-8 file list w/o BOM.
2. create a file with a trademark character (U+2122) in the file name (use charmap application to lookup character and copy it to the clipboard).
3. add the file to the file list.

If add the BOM it starts to work, which means I have to always prefix the BOM into the list.

Can you remove that requirement and always treat the list file as utf-8 file even if BOM is not prefixed?
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *ghisler(Author) »

Please use UTF-16 instead, it always has BOM.

The problem is that some characters valid in UTF-8 are also valid in Chinese or Korean ANSI.
Author of Total Commander
https://www.ghisler.com
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *andry81 »

I am not quite get it. Why not to force treat it as utf-8?
Fla$her
Power Member
Power Member
Posts: 2244
Joined: 2020-01-18, 04:03 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *Fla$her »

UTF-16 ... always has BOM.
It's doesn't correspond to reality.
Overquoting is evil! 👎
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *Usher »

andry81 wrote: 2021-04-09, 20:51 UTC Why not to force treat it as utf-8?
Because of backward compatibility.

However, there might be an option in wincmd.ini for 8-bit text files without BOM - something like "Define charset by file type" which would allow to choose at least Autodetect/ANSI/DOS/UTF-8 for specific file extensions.

Notice that both Unicode and its UTF-8 form aren't perfect in design. That's why automatic detection fails in some cases and we must live with it.
Andrzej P. Wozniak
Polish subforum moderator
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *andry81 »

Usher wrote: 2021-04-12, 20:16 UTC
andry81 wrote: 2021-04-09, 20:51 UTC Why not to force treat it as utf-8?
Because of backward compatibility.
Backward compatibility with DOS? The file system in the Windows already Unicode.
Usher wrote: 2021-04-12, 20:16 UTC However, there might be an option in wincmd.ini for 8-bit text files without BOM - something like "Define charset by file type" which would allow to choose at least Autodetect/ANSI/DOS/UTF-8 for specific file extensions.
This has no sense, a windows files path already in unicode.
Usher wrote: 2021-04-12, 20:16 UTC Notice that both Unicode and its UTF-8 form aren't perfect in design. That's why automatic detection fails in some cases and we must live with it.
Automatic detection of that? Why you need automatic detection of unicode characters?
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *Usher »

Why TC should stop work in Windows 95? The file system is in UTF-16, not in UTF-8.

And a file list is just a text file. How TC would know that you'd like to deal with a very specific kind of text?
Andrzej P. Wozniak
Polish subforum moderator
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *andry81 »

Usher wrote: 2021-04-16, 21:24 UTC Why TC should stop work in Windows 95? The file system is in UTF-16, not in UTF-8.
How is it related to the question about the BOM presence?
Usher wrote: 2021-04-16, 21:24 UTC And a file list is just a text file. How TC would know that you'd like to deal with a very specific kind of text?
To not know it the unicode is exist in first place.
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *Usher »

I'm afraid I don't understand your English. I can't help you.
Andrzej P. Wozniak
Polish subforum moderator
andry81
Junior Member
Junior Member
Posts: 97
Joined: 2018-11-22, 19:17 UTC

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *andry81 »

Usher wrote: 2021-04-17, 11:06 UTC I'm afraid I don't understand your English. I can't help you.
let me help you: translate.google.com
User avatar
Hacker
Moderator
Moderator
Posts: 13052
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Re: [TC9.51] LOADSEARCH/LOADSELECTION does not work over utf-8 file list w/o BOM

Post by *Hacker »

Moderator message from: Hacker

andry81,
Please stay polite.
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.
Post Reply