[Solved] Remind user once the bound file in the descript.ion disappears but the comment still exist

[tuska]

2Hacker
After I did two successful tests and gave my feedback, it turned out now unfortunately that in the evaluation file "MissingFiles.txt"
there are entries where characters („”™áŽ), instead of german umlauts (äöüÄÖÜ) and special characters (ߧ&) are present in the folder or file name.

A detailed check has shown that these entries are NOT relevant.
Unfortunately, there are a very large number of such entries - but this also means that considerably fewer descript.ion files
are affected than assumed.

These entries should not be in the evaluation at all.

Maybe you can do something else, otherwise I have no choice but to search for special characters and delete the lines manually.
Please let me know.

[Hacker]

tuska,
Can you please provide a sample .7z with a folder structure and a descript.ion file, please?

TIA
Roman

[Usher]

When reviewing and replacing characters with german umlauts (äöüÄÖÜ) and special characters (ß, §), the letter "ü" could not be replaced,
because it was removed from the filename.

That's why I insist on using Unicode version of any software, if possible. Otherwise accented and special characters may be broken or removed when editing text on another PC with different codepage set…

Furthermore, a "special character" occurred with a file name, although this was not present in the file name:
D:\Daten\Programme\Adobe\Acrobat\Adobe-Acrobat-11-Pro\Preise\Preise_Adobe AcrobatÿDC.url <-> Preise_Adobe Acrobat DC.url

It looks like 0xA0 character, non-breaking-space, &nbsp; copied from HTML code.

[tuska]

tuska,
Can you please provide a sample .7z with a folder structure and a descript.ion file, please?

Thank you for taking care of this.
I have now uploaded a new test file zTEST.7z

The following entries should be present in the MissingFiles.txt:

Code: Select all

D:\zTEST\zTEST_root_§22.txt
D:\zTEST\2010\15% Rabatt für ein Wiedersehen mit Ihnen....msg
D:\zTEST\2010\🎁Gesunde Geschenke für Weihnachten🎁.msg  -> descript.ion: D:\zTEST\2010\🎁Gesunde Geschenke für Weihnachten🎁.msg
D:\zTEST\2010\Folder A\aa.txt
D:\zTEST\2010\Folder A\Fußball.txt
D:\zTEST\2010\Folder A\#_Raute.txt
D:\zTEST\2010\Folder E\ee.txt
D:\zTEST\2011\FolderZ
D:\zTEST\2011\FolderZ&
D:\zTEST\2012\Überstunden.txt
D:\zTEST\2012\Folder A\Gelöscht.txt
D:\zTEST\2012\Folder A\Mängel.txt
D:\zTEST\2012\Folder B\Bügel.txt
D:\zTEST\2012\Folder D\d.txt
D:\zTEST\2012\Folder D\Folder X\äöüÄÖÜߧ&_Umlaute_Sonderzeichen.txt
D:\zTEST\2012\Folder D\Folder X\Folder Y\yy.txt
D:\zTEST\2012\Folder D\Folder X\Folder Y\Öffnungszeiten
D:\zTEST\2012\Folder D\Folder X\Folder Y\Folder A\Änderung.txt
D:\zTEST\2013

Thank you for your efforts!

PS: Unfortunately the above mentioned special characters are not complete, because you can also use '%' in a file or folder name.
      I googled, but there are a lot of lists and I didn't know which to refer to as support.

EDIT:
Removed entry "D:\zTEST\2010\Folder A\Bauer&Co.txt" from code field above because this file was NOT moved
and therefore an entry in the MissingFiles.txt file was NOT required.

[Usher]

You should kill stupid marketoids using webmail to put emoji in a (spam) message subject. Emoji are currently included in the lastest version of Unicode, but it's supported only in the newest, fully updated versions of programs and systems. Browsers can display emoji properly using webfonts, but you must install some emoji font in your system to see those icons in other programs.

Whatever comes from the web, it may be written using HTML entities like the green parcel above &#127873; (copied from misformatted text) or urlencoded using % like %F0%9F%8E%81 (the same parcel) when used as a path/filename in an URL, see http://www.test.invalid/Merry_Christmas.%F0%9F%8E%81 (go to that non-existing page and check the address bar).
Some programs may convert such entities on the fly but it may be done improperly in many ways…
Umlauts in web folder/file names are obviously urlencoded…

[tuska]

2Usher
Thank you very much for your explanations.

I just wanted to point out that Outlook messages with a subject like this can be received as promotional emails
and might in some way have an influence on the entries in the MissingFiles.txt.
There I would like to see the path and file names as they are stored so that they can be found more easily later.

I don't want to start a big discussion about Emojis.

[Hacker]

tuska,

Code: Select all

StartDir = C:\Test
Recurse = True
ResultFile = MissingFiles.txt
UseAbsolutePaths = True
UseCustomDescriptIonCodePage = 437	; Empty for default

SetWorkingDir, %StartDir%

Loop, Files, descript.ion, % Recurse = "True" ? "R" : ""
{
	If (UseCustomDescriptIonCodePage)
		FileRead, DescriptIon, *P437 %A_LoopFilePath%
	Else
		FileRead, DescriptIon, %A_LoopFilePath%

	Loop, Parse, DescriptIon, `n, `r
	{
		FileName := SubStr(A_LoopField, 1, 1) != """" ? SubStr(A_LoopField, 1, InStr(A_LoopField, " ", , 2) - 1) : SubStr(A_LoopField, 2, InStr(A_LoopField, """ ", , 2) - 2)
		If (FileName != "")
			If (A_LoopFileDir = "")
				MissingFiles .= FileExist(FileName) = "" ? UseAbsolutePaths = "True" ? SubStr(A_LoopFileLongPath, 1, InStr(A_LoopFileLongPath, "\", , 0)) . FileName . "`n" : FileName . "`n" : ""
			Else
				MissingFiles .= FileExist(A_LoopFileDir . "\" . FileName) = "" ? UseAbsolutePaths = "True" ? SubStr(A_LoopFileLongPath, 1, InStr(A_LoopFileLongPath, "\", , 0)) . FileName . "`n" : A_LoopFileDir . "\" . FileName . "`n" : ""
	}
}

If (MissingFiles)
{
	MissingFiles := SubStr(MissingFiles, 1, -1)
	FileAppend, %MissingFiles%, %ResultFile%, UTF-8
}

Works with your descript.ion files.

HTH
Roman

[tuska]

2Hacker
For me this script is an excellent control tool for inconsistent descript.ion files!
Thank you again very much for your efforts!

Notes:
Sometimes there were still problems with the representation of german umlauts and special characters.

Code: Select all

Test file:	§⌡  	(D:\zTEST\zTEST_root_⌡22.txt)

The §-character was not correctly rendered in the MissingFiles.txt file and in this test scenario as the only character.

Code: Select all

Drive D:	ö÷,Ä─   (..Ver÷ffentlichung-─nderung.png)
		Ö╓	(╓sterreich)
                üⁿ     	(Bezⁿglich Ihres Inserates..., fⁿr)
                äΣ     	(...EinkommensteuererklΣrung.pdf)
		ß▀	(Fu▀ball)

The files of drive D: with display problems do not have any special features for me after viewing them
in TC and Everything. In addition, the german umlauts and "ß" were very often represented correctly! Mmhh..

For me, these are in any case negligible points for which one should not invest any more time!

I will leave the test file in OneDrive until 31.1.2019 if anyone else is interested.
Then I delete the folder on OneDrive.
---------------------------------------------------------------------------------------------------------------------------------
For me as TC user the following results (see also above:...):

- I have now run the script again for drive D:, with the result of 159 entries(!) in MissingFiles.txt regarding inconsistent descript.ion files.
    That's still a lot of entries to clean up.

- As long as there is no other option in TC that allows renaming, moving or deleting files with TC file comments
    and backup the descript.ion files > automatically < via TC synchronization,
    in my case it will be necessary almost daily and several times to switch the required option on or off.
    and to exercise the utmost concentration in doing so.

- Furthermore, it will be necessary to run this script from time to time and make corrections if necessary.

Many thanks again for this excellent script!

Regards
Karl

[Hacker]

tuska,
Welcome, glad to help.
I think the remanining problem is still just a code page problem. I can look at it, no problem, just, again, please provide a .7z.

Thanks
Roman

[tuska]

I think the remanining problem is still just a code page problem. I can look at it, no problem, just, again, please provide a .7z.

Now you're taking it sporty

I have now made a file move for the files in question
and in the MissingFiles.txt the following entries should be present:

Code: Select all

D:\zzTEST\§_Sonderzeichen.txt
D:\zzTEST\People & Porträt_1von2_CHIP-7-2010\
D:\zzTEST\_2011 11 29_195442_Anzeige_Veröffentlichung-Änderung.png
D:\zzTEST\2010 06 04_Becken und Hüfte.pdf
D:\zzTEST\Abo-Nummer 123 - Bitte Adreßkorrektur zu Rechnungsadresse.msg
D:\zzTEST\AW  Anfrage Geschenkabo PC go --- Bestellung im WEKA-Shop -   Österreich.msg

I uploaded only the file descript.ion to OneDrive.
Unfortunately I cannot offer the actual files in this case, because they contain private data.
I hope that's enough for you in this case.
…
The interesting thing for me about the file descript.ion is that
- if the content is displayed with Ctrl+Q, the content is displayed correctly
- after opening the file with Notepad, Notepad++, EmEditor the content is displayed with special characters

[Hacker]

tuska,
The descript.ion you provided does not contain the first two lines from your post. Apart from that, the script seems to work completely fine with it. Where do you see a problem?

Roman

[tuska]

We can leave the script as it is - I am very satisfied.
I just thought you wanted to look at it again because I had notes.

descript.ion (updated on OneDrive)
- the script works except for "§" in this test scenario
- I shortened the filenames ... and now the script also works for these files I mentioned above

As I have already said above, any further effort is not necessary for me.
Let's leave the script as it is - it works great.

Thank you!

[Usher]

2tuska
Why are you still using DOS codepage in descript.ion? I have already explained that Unicode is a must:

That's why I insist on using Unicode version of any software, if possible. Otherwise accented and special characters may be broken or removed when editing text on another PC with different codepage set…

Use the following values in [Configuration] section of your wincmd.ini

Code: Select all

CmtOemFont=0 
CommentPreferredFormat=1

and you will see all umlauts, ogonki, special characters, emoji etc. (in a good Unicode text editor, of course)…

[tuska]

2Usher
Thank you for taking up this topic again and pointing me to these settings!

I am only a user and would never have thought that such entries are required in wincmd.ini,
to display all characters correctly when opening a descript.ion file with an editor (in my case EmEditor Prof. x64).

After I made this setting, everything is now displayed as desired.
---
Unfortunately I don't understand why one have to enter this parameter:
CmtOemFont=0 --> Use OEM (DOS) character set in descript.ion files (only recommended if you use the tool 4dos or 4nt),
because when I add a new comment, there is still ☐ Use OEM (DOS) font available.
No matter what - without the entry CmtOemFont=0 or CommentPreferredFormat=1 it doesn't work.
---
Now I'm all confused:
If I create a new TC file comment for a file with an umlaut with these settings,
then the umlaut is not displayed correctly in the descript.ion with F4 (EmEditor) and Ctrl+Q Quick View.
---
Finally, I removed the Unicode settings again for the following reasons:
- I prefer the correct view in the tooltip for the file/folder
- the view for the hidden file descript.ion with F4 editor or Strg-Q is secondary for me
---

Regards
Karl

[Usher]

Well, I must do more tests, so only some general remarks follow.

CmtOemFont=0

Zero is a default value. You have set 1, so I suggested to change the value rather than to delete the whole line.

there is still ☐ Use OEM (DOS) font available.

This option is always displayed, but it may be grayed out (visible and impossible to change, that is - unavailable), so in such a case you don't use OEM codepage.

If I create a new TC file comment for a file with an umlaut with these settings, then the umlaut is not displayed correctly in the descript.ion with F4 (EmEditor) and Ctrl+Q Quick View.

I think that you should restart TC to make new settings valid, and create a new descript.ion file rather than add new comments to the old file. If you set ANSI/Unicode for comments, TC will add Unicode comments to new files or existing Unicode ones and keep ANSI code page for existing 8-bit descript.ion files. It means that all your old files using DOS/OEM codepage will be displayed wrong! So you observation result is expected.

Finally, I removed the Unicode settings again

I for sure cannot do it - too many files (ebooks, documents, test files, etc.) in different languages. I think you can easy find a way to convert your files from DOS to ANSI or Unicode if they contain only German accented characters – you have probably more than one such converter in your archives…