Search files compressed with compact.exe in Windows 10

English support forum

Moderators: white, Hacker, petermad, Stefan2

Post Reply
User avatar
LonerD
Senior Member
Senior Member
Posts: 381
Joined: 2010-06-19, 20:18 UTC
Location: Makeyevka, Russia
Contact:

Search files compressed with compact.exe in Windows 10

Post by *LonerD »

Windows 10 has possibility to use NTFS files compression with built-in console command compact.exe and parameter /exe (with algoritms xpress4k, xpress8k, xpress16k, lzx).
Windows can't recognize compression, show overlay image on file icon in this case and decompress files with menu.
Total Commander have option to search compressed files, but it not works with this parameter.
Is there any way to search ? Plugins, options, scripts...?
"I used to feel guilty in Cambridge that I spent all day playing games, while I was supposed to be doing mathematics. Then, when I discovered surreal numbers, I realized that playing games IS math." John Horton Conway
User avatar
ghisler(Author)
Site Admin
Site Admin
Posts: 48021
Joined: 2003-02-04, 09:46 UTC
Location: Switzerland
Contact:

Re: Search files compressed with compact.exe in Windows 10

Post by *ghisler(Author) »

If I understand you correctly, you want to find NTFS-file system compressed files.

TC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.
Author of Total Commander
https://www.ghisler.com
User avatar
LonerD
Senior Member
Senior Member
Posts: 381
Joined: 2010-06-19, 20:18 UTC
Location: Makeyevka, Russia
Contact:

Re: Search files compressed with compact.exe in Windows 10

Post by *LonerD »

ghisler(Author) wrote: 2019-01-07, 15:18 UTCTC should show "c" character in "Attributes" column for NTFS-compressed files. You can also find them via the second tab of the search page.
It works with standard compression.

But my question - about NTFS-compression with parameter /exe in Windows 10, for example

Code: Select all

compact /c /a /f /i /exe:xpress16k /s:"d:\MyDirectory"
Files in Windows Explorer don't change color, and TC doesn't show directory and files in directory as "compressed".
In Windows file properties "Size" and "Size on disk" are diferent.
TC command cm_GetFileSpace show incorrect "Actual space used on source drive".
But maybe some tricks exist... :?
"I used to feel guilty in Cambridge that I spent all day playing games, while I was supposed to be doing mathematics. Then, when I discovered surreal numbers, I realized that playing games IS math." John Horton Conway
User avatar
Stefan2
Power Member
Power Member
Posts: 4133
Joined: 2007-09-13, 22:20 UTC
Location: Europa

Re: Search files compressed with compact.exe in Windows 10

Post by *Stefan2 »

LonerD wrote: 2019-01-08, 22:25 UTC...
But my question - about NTFS-compression with parameter /exe in Windows 10, for example
...
But maybe some tricks exist... :?
How would you detect such files? (binary signatures, Header Hexdump, magic numbers)

See f.ex.:
http://file-extension.net/seeker/file_extension_zip
http://mark0.net/soft-tridnet-e.html



 
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV »

Stefan2,
As I understand, this 'brand new compression' doesn't change files itself, it only stores them in a new special compressed form that provides better compression for PE files. And it seems that it uses some other file attributes than reqular NTFS compressed files use (C attribute flag), and this is quite strange for me.
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher »

2Stefan2
2MVV
It's not "brand new", /exe switch is just for internal executable compression, now incorporated into the program for NTFS compression, see https://en.wikipedia.org/wiki/Executable_compression
The options meaning:
- lzx - LZ compression like old DOS LZEXE packer, see https://bellard.org/lzexe.html
- express4k, express8k, express16k - LZMA compression like UPX and other modern exe packers, 4k, 8k, 16k - is block size or alignment.

If it's full exe packer, every compacted executable (exe, dll etc.) should start with unpacking stub containing its unique signature (as @Stefan2 suggests). In this case there won't be "c" NTFS attribute, and TC should always show compressed size.
Maybe Total7zip plugin with additional codecs will show uncompressed size, when pressing Ctrl+PgDn to open archive…

I may be wrong, didn't test yet.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV »

I think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool... According to this article, compression is done on file system level, so files are unpacked transparently and you can't detect the packer by signatures. Also, it is said that Explorer doesn't highlight LZX-compressed files, i.e. it doesn't really use regular C attribute, and it is a real question is there a way to detect such files or not...
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher »

MVV wrote: 2019-01-12, 23:39 UTCI think that it would be a bad idea to integrate an EXE file packer into file system level NTFS compression tool...
But compact.exe IS exe packer when called with /exe switch. And now I'm sure that decompression module cannot be added as a stub to exe. I've forgotten that executable files are digitally signed with certificate so they cannot be changed. That's why decompression must be transparent.
MVV wrote: 2019-01-12, 23:39 UTCAccording to this article, compression is done on file system level
It looks like I'm wrong about LZX algorithm - it's probably LZMA compression of the whole file, while express compression works on the fly with smaller LZMA compressed data blocks. Unfortunately, the article doesn't explain the whole thing.
MVV wrote: 2019-01-12, 23:39 UTCit is a real question is there a way to detect such files or not...
If it's done on the system level, the NTFS driver must somehow recognize execompressed files so there should be some additional attribute in MFT. I know that MS coders can make it completely dumb and the NTFS driver may work like old doublspace/drivespace now checking every cluster for compression signature, but it would be really stupid. There may also be problems with defragmentation, so there should be some API dealing with new compression methods…
Andrzej P. Wozniak
Polish subforum moderator
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV »

The most logical solution would using same C attribute but with some algorithm identifier within compressed sector header, but perhaps it would break backward compatibility so they did this... It is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check. :)
But compact.exe IS exe packer when called with /exe switch.
It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory, it is file system driver who decompresses original file before mapping it onto the memory.
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher »

MVV wrote: 2019-01-14, 06:17 UTCIt is interesting what will pre-W10 system see for such compressed files. You can try compressing some EXE on a removable disk and check. :)
It's possible that you should first create partition and format it under newest Win10 build. It may be newer NTFS version marked somewhere there… Guessing only…
MVV wrote: 2019-01-14, 06:17 UTC
But compact.exe IS exe packer when called with /exe switch.
It is not an exe packer in usual meaning, because it has no a stub that decompresses payload into memory,
You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?
MVV wrote: 2019-01-14, 06:17 UTCit is file system driver who decompresses original file before mapping it onto the memory.
To be honest, I don't know (and don't want to guess any more here) whether exe pack/unpack libs are separate dlls or modules linked statically here or there. That's why I've mentioned lack of documentation.
Andrzej P. Wozniak
Polish subforum moderator
User avatar
MVV
Power Member
Power Member
Posts: 8702
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Search files compressed with compact.exe in Windows 10

Post by *MVV »

Since we are talking about NTFS compression, I assume that it is a transparent lossless compression that doesn't make any file changes that are visible for applications.
You are talking about two things - packer and unpacker. There's no need to keep both features in a single file. Remember old pkzip/pkunzip separate files?
Usually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.
User avatar
LonerD
Senior Member
Senior Member
Posts: 381
Joined: 2010-06-19, 20:18 UTC
Location: Makeyevka, Russia
Contact:

Re: Search files compressed with compact.exe in Windows 10

Post by *LonerD »

LonerD wrote: 2019-01-03, 17:48 UTCIs there any way to search ? Plugins, options, scripts...?
Solution.
Search with Dirsizecalc plugin
Property "Compression rate"
OP: <
Value: 100
"I used to feel guilty in Cambridge that I spent all day playing games, while I was supposed to be doing mathematics. Then, when I discovered surreal numbers, I realized that playing games IS math." John Horton Conway
User avatar
Usher
Power Member
Power Member
Posts: 1675
Joined: 2011-03-11, 10:11 UTC

Re: Search files compressed with compact.exe in Windows 10

Post by *Usher »

MVV wrote: 2019-01-15, 05:49 UTCUsually EXE packers that modify binaries and leave EXE extension produce "self-extracting archives" that unpack and call compressed binaries when you execute them on the fly. Otherwise compressed files would have another extension.
1. In the past there were some installers that contained packed files with extension unchanged or the archive contained only file data stream with file metadata provided separately.
2. We already know that it's not standard exepacking, so compact.exe may be only packer and some lib linked with ntfs driver may contain only exe unpacker.

It looks like corpo thinking strikes back. Maybe someone should check alternate data streams for such files?
Andrzej P. Wozniak
Polish subforum moderator
User avatar
LonerD
Senior Member
Senior Member
Posts: 381
Joined: 2010-06-19, 20:18 UTC
Location: Makeyevka, Russia
Contact:

Re: Search files compressed with compact.exe in Windows 10

Post by *LonerD »

In any case TC command cm_GetFileSpace doesn't work right in current reality and essentially useless.
TC shows phrase "Actual space used", but in fact space used is different from what we see.
Image: https://i.imgur.com/2dUIXSI.png

2ghisler(Author)
Probably it's time to fix command or at least change phrase.
"I used to feel guilty in Cambridge that I spent all day playing games, while I was supposed to be doing mathematics. Then, when I discovered surreal numbers, I realized that playing games IS math." John Horton Conway
Post Reply