Packer plugins are slow for archives with many files.

[Terry Philips]

I'm having a problem with the design of the Packer Plugin interface:

The way the files in an archive are enumerated by repeatedly calling the ReadHeader-functions is very slow when archives contain many files (I'm talking 1 million files).

Every time you navigate into a subdirectory inside the archive Total Commander will re-enumerate all entries. That makes an archive with this many files VERY slow to navigate. If I want to navigate 7 levels down into the archive, I'll have to sit and wait for 5-10 seconds for every level. (The time is spend inside Total Commander, not inside my plugin - I have checked).

I can think of a couple of ways this could be fixed:

Without requiring changes to the plugin-interface Total Commander could retrieve the information only once, and sort it in a tree structure and keep it internally - but that would take quite a bit of memory, and it would still take a long time for the very first enumeration when opening the archive - probably even much longer since a tree for further navigation would have to be built in memory.

So a better way would be if the packer plugin could export a function called something like SetInternalPath which would take as a parameter a relative path inside the archive (or "\" or an empty string for the root). Then TC would call this function PRIOR to calling ReadHeader the first time. Then ReadHeader would only return files from the archive located in the specified sub-directory, and return E_END_ARCHIVE when there are no more files in that dir.

So, first TC would call SetInternalPath("\"), and then ReadHeader until it returns E_END_ARCHIVE, which would list all files and sub-dirs in the root of the archive. Then when the user navigates into SomeSubDir, TC would call SetInternalPath("\SomeSubDir"), and then ReadHeader until all files in the subdir are listed...

Would you consider supporting such an interface?

Kind regards,
Terry

[ghisler(Author)]

I'm aware of the problem. Currently all I can recommend is to write a file system plugin, and let the user choose the file to browse in some kind of configuration dialog. File system plugins return the content of one directory at a time.