[WCX] ZPAQ

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

[WCX] ZPAQ

Post by *milo1012 »

ZPAQ 1.5.5

A packer plug-in (wcx) for Total Commander for the ZPAQ format.
Based on zpaq 7.07.

See http://mattmahoney.net/dc/zpaq.html for more details.

In short: you can't delete files, only add new ones, but it's very efficient for detecting duplicate file fragments,
when you'd add several versions of a file to an archive.
Additionally it has very efficient compression modes (level 4/5), which in most cases creates even smaller files than 7-Zip in it's best mode, but can be quite slow.
The lower compression methods (up to 2) compress fast, and decompress very fast.

In default mode, when you open a zpaq archive in TC, you can only see either the latest update's files, or all "collected" files from all updates you made,
with each file showing it's newest version, just like you would expect from other archivers (zip, rar, ...).
This view depends on how files were added to an archive.

To use the unique journaling feature, i.e. when you want to get your old file versions, enable the 'Show all archive versions' option
(through 'Pack files' -> select 'zpaq' -> 'Configure'), so that you will be able to view all of the archive's updates.
For that you will now see unique dir names in the root of the archive, (either ascending eight-digit strings, or a date string representing the
date when that version was added to the archive) which will portray every update you made to it.
This means you can decompress your old file versions out of the archive, like you would with every other file.
This will also mean that you won't see all your files "collected" in one of
those single dirs. Use e.g. TC's branch view (Ctrl + B) for that,
or *NEW* the 'Show the default archive view in extra dir' option.

It's probably best to leave the default view enabled most of the time, when you
just add files to the archive, but enable it if you need an old version out of it.

For using more than just one compression/decompression thread, the x64 version is highly recommended.
While two parallel threads might initially work on 32-bit, they could fail to work until the end, crashing the plug-in.

Requires a SSE2 capable CPU, or use the non-SSE version from the plug-in archive.

Compression level 4 is very slow, and level 5 is extremely slow, so take care before starting them with huge file collections.



Download (32+64 bit, source available)
SHA1: 15e705fdb3bc2b5adce266404765a6cb7c7f59ab




Old version 1.5
Download (32+64 bit)
SHA1: da0b3a28f52232ef57a737b24ef2784739563490

Old version 1.4a
Download (32+64 bit)
SHA1: dcd2467e71fc3723fe7aff2549a45f2ba93c6a6c

Old version 1.4
Download (32+64 bit)
SHA1: 12cb3eb1af9f97dc1486d4f1e466955f2ec92dc7

Old version 1.3.6
Download (32+64 bit)
SHA1: 436bd88261d1f60fb6e195abaa62eb8c92767d32

Old version 1.3.5
Download (32+64 bit)
SHA1: f9944d76e135f22fc95e27a88b67aa3a1b9e8370

Old version 1.3
Download (32+64 bit)
SHA1: a51ec9cd17b6aa48990dc313b02e1710053f3405

Old version 1.2a
Download (32+64 bit)
SHA1: ad6d9611a3bcaf37de76a4b871239ff829ff9c71

Old version 1.2
Download (32+64 bit)
SHA1: 4f6f8c781cf67123a154ba990ebc96614b7a51fd

Old version 1.1c
Download (32+64 bit)
SHA1: 8f7018545393c9f50be0c6e98bffb2bed9087a7c

Old version 1.1b
Download (32+64 bit)
SHA1: b5ded9eeff897b6a5c41e448278c2447b8f2f97e

Old version 1.1a
Download (32+64 bit)
SHA1: ae381734c792daaca2a5a759a1c726467f82016d

Old version 1.1
Download (32+64 bit)
SHA1: fc2d678c9dee14401a1a7fe205610daaf9892f04

Old version 1.0
Download (32+64 bit)
SHA1: 18960761d4b44d84b20ed4a8704a1cf11dc73388

Old version 0.8
Download (32+64 bit)
SHA1: c5421cc6d7c270808a432b38ca6ce82c7edd6a9a




Please report bugs and give me some feedback.
Last edited by milo1012 on 2017-04-15, 20:54 UTC, edited 20 times in total.
TC plugins: PCREsearch and RegXtract
User avatar
xmeron
Junior Member
Junior Member
Posts: 90
Joined: 2009-06-17, 19:43 UTC

Post by *xmeron »

2milo1012

Thanks for this nice plugin, already installed, incremental backup/pack is amazing.

At the first beginning, I cannot find password option and want to feedback here to add this option. Only after read this thread, I knew the real trick, so maybe adding some notes is better, such as: Please go back to TC Packer Interface and tick Encrypt if you need password.

Then, hope to see language files soon.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

xmeron wrote:so maybe adding some notes is better, such as: Please go back to TC Packer Interface and tick Encrypt if you need password.
Well, plug-ins are actually supposed to use the settings from the "Pack" dialog box,
so that you can use them like TC's internal packers (zip, tar, ...).
But agreed: only a few actually use it, and e.g. the 7-Zip plug-in can accept a password it in it's own packer config.

In any case, I'll add a readme file in the next version.
xmeron wrote:Then, hope to see language files soon.
It is planned. I'll have to see if I'm able to make the important dialogs and messages translatable though.


Thx for testing!
TC plugins: PCREsearch and RegXtract
User avatar
xmeron
Junior Member
Junior Member
Posts: 90
Joined: 2009-06-17, 19:43 UTC

Post by *xmeron »

milo1012 wrote:only a few actually use it
For me, incremental backup is very useful.
milo1012 wrote:I'll add a readme file in the next version.
Readme file is necessary, but what I mean is, not all people will read readme file, so also put this notes inside zpaq pack window, maybe it's better.

And,
A question(or a bug?), after restart TC, TC will not show different versions/folders again inside apaq package, can only see all files, how to back to show these versions/folders again? Or just always show different versions/folders as default?
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

xmeron wrote:not all people will read readme file, so also put this notes inside zpaq pack window, maybe it's better.
I'll see what I can do.
xmeron wrote:after restart TC, TC will not show different versions/folders again inside apaq package, can only see all files, how to back to show these versions/folders again? Or just always show different versions/folders as default?
You're right: it's not intended to reset after restart, but that setting should be kept in the ini file.
Just enable "Show all versions" in the pack config, to show the versions again. (just confirm that dialog with "OK", and cancel the "Pack" dialog)
I'll fix it in the next version. Thanks for reporting!
TC plugins: PCREsearch and RegXtract
TW
Senior Member
Senior Member
Posts: 383
Joined: 2005-01-19, 13:35 UTC

Post by *TW »

thanks a lot, will give it a try. :)
licenced and happy TC user since 1994 (#11xx)
User avatar
hlloyge
Member
Member
Posts: 131
Joined: 2006-11-02, 23:14 UTC

Post by *hlloyge »

Is it possible for unpacker to show pack times inside archive, rather than generic 00000000 and 000000001 backup 'folder' names?
Something like 2015-09-12-15-23 and 2015-09-15-17-39?
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

hlloyge wrote:Is it possible for unpacker to show pack times inside archive, rather than generic 00000000 and 000000001 backup 'folder' names?
Something like 2015-09-12-15-23 and 2015-09-15-17-39?
Good idea!
I'll see for an option for that in the next version.
Should be ready soon.
TC plugins: PCREsearch and RegXtract
User avatar
xmeron
Junior Member
Junior Member
Posts: 90
Joined: 2009-06-17, 19:43 UTC

Post by *xmeron »

2milo1012
For saving much time, sometimes we just need to store/backup different versions of files especially when files are very huge, so can you please add compression level 0 in this plugin?

And, when pack huge files, seems TC don't show the progress from 0 to 100%.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

xmeron wrote:...can you please add compression level 0 in this plugin?
I already implemented it.
Wait for the next version.
xmeron wrote:And, when pack huge files, seems TC don't show the progress from 0 to 100%.
There's not much I can do, simply because I don't get more status updates from the zpaq compressor
than the standalone exe would (you can see it in it's console output).
I'll try to take another look at it at some later time, but I can't make any promises,
since it's something that the ZPAQ author should implement, and not me.
(it'd need a very complicated thread synchronization)
TC plugins: PCREsearch and RegXtract
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

New Version 1.0!
  • added compression method/level 0 support (store / dedupe only)
  • added 'Force add identical file versions' option
  • added 'Store drive letter / UNC name' option
  • added 'Show archive version names as timestamp' option
  • added 'Mask stored drive letters / UNC names' option
  • added 'Extract and set directory timestamps' option
  • added 'Warn about memory requirements' option
  • added 'Keep last used archive password in memory' option
  • added encrypt clue in option dialog
  • memory warning now also triggered before any pack or unpack operation, for safety reasons
  • memory warning now always considers available system memory
  • fixed: 'Show all versions' option wasn't saved correctly (reversed after TC restart)
  • fixed: all Windows forbidden characters for TC list are now masked
  • fixed: extremely slow archive listing for archives with a huge file set (50k+ files)
  • fixed: extremely slow extract operation when trying to set dir timestamps for archives with a huge file set (50k+ files)
  • added translation support ('zpaq.lng'):
    -> just create a new section with language name as section name and use
    the keys from the '(default)' section
    -> can set dialog font size for each language (between 8 and 11)
  • added German translation to zpaq.lng
  • added '(collecting file entries...)' message to progress dialog, before starting an unpack operation
  • added password encoding to pw dialog, reason: standalone zpaq uses UTF-8 when using "-key xxx", but ANSI encoding when using "-key" (empty, prompts for password)
  • several fixes and improvements
  • added non-SSE2 x86 plug-in version
  • released source
Check the first post for the new file.
TC plugins: PCREsearch and RegXtract
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6429
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Problems with Show all archive versions

Post by *Horst.Epp »

The following scenario:
1. Don't set "Show all archive versions"
2. Add modified file to an existing archive by selecting "Overwrite all older" in the TC dialog which comes up updating the archive.
3. Set "Show all archive versions" and try to view a file
It gives you an "Disk read error" and no archive versions are shown in the root of the archive.
4. Removing the option "Show all archive versions" allows to read the files without errors but still doesn't show archive versions in the root.

Setting "Show all archive versions" and making updates works fine
and gives no errors.

I'm using the latest x64 version under TC 8.52a
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: Problems with Show all archive versions

Post by *milo1012 »

Horst.Epp wrote:The following scenario:
1. Don't set "Show all archive versions"
2. Add modified file to an existing archive by selecting "Overwrite all older" in the TC dialog which comes up updating the archive.
3. Set "Show all archive versions" and try to view a file
It gives you an "Disk read error" and no archive versions are shown in the root of the archive.
4. Removing the option "Show all archive versions" allows to read the files without errors but still doesn't show archive versions in the root.
I can't reproduce it in the exact order that you described it,
but I guess you opened/viewed the archive before changing the option in step 3.

That's typical TC behavior.
Explanation: TC caches the content of an archive in memory,
and doesn't tell the plug-in to read it again when you would re-open it with Return/Ctrl+PgDown.
Now TC can't know that the archive changed it's internal structure.
Therefore opening it again will show the cached "old" paths, and trying to extract files,
which are not accessible by that old path any more, will trigger an error message.

One solution:
Opening any different archive, no matter if internals like zip/tar, or external plug-in ones, then leaving it immediately,
and opening your zpaq archive again, should trigger a re-read correctly, because TC's archive cache is overridden now.
I tested it to sometimes not work though.

I'm not sure what else you could do do empty that cache, besides TC restart of course.
Maybe Christian could give a hint.
(is there some ini option for that?)

Anyway, please try that and tell me if the issue is still there.


I don't think I can add some logic to detect it somehow,
because I couldn't distinct normal read errors from that "path problem".

But thanks for reminding me of that issue.
I'll add it to the readme for the next version.
TC plugins: PCREsearch and RegXtract
User avatar
hlloyge
Member
Member
Posts: 131
Joined: 2006-11-02, 23:14 UTC

Post by *hlloyge »

Could you change font color for unpack memory warning from yellow to something more readable, like red? I don't see anything :)
User avatar
Ovg
Power Member
Power Member
Posts: 756
Joined: 2014-01-06, 16:26 UTC

Post by *Ovg »

2hlloyge
+100500
It's impossible to lead us astray for we don't care even to choose the way.
#259941, TC 11.01 x64, Windows 7 SP1 x64
Post Reply