[WCX] ZPAQ

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: sheep, Hacker, Stefan2, white

Post Reply
User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-10-30, 02:28 UTC

Hacker wrote:Which two concepts do you mean?
Either showing time stamps only for multiple version of the same file (same file path), or all files in general (meaning also files showing up only once in the archive).
The first one will slightly slow down opening an archive with many files, but since ZPAQ needs a modern/powerful system anyway, I think I'll stick to it.
And I think I'll stick with a user configurable timestamp and not showing/allowing Windows forbidden chars, as we have enough Unicode chars (like this) to customize the timestamp and making the probability of generating duplicate filenames (name collisions) negligible.

Hacker wrote:Well, without such an option, I really don't have any specific preference, can be as it is now?
Well, I wanted to make sure we both understand that such option would only work in connection with the 'Show all archive versions' option, otherwise it wouldn't make much sense.

Hacker wrote:2. Never create filenames with timestamps appended
...
So like we have it now, with the version sub-dirs ('Show all archive versions')? Or do you mean simply merging the version dirs, so that we'd have the same filename all over again in a dir?
This means, listing in TC

Code: Select all

...
c:\Backup.zpaq\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument2.docx
...
will work, but extracting a specific "ImportantDocument.docx" wouldn't, cause TC (the wcx interface) can't request a specific file that way and would always ask for the first entry with such path (no way to know what the user requested for real).

Hacker wrote:- problems if the filenames are really long and we reach the 255 char limit
I think I simply inform the user with a message box if such case is detected. In practice, such long names are not w/o troubles anyway, and the user could still re-customize the timestamp to make it a bit shorter. I might even make the timestamp position optional, like

Code: Select all

$n_≪$Y$M$D$h$m$s≫
$Y$M$D$h$m$s-$n
$n,(%v)
...
(%n being the filename)

Hacker wrote:3. Only append timestamps to filenames in case multiple versions of the same file are extracted
How is that supposed to work? Like I said earlier, when merging all version dirs, TC can't request an individual file when the same file is listed multiple times with the very same path in the archive. And of course, when overwriting existing files, TC will ask the user itself for what to do, it's nothing I could intercept or bias with the plug-in.

Hacker wrote:Concerning files that were deleted and are not present in newest versions of the backup I would not treat them any differently - consider the newest version as the current one.
If I simply merge the version dirs, they would show up anyway. I'd use the same rules as for any other file set.
But what do you mean exactly anyway? Having only "older" file versions the timestamp appended

Code: Select all

c:\Backup.zpaq\ImportantDir\ImportantDocument.docx      (being the latest version)
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-08-10 19:33:02>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-07-16 15:21:31>.docx
or all files get the timestamp?
That's exactly why I asked for how a sample archive structure would look like in TC with/without the option.


So all in all I will try to make it work as described: Merging the version dirs (i.e. removing them and moving all file paths to the base of the archive, like with the normal/non-version view)), all files with multiple version will have the customizable timestamp appended, warn user if some files are exceeding 255 chars due to the timestamp when opening the archive (no matter if for listing or extracting).
TC plugins: PCREsearch and RegXtract

User avatar
Hacker
Moderator
Moderator
Posts: 11381
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Post by *Hacker » 2016-10-30, 10:21 UTC

milo1012,
I wanted to make sure we both understand that such option would only work in connection with the 'Show all archive versions' option, otherwise it wouldn't make much sense.
Agreed, but please do not show the top level "timestamp-dirs" for each version when this new view is activated, this would defeat the purpose.

I think there is a small misunderstanding about this section which I wrote:
I'd love to have three options for filenames upon extraction:

1. Always create filenames with timestamps appended
- no problems with extracting several versions of one file
- problems if the filenames are really long and we reach the 255 char limit

2. Never create filenames with timestamps appended
- when extracting several versions of one file the overwrite dialog would be presented
- no problems with filename length
- you get the original filenames and don't need to use the MRT to remove the timestamp

3. Only append timestamps to filenames in case multiple versions of the same file are extracted
- IMHO the best of the two above mentioned options
This whole section above is about how files should look like when created on disk after extraction, not about how they should be shown when browsing the ZPAQ archive. (I do not know if the WCX interface allows for such a thing, though.)
Can you please reread the section with this in mind?
That's exactly why I asked for how a sample archive structure would look like in TC with/without the option.
OK, so, the default view would be like this:
[ ] Do not show timestamp for newest version
[ ] Do not show timestamp for files with only one version

Code: Select all

c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-09-18 12:32:11>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-08-10 19:33:02>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-07-16 15:21:31>.docx
c:\Backup.zpaq\ImportantDir\OnlyOneVersion <2016-10-02 04:18:59>.docx
[x] Do not show timestamp for newest version
[x] Do not show timestamp for files with only one version

Code: Select all

c:\Backup.zpaq\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-08-10 19:33:02>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-07-16 15:21:31>.docx
c:\Backup.zpaq\ImportantDir\OnlyOneVersion.docx
[x] Do not show timestamp for newest version
[ ] Do not show timestamp for files with only one version

Code: Select all

c:\Backup.zpaq\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-08-10 19:33:02>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-07-16 15:21:31>.docx
c:\Backup.zpaq\ImportantDir\OnlyOneVersion <2016-10-02 04:18:59>.docx

[ ] Do not show timestamp for newest version
[x] Do not show timestamp for files with only one version

Code: Select all

c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-09-18 12:32:11>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-08-10 19:33:02>.docx
c:\Backup.zpaq\ImportantDir\ImportantDocument <2016-07-16 15:21:31>.docx
c:\Backup.zpaq\ImportantDir\OnlyOneVersion.docx
Please let me know if it is a bit clearer now.

Thank you
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-10-31, 12:37 UTC

Hacker wrote:OK, so, the default view would be like this:
...
Thanks. That's how I understood your suggestion and how it will work. And these two "sub-options" for the timestamps should be viable as well.
But for the sake of completeness: I also asked about how the archive structure would look like in TC without the overall option, just to make sure that we're both talking about merging the version dirs ("timestamp-dirs", if you enabled the custom string)
So I guess sth. like

Code: Select all

c:\Backup.zpaq\...other files...
c:\Backup.zpaq\2016-10-02-05-18-59\ImportantDir\OnlyOneVersion.docx
c:\Backup.zpaq\2016-09-18-13-32-11\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\2016-08-10-20-33-02\ImportantDir\ImportantDocument.docx
c:\Backup.zpaq\2016-07-16-16-21-31\ImportantDir\\ImportantDocument.docx
c:\Backup.zpaq\...other files...
(assuming each backup is done exactly one hour after the file creation)
Hacker wrote:but please do not show the top level "timestamp-dirs" for each version when this new view is activated
Sure.
Hacker wrote:Can you please reread the section with this in mind?
I see and I did, but this doesn't change the approach I have in mind


milo1012 wrote:So all in all I will try to make it work as described: Merging the version dirs (i.e. removing them and moving all file paths to the base of the archive, like with the normal/non-version view)), all files with multiple version will have the customizable timestamp appended, warn user if some files are exceeding 255 chars due to the timestamp when opening the archive (no matter if for listing or extracting).
So with adding the two "sub-options", would this be what you had in mind?
TC plugins: PCREsearch and RegXtract

User avatar
Hacker
Moderator
Moderator
Posts: 11381
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Post by *Hacker » 2016-10-31, 13:13 UTC

milo1012
I also asked about how the archive structure would look like in TC without the overall option, just to make sure that we're both talking about merging the version dirs ("timestamp-dirs", if you enabled the custom string)
So I guess sth. like
Yeah, pretty much like that, depending on the custom timestamp string the user defined (and without the double backslash you added there to check if I pay attention ;) ).
I see and I did, but this doesn't change the approach I have in mind
Not for browsing the archive but the section was rather about if it is possible for the extracted file to have a different name than is shown in the archive. I mean, can I see "ImportantDocument <2016-09-18 12:32:11>.docx" in the archive but get "ImportantDocument.docx" upon extraction? And also have some options regarding this (as outlined in the section)?
So with adding the two "sub-options", would this be what you had in mind?
Not precisely, since I wonder if we optionally (as described in the section regarding names of extracted files) could skip the WYSIWYG approach and while seeing the timestamps in the file names in the archive, not have the files named with timestamps upon extraction.

Thanks
Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-10-31, 14:39 UTC

Hacker wrote:Not precisely, since I wonder if we optionally (as described in the section regarding names of extracted files) could skip the WYSIWYG approach and while seeing the timestamps in the file names in the archive, not have the files named with timestamps upon extraction.
One thing at a time. I will try to implement the solution as I described so far, and after that is working I will test the practical consequences of using the original filename when extracting and how the options for this could look like. But as we discussed earlier, with such approach, neither TC nor the plugin will show warnings about overwriting existing files. So it's the user's responsibility. Sure, I could go beyond the pure wcx interface and check if files are existing prior to extraction and show my own message box for overwrite confirmation, but this is really sth. I prefer not to do, as it's a lot of work and the message box can't be purely modal that way.
TC plugins: PCREsearch and RegXtract

User avatar
Hacker
Moderator
Moderator
Posts: 11381
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Post by *Hacker » 2016-10-31, 14:59 UTC

milo1012,
I will try to implement the solution as I described so far, and after that is working I will test the practical consequences of using the original filename when extracting and how the options for this could look like.
Ah, sure, OK. Let's see then. :)

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-11-21, 06:33 UTC

New Version 1.5!
  • tadded customizable "-until" listing option for archives, working along with the Show all archive versions option, by providing "dummy" files with all version dir names, located in a custom named sub-dir in the archive's root:
    -> extracting one of such dummy files will remember the underlying version choice in the plug-in memory and apply the "-until" option (same as standalone zpaq) the next time TC opens the archive (opening another archive before that might be necessary, due to TC's archive content cache)
    -> greatly helps viewing/extracting an older archive state with the full dir structure, instead of only the selected file versions belonging to the archive update in question and missing files and dir structures taken along from previous updates, either by using the Show the default archive view in extra dir option or by disabling the Show all archive versions option before opening the archive again
    -> will not "rollback" the archive state when used in conjunction with adding files to an existing archive (doesn't apply the "-until" option then)
  • added optional "full archive state" extraction:
    -> when trying to extract/copy nothing but a complete version dir out of a Show all archive versions listing, the plug-in will ask if it should extract the complete archive state in the selected version instead of only the selected file versions belonging to the archive update in question and missing files and dir structures taken along from previous updates
    -> may work in conjunction with the Show the default archive view in extra dir option, but only when none of the individual files from the targeted version move to the extra dir
  • added optional (and limited) support for individual file extraction (PK_CAPS_SEARCHTEXT), which will enable TC to:
    1. search archive content for text in the main search function (Find files)
    2. compare archive's file content in the Synchronize dirs function
    -> can be VERY slow for archives with a lot of files and created with larger block sizes and/or higher compression modes, as for each file you'd possibly read and decompress the major part of an archive block all over again, since there is currently no way to optimize the order of decompression requests
  • added option to ALWAYS store full (absolute) paths when packing, and not the path structure TC is requesting
    -> might help in cases like using different base dirs for different archive updates, although the same file set was meant to be added to the archive
  • added: for the x64 plug-in the CPU information in the config dialog will now additionally show the number of physical/real CPUs VS the (former) simple number of CPUs reported by the OS (for identifying SMT/HTT processor configuration)
  • added tooltips with additional information for some elements in the config dialog
  • when extracting files due to directly viewing (F3) or editing (F4) a file from inside an archive listing, and when text search in archives is enabled, the progress dialog might show intermediate steps (depending on the file size)
  • fixed: filenames with characters above U+FFFF were not valid for pack operations and skipped; listing/extracting archives with such filenames recoded the characters to an improper sequence of replacement characters (reason: standalone zpaq uses a non-standard UTF-8 encoding scheme for codepoints above U+FFFF)
  • fixed: the Mask empty archive versions option didn't mask some empty dirs when used in conjunction with the Show the default archive view in extra dir and Include deleted files option
  • fixed possible buffer overflow when listing archives with file path lengths above TC's current limit (1023 characters)
  • added workaround for archive entries containing relative paths and starting with a current directory period, e.g. "./dir1/file...", where TC has problems extracting the archive (when not showing all versions)
  • the password caching will now also take the file creation timestamp into account
  • added encoding protection key for the pkplugin.ini file, to prevent external editors from saving it as full UTF-8 (start BOM)
  • 32-bit default plug-in (SSE) compiled with different settings; may result in a slight speed increase for the faster compression methods
  • some code improvements
Check the first post for the new file.
TC plugins: PCREsearch and RegXtract

reg2s
Junior Member
Junior Member
Posts: 9
Joined: 2016-05-23, 19:26 UTC

Post by *reg2s » 2016-11-21, 08:20 UTC

Needed some time to realize, that both "Show all archive versions", "Show the default archive view" and "Show dir for customizable "-unti"" options must be on for "-until" functionality to work.) Finaly, there is possibility to see full previous archive states without command-line zpaq utility.

Also "full archive state" unpacking and content search are very helpfull. Thank You very much for the great update.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-11-22, 14:58 UTC

reg2s wrote:Needed some time to realize, that both "Show all archive versions", "Show the default archive view" and "Show dir for customizable "-unti"" options must be on for "-until" functionality to work.)
Yes, or you completely switch off "Show all archive versions" before opening the archive again.
I considered showing the custom -until dir in the normal archive view as well, but I really want to keep all "special" views bound to showing all versions for now, and keeping the normal view, well, a normal view. Plus I think that it's still rather unfrequented that you really need a full view of an old archive state.
TC plugins: PCREsearch and RegXtract

reg2s
Junior Member
Junior Member
Posts: 9
Joined: 2016-05-23, 19:26 UTC

Post by *reg2s » 2016-11-22, 15:23 UTC

Well, this makes sense. But I'm sure I will use both modes. Thank You again.

Skif_off
Member
Member
Posts: 118
Joined: 2013-09-30, 13:13 UTC

Post by *Skif_off » 2016-11-23, 23:44 UTC


User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-11-24, 17:28 UTC

2Skif_off
Many thanks!
As usual, I won't release a new version just for a language update.
In the meantime you can download the combined language file here:
http://wincmd.ru/files/9924355/zpaq_lng_15.zip
TC plugins: PCREsearch and RegXtract

User avatar
petermad
Power Member
Power Member
Posts: 8964
Joined: 2003-02-05, 20:24 UTC
Location: Valsted, Denmark
Contact:

Post by *petermad » 2016-12-07, 22:51 UTC

2milo1012
Opdated Danish language: http://madsenworld.dk/tcmd/zpaq_lng_15.zip
License #524 (1994)
Danish Total Commander Translator
TC 9.5b3 32+64bit on Win XP 32bit, Win 7, 8.1 & 10 (1909) 64bit, 'Everything' 1.4.1.935 (x64)
TC 3.0b12 on Android 6.0
Get: Extended Total Commander Menus | PHSM-Calendar

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2016-12-08, 11:54 UTC

2petermad
Thanks a lot!
I've updated the combined language file (link in my previous post).
TC plugins: PCREsearch and RegXtract

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2017-04-15, 20:45 UTC

New Version 1.5.5!
  • fixed: the progress dialog randomly stopped getting any updates whatsoever in the middle of a long pack/unpack process, especially in the x64 version (reason: QueueUserAPC() starts to fail for unknown reasons)
  • progress dialog code overhaul: dialog will now get updates every ~0.1 seconds (formerly 0.5 seconds), which allows faster response time for bg/fg switching; no more superfluous delay when creating or extracting (very) small archives
  • fixed: the custom "-until" listing option was accessible in config dialog even with Show all archive versions disabled
  • fixed: wrong memory usage calculation in config dialog when switching from compression level >= 2 to 0/1
  • all message boxes asking the user are now in Yes/No/Cancel style, to be able to cancel them quickly with esc key
  • reimplemented basic listing/extraction support for old streaming (non-journaling) zpaq archives:
    -> will show all file sizes as zero bytes and all file timestamps as 01/01/1980
    -> extracting such files will set the file timestamps to the then current system time
  • added support for an own (separate) settings file for the plug-in:
    -> when creating a file "zpaq.ini" in the same dir/location where the main plug-in file resides, all settings will be saved to and loaded from that very file; otherwise the TC default file ("pkplugin.ini") will be used
  • added support for the wcx interface omitting a PackSetDefaultParams() function call before any other operation (default options will take place; might help for tools like cmdTotal, though it will quit after 300 seconds due to extraction taking place in CloseArchive())
  • enabled Large Address Aware flag for 32-bit plug-in (probably no use for now, as TC itself is not flagged)
  • Russian translation update (by Skif_off)
  • Danish translation update (by petermad)
Check the first post for the new file.

Just a minor update, because this progress dialog bug bothered me.
New major feature(s) maybe later this year.
TC plugins: PCREsearch and RegXtract

Post Reply