[WCX] ZPAQ

[milo1012]

Could you change font color for unpack memory warning from yellow to something more readable, like red? I don't see anything

Well, red is already used for the "real" warning.
But sure, I can make both colors configurable.

[Horst.Epp]

The following scenario:
1. Don't set "Show all archive versions"
2. Add modified file to an existing archive by selecting "Overwrite all older" in the TC dialog which comes up updating the archive.
3. Set "Show all archive versions" and try to view a file
It gives you an "Disk read error" and no archive versions are shown in the root of the archive.
4. Removing the option "Show all archive versions" allows to read the files without errors but still doesn't show archive versions in the root.

I can't reproduce it in the exact order that you described it,
but I guess you opened/viewed the archive before changing the option in step 3.

That's typical TC behavior.
Explanation: TC caches the content of an archive in memory,
and doesn't tell the plug-in to read it again when you would re-open it with Return/Ctrl+PgDown.
Now TC can't know that the archive changed it's internal structure.
Therefore opening it again will show the cached "old" paths, and trying to extract files,
which are not accessible by that old path any more, will trigger an error message.

One solution:
Opening any different archive, no matter if internals like zip/tar, or external plug-in ones, then leaving it immediately,
and opening your zpaq archive again, should trigger a re-read correctly, because TC's archive cache is overridden now.
I tested it to sometimes not work though.

I'm not sure what else you could do do empty that cache, besides TC restart of course.
Maybe Christian could give a hint.
(is there some ini option for that?)

Anyway, please try that and tell me if the issue is still there.


I don't think I can add some logic to detect it somehow,
because I couldn't distinct normal read errors from that "path problem".

But thanks for reminding me of that issue.
I'll add it to the readme for the next version.

Only a TC restart solved the problem for me.
There was no archive open while change the setting !
Anyway, not a big problem.
The speed of archiving is impressing fast, at least with mode 1 or 2.

[krasusczak]

Hi I have found one problem/error when you change language to Deutsch the changed & supported variable in pkplugin.ini is not "Deutsch" but "?Deutsch" & works only with this second one, exactly thesame is happens when you try to use other language like "?Spanish" etc.

this "?" may be invisible in some text editors


btw. Thanks for this plugin looks very interesting

[milo1012]

Hi I have found one problem/error when you change language to Deutsch the changed & supported variable in pkplugin.ini is not "Deutsch" but "?Deutsch" & works only with this second one, exactly thesame is happens when you try to use other language like "?Spanish" etc.

this "?" may be invisible in some text editors

Yes, this is by design.
I use the simpleini library in order to use UTF-8 in INI files (plus to have some additional advantages).
This means that you can also use most Unicode characters as section names (which will be the language name you can see) in the zpaq.lng file.
But the pkplugin.ini file is in ANSI encoding. So in order to clearly identify and insert the language string, I insert a Byte Order Mark before it.

I actually didn't expect users to manually change the language string in the pkplugin.ini file (use the config dialog for that).
So if you manually remove the BOM characters, I can't read it as UTF-8, and the language will be reset to default.

[batchman61]

Hi,

regarding archive content cache it could be worth a try to:
close zpaq archive, run cm_UnloadPlugins, reopen archive.

A question regarding branch view:
Is it possible to implement an equivalent of cm_DirBranchSel (Shift+Ctrl+B) to list marked files and folders including sub-folders and -files only ?

Thank you for this plugin !

[milo1012]

Is it possible to implement an equivalent of cm_DirBranchSel (Shift+Ctrl+B) to list marked files and folders including sub-folders and -files only ?

TC doesn't send any more requests to the plug-in, once the content of the archive is loaded.
So it's something that Christian should implement; nothing I can do.

But it's a good point.
I didn't realize that this doesn't work in archives, just like cm_LeftDirBranchSel and cm_RightDirBranchSel.
Not sure if it falls in the bug or a missing feature category.
(and even things like TWinKey wouldn't help here)

regarding archive content cache it could be worth a try to:
close zpaq archive, run cm_UnloadPlugins, reopen archive.

The archive content is cached by TC, not by any plug-in, so it doesn't help.
But like I said, opening and closing a different archive to clear the cache works for me quite well.

[kulmegil]

Looks like atm it doesn't handle extracting ZPAQ archives separated into multiple files (? or * syntax), does it?
A shame since it's a nice zpaq feature, very useful for backups.

I does support creating ones.

[milo1012]

Looks like atm it doesn't handle extracting ZPAQ archives separated into multiple files (? or * syntax), does it?

I'm not exactly sure what you mean.
(Remote) Multi-part archives?

I know that they can be created since 6.5 or so, but I already stated in the readme that I didn't test them yet.
Personally I don't see the advantage of it, compared to a single file, even for backup purposes.
I'm not sure if the TC interface would handle separate archives, a lot of logic is needed there.
But I will take a look at it at a later time of course.

I does support creating ones.

What do you mean?



Update:
A quick test revealed:
You can extract of course part001, part002, ... individually, if they don't rely on data from different parts, which should be fair enough for now.
But when trying to extract from the index (part000), it will fail.

And of course, trying to add to part001, part002, ... will not update the index, but at least it doesn't destroy any data.
FYI: This is even possible in the standalone program.
But: a concatenation of such "updated" parts will result in a broken archive.
I wonder why the standalone program doesn't detect when you try to update a single part.
That's the reason why I don't like that multi-part concept: you add a lot of possibilities for trouble.

I could try to detect multi-part archives and replace the numbers with wildcards internally,
so that opening the index will at least try to load part1, etc.,
but the problem is that the user may use any arbitrary number scheme, e.g.
001part, p01art, part0000001, ...
I'm not sure if it's possible to detect all cases.

As for creating multi-part archives from the scratch, I'm not sure how to handle it.
I could let the user choose the position and maybe length for the wildcard.

[kulmegil]

You can extract of course part001, part002, ... individually, if they don't rely on data from different parts, which should be fair enough for now.
But when trying to extract from the index (part000), it will fail.

This looks like a bad idea. I would probably end up overwriting some files with their older versions and ended up with some previously deleted ones.
(opening index would give me actual state of archive but won't let me extract anything. Parts contains only new/changed files added in given update)

And of course, trying to add to part001, part002, ... will not update the index, but at least it doesn't destroy any data.

When I create/add to archive I can edit archive filename and manually insert wildcards where needed. Unfortunately I cannot do that when browsing / extracting.

I could try to detect multi-part archives and replace the numbers with wildcards internally,
so that opening the index will at least try to load part1, etc.,
but the problem is that the user may use any arbitrary number scheme, e.g.
001part, p01art, part0000001, ...
I'm not sure if it's possible to detect all cases.

You can parse archive filename looking for number-like strings (incl. leading zeros). Preferably backwards since the part number is likely last number in naming scheme. Replace all digits with zeros and check same dir for filename. If exist assume it's multipart archive and replace digits with "?".

It's not perfect because in a rare case it's possible to run into a user that stores different, independent archives that would match this naming scheme. Unlikely, people don't start counting from "00". But maybe ZPAQ lib includes some method to call to verify if the files belong to the same archive?

I think support for such important feature, in case of ZPAQ archives, is far more important and imho justifies small scarifies. Can always be opt-in if that's a concern.

[milo1012]

You can extract of course part001, part002, ... individually, if they don't rely on data from different parts, which should be fair enough for now.
But when trying to extract from the index (part000), it will fail.

This looks like a bad idea. I would probably end up overwriting some files with their older versions and ended up with some previously deleted ones.
(opening index would give me actual state of archive but won't let me extract anything. Parts contains only new/changed files added in given update)

What are you talking about? I mean extraction, not writing to it.
If you willingly extract from such individual parts, it is your responsibility where you extract them, just like with any other archive.
And you can see the file date anyway, and even the archive version/date string in the root, if you enabled that option,
plus the archive version from the file name.
I just stated that it is/was possible to still get your data, if it doesn't rely on data from different parts.

Anyway, I've already given it some thoughts.
I'll try to detect multi-part archives and make it an option, for the next version.
So if the user willingly created a normal/standalone file, like "myarchive000", it would fail adding to it or extracting from it with that option enabled, but should work normal when you unset it.

But maybe ZPAQ lib includes some method to call to verify if the files belong to the same archive?

Sadly not, and that's why I stated that I don't like that concept as it is for now.
In my concept you would insert some sort of mark in each individual part, which is normally skipped,
but which clearly identifies it as individual part when opening it, so that you can't manipulate those individual parts accidentally.
(like e.g. Rar does with multi-part archives)
But that isn't the case here, even in the standalone program.
Sure, the current state has the advantage that you can concatenate all parts and still get a working archive, but that's about it.

[batchman61]

Hi,

how did you implement Option "Force add identical file version" ?

It's a perfect fit for some of my automated backup requirements. As far as tested the plugin logically stores all Data with each version, physically only differences are stored.

Wasn't able to implement this behaviour using the standalone program. As of the docs, option
-force attempt to add files even if the last-modified date has not changed. Files are added only if they really are different, based on comparing the computed and stored SHA-1 hashes.

"added only if they really are different" is what I`ve seen in my tests.

How do you logically store all Data with each version ?

Thanks

[milo1012]

how did you implement Option "Force add identical file version" ?

It's a perfect fit for some of my automated backup requirements. As far as tested the plugin logically stores all Data with each version, physically only differences are stored.

Wasn't able to implement this behaviour using the standalone program. As of the docs, option
-force attempt to add files even if the last-modified date has not changed. Files are added only if they really are different, based on comparing the computed and stored SHA-1 hashes.

"added only if they really are different" is what I`ve seen in my tests.

How do you logically store all Data with each version ?

First of all: congratulations, you (indirectly) just found a bug

But first the explanation of the "Force add identical file versions":
Without it, a file is not even read at all, if the timestamp AND file size AND attributes (hidden, write-only, etc.) are identical to a file of the same name (and path) already found in the archive.
This might be dangerous, if a file content change doesn't affect the file size / timestamp / attributes.
(there are some applications which allow modifying a file's content without such change).
With that option enabled, the file's content is always checked for differences, so that you're definitely on the safe side if such situation occurs.

I think that explanation found on the Zpaq website and console program help is quite ambiguous, mostly because it doesn't mention the file size at all,
only the man page does (indirectly, because Windows file attributes contain the file size):

A change is an addition, update, or deletion of any file or directory in files or any of its subdirectories to any depth. A file or directory is considered changed if its size or last-modified date (with 1 second resolution), or Windows attributes or Unix/Linux permissions (if saved) and differ between the internal and external versions. File contents are not compared. If the attributes but not the date has changed, then the attributes are updated in the archive with the assumption that the file contents have not changed.

BTW, this means that a change can also be skipped if a timestamp changes only slightly (<0.5 s), because the base time resolution of the format is one second, but e.g. NTFS has 100 ns.


I should probably rename that option to sth. like "always compare identical files's content"




Concerning the bug:
First of all, it is also present in the standalone program.
If you enable the "-all" option together with the "add" command, files can't be compared by their paths any more,
because they are internally prefixed with that version number scheme.
So you will end up with the same file from the same path in the archive stored twice, although being completely identical, due to different internal paths.
I used that same code, and thought that version listing is disabled for "add" mode, so I have this problem too:
If you enabled "Show all archive versions", which is the equivalent to "-all 8", identical files are being stored again.

I already fixed it.
A new version should be ready soon.

[batchman61]

Hi,
thanks for the explanation regarding -force and -all.

Did a few tests using the standalone program and became aware of a glitch in option Show archive version names as timestamp.

zpaq64 add TARGET.zpaq SRCDIR -to "" -all 8 executed 3 times.

Option enabled shows a single timestamp (of SRCDIR) and each file 3 times.
Option disabled shows 00000001 - 3 and contained files as expected.

As of zpaq list the root folder of each add seems to have the timestamp of SRCDIR:

zpaq v7.05 journaling archiver, compiled Apr 17 2015
D:/wip/zpaq/tgt/test01_all8.zpaq: 3 versions, 15 files, 3 fragments, 0.002429 MB

- 2015-10-06 18:14:18 84 D 00000001/ +5 -0 -> 1241
- 2015-10-06 14:31:52 84 D 00000001/A/
- 2015-10-06 16:51:56 28 A 00000001/A/a1.txt
- 2015-10-06 16:51:56 28 A 00000001/A/a2.log
- 2015-10-06 16:51:56 28 A 00000001/A/a3.err
- 2015-10-06 18:14:18 84 D 00000002/ +5 -0 -> 594
- 2015-10-06 14:31:52 84 D 00000002/A/
- 2015-10-06 16:51:56 28 A 00000002/A/a1.txt
- 2015-10-06 16:51:56 28 A 00000002/A/a2.log
- 2015-10-06 16:51:56 28 A 00000002/A/a3.err
- 2015-10-06 18:14:18 84 D 00000003/ +5 -0 -> 594
- 2015-10-06 14:31:52 84 D 00000003/A/
- 2015-10-06 16:51:56 28 A 00000003/A/a1.txt
- 2015-10-06 16:51:56 28 A 00000003/A/a2.log
- 2015-10-06 16:51:56 28 A 00000003/A/a3.err

0.000252 MB of 0.000252 MB (15 files) shown
-> 0.000084 MB (9 refs to 3 of 3 frags) after dedupe
-> 0.002429 MB compressed.
0.078 seconds (all OK)

The manpage states for -all:
"The date shown on the root directory of each version is the date of the update."
which seems to be not perfectly true in the above example.

Maybe a version + timestamp pattern like
<VERSION>_<YYYY>-<MM>-<DD>_<HR>-<MN>-<SD>
can be helpful ?

[milo1012]

zpaq64 add TARGET.zpaq SRCDIR -to "" -all 8 executed 3 times.

First of all I can't recreate it with your command line.
No matter what I try, the timestamp always differs by at least one second.
That's what I've seen in all tests I did so far.

But anyway, why are you using "add" together with "-all", when I already explained that it's bugged, and not supposed to be used together?

I don't think that I should support such special case, because:
Normally the archiver uses unique timestamps for each version, which will advance at least one second each (base time resolution).
This is important if you'd use "add" in a batch operation.
Just try it w/o "-all", and you'll see that it might even create version dates lying in the future, depending how fast "add" works of course.
Therefore every sequential update will and needs to add at least one second to the timestamp.

So that situation you created invalidates the archive specs AFAIK.


Try your command line with single files and w/o "-all", and tell me if it still produces the same timestamp.
As I said, it works for me for any input.




Update:
I found the culprit:
the [face=courier]-to ""[/face] command creates such archives.

Not sure if it's even allowed to use an empty string.

Anyway, like I said, I really don't want to create a workaround for a baloney situation, that violates the specs.

[milo1012]

New Version 1.1!

• added optional support for NTFS Alternate Data Streams (storing and extracting)
  -> works for files AND dirs, but extracting for empty directories doesn't (TC limitation)
  -> checks target file system for ADS being available, before extracting
  -> for compression it needs Windows Server 2003 / Vista or higher, extracting works for all OSes
• added optional (remote) multi-part detection
  -> needs at least two part files for detecting multi-part w/o index part file
  -> asks user before packing to detected multi-part archive but no index part found
  -> single index part file always treated as multi-part (error message when trying to extract from non-index file accidentaly having the '...xxx000xxx' name)
  -> configurable minimum digit group length
• added config dialog options fpr ADS packing and extracting
• added config dialog option for multi-part detection (minimum number of digits, or disabled)
• fixed: when packing and 'Show all archive versions' enabled, files with the same paths were stored again (but deduplicated),
  even if they were identical to existing ones from an old version; reason: unique different internal paths due to prefixed version number scheme;
  behavior is now identical to standalone zpaq
• fixed: packing files with path lenght > 259 to an archive didn't work
• changed mem pre-warning color in config dialog to blue
• config dialog mem warning colors can now be set through ini file (RGB integer)
• changed: 'Language' entry in 'pkplugin.ini' is now also valid without BOM (when user removed it) when loading,
  but only if it transfers lossless to the UTF-8 section string found in the language file
• renamed 'Force add identical file versions' option to 'Force content compare for identical files'
• added Chinese simplified translation
• some internal optimizations

Check the first post for the new file.