[WCX] ZPAQ

[fcorbelli]

[*] speedup of the SHA calculation (which was discussed quite intense in the encoding forums), but again: not completed yet

Can you please link exactly... where?
Maybe I can get some useful ideas

[milo1012]

Can you please link exactly... where?
Maybe I can get some useful ideas

Probably nothing new to you, I mainly meant the discussions on encode.su, like here
https://encode.su/threads/456-zpaq-updates/page71
and linked there was
https://www.nayuki.io/page/fast-sha1-hash-implementation-in-x86-assembly
I made some benchmarks, and can confirm some slight speedups, but overall still mixed results when it comes to different machines.
Anyway, I'm trying the Intel SHA extensions too, but not sure if I can implement it w/o breaking the stock libzpaq. We'll see.

Already in zpaqfranz.

Yes, I know, but I meant more from TC's side, not in the CLI, i.e. one could implement a wdx interface with a field for file-hash, for quick comparision. I'm not sure if this is possible w/o breaking the wcx side.

You can store every kind of hash, and CRC too to detect SHA-1 collisions.

Tbh, the whole command line for zpaqfranz becomes more and more cryptic. I'm aware of the examples on the forums, but the man/help is too "undisclosed" by default for most users IMO.
I agree: it's a shame that zpaq doesn't get more attention, and I'm glad that zpaqfranz tries to improve things, but that puzzle box of different CLI switches doesn't really help here. In fact I tried to advertize it to some friends, and some just answered: "nah, too much of a hassle".

Here's, BTW, the deduplicator optimization

Thx, I'll take a look into that.

[fcorbelli]

Probably nothing new to you, I mainly meant the discussions on encode.su, like here
https://encode.su/threads/456-zpaq-updates/page71

Nothing essentially changes, the implementation in zpaq is already good enough, due to limited latency and "cache-like" buffering. You will find on encode.su the various threads where I have experimented with (about) all of them, up to 7zip's. You can get a good improvement using SHA instructions, if your CPU supports it. zpaqfranz does this way now.

Anyway, I'm trying the Intel SHA extensions too, but not sure if I can implement it w/o breaking the stock libzpaq. We'll see.

Of course you can.
I already do

Tbh, the whole command line for zpaqfranz becomes more and more cryptic. I'm aware of the examples on the forums, but the man/help is too "undisclosed" by default for most users IMO.

They are almost identical to zpaq.
About a drop in replacement, at 99% if you use the -715 switch and you do not rely on different output

I agree: it's a shame that zpaq doesn't get more attention, and I'm glad that zpaqfranz tries to improve things, but that puzzle box of different CLI switches doesn't really help here. In fact I tried to advertize it to some friends, and some just answered: "nah, too much of a hassle".

There are more switches because there are more commands. Many, many more
If you use it like zpaq, the switches are essentially identical.
If you know how to use zpaq, you can use zpaqfranz in exactly the same way, without having to use new switches or command
The only "real" difference is the t (test) instead of x -test (I really, really hate the x -test)

Code: Select all

zpaqfranz t something-to-be-tested.zpaq

instead of

Code: Select all

zpaq x something-to-be-tested.zpaq -test

That's all

(and, BTW, runs now on multiple files at time, zpaqfranz t *.zpaq)

If you happen to come across a situation where the operation is different vs zpaq, please let me know and I will fix it.
I spended a lot of time to maintain full backward compability, in file format AND CLI commands
The output is really different (much, much better) of course

If, on the other hand, you want to do things that are simply impossible on zpaq, from sfx modules, to VSS backups, to ZFS support, to additional hashes, to paranoid file testing, to RAM extraction, to stdin, to stdout... etc., "you" can simply ... read the manual or do zpaqfranz h h or ... read the examples (zpaqfranz h t for example) or... ask me
(yes, the embedded help does have "almost" every example for every command-switch)

At least it cannot be said to be a fork that only changed just a few LOCs

Here's, BTW, the deduplicator optimization

Thx, I'll take a look into that.

[fcorbelli]

Why there are just a "bit" of new things?

Because

zpaq 7.15 commands

Code: Select all

a          Add or append files to archive
l          List file(s)
x          Extract the file(s)

zpaqfranz new commands (on Windows)

Code: Select all

1on1       Delete folder2's files with same name/hash of folder1
autotest   Check for hidden errors after compiling from source
b          CPU benchmark, speed index in (yes!) franzomips
backup     Backup with hardened multipart
c          Compare one master dir against one or more slaves dir(s)
collision  Inspect archive for SHA-1 collisions
consolidatebackup Consolidate multipart backup
cp         Friendly file copy with ETA (resumable)
d          Deduplicate files inside a single folder WITHOUT MERCY
dir        A better dir (yes, Windows' dir)
dirsize    Show cumulative folder(s) size
dump       Inspect archive for SHA-1 collisions
f          Free disk space fills (=reliability test) or wipe (privacy)
find       Search file(s) with the wildcards
fzf        List archive filenames 'plain-and-dirty'
g          Windows C: archiver from a shell without admin rights
gui        Windows text-based GUI (listing-extraction)
e          Extract the file(s) in the current folder
i          File (archive) information
isopen     Check if a file is open (by other software)
k          Kill (delete) everything not in the archive (RISKY!)
last       Get the last multipart filename
last2      Compare last2 rows
m          Merge (consolidate) multipart archive into one
n          Decimate (keeping the newer X) older files
p          Paranoid test (slow, a lot of RAM needed)
password   Change/remove password of single archive (no multipart)
pause      Halt script execution until timeout expires or keypress
q          Windows archive of C: with VSS
r          Robocopy one master to multiple slave folders
rd         Remove hard-to-delete Windows' folder (ex. path too lengthy)
redu       Inspect archive for SHA-1 collisions
rsync      Delete rsync's dangling temporary files
s          Get dir(s) size, return free disk space
sfx        Create SFX module (with encryption support)
sum        Calc hash/checksums, find duplicated files
t          Test archive integrity
testbackup Multipart hardening
trim       Trim .zpaq archive from the incomplete transaction
utf        Convert filenames to Latin, fix too long filenames etc.
v          Verify an archive (against filesystem)
versum     Hashdeep-like double check of hashes
w          Very big files chunked extraction/test
z          Remove empty directories

If you want to discover the switches...
zpaqfranz h franz

But the best choice is always the examples

[milo1012]

Thanks for the explanations. Yes, I'm aware that you can get examples from CLI and get some kind of explanation for all switches. But that's the point: the zpaq CLI already is not that much of a beauty.
In fact, what bugs me the most is listing: I was never sure if the CLI shows me an archive listing with or w/o merging the local file path or not. And yes, this is marked as "external" files, but even this basic features is not what I'm used to from other CLI archivers. Personally I want a program to never mix archive content with local file system content, I'll want do that on my own, but I guess that comes with the nature of being more a "backup" or snapshot program, for which it works great of course.
Same with extraction: never really sure which files are being overwritten and which not. Maybe I'm a bit paranoid at that point, but I always avoid calling such things over CLI directly, but use a temporary location first.

But anyway, I meant ZPAQ as itself is already a "need to get used to it" hurdle, which (sadly) many people seem to avoid.

Of course you can.
I already do

I meant the STOCK libzpaq which isn't merged into a single CPP file now. But sure, it will work somehow.

[fcorbelli]

1) extraction
Use zpaqfranz with -longpath
This will make the behavyour... 100% predictable
I suggest to read this
https://github.com/fcorbelli/zpaqfranz/issues/86
Of course the default is just like zpaq's
That's all

2) if you want I can patch the very libzpaq of your plugin for HW accelerated SHA (if supported)
Not a big deal

3) zpaq is not a backup software. Not at all

[fcorbelli]

After looking at the plug in... it is a virtually complete rewrite
WOW

Updating it to version 7.15 is already a half disaster, for zpaqfranz I'd say even worse
It would probably take me less time to make a GUI with Borland C++ (does it still exist?)

[nsp]

@fcorbelli using WCX is a way to have a common gui for all archives as it is handled by the file manager. So this is not a matter of having a dedicated gui for each archive pgm. Currently the main usage we have of zpaqranz is to have in a compressed file a complete history like using git and commits in a single file for data files or sources. We even have people that use it automatically for each git release to distribute sources keeping track of versions. Having it under total commander as wcx is a very good way to explore the archive first with the latest version only (like any other archive file) and advanced view to get track of version. WE mostly use the command line to create archive and once the archive is distributed wcx or command line to "explore" it. ZpaqFranz is really a big step in term of performance and versatility over original ZPAQ. We have users who did not yet made the switch because this WCX do not handle new archive correctly with newer option...
So it would be great if you could provide a kind of libzpakfranz or helps to handle directly or through instrumented command line new archives.
anyway for all your work !

[fcorbelli]

The total commander zpaq plugin is a masterpiece, indeed
Sadly it is a... rewrite
Therefore it is indeed possible to backport some new functions
Like multipart checked (aka: backup command) archives

But why?
Because it runs with storagebox
Aka 3,90 euro/month/terabyte of cloud backup (yep, I am a senior freebsd user)

If you do not use rsync-based backups this function is almost useless
If you do, it is very very precious (aka: unique)
Same for g, aka Windows VSS backups without programs

Or the -longpath (aka filenames longher than 255)

There is even a specific command to rd folders. Longpath folders, that cannot be deleted with a straight rd something /s

Soneone really cares of SHA1 collision? Maybe not many people

Therefore, instead of try to make a new zpaqfranz plugin (very hard) some improvement in the current one seems more...doable

[Brumm]

Soneone really cares of SHA1 collision? Maybe not many people

We exist.

[Horst.Epp]

Hello Franco
do you have an English language version of your Pakka tool ?
I have the x64 version build 34.2

[fcorbelli]

Soneone really cares of SHA1 collision? Maybe not many people

We exist.

Touchè
I am working on splitting into pieces of a fixed size, it is worse than a minefield but OK

[fcorbelli]

Hello Franco
do you have an English language version of your Pakka tool ?
I have the x64 version build 34.2

In the "Impostazioni" tab you will find a "Italian interface" checkbox
Turn off and voilà (+ o -) english will be selected (and a .ini file get written)
Yep, I should make english as default, but you are the VERY FIRST user reported in just about 10 years
https://encode.su/threads/1860-PAKKA-(ZPAQ-s-Win32-quot-versioned-quot-unpacker)?highlight=pakka

WOW, about 10 years of "fighting" with zpaq
Here's my very first enhancement, SPAKKA (this will become the standard multipart archive)
https://encode.su/threads/456-zpaq-updates?p=36202&viewfull=1#post36202

SPACCA means in Italian SPLIT, BREAK, not PACCA

[Horst.Epp]

I already have the "Italian Interface" switched off
but there are stiil Italian descriptions and buttons.

[fcorbelli]

I already have the "Italian Interface" switched off
but there are stiil Italian descriptions and buttons.

The pakka.ini is present?