[WCX] ZPAQ

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: sheep, Hacker, Stefan2, white

Post Reply
User avatar
Hacker
Moderator
Moderator
Posts: 11396
Joined: 2003-02-06, 14:56 UTC
Location: Bratislava, Slovakia

Post by *Hacker » 2017-04-15, 21:15 UTC

milo1012,
Thank you.

Roman
Mal angenommen, du drückst Strg+F, wählst die FTP-Verbindung (mit gespeichertem Passwort), klickst aber nicht auf Verbinden, sondern fällst tot um.

wwj402
Junior Member
Junior Member
Posts: 3
Joined: 2015-06-25, 13:36 UTC

Post by *wwj402 » 2017-07-10, 12:58 UTC

chinese language update
https://pastebin.com/1YjMFMr1

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2017-07-14, 17:41 UTC

2wwj402
Thanks a lot!

You can download the combined language file here:
http://wincmd.ru/files/9924355/zpaq_lng_155.zip
TC plugins: PCREsearch and RegXtract

ykhabins
Junior Member
Junior Member
Posts: 34
Joined: 2003-10-09, 13:12 UTC

ZPaq packer plug-in error: bad allocation

Post by *ykhabins » 2017-10-18, 17:59 UTC

Hello,

I am using ZPaq packer plug-in v.1.5.5
http://totalcmd.net/plugring/ZPAQ.html

TC version is 9.10RC3, 32-bit edition.
Windows 10, 64-bit, build 15063.632

Previously, in TC 9.0a, ZPaq was able to handle compression of 3GB files.
Now, with the TC 9.10RC?, Zpaq is failing to handle 800 MB files with the error: bad allocation

Please advise.

Message from moderator

Merged thread ZPaq packer plug-in error: bad allocation.

Hacker (Moderator)

User avatar
Horst.Epp
Power Member
Power Member
Posts: 3474
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Post by *Horst.Epp » 2017-10-18, 18:07 UTC

Not confirmed for the x64 version of the ZPAQ plugin under TC 9.10RC3
I tested with a 3.5GB file and it runs fast and with no problems.

Why do you use a x86 TC and plugins if you have an x64 OS ?
x86 TC and plugins have less memory available
and you also may have strange ZPQA settings.
Windows 10 Home x64 November 2019 Update, Version 1909 (OS Build 18363.476)
Intel(R) Core(TM) i7-4770 CPU @ 3.40GH, 16GB RAM
TC 9.50ß8 x64 / x86, Everything 1.4.1.959 (x64)

ykhabins
Junior Member
Junior Member
Posts: 34
Joined: 2003-10-09, 13:12 UTC

Post by *ykhabins » 2017-10-18, 18:15 UTC

TC 32-bit because I am using one plugin that is only available in 32-bit edition: hpg_ed

Again, previously, in TC 9.0a 32-bit edition, ZPaq was able to handle compression of 3GB files.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Re: ZPaq packer plug-in error: bad allocation

Post by *milo1012 » 2017-10-19, 18:06 UTC

ykhabins wrote:Previously, in TC 9.0a, ZPaq was able to handle compression of 3GB files.
Now, with the TC 9.10RC?, Zpaq is failing to handle 800 MB files with the error: bad allocation
Well, first of all: what were your ZPAQ compression settings (compression method, block size, number of threads)?
The default settings barely use more than 128 MB RAM when (one thread), so I doubt that you used these.

But for a possible explanation: The "bad allocation" message means that the plug-in wasn't able to allocate a large enough block of (heap) memory in the TC process address space. I can only guess that TC 9.10 somehow tends to use more heap memory than 9.0a, or, even more important, tends to fragment the process address space more than TC 9.0 does. Fragmentation is an important factor here: you can have 1,9 GiB of the 2 GiB¹ address space free, but if the active allocated 0.1 GiB are fragmented all over the space, leaving no free memory block larger than maybe 100 MB, you're probably out of luck for ZPAQ compression mode 1, or a block of 400 MB for mode 2, and so on.
I can only guess that something like this is the explanation.
But I can recommend VMMap to diagnose the cause of this.
It's actually ideal for such situations, since TC doesn't crash after this error. So just use the tool on the TC process directly after you see this message and look for how much and how large memory blocks you have available (and maybe compare this to TC 9.0).



¹ Sadly TC 32-bit is not flagged as Large Address Aware, so it can't use 4 GiB on a x64 OS, but is still limited to 2 GiB.
I think Christian somewhere explained that he doesn't want to enable this flag, but I can't seem to find the thread.
TC plugins: PCREsearch and RegXtract

User avatar
Horst.Epp
Power Member
Power Member
Posts: 3474
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Post by *Horst.Epp » 2017-10-19, 19:54 UTC

ykhabins wrote:TC 32-bit because I am using one plugin that is only available in 32-bit edition: hpg_ed

Again, previously, in TC 9.0a 32-bit edition, ZPaq was able to handle compression of 3GB files.
You can install the combined x86/x64 TC.
That solves your ZPAQ problem and you can easy access the 32bit plugin.
Windows 10 Home x64 November 2019 Update, Version 1909 (OS Build 18363.476)
Intel(R) Core(TM) i7-4770 CPU @ 3.40GH, 16GB RAM
TC 9.50ß8 x64 / x86, Everything 1.4.1.959 (x64)

ykhabins
Junior Member
Junior Member
Posts: 34
Joined: 2003-10-09, 13:12 UTC

Post by *ykhabins » 2017-10-19, 22:00 UTC

Hi milo1012,

My machine has 24 GB of RAM.

Settings:
- Compression method: 3
- Block Size: 64 MB
- Threads: 2 (max for 32-bit)

Everything else default settings.

And the error crashes TC.
TC needs to be restarted after the error.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2017-10-20, 05:40 UTC

ykhabins wrote:- Compression method: 3
- Block Size: 64 MB
- Threads: 2 (max for 32-bit)
Well, then it's obvious: these settings are too high for 32-bit. Actually I covered this very case in the readme file! With such settings, already a few MB more used by TC 9.10 will make the probability for a crash more likely. You should switch to one thread!

Your overall amount of RAM doesn't matter, a 32-bit program can never use more than 2/4 GB.

And BTW, you can still use VMMap, even if TC crashes: just don't confirm the Windows crash error message yet and start VMMap on the now frozen TC process anyway. You will probably see what I descibed: not enough and/or large enough memory blocks available.
TC plugins: PCREsearch and RegXtract

ykhabins
Junior Member
Junior Member
Posts: 34
Joined: 2003-10-09, 13:12 UTC

Post by *ykhabins » 2017-10-20, 19:57 UTC

Hi milo1012,

Based on your suggestion, I changed settings to 1 thread, and it worked.
Painfully slow, but worked.

So it seems that TC 9.10 (already RTM) is using more memory than 9.0a.

Thanks.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2017-10-21, 10:51 UTC

ykhabins wrote:So it seems that TC 9.10 (already RTM) is using more memory than 9.0a.
I made some tests now and can't find any differences in the memory consumption between 9.0 and 9.10. I tested on a blank TC installation. I can only guess that you use some additional plug-ins, or use Everything within TC, where the behavior might have changed in TC 9.10 and therefore allocates memory differently.

But still: even on my blank test installation of TC 9.0 I get this very crash with method 3 and two threads. So in any case you were quite lucky not getting a crash on your 9.0 installation.
That's the exact reason why I recommend one thread for 32-bit TC and made a distinct warning about this in the readme file - I mention it even in the description text on totalcmd.net and the beginning of this thread! I really wish that people would read those things more carefully...
TC plugins: PCREsearch and RegXtract

xpk
Junior Member
Junior Member
Posts: 4
Joined: 2015-08-28, 06:55 UTC

Post by *xpk » 2018-03-20, 16:51 UTC

milo1012 wrote: Plus I think that it's still rather unfrequented that you really need a full view of an old archive state.
I have just recently discovered ZPAQ and I'm trying to evaluate it alongside this great addon. I'd like to present my different opinion.

The important thing is the files, not the archives and their internal technicals. The archive is just a tool for backup and restore. The more transparent the archiver is, the better. This transparency and ease of use is achieved by using TC packer addons such as this, for which you have my gratitude.

  • - The whole deduplication thingy is important to have inside archive processing to save total archive space, but it is not relevant to the actual files and my primary interest is not to see the files without deduplicated files. If I backup all files, I want to SEE all those files in the archive for all the dates when I have backed them up and I want to be able to EXTRACT all those files.
    If I could have just one option

    Code: Select all

    Show full archive state for every version
    which would both show and consequently extract full snapshot that would be awesome.
    Then you could get rid of these 2 clumsy options:

    Code: Select all

    Show dir for customizable "-until" archive view
    Support full archive state extraction
    - The date of archiving is not important, the date of fileset snapshot is. I may obviously re-archive the files in the future using other archiver, but that is no reason to discard the dates when those files were first backed up. I need to be able to set the date to the fileset (zpaq index/version) or let the archiver set it by the newest file in the snapshot. This is even more important when there is no other anotation feature in zpaq like version comments. The only possible workaround - messing with the system time before adding each backup version is not exactly user friendly.

User avatar
milo1012
Power Member
Power Member
Posts: 1109
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 » 2018-03-21, 23:15 UTC

2xpk
I perfectly understand your points, but there are certain technical aspects to consider here.
I think I already mentioned one of the main reason for not doing this in the past:
milo1012 wrote:This might be OK for a < 100 archive updates, but for an archive with > 10000 files and/or > 1000 updates this will be too much, or in other words: making archive listing unnecessarily complex. TC already has palpable slowdowns when listing an archive with > 50000 files inside.
So TC is just too slow when listing archives with more than a few (ten) thousand files. So no matter if you have a huge base file set with just a few updates, or an archive with a small file set and many updates: In both cases navigating through the virtual archive structure would be a pain.

The 2nd aspect is the internal data structures in the main zpaq code. Matt uses basic (but efficient) structures to hold just the minimum amount of data needed for listing until to a certain archive state, but I already made a quite complicated wrapper for this to make all available options for archive listing in the plug-in possible. Adding another layer on top of that is a huge amount of work (and makes the wrapper code even more complex and hard to read).

I would love to see Matt building such feature into ZPAQ on it's own, but I'm not sure if he really is interested in developing ZPAQ any further (at least when it comes to new usability features).

I'm currently busy trying to add the versioned file name listing as Hacker suggested. When this is done I will take another look at a possible "list all archive states" option, but I REALLY can't make any promises.
TC plugins: PCREsearch and RegXtract

Post Reply