[WCX] ZPAQ

Discuss and announce Total Commander plugins, addons and other useful tools here, both their usage and their development.

Moderators: white, Hacker, petermad, Stefan2

Post Reply
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

New Version 1.3.5!
  • update to ZPAQ 7.07 core
    -> fixes handling of some corrupted archives
    -> check if an archive unexpectedly appears or disappears before starting updating it (like when plugging in an external drive)
    -> prevent updates to mixed journaling/streaming archives
    -> pure (old) streaming archives will not list or extract any more in TC
  • directory timestamps now extract correctly for the versioning view (i.e. 'Show all archive versions'
    enabled), no matter which and how many versions are extracted
    -> they will still show correctly in TC only for the first (oldest) version
    with an individual (a new) timestamp
  • added new special field for the custom format string: the archive name (w/o extension)
    -> length modifier will determine the number of characters used, zero would use the complete name
  • the 'Show the default archive view in extra dir' option will now also work w/o the
    'Show archive version names as detailed timestamp' option
  • the 'Show the default archive view in extra dir' custom string can now use the same
    format string as for the 'Show archive version names as detailed timestamp' option
  • added Danish translation (by 'petermad')
  • config dialog overhaul: readjusted elements horizontally, no crop on low-res screens (down to 800x600)
Check the first post for the new file.
TC plugins: PCREsearch and RegXtract
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Post by *Horst.Epp »

There is a major bug in this version if I update an existing archive.
It updates the time stamps of sub-dirs with new or changed files in the archive
but doesn't include any changed files from this dirs.
Only new files are inserted.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

Horst.Epp wrote:It updates the time stamps of sub-dirs with new or changed files in the archive
but doesn't include any changed files from this dirs.
Only new files are inserted.
I'm not sure if I understand the problem.
But when I create some test archives and do multiple updates, it works just as usual, adding all files that are requested.
(and I didn't change any code that would cause this)

Can you give detailed instructions how this occurs,
or what you would except to happen?


Update:
Could it be
http://encode.ru/threads/456-zpaq-updates?p=46873&viewfull=1#post46873 wrote:v706d will no longer append an empty 104 byte header when there are no files to update.
?
In short: if all files you try to add are the same as in the last update, they archive now stays unaltered.

If this is the problem, maybe I could add some option for that.
TC plugins: PCREsearch and RegXtract
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Post by *Horst.Epp »

In short:
Existing files in the archive are never updated if newer versions are found
in the tree to be stored in the archive.

I have an archiv of a tools tree which is about 800MB archive size.
If files are changed in this tree they are no longer added
when I update the archive.
Only the time stamp of the dirs in the archive are updated in which the changed files are.
New files are found and stored correct into the archive.
This worked perfect with the previous x64 version.
I select an dir in TC and and existing archive on the other side.
The TC options "Also pack pathnames"
and "Recursively pack subdirectories" are set.
The pkplugin.ini is as follows:

; TOTAL COMMANDER packer plugin common INI file
[ZPAQ]
ShowAllVersions=1
ForceAddIdenticalFiles=0
StoreDriveLetters=0
MemoryWarning=1
SetDirDates=1
MaskDriveLetters=1
RememberPassword=0
ShowVersionsAsDate=1
Threads=2
CompressionLevel=1
StoreAds=0
ExtractAds=0
Color1=255
Color2=16711680
MultiPartDigits=0
MaskEmptyVersions=0
KeepDeletedFiles=0
RollbackSupport=0
BackgroundOpWarning=1
ShowArchiveDefaultView=1
DefaultViewIncludeDeleted=0
DefaultViewDirName=All

I also checked the archive with the Pakka utility
and the changed files are for shure not updated in the archive.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

I can confirm the issue, but only for the x64 version, the 32-bit version works just fine.

This is really one of the most [face=impact]ugly[/face] situations that could occur, programming-wise, because the code is actually quite fine,
with basically no raw pointer arithmetic that would explain the difference, and everything else is pretty much portable to both target platforms.

I guess it's some 64-bit integer treatment problem, because Matt changed some arithmetics in the 7.06 version,
and nothing in my own wrapper code could cause this (normally).
In any case I have to test this in detail, which might take some time.

I will also test a different compiler and the official zpaq64.exe.
If anyone can test this standalone exe for this issue before I do, please leave a message,
also if you have an idea which part of the code could cause this.


So if you use the x64 version, I recommend to use plugin version 1.3 until I can fix this bug.
TC plugins: PCREsearch and RegXtract
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

I found the new 7.07 core to be the culprit, just as I expected.
The standalone x64 version seems not affected.

However I'm not sure if I can find the exact source of the problem soon, so I decided to revert to 7.05 core for now.
All new features from 1.3.5 will work, of course.


New Version 1.3.6!
  • revert to ZPAQ 7.05 core due to unresolved bug in x64 plug-in
Check the first post for the new file.
TC plugins: PCREsearch and RegXtract
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Post by *Horst.Epp »

milo1012 wrote:I found the new 7.07 core to be the culprit, just as I expected.
The standalone x64 version seems not affected.

However I'm not sure if I can find the exact source of the problem soon, so I decided to revert to 7.05 core for now.
All new features from 1.3.5 will work, of course.


New Version 1.3.6!
  • revert to ZPAQ 7.05 core due to unresolved bug in x64 plug-in
Check the first post for the new file.
Thanks for the quick fix.
Works again now as expected.
krasusczak
Senior Member
Senior Member
Posts: 282
Joined: 2011-09-23, 10:35 UTC

Post by *krasusczak »

milo1012
So as you mention before, is it safe to use 32bit version of 1.3.5 without any risk right?
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

krasusczak wrote:So as you mention before, is it safe to use 32bit version of 1.3.5 without any risk right?
Yes, I tested it to be stable before I released it.
That's why I didn't expect the x64 version to behave like that, as there is no real reason for it.
Also it's mostly the same code that is used in the 7.07 zpaq.exe, and this works just fine.

In the meantime I found out that Visual Studio 2005 and 2008 can compile it to a correctly working DLL,
but VS 2012 will produce a non-working one too (as VS 2010, which I used for the release in question) .
This is getting hilarious :roll:
It really is some kind of compiler issue, which I never thought I would see.

I will do some additional tests when I find more time.
TC plugins: PCREsearch and RegXtract
ykhabins
Junior Member
Junior Member
Posts: 35
Joined: 2003-10-09, 13:12 UTC

ZPaq Progress Bar is not working

Post by *ykhabins »

Hi there,

I am using TC 8.52a 32-bit and ZPaq plugin 1.3.6
Unfortunately, the ZPaq plugin progress bar is not moving at all in the dialog box. Impossible to know where is the process, specially for large files.

Is it possible to fix it?
ykhabins
Junior Member
Junior Member
Posts: 35
Joined: 2003-10-09, 13:12 UTC

Adding new files to the existing ZPaq archive is malfunction

Post by *ykhabins »

Hi there,

It seems I bumped into a bug while trying to add new files to the existing ZPaq archive.

Scenario: TC left pane has list of files, TC right pane shows files in the existing ZPaq archive. Trying to copy, i.e. meaning trying to add new file(s) from the left pane, deletes original archive files and replaces them with the new file from the left pane.

[Edit] My bad. I went to the ZPaq configuration and checked off "Show all archive versions". Problem solved.
User avatar
Horst.Epp
Power Member
Power Member
Posts: 6450
Joined: 2003-02-06, 17:36 UTC
Location: Germany

Re: Adding new files to the existing ZPaq archive is malfunc

Post by *Horst.Epp »

ykhabins wrote:Hi there,

It seems I bumped into a bug while trying to add new files to the existing ZPaq archive.

Scenario: TC left pane has list of files, TC right pane shows files in the existing ZPaq archive. Trying to copy, i.e. meaning trying to add new file(s) from the left pane, deletes original archive files and replaces them with the new file from the left pane.

[Edit] My bad. I went to the ZPaq configuration and checked off "Show all archive versions". Problem solved.
Confirmed for the x64 version 1.3.6
The new files are show in the newest version of the archive
but the previous files are gone from the sub-dir
(still found in the older archive entry).
Show all archive versions is set.
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Re: Adding new files to the existing ZPaq archive is malfunc

Post by *milo1012 »

ykhabins wrote:...My bad. I went to the ZPaq configuration and checked off "Show all archive versions". Problem solved.
Yes, that's one solution, or just use the 'Don't mark files found only in the last update as deleted' option when packing files.
This would "keep" all files from the past (the last archive "state"), which would now be missing in the list of files you'd try to add.
That is the whole concept of the journaling archiver, to always "mirror" exactly what you feed into the archive,
marking all files not found any more as deleted (but not deleting them for real of course).
I described all of this in the readme.

Just note that at some point the standalone program additionally tries to look for files on the disk, and shows a combination of internal and external files when listing files (command "-list").
I can't implement the plug-in that way, as I wouldn't know how to deal with non-archive files through the wcx interface.
I might take another look into that in the (far) future, but the current concept works well IMO.


ykhabins wrote:Unfortunately, the ZPaq plugin progress bar is not moving at all in the dialog box. Impossible to know where is the process, specially for large files.
I already stated here that there's basically nothing I can do.
The ZPAQ author should implement more status updates, as I would probably need weeks to do it, not even knowing if it's even possible at all.
I recommend to do some test runs, so that you get a "feeling" for how long each compression mode takes for a specific amount of data (e.g. 100 MB),
so that you can estimate the rough time by yourself.
I described this in the readme too.

Horst.Epp wrote:The new files are show in the newest version of the archive
but the previous files are gone from the sub-dir
(still found in the older archive entry).
Yes, and that's how it always worked, and is supposed to work, as I described above.
Or is this behavior different in an older plug-in version?
TC plugins: PCREsearch and RegXtract
ykhabins
Junior Member
Junior Member
Posts: 35
Joined: 2003-10-09, 13:12 UTC

Post by *ykhabins »

Hi milo1012

I am using c# for my own needs to run ZPaq.
Please find below how I do it and intercept all what ZPaq emits to the console.
It includes % of the archiving progress.
Maybe you can reuse some of its technique.

Code: Select all

void Main()
{
	bool rc = true;
	string parInputXMLFile = @"d:\Temp\CDW\SubsidyTrackingFact\Paraguay\201601\id2095_PY201601_SubsidyTrackingFact.xml";
	string parInputRECEIPTFile = ""; //@"d:\Temp\CDW\SubsidyTrackingFact\Paraguay\201601\id2095_PY201601_SubsidyTrackingFact_RECEIPT_2016-02-17T22-40-32.xml";
	string parOutputDirectory = @"d:\Tmp";
	
	string parZPaq = @"d:\Kit\ZPaq\v7.07\zpaq64.exe";
	string parThreads= "8";	
	string parMethod = "34";
	if (String.IsNullOrEmpty(parMethod))
                parMethod = "34";
	
	try
	{	
		string outputZPaqFile = Path.Combine(parOutputDirectory, Path.GetFileNameWithoutExtension(parInputXMLFile) + ".zpaq");
		string parInputXMLFileName = Path.GetFileName(parInputXMLFile);
		string parInputRECEIPTFileName = Path.GetFileName(parInputRECEIPTFile);
		
		ProcessStartInfo psi = new ProcessStartInfo();
		psi.FileName = parZPaq;
	//	psi.WorkingDirectory = parWorkingDirectory;
		psi.WindowStyle = ProcessWindowStyle.Hidden;
		psi.RedirectStandardError = true;
		psi.RedirectStandardOutput = true;
		psi.RedirectStandardInput = true;
		psi.UseShellExecute = false;
		psi.CreateNoWindow = true;
		psi.ErrorDialog = false;
		psi.Arguments = string.Format("add {0} {1} {2} -to {3} {4} -method {5} -threads {6}"
				, outputZPaqFile
				, parInputXMLFile
				, parInputRECEIPTFile
				, parInputXMLFileName
				, parInputRECEIPTFileName
				, parMethod
				, parThreads);
	
		// will cause to pickup default value of threads, i.e. # of cores
		if (String.IsNullOrEmpty(parThreads))
		{
			psi.Arguments = psi.Arguments.Replace("-threads", string.Empty);
		}
		psi.Arguments.Dump("SSIS log");
		
		using (Process process = Process.Start(psi))
		{
			// to capture console output
			// http://csharptest.net/532/using-processstart-to-capture-console-output/
	
			ManualResetEvent mreOut = new ManualResetEvent(false);
			ManualResetEvent mreErr = new ManualResetEvent(false);
			process.OutputDataReceived += (o, e) => { if (e.Data == null) mreOut.Set(); else e.Data.Dump("Out Log to SSIS"); };
			process.BeginOutputReadLine();
			process.ErrorDataReceived += (o, e) => { if (e.Data == null) mreErr.Set(); else 
						{
							if(e.Data.Length > 0)
							{
								if(e.Data.Contains("(with errors)"))
									e.Data.Dump("Err Log to SSIS");
								else
									e.Data.Dump("Out Log to SSIS");
							}
						}
				};
			process.BeginErrorReadLine();
			
			process.StandardInput.Close();
			process.WaitForExit();
		
			mreOut.WaitOne();
			mreErr.WaitOne();
			
			if (process.ExitCode != 0)
			{
				"Running ZPaq was abruptly terminated. Something is wrong.".Dump("Log to SSIS");
				rc = false;
			}
		}
	}
	catch (Exception ex)
	{
		rc = false;
		ex.Message.Dump("ZPaq process launch failed");
	}
	
	return;
}
It produces the following output:


Code: Select all

SSIS log

add d:\Tmp\id2095_PY201601_SubsidyTrackingFact.zpaq d:\Temp\CDW\SubsidyTrackingFact\Paraguay\201601\id2095_PY201601_SubsidyTrackingFact.xml  -to id2095_PY201601_SubsidyTrackingFact.xml  -method 34 -threads 8 


Out Log to SSIS

zpaq v7.07 journaling archiver, compiled Mar 18 2016 


Out Log to SSIS

Adding 634.282956 MB in 1 files -method 34 -threads 8 at 2016-03-22 15:49:31. 


Out Log to SSIS

 2.68% 0:00:06 [1..81] 16681058 -method 34,229,0 


Out Log to SSIS

 5.29% 0:00:06 [82..170] 16527372 -method 34,228,0 


Out Log to SSIS

 7.89% 0:00:06 [171..257] 16288611 -method 34,228,0 


Out Log to SSIS

10.51% 0:00:06 [258..355] 16657880 -method 34,227,0 


Out Log to SSIS

13.08% 0:00:06 [356..452] 16641244 -method 34,228,0 


Out Log to SSIS

15.71% 0:00:06 [453..554] 16753848 -method 34,227,0 


Out Log to SSIS

18.36% 0:00:06 [555..700] 16697945 -method 34,228,0 


Out Log to SSIS

21.00% 0:00:07 [701..990] 16673117 -method 34,231,0 


Out Log to SSIS

23.63% 0:00:07 [991..1121] 16679476 -method 34,226,0 


Out Log to SSIS

26.26% 0:00:08 [1122..1249] 16707839 -method 34,223,0 


Out Log to SSIS

28.87% 0:00:09 [1250..1381] 16651863 -method 34,225,0 


Out Log to SSIS

31.50% 0:00:09 [1382..1500] 16693578 -method 34,227,0 


Out Log to SSIS

34.16% 0:00:09 [1501..1594] 16770395 -method 34,230,0 


Out Log to SSIS

36.80% 0:00:09 [1595..1683] 16554940 -method 34,228,0 


Out Log to SSIS

39.38% 0:00:09 [1684..1770] 16685511 -method 34,227,0 


Out Log to SSIS

42.04% 0:00:09 [1771..1872] 16712151 -method 34,229,0 


Out Log to SSIS

44.70% 0:00:11 [1873..1975] 16611421 -method 34,229,0 


Out Log to SSIS

47.29% 0:00:10 [1976..2075] 16710118 -method 34,226,0 


Out Log to SSIS

49.90% 0:00:09 [2076..2213] 16699763 -method 34,230,0 


Out Log to SSIS

52.56% 0:00:09 [2214..2364] 16741266 -method 34,227,0 


Out Log to SSIS

55.18% 0:00:08 [2365..2534] 16713517 -method 34,227,0 


Out Log to SSIS

57.83% 0:00:07 [2535..2667] 16418897 -method 34,225,0 


Out Log to SSIS

60.41% 0:00:07 [2668..2745] 16720764 -method 34,227,0 


Out Log to SSIS

63.06% 0:00:07 [2746..2834] 16484330 -method 34,227,0 


Out Log to SSIS

65.67% 0:00:09 [2835..2933] 16644024 -method 34,228,0 


Out Log to SSIS

68.25% 0:00:08 [2934..3032] 16646028 -method 34,229,0 


Out Log to SSIS

70.88% 0:00:07 [3033..3120] 16643583 -method 34,228,0 


Out Log to SSIS

73.56% 0:00:07 [3121..3205] 16637996 -method 34,228,0 


Out Log to SSIS

76.14% 0:00:06 [3206..3284] 16596888 -method 34,226,0 


Out Log to SSIS

78.81% 0:00:05 [3285..3385] 16717667 -method 34,229,0 


Out Log to SSIS

81.38% 0:00:04 [3386..3490] 16660173 -method 34,227,0 


Out Log to SSIS

84.03% 0:00:04 [3491..3606] 16749474 -method 34,227,0 


Out Log to SSIS

86.65% 0:00:04 [3607..3708] 16709619 -method 34,229,0 


Out Log to SSIS

89.28% 0:00:03 [3709..3790] 16729867 -method 34,227,0 


Out Log to SSIS

91.91% 0:00:02 [3791..3882] 16664174 -method 34,227,0 


Out Log to SSIS

94.60% 0:00:01 [3883..3991] 16678227 -method 34,229,0 


Out Log to SSIS

97.20% 0:00:00 [3992..4082] 16583619 -method 34,228,0 


Out Log to SSIS

99.80% 0:00:00 [4083..4164] 16488292 -method 34,227,0 


Out Log to SSIS

100.00% 0:00:00 + d:/Temp/CDW/SubsidyTrackingFact/Paraguay/201601/id2095_PY201601_SubsidyTrackingFact.xml -> id2095_PY201601_SubsidyTrackingFact.xml 634282956 


Out Log to SSIS

100.00% 0:00:00 [4165..4173] 1673425 -method 34,225,0 


Out Log to SSIS

1 +added, 0 -removed. 


Out Log to SSIS

0.000000 + (634.282956 -> 634.282956 -> 19.655740) = 19.655740 MB 


Out Log to SSIS

45.755 seconds (all OK) 
User avatar
milo1012
Power Member
Power Member
Posts: 1158
Joined: 2012-02-02, 19:23 UTC

Post by *milo1012 »

ykhabins wrote:Please find below how I do it and intercept all what ZPaq emits to the console.
...
Thanks for the example, but it doesn't change the basic problem.
I'm using the direct C++ code, modified to work as a wcx plug-in.
In all places where a console output would send an update (percentage-wise) I update the TC progress dialog too.

It seems that you use -method 34 in your example,
which means that it uses compression level 3 with a block size of 2^4 = 16 MiB,
while the plug-in always uses the default size, which is 2^6 = 64 MiB for level 3.
Please try with -method 3 and compare it with the plug-in output (and same amount of threads!).
Roughly speaking, the smaller the block size, the more updates you get,
because the main compression routine spawns compression threads, and instructs them to process a specific block.

You're free to look at my code and see if you can find something I forgot.
The main problem for the updates is that block size mentioned above.

I will take another look at the source I have now.
On top of that, the only thing I could do is to look in each compression thread for how many data was processed,
but this would need a thread synchronization and a callback to the main loop, in order to send status updates to the dialog,
and I'm not even sure if I could get the number w/o modifying the libzpaq itself, which is the last thing I'd intend to do.


Update:
Indeed there was a special reason why I couldn't do more updates than I do now
(i.e. a big file will only update the dialog at start end at the very end),
but I need to reanalyze the exact reason, as this is code that I wrote half a year ago.
TC plugins: PCREsearch and RegXtract
Post Reply