Copy files with their relative path-structure after duplicate-files-search?

[georgeb]

Recently I again had to undergo the tedious task of consolidating 2 large (TB) USB-media-sticks with both some different and also mainly identical files, albeit many of them renamed and/or moved following different or changing naming-conventions. A typical task where TC's "Sync-Dirs" should be able to excel you may think - but not so fast. Yes, there are those identical files from both structures - but other than that "SyncDirs" has its known limits as it will find hundreds of GBs as - seemingly - "unique" files within both structures left and right.

But are they really "unique" - or are they in fact just binary duplicates having been renamed and/or moved to someplace else?

As my previous suggestions about how to remedy such a situation have gone unanswered so far this is still where things start getting really complicated as the goal of such a consolidation-operation is of course to merge these data-structures in a way that would keep "best of both worlds", regarding sound-quality, in the end.

So the crucial component to remedy that problem successfully will always be a binary-duplicates-search between both structures with the goal of separating the "truly unique"-files from the renamed and/or moved dupes which should remain in only one singular position following the optimized, current naming-convention afterwards.

Since such a binary-duplicates-search most unfortunately cannot be performed from within "Sync-Dirs" my best - but actually quite complicated - solution so far is to copy all those renamed/moved versions to several auxiliary-directories, delete them in the wrong places in both file-structures and copying the consolidated, new subtree-structure from those finally disentangled auxiliary-directories to both tree-structures in the end (while only the "truly unique" versions on both sides are actually "synchronized"=merged to result in identical structures on both sides following the basic intention of the "Sync-Dirs"-tool).

Now when performing the necessary and essential binary-duplicates-search outside of "Sync-Dirs" between auxiliary copies of the perceivedly (but not necessarily truly) "unique" left/right categories from "Sync-Dirs" I'll get hundreds of pairs, even triples, of binary identical files - together with their - often different - subtree/path-structure specified behind the actual names.

Now the incredibly powerful tool of de-/selecting those files folder-wise wit the Num+-method comes into play. But unfortunately - when that is all set and done - it seems that all files now selected can only be copied into one single directory from there thereby neglecting the individual paths of those files as initially indicated in the dupe-search results.

I have to apologize for so far having only having elaborated on the situation in such a comprehensive manner but otherwise I'm afraid the actual problem/question might have easily been misunderstood - as here finally comes the actual question:

Is it possible from the binary-duplicates-results-field (after proper file-selection via Num+) to copy those files selected, together with their relative path-structure as indicated after the filename so as to replicate the whole subtree-structure, perhaps to a different storage-medium/drive-letter, instead of piling them up altogether in a single location/directory by simply applying the F5-Copy-command, thereby apparently losing the entire relative-path-structure as identified by the binary-dupes-result-page???

[Dalai]

TC can keep relative paths only in branch view (Ctrl+B) but not from search results.

You can make use of a packer plugin like TreeCopyPlus or CopyTree. After installing either of these plugins you can do the following: Instead of the regular F5 press Alt+F5 and select the appropriate extension from packer list. Both plugins have some options like directory level (CopyTree has a bunch more) which can help with keeping the directory structure. The files won't be packed but copied by the plugin instead.

Regards
Dalai

[georgeb]

TC can keep relative paths only in branch view (Ctrl+B) but not from search results.

You can make use of a packer plugin like TreeCopyPlus or CopyTree.

Thanks a lot @Dalai for pointing this out. I tried the latter b/c more recent plugin - and it did the trick! This will save me a lot of copying-work in the future as opposed to my clumsy current approach of deleting the found duplicates first in one auxiliary copy and then having to generate the complementary-file-set against a second (still complete) auxiliary copy which in the end will also leave 2 subtrees of only the duplicates found on either side behind.

I'm a little bit confused though about your mention of "branch view (Ctrl+B)". Isn't that the one that would on the contrary entirely flatten the subtree-structure showing all files from all subdirs piled up in a single heap? So that for instance by sorting via file-size or date one could then determine the largest or newest file within that whole subtree-structure? But this (Ctrl+B)-view AFAIK does not offer to also show the path in any extra column, not even via some custom-view-mode?

[Horst.Epp]

I'm a little bit confused though about your mention of "branch view (Ctrl+B)". Isn't that the one that would on the contrary entirely flatten the subtree-structure showing all files from all subdirs piled up in a single heap? So that for instance by sorting via file-size or date one could then determine the largest or newest file within that whole subtree-structure? But this (Ctrl+B)-view AFAIK does not offer to also show the path in any extra column, not even via some custom-view-mode?

It looks like you never tried to make a custom column
Of course, you can add the field [=tc.path] in such a definition.

For example, I have this in my wincmd.ini under
[ViewModes]
2_name=BranchView
2_icon=
2_commands=*|cm_SrcViewMode0
2_options=6|-1|0||-1|-1|-1|-1|-1

The following defines the fields of my BranchView
[CustomFields]
Widths1=135,20,130,-47,-63
Headers1=Path\nSize\nDate
Contents1=[=tc.path]\n[=tc.size.bkMG2]\n[=tc.writedate]

Additionally, I have redefined Ctrl-B to the following em_command from the usercmd.ini
[em_branchview]
cmd=cm_DirBranch,cm_SrcViewModeList BranchView

[Hacker]

georgeb,

I'm a little bit confused though about your mention of "branch view (Ctrl+B)".

Dalai only mentioned it because that's where TC does not need a plugin like CopyTree to copy a "flattened" structure with the original paths included.

Roman

[Dalai]

Dalai only mentioned it because that's where TC does not need a plugin like CopyTree to copy a "flattened" structure with the original paths included.

Exactly. After pressing F5 in branch view TC provides the additional option "Keep relative paths (relative to current directory)" in the copy dialog. This is not the case for regular copy and search results which is why a plugin is necessary to keep the dir structure in these cases.

Custom columns can be used in normal view, branch view and even in search results. Just right-click on the column headers and switch to a custom columns view (or create a new one). However, this doesn't help with copying the directory structure.

Regards
Dalai

[oko]

Consolidation means also to get rid of duplicate files. Why do you want to copy duplicate files to have "triplicate" files?
So I do not understand exactly what do you want to achieve? To have the same content on both disks without duplicates but with all unique files originally from disk1 and disk2?
Find duplicates (Alt+F7) and choose from which disk or folders should be deleted. After duplicates do not exist you can synchronize disks (Syncdir) to have the same contents.

[georgeb]

Thanks guys for all that additional info.

To @Horst.Epp
Contrary to what it may look like I have applied several custom-colum-views in the past.
It just never came to my mind in connection with the Ctrl-B view.

[beb]

There's an ideal tool to do exactly what the OP is asking for, which at the same time is a) simple, b) quick, and c) tiny (< 1MB).
Alas, they provide such a weird licensing approach, that I cannot recommend it, while still forced to stay with its 2006 year version (so there's no chance you would get it, and therefore no sense in sharing it).
Sad, such a powerful tool as Total Commander cannot provide similar functionality regarding user-friendly choices upon post-processing the duplicates search results, which itself (the searching) Total Commander is doing ok. And this post reminded me that I need once to make a post on the suggestion forum. Thus, thank you.

[georgeb]

Consolidation means also to get rid of duplicate files. Why do you want to copy duplicate files to have "triplicate" files?

No, you seemingly misunderstood. Those "triplicate"-files are only auxiliary copies. The situation described is much more complicated than just "synchronizing"=making equal both branches (memory sticks). The goal is to consolidate the whole structure - as already described - to result in 2 equal "best-of-both-worlds"-structures in the end. And also to get rid of renamed/moved content of binary equal duplicates with different names or in different/moved locations.

"SyncDirs" doesnt help a lot here as it cannot differentiate between moved or renamed files with identical binary content - and will therefore list all of them as (wrongfully) "unique" (left/right).

So these pseudo-"unique" hits in "SyncDirs" have to be disentangled into (at least) 4 different subgroups:
1. "truly unique" left/right, that is truly singular items on either side - which will in the end copied to both structures, thereby actually meeting the synchronization-purpose in "SyncDirs" in a narrower sense.

2. Files with binary duplicates on the other side - because of renaming or having been moved to different locations (left/right)
Those different versions obviously must not be redistributed to both final versions in the end. So they must be "disentangles" first before merging the "better/newer/desired_final_versions" back to both sticks in only one permanent fashion.

So let's assume for the sake of simplicity that one stick/thumbdrive will usually hold the "newer" version named after the most recent naming-convention and already in the correct place within the desired directory-structure. Then the final consolidation process will run as follows:

1. Delete both pseudo-"unique" colums from "SyncDirs" left/right altogether as in there singular and duplicate versions are mingled together inseparably.

2. Copy the 2 auxiliary-folders containing only the truly singular items from either side (without any duplicates) in a kind of crosswise manner to the opposite USB-stick - and vice versa.

3. Finally determine which one (if any) USB-stick holds the newer/desired structure regarding filenames and the storage-location for those duplicate (triplicate?) versions of equal files. If that is the case then you can simply delete the other auxiliary folder - and finally distribute the folder with the prevailing naming/location-structure to BOTH USB-stick versions.

Voilà! That procedure will finally produce 2 identical copies of the latest version on both sides ("best of both worlds")! And don't worry about redundancy! After completion all auxiliary folders containing the disentangled subgroups can be deleted entirely.

[georgeb]

And this post reminded me that I need once to make a post on the suggestion forum. Thus, thank you.

In case you're interested, my proposals are already there.
viewtopic.php?p=425346#p425346 (and related discussions)
The core-element would always be to include a "binary duplicates-by-content"-search into the "SyncDirs"-process which could be triggered manually (and only on special user-request so as to not unnecessarily complicate the basic "SyncDirs"-process) after an initial "SyncDirs"-run and only between those "mingled" pseudo-"unique"-columns left/right. As a result the blue and green selection-buttons on top would be split into dark-green/light-green thereby making those then split-columns separately selectionable - "dark-green" standing for "truly-unique"/singular on the left side while "light-green" would stand for "only-locally-unique"-with binary duplicates to be found somewhere else. And the same differentiation would apply to the right-side-column, then using "dark-blue" and "light-blue" respectively. Now when right-clicking on some duplicate-filename a pop-up/new window could then be offered showing ONLY all the binary duplicates of that single file in their "SyncDirs"-fashion, together with full-path-names and much easier to visually compare and select - as opposed to many-thousands-of-file-pairs/triples as listed in an endless chain (nearly impossible to inspect individually) in the "Search-for-binary-duplicates"-result-panel.

[oko]

to georgeb

I understand that sync dir do not find duplicates if they are not in the same folder on both sides and process them as unique. Therefore I wrote process in too steps:
1/ find duplicates (with alt+f7) (and delete one of their pair according some criteria) and then
2/ sync dir
But if I understand well, in "find duplicates" you are not able to choose which one of two duplicates to delete and which one to remain according to automated criteria (is in folder, is newer or so), therefore you need to take duplicates out (with their original structure) and then browse to choose individually which are the right.

[georgeb]

But if I understand well, in "find duplicates" you are not able to choose which one of two duplicates to delete and which one to remain according to automated criteria (is in folder, is newer or so), therefore you need to take duplicates out (with their original structure) and then browse to choose individually which are the right.

In theory the duplicates-search-result-window allows to choose by an automated process (Num+ selection of folders) and by FIX CRITERIA which structure to keep and which one to delete BUT in reality some kind of visual (and possibly even auditive by playing critical differences) inspection may have to follow. In particular as even the "truly unique" files on both sides are not a "done deal" and have sometimes be inspected visually, too. Also they may in fact be pseudo-unique because of mis-spellings which ought to be detected.

And that is practically impossible given the somewhat obscure nature of representation in the Find-Duplicates-result-window, consisting of an endless list of perhaps thousands of identical pairs/triples of filenames with no actual tree-/path-structure (ok, the paths are listed behind the filenames - but this can be no substitute for a real subtree-structure).

As opposed to the TC-file-panels which provide much better oversight. For instance between those auxiliary file panels additional searches can be run to facilitate the visual inspection. For example some secondary searches may be performed between them, such as
*Elvis*Pr*.*
thereby revealing if perhaps on one side only some crap-mp3 version might exist whereas on the other side a high-res-flac-version of the same title might be present - in which case of course the high-res-flac-version would be destined to prevail in the end-version. While at the same time such a specific search would also reveal any commonplace mis-spellings of "Presley" (like "Pressly" or "Presly"). The Find-Duplicates-result-window simply will never be able to offer that level of versatility. Hence the auxiliary copies of the separated file-categories of "truly unique"/"files with duplicates due to moving and/or renaming", each for the left and right substructure. Hope that will help to clarify my approach which again could be facilitated considerably by a more sophisticated and much more capable "SyncDirs"-tool as the structure it provides for displaying and grouping of the different categories is IMHO already next to optimal.