Fastest way to synchronize of 2 disks?

[Terka]

Hi all, do you know about solution which is capable of:

example:
2 drives
o: (original), b: (backup)

at the beginning the o: and b: are exactly the same

have large files (eg. home videos) on o: and im moving them to different directories on o: only.
That means, all files on o: are still backuped on b:, but they are located in wrong folders.

I want to make o: and b: exactly the same again, by MOVING files on b: to proper folders.
Dont want to copy anything from original disk to backup one, because copying takes looong time.
Want only move files inside the b: drive to proper directory, because moving on SAME disk is fast.
thank you,
Terka

[ZoSTeR]

Here's a PowerShell script for that. Paste it in the PowerShell ISE and see how it performs. The critical commands are in WhatIf mode so they won't do anything until you remove the two "-WhatIf".

It looks at all files on B: and if there is a file with the same name on O: it will move it to a folder with the same path.

It assumes the same content on the drives, only with a different folder structure. There is no error or collision checking.

This is very risky and you should be sure that it performs as expected.

Code: Select all

$files_S = Get-ChildItem -Path "O:\" -File -Recurse
$files_D = Get-ChildItem -Path "B:\" -File -Recurse

foreach ($file_D in $files_D)
{
    foreach ($file_S in $files_S)
    {
        if ($file_S.Name -eq $file_D.Name)
        {
            $destDrive = Split-Path $file_D.Directory -Qualifier
            $destFolder = Split-Path $file_S.Directory -NoQualifier
            $finalDest = $destDrive + $destFolder + '\'
            if (!(Test-Path $finalDest))
            {
                New-Item -Path $finalDest -ItemType Directory -Force -WhatIf
            }
            Move-Item -Path $file_D.FullName -Destination $finalDest -WhatIf
        }
    }
}

Edit: Add "\" to the drive.

[Terka]

hi ZoSTeR, thank you.
I wrote similar script in Autohotkey, with checking the date and size to ensure the file is the correct one. Its working fine, but the performance is really poor.
I think someone already solved similar problem, thats why im asking.

[MVV]

I would sort files by size before comparing to reduce complexity from O(n^2) to O(n) in best case (when all files have different sizes). While sorted, arrays may be enumerated simultaneously w/o any need to start from the beginning again (only files from current position of second array up to position where size differs should be compared with current file from first array).

[Terka]

Usually the differences between the two directories are quite small
eg. renamed directory or few files moved to another directory.

[MVV]

I still think that comparing sorted arrays should be very fast. Something like this:

Code: Select all

$A = (dir -Path "O:\" -File -Recurse) | sort -Property Length;
$B = (dir -Path "B:\" -File -Recurse) | sort -Property Length;
for ($i, $j = 0, 0; $i -lt $A.Length; ++$i) {
    $fa = $A[$i];
    $n = $fa.Length;
    for (; $j < $B.Length -and $B[j].Length -le $n; ++$j) {
        if ($B[j].Length -eq $n) {
            $fb = $B[j];
            # compare $fa and $fb and move $fb to required place
        }
    }
}

(errors may be there, I haven't debugged it)

[Terka]

Sadly im quite poor in progamming, i use only Autohotkey a bit (and bash).

[ZoSTeR]

If this this not a backup of a few hundred media files we're talking about, but a full system backup, you should definitely look for some other options.

And ensuring that the two files being compared are really identical is a whole different ball game.

[milo1012]

There is an alternative solution, by using TC's "Find duplicates" option, and some RegEx tool.

But it will only work if you have no duplicate files (same name, size) inside either o: or b:
This means e.g. fileX is only found once in o:, and once in b:


Use TC's find files (Alt+F7).

Search in:
o:\;b:\
Advanced tab:
Attributes: [ ]Directories
[x] Find duplicate files:
[x] same name
[x] same size
For more security you may also use:
[x] same plugin fields: [=tc.writedate]

Let the search finish, use "Feed to listbox"

Mark ALL files.
Use cm_SaveSelectionToFile to write all marked files with full paths to an output file.

Now use (parse) this file with a sophisticated RegEx tool, to find all file pairs that differ in their path, ignoring the drive letter.
For my RegXtract plug-in you would use:
Regular Expression:

Code: Select all

(?m)b:((.*?\\)([^\\\n\r]+))\Ro:(?!\g{2})(.*?\\)([^\\\n\r]+)$

Replace String:

Code: Select all

MOVE "b:$3$4" "b:$5$6"

Leave everything else as default settings,
and modify the drive letter to your actual drive letters (((?m)b... and ...\Ro... ).

TC seems to always sort duplicate files by name.
So, if your backup drive letter does NOT sort before the original drive letter, use:
(x: is backup)

Code: Select all

(?m)(o:)((.*?\\)([^\\\n\r]+))\Rx:(?!\g{2})(.*?\\)([^\\\n\r]+)$
MOVE "b:$5$6" "b:$3$4"

The plug-in will now output a txt file, which you can rename to .bat/.cmd and execute.
You probably need to recode that output file to the DOS/ANSI code page first, because it's in UTF-8 after output.




The move command will of course only work if the target directory exists,
and probably not for read-only files, will possibly ask for overwrite, and so on.
Feel free to modify this (like using robocopy, etc.).
(alternatively you could output that file pair only (skip the MOVE command) and let that output file parse by a standalone batch)


And like ZoSTeR said: it's still quite risky, especially if you don't want to compare file content, but it should work,
if TC really finds file pairs only (and not any triplets, quadruples, etc.).

[Phred]

I've long been intrigued by the claims of XXCopy.

The author says that his program can duplicate a disk, down to the boot sector.

I've not tried it, fully, but it could be useful here. It has over a hundred C/L switches, so is likely to be able to accommodate the task required in the OP.

www.xxcopy.com

Anyone else used it?

[bobputnam]

I've long been intrigued by the claims of XXCopy.

The author says that his program can duplicate a disk, down to the boot sector.

I've not tried it, fully, but it could be useful here. It has over a hundred C/L switches, so is likely to be able to accommodate the task required in the OP.

www.xxcopy.com

Anyone else used it?

I've used xxcopy for about a decade. It works fine for duplicating a folder(s) or disk. I run a batch file as a task at night, so I couldn't tell you whether it's fast or not.

[jvh]

FreeFileSync with enabled option Detect Moved Files can do this job. Probably not the best variant but looks safe enough.

Note: Don't forget to disable instalation of advertisement software (Skype?) during installation.