Page 1 of 1

Move directory atomically or fail at once

Posted: 2019-09-26, 12:18 UTC
by ono
I have this problem again today. I wanted to move some big dataset directory, but I had some file locked there. (Lovely Windows file locking) And TC starts to move the stuff one-by-one :evil: I would really wish it failed instead the move individual files around when the directory cannot be moved atomically completely at once.

I tried to look for the setting that controls this behavior, but cannot find one - neither in the program itself and these forums. Is there any wincmd.ini setting that will instruct TC to NOT to move individual items if the directory move fails? :cry:

Re: Move directory atomically or fail at once

Posted: 2019-09-29, 13:35 UTC
by Usher
TC has no such option. It creates file list and tries to copy as many files as possible without prior lock or file attributes checking.

In such cases, when problems are expected, most users do as follows:
Copy files in background, ignore (skip) all locked or unavailable files, close programs which lock ignored files, copy newer files only (some unlocked files might be also updated). If any files are still locked, prepare pendrive with windows installer and TC, then run command line and TC from that pendrive and copy newer files once again. When the copying is finished, compare source and target directory, and delete the source if all is OK.
Some users use Synchronize option for such actions.

Re: Move directory atomically or fail at once

Posted: 2019-09-30, 06:22 UTC
by MVV
When I expect that some large folder should be moved in a moment, it is very sad to see how TC does moving file-by-file. Especially often it does happen when moving requires elevation.

So I would love an option to confirm slow move too, some kind of a message 'Hey, fast moving failed, folder will be moved file by file, OK?'. Perhaps it should be used for large folders only because it is faster to move small folder file by file... But if there are locked files, file by file moving will be failed too.

Re: Move directory atomically or fail at once

Posted: 2019-09-30, 12:56 UTC
by Usher
Windows is a multitasking system so TC don't know which files will be available when it starts to copy them. They may be locked or removed by another program after TC starts its operation. What is more, you can even remove files yourself from TC if you ran copy in background.

For extremely large directories copying may take several hours so the final state of copied files may be far from their state on the start. In many cases you will never have so much time to completely exclude computer from normal work. Reorganizing of such large (or important) volume of data without prior planning is just stupid in my opinion.

Finally, talking about fast/slow moving is misleading if it's not a special case of moving inside the same partition.

Re: Move directory atomically or fail at once

Posted: 2019-10-12, 10:34 UTC
by ono
Usher wrote:
2019-09-30, 12:56 UTC
Windows is a multitasking system so TC don't know which files will be available when it starts to copy them. They may be locked or removed by another program after TC starts its operation. What is more, you can even remove files yourself from TC if you ran copy in background. (...)
With all the respect, I think this is not a problem. What I am saying is that TC should not attempt to copy anything by itself when requested directory move fails. And I am talking about moving directory within single file-system, which is pretty much atomic operation, unless it fails :lol:. So not talking about copying or moving anything across filesystems.

I am pretty sure TC knows that move failed, and it is TC fallback mechanism that starts moving contained files and subfolders one by one.

I don't want this fall-back behavior, it very very undesired as it leads to inconsistencies when some files are locked/etc. At least I want some sort of a confirmation for this behavior.

Re: Move directory atomically or fail at once

Posted: 2019-10-12, 14:00 UTC
by Dalai
ono wrote:
2019-10-12, 10:34 UTC
I don't want this fall-back behavior, it very very undesired as it leads to inconsistencies when some files are locked/etc. At least I want some sort of a confirmation for this behavior.
I'm not sure but IIRC I made such a suggestion a long time ago (don't remember when and if it was in English or German). Unfortunately nothing has happened since then.

Regards
Dalai

Re: Move directory atomically or fail at once

Posted: 2019-10-12, 18:27 UTC
by Usher
ono wrote:
2019-10-12, 10:34 UTC
So not talking about copying or moving anything across filesystems.
It doesn't matter. Checking files in a large directory may take several seconds or minutes. Before you finish checking, files already checked may be locked by another program or background task. If you want to finish moving with success, you should either lock all checked files or ensure that they won't be used (close all programs and stop services which may use the files or run system from pendrive or Live CD/DVD). Otherwise results are unpredictable.

Finally, on NTFS partition you can create junction to existing directory rather than moving/renaming it in any way.

Re: Move directory atomically or fail at once

Posted: 2019-10-12, 19:57 UTC
by ono
Usher wrote:
2019-10-12, 18:27 UTC
ono wrote:
2019-10-12, 10:34 UTC
So not talking about copying or moving anything across filesystems.
It doesn't matter. Checking files in a large directory may take several seconds or minutes. Before you finish checking, files already checked may be locked by another program or background task. (...)
What checking? To me TC simply calls MoveFileEx(srcDir, dstDir) (a kernel function) to move srcDir entry into dstDir which is instant (atomic?) operation. I don't see any checking here, just one kernel function call.

Only when MoveFileEx fails with ERROR_SHARING_VIOLATION (meaning some file is locked in srcDir), TC tries to move srcDir contents one-by-one - this is some sort of a fallback behavior and I don't want such behavior to be enforced on me.

Re: Move directory atomically or fail at once

Posted: 2019-10-13, 20:49 UTC
by Usher
ono wrote:
2019-10-12, 19:57 UTC
To me TC simply calls MoveFileEx(srcDir, dstDir)
Oh, you mean this thing… TC may call this function, but does it work in Windows 9x as you expect? Maybe there is still some code from Win3x times there. Maybe you are right, I only know that results may depend on file system drivers, and they are different in Win9x and NT family.
ono wrote:
2019-10-12, 19:57 UTC
Only when MoveFileEx fails with ERROR_SHARING_VIOLATION (meaning some file is locked in srcDir), TC tries to move srcDir contents one-by-one - this is some sort of a fallback behavior and I don't want such behavior to be enforced on me.
If you are right, there should be one more confirmation added for this case to TC options.

Thanks for explanations and sorry for misunderstanding.

Re: Move directory atomically or fail at once

Posted: 2019-10-18, 08:38 UTC
by MVV
By the way, I found this old request that is related to current one (though is only for as admin action).