Move directory atomically or fail at once

Here you can propose new features, make suggestions etc.

Moderators: sheep, Hacker, Stefan2, white

Post Reply
ono
Junior Member
Junior Member
Posts: 49
Joined: 2003-12-22, 11:37 UTC
Location: Zurich Area, Switzerland

Move directory atomically or fail at once

Post by *ono » 2019-09-26, 12:18 UTC

I have this problem again today. I wanted to move some big dataset directory, but I had some file locked there. (Lovely Windows file locking) And TC starts to move the stuff one-by-one :evil: I would really wish it failed instead the move individual files around when the directory cannot be moved atomically completely at once.

I tried to look for the setting that controls this behavior, but cannot find one - neither in the program itself and these forums. Is there any wincmd.ini setting that will instruct TC to NOT to move individual items if the directory move fails? :cry:
TC Personal license #99581

User avatar
Usher
Power Member
Power Member
Posts: 584
Joined: 2011-03-11, 10:11 UTC

Re: Move directory atomically or fail at once

Post by *Usher » 2019-09-29, 13:35 UTC

TC has no such option. It creates file list and tries to copy as many files as possible without prior lock or file attributes checking.

In such cases, when problems are expected, most users do as follows:
Copy files in background, ignore (skip) all locked or unavailable files, close programs which lock ignored files, copy newer files only (some unlocked files might be also updated). If any files are still locked, prepare pendrive with windows installer and TC, then run command line and TC from that pendrive and copy newer files once again. When the copying is finished, compare source and target directory, and delete the source if all is OK.
Some users use Synchronize option for such actions.
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8394
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Move directory atomically or fail at once

Post by *MVV » 2019-09-30, 06:22 UTC

When I expect that some large folder should be moved in a moment, it is very sad to see how TC does moving file-by-file. Especially often it does happen when moving requires elevation.

So I would love an option to confirm slow move too, some kind of a message 'Hey, fast moving failed, folder will be moved file by file, OK?'. Perhaps it should be used for large folders only because it is faster to move small folder file by file... But if there are locked files, file by file moving will be failed too.

User avatar
Usher
Power Member
Power Member
Posts: 584
Joined: 2011-03-11, 10:11 UTC

Re: Move directory atomically or fail at once

Post by *Usher » 2019-09-30, 12:56 UTC

Windows is a multitasking system so TC don't know which files will be available when it starts to copy them. They may be locked or removed by another program after TC starts its operation. What is more, you can even remove files yourself from TC if you ran copy in background.

For extremely large directories copying may take several hours so the final state of copied files may be far from their state on the start. In many cases you will never have so much time to completely exclude computer from normal work. Reorganizing of such large (or important) volume of data without prior planning is just stupid in my opinion.

Finally, talking about fast/slow moving is misleading if it's not a special case of moving inside the same partition.
Regards from Poland
Andrzej P. Wozniak

ono
Junior Member
Junior Member
Posts: 49
Joined: 2003-12-22, 11:37 UTC
Location: Zurich Area, Switzerland

Re: Move directory atomically or fail at once

Post by *ono » 2019-10-12, 10:34 UTC

Usher wrote:
2019-09-30, 12:56 UTC
Windows is a multitasking system so TC don't know which files will be available when it starts to copy them. They may be locked or removed by another program after TC starts its operation. What is more, you can even remove files yourself from TC if you ran copy in background. (...)
With all the respect, I think this is not a problem. What I am saying is that TC should not attempt to copy anything by itself when requested directory move fails. And I am talking about moving directory within single file-system, which is pretty much atomic operation, unless it fails :lol:. So not talking about copying or moving anything across filesystems.

I am pretty sure TC knows that move failed, and it is TC fallback mechanism that starts moving contained files and subfolders one by one.

I don't want this fall-back behavior, it very very undesired as it leads to inconsistencies when some files are locked/etc. At least I want some sort of a confirmation for this behavior.
TC Personal license #99581

User avatar
Dalai
Power Member
Power Member
Posts: 6745
Joined: 2005-01-28, 22:17 UTC
Location: Meiningen (Südthüringen)

Re: Move directory atomically or fail at once

Post by *Dalai » 2019-10-12, 14:00 UTC

ono wrote:
2019-10-12, 10:34 UTC
I don't want this fall-back behavior, it very very undesired as it leads to inconsistencies when some files are locked/etc. At least I want some sort of a confirmation for this behavior.
I'm not sure but IIRC I made such a suggestion a long time ago (don't remember when and if it was in English or German). Unfortunately nothing has happened since then.

Regards
Dalai
#101164 Personal licence
Ryzen 5 2600, 16 GiB RAM, ASUS Prime X370-A, Win7 x64

Plugins: Services2, Startups

User avatar
Usher
Power Member
Power Member
Posts: 584
Joined: 2011-03-11, 10:11 UTC

Re: Move directory atomically or fail at once

Post by *Usher » 2019-10-12, 18:27 UTC

ono wrote:
2019-10-12, 10:34 UTC
So not talking about copying or moving anything across filesystems.
It doesn't matter. Checking files in a large directory may take several seconds or minutes. Before you finish checking, files already checked may be locked by another program or background task. If you want to finish moving with success, you should either lock all checked files or ensure that they won't be used (close all programs and stop services which may use the files or run system from pendrive or Live CD/DVD). Otherwise results are unpredictable.

Finally, on NTFS partition you can create junction to existing directory rather than moving/renaming it in any way.
Regards from Poland
Andrzej P. Wozniak

ono
Junior Member
Junior Member
Posts: 49
Joined: 2003-12-22, 11:37 UTC
Location: Zurich Area, Switzerland

Re: Move directory atomically or fail at once

Post by *ono » 2019-10-12, 19:57 UTC

Usher wrote:
2019-10-12, 18:27 UTC
ono wrote:
2019-10-12, 10:34 UTC
So not talking about copying or moving anything across filesystems.
It doesn't matter. Checking files in a large directory may take several seconds or minutes. Before you finish checking, files already checked may be locked by another program or background task. (...)
What checking? To me TC simply calls MoveFileEx(srcDir, dstDir) (a kernel function) to move srcDir entry into dstDir which is instant (atomic?) operation. I don't see any checking here, just one kernel function call.

Only when MoveFileEx fails with ERROR_SHARING_VIOLATION (meaning some file is locked in srcDir), TC tries to move srcDir contents one-by-one - this is some sort of a fallback behavior and I don't want such behavior to be enforced on me.
TC Personal license #99581

User avatar
Usher
Power Member
Power Member
Posts: 584
Joined: 2011-03-11, 10:11 UTC

Re: Move directory atomically or fail at once

Post by *Usher » 2019-10-13, 20:49 UTC

ono wrote:
2019-10-12, 19:57 UTC
To me TC simply calls MoveFileEx(srcDir, dstDir)
Oh, you mean this thing… TC may call this function, but does it work in Windows 9x as you expect? Maybe there is still some code from Win3x times there. Maybe you are right, I only know that results may depend on file system drivers, and they are different in Win9x and NT family.
ono wrote:
2019-10-12, 19:57 UTC
Only when MoveFileEx fails with ERROR_SHARING_VIOLATION (meaning some file is locked in srcDir), TC tries to move srcDir contents one-by-one - this is some sort of a fallback behavior and I don't want such behavior to be enforced on me.
If you are right, there should be one more confirmation added for this case to TC options.

Thanks for explanations and sorry for misunderstanding.
Regards from Poland
Andrzej P. Wozniak

User avatar
MVV
Power Member
Power Member
Posts: 8394
Joined: 2008-08-03, 12:51 UTC
Location: Russian Federation

Re: Move directory atomically or fail at once

Post by *MVV » 2019-10-18, 08:38 UTC

By the way, I found this old request that is related to current one (though is only for as admin action).

Post Reply