WFX multithread access troubles

[MVV]

I have serious problem with background operations for my virtual panel.

I have critical section that prevents multiple threads from access to virtual system simultaneously.

When I check 'in background' box and start copying files to plugin, TC moves operation to background thread and activates itself, but when I'm trying to change plugin folder, TC calls FsFindExecute function which needs to enter critical section in order to search file(s) inside virtual system (this is necessary to prevent reading from map while it changes). And if critical section is locked, TC hangs... Main thread falls into waiting for critical section loop while background thread calls ProgressProc from FsGetFileW function and hands too.

If I remove calls to ProgressProc function, it works perfect! I may browse virtual panel, add/delete links, execute internal commands, save panel contents while background operation is running.

Also TC hangs when I'm trying to pause backgroung operation and access virtual panel - it seems that TC pauses background thread that have critical section locked and main thread falls into waiting loop again.

Any suggestions would be appreciated. Maybe I'm doing something wrong, maybe its because of TC nature and may be fixed.

Currently I see only one solution - TC should not enter many plugin functions by multiple threads simultaneously (or at least prevent main thread from entering plugin function when backgroung thread is already executing some plugin function).

[ghisler(Author)]

Well, it's not easy to add multi-threading, that's why I have hesitated to add it for plugins for such a long time.

What you need to do is simple: Do NOT call any callbacks as long as you are in your critical section! Keep the critical sections to a minimum.

Example in pseudo code:

EnterCriticalSection
access your data, store data in local variable for ProgressProc
LeaveCriticalSection
Call ProgressProc
EnterCriticalSection
access your data again
LeaveCriticalSection
etc.

I'm doing such things all the time in TC itself to avoid deadlocks...

[MVV]

Well, you're right. I should minimize life time of critical section lock.
But here some problem remains anyway - what if I pause background thread just when it has critical section locked?


When using standard map, it is hard to determine is iterator valid or not. It seems that better way to minimize bug count - is to find all files on FindFirstFile and return them one by one.

[MVV]

Thanks, I've realized right multi-thread access, but I have one question.

Does TC make {down|up}load list before every background operation? And, does TC pause background operations while retrieving folder contents?

If it's not so, it is not safe to use same iterator in FsFindNextFile calls because background operation may delete item that iterator points to. So, there are two solutions I see - TC should pause background operations between FsFindFirst and FsFindClose calls or each plugin should use safe methods or retrieve full folder listing on FsFindFirst call (so I will need to write additional container for such filelist).

[ghisler(Author)]

Downloads work like this:
1. TC gets complete list of selected files+dirs
2. It downloads the files, and for the dirs, it gets the content of the complete dir, then downloads all the files. For subdirs, it gets again all the files in the subdir, etc.

[MVV]

So, each operation that runs in separate thread fully independent from others, is it correct?
And, in theory such situation may come when one thread will call FsFindNext and recieve e.g. info about file 1.txt and before it will call FsFindNext again another thread will delete this file so file handle become invalid?

[ghisler(Author)]

Yes, this can indeed happen - but the same can happen when you access an ftp server, or even with local files when you use multiple threads! You need to handle this situation by returning an error, or by skipping the file.

Alternatively, build a file list in FsFindFirst, and then return all the names in FsFindNext even if they no longer exist. The user will then get an error when the file should be downloaded.

[MVV]

The problem is that it is hard to detect if memory pointed to by iterator was freed (also, it is possible that this memory block is already allocated). So it seems that best decision in this case is to get full list at once with preventing other threads to access virtual system during enumerating files.