Speed up searches with FindFirstFileEx

[MarcinW]

Since Windows 7 / Windows Server 2008 R2 (version 6.1) there are new functionalities of FindFirstFileExA/FindFirstFileExW function:

1) fInfoLevelId parameter can have now FindExInfoBasic value - searches don't query short file names, improving filesystem enumeration speed.

2) dwAdditionalFlags parameter can have now FIND_FIRST_EX_LARGE_FETCH flag - so all searches use internally buffer of size 0x10000 instead of 0x1000 (as described in Crash on VirtualPC thread) - this decreases significantly number of calls to NtQueryDirectoryFile function, and - in consequence - decreases user-kernel-user transitions, when NtQueryDirectoryFile calls a filesystem driver.

You can even use SwapImportedFunction from PE.pas unit, build a small wrapper, hook all calls to FindFirstFileA/FindFirstFileW and redirect them to FindFirstFileExA/FindFirstFileExW.

Regards

[ghisler(Author)]

Thanks for the info!
1) TC needs the short names in various places like file_id.diz files or in case of very long paths, so I cannot use this.

2) I will try that. It should be no problem because my calls to FindFirstFileW are encapsulated in another function.

Btw, are there any differences between FindFirstFileW and FindFirstFileExW I need to be aware of? I don't want to break anything by switching to this function, e.g. access to network drives with non-Microsoft software like NAS devices...

[Hacker]

Christian,

1) TC needs the short names in various places like file_id.diz files or in case of very long paths, so I cannot use this.

Could you not only use short file names when there is actually a need for them? Ie. when a long path becomes problematic or when file_id.diz files are turned on?

Roman

[Lefteous]

I really hope 8.5 will do something about the search in terms of indexed based search - can't wait...

[Horst.Epp]

I really hope 8.5 will do something about the search in terms of indexed based search - can't wait...

I find using Everything or VTfind is enough for that purpose.
Both integrate fine with TC.
You need NTFS volumes of course, for others Locate32 does the job.

[jjk]

Unfortunately Everything or UltraSearch and probably VTfind need admin rights to work. But in work environments we have them rarely.
Does anybody know the contribution of FindFirstFileW and FindFirstFileExW in accelerating search ? Can we hope great improvements with their ?

[Horst.Epp]

Unfortunately Everything or UltraSearch and probably VTfind need admin rights to work. But in work environments we have them rarely.
Does anybody know the contribution of FindFirstFileW and FindFirstFileExW in accelerating search ? Can we hope great improvements with their ?

Everything can run as a service.
Then you don't need Admin rights for using it.
You must of course have admin rights to install it as a service

[Flint]

You need NTFS volumes of course

With latest betas you can add FAT volumes for "non-journal-based" monitoring in Everything (though it will be slower and won't always contain 100% up-to-date data).

[Hacker]

Lefteous,
You could perhaps also support the following idea:
Indexed search using WDX plugins

Roman

[Lefteous]

2Horst.Epp

Both integrate fine with TC.

I'm not happy with these solutions. How on earth are they 'integrated fine'?

2Hacker

You could perhaps also support the following idea

Yes I support this idea.

[Horst.Epp]

2Horst.Epp

Both integrate fine with TC.

I'm not happy with these solutions. How on earth are they 'integrated fine'?
...

For VTfind:
You can directly jump to any file or dir found in a new TC tab.
You can feed to list box the search results.
You can view any file from the search result in TC viewer.

What else do you want ?

[MarcinW]

Btw, are there any differences between FindFirstFileW and FindFirstFileExW I need to be aware of? I don't want to break anything by switching to this function, e.g. access to network drives with non-Microsoft software like NAS devices...

Since Windows 2000 (or even Windows NT 4) FindFirstFile is just a wrapper around FindFirstFileEx (this is a common practice for "Ex" functions). So, in fact, Total Commander uses FindFirstFileEx since many years. So I'm sure that there won't be any problems when using it directly.

Code: Select all

.text:7D86ABEB ; Exported entry 162. FindFirstFileW
.text:7D86ABEB
.text:7D86ABEB ; =============== S U B R O U T I N E =======================================
.text:7D86ABEB
.text:7D86ABEB ; Attributes: bp-based frame
.text:7D86ABEB
.text:7D86ABEB ; HANDLE __stdcall FindFirstFileW(LPCWSTR lpFileName, LPWIN32_FIND_DATAW lpFindFileData)
.text:7D86ABEB                 public FindFirstFileW
.text:7D86ABEB FindFirstFileW  proc near
.text:7D86ABEB
.text:7D86ABEB lpFileName      = dword ptr  8
.text:7D86ABEB lpFindFileData  = dword ptr  0Ch
.text:7D86ABEB
.text:7D86ABEB                 mov     edi, edi
.text:7D86ABED                 push    ebp
.text:7D86ABEE                 mov     ebp, esp
.text:7D86ABF0                 xor     eax, eax
.text:7D86ABF2                 push    eax                  ; dwAdditionalFlags
.text:7D86ABF3                 push    eax                  ; lpSearchFilter
.text:7D86ABF4                 push    eax                  ; fSearchOp
.text:7D86ABF5                 push    [ebp+lpFindFileData] ; lpFindFileData
.text:7D86ABF8                 push    eax                  ; fInfoLevelId
.text:7D86ABF9                 push    [ebp+lpFileName]     ; lpFileName
.text:7D86ABFC                 call    FindFirstFileExW
.text:7D86AC01                 pop     ebp
.text:7D86AC02                 retn    8
.text:7D86AC02 FindFirstFileW  endp
.text:7D86AC02
.text:7D86AC02 ; ---------------------------------------------------------------------------

Regards

[ghisler(Author)]

That's great, thanks for investigating!