[BUG]: Cannot open directories with large file counts

[sirksel]

First, let me say, I love Total Commander (both your desktop version and Android version), and I've been using the desktop version as a licensed user for years...

Lately, with the Android version, I've had trouble opening an MP3 directory containing 3,798 mp3 files on extSdCard mount. I'm fairly certain it's not a file system issue as other explorer apps (like ES) will open the directory. On TC, it just reports that the directory cannot be opened. It seemed to me like it might be an array upper bound issue, but I'm not sure. Might you be able to replicate/fix the issue? I really appreciate your help.

[ghisler(Author)]

You need to be patient - if Android claims that the program doesn't respond, you need to tap on "Wait". I have a directory with 8515 files inside - it works, but takes a while to load.

[sirksel]

Mr. Ghisler, thank you for your quick response! I don't get the option to wait, and it doesn't ever attempt to force close the app. The wait/close dialog never comes up. I'll give you more specifics below, since it appears I may have a different issue...

Using a stock-build but rooted Samsung Galaxy Note 2 16GB SGH-T889, running Jelly Bean 4.1.1, here's what happens. I'm in the /storage/extSdCard directory, and I tap on the "mp3" folder (without quotes) displayed, containing ~4,000 files. It whirs for about 3 seconds with no obvious response, then displays the following box for about 4 seconds:

Getting directory {title bar}
------------------------
From:
/storage/extSdCard/mp3
{below displaying the whirly progress animation}

After these 4 seconds have elapsed, it displays the following dialog:

Error {title bar}
------------------------
Cannot open file/folder!
mp3
[OK]

I would wait if I could, and even if it took 20-30 seconds to load, that would be fine. Unfortunately, I don't seem to have that option. I can leave the error dialog up there for 3 minutes, and nothing happens once that error condition is reached. When I click OK it goes back to the prior directory display (storage/extSdCard).

It may be irrelevant, but may help to diagnose... When I repeat this process, it goes immediately to the Error dialog, and it doesn't even attempt the dialog box with the whirly progress indicator. Acting as if maybe it's finding the error result in a cache somewhere. (Also, if I exit TC and use a task manager to kill all tasks, then try again, it repeats the prior behavior with the whirly progress dialog followed by the error dialog.)

Any idea what could be causing this? Is the Error dialog coming from TC or the OS? Is there a build.prop modification I could make to help extend the timeout interval? I really appreciate whatever help you could provide, as I love your file manager, but I have to use two different ones until this is resolved... Thanks again for your help!

[ghisler(Author)]

The "Getting directory" dialog is shown when reading takes a lot of time. The error dialog is shown when listFiles() crashes or returns null. I guess that the file names are just too long to fit into the limited memory space available to TC, sorry.

[CatalinC]

Here are more details:

- S3 (i9300) with Omega ROM v44.3 (= 4.1.2), rooted and everything else, TC has root

- 64GB extSdCard , exFAT (file-system is perfect on PC and mounts perfect in this ROM)

- all folders EXCEPT one are OK

- ONE folder (TitaniumBackup) has a problem in TC (and it reports "cannot open folder TitaniumBackup") - but that same folder is OK for instance in ES File Explorer (which reports 2800+ files in a single level).

I have tested with 2.01 and now with 2.02 beta 2.

Please let me know if I can provide more details or do more tests.

[ghisler(Author)]

I'm really sorry, I cannot help you with this limited information.

[CatalinC]

I'm really sorry, I cannot help you with this limited information.

Here is some more info and a simplified procedure to repeat the problem:

- the 'magic number" seems to be around 2000 files in a single folder

- the second trick seems to be that the problem shows only on extSdCard (FAT32/exFAT), while on the internal (ext4 ?) partition things seem to be OK as high as I was able to test (2810 files or so)

- if I have around 1990 or so files in a single folder on extSdCard it works the first time when the program enters the folder, but if you exit and try to enter again it no longer works; if TC is closed and restarted again it works the first time, then stops being able to read the folder (so I guess there is a limited resource somewhere, possibly in TC or between OS and TC, but there is something about the way that resource is used/released since on both extended and internal ES File Explorer worked no matter the number of files).

[ghisler(Author)]

In principle, there is no difference in reading the file list on the internal and external SD card. No root rights are needed in either case. Maybe TC doesn't like one of the file names?

[CatalinC]

In principle, there is no difference in reading the file list on the internal and external SD card. No root rights are needed in either case. Maybe TC doesn't like one of the file names?

Hmm ... the thing was that I did not have another folder with so many files, so I just copied (with another file manager) that precise same folder with the problem from the external on the internal - so I doubt it was from the file name. Also that would not explain how when at 1990 files or so it works the first time - but not for the second one.

One of the things that I was afraid was from the way those are "mounted" - on the internal the UID of the folder creator is shown as 0 (root), but on external it is 1000 (system) - and the GID is also slightly different. But again it does not explain what happens at the level of 1990.

Other than a fixed-buffer in TC my wild guess was something inside the system in something like the "enumeration callback" - but I never looked on how that is done on Android so that is just a very wild guess

[ghisler(Author)]

So could you access the large folder on the internal file system with Total Commander?

[CatalinC]

So could you access the large folder on the internal file system with Total Commander?

Yes, as described above the same folder seemed to work OK with TC from internal but not from external. It worked OK on both with ES File Explorer.

[ghisler(Author)]

OK, can you try to create a new folder on the external memory and copy the folder there from the internal? And then try to access that from Total Commander?

[CatalinC]

OK, can you try to create a new folder on the external memory and copy the folder there from the internal? And then try to access that from Total Commander?

Already did that - after the initial test on internal I decided to test on external with a variable number of files, and most certainly I was not going to delete my original folder

I have now just repeated the procedure:

a) initial /storage/extSdCard/TitaniumBackup is 2804 files, 1733392313 bytes

b) that is copied as a folder (using ES File Explorer) to /sdcard/TMP , where it has exactly the same number of files and bytes

c) that folder can be opened any number of times in TC; it takes some time (2-4 seconds) but it works; also with a long-press in TC and then Properties it shows the correct size

d) even more astounding - that folder can be copied with Total Commander itself to /storage/extSdCard/TMP

e) the copy procedure ends successfully in TC (takes a lot of time but I feel it takes less than the copy with ES File Explorer, even if writing on internal is about twice as fast as writing on external)

f) however after the copy operation ends, if I go to the destination panel and try to enter the folder /storage/extSdCard/TMP/TitaniumBackup it no longer works - it waits for like 5 seconds, then it sometimes it pops a nice dialog "Getting directory", and then (first time after 3-4 seconds, then immediately) switches that to a dialog that is saying "ERROR - Cannot open file/folder Titaniumbackup" (which has to be dismissed with OK); just getting the Properties gives some info but the size in 0 bytes, 0 files

g) that folder /storage/extSdCard/TMP/TitaniumBackup can be seen perfectly with ES File Explorer and it shows the same number of files and bytes as the original; I do not think that ES File Explorer has a procedure to compare the contents so I can not say 100% certain that the binary content is the same but it looks that way. (EDIT - I just checked on PC with TC in Windows and those two are identical).


Have you tried to create a folder on one of your external SD cards (ideally on a Samsung device - those might be using an interesting FUSE mounting trick) with 2800 files or so? Even on a non-Samsung device that would be an interesting test!

[ghisler(Author)]

Just had an idea: Do you use the option "root functions everywhere" in Total Commander? Then TC will get the directory listing from a Linux shell command in all paths EXCEPT on the internal SD-Card. If you use this, try to uncheck it.

[CatalinC]

Just had an idea: Do you use the option "root functions everywhere" in Total Commander? Then TC will get the directory listing from a Linux shell command in all paths EXCEPT on the internal SD-Card. If you use this, try to uncheck it.

YES - that was it - with an interesting twist (regarding the root shell command) but just as I said above:

Other than a fixed-buffer in TC my wild guess was something inside the system in something like the "enumeration callback" - but I never looked on how that is done on Android so that is just a very wild guess

Anyway unchecking that fixed the immediate problem