Multi-threading for thumbnail generation

[HighDiver]

I have a folder with large (2560x1600) images which I use as wallpaper - when I get bored of my desktop, I like to use Total Command's thumbnail view on this folder to preview a new one.

I've configured TC to use IrfanView to generate the thumbs, but it does take a couple of minutes for every file to be processed, during which time one CPU core is maxed out.

This seems like one example of where multi-threading could provide a useful performance boost, processing graphics files in parallel. Would it be possible to configure TC to spawn multiple threads for thumbnail generation?

Thanks!

[milo1012]

t does take a couple of minutes for every file to be processed, during which time one CPU core is maxed out

Then there's sth. seriously wrong with your configuration.
Take a look at the tcthumbs.idb file, it might be very large or fragmented and could slow down image caching.

I don't use IrfanView and can't test it right now,
but with the Imagine plugin I can load Thumbnails of 80 jpeg files,
each > 10 Mpx, in less than 10 seconds (Thumbnail cache cleared).

Yes, MT can improve performance, but this will only be true if the I/O-performance is high enough to provide enough data in the time frame.

[HighDiver]

t does take a couple of minutes for every file to be processed, during which time one CPU core is maxed out

Take a look at the tcthumbs.idb file, it might be very large or fragmented and could slow down image caching.

The processing only occurs without the tcthumbs.db file. It is perfectly quick with it present, but since I only use the thumbnail view infrequently, that file is stored in a temp folder on a ramdisk.

...with the Imagine plugin I can load Thumbnails of 80 jpeg files,
each > 10 Mpx, in less than 10 seconds (Thumbnail cache cleared).

I tried the Imagine plugin (and running Imagine.exe with the %APPDATA% cache cleared) and saw identical results to IrfanView - 3-4 files/second with 2560x1600 images (a folder with 360 files from 350KB to 14MB in size). Lower resolution files in another folder were far faster - what resolution were yours?

Yes, MT can improve performance, but this will only be true if the I/O-performance is high enough to provide enough data in the time frame.

In my case, CPU is definitely the bottleneck - the drive is capable of 120MB/s data transfer and Process Hacker reports total I/O (read+write) of 90KB/s during thumbnail generation.

[milo1012]

but it does take a couple of minutes for every file to be processed

3-4 files/second with 2560x1600 images

That's contradictory. So what is the case with your system, minutes or seconds per file?

what resolution were yours?

From 3600 x 2400 to 10000 x 7500, all JPGs between 8 MB and 300k.
Repeatable: 80 files in about 9 seconds and default Thumbnail size (96 x 96).
Core i5, 3200 MHz.

the drive is capable of 120MB/s data transfer and Process Hacker reports total I/O (read+write) of 90KB/s

I/O-performance isn't just about raw transfer rate, it's also about access speed, and IOPS (Input/Output Operations Per Second), like for modern SSDs.
If you would try to access several files in parallel on slow and uncached drives you will probably have serious slowdowns,
like for CD/DVD/BD and, most important, unbuffered flash-bases drives, which includes nearly all USB sticks, SD cards, CF cards etc.
So it wouldn't be that good to enable MT in general.

[HighDiver]

That's contradictory. So what is the case with your system, minutes or seconds per file?

3-4 files/second for a folder with 360 files gives 80-120 seconds processing time. Where's the contradiction?

From 3600 x 2400 to 10000 x 7500, all JPGs between 8 MB and 300k.
Repeatable: 80 files in about 9 seconds and default Thumbnail size (96 x 96).

Just to confirm, this is with thumbs.tc deleted and on an HDD rather than SDD?

I did retest with 96x96 thumbs and saw no perceptible difference (I was using 96x80) but using IrfanView to generate the thumbs ran noticeably faster than Imagine. CPU is a Core i7 running at 3.36GHz.

I/O-performance isn't just about raw transfer rate, it's also about access speed, and IOPS (Input/Output Operations Per Second), like for modern SSDs.

Agreed - but if you are seeing a core maxed out for an extended period of time on an operation, it should be pretty safe to say that CPU is the bottleneck. However I did copy the folder to the SSD RAID I use as my main disk and saw no speed improvement.

[Hacker]

HighDiver,

3-4 files/second for a folder with 360 files gives 80-120 seconds processing time. Where's the contradiction?

Well, you said "a couple of minutes for every file". Perhaps you meant folder instead of file?

Roman

[HighDiver]

Well, you said "a couple of minutes for every file". Perhaps you meant folder instead of file?

I would have thought "a couple of minutes for every file to be processed" to be quite distinct from "a couple of minutes for each file to be processed". However it can be easy to overlook a word, so thanks for pointing this out.

To be completely clear, I'm seeing thumbnails generated for 3-4 files/second (2560x1600 JPGs), taking roughly a couple of minutes total for a 360-file folder. It's not a feature I use often, so not a deal-breaker for me, but it would be a nice improvement to have multithreading available for such a CPU-intensive task.

[milo1012]

Well, my English is definitely not perfect, but using "all files" would be much less ambiguous.

Just to confirm, this is with thumbs.tc deleted and on an HDD rather than SDD?

Yes, on a plain 3 TB HDD with db file deleted (and TC restart).
But it's obvious that transfer speed wouldn't matter in that case, since you're having some CPU bottleneck.
I merely use the Imagine.wlx as the Lister plugin, which links to the Imagine.dll for Thumbnail generation (32 bit TC and plugin).
And of course turned off Use Explorer method (OLE2) for image files.
There actually isn't much of a speed difference when using Imagine VS Explorer method (Windows 7).

So I don't know what it is, but it's obviously not as fast as it could be for your system. (Virus Scanner?)

[Hacker]

HighDiver,

I would have thought "a couple of minutes for every file to be processed" to be quite distinct from "a couple of minutes for each file to be processed".

Every can be quite ambiguous this way.
"Ten pence is donated to charity for every bottle sold."

Roman

[HighDiver]

And of course turned off Use Explorer method (OLE2) for image files.
There actually isn't much of a speed difference when using Imagine VS Explorer method (Windows 7).

OLE2 disabled here also - in fact, using it doesn't work (only showing the IrfanView application icon). Be curious to hear if using IrfanView is faster than Imagine in your case also.

So I don't know what it is, but it's obviously not as fast as it could be for your system. (Virus Scanner?)

No AV (I use other, low-impact, security software). However what frequency is your CPU running at? For single-threaded CPU-intensive applications, there's little difference between an i5 and an i7, so a higher-clocked i5 could provide one reason.

Another aspect to thumbnail generation worth mention is that TC generates them for displayed files rather than the full folder (so scrolling down, even after waiting a couple of minutes, means waiting on thumb generation again). Having every thumb in a selected folder generated (as long as that folder remains selected) would make file viewing seem faster.

[siealex]

Check your HDD interface mode, this looks like it is PIO.

[HighDiver]

Check your HDD interface mode, this looks like it is PIO.

No, it's DMA. PIO would mean higher CPU utilisation all the time, and I did note above that copying the folder to SSD made no appreciable difference.

[milo1012]

Alright, some quick benchmark:

The same 80 files, deleted db file (or just renaming folder/dir where the pictures reside is enough)
on a Core-i5-3470 (3200 MHz, no OC), Windows 7 x64, TC 32 bit

IrfanView: roughly ~16 seconds
Explorer method: ~7 seconds
Imagine: ~8.5-9 seconds

For making this clear: TC must use the Clipboard and resize any image manually when using IrfanView.
(see help file for: Dialog box: Configure lister or Configuration - Thumbnails)
That's why I prefer Imagine, it's portable and uses a function call ListGetPreviewBitmap instead of clipboard.

Another aspect to thumbnail generation worth mention is that TC generates them for displayed files rather than the full folder

There is an ini setting PreloadThumbnails

1: Loads thumbnail images already when opening a dir, not only when scrolling to that line.
2: Same as 1, but also in branch view
3: Preload just the next two lines below the visible thumbnails
0: Disabled, load thumbnails only when they scroll into view

Are you sure you're actually using IrfanView?
Maybe some other Lister-Plugin is interfering with thumbnail creation.
So check the wlx-plugin list or try it with a clean TC installation.
Otherwise it must be your system's settings.

[HolgerK]

TC x64 851a, Windows 7 x64, AMD Phenom II x4 945 (3GHz), SSD, Virus Scanner MS Security Essentials (no need for overkill..)

- 80 JPGs (3456*2340), size between 2.2MB and 4MB
- Thumbnail size 96*64
- only [x] Use Explorer method(OLE2) for: *.*

SSD(Samsung 830 256GB): Less than 8 seconds.
HDD(Toshiba DT01ABA 3TB, 5940rpm): about 9.5 seconds.

PreloadThumbnails=2
ThumbWidthNoCache=96
ThumbHeightNoCache=64
ThumbOptions=1
ThumbsLocation=%$LOCAL_APPDATA%\GHISLER
ThumbsInDlg=1
ThumbsCopyDel=1
ThumbsCustomFieldsEnabled=1
ThumbnailMemoryLimit=5000
ThumbnailPrealloc=2000000
ThumbIrfXnTypes=*.*
ThumbPlgTypes=*.* *\
ThumbTxtTypes=*.txt *.ini *.reg *.cpp *.h *.cs

Regards
Holger

Edit: Added time for HDD

[mrle]

Test here: 863 JPGs in 5-6 seconds, on 4-5 years old Pentium laptop.
Size of the images does not matter so much if they have thumbnail in EXIF.

decFastThumbs.wlx ( http://www.totalcmd.net/plugring/decftumb.html ) knows to read thumbnails from EXIF.
IrfanView (newer versions) and XnViev also knows that, but only in their own browsers, not through Total Commander.
Imagine.wlx do not know to read thumbnails from EXIF.

Typically EXIF thumbnail is 160x120 pixels, so 160x160 should be best size (for speed) in TC configuration.

---

"Exifer" ( http://www.exifer.friedemann.info/ ) can create, if they does not exist, the JPEG thumbnails.

Another freeware - "EXIF Thumbnail Creator" ( http://exifthumbnailcreator.suconsultorti.es/ ).
Unfortunately decFastThumbs.wlx can not see this thumbs, while the Irfan and XnViev can (but only in their own browsers).


EDIT:

Ups, I forgot "Photo Studio" ( http://www.stuffware.co.uk/photostudio/index.html ), perhaps the best program for creating thumnails in EXIF.

Also, "ExifTool" (command-line app) would have to know to make thumbnail, but I have never found a command for that ( http://www.sno.phy.queensu.ca/~phil/exiftool/#writing ).

Also it's possible a combination of the following two apps:
- "jhead" command-line app ( http://www.sentex.net/~mwandel/jhead/ ) - can insert external thumbnail in JPEG,
- XnView - can re-create, from original image, already existing thumbnail.