slow - compare file by content (and synchronize dir.)

[peterb]

Hi,

I noticed TC7 ist slower at file compares.
Specially slow on CD drives (this is where I noticed it
during a synchronize directory incl. content compare).

TC6 reads CD smoothly at comparing a file between CD and HD,
TC7 seems to read CD in chunks, more head movement, slower.

Has TC 7b3 a different file compare method (comparing single file
or in synchronize directories with content compare)? I remember
reading something, can not find it anymore.

Is it changeable back to TC6 behaviour by INI or configuration menu?

thanks
Peter

[ghisler(Author)]

TC7 reads data in larger blocks, this speeds up comparing two files on the same harddisk quite a lot. I did't notice any speed difference with CDs - have you disabled the read cache?

[peterb]

> have you disabled the read cache
How to do that?

Hmm, 2 hours later, lots of testing....
I cannot confirm speedup of compare.
TC 6.56 is appr. 2 - 3 times faster than TC 7b3.
That are very large differences.

Either there is an issue, or the failure is with my 2 tested systems,
or between my earphones

I tried single large files (500MB ZIP) for simple Compare by Contents.
For subdir using CbC I used directories (each total of appr 400 MB)
with WAV files (appr 60MB each) or normal MP3 files (3- 6 MB).
I tried to avoid caching issues as far as possible.

Tests between:
2 separate HD; same HD; HD <-> internal DVD (XPproSP1 and W98)
HD <-> USB DVD burner; HD <-> USB HD (XP)
XPproSP1: Thinkpad T41 (1GB RAM)
Win98: Desktop Intel P700 (512MB RAM)

Only on the slower W98 PC, the compare between 2 separate HD
resulted in simmiliar times. All other test times are far appart.
Tests over 100M LAN showed simmiliar times for TC6 and TC7.

Again, I noticed the internal DVD drive in the Thinkpad to create
much more access when using TC7b3 than TC6.56. The DVD at the
Win98 desktop seemed to be simmiliar when accessing.

Regarding setting, both TC are set to "copy method" standard, 32k,
no "compatibility mode", "USB stick" ticked (XP). INI files seems
to have no major differences. For TC7, I also tried to change
CompareIgnoreRepSpace and CompareIgnoreRepeatedLines
from 1 (standard?) to 0, no change.

Do you need any more tests? Do you want the INI files?

Have a good weekend,
Peter

[ghisler(Author)]

I checked my code - the only difference in the compare function is indeed the bugger size: 32kBytes in TC 6.56, and 1 MByte in TC7. A larger buffer should lead to a much faster compare speed than a smaller buffer when the files are on the same disk. Why? With a 32 times smaller buffer, the harddisk head must jump between the two files 32x as often as with the larger buffer! At least here this gives a huge speed improvement! I really don't see why it could be slower...

Regarding setting, both TC are set to "copy method" standard, 32k,

The copy mode has no influence on the compare by contents function.

In your case, is copying with a 32k buffer also faster than with a 1 MByte buffer (set sizes in the section "Standard method" both to 1024)?

[peterb]

In your case, is copying with a 32k buffer also faster than with a 1 MByte buffer (set sizes in the section "Standard method" both to 1024)?

Which settings do you refer to?
If you mean settings "Copy/Delete", which settings should I test?
For TC 6, TC 7, or both?

Copy method
- use default methode only
-- Buffer size for small files
or
-Also use big file copy mode
-- Buffer size for big files

thanks
Peter

[ghisler(Author)]

Which settings do you refer to?
If you mean settings "Copy/Delete", which settings should I test?
For TC 6, TC 7, or both?

It doesn't matter, both use the same.

- use default methode only
-- Buffer size for small files

This one, set both to 1024.

[peterb]

Christian,

first tests done, XPproSP1, Thinkpad T41 1.6Mhz CPU, 1GB Memory.
Settings as recomended, TC6 and 7, buffer 1024.
Same files as before (400MB .mp3).
HD internal <-> HD USB
HD internal 1 <-> HD internal 2

Sorry, no change, same results as before. TC6 still faster than TC7.
I will test DVD on XP and Win98 as well, but I do not expect changes.

Did you try large files (60MB) or many .mp3 ?

regards
Peter

[ghisler(Author)]

I was talking about copying on the same disk, not between two disks. When copying between two disks, small buffers have an advantage because you can read+write almost at the same time.

[peterb]

Ok, same disk now. Similiar results.

Synchronize directories with contents, 32 .mp3, 310MB total:
TC6 - 49 sec - 32k
TC6 - 40 sec - 1024k
TC7 - 82 sec - 32k
TC7 - 87 sec - 1024k

Zip file, 410MB
TC6 - 24 sec - 32k
TC6 - 23 sec - 1024k
TC7 - 48 sec - 32k
TC7 - 51 sec - 1024k

Do you want any more testing?

The very first test comparing disk to DVD was done at 32k setting.

Also, I noticed some differences between TC6 and TC7
when zipping large files.

Enjoy weekend
Peter

[Hacker]

Christian,
Hm, I also have very slow speeds when using synchronize dirs with "by content" activated between HDD <--> DVD-R. I never suspected TC, though. I will try it later in the evening with 6.56 to see if I can confirm peterb's findings.

Roman

[petermad]

I have done some tests too - all with these settings for Copy/Delete: http://madsenworld.dk/tcmd/copy-del.png

Synchronizing two zip files containing 3.8 Gb with 27.500 files on same drive (FAT32) (temp dir and swap file on other harddisk (NTFS)):
TC6.56: 8min 02 sec
TC7pb3: 9min 29sec

Synchronizing two directories containing 3.8 Gb with 27.500 files on different drives (FAT32 -> FAT32):
TC6.56: 3min 56sec
TC7pb3: 7min 15sec

Synchronizing two directories containing 3.8 Gb with 27.500 files on same drive (FAT32):
TC6.56: 18min 05 sec
TC7pb3: 13min 39sec

Synchronizing two directories containing 3.8 Gb with 27.500 files on same drive (NTFS):
TC6.56: 19min 15sec
TC7pb3: 15min 19sec

So TC7 is faster at synchronizing than TC6 on same drive but only half as fast on different drives.



Since I had to do some zip unpacking to perform the tests here are the timings for unpacking:

Unpacking a zip file containing 3.8 Gb with 25.500 files to other harddisk (FAT32 -> NTFS):
TC6.56: 4min 58sec
TC7pb3: 5min 13sec

Unpacking a zip file containing 3.8 Gb with 27.500 files to other harddisk (FAT32 -> FAT32):
TC6.56: 5min 18sec
TC7pb3: 6min 45sec

Unpacking a zip file containing 3.8 Gb with 25.500 files to same harddisk (FAT32):
TC6.56: 10min 14sec
TC7pb3: 12min 52sec

So in general TC6 is faster at unpacking than TC7 !

[ghisler(Author)]

OK, I will revert to 32k buffers, although I have the exact opposite effect here...

[Hacker]

OK, I just tested Synchronize dirs with "by content" checked on a DVD+R backup (four gigs of 350 MB files, the rest 20 MB files), DVD+R <--> HDD:

TC 6.56 - 08:48
TC 7pb3 - 66:48

Roman

[peterb]

Hi,

7b4 is back to normal speed when using sync/compare by contents.
Speed is like with 6.56, if not a little bit faster.

I have done only a few quick tests, but I believe that is enough.
disk to disk, disk to usb disk, disk to dvd/cd-rom

DVD/CD-Rom access is also back to normal (checked on Thinkpad).
Reading is smooth, continous. 7b3 made the cd drive reading in
small chunks, at least it sounded like. The cd drive / disk was
constantly reaccessed, I guess that problem made the disk to disk
compare also slow.

Anyhow, thanks for fixing,
Peter

PS: can anybody else confirm this fix, petermad, hacker ?

[Hacker]

Yep, I also confirm.

Roman