Sparse file support

[ghisler(Author)]

This was suggested via e-mail, and I have implemented it now in TC 11.55 rc1. Therefore it would be great if someone could try it!

The NTFS file system supports so-called sparse files, there are files where not all the space has been allocated - instead, only blocks of data containing non-null bytes are allocated. This can save a lot of space, e.g. in large disk images which are mostly empty.

Windows 11 22H2 and newer now allows to copy such sparse files (preserving the allocated parts only) with function CopyFile2. On older versions of Windows, I'm copying the parts manually.

To experiment with sparse files, I have created a small tool to create them:
https://www.totalcommander.ch/win/tools/mksparse.zip

The upper part of the dialog allows to create one or more completely empty sparse files which take up 0 bytes on disk. You can see the allocated space with Alt+Enter in Total Commander.

The lower part allows to patch data in existing files, which would allocate the data in sparse files, so their size is then larger than 0. You will notice that Windows allocates an entire block of, for example, 64 kBytes.

When you copy such a file with older versions of Total Commander or with the Explorer, the target file will allocate the entire file size. With TC 11.55 rc1, however, the copied file will not occupy more space than the source file.

[white]

I did some testing on this. So far it seems to work OK using TC 11.55rc5 32-bit and 64-bit.

I tested:

• Copying sparse files sized 1MB, 1GB, 1TB. Also when patched with a little data somewhere in it.
• Moving files.
• Copying and moving among 2 different physical drives.
• Renaming files.

Everything seemed to work OK, apart from when on Sandbox, things went sour and Windows reported no space left on the drive. Then navigating in Lister to for example 50% resulted in disk read errors (although skipping right to the end worked fine and I could still copy 1TB sparse files).

To other users wanting to test, here are some tips:

• As said by Ghisler, you can check how much space is really allocated on disk by looking at the file(s) properties (Alt+Enter).
• Create a custom column view in TC that includes "tc.sparse" and "tc.size".
• Be careful when creating very "large" sparse files on your real system. It's safer to work with smaller files or test on a virtual environment.
• mksparse.exe seems to default to location %USERPROFILE%\bigfile. You can simply enter a filename without path to create or patch a file in the current folder (if you create a button for mksparse.exe, leave the Start path empty).
• It's more convenient to leave the mksparse.exe open, so you don't have to edit the filename every time.
• The lower part of the mksparse.exe program uses the same filename specified at the top.
• The offset you can specify in the lower part is in decimals.
• If you use an offset bigger than the file size, the file is made bigger.

I don't know what went wrong on my Sandbox. It seems that some of my sparse files ended up to occupy all remaining space while Windows reported little space to be allocated for them. After deleting the sparse files, I got the space back and could not reproduce it again.

[ghisler(Author)]

Thanks for your tests! What did you use to create a Sandbox? Sandboxie?

[white]

Thanks for your tests! What did you use to create a Sandbox? Sandboxie?

I used Windows Sandbox, included (if enabled) in my Windows version (Windows 11 Pro).
(I did also do tests on my real system, but not with 1TB files)

[ghisler(Author)]

I see, I will try that.

You WILL get a warning about not enough space if the free space on the disk is less than the total size of the sparse file (not the allocated size), because not all target drives support sparse files, and there can be mount points anywhere in the target which do not support sparse files. I assume that this is what you experienced with the sandbox.

[white]

You WILL get a warning about not enough space if the free space on the disk is less than the total size of the sparse file (not the allocated size), because not all target drives support sparse files, and there can be mount points anywhere in the target which do not support sparse files. I assume that this is what you experienced with the sandbox.

I think the problem was that I tried to do some other operations on 1TB files, like viewing in Lister and then searching for text, and creating a checksum and then verifying it. It doesn't seem surprising that things go wrong when testing this with 1TB files on a 80GB virtual drive. Copying worked fine.

[ghisler(Author)]

Did you create a blake3 checksum? I ask because it maps parts of the file into memory, so it couldn't be accessed from Lister at the same time.

[white]

Did you create a blake3 checksum? I ask because it maps parts of the file into memory, so it couldn't be accessed from Lister at the same time.

Yes, but I did not use them at the same time.

[ghisler(Author)]

I made a strange discovery: When I create a blake3 checksum of a huge sparse file in the sandbox, the available disk space in there goes down from about 80 GB to 0. I can get that back only by deleting the sparse file. This seems to be a bug of the sandbox when mapping a file into memory. The file is mapped as read only, so no data should be allocated. The sparse file still shows only 64k allocated. People who have this problem should disable blake3 multi-threading.

[white]

When I create a blake3 checksum of a huge sparse file in the sandbox, the available disk space in there goes down from about 80 GB to 0.

Same when searching with lister. I created an 1TB file, patched some data at the end, opened the file in lister and searched for the text at the end. The free space dropped to zero and a disk error occurred. When I then deleted the 1TB file, the free space was back.

[ghisler(Author)]

Same when searching with lister.

Not here - Lister doesn't map the data into memory. Maybe you searched in "Compare by content"? It maps large files into memory like the blake3 checksum function. Or maybe you used a Lister plugin?

[white]

Same when searching with lister.

Not here - Lister doesn't map the data into memory. Maybe you searched in "Compare by content"? It maps large files into memory like the blake3 checksum function. Or maybe you used a Lister plugin?

Interesting. It only happens when using 64-bit TC. I tested it again with both 64-bit TC and 32-bit TC.

I usually start out using 64-bit TC for testing, I guess you start out with 32-bit TC?

[ghisler(Author)]

No, I tried with both 64-bit and 32-bit TC. While the blake3 problem only happens with 64-bit (because only 64-bit uses file mapping), there is no problem here with Lister search in both 64-bit and 32-bit.

[white]

.. there is no problem here with Lister search in both 64-bit and 32-bit.

I found out the reason. It happens when I first check the properties of the file (and close the properties) and then open Lister and do the search. It happens with 32-bit TC as well.

[ghisler(Author)]

So apparently the Properties function (which is part of Windows) does something causing the duplication of the data? I don't think there is anything I can do to fix that.