Sparse file support

[white]

So apparently the Properties function (which is part of Windows) does something causing the duplication of the data?

It seems that way. I get the same behavior with 40GB files, also on my real system on a physical harddisk (2 TB, not an SSD drive). There must be some real data patched in the file for it to happen.

Edit:
I tested a sparse file created with fsutil and the issue did not occur. So there seems to be an issue with mksparse.exe when patching a file.

[j7n]

I heard about this feature being considered for removal immediately after it was added. The use cases for sparse files aren't very clear. I'm sure someone can use them if once they discover the feature later. It seems to be more useful in the program that makes large files for whatever reason, to reduce writes to the disk. Once these files already exist, TC can now make compacted copies of them. Right? Such as reducing a 10 gig flat file to 1 gig + 9 gig of zeros remembered. But the intial problem of writing to the disk has already been done.

[ghisler(Author)]

also on my real system on a physical harddisk

What do you mean with that? Nothing happens here on a real system when I alt+enter on a sparse file.

[white]

also on my real system on a physical harddisk

What do you mean with that? Nothing happens here on a real system when I alt+enter on a sparse file.

The thing I did was: First view the properties, then close the properties window, then search for a text in the file in TC's Lister (for example for a non-existing text).

But as said, it seems to be caused by your mksparse.exe program. When I use fsutil to make a sparse file, the issue doesn't occur. When I use mksparse.exe, the issue does occur. So it seems like your program is creating invalid files when patching files.

[ghisler(Author)]

I can reproduce the problem now also with fsutil: The problem only occurs if the sparse file isn't completely empty.

Example creation:

Type NUL > temp
FSUtil Sparse SetFlag temp
FSUtil Sparse SetRange temp 0 0x40000000
FSUtil File SetEOF temp 0x40000000
echo abcd >> temp

After Alt+Enter and searching, the available disk space is reduced by the nominal size of the sparse file, although the size hasn't changed with Alt+Enter, and the swap file hasn't grown either.

[white]

I can reproduce the problem now also with fsutil: The problem only occurs if the sparse file isn't completely empty.

Example creation:
..

Try it like this:
(Example with 40GB file, make sure you have 40GB free space or use a smaller size)
(The copy operation takes a while but doesn't need an additional 40GB space)

Code: Select all

fsutil file createnew testfile.bin 42949672956

copy con 4bytes
1234^Z

copy /b testfile.bin+4bytes

fsutil sparse setflag testfile.bin

fsutil sparse setrange testfile.bin 0 42949672956

[white]

Try it like this:

The file created like this, doesn't create the problem after viewing the file's properties. But after creating a copy of this file using TC, the copy does have the problem..

[white]

But after creating a copy of this file using TC..

Or robocopy..

The same things happens when using "copy testfile.bin nul" instead of searching with TC's Lister.

So it seems that disk space reducing after viewing properties, is not related to TC (and mksparse.exe) because it also happen when using Windows tools only.

Preliminary conclusion:

There may be a Windows bug where opening the Properties window of a sparse file with at least some allocated space, can cause incorrect reservation of disk space during subsequent reads (e.g. when running the command: copy testfile.bin nul). Selecting 2 files and viewing the properties of the 2 files together does not trigger this behavior.

The space is added to the "Total Reserved Clusters" as can be seen when issuing these commands (as Administrator):
fsutil volume diskfree c:
fsutil fsinfo ntfsinfo c:

The space is freed when the sparse file is deleted. (You can copy the file first.)

Recommendation:

Don't view the properties of sparse files, but use other tools, for example Sysinternals' Disk Usage, if you want to know the allocated disk space.