File.CopyTo: try to pre-allocate disk space for large destination files #61676
Conversation
adamsitnik
added
area-System.IO
os-linux
|
Tagging subscribers to this area: @dotnet/area-system-io Issue DetailsI am currently re-running all the dotnet/performance#2134 benchmarks and will post the results soon. From initial quick run I was able to get up to x2 for large files.
|
adamsitnik
force-pushed
the
fileCopyUsePreallocationSize
branch
from
15922c7
to
3a27e9f
Compare
I see these benchmarks do a single copy. What file system is used in on the benchmark machine? ext4?
Looking at the absolute numbers: |
Is there any chance that your benchmarks were using Or performing an operation that runs very slow at first execution but performs much better for every next call? (in such case BDN could under estimate the number of invocations per iteration). Do you have the source code by any chance? |
Yes, ext4.
To me as well. @tmds Is there any chance you could run them on your machine and share the results? git clone https://github.com/dotnet/performance.git
python3 ./performance/scripts/benchmarks_ci.py -f net7.0 --filter '*File.CopyTo*' |
|
My macOS and Ubuntu 18.04 results: BenchmarkDotNet=v0.13.1.1616-nightly, OS=macOS Big Sur 11.4 (20F71) [Darwin 20.5.0]
Intel Core i7-5557U CPU 3.10GHz (Broadwell), 1 CPU, 4 logical and 2 physical cores
.NET SDK=7.0.100-alpha.1.21566.20
[Host] : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
Job-VREZES : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
BenchmarkDotNet=v0.13.1.1616-nightly, OS=ubuntu 18.04
Intel Xeon CPU E5-1650 v4 3.60GHz, 1 CPU, 12 logical and 6 physical cores
.NET SDK=7.0.100-alpha.1.21566.20
[Host] : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
Job-GTINUV : .NET 7.0.0 (7.0.21.56201), X64 RyuJIT
|
|
On my machine, which has Fedora 34 with btrfs Adding the
I see the same thing when using
The performance improvement is ext4 specific. From the benchmark results, it seems it is faster to delete and create a new file than it is to overwrite a file. It is surprising. |
The code is placed after the usage of I'll run the benchmarks against tmpfs and see if I can get the btrfs numbers as well. |
It would be interesting to see a flame graph that shows where this unexpected slowness comes from. If I'd make a guess, the ftruncate has a cost on ext4 that depends on the file size that gets truncated. And the fallocate added in this PR undoes that cost. |
|
If we add The tricky part is that For |
|
Regardless of the performance delta, preallocation can be highly advantageous on NTFS to reduce fragmentation. NTFS can fragment absolutely hideously, like placing each cluster in a separate fragment (without necessity). I had good experiences with preallocating. |
Preallocation performance benefits are file system type dependent. afaik, for 'Linux filesystems' improvements have only been observed with ext4. |
…ationSize # Conflicts: # src/native/libs/System.Native/pal_io.c
|
I can't finish this experiment as I simply have a lot of other more important things to deliver, so I am going to close the PR. To make sure this opportunity is not lost I've created a new |
ghost
locked as resolved and limited conversation to collaborators
Using the dotnet/performance#2134 benchmarks I got the following results:
As we can see there is no regression for small files. For other files we can see:
File.CopyTowhere we write to a new file.File.CopyTo(overwrite: true)where we overwrite an existing file. To be honest I am surprised that the gain is so huge.