[Breaking change] FileStream does not synchronize file offset with OS anymore

[adamsitnik]

Before .NET 6, FileStream was using expensive syscall called SetFilePointer to synchronize it's private offset with Windows OS.

A blog post from Windows Performance Team from 2008 calls it an anachronism:

The old DOS SetFilePointer API is an anachronism. One should specify the file offset in the overlapped structure even for synchronous I/O. It should never be necessary to resort to the hack of having private file handles for each thread.

In order to improve the performance of async reads and writes and solve the following issues:

• Win32 FileStream will issue a seek on every ReadAsync call #16354 Win32 FileStream will issue a seek on every ReadAsync call
• FileStream.Windows useAsync WriteAsync calls blocking apis #25905 FileStream.Windows useAsync WriteAsync calls blocking apis

FileStream is no longer doing that (#49975) and the offset is just kept in memory. This has allowed for up to two times faster ReadAsync and up to five time faster WriteAsync! See #49975 for full details.

FileStream.Position always returns the current offset, but if the user obtains the file handle via call to FileStream.SafeFileHandle and asks the OS for the current file offset of the given handle by performing a syscall, the value will return 0.

[DllImport("kernel32.dll")]
private static extern bool SetFilePointerEx(SafeFileHandle hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);

byte[] bytes = new byte[10_000];
string path = Path.Combine(Path.GetTempPath(), Path.GetTempFileName());

using (FileStream fs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite, FileShare.None, bufferSize: 4096, useAsync: true))
{
    SafeFileHandle handle = fs.SafeFileHandle;

    await fs.WriteAsync(bytes, 0, bytes.Length);
    Console.WriteLine(fs.Position);

    if (SetFilePointerEx(handle, 0, out long currentOffset, 1 /* get current offset */))
    {
        Console.WriteLine(currentOffset);
    }
}

Pre-change behavior:

10000
10000

Post-change behavior:

10000
0

It works in the other direction as well: if the user obtains filehandle and performs a syscall that moves the file offset, the offset returned by FileStream.Position won't be changed.

To enable the .NET 5 behavior, users can specify an AppContext switch or an environment variable:

{
    "configProperties": {
        "System.IO.UseNet5CompatFileStream": true
    }
}

set DOTNET_SYSTEM_IO_USENET5COMPATFILESTREAM=1

@dotnet/compat @stephentoub @jozkee @carlossanlop @jeffhandley

FWIW I've ensured that this change is not breaking ASP.NET (dotnet/aspnetcore#31441), WinForms (dotnet/winforms#4756) and the SDK (dotnet/sdk#16684)

[ghost]

Tagging subscribers to this area: @carlossanlop
See info in area-owners.md if you want to be subscribed.

Issue Details

---

Before .NET 6, FileStream was using expensive syscall called SetFilePointer to synchronize it's private offset with Windows OS.

A blog post from Windows Performance Team from 2008 calls it an anachronism:

The old DOS SetFilePointer API is an anachronism. One should specify the file offset in the overlapped structure even for synchronous I/O. It should never be necessary to resort to the hack of having private file handles for each thread.

In order to improve the performance of async reads and writes and solve the following issues:

• Win32 FileStream will issue a seek on every ReadAsync call #16354 Win32 FileStream will issue a seek on every ReadAsync call
• FileStream.Windows useAsync WriteAsync calls blocking apis #25905 FileStream.Windows useAsync WriteAsync calls blocking apis

FileStream is no longer doing that (#49975) and the offset is just kept in memory. This has allowed for up to two times faster ReadAsync and up to five time faster WriteAsync! See #49975 for full details.

FileStream.Position always returns the current offset, but if the user obtains the file handle via call to FileStream.SafeFileHandle and asks the OS for the current file offset of the given handle by performing a syscall, the value will return 0.

[DllImport("kernel32.dll")]
private static extern bool SetFilePointerEx(SafeFileHandle hFile, long liDistanceToMove, out long lpNewFilePointer, uint dwMoveMethod);

byte[] bytes = new byte[10_000];
string path = Path.Combine(Path.GetTempPath(), Path.GetTempFileName());

using (FileStream fs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite, FileShare.None, bufferSize: 4096, useAsync: true))
{
    await fs.WriteAsync(bytes, 0, bytes.Length);
    Console.WriteLine(fs.Position);

    if (SetFilePointerEx(fs.SafeFileHandle, 0, out long currentOffset, 1 /* get current offset */))
    {
        Console.WriteLine(currentOffset);
    }
}

Pre-change behavior:

10000
10000

Post-change behavior:

10000
0

It works in the other direction as well: if the user obtains filehandle and performs a syscall that moves the file offset, the offset returned by FileStream.Position won't be changed.

To enable the .NET 5 behavior, users can specify an AppContext switch or an environment variable:

{
    "configProperties": {
        "System.IO.UseNet5CompatFileStream": true
    }
}

set DOTNET_SYSTEM_IO_USENET5COMPATFILESTREAM=1

@dotnet/compat @stephentoub @jozkee @carlossanlop @jeffhandley

FWIW I've ensured that this change is not breaking ASP.NET (dotnet/aspnetcore#31441), WinForms (dotnet/winforms#4756) and the SDK (dotnet/sdk#16684)

Author:	adamsitnik
Assignees:	-
Labels:	area-System.IO, breaking-change
Milestone:	6.0.0

[carlossanlop]

@adamsitnik since both #16354 and #25905 got fixed, is there anything else we should address before we close this issue?

Since this is a breaking change that needs to be documented, I opened this issue in dotnet/docs: dotnet/docs#23857

[jozkee]

The current code doesn't actually shows that the Position is not being synchronized with the OS because you are accessing the SafeFileHandle which issues a Seek before returning the handle and thus, the OS knows that you changed the position.

We should expose the handle before writing:

using (FileStream fs = new FileStream(path, FileMode.Create, FileAccess.ReadWrite, FileShare.None, bufferSize: 4096, useAsync: true))
{
+    SafeFileHandle handle = fs.SafeFileHandle;

    await fs.WriteAsync(bytes, 0, bytes.Length);
    Console.WriteLine(fs.Position);

+    if (SetFilePointerEx(handle, 0, out long currentOffset, 1 /* get current offset */))
-    if (SetFilePointerEx(fs.SafeFileHandle, 0, out long currentOffset, 1 /* get current offset */))
    {
        Console.WriteLine(currentOffset);
        Console.WriteLine(fs.Position);
    }
}

[adamsitnik]

@jozkee great point! It just shows how hard it is to hit the breaking change scenario ;)

[adamsitnik]

Closing (dotnet/docs#24060)