Linux - FileStream.Length, FileStream.Seek & File.ReadAllBytes misbehaves with files in procfs

[VasiliyNovikov]

I did the following experiment:

using System;
using System.IO;

namespace TestProcStatFile
{
    class Program
    {
        static void Main(string[] args)
        {
            const string procStat = "/proc/stat";
            using (var file = File.OpenRead(procStat))
            {
                Console.WriteLine($"Stream length: {file.Length}");
                Console.WriteLine($"Actual length (via File.ReadAllBytes): {File.ReadAllBytes(procStat).Length}");
                Console.WriteLine($"Actual length (via File.ReadAllText): {File.ReadAllText(procStat).Length}");

                Console.WriteLine($"Can seek: {file.CanSeek}");
                Console.WriteLine($"Seek(8192), Position: {file.Seek(8192, SeekOrigin.Begin)}");
                try
                {
                    file.Seek(0, SeekOrigin.End);
                }
                catch (Exception e)
                {
                    Console.WriteLine($"Seek from end: {e}");
                }
            }
        }
    }
}

Which reports the following:

Stream length: 0
Actual length (via File.ReadAllBytes): 0
Actual length (via File.ReadAllText): 5085
Can seek: True
Seek(8192), Position: 8192
Seek from end: System.IO.IOException: Invalid argument
   at System.IO.FileStream.CheckFileCall(Int64 result, Boolean ignoreNotSupported)
   at System.IO.FileStream.Seek(Int64 offset, SeekOrigin origin)
   at TestProcStatFile.Program.Main(String[] args) in /home/netem/RiderProjects/TestProcStatFile/Program.cs:line 21

https://github.com/dotnet/corefx/blob/a8cfc6a498f0455c59b0030f74cba4dabeee86e1/src/System.IO.FileSystem/src/System/IO/File.cs#L320
This guy relays on FileStream.Length. I'm thinking if FileStream.Length should actually throw for files in procfs and CanSeek should return false for them and File.ReadAllBytes should check CanSeek and if it is false then don't pre-allocate buffer of the final size but allocate it dynamically while reading

[tmds]

Almost all files in proc have zero size and should be read till EOF.
These files represent some live state of the system, and they don't have a content until you request the state (i.e. read).
You can seek these files. Seeking to the start is a way of refreshing the content of the file.

Since this is a special file system, I suggest to not make FileStream more complex to support this.
We could do some internal implementation that could be shared between various readers of proc in the corefx repo.

[VasiliyNovikov]

Shouldn't we at least fix File.ReadAllBytes to not return empty result?

[tmds]

The challenge is that for proc (and similar) files we need to allocate a buffer up-front (and deal with it maybe being too small). Whereas for real empty files we don't want to be allocating a buffer at all.
And there isn't a good way afaik to distinguish between these files.

[VasiliyNovikov]

Maybe there is some API for that... E.g. stat -f /proc/stat shows it:

  File: "/proc/stat"
    ID: 0        Namelen: 255     Type: proc
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 0          Free: 0          Available: 0
Inodes: Total: 0          Free: 0

[tmds]

You mean, stat shows the Type 'proc'? This behavior isn't limited to the proc file system, any file system can do this.

[VasiliyNovikov]

You mean, stat shows the Type 'proc'?

Yes

This behavior isn't limited to the proc file system, any file system can do this.

What else can behave this way? Here is only proc mentioned:

For most files under the /proc directory, stat() does not return the file size in the st_size field; instead the field is returned with the value 0.

And I also tried to get size of files under /sys - it returns non-0 size

[tmds]

What else can behave this way? Here is only proc mentioned:

Any file system can. I have seen application that export a file system via fuse (https://github.com/libfuse/libfuse) which behave the same way.

[VasiliyNovikov]

I see. Thanks
What if we have some hardcoded list of well-know file systems - for all of them we know the behavior of file size. If file system is not in that list then assume file size is not reliable .... or vice versa?

[VasiliyNovikov]

Would it make sense to fix it not 100% of cases but 99 - only for known file systems?

[svick]

@tmds

Whereas for real empty files we don't want to be allocating a buffer at all.
And there isn't a good way afaik to distinguish between these files.

Would it work if you tried to read a single byte (or read into a small stack-allocated buffer)? If that succeeds, you know the file size is lying and you have to deal with it in a special way.

This would mean some seemingly unnecessary allocations and copying for proc and similar cases, but I think that's still better than returning the wrong result. And for actual empty files, there would be no heap allocations, and the total overhead should be fairly small.

[VasiliyNovikov]

Another thing we could try is to use the fact that seeking from the end fails for such files (but this needs to be tested with other file systems that return 0 size to make sure they behave the same way)

[tmds]

@svick it depends on the file system, but I think it should work for special file systems from the kernel. A small stack-allocated buffer would be good. Then we may have the full data already and we only need to copy it into a byte array.

[VasiliyNovikov]

And we can combine 2 approaches - for well-known file systems use hardcoded behavior, for unknown - test the file itself

[VasiliyNovikov]

Maybe I'm going too crazy with checking file system type. Probably reading just small buffer is even cheaper than determining file system type and it is definitely much simpler in terms of implementation
Q. Can Windows & OSX have the same behavior for some special file systems/files?

[tmds]

Probably reading just small buffer is even cheaper than determining file system type and it is definitely much simpler in terms of implementation

Yes, a stackalloced read should be enough.
If you implement this, please verify it gives the same result for every file in proc (when using a tiny stackalloced buffer).

Q. Can Windows & OSX have the same behavior for some special file systems/files?

I don't know for sure, but I'd assume this is Linux specific behavior.

[VasiliyNovikov]

B.T.W. You mentioned FUSE and I found this: https://osxfuse.github.io/ . Could this mean that OSX can also have the same behavior for those file systems?

[tmds]

Probably it can. It's fine if the change applies to all Unixes.

[VasiliyNovikov]

Thanks!