perf(python): Use underlying fileno for Python files when possible

[ruihe774]

If a Python file object is passed as the argument of IO functions (e.g. read_* & write_*), currently it is wrapped into a PyFileLikeObject, and each read & write will acquire the GIL and call into Python, which is inefficient. This PR introduces a improvement that extracts the underlying fd through fileno() and opens the fd as a Rust File, which does not require GIL and have better performance.

[codecov]

Codecov Report

Attention: Patch coverage is 93.27731% with 8 lines in your changes missing coverage. Please review.

Project coverage is 80.70%. Comparing base (59d2529) to head (ca6a1f6).
Report is 9 commits behind head on main.

Files	Patch %	Lines
py-polars/src/file.rs	91.30%	6 Missing ⚠️
py-polars/src/functions/io.rs	50.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main   #17315      +/-   ##
==========================================
- Coverage   80.73%   80.70%   -0.03%     
==========================================
  Files        1475     1475              
  Lines      193238   193388     +150     
  Branches     2760     2760              
==========================================
+ Hits       156013   156080      +67     
- Misses      36715    36798      +83     
  Partials      510      510              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[ghuls]

You can't always skip the python read functions as they might do something more complex than just reading the file descriptor as it is: e.g. reading a compressed file format not supported by Polars, or nested compressed files.

$ head -n 1000 test.csv | gzip | gzip | gzip > test.csv.gz.gz.gz


In [19]: with gzip.open("test.csv.gz.gz.gz") as fh:
    ...:     with gzip.open(fh) as fh2:
    ...:         df = pl.read_csv(fh2)

[ruihe774]

You can't always skip the python read functions as they might do something more complex than just reading the file descriptor as it is: e.g. reading a compressed file format not supported by Polars, or nested compressed files.

You're right. I limit this optimization to builtin file IO types in 4257873.

[ruihe774]

I also add an improvement: use inode number in MMapSemaphore instead of path, which better handles hard links and symbolic links.

[ruihe774]

I also add an improvement: use inode number in MMapSemaphore instead of path, which better handles hard links and symbolic links.

A better way is to use open file description locks, and apply it to all file IO, not only mmap, to avoid data race in multithread/multiprocess conditions. I can implement it in the future.

[ritchie46]

Thanks @ruihe774, this seems like an improvement indeed.