read_excel leaves ray logs open

[jeffreykennethli]

Describe the problem

After running read_excel, there are open file handles for ray session logs:

in ipython:

in test_io.py:

Interestingly for test_io.py::TestExcel, only one of the test cases fails with this assertion if you run them together, but if you run them individually, they all fail.

Is related to #2559 but the root cause is probably different. To check open files, use

import psutil
p = psutil.Process()
# run code
print(p.open_files())

[devin-petersohn]

@modin-project/modin-ray Do we need to manually close the remaining open file handle(s) for ray logs?

[rkooo567]

Files are open because Ray keeps reading them to stream logs to the driver I think. You can probably do ray.init(log_to_driver=False), then it is not open anymore?

[jeffreykennethli]

Tried log_to_driver=False, initializing then shutting down ray (ray should be shutdown when the process that uses it (read_excel) doesn't need it, and the open file still appears there:

More generally, is it intended for the stream logs to still be open after some work is done? If so, maybe the solution on our end is to ignore open driver logs. If not, then @rkooo567 do you have any other ideas how to make sure the stream logs are closed?

[rkooo567]

Yes it is intended to keep opening files after some work is done. https://github.com/ray-project/ray/blob/5298a9046c68c8577ee5b01371c7aa0a95a915e3/python/ray/_private/log_monitor.py#L235

Seems like files are only closed when there are too many open files (threshold is 200 by default). Otherwise, we keep them open so that we can keep reading new log lines from the file to stream logs to drivers.

However, I think files keep being open after ray.shutdown could be a bug. Files being open with log_to_driver=False seems a bit weird to me. I suspect the log monitor component doesn't have a signal handler to close all opened files.

For this one, can you create an issue to Ray Github?

[jeffreykennethli]

Thank you! I will refactor the tests to ignore the ray driver logs then.

Opened issue ray-project/ray#20991.