Media repository reads files from disk on the main thread.

[erikjohnston]

It does use FileSender, which reads the file in chunks and so shouldn't be too slow, but nonetheless there is potential for disk I/O to take a long time (especially with networked storage).

We should look into either a) doing async file I/O (though that is likely hard) or b) do the reads from a thread.

[richvdh]

doing async file I/O (though that is likely hard)

it really shouldn't be: twisted already has support for doing async socket I/O: files are no different. Let's not be scared of looking into it, anyway.

[erikjohnston]

In Linux AFAIK setting the open file descriptor to non blocking has no effect for files, I can't really find a good source for that other than random blogs and SO answers https://stackoverflow.com/a/16513897

I believe the three options for doing async file I/O on linux:

1. Use POSIX AIO interface, which is a glibc thing that works in userspace via thread pools;
2. Linux AIO, which uses io_submit and the like https://kkourt.io/blog/2017/10-14-linux-aio.html; or
3. Using the new io_uring interface, which requires a new kernel

[richvdh]

ouch. TIL. thanks.

[clokep]

Using the new io_uring interface, which requires a new kernel

There was an interesting mailing list thread about adding support to this to Twisted: https://twistedmatrix.com/pipermail/twisted-python/2021-January/065382.html