-
-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tokio::io::bsd::Aio can cause multithreaded runtime to busy-loop #4728
Comments
I've confirmed that using EV_ONESHOT fixes the problem. That means that the major fix will happen in the tokio-file crate, but Tokio will still need a doc and tests fix. The hard part is that this involves C unions, and backwards-incompatible changes to libc. Sigh. |
I guess documenting that you must use EV_ONESHOT is a possible solution. |
asomers
added a commit
to asomers/nix
that referenced
this issue
Jun 4, 2022
asomers
added a commit
to asomers/nix
that referenced
this issue
Jun 4, 2022
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Jun 4, 2022
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 11, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 11, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 12, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 20, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 20, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
asomers
added a commit
to asomers/nix
that referenced
this issue
Aug 26, 2023
Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728
github-merge-queue bot
pushed a commit
to nix-rust/nix
that referenced
this issue
Aug 27, 2023
* Allow setting kevent_flags on struct sigevent Also, disallow using SigevNotify::SigevThreadId on musl. I don't think it ever worked. Depends on rust-lang/libc#2813 Blocks tokio-rs/tokio#4728 * Inline libc::sigevent Because the PR to libc is stalled for over one year, with no sign of progress.
asomers
added a commit
to asomers/tokio-file
that referenced
this issue
Aug 29, 2023
So tokio-file can be used by a multithreaded tokio runtime. See also tokio-rs/tokio#4728
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Version
Tokio git master at f6c0405
Platform
FreeBSD 14.0-CURRENT amd64
Description
Background
The funny thing about AIO is that you don't register it with
kevent
. Instead, you register it by setting a certain field within thestruct aiocb
that you submit with a syscall likeaio_write
. When it's ready,kevent
will return an event withEVFILT_AIO
set. The user must then callaio_return
to free the kernel resource, or elsekevent
will keep reporting that event as ready. As written now, theaio_return
call happens outside of Tokio, in thetokio-file
crate, in a custom Future'spoll
method.Problem
With the multi-threaded runtime, it's possible that the future's task is on one thread, but the kqueue it's registered to is being polled by a different thread. When the kernel notifies the polling thread, that thread will signal (how I'm not sure) the thread with the future. Then it will go back to sleep waiting for more events. But if the thread with the future doesn't poll it quickly, then polling thread will immediately get woken again. This can lead to many more
kevent
calls than should be necessary. In my application, I see up to 200x morekevent
calls thanaio_return
calls.Possible Solutions
struct aiocb
into the IODriver
, and arrange for the IO driver to promptly callaio_return
upon notification, storing the result.EV_ONESHOT
to make the aio operation edge-triggered. I'm not sure if this works with POSIX AIO.Example
This test case will obviously never finish. Ideally it would simply hang. But because nothing ever calls
aio_return
, the IO driver will busy loop aroundkevent
, and it will spin the cpu.The text was updated successfully, but these errors were encountered: