-
Notifications
You must be signed in to change notification settings - Fork 592
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Optimize thread_worker #13056
Optimize thread_worker #13056
Conversation
392564c
to
4980680
Compare
dfab236
to
9e3490d
Compare
9e3490d
to
76ab1e6
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
lgtm
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks great.
I wonder if it makes sense to add an abort_source
, or fail pending requests on stop()
?
Use a ring buffer to bulk pass tasks around, and also support creating a twin worker thread for every reactor thread. Before: ``` test iterations median mad min max allocs tasks inst thread_worker_test.1 3354 216.749us 47.062ns 216.590us 216.951us 5.000 4.000 1610844.2 thread_worker_test.10 460 2.088ms 870.335ns 2.087ms 2.089ms 70.000 58.000 25391981.4 thread_worker_test.100 61 16.465ms 1.733us 16.461ms 16.469ms 700.000 598.000 208106393.1 thread_worker_test.1000 50 19.925ms 6.187us 19.919ms 19.937ms 7007.000 5998.000 214776807.2 ``` After: ``` test iterations median mad min max allocs tasks inst thread_worker_test.1 1855 216.515us 77.552ns 216.330us 216.593us 4.000 4.000 1549639.0 thread_worker_test.10 413 2.085ms 394.971ns 2.084ms 2.086ms 40.000 40.000 24921287.6 thread_worker_test.100 113 8.518ms 3.891us 8.513ms 8.524ms 400.000 400.000 106121583.5 thread_worker_test.1000 93 10.424ms 605.968ns 10.423ms 10.426ms 4880.000 5744.000 127114775.7 ``` Signed-off-by: Tyler Rockwood <rockwood@redpanda.com>
76ab1e6
to
1edf4c8
Compare
I am not really sure how to do this correctly, as I don't think seastar classes are designed to support work outside of seastar, unless there is a different technique you're suggesting. Same goes for failing pending stuff on |
The stop signal was previously passed via the I'm not saying it has to be done, it's just a thought. Propagating an abort (somehow) might be more useful than just cancelling the work. Or it might just make sense to drain the queue; it probably depends what kind of work it's running. |
That's a little hard to do now that there are multiple signals in the normal
Agreed, I don't know about kerberos, but for wasm, cancelling does make since, but that will have already happened as the transform processor needs to shutdown before wasm does. |
Kerberos is usually just reading a local file, so I/O, but not too bad, and unlikely to build a large backlog. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM, let's keep an eye on the queue length at shutdown. Should there be metrics for queue length, time in queue, etc?
Feel free to add a followup issue.
I added a followup task for metrics - Also the queue is limited to 128 items for now. |
Use a ring buffer to bulk pass tasks around, and also support creating a
twin worker thread for every reactor thread.
This amortizes the cost of running a task on an alien thread, this came about because we're going to be switching transforms over to Wasmtime running on an alien thread.
Before:
After:
Backports Required
Release Notes