timely-util/builder_async: offer automatically yielding give API #27866
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
petrosagg
requested review from
aljoscha,
danhhz,
jkosh44 and
a team
as code owners
petrosagg
force-pushed
the
fueled-give2
branch
2 times, most recently
from
a1f6b22
to
d568e3d
Compare
Normal `Capability` values are the only thing currently exposed to users so there is no use for that trait anymore. Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
petrosagg
requested review from
antiguru
and removed request for
aljoscha,
danhhz and
a team
They are now synchronous so this commit removes all the `.await`s. Signed-off-by: Petros Angelatos <petrosagg@gmail.com>
antiguru
approved these changes
Jun 26, 2024
|
|
||
| /// Provides one record at the time specified by the capability. This method will automatically | ||
| /// yield back to timely after [Self::MAX_OUTSTANDING_BYTES] have been produced. | ||
| pub async fn give_fueled<D2>(&mut self, cap: &Capability<T>, data: D2) |
There was a problem hiding this comment.
Note to future selves: Should this be #[inline]?
There was a problem hiding this comment.
I believe this will always be considered for inlining (at least according to this article https://matklad.github.io/2021/07/09/inline-in-rust.html#Inlining-in-Rust) because the method is generic
5 tasks
7 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Motivation
This PR offers a fueled give API that automatically yields back to timely after certain number of bytes have been emitted into the dataflow. The value is currently capped at 128MB, which is the size of a persist part. This API is only available for operators that make use of
StackWrappercontainers which can accurately track their heap size.As part of this PR I also made the original
giveandgive_containermethods synchronous. The only reason they were async in the first place was because I had initially envisioned this auto-yield API to be the default and so the async-ness was reserved. Since I no longer think this is a good idea I removed the async keyword.Tips for reviewer
The diff is big due to the last commit that contains all the removed
.awaitcalls. Only the changes intimely-utilneed review. Everything else is mechanical.Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.