-
Notifications
You must be signed in to change notification settings - Fork 95
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
rust: add async reading functionality #1211
Conversation
17cf451
to
0f675fe
Compare
bfe547a
to
c534658
Compare
7f0ef87
to
75ba3c0
Compare
04233ed
to
39b00f9
Compare
@mrkline do you have strong opinions on |
10264f3
to
f96c691
Compare
f96c691
to
41bf547
Compare
41bf547
to
1056c7c
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think this needs one change to a test, and the rest is optional
if remaining == 0 { | ||
return std::task::Poll::Ready(Ok(())); | ||
} | ||
let to_fill = min(min(remaining, buf.remaining()), max_read_len); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The problem with this is it fills the buffer in one shot. So the read loop doesn't ever run more than once. It would be a better test if it read e.g. a byte at a time into the buffer.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't get it - the reader doesn't fill the buffer in one shot, it only yields up to max_read_len
at a time. Is there something i'm missing?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So it reads up to max_read_len bytes, which in the tests would usually fit in the buffer perfectly, so it exits the read loop without trying to read more (reads all the data on one pass) so if there was an issue with the subsequent iterations of the read loop, as would be exercised in the real world, then you wouldn't see it with this test.
It would be a better test if this required at least two read calls to get all the data, then the caller would have to loop twice at least and exercise that code.
Speaking from long experience of having had a bug exactly like this that wasn't caught by the tests.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm going to approve the PR so this doesn't hold up your work, it's up to you if you want to improve on the test or not
Changelog
lz4_flex
tolz4-rs
. This moves us from using a pure-rust lz4 implementation to C bindings. I believe this is worthwhile becauselz4_flex
does not support LZ4 "high compression mode". The practical reason for doing so in this PR is thatlz4_flex
does not expose interfaces that make it easy to build an AsyncRead adapter for it, butlz4-rs
does.Docs
Description
Adds an async
RecordReader
implementation, for reading MCAP data asynchronously. This is an optional feature, namedtokio
. I chose this feature flag name and this module name because this functionality is tied heavily into the Tokio ecosystem. If at some point we rebuild this to be async-executor-agnostic, we can add that functionality under a new module and feature flag name.