Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature/generic stream #414

Merged
merged 13 commits into from
Jul 27, 2021
Merged

Feature/generic stream #414

merged 13 commits into from
Jul 27, 2021

Conversation

ryan-summers
Copy link
Member

@ryan-summers ryan-summers commented Jul 23, 2021

This PR fixes #385 by improving the semantics by which streams are generated. Specifically, serialization format is now application-defined and data is only copied once.

TODO:

  • Add documentation for streaming architecture
  • Add format specifiers
  • Clean up reception script
  • Update lockin serialization

@ryan-summers ryan-summers marked this pull request as ready for review July 23, 2021 13:30
Copy link
Member

@jordens jordens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work. Let me know what you think about the two bigger things (MaybeUninit and the genericism/light-weight queue items).

src/bin/dual-iir.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
format,
offset: 7,
sequence_number,
buffer: unsafe { buffer.assume_init() },
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder whether it would be better to follow through with the notion that this is uninitialized data. I would not allow the closure (to add()) to read stale/previous data in the buffers.
The closure is the recipient of the "out-pointer" in https://doc.rust-lang.org/std/mem/union.MaybeUninit.html#out-pointers. And the closure will be doing unsafe stuff in general.

Copy link
Member Author

@ryan-summers ryan-summers Jul 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is not a MaybeUninit structure, but rather a heapless::pool::Box<T, Uninit>. The only way to get a Box<T, Init> is to call Box::init(t: T), where a copy from t -> inner data is performed. If we actually did that, we'd have to enforce a 1KB copy (of zero'd data) into the buffer whenever pulling a buffer out of the pool (e.g. in the high prio task). I wanted to avoid that overhead, since we never read the buffer anyways.


Edit: While the above is true, we could still follow-through with using a MaybeUninit<[u8; FRAME_SIZE]> type in the pool. Then we can initialize it with MaybeUninit::uninit() and use the concept of out-pointers you referenced. I don't know if this would work, but I'll definitely look into it.

Copy link
Member

@jordens jordens Jul 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you sure? The only example I'm aware of is this: https://github.com/stm32-rs/stm32l4xx-hal/blob/master/examples/rtic_frame_serial_dma.rs#L92
DMAFrame::new() has a MaybeUninit::uninit() member. The dma_buf.init() (ptr::write(...)) would copy uninitialized data? I suspect that this is not the case and this is actually the pattern we also want to follow here.

Copy link
Member Author

@ryan-summers ryan-summers Jul 24, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

core::ptr::write() is just a call to memcpy underneath. Thus, even if we have a MaybeUninit::uninit() object, we're still copying the entire MaybeUninit structure, which contains the underlying data buffer (it's a union type). When memcpy-ing a union, you have to memcpy the whole size of the union, which in this case is the 1024 bytes for the buffer.

Check out godbolt for the compiler output: https://godbolt.org/z/n8r6EexK6 - both do a memcpy of 1024 bytes.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. Right.
Would be interesting to discuss with @korken89 what the intended usage pattern is here without assume_init.

src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
src/net/data_stream.rs Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
scripts/stream_throughput.py Outdated Show resolved Hide resolved
scripts/stream_throughput.py Outdated Show resolved Hide resolved
src/net/data_stream.rs Outdated Show resolved Hide resolved
scripts/stream_throughput.py Outdated Show resolved Hide resolved
src/net/mod.rs Outdated Show resolved Hide resolved
Copy link
Member

@jordens jordens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice

@jordens
Copy link
Member

jordens commented Jul 27, 2021

bors r+

@bors
Copy link
Contributor

bors bot commented Jul 27, 2021

Build succeeded:

@bors bors bot merged commit 62d1a16 into master Jul 27, 2021
@bors bors bot deleted the feature/generic-stream branch July 27, 2021 11:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

streaming: better buffering
2 participants