Add AsyncWriteExt::write_all_vectored utility #1741

Thomasdezeeuw · 2019-07-16T16:26:52Z

Since vectored I/O has been stabilised I was missing this. Two things to note;

this takes &mut [IoSlice], where the call in io::Read takes &[IoSlice]. This is needed to advance the buffer inside the IoSlice.
this uses some unsafe code, I couldn't get around without it.

TODO:

Docs.

futures-util/src/io/write_all.rs

Thomasdezeeuw · 2019-08-06T09:37:23Z

@Nemo157 I've created {IoSlice, IoSliceMut}::advance which does the most heavy lifting of this change in rust-lang/rust#62987. However it's an unstable feature, now that Future is also available on stable how are nightly features supposed to be handled?

Thomasdezeeuw · 2019-09-29T09:09:17Z

@Nemo157 sorry about the delay. I've rebased this on master, updated the code to use the newly created io_slice_advance Rust feature and added documentation.

Thomasdezeeuw · 2019-09-29T09:10:57Z

futures-util/src/io/mod.rs

+    ///
+    /// [vectored writes]: std::io::Write::write_vectored
+    ///
+    /// # Notes


Please carefully review this section, I don't if it clearly states what I mean.

If the future successfully completes, bufs is an empty slice, right?

On failure bufs may have been modified, but is not guaranteed to match with what has actually been written out (because I believe the underlying IO doesn't guarantee that it either writes some data or returns an error, it could have written some of the data and then returned an error). That's pretty much equivalent to what write_all guarantees, if it returns an error some part of the data will have been written with no way to know how much.

I think this should try to avoid the "undefined" term since that has such strong memory-safety connotations, and the failure cases are relatively well-defined.

If the future successfully completes, bufs is an empty slice, right?

That is correct, but in the future we could optimise to just return ready if the entire (remaining) buffer is written, not modifying bufs.

On failure bufs may have been modified, but is not guaranteed to match with what has actually been written out (because I believe the underlying IO doesn't guarantee that it either writes some data or returns an error, it could have written some of the data and then returned an error). That's pretty much equivalent to what write_all guarantees, if it returns an error some part of the data will have been written with no way to know how much.

Do you want this to be reflected in the documentation somehow, or leave it as is?

I think this should try to avoid the "undefined" term since that has such strong memory-safety connotations, and the failure cases are relatively well-defined.

Hence I added the it was not about memory safety, but I do understand the concern. Should I replace "undefined" with "unknown" (or something else) or you want to provide the guarantee that bufs will be empty after it successfully returns (also see my first remark in this comment)?

futures-util/src/io/write_all.rs

Thomasdezeeuw · 2019-10-04T15:39:59Z

Let me know if you want me to rebase this on the master branch.

Thomasdezeeuw · 2019-11-22T14:43:26Z

@Nemo157 this pr has fallen out of date, but can I get an OK/not OK before spending more time on this?

Thomasdezeeuw · 2020-01-26T13:10:21Z

@Nemo157 @taiki-e ping :)

Thomasdezeeuw · 2020-03-21T12:43:56Z

@cramertj do you have time to review this?

cramertj · 2020-03-26T15:31:17Z

Sure, sorry for the delay! Thanks for the ping.

futures-util/Cargo.toml

cramertj · 2020-03-26T15:38:31Z

futures-util/src/io/mod.rs

+    ///
+    /// Different to `io::Write::write_vectored` this takes a *mutable*
+    /// reference to a slice of `IoSlice`s, not a non-mutable reference, because
+    /// we need to modify the slice to keep track of the bytes already written.


Who needs to? The future can store that state internally. Is this for the user to know? IMO we should try to match the API of write_all as closely as possible. Actually, for that matter, what we should be trying to match would be the yet-to-exist std::io::Write::write_all_vectored. If there isn't yet an issue for creating that API, would you mind creating one and linking it here? I'd really like to keep these extension traits from diverging from the API in the standard library, but I'm fine landing it here if it's already under a feature gate, so long as we make sure to tag that feature gate as _unstable / have it require an unstable feature flag in order to compile. Does that make sense?

The problem with writing all buffers is that is we need to deal with partially written IoSlices. For which I wrote IoSlice::advance, which internally updates the slices, but needs mutable access. Basically WriteAllVectored needs ownership of bufs, to keep track of what bytes are already written. Also see the next section in the documentation.

In practice I don't think this will be a problem as IoSlice slices are often created just before writing, e.g. see the example. But if you know of a better solution I'll love to hear it.

Created rust-lang/rust#70436.

cramertj · 2020-03-26T15:39:36Z

futures-util/src/io/write_all_vectored.rs

@@ -0,0 +1,200 @@
+use futures_core::future::Future;


Implementation and tests here look good to me, thanks!

cramertj · 2020-03-26T15:40:00Z

futures-util/src/io/mod.rs

+    /// # Examples
+    ///
+    /// ```
+    /// # futures::executor::block_on(async {


Thanks for including this example!

Thomasdezeeuw · 2020-04-07T13:26:57Z

@cramertj I've rebased this on master and changed the docs to match io::Write::write_all_vectored in the std lib (as merged in #70612).

Thomasdezeeuw · 2020-04-08T13:18:43Z

An unrelated tests failed select_on_non_unpin_size on the CI. @cramertj should I just increase the expected size in test? Or is this being tracked elsewhere?

Nemo157 · 2020-04-08T13:20:22Z

Fixed in #2119, though that's still failing because of a nightly bug in the rustdoc check.

This adds a new feature to future-util, named write_all_vectored, to enable the utility since it requires the unstable io_slice_advance Rust feature. This matches the same API found in io::Write::write_all_vectored in the std lib.

Thomasdezeeuw · 2020-04-15T18:04:19Z

Now that #2119 is merged I've rebased this.

Thomasdezeeuw · 2020-04-18T12:36:12Z

Friendly ping @cramertj, I don't want to have to rebase this again :)

cramertj · 2020-04-22T21:16:20Z

Thanks for the ping, and for the changes! Sorry for the delay on my end.

Nemo157 reviewed Jul 16, 2019

View reviewed changes

futures-util/src/io/write_all.rs Outdated Show resolved Hide resolved

Thomasdezeeuw mentioned this pull request Jul 16, 2019

Tracking issue for io_slice_advance rust-lang/rust#62726

Closed

3 tasks

Thomasdezeeuw mentioned this pull request Jul 26, 2019

Add {IoSlice, IoSliceMut}::advance rust-lang/rust#62987

Merged

Thomasdezeeuw commented Sep 29, 2019

View reviewed changes

taiki-e reviewed Sep 29, 2019

View reviewed changes

futures-util/src/io/write_all.rs Outdated Show resolved Hide resolved

Thomasdezeeuw requested a review from Nemo157 January 6, 2020 14:49

cramertj reviewed Mar 26, 2020

View reviewed changes

futures-util/Cargo.toml Outdated Show resolved Hide resolved

cramertj reviewed Mar 26, 2020

View reviewed changes

futures-util/src/io/mod.rs

/// # Examples

///

/// ```

/// # futures::executor::block_on(async {

Copy link

Member

cramertj Mar 26, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for including this example!

Thomasdezeeuw mentioned this pull request Mar 26, 2020

Tracking issue for Write::write_all_vectored rust-lang/rust#70436

Open

3 tasks

Add AsyncWriteExt::write_all_vectored utility

564522c

This adds a new feature to future-util, named write_all_vectored, to enable the utility since it requires the unstable io_slice_advance Rust feature. This matches the same API found in io::Write::write_all_vectored in the std lib.

cramertj merged commit d7d8216 into rust-lang:master Apr 22, 2020

Thomasdezeeuw deleted the write_all_vectored branch April 23, 2020 07:57

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AsyncWriteExt::write_all_vectored utility #1741

Add AsyncWriteExt::write_all_vectored utility #1741

Thomasdezeeuw commented Jul 16, 2019 •

edited

Loading

Thomasdezeeuw commented Aug 6, 2019

Thomasdezeeuw commented Sep 29, 2019

Thomasdezeeuw Sep 29, 2019

Nemo157 Sep 30, 2019

Thomasdezeeuw Oct 4, 2019

Thomasdezeeuw commented Oct 4, 2019

Thomasdezeeuw commented Nov 22, 2019

Thomasdezeeuw commented Jan 26, 2020

Thomasdezeeuw commented Mar 21, 2020

cramertj commented Mar 26, 2020

cramertj Mar 26, 2020

Thomasdezeeuw Mar 26, 2020

Thomasdezeeuw Mar 26, 2020

cramertj Mar 26, 2020

cramertj Mar 26, 2020

Thomasdezeeuw commented Apr 7, 2020

Thomasdezeeuw commented Apr 8, 2020

Nemo157 commented Apr 8, 2020

Thomasdezeeuw commented Apr 15, 2020

Thomasdezeeuw commented Apr 18, 2020

cramertj commented Apr 22, 2020

Add AsyncWriteExt::write_all_vectored utility #1741

Add AsyncWriteExt::write_all_vectored utility #1741

Conversation

Thomasdezeeuw commented Jul 16, 2019 • edited Loading

Thomasdezeeuw commented Aug 6, 2019

Thomasdezeeuw commented Sep 29, 2019

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thomasdezeeuw commented Oct 4, 2019

Thomasdezeeuw commented Nov 22, 2019

Thomasdezeeuw commented Jan 26, 2020

Thomasdezeeuw commented Mar 21, 2020

cramertj commented Mar 26, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Thomasdezeeuw commented Apr 7, 2020

Thomasdezeeuw commented Apr 8, 2020

Nemo157 commented Apr 8, 2020

Thomasdezeeuw commented Apr 15, 2020

Thomasdezeeuw commented Apr 18, 2020

cramertj commented Apr 22, 2020

Thomasdezeeuw commented Jul 16, 2019 •

edited

Loading