Hook up std::net to wasi-libc on wasm32-wasip2 target

[nickrum]

One of the improvements of the wasm32-wasip2 target over wasm32-wasip1 is better support for networking. Right now, p2 is just re-using the std::net implementation from p1. This PR adds a new net module for p2 that makes use of net from sys_common and calls wasi-libc functions directly.

There are currently a few limitations:

• Duplicating a socket is not supported by WASIp2 (directly returns an error)
• Peeking is not yet implemented in wasi-libc (we could let wasi-libc handle this, but I opted to directly return an error instead)
• Vectored reads/writes are not supported by WASIp2 (the necessary functions are available in wasi-libc, but they call WASIp1 functions which do not support sockets, so I opted to directly return an error instead)
• Getting/setting TCP_NODELAY is faked in wasi-libc (uses the fake implementation instead of returning an error)
• Getting/setting SO_LINGER is not supported by WASIp2 (directly returns an error)
• Setting SO_REUSEADDR is faked in wasi-libc (since this is done from sys_common, the fake implementation is used instead of returning an error)
• Getting/setting IPV6_V6ONLY is not supported by WASIp2 and will always be set for IPv6 sockets (since this is done from sys_common, wasi-libc will return an error)
• UDP broadcast/multicast is not supported by WASIp2 (since this is configured from sys_common, wasi-libc will return appropriate errors)
• The MSG_NOSIGNAL send flag is a no-op because there are no signals in WASIp2 (since explicitly setting this flag would require a change to sys_common and the result would be exactly the same, I opted to not set it)

Do those decisions make sense?

While working on this PR, I noticed that there is a std::os::wasi::net::TcpListenerExt trait that adds a sock_accept() method to std::net::TcpListener. Now that WASIp2 supports standard accept, would it make sense to remove this?

cc @alexcrichton

[rustbot]

Thanks for the pull request, and welcome! The Rust team is excited to review your changes, and you should hear from @tgross35 (or someone else) some time within the next two weeks.

Please see the contribution instructions for more information. Namely, in order to ensure the minimum review times lag, PR authors and assigned reviewers should ensure that the review label (S-waiting-on-review and S-waiting-on-author) stays updated, invoking these commands when appropriate:

• @rustbot author: the review is finished, PR author should check the comments and take action accordingly
• @rustbot review: the author is ready for a review, this PR will be queued again in the reviewer's queue

[tgross35]

Thanks for the PR! The reasoning for these changes makes sense, but why are all the extern interfaces added here? That part of the changes should go into our libc https://github.com/rust-lang/libc/blob/72cb7aaf50bf70b5d360b1a64b0d52d3ed3fbf15/src/wasi.rs.

Pinging was[mi] expert @alexcrichton, could you double check the reasoning here?

[tgross35]

More specifically, if these are available on wasi-p2 but not p1 then the libc changes would mean:

1. Move wasi.rs to wasi/mod.rs
2. Create wasi/p2.rs that includes things only available on p2
3. If there is anything that is available on p1 but not p2, move it to wasi/p1.rs
4. In the new wasi/mod.rs, import everything from either p1 or p2 based on the target (like https://github.com/rust-lang/libc/blob/72cb7aaf50bf70b5d360b1a64b0d52d3ed3fbf15/src/unix/mod.rs#L1570-L1612)

[nickrum]

I was planning to work on bringing those extern interfaces to rust-lang/libc next by doing exactly that.
Since splitting libc between WASIp1 and WASIp2 and verifying which types/structs/functions are available where felt like a bigger effort, since there already is a netc module inside WASIp1's net and since there are other targets having their network-related extern interfaces directly within std, my thought was that it might be fine to add them here for the time being to get things started and replace them once the changes to libc are ready. But if you prefer to have them in libc first, that makes total sense as well.

[tgross35]

It would be better to update libc first so definitions only live in a single place, I would be concerned about things getting out of sync otherwise. Doing that shouldn't be much worse than adding them here - you can probably just do steps 1, 2, and 4 in my list above, skipping step 3 unless it turns out to be a problem.

The turnaround is pretty quick getting a new libc version and updating std.

[tgross35]

Also if you have a link to the headers please post them in the libc PR, sometimes it's tough to track down where definitions came from

[alexcrichton]

r? @alexcrichton

Thanks so much for working on this! I'll do a review tomorrow

[alexcrichton]

Thank you so much for working on this! I've left some comments below with a few thoughts here and there. Mostly I'm hoping we can delegate to libc to return errors where possible instead of explicitly returning errors here since that'll make it easier to support in the future by just changing wasi-libc. Otherwise most of your other changes also makes sense to me, but let me know if you have any more questions.

As for libc bindings, I agree it'd be great to get these bindings upstream into the libc crate itself. I can try to take a look at setting that up CI-wise, and/or please feel free to cc me on PRs.

[alexcrichton]

I've started a bit with wasi-libc by submitting rust-lang/libc#3869 to re-enable CI. Once that's ready I'll also submit a new target of wasm32-wasip2 and get that configured/ready too. After that it should be a simpler to add p2-specific constants and such.

[alexcrichton]

Oh I can also mention, have you been able to test this? For example are you able to run perhaps the std::net-related portions of the testsuite in the standard library? The p2 target isn't tested on CI so it's just for local testing and double-checking, but one day the p1 tests may migrate to p2 tests in CI.

[nickrum]

Thanks for the review and starting the work on libc. I'll have a look at moving over the brindings next.

Unfortunately, building the standard library test suit for the wasm32-wasip2 target fails for me, so I was only able to test this outside of the std lib. Is there a way to only build the std::net-related tests? Otherwise I can look into the failing tests as well.

[tgross35]

Thanks for the review and starting the work on libc. I'll have a look at moving over the brindings next.

Unfortunately, building the standard library test suit for the wasm32-wasip2 target fails for me, so I was only able to test this outside of the std lib. Is there a way to only build the std::net-related tests? Otherwise I can look into the failing tests as well.

You should be able to filter them with something like ./x t library/std -- net. But do they fail to build or fail to run? I'm not sure why there would be a build failure, but if you post some logs we might be able to help.

[alexcrichton]

Ok all looks good to me, thanks again! Before landing though I think it'd be good to run this through its paces with the test suite of the standard library (or another library if you're interested in another library perhaps?). If the standard library's tests don't build at all is it possible to temporarily comment out the pieces that don't build? This won't be run on CI anyway so it's more of a one-time check to make sure things are working.

[tgross35]

Are the bindings able to go into libc? I'm working on rolling a release today, happy to hold off on that.

[nickrum]

I'm working on the bindings right now. If you could hold off on the release, that'd be great.

[tgross35]

No problem! They're pretty quick to do whenever

[nickrum]

But do they fail to build or fail to run? I'm not sure why there would be a build failure, but if you post some logs we might be able to help.

They fail to build. Here are the errors I'm getting:

error: cannot find macro `error` in this scope
    --> std/src/fs/tests.rs:1147:5
     |
1147 |     error!(blank.create(true).open(&tmpdir.join("f")), invalid_options);
     |     ^^^^^

error[E0433]: failed to resolve: could not find `unix` in `os`
   --> std/src/process/tests.rs:574:16
    |
574 |     crate::os::unix::process::CommandExt::arg0(&mut command, "exciting-name");
    |                ^^^^ could not find `unix` in `os`
    |
note: found an item that was configured out
   --> std/src/os/mod.rs:27:9
    |
27  | pub mod unix {}
    |         ^^^^
note: the item is gated here
   --> std/src/os/mod.rs:19:1
    |
19  | / #[cfg(all(
20  | |     doc,
21  | |     any(
22  | |         all(target_arch = "wasm32", not(target_os = "wasi")),
23  | |         all(target_vendor = "fortanix", target_env = "sgx")
24  | |     )
25  | | ))]
    | |___^
note: found an item that was configured out
   --> std/src/os/mod.rs:65:9
    |
65  | pub mod unix;
    |         ^^^^
note: the item is gated here
   --> std/src/os/mod.rs:64:1
    |
64  | #[cfg(all(not(target_os = "hermit"), any(unix, doc)))]
    | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

error[E0405]: cannot find trait `AsRawFd` in this scope
   --> std/src/net/tcp/tests.rs:678:32
    |
678 |     fn render_inner(addr: &dyn AsRawFd) -> impl fmt::Debug {
    |                                ^^^^^^^ not found in this scope
    |
help: consider importing one of these traits
    |
1   + use crate::os::fd::AsRawFd;
    |
1   + use crate::realstd::os::fd::AsRawFd;
    |

error[E0425]: cannot find function `symlink_file` in this scope
  --> std/src/fs/tests.rs:76:11
   |
76 |     match symlink_file(r"nonexisting_target", link) {
   |           ^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `junction_point` in this scope
   --> std/src/fs/tests.rs:595:12
    |
595 |     check!(junction_point(&d2, &dt.join("d2")));
    |            ^^^^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `symlink_dir` in this scope
   --> std/src/fs/tests.rs:702:17
    |
702 |         let _ = symlink_dir(attack_dest_dir, &victim_del_path);
    |                 ^^^^^^^^^^^ not found in this scope

error[E0425]: cannot find function `env_cmd` in this scope
   --> std/src/process/tests.rs:276:19
    |
276 |     let mut cmd = env_cmd();
    |                   ^^^^^^^ not found in this scope

It looks like the standard library test suite expects the target_family to be either unix or windows, which is not true for the wasm targets.

Before landing though I think it'd be good to run this through its paces with the test suite of the standard library (or another library if you're interested in another library perhaps?). If the standard library's tests don't build at all is it possible to temporarily comment out the pieces that don't build?

I agree, that would be good! I can't think of a library to thoroughly test this with off the top of my head, but I'll look into getting the test suite to build 👍

[nickrum]

Should be good to go now 👍

[alexcrichton]

@bors: r+

[bors]

📌 Commit 87f17f3 has been approved by alexcrichton

It is now in the queue for this repository.

[bors]

⌛ Testing commit 87f17f3 with merge e8c85f3...

[rust-log-analyzer]

The job x86_64-msvc-ext failed! Check out the build log: (web) (plain)

Click to see the possible cause of the failure (guessed by this bot)

[bors]

💔 Test failed - checks-actions

[tgross35]

@bors retry MSVC spurious issue

[bors]

⌛ Testing commit 87f17f3 with merge c87004a...