Consider supporting advanced networking APIs

[Ralith]

Most Unixes today implement the RFC2292 Advanced Sockets API to allow ancillary data to be sent/received alongside datagrams using different control messages. Examples include explicit congestion notification flags and hardware packet timestamps on UDP/IP sockets, and file descriptors on Unix sockets.

Is tokio interested in providing direct support for these use cases? The API is idiosyncratic, requiring large amounts of aggressively unsafe code (see e.g. the draft implementation of ECN handling in quinn), so a safe binding would be valuable. On the other hand, platform support for specific control messages varies, and libc is currently missing bindings for the macros involved in encoding/decoding control messages on all platforms but Linux.

Such an API would allow any write operation on a datagram socket to be supplied with a stack-allocated buffer of control messages constructed in a memory- and type-safe manner, and any read operation supplied with a stack-allocated buffer to which it can write arbitrary control messages, which can be safely iterated over on success. Accomplishing this with minimal overhead wouldn't be trivial. For example, we could define UdpSocket::poll_send_to_advanced which takes a slice of enum ControlMessage { ... } values, but this would require conversion internally, perhaps using a SmallVec to avoid the heap most of the time. Returning a set of control messages has similar challenges.

[carllerche]

I definitely want to support a way to use these advanced APIs. The question is going to be what that way is :).

I'm not sure what the latest and greatest way to provide platform specific APIs is. Is an extension trait still the strategy?

[Ralith]

@aturon's post on portability seems relevant. He advocates for just cfging things out and letting a (not yet implemented?) lint handle potential surprises.

[carllerche]

@Ralith sounds good to me! Looking forward to any PRs 👍

[pusateri]

In nix-rust/nix#990, I recently added IP_PKTINFO & IP6_PKTINFO ancillary data cmsg support with recvmsg() and sendmsg(). Then in nix-rust/nix#1002, I'm adding IP_RECVIF and IP_RECVDSTADDR (IP_SENDSRCADDR). I would like to see this get migrated to socket2 and eventually into std.

Other ancillary data can be added there easily. If it was easy to swap in a different underlying socket implementation, this could be a pathway.

[Ralith]

Nix's approach seems to be unsound (per nix-rust/nix#999), and the CmsgSpace trick, while clever, is pretty ugly. I'm not sure what a better API would look like, though. The ABIs here are practically a diverse set of ever so slightly incompatible binary network protocols. Maybe something flyweight-y that doesn't expose references?

[pusateri]

I used nix because it was easy to add additional control messages to but I am flexible and ultimately, just need a solution for tokio (preferably with async/await). I tested nix on lots of operating systems and architectures including actual hardware amd64, armv7, powerpc, & mips64 using *BSD and Linux variants. So while there may be a theoretical alignment issue with certain control messages on certain architectures, I never was able to hit it. But I don’t mind fixing it if it can be reproduced.

[asomers]

You're damn right that Nix's cmsg code is ugly! But have you ever looked at cmsg code written in C? It's ugly too. The entire sendmsg/recvmsg API is very cumbersome. If you have a better idea about how to Rustify it, please raise an issue in Nix!

[Ralith]

But have you ever looked at cmsg code written in C?

Yes, I'm currently using C-style code in Quinn. It's not great either.

If you have a better idea about how to Rustify it, please raise an issue in Nix!

It's a hard problem, for sure. Time permitting, I plan to experiment with a flyweight-y by-value approach, sort of like capnp (de)serializing if you've ever used that.

[carllerche]

I'm fine adding more APIs, however I'm going to close this issue as there is no immediate bandwidth to implement this.

If someone wishes to champion this, we can re-open it or they can discuss in Gitter and submit a PR.